file_io

Submodules

Classes

`ADESXMLDataReader`	Read MPC ADES observation data stored as XML into a structured array.
`CSVDataReader`	A class to read in object data files stored as CSV or whitespace
`HDF5DataReader`	A class to read in object data files stored as HDF5 files.
`Obs80DataReader`	A class to read in object data files stored in the MPC's obs80

Package Contents

class ADESXMLDataReader(filename, sep=None, **kwargs)[source]

Bases: layup.utilities.file_io.ObjectDataReader.ObjectDataReader

Read MPC ADES observation data stored as XML into a structured array.

filename

_records = None

_id_map_built = False

obj_id_counts

get_reader_info()[source]: Return a string identifying the reader and its input file.

_parse()[source]: Parse (once) and cache the observation records.

get_row_count()[source]: Return the total number of observation records in the file.

_get_fixed_dtypes()[source]

Columns forced to str (identifiers, not numbers).

Mirrors CSVDataReader._get_fixed_dtypes() so the primary-id and station columns are never coerced to numeric (e.g. station "024" must stay a string, not become 24).

_records_to_array(records)[source]: Convert a list of record dicts into a numpy structured array.

_read_rows_internal(block_start=0, block_size=None, **kwargs)[source]: Read a contiguous block of observation rows.

_build_id_map()[source]: Populate obj_id_counts (rows per object id) for create_chunks.

_read_objects_internal(obj_ids, **kwargs)[source]: Read all rows belonging to the given object ids.

_process_and_validate_input_table(input_table, **kwargs)[source]: Run the shared validation and strip whitespace from column names.

class CSVDataReader(filename, sep='csv', **kwargs)[source]

Bases: layup.utilities.file_io.ObjectDataReader.ObjectDataReader

A class to read in object data files stored as CSV or whitespace separated values.

Note that we require the header line to be the first line of the file

filename

sep = 'csv'

data_separator = ','

num_pre_header_lines = 0

header_row_index = 0

obj_id_table = None

obj_id_counts

get_reader_info()[source]

Return a string identifying the current reader name and input information (for logging and output).

Returns:: name – The reader information.
Return type:: string

get_row_count()[source]

Return the total number of rows in the [C|P|W]SV file.

Returns:: Total rows in the first key of the input [C|P|W]SV file.
Return type:: int

_validate_header_line()[source]: Read and validate the header line (first line of the file)

_check_header_line(header_line)[source]

Check that a given header line is valid and exit if it is invalid.

Parameters:: header_line (str) – The proposed header line.

_get_fixed_dtypes()[source]

Get a dictionary of the fixed dtypes for the columns in the CSV file.

Returns:: fixed_dtypes – A dictionary of the fixed dtypes for the columns in the CSV file. The keys are the column names and the values are assigned dtype.
Return type:: dict

_read_rows_internal(block_start=0, block_size=None, **kwargs)[source]

Reads in a set number of rows from the input.

Parameters:

block_start (integer, optional) – The 0-indexed row number from which to start reading the data. For example in a CSV file block_start=2 would skip the first two lines after the header and return data starting on row=2. Default =0
block_size (integer, optional, default=None) – The number of rows to read in. Use block_size=None to read in all available data. default =None
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_build_id_map()[source]: Builds a table of just the object IDs

_read_objects_internal(obj_ids, **kwargs)[source]

Read in a chunk of data for given object IDs.

Parameters:

obj_ids (list) – A list of object IDs to use.
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_process_and_validate_input_table(input_table, **kwargs)[source]

Perform any input-specific processing and validation on the input table. Modifies the input table in place.

Notes

The base implementation includes filtering that is common to most input types. Subclasses should call super.process_and_validate() to ensure that the ancestor’s validation is also applied.

Parameters:

input_table (numpy structured array) – A loaded table.
**kwargs (dictionary, optional) – Extra arguments

Returns:

input_table – Returns the input table modified in-place.

Return type:

numpy structured array

class HDF5DataReader(filename, **kwargs)[source]

Bases: layup.utilities.file_io.ObjectDataReader.ObjectDataReader

A class to read in object data files stored as HDF5 files.

filename

obj_id_table = None

obj_id_counts

get_reader_info()[source]

Return a string identifying the current reader name and input information (for logging and output).

Returns:: name – The reader information.
Return type:: string

get_row_count()[source]

Return the total number of rows in the first key of the input HDF5 file.

Returns:: Total rows in the first key of the input HDF5 file.
Return type:: int

_read_rows_internal(block_start=0, block_size=None, **kwargs)[source]

Reads in a set number of rows from the input.

Parameters:

block_start (integer, optional) – The 0-indexed row number from which to start reading the data. For example in a CSV file block_start=2 would skip the first two lines after the header and return data starting on row=2. Default=0
block_size (integer, optional) – the number of rows to read in. Use block_size=None to read in all available data. Default = None
**kwargs (dictionary, optional) – Extra arguments

Returns:

res_df – Dataframe of the object data.

Return type:

pandas dataframe

_build_id_map()[source]: Builds a table of just the object IDs

_read_objects_internal(obj_ids, **kwargs)[source]

Read in a chunk of data for given object IDs.

Parameters:

obj_ids (list) – A list of object IDs to use.
**kwargs (dictionary, optional) – Extra arguments

Returns:

res_df – The dataframe for the object data.

Return type:

Pandas dataframe

_process_and_validate_input_table(input_table, **kwargs)[source]

Perform any input-specific processing and validation on the input table. Modifies the input dataframe in place.

Notes

The base implementation includes filtering that is common to most input types. Subclasses should call super.process_and_validate() to ensure that the ancestor’s validation is also applied.

Parameters:

input_table (pandas dataframe) – A loaded table.
**kwargs (dictionary, optional) – Extra arguments

Returns:

input_table – Returns the input dataframe modified in-place.

Return type:

pandas dataframe

class Obs80DataReader(filename, **kwargs)[source]

Bases: layup.utilities.file_io.ObjectDataReader.ObjectDataReader

A class to read in object data files stored in the MPC’s obs80 format.

Note that we will ignore the header lines that might accompany the file.

filename

output_dtype

col_names

obj_id_table = None

obj_id_counts

_is_header_row(line)[source]

Check if the line is a header row.

Parameters:: line (str) – The line to check.
Returns:: True if the line is a header row, False otherwise.
Return type:: bool

get_reader_info()[source]

Return a string identifying the current reader name and input information (for logging and output).

Returns:: name – The reader information.
Return type:: string

_iter_records(f)[source]

Yield one (main_line, second_line) tuple per logical obs80 record.

This is the single source of truth for how the raw lines of the file group into records; every read path (get_row_count, read_rows, read_objects) walks it so their record counts and ordering always agree. It pairs each two-line record (S/R/V first line + its lower-case s/r/v continuation line) and, crucially, refuses to emit a malformed observation:

a deleted/replaced observation (note 2 code X / x) is skipped;
an orphan continuation line (one with no matching preceding first line) is skipped – it carries no astrometry of its own and would otherwise be emitted as a positionless observation whose position columns are misread as RA/Dec;
a first line whose continuation is missing is dropped rather than paired with the next unrelated line.

main_line is the astrometry line; second_line is the observer-position line, or None for a single-line observation.

get_row_count()[source]

Return the total number of rows in the file.

Note that the obs 80 format allows for two-line rows, so the number of lines used to store the data is not the same as the number of rows.

Returns:: Total rows in the file.
Return type:: int

_read_rows_internal(block_start=0, block_size=None, **kwargs)[source]

Reads in a set number of rows from the input.

Parameters:

block_start (integer, optional) – The 0-indexed row number from which to start reading the data. For example in a CSV file block_start=2 would skip the first two lines after the header and return data starting on row=2. Default =0
block_size (integer, optional, default=None) – The number of rows to read in. Use block_size=None to read in all available data. default =None
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_build_id_map()[source]: Builds a table of just the object IDs

_read_objects_internal(obj_ids, **kwargs)[source]

Read in a chunk of data for given object IDs.

Parameters:

obj_ids (list) – A list of object IDs to use.
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_process_and_validate_input_table(input_table, **kwargs)[source]

Perform any input-specific processing and validation on the input table. Modifies the input table in place.

Notes

The base implementation includes filtering that is common to most input types. Subclasses should call super.process_and_validate() to ensure that the ancestor’s validation is also applied.

Parameters:

input_table (numpy structured array) – A loaded table.
**kwargs (dictionary, optional) – Extra arguments

Returns:

input_table – Returns the input table modified in-place.

Return type:

numpy structured array

get_obs80_id(line)[source]

Get the object ID from the Obs80 line. Note that we have already confirmed that self.primary_id_column_name is in self.col_names. :param line: The line of obs80 data to extract the object ID from. :type line: str

Returns:: The object ID extracted from the line.
Return type:: str

convert_obs80(line, second_line=None)[source]

Converts a row of obs80 data to a tuple of values. The second line is optional and may contain the observatory position.

Parameters:

line (str) – The line of obs80 data to convert.
second_line (str, optional) – The optional second line of obs80 data to convert. Default is None.

Returns:

A tuple of values containing the object ID, ISO time, RA in degrees, Dec in degrees, magnitude, filter, observatory code, catalog, program, and observatory position (x, y, z).

Return type:

tuple