file_io.CSVReader

Attributes

`_INVALID_COL_CHARS`
`VALID_FILE_FORMATS`
`PRE_HEADER_COMMENT_AND_EXCLUDE_STRINGS`
`logger`

Classes

CSVDataReader

A class to read in object data files stored as CSV or whitespace

Module Contents

_INVALID_COL_CHARS = '!#$%&‘()*+, ./:;<=>?@[\\]^{|}~'[source]

VALID_FILE_FORMATS[source]

PRE_HEADER_COMMENT_AND_EXCLUDE_STRINGS = ('#', '!')[source]

logger[source]

class CSVDataReader(filename, sep='csv', **kwargs)[source]

Bases: layup.utilities.file_io.ObjectDataReader.ObjectDataReader

A class to read in object data files stored as CSV or whitespace separated values.

Note that we require the header line to be the first line of the file

filename[source]

sep = 'csv'[source]

data_separator = ','[source]

num_pre_header_lines = 0[source]

header_row_index = 0[source]

obj_id_table = None[source]

obj_id_counts[source]

get_reader_info()[source]

Return a string identifying the current reader name and input information (for logging and output).

Returns:: name – The reader information.
Return type:: string

get_row_count()[source]

Return the total number of rows in the [C|P|W]SV file.

Returns:: Total rows in the first key of the input [C|P|W]SV file.
Return type:: int

_validate_header_line()[source]: Read and validate the header line (first line of the file)

_check_header_line(header_line)[source]

Check that a given header line is valid and exit if it is invalid.

Parameters:: header_line (str) – The proposed header line.

_get_fixed_dtypes()[source]

Get a dictionary of the fixed dtypes for the columns in the CSV file.

Returns:: fixed_dtypes – A dictionary of the fixed dtypes for the columns in the CSV file. The keys are the column names and the values are assigned dtype.
Return type:: dict

_read_rows_internal(block_start=0, block_size=None, **kwargs)[source]

Reads in a set number of rows from the input.

Parameters:

block_start (integer, optional) – The 0-indexed row number from which to start reading the data. For example in a CSV file block_start=2 would skip the first two lines after the header and return data starting on row=2. Default =0
block_size (integer, optional, default=None) – The number of rows to read in. Use block_size=None to read in all available data. default =None
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_build_id_map()[source]: Builds a table of just the object IDs

_read_objects_internal(obj_ids, **kwargs)[source]

Read in a chunk of data for given object IDs.

Parameters:

obj_ids (list) – A list of object IDs to use.
**kwargs (dictionary, optional) – Extra arguments

Returns:

res – The data read in from the file.

Return type:

numpy structured array

_process_and_validate_input_table(input_table, **kwargs)[source]

Perform any input-specific processing and validation on the input table. Modifies the input table in place.

Notes

The base implementation includes filtering that is common to most input types. Subclasses should call super.process_and_validate() to ensure that the ancestor’s validation is also applied.

Parameters:

input_table (numpy structured array) – A loaded table.
**kwargs (dictionary, optional) – Extra arguments

Returns:

input_table – Returns the input table modified in-place.

Return type:

numpy structured array