dqa.connectors.importer.PickledDataFrameImporter

class dqa.connectors.importer.PickledDataFrameImporter(import_index: str | None = 'Index', **kwargs)

Importer for a pandas data frame that has been stored using the pickle library.

Parameters:

import_index (str, optional, default='Index') – Name to load the index of the pickled dataframe as. If None, then the index is not loaded at all.

Methods

filter_filenames(files, start, end, step)	For a list of available filenames, take a subset specified by start, end, step.
filter_filenames_regex(files)	Filter filenames by the regular expression specified in the constructor.
finish()	Can perform actions that are required to clean up after the task has finished, e.g. close network connections etc.
get_import_parameters(folder_index)	Returns filename, folder_name, start, end, step for the folder with index folder_index.
get_start_end_step(start, end, step, ...)	Adjusts start, end, step if the num_files parameter is set.
import_file(folder, filename, parameters, ...)	Imports a single file.
import_files(folder, filenames, parameters)	Imports all files given by a list of filenames.
import_folder(folder[, start, end, step, ...])	Imports all files from a folder.
list_folder(folder)	Lists all files in a folder. By default, this lists the files on the file system that are relevant (by checking file ending or regex pattern). For other purposes this could also be used, for example, to list the contents of a database. :param folder: The name of the folder to be listed. :return: A tuple containing: - A list of all relevant filenames. - A list containing a dictionary of paramters for each filename. By default, these are obtained by parsing the named groups in the filename regex.

import_file2
in_out_default
input_output_dataset
input_output_machine
input_output_mode
input_output_name
list_batch_ids
list_total
log
modify_data_row
modify_dataset
modify_dataset_dict
modify_machine
modify_measurement
set_logging_level
transfer_metadata