Internal data format
====================

.. include:: <isonum.txt>

The Knowlestry Application Framework works with an internal data representation format, which is implemented in the module
:mod:`dqa.data_representation.data_representation`. The main purpose of this data format is a standardized interface to
pass data between processing units ("Tasks", implemented in the :class:`dqa.tasks.tasks.Task` class).

The data structure follows a hierarchy with multiple levels. For each of these levels, the ``Task`` class provides a
corresponding method ``modify_*``. By overriding this method, one can implement a task by modifying the data for each
unit on the particular level separately.

The following table summarizes the hierarchy of the data format along with the corresponding methods in the task class.

======================================================================================== ============================================
Data structure level                                                                      Method in :class:`dqa.tasks.tasks.Task`
======================================================================================== ============================================
                                                                                          ``Task``
 ``dataset_dict: Dict[str, Dataset]``                                                     ``modify_dataset_dict(datasets)``
 |nbsp| |nbsp| |darr| |nbsp| ``str key``                                                  |nbsp| |nbsp| |darr|
 ``dataset: Dataset``                                                                     ``modify_dataset(dataset)``
 |nbsp| |nbsp| |darr| |nbsp| ``str key`` (in ``dataset.machines``)                        |nbsp| |nbsp| |darr|
 ``machine: Machine``                                                                     ``modify_machine(machine)``
 |nbsp| |nbsp| |darr| |nbsp| ``int index`` (in ``machine.measurements``)                  |nbsp| |nbsp| |darr|
 ``measurement: Dict[str, Dict[str, Any]]``                                               ``modify_measurement(measurement)``
 |nbsp| |nbsp| |darr| |nbsp| |nbsp| |darr| |nbsp| ``'metadata'`` key                      |nbsp| |nbsp| |darr|
 |nbsp| |nbsp| |darr| |nbsp| ``metadata: Dict[str, Any]``                                 |nbsp| |nbsp| |darr|
 |nbsp| |nbsp| |darr| |nbsp| ``'data'`` key                                               |nbsp| |nbsp| |darr|
 ``data: Dict[str, np.ndarray]``                                                          |nbsp| |nbsp| |darr|
 |nbsp| |nbsp| |darr| |nbsp| ``str key``                                                  |nbsp| |nbsp| |darr|
 ``data_row: np.ndarray``                                                                 ``modify_data_row(data_row)``
======================================================================================== ============================================


The total data processed by the Knowlestry Application Framework is contained in a dictionary of :class:`dqa.data_representation.data_representation.Dataset`
objects whose keys are strings. Each such Dataset contains a dictionary ``machines`` mapping string identifiers
to :class:`dqa.data_representation.data_representation.Machine` objects. 

The ``Dataset`` and ``Machine`` class both implement the ``[]`` operator, allowing to access the includes machines or
measurements, respectively. For example,

.. code-block:: python
    :caption: Access operator example

    dataset_dict['Data']['M0'][0]['data']['my_data_row']

accesses the data row ``my_data_row`` in the measurement with index ``0`` of the machine ``M0`` in the dataset ``Data``.