trick/docs/developer_docs/DesDataRecording.md

Home → Developer Docs → Data Recording

Data Recording Design

The data recording design consists of 3 object types working together. At the bottom level are the data recording groups. The data recording groups contain the user specified variables to record, the frequency and the format to write the data in. Each data recording group is associated with a buffering technique object. These objects determine when the data recording groups write their data. Finally at the top is the data recording dispatcher. The dispatcher holds the list of all the buffering techniques and serves as a single calling point for the simulation executive to fire off all data recording group activities.

DataRecordGroup

The first objects are a set of DataRecordGroups. The DataRecordGroups are responsible for copying simulation data into a temporary memory buffer while the simulation is running. The DataRecordGroups also are responsible for formatting the simulation data into a recording format.

A base DataRecordGroup object provides functions common to all recording groups and provides an interface for the specific recording formats to follow. The base DataRecordGroup derives from the SimObject class. This allows new data recording groups to be added and removed from the scheduler as other SimObjects.

The base class provides the following common functions:

1. Common initialization of a data record group.
2. Common copying of simulation data to memory buffers.

The base class provides function definitions for the specific formats to define:

1. Format specific initialization
2. Format specific write data from memory buffer to disk
3. Format specific shutdown of data recording files.

DataRecordGroup Common Initialization

Initialization of a DataRecord group is handled in three phases.

The first is during the construction of the object. New data recording group objects are typically created within the run input file read in during simulation initialization.

@copydetails Trick::DataRecordGroup::DataRecordGroup(string) Trick::DataRecordGroup::DataRecordGroup(string)

The second part of a DataRecordGroup initialization is supplied by the user. The user typically sets these in the run input file read in during simulation initialization. Typically the user will:

1. Add variables to be recorded.
2. Add variables to be added to the changes watch list.
3. Assign a different recording frequency.
4. Change the amount of memory data record is to allocate for memory buffers

The third part of initialization occurs at the end of simulation initialization. During this phase all data recording groups finish their initialization. Format specific initialization is also called during this phase.

@copydetails Trick::DataRecordGroup::init() Trick::DataRecordGroup::init()

DataRecordGroup Common Copy Simulation Data

This routine copies simulation data into a large buffer which will be written to disk when commanded by the DataRecordDispatcher. Copying the simulation off into a buffer allows the simulation to continue executing and data recording to use a separate thread of execution to write the data to disk. The following algorithm

-# Return if data recording is not enabled. -# If the recording frequency is to record on parameter changes -# For each change parameter compare the current value against the previous frame value. -# If the value has changed mark the change_detected flag true -# Copy the current value to the previous frame value. -# If the recording frequency is set to always or the change_detected flag is true -# If the recording frequency is to record on parameter changes step -# Copy to memory the previous value with the current time stamp -# If the copy memory pointer is at the end of the buffer, set the pointer to the beginning of the buffer -# Copy to memory the current value with the current time stamp

Data Recording Specific Formats Routines

ASCII Recording

@copydoc Trick::DRAscii Trick::DRAscii

ASCII Initialization

@copydetails Trick::DRAscii::format_specific_init() Trick::DRAscii::format_specific_init()

ASCII Write Data

@copydetails Trick::DRAscii::write_data() Trick::DRAscii::write_data()

ASCII Shutdown

@copydetails Trick::DRAscii::shutdown() Trick::DRAscii::shutdown()

Binary Recording

@copydoc Trick::DRBinary Trick::DRBinary

Binary Initialization

@copydetails Trick::DRBinary::format_specific_init() Trick::DRBinary::format_specific_init()

Binary Write Data

@copydetails Trick::DRBinary::write_data() Trick::DRBinary::write_data()

Binary Shutdown

@copydetails Trick::DRBinary::shutdown() Trick::DRBinary::shutdown()

HDF5 Recording

@copydoc Trick::DRHDF5 Trick::DRHDF5

HDF5 Initialization

@copydetails Trick::DRHDF5::format_specific_init() Trick::DRHDF5::format_specific_init()

HDF5 Write Data

@copydetails Trick::DRHDF5::write_data() Trick::DRHDF5::write_data()

HDF5 Shutdown

@copydetails Trick::DRHDF5::shutdown() Trick::DRHDF5::shutdown()

DataRecordBuffering

The second objects are a set of DataRecordBuffering objects. These objects are responsible for writing the copied simulation data to disk. There are three techniques to write data to disk.

A base DataRecordBuffering object provides a common interface for each of the buffering techniques. The common object provides routines for. These base routines may be overridden by the specific buffering techniques.

1. Processing simulation arguments
2. Initialization
3. Adding a data recording group

The base DataRecordBuffering provides hooks for:

1. Processing data at the end of each software frame
2. Processing data at shutdown.

An empty initialization routine is provided by the base object if the specific buffering techniques do not require a specialized initialization routine.

The buffering classes do not write the data themselves, rather they call the data record groups write_data routines to do the writing.

The 3 specific buffering techniques are

1. Write directly to disk immediately.
2. Use a thread to write data to disk asynchronously
3. Do not write data until simulation shutdown. Allow data copying to overwrite data stored in the memory buffer.

DataRecordBuffering Base Routines

DataRecordBuffering Process Simulation Arguments

@copydetails Trick::DataRecordBuffering::process_sim_args() Trick::DataRecordBuffering::process_sim_args()

DataRecordBuffering Initialization

@copydetails Trick::DataRecordBuffering::init() Trick::DataRecordBuffering::init()

DataRecordBuffering Adding a Recording Group

@copydetails Trick::DataRecordBuffering::add_group() Trick::DataRecordBuffering::add_group()

DataRecordDisk

This buffering technique calls all of the data record write_data routines in the main simulation thread of execution.

DataRecordDisk End of Frame Processing

@copydetails Trick::DataRecordDisk::write_data() Trick::DataRecordDisk::write_data()

DataRecordDisk Shutdown Processing

@copydetails Trick::DataRecordDisk::shutdown() Trick::DataRecordDisk::shutdown()

DataRecordRing

This buffering technique does not call the write_data routines during runtime. Only at shutdown does the write_data routines get called. The last contents of the memory buffer are then dumped to disk.

DataRecordRing End of Frame Processing

@copydetails Trick::DataRecordRing::write_data() Trick::DataRecordRing::write_data()

DataRecordRing Shutdown Processing

@copydetails Trick::DataRecordRing::shutdown() Trick::DataRecordRing::shutdown()

DataRecordThreaded

This buffering technique uses a separate thread to write data to disk. The main thread of execution attempts to signal the thread at the end of each frame. The thread calls the group write_data routines when signaled.

DataRecordThreaded Writer

@copydetails Trick::DataRecordThreaded::drt_writer() Trick::DataRecordThreaded::drt_writer()

DataRecordThreaded End of Frame Processing

@copydetails Trick::DataRecordThreaded::write_data() Trick::DataRecordThreaded::write_data()

DataRecordThreaded Shutdown Processing

@copydetails Trick::DataRecordThreaded::shutdown() Trick::DataRecordThreaded::shutdown()

DataRecordDispatcher

The third object is the DataRecordDispatcher. The DataRecordDispatcher calls each data recording buffering object's end of frame and shutdown jobs. It serves as a single entry point for all data recording activities.

The dispatcher is intended to be the only data recording object called by the %Trick scheduler. The dispatcher contains routines to

1. Process simulation arguments
2. Add data recording group
3. Initialization
4. End of Frame processing
5. Shutdown

DataRecordDispatcher Process Simulation Arguments

@copydetails Trick::DataRecordDispatcher::process_sim_args() Trick::DataRecordDispatcher::process_sim_args()

DataRecordDispatcher Add Data Recording Group

When a data recording group is added to the DataRecordDispatcher the group is tied to the requested buffering technique.

@copydetails Trick::DataRecordDispatcher::add_group() Trick::DataRecordDispatcher::add_group()

DataRecordDispatcher Initialization

@copydetails Trick::DataRecordDispatcher::init() Trick::DataRecordDispatcher::init()

DataRecordDispatcher End of Frame Processing

@copydetails Trick::DataRecordDispatcher::write_data() Trick::DataRecordDispatcher::write_data()

DataRecordDispatcher Shutdown

@copydetails Trick::DataRecordDispatcher::shutdown() Trick::DataRecordDispatcher::shutdown()