From 2ef76cf42aabbf920ba25993ad1bfb39e4115daf Mon Sep 17 00:00:00 2001 From: Jacqueline Deans Date: Thu, 4 May 2023 08:53:56 -0500 Subject: [PATCH] Add old design documentation under developer_docs --- docs/developer_docs/DesConvertSwig.md | 333 ++++++++++++++++++ docs/developer_docs/DesDataRecording.md | 290 +++++++++++++++ .../DesInterfaceCodeGenerator.md | 110 ++++++ docs/developer_docs/DesMasterSlave.md | 168 +++++++++ docs/developer_docs/DesMonteCarlo.md | 85 +++++ docs/developer_docs/DesRealTimeClock.md | 50 +++ .../DesRealTimeSynchronization.md | 98 ++++++ docs/developer_docs/DesScheduledJobQueue.md | 87 +++++ docs/developer_docs/DesSleepTimer.md | 91 +++++ docs/developer_docs/Developer-Docs-Home.md | 11 + 10 files changed, 1323 insertions(+) create mode 100644 docs/developer_docs/DesConvertSwig.md create mode 100644 docs/developer_docs/DesDataRecording.md create mode 100644 docs/developer_docs/DesInterfaceCodeGenerator.md create mode 100644 docs/developer_docs/DesMasterSlave.md create mode 100644 docs/developer_docs/DesMonteCarlo.md create mode 100644 docs/developer_docs/DesRealTimeClock.md create mode 100644 docs/developer_docs/DesRealTimeSynchronization.md create mode 100644 docs/developer_docs/DesScheduledJobQueue.md create mode 100644 docs/developer_docs/DesSleepTimer.md diff --git a/docs/developer_docs/DesConvertSwig.md b/docs/developer_docs/DesConvertSwig.md new file mode 100644 index 00000000..4264384c --- /dev/null +++ b/docs/developer_docs/DesConvertSwig.md @@ -0,0 +1,333 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Convert Swig | +|------------------------------------------------------------------| + +# convert_swig + +The purpose of convert_swig is to create SWIG interface files for the given +C/C++ header file (usually S_source.hh) and each of the header files that it +(recursively) includes. SWIG (Simplified Wrapper and Interface Generator) is +an interface compiler that connects programs written in C and C++ with scripting +languagues such as Perl and Python. + +In Trick the list of header files to be processed is usually produced by the script +make_swig_makefile.pm, as it's creating Makefile_swig. This list is stored in +the file ".S_library_swig". So, if .S_library_swig exists, we can just open and read it. + +Otherwise we need to process S_source.hh to produce the list of header files. + +Specifically, we want to generate SWIG interfaces for those header files that are: + +1. actual dependencies of S_source.hh, GIVEN THE CURRENT environment and +1. not excluded from ICG processing ( by ICG_NO or ICG_EXCLUDE). + +The header files that are actually included are the dependencies we care +about. Keep in mind that the preprocessor and the current ENVIRONMENT +may cause some headers to be conditionally included or excluded. We only +want to generate SWIG interfaces for headers that are ACTUALLY included. + +Whereas the preprocessor can (using the gcc -MM option) generate a list +of dependencies that satisfy 1) (above), it can't handle that ICG exclusions. +And, whereas the function get_headers() can generate a list of dependences +which are flagged if they contain ICG_NO, it doesn't handle conditional includes. + +So, the strategy that we employ is to generate and then find the +intersection of both lists. Then we eliminate those that are in 1) +$TRICK_HOME/trick_source, or 2) flagged as containing ICG_NO or 3) are +in ICG_EXCLUDE'd directories. + +First, create a list headers using the GCC with the -MM option. GCC will +handle conditional inclusion. + +Second, create a list where the files are flagged if they contain ICG_NO. + +Then we generate the intersection of the two lists and then eliminate the dependencies that: + +1. are in $TRICK_HOME/trick_source. +1. contain ICG_NO. +1. are in ICG_EXCLUDE'd directories. + +to create the final list of header dependencies that we need to convert into SWIG interfaces. + +Next we need to determine which of the files do not have up-to-date SWIG files. +For each header file in final dependency list, if the corresponding SWIG (.i) file +doesn't exist or the header file is newer than the existing SWIG file, then record +that a new SWIG file needs needs to be created. The global hash %out_of_date +represents a list of header files whose corresponding .i files need to be regenerated. + +Finally, call process_file() to create SWIG interface files for each of the out_of_date headers. + +## process_file + +### Synopsis + +This subroutine processes S_source.h and each of it's requisite header files to +generate the corresponding SWIG interfaces files. + +### Parameters + +sim_ref Is this parameter ever used? + +in_file The name of input file, invariably "S_source.hh". + +### Design + +This function first reads the contents of and each of the header files +that it (recursively) includes into a hash (%file_contents), keyed by the +corresponding filenames. It then converts the contents of the each header whose +corresponding SWIG interface file is out of date into a SWIG interface file. + +#### Creating the %file_contents hash + +To create the file_contents hash, we first run the input file through the +C/C++ preprocessor with the -dI option. This creates one file containing +the contents of all of the included header files delimited by "line markers". +The line markers indicate which header file each of content came from. +Preprocessor line markers are of the form: '#' +They are described in Chapter 9, "Preprocessor Output" of the GCC document, +"Preprocessor Output - The C Preprocessor". Whether in_file is run through +the C or C++ preprocessor depends on its name suffix. + +For each line in the ONE big file, check whether it's a linemarker or not. +if it's a linemarker ( telling us where the following content is from) +extract the header filename. This will be our current filecontents hash key. +If it's not a linemarker, then it must be content. So, append it to the string, +whose key is the current file name ($curr_name). + +#### Creating SWIG interface files + +The global hash \%out_of_date represents the list of header files whose +corresponding SWIG interface files are out of date. It is generated in the main +part of the convert_swig program. + +For each of these out of date header files, we generate a SWIG interface file +from the contents stored in the \%file_contents hash. + +First we remove the friend init_attr functions from the headers content. +They don't need to be wrapped. + +Then, for each of the #includes in the out_of_date header file +create a corresponding %import directive. + +Next, we generate a module name and path for the SWIG interface file. The module +name is generated from an md5 hash of the header file's full name. + +Finally we open the SWIG interface file, and in it we: + +1. write a %module directive that identifies the module. +1. write a #include directive to include trick_swig interface utilities. +1. write a #include directive to include the header file to which this + interface file corresponds and from which it was derived. +1. create a SWIG interface for each class declared in the corresponding + header file using the %trick_swig_class_typemap() macro. This macro is + defined in swig_class_typedef.i, included by trick_swig.i (see above). +1. Write the SWIG interface code (processed header file) and the header + file contents. + + +## process_contents + +### Synopsis + +Process header file contents for use in the corresponding SWIG interface file. + +### Parameters + +contents_ref + (IN) reference to header file contents that are to be converted to a SWIG interface. + +new_contents_ref + (OUT) SWIG interface code, derived from the header file contents. + +curr_namespace + (IN) current namespace. + +class_names_ref + (OUT) reference to an array of class and/or struct names encountered when + processing the header file contents. + +template_typedefs_ref + (OUT) Series of SWIG %template directives. %template directives create a type in + the target language that corresponds to a C++ template instance. + +### Description + +While there's header file content remaining to be processed, repeatedly make the +best match with the following available patterns: + +Case of :
+typedef existing-type-name new-type-name ';'
+Concatenate the matched text to the SWIG interface text. + +Case of :
+typedef enum optional-name '{' bracketed-content '}' enum-name';'
+Concatenate the matched text to the SWIG interface text. + +Case of :
+typedef ( struct | union ) name '{'
+Call process_typedef_struct() to process the matched text and the bracketed +content of the struct that follows in the header file contents and update the +SWIG interface text accordingly. + +Case of :
+template '<' template-parameters '>' class class-name
+then just concatenate the matched text to the SWIG interface text. + +Case of:
+namespace name
+then call process_namespace() to process the matched text and the +bracketed content that follows in the header file contents and update the the +SWIG interface text accordingly. + +Case of:
+( class | struct ) class-name ( '{' | ':' )
+Call process_class() to process the matched text and the bracketed +that follows in the header file contents and update the the SWIG interface text. + +Default: +Match anything that doesn't match the other patterns and concatenate it to the +to the SWIG interface text. Note that (in Perl) *? in the regular +expression (.*?) is a non-greedy quantifier, so it gobbles up text only +until another match can be made. + +## process_namespace + +### Synopsis + +Process namespaces found in a header file for use in the corresponding SWIG +interface file. + +### Parameters + +namespace_string +(IN) This is a string of the form: namespace name, that was +extracted from the header file contents. In the contents there should remain the bracketed +content to which this namespace applies. + +contents_ref +(IN) This is a reference to the remainder of the header file (following the + above string) to be processed. + +new_contents_ref +(OUT) The SWIG code generated so far. + +curr_namespace +(IN) current namespace. + +class_names_ref +(OUT) reference to an array of class and/or struct names encountered when +processing the header file contents. + +template_typedefs_ref +(OUT) Series of SWIG %template directives. %template directives create a type +in the target language that corresponds to a C++ template instance. + +### Description + +Extract the name from the namespace_string and append it to the current namespace's name. + +Add the namespace_string to the SWIG interface text. + +Call extract_bracketed() to extract the contents of the namespace from the header text. + +Call process_contents() to convert the extracted namespace contents to a SWIG interface. + +Append whatever wasn't matched in process contents to the SWIG interface text. + +## process_class + +### Synopsis +Process classes declarations found in a header file for use in the corresponding +SWIG interface file. + +### Parameters + +class_string +(IN) This is a string of the form:
+( class | struct ) class-name ( '{' | ':' )
+ +contents_ref +(IN) This is a reference to the remainder of the header file (following the +class_string) to be processed. + +new_contents_ref +(OUT) The SWIG code generated so far. + +curr_namespace +(IN) current namespace. + +class_names_ref +(OUT) reference to an array of class and/or struct names encountered when +processing the header file contents. + +template_typedefs_ref +(OUT) Series of SWIG %template directives. %template directives create a type +in the target language that corresponds to a C++ template instance. + +### Description + +process_class() processes class declarations with the following steps: + +1. Extract the class_name from the class_string. +Add the class_string to the SWIG interface text. + +1. Call extract_bracketed() to extract the class contents between '{' and '}'. + +1. While there's class content text remaining to be processed, +repeatedly search for data members that match : template_name '<' template-params '>' name ; +For each match, create a SWIG %template directive to create an instantiation +of the specific templated type used by the data member. Add the +SWIG %template directive to the templated typedefs string that +Otherwise append whatever wasn't matched in process contents to +the SWIG interface text. + +## process_typedef_struct + +### Synopsis + +Process a type definition of a struct or union to make it suitable as SWIG +interface code. Extract the struct (or union) name and bracketed contents from +the header file text (typedef_struct_string and contents_ref) . Record the +extracted names in the list referenced by class_names_ref, and then reconsistute +the type definition, via the new_contents_ref. + +### Parameters + +typedef_struct_string + (IN) This is a string of the form:
+ + 1. typedef struct [ optional-name ] "{" OR
+ 1. typedef union [ optional-name ] "{"
+ +contents_ref + (IN) This is a reference to the remainder of the header file (following the + above string) to be processed. + +new_contents_ref + (OUT) The SWIG interface code generated so far. + +class_names_ref + (OUT) reference to an array of class and/or struct names encountered when + processing the header file contents. + +### Description + +process_typedef_struct() processes a type definition of a struct or union +with the following steps: + +1. Append the typedef_struct_string to the SWIG interface text +(via new_contents_ref). + +1. Extract the optional-name from the typedef_struct_string. + +1. Call extract_bracketed() to extract the struct contents from the header +text (via contents_ref), including the right bracket. + +1. Extract the one or more typedef'ed names followed by a semi-colon, that +should still be in the header text. + +1. Push the optional-name and the typedef'ed names into the class_names list +(via class_names_ref). + +1. Append the bracketed struct contents and the one or more typedef'ed names +and the semi-colon that we just extracted to the SWIG interface text. + diff --git a/docs/developer_docs/DesDataRecording.md b/docs/developer_docs/DesDataRecording.md new file mode 100644 index 00000000..413f468e --- /dev/null +++ b/docs/developer_docs/DesDataRecording.md @@ -0,0 +1,290 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Data Recording | +|------------------------------------------------------------------| + +# Data Recording Design + +The data recording design consists of 3 object types working together. At the bottom +level are the data recording groups. The data recording groups contain the user +specified variables to record, the frequency and the format to write the data in. +Each data recording group is associated with a buffering technique object. These objects +determine when the data recording groups write their data. Finally at the top is the +data recording dispatcher. The dispatcher holds the list of all the buffering techniques +and serves as a single calling point for the simulation executive to fire off all +data recording group activities. + +## DataRecordGroup + +The first objects are a set of DataRecordGroups. The DataRecordGroups are +responsible for copying simulation data into a temporary memory buffer +while the simulation is running. The DataRecordGroups also are responsible +for formatting the simulation data into a recording format. + +A base DataRecordGroup object provides functions common to all recording +groups and provides an interface for the specific recording formats to follow. +The base DataRecordGroup derives from the SimObject class. This allows +new data recording groups to be added and removed from the scheduler as +other SimObjects. + +The base class provides the following common functions: + +1. Common initialization of a data record group. +1. Common copying of simulation data to memory buffers. + +The base class provides function definitions for the specific formats to define: + +1. Format specific initialization +1. Format specific write data from memory buffer to disk +1. Format specific shutdown of data recording files. + +## DataRecordGroup Common Initialization + +Initialization of a DataRecord group is handled in three phases. + +The first is during the construction of the object. New data recording group +objects are typically created within the run input file read in during simulation +initialization. + +@copydetails Trick::DataRecordGroup::DataRecordGroup(string) +Trick::DataRecordGroup::DataRecordGroup(string) + +The second part of a DataRecordGroup initialization is supplied by the user. +The user typically sets these in the run input file read in during simulation +initialization. Typically the user will: + +1. Add variables to be recorded. +1. Add variables to be added to the changes watch list. +1. Assign a different recording frequency. +1. Change the amount of memory data record is to allocate for memory buffers + +The third part of initialization occurs at the end of simulation initialization. +During this phase all data recording groups finish their initialization. Format +specific initialization is also called during this phase. + +@copydetails Trick::DataRecordGroup::init() +Trick::DataRecordGroup::init() + +### DataRecordGroup Common Copy Simulation Data + +This routine copies simulation data into a large buffer which will be written to +disk when commanded by the DataRecordDispatcher. Copying the simulation off into +a buffer allows the simulation to continue executing and data recording to use +a separate thread of execution to write the data to disk. The following algorithm + +-# Return if data recording is not enabled. +-# If the recording frequency is to record on parameter changes + -# For each change parameter compare the current value against the previous frame value. + -# If the value has changed mark the change_detected flag true + -# Copy the current value to the previous frame value. +-# If the recording frequency is set to always or the change_detected flag is true + -# If the recording frequency is to record on parameter changes step + -# Copy to memory the previous value with the current time stamp + -# If the copy memory pointer is at the end of the buffer, set the pointer to the + beginning of the buffer + -# Copy to memory the current value with the current time stamp + +### Data Recording Specific Formats Routines + +#### ASCII Recording + +@copydoc Trick::DRAscii +Trick::DRAscii + +##### ASCII Initialization + +@copydetails Trick::DRAscii::format_specific_init() +Trick::DRAscii::format_specific_init() + +##### ASCII Write Data + +@copydetails Trick::DRAscii::write_data() +Trick::DRAscii::write_data() + +##### ASCII Shutdown + +@copydetails Trick::DRAscii::shutdown() +Trick::DRAscii::shutdown() + +#### Binary Recording + +@copydoc Trick::DRBinary +Trick::DRBinary + +##### Binary Initialization + +@copydetails Trick::DRBinary::format_specific_init() +Trick::DRBinary::format_specific_init() + +##### Binary Write Data + +@copydetails Trick::DRBinary::write_data() +Trick::DRBinary::write_data() + +##### Binary Shutdown + +@copydetails Trick::DRBinary::shutdown() +Trick::DRBinary::shutdown() + +#### HDF5 Recording + +@copydoc Trick::DRHDF5 +Trick::DRHDF5 + +##### HDF5 Initialization + +@copydetails Trick::DRHDF5::format_specific_init() +Trick::DRHDF5::format_specific_init() + +##### HDF5 Write Data + +@copydetails Trick::DRHDF5::write_data() +Trick::DRHDF5::write_data() + +##### HDF5 Shutdown + +@copydetails Trick::DRHDF5::shutdown() +Trick::DRHDF5::shutdown() + +## DataRecordBuffering + +The second objects are a set of DataRecordBuffering objects. These objects +are responsible for writing the copied simulation data to disk. There are +three techniques to write data to disk. + +A base DataRecordBuffering object provides a common interface for each of +the buffering techniques. The common object provides routines for. These +base routines may be overridden by the specific buffering techniques. + +1. Processing simulation arguments +1. Initialization +1. Adding a data recording group + +The base DataRecordBuffering provides hooks for: + +1. Processing data at the end of each software frame +1. Processing data at shutdown. + +An empty initialization routine is provided by the base object if the +specific buffering techniques do not require a specialized initialization routine. + +The buffering classes do not write the data themselves, rather they call the data record +groups write_data routines to do the writing. + +The 3 specific buffering techniques are + +1. Write directly to disk immediately. +1. Use a thread to write data to disk asynchronously +1. Do not write data until simulation shutdown. Allow data copying to + overwrite data stored in the memory buffer. + +### DataRecordBuffering Base Routines + +#### DataRecordBuffering Process Simulation Arguments + +@copydetails Trick::DataRecordBuffering::process_sim_args() +Trick::DataRecordBuffering::process_sim_args() + +#### DataRecordBuffering Initialization + +@copydetails Trick::DataRecordBuffering::init() +Trick::DataRecordBuffering::init() + +#### DataRecordBuffering Adding a Recording Group + +@copydetails Trick::DataRecordBuffering::add_group() +Trick::DataRecordBuffering::add_group() + +### DataRecordDisk + +This buffering technique calls all of the data record write_data routines in the +main simulation thread of execution. + +#### DataRecordDisk End of Frame Processing + +@copydetails Trick::DataRecordDisk::write_data() +Trick::DataRecordDisk::write_data() + +#### DataRecordDisk Shutdown Processing + +@copydetails Trick::DataRecordDisk::shutdown() +Trick::DataRecordDisk::shutdown() + +### DataRecordRing + +This buffering technique does not call the write_data routines during runtime. Only +at shutdown does the write_data routines get called. The last contents of the memory +buffer are then dumped to disk. + +#### DataRecordRing End of Frame Processing + +@copydetails Trick::DataRecordRing::write_data() +Trick::DataRecordRing::write_data() + +#### DataRecordRing Shutdown Processing + +@copydetails Trick::DataRecordRing::shutdown() +Trick::DataRecordRing::shutdown() + +### DataRecordThreaded + +This buffering technique uses a separate thread to write data to disk. The main +thread of execution attempts to signal the thread at the end of each frame. The +thread calls the group write_data routines when signaled. + +#### DataRecordThreaded Writer + +@copydetails Trick::DataRecordThreaded::drt_writer() +Trick::DataRecordThreaded::drt_writer() + +#### DataRecordThreaded End of Frame Processing + +@copydetails Trick::DataRecordThreaded::write_data() +Trick::DataRecordThreaded::write_data() + +#### DataRecordThreaded Shutdown Processing + +@copydetails Trick::DataRecordThreaded::shutdown() +Trick::DataRecordThreaded::shutdown() + +## DataRecordDispatcher + +The third object is the DataRecordDispatcher. The DataRecordDispatcher +calls each data recording buffering object's end of frame and shutdown +jobs. It serves as a single entry point for all data recording activities. + +The dispatcher is intended to be the only data recording object called by +the %Trick scheduler. The dispatcher contains routines to + +1. Process simulation arguments +1. Add data recording group +1. Initialization +1. End of Frame processing +1. Shutdown + +### DataRecordDispatcher Process Simulation Arguments + +@copydetails Trick::DataRecordDispatcher::process_sim_args() +Trick::DataRecordDispatcher::process_sim_args() + +### DataRecordDispatcher Add Data Recording Group + +When a data recording group is added to the DataRecordDispatcher the group is +tied to the requested buffering technique. + +@copydetails Trick::DataRecordDispatcher::add_group() +Trick::DataRecordDispatcher::add_group() + +### DataRecordDispatcher Initialization + +@copydetails Trick::DataRecordDispatcher::init() +Trick::DataRecordDispatcher::init() + +### DataRecordDispatcher End of Frame Processing + +@copydetails Trick::DataRecordDispatcher::write_data() +Trick::DataRecordDispatcher::write_data() + +### DataRecordDispatcher Shutdown + +@copydetails Trick::DataRecordDispatcher::shutdown() +Trick::DataRecordDispatcher::shutdown() + diff --git a/docs/developer_docs/DesInterfaceCodeGenerator.md b/docs/developer_docs/DesInterfaceCodeGenerator.md new file mode 100644 index 00000000..f15fe050 --- /dev/null +++ b/docs/developer_docs/DesInterfaceCodeGenerator.md @@ -0,0 +1,110 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → ICG | +|------------------------------------------------------------------| + +# Interface Code Generator Design + +The Interface Code Generator (ICG) creates class/struct layout information, called +attributes, used to convert string variable names to simulation memory +addresses. The attributes are used with several of Trick's services +including checkpointing, data recording, and the variable server. + +ICG execution is divided into 2 parts. + +1. Executing the Clang C++ frontend to parse all header files and extract + class/struct information. +1. Print class/struct information as attributes in input/output source code. + +## Clang C++ Frontend + +ICG uses the Clang frontend. Clang is a C/C++ frontend to the LLVM compiler. +Clang and LLVM provide libraries that allow developers to create their own tools +using the compiler. ICG calls the clang parser to process the header +files and converts the source text into a tree form called an abstract syntax tree +(AST). Clang completely handles all textual parsing. ICG provides classes to +provide input parameters to the compiler and to extract the information out of the +AST. + +### Parser Setup + +The frontend has some parameters that must be set in order for the parser to +execute. + +#### Parsing Language + +The S_source.hh file where parsing begins is written in C++. The frontend is set to +parse C++. + +#### Target Architecture + +The frontend requires a target architecture set. ICG does not pass the AST to the +next stage of compiler tools after the frontend, so it doesn't matter what the +target architecture is set to. Setting the target to the the default architecture +works. + +### Preprocessor + +The frontend includes a preprocessor. The preprocessor handles include and define +statements. ICG provides a pair of classes that interact with the clang preprocesor. +The HeaderSearchDirs class to provide the preprocessor with include search +directories and default defines. The CommentSaver class saves all comments parsed +by the preprocessor. These classes are used throughout the rest of ICG execution. + +#### HeaderSearchDirs + +The HeaderSearchDirs manages the header search directories used by the preprocessor +and provides an interface for querying the inclusion status of a directory. Since +this class is closely related to the preprocessor, it also handles define macros. + +The HeaderSearchDirs class provides a list of include search directories to the +clang preprocessor. There are two types of include directories, system include +directories and user provided directories. Both types must be supplied to the +preprocessor. + +The system include directories are retrieved by running the TRICK_CPPC compiler +in verbose preprocessor mode with no input code. When the compiler is run +in this mode, it prints the system include directories built into the compiler. +This class parses that output for the system include directories and feeds the +list into the preprocessor. + +The user include directories are retrieved from the command line when executing +ICG. Typically ICG is called with the TRICK_CXXFLAGS environment variable +included as an argument. TRICK_CXXFLAGS may include -I strings. +These include directories are parsed from the command line and fed to the +preprocessor. + +This class parses the TRICK_ICG_EXCLUDE environment variable and manages a list +of directories ICG should not write any output source code. + +The HeaderSearchDirs class provides boolean responses to questions to whether a +path is in a user directory or is in an TRICK_ICG_EXCLUDE directory. + +#### CommentSaver + +The CommentSaver class saves comments found in user code and provides an iterface +for retrieving comments based on file name/line number. The class also provides +an interface to retrieving several fields from a Trick header comment including +the ICG and ICG_EXCLUDE_TYPES fields. + +The clang preprocessor includes a comment handler interface. ICG uses +this interface to attach the CommentSaver class. Each comment handled by this +class is stored in a map indexed by file name and line number. + +### Abstract Syntax Tree (AST) Traversal + +Traversing the resulting AST is bulk of ICG's work. The clang compiler provides +an interface allowing ICG to attach handlers to process interesting events during +code parsing. In this case, ICG only attaches a handler when clang is finished +parsing the entire input file, called HandleTranslationUnit. + +When HandleTranslationUnit is start parsing the AST searching for top level +declarations we want to process. From the top level recursively calls +tree parsers that specialize in extracting information about typedefs, +classes and structs, fields within each class/struct, and enumerations. + +The following diagram shows the recursive class instantiation sequence as well +as the data returned to the caller. + +## Writing Class/Struct Attributes + + + diff --git a/docs/developer_docs/DesMasterSlave.md b/docs/developer_docs/DesMasterSlave.md new file mode 100644 index 00000000..10cca1f3 --- /dev/null +++ b/docs/developer_docs/DesMasterSlave.md @@ -0,0 +1,168 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Master-Slave | +|------------------------------------------------------------------| + +# Trick Master/Slave Synchronization Design + +Master/Slave synchronization is an optional class when running simulations. + +Master/Slave synchronization synchronizes a master simulation to one or more slave +simulations. While synchronized, the master controls the overall simulation +mode as well as the current simulation time for all simulations. The set of +simulations may change mode if one of the simulations in the set freezes or exits +unexpectedly. + +Master/Slave syncrhonization uses the Trick::MSConnect class for communications. +The MSConnect class is an abstract base class that defines functions for +connecting the simulations together and passing the simulation mode and +simulation time between simulations. The Trick::MSSocket class derives from the +Trick::MSConnect class, and uses sockets as the transport medium for master/slave communications. + +## Master + +The master simulation starts slave simulations during initialization. At the +end of each software frame during Run mode and at the end of each freeze frame +during Freeze mode the master reads the mode of all connected slave simulations. +The master will determine the mode of the simulations for the next frame +according to the slave modes and the master mode. The new mode and the +current simulation time is sent to the slaves. + +Master/Slave synchronization is divided into code based on the mode of the +simulation. The modes are Initialization, Run, Freeze, and Shutdown. + +Master/Slave functions are designed to be called from the S_define file. + +Listed below are the various modes of operation and the Master +responsibilities within each. + +### Initialization -- Master + +During Initialization slave connections are allocated and defined. Default +values to start each slave using ssh are set when a slave connection is +allocated. At the mimimum a slave must define a remote machine, the path +to the simulation executive, and a Trick::MSConnect class to use for +communications with the slave. Users may also define the remote shell to use, +a simulation executable name, a remote run directory, and additional user arguments +to pass to the slave. The allocation and definition of the slave +connections are typically done through Trick's input processor, but may also be +done in default_data or initialization jobs. + +At the end of the scheduler initialization phase the master starts the slaves. +For each slave the master creates a slave startup command and executes the +slave startup command. This is how the slave startup command is created: + +1. If the MSConnect class is not defined, return an error. +1. Begin the slave startup command with the slave remote shell. +1. Add the remote display environment variable to the startup command if + one was defined. +1. Add a cd command to the slave simulation path directory to the slave starup + command. +1. Add the S_main executable name to the slave startup command. +1. Add the Run Input File to the slave startup command. +1. Add additional user arguments to the slave startup command. +1. Execute the slave startup command. + +The master sends its software frame to each slave. + +### Run -- Master + +During Run mode the master reads the simulation mode of each slave. The +master determines the new simulation mode and sends the new mode and the +current simulation time to the slaves. + +1. For each slave + 1. Read the simulation command of the slave. + 1. If the master mode command is not ExitCmd take action if the slave + returns the following: + 1. If the slave command was not readable, returned ErrorCmd, and: + 1. Terminate the sim if sync_error_terminate == true. + 1. Deactivate the slave if sync_error_terminate == false. + 1. If the slave command is ExitCmd: + 1. Terminate the sim if sync_error_terminate == true. + 1. Deactivate the slave if sync_error_terminate == false. + 1. If the slave command is FreezeCmd: + 1. Freeze the master sim + +After gathering the simulation command from all of the slaves, send the +command to all of the slaves. + +1. For each slave + 1. Write the simulation time to the slave. + 1. Write the simulation mode command to the slave. + +### Freeze -- Master + +Upon entering freeze mode the master turns off sync_time_limit because while +in freeze, master/slave communications may not be regular and periodic. + +At the end of each freeze frame call Trick::Master::end_of_frame() to perform the same +synchronization as in run mode. + +When exiting freeze mode the master sets sync_time_limit back to the user +set value ready for running again. + +### Shutdown -- Master + +The master tells the slaves that it is shutting down. + +1. For each slave + 1. Write the simulation time to the slave. + 1. Write the simulation mode command ExitCmd to the slave. + + +## Slave + +At the end of each software frame during Run mode and at the end of each +freeze frame during Freeze mode slave simulations send the master the current +mode of the slave. The slaves wait for the new mode and simulation time from +the master. + +### Initialization -- Slave + +The Trick::Slave class is enabled by calling the MSConnect::process_sim_args() +routine. If the MSConnect::process_sim_args() finds command line arguments +specifying this simulation is a slave, then the Trick::Slave is enabled. + +At the end of the scheduler initialization phase the slave connects to the +master. + +1. Call Trick::MSConnect::connect() to connect the slave to the master. +1. Search for a RealtimeSync class. If one exists, turn the object off. +1. Search for a Master class. If one exists, turn the object off. +1. Read and set the slave's software frame from the master. + +### Run -- Slave + +In Run mode the slave writes its next executive command to the master. The +master will return the overall next simulation mode to the slave. The slave +will also read the current simulation time from the master. + +1. Write the simulation command to the master. +1. Read the simulation time from the master. +1. if the simulation time read was not successful: + 1. Terminate the sim if sync_error_terminate == true. + 1. Disable master/slave communication and command the sim to freeze if sync_error_terminate == false. +1. If the simulation time read was successful set the slave simulation + time to the time of the master. +1. Read the simulation command from the master. +1. If the simulation command was not successful: + 1. Terminate the sim if sync_error_terminate == true. + 1. Disable master/slave communication and command the sim to freeze if sync_error_terminate == false. +1. If the simulation command read was successful set the slave simulation + command to the command of the master. + +### Freeze -- Slave + +Upon entering freeze mode the slave turns off sync_time_limit because while +in freeze, master/slave communications may not be regular and periodic. + +At the end of each freeze frame call Trick::Slave::end_of_frame() to perform the same +synchronization as in run mode. + +When exiting freeze mode the slave sets sync_time_limit back to the user +set value ready for running again. + +### Shutdown -- Slave + +If a slave is shutting down, inform the master that it is exiting. +[@anchor d_slave_shutdown_0 d_slave_shutdown_0] + diff --git a/docs/developer_docs/DesMonteCarlo.md b/docs/developer_docs/DesMonteCarlo.md new file mode 100644 index 00000000..344a691f --- /dev/null +++ b/docs/developer_docs/DesMonteCarlo.md @@ -0,0 +1,85 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Monte Carlo | +|------------------------------------------------------------------| + +# Monte Carlo and Optimization + +Monte Carlo is the process of iteratively calling a simulation over a set of predetermined or auto-generated inputs. +Trick has designed its Monte Carlo capability to run distributed. + +## Structure +In particular, Monte Carlo is designed after a "master/slave" model. The master is in charge of creating slaves and +tasking them to work. There may be any number of slaves distributed over a network. Master and slave communicate through +sockets. Theoretically, a master and slave need not have the same file system. Each slave is responsible for requesting +work, accomplishing work and reporting results. The work at hand is running a single simulation iteratively over an input space. + +### The Master +The master is the command center of a Monte Carlo simulation. The master tasks slaves to run the simulation with a given set of inputs. The master will task slaves to run in parallel. The master is responsible for keeping the slaves as busy as possible. To keep things running smoothly, the master is designed to reassign work when a slave is either dead or running too slowly. the master is only in charge of tasking work. The master does not run the simulation itself. The master will continue issuing work to the slaves until it is satisfied all simulation runs are complete. + +The master's life cycle consists of the following: + + - Initialize + - While there are unresolved runs: + - Spawn any uninitialized slaves. + - Dispatch runs to ready slaves. + - Resolve run based on slave's exit status. + - Receive results from finished slave's child. + - Check for timeouts. + - Shutdown the slaves and terminate. + +@see Trick::MonteCarlo + +### Slaves + +A slave consists of a parent and fork()ed children. A slave parent spawns a child using the fork() system call. A +slave child runs the simulation in its own address space. Only one child exists at a time in a slave. Per slave, +simulation execution is sequential. + +A slave is responsible for requesting work from the master, running a Trick simulation with inputs given by the master, +dumping recorded data to disk and informing the master when it is finished running its task. + +The slave's life cycle consists of the following: + + - Initialize + - Connect to and inform the master of the port over which the slave is listening for dispatches. + - Until the connection to the master is lost or the master commands a shutdown: + - Wait for a new dispatch. + - Process the dispatch. + - Slave fork()s child. + - Child runs simulation with varied input. + - Write the run number processed to the master at child shutdown. + - Write the exit status to the master. + - Run the shutdown jobs and terminate. + +@see Trick::MonteSlave + +## Simulation Inputs + +The goal of Monte Carlo is to run the simulation over a set of inputs. The inputs that the master passes to the slaves +are either generated by a statistical algorithm or they are hard-coded by the user in a data file. Inputs may also be +generated exclusively by user calculations. + +## Monte Carlo Output + +For each simulation run within a Monte Carlo suite of runs, a directory called "MONTE_" is created. Slave output +is directed to this "MONTE_" directory. Trick recorded data is written in a set of "RUN_" directories within the parent +"MONTE_" directory. Along with recorded data, stdout, stderr, and send_hs files are written. A file that contains the +summary of all runs is written to the "MONTE_" directory. + +### Data Processing + +The trick_dp is desinged to understand "MONTE_" directories. When choosing to plot a "MONTE_" directory, trick_dp +will overlay all curves from each "RUN_" directory within the parent "MONTE_" directory. The plot widget has built +in features that allow the developer to distinguish what curve goes with what simulation run. + +### Optimization + +Optimization is made possible by creating a framework whereby the developer can change simulation inputs based on +simulation results. Trick offers a set of job classes that allow the developer to enter the Monte Carlo loop and +thereby enter the decision making on-the-fly. No canned optimization is available. + +This special set of job classes work in concert together in master and slaves. Trick schedules jobs within the master +at critical points so that they may create inputs to send to the slave as well as receive results from the slave. +Slave jobs are scheduled to receive simulation inputs from the master as well as send simulation results back to the +master. + +The jobs are specified in the S_define. The jobs are created by the developer. diff --git a/docs/developer_docs/DesRealTimeClock.md b/docs/developer_docs/DesRealTimeClock.md new file mode 100644 index 00000000..a8d415b6 --- /dev/null +++ b/docs/developer_docs/DesRealTimeClock.md @@ -0,0 +1,50 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Real-Time Clock | +|------------------------------------------------------------------| + +# Real-Time Clock Design + +## Base Real-Time Clock + +The base Clock object provides an interface for the following operations. The +base Clock does not implement the following operations. + +1. Initializing the clock hardware +1. Resetting the clock (setting the reference time) +1. Getting the current real time +1. Spinning on the clock
+ Spinning on the clock means to continually check the clock waiting for real-time to + catch up to the desired simulation elapsed time. +1. Stopping the clock + +## Gettimeofday Clock + +The GetTimeOfDayClock is a clock object that derives from the Base Real-Time Clock. The +GetTimeOfDayClock uses system clock as the real-time clock. + +The GetTimeOfDayClock implements the interface laid out by the base clock class + +### Initializing the Clock + +@copydetails Trick::GetTimeOfDayClock::clock_init() +Trick::GetTimeOfDayClock::clock_init() + +### Resetting the Clock (setting the reference time) + +@copydetails Trick::GetTimeOfDayClock::clock_reset() +Trick::GetTimeOfDayClock::clock_reset() + +### Getting the Current Real Time + +@copydetails Trick::GetTimeOfDayClock::clock_time() +Trick::GetTimeOfDayClock::clock_time() + +### Spinning on the Clock + +@copydetails Trick::GetTimeOfDayClock::clock_spin() +Trick::GetTimeOfDayClock::clock_spin() + +### Stopping the Clock + +@copydetails Trick::GetTimeOfDayClock::clock_stop() +Trick::GetTimeOfDayClock::clock_stop() + diff --git a/docs/developer_docs/DesRealTimeSynchronization.md b/docs/developer_docs/DesRealTimeSynchronization.md new file mode 100644 index 00000000..2031a901 --- /dev/null +++ b/docs/developer_docs/DesRealTimeSynchronization.md @@ -0,0 +1,98 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Real-Time Synchronization | +|------------------------------------------------------------------| + +# Real-Time Synchronization Design + +A simulation is "real-time" when it can consistently and repetitively execute +its scheduled math models to completion within some predetermined interval +time frame for an indefinite period of time. This predetermined interval +time frame is referred to as the real-time frame. In this design the real-time +frame is set equal to the simulation software frame. Real-time is achieved +by periodically pausing simulation execution at the end of each real-time +frame. If simulation execution has not finished by the end of each real-time +frame then this is an overrun. The real-time synchronization capability has +several user settable options when a simulation overruns. If simulation +execution finishes before the real-time frame expires, this is an underrun. If +the simulation is underruning, the real-time synchronization capability allows +the simulation to "sleep" while waiting for the real-time frame to expire. + +Real-time synchronization defaults off when initializing the simulation. +To enable real-time synchronization users or models may call enable() at any +time during the simulation. To exit real-time synchronization users or +models may call disable() at any time during the simulation. The actual +entering or exiting real-time happens at the end of the frame. + +Listed below are the various modes of operation in a simulation and the +responsibilities of the real-time synchronization class in each. + +## Pre-Initialization + +During pre-initialization the user may specify the behavior of the real-time +synchronization class during overruns and underruns. + +1. Set real-time synchronization enabled or disabled. +1. Set real-time sleep timer enabled or disabled. +1. Set the maximum number of consecutive overruns allowed. (default 100,000,000) +1. Set the maximum time allowed for single overruns (default 1e37 seconds) +1. Set action taken when maximum overruns achieved to freeze/terminate or + terminate immediately (default terminate immediately) + +## Initialization and Restart + +The real-time initialization and restart jobs are designed to be the last +jobs run before exiting initialization or restart mode. + +Initialize the underlying hardware. + +@copydetails Trick::RealtimeSync::initialize(long long , long long ) +Trick::RealtimeSync::initialize(long long , long long ) + +## Start Real-Time + +@copydetails Trick::RealtimeSync::initialize(long long , long long ) +Trick::RealtimeSync::initialize(long long , long long ) + +## End of Frame + +At the end of the real-time frame (also the software frame) the +real-time synchronization class will determine if the simulation is +underruning or overrunning and take actions to keep simulation time +and real-time in sync. + +@copydetails Trick::RealtimeSync::rt_monitor( long long ) +Trick::RealtimeSync::rt_monitor( long long ) + +## Freeze Mode + +### Freeze Initialization + +During freeze initialization the real-time synchronization class resets the real-time +frame to the freeze frame and restarts the real-time clock and timer. + +@copydetails Trick::RealtimeSync::freeze_init( long long ) +Trick::RealtimeSync::freeze_init( long long ) + +### Freeze Loop + +During freeze loop processing the real-time synchronization pauses the freeze loop. +The pause is released when real-time equals the freeze loop period. No checks are +done for overrunning or underruning a freeze period. + +@copydetails Trick::RealtimeSync::freeze_pause( long long ) +Trick::RealtimeSync::freeze_pause( long long ) + +### UnFreeze + +During unfreeze the real-time synchronization class resets the real-time +frame to the software frame and restarts the real-time clock and timer. + +@copydetails Trick::RealtimeSync::unfreeze( long long , long long ) +Trick::RealtimeSync::unfreeze( long long , long long ) + +## Shutdown Mode + +During shutdown, the real-time synchronization class stops the clock and sleep +timer hardware. + +@copydetails Trick::RealtimeSync::shutdown() +Trick::RealtimeSync::shutdown() diff --git a/docs/developer_docs/DesScheduledJobQueue.md b/docs/developer_docs/DesScheduledJobQueue.md new file mode 100644 index 00000000..d81e0e87 --- /dev/null +++ b/docs/developer_docs/DesScheduledJobQueue.md @@ -0,0 +1,87 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Scheduled Job Queue | +|------------------------------------------------------------------| + +# ScheduledJobQueue Design + +The Scheduler typically calls upon the ScheduledJobQueue to add +a job, find the next job that matches the current sim time, get the next +job call time, and instrument the job queue. + +Listed below are the design for each use case of the ScheduledJobQueue. + +## Adding a job + +Adding a job to the queue increases the queue size by 1. Jobs are ordered by +job class, then phase, then sim object id, and finally job_id. The job_id increases +with the ordering of jobs in a sim object. + +@copydetails Trick::ScheduledJobQueue::push() +Trick::ScheduledJobQueue::push() + +## Adding a job ignoring sim object order. + +Somethimes To add a job to the queue, Setting the sim object id to a large nnumber effectively +removes the sim object id in the sorting. + +@copydetails Trick::ScheduledJobQueue::push_ignore_sim_object() +Trick::ScheduledJobQueue::push_ignore_sim_object() + +## Find Next Job with Current Sim Time + +Advances curr_index to next job in list who's next call time matches the incoming time +argument. If the job is not a system_job as determined by the scheduler, the next job +call time is calculated before returning. (system_jobs set their own next job call time). +Also track the next_job_time which will be used to advance simulation time +at the end of frame. + +@copydetails Trick::ScheduledJobQueue::find_next_job() +Trick::ScheduledJobQueue::find_next_job() + +## Get the Next Job + +A simple get the next job in the list. + +@copydetails Trick::ScheduledJobQueue::get_next_job() +Trick::ScheduledJobQueue::get_next_job() + +## Get the next job call time + +As the find_next_job routine traverses the queue of jobs searching for jobs that match the +current simulation time step, it also saves the time of the next lowest job +call time. This is the next time step that the scheduler will use when +advancing simulation time. + +@copydetails Trick::ScheduledJobQueue::get_next_job_call_time() +Trick::ScheduledJobQueue::get_next_job_call_time() + +## Test Next Job Call Time + +Some job types reschedule themselves and the next job call time are not tested by the find next job routine. +The test next job call time allows these job to check their next call time against the overall next +job call time. If the current job next job call time is lower than the overall job call time, the +overall next job call time is set to the current job's next call time. + +@copydetails Trick::ScheduledJobQueue::test_next_job_call_time() +Trick::ScheduledJobQueue::test_next_job_call_time() + +## Instrument a job before all jobs in the queue + +To add an instrumentation job before a job or all jobs in the queue. Requirement [@ref r_exec_instrument_0] + +@copydetails Trick::ScheduledJobQueue::instrument_before(JobData * , string , InstrumentationEvent * ) +Trick::ScheduledJobQueue::instrument_before(JobData * , string , InstrumentationEvent * ) + +## Instrument a job after all jobs in the queue + +To add an instrumentation job after a job or all jobs in the queue. Requirement [@ref r_exec_instrument_2] + +@copydetails Trick::ScheduledJobQueue::instrument_after(JobData * , string , InstrumentationEvent * ) +Trick::ScheduledJobQueue::instrument_after(JobData * , string , InstrumentationEvent * ) + +## Remove an instrumentation job + +To remove an instrumentation job from the queue. Requirement [@ref r_exec_instrument_4] + +@copydetails Trick::ScheduledJobQueue::instrument_remove(string , InstrumentationEvent * ) +Trick::ScheduledJobQueue::instrument_remove(string , InstrumentationEvent * ) + diff --git a/docs/developer_docs/DesSleepTimer.md b/docs/developer_docs/DesSleepTimer.md new file mode 100644 index 00000000..92364a94 --- /dev/null +++ b/docs/developer_docs/DesSleepTimer.md @@ -0,0 +1,91 @@ +| [Home](/trick) → [Developer Docs](Developer-Docs-Home) → Sleep Timer | +|------------------------------------------------------------------| + +# Sleep Timer Design + +After all jobs in an execution frame have run, the default behavior (the timer is disabled) +is that the simulation will spin on the clock until real-time has caught up to simulation time. +Enabling the sleep timer gives control to the timer's pause() function, in which it may +release the processor and regain control when the timer elapses. + +## Base Sleep Timer + +The base Timer object provides a common interface for all Timers for the +following functions. + +1. Enabling the Timer +1. Disabling the Timer +1. Getting the enabled/disabled status + +The base Timer object provides an interface for the following operations. The +base Timer does not implement the following operations. + +1. Initializing the Timer hardware +1. Setting the frame time for timer +1. Starting the timer +1. Resetting the timer +1. Pausing for the timer to expire +1. Stopping the timer +1. Shutting down the Timer hardware + +## ITimer + +The ITimer is a Timer object that derives from the Base Sleep Timer. The +ITimer uses system itimer functionality to allow the simulation to release the +cpu while waiting for the timer to expire. When the itimer expires it uses +the SIGALRM signal to set a semaphore. When the semaphore is set the simulation +wakes up. + +The itimer does not expire exactly at the desired period. It actually exipres after +the desired period by up to 1ms. To compensate for the delay we subtract 2ms from +the desired period. The RealtimeSync class "spins" on the clock the remaining +1-2ms the timer does not sleep for. + +The ITimer implements the interface laid out by the base Timer class + +### Initialization + +@copydetails Trick::ITimer::init() +Trick::ITimer::init() + +### Setting the Frame Time + +@copydetails Trick::ITimer::set_frame_times() +Trick::ITimer::set_frame_times() + +### Starting the timer. + +@copydetails Trick::ITimer::start() +Trick::ITimer::start() + +### Restarting the timer. + +@copydetails Trick::ITimer::reset() +Trick::ITimer::reset() + +### Pausing for the Timer to Expire + +Pausing for the timer has 2 parts. The first is the signal handler that posts +a semaphore in response to a SIGALRM signal. The signal handler is assigned +to SIGALRM during the initialization routine. The second is the pause routine +that waits for the semaphore to be posted. + +#### Signal Handler + +@copydetails Trick::ITimer::it_handler() +Trick::ITimer::it_handler() + +### Pause for Semaphore + +@copydetails Trick::ITimer::pause() +Trick::ITimer::pause() + +### Stopping the timer. + +@copydetails Trick::ITimer::stop() +Trick::ITimer::stop() + +### Shutting Down the Timer + +@copydetails Trick::ITimer::shutdown() +Trick::ITimer::shutdown() diff --git a/docs/developer_docs/Developer-Docs-Home.md b/docs/developer_docs/Developer-Docs-Home.md index 155eed4f..c3ccdc6f 100644 --- a/docs/developer_docs/Developer-Docs-Home.md +++ b/docs/developer_docs/Developer-Docs-Home.md @@ -12,4 +12,15 @@ Link documentation for Trick internals, processes, and plans here. - [Python Environment](Python-Environment-Issues) - [CI](CI) +Old design documentation - may or may not be up to date + +- [Convert Swig](DesConvertSwig) +- [ICG](DesInterfaceCodeGenerator) +- [Realtime Clock](DesRealTimeClock) +- [Realtime Synchronization](DesRealTimeSynchronization) +- [Scheduled Job Queue](DesScheduledJobQueue) +- [Sleep Timer](DesSleepTimer) +- [Data Recording](DesDataRecording) +- [Monte Carlo](DesMonteCarlo) +- [Master Slave](DesMasterSlave)