mirror of
https://github.com/nasa/trick.git
synced 2024-12-18 20:57:55 +00:00
Fix review issues.
This commit is contained in:
John M. Penn
parent
12d483f094
commit
92f1234deb
@ -1,4 +1,4 @@
|
||||
# Trick Real-time Best Practices
|
||||
# Trick Realtime Best Practices
|
||||
|
||||
**Contents**
|
||||
|
||||
@ -10,87 +10,87 @@
|
||||
|
||||
---
|
||||
## Purpose
|
||||
The intention of this document is to compile and share practical knowledge, based on the experience of people in the Trick simulation community regarding the development of real-time computer simulations.
|
||||
The intention of this document is to compile and share practical knowledge, based on the experience of people in the Trick simulation community regarding the development of realtime computer simulations.
|
||||
|
||||
<a id=prerequisite-knowledge></a>
|
||||
## Prerequisite Knowledge
|
||||
(Assuming you've completed the [Trick Tutorial] (https://nasa.github.io/trick/tutorial/Tutorial))
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=simulation-time-vs-real-time></a>
|
||||
### Simulation Time vs Real Time
|
||||
<a id=simulation-time-vs-realtime></a>
|
||||
### Simulation Time vs Realtime
|
||||
|
||||
Real world dynamic systems change in real time (the time that you and I experience). A simulated dynamic system changes in simulation-time. Simulation time begins at t=0, and runs until we stop it. Simulation time can elapse faster or slower than real-time.
|
||||
Real world dynamic systems change in realtime (the time that you and I experience). A simulated dynamic system changes in simulation time. Simulation time begins at t=0, and runs until we stop it. Simulation time can elapse faster or slower than realtime.
|
||||
|
||||
If we want to interact with a simulation as if it were real, we need to synchronize simulation time to real time. This requires that a simulation is capable of running at least as fast as real time. If it is incapable, then it can not be made to run in real time.
|
||||
If we want to interact with a simulation as if it were real, we need to synchronize simulation time to realtime. This requires that a simulation is capable of running at least as fast as realtime. If it is incapable, then it can not be made to run in realtime.
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=real-time-clock></a>
|
||||
### Real Time Clock
|
||||
* By default, the Trick real-time scheduler will synchronize to the system clock:
|
||||
<a id=realtime-clock></a>
|
||||
### Realtime Clock
|
||||
* By default, the Trick realtime scheduler will synchronize to the system clock:
|
||||
* ```clock_gettime(CLOCK_REALTIME,…)``` [Linux]
|
||||
* ```gettimeofday()``` [Mac OS]
|
||||
|
||||
* The Trick real-time scheduler can also be configured to synchronize to a [custom real-time clock] (https://nasa.github.io/trick/documentation/simulation_capabilities/Realtime-Clock).
|
||||
* The Trick realtime scheduler can also be configured to synchronize to a [custom realtime clock] (https://nasa.github.io/trick/documentation/simulation_capabilities/Realtime-Clock).
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=enabling-real-time></a>
|
||||
### Enabling Real-time
|
||||
<a id=enabling-realtime></a>
|
||||
### Enabling Realtime
|
||||
|
||||
Trick tries to consistently and repetitively execute its scheduled math models to completion within some predetermined real-time interval for an indefinite period. This real-time interval is called the **real-time software frame**.
|
||||
Trick tries to consistently and repetitively execute its scheduled math models to completion within some predetermined realtime interval for an indefinite period. This realtime interval is called the **realtime software frame**.
|
||||
|
||||
To enable real-time synchronization, call ```trick.real_time_enable()``` in the input file.
|
||||
To enable realtime synchronization, call ```trick.real_time_enable()``` in the input file.
|
||||
|
||||
[Ref: Realtime](https://nasa.github.io/trick/documentation/simulation_capabilities/Realtime)
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=real-time-software-frame></a>
|
||||
### Real-time Software Frame
|
||||
The real-time software frame determines how often Trick will synchronize simulation time to the real time clock. Simulation time will run as fast as it can in the intervals between real time synchronizations.
|
||||
<a id=realtime-software-frame></a>
|
||||
### Realtime Software Frame
|
||||
The realtime software frame determines how often Trick will synchronize simulation time to the realtime clock. Simulation time will run as fast as it can in the intervals between realtime synchronizations.
|
||||
|
||||
To set the real time software frame, call the following in the input file:
|
||||
To set the realtime software frame, call the following in the input file:
|
||||
|
||||
```python
|
||||
trick.exec_set_software_frame(double time)
|
||||
```
|
||||
[Ref: Software Frame](https://nasa.github.io/trick/documentation/simulation_capabilities/Executive-Scheduler#software-frame)
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=under-runs-and-over-runs></a>
|
||||
### Under-runs and Over-runs
|
||||
|
||||
An **under-run** occurs when the Trick executive finishes running all of its scheduled jobs, between synchronizations to the real time clock. This is a **good thing**. In this case the executive will enter a spin loop, waiting for the next real-time frame to start.
|
||||
An **under-run** occurs when the Trick executive finishes running all of its scheduled jobs, between synchronizations to the realtime clock. This is a **good thing**. In this case the executive will enter a spin loop, waiting for the next realtime frame to start.
|
||||
|
||||
<a id=figure-Real-time-under-run></a>
|
||||
|
||||
<a id=figure-realtime-under-run></a>
|
||||
|
||||
|
||||
An **over-run** occurs if the executive does not finish running all of its scheduled jobs. This is a **bad-thing**. In this case, the executive will immediately start the next frame in an attempt to catch up.
|
||||
|
||||
<a id=figure-Real-time-over-run></a>
|
||||
|
||||
<a id=figure-Realtime-over-run></a>
|
||||
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=itimers></a>
|
||||
### Itimers ( Being Nice to Other Processes On Your System )
|
||||
|
||||
During real time under runs you may want to release the processor for other tasks to use instead of spinning waiting for the clock. Trick provides a real-time sleep timer based on itimers. You might think of it as a “snooze button”.
|
||||
During real time under runs you may want to release the processor for other tasks to use instead of spinning waiting for the clock. Trick provides a realtime sleep timer based on itimers. You might think of it as a “snooze button”.
|
||||
|
||||
To enable itimers call ```trick.itimer_enable()``` from the input file.
|
||||
|
||||
With itimer_enabled, the simulation will sleep() after completing the jobs scheduled for the current frame. The itimer will then wake the sim 2ms before the real-time frame is to expire. The executive will spin for the final 2ms. The 2ms spin is there because an itimer interval is not guaranteed to be perfectly precise.
|
||||
With itimer_enabled, the simulation will sleep() after completing the jobs scheduled for the current frame. The itimer will then wake the sim 2ms before the realtime frame is to expire. The executive will spin for the final 2ms. The 2ms spin is there because an itimer interval is not guaranteed to be perfectly precise.
|
||||
|
||||
<a id=figure-Real-time-with_itimer></a>
|
||||
|
||||
<a id=figure-Realtime-with_itimer></a>
|
||||
|
||||
|
||||
[Ref: Itimer](https://nasa.github.io/trick/documentation/simulation_capabilities/Realtime-Timer)
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
<a id=frame-logging></a>
|
||||
### Frame-Logging ( Critical For Improving Sim Performance )
|
||||
@ -112,6 +112,8 @@ Frame logging records the following data files in your sim’s RUN_ directory:
|
||||
* [```log_frame.trk```](#log-frame-trk)<br>
|
||||
* [```log_frame_trickjobs.trk```](#log-frame-trickjobs-trk)
|
||||
* [```log_frame_frame_userjobs_main.trk```](#log-frame-userjobs-main-trk)
|
||||
* [```log_timeline.csv```](#log-timeline-csv)
|
||||
* [```log_timeline_init.csv```](#log-timeline-csv)
|
||||
|
||||
Note that **main** in this file name refers to the main thread.
|
||||
If child threads (for example: C1, C2, ...) have been specified in the sim then frame log files for those threads will also be created ( that is:```log_trick_frame_userjobs_C1.trk```, ```log_trick_frame_userjobs_C2.trk```, ```...```).
|
||||
@ -125,8 +127,8 @@ If child threads (for example: C1, C2, ...) have been specified in the sim then
|
||||
|# | Name | Type | Units | Description |
|
||||
|--:|:---- |:-----|:-----:|-------------|
|
||||
| 1| ```sys.exec.out.time``` | double | seconds |Simulation Time |
|
||||
| 2| ```trick_real_time.rt_sync.frame_time``` | double | seconds | This badly named parameter expresses the amount of time that the scheduled jobs in this frame took to execute. See: [figure](#figure-Real-time-under``-run)|
|
||||
|3| ```trick_real_time.rt_sync.frame_overrun ``` | double | seconds | The magnitude of the current overrun. See: [figure](#figure-Real-time-over-run) |
|
||||
| 2| ```trick_real_time.rt_sync.frame_time``` | double | seconds | This badly named parameter expresses the amount of time that the scheduled jobs in this frame took to execute. See: [figure](#figure-realtime-under``-run)|
|
||||
|3| ```trick_real_time.rt_sync.frame_overrun ``` | double | seconds | The magnitude of the current overrun. See: [figure](#figure-realtime-over-run) |
|
||||
|4| ```JOB_data_record_group_frame_userjobs.data_record...``` | double | s | How long the write job for the user Jobs data recording group took. |
|
||||
|5| ```JOB_data_record_group.trickjobs...``` | double | seconds | How long did the write job for the Trick Jobs data recording group take. |
|
||||
|
||||
@ -152,6 +154,48 @@ If child threads (for example: C1, C2, ...) have been specified in the sim then
|
||||
| 1| ```sys.exec.out.time``` | double | seconds |Simulation Time |
|
||||
| n| *trick-job-name* | double | seconds |How long the trick-job took to execute. |
|
||||
|
||||
<a id=log-timeline-csv></a>
|
||||
#### ```log_timeline.csv``` & ```log_timeline_init.csv```
|
||||
These files contain start and stop times for each of the jobs executed in a trick sim.
|
||||
```log_timeline.csv``` contains times for jobs run during run-time. ```log_timeline_init.csv``` contains times for jobs run at initialization time.
|
||||
|
||||
Frankly this format is **weird**, but it contains useful information.
|
||||
It's weird because of its redundancy, and that each job timing "record" consists of four CSV lines.
|
||||
|
||||
Both files have the same format. They contain three columns, of ```float``` formatted numbers representing (in order, left to right):
|
||||
|
||||
1. time-stamp
|
||||
2. trick job ID
|
||||
3. user job ID
|
||||
|
||||
Each record consists of four rows in the CSV file representing the start and stop times of a job.
|
||||
|
||||
|row#|time-stamp|trick-job-id|user-job-id|
|
||||
|---:|---:|---:|---:|
|
||||
|4xRecord#+0|start-job-time|0|0|
|
||||
|4xRecord#+1|start-job-time|trick job id|user-job-id|
|
||||
|4xRecord#+2|stop-job-time|trick-job-id|user-job-id|
|
||||
|4xRecord#+3|stop-job-time|0|0|
|
||||
|
||||
If **trick** job ID is non-zero, then the **user** job ID will be zero, and vice versa.
|
||||
Within any four line record the job-ID will be recorded twice.
|
||||
|
||||
#####Example
|
||||
|
||||
The following is one four-line record from a ```log_timeline.csv``` file.
|
||||
|
||||
```
|
||||
...
|
||||
0.000026,0,0
|
||||
0.000026,16.010000,0
|
||||
0.000027,16.010000,0
|
||||
0.000027,0,0
|
||||
...
|
||||
```
|
||||
The first line of the record indicates that **some** job started at time=0.000026 seconds. It's not until the second line of the record that you find that the start time (0.000026) refers to the job with ID=16.01. The third line in the record indicates that the job with ID=16.01 stopped at 0.000027. The fourth line redundantly conveys that **some** job stopped at 0.000027. (I told you it was weird.)
|
||||
|
||||
To match the job ID's with the job names, see the ```S_job_execution``` file.
|
||||
|
||||
### Analyzing the Frame Log Files
|
||||
There are several ways we can examine/ analyze the data logged in these files.
|
||||
|
||||
@ -167,9 +211,9 @@ When you build your Trick sim, the following data-product files are created for
|
||||
#### [koviz](https://github.com/nasa/koviz)
|
||||
**koviz** Is an open-source project for plotting Trick data-recording files. It's particularly good for Monte Carlo data plotting. It also has a nifty feature for processing frame-log files. Not only can it plot them, but ```% koviz -rt RUN_directory``` will generate a report that will list the top 10 spikes in your frame log.
|
||||
|
||||
**TIP:**`` ```trick_real_time.rt_sync.rt_monitor``` (the Trick real-time monitor) should show the max job time. If not, you might have problems to solve.
|
||||
**TIP:**`` ```trick_real_time.rt_sync.rt_monitor``` (the Trick realtime monitor) should show the max job time. If not, you might have problems to solve.
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
### Other Useful Files Generated by a Trick Sim Run
|
||||
|
||||
@ -179,11 +223,11 @@ When you build your Trick sim, the following data-product files are created for
|
||||
|```RUN_*/S_run_summary ```|(Should be called build_summary) - Documents the name and path of the executable and the input file, the build time of the simulation executable, and the Trick version. It also contains the list of environment variables used when the simulation was built and the model versions.|
|
||||
|```RUN_*/send_hs ```|the end of this file contains run statistics that may be useful.|
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
### Trick Executive Scheduler
|
||||
|
||||
The
|
||||
The
|
||||
[Executive Scheduler](https://nasa.github.io/trick/documentation/simulation_capabilities/Executive-Scheduler) determines how, when, and where (which CPU) the jobs in your Trick sim are executed.
|
||||
|
||||
|
||||
@ -206,13 +250,13 @@ Ref: [RedHat: Isolating CPUs Using tuned-profiles-realtime](https://access.redha
|
||||
|
||||
#### 1.1 Trick events are computationally expensive. Use them judiciously.
|
||||
|
||||
Trick events can provide a quick, easy, way to customize the behavior of a sim, based on some condition. But, because they require Python interpretation, they are slow. They are not intended for implementation of permanent sim functionality. If they are over used, they can seriously degrade simulation performance. So, take it easy with the events.
|
||||
Trick events can provide a quick and easy way to customize the behavior of a sim, based on some condition. But, because they require Python interpretation, they are slow. They are not intended for implementation of permanent sim functionality. If they are over used, they can seriously degrade simulation performance. So, take it easy with the events.
|
||||
|
||||
See [Event Manager] (https://nasa.github.io/trick/documentation/simulation_capabilities/Event-Manager).
|
||||
|
||||
#### 1.2 Disable Trick run-time components that your sim doesn't need.
|
||||
|
||||
```default_trick_sys.sm```, the file included at the top of any Trick ```S_define``` file defines numerous "modules" that provide functionality to a Trick sim. Whereas some of these modules ( like the Executive, MemoryManager, CommandLineArguments) are required for any Trick Simulation to function, many are optional. If the modules are not needed, then disabling them can improve simulation perfromance.
|
||||
```default_trick_sys.sm```, the file included at the top of any Trick ```S_define``` file defines numerous "modules" that provide functionality to a Trick sim. Whereas some of these modules ( like the Executive, MemoryManager, CommandLineArguments) are required for any Trick Simulation to function, many are optional. If the modules are not needed, then disabling them can improve simulation performance.
|
||||
|
||||
Inserting one or more of the ```#define``` statements listed below to the top of the ```S_define```, just before the inclusion of ```default_trick_sys.sm``` will disable those modules.
|
||||
|
||||
@ -258,13 +302,13 @@ LIBRARY DEPENDENCIES:
|
||||
|
||||
class SubmarineSimObject : public Trick::SimObject {
|
||||
public:
|
||||
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
#### 1.3 Consider running Trick variable server clients and Sims on different machines.
|
||||
Trick variable server clients communicate with a simulation via a TCP/IP connection.
|
||||
The client process may, but isn't required to run on the same machine as your simulation process. On the same machine, both will competing for the same resources. This can degrade sim performance, especially when clients are rendering high-definition graphics.
|
||||
The client process may, but isn't required to, run on the same machine as your simulation process. On the same machine, both will competing for the same resources. This can degrade sim performance, especially when clients are rendering high-definition graphics.
|
||||
|
||||
|
||||
#### 1.4 Compile Trick and Trick sims with optimizations turned on
|
||||
@ -277,7 +321,7 @@ TRICK_CFLAGS += -Imodels -O2
|
||||
TRICK_CXXFLAGS += -Imodels -O2
|
||||
```
|
||||
|
||||
See:
|
||||
See:
|
||||
|
||||
* [GCC Optimization Options](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html)
|
||||
* [Clang Optimization Options](https://clang.llvm.org/docs/CommandGuide/clang.html#code-generation-options)
|
||||
@ -312,7 +356,7 @@ Suppose ```SIM_submarine```'s S_define file contains the job ```submarine.diagno
|
||||
SubmarineSimObject dyn;
|
||||
...
|
||||
```
|
||||
This job only transmits information. It doesn't effect the simulation, but does degrade real-time performance slightly. To disable it, we'll add the following to our input file:
|
||||
This job only transmits information. It doesn't affect the simulation, but does degrade realtime performance slightly. To disable it, we'll add the following to our input file:
|
||||
|
||||
```trick.exec_get_job("dyn.submarine.diagnostics", 0).disable()```
|
||||
|
||||
@ -355,11 +399,11 @@ Disk access is slow. If you need to read from disk, do it in a ```default_data``
|
||||
|
||||
#### 2.2 Try to reduce variation in job cycle times.
|
||||
Realtime performance is largely about minimizing the worst case, rather than the average case.
|
||||
The most well behaved job takes the same time, every time.
|
||||
The most well behaved job takes the same amount of time, every time.
|
||||
|
||||
|
||||
#### 2.3 Minimize dynamic memory allocation during run-time
|
||||
The time to dynamically allocate memory can vary, and in the worst-case is unbounded. This is bad for real-time performance.
|
||||
The time to dynamically allocate memory can vary, and in the worst-case is unbounded. This is bad for realtime performance.
|
||||
|
||||
|
||||
#### 2.4 Allow the compiler to help you find problems
|
||||
@ -388,14 +432,14 @@ Getting the best performance from a simulation on a multi-CPU machine requires u
|
||||
##### Uniform Memory Access (UMA)
|
||||
Uniform memory access (UMA) is a multi-processor model in which all processors share the physical memory uniformly. All memory accesses have the same latency.
|
||||
|
||||
|
||||
|
||||
|
||||
In an UMA architecture, as the number of CPUs increases, the higher tha chance that the system bus will become a bottle-neck.
|
||||
In an UMA architecture, as the number of CPUs increases, the higher the chance that the system bus will become a bottle-neck.
|
||||
|
||||
##### Non Uniform Memory Access (NUMA)
|
||||
Non-Uniform Memory Access (NUMA) is a multiprocessor model in which each processor is connected to dedicated memory but may access memory attached to other processors in the system. A NUMA architecture is one in which accesses to different addresses may have different latencies depending on where the data is stored. NUMA essentially connects UMA elements via a data-transfer interconnect. For best performance, applications should be “NUMA aware”.
|
||||
|
||||
|
||||
|
||||
|
||||
On a Linux system the following will display the available nodes, CPUs, memory, and a normalized measure of access latency between nodes.
|
||||
|
||||
@ -434,7 +478,7 @@ This computer has two NUMA nodes, each with 24 CPUs, and each with about 16 giga
|
||||
Energy efficiency and performance are usually trade-offs. Turn off any energy-efficiency settings the computer may have enabled, usually in the BIOS.
|
||||
|
||||
|
||||
#### 3.3 Fewer, Faster Cores Are Preferable to More, Slower Cores
|
||||
#### 3.3 Fewer, Faster Cores Are Preferable to More, Slower Cores
|
||||
For simulation hosts, fewer but faster cores is usually preferable to more but slower cores
|
||||
|
||||
* **Server-class** machines are designed for throughput.
|
||||
@ -442,11 +486,11 @@ For simulation hosts, fewer but faster cores is usually preferable to more but s
|
||||
|
||||
|
||||
#### 3.4 Buy More Memory
|
||||
Insufficient random access memory (RAM) leads to virtual memory page swapping, which degrades real-time performance. More RAM is one of the easiest and cheapest ways of boosting machine performance.
|
||||
Insufficient random access memory (RAM) leads to virtual memory page swapping, which degrades realtime performance. More RAM is one of the easiest and cheapest ways of boosting machine performance.
|
||||
|
||||
|
||||
#### 3.5 Avoid Intel “Efficiency” Cores
|
||||
Intel "efficiency" cores aren’t currently recognized by most Linux OS’s and will cause a lot of problems. They are more energy efficient, but slower, the opposite of what hard real-time needs. RHEL8 is unable to determine which cores are “E” (Efficiency) and which are “P” (Performance). Buy more memory. Do it!
|
||||
Intel "efficiency" cores aren’t currently recognized by most Linux OS’s and will cause a lot of problems. They are more energy efficient, but slower, the opposite of what hard realtime needs. RHEL8 is unable to determine which cores are “E” (Efficiency) and which are “P” (Performance). Buy more memory. Do it!
|
||||
|
||||
---
|
||||
### 4. Network
|
||||
@ -457,7 +501,7 @@ Isolate all sim-to-sim traffic to one network interface. Leave the other for con
|
||||
|
||||
|
||||
#### 4.2 Use One Master Clock For All Machines On The Network
|
||||
All clocks will drift apart unless periodically synchronized. Synchronization means that one of the clocks must be the reference, or "master". Multiple unsynchronized clocks in a real-time system is nightmare fuel.
|
||||
All clocks will drift apart unless periodically synchronized. Synchronization means that one of the clocks must be the reference, or "master". Multiple unsynchronized clocks in a realtime system is nightmare fuel.
|
||||
|
||||
---
|
||||
### 5. Miscellaneous
|
||||
@ -465,7 +509,7 @@ All clocks will drift apart unless periodically synchronized. Synchronization me
|
||||
|
||||
#### 5.1 Maintain a history of simulation performance.
|
||||
|
||||
Maintain a performance history of your sim as development procedes. This can be very useful evidence in solving problems. Begin frame logging the sim even before implementing real-time.
|
||||
Maintain a performance history of your sim as development procedes. This can be very useful evidence in solving problems. Begin frame logging the sim even before implementing realtime.
|
||||
|
||||
|
||||
#### 5.2 Take care when "tuning" operating system behavior
|
||||
@ -494,4 +538,4 @@ Use ```sudo``` command to give root privileges to the simulation executable usin
|
||||
```
|
||||
chown root S_main_${TRICK_HOST_CPU}.exe
|
||||
chmod 4775 S_main_${TRICK_HOST_CPU}.exe
|
||||
```
|
||||
```
|
||||
|
||||
Loading…
Reference in New Issue
Block a user