trick/docs/developer_docs/DesMasterSlave.md

Home → Developer Docs → Master-Slave

Trick Master/Slave Synchronization Design

Master/Slave synchronization is an optional class when running simulations.

Master/Slave synchronization synchronizes a master simulation to one or more slave simulations. While synchronized, the master controls the overall simulation mode as well as the current simulation time for all simulations. The set of simulations may change mode if one of the simulations in the set freezes or exits unexpectedly.

Master/Slave syncrhonization uses the Trick::MSConnect class for communications. The MSConnect class is an abstract base class that defines functions for connecting the simulations together and passing the simulation mode and simulation time between simulations. The Trick::MSSocket class derives from the Trick::MSConnect class, and uses sockets as the transport medium for master/slave communications.

Master

The master simulation starts slave simulations during initialization. At the end of each software frame during Run mode and at the end of each freeze frame during Freeze mode the master reads the mode of all connected slave simulations. The master will determine the mode of the simulations for the next frame according to the slave modes and the master mode. The new mode and the current simulation time is sent to the slaves.

Master/Slave synchronization is divided into code based on the mode of the simulation. The modes are Initialization, Run, Freeze, and Shutdown.

Master/Slave functions are designed to be called from the S_define file.

Listed below are the various modes of operation and the Master responsibilities within each.

Initialization -- Master

During Initialization slave connections are allocated and defined. Default values to start each slave using ssh are set when a slave connection is allocated. At the mimimum a slave must define a remote machine, the path to the simulation executive, and a Trick::MSConnect class to use for communications with the slave. Users may also define the remote shell to use, a simulation executable name, a remote run directory, and additional user arguments to pass to the slave. The allocation and definition of the slave connections are typically done through Trick's input processor, but may also be done in default_data or initialization jobs.

At the end of the scheduler initialization phase the master starts the slaves. For each slave the master creates a slave startup command and executes the slave startup command. This is how the slave startup command is created:

1. If the MSConnect class is not defined, return an error.
2. Begin the slave startup command with the slave remote shell.
3. Add the remote display environment variable to the startup command if one was defined.
4. Add a cd command to the slave simulation path directory to the slave starup command.
5. Add the S_main executable name to the slave startup command.
6. Add the Run Input File to the slave startup command.
7. Add additional user arguments to the slave startup command.
8. Execute the slave startup command.

The master sends its software frame to each slave.

Run -- Master

During Run mode the master reads the simulation mode of each slave. The master determines the new simulation mode and sends the new mode and the current simulation time to the slaves.

1. For each slave
   1. Read the simulation command of the slave.
   2. If the master mode command is not ExitCmd take action if the slave returns the following:
      1. If the slave command was not readable, returned ErrorCmd, and:
         1. Terminate the sim if sync_error_terminate == true.
         2. Deactivate the slave if sync_error_terminate == false.
      2. If the slave command is ExitCmd:
         1. Terminate the sim if sync_error_terminate == true.
         2. Deactivate the slave if sync_error_terminate == false.
      3. If the slave command is FreezeCmd:
         1. Freeze the master sim

After gathering the simulation command from all of the slaves, send the command to all of the slaves.

1. For each slave
   1. Write the simulation time to the slave.
   2. Write the simulation mode command to the slave.

Freeze -- Master

Upon entering freeze mode the master turns off sync_time_limit because while in freeze, master/slave communications may not be regular and periodic.

At the end of each freeze frame call Trick::Master::end_of_frame() to perform the same synchronization as in run mode.

When exiting freeze mode the master sets sync_time_limit back to the user set value ready for running again.

Shutdown -- Master

The master tells the slaves that it is shutting down.

1. For each slave
   1. Write the simulation time to the slave.
   2. Write the simulation mode command ExitCmd to the slave.

Slave

At the end of each software frame during Run mode and at the end of each freeze frame during Freeze mode slave simulations send the master the current mode of the slave. The slaves wait for the new mode and simulation time from the master.

Initialization -- Slave

The Trick::Slave class is enabled by calling the MSConnect::process_sim_args() routine. If the MSConnect::process_sim_args() finds command line arguments specifying this simulation is a slave, then the Trick::Slave is enabled.

At the end of the scheduler initialization phase the slave connects to the master.

1. Call Trick::MSConnect::connect() to connect the slave to the master.
2. Search for a RealtimeSync class. If one exists, turn the object off.
3. Search for a Master class. If one exists, turn the object off.
4. Read and set the slave's software frame from the master.

Run -- Slave

In Run mode the slave writes its next executive command to the master. The master will return the overall next simulation mode to the slave. The slave will also read the current simulation time from the master.

1. Write the simulation command to the master.
2. Read the simulation time from the master.
3. if the simulation time read was not successful:
   1. Terminate the sim if sync_error_terminate == true.
   2. Disable master/slave communication and command the sim to freeze if sync_error_terminate == false.
4. If the simulation time read was successful set the slave simulation time to the time of the master.
5. Read the simulation command from the master.
6. If the simulation command was not successful:
   1. Terminate the sim if sync_error_terminate == true.
   2. Disable master/slave communication and command the sim to freeze if sync_error_terminate == false.
7. If the simulation command read was successful set the slave simulation command to the command of the master.

Freeze -- Slave

Upon entering freeze mode the slave turns off sync_time_limit because while in freeze, master/slave communications may not be regular and periodic.

At the end of each freeze frame call Trick::Slave::end_of_frame() to perform the same synchronization as in run mode.

When exiting freeze mode the slave sets sync_time_limit back to the user set value ready for running again.

Shutdown -- Slave

If a slave is shutting down, inform the master that it is exiting. [@anchor d_slave_shutdown_0 d_slave_shutdown_0]