trick/docs/documentation/building_a_simulation/Trickified-Project-Libraries.md

During a simulation build, Trick generates several rounds of files to support data recording, checkpointing, and Python access:

• Trick generates S_source.hh from the S_define
• ICG recursively builds a tree of all header files included from S_source.hh and generates an io_*.cpp and py_*.i file for each
• SWIG converts all py_*.i to py_*.cpp files
• Trick compiles all io_*.cpp and py_*.cpp files

The time required grows with the number of included header files and can represent a significant portion of the build process. During subsequent builds, only headers files that have changed are reprocessed. However, make clean removes all of the generated files, so the next build will have to run the entire process again.

For external libraries, which do not change from build to build, this is unnecessary and wasteful. In this case, we would like to compile the io_*.cpp and py_*.cpp files once and simply link against them during simulation build. As of version 17.1, Trick supports "Trickifying" a set of headers via the trickify.mk makefile.

Who's Responsible?

Support of Trickification is the project owner's responsibility. While it is possible for anyone to Trickify any set of headers, you really want the project to maintain the files we're going to talk about. This will ensure that's it done correctly, in one place, and stays synchronized with the project.

Trickifying Your Project

Trickifying your project requires a list of headers and the paths at which they can be found. This list should include the headers for which you want data recording, checkpointing, and Python access, which probably means most or even all of them. Specify the list by creating a file named S_source.hh that includes the headers you want processed and nothing else. This is not the same S_source.hh generated by Trick during the simulation build process, but Trick's tools expect that name, so you have to use it for now. Specify the header paths via the TRICKIFY_CXX_FLAGS variable, which you can set when you call make or export from your own makefile. I don't recommend setting it in your shell via export or setenv. All of Trick's environment variables can be specified at compile time, exposed at run time, and limited to the duration of the executable. There's no reason to permanently pollute your environment from your .cshrc, .bashrc, etc.

For example, say we have a tiny project that looks like this:

${HOME}/myproject/
    include/
        Foo.hh
        Bar.hh
        Baz.hh

To Trickify this project, we'll make a file called S_source.hh which includes all three headers:

#include "Foo.hh"
#include "Bar.hh"
#include "Baz.hh"

I recommend putting your S_source.hh in its own directory since Trick is going to generate a bunch of files, for some of which you can't yet specify the output directory. Let's call it trickified:

${HOME}/myproject/
    include/
        Foo.hh
        Bar.hh
        Baz.hh
    trickified/
        S_source.hh

In the trickified directory, run:

make -f ${TRICK_HOME}/share/trick/makefiles/trickify.mk TRICKIFY_CXX_FLAGS=-I${HOME}/myproject/include

The result should be:

${HOME}/myproject/
    include/
        Foo.hh
        Bar.hh
        Baz.hh
    trickified/
        S_source.hh
        trickified.o
        build/
        python/

trickified.o contains all of the io_*.cpp and py_*.cpp code. The name is configurable via the TRICKIFY_OBJECT_NAME variable, which can include directories, which will automatically be created if necessary. build contains a lot of ICG and SWIG artifacts. You can't change its name or location at this time, but it's useful to keep around as it will allow you to rebuild only the parts of the project that change in the future. python includes a bunch of crazily-named Python modules which serve as the input file interface to the content of the header files. You can configure its location via the TRICKIFY_PYTHON_DIR variable. Directories will be automatically created as needed.

Your Trickified library can be produced in three different formats based on the value of TRICKIFY_BUILD_TYPE:

1. STATIC (.a)
   Create a static library. This will require the use of --whole-archive (on Linux) or -all_load/-force_load (on Mac) when linking the sim executable. Trick uses dlsym to dynamically load symbols at run time, but the linker, by default, will not include symbols from static libraries that are not known to be needed at compile time.
2. SHARED (.so)
   Create a shared object (dynamically linked library). This may require the use of -rpath to ensure the linker can find the shared object at runtime unless you explicitly link against it (as opposed to using -L and -l) during compilation.
3. PLO (.o)
   Create a partially-linked object (see the --relocatable option of ld). No special linker options are required. This is the default build type.

Note that Trick does not automatically append file extensions and will use the value of TRICKIFY_OBJECT_NAME regardless of the value of TRICKIFY_BUILD_TYPE.

Simplify with a Makefile

Let's be honest. You're not going to remember that command line. And who wants to type all that stuff every time? Let's do it once in our own makefile and just call make on that. It seems sensible to put this in the trickified directory right next to S_source.hh.

${HOME}/myproject/
    include/
        Foo.hh
        Bar.hh
        Baz.hh
    trickified/
        Makefile
        S_source.hh
        trickified.o
        build/
        python/

ifndef TRICK_HOME
    $(error TRICK_HOME must be set)
endif

TRICKIFY := $(TRICK_HOME)/share/trick/makefiles/trickify.mk

ifeq ($(wildcard $(TRICKIFY)),)
    $(error This makefile requires at least Trick 17.1)
endif

export TRICKIFY_OBJECT_NAME := trickified_myproject.o
export TRICKIFY_CXX_FLAGS := -I$(HOME)/myproject/include

all:
        @$(MAKE) -s -f $(TRICKIFY)

clean:
        @rm -rf build python $(TRICKIFY_OBJECT_NAME)

Now just type make in trickified and everything is taken care of. I even added a check to make sure you're using a recent enough version of Trick. I've silenced a lot of make's output because I prefer to see echoed commands only when debugging, but you're welcome to get rid of the @ and -s if you enjoy such verbosity. Note that I've used TRICKIFY_OBJECT_NAME to rename the default trickified.o to something a little less generic. If you're following along, you can remove the trickified.o we built earlier.

Don't Version Control Build Artifacts!

The only Trickification-related files you want under version control are S_source.hh and Makefile. You should ignore all of the generated files. For instance, the appropriate .gitignore for the trickified directory when using default values for the Trickification variables is:

build
python
trick
*.o

The generated Python modules can be particularly problematic if they are accidentally version controlled. The names are created by hashing the full file path, both during Trickification and again when the sim is built. If the paths change between Trickification and sim compilation, the names won't match, and you'll get confusing linker errors.

Using a Trickified Project

Using a project that's been Trickified is a lot like using an external library, but there are a couple of extra things to take care of. Continuing with the example above, we would need to add the following to our sim's S_overrides.mk:

TRICK_LDFLAGS += $(HOME)/myproject/trickified/trickified_myproject.o

This line links in the Trickified object. Note that you may need additional flags if you used TRICKIFY_BUILD_TYPE to build a static library or shared object.

TRICK_EXT_LIB_DIRS += :$(HOME)/myproject

This line tells Trick to expect io_* and py_* code for the headers in the specified directory (and all directories below it), but not to generate it itself. This is different than TRICK_ICG_EXCLUDE and TRICK_EXCLUDE, which cause ICG to ignore the headers entirely. It also tells Trick not to compile any source files in the specified directory (and all directories below it) that may be referenced as LIBRARY_DEPENDENCIES in user files. Note that it is a colon-delimited list of paths.

You'll need to be more selective if the sim itself or additional non-Trickified headers or source are under the same directory as Trickified headers or source. You may have to resort to individually specifying the full path to every file to be excluded, perhaps using some fancy find options to automatically generate the list.

TRICK_PYTHON_PATH += :$(HOME)/myproject/trickified/python

This line tells Trick where to find the Python modules generated by SWIG so that you can access the Trickified project from the input file. It is also a colon-delimited list of paths.

TRICK_SWIG_FLAGS += -I$(HOME)/myproject/trickified

This line tells Trick where to find the py_*.i files generated by ICG and should point to the directory containing build. These are necessary if any of your headers include headers from the Trickified project, which is likely, since you otherwise wouldn't be using it. It is a space-delimited list of options. Don't forget to prepend the path with -I.

Simplify Your Users' Lives

While the above is sufficient to use a Trickified project, it's awfully inconvenient to have to add all that stuff to every sim's S_overrides.mk. Plus, there are probably more things that need to be added, like header paths and linker flags for any additional libraries on which the project depends. Your users will love your project even more if you provide them with a makefile they can simply include from their S_overrides.mk. And it's not just the users that benefit! You'll have to answer far fewer questions about why they can't get your project compiled into their sim if, when your project inevitably changes, all they have to do is pull down the new makefile instead of changing all of their S_overrides.mk. Everybody wins! Turning once again to our example, let's call the makefile myproject.mk and put it in the trickified directory. You may have a name or location that makes more sense for your project. Maybe you already have a makefiles directory, or maybe your project supports a number of third party tools and you have a directory for Trick support. But let's keep the example simple:

${HOME}/myproject/
    include/
        Foo.hh
        Bar.hh
        Baz.hh
    trickified/
        Makefile
        myproject.mk
        S_source.hh
        trickified_myproject.o
        build/
        python/

Here's the contents of myproject.mk. It's everything from the previous section plus some other things you might find useful.

# We know this file's position relative to the root directory of the project,
# and MAKEFILE_LIST will give us the full path to this file no matter where the
# user has installed this project.
export MYPROJECT_HOME := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/..)

# Make MYPROJECT_HOME available to the sim at run time. This isn't necessary,
# but it can be useful if your users need to reference something from your
# project at run time.
TRICK_GTE_EXT += MYPROJECT_HOME

# Specify include paths for your headers.
MYPROJECT_INCLUDE := -I$(MYPROJECT_HOME)/include

# Specify include paths for your source, which users will need if they list
# any of your project's source files in a LIBRARY_DEPENDENCIES section.
MYPROJECT_SOURCE := -I$(MYPROJECT_HOME)/source

# Users may set different flags for C and C++, so you should really modify both
# to be safe.
TRICK_CFLAGS   += $(MYPROJECT_INCLUDE) $(MYPROJECT_SOURCE)
TRICK_CXXFLAGS += $(MYPROJECT_INCLUDE) $(MYPROJECT_SOURCE)

# Enable Trickification support if Trick >= 17.1.
# Otherwise, let Trick generate all of the io_* and py_* code as usual.
ifneq ($(wildcard $(TRICK_HOME)/share/trick/makefiles/trickify.mk),)

    MYPROJECT_TRICK := $(MYPROJECT_HOME)/trickified/trickified_myproject.o

    # Tell Trick the headers and source at this location are part of a
    # Trickified project
    TRICK_EXT_LIB_DIRS += :$(MYPROJECT_HOME)

    # Tell Trick where to find the Python modules generated by SWIG
    TRICK_PYTHON_PATH += :$(MYPROJECT_HOME)/trickified/python

    # Tell SWIG where to find py_*.i files
    TRICK_SWIG_FLAGS += -I$(MYPROJECT_HOME)/trickified

    # Link in the Trickified object
    TRICK_LDFLAGS += $(MYPROJECT_TRICK)

    # Append a prerequisite to the $(SWIG_SRC) target. This will build the
    # Trickified library along with the sim if it does not already exist. Using
    # $(SWIG_SRC) ensures that all Trickified .i files are created before SWIG is
    # run on any simulation .i files, which may %import them. Note that this does
    # NOT cause the Trickified library to be rebuilt if it already exists, even if
    # the Trickified source code has changed.
    $(SWIG_SRC): $(MYPROJECT_TRICK)

endif

$(MYPROJECT_TRICK):
        @$(MAKE) -s -C $(MYPROJECT_HOME)/trickified

Now to use your project, all the user has to do is add one line to his S_overrides.mk:

include <myproject>/trickified/myproject.mk

They'll have to replace <myproject> with the location to which they installed your project, of course. They might choose to hardcode the path or use a variable, but that's up to them.

You Still Need a Core Library

Trickification is great and all, but it only builds the io_* and py_* code into an object. And because your project's headers and source are now under TRICK_EXT_LIB_DIRS, Trick won't be determining dependencies or compiling source code. Trickification thus necessitates that you build all of your source code into a library. There are plenty of internet tutorials available on that topic, so I won't be suggesting anything here. But once you've got that taken care of, you should incorporate it into your user-facing makefile by adding it to TRICK_LDFLAGS. You can also create a rule to call its build system and add the library as a prerequisite to $(S_MAIN) to have it built along with the sim if necessary.

A Real Life Trickified Project

Here's a real project we used as the guinea pig for Trickification. It provides a makefile that a user can include from his S_overrides.mk that causes both a core library and Trickified object to be built if they don't already exist whenever the sim is compiled. The makefile is located at 3rdParty/trick/makefiles/trickified.mk. The Trickified stuff is at 3rdParty/trick/lib.

https://github.com/nasa/IDF

Continue to Running A Simulation