docs: Add strategies for assembling root filesystems.

Expand the documentation for using a crosstool-NG-generated toolchain for
building a root filesystem for a target device.

Signed-off-by: "Trevor Woerner" <twoerner@gmail.com>
yann.morin.1998@anciens.enib.fr: some eye-candy]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>

docs/5 - Using the toolchain.txt

@@ -25,16 +25,139 @@ and so on...
 not the host of the toolchain; and 'build' refers to the machine where
 you build your program, that is the host of the toolchain.)
 
-It is strongly advised not to use the toolchain sysroot directory as an
-install directory for your programs/packages. If you do so, you will not be
-able to use your toolchain for another project. It is even strongly advised
-that your toolchain is chmod-ed to read-only once successfully build, so that
-you don't go polluting your toolchain with your programs/packages' files.
 
-Thus, when you build a program/package, install it in a separate directory,
+Assembling a root filesystem  /
-eg. /your/root. This directory is the /image/ of what would be in the root file
+_____________________________/
-system of your target, and will contain all that your programs/packages have
+
-installed.
+
+Assembling a root filesystem for a target device requires the successive
+building of a set of software packages for the target architecture. Building
+a package potentially requires artifacts which were generated as part of an
+earlier build. Note that not all artifacts which are installed as part of a
+package are desirable on a target's root filesystem (e.g. man/info files,
+include files, etc.). Therefore we must distinguish between a 'staging'
+directory and a 'rootfs' directory.
+
+A 'staging' directory is a location into which we install all the build
+artifacts. We can then point future builds to this location so they can find
+the appropriate header and library files. A 'rootfs' directory is a location
+into which we place only the files we want to have on our target.
+
+There are four schools of thought here:
+
+1) Install directly into the sysroot of the toolchain.
+
+   By default (i.e. if you don't pass any arguments to the tools which
+   would change this behaviour) the toolchain that is built by
+   crosstool-NG will only look in its toolchain directories for system
+   header and library files:
+
+#include "..." search starts here:
+#include <...> search starts here:
+<ct-ng install path>/lib/gcc/<host tuple>/4.5.2/include
+<ct-ng install path>/lib/gcc/<host tuple>/4.5.2/include-fixed
+<ct-ng install path>/lib/gcc/<host tuple>/4.5.2/../../../../<host tuple>/include
+<ct-ng install path>/<host tuple>/sysroot/usr/include
+
+   In other words, the compiler will automagically find headers and
+   libraries without extra flags if they are installed under the
+   toolchain's sysroot directory.
+
+   However, this is bad because the toolchain gets poluted, and can
+   not be re-used.
+
+   $ ./configure --build=<build tuple> --host=<host tuple> \
+         --prefix=/usr --enable-foo-bar...
+   $ make
+   $ make DESTDIR=/<ct-ng install path>/<host tuple>/sysroot install
+
+2) Copy the toolchain's sysroot to the 'staging' area.
+
+   If you start off by copying the toolchain's sysroot directory to your
+   staging area, you can simply proceed to install all your packages'
+   artifacts to the same staging area. You then only need to specify a
+   '--sysroot=<staging area>' option to the compiler of any subsequent
+   builds and all your required header and library files will be found/used.
+
+   This is a viable option, but requires the user to always specify CFLAGS
+   in order to include --sysroot=<staging area>, or requires the use of a
+   wrapper to a few select tools (gcc, ld...) to pass this flag.
+
+   Instead of polluting the toolchain's sysroot you are copying its contents
+   to a new location and polluting the contents in that new location. By
+   specifying the --sysroot option you're effectively abandoning the default
+   sysroot in favour of your own.
+
+   Incidentally this is what buildroot does using a wrapper, when using an
+   external toolchain.
+
+   $ cp -a $(<host tuple>-gcc --your-cflags-except-sysroot -print-sysroot) \
+         /path/to/staging
+   $ ./configure --build=<build tuple> --host=<host tuple>          \
+                 --prefix=/usr --enable-foo-bar...                  \
+                 CC="<host tuple>-gcc --syroot=/path/to/staging"    \
+                 CXX="<host tuple>-g++ --sysroot=/path/to/staging"  \
+                 LD="<host tuple>-ld --sysroot=/path/to/staging"    \
+                 AND_SO_ON="tuple-andsoon --sysroot=/path/to/staging"
+   $ make
+   $ make DESTDIR=/path/to/staging install
+
+3) Use separate staging and sysroot directories.
+
+   In this scenario you use a staging area to install programs, but you do
+   not pre-fill that staging area with the toolchain's sysroot. In this case
+   the compiler will find the system includes and libraries in its sysroot
+   area but you have to pass appropriate CPPFLAGS and LDFLAGS to tell it
+   where to find your headers and libraries from your staging area (or use
+   a wrapper).
+
+   $ ./configure --build=<build tuple> --host=<host tuple>          \
+                 --prefix=/usr --enable-foo-bar...                  \
+                 CPPFLAGS="-I/path/to/staging/usr/include"          \
+                 LDFLAGS="-L/path/to/staging/lib -L/path/to/staging/usr/lib"
+   $ make
+   $ make DESTDIR=/path/to/staging install
+
+4) A mix of 2) and 3), using carefully crafted union mounts.
+
+   The staging area is a union mount of:
+      - the sysroot as a read-only branch
+      - the real staging area as a read-write branch
+   This also requires passing --sysroot to point to the union mount, but has
+   other advantages, such as allowing per-package staging, and a few more
+   obscure pros. It also has its disadvantages, as it potentially requires
+   non-root users to create union mounts. Additionally, union mounts are not
+   yet mainstream in the Linux kernel, so it requires patching. There is a
+   FUSE-based unionfs implementation, but development is almost stalled,
+   and there are a few gotchas...
+
+   $ (good luck!)
+
+
+It is strongly advised not to use the toolchain sysroot directory as an
+install directory (i.e. option 1) for your programs/packages. If you do so,
+you will not be able to use your toolchain for another project. It is even
+strongly advised that your toolchain is chmod-ed to read-only once
+successfully install, so that you don't go polluting your toolchain with
+your programs'/packages' files. This can be achieved by selecting the
+"Render the toolchain read-only" from crosstool-NG's "Paths and misc options"
+configuration page.
+
+Thus, when you build a program/package, install it in a separate, staging,
+directory and let the cross-toolchain continue to use its own, pristine,
+sysroot directory.
+
+When you are done building and want to assemble your rootfs you could simply
+take the full contents of your staging directory and use the 'populate'
+script to add in the necessary files from the sysroot. However, the staging
+area you have created will include lots of build artifacts that you won't
+necessarily want/need on your target. For example: static libraries, header
+files, linking helper files, man/info pages. You'll also need to add various
+configuration files, scripts, and directories to the rootfs so it will boot.
+
+Therefore you'll probably end up creating a separate rootfs directory which
+you will populate from the staging area, necessary extras, and then use
+crosstool-NG's populate script to add the necessary sysroot libraries.
 
 
 The 'populate' script |