crosstool-ng/docs/8 - Internals.txt

File.........: 8 - Internals.txt
Copyright....: (C) 2010 Yann E. MORIN <yann.morin.1998@free.fr>
License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5


Internals  /
__________/


Internally, crosstool-NG is script-based. To ease usage, the frontend is
Makefile-based.


Makefile front-end |
-------------------+

The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
script with an action will act exactly as if the Makefile was in the current
working directory and make was called with the action as rule. Thus:
  ct-ng menuconfig

is equivalent to having the Makefile in CWD, and calling:
  make menuconfig

Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
traditional command.

ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
at configuration time with ./configure.

ct-ng also searches for config files, sub-tools, samples, scripts and patches in
that library directory.

Because of a stupid make behavior/bug I was unable to track down, implicit make
rules are disabled: installing with --local would trigger those rules, and mconf
was unbuildable.


Kconfig parser |
---------------+

The kconfig language is a hacked version, vampirised from the Linux kernel
(http://www.kernel.org/), and (heavily) adapted to my needs.

The list of the most notable changes (at least the ones I remember) follows:
- the CONFIG_ prefix has been replaced with CT_
- a leading | in prompts is skipped, and subsequent leading spaces are not
  trimmed; otherwise leading spaces are silently trimmed
- removed the warning about undefined environment variable

The kconfig parsers (conf and mconf) are not installed pre-built, but as
source files. Thus you can have the directory where crosstool-NG is installed,
exported (via NFS or whatever) and have clients with different architectures
use the same crosstool-NG installation, and most notably, the same set of
patches.


Architecture-specific |
----------------------+

Note: this chapter is not really well written, and might thus be a little bit
complex to understand. To get a better grasp of what an architecture is, the
reader is kindly encouraged to look at the "arch/" sub-directory, and to the
existing architectures to see how things are laid out.

An architecture is defined by:

 - a human-readable name, in lower case letters, with numbers as appropriate.
   The underscore is allowed; space and special characters are not.
     Eg.: arm, x86_64
 - a file in "config/arch/", named after the architecture's name, and suffixed
   with ".in".
     Eg.: config/arch/arm.in
 - a file in "scripts/build/arch/", named after the architecture's name, and
   suffixed with ".sh".
     Eg.: scripts/build/arch/arm.sh

The architecture's ".in" file API:
 > the config option "ARCH_%arch%" (where %arch% is to be replaced with the
   actual architecture name).
   That config option must have *neither* a type, *nor* a prompt! Also, it can
   *not* depend on any other config option (EXPERIMENTAL is managed as above).
     Eg.:
       config ARCH_arm
   + mandatory:
       defines a (terse) help entry for this architecture:
       Eg.:
         config ARCH_arm
           help
             The ARM architecture.
   + optional:
       selects adequate associated config options.
       Note: 64-bit architectures *shall* select ARCH_64
       Eg.:
         config ARCH_arm
           select ARCH_SUPPORTS_BOTH_ENDIAN
           select ARCH_DEFAULT_LE
           help
             The ARM architecture.
       Eg.:
         config ARCH_x86_64
            select ARCH_64
            help
              The x86_64 architecture.

 > other target-specific options, at your discretion. Note however that to
   avoid name-clashing, such options shall be prefixed with "ARCH_%arch%",
   where %arch% is again replaced by the actual architecture name.
   (Note: due to historical reasons, and lack of time to clean up the code,
    I may have left some config options that do not completely conform to
    this, as the architecture name was written all upper case. However, the
    prefix is unique among architectures, and does not cause harm).

The architecture's ".sh" file API:
 > the function "CT_DoArchTupleValues"
   + parameters: none
   + environment:
     - all variables from the ".config" file,
     - the two variables "target_endian_eb" and "target_endian_el" which are
       the endianness suffixes
   + return value: 0 upon success, !0 upon failure
   + provides:
     - mandatory
     - the environment variable CT_TARGET_ARCH
     - contains:
       the architecture part of the target tuple.
       Eg.: "armeb" for big endian ARM
            "i386" for an i386
   + provides:
     - optional
     - the environment variable CT_TARGET_SYS
     - contains:
       the system part of the target tuple.
       Eg.: "gnu" for glibc on most architectures
            "gnueabi" for glibc on an ARM EABI
     - defaults to:
       - for glibc-based toolchain: "gnu"
       - for uClibc-based toolchain: "uclibc"
   + provides:
     - optional
     - the environment variables to configure the cross-gcc (defaults)
       - CT_ARCH_WITH_ARCH    : the gcc ./configure switch to select architecture level         ( "--with-arch=${CT_ARCH_ARCH}"   )
       - CT_ARCH_WITH_ABI     : the gcc ./configure switch to select ABI level                  ( "--with-abi=${CT_ARCH_ABI}"     )
       - CT_ARCH_WITH_CPU     : the gcc ./configure switch to select CPU instruction set        ( "--with-cpu=${CT_ARCH_CPU}"     )
       - CT_ARCH_WITH_TUNE    : the gcc ./configure switch to select scheduling                 ( "--with-tune=${CT_ARCH_TUNE}"   )
       - CT_ARCH_WITH_FPU     : the gcc ./configure switch to select FPU type                   ( "--with-fpu=${CT_ARCH_FPU}"     )
       - CT_ARCH_WITH_FLOAT   : the gcc ./configure switch to select floating point arithmetics ( "--with-float=soft" or /empty/  )
   + provides:
     - optional
     - the environment variables to pass to the cross-gcc to build target binaries (defaults)
       - CT_ARCH_ARCH_CFLAG   : the gcc switch to select architecture level                     ( "-march=${CT_ARCH_ARCH}"            )
       - CT_ARCH_ABI_CFLAG    : the gcc switch to select ABI level                              ( "-mabi=${CT_ARCH_ABI}"              )
       - CT_ARCH_CPU_CFLAG    : the gcc switch to select CPU instruction set                    ( "-mcpu=${CT_ARCH_CPU}"              )
       - CT_ARCH_TUNE_CFLAG   : the gcc switch to select scheduling                             ( "-mtune=${CT_ARCH_TUNE}"            )
       - CT_ARCH_FPU_CFLAG    : the gcc switch to select FPU type                               ( "-mfpu=${CT_ARCH_FPU}"              )
       - CT_ARCH_FLOAT_CFLAG  : the gcc switch to choose floating point arithmetics             ( "-msoft-float" or /empty/           )
       - CT_ARCH_ENDIAN_CFLAG : the gcc switch to choose big or little endian                   ( "-mbig-endian" or "-mlittle-endian" )
     - default to:
       see above.
   + provides:
     - optional
     - the environment variables to configure the core and final compiler, specific to this architecture:
       - CT_ARCH_CC_CORE_EXTRA_CONFIG   : additional, architecture specific core gcc ./configure flags
       - CT_ARCH_CC_EXTRA_CONFIG        : additional, architecture specific final gcc ./configure flags
     - default to:
       - all empty
   + provides:
     - optional
     - the architecture-specific CFLAGS and LDFLAGS:
       - CT_ARCH_TARGET_CLFAGS
       - CT_ARCH_TARGET_LDFLAGS
     - default to:
       - all empty

You can have a look at "config/arch/arm.in" and "scripts/build/arch/arm.sh" for
a quite complete example of what an actual architecture description looks like.


Kernel specific |
----------------+

A kernel is defined by:

 - a human-readable name, in lower case letters, with numbers as appropriate.
   The underscore is allowed; space and special characters are not (although
   they are internally replaced with underscores.
     Eg.: linux, bare-metal
 - a file in "config/kernel/", named after the kernel name, and suffixed with
   ".in".
     Eg.: config/kernel/linux.in, config/kernel/bare-metal.in
 - a file in "scripts/build/kernel/", named after the kernel name, and suffixed
   with ".sh".
     Eg.: scripts/build/kernel/linux.sh, scripts/build/kernel/bare-metal.sh

The kernel's ".in" file must contain:
 > an optional lines containing exactly "# EXPERIMENTAL", starting on the
   first column, and without any following space or other character.
   If this line is present, then this kernel is considered EXPERIMENTAL,
   and correct dependency on EXPERIMENTAL will be set.

 > the config option "KERNEL_%kernel_name%" (where %kernel_name% is to be
   replaced with the actual kernel name, with all special characters and
   spaces replaced by underscores).
   That config option must have *neither* a type, *nor* a prompt! Also, it can
   *not* depends on EXPERIMENTAL.
     Eg.: KERNEL_linux, KERNEL_bare_metal
   + mandatory:
       defines a (terse) help entry for this kernel.
       Eg.:
         config KERNEL_bare_metal
           help
             Build a compiler for use without any kernel.
   + optional:
       selects adequate associated config options.
       Eg.:
         config KERNEL_bare_metal
           select BARE_METAL
           help
             Build a compiler for use without any kernel.

 > other kernel specific options, at your discretion. Note however that, to
   avoid name-clashing, such options should be prefixed with
   "KERNEL_%kernel_name%", where %kernel_name% is again tp be replaced with
   the actual kernel name.
   (Note: due to historical reasons, and lack of time to clean up the code,
    I may have left some config options that do not completely conform to
    this, as the kernel name was written all upper case. However, the prefix
    is unique among kernels, and does not cause harm).

The kernel's ".sh" file API:
 > is a bash script fragment

 > defines the function CT_DoKernelTupleValues
   + see the architecture's CT_DoArchTupleValues, except for:
   + set the environment variable CT_TARGET_KERNEL, the kernel part of the
     target tuple
   + return value: ignored

 > defines the function "do_kernel_get":
   + parameters: none
   + environment:
      - all variables from the ".config" file.
   + return value: 0 for success, !0 for failure.
   + behavior: download the kernel's sources, and store the tarball into
     "${CT_TARBALLS_DIR}". To this end, a functions is available, that
     abstracts downloading tarballs:
     - CT_DoGet <tarball_base_name> <URL1 [URL...]>
       Eg.: CT_DoGet linux-2.6.26.5 ftp://ftp.kernel.org/pub/linux/kernel/v2.6
     Note: retrieving sources from svn, cvs, git and the likes is not supported
     by CT_DoGet. For now, you'll have to do this by hand.

 > defines the function "do_kernel_extract":
   + parameters: none
   + environment:
      - all variables from the ".config" file,
   + return value: 0 for success, !0 for failure.
   + behavior: extract the kernel's tarball into "${CT_SRC_DIR}", and apply
     required patches. To this end, a function is available, that abstracts
     extracting tarballs:
     - CT_ExtractAndPatch <tarball_base_name>
       Eg.: CT_ExtractAndPatch linux-2.6.26.5

 > defines the function "do_kernel_headers":
   + parameters: none
   + environment:
      - all variables from the ".config" file,
   + return value: 0 for success, !0 for failure.
   + behavior: install the kernel headers (if any) in "${CT_SYSROOT_DIR}/usr/include"

 > defines any kernel-specific helper functions
   These functions, if any, must be prefixed with "do_kernel_%CT_KERNEL%_",
   where '%CT_KERNEL%' is to be replaced with the actual kernel name, to avoid
   any name-clashing.

You can have a look at "config/kernel/linux.in" and "scripts/build/kernel/linux.sh"
as an example of what a complex kernel description looks like.


Adding a new version of a component |
------------------------------------+

When a new component, such as the Linux kernel, gcc or any other is released,
adding the new version to crosstool-NG is quite easy. There is a script that
will do all that for you:
  scripts/addToolVersion.sh

Run it with no option to get some help.


Build scripts |
--------------+

To Be Written later...