Apply the approach to join the argument list 'args' to pass a single
list argument to _collect_from_depot. Actually, this story teaches me to
shy away from {*} because of its special semantics, which are
{*} makes each item in a list an individual argument of the current
command (https://wiki.tcl.tk/17158)
This is a direct follow up to "run: let import_from_depot accept list
variables", which broke calling import_from_depot with a number of
individual arguments.
Issue #2619
When doing something like this in a run script ...
lappend arg X
lappend arg Y
lappend arg Z
import_from_depot arg
... the internals of import_from_depot do not treat $arg as list but as one
string and interprets it as malformed depot path. We can handle this by
applying {*} to $arg inside the procedure.
Issue #2619
This patch changes the depot layout such that each archive is
represented as a directory that contains the versions of the archive as
subdirectories.
Issue #2610
The run tool now by default checks configurations with target-specific
XML schemata. Each component may define a config schema file in its
target.mk via the CONFIG_XSD variable. When the run tool has checked an
configuration of an init instance, it additionally goes through the
start nodes of the config. For each start node it checks whether there
is an XSD file that matches. If so, the run tool also checks the config
of the start node (if existant). This is done recursively. I.e., also
the child configs of a sub-init of a sub-init of the top-level init
receive a config check.
Issue #2600
If there is no debug/core-<kernel>.o library available, e.g., when
running a scenario in a fresh build directory where core is provided
via a depot package, the build_core_image step of the run tool was
canceled in the past, when trying to access the non-existing library.
This commit changes the semantic of build_core_image to link a core
binary with debug symbols only when the designated library is in place.
This fixup is related to issue #2339
* introduce new syscall (core-only) to create privileged threads
* take the privilege level of the thread into account
when doing a context switch
* map kernel segment as accessable for privileged code only
Ref #2091
* introduces central memory map for core/kernel
* on 32-bit platforms the kernel/core starts at 0x80000000
* on 64-bit platforms the kernel/core starts at 0xffffffc000000000
* mark kernel/core mappings as global ones (tagged TLB)
* move the exception vector to begin of core's binary,
thereby bootstrap knows from where to map it appropriately
* do not map boot modules into core anymore
* constrain core's virtual heap memory area
* differentiate in between user's and core's main thread's UTCB,
which now resides inside the kernel segment
Ref #2091
In the image.elf file, the very last boot module is followed
by arbitrary other core-local data. Since those boot modules are
exported as page-granular dataspace to the outside of core via core's
ROM service, we need to ensure that the last page is padded with zeros.
When building Genode on a Linux system running in a Xen Dom0, the 'xen'
run target can run a Genode scenario in a Xen DomU.
Usage: in build/x86_*/etc/build.conf, define:
RUN_OPT = --include boot_dir/$(KERNEL) --include image/iso --include power_on/xen --include log/xen --include power_off/xen
The Xen DomU runs in HVM mode and loads Genode from an ISO image. Serial
log output is printed to the console and graphical output is shown in an
SDL window.
The Xen DomU ist managed using the 'xl' command line tool and it is
possible to add configuration options in the 'xen_args' variable in a run
script. Common options are:
- disabling the graphical output:
append xen_args { sdl="0" }
- configuring a network device:
append xen_args { vif=\["model=e1000,mac=02:00:00:00:01:01,bridge=xenbr0"\] }
- configuring USB input devices:
append xen_args { usbdevice=\["mouse","keyboard"\] }
Note: the 'xl' tool requires super-user permissions and interactive
password input can be troublesome in combination with 'expect' and is not
practical for automatic tests. For this reason, the current implementation
assumes that no password input is needed when running 'sudo xl', which can
be achieved by creating a file '/etc/sudoers.d/xl' with the content
'user ALL=(root) NOPASSWD: /usr/sbin/xl'
(where 'user' is the Linux user name).
Fixes#2504
This patch enables one to supply a list of arguments to
'install_config'. The arguments appear concatenated in the resulting
config file. This is useful for embedding dynamically generated content
into an otherwise static configuration. For example, in the following
config, the 'pkg' attribute of the '<query>' config node is defined by
the return value of the 'query_pkg' function:
install_config {
<config>
...
<start name="depot_query">
...
<config>
<query pkg="genodelabs/pkg/} [query_pkg] {"/>
</config>
</start>
</config>}
relocation is required, to avoid overlapping phdr in the elf binaries, but
sel4 will fail to boot if bender relocates the images at the end of physical
memory. Limit the physical relocation address to a fixed 256M value, so that
sel4 will work.
Issue #2451
The former use of 'arm_v7' as architecture was too unspecific. Depot
binary archives for Cortex-A8/A9 platforms should be generated for
the 'arm_v7a' architecture to define the required compile flags
(i.e., -march).
We do not have debug symbols in the bin/ binaries anymore. Thus, use the
debug/ binaries instead. We now have kernel specific binary names for
ld.lib.so and some others. Adapt, to this fact as well. For doing so
without unnecessary output, provide a new parameter "silent" at the
"kernel_specific_binary" procedure.
Ref #2398
Previously, the debugging version of the core image
(<build_dir>/var/run/<run_name>.core) was created by the run tool from
the common core object file
(<build_dir>/var/run/<run_name>/core-<kernel>.o). This object file,
however, was already stripped by the build system and thus didn't
contain debug info anymore. The build system also creates an unstripped
version in <build_dir>/debug/core-<kernel>.o, so we now use this one
for <build_dir>/var/run/<run_name>.core.
Ref #2254
This way, files copied from the depot are incorporated in addition to
the files explicitly specified as boot modules.
The patch also adds an automatic check for the validity of the XML
syntax of boot modules ending with '.config'.
Issue #2339
Run scripts can use the new 'import_from_depot' function to incorporate
archive content from the depot into a scenario. The function must be
called after the 'create_boot_directory' function and takes any number
of pkg, src, or raw archives as arguments. An archive is specified as
depot-relative path of the form <user>/<type>/name. Run scripts may
call 'import_from_depot' repeatedly.
An argument can refer to a specific version of an archive or just the
version-less archive name. In the latter case, the current version (as
defined by a corresponding archive recipe in the source tree) is used.
If a 'src' archive is specified, the run tool integrates the content of
the corrsponding binary archive into the scenario. The binary archives
are selected according the spec values as defined for the build directory.
As of now, only x86_32 and x86_64 are supported by the 'depot_spec'
function.
Issue #2339
By installing the core object to bin/, we follow the same convention as
for regular binaries. This, in turn, enables us to ship core in a
regular binary archive. The patch also adjusts the run tool to pick up
the core object from bin/ for the final linking stage.
'unzip' can uncompress and load raw multiboot images and ELF files.
Usage together with the PXE bootloader ease life running the muen hypervisor
together with a Genode/x86_64 VM, where the raw boot image is typically quite
large.
Ref #2358
This enables to use append_if like the append command in the following
situation.
proc optional_binary {} {...}
append_if $optional config {
<start name="} [optional_binary] {">
...
}
Put the initialization of the cpu cores, setup of page-tables, enabling of
MMU and caches into a separate component that is only used to bootstrap
the kernel resp. core.
Ref #2092
Instead of implicitly asking for core's link address when linking
core within the run tool, deliver it explicitly to the build_core
routine. Thereby we gain the freedom to use the build_core tool for
different targets like core, and bootstrap.
Ref #2092
In combination with run/image/uboot, run/boot_dir/foc expected a file link it
created itself to be a directory by trying to create another file link inside
it.
Ref #1987
This commit mostly removes the globally visible NR_OF_CPUS define
from the global makefile specifiers defined in the base-hw repository.
Whereever necessary it adds platform specific makefiles to the base
repository when they were missing.
Ref #2190
This patch removes possible ambiguities with respect to the naming of
kernel-dependent binaries and libraries. It also removes the use of
kernel-specific global side effects from the build system. The reach of
kernel-specific peculiarities has thereby become limited to the actual
users of the respective 'syscall-<kernel>' libraries.
Kernel-specific build artifacts are no longer generated at magic places
within the build directory (like okl4's includes, or the L4 build
directories of L4/Fiasco and Fiasco.OC, or the build directories of
various kernels). Instead, such artifacts have been largely moved to the
libcache. E.g., the former '<build-dir>/l4/' build directory for the L4
build system resides at '<build-dir>/var/libcache/syscall-foc/build/'.
This way, the location is unique to the kernel. Note that various tools
are still generated somewhat arbitrarily under '<build-dir>/tool/' as
there is no proper formalism for building host tools yet.
As the result of this work, it has become possible to use a joint Genode
build directory that is usable with all kernels of a given hardware
platform. E.g., on x86_32, one can now seamlessly switch between linux,
nova, sel4, okl4, fiasco, foc, and pistachio without rebuilding any
components except for core, the kernel, the dynamic linker, and the timer
driver. At the current stage, such a build directory must still be
created manually. A change of the 'create_builddir' tool will follow to
make this feature easily available.
This patch also simplifies various 'run/boot_dir' plugins by removing
the option for an externally hosted kernel. This option remained unused
for many years now.
Issue #2190
This patch decouples the kernel-specific implementation of the dynamic
linker from its kernel-agnostic binary interface. The name of the
kernel-specific dynamic linker binary now corresponds to the kernel,
e.g., 'ld-linux.lib.so' or 'ld-nova.lib.so'. Applications are no longer
linked directly against a concrete instance of the dynamic linker but
against a shallow stub called 'ld.lib.so'. This stub contains nothing
but the symbols provided by the dynamic linker. It thereby represents
the Genode ABI.
At system-integration time, the kernel-specific run/boot_dir back ends
integrate the matching the kernel-specific variant of the dynamic linker
as 'ld.lib.so' into the boot image.
The ABI symbol file for the dynamic linker is located at
'base/lib/symbols/ld'. It contains the joint ABI of all supported
architectures. The new utility 'tool/abi_symbols' eases the creation of
such an ABI symbol file for a given shared library. Its result should be
manually inspected and edited as needed.
The patch removes the 'syscall' library from 'base_libs.mk' to avoid
polluting the kernel-agnostic ABI with kernel-specific interfaces.
Issue #2190
Issue #2195
Instead of solving the problem to deliver ROM modules to core while booting
differently for the several kernels (multi-boot, elfweaver, core re-linking),
this commit unifies the approaches. It always builds core as a library, and
after all binaries are built from a run-script, the run-tool will link an
ELF image out of the core-library and all boot modules. Thereby, core can
access its ROM modules directly.
This approach now works for all kernels except Linux.
With this solution, there is no [build_dir]/bin/core binary available anymore.
For debugging purposes you will find a core binary without boot modules, but
with debug symbols under [run_dir].core.
Fix#2095
* Remove 'test' routine from kernel/core
* Move 'cpu_scheduler' and 'double_list' test to user-land
* Remove 'hw_info' target at all (can be recycled in a topic branch)
The new parameter specifies the additional timeout duration in seconds
which is incurred by AMT log processing, e.g. time spent waiting for the
system to boot.
- disable iommu
- increase root_cnode further for native boot
- support vesa driver on native hardware
- don't mask edge triggered ioapic irqs
- increase various allocators to get noux_tool_chain_* booting natively
Issue #2044
- adjust syscall bindings to support -fPIC
- read serial i/o ports from BIOS data area
- use autoconf.h provided by sel4
-- to avoid ambiguity between sel4 kernel and user libraries
-- remove manual set defines
- remove debug messages
- increase user virtual area to 3GB
Issue #1720
Issue #2044
When running the same kernel in a VM as on the host system and the
kernel boot message from the VM appears on the log output, the run tool
assumes that the host machine has rebooted unexpectedly. With this
commit, an unexpected reboot is assumed only if the kernel boot message
appears at the beginning of a line. On base-hw, we enforce a line feed
at the beginning of the boot message as the SPIKE emulator log starts
with the first message of the kernel lacking a line feed.
Fixes#2041
Enable the ACPI functionality in the platform_drv on hw_x86_64_muen and
provide a simple generated XML report as ROM session in order to make
the PCI configuration space available.
This is a requirement to implement support for MSI on hw_x86_64_muen.
The script takes the following RUN_OPT parameters:
--image-muen-external-build Muen system is built automatically or externally
--image-muen-system Muen system policy
--image-muen-components Muen system components
--image-muen-hardware Muen hardware platform
--image-muen-gnat-path Path to GNAT toolchain
--image-muen-spark-path Path to SPARK toolchain
Causes trouble if a gz image is loaded via grub and later used as initrd for a
Linux VM (e.g. with Seoul VMM)
Discovered during Turmvilla scenario #1552 and issue #1733.
This makes use of the iPXE sanboot command [1] which downloads and
boots an ISO image directly via HTTP. Therefore, your RUN_OPT needs
both
--include image/iso and
--include load/ipxe
NOTE: The webserver serving the ISO image must support ranged requests,
see [2].
[1] - http://ipxe.org/cmd/sanboot
[2] - http://forum.ipxe.org/showthread.php?tid=7295&pid=10482#pid10482
iPXE is an open source network boot firmware which supports booting from
a web server via HTTP [1].
The following two parameters can be used to specify the iPXE/HTTP setup:
--load-ipxe-base-dir
This parameter specifies the base directory of the HTTP server from
which the target machine downloads the files.
--load-ipxe-boot-dir
The directory relative to iPXE base dir which contains the iPXE
chainload configuration and all necessary files.
The target machine is expected to request the following iPXE
configuration via HTTP:
http://${HOST_URL}/${ipxe-boot-dir}/boot.cfg
This can be achieved by building iPXE with the following embedded
script:
#!ipxe
dhcp
chain http://${HOST_URL}/${ipxe-boot-dir}/boot.cfg
See also [2] for additional information.
[1] - http://ipxe.org/
[2] - http://ipxe.org/howto/chainloading#breaking_the_loop_with_an_embedded_scriptFixes#1708
Instead of holding SPEC-variable dependent files and directories inline
within the repository structure, move them into 'spec' subdirectories
at the corresponding levels, e.g.:
repos/base/include/spec
repos/base/mk/spec
repos/base/lib/mk/spec
repos/base/src/core/spec
...
Moreover, this commit removes the 'platform' directories. That term was
used in an overloaded sense. All SPEC-relative 'platform' directories are
now named 'spec'. Other files, like for instance those related to the
kernel/architecture specific startup library, where moved from 'platform'
directories to explicit, more meaningful places like e.g.: 'src/lib/startup'.
Fix#1673
The plugin works just like the netio plugin and uses the following
parameters
--power-off-energenie-host network address of device
--power-off-energenie-password password for device
--power-off-energenie-port target port of device
The run plugin is not generic and works for NETIO4/NETIO230 powerplugs
only. Further, this opens the path for other vendor-specific powerplug
plugins.
Note, the plugin parameter for the addressed powerplug was renamed to
--power-on-netio-host resp.
--power-off-netio-host
If a user has e.g. /tftpboot/x86 as directory and configures
base_dir=/tftboot and offset_dir=/x86, this leads to bad behavior
as the load module creates a symlink
/tftpboot/x86/<builddir> -> <absolut_builddir>
in this case instead of the desired
/tftpboot/x86 -> <absolut_builddir>
Furthermore, the module works on
/tftpboot/x86/config-00-00-00-00-00-00
and
/tftpboot/x86/<builddir>/config-00-00-00-00-00-00
afterwards, which looks bad too. As there is no warning at all, this can
be hard to debug. The commit adds an appropriate check with error message and
exit -1 on an existing directory.
Fixes#1630
The base-hw kernel on x86_64 currently assumes 254 MiB of RAM. The RAM
region is subtracted from the I/O mem allocator and therefore this range
is not available for device I/O.
If qemu is started with -m 128, the region for (emulated) PCI config
space access lies within this region and I/O mem allocation in the
pci_drv will fail. Giving qemu more RAM moves the PCI config space out
of the 254 MiB region, making the run/libc_ffat scenario with acpi work.
The boot modules assembled by the generated boot_modules.s file is
accessed from core using struct Bm_header. Unfortunately the assembler
.long directive is synonym to .int [1] and thus has the same size as the
C++ int type and *not* long. Use the matching assembly type .quad in
boot_modules.s when generating the file for 64-bit platforms such as
x86_64.
[1] - https://sourceware.org/binutils/docs/as/Long.html
This patch contains the initial code needed to build and bootstrap the
base-hw kernel on x86 64-bit platforms. It gets stuck earlier
because the binary contains 64-bit instructions, but it is started in
32-bit mode. The initial setup of page tables and switch to long mode is
still missing from the crt0 code.
On each execution the output buffer is overriden when run_genode_until
is executed multiple times. Run scripts, that depend on extracting
information after several execution steps, e.g. noux_net_netcat, will
fail to do so. Therefore, we append the newly captured log output to
the old output buffer.
Issue #1327.
The mkfs.ext2 heuristics select the "small" ext2 usage type, which does
not fit well with GiB-sized pen drives. For example, the block size is
just 1024 bytes compared to 4096 for "default". Therefore, we enforce
the default usage type as this fits our use case of dumping the image to
USB sticks better.
While booting up, some devices send garbage over the serial connection.
This can result in an unexpected EOF event within expect(1). A filter
program may be specified with '--log-serial-filter' to circumvent this
problem. Then, the output of the serial program is piped through
the filter, which may sanitize the character stream.
Fixes#1395.
and move it close to run_genode_until implementation.
Somehow the "match_max -d" gets ignored if it is to far away and some
our run scripts fails because the buffer contains not all information
(seen for affinity.run on a 8 core machine)
Due to commit "run: relax IP power plug recognition + serial EOF",
when piping the serial command through 'tr', some characters might
get buffered, thereby preventing some run scripts to finish correctly.
This commit removes the 'tr' hack. Instead, to circumvent the 'expect'
problem, which kills under special conditions spawned childs used to
obtain serial line content, whenever EOF of the serial command is
recognized during the boot phase, the child process gets re-spawned.
* To communicate with IP power plug devices from Koukaam, to remote
control power supply of test boards, don't parse minor versions, but
support different IP power plugs
* TCL's expect may report an EOF when reading from the spawned serial
terminal, especially when using 'socat' to stream serial over TCP/IP,
although the spawned child is still running and delivering content.
This problem is clearly not dependent on the characters send, but
possibly due to strange pipe signals. When piping the serial output
through 'tr' the problem vanishs.
The older SOAP EOI protocol is not supported with AMT version 9+. By default
the wsman tool will be used if installed.
RUN_OPT examples to enforce amtool or wsman:
--target amt --amt-tool wsman
--target amt --amt-tool amttool
Fixes#1251
This provides bootable disk images for x86 platforms via
! RUN_OPT="--target disk"
The resulting disk image contains one ext2 partition with binaries from
the GRUB2 boot loader and the run scenario. The default disk size fits
all binaries, but is configurable via
! --disk-size <size in MiB>
in RUN_OPT.
The feature depends on an grub2-head.img, which is part of the commit,
but may also be generated by executing tool/create_grub2. The script
generates a disk image prepared for one partition, which contains files
for GRUB2. All image preparation steps that need superuser privileges
are conducted by this script.
The final step of writing the entire image to a disk must be executed
later by
sudo dd if=<image file> of=<device> bs=8M conv=fsync
Fixes#1203.
By adding: "--target jtag \
--jtag-debugger <debugger configuration> \
--jtag-board <board configuration>" to the RUN_OPTs
this commits enables the run-tool to load and execute an ELF image
via JTAG to the target platform.
Fixes#1191
Using 'upvar' instead of 'global' in the 'append_if' and 'lappend_if'
functions makes it possible to use these functions with local variables
of the calling function.
Fixes#1137.
'check_installed' takes a command name as argument and tries to call 'which' in
order to find the command path. If that does not succeed, paths like '/sbin' are
'/usr/sbin' searched. On success the absolute path of the command is returned,
on failure 'exit' is called with an error message.
Issue #1048
Unless '-ignorestderr' is set on 'exec' in expect, any output on stderr
is interpreted as execution failure. In this case, 'create_iso' logs
some diagnostics but does not fail.
We repeatedly promote our ISOs to be bootable from USB storage.
Therefore, we have a hard dependency on 'isohybrid' and should fail in
the case that the tool is missing.
For PBXA9 qemu adjusts provided RAM chips to the -m arg. Thus we
filter user values and force value that enables all chips that Genode
expects to be available. Not doing so leads to inexplicable errors.
ref #964
Instead of using a fixed command like picocom in the run tool,
it's better to have the flexibility to set the whole command that
connects to the serial device. Therefore, it's possible for instance
to connect to a remote serial device exported via TCP/IP by using tools
like socat.
If the target machine is connected locally one may specify
"serial" as target in the RUN_OPT variable to get the test output.
Used for panda and arndale on foc and hw.
Use RUN_OPT="--target ..." to select the backend test method.
Supported values so far:
qemu - qemu + grub bootloader (default)
qemu+pxe - qemu + pulsar bootloader (PXE)
amt - Intel AMT (reset+serial output) + pulsar bootloader
Related to issue #598
genode_until_run can be called now with a spawn id to able to reattach to a
spawned process (amt, serial output). Run scripts can now call genode_until_run
multiple times.
Make pxe optional and use by default grub.
For that to work we use objcopy to repack the elf64
file into elf32.
With this commit more tests succeed. Most
tests use 64M and with that pulsar even does not start
the hypervisor. With 96M more test run however that would
mean to adjust most of the run scripts ...
Use git to get recent kernels from github. Adjust NOVA patch to compile
with recent github version. Patch and use makefile of NOVA microkernel
to avoid duplicated (and outdated) makefile in Genode
Furthermore, this patch adds support for using NOVA on x86_64. The
generic part of the syscall bindings has been moved to
'base-nova/include/nova/syscall-generic.h'. The 32/64-bit specific
parts are located at 'base-nova/include/32bit/nova/syscalls.h' and
'base-nova/include/64bit/nova/syscalls.h' respectively.
On x86_64, the run environment boots qemu using the Pulsar boot loader
because GRUB legacy does not support booting 64bit ELF executables.
In addition to the NOVA-specific changes in base-nova, this patch
rectifies compile-time warnings or build errors in the 'ports' and
'libports' repositories that are related to NOVA x86_64 (i.e., Vancouver
builds for 32bit only and needed an adaptation to NOVAs changed
bindings)
Fixes#233, fixes#234
In the compare_output_to function in the run tool a check was introduced,
whether the given arguments are empty, and if so if the output string is it
too. Without this patch compare_output_to succeeded when the given pattern
was empty but output wasn't. Please refer to issue #162.
