ExternalVendorCode/genode
1
0
Fork 0
mirror of https://github.com/genodelabs/genode.git synced 2025-01-30 16:14:13 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Release notes for version 19.05

Browse Source
This commit is contained in:
Norman Feske 2019-05-27 13:49:56 +02:00
parent 44181fedf7
commit 4dbb18bf5e
1 changed files with 886 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

886
doc/release_notes-19-05.txt Normal file
View File

@ -0,0 +1,886 @@
===============================================
Release notes for the Genode OS Framework 19.05
===============================================
Genode Labs
The Genode release 19.05 is primarily focused on platform support.
It adds compatibility with the 64-bit ARM architecture (AARCH64),
comes with improvements of the various kernels targeted by the framework,
and extends the list of supported hardware. The increased diversity of base
platforms calls for unifications to keep the hardware and kernel landscape
manageable.
On that account, Genode uses one reference tool chain across all kernels
and CPU architectures. The current release upgrades this tool chain to
*GCC 8.3* with C++17 enabled by default
(Section [Tool chain based on GCC 8.3.0 and binutils 2.32]).
To increase the velocity of Genode system scenarios across different boards
of a given CPU architecture, the release introduces the notion
of *board and kernel-agnostic build directories* presented in Section
[Unified build directories for ARM]. Once built for one particular
CPU architecture, the same binaries can be deployed at any supported board or
kernel without recompilation. This vastly accelerates the workflow when
targeting multiple boards and emulators at once.
As another major unification effort, the current release introduces a new
*kernel-agnostic virtualization* interface. Up until now, virtualization
used to be inherently tied to a specific kernel. Thanks to the new interface,
however, one virtual machine monitor implementation can be combined with
kernels as different as NOVA, seL4, or Fiasco.OC. No recompilation needed.
As outlined in Section [Kernel-agnostic virtual-machine monitors], Genode
has now become able to run the Seoul VMM on all those kernels, while
VirtualBox is planned to follow.
On our [https://genode.org/about/road-map - road map], we originally
planned several user-facing features related to Sculpt OS. However, in the
light of the major platform efforts, we decided to defer those topics instead
of rushing them.
That said, the release is not without new features. For example, our port
of *OpenJDK* has become able to host the Spring framework and the Tomcat web
server, there are welcome improvements of the *package-management tooling*,
and we added new options for user-level networking.
Finally, version 19.05 is accompanied with the annual revision of the *Genode*
*Foundations book* (Section [New revision of the Genode Foundations book]),
which is now available as an online version in addition to the regular PDF
document.
Kernel-agnostic virtual-machine monitors
########################################
Since the introduction of Genode's
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - Application Binary Interface]
in the 17.02 release,
Genode components can be assembled once for a given hardware platform and
executed without further adjustments on all the supported kernels. However, at
that time, the supported virtual machine monitors - a port of VirtualBox 4 & 5,
Seoul, and our
[https://genode.org/documentation/articles/arm_virtualization - custom VMM] -
remained kernel specific.
Of course, last remaining bastions tempt to be taken! So last year, we started
the venture to unify our virtualization interface across different kernels.
Starting point was the already existing Genode VM interface of our custom VMM
on ARM. We took it and extended the interface with caution to the x86 world.
Having an eye on the requirements of our already supported VMMs on NOVA(x86),
namely VirtualBox and Seoul, the VM interface got extended with missing
features like multiple vCPU support and specific VM handlers per vCPU.
In parallel, we started to investigate the other x86 microkernels with regard
to hardware-assisted virtualization features, namely seL4 and Fiasco.OC.
Over several weeks, we iteratively extended the interface. On the one hand
we familiarized ourself with the kernel interfaces of seL4 & Fiasco.OC while
on the other hand considered known requirements of the NOVA microhypervisor.
Additionally, we kept our custom VMM for ARM still compatible with the new VM
interface.
During this time, it became apparent that the control flow on a VM resume/pause
and a VM event(exit) are different between seL4/Fiasco.OC and NOVA/base-hw.
For seL4 and Fiasco.OC, a VM is resumed by making a blocking syscall on the
kernel. On a VM event, the blocking syscall would return. Logically, on both
kernels the VMM 'calls' into the VM.
On base-hw and NOVA, it is the other way around. Whenever a VM causes a VM
event, the kernels set up either an asynchronous notification (base-hw) or a
synchronous IPC call (NOVA) to the VMM. In both cases the VMM executes a prior
registered VM event handler as response.
Upon return of the VM event handler, the kernel resumes the VM. Logically, on
NOVA and base-hw the VM 'calls' into the VMM. The following two figures
contrast the different flows of control between a user-level virtual machine
monitor and the respective kernels.
[image vm_seq_foc_sel4]
Control flow of handling virtualization events on Fiasco.OC and seL4
[image vm_seq_nova_hw]
Control flow of handling virtualization events on NOVA and the base-hw kernel
Hiding this differences behind a common VM interface was the challenge we were
faced, accepted, and won. Finally, at one point in December we had all 3
x86 kernels running with a test VMM - without re-compilation. The toy VMM
(vmm_x86.run) runs multiple vCPUs on multiple physical CPUs and tests several
VM events/exits.
After this major breakthrough, we spent the days left before Christmas to
adjust the Seoul VMM to the new VM interface, freeing it from the ties to the
NOVA kernel. The choice to start with Seoul stems from the fact that it is -
compared to VirtualBox - much smaller and therefore easier to debug if things
go wrong in the beginning. After one week, the Seoul VMM became in principle
kernel independent and worked again on NOVA. After some more days, it started
to hobble on seL4 and Fiasco.OC as well.
With the New Year, VirtualBox was the next target where all NOVA kernel
specific calls were replaced with the new Genode VM interface. Mid of January,
the work showed first results by having a prototype running simple VMs on NOVA
again. At this point, it became apparent that this venture is not anymore an
adventure. All the findings and technical details so far got condensed to a
[https://fosdem.org/2019/schedule/event/microkernel_virtualization - presentation]
given and recorded at the [https://fosdem.org/2019 - FOSDEM 2019] in Brussels
in February in the
[https://fosdem.org/2019/schedule/track/microkernels_and_component_based_os - Microkernel and Component based OS]
developer room.
At this point, we started transforming our prototype for the 4 kernels into a
clean solution to be featured in Genode 19.05. Eventually, the kernel-agnostic
Seoul VMM runnable on seL4, Fiasco.OC, and NOVA entered Genode master. In the
Genodians article
[https://genodians.org/alex-ab/2019-05-09-seoul-vmm - Seoul VMM and the new VM interface],
we conserved the current state and a few performance measurements.
Shortly before this release, the kernel-agnostic VirtualBox VMM version on
Genode/NOVA got ready. The kernel-agnostic version is in principle capable to
run Linux VMs and Windows 7/10 VMs on Genode/NOVA. Currently, this version
must still be considered as experimental and does not run on seL4 or
Fiasco.OC.
Because of the experimental nature of the kernel-agnostic VirtualBox VMM
version, we decided to keep the kernel-specific version for NOVA for the
moment. This gives us time to test and improve the kernel-agnostic version. It
also allows us to compare both versions to each other.
If time and interest permits, we will consider bringing the virtualization
support on Genode/seL4 and Genode/Fiasco.OC on par with Genode/NOVA.
When building VirtualBox with Genode 19.05,
you will find both the 'virtualbox5-nova' and the new 'virtualbox5' binaries
in the build directory. The former relies on NOVA's kernel interface whereas
the latter uses Genode's kernel-agnostic VM interface. Nightly tested run
scenarios with the new VM interface are named 'vbox5_vm*.run' and can be found
in the 'repos/ports/run' directory.
Broadened CPU architecture support and updated tool chain
#########################################################
With the major update of Genode's tool chain and library infrastructure in
tandem, the framework gains a consistent architecture support across x86-32,
x86-64, ARM-32, RISC-V, and the newly added AARCH64. This includes the tool
chain (Section [Tool chain based on GCC 8.3.0 and binutils 2.32]), the base
framework, the dynamic linker, and the C runtime
(Section [Updated dynamic linker and C runtime]).
Together with this update, we took the chance to wrap up our long-time move
away from board-specific build directories to one generic build directory
shared by multiple kernels and boards for a given CPU architecture
(Section [Unified build directories for ARM]).
Tool chain based on GCC 8.3.0 and binutils 2.32
===============================================
Genode uses a tailored tool chain based on GCC and binutils that is used
across all supported kernels and architectures. Since the previous tool-chain
update in version
[https://genode.org/documentation/release-notes/17.05#Tool_chain - 17.05],
we relied on GCC 6.3. After two years, it was time for an update, motivated by
three major reasons. First, the C++17 standard is common-place now. We Genode
developers anticipate the improvements that come with it. Second, RISC-V and
AARCH64 are now supported by mainline GCC. Up till now, we had to maintain a
custom patch set for Genode's RISC-V support. AARCH64 was not supported yet.
Third, our increasing engagement with SPARK depends on recent improvements of
the Ada compiler that is part of GCC.
With Genode 19.05, the tool chain is now based on binutils version 2.32, GCC
version 8.3.0, GDB version 8.2.1, gcov version 8.3.0, standard C++ library
version 8.3.0.
The tool chain supports x86 (32 and 64 bit), ARM, AARCH64, and RISC-V.
For C++ code, the C++17 standard is enabled by default.
The update of the tool chain provided a perfect opportunity to replace the
former use of gnatmake with a much more natural integration of Ada in Genode's
build system, using a custom ali2dep dependency-extraction tool developed
by [https://github.com/Componolit/ali2dep - Componolit].
In contrast to the previous versions, we switched to a versioned installation
directory for the new tool chain. By default, it is now installed to
_/usr/local/genode/tool/19.05/_. This eases the use of different tool-chain
versions for different development branches.
:Tool-chain installation:
[https://genode.org/download/tool-chain]
Caveats
-------
The tool-chain update required a number of adaptations throughout the source
tree, and may affect Genode users too:
* The silent fall-though within switch statements must now be replaced
by an explicit annotation of the form
! [[fallthrough]]
* The 'register' keyword is no longer valid with C++17. Hence, it must
be removed from the code.
* Types marked as 'Noncopyable' can no longer have an implicit default
constructor. A default constructor must be provided manually.
Updated dynamic linker and C runtime
====================================
The tool-chain update is accompanied with a major update of the dynamic linker
and the C runtime to cover both the AARCH64 and RISC-V architectures in
addition to the traditional x86 and ARM architectures.
FreeBSD 12 supports AARCH64 and RISC-V. Hence, by updating our C runtime to
this version, Genode's libc support extends to those architectures now.
Until now, Genode's dynamic linker supported only the eager binding of symbols
at loading time on the *RISC-V* architecture. With the current version, we
lifted this limitation in favor of lazy binding as used on all other CPU
architectures.
Unified build directories for ARM
=================================
In version
[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - 17.02],
we introduced unified build directories for x86, which allow us to build and
run Genode scenarios on various kernels while using only one build directory.
This concept leverages Genode's cross-kernel binary compatibility to make
the switch from one kernel to another - like developing on base-linux and
deploying on base-nova - a seamless experience.
On ARM, this concept was held back by a third dimension. The
system-integration step does not only depend on the CPU architecture and
the kernel but also on the used board. Our traditional approach was the
use of one build directory per board. Granted, within such a build directory,
one could easily switch between different kernels like Fiasco.OC and seL4.
But on ARM, we find an extreme proliferation of different board
configurations, which share the same CPU architecture but demand different
integration steps. This ensues large redundancies among different build
directories. Switching from one board to another - even when most binaries
happen to be exactly the same - requires an additional rebuilding effort.
With version 19.05, we took the chance to generalize the unified build
directory concept to support multiple different boards per build directory,
greatly reducing the friction when switching kernels and boards for a given
CPU architecture (like ARMv7a). This change has the following implications:
* Drivers no longer depend on the SPEC values as configured for a build
directory.
* All *binaries* are now *named unambiguously*. For example, the USB drivers
for the Panda (OMAP) and Arndale (Exynos) boards were formerly called
'usb_drv' but were different programs. They just never happened to
appear in the same build directory. In the new version, they are named
'panda_usb_drv' and 'arndale_usb_drv' respectively and can thereby
peacefully co-exist within the same 'armv7a' build directory.
Note that this binary renaming will likely affect existing run scripts.
* Include paths no longer hide the board details, which makes the included
code much more easy to follow.
* Run scripts need to pick the right binary, depending on the used board.
Since the board is no longer tied to a build directory, the selection
of the used board has become a build-time variable 'BOARD' following
the successful pattern of how we specify the targeted 'KERNEL'.
To avoid the pollution of run scripts with difficult conditions, we wrap
the drivers needed for a particular board and use case into so-called
_drivers_ packages. Such a package can be instantiated within a generic
scenario using a nested init instance. The details about the drivers and
how they access the hardware remain nicely hidden inside this building block.
Currently, there exist _drivers_ packages for two distinct use cases:
:drivers_interactive pkgs: contain all drivers needed for simple
interactive scenarios, including graphical output and user input.
:drivers_nic pkgs: contain the drivers needed for communication over the
network.
Whenever a run script fits one of these use cases, it can rely on the
corresponding ready-to-use drivers packages via:
! import_from_depot [depot_user]/src/[base_src] \
! [depot_user]/pkg/[drivers_nic_pkg] \
! ...
With the drivers package incorporated, the drivers subsystem can be
instantiated as follows (note the absence of any board or kernel-specific
details):
! <start name="drivers" caps="1000">
! <resource name="RAM" quantum="32M" constrain_phys="yes"/>
! <binary name="init"/>
! <route>
! <service name="ROM" label="config">
! <parent label="drivers.config"/> </service>
! <service name="Timer"> <child name="timer"/> </service>
! <any-service> <parent/> </any-service>
! </route>
! <provides> <service name="Nic"/> </provides>
! </start>
Using the 'BOARD' build variable
--------------------------------
The new 'BOARD' variable selects the board to use. It can be specified either
as a 'make' command-line argument (or environment variable), or defined in the
build-directory configuration (_etc/build.conf_). The following boards are
available:
:arm_v6: rpi
:arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre,
nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory,
wand_quad, zynq_qemu
:arm_v8a: rpi3
:x86_64: pc, linux, muen
:x86_32: pc, linux
:riscv: spike
Please note, when running Genode on Linux or the Muen separation kernel -
although it is run on common x86 PC hardware - we treat both runtime
environments as separate "boards" because their device driver environments
are fundamentally different.
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual update, which is actually
rather a refinement than a revision. The noteworthy additions and changes are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Component health monitoring
* Static code analysis
* Documentation of --depot-user and --depot-auto-update
* Minor adjustments in the under-the-hood chapter
* Changes of the build system
* Updated tool requirements
* Updated API reference
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
New online version of the book
------------------------------
We are happy to announce that the Genode Foundations book is now available
as an online version in addition to the regular PDF version.
:Browse the Genode Foundations book online:
[https://genode.org/documentation/genode-foundations/19.05/index.html]
Thanks a lot to Edgard Schmidt for creating the tooling for the HTML version
of the book!
Base framework and OS-level infrastructure
##########################################
Modernized block-storage interfaces
===================================
With the current release, we revisited Genode's interfaces for accessing
block devices to ease the implementation of asynchronous I/O, to accommodate
zero-copy block drivers, and to support trim and sync operations.
Revised RPC interface
---------------------
The 'Block::Session' RPC interface remained untouched for a long time.
We have now rectified long-standing deficiencies.
First, *sync requests* used to be handled as synchronous RPCs. This is bad
for components like part_block that multiplex one block device for multiple
clients. One long-taking sync request of one client could stall the I/O for
all other clients. The new version handles sync requests as asynchronous
block-request packets instead.
Second, the new version allows a server to dictate the *alignment* of
block-request payload. This way, a driver becomes able to use the payload
buffer shared between client and server directly for DMA transfers while
respecting the device's buffer-alignment constraints.
Third, we added support for *trim* as an asynchronous block operation.
However, as of now, this operation is ignored by all servers.
Fourth, each block operation can now be accompanied with a client-defined
request tag independent from the other parameters of the operation. The tag
allows a block-session client to uniquely correlate acknowledgments with
outstanding requests. Until now, this was possible for read and write
operations by taking the value of the request's packet-stream offset. However,
sync and trim requests do not carry any packet-stream payload and thereby lack
meaningful and unique offset values. By introducing the notion of a tag, we
can support multiple outstanding requests of any type and don't need to
overload the meaning of the offset value.
New client-side API
-------------------
We have now equipped the 'Block::Connection' with a framework API for the
implementation of robust block-session clients that perform block I/O in an
asynchronous fashion.
An application-defined 'JOB' type, inherited from 'Connection::Job',
encapsulates the application's context information associated with a block
operation.
The life cycle of the jobs is implemented by the 'Connection' and driven by
the application's invocation of 'Connection::update_jobs'. The 'update_jobs'
mechanism takes three hook functions as arguments, which implement the
applications-defined policy for producing and consuming data, and for the
completion of jobs.
We plan to gradually move the existing block clients to the new API to benefit
from the latency-hiding effects of asynchronous I/O. The first updated client
is the _block_tester_ component located at _os/src/app/block_tester/_, which
received a number of new features like the choice of the batch size. Please
refer to the accompanied README for a detailed description of the
block-tester.
Unified types for time values
=============================
[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - Two years ago],
we introduced the so-called timeout framework to provide a general solution
for requirements unmet by the bare timer-session interface - most notably
timer-session multiplexing amongst multiple timeouts, and microseconds
accuracy. Up to this day, the timeout framework has proved itself many times
in both real-life appliances and artificial tests and has become the standard
front end for timing in Genode applications.
With this release, we solved one of the few remaining limitations with the
framework by enabling timeouts of up to 2^64 microseconds (> 500000 years)
across all supported architectures. In order to achieve this, we replaced the
former machine-word-wide types used for plain time values by unsigned 64-bit
integers. We did this not only inside the timeout framework but also to almost
all code in the basic Genode repositories that uses the framework.
By doing so, we also paved the way for a second step, in which we are planning
to replace plain time values as far as possible with the abstract 'Duration'
type. With this type in place, the user wouldn't have to worry anymore about
any plain-integer implications when calculating with time values.
Support for chained EBR partitions
==================================
Having an active community around Sculpt leads to bugfixes in unexpected
places. By now we prefer to use a GPT rather than an MBR based partition table
and although we test 'part_block', the component that parses the tables, on
regular basis, the handling of chained EBR's was flawed. Community member
[https://genodians.org/valerius/index - Valery Sedletski] who relies on such a
setup encountered this flaw and provided a bug report, which enabled us to
quickly reproduce and fix the problem.
IP forwarding with port redirection
===================================
The NIC router can now be used to redirect to individual destination ports on
port-forwarding. To express the redirection, the new 'to_port' attribute can
be added to '<tcp-forward>' and '<udp-forward>' rules in the NIC router
configuration. If the new attribute isn't added, the rules behave as usual and
forward with an unaltered destination port.
Libraries, languages, and applications
######################################
Ada/SPARK runtime and SPARK-based cryptography
==============================================
The SPARK runtime has been updated to GCC 8.3. SPARK components do not require
'Genode::Env' or a terminal session anymore. Debug messages can still be
printed using 'GNAT.IO', which uses 'Genode::log' and 'Genode::error'
internally now.
Threading support, which was never fully implemented, has been removed to
further simplify the runtime. This simplification allowed us to prove absence
of runtime errors for the secondary stack allocator and other parts of the
runtime.
[https://github.com/Componolit/libsparkcrypto.git - Libsparkcrypto] is a
library of common cryptographic algorithms implemented in SPARK. It is
free-standing and has a very small footprint. The port of libsparkcrypto for
Genode has been added to the libports repository. Thanks to Alexander Senier
and Johannes Kliemann of [https://componolit.com - Componolit] for maintaining
the Ada/SPARK runtime and libsparkcrypto.
To accommodate the use case of block encryption, we added the small wrapper
library 'aes_cbc_4k' around libsparkcrypto that provides a simple C++
interface for the en/decryption of 4 KiB data blocks. It uses AES-CBC while
incorporating the block number and the private key as salt values.
Improved resilience of the sequence tool
========================================
We have a simple component that starts other components sequentially. It
will exit whenever one of those components has exited with an error.
However, this component is used by our [https://genodians.org - Genodians]
appliance where it controls the content-update mechanism. Since updating
involves fetching content via HTTP/S depending on external events, e.g.,
the remote site is not reachable, the sequence tool might exit. In a long
running appliance, this is obviously not a useful action where no one is
in place to restart the sequence tool. Rather than increasing the overall
complexity of the appliance by introducing such a management component, we
added a _keep-going_ feature to the sequence tool that will instruct it
to carry on even if one of the started components has failed.
Please look at _repos/os/src/app/sequence/README_ for instructions on
how to use the feature.
NIC-bus server for private LANs
===============================
The 'nic_bus' server was added to the world repository as an alternative
to the 'nic_router' and 'nic_bridge' components. The name may be a slight
misnomer, but this component acts neither as a hub, switch, or router.
The 'nic_bus' implements unicast and multicast Ethernet packet switching
between sessions, but drops any unicast packet not destined for a session
attached to the bus. This is in opposition to the behavior of a typical
Ethernet switch and is intending to create simple, software-defined
local-area-networks for native components as well as virtual machines.
In practice the component has been used for attaching VMs to the
[https://yggdrasil-network.github.io/ - Yggdrasil] overlay network via
a bus-local IPv6 prefix.
Distributed Genode
==================
In
[https://genode.org/documentation/release-notes/16.08#Network-transparent_ROM_sessions_to_a_remote_Genode_system - 16.08],
we initially released the _remote_rom_ components that act as communication
proxies. A communication proxy transparently relays a particular service to
another Genode system. As the name suggests, the remote_rom relays ROM
sessions.
Originally implemented as a proof of concept using bare IP packets, broadcast
MACs and static configuration of IP addresses, we added several improvements
to allow a more general use. First, we adopted the size-guard idea for packet
construction and processing from the NIC router. Furthermore, we adopted the
single-threaded implementation style that was already established in other NIC
components. Thanks to Edgard Schmidt for this contribution. Second, we
implemented ARP requests to eliminate broadcasting. Third, we moved from bare
IP packets to UDP/IP and implemented a go-back-N ARQ strategy in order to
reliably transmit larger ROM dataspaces.
As the remote_rom proved valuable for distributing functionality across
multiple Genode devices, we also applied this concept to the LOG session in
order to transmit LOG output from a headless Genode device to a
[https://genode.org/download/sculpt - Sculpt] system for instance. The udp_log
component provides a LOG service and sends the LOG messages as UDP packets to
another machine. The log_udp reverses this process by receiving these UDP
packets and forwarding the messages to a LOG service. An example can be found
in the world repository at _run/udp_log.run_ and _run/log_udp.run_.
Seoul and VirtualBox virtual machine monitors
=============================================
Besides the conversion of the Genode back end of Seoul to the new VM
interface, we added mouse-wheel support to the PS/2 model and changed the VMM
to request a single GUI/nitpicker session rather than distinct framebufer and
input sessions.
Similar to the Seoul VMM, the VirtualBox VMM was adjusted to the new VM
interface and now uses the GUI/nitpicker session. The original kernel-specific
VirtualBox version tied to the NOVA kernel is still available. Both versions
can be used simultaneously.
Use of Nim decoupled from Genode build system
=============================================
With this release, all integration with Nim tooling has been removed from the
Genode build system as a result of maturing support for additional languages
via Genode SDKs. Building Nim components independently of the Genode source
tree has the benefit of smaller upstream checkouts and faster build times, and
has yielded components such as the
[https://genodians.org/ehmry/2019-03-22-depot_9P - 9P server] used in some
Sculpt developer workflows. An example of an independent build system for Nim
components is
[https://genodians.org/ehmry/2019-04-27-nim_packaging - documented on the Genodians blog].
OpenJDK improvements
====================
Within the 19.05 release cycle, we further improved Genode's OpenJDK support
by enabling additional networking infrastructure required by the
[https://spring.io - Spring Framework]. The improvements especially concern
support for SSL connections, which enabled us to successfully execute an embedded
[https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-web-servers.html - Tomcat]
server natively on Genode x86 and ARMv7 platforms using the same JAR archive.
This line of work continues our Java for embedded systems effort as described in
our [https://genodians.org/ssumpf/2019-02-27-java-19-02 - Boot2Java] article.
Having these features in place, our Java efforts will continue in the direction
of Java Swing and the support of input devices in the future, with the ultimate
goal of seamless Java application integration into
[https://genode.org/download/sculpt - Sculpt OS].
Device drivers
##############
Improved Zynq board support
===========================
The initial support of the Xilinx Zynq-7000 SoC was added to our custom kernel
in 15.11. Since then, the support of this hardware has been incrementally
extended. The definitions of memory maps, frequencies, and RAM sizes for
different Zynq-based boards are found in the world repository.
One of the major additions in this release is the initialization of the L2
cache. In this context, we also added a simple cache benchmark at
_repos/os/run/cache.run_ that measures the access times for memory regions of
different size and thereby reveals the number of cache levels and their sizes.
With the latest improvements of the network driver in 18.11, a zero-copy
approach was introduced as an effort to eliminate bottlenecks in the driver's
performance. However, this modification also introduced a kernel dependency of
the driver in order to flush packet-buffer memory from the cache before handing
it over to the DMA-controller. With this release, we moved back to using
uncached dataspaces in order to eliminate the cache flushes and the kernel
dependency. Interestingly, we could not recognize a significant impact on the
driver's performance, which confirms the presumption that flushing the cache
nullifies the gain from using cached dataspaces.
In order to enable the continuous operation of the network driver, we extended
the driver-internal error handling that is necessary to recover the network
driver in certain situations.
_Thanks to Johannes Schlatow for contributing and maintaining Genode's Zynq support!_
Updated Intel network drivers
=============================
As a result of recurring issues with modern Intel i219 laptop NICs, we
updated the driver sources for Intel chipsets to the latest upstream
iPXE version. This update also enables all NIC variants, which were
missing from our manually maintained PCI ID whitelist before.
New drivers-nic and drivers-interactive depot packages
======================================================
As already described in section [Unified build directories for ARM],
_drivers_nic_ packages nicely hide the driver configuration internals needed for
a specific board to communicate over the network. Until now there was only one
package available for x86 based PCs. Now, additional _drivers_nic_ packages
are available for:
:boards: imx53_qsb imx6q_sabrelite linux muen pbxa9 rpi zynq
Beside the formerly available _drivers_interactive_ packages for linux, pbxa9
and pc, there are now additional ones for the following:
:boards: imx53_qsb rpi muen
Platforms
#########
For most kernel environments, the core component provides a ROM module named
'platform_info', which comprises information provided by either the kernel or
the bootloader. The information entails e.g., the TSC clock frequency and
framebuffer dimensions. Most of the information is of interest for special
device driver components only.
Over the time, there was an increasing need to incorporate the information
about which kernel Genode runs on top of. Thereby, special test components,
like depot_autopilot could use the information to, e.g., skip certain tests
on kernels known to not support them. Moreover, there are rare corner-cases
where kernels behave differently, for instance, interrupts are enumerated
differently on certain ARM platforms. Rather than maintaining multiple driver
binaries with different names depending on specific kernels, the
'platform_info' ROM module can now be used to differentiate between kernels
when necessary.
Execution on bare hardware (base-hw)
====================================
This release comes with fundamental optimizations and corrections for
executing Genode on bare hardware when using the core component as the actual
kernel.
In the past, we could observe some serious peculiarities regarding the timing
behavior on the hw kernel. After a careful review, we identified the obstacles
that led to time drifts on several platforms and to quite different runtime
execution.
First and foremost, we limited the CPU-load wasted by the kernel, which
unnecessarily made new scheduling decisions quite often. When the hw kernel
was started as an experiment, there was less focus on performance, but more on
simplicity. Instead of caring about state changes that make a scheduling
decision necessary, the scheduler was asked for the next execution context
unconditionally, whenever the kernel was entered. Now, the scheduler gets
invoked only whenever an execution context gets blocked, or unblocked, or if
the kernel's timer fires due to a timeout. This dramatically influences the
CPU-load caused by the hw kernel in a positive way.
The timing accuracy got increased by reworking most hardware timer drivers
used in the kernel to let the timer never stop counting. Moreover, we limit
the scope in between reading the clock and adjusting the next timeout to a
minimum. The whole internal time representation got widened to 64-bit.
In some rare use cases, we could observe components that do I/O polling, and
thereby actively ask for pending signals, to starve. The reason was a gap in
the hw kernel's syscall API. Beside the ability to wait for signals, the
base-library offers the ability to check for pending signals without blocking
in the case of no available signals. The equivalent call in the kernel was
still missing, and is now present and integrated in the base-library of
base-hw.
ARM architecture
----------------
With this release, we add the i.MX 7 Dual SABRE reference board to the rich
hardware zoo Genode runs directly on top of. This includes the use of the
virtualization extensions available on this platform.
Apart from the new board support, several optimizations were added
specifically for the ARM architecture. Several unnecessary cache maintenance
operations were eliminated, which resided in the code base since the time when
the kernel used a separate address-space only. Moreover, the kernel-lock -
used when several execution contexts on different CPU-cores try to enter the
kernel - does not spin anymore. Instead, the CPU goes into a sleep-state to
save energy. As a side-effect, multi-core scenarios become usable when
executed in Qemu.
X86 architecture
----------------
Since the newly used compiler version makes aggressive use of FPU instructions
including the core component, the kernel itself makes use of FPU registers and
state. Therefore, lazy FPU switching becomes a no go for base-hw. Although, we
incorporated eager FPU switching into the ARM-specific part of the hw kernel
already, the x86 version was still missing it. Now, the FPU context of a thread
gets saved and restored on every kernel entry and exit on x86 too.
Updated Muen separation kernel
==============================
The Muen port has been updated to the latest development version, which comes
with many improvements under the hood. Most notably this version of Muen brings
support for Linux SMP subjects, GNAT Community 2018 toolchain support as well
as much improved build speed, which is most noticeable during autopilot runs.
Additionally, the debug server buffer size in the Genode system policy has been
increased to avoid potential message loss in case of rapid successive logging.
_Thanks to Adrian-Ken Rueegsegger of [https://codelabs.ch - Codelabs] for_
_this welcome contribution!_
NOVA microhypervisor
====================
The kernel got updated due to the tool-chain update to GNU G++ 8.3.0.
Additionally, several issues reported by Julian Stecklina regarding FPU and
page-table synchronization got addressed. The kernel memory allocation at boot
time got more flexible to address target machines with fragmented physical
memory. Additionally, the vTLB implementation is no longer used on AMD
machines whenever nested paging is available.
seL4 microkernel
================
With this release, we extend the variety of hardware to run Genode on top of
the seL4 kernel with NXP's i.MX 7 Dual SABRE reference board. To do so, we had
to update the seL4 tools used to craft a bootable ELF image to a state that is
consistent with the currently supported seL4 kernel version 9.0.1.
As a side-effect of this development work, the General Purpose Timer (GPT) used
in the i.MX series can now be used as a timer service component.
Fiasco.OC microkernel
=====================
As with base-hw and seL4, we add the i.MX 7 Dual SABRE reference board to the
list of working hardware for Genode running on top of the Fiasco.OC
microkernel. Moreover, with Fiasco.OC it is now possible to take the first
steps using Genode on the ARM 64-bit architecture. Therefore, we add Raspberry
Pi 3 as a candidate board to be used with Genode/Fiasco.OC. Currently, only
basic tests without peripheral dependencies are supported.
Tooling and build system
########################
Improved handling of missing ports
==================================
The depot tools _tool/depot/create_ and _tool/depot/extract_ now detect and
report all missing third-party sources - called ports - for a given set of
archives at once. Additionally, the user can tell the tools to download and
prepare such missing ports automatically by setting the argument
'PREPARE_PORTS=1'. Please be aware that doing so may cause downloads and
file operations in your _contrib/_ directory without further interaction.
These features make building archives with dependencies to many ports more
enjoyable. If you merely need a list of ports that are missing for your
archives, you can use the new tool _tool/depot/missing_ports_.
For more details you may read the
[https://genodians.org/m-stein/2019-05-21-depot-missing-ports - article on genodians.org].
Automated depot management
==========================
When using the 'import_from_depot' mechanism of the run tool, one frequently
encounters a situation where the depot lacks a particular archive. Whenever
the run tool detects such a situation, it prompts the user to manually curate
the depot content via the _tool/depot/create_ tool. The need for such manual
steps negatively interferes with the development workflow. The right manual
steps are sometimes not straight-forward to find, in particular after
switching between Git branches.
To relieve the developer from this uncreative manual labor, we extended the
run tool with the option '--depot-auto-update' for managing the depot
automatically according to the needs of the executed run script. To enable
this option, use the following line in the build configuration:
! RUN_OPT += --depot-auto-update
If enabled, the run tool automatically invokes the right depot-management
commands to populate the depot with the required archives, and to ensure the
consistency of the depot content with the current version of the source tree.
The feature comes at the price of a delay when executing the run script
because the consistency check involves the extraction of all used source
archives from the source tree. In regular run scripts, this delay is barely
noticeable. Only when working with a run script of a large system, it may be
better to leave the depot auto update disabled.
Please note that the use of the automated depot update may result in version
updates of the corresponding depot recipes in the source tree (recipe hash
files). It is a good practice to review and commit those hash files once the
local changes in the source tree have reached a good shape.