ExternalVendorCode/genode
1
0
Fork 0
mirror of https://github.com/genodelabs/genode.git synced 2024-12-30 10:38:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
baea48fbec
genode/doc/release_notes/20-05.txt
Norman Feske 98211db63d doc: move release notes to sub directory 
This keeps the doc/ directory tidy and neat.
2020-11-27 09:19:09 +01:00

1169 lines
51 KiB
Plaintext
Raw Blame History

===============================================
Release notes for the Genode OS Framework 20.05
===============================================
Genode Labs
Genode 20.05 takes our [https://genode.org/about/road-map - road map's] focus
on the consolidation and optimization of the framework and its API to heart.
It contains countless of under-the-hood improvements, mostly on the account of
vastly intensified automated testing, the confrontation of Genode with
increasingly complex software stacks, and stressful real-world work loads. You
will find this theme throughout the release notes below. The result of this
overhaul is captured in the updated version of the Genode Foundations book
(Section [New revision of the Genode Foundations book]).
Since we achieved the first version of Sculpt OS running on 64-bit ARM
hardware with the previous release, we have intensified our engagement with
ARM platforms. Section [New platform driver for the ARM universe] introduces a
proper device-driver infrastructure for ARM platforms, taking cues from our
x86 platform driver, while taking a clean-slate approach. Section
[Consolidated virtual machine monitor for ARMv7 and ARMv8] shows the
continuation of our ARM virtualization support. In
[https://genodians.org/nfeske/2020-04-30-porting-sculpt - anticipation] of a
more diverse ARM SoC support in the future, maintained not only by Genode Labs
but by independent groups,
Section [Board support outside the Genode main repository] describes the new
ability to host board-support packages outside the Genode source tree.
Even though Genode is able to run on top of the Linux kernel since the very
beginning, Linux was solely meant as a development vehicle. In particular,
Genode's capability-based security model was not in effect on this kernel.
Thanks to the work described in
Section [Capability-based security using seccomp on Linux], this has changed
now. By combining seccomp with an epoll-based RPC mechanism, each Genode
component is sandboxed individually and access control is solely managed by
Genode capabilities now.
The most significant functional addition is a new version of our custom
block-level encryption component described in
Section [Feature-completeness of the consistent block encrypter]. This
long-term project reached the important milestone of feature completeness so
that we are well prepared to introduce this much anticipated feature to Sculpt
OS in the upcoming release.
Given the focus on architectural work, we deferred a few points originally
mentioned on our [https://genode.org/about/road-map - road map] for this
release. In particular, the topics related to Sculpt OS and its related use
cases (network router and desktop) had to be postponed. On the other hand, our
focus on new tooling for streamlining the development and porting of Genode
software has remained steady, which is illustrated by the series of articles
at [https://genodians.org]. For example, the articles about
[https://genodians.org/chelmuth/2020-03-16-lets-encrypt - the development of network applications], or
[https://genodians.org/nfeske/2020-05-27-autoconf-git - the porting of Git].
Capability-based security using seccomp on Linux
################################################
_This section was written by Stefan Thöni of_
_[https://gapfruit.com/ - gapfruit AG] who conducted the described line of_
_work in close collaboration with Genode Labs._
My goal for the Genode
[https://genodians.org/m-stein/2019-08-13-community-summer - Community Summer 2019]
was to enable seccomp for base-linux to achieve an intermediate level of
security for a Genode system running on Linux. To get any security benefit
from seccomp, it turned out the RPC mechanisms of base-linux needed to be
significantly reworked to prevent processes from forging any capabilities.
The new implementation of capability-based security on Linux maps each
capability to a pair of socket descriptors, one of which can be transferred
along socket connections using kernel mechanisms. Each invocation of a
capability uses the received socket descriptor to address the server which in
turn uses the epoll framework of the Linux kernel to get notification of
incoming messages and the server side socket descriptor to securely determine
the invoked RPC object. Capabilities which are passed back to the server
rather than invoked can be securely identified by their inode number. This
way, no client can forge any capability.
With the hard part finally finished thanks to a
[https://github.com/genodelabs/genode/pull/3581 - concerted effort] led by
Norman Feske, I could turn back to seccomp. This Linux kernel mechanism
restricts the ability of a process to use syscalls. Thanks to the small
interface required by Genode processes, the whitelist approach worked nicely.
All Genode processes get restricted to just 25 syscalls on x86, none of which
can access any file on the host system. Instead, all accesses to the host
system must go through Genode's RPC mechanisms to one of the hybrid
components, which are not yet subject to seccomp. Although some global
information of the host system may still be accessed, the possibilities of
escaping a sandboxed Genode process are vastly reduced.
Note that these changes are transparent to any user of base-linux in all but
one way: The Genode system might run out of socket descriptors in large
scenarios. If this happens, you need to increase the hard open file descriptor
limit. See 'man limits.conf' for further information.
Feature-completeness of the consistent block encrypter
######################################################
With
[https://genode.org/documentation/release-notes/19.11#Preliminary_block-device_encrypter - Genode 19.11],
we introduced a preliminary version of the consistent block encrypter (CBE).
The CBE is a block-device encryption component with its core logic entirely
written in SPARK. It combines multiple techniques to ensure confidentiality,
integrity, consistency, and freshness of the block data efficiently.
Furthermore, the whole core logic so far passes 'gnatprove' in mode "flow".
With Genode 20.05, the CBE reached not only feature completeness by receiving
support for trust-anchor integration, online rekeying and online resizing,
it also has been integrated as VFS plugin into a real-world scenario running
an encrypted Linux VM.
As the full story would not fit in the release notes, we decided to dedicate
an extra article at genodians.org that will be published soon and that will
include a tutorial for deploying the CBE in your own Sculpt OS. At this point,
however, we will give you a brief summary of the update. You can check out the
code and run scripts that showcase the new features using the branches
[https://github.com/m-stein/genode/commits/cbe_20.05 - genode/cbe_20.05] and
[https://github.com/m-stein/cbe/commits/cbe_20.05 - cbe/cbe_20.05].
Trust-anchor integration
------------------------
For several reasons, the management of the encryption keys is an aspect that
should not be part of the CBE. Instead, an external entity, called trust
anchor like a smartcard or an USB dongle shall be available to the CBE for
this purpose.
First of all, the trust anchor is responsible for generating the symmetric
block-encryption keys for the CBE via a pseudo-random number generator. The
confidentiality of the whole CBE device depends on these keys. Primarily, they
must never leave the CBE component unencrypted, but even more importantly, the
trust anchor must be the only one able to decrypt them when they got read from
the back-end device.
That said, the trust anchor carries the private key of the user exclusively
and does the asymmetric encryption and decryption of the symmetric keys every
time they leave or enter the CBE component.
Furthermore, the trust anchor is also the key element to the integrity,
consistency, and freshness of an existing CBE device as it always holds the
top-level hash of the most recent CBE state known to be consistent. On
startup, a CBE component matches this hash against the superblocks found on
the device to find the correct one and exclude broken or modified data.
In order to stay independent from the concrete implementation of different
trust anchors, the CBE defines a simple and generic interface to cover the
above mentioned tasks.
Online rekeying
---------------
It is desirable to be able to replace the symmetric key of a CBE device. For
instance, when the trust anchor got compromised or lost. But also as a
preventive requirement on a regular basis.
In order to rekey a CBE device, each block in the CBE must be decrypted and
then re-encrypted. Obviously, this can be a time-intensive task for larger
devices or in the presence of many states of a device.
Rekeying in the CBE is therefore implemented in a way that doesn't block all
other operations on the device. The CBE will continue handling requests from
the user to read blocks, write blocks, synchronize the back end, or discard
snapshots during the rekeying process.
If the system should be turned off while rekeying, be it cleanly or
unintentionally, the process is automatically and safely continued at the next
startup without losing much progress. Once the CBE is done with a rekeying,
the old symmetric key is removed completely from the system.
Online resizing
---------------
There are two pools in the CBE that one might want to re-dimension after
initialization. One is the most recent state of the CBE device, i.e., the
range of addressable virtual blocks. The other is the pool of sparse blocks
that are used to hold older states of the above mentioned blocks. If the pool
of sparse blocks is bigger, the user can keep more snapshots of the device
respectively snapshots that differ more from the current state.
As with rekeying, resizing is done online. I.e., the same operations that get
executed in parallel to rekeying can be executed in parallel to resizing too.
Also on system shutdowns or crashes, resizing behaves like rekeying and
automatically continues at the next startup.
VFS plugin and demo scenario
----------------------------
During the previous months, we phased out the former CBE server components and
solely focused on the CBE-VFS plugin together with the tooling components
'cbe_tester', 'cbe_init', 'cbe_check', and 'cbe_dump'.
As the introduction of the trust anchor simplified the key management, the
plugin was adapted in this regard. It still exposes the same three main
directories _control/_, _snapshots/_ and _current/_. However, the _control/_
directory now only contains the files _create_snapshot_, _discard_snapshot_,
_extend_, and _rekey_. The former two files are used for snapshot management
whereas the other ones address rekeying and extending the size of the block
pools of the CBE. The file that dealt with key management is gone.
There is a new integrated test scenario _vfs_cbe.run_ for base-linux where
the new features of the CBE are exercised concurrently by a shell-script. As
preliminary step, the _vfs_cbe_init.run_ has to be executed to create the
CBE image file.
Furthermore, we finally put the CBE to good use by employing it as backing
storage for VMs. In this exemplary scenario, the CBE-VFS plugin is running in
its own VFS server and is accessed by a management component and the VM. The
CBE is initialized with a 1 GiB virtual block device (VBD) and 256 MiB worth
of sparse blocks.
[image cbe_vbox]
The exemplary CBE-VFS server configuration looks like this:
! <start name="vfs_cbe">
! <binary name="vfs"/>
! <resource name="RAM" quantum="8M"/>
! <config>
! <vfs>
! <fs buffer_size="1M" label="cbe_file"/>
! <dir name="dev">
! <cbe name="cbe" block="/cbe.img"/>
! </dir>
! </vfs>
! <policy label_prefix="bash_cbe" root="/dev/cbe" writeable="yes"/>
! <policy label_prefix="vbox_cbe" root="/dev/cbe/current" writeable="yes"/>
! </config>
! <route>
! <service name="File_system" label="cbe_file">
! <child name="fs_server"/>
! </service>
! <any-service> <parent/> </any-service>
! </route>
! </start>
The file serving as backing storage for the CBE is accessed via the FS-VFS
plugin but thanks to the nature of Genode's VFS, it could as easily be a
normal block device simply by replacing the FS-VFS plugin with the block-VFS
plugin.
That being said, the complete 'cbe' directory is made accessible to a
management component. In this example a 'bash' environment steps in to fulfill
this role:
! <start name="bash_cbe" caps="1000">
! <binary name="/bin/bash"/>
! <resource name="RAM" quantum="64M"/>
! <config>
! <libc stdin="/dev/terminal" stdout="/dev/terminal" stderr="/dev/terminal"
! rtc="/dev/rtc" pipe="/dev/pipe"/>
! <vfs>
! […]
! <dir name="dev">
! <fs label="cbe" buffer_size="1M"/>
! </dir>
! </vfs>
! <arg value="bash"/>
! <env key="TERM" value="screen"/>
! <env key="HOME" value="/"/>
! <env key="PATH" value="/bin"/>
! </config>
! <route>
! <service name="File_system" label="cbe"> <child name="vfs_cbe"/> </service>
! […]
! <any-service> <parent/> <any-child/> </any-service>
! </route>
! </start>
Within the confinement of the shell, the user may access, create and discard
snapshots, extend the CBE and last but not least may issue rekey requests by
interacting with the files in the _/dev/cbe/_ directory.
On the other hand, there is VirtualBox where access to the CBE is limited to
the common block operations:
! <start name="vbox_cbe" caps="1000">
! <binary name="virtualbox5-nova"/>
! <resource name="RAM" quantum="4G"/>
! <config vbox_file="machine.vbox" vm_name="linux">
! <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
! <vfs>
! <fs label="vm" buffer_size="4M" writeable="yes"/>
! <dir name="dev">
! <fs label="cbe" buffer_size="4M" writeable="yes"/>
! <log/> <rtc/>
! </dir>
! </vfs>
! </config>
! <route>
! <service name="File_system" label="vm"> <child name="vm_fs"/> </service>
! <service name="File_system" label="cbe"> <child name="vfs_cbe"/> </service>
! […]
! <any-service> <parent/> </any-service>
! </route>
! </start>
The VM is configured for raw access in its 'machine.vbox' configuration. The
configuration references the CBE's current working state. Since its access is
limited by the 'policy' at the VFS server, the used VMDK file looks as
follows:
! […]
! RW 2097152 FLAT "/dev/cbe/data" 0
! […]
The block device size is set to 1 GiB, given as number of 512 byte sectors.
Any extending operations on the CBE require the user to update the VMDK
configuration to reflect the new state of the CBE.
All in all, in this scenario, the VM is able to read from and write to the CBE
while the user may perform management operations concurrently. Please note,
that the interface of the CBE-VFS plugin is not yet set in stone and may
change as the plugin as well as the CBE library itself continue to mature. For
now, the interface is specially tailored towards the interactive demonstration
use-case.
Together with the upcoming Genodians article about the CBE, we will also
publish depot archives that can be used to conveniently reproduce the
described demo scenario.
Limitations
-----------
Please note that the actual block encryption is not part of the CBE itself as
the CBE should not depend on a concrete encryption algorithm. Instead, it
defines a generic interface to hand over the actual encryption or decryption
of a block to a separate entity. Therefore, the encryption module employed in
our scenarios should be seen as mere placeholder.
An advantage of this design is that encryption can be individually adapted to
the use case without having to change the management of the virtual device.
This also includes the possible use of encryption hardware. Furthermore, in
the case that an encryption method is found to be obsolete, devices using the
CBE can be updated easily by doing a rekeying with the new encryption method.
The same applies for the trust anchor. In our CBE scenarios it is represented
by a placeholder module that is meant only to demonstrate the interplay with
the CBE and that should be replaced in a productive context.
Furthermore, the CBE leaves a lot of unused potential for optimization
regarding performance since we focused primarily on robustness and
functionality, postponing optimizations at this stage. For instance, even
though the CBE is designed from the ground up to operate asynchronously, the
conservative parametrization of the current version deliberately doesn't take
advantage of it. We are looking forward to unleash this potential during the
next release cycle.
Last but not least, note that the CBE has not undergone any security
assessment independent from Genode Labs yet.
New revision of the Genode Foundations book
###########################################
The "Genode Foundations" book received its annual update. It is available at
the [https://genode.org] website as a PDF document and an online version.
The most noteworthy additions and changes are:
: <div class="visualClear"><!-- --></div>
: <p>
: <div style="clear: both; float: left; margin-right:20px;">
: <a class="internal-link" href="https://genode.org">
: <img class="image-inline" src="https://genode.org/documentation/genode-foundations-title.png">
: </a>
: </div>
: </p>
* Description of the feedback-control-system composition
* Removal of outdated components and APIs (e.g., Noux, slave API)
* Additional features ('<alias>', unlabeled LOG sessions)
* Recommended next steps after reading of the getting-started section
* Updated API reference ('Mutex', 'Blockade', 'Request_stream', 'Sandbox')
: <div class="visualClear"><!-- --></div>
To examine the changes in detail, please refer to the book's
[https://github.com/nfeske/genode-manual/commits/master - revision history].
The great consolidation
#######################
On Genode's [https://genode.org/about/road-map - road map] for 2020, we stated
that "during 2020, we will intensify the consolidation and optimization of the
framework and its API, and talk about it." This ambition is strongly reflected
by the current release as described as follows.
Updated block servers using 'Request_stream' API
================================================
During the previous release cycle, we did adjust Genode's AHCI driver as well
as the partition manager to take advantage of the modern 'Request_stream' API,
and thus, deprecating the ancient 'Bock::Driver' interface. With the current
release, we have continued this line of work by introducing the
'Request_stream' API in our USB block driver and Genode's native NVMe driver
with the ultimate goal to eliminate the 'Block::Driver' interface within all
Genode components.
Additionally, the NVMe driver received a major polishing regarding the
handling of DMA memory. The handling of this type of memory got substantially
simplified and the driver now can use the shared memory of a client session
directly, and therefore, save copy operations.
Migration from 'Lock' to 'Mutex' and 'Blockade'
===============================================
Since introducing the 'Mutex' and 'Blockade' types in the
[https://genode.org/documentation/release-notes/20.02#Base-framework_refinements - previous release],
we continued the cultivation of using those types across the _base_ and _os_
repositories of the framework. The changes are mostly transparent to a Genode
developer. One noticeable change is that the 'Genode::Synced_interface' now
requires a 'Mutex' as template argument, the 'Lock' class is not supported
anymore.
Retired Noux runtime environment
================================
We introduced Noux in Genode
[https://genode.org/documentation/release-notes/11.02#Noux_-_an_execution_environment_for_the_GNU_userland - version 11.02]
as a runtime environment for executing command-line-based GNU userland
software on top of Genode. It soon became an invaluable feature that nicely
bridged the gap between Genode's rigid component architecture and the use of
broadly popular Unix tools such as Vim, bash, and make. In particular, without
Noux as a stepping stone, we couldn't have conceived the
[https://genode.org/download/sculpt - Sculpt] operating system.
Code-wise, Noux was the starting point of Genode's unique
[https://genode.org/documentation/release-notes/14.05#Per-process_virtual_file_systems - VFS]
infrastructure that we take for granted today.
That said, the success story of Noux was not without problems. For example,
despite significant feature overlap between Noux and Genode's libc, both
runtime environments remained distinct from each other. Programs had to be
targeted to either environment at build time. As another problem, the
fork/execve mechanism of Noux required a few special hooks in Genode's base
system that are complicated. Still, those hooks remained insufficient to
accommodate Noux on top of the Linux kernel.
One year ago, a "divine" plan of how the feature set of Noux could be
implemented in our regular C runtime struck us. It
[https://genode.org/documentation/release-notes/19.11#C_runtime_with_improved_POSIX_compatibility - turned out]
to work as we hoped for. During the release cycles of versions
[https://genode.org/documentation/release-notes/19.08#Consolidation_of_the_C_runtime_and_Noux - 19.08],
[https://genode.org/documentation/release-notes/19.11#C_runtime_with_improved_POSIX_compatibility - 19.11], and
[https://genode.org/documentation/release-notes/20.02#POSIX_compatibility_improvements - 20.02],
we gradually moved closer to our vision. With the current release, we are
proud to announce that Noux has become obsolete without sacrificing its
feature set! All use cases of Noux can now be addressed by combining Genode's
generic building blocks, in particular the VFS server, VFS plugins for pipes
and terminal access, the 'fs_rom' server, and the C runtime.
[image from_noux_to_libc]
Figure [from_noux_to_libc] illustrates the different approaches. On the left,
the Noux runtime environment provides the traditional Unix system-call
interface to the Unix process(es), taking the position of a Unix kernel. Noux
implements concepts like a virtual file system, file descriptors, pipes, and
execve/fork. Structurally, it looks very traditional. The scenario on the
right achieves the same functionality without a Unix-kernel-like component.
The VFS is provided by a standalone file-system server. For obtaining
executables from the VFS, the 'fs_rom' server is used. The Unix program (bash
in the example) is executed as a plain Genode component that is linked against
Genode's C runtime. This runtime transparently implements fork/execve for
spawning child processes (Vim in the example). All inter-component
communication is achieved via generic Genode session interfaces like the
file-system session. No Unix-like system call interface between components is
needed. The scenario on the right works on all kernels supported by Genode,
including Linux.
The retirement of Noux touches a lot of the existing system scenarios, which
had to be revisited one by one. Among the many examples and test cases updated
to the structure depicted above are _fs_query.run_, _ssh_terminal.run_,
_vim.run_, and _tool_chain_auto.run_. The latter is currently the most complex
scenario, which self-hosts the Genode tool chain and build/packaging system on
top of Genode.
Sculpt OS as the most prominent use case of Noux had to be adjusted as well.
The _noux-system_ package has been replaced by the new _system_shell_ package
as drop-in replacement. The log-noux instance of the Leitzentrale has been
replaced by a simple new component called _stdin2out_. Disk-management
operations are now performed by executing the file-system utilities _e2fsck_,
_resize2fs_, and _mke2fs_ as stand-alone components. The prepare step and the
inspect tab are realized as a system composition like depicted above. On the
user-visible surface, this profound change is barely noticeable.
Removed components and features
===============================
RAM file-system server
~~~~~~~~~~~~~~~~~~~~~~
The 'ram_fs' file-system server has become obsolete because its feature set is
covered by the generic 'vfs' server when combined with the import VFS plugin:
! <start name="ram_fs"...>
! ...
! <config>
! <vfs>
! <ram/>
! <import>
! ...
! </import>
! </vfs>
! ...
! </config>
! </start>
Since the VFS server is a full substitute, the current release drops the
original 'ram_fs' server and migrates all remaining use cases to the VFS
server.
Input-merger component
~~~~~~~~~~~~~~~~~~~~~~
The input-merger component was introduced in version
[https://genode.org/documentation/release-notes/14.11#New_input_merger - 14.11]
as a mechanism for merging PS/2 and USB HID input streams.
It was later superseded by the generic input filter in version
[https://genode.org/documentation/release-notes/17.02#Input-event_filter - 17.02].
The functionality of the input merger can be achieved with the input filter
using a configuration like this:
! <config>
! <input label="ps2"/>
! <input label="usb"/>
! <output>
! <merge>
! <input name="ps2"/>
! <input name="usb"/>
! </merge>
! </output>
! </config>
The current release removes the input merger.
OpenVPN moved to genode-world repository
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since we do not consider our initial
[https://genode.org/documentation/release-notes/14.08#New_port_of_OpenVPN - port of OpenVPN]
as an officially supported feature of Genode, we moved it to the Genode-world
repository.
Rust support removed
~~~~~~~~~~~~~~~~~~~~
Support for the Rust programming language was added as a community
[https://github.com/genodelabs/genode/pull/1899 - contribution] in 2016. It
included the ability to supplement Rust code to Genode components via Genode's
build system, a few runtime libraries, and a small test case. However, the
addition of Rust remained a one-off contribution with no consecutive
engagement of the developer. Over the years, we kept the feature alive - it
used to be exercised as part of our nightly tests - but it was never picked up
by any regular Genode developer. Once it eventually became stale, it was no
longer an attractive feature either because it depended on an outdated nightly
build of the Rust tool chain.
The current release removes Rust to lift our maintenance burden. To
accommodate Rust developers in the future, we may consider supporting Rust on
Genode via the [https://github.com/nfeske/goa - Goa] tool, and facilitating
regular tools and work flows like cargo.
Python2 removed
~~~~~~~~~~~~~~~
With Python3 present in the Genode world repository since version
[https://genode.org/documentation/release-notes/18.08#Python_3 - 18.08],
the time was overdue to remove our original port of Python2 from Genode's main
repository.
Init's ancient '<configfile>' feature removed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The '<configfile>' feature of init allows the use of a ROM session for
obtaining the configuration for a started component. It has long been replaced
by the label-matching-based session routing.
! <route>
! <service name="ROM" label="config">
! <parent label="another.config"/> </service>
! ...
! </route>
Since its naming is rather inconsistent with our terminology (ROM modules are
not files, the "name" is actually a "label") and its use case is covered by
init's generic session-routing mechanism, we took the chance to remove this
legacy with the current release.
Board support outside the Genode main repository
================================================
During the last decade of development, a wide range of hardware support
entered the Genode OS framework. However, the limited resources of a small
core-developer team at Genode Labs makes it impossible to guarantee the same
daily test-coverage for every single board that entered the framework at some
point in time. On the other hand, it is crucial for the acceptance of a
project like Genode to deliver high-quality and carefully tested components.
To balance the act in between retaining hardware support not necessarily used
by the core-team, and having thoroughly tested, recommended hardware on the
other hand, we decided to continuously move support for hardware that is no
longer tested regularly to the
[https://github.com/genodelabs/genode-world - Genode-World] repository.
We started this process by migrating the OMAP4 and Exynos support. The support
for Odroid X2 was eventually dropped. Support for Odroid XU, Arndale board,
and Panda board on top of the base-hw kernel got moved to Genode-World. The
Fiasco.OC kernel support for these boards is no longer part of the framework.
If you like to build scenarios for one of the boards that are now located in
Genode-World, you have to add the world repository to the 'REPOSITORIES'
variable of your build environment, e.g., by uncommenting the following line
in the _etc/build.conf_ file of your build directory:
! #REPOSITORIES += $(GENODE_DIR)/repos/world
To take advantage of the full convenience of using the run tool with one of
these world-located boards, you have to tweak the location of the kernel's
run-tool plugin by replacing the following part:
! ifdef KERNEL
! RUN_OPT += --include boot_dir/$(KERNEL)
! endif
with the absolute path of the kernel-specific file in your Genode-World
repository. Typically this would be:
! ifdef KERNEL
! RUN_OPT += --include $(GENODE_DIR)/repos/world/tool/run/boot_dir/$(KERNEL)
! endif
In the near future, the following hardware targets will be moved to
Genode-World too:
* nit6_solox
* usb_armory
* wand_quad
* zynq_qemu
For the community, this step has the advantage of improved transparency: All
hardware support found inside the main Genode repository gets tested each
night using the CI tools at Genode Labs. The targets hosted at Genode-World
are only guaranteed to compile on a regular basis. On the other hand, the
barrier for contributors to introduce and maintain their own hardware support
is lowered because the quality assurance with respect to Genode-World
components is less strict compared with the inclusion of patches in Genode's
main repository.
Base framework and OS-level infrastructure
##########################################
New platform driver for the ARM universe
========================================
In contrast to x86 PCs where most devices are discovered at runtime using
PCI-bus discovery, within ARM SoCs most devices as well as their I/O resources
are known in advance. Given this fact and for the ease-of-use, until now, most
Genode drivers on ARM used hard-coded I/O resource values, in particular
interrupt and memory-mapped I/O registers.
But in principle, the same device can actually appear within different SoCs.
Even though it may use different I/O resources, its inner-working is
technically identical. By now, we had to re-compile those drivers for
different SoCs, or configure the driver appropriately. A more substantial
downside of this approach is that the drivers directly open up interrupt and
I/O memory sessions at the core component, which is by definition free of any
policy. Therefore, it is not impossible that a driver requests I/O resources
not intended for it. Last but not least, device drivers need DMA-capable
memory along with the knowledge about the corresponding bus address to program
the device appropriately. But this knowledge should not be made available to
arbitrary components given any memory.
Because of these arguments, there was a need for a platform driver component
for ARM similar to the existing one addressing x86 PCs. Actually, it would be
preferable to have a common API that covers both architectures because there
are devices present in both, e.g., certain PCI devices. Nevertheless, the
platform session and platform device interface currently used in the x86
variant emerged from the originally called PCI session and got slightly
frayed. Therefore, we started with a clean and consolidated API for the new
generic ARM platform driver. And although this first version's API is not yet
carved in stone, it will serve as a blueprint for a consolidation of the x86
platform driver in future.
The new platform driver serves a virtual bus to each of its clients in form of
a platform session. The session's API allows the client to request a ROM
session capability. Inside the corresponding ROM dataspace, a driver can find
all information related to its devices in XML-structured form, like in the
following example for the framebuffer driver on pbxa9:
! <devices>
!
! <device name="clcd">
! <property name="compatible" value="arm,pl111"/>
! </device>
!
! <device name="sp810_syscon0">
! <property name="compatible" value="arm,sp810"/>
! </device>
!
! <devices>
The device information in the example above is quite lean, but might get
enriched dependent on the platform driver variant for instance by:
* PCI device information in case of PCI devices
* Device-specific configuration values, e.g., hardware RX/TX queue-sizes
* Device-environment information, e.g., peripheral clock frequency
Given the information about available devices, a driver requests a
platform-device interface for one specific device using its unique name. If a
device is not used anymore, it should be released again. The side effect of
acquisition and release of a device can affect the powering, clock-gating, and
I/O pin-setting of the corresponding device.
Besides device discovery and acquisition, the platform session allows the
driver to allocate and free DMA-capable memory in form of RAM dataspaces, and
to obtain the corresponding bus addresses.
Currently, the platform device interface is limited to obtain interrupt and
I/O memory resources. Information about the available I/O resources is either
integral part of the driver implementation or can be derived from the
device-specific part of the platform session's devices ROM. There is no
additional naming required to request a device's interrupt or I/O memory
session. They are simply referenced by indices.
The new platform driver for ARM uses a config ROM to obtain its device
configuration, as well as the policy rules that define which devices are
assigned to whom. The following example shows how the configuration is used in
the _drivers_interactive_ package for pbxa9 to define and deliver necessary
device resources for framebuffer and input drivers:
! <config>
!
! <!-- device resource declarations -->
!
! <device name="clcd">
! <resource name="IO_MEM" address="0x10020000" size="0x1000"/>
! <property name="compatible" value="arm,pl111"/>
! </device>
!
! <device name="sp810_syscon0">
! <resource name="IO_MEM" address="0x10001000" size="0x1000"/>
! <property name="compatible" value="arm,sp810"/>
! </device>
!
! <device name="kmi0">
! <resource name="IO_MEM" address="0x10006000" size="0x1000"/>
! <resource name="IRQ" number="52"/>
! <property name="compatible" value="arm,pl050"/>
! </device>
!
! <device name="kmi1">
! <resource name="IO_MEM" address="0x10007000" size="0x1000"/>
! <resource name="IRQ" number="53"/>
! <property name="compatible" value="arm,pl050"/>
! </device>
!
! <!-- policy part, who owns which devices -->
!
! <policy label="fb_drv -> ">
! <device name="clcd"/>
! <device name="sp810_syscon0"/>
! </policy>
!
! <policy label="ps2_drv -> ">
! <device name="kmi0"/>
! <device name="kmi1"/>
! </policy>
! </config>
The platform driver is dynamically re-configurable. By now, if it detects
changes in the policy of an already used session, or one of its devices, that
session is closed. Open sessions without configuration changes are not
affected.
The device resource information in the above configuration is one-by-one
derived from the flattened device tree of the ARM Realview PBX A9 board. In
the near future, we approach to provide tooling for the automated derivation
of arbitrary flattened device trees to a platform driver configuration.
By now, the drivers for pl11x framebuffer devices, pl050 PS/2-devices, and the
lan9118 Ethernet device got converted to use the new platform session API, and
are therefore no longer dependent on a specific SoC or board definition.
Within the upcoming releases, we plan to extend the support of different SoCs
using the new platform session API. Thereby, the existent platform session API
specific to Raspberry Pi and i.MX53 boards will vanish, and its functionality
will be incorporated into the new one.
Block-device sync-operation support
===================================
Support for 'SYNC' block requests was extended throughout the block storage
stack. Such a request instructs the block components to flush their internal
buffers. Although it was already implemented in most prominent device drivers
such as AHCI and NVMe, we encountered problems with older AHCI controllers. A
'SYNC' request is now only issued when the NCQ queue of the AHCI controller is
empty.
In addition, the block-VFS plugin as well as the 'lx_fs' file system component
now respect a sync request at the VFS or rather file-system-session level.
Base API refinements
====================
Deprecation of unsafe 'Xml_node' methods
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
One year ago, we
[https://genode.org/documentation/release-notes/19.02#Improved_API_safety - revised]
the interface of Genode's XML parser to promote memory safety. The current
release marks the risky API methods as deprecated and updates all components
to the modern API accordingly.
Replaced 'Genode::strncpy' by 'Genode::copy_cstring'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since the first version of Genode, the public API featured a few
POSIX-inspired string-handling functions, which were usually named after their
POSIX counterparts. In the case of 'strncpy', this is unfortunate because
Genode's version is not 100% compatible with the POSIX 'strncpy'. In
particular, Genode's version ensures the null-termination of the resulting
string as a mitigation against the most prevalent memory-safety risk of POSIX
'strncpy'. The current release replaces 'Genode::strncpy' with a new
'Genode::copy_cstring' function to avoid misconceptions caused by the naming
ambiguity in the future.
LOG session
~~~~~~~~~~~
The return value of 'Log_session::write' got removed. It was never meaningful
in practice. Yet ignoring the value at the caller site tends to make static
code analyzers nervous.
Removed 'Allocator_guard'
~~~~~~~~~~~~~~~~~~~~~~~~~
The 'Allocator_guard' was an
[https://genode.org/documentation/release-notes/11.02#Comprehensive_accounting_of_core_resources - early]
take on a utility for tracking and constraining the consumption of memory
within a component. However, we later
[https://github.com/genodelabs/genode/issues/1039 - got aware] of several
limitations of the taken approach. In short, the 'Allocator_guard' tried
to attack the resource-accounting problem at the wrong level of abstraction.
In Genode
[https://genode.org/documentation/release-notes/17.05#Merged_RAM_and_PD_services_of_the_core_component - 17.05],
we introduced a water-tight alternative in the form of the
'Constrained_ram_allocator', which was gradually being picked up by new
components. However, the relic from the past still remained present in several
time-tested components including Genode's core component. With the current
release, we finally removed the 'Allocator_guard' from the framework and
migrated all former use cases to the 'Constrained_ram_allocator'.
The adjustment of core in this respect has the side effect of a more accurate
capability accounting in core's CPU, TRACE, and RM services. In particular,
the dataspace capabilities needed for core-internal allocations via the
'Sliced_heap' are accounted to the respective client now. The same goes for
nitpicker and nic_dump as other former users of the allocator guard. Hence,
the change touches code at the client and server sides related to these
services.
C runtime
=========
Decoupling C++ runtime support from Genode's base ABI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traditionally all Genode components are linked (explicitly or implicitly) to
the platform library and dynamic linker ld.lib.so. This library provides the
kernel-agnostic _base_ API/ABI and hides platform-specific adaptations from
the components. Also, ld.lib.so includes the C++ runtime which is provided via
the C++ ABI and implements support for runtime type information, guard
variables, and exceptions. So, C++ programs using the named features are
guaranteed to work on Genode as expected.
Since the introduction of our
[https://genode.org/documentation/developer-resources/package_management - package management]
and depot archives in
[https://genode.org/documentation/release-notes/17.05#Package_management - release 17.05]
software packages are required to explicitly specify API dependencies. For
Genode components using the base framework, e.g., to implement signal handlers
in the entrypoint, the dependency to the _base_ API is natural.
Genode-agnostic programs using only the LibC or C++ STL on the other hand
shall not depend on one specific version of the platform library.
Unfortunately up to this release, this independence could only be accomplished
for C or simple C++ programs for the reasons named above. Therefore, all C++
components had to specify the dependency to the base ABI. The unwanted result
of this intermixture was that complex POSIX programs (e.g., Qt5 applications)
had to be updated every time the base ABI changed a tiny detail.
In this release, we cut the dependency on the LibC runtime level and provide
the C++ ABI symbols also in the LibC ABI. The implementation of the functions
remains in ld.lib.so and, thus, is provided at runtime in any case. We also
cleaned up many POSIX-level depot archives from the base dependency already,
which paves the way for less version updates in the future of only the base
API changes.
Redirection of LOG output to TRACE events
=========================================
During the debugging of timing-sensitive system scenarios such as high-rate
interrupt handling of device drivers, instrumentation via the regular logging
mechanism becomes prohibitive because the costs of the logging skew the system
behavior too much.
For situations like this, Genode features a
[https://genode.org/documentation/release-notes/13.08#Light-weight_event_tracing - light-weight tracing]
mechanism, later supplemented with easy-to-use
[https://genode.org/documentation/release-notes/18.02#New_trace-logging_component - tooling].
Still, the tracing facility remains underused to this day.
As one particular barrier of use, manual instrumentation must explicitly
target the tracing mechanism by using the 'trace' function instead of the
'log' function. The process of switching from the logging approach to the
tracing mechanism is not seamless. The current release overcomes this obstacle
by extending the trace policy interface with a new policy hook of the form
! size_t log_output (char *dst, char const *log_message, size_t len);
The hook is invoked each time the traced component performs log output. Once
the trace monitor installs a trace policy that implements this hook into the
traced component, log output can be captured into the thread-local trace
buffer. The size of the captured data is returned. If the hook function
returns 0, the output is directed to the LOG as usual. This way, the
redirection policy of log output can even be changed while the component is
running!
The new feature is readily available for the use with the trace logger by
specifying the 'policy="log_output"' attribute at a '<policy>' node in the
trace logger's configuration. (excerpt taken from the example at
_os/recipes/pkg/test-trace_logger/runtime_):
! <config verbose="yes"
! session_ram="10M"
! session_parent_levels="1"
! session_arg_buffer="64K"
! period_sec="3"
! activity="yes"
! affinity="yes"
! default_policy="null"
! default_buffer="1K">
!
! <policy label="init -> dynamic -> test-trace_logger -> dynamic_rom"
! thread="ep"
! buffer="8K"
! policy="log_output"/>
! </config>
Thanks to Tomasz Gajewski for contributing this handy feature.
MSI-X support on x86
====================
The platform driver now supports the scan for the MSI-X capability of PCI
devices as well as the parsing and setup of the MSI-X structure to make use of
it. With this change, MSI-X style interrupts become usable by kernels
supporting MSI already.
The feature was tested with NVMe devices so far. If MSI-X is available for a
PCI device, the output looks like this:
! [init -> platform_drv] nvme_drv -> : assignment of PCI device 01:00.0 succeeded
! [init -> platform_drv] 01:00.0 adjust IRQ as reported by ACPI: 11 -> 16
! [init -> platform_drv] 01:00.0 uses MSI-X vector 0x7f, address 0xfee00018
! [init -> nvme_drv] NVMe PCIe controller found (0x1987:0x5007)
Optimized retrieval of TRACE subject information
================================================
The trace infrastructure of Genode allows for the tracking and collection of
information about the available subjects (e.g., threads) in the system. Up to
now, the retrieval of information about all subjects of count N required to
issue N RPC calls to Genode's core component, which imply N times the overhead
for inter-component context switching. With this release, the trace session
got extended with the ability to request the information of all subjects as
one batch, thereby dramatically reducing the overhead in large scenarios such
as Sculpt OS. The new 'for_each_subject_info' method of the trace-client side
makes use of the new optimization and is used by the top component.
Library updates
===============
We updated the OpenSSL patch level from 1.0.2q to the latest version 1.0.2u as
an intermediate step to a future update to 1.1.1.
Platforms
#########
Execution on bare hardware (base-hw)
====================================
Consolidated virtual machine monitor for ARMv7 and ARMv8
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The virtual machine monitor for ARMv8 introduced and enriched during the past
two releases got consolidated within this release cycle to support ARMv7 as
well. The old ARMv7 proof-of-concept implementation of the virtual machine
monitor is now superseded by it. Most of the code base of both architecture
variants is shared and only certain aspects of the CPU model are
differentiated. As a positive side effect, the ARMv7 virtual machine is now
supporting recent Linux kernel versions too.
Note that the virtualization support is still limited to the base-hw kernel.
The ARMv7 variant can be used on the following boards:
* virt_qemu
* imx7d_sabre
* arndale (now in genode-world!)
The ARMv8 variant is now available on top of these boards:
* virt_qemu_64
* imx8q_evk
Write-combined framebuffer on x86
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Motivated by a chewy GUI performance of Sculpt on
[https://genode.org/documentation/release-notes/20.02#Execution_on_bare_hardware__base-hw_ - Genode/Spunky],
the write-combining support for base-hw got enabled. To achieve better
throughput to the framebuffer memory, we set up the x86 page attribute table
(PAT) with a configuration for write combining and added the corresponding
cacheability attributes to the page-table entries of the framebuffer memory
mappings. With these changes the GUI became much more snappy.
Improved cache maintenance on ARM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While conducting performance measurements, in particular when using a lot of
dataspaces, it became obvious that the usage of ARM cache maintenance
operations was far from optimal in the base-hw kernel. As a consequence, we
consolidated all cache maintenance operations in the kernel across all ARM
processor variants. Thereby we also found some misunderstandings of the
hardware semantics related to making the instruction and data cache coherent.
The latter is necessary in the presence of self-modifying code, for example in
the Java JIT compiler.
Therefore, we also had to change the system call interface for this purpose.
The now called 'cache_coherent_region' system call is limited to target
exactly one page at a time. However, the 'Genode::cache_coherent' function of
the base API abstracts away from this kernel-specific system call anyway.
As another side effect, we tweaked the clearing of memory. Whenever core hands
out a newly allocated dataspace, its memory needs to be filled with zeroes to
prevent crosstalk between components. Now, the base-hw kernel contains
architecture specific ways of effectively clearing memory, which dramatically
increased the performance for memory-allocation-intensive scenarios.
Qemu-virt platform support
~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks to the contribution by Piotr Tworek, the base-hw kernel now runs on top
of the Qemu virt platform too. It comes in two flavours. The 32-bit 'virt_qemu'
board consists of 2GB of RAM, 2 Cortex A15 cores and uses the GICv2 interrupt
controller. The 'virt_qemu_64' board is the 64-bit variant with 4 Cortex A53
cores, a GICv3 interrupt controller, and has also 2GB of RAM.
Both machine models support the ARM virtualization extensions, which can be
utilized by using Genode's virtual machine monitor for ARM as discussed in
Section [Consolidated virtual machine monitor for ARMv7 and ARMv8].
NOVA microhypervisor
====================
Genode's present CPU affinity-handling concept was originally introduced in
[https://genode.org/documentation/release-notes/13.08#Management_of_CPU_affinities - release 13.08].
With the current release, we added support to leverage the two dimensional
version of the concept by grouping
[https://en.wikipedia.org/wiki/Hyper-threading - hyper-threads] of one CPU
core on the y-axis in the affinity space of Genode.
Genode's core (roottask) for NOVA got adjusted to scan the NOVA kernel's
hypervisor information page (HIP) for hyper-thread support. If available, all
CPUs belonging to the same core get grouped now in Genode's affinity space
along the y-axis. An example output on a machine with hyper-thread support now
looks like this:
! Hypervisor reports 4x2 CPUs
! mapping: affinity space -> kernel cpu id - package:core:thread
! remap (0x0) -> 0 - 0:0:0) boot cpu
! remap (0x1) -> 1 - 0:0:1)
! remap (1x0) -> 2 - 0:1:0)
! remap (1x1) -> 3 - 0:1:1)
! remap (2x0) -> 4 - 0:2:0)
! remap (2x1) -> 5 - 0:2:1)
! remap (3x0) -> 6 - 0:3:0)
! remap (3x1) -> 7 - 0:3:1)
From the output, one can determine Genode's affinity notation in form of (x,y)
mapped to the corresponding CPU. The package:core:thread column represents the
report by the NOVA kernel about CPU characteristics collected by utilizing
[https://en.wikipedia.org/wiki/CPUID - CPUID] during system boot up.
To utilize all hyper-threads in an init configuration, the affinity-space can
now be configured with a height of 2 and the y-axis of a 'start' node with 0
to 1, e.g.
! <config>
! <affinity-space width="4" height="2"/>
! ...
! <start name="app">
! <affinity xpos="0" ypos="1"/> <!-- CPU on Core 0, Hyper-thread 1 -->
! ...
!
! <start name="app"> <!-- CPU on Core 3, Hyper-thread 0 -->
! <affinity xpos="3" ypos="0"/>
! ...
! </start>
!
! <start name="app"> <!-- CPU on Core 3, Hyper-thread 1 -->
! <affinity xpos="3" ypos="1"/>
! ...
! </start>
! </config>
Note: With this new feature, the former sorting of hyper-threaded CPUs for
Genode/NOVA is removed, which got introduced with
[https://genode.org/documentation/release-notes/16.11#NOVA_hypervisor - release 16.08].
Linux
=====
When executed on top of the Linux kernel, Genode's core component used to
assume a practically infinite amount of RAM as the basis for the RAM-quota
trading within the Genode system. The current release introduces the option to
manually supply a realistic value of the total RAM quota in the form of an
environment variable to Genode's core component. If the environment variable
GENODE_RAM_QUOTA is defined, its value is taken as the number of bytes
assigned to the init component started by core.
Thanks to Pirmin Duss for this welcome contribution.
Reference in New Issue View Git Blame Copy Permalink