This patch equips the publish tool to handle system images.
System images reside at <depot-user/image/. The directory contains an
index file that contains the meta information of the available images
in XML form, e.g.,
<index>
<image name="sculpt-pinephone-2023-01-19" board="pinephone">
<info text="initial version"/>
</image>
</index>
To publish a new version of the index:
./tool/depot/publish <depot-user>/image/index
Each system image comes in two forms, a bootable disk image and an
archive of the boot directory. The bootable disk image can be used to
install a new system from scratch by copying the image directly ot a
block device. It contains raw block data. The archive of the boot
directory contains the content needed for an on-target system update to
this version. Within the depot, this archive a directory - named after
the image - that contains the designated content of the boot directory
on target. Depending on the board, it may contain only a single file
loaded by the boot loader (e.g., uImage), or several boot modules, or
even the boot-loader configuration.
To publish both forms:
./tool/depot/publish <depot-user>/image/<image-name>
This results in the following - accompanied by their respective .sig
files - in the public directory:
<depot-user>/image/<image-name>.img.gz (disk image)
<depot-user>/image/<image-name>.tar.xz (boot archive)
<depot-user>/image/<image-name>.zip (disk image)
The .zip file contains the .img file. It is provided for users who
download the image on a system with no support for .gz.
Fixes#4735
Issue #4744
The semantic of .NOPARALLEL has changed in GNU Make 4.4
Quote:
New feature: .NOTPARALLEL accepts prerequisites If the .NOTPARALLEL
special target has prerequisites then all prerequisites of those targets
will be run serially (as if .WAIT was specified between each
prerequisite).
This means that only prerequisites are made sequential. Before
everything within a Makefile would be done in sequential order.
Therefore, we had to add the *.hash target (appears multiple times) to
the .NOPARALLEL prerequisites.
issue #4725
Normally, the APIs listed in 'used_apis' files do not carry a version
but implictly refer to the current version. This patch allows for
specifiying an concrete version. This is a useful feature in the context
of the Goa tool.
By replacing the formerly hard-coded $(GENODE_DIR)/tool/depot/ by the
variable DEPOT_TOOL_DIR, the depot tools can be hosted outside the
Genode source tree, i.e., as part of the Goa tool.
.SHELLFLAGS is extended by option pipefail to make pipes fail if any pipe
element fails. As .SHELLFLAGS is exported into sub-make instances it
must be unexported before calling third-party build systems recursively.
This change keeps the version-controlled 'pubkey' and 'download' files
separate from files generated via depot/create or downloaded via
depot/download. So one can remove the entire depot/ directory without
interfering with git.
Furthermore, depot keys can now be hosted in supplemental repositories
independent from Genode's main repository.
Fixes#4364
The depot tool 'build_bin_archive' used to ask only whether CCACHE is empty or
not and, if it wasn't empty, the tool added 'CCACHE := yes' to the build.conf
of the temporary build dir. This caused the sub-build processes to use ccache
even when the top-level CCACHE variable was set to a bogus value.
Also adds documentation of the CCACHE variable to the help message of the
'build' and 'build_bin_archive' tool.
Ref #4212
We used to export 'LC_COLLATE=C' for the environment of Genode tools. This was
meant to ensure that sorting is always done C-style and not dependent on the
users locale settings. This is required, for instance, to ensure that the same
archive always yields the same hash.
However, 'export LC_COLLATE=C' is not sufficient. It can be outruled by an
'LC_ALL' setting in the users environment. The manual of 'sort' recommends to
set 'LC_ALL=C' locally if you want reliable results and this is what this
commit does. Furthermore it removes the former 'export LC_COLLATE=C'
directives.
Note that I couldn't find a way to set 'LC_ALL' local to 'exec ... sort' in
TCL. This is why I set it global instead using TCLs 'env' array.
Note that the Make directive '$(sort ...)' and the TCL directive 'lsort',
unlike the Shell directive 'sort', are not affected by the users locale
settings.
Fixes#4144
This commit complements issue #4004 with support for the depot/create
tool. If CCACHE=1 is specified at the command line, the build directory
for building a binary archive is configured with 'CCACHE := yes'.
Introduce 'XZ_THREADS' to override the default number of threads used
for compression, which is still set to '1'. As using multiple threads
will increase the amount of memory needed during the compression and
potentially influences the size of the created archive, setting the
variable limits the impact.
Fixes#3431.
* The extract tool determines and reports all missing ports at once.
* The extract tool automatically prepares all missing ports if PREPARE_PORTS=1.
* The missing_ports tool prints a list of missing ports for given archives.
Fixes#3353
nullglob is needed to get an empty shell glob in case no matching binary
archive (path) exists. Otherwise, the original glob string including the
* is returned and used in the for loop.
This patch handles version collisions of binary archives. If a binary
archive for an (automatically) updated src-archive version already
exists in the depot, the extract tool removes the binary archive because
its existing content may stem from another src content (a version from a
different topic branch). This ensures that the new version is always
built, not skipped, when 'UPDATE_VERSIONS=1 REBUILD=' is specified.
Fixes#3267
In cases where the signing failed (maybe the passphrase for the key was
not available at this time), there remained an empty .sig file, which is
newer than the to-be-signed file and, therefore, prevents subsequent
signing processes.
The input for the pkg index is located at gems/run/sculpt/index.
The sculpt.run script uses this input for generating the depot index
file at depot/<user>/index/<version>.
The tool/depot/publish tool support arguments of the form
<user>/index/<version> where <version> corresponds to the Sculpt
version.
Issue #3172
Allow the gpg utility to be overridden on the command line. For
example, './tool/depot/publish GPG="ssh 10.0.1.3 gpg" ...' would invoke
a remote gpg over SSH.
Fix#2981
The timestamp-based rules of make do not work well for the publish tool
because depot archives are often re-created with the same content but a
different modification time, in particular when creating archives via
'FORCE'. This way, those archives are re-published every time, which
becomes a time-consuming operation since all archives must be signed.
This patch filters the targets based on the content that is already
present in the public/ location. All existing archives are skipped.
This patch changes the depot layout such that each archive is
represented as a directory that contains the versions of the archive as
subdirectories.
Issue #2610
In general, make the message conform to the according message of the
common build system. Adds instruction how to fix the missing/outdated
port.
Fixes#2573
When building with multiple jobs, the 'Makefile' rule may be executed
before any of its sibling rules, which implicitly create the target
directory. In this case, the attempt to create the symlink for
'Makefile' fails and the 'build_bin_archive' aborts. Analogously to the
sibling rules, this fix creates the target directory as a side effect of
the 'Makefile' rule.
This patch makes the build step of the create tool conditional. If
merely creating api, src, pkg, or raw archives, the invocation of the
'build' tool can be skipped. Otherwise, the heap message for the 'build'
tool is displayed (because it is called w/o any arguments), which is
misleading to the user.
The depot tool did not consistently distinct libraries from regular
targets when handling binary archives. The binary archive of a regular
target is located at <user>/<bin>/<arch>/<name> whereas a library is
located at <user>/<bin>/<arch>/<api>/<name>.
Fixes#2438
This patch replaces the toy downloader (that merely copied files
locally, for testing) with the mechanics needed to download files from a
www server.
It also changes the use of GPG to use detached signatures.
Issue #2339
