When using the depot/build tool with the 'REBUILD=' argument as done by
the run tool's --depot-auto-update feature, the depot/build tool now
checks that an existing bin archive was indeed created via the src and
api archives present in the depot for the given version. This addresses
consistency issues when switching between different git branches that
refer to the same depot versions but with different content.
Fixes#5379
This patch records the content hashes of the src and api ingredients
used for creating a bin archive. This information allows for the
detection of possible inconsistencies between a bin archive and its
corresponding src archive within the depot.
Issue #5379
When extracting depot archives from the source tree, retain the content
hash of the extracted information along the archive in <version>.hash
file. The .hash files of src and api archives become thereby usable for
checking the consistency of bin archives with their ingredients.
Issue #5379
On systems with the option `use-keyboxd` enabled in config, option
`--no-keyring` in gpg command line doesn't work. The result of the
pubkey_id function will be the first key in the keybox keyring instead
of the id of pubkey from the depot user.
Fixesgenodelabs/genode#5235genodelabs/goa#92
This patch removes the implicit build of all shared libraries a target
depends on. Targets only depend on the respective ABIs instead. This
alleviates the need to locally build complex shared libraries (think of
Qt) when developing applications. Instead, application developers can
use binary depot archives.
The implementation splits the mk/lib.mk file into three files:
- mk/a.mk for building one static library (.lib.a)
- mk/so.mk for building one shared object (.lib.so)
- mk/abi.mk for building one ABI stub (.abi.so)
Furthermore, the commit moves messages and the collection of build
artifacts to var/libdeps, triggers the build of kernel-specific
ld-<kernel>.lib.so, and prunes the lib-dependency tree at ABIs.
Fixes#5061
Without this change gpg aborted with
gpg: Sorry, no terminal at all requested - can't get input
and after adding --batch onlye with
gpg: signing failed: File exists
Downloading non-existent archives from 'depot.genode.org' leads to
files in the 'public' directory containing the 404 error website
delivered by the HTTP server. Specifying the '--fail' option results
in curl generating an exit-code that leads to the appropriate action
by the tool, e.g.:
Error: failed to download 'https://depot.genode.org/user/pkg/x/version.tar.xz'
Issue #4865.
If GPG signing is aborted, e.g., by pressing CTRL-C, an empty .sig file
remained in PUBLIC_DIR because the file was created by the Bash
redirection. By using '--output $@', gpg maintains the .sig file.
If a content.mk file wrongly refers to a non-existing directory in a
call of the 'port_dir' function, the 'content_env_missing_ports.mk' gets
stuck while invoking 'cat' without argument, which makes such mistakes
difficult to diagnose. This patch adds a sanity check for the existence
of the port hash file at the specified location before attempting to
'cat' the hash-file content.
This patch equips the publish tool to handle system images.
System images reside at <depot-user/image/. The directory contains an
index file that contains the meta information of the available images
in XML form, e.g.,
<index>
<image name="sculpt-pinephone-2023-01-19" board="pinephone">
<info text="initial version"/>
</image>
</index>
To publish a new version of the index:
./tool/depot/publish <depot-user>/image/index
Each system image comes in two forms, a bootable disk image and an
archive of the boot directory. The bootable disk image can be used to
install a new system from scratch by copying the image directly ot a
block device. It contains raw block data. The archive of the boot
directory contains the content needed for an on-target system update to
this version. Within the depot, this archive a directory - named after
the image - that contains the designated content of the boot directory
on target. Depending on the board, it may contain only a single file
loaded by the boot loader (e.g., uImage), or several boot modules, or
even the boot-loader configuration.
To publish both forms:
./tool/depot/publish <depot-user>/image/<image-name>
This results in the following - accompanied by their respective .sig
files - in the public directory:
<depot-user>/image/<image-name>.img.gz (disk image)
<depot-user>/image/<image-name>.tar.xz (boot archive)
<depot-user>/image/<image-name>.zip (disk image)
The .zip file contains the .img file. It is provided for users who
download the image on a system with no support for .gz.
Fixes#4735
Issue #4744
The semantic of .NOPARALLEL has changed in GNU Make 4.4
Quote:
New feature: .NOTPARALLEL accepts prerequisites If the .NOTPARALLEL
special target has prerequisites then all prerequisites of those targets
will be run serially (as if .WAIT was specified between each
prerequisite).
This means that only prerequisites are made sequential. Before
everything within a Makefile would be done in sequential order.
Therefore, we had to add the *.hash target (appears multiple times) to
the .NOPARALLEL prerequisites.
issue #4725
Normally, the APIs listed in 'used_apis' files do not carry a version
but implictly refer to the current version. This patch allows for
specifiying an concrete version. This is a useful feature in the context
of the Goa tool.
By replacing the formerly hard-coded $(GENODE_DIR)/tool/depot/ by the
variable DEPOT_TOOL_DIR, the depot tools can be hosted outside the
Genode source tree, i.e., as part of the Goa tool.
.SHELLFLAGS is extended by option pipefail to make pipes fail if any pipe
element fails. As .SHELLFLAGS is exported into sub-make instances it
must be unexported before calling third-party build systems recursively.
This change keeps the version-controlled 'pubkey' and 'download' files
separate from files generated via depot/create or downloaded via
depot/download. So one can remove the entire depot/ directory without
interfering with git.
Furthermore, depot keys can now be hosted in supplemental repositories
independent from Genode's main repository.
Fixes#4364
The depot tool 'build_bin_archive' used to ask only whether CCACHE is empty or
not and, if it wasn't empty, the tool added 'CCACHE := yes' to the build.conf
of the temporary build dir. This caused the sub-build processes to use ccache
even when the top-level CCACHE variable was set to a bogus value.
Also adds documentation of the CCACHE variable to the help message of the
'build' and 'build_bin_archive' tool.
Ref #4212
We used to export 'LC_COLLATE=C' for the environment of Genode tools. This was
meant to ensure that sorting is always done C-style and not dependent on the
users locale settings. This is required, for instance, to ensure that the same
archive always yields the same hash.
However, 'export LC_COLLATE=C' is not sufficient. It can be outruled by an
'LC_ALL' setting in the users environment. The manual of 'sort' recommends to
set 'LC_ALL=C' locally if you want reliable results and this is what this
commit does. Furthermore it removes the former 'export LC_COLLATE=C'
directives.
Note that I couldn't find a way to set 'LC_ALL' local to 'exec ... sort' in
TCL. This is why I set it global instead using TCLs 'env' array.
Note that the Make directive '$(sort ...)' and the TCL directive 'lsort',
unlike the Shell directive 'sort', are not affected by the users locale
settings.
Fixes#4144
This commit complements issue #4004 with support for the depot/create
tool. If CCACHE=1 is specified at the command line, the build directory
for building a binary archive is configured with 'CCACHE := yes'.
Introduce 'XZ_THREADS' to override the default number of threads used
for compression, which is still set to '1'. As using multiple threads
will increase the amount of memory needed during the compression and
potentially influences the size of the created archive, setting the
variable limits the impact.
Fixes#3431.
* The extract tool determines and reports all missing ports at once.
* The extract tool automatically prepares all missing ports if PREPARE_PORTS=1.
* The missing_ports tool prints a list of missing ports for given archives.
Fixes#3353
nullglob is needed to get an empty shell glob in case no matching binary
archive (path) exists. Otherwise, the original glob string including the
* is returned and used in the for loop.
This patch handles version collisions of binary archives. If a binary
archive for an (automatically) updated src-archive version already
exists in the depot, the extract tool removes the binary archive because
its existing content may stem from another src content (a version from a
different topic branch). This ensures that the new version is always
built, not skipped, when 'UPDATE_VERSIONS=1 REBUILD=' is specified.
Fixes#3267
In cases where the signing failed (maybe the passphrase for the key was
not available at this time), there remained an empty .sig file, which is
newer than the to-be-signed file and, therefore, prevents subsequent
signing processes.
The input for the pkg index is located at gems/run/sculpt/index.
The sculpt.run script uses this input for generating the depot index
file at depot/<user>/index/<version>.
The tool/depot/publish tool support arguments of the form
<user>/index/<version> where <version> corresponds to the Sculpt
version.
Issue #3172
Allow the gpg utility to be overridden on the command line. For
example, './tool/depot/publish GPG="ssh 10.0.1.3 gpg" ...' would invoke
a remote gpg over SSH.
Fix#2981
The timestamp-based rules of make do not work well for the publish tool
because depot archives are often re-created with the same content but a
different modification time, in particular when creating archives via
'FORCE'. This way, those archives are re-published every time, which
becomes a time-consuming operation since all archives must be signed.
This patch filters the targets based on the content that is already
present in the public/ location. All existing archives are skipped.
This patch changes the depot layout such that each archive is
represented as a directory that contains the versions of the archive as
subdirectories.
Issue #2610
