Heads buildstystem:
Makefile logic will download modules packages under ./packages, check itheir integrity, then extract it and patch extraction directory ONLY if no corresponding .*_verify files are found under ./packages directory. They are extracted under build/modulename-ver/ where patches are applied prior of building them.
build/module* .configured is written when packages are configured under build/modulename-ver/.configured
build/modules* .build is written when packages are built under build/modulename-ver/.build
CircleCI caching subsystem notes:
A cache name tag is calculated in the prep_env stage early at each beginning of a workflow, and consists of a cache name, appended by a calculated digest signature (which is the final hash of hashed files (the hash of a digest).
Look for the following under .circleci/config.yml:
"Creating .... digest statements" : they are basically files passed under sha256sum to create a digest.
restore_cache keys: they are basically a string concatenating: name + checksum of digest + CACHE_VERSION. Only the first cache is extracted following declared order.
save_cache keys: same as above, only saving non-existing caches. That is, skipping existing ones and creating missing ones.
A cache is extracted at the beginning of a workflow if an archive matches an archive name, which consists of a name tag + digest hash + CACHE_VERSION
A cache is created only at the end of a workflow ("Saving cache...").
Caches are specialized. Caches are linked to checkumming of some content. And the largest available cache is extracted on next workflow, only extracting the directories/files that were contained in that cache.
A workspace cache ("Attaching workspace..."), as opposed to a end workflow cache, is passed along steps that depends on prior workflow, as specified under CirclecI config. The current CircleCI config creates a workspace cache for:
make + gawk + musl-cross-make (passed along next)
the most massive board config for each coreboot version (passed along next)
which is finally leading to the workflow cache, specialized for different content that should not change across builds.
That is 3 caches
musl-cross-make and bootstrapping tools (builds make and gawk locally) as long as musl-cross module has same checksum
a coreboot cache, containing all coreboot building directories, as long as coreboot module and patches are having the same hashes
a global cache containing alla builds artifacts (build dir, install dir, musl-cross dir etc)
Consequently, a workspace cache contains all the files under a path that is specified. For heads running under CircleCI, this is ~/project, which is basically "heads" checked out GitHub project, and everything being built under it.
When a workflow is successful, save_cache is ran, constructing caches for digest hashes that are not yet saved (which corresponds to a hash matching muslc-cross module hash, coreboot+patches digest hash and another one for all modules and patches digest hash.
On next workspace iteration, pre_env step will include a "Restore cache" step, which will use the largest cache available and extract it prior of passing it as workspace caches. This is why there is no such different in build time when building on a clean build (the workspace caches layers are smaller, and passed along. This means saving it, passing it. next workspace downloads extracts and builds on top of those smaller layers), as opposed to a workspace reusing and repassing the bigger workspaces containing the whole cache (bigger initial cache extract, then compressing and saving it to be passed as a workspace layer that is then downloaded, extracted, building on top, compressing and saving which then passed as a workspace cache to the next layer depending on it).
And finally, the caching system (save_cache, restore_cache) is based on a CircleCI environment variable named CACHE_VERSION which is appended at the end of the checkum fingerprint of a named cache. It can at any moment be changed to wipe actually used cache, if for some reason it is broken.
Consequently:
CircleCI cache should include packages cache (so that packages are downloaded and verified only once.)
Heads Makefile only downloads, checks and extracts packages and then patch extracted directory content if packages/.module-version_verify doesn't exist. This was missing, causing coreboot tarballs to be redownloaded (not present under packages) and reextracted and repatched (since _verify file was not present under packages/*_verify)
- zlib 1.2.12 release is not respecting cross compiling. 1.2.11 disappeared from servers: taking another archive link, same hash.
- busybox 1.32.0 was not patched with 1.28.0 patch. Renaming patch so that its applied in fresh builds.
- initrd/bin/oem-factory-reset: adds a measured integrity output prior of prompts. Goal is for stating TOTP/HOTP/boot detached signed measurements prior of initiating a Re-Ownership, validating provisioned OEM state.
- initrd/bin/gui-init : Add two additional menu options to LUKS reencrypt and LUKS passphrase change, calling functions of initrd/bin/reencrypt-luks
- initrd/bin/gui-init : Add option F for EOM Factory Reset / Reownership when no public key is exported by key-init
oem-factory-reset: adapt code so that custom passphrases can be provided by user without changing oem factory reset workflow.
oem-factory-reset: output provisioned secrets on screen at the end of of the process.
oem-factory-reset: warn user of what security components will be provisioned with defaults/customs PINs prior of choosing not after
gui-init and oem-factory-reset: change OEM Factory Reset -> OEM Factory Reset / Re-Ownership to cover actual use cases
bin/kexec-parse-boot: test 2bb1f52bf5 that fix correctly comma seperated arguments.
Still TODO: when booting, Heads tries to find where the ISO with /dev/disk/by-label/ARCH_202202 wich is never brought up. uuids could, not sure why the label is not brought up correctly. Maybe an issue in the way Arch makes the ISO.
@tslilc : Any idea to continue #584 or modify #762?
It is going to be enabled later anyway (if CONFIG_HOTPKEY=y), so it can
also be simplified by enabling it at the very beginning.
This enables USB keyboard consistently during all boot menus, including
the "No Bootable OS Found" prompt. It isn't a big deal for "normal"
laptop usage, but it is important for automatic tests and also
non-laptop systems.
Heads build system is reextracting archives and reapplying patches on each iteration.
CircleCI optimizes building time by providing cache mechanisms and forces users to build a target under an hour.
This is to force Open Source projects (free tier) to not be leechers of the free tier.
In the past days, CircleCI bails on building coreboot 4.11 boards because some files being cached are already being present (created files from patches).
In those, two files were unwanted artifacts, recreated on top of coreboot 4.11 extracted original files (undesired .orig files), while bailing on the creating of src/security/tpm/sha1.c from patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch.
Hopefully, this is CircleCI having a maximum of 3 automatically entered input (it fails on the 3rd)... And this fix will permit src/security/tpm/sha1.c and src/security/tpm/sha1.h to be skipped if existing.
Below, we see that CircleCI fills patch prompts with EOF 2 times, and then waits for input and then timeouts.
Here is the failing log trace from https://app.circleci.com/pipelines/github/tlaurion/heads/990/workflows/f2a430fd-dc8c-4e95-abe3-364a0e825533/jobs/4914/parallel-runs/0/steps/0-103:
Exerpt of that log:
if [ -d patches/coreboot-4.11 ] && [ -r patches/coreboot-4.11 ] ; then for patch in patches/coreboot-4.11/*.patch ; do echo "Applying patch file : $patch " ; ( cd /root/project/build/coreboot-4.11/ ; patch -p1 ) < $patch || exit 1 ; done ; fi
Applying patch file : patches/coreboot-4.11/0000-cpu-x86-smm-Use-PRIxPTR-to-print-uintptr_t.patch
patching file src/cpu/x86/smm/tseg_region.c
Applying patch file : patches/coreboot-4.11/0001-Add-Heads-TPM-measured-boot-support.patch
patching file src/Kconfig
The next patch would create the file src/Kconfig.orig,
which already exists! Assume -R? [n] EOF
Apply anyway? [n] EOF
Skipping patch.
1 out of 1 hunk ignored
patching file src/include/program_loading.h
patching file src/lib/cbfs.c
patching file src/lib/hardwaremain.c
Hunk #2 succeeded at 549 (offset 8 lines).
patching file src/lib/rmodule.c
patching file src/security/tpm/Makefile.inc
The next patch would create the file src/security/tpm/sha1.c,
which already exists! Assume -R? [n] make: *** [Makefile:507: /root/project/build/coreboot-4.11/.canary] Hangup
context deadline exceeded
Those boards now produce 4MB coreboot ROM and according CBFS small size, and remove the logic to extract 4Mb ROM out of the 12Mb rom which for some reason, was now misaligned.
config/coreboot-xx30-flash : remove all unnneded stuff to xx30-flash boards.
config/linux-x230-flash: used commonly for all xx30-flash boards, this is now finally saved with savedeconfig, and removes another bunch of unneeded stuff.
Tested working. Fixes#1095
- Add kgpe-d16 patch to remove HID for PCI devices (successful build on top of #1101 and #1012 per https://app.circleci.com/pipelines/github/tlaurion/heads/937/workflows/de49bea0-3f58-4a91-8891-87622f5a0eed)
- CircleCI modified to build for coreboot 4.11 kgpe-d16_workstation on top of 4.15 passed workspace
- CircleCI modified so that we still archive all the logs in artifacts for the current build even if failing. We now exit 1 after having archived all the log files under build/
- Add xx30 vbios extract scripts to test. Expecting musl-cross target to fail since make and gawk aren't built
- CircleCI: gawk was not installed in apt statements under Debian. Installing
- Makefile: seperate and fix local make and gawk building pror of using. Otherwise, impossible to build musl-cross target seperatly.
- Also give some debugging info at start of Heads builds to tell which local gawk and make are used, also telling which make call will be propagated in the rest of the builds
- Fix gawk version checking, reporting bad version even if 4.2.1 as expected on debian-10 (debian-10 OS deploys gawk and make in version 4.2.1)
- CircleCI: Changing musl-cross taget to bootstrap (gawk+make) and musl-cross-make (bootstrap_musl-cross-make) for clarity
- Add kgpe-d16 patch to remove HID for PCI devices (successful build on top of #1101 and #1012 per https://app.circleci.com/pipelines/github/tlaurion/heads/937/workflows/de49bea0-3f58-4a91-8891-87622f5a0eed)
- CircleCI modified to build for coreboot 4.11 kgpe-d16_workstation on top of 4.15 passed workspace
- CircleCI modified so that we still archive all the logs in artifacts for the current build even if failing. We now exit 1 after having archived all the log files under build/
- remove https patch that was made as temporarily fix for bad cert
- upgrade crossgcc's iasl to 2021 so toolchain can be built on debian 11+
- make iasl report itself as being part of coreboot crossgcc build stack.
- remove acpinames from buildgcc make jobss
- add missing string include for binutils gold
- add gnat statements workarounds
- patch Librem L1UM ACPI for newer IASL
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
Older coreboot versions (pre-4.11) required IASL to be built separate
from the main toolchain (crossgcc), but that is no longer the case,
and doing so causes random failures from trying to build IASL as
part of the toolchain and separately, in parallel, each using
-j$(CPUS) threads.
Test: build any board using coreboot 4.15 under Debian 11, observe
no random failures from building the toolchain due to false positives
for a missing depencency .
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
Add patches to coreboot 4.15 to:
- show ME status even when device is disable
- fix PCIe RP hotplug on Librem 14
- fix ME reset timeout on Librem 13/15
This synchronizes with Purism's coreboot 4.15-Purism-3 tag.
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
This commit adds explanatory notes and updates existing t530 and w530 boards to generally align them with the dGPU points and provide signposting for those with and those without dGPU boards. It also adds an additional README in the blobs directory to explain the vbios extraction and building process.
This commit adds support for the t530 and w530 boards to enable dGPUs. dGPU's are required for DisplayPort external displays in the t530 dgpu model, and for both the VGA dn DisplayPort external displays in the W530 (which has two dGPUs, the K1000M and K2000M, hence two boards). The commit does the following:
1. Adds automated extraction scripts for vbios modelled on the me script in the blobs directory (one per board is necessary as it is based on board-specific bios updates).
2. Adds specific boards for the various dGPU models and corresponding coreboot configs.
3. Updates circleci config.yaml to run scripts and test boards.
Tested and working on T530 dgpu and W530 K1000M. dGPU scripts tested on Debian 10 and Ubuntu 21.04
Unify the CONFIG_BOOT_KERNEL_ADD/REOVE parameters for all
Librem boards. Ensure IOMMU disabled for the GPU, and that
duplicated IOMMU params are not passed to the kernel.
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
Resolves issues booting Qubes 4.1-RC1 installer. Drop log level
to be consistent with the Librem 14/Mini v1/v2.
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
Blobs needed to build the Librem 13/15 boards are now
handled by the purism_blobs module; these scripts are
no longer needed.
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
Update all Purism Librem boards except the L1UM server
to coreboot 4.15:
- update coreboot version from 4.8.1/4.13 to 4.15
- use purism_blobs module (if not already)
- update board coreboot defconfig files (Librem 13/15)
Signed-off-by: Matt DeVillier <matt.devillier@puri.sm>
