mirror of
https://github.com/linuxboot/heads.git
synced 2025-01-18 10:46:44 +00:00
CircleCI cache: have all cache layers caching packages directory
Heads buildstystem: Makefile logic will download modules packages under ./packages, check itheir integrity, then extract it and patch extraction directory ONLY if no corresponding .*_verify files are found under ./packages directory. They are extracted under build/modulename-ver/ where patches are applied prior of building them. build/module* .configured is written when packages are configured under build/modulename-ver/.configured build/modules* .build is written when packages are built under build/modulename-ver/.build CircleCI caching subsystem notes: A cache name tag is calculated in the prep_env stage early at each beginning of a workflow, and consists of a cache name, appended by a calculated digest signature (which is the final hash of hashed files (the hash of a digest). Look for the following under .circleci/config.yml: "Creating .... digest statements" : they are basically files passed under sha256sum to create a digest. restore_cache keys: they are basically a string concatenating: name + checksum of digest + CACHE_VERSION. Only the first cache is extracted following declared order. save_cache keys: same as above, only saving non-existing caches. That is, skipping existing ones and creating missing ones. A cache is extracted at the beginning of a workflow if an archive matches an archive name, which consists of a name tag + digest hash + CACHE_VERSION A cache is created only at the end of a workflow ("Saving cache..."). Caches are specialized. Caches are linked to checkumming of some content. And the largest available cache is extracted on next workflow, only extracting the directories/files that were contained in that cache. A workspace cache ("Attaching workspace..."), as opposed to a end workflow cache, is passed along steps that depends on prior workflow, as specified under CirclecI config. The current CircleCI config creates a workspace cache for: make + gawk + musl-cross-make (passed along next) the most massive board config for each coreboot version (passed along next) which is finally leading to the workflow cache, specialized for different content that should not change across builds. That is 3 caches musl-cross-make and bootstrapping tools (builds make and gawk locally) as long as musl-cross module has same checksum a coreboot cache, containing all coreboot building directories, as long as coreboot module and patches are having the same hashes a global cache containing alla builds artifacts (build dir, install dir, musl-cross dir etc) Consequently, a workspace cache contains all the files under a path that is specified. For heads running under CircleCI, this is ~/project, which is basically "heads" checked out GitHub project, and everything being built under it. When a workflow is successful, save_cache is ran, constructing caches for digest hashes that are not yet saved (which corresponds to a hash matching muslc-cross module hash, coreboot+patches digest hash and another one for all modules and patches digest hash. On next workspace iteration, pre_env step will include a "Restore cache" step, which will use the largest cache available and extract it prior of passing it as workspace caches. This is why there is no such different in build time when building on a clean build (the workspace caches layers are smaller, and passed along. This means saving it, passing it. next workspace downloads extracts and builds on top of those smaller layers), as opposed to a workspace reusing and repassing the bigger workspaces containing the whole cache (bigger initial cache extract, then compressing and saving it to be passed as a workspace layer that is then downloaded, extracted, building on top, compressing and saving which then passed as a workspace cache to the next layer depending on it). And finally, the caching system (save_cache, restore_cache) is based on a CircleCI environment variable named CACHE_VERSION which is appended at the end of the checkum fingerprint of a named cache. It can at any moment be changed to wipe actually used cache, if for some reason it is broken. Consequently: CircleCI cache should include packages cache (so that packages are downloaded and verified only once.) Heads Makefile only downloads, checks and extracts packages and then patch extracted directory content if packages/.module-version_verify doesn't exist. This was missing, causing coreboot tarballs to be redownloaded (not present under packages) and reextracted and repatched (since _verify file was not present under packages/*_verify)
This commit is contained in:
parent
303dc26c91
commit
f6d049b3c0
@ -160,24 +160,26 @@ jobs:
|
||||
paths:
|
||||
- crossgcc
|
||||
- build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
|
||||
- packages
|
||||
- save_cache:
|
||||
#Generate cache for the same coreboot mnd musl-cross-make modules definition if hash is not previously existing
|
||||
#CircleCI removed their wildcard support, so we have to list precise versions to cache in directory names
|
||||
key: heads-coreboot-musl-cross-{{ checksum "./tmpDir/coreboot_musl-cross.sha256sums" }}{{ .Environment.CACHE_VERSION }}
|
||||
paths:
|
||||
- crossgcc
|
||||
- build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
|
||||
- packages
|
||||
- build/coreboot-4.11
|
||||
- build/coreboot-4.13
|
||||
- build/coreboot-4.14
|
||||
- build/coreboot-4.15
|
||||
- crossgcc
|
||||
- build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
|
||||
- save_cache:
|
||||
#Generate cache for the exact same modules definitions if hash is not previously existing
|
||||
key: heads-modules-and-patches-{{ checksum "./tmpDir/all_modules_and_patches.sha256sums" }}{{ .Environment.CACHE_VERSION }}
|
||||
paths:
|
||||
- packages
|
||||
- crossgcc
|
||||
- build
|
||||
- packages
|
||||
- install
|
||||
|
||||
workflows:
|
||||
|
Loading…
Reference in New Issue
Block a user