ExternalVendorCode/heads
1
0
Fork 0
mirror of https://github.com/linuxboot/heads.git synced 2024-12-19 21:17:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
6012e7724c
heads/.circleci/config.yml
Thierry Laurion f6d049b3c0
CircleCI cache: have all cache layers caching packages directory 
Heads buildstystem:

    Makefile logic will download modules packages under ./packages, check itheir integrity, then extract it and patch extraction directory ONLY if no corresponding .*_verify files are found under ./packages directory. They are extracted under build/modulename-ver/ where patches are applied prior of building them.
    build/module* .configured is written when packages are configured under build/modulename-ver/.configured
    build/modules* .build is written when packages are built under build/modulename-ver/.build

CircleCI caching subsystem notes:

    A cache name tag is calculated in the prep_env stage early at each beginning of a workflow, and consists of a cache name, appended by a calculated digest signature (which is the final hash of hashed files (the hash of a digest).
        Look for the following under .circleci/config.yml:
            "Creating .... digest statements" : they are basically files passed under sha256sum to create a digest.
            restore_cache keys: they are basically a string concatenating: name + checksum of digest + CACHE_VERSION. Only the first cache is extracted following declared order.
            save_cache keys: same as above, only saving non-existing caches. That is, skipping existing ones and creating missing ones.
    A cache is extracted at the beginning of a workflow if an archive matches an archive name, which consists of a name tag + digest hash + CACHE_VERSION
    A cache is created only at the end of a workflow ("Saving cache...").
        Caches are specialized. Caches are linked to checkumming of some content. And the largest available cache is extracted on next workflow, only extracting the directories/files that were contained in that cache.
    A workspace cache ("Attaching workspace..."), as opposed to a end workflow cache, is passed along steps that depends on prior workflow, as specified under CirclecI config. The current CircleCI config creates a workspace cache for:
        make + gawk + musl-cross-make (passed along next)
        the most massive board config for each coreboot version (passed along next)
        which is finally leading to the workflow cache, specialized for different content that should not change across builds.
            That is 3 caches
                musl-cross-make and bootstrapping tools (builds make and gawk locally) as long as musl-cross module has same checksum
                a coreboot cache, containing all coreboot building directories, as long as coreboot module and patches are having the same hashes
                a global cache containing alla builds artifacts (build dir, install dir, musl-cross dir etc)
    Consequently, a workspace cache contains all the files under a path that is specified. For heads running under CircleCI, this is ~/project, which is basically "heads" checked out GitHub project, and everything being built under it.
    When a workflow is successful, save_cache is ran, constructing caches for digest hashes that are not yet saved (which corresponds to a hash matching muslc-cross module hash, coreboot+patches digest hash and another one for all modules and patches digest hash.
    On next workspace iteration, pre_env step will include a "Restore cache" step, which will use the largest cache available and extract it prior of passing it as workspace caches. This is why there is no such different in build time when building on a clean build (the workspace caches layers are smaller, and passed along. This means saving it, passing it. next workspace downloads extracts and builds on top of those smaller layers), as opposed to a workspace reusing and repassing the bigger workspaces containing the whole cache (bigger initial cache extract, then compressing and saving it to be passed as a workspace layer that is then downloaded, extracted, building on top, compressing and saving which then passed as a workspace cache to the next layer depending on it).
    And finally, the caching system (save_cache, restore_cache) is based on a CircleCI environment variable named CACHE_VERSION which is appended at the end of the checkum fingerprint of a named cache. It can at any moment be changed to wipe actually used cache, if for some reason it is broken.

Consequently:

    CircleCI cache should include packages cache (so that packages are downloaded and verified only once.)
    Heads Makefile only downloads, checks and extracts packages and then patch extracted directory content if packages/.module-version_verify doesn't exist. This was missing, causing coreboot tarballs to be redownloaded (not present under packages) and reextracted and repatched (since _verify file was not present under packages/*_verify)
2022-04-02 14:57:54 -04:00

526 lines
16 KiB
YAML
Raw Blame History

version: 2.1
commands:
build_board:
parameters:
target:
type: string
subcommand:
type: string
steps:
- run:
name: Install dependencies
command: |
ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
apt update
apt install -y build-essential zlib1g-dev uuid-dev libdigest-sha-perl libelf-dev bc bzip2 bison flex git gnupg gawk iasl m4 nasm patch python python2 python3 wget gnat cpio ccache pkg-config cmake libusb-1.0-0-dev autoconf texinfo ncurses-dev doxygen graphviz udev libudev1 libudev-dev automake libtool rsync innoextract sudo
- run:
name: Make Board
command: |
rm -rf build/<<parameters.target>>/* build/log/* && make V=1 BOARD=<<parameters.target>> <<parameters.subcommand>> || touch ./tmpDir/failed_build
no_output_timeout: 3h
- run:
name: Output hashes
command: |
cat build/<<parameters.target>>/hashes.txt || echo "No hashes for this build step..."\
- run:
name: Archiving build logs.
command: |
tar zcvf build/<<parameters.target>>/logs.tar.gz $(find build/ -name "*.log")
- run:
name: Output build failing logs
command: |
if [[ -f ./tmpDir/failed_build ]]; then find ./build/ -name "*.log" -type f -mmin -1|while read log; do echo ""; echo '==>' "$log" '<=='; echo ""; cat $log;done; exit 1;else echo "Step hasn't failed. Continuing with next step..."; fi \
- store_artifacts:
path: build/<<parameters.target>>
jobs:
prep_env:
docker:
- image: debian:11
resource_class: large
steps:
- run:
name: Install dependencies
command: |
ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
apt update
apt install -y build-essential zlib1g-dev uuid-dev libdigest-sha-perl libelf-dev bc bzip2 bison flex git gnupg gawk iasl m4 nasm patch python python2 python3 wget gnat cpio ccache pkg-config cmake libusb-1.0-0-dev autoconf texinfo ncurses-dev doxygen graphviz udev libudev1 libudev-dev automake libtool rsync innoextract sudo
- checkout
- run:
name: git reset
command: |
git reset --hard "$CIRCLE_SHA1" \
- run:
name: Make tmp dir
command: |
mkdir ./tmpDir \
- run:
name: Creating all modules and patches digest (All modules cache digest)
command: |
find ./patches/ ./modules/ -type f | sort -h |xargs sha256sum > ./tmpDir/all_modules_and_patches.sha256sums \
- run:
name: Creating coreboot (and associated patches) and musl-cross-make modules digest (musl-cross-make and coreboot cache digest)
command: |
find ./modules/coreboot ./modules/musl-cross* ./patches/coreboot* -type f | sort -h | xargs sha256sum > ./tmpDir/coreboot_musl-cross.sha256sums \
- run:
name: Creating musl-cross-make and musl-cross-make patches digest (musl-cross-make cache digest)
command: |
find modules/musl-cross* -type f | sort -h | xargs sha256sum > ./tmpDir/musl-cross.sha256sums \
- restore_cache:
keys:
#Restore existing cache for matching modules digest, validated to be exactly the same as in github current commit.
#This cache was made on top of below caches, if previously existing. If no module definition changed, we reuse this one. Otherwise...
- heads-modules-and-patches-{{ checksum "./tmpDir/all_modules_and_patches.sha256sums" }}{{ .Environment.CACHE_VERSION }}
#If precedent cache not found, restore cache for coreboot module (and patches) and musl-cross-make digests (bi-yearly modified)
#Otehrwise....
- heads-coreboot-musl-cross-{{ checksum "./tmpDir/coreboot_musl-cross.sha256sums" }}{{ .Environment.CACHE_VERSION }}
#If precedent cache not found. Restore cache for musl-cross-make module digest (rarely modified).
#Otherwise, we build cleanly.
- heads-musl-cross-{{ checksum "./tmpDir/musl-cross.sha256sums" }}{{ .Environment.CACHE_VERSION }}
- run:
name: Download and neuter xx20 ME (keep generated GBE and extracted IFD in tree)
command: |
./blobs/xx20/download_parse_me.sh
- run:
name: Download and neuter xx30 ME (keep generated GBE and extracted IFD in tree)
# me_cleaner.py present under heads xx30 blobs dir comes from https://github.com/corna/me_cleaner/blob/43612a630c79f3bc6f2653bfe90dfe0b7b137e08/me_cleaner.py
command: |
./blobs/xx30/download_clean_me.sh -m $(readlink -f ./blobs/xx30/me_cleaner.py)
- run:
name: Download and extract t530 vbios roms for dgpu boards
command: |
./blobs/xx30/vbios_t530.sh
- run:
name: Download and extract w530 vbios roms for dgpu boards
command: |
./blobs/xx30/vbios_w530.sh
- persist_to_workspace:
root: ~/
paths:
- .
build_and_persist:
docker:
- image: debian:11
resource_class: large
parameters:
target:
type: string
subcommand:
type: string
steps:
- attach_workspace:
at: ~/
- build_board:
target: <<parameters.target>>
subcommand: <<parameters.subcommand>>
- persist_to_workspace:
root: ~/
paths:
- .
build:
docker:
- image: debian:11
resource_class: large
parameters:
target:
type: string
subcommand:
type: string
steps:
- attach_workspace:
at: ~/
- build_board:
target: <<parameters.target>>
subcommand: <<parameters.subcommand>>
save_cache:
docker:
- image: debian:11
resource_class: large
steps:
- attach_workspace:
at: ~/
- save_cache:
#Generate cache for the same musl-cross module definition if hash is not previously existing
#CircleCI removed their wildcard support, so we have to list precise versions to cache in directory names
key: heads-musl-cross-{{ checksum "./tmpDir/musl-cross.sha256sums" }}{{ .Environment.CACHE_VERSION }}
paths:
- crossgcc
- build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
- packages
- save_cache:
#Generate cache for the same coreboot mnd musl-cross-make modules definition if hash is not previously existing
#CircleCI removed their wildcard support, so we have to list precise versions to cache in directory names
key: heads-coreboot-musl-cross-{{ checksum "./tmpDir/coreboot_musl-cross.sha256sums" }}{{ .Environment.CACHE_VERSION }}
paths:
- crossgcc
- build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1
- packages
- build/coreboot-4.11
- build/coreboot-4.13
- build/coreboot-4.14
- build/coreboot-4.15
- save_cache:
#Generate cache for the exact same modules definitions if hash is not previously existing
key: heads-modules-and-patches-{{ checksum "./tmpDir/all_modules_and_patches.sha256sums" }}{{ .Environment.CACHE_VERSION }}
paths:
- crossgcc
- build
- packages
- install
workflows:
version: 2
build_and_test:
jobs:
- prep_env
# Below, sequentially build one board for each coreboot
# version. The last board in the sequence is the dependency
# for the parallel boards built at the end, and also save_cache.
# Prerequisites
- build_and_persist:
name: bootstrap_musl-cross-make
target: x230-hotp-maximized
subcommand: bootstrap musl-cross
requires:
- prep_env
# Coreboot 4.13
- build_and_persist:
name: x230-hotp-maximized
target: x230-hotp-maximized
subcommand: ""
requires:
- bootstrap_musl-cross-make
# Coreboot 4.15
- build_and_persist:
name: librem_14
target: librem_14
subcommand: ""
requires:
- x230-hotp-maximized
# Coreboot 4.11
- build_and_persist:
name: kgpe-d16_workstation
target: kgpe-d16_workstation
subcommand: ""
requires:
- librem_14
#Cache one workspace per Coreboot version, ideally the boards including the highest number of modules, since not rebuilt across builds.
#Below, 4.11, 4.13, 4.15
- save_cache:
requires:
- kgpe-d16_workstation
#
#
# Those onboarding new boards should add their entries below.
#
#
- build:
name: x220-hotp-maximized
target: x220-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: x220-maximized
target: x220-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t420-hotp-maximized
target: t420-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t520-hotp-maximized
target: t520-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t420-maximized
target: t420-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t520-maximized
target: t520-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: x230-flash
target: x230-flash
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t430-flash
target: t430-flash
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t430
target: t430
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t430-hotp-verification
target: t430-hotp-verification
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: x230
target: x230
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: x230-hotp-verification
target: x230-hotp-verification
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t430-hotp-maximized
target: t430-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: x230-maximized
target: x230-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t530-hotp-maximized
target: t530-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t530-dgpu-hotp-maximized
target: t530-dgpu-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-hotp-maximized
target: w530-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-dgpu-K1000m-hotp-maximized
target: w530-dgpu-K1000m-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-dgpu-K2000m-hotp-maximized
target: w530-dgpu-K2000m-hotp-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t430-maximized
target: t430-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t530-maximized
target: t530-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: t530-dgpu-maximized
target: t530-dgpu-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-maximized
target: w530-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-dgpu-K1000m-maximized
target: w530-dgpu-K1000m-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: w530-dgpu-K2000m-maximized
target: w530-dgpu-K2000m-maximized
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: qemu-coreboot
target: qemu-coreboot
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: qemu-coreboot-fbwhiptail
target: qemu-coreboot-fbwhiptail
subcommand: ""
requires:
- x230-hotp-maximized
- build:
name: librem_13v2
target: librem_13v2
subcommand: ""
requires:
- librem_14
- build:
name: librem_15v3
target: librem_15v3
subcommand: ""
requires:
- librem_14
- build:
name: librem_13v4
target: librem_13v4
subcommand: ""
requires:
- librem_14
- build:
name: librem_15v4
target: librem_15v4
subcommand: ""
requires:
- librem_14
- build:
name: librem_mini
target: librem_mini
subcommand: ""
requires:
- librem_14
- build:
name: librem_mini_v2
target: librem_mini_v2
subcommand: ""
requires:
- librem_14
- build:
name: kgpe-d16_workstation-usb_keyboard
target: kgpe-d16_workstation-usb_keyboard
subcommand: ""
requires:
- kgpe-d16_workstation
- build:
name: kgpe-d16_server
target: kgpe-d16_server
subcommand: ""
requires:
- kgpe-d16_workstation
- build:
name: kgpe-d16_server-whiptail
target: kgpe-d16_server-whiptail
subcommand: ""
requires:
- kgpe-d16_workstation
- build:
name: librem_l1um
target: librem_l1um
subcommand: ""
requires:
- kgpe-d16_workstation
########################
########################
### OLD STUFF ###
########################
########################
# linuxboot steps need something to pass in the kernel header path
# skipping for now
# - run:
# name: qemu-linuxboot-edk2
# command: |
# ./build/make-4.2.1/make \
# CROSS=/cross/bin/x86_64-linux-musl- \
# BOARD=qemu-linuxboot \
# `/bin/pwd`/build/linuxboot-git/build/qemu/.configured \
# # Run first to avoid too many processes
#
# - run:
# name: qemu-linuxboot
# command: |
# ./build/make-4.2.1/make \
# CROSS=/cross/bin/x86_64-linux-musl- \
# CPUS=16 \
# V=1 \
# BOARD=qemu-linuxboot \
#
# - store-artifacts:
# path: build/qemu-linuxboot/linuxboot.rom
# - store-artifacts:
# path: build/qemu-linuxboot/hashes.txt
Reference in New Issue View Git Blame Copy Permalink