Test each subtarget on push events to improve testing and to refresh
ccache of each subtarget.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 5bafc4352f)
Add support for getting ccache cache from S3.
ccache is archieved in a tar and downloaded from S3 Cloud Storage.
For push events, ccache is then uplodaed back to S3 to refresh and have
a ccache cache always fresh.
An additional workflow is added to upload files to an S3 Cloud Storage
from artifacts uplodaed to github. The minio tool is used to upload
files to S3.
If the ccache can't be downloaded from s3, we fallback to github cache
system.
Also limit s3 upload to the openwrt repository since external fork won't
have (obviously) the required secrtes to upload data to the S3 Cloud
Storage.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit ebbc806d30)
Limit ccache cache save/delete only on push events. Saving ccache
cache for pull request will result in bloat and refreshing ccache is not
possible due to security measure on enforcing read permission on
pull_request events.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit ff66a7c1c0)
Disable ccache usage for coverity workflow as it may cause side effect
in the produced bins.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 2129ee1879)
CCache cache is currently broken due to a funny bug in ccache compiler
type detection. It seems ccache compiler type detection is very fragile
and with the use of external toolchain doesn't correctly detect the
type.
The type detected is set to other instead of gcc resulting in ccache
complaining for unsupported compiler options.
To handle this problem, force the compiler type to gcc to make ccache
correctly work and speedup compilation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit ae7b05328c)
Add new input to define custom ccache cache type. This is useful to use
a different ccache cache for some special workflow that may do more test
than simple kernel compilation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 07b52a8a25)
Add option to disable use of ccache. This can be useful for some
sensible test that should not use ccache as they can cause side effects
of any sort. (example Coverity Scan)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit b9a41c1e84)
Github Actions cache doesn't permit to overwrite cache if it does
already exist. As a trick to refresh and have fresh ccache pool,
delete the ccache cache if it does exist with the help of Github REST
API. An additional permission is needed to access this API. Add this
permittion to each user of the build workflow.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 203cc0a7ef)
Split caching ccache in separate restore and save jobs to always refresh
the ccache across different runs. Currently if a key is restored, cache
is not saved resulting in a less useful ccache that benefits from
multiple runs.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 6321361c6b)
Due to problem with migrating from master to main as the default branch
and downstream project still requiring the master branch to be present,
we currently have for push events double CI runs, one for main and one
for master. To solve this ignore any push event to the master branch for
every workflow that react on push events.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit f5a5ce8822)
Add support to use container included external toolchain and skip
redownloading external sdk for each test.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 0fe5776f4a)
Build and push container with external toolchain embedded in the
container image.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit e1370cdd49)
Add checks to test if toolchain container can be used.
This is to handle case of new target or migration of any sort.
If the toolchain container can't be found, the tools container is used
instead.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 23a5c715a9)
Add option to configure container to use for build test.
By default the tools container is used if no option is provided.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 803b011048)
Drop redundant generare ccache hash job as that can be done by
integrated github expressions to generate an hash.
The only change is that the integrated way generate a sha256 hash
instead of an md5 sum.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 457f6b0b9c)
Refresh containers also on modify of cmake options in the include file.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit b40c0b54bd)
Fix concurrency group for push-containers workflow to handle running on
different branches.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 4c2eab1c27)
Add support for 'sifiveu' target and its specific packages in labeler.
Signed-off-by: Piotr Dymacz <pepe2k@gmail.com>
(cherry picked from commit 92b8b18c26)
Use toolchain container for label workflow to skip downloading external
toolchain from openwrt servers.
Fixes: 0fe5776f4a ("CI: build: Add support to use container included external toolchain")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 66fd0aa6ef)
Don't add "" in target and subtarget for label workflow from label
detection as it does cause problem in build workflow on container
target/subtarget matching.
Fixes: bf8187d5dc ("CI: use split target and subtarget in label workflow")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 1fa84354a9)
Commit 1cb8cdb ("ci: use new buildbot worker images with Debian 11")
introduced new Git version with strict rules for owner of the git
directory.
To handle this and not cause major change, just move the parsing before
the change of ownership of the openwrt directory permitting the correct
run of git fetch command with the same user that did the repository
checkout.
Fixes: 1cb8cdb ("ci: use new buildbot worker images with Debian 11")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 0063e71d66)
Commit bf8187d5dc ("CI: use split target and subtarget in label
workflow") didn't correctly output subtarget resulting in calling with
an empty subtarget. Fix this and correctly output generated subtarget.
Fixes: bf8187d5dc ("CI: use split target and subtarget in label workflow")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 8aa5a86010)
With eecc6e4811 ("CI: rework build workflow to have split target and
subtarget directly") target and subtarget are split in 2 different
variables. Label workflow were not aligned to this change and are
currently broken.
Fix them and correctly pass split target and subtarget.
Fixes: eecc6e4811 ("CI: rework build workflow to have split target and subtarget directly")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit bf8187d5dc)
Instead of referring to a redundant job and ENV variables, rework build
workflow to accept and require split target and subtarget and use them
directly from inputs.
Rework each user and pass a JSON of tuple to matrix include with each
target/subtarget combination to test. Special notice this doesn't use
the github actions matrix combination feature but reference each
specific tuple of target and subtarget to test.
Just a cleanup no behaviour change intended.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit eecc6e4811)
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Use buildbot user on git diff check instead of using git config
safe directory.
This should accomplish the same result but should be a better approach
following safe practice enforced by git.
Fixes: a7747e8670 ("ci: fix check kernel patches job")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 6c80a578a4)
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Currently all 23.05 related CI jobs are failing as the containers are
not available, so lets fix it by pushing those containers when the
version.mk changes.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 8fc2a0f00f)
Currently the check fails due to the following error:
warning: Not a git repository. Use --no-index to compare two paths outside a working tree
usage: git diff --no-index [<options>] <path> <path>
Thats likely caused by commit 1cb8cdbf07 ("ci: use new buildbot worker
images with Debian 11") which contains a patched Git version with CVE
security fixes introduced in DLA-3239-2:
Multiple issues were found in Git, a distributed revision control
system. An attacker may cause other local users into executing arbitrary
commands, leak information from the local filesystem, and bypass
restricted shell.
Note: Due to new security checks, access to repositories owned and
accessed by different local users may now be rejected by Git; in case
changing ownership is not practical, git displays a way to bypass these
checks using the new "safe.directory" configuration entry.
So lets opt-out of this new behavior by setting `safe.directory=*` and
thus force Git to consider all Git repositories as safe regardless of
their owner, since we need to trust those sources anyway and it should
be likely more robust solution, then fiddling with filesystem
permissions.
Fixes: 1cb8cdbf07 ("ci: use new buildbot worker images with Debian 11")
References: https://www.debian.org/lts/security/2022/dla-3239-2
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Debian 10 LTS support ends on 6/2024, so it makes no sense to use it as
a base for 23.05 release, so lets switch to Debian 11 which should've
LTS support till 6/2026.
References: f2744543fa
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Coverity Scan is a static code analysis service focused on open source
software quality and security, so lets scan various OpenWrt components
every Friday for the start.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Upload proposed refreshed patches if the check fails.
This should help devs refresh the patches if they don't have access to a
buildroot.
Devs should ALWAYS refresh the patches before submitting and merging
commits.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Our buildbot build a different external toolchain/sdk for each build.
This cause the idea of using the tar hash to cache it broken and wrong.
This makes the github cache bloated and remove space for ccache cache.
Drop cache for external toolchain/sdk as the feature is broken and cause
problems to ccache cache.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use openwrt official tools container by default.
Fork will use openwrt tools container by default.
This can be disabled by setting the option use_openwrt_container to
false for the build.yml and check-kernel-patches.yml.
The push-containers workflow is disabled on forks. The workflow can be
reenabled by commenting the condition in push-containers.yml.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Current job triggers based on matching of changed paths is quite
limited, so lets make it possible to additionally trigger manual CI jobs
by adding CI specific pull request build labels:
* `ci:target:x86:64` label is going to trigger CI target check jobs for
x86/64 (sub)target.
* `ci:kernel:x86:64` label is going to trigger CI kernel check jobs for
x86/64 (sub)target.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
We may find in a situation where due the queue an old run finish after
the last run, resulting in the containers getting overwritten with an
old version.
Limit the push-containers workflow to one concurrent run and cancel any
run in progress.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Directly copy prebuilt tools in container instead of creating an
archieve and extracting it later in other workflows.
Update build workflow to support this new implementation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
We can now drop the dl dir in the prebuilt tools tar as package archieve
is not a requirement anymore and won't trigger a package recompile.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Testing target changes was only set for push events. Enable this also
for pull request events to enable testing pr making specific target
changes.
Fixes: 57a02cbbff ("CI: kernel: test each target with additional changes than target/linux")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Test each target if there are additional changes than target/linux.
This is needed to do wide test with changes to kmods, include/kernel and
changes to the workflow files.
While at it also cleanup and rework the code to drop duplication.
Also drop since_last_remote_commit to better track changes.
Fixes: 04ada8bc41 ("CI: kernel: build only changed targets")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add concurrency limits for pull request test so that on pull request
refresh old jobs are cancelled.
The group is created based on the github ref + workflow name and the
workflow is cancelled only it it comes from a pull_request event.
Push events are not affected by this limit.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Detect changes in commit and build only changed targets.
If a change is related to the generic target, build test each target.
The matrix json is split. For target check patch only the first
subtarget is selected, for build test each target subtarget is built.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
If for whatever reason external toolchain can't be found or downloaded,
fallback to internal toolchain build.
This can be useful when new target are introduced and external toolchain
are not present in openwrt fileserver.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
