Add checks to test if toolchain container can be used.
This is to handle case of new target or migration of any sort.
If the toolchain container can't be found, the tools container is used
instead.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add option to configure container to use for build test.
By default the tools container is used if no option is provided.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Currently all 23.05 related CI jobs are failing as the containers are
not available, so lets fix it by pushing those containers when the
version.mk changes.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Instead of referring to a redundant job and ENV variables, rework build
workflow to accept and require split target and subtarget and use them
directly from inputs.
Rework each user and pass a JSON of tuple to matrix include with each
target/subtarget combination to test. Special notice this doesn't use
the github actions matrix combination feature but reference each
specific tuple of target and subtarget to test.
Just a cleanup no behaviour change intended.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use buildbot user on git diff check instead of using git config
safe directory.
This should accomplish the same result but should be a better approach
following safe practice enforced by git.
Fixes: a7747e8670 ("ci: fix check kernel patches job")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Currently the check fails due to the following error:
warning: Not a git repository. Use --no-index to compare two paths outside a working tree
usage: git diff --no-index [<options>] <path> <path>
Thats likely caused by commit 1cb8cdbf07 ("ci: use new buildbot worker
images with Debian 11") which contains a patched Git version with CVE
security fixes introduced in DLA-3239-2:
Multiple issues were found in Git, a distributed revision control
system. An attacker may cause other local users into executing arbitrary
commands, leak information from the local filesystem, and bypass
restricted shell.
Note: Due to new security checks, access to repositories owned and
accessed by different local users may now be rejected by Git; in case
changing ownership is not practical, git displays a way to bypass these
checks using the new "safe.directory" configuration entry.
So lets opt-out of this new behavior by setting `safe.directory=*` and
thus force Git to consider all Git repositories as safe regardless of
their owner, since we need to trust those sources anyway and it should
be likely more robust solution, then fiddling with filesystem
permissions.
Fixes: 1cb8cdbf07 ("ci: use new buildbot worker images with Debian 11")
References: https://www.debian.org/lts/security/2022/dla-3239-2
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Debian 10 LTS support ends on 6/2024, so it makes no sense to use it as
a base for 23.05 release, so lets switch to Debian 11 which should've
LTS support till 6/2026.
References: f2744543fa
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Coverity Scan is a static code analysis service focused on open source
software quality and security, so lets scan various OpenWrt components
every Friday for the start.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Upload proposed refreshed patches if the check fails.
This should help devs refresh the patches if they don't have access to a
buildroot.
Devs should ALWAYS refresh the patches before submitting and merging
commits.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Our buildbot build a different external toolchain/sdk for each build.
This cause the idea of using the tar hash to cache it broken and wrong.
This makes the github cache bloated and remove space for ccache cache.
Drop cache for external toolchain/sdk as the feature is broken and cause
problems to ccache cache.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use openwrt official tools container by default.
Fork will use openwrt tools container by default.
This can be disabled by setting the option use_openwrt_container to
false for the build.yml and check-kernel-patches.yml.
The push-containers workflow is disabled on forks. The workflow can be
reenabled by commenting the condition in push-containers.yml.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Current job triggers based on matching of changed paths is quite
limited, so lets make it possible to additionally trigger manual CI jobs
by adding CI specific pull request build labels:
* `ci:target:x86:64` label is going to trigger CI target check jobs for
x86/64 (sub)target.
* `ci:kernel:x86:64` label is going to trigger CI kernel check jobs for
x86/64 (sub)target.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
We may find in a situation where due the queue an old run finish after
the last run, resulting in the containers getting overwritten with an
old version.
Limit the push-containers workflow to one concurrent run and cancel any
run in progress.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Directly copy prebuilt tools in container instead of creating an
archieve and extracting it later in other workflows.
Update build workflow to support this new implementation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
We can now drop the dl dir in the prebuilt tools tar as package archieve
is not a requirement anymore and won't trigger a package recompile.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Testing target changes was only set for push events. Enable this also
for pull request events to enable testing pr making specific target
changes.
Fixes: 57a02cbbff ("CI: kernel: test each target with additional changes than target/linux")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Test each target if there are additional changes than target/linux.
This is needed to do wide test with changes to kmods, include/kernel and
changes to the workflow files.
While at it also cleanup and rework the code to drop duplication.
Also drop since_last_remote_commit to better track changes.
Fixes: 04ada8bc41 ("CI: kernel: build only changed targets")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add concurrency limits for pull request test so that on pull request
refresh old jobs are cancelled.
The group is created based on the github ref + workflow name and the
workflow is cancelled only it it comes from a pull_request event.
Push events are not affected by this limit.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Detect changes in commit and build only changed targets.
If a change is related to the generic target, build test each target.
The matrix json is split. For target check patch only the first
subtarget is selected, for build test each target subtarget is built.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
If for whatever reason external toolchain can't be found or downloaded,
fallback to internal toolchain build.
This can be useful when new target are introduced and external toolchain
are not present in openwrt fileserver.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
We're currently missing log output in cases where `configure` fails
which returns 77 as its error code:
make[3]: *** [Makefile:118: elfutils-0.188/.configured_889556d2f423f99e091beece9c8d870a] Error 77
So lets adjust the regexps so they can handle multiple digits.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
When a new tag for a release is created, the just checkout repo from
github actions will already have such tag locally created.
This will result in git fetch --tags failing with error rejecting the
remote tag with (would clobber existing tag).
Add -f option to overwrite any local tags and always fetch them from
remote.
Fixes: e24a1e6f6d ("CI: build: add support for external toolchains from stable branch")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
This adds the target label also to changes in packages which are target
specific like the boot loader of a target or some drivers which are only
use on one target.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Generilize tools workflow for future usage in shared workflow for tools
build.
Split tools workflow to tools and push-containers:
- tools just execute build test
- push-containers build and push prebuilt containers
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
This should have been amended in the previous commit.
Fixes: 65e72f8e38 ("github: add issue template")
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Add an issue template with required fields, instructions how to easily
get some of that required data from the device. Aside from that, also
add some links to the OpenWrt contact page, and for feature requests,
link to the forum.
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
We don't need to checkout feed and install feeds for kernel tests. This
saves up to 2 minutes for each target kernel build test.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
On cache hit, skip sdk adapt to external toolchain. This is needed because we
cache the already extracted sdk and that is already adapted to be used
as external toolchain.
Rerunning the adap step will result in the test to fail for missing file
as the file are already got wrapped to the external toolchain format.
Fixes: 42f0ab028e ("CI: build: fix use of sdk as toolchain")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The toolchain included in a sdk have a different format than an external
toolchain tar.
Since sdk is a more integrated setup doesn't use and include wrapper bin
that use the external toolchain config and use an alternative and more
standard way to include all the toolchain headers.
External toolchain use wrapper.sh to append the configured include
header when each tool is called.
Fix the sdk toolchain by reverting their own sdk wrapper scripts and to
simulate an external toolchain build copying what is done in the
toolchain target makefile.
This handle compilation error and warning caused by not using fortify
header on building packages.
Fixes: 006e52545d ("CI: build: add support to fallback to sdk for external toolchain")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Since kernel and packages workflow now use a shared build workflow, they
also need to react on changes on these shared workflow.
Fix this and add these shared workflow to the event paths to check.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The current match logic doesn't handle test for push events related to
stable release (example openwrt-22.03) but only fork with the related
prefix (example openwrt-22.03-fixup)
Fix wrong matching and while at it also add extra checks to other
matching (check if the branch name actually start with the requested
prefix)
Fixes: e24a1e6f6d ("CI: build: add support for external toolchains from stable branch")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The current match logic doesn't handle test for push events related to
stable release (example openwrt-22.03) but only fork with the related
prefix (example openwrt-22.03-fixup)
Fix wrong matching and while at it also add extra checks to other
matching (check if the branch name actually start with the requested
prefix)
Fixes: abe8a48242 ("CI: build: add support for per branch tools container")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to use sdk as external toolchain if the packaged external
toolchain tar is not found on openwrt servers for build shared workflow.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to use external toolchains from stable branch if we are
testing commit targeting stable openwrt branch in kernel and packages
workflow.
With pr the target branch is parsed and the right toolchain is used.
To use the stable toolchain for local testing the branch needs to have
the prefix openwrt-[0-9][0-9].[0-9][0-9]- (example openwrt-21.02-fixup)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support in build shared workflow for per branch tools container.
With pr the target branch is parsed and the right container is used.
To use the stable container for local testing the branch needs to have
the prefix openwrt-[0-9][0-9].[0-9][0-9]- (example openwrt-21.02-fixup)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to push per branch container tools.
For anything not official stick to latest tag that correspond to test
run from master.
If we are testing something for one of the openwrt stable branch, parse
the branch name or the tag and push dedicated tools containers.
To use the stable container for local testing the branch needs to have
the prefix openwrt-[0-9][0-9].[0-9][0-9] (example openwrt-21.02-fixup)
Any branch that will match this pattern openwrt-[0-9][0-9].[0-9][0-9]
will refresh the tools container with the matching tag.
(example branch openwrt-22.03 -> tools:openwrt-22.03)
(example branch openwrt-22.03-test -> tools:openwrt-22.03)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
This adds options to build all boards of a selected target and an
additional option to build the testing kernel instead of the normal
kernel. This can be used by other trigger work flows.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This adds an option to build with internal toolchain. This can be used
to build targets which are currently not build by the OpenWrt build bots
and which needs their own toolchain build for every build.
Building the toolchain takes about 30 minutes compared to using the
external toolchain which takes some seconds.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Extract the building of OpenWrt into an own workflow which is then
triggered by the kernel.yml and packages.yml workflow with different
inputs. This allows us to share much of the code of the workflow.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
The label used for stable branch is in the form of
release/[0-9][0-9].[0-9][0-9]
Currently we apply the name of the target branch as the label, fix this
and correctly use the current label.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to tag pr targeting stable branch matching the simple regex
of openwrt-[0-9][0-9].[0-9][0-9]. The tag that will be added will match
the pr target branch.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Activate CONFIG_AUTOREMOVE to match the settings used to build the
pre-build tools. This has to match the pre-build tools to not rebuild
them.
This prevents the tools being rebuild in packages.yml.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This will build OpenWrt for MIPS malta BE and x86 64 Bit with all
packages and kernel modules activated. It is triggered when something
changes in the build system or when a package definition is changed.
This task probably needs 90 minutes to execute, but I hope that it
will find build problems in pull requests early.
This intentionally does not activate the feeds, because building them
too would take too long. We only build x86/64 and malta/be to save
resources.
I would like to detect build problems when a package is changed. We
often had build breaks when a package version was increased sometime
even in other packages which used it as a dependency.
This is based on the .github/workflows/packages.yml workflow.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Target major version of labeler to include minor fixes and use always
the latest major version with included fixes.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Bump actions/download,upload-artifact action to v3 on every workflow
to mute node deprecation warning.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
To actually use ccache cache on kernel test from pr, the kernel workflow
has to be run first from a push action.
This will permit as a side effect to test merged commits and catch commit
that may cause regression in kernel compilation even outside the github
system.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use ccache to speedup kernel compilation.
Ccache dir is cached across each build test. To refresh ccache directory
we generate an hash of the kernel include files, that includes the
kernel versions of every kernel supported and the kernel compile
includes.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use tools:latest container with prebuilt host tools to speedup kernel
compilation in kernel workflow.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Currently each Kernel compilation takes about 30 minutes of which 20
minutes are used to compile our tools. While the toolchain is downloaded
and instantly ready the tools are missing.
This commit starts uploading a Docker container including compiled tools
which are ready to use. It is automatically updated whenever any tools
are changed.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Co-Developed-by: Christian Marangi <ansuelsmth@gmail.com>
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Instead of using a fresh Linux installation which is setup every time
use the Buildbot container which is used for our own Buildbot
infrastructure, too.
While at it also tidy up the workflow to make it more consistent with
other workflow.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Co-Developed-by: Christian Marangi <ansuelsmth@gmail.com>
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Instead of waiting for complete workflow finish, then downloading the
artifacts, unpacking them and inspecting them, lets try to make the
build failure immediately visible in the log output:
====== Make errors from logs/target/linux/compile.txt ======
* Legacy (non-UHI/non-FIT) Boards
*
Support MIPS SEAD-3 boards (LEGACY_BOARD_SEAD3) [N/y/?] (NEW)
Error in reading or end of file.
make[6]: *** [scripts/kconfig/Makefile:77: syncconfig] Error 1
make[5]: *** [Makefile:616: syncconfig] Error 2
make[4]: *** [Makefile:736: include/config/auto.conf.cmd] Error 2
make[3]: *** [Makefile:24: build_dir/target-mipsel-openwrt-linux-musl_musl/linux-ramips_mt7620/linux-5.15.62/.modules] Error 2
make[2]: *** [Makefile:11: compile] Error 2
time: target/linux/compile#30.09#11.30#37.92
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Grant pull-requests write permission to the labeler workflow and
read-only to everything else.
Signed-off-by: Alex Low <aleksandrosansan@gmail.com>
[ wrap to 80 columns and fix wrong author as requested by author itself ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
This adds GitHub CI action which makes use of 'Labeler', allowing
automatic labeling of new PRs, based on the modified files paths.
Below labels are supported and more can be added later:
- 'target/*'
- 'target/imagebuilder'
- 'kernel'
- 'core packages'
- 'build/scripts/tools'
- 'toolchain'
- 'GitHub/CI'
For more information:
https://github.com/marketplace/actions/labeler
Signed-off-by: Piotr Dymacz <pepe2k@gmail.com>
Actually package kmods in kernel workflow to catch dependency error and
other problem that may arise from kmods packaging.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Cache external toolchain for each target to remove load from openwrt cdn
server and make the external toolchain setup quicker.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Activate building all kernel modules.
This builds all kernel modules from the core packages and the feeds.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Instead of cloning the feeds from the default location at
git.openwrt.org use the github action to clone them directly from
github. We saw some error messages when cloning from git.openwrt.org,
probably related to some rate limiting applied. Cloning from github
within a github action should work more stable.
The "./scripts/feeds update -a" script will use the already checked out
feed repositories and not clone them again from git.openwrt.org, but it
will also not change the branch name.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Show the configuration used to build OpenWrt before starting the build.
This should make it easier for people to reproduce problems when it
fails.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Run github actions insider buildbot docker container.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[ run container under buildbot user ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add Github Actions yaml script to build test kernel PR changes for
each target.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[ add commit description ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
* Always store build logs
* Store .config as an artifact
* Rename job to `tools-{ os }` for log archive without spaces
* Run CI job on changes to the CI file itself
Signed-off-by: Paul Spooren <mail@aparcar.org>
Artifacts can only be uploaded from inside the GITHUB_WORKSPACE. While
the Linux CI jobs run inside that per default, a special case-sensitive
mount outside the GITHUB_WORKSPACE is used for macOS builds.
To make log artifacts work for both macOS and Linux, move logs/ folder
to GITHUB_WORKSPACE on failures.
Signed-off-by: Paul Spooren <mail@aparcar.org>
This commits adds GitHub CI to check that all tools compile on both
Ubuntu and macOS. Since running in parrallel this should also detect
badly set depdendencies within tools/Makefile.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Due to a limited grep pattern, subjects containing dots like `image.mk`
were falsely reported as problematic. Extend pattern to allow dots.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The formal checks verify the following things:
- Commits does not contain any merge commits
- Signed by a real name
- Commit titles starts with an `<area>:`
- Author name matches signed of name
- Commit message is not empty
Signed-off-by: Paul Spooren <mail@aparcar.org>
By adding this file a badge should appear in the GitHub web interface to
motivate people donate money to the OpenWrt project.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Update README to include Openwrt branding and improve wording.
Point at the Openwrt wiki in .github templates.
Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
This text is used by GitHub to remind important things to
people sending PRs through the GitHub's web interface.
See here for more information
https://github.com/blog/2111-issue-and-pull-request-templates
It links to the wiki page about submission rules.
Signed-off-by: Alberto Bursi <alberto.bursi@outlook.it>
