CDNs are known to ship outdated or corrupted files, if it unpacks
correctly, it necessarily doesn't mean, that we're using the desired
content. So lets fix it by checking the tarball as well.
I'm adding GPG checking explicitly, its not needed, but just double
checking, that everything is working as expected on build
infrastructure.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Add support for getting ccache cache from S3.
ccache is archieved in a tar and downloaded from S3 Cloud Storage.
For push events, ccache is then uplodaed back to S3 to refresh and have
a ccache cache always fresh.
An additional workflow is added to upload files to an S3 Cloud Storage
from artifacts uplodaed to github. The minio tool is used to upload
files to S3.
If the ccache can't be downloaded from s3, we fallback to github cache
system.
Also limit s3 upload to the openwrt repository since external fork won't
have (obviously) the required secrtes to upload data to the S3 Cloud
Storage.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Limit ccache cache save/delete only on push events. Saving ccache
cache for pull request will result in bloat and refreshing ccache is not
possible due to security measure on enforcing read permission on
pull_request events.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
CCache cache is currently broken due to a funny bug in ccache compiler
type detection. It seems ccache compiler type detection is very fragile
and with the use of external toolchain doesn't correctly detect the
type.
The type detected is set to other instead of gcc resulting in ccache
complaining for unsupported compiler options.
To handle this problem, force the compiler type to gcc to make ccache
correctly work and speedup compilation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add new input to define custom ccache cache type. This is useful to use
a different ccache cache for some special workflow that may do more test
than simple kernel compilation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add option to disable use of ccache. This can be useful for some
sensible test that should not use ccache as they can cause side effects
of any sort. (example Coverity Scan)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Github Actions cache doesn't permit to overwrite cache if it does
already exist. As a trick to refresh and have fresh ccache pool,
delete the ccache cache if it does exist with the help of Github REST
API. An additional permission is needed to access this API. Add this
permittion to each user of the build workflow.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Split caching ccache in separate restore and save jobs to always refresh
the ccache across different runs. Currently if a key is restored, cache
is not saved resulting in a less useful ccache that benefits from
multiple runs.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Drop redundant generare ccache hash job as that can be done by
integrated github expressions to generate an hash.
The only change is that the integrated way generate a sha256 hash
instead of an md5 sum.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Commit 1cb8cdb ("ci: use new buildbot worker images with Debian 11")
introduced new Git version with strict rules for owner of the git
directory.
To handle this and not cause major change, just move the parsing before
the change of ownership of the openwrt directory permitting the correct
run of git fetch command with the same user that did the repository
checkout.
Fixes: 1cb8cdb ("ci: use new buildbot worker images with Debian 11")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to use container included external toolchain and skip
redownloading external sdk for each test.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add checks to test if toolchain container can be used.
This is to handle case of new target or migration of any sort.
If the toolchain container can't be found, the tools container is used
instead.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add option to configure container to use for build test.
By default the tools container is used if no option is provided.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Instead of referring to a redundant job and ENV variables, rework build
workflow to accept and require split target and subtarget and use them
directly from inputs.
Rework each user and pass a JSON of tuple to matrix include with each
target/subtarget combination to test. Special notice this doesn't use
the github actions matrix combination feature but reference each
specific tuple of target and subtarget to test.
Just a cleanup no behaviour change intended.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Coverity Scan is a static code analysis service focused on open source
software quality and security, so lets scan various OpenWrt components
every Friday for the start.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Our buildbot build a different external toolchain/sdk for each build.
This cause the idea of using the tar hash to cache it broken and wrong.
This makes the github cache bloated and remove space for ccache cache.
Drop cache for external toolchain/sdk as the feature is broken and cause
problems to ccache cache.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Use openwrt official tools container by default.
Fork will use openwrt tools container by default.
This can be disabled by setting the option use_openwrt_container to
false for the build.yml and check-kernel-patches.yml.
The push-containers workflow is disabled on forks. The workflow can be
reenabled by commenting the condition in push-containers.yml.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Directly copy prebuilt tools in container instead of creating an
archieve and extracting it later in other workflows.
Update build workflow to support this new implementation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
If for whatever reason external toolchain can't be found or downloaded,
fallback to internal toolchain build.
This can be useful when new target are introduced and external toolchain
are not present in openwrt fileserver.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
When a new tag for a release is created, the just checkout repo from
github actions will already have such tag locally created.
This will result in git fetch --tags failing with error rejecting the
remote tag with (would clobber existing tag).
Add -f option to overwrite any local tags and always fetch them from
remote.
Fixes: e24a1e6f6d ("CI: build: add support for external toolchains from stable branch")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
On cache hit, skip sdk adapt to external toolchain. This is needed because we
cache the already extracted sdk and that is already adapted to be used
as external toolchain.
Rerunning the adap step will result in the test to fail for missing file
as the file are already got wrapped to the external toolchain format.
Fixes: 42f0ab028e ("CI: build: fix use of sdk as toolchain")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The toolchain included in a sdk have a different format than an external
toolchain tar.
Since sdk is a more integrated setup doesn't use and include wrapper bin
that use the external toolchain config and use an alternative and more
standard way to include all the toolchain headers.
External toolchain use wrapper.sh to append the configured include
header when each tool is called.
Fix the sdk toolchain by reverting their own sdk wrapper scripts and to
simulate an external toolchain build copying what is done in the
toolchain target makefile.
This handle compilation error and warning caused by not using fortify
header on building packages.
Fixes: 006e52545d ("CI: build: add support to fallback to sdk for external toolchain")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The current match logic doesn't handle test for push events related to
stable release (example openwrt-22.03) but only fork with the related
prefix (example openwrt-22.03-fixup)
Fix wrong matching and while at it also add extra checks to other
matching (check if the branch name actually start with the requested
prefix)
Fixes: e24a1e6f6d ("CI: build: add support for external toolchains from stable branch")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The current match logic doesn't handle test for push events related to
stable release (example openwrt-22.03) but only fork with the related
prefix (example openwrt-22.03-fixup)
Fix wrong matching and while at it also add extra checks to other
matching (check if the branch name actually start with the requested
prefix)
Fixes: abe8a48242 ("CI: build: add support for per branch tools container")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to use sdk as external toolchain if the packaged external
toolchain tar is not found on openwrt servers for build shared workflow.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support to use external toolchains from stable branch if we are
testing commit targeting stable openwrt branch in kernel and packages
workflow.
With pr the target branch is parsed and the right toolchain is used.
To use the stable toolchain for local testing the branch needs to have
the prefix openwrt-[0-9][0-9].[0-9][0-9]- (example openwrt-21.02-fixup)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support in build shared workflow for per branch tools container.
With pr the target branch is parsed and the right container is used.
To use the stable container for local testing the branch needs to have
the prefix openwrt-[0-9][0-9].[0-9][0-9]- (example openwrt-21.02-fixup)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
This adds options to build all boards of a selected target and an
additional option to build the testing kernel instead of the normal
kernel. This can be used by other trigger work flows.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This adds an option to build with internal toolchain. This can be used
to build targets which are currently not build by the OpenWrt build bots
and which needs their own toolchain build for every build.
Building the toolchain takes about 30 minutes compared to using the
external toolchain which takes some seconds.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Extract the building of OpenWrt into an own workflow which is then
triggered by the kernel.yml and packages.yml workflow with different
inputs. This allows us to share much of the code of the workflow.
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
