In 451e2ce006 the code would only move packages separated by dashes,
however OPKG uses underscores to split between package name and version.
Result was the lack of such packages in the packed ImageBuilder.
Use dashes or underscores depending on the select package manger.
Fixes: 451e2ce006 "imagebuilder: fix package inclusion and APK index"
Signed-off-by: Paul Spooren <mail@aparcar.org>
Previously three different ways to include packages in an ImageBuilder
existed:
* buildbot: include libc, kernel (and base-files) in $(IB_LDIR)
* not buildbot, standalone: include all packages in ./packages/
* not buildbot, not standalone: include libc, kernel (and base-files) in
./packages/
First of, the separation between *buildbot* and *not buildbot, not
standalone* is not required, we can just always copy packages to
./packages instead of ever using the special place $(IB_LDIR).
Doing so drops the need to handle the extra case and also allows to
clean up the OPKG package installation, which no longer requries the
`firstword` logic, things are now always at ./packages.
Signed-off-by: Paul Spooren <mail@aparcar.org>
We need to exclude packages from the APK index which must not be
upgraded. To do so, the packages `libc`, `kernel` and `base-files` are
excluded to APK never suggestes them as upgradable.
The previous logic would however match packages like `libcomerr0`,
causing build failures. Make the copying and excluding logic more
precise by adding a single dash.
Signed-off-by: Paul Spooren <mail@aparcar.org>
This commit solves multiple issues. First of just install the three
special packages base-files, libc and kernel directly from the index. In
upstream indexes, those will never appear to prevent accidental upgrades
may breaking the system.
Next, enable caching for the ImageBuilder, which speeds up consecutive
builds from ~33 seconds to ~5 seconds. Using cache however makes APK
create the folder `/var/cache/apk/` which conflicts with the base-files
installation, which ships a symlink from `/var` to `/tmp`, so specify
`--no-cache` for the rootfs initialization.
Lastly, drop the use of `apk update` since APK automatically does that.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Those packages were not copied due to OPKG using an underscore while APK
uses dashes. Remove that char to copy kernel/libc for either APK/OPKG.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Also remove vmlinuz-initramfs files from final imagebuilder image as
these file are not needed.
Fixes: c85348d9ab ("imagebuilder: remove initramfs image files")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Initramfs images are not supported by imagebuilder. With recent changes
to support Per Device Rootfs, we now generate an image and a vmlinux for
each Rootfs and these additional files are all shipped in the
imagebuilder tar.
Drop these new file and any vmlinux-initramfs as they are not used and
increase the final size of the imagebuilder archive.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Fix broken IB_STANDALONE option for OPKG due to an error in ifdef logic
where we weren't adding the required entry to reference the local files
in repositories.conf
Rework the ifdef to more explicit and restore original functionality of
this option.
While at it also provide different README for APK or OPKG.
Fixes: d788ab376f ("build: add APK package build capabilities")
Link: https://github.com/openwrt/openwrt/pull/15599
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
A new option called `USE_APK` is added which generated APK packages
(.apk) instead of OPKG packages (.ipk).
Some features like fstools `snapshot` command are not yet ported
Signed-off-by: Paul Spooren <mail@aparcar.org>
Historically it's possible to leave the `SUBTARGETS` undefined and
automatically fallback to a "generic" subtarget. This however breaks
various downstream scripts which may have expectations around filenames:
While some targets with an explicit generic subtarget contain `generic`
in the filenames of artifacts, implicit "subtargets" don't.
Right now this breaks the CI[1], possibly also scripts using the ImageBuilders.
This commit removes all code that support implicit handling of
subtargets and instead requires every target to define "SUBTARGETS".
[1]: https://github.com/openwrt/openwrt/actions/runs/8592821105/job/23548273630
Signed-off-by: Paul Spooren <mail@aparcar.org>
Fixes the regression so that targets that were installed via a feed can
also be build again with the Image Builder.
Fixes: 84ec8c4 ("imagebuilder: copy from buildroot only target/linux")
Signed-off-by: Florian Eckert <fe@dev.tdt.de>
Tested-by: Thomas Richard <thomas.richard@bootlin.com>
This reduces the size of a single imagebuilder by about 40MB
In example for the target ath79 it would be the sum of generic and <target> directories, so about 16MB,
instead of the whole size of the target directory, about 53MB:
11M target/linux/generic/
3.9M target/linux/ath79/
Signed-off-by: a-gave <agave@dracaena.it>
When using external targets there is a symlink being created for the
target under target/linux which then becomes dangling under Image
Builder. Fix it by dereferencing the possible symlink.
Tested on IB with external target, ipq40xx and mvebu.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
This commit fixes commit "2999f810ff: build,IB: include kmods only in
local builds" which cause the local packages/ folder only to be added
for local builds but no longer for ImageBuilder created by the Buildbot.
The commits intention was to use remote kmods repositories rather than
storing them locally. Accidentally the entire handling of the local
`packages/` was removed.
Re-add the folder and include a README describing what it can be used
for.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Using these config-options to customize the folders used at build-time
makes these folder settings appear in generated archive. This causes the
imagebuilder to be not portable, as it's going to use the build-time folders
on the new systems. Errors look like:
mkdir: cannot create directory '/mnt/build': Permission denied
Makefile:116: recipe for target '_call_image' failed
make[2]: *** [_call_image] Error 1
Makefile:241: recipe for target 'image' failed
make[1]: *** [image] Error 2
The build-time settings of these folders are passed into the archives via
.config file.
The expected behavior is that after unpacking the imagebuilder acts like
these settings have their defaults, using intree folders. So unset the
build-time settings.
Signed-off-by: Sven Roederer <devel-sven@geroedel.de>
Invoke bundle-libraries.sh with any buildroot related directory entries
removed from $PATH to avoid picking up cross versions of utilities like
ldd which will not properly work when used against host executables.
This should fix executable bundling for glibc-target imagebuilders.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
When building images with the imagebuilder, the partition signature
never changes. The signature is generated by hashing SOURCE_DATE_EPOCH
and LINUX_VERMAGIC which are undefined. Prepopulate these variables, as
done by the SDK.
Signed-off-by: Matthew Gyurgyik <matthew@gyurgyik.io>
The ImageBuilder downloads pre-built packages and adds them to images.
This process uses `opkg` which has the capability to verify package list
signatures via `usign`, as enabled per default on running OpenWrt
devices.
Until now this was disabled for ImageBuilders because neither the `opkg`
keys nor the `opkg-add` script was present during first packagelist
update.
To harden the ImageBuilder against *drive-by-download-attacks* both keys
and verification script are added to the ImageBuilder allowing `opkg` to
verify downloaded package indices.
This commit adds `opkg-add` to the ImageBuilder scripts folder. The keys
folder is added to ImageBuilder $TOPDIR to have an obvious place for users to
store their own keys. The `option check_signature` is appended to the
repositories.conf file. All of the above only happens if the Buildbot
runs with the SIGNATURE_CHECK option.
The keys stored in the ImageBuilder keys/ are the same as included in
the openwrt-keyring package. To avoid the chicken-egg problem of
downloading and verifying a package, containing signing keys, the keys
are added during the ImageBuilder generation. They are same as in
shipped images (stored at `/etc/opkg/keys/`).
To allow a local package feed in which the user can add additional
packages, a local set of `usign` and `ucert` keys is generated, same as
building OpenWrt from source. The private key signs the local repository
inside the packages/ folder. The local public key is added to the keys/
folder to be considered by `opkg` when updating repositories. This way a
local package feed can be modified while requiring `opkg` to check
signatures for remote feed, making HTTPS optional.
The new option `ADD_LOCAL_KEY` allows to add the local key inside the
created images, adding the advantage that sysupgrades can validate the
ImageBuilders local key.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Without an absolute path to staging_dir/host/bin/sstrip the Makefile
tries to run a host installed version of sstrip, which is likely not
available.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The buildbots generate a kmod archive which should be used instead of a
local copy. This is possible due to the introduction of a kernelversion
specific feed.
This commit adds the ability of using only signed package feeds.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The `libfakeroot` files are currently missing in the ImageBuilder. As
`fakeroot` is always built, copy those files unconditionally.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Both IB and SDK now use the same logic for packing.
This commit add reproducible multithread compression to the SDK and
corrects the file mtime for both. Previously all files where just copied
over from the build system, generating random mtimes.
Signed-off-by: Paul Spooren <mail@aparcar.org>
This speeds up the packing of the imagebuilder a lot:
imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s
To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.
Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Refer to LINUX_KARCH instead of ARCH when bundling DTS files in the image
builder tarball.
While we're at it, also dereference symbolic links when copying as some
kernel architectures contain symbolic links in their DTS directories.
This fixes aarch64 imagebuilders such as brcm2708/bcm2710 ones in particular
as the kernel refers to "aarch64" as "arm64" internally.
Ref: https://forum.lede-project.org/t/lede-image-builder-problem/3680
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Introduce a new location STAGING_DIR_IMAGE which is intended to be used by
bootloader iamges and similar image-related artifacts.
This directory is guaranteed to be persistent across kernel upgrades which
might involve a removal of KERNEL_BUILD_DIR and is guranteed to be bundled
with the image builder.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Rework the bundle-libraries.sh implementation to use a more robust approach
for executing host binaries through the shipped ELF loader and libraries.
The previous approach relied on symlinks pointing to a wrapper script which
caused various issues, especially with multicall binaries as the original
argv[0] name was not preserved through the ld.so invocation. Another down-
side was the fact that the actual binaries got moved into another directory
which caused executables to fail looking up resources with paths relative
to the executable location.
The new library wrapper implements the following improvements:
- Instead of symlinks pointing to a common wrapper, each ELF executable
is now replaced by a unqiue shell script which retains the original
program name getting called
- Instead of letting ld.so invoke the ELF executable directly, launch
the final ELF binary through a helper program which fixes up the argv[0]
argument for the target program
- Support sharing a common location for the bundled libraries instead of
having one copy in each directory containing wrapped binaries
Finally modify the SDK build to wrap the staging_dir and toolchain binaries
which allows to use the SDK on systems with a different glibc version.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
With symlink tree some directories are just symlinked which
means IB and SDK end up with a symlink instead of an actual
directory; this fixes the missing files by dereferencesing
the directories instead of copying the symlinks.
Signed-off-by: Daniel Dickinson <lede@cshore.thecshore.com>
The libc and kernel package files moved since the introduction of shared
packages and the changed output directory layout. This causes the generated
ImageBuilder archive to lack the necessary "libc" and "kernel" meta packages,
leading to opkg install errors later on.
Use the FeedPackageDir macro to figure out the proper source directory to use.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
We allow to configure the version distribution name;
let's also use it for the tarballs (SDK, ImageBuilder,
and SDK).
Signed-off-by: Daniel Dickinson <openwrt@daniel.thecshore.com>
SVN-Revision: 48081
This introduces a common macro to assemble the correct url templates to
avoid code duplication and have the feed config handling in a central place.
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>
SVN-Revision: 45799
Change the IB packaging to only embed libc, kernel and kmod packages by default
and generate repositories.conf to refer to the remote package repositories.
Introduce a new config option CONFIG_IB_STANDALONE which restores the old
behaviour of building self contained IB archives.
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>
SVN-Revision: 45772
The package index is generated on first use anyway, therefore it makes no
sense to continue shipping it.
Also sstrip the bundled host binaries when packing the IB to save some
additional space.
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>
SVN-Revision: 44293
This commit introduces a new option CONFIG_VERSION_FILENAMES which causes
OpenWrt to embed the version number in generated image files, SDK- and
ImageBuilder archives.
The option is enabled by default if CONFIG_VERSIONOPT is set.
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>
SVN-Revision: 43869
