Now that we copy and then delete the Per-Device rootfs linux directory,
it's not valid anymore placing the generated cpio there as artifacts
or subsequent commands need the generated cpio.
To handle this, rework Initramfs compile cpio handling by placing them in
the KERNEL_BUILD_DIR but add to the name the rootfs HASH ID.
To also prevent race condition, generate and access these file under a
lock to prevent fit command to reference a cpio while a parallel
execution is genereting it.
Fixes: 52cc9d82f1 ("kernel: rework Initramfs locking logic")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Fix flock name for gen-initramfs lock as $(2) starts with .HASH making
the lock name gen-initramfs-.HASH.flock
Fix this to a better name of gen-initramfs.HASH.flock
Fixes: 52cc9d82f1 ("kernel: rework Initramfs locking logic")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Fix wrong rework for Initramfs cpio xz compression where it was wrongly
dropped the Per-Device linux directory if condition.
Fixes: 52cc9d82f1 ("kernel: rework Initramfs locking logic")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
To speedup compilation of Per-Device Initramfs, copy the kernel build
directory for each rootfs ID.
This permits concurrent execution of kernel build without conflicting
with each other at the expense of additional disk space usage.
To limit disk space usage, the copied kernel directory is deleted after
the Per-Device Initramfs image is generated and saved.
Link: https://github.com/openwrt/openwrt/pull/16404
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Rework CompileImage/Initramfs locking logic to fix race condition on
concurrent access/compilation.
Rework each intermediate step and group them under one single execution
block. Protect this with a new lock, 'gen-initramfs' and rename the
compile initramfs to compile-initramfs lock name.
This is done to handle corner case scenario where a rootfs with the same
hash ID is generated at the same time. To handle this, we execute
everything under lock and use a specific hash ID lock to prevent
concurrent access/compilation.
Link: https://github.com/openwrt/openwrt/pull/16404
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Commit 0137fbd74b ("kernel: skip rebuilding kernel with
ROOTFS_INITRAMFS_SEPARATE") had a logic error error and didn't account
that the generic initramfs is still needed to be built to enable support
for the cpio compression and other config specific to initramfs. With
that commit we completely skip rebuilding the kernel with those new
options.
To better handle this, skip kernel build ONLY when we are handling Per
Device Rootfs, permitting the first generic build to be actually called
with the new options.
Fixes: 0137fbd74b ("kernel: skip rebuilding kernel with ROOTFS_INITRAMFS_SEPARATE")
Link: https://github.com/openwrt/openwrt/pull/15912
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Also fix CopyImage for Separate Initramfs with Per Device Rootfs.
Fixes: 7bffb3f72b ("kernel: fix CopyImage function with Per Device Rootfs")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Some target define custom kernel images with KERNEL_INITRAMFS_NAME to
reference .elf variant of vmlinux.
With Per Device Rootfs, the expected format is
image.suffix.extension.ROOTFS_ID, while in CopyImage we are currently
generating images with image.suffix.ROOTFS_ID.extension making some
target failing.
Fix CopyImage function to correctly follow the expected pattern.
Fixes: 97fd059e7e ("image: respect TARGET_PER_DEVICE_ROOTFS for initramfs")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
With PER_DEVICE_ROOTFS, rebuilding kernel is needed to embed the cpio
image in the kernel image. With ROOTFS_INITRAMFS_SEPARATE, the cpio
image is external hence we can reuse the same kernel image without
rebuilding it.
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Fit command makes use of CONFIG_TARGET_ROOTFS_INITRAMFS_SEPARATE as the
cpio is provided externally and is not embedded in the kernel image.
As done with embedded cpio, also handle PER_DEVICE_ROOTFS by generating
a cpio for each rootfs and reference them by the ROOTFS_ID generated
previously. The generated cpio are placed in the linux directory + the
package ID.
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Initramfs images were using a common rootfs (TARGET_DIR) for all
devices, ignoring TARGET_PER_DEVICE_ROOTFS. If a single device required
a package to build a functional initramfs image, it should be included
by default for all devices or that device should be isolated into a new
subtarget. Now the initramfs will be built using the target-specific
Implementing Per Device Rootfs for Initramfs is not trivial as the
rootfs needs to be embedded in the kernel image. The kernel supports an
option to define the initramfs location and the image generation for the
kernel can't be run in parallel as other checks are done to config and
other arch dependent files.
To handle this, we prepare a config for each rootfs and we generate the
images under lock to prevent problem with parallel execution.
Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
[ rework implementation for locking support ]
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Simplify SetInitramfs compression ALGO config setup by using Makefile
foreach.
While at it also make it more readable.
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Rework SetInitramfs functions to take a second arg to define the
location of the .config. This is needed in preparation for PER_ROOTFS
Initramfs support as we will prepare .config in dedicated directory and
use them only later when the image is actually built.
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Allow Kernel/CompileImage/Initramfs to use a different rootfs location.
If the additional arg is not defined, TARGET_DIR is used by default.
This allows the caller to customize the kernel initramfs for different
rootfs.
Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
[ simplify commit and rework commit description ]
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Kernel module packages compiling is not cached (e.g. mac80211)
even with CONFIG_CCACHE on.
CC should be set to KERNEL_CC in KERNEL_MAKE_FLAGS at kernel.mk
to allow kernel module packages using ccache.
Signed-off-by: Zeyu Dong <dzy201415@gmail.com>
New kernel version 6.1 introduced new INITRAMFS option. Add them to the
Initramfs functions to correctly compile initramfs images.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Commit 54070a1 was added to allow building proper SDKs with kernels <
5.10. Now that all targets use at least kernel 5.10 it can be reverted.
Signed-off-by: Sebastian Kemper <sebastian_ml@gmx.net>
Requires: tools/libdeflate
fix consistency of executable to use
`$(STAGING_DIR_HOST)/bin/libdeflate-gzip`, and not system-installed ones
from the usual environment `PATH`;
this affects option `CONFIG_KERNEL_INITRAMFS_COMPRESSION_GZIP`
this may have worked in the past but only via side effect of having the
binaries on the host system (and whatever unpredictable version or
patchset those might be), and did not use the improved but totally
compatible libdeflate-gzip
Fixes: 330bd380e8 ("image: allow building FIT and uImage with ramdisk")
Signed-off-by: Tony Butler <spudz76@gmail.com>
Requires: tools/bzip2
fix consistency of executable to use `$(STAGING_DIR_HOST)/bin/bzip2`, and
not system-installed ones from the usual environment `PATH`;
this affects option `CONFIG_KERNEL_INITRAMFS_COMPRESSION_BZIP2`
this may have worked in the past but only via side effect of having the
binaries on the host system (and whatever unpredictable version or
patchset those might be)
Fixes: 330bd380e8 ("image: allow building FIT and uImage with ramdisk")
Signed-off-by: Tony Butler <spudz76@gmail.com>
libdeflate decompresses much faster than gzip.
Example:
~/d/openwrt> time gzip -dc dl/cmake-3.25.1.tar.gz > /dev/null
________________________________________________________
Executed in 1.01 secs fish external
usr time 912.61 millis 1.67 millis 910.94 millis
sys time 32.21 millis 0.25 millis 31.96 millis
~/d/openwrt> time libdeflate-gzip -dc dl/cmake-3.25.1.tar.gz > /dev/null
________________________________________________________
Executed in 523.04 millis fish external
usr time 415.48 millis 1.07 millis 414.41 millis
sys time 107.74 millis 0.15 millis 107.59 millis
~/d/openwrt> time gzip -dc dl/tessdata-4.1.0.tar.gz > /dev/null
________________________________________________________
Executed in 8.99 secs fish external
usr time 8.90 secs 530.00 micros 8.90 secs
sys time 0.07 secs 63.00 micros 0.07 secs
~/d/openwrt> time libdeflate-gzip -dc dl/tessdata-4.1.0.tar.gz > /dev/null
________________________________________________________
Executed in 2.74 secs fish external
usr time 2.38 secs 537.00 micros 2.38 secs
sys time 0.35 secs 66.00 micros 0.35 secs
Place libdeflate into tools-core as it is needed to decompress other
archives.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Requires: tools/lz4, tools/lzop
complete the wiring so that these options work:
* `CONFIG_KERNEL_INITRAMFS_COMPRESSION_LZO`
* `CONFIG_KERNEL_INITRAMFS_COMPRESSION_LZ4`
Signed-off-by: Tony Butler <spudz76@gmail.com>
[remove blocking dependencies for separate ramdisk, fix lzop options]
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
When building UML this showed up as of late (2022-09-01):
| Makefile:1335: *** Headers not exportable for UML. Stop.
This message popped up because of an upstream patch that
fixed this error not showing up sooner. For more information see:
<https://lore.kernel.org/lkml/20220901011252.550830-1-masahiroy@kernel.org/T/>
Now, I don't think we can just skip the kernel headers.
Some userspace applications are going to need these. So,
The next best alternative I can think of, is to go with
the headers by the host arch the UML is compiled for.
(Technically, ARCH= should be safe for all other targets
as well... But let's not poke a hornet's nest)
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Make sure the timestamp of the root directory of the initramfs is set
to SOURCE_DATE_EPOCH as well.
Fixes: 29d7461d11 ("kernel: set options to make external initramfs reproducible")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Make sure xz uses at least 2 threads so compression always runs in
multi-threaded mode as the resulting file in single-threaded mode
differs.
Fixes: 29d7461d11 ("kernel: set options to make external initramfs reproducible")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
CONFIG_EXTERNAL_CPIO is a string variable, hence testing for 'y'
doesn't make much sense here.
Fixes: 330bd380e8 ("image: allow building FIT and uImage with ramdisk")
Reported-by: Huangbin Zhan <zhanhb88@gmail.com>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The $(LINUX_DIR)/.config timesptamp changes between runs of
make target/compile and make target/install (which builds the image).
Kernel-dependent packages and out of tree modules are built in between
those runs, and they check the .config timestamp to decide if they need
to be rebuilt.
Save the target/compile .config to use its timestamp if the file does
not change between runs. That way the subsequent kernel packages are
not unnecessarily rebuilt when you run 'make' back to back.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
This fixes a regression introduced with commit
5ed1e5140a ("build: build kernel image
before building modules/packages").
Before this commit the make target would always include "modules",
resulting in a MODPOST and a complete Module.symvers file. Since this
commit a MODPOST of the kernel modules is not guaranteed for kernels <
5.10. This results in some broken SDKs in which external packages that
depend on exported symbols from kernel modules fail to compile.
Adding "modules" back to the calls to the CompileImage defines fixes the
regression. For kernels > 5.10 this is not needed, but it doesn't cause
any harm either.
Tested with kernels 5.4.x and 5.10.x.
Signed-off-by: Sebastian Kemper <sebastian_ml@gmx.net>
Use an 'if' so the absence of $(LINUX_DIR)/user_headers doesn't make the
line evaluate to false and cause the build to fail.
Signed-off-by: Matt Merhar <mattmerhar@protonmail.com>
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if
make TOPDIR="$(pwd)" -C "$pkgdir" compile
was called manually. In most of the cases, I just saw warnings like this:
make: Entering directory '/home/.../package/gluon-status-page'
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
[...]
While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.
After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.
Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
Instead of embedding the initrd cpio archive into the kernel, allow
for having an external ramdisk added to the FIT or uImage.
This is useful to overcome kernel size limitations present in many
stock bootloaders, as the ramdisk is then loaded seperately and doesn't
add to the kernel size. Hence we can have larger ramdisks to host ie.
installers with all binaries to flash included (or a web-based
firmware selector).
In terms of performance and total size the differences are neglectible.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Linux 5.9 introduces support for ZSTD ramdisk and initrd compression,
make sure we enable/disable the relevant options when building an
initramfs enabled kernel.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
When we use an external kernel tree which may not have been fully
cleaned, there may be user_headers left which do not match the target
architecture, leading to build failures for packages that do an explicit
inclusion of user_headers (such as iproute2 or iptables). Make sure we
delete them while preparing the directory.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
On non-GNU systems, zcat often does not handle gzip decompression.
Use gzip -dc like the regular unpack command
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is a follow up of 8cb13f4e6d which sets the right timestamp for the
/init file in initramfs. The previous patch doesn't cover it as the
files appear to come from a later step during compilation.
CC: Alexander Couzens <lynxis@fe80.eu>
Signed-off-by: Paul Spooren <mail@aparcar.org>
They are no longer stored in the "testing" subdirectory and are not
available as .tar.xz archives. If -rc is detected download it from the
git.kernel.org and use .tar.gz.
Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
In commit fce35bce0f ("config: support new symbol intro'd in kernel
4.12")
I forgot to remove the initial debug test line.
This clearly is wrong as the same symbol is defined conditionally in the
line below as it should be.
I looked over it as I just checked if the symbol was present now upon
testing it.
Fixes: fce35bce0f ("config: support new symbol intro'd in kernel
4.12")
Signed-off-by: Koen Vandeputte <koen.vandeputte@ncentric.com>
Symbol CONFIG_INITRAMFS_FORCE allows to ignore the value passed by the
bootloader.
By default, all symbols containing INITRAMFS are wiped from the final
config and then re-added conditionally.
Add support for this symbol, as the build will stop otherwise
questioning the user about this option:
* Restart config...
*
*
* General setup
*
Cross-compiler tool prefix (CROSS_COMPILE) []
Compile also drivers which will not load (COMPILE_TEST) [N/y/?] n
...
Initial RAM filesystem and RAM disk (initramfs/initrd) support
(BLK_DEV_INITRD) [Y/n/?] y
Initramfs source file(s) (INITRAMFS_SOURCE) []
Ignore the initramfs passed by the bootloader (INITRAMFS_FORCE)
[N/y/?] (NEW)
Signed-off-by: Koen Vandeputte <koen.vandeputte@ncentric.com>
This allows packages to use kernel make options without the forced
-C $(LINUX_DIR). It also makes it more clear that it to be called from
kernel module packages directly.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
In case we use external and/or git cloned kernels, let the kernel
determine the appropriate KERNELRELEASE. We cannot used
LINUX_UNAME_VERSION because that one gets determined at a later time,
when the kernel is already built proper.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
This reverts commit 0df2c6563a since it
gets in the way of identifying properly which kernel we are running.
This is particularly important if LEDE is using external kernels/git
cloned kernels. We want to make sure we only load modules from that
specific kernel.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
