Add comments to build system makefile functions and variables to help
developers in understanding build system internals and ease the
development process.
This patch adds some documentation examples with proposed doxygen-like
syntax. Hopefully, this would start the discussion and result in
generation of the makefile documentation guidelines.
Signed-off-by: Sergey Matsievskiy <matsievskiysv@gmail.com>
Link: https://github.com/openwrt/openwrt/pull/16888
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit a72c8c7705)
Link: https://github.com/openwrt/openwrt/pull/17097
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Correctly ignore Initramfs image copy on error. This follows the pattern with
sysupgrade image where an image might fail as it's too big or the
generation command fails and there is nothing to copy to the bin
directory.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
It seems some target started declaring package in DEVICE_PACKAGES just
to call InstallDev and generate binary for the image firmware.
This is very much used by layerscape target where trusted-firmware-a and
dependency are called for final image generation.
This is problematic for APK since it's more sensible to non exisiting
package.
To handle this, introduce a prefix '~' for a package that will signal to
build the package but not install it in the final image.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
To better support imagebuilder declaring --repositories-file on calling
apk macro, detach this and --repository from rootfs.mk macro and move it
to package Makefile and image.mk where they are used to permit a more
generic usage.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Exclude initramfs-images dependency with IB as the target is not defined
in such context.
Fixes: cc6a0abcab ("image: make images and artifacts dependent of initramfs")
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
There is currently a BIG bug in how the images dependency is handled and
recent Per Device Rootfs made this more clear and less statistical.
There is currently no dependency between images/artifacts build with
initramfs build. This cause whatever additional image that depends on an
initramfs image to fail as it might happen that image and initramfs
build are called at the same time and the additional image is called
before initramfs build has finished.
Each image-command assume the source image to be taken from the /bin
directory but that is only copied from the /tmp directory only at the
end of the process.
Artifacts currently depends on image with the use of the
BOARD-NAME-images Makefile target, but this is not the case for
initramfs that also define a -images Makefile target but that is not
accounted in images (that might depend on some initramfs images)
To actually fix this, introduce a new Makefile target, -initramfs-images
and make image and artifacts build to depend on this. Since initramfs
images are optional, this dependency is actived only when initramfs
image are built.
With this change we correctly enforce the build order:
- Initramfs Images (optional)
- Images
- Artifacts
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Initramfs images were using a common rootfs (TARGET_DIR) for all
devices, ignoring TARGET_PER_DEVICE_ROOTFS. If a single device required
a package to build a functional initramfs image, it should be included
by default for all devices or that device should be isolated into a new
subtarget. Now the initramfs will be built using the target-specific
Implementing Per Device Rootfs for Initramfs is not trivial as the
rootfs needs to be embedded in the kernel image. The kernel supports an
option to define the initramfs location and the image generation for the
kernel can't be run in parallel as other checks are done to config and
other arch dependent files.
To handle this, we prepare a config for each rootfs and we generate the
images under lock to prevent problem with parallel execution.
Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
[ rework implementation for locking support ]
Link: https://github.com/openwrt/openwrt/pull/12959
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
A new option called `USE_APK` is added which generated APK packages
(.apk) instead of OPKG packages (.ipk).
Some features like fstools `snapshot` command are not yet ported
Signed-off-by: Paul Spooren <mail@aparcar.org>
include-prefixes were moved to a common directory in linux v4.12, see
d5d332d3f7
This is needed for bcm27xx kernel v6.6 support.
Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
The needed file '.packageinfo' for creating the CycloneDX SBOM in the
imagebuilder is available in the top directory of the imagebuilder and
not in the tmp directory.
For this reason, the creation of the CycloneDX SBOM file is not available
for the imagebuilder.
To fix this, it is now first checked whether the CycloneDX SBOM should be
built at all and then second decided by checking the IB variable where the
'.packageinfo' file is to be found.
With this change, it is now possible to create the CycloneDX SBOM also for
the imagebuilder as well.
Signed-off-by: Florian Eckert <fe@dev.tdt.de>
Historically it's possible to leave the `SUBTARGETS` undefined and
automatically fallback to a "generic" subtarget. This however breaks
various downstream scripts which may have expectations around filenames:
While some targets with an explicit generic subtarget contain `generic`
in the filenames of artifacts, implicit "subtargets" don't.
Right now this breaks the CI[1], possibly also scripts using the ImageBuilders.
This commit removes all code that support implicit handling of
subtargets and instead requires every target to define "SUBTARGETS".
[1]: https://github.com/openwrt/openwrt/actions/runs/8592821105/job/23548273630
Signed-off-by: Paul Spooren <mail@aparcar.org>
Add support for compiling DTS for the selected target. This can be
useful for testing if the DTS correctly compile and doesn't produce any
error.
This adds a new make target. To compile only DTS use:
make target/linux/dtb
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Many NAND devices use a build recipe with "append-ubi | check-size" to
ensure factory images don't exceed the target flash partition size.
However, UBI reserves space for bad block handling and other operational
overhead, and thus 'check-size' can overestimate the space available by
several MB. In practice, this means a failed check is definitely a failure,
while a passing check is only probably a pass.
Improve the situation by teaching 'Build/append-ubi' to check image sizes
while accounting for UBI reserved blocks. Add new device variable NAND_SIZE
and use with existing IMAGE_SIZE to derate the available space. Each UBI
device reserves 20 PEBs per 1024 PEBs of the entire NAND device for bad
blocks, plus an additional 4 PEBs overhead.
Many devices can transparently enable this check by setting NAND_SIZE based
on their flash storage, and may then remove any unneeded 'check-size'.
Link: http://www.linux-mtd.infradead.org/doc/ubi.html#L_overhead
Suggested-by: Shiji Yang <yangshiji66@qq.com>
Suggested-by: Robert Marko <robimarko@gmail.com>
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Add the make function 'exp_units' for helping evaluate k/m/g size units in
expressions, and use this to consistently replace many ad hoc substitutions
like '$(subst k,* 1024,$(subst m, * 1024k,$(IMAGE_SIZE)))' in makefiles.
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Changes introduced in commit d604a07225 ("build: add CycloneDX SBOM
JSON support") broke ImageBuilder:
Cannot open '/openwrt-imagebuilder-ath79-generic.Linux-x86_64/tmp/.packageinfo': No such file or directory
So lets fix it by wrapping the BOM generation behind condition of IB
feature check.
Fixes: #13881
Fixes: d604a07225 ("build: add CycloneDX SBOM JSON support")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
CycloneDX is an open source standard developed by the OWASP foundation.
It supports a wide range of development ecosystems, a comprehensive set
of use cases, and focuses on automation, ease of adoption, and
progressive enhancement of SBOMs (Software Bill Of Materials) throughout
build pipelines.
So lets add support for CycloneDX SBOM for packages and images
manifests.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Handle compiling device tree overlay blobs separate to allow for
overlays being compiled with different parameters, mostly to safe
space.
Allow defining DEVICE_DTC_FLAGS and DEVICE_DTCO_FLAGS as per-device
parameters to be passed to dtc. Previously some boards directly used
DTC_FLAGS in their build recipe which then also affected other boards.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Instead of generating full config nodes incl. kernel, generate minimal
config nodes for device tree overlays to be applied to the main config.
In this way, multiple device tree overlays can be applied more easily.
While at it change filenames to upstream style, ie. use dtso and dtbo
suffix for device tree overlays.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Some U-Boot checks for a specified uImage time and refuses to boot if
mismatched. This patch fixes it by recognizing UIMAGE_TIME parameter.
Signed-off-by: David Yang <mmyangfl@gmail.com>
This reverts commit a33b97dcb1.
A new version of the squashfs4 tool patch reintroduced the -Xe option.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The -Xpreset option changed format and dropped the -Xe and just require
the extreme string to be passed to the -Xpreset option.
Update the LZMA_XZ_OPTIONS to reflect this change.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
FACTORY_SIZE is used as a device recipe variable on both the D-Link
DIR-825-B1 and Trendnet TEW-673GRU, but is not listed as a device
variable, neither globally, nor for ath79. Being assigned the same
value, this probably hasn't caused any issues.
Add FACTORY_SIZE to the global list DEFAULT_DEVICE_VARS, to ensure the
variable is reset for every device, and to allow it to be used outside
of the ath79 target.
Signed-off-by: Sander Vanheule <sander@svanheule.net>
Some vendors use basic gzip metadata (original filename and timestamp)
to verify valid images, along with the size of it's contents.
Also, add a new device profile variable FACTORY_IMG_NAME
which would be ideal to use with this new recipe.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Commit a01d23e75 ("image: always rebuild kernel loaders")
is a step in the right direction, but exposed some issues
and regressions in the makefile.
Some of the files made by device specific COMPILE targets
start with an "append" command (i.e. >> instead of > redirection)
and if the file already exists, the target file is the
input to itself before the first recipe-specified input.
Fixes: a01d23e75 ("image: always rebuild kernel loaders")
Fixes: a7fb589e8 ("image: always rebuild kernel loaders")
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Support defining a per-device loadaddress for the DTB. This is required
for devices which to not align the DTB from the bootloader correctly.
Signed-off-by: David Bauer <mail@david-bauer.net>
Kernel loaders like the lzma-loader currently don't track changes to
their sources. This can lead to an old version of a loader to be used
when a build tree is not clean between builds.
As the loaders are tiny and the build times are insignificant, simply
force rebuilding them on every build to avoid this problem.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Building images usally stores them in KDIR_TMP and then copies them over
to BIN_DIR. This is fine as rebuilding of images overwrites existing
images. When using the EXTRA_IMAGE_NAME variable frequently this fills
up the ImageBuilder KDIR_TMP folder since every built image is stored
forever.
This commit clears the KDIR_TMP folder before building a new image.
Below an example how sysupgrade.openwrt.org filled up after the release
of 22.03.0 where every created image contains a hash of the package
selction in the filename:
aparcar@asu-01:~/asu/worker1/cache/22.03.0$ du -d 1 -h
400M ./kirkwood
260M ./gemini
2.0G ./ipq806x
1.7G ./ipq40xx
8.1G ./ramips
4.0K ./octeon
495M ./sunxi
728M ./lantiq
1.8G ./rockchip
3.7G ./mediatek
4.0K ./realtek
5.4G ./mvebu
8.9G ./ath79
3.0G ./bcm47xx
14G ./bcm27xx
11G ./x86
4.0K ./bcm63xx
312M ./mpc85xx
600M ./apm821xx
5.4G ./bcm53xx
66G .
Signed-off-by: Paul Spooren <mail@aparcar.org>
Introduce `sha256_unsigned` which is a checksum of the image _before_ a
signature is attached. This is helpful to compare image reproducibility.
Since the `.sha256sum` file is located in the $(KDIR) folder, switch
$(BIN_DIR) with $(KDIR) to simplify the code. The value of $(BIN_DIR)
itself is not stored inside the resulting JSON file, so it can be
replaced.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Drop the -processors argument from the mksquashfs4 call, so it will use
all available processors. This dramatically reduces the time to create
squashfs filesystems.
The times below are observed when building an image for my main router,
the WatchGuard Firebox M300 (qoriq target):
Before:
real 4m45,973s
After:
real 0m23,497s
With this commit `mksquashfs` may use more cores than defined via `-j`.
This is the same behaviour as for archive creation of ImageBuilder, SDK
or toolchain. There is no trivial way to limit `mksquashfs` CPU core
usage to the amount of "free" make jobs since two running `mksquashfs`
instances would each run with the total allowed number (-j) of threads.
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
[extended reasoning in commit message]
Signed-off-by: Paul Spooren <mail@aparcar.org>
DEVICE_DTS_DELIMITER needs to be in the DEFAULT_DEVICE_VARS
list to work as expected. This was missing from the original
version and got overlooked.
Fixes: fd67908647 ("scripts: mkits.sh: Allow legacy @ mode for dts creation")
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Multiple profiles create artifacts, these should be stored in the JSON
file as well, allowing downstream tooling to show those files, too.
Artifacts don't have specific filesystems so only the fields `name`,
`type` and `sha256` are available.
Rename env variable names from IMAGE_ to FILE_ prefixes to reflect that
images, kernels and artifacts are added with the same command.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The option '-xattr' for mksquashfs4 should be '-xattrs' which lead to
build failure with SELinux enabled. Add the missing 's'.
Fixes: 4baf47b9a8 ("images: squashfs: xattrs should not depend on buld host")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Enable xattr for the generated squashfs only if needed for SELinux.
This eliminates warnings during boot on target when building
(non-SELinux) OpenWrt on SELinux-enabled hosts like Fedora.
Reported-by: fda77 <fda77@users.noreply.github.com>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Don't attempt to copy initramfs images for devices which do not output
an initramfs image.
This was breaking builds for mpc85xx-p1010 since mid-march.
Signed-off-by: David Bauer <mail@david-bauer.net>
The variable was missing in the definition of DEFAULT_DEVICE_VARS which
caused it to contain wrong values, messing up the resulting JSON files.
This patch adds the variable DEVICE_PACKAGES to DEFAULT_DEVICE_VARS.
Suggested-by: Baptiste Jonglez <git@bitsofnetworks.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if
make TOPDIR="$(pwd)" -C "$pkgdir" compile
was called manually. In most of the cases, I just saw warnings like this:
make: Entering directory '/home/.../package/gluon-status-page'
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
[...]
While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.
After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.
Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
In case CONFIG_TARGET_MULTI_PROFILE is set, IMG_PREFIX cannot be
expanded. Use DEVICE_IMG_PREFIX instead and make sure it's defined.
Fixes: 8f89b1ab0f ("image: add 'append-image' build command")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add possibility to use images and initramfs in artifacts.
Signed-off-by: Oskari Lemmela <oskari@lemmela.net>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add new target feature 'dt-overlay' which makes DTC keep the symbol
names in the generated dtb.
Make sure additional DT overlay sources specified by the new device
variable DEVICE_DTS_OVERLAY get compiled together with the main DTS
(currently overlays got to be in the same folder). Let Build/fit pass
the generated DT overlay blobs to mkits.sh.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
U-boot will reject all nodes with @ since commit:
79af75f777
This will cause the OpenWrt images to fail booting,
to rectify use the config-1 as default.
Signed-off-by: Robert Marko <robert.marko@sartura.hr>
We so far had two variables IMG_PREFIX and IMAGE_PREFIX with
different content. Since these names are obviously quite
confusing, this patch renames the latter to DEVICE_IMG_PREFIX,
as it's a device-dependent variable, while IMG_PREFIX is only
(sub)target-dependent.
For consistency, also rename IMAGE_NAME to DEVICE_IMG_NAME, as
that's a device-dependent variable as well.
Cc: Paul Spooren <mail@aparcar.org>
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Previously, build would fail for targets containing devices with not
initramfs image (such as mpc85xx-p1010). Only generate the JSON image
info for the initramfs image when we have one to avoid breaking the
builds.
Fixes commit d3140d0529 ("build/json: generate json file for initramfs")
Signed-off-by: David Bauer <mail@david-bauer.net>
