While experimenting with the AUTOREMOVE option in search of a way to use
prebuilt host tools in different buildroot, it was discovered that the
md5 generated by find_md5 in depends.mk is not reproducible.
Currently the hash is generated by the path of the file in addition to
the file mod time. Out of confusion, probably, there was an idea that
such command was used on the package build_dir. Reality is that this
command is run on the package files. (Makefile, patches, src)
This is problematic because the package Makefile (for example) change at
each git clone and base the hash on the Makefile mtime doesn't really
reflect if the Makefile actually changes across a buildroot or not.
A better approach is to generate an hash of each file and then generate
an hash on the sort hash list. This way we remove the problem of git
clone setting a wrong mtime while keeping the integrity of checking if a
file changed for the package as any change will result in a different
hash.
Introduce a new kind of find_md5 function, find_md5_reproducible that
apply this new logic and limit it only with AUTOREMOVE option set to
prevent any kind of slowdown due to additional hash generation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
(cherry picked from commit 53a08e3743)
4e19cbc553: [download: handle possibly invalid local tarballs] added a
FORCE rule to downloaded files, so that they will be always checked by
download.pl.
As a side-effect, check-compile will fail, forcing unnecessary package
rebuilds.
The check-compile.txt log shows (for libxml2 for example):
Considering target file '.../dl/libxml2-2.9.12.tar.gz'.
...
prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does
not exist.
Must remake target '.../dl/libxml2-2.9.12.tar.gz'.
...
Giving up on target file '...libxml2-2.9.12/.prepared_...'.
Giving up on target file '...libxml2-2.9.12/.configured_...'.
Giving up on target file '...libxml2-2.9.12/.built'.
Giving up on target file '...stamp/.libxml2_installed'.
Giving up on target file '.compile'.
Then the package is rebuilt even if it is not otherwise needed.
To fix this, instead of always forcing the download target to be remade,
check its hash first: if it matches, then the FORCE is not added.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
This was accidentally dropped in 27a4a71c24
("metadata: handle ABI version rebuild tracking for transient dependencies")
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This reverts commit b12288fa69.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up
Signed-off-by: Felix Fietkau <nbd@nbd.name>
A stray comma was being appended to the last package version dependency,
causing it to be missed for ABI version checks
Signed-off-by: Felix Fietkau <nbd@nbd.name>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
With commit 2ca084cc ("build: improve ccache support") these variables
are being set globally and we don't need them for specific targets.
Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Adding inline shell invocations in per-target variables causes them to be
executed over and over again, which causes a significant slowdown.
Fix this by evaluating it only once per package directory
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
Build/InstallDev no longer places a file list in
$(STAGING_DIR_HOST)/packages; this change removes the creation of
$(STAGING_DIR_HOST)/packages and the attempted removal of a
STAGING_DIR_HOST file list during package clean.
This also changes the host directory passed to Build/UninstallDev from
$(STAGING_DIR_HOST) to $(STAGING_DIR)/host, to match the directory
passed to Build/InstallDev.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85)
Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e)
Revert "ramips: add preliminary support for WIO ONE" (085141dc5b)
Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539)
Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d)
Revert "toolchain: add autoconf-lean" (fdb30eed03)
Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758)
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This changes the default PKG_BUILD_DIR to take BUILD_VARIANT into
account (if set), so that packages do not need to manually override
PKG_BUILD_DIR just to handle variants.
This also updates most base packages with variants to use the updated
default PKG_BUILD_DIR.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This feature is similar to the SRC_TREE_OVERRIDE. However instead of having
to manually create a symlink inside the package folder, the buildsystem
will check if $(TOP_DIR)/src-checkout/$(PKG_NAME)/.git is present and use
this instead.
Signed-off-by: John Crispin <john@phrozen.org>
Extend the packageauxvars database to keep a list of possible package
dependencies for each provider, then utilize this information in buildroot
to resolve the ABI version dependencies of dependent packages up to five
levels deep.
This should properly trigger rebuilds for packages indirectly depending
on other packages whose ABI_VERSION changed.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This fixes two issues with cleaning package files from STAGING_DIR:
* CleanStaging currently can only remove files and not directories. This
changes CleanStaging to use clean-package.sh, which does remove
directories.
* Because of the way directories are ordered in the staging files list,
clean-package.sh currently tries (and fails) to remove parent
directories before removing subdirectories. This changes
clean-package.sh to process the staging files list in reverse, so that
subdirectories are removed first.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
When CONFIG_AUTOREMOVE is enabled, the build system touches an additional
".autoremove" stamp file in the cleaned build directory.
Since the autoremove stamp file is touched after the compile process
completed, it ends up being the most recent file of the package build
directory, causing the timestamp.pl check of depends.mk to erroneously
declare the ".built" and ".prepared" stamp files as stale, triggering
a forced clean-build submake process.
Fix the problem by using the ".built" stamp file as modification time
reference when touching the ".autoremove" stamp file.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Some packages, e.g. busybox, explicitly remove old .configured stamps
before attempting configuration, rather than after the actual configuration
step. This seems like a good idea, as there will be no stamp left if
configuration fails. Change generic rules to work like this, so package-
specific rules can be dropped.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Recent attempts to use it have shown that it does not work properly
except for a few undocumented cases. It's better to remove this now to
avoid having more people fall into the same trap
Signed-off-by: Felix Fietkau <nbd@nbd.name>
quilt.mk needs to be included first, to ensure that STAMP_PREPARED does
not include the hash if quilt is used.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Instead of using a separate .clean stamp file, remove the install info
file on compile, then append the install package list afterwards
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Since the main stamp file depends on the _check stampfile, the _check
stampfile needs to be created on the first run as well.
Fixes spurious rebuilds with CONFIG_AUTOREBUILD=y
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This avoids repeatedly unpacking and rebuilding packages that are
failing the build. Re-running the failing step should be much faster.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Calling the clean target removes all .ipk files and un-stages the
package. Add a new target just for clearing the build dir and call that
one instead of the full clean target
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is used to save space on buildbot instances.
If any part of a package needs to be rebuild, the whole package is
rebuilt from scratch. Stamp files are preserved to allow dependency
checks to work
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Since we've switched to preferring SHA256 over MD5, the old variable
name is misleading. Packages using the old name remain compatible.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
We have packages with their own parts appended to standard STAMP_CONFIGURED
(mostly with an underscore character). This will render the current
STAMP_CONFIGURED_WILD setting invalid and the build system may miss a rebuild
on config change
1. Build with config A
2. Build with config B, yet .configured_A_xx did not get cleaned
3. Return to config A, but rebuild will not happen because stamp file
of config A still exists
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
Make usually passes -j and jobserver arguments as part of MAKEFLAGS. LEDE
removes MAKEFLAGS to have better control of the build, and re-adds the
jobserver arguments with -j to the Make commandline where desired.
Make 4.2+ behave differently with these arguments passed on the commandline
than in MAKEFLAGS: -j will override the jobserver argument and the job
count will be unlimited.
Moving the flags to MAKEFLAGS will need many packages to be changed and
tested; therefore, we opt for a less invasive change for now and just
remove -j for Make 4.2+, as the jobserver argument alone is enough to
enable parallel builds for these Make versions.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
