Add HOST_MAKE_PATH and use it in order to execute Make
in a subdirectory of the build directory
and in a similar way that MAKE_PATH is used for target building.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15991
Signed-off-by: Robert Marko <robimarko@gmail.com>
Add generic recipes for incorporating gnulib into a build
for simplification and readability of the individual build Makefile.
Recipes for configuring and installing are purposefully missing
since "configuring" gnulib is done with standard autoreconf
and gnulib is not a final build target meant for installing.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15853
Signed-off-by: Robert Marko <robimarko@gmail.com>
Add support for overriding the SUBDIRS variable while invoking Make
by defining it after Make in the command line.
This is useful for builds that have previously patched out
the building of certain subdirectories in projects
that use recursive and independent Makefiles,
for example, to block the building of docs or test suites.
The wildcard function is used in the case of there being
subdirectories within any of the subdirectories,
for example, in the building of gengetopt,
in order to avoid the problem where Make will attempt
to execute a Makefile in a subdirectory that does not exist
within the subdirectory it is currently running from
because it really exists at the top-level, or one that exists
within one of the subdirectories when ran from top-level.
There are also cases where the Makefiles in the subdirectories
have the recursive building rules even though there are no more
subdirectories beyond that point, for example, with gnulib.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15853
Signed-off-by: Robert Marko <robimarko@gmail.com>
Several GNU tools such as tar, coreutils, and findutils
now build with support for 64-bit time by default
and otherwise require reconfiguring with a flag
--disable-year2038 in order to build without 64-bit time.
Some standard C libraries, for example,
certain older versions of glibc such as 2.31
have large file support but not long time bits support:
checking for ... option to enable large file support... -D_FILE_OFFSET_BITS=64
checking for ... option for timestamps after 2038... support not detected
This test using C code taken from largefile.m4 in gnulib
uses math and casting to check for overflow
with a macro and array pair that can only be defined
when 64-bit time support is present, and otherwise errors.
It is the exact same code used to test for 64-bit time
during the configure stage of building these tools,
so the results of this test before configure takes place
will always be in concordance with the results of
the test that takes place during the configure script.
Based on the test, the configure flag --disable-year2038
is added to every host tool build depending on the host system.
When the year 2038 problem finally comes around,
the effect of the test can be converted
from the toggling of a configure option into a build prerequisite,
requiring it to pass in order to continue building.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15799
Signed-off-by: Robert Marko <robimarko@gmail.com>
We currently skip defining Host/Prepare/Default if HOST_UNPACK is not
defined.
This is mostly the case for Host packages that just provide files with
the src directory and don't need to be downloaded/extracted.
This was probably done lots of times ago due to quilt causing error as
the patches directory wasn't present.
This has changed now and quilt can correctly detect if no patches needs
to be applied (instead of terminating with error)
Always define Host/Prepare/Default to make tools/refresh correctly works
as HOST_QUILT is hardcoded enabled for this make target and will
complain for tool not prepared for quilt patches.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
A funny bug was discovered where if the buildroot's path
has the name of the build target within it, it will also be substituted
along with the stampfile's name for each program,
causing an attempt to touch a file in a directory that doesn't exist.
...
...
touch: cannot touch '/Volumes/touch/openwrt/staging_dir/host/stamp/.touch_installed': No such file or directory
touch: cannot touch '/Volumes/ln/openwrt/staging_dir/host/stamp/.ln_installed': No such file or directory
touch: cannot touch '/Volumes/chown/openwrt/staging_dir/host/stamp/.chown_installed': No such file or directory
make[2]: *** [Makefile:50: /Volumes/coreutils/openwrt/staging_dir/host/stamp/.coreutils_installed] Error 1
...
...
Split up the path with $(dir) and $(notdir) before substitution to fix
the syntax.
Reported-by: Georgi Valkov <gvalkov@gmail.com>
Tested-by: Georgi Valkov <gvalkov@gmail.com> # MacOS
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Some individual build items install a group of programs
instead of a program matching the name of the build item.
Add support for installing stampfiles for each of the
programs installed by that build item,
which will allow more control and awareness
of what is installed by the rest of the build system,
if, for example, prereq symlink checks are looking
for the same program which is built already.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
A package may run git as part of its build process, and if the package
source code is not from a git checkout, then git may traverse up the
directory tree to find buildroot's repository directory (.git).
For instance, Poetry Core, a Python build backend, will read the
contents of .gitignore for paths to exclude when creating a Python
package. If it finds buildroot's .gitignore file, then Poetry Core will
exclude all of the package's files[1].
This exports GIT_CEILING_DIRECTORIES for both package and host builds so
that git will not traverse beyond $(BUILD_DIR)/$(BUILD_DIR_HOST).
[1]: https://github.com/python-poetry/poetry/issues/5547
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Recent versions of Automake
have changed dependency tracking significantly
(reference commit below)
causing breakage in some package builds
when using newer Automake with packages that need autoreconf
that were bootstrapped with an old version of Automake.
Those changes cause a great inconsistency between packages over time
where some packages may or may not use this feature,
and may or may not update the .ac and .am files
to work with the new methods.
This problem might exist in many packages
where autoreconf is not currently required,
but would cause build failure if autoreconf is used.
Fortunately, this feature is practically useless
for the purposes of Openwrt and the average developer,
so we can disable it.
GNU Automake manual states in part:
"Because dependencies are only computed as a side-effect of compilation...
no dependency information exists the first time a package is built...
dependency tracking is completely useless for one-time builds..."
A nice side-effect is that build times are slightly faster.
Ref: 6a675ef17edf7109da189f5ae70e2dc6b7665896 (automake.git)
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Packages in general use 4 check to trigger a recompile:
- timestamp for the build_dir
- timestamp for the staging stamp dir
- depends hash for the build_dir prepared file
- presence of package archieve in dl
If host tools are prebuilt and shipped in a container or manually
installed from an archieve, it would be ideal to skip including the
package archieve and just provide the build_dir prepared files and the
staging stamp file (and the actualy prebuilt tools).
Add some logic to skip dl download for host tools if AUTOREMOVE is
selected and checks for the presence of staging dir stamp file and build
dir stamp file.
If one of these requirements are not met, the package is redownloaded
and rebuilt.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
While experimenting with the AUTOREMOVE option in search of a way to use
prebuilt host tools in different buildroot, it was discovered that the
md5 generated by find_md5 in depends.mk is not reproducible.
Currently the hash is generated by the path of the file in addition to
the file mod time. Out of confusion, probably, there was an idea that
such command was used on the package build_dir. Reality is that this
command is run on the package files. (Makefile, patches, src)
This is problematic because the package Makefile (for example) change at
each git clone and base the hash on the Makefile mtime doesn't really
reflect if the Makefile actually changes across a buildroot or not.
A better approach is to generate an hash of each file and then generate
an hash on the sort hash list. This way we remove the problem of git
clone setting a wrong mtime while keeping the integrity of checking if a
file changed for the package as any change will result in a different
hash.
Introduce a new kind of find_md5 function, find_md5_reproducible that
apply this new logic and limit it only with AUTOREMOVE option set to
prevent any kind of slowdown due to additional hash generation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Package with whitespace in their build directory are not correctly
removed when CONFIG_AUTOREMOVE is enabled. This is caused by xargs that
use whitespace as delimiters. To handle this use \0 as the delimiter and
set find to use \0 as the delimiter.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Added HOST_CXXFLAGS to specify CXXFLAGS during host-compile
(e.g. to specify c++ standard: HOST_CXXFLAGS += -std=c++11)
Signed-off-by: Sergey V. Lobanov <sergey@lobanov.in>
4e19cbc553: [download: handle possibly invalid local tarballs] added a
FORCE rule to downloaded files, so that they will be always checked by
download.pl.
As a side-effect, check-compile will fail, forcing unnecessary package
rebuilds.
The check-compile.txt log shows (for libxml2 for example):
Considering target file '.../dl/libxml2-2.9.12.tar.gz'.
...
prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does
not exist.
Must remake target '.../dl/libxml2-2.9.12.tar.gz'.
...
Giving up on target file '...libxml2-2.9.12/.prepared_...'.
Giving up on target file '...libxml2-2.9.12/.configured_...'.
Giving up on target file '...libxml2-2.9.12/.built'.
Giving up on target file '...stamp/.libxml2_installed'.
Giving up on target file '.compile'.
Then the package is rebuilt even if it is not otherwise needed.
To fix this, instead of always forcing the download target to be remade,
check its hash first: if it matches, then the FORCE is not added.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
With commit 2ca084cc ("build: improve ccache support") these variables
are being set globally and we don't need them for specific targets.
Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
This causes various issues in other places that assume that host
binaries are staged in STAGING_DIR_HOST.
Since all the right places use HOST_BUILD_PREFIX, override that instead.
This fixes some issues with quilt on toolchain dirs
Signed-off-by: Felix Fietkau <nbd@nbd.name>
quilt.mk needs to be included first, to ensure that STAMP_PREPARED does
not include the hash if quilt is used.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Calling the clean target removes all .ipk files and un-stages the
package. Add a new target just for clearing the build dir and call that
one instead of the full clean target
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Instead of passing HOST_CONFIGURE_VARS as arguments to the configure script,
pass it as environment variables which brings the logic in line with the
behaviour of package-defaults.mk.
The change is needed since passing environment variables as configure
parameters only works with GNU autoconf which evaluates command line arguments
looking like variable assignments. Doing the same with non-autoconf configure
scripts is not guaranteed to work since such scripts might terminate due to
unknown argument errors.
One example case is the cmake configure script which bails out when called
as "./configure LDFLAGS=..." but not when called as "LDFLAGS=... ./configure".
Also change the SHELL override to CONFIG_SHELL in the default
HOST_CONFIGURE_VARS as the former is not properly propagated through the
various GNU configure invocations since it gets lost when configure re-
executes itself.
A prior attempt to change the variable placement had to be reverted due to
the missing SHELL -> CONFIG_SHELL change, leading to misgenerated libtool
executables in various packages.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This reverts commit 8395b63aac616f72fd835c59240fc2a4a6b28106.
Various host builds currently rely on the broken behaviour of
HOST_CONFIGURE_VARS so roll back to the previous state.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Ensure that HOST_CONFIGURE_VARS are set before the actual configure command
instead of passing them as configure command arguments.
This change brings host-build.mk in line with package-defaults.mk and makes
host configure environment variables work as expected.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Instead of hardcoding $(STAGING_DIR)/host, use the new $(STAGING_DIR_HOSTPKG)
variable to refer to the directory.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
The normal Prepare step for a build is unpack, apply patches.
But for certain packages, patches contain whole files, which
would be nice to have separately and copied over as a last step
in the Prepare phase.
We need it for some other packages + patches, but I think
the 'hostapd' package can be used as a test for this.
As a quick note:
the reason the condition is being evaluated as
`[ ! -d ./src/ ] || $(CP) ./src/* $(HOST_BUILD_DIR)`
and not with
`[ -d ./src/ ] && $(CP) ./src/* $(HOST_BUILD_DIR)`
is that the latter would translate in a build failure if the `src`
folder is not present (the exit code would be 1).
The first one, succeeds for both cases (if `src` present or not).
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
