It seems like after a build the /dl dir seems to now contain a .hash
file for each source file due to inproper cleanup so fix it by removing
those intermediate files before leaving the download action.
Fixes: 4e19cbc553 ("download: handle possibly invalid local tarballs")
Reported-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Silence the warning in git 2.27 about undefined fast-forward style
in git pull. Define "ff-only" as the style.
Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
As package size changes are a continuous topic on the mailing list this
scripts helps developers to compare their local package modifications
against latest upstream.
The script downloads the latest package indexes based on env variables
or the `.config` file. The script compares the actual installed size
(data.tar.gz) or the IPK package size.
An example output is found below:
```
user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh
Compare packages of ath79/tiny/mips_24kc:
dropbear busybox iw ubus
Checking configuration difference
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 554 100 554 0 0 336 0 0:00:01 0:00:01 --:--:-- 336
--- start config diff ---
--- /tmp/config.DDjwVh-LOCAL 2020-11-23 09:08:28.913203068 -1000
+++ /tmp/config.DDjwVh-UPSTREAM 2020-11-23 09:08:36.369240887 -1000
@@ -1,5 +1,9 @@
+CONFIG_ALL_KMODS=y
+CONFIG_ALL_NONSHARED=y
CONFIG_AUTOREBUILD=y
+CONFIG_AUTOREMOVE=y
--- 8< ---
CONFIG_BINARY_FOLDER=""
+CONFIG_BUILDBOT=y
+CONFIG_TARGET_ALL_PROFILES=y
CONFIG_TARGET_ROOTFS_DIR=""
CONFIG_USE_SSTRIP=y
CONFIG_USE_UCLIBCXX=y
--- end config diff ---
Checking installed size
Fetching latest package indexes...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252
Comparing package sizes...
Change Local Remote Package
+271 51386 51115 base-files
+123 705241 705118 bnx2-firmware
+86 17209 17123 fstools
+22 47989 47967 procd
+21 208311 208290 busybox
+19 67181 67162 netifd
```
I plan to integrate this script into the CI so we have a summary how
sizes change over different architectures.
Signed-off-by: Paul Spooren <mail@aparcar.org>
With this commit, the download script will try downloading source files
using the filename instead of the url-filename in case the previous
download attempt using the url-filename failed.
This is required, as the OpenWrt sources mirrors serve files using the
filename files might be renamed to after downloading. If the original
mirror for a file where url-filename and filename do not match goes
down, the download failed prior to this patch.
Further improvement can be done by performing this only for the
OpenWrt sources mirrors.
Signed-off-by: David Bauer <mail@david-bauer.net>
Kernel has separated the structs that are reported to be const in
checkpatch.pl into a file of its own, const_structs.checkpatch.
This file has been missing after the recent update of checkpatch.pl,
leading to the following message:
No structs that should be const will be found - file
'/data/openwrt/scripts/const_structs.checkpatch': No such file
or directory
This commit adds the relevant file from v5.10-rc4.
Fixes: 086ee09bbc ("scripts: Update checkpatch.pl to 2020-06-11")
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
The kernel expects changes to MAINTAINERS for all removed or added
files, printing warnings like:
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#828:
deleted file mode 100644
Since this does not apply to "our" files in OpenWrt repo, this
warning should be disabled.
This can be achieved easiest by setting $reported_maintainer_file
to 1. While this is a hack that tricks the script into believing
the proper MAINTAINERS changes have been made, it's the easiest
solution as it does not require to touch any other code.
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Running the updated checkpatch version with spelling.txt show that
spelling mistakes happen to everyone.
> /target/linux$ fd .*.patch | xargs ../../scripts/checkpatch.pl | rg spell
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'begining' may be misspelled - perhaps 'beginning'?
WARNING: 'superflous' may be misspelled - perhaps 'superfluous'?
WARNING: 'multipe' may be misspelled - perhaps 'multiple'?
WARNING: 'recieves' may be misspelled - perhaps 'receives'?
WARNING: 'retreive' may be misspelled - perhaps 'retrieve'?
WARNING: 'tranfer' may be misspelled - perhaps 'transfer'?
WARNING: 'additonal' may be misspelled - perhaps 'additional'?
WARNING: 'accomodate' may be misspelled - perhaps 'accommodate'?
[...]
Signed-off-by: Paul Spooren <mail@aparcar.org>
Remove checks for device tree documentation as the OpenWrt tree comes
withouth the ./Documentation folder.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Checkpatch grown in functionallity and we should make use of that. If
OpenWrt patches should be upstream material they should also be checked
based on upstream checkpatch.pl instead of 2013.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The source date epoch is the only reproducible date close to the actual
build date. It can be used for tooling like the firmware wizard to show
the image age.
Signed-off-by: Paul Spooren <mail@aparcar.org>
mkhash currently returns the hash of an empty input when trying to hash
a folder. This can be missleading in caseswhere e.g. an env variable is
undefined which should contain a filename. `mkhash ./path/to/$FILE`
would exit with code 0 and return a legit looking checksum.
A better behaviour would be to fail with exit code 1, which imitates the
behaviour of `md5sum` and `sha256sum`.
To avoid hashing of folders the `stat()` is checked.
Hashing empty inputs result in the following checksums:
md5: d41d8cd98f00b204e9800998ecf8427e
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Signed-off-by: Paul Spooren <mail@aparcar.org>
If hashing a file fails mkhash shouldn't just silently fail. Now check
after each call of `hash_file()` the return and exit early in case of
errors. The return value which was previously ignored and would always
return 0.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The -n option prints the filename of hashed files next to the calculated
checksum. Reflect that in the usage message.
user@dawn:~/src/openwrt/openwrt$ ./a.out md5 -n .config
eb06db36e7b6751cb18801945e46bf5d .config
Signed-off-by: Paul Spooren <mail@aparcar.org>
Recent versions (> 5.33) of `file` report liblto_plugin.so as
executable:
$ file liblto_plugin.so
liblto_plugin.so.0.0.0: ELF 64-bit LSB pie executable ...
Which then leads to improper packaging of the plugin, resulting in the
broken compiler:
configure: checking whether the C compiler works
mips-openwrt-linux-musl/bin/ld: liblto_plugin.so: error loading plugin: liblto_plugin.so: invalid ELF header
As the LTO compiler plugin library is incorrectly packaged as SDK
executable:
$ head -1 ~/staging_dir/toolchain...libexec/gcc/.../liblto_plugin.so
#!/usr/bin/env bash
Fix this by filtering out shared libraries from the patching.
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1296868
Acked-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Shallow copies are quite common on CI platforms nowadays, making REBOOT
tag unavailable, thus producing following confusing errors in the build
logs:
fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..HEAD
fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..0493d57e04774d47921a7d2014b567455d5dc16b
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Explicitly set the QEMU cpu type and support the MIPS R2 ISA, for both
64-bit and 32-bit targets. The later previously supported MIPS R2 by
implicit default.
This is needed after commit 93608697f3 ("malta: update MIPS64 ISA to R2"),
otherwise booting malta images with scripts/qemustart will hang.
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
As the PWM has its own sub-system in the Linux kernel,
I think it should be handled in the same way as GPIO, RTC, PCI...
This patch introduces a specific feature flag "pwm" and the
"leds-pwm" kernel module as the first customer.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Some bootloaders are really keen on just one special
fdt in a multi-image fit image. This is a problem, because
currently this is fixed to "fdt@1".
This patch introduces a new device variable:
DEVICE_FDT_NUM that allows to specify the right
fdt number.
If the value is absent "1" will be chosen.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
SPDX moved from GPL-2.0 to GPL-2.0-only and from GPL-2.0+ to
GPL-2.0-or-later. Reflect that in the SPDX license headers.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[adjust commit title, update remaining files]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Use the prepared .packageusergroup file to lookup user and group names
when processing the passed file mode.
Also replace the various subshell/cut invocations with a sequence of
standard variable interpolations which fixes paths with embedded colons
as a side-effect.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
With the introduction of `./tmp/userids` the `ipkg-build` script can now
resolve values of "PKG_FILE_MODES", allowing users to set names rather
than numeric values.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Since the introduction of fakeroot support, wrapped SDK executables might
be invoked from a shell that has libfakeroot.so preloaded.
Since we're using preloading as well in order to mangle argv[0] when
invoking the shipped ELF interpreter directly, we must take care of
preloading the already preloaded libraries as well, to avoid invoked
programs losing their fakeroot capabilities.
Extend the bundle-libraries.sh script to take any existing $LD_PRELOAD
into account when invoking the target ELF executable with a preloaded
runas.so library.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
With the new `SOURCE` argument of `get_source_date_epoch` it is possible
to set package timestamps based on actual package changes rather thane
$TOPDIR changes.
This commit adds a new variable PKG_SOURCE_DATE_EPOCH which is used by
the `ipkg` build script. As a fallback the existing SOURCE_DATE_EPOCH is
used or as last resort the current time.
The redundant checks for `.git/` and `.svn/` are removed.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The SOURCE_DATE_EPOCH variable is used to make builds reproducible even
if rebuild at different times. Instead of using the current timestamp,
the time of the last source change is used.
Created packages are `touch`ed with a specific timestamp so resulting
packages have the same checksums.
The `get_source_date_epoch.sh` script tries multiple ways (file, git,
hg) to determine the correct timestamp.
Until now the script would only consider the $TOPDIR instead of package
specific changes. Resulting in packages with same versions but different
timestamps, as $TOPDIR (openwrt.git) received changes not affecting
package versions. This results in warning/erros in `opkg` as the package
versions stay the same but checksums changed.
This commit adds an optional argument to get the `SOURCE_DATE_EPOCH` of
a specific path (e.g. package SOURCE) rather than the $TOPDIR. As a
consequence this allows granular but still reproducible timestamps.
As packages might be distributed over multiple repositories the check
for `.git/` becomes unfeasible. Instead tell `git` and `hg` to change
their working directories and automatically traverse the repo folder.
Signed-off-by: Paul Spooren <mail@aparcar.org>
If a `cd` to `TOPDIR` fails the script should quit.
Also unify `try_mtime` function by storing it in a variable.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The `ipkg-build` script converts a folder into a `opkg` installable
package. Until now it would use root:root for all packages and try to
preserve file modes.
This has the two drawbacks of packages want to add non-root files or add
SUID files, like the `sudo` package does.
To give more flexibility regarding file modes and avoid init script
hacks, a new variable called `PKG_FILE_MODES`. The variable contains a
list of files modes in the format `path:owner:group:mode`.
An example for the `sudo` package below:
```
PKG_FILE_MODES:=\
/usr/bin/sudo:root:root:4755 \
/etc/sudoers:root:root:0440
```
The `ipkg-build` now runs within a fakeroot environment to set any mode
and directly store it in the resulting `ipk` package archive.
Both options `-o` and `-g` are no longer required due to the introduction
of the more flexible `-m` options, which takes the `PKG_FILE_MODES` as
input.
Lastly the option `-c` is removed as it's unused within the script.
Signed-off-by: Paul Spooren <mail@aparcar.org>
This reverts commit c737a9ee6a.
The source CDN has been discontinued in its current form and will take a
while to be reestablished. Even then it makes little sense to put a CDN
before other CDNs such as kernel.org, apache.org, sourceforge etc.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Fix shellcheck SC2230
> which is non-standard. Use builtin 'command -v' instead.
Using `command -v` is POSIX compliant while `which` is not. Also to
mention, `command -v` is a shell builtin whereas `which` is a separate
busybox applet.
Once applied to everything concerning OpenWrt we can disable the busybox
feature `which` and save 3.8kB.
Acked-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Paul Spooren <mail@aparcar.org>
[also replace cases in zram-swap]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
That file was added accidentally in v3 of the patch and I haven't
properly reviewed that patch before pushing it.
Fixes: 656b562aff ("scripts: Add Buildbot dump-target-info.pl script")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
The script comes from buildbot.git[0] and is used to print available
targets and architectures, which are then build.
As the buildbot clones openwrt.git anyway, the script might as well live
here to be used for other cases as well, e.g. determining what
architectures are available when building Docker containers or show
developers an overview which architectures are used by which target.
It's called with either the parameter `architectures` or `targets`,
showing architectures followed by supported targets or targets, followed
by the supported architectures:
$ ./scripts/dump-target-info.pl architectures
aarch64_cortex-a53 bcm27xx/bcm2710 mediatek/mt7622 mvebu/cortexa53 sunxi/cortexa53
aarch64_cortex-a72 bcm27xx/bcm2711 mvebu/cortexa72
...
$ ./scripts/dump-target-info.pl targets
apm821xx/nand powerpc_464fp
apm821xx/sata powerpc_464fp
...
In the future the the script could be removed from the buildbot
repository and maintained only here.
Rename `dumpinfo.pl` to `dump-target-info.pl` to improve verbosity of
filename.
[0]: https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/dumpinfo.pl;h=aa97f8d60379076a41b968402e9337cea824ece5;hb=HEAD
Signed-off-by: Paul Spooren <mail@aparcar.org>
checkpatch.pl uses a list of files to detect the root OpenWrt
directory. This includes README, which has been renamed to
README.md in the previous commit.
Update the file name in checkpatch.pl to prevent errors like the
following when running the script:
Must be run from the top-level dir. of a OpenWrt tree
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
By specifying "BROKEN := 1" or "BROKEN := y" for a device, it will be
hidden (and deselected) by default. By that, it provides a stronger
option to "disable" a device beyond just using DEFAULT := n.
To make these devices visible, just enable the BROKEN option in
developer settings as already implemented for targets and packages.
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
The f-string feature was introduced in Python 3.6. As Buildbots may run
on Debian 9, which comes per default with Python 3.5, this would cause
an issue. Instead of f-strings use the *legacy* `.format()` function.
Signed-off-by: Paul Spooren <mail@aparcar.org>
We don't need to see how git will be executed and it produces non silent
output on 'scripts/env diff' commands when there are no differences
unlike before.
Re-introduce original silent behaviour.
Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
Due to a line break, printf was accidentally called with three
arguments instead of two, causing a different output than before.
Fix it by splitting the printf command into two lines.
Fixes: 907053193a ("scripts/mkits.sh: replace echo -e with printf")
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Fixes following shellcheck warning:
In scripts/mkits.sh line 19:
"-k kernel [-D name -d dtb] -o its_file" "$(basename $0)"
^-- SC2086: Double quote to prevent globbing and word splitting.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Fixes following shellcheck warning:
In scripts/env line 25:
exit ${1:-1}
^-----^ SC2086: Double quote to prevent globbing and word splitting.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
