Commit Graph

814 Commits

Author SHA1 Message Date
Álvaro Fernández Rojas
258cb7f169 scripts: cfe-wfi-tag: fix shebang
Fixes: 72985233a6 ("scripts: support CFE WFI images")
Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2020-05-19 08:06:29 +02:00
Álvaro Fernández Rojas
30fcb62840 scripts: cfe-bin-header: fix shebang
Fixes: e575a7f777 ("scripts: support tags for CFE binaries")
Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2020-05-19 08:05:22 +02:00
Álvaro Fernández Rojas
72985233a6 scripts: support CFE WFI images
CFE WFI (Whole Flash Image) consists in 20 byte footer:
- u32: CRC32
- u32: WFI Version
- u32: Chip ID
- u32: Flash Type
- u32: Flags

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2020-05-18 18:24:06 +02:00
Álvaro Fernández Rojas
e575a7f777 scripts: support tags for CFE binaries
CFE loads binaries based on a 12 byte header which corresponds to:
- u32: Load Address
- u32: Entry Address
- u32: Size

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2020-05-18 18:24:06 +02:00
Paul Spooren
c737a9ee6a scripts/download: add sources CDN as first mirror
OpenWrt now has a CDN for sources at sources.cdn.openwrt.org which
mirrors sources.openwrt.org.

Downloading sources outside Europe or US (mainland) could
result in low throughput, extremely slowing down the first compilation of
the build system.

This patch adds sources.cdn.openwrt.org as the first mirror to offer
worldwide fast download speeds by default. If the CDN goes down for
whatever reason, the script jumps to the next available mirror and
downloads requested files as before (in regional varying speed).

Signed-off-by: Paul Spooren <mail@aparcar.org>
Acked-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-04-14 23:16:55 +01:00
Paul Spooren
14cbd8fb2d scripts: JSON merge don't crash if no JSON found
The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new
image generation methods from `image.mk` are used. However some targets
like `armvirt` do not use it yet, so the folder is never created.

The `json_overview_image_info.py` script used to raise an error if the
given `WORK_DIR` isn't a folder, however it should just notify about
missing JSON files.

This patch removes the Python assert and exists with code 0 even if no
JSON files were found, as this is not necessarily an error but simply
not yet implemented. Using `glob` on an not existing `Path` results in
an empty list, therefore the for loop won't run.

Signed-off-by: Paul Spooren <mail@aparcar.org>
CC: Petr Štetiar <ynezz@true.cz>
2020-04-14 23:16:55 +01:00
Jo-Philipp Wich
7b1d809a8d Revert "build: scripts/config - update to kconfig-v5.6"
This reverts commit dcf3e63a35.

The kconfig update requires further testing and refinement until it can
remain in tree. Main problems are:

 - Recursive deps are now fatal instead of a warning
 - Previously legal syntax now leads to hard failures
 - It fails all package builds since multiple days

The updated kconfig implementation needs to cope with the current status
quo in the various package feeds before we can reconsider it for master.

It is not desirable that single broken packages can hard-fail the entire
build pipeline.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2020-04-12 00:10:47 +02:00
Jo-Philipp Wich
6c16d64b2a Revert "build: add option to warn on recursive dependency"
This reverts commit 3204430e38.

Reverting this commit in preparation for reverting
dcf3e63a35 ("build: scripts/config - update to kconfig-v5.6") which
introduces various unaddressed build breakages.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2020-04-12 00:10:30 +02:00
Jo-Philipp Wich
fbc01f2a4c Revert "build: config: allow bool to select a module pkg"
This reverts commit 8514b6b42c.

Reverting this commit in preparation for reverting
dcf3e63a35 ("build: scripts/config - update to kconfig-v5.6") which
introduces various unaddressed build breakages.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2020-04-12 00:10:16 +02:00
Eneas U de Queiroz
8514b6b42c build: config: allow bool to select a module pkg
kconfig-v5.6 disallowed a bool symbol to select another symbol that
'depends on m' (i.e. can be only 'm' on 'n').  It is, in fact, an unmet
dependency to have set to 'y'.  However, openwrt depends on the previous
behavior, to be able to build a package that can be a module or built-in
depending on a bool config.  This restores the previous behavior.

Ref: https://forum.openwrt.org/t/wireless-fails-on-snapshot-r12900-kernel-5-4-on-c2600-ipq806x
Tested-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz> [added forum ref]
2020-04-11 19:06:26 +02:00
Eneas U de Queiroz
3204430e38 build: add option to warn on recursive dependency
This addes the option to treat recursive dependencies as warnings
instead of errors, by running make with WARN_RECURSIVE_DEP=1.

Note that the script/config targets will not get rebuilt when you add or
remove WARN_RECURSIVE_DEP while running make.  One must run
'make config-clean' before building config with a different setting.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-04-09 12:51:10 +02:00
Eneas U de Queiroz
dcf3e63a35 build: scripts/config - update to kconfig-v5.6
Major changes include:
 - Much more readable reverse dependencies separated in groups
 - Improved recursive dependency report
 - More readable .config files: add comments to signal end of menus
 - More warnings for incorrect Config.in entries, such as a 'choice'
   default not contained in the 'choice'
 - Hability to properly display pseudographics with non-latin locales
 - Recursive dependencies are now treated as errors - this should make
   it harder for them to creep in.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-04-09 12:51:10 +02:00
Eneas U de Queiroz
8636a17214 build: simplify building *config targets
Instead of passing pkg-config location through a variable when building
qconf (make xconfig), prepend its parent directory to the PATH, as it is
being done for other conf targets.

Use a Makefile pattern rule to group all 'scripts/config/%onf'
(currently conf, mconf, qconf) targets in a single rule.  Add -O2 to
CFLAGS when building them as well.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2020-04-09 12:51:10 +02:00
Paul Spooren
07449f692c build: refactor JSON info files to profiles.json
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").

They are useful for firmware wizards and script checking for
reproducibility.

Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.

This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.

Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.

Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.

As before, this creation is enabled by default only if `BUILDBOT` is set.

Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-04-03 12:17:45 +02:00
李国
a6b7c3e672 x86: generate EFI platform bootable images
Add EFI platform bootable images for x86 platforms. These images can
also boot from legacy BIOS platform.

EFI System Partition need to be fat12/fat16/fat32 (not need to load
filesystem drivers), so the first partition of EFI images are not ext4
filesystem any more.

GPT partition table has an alternate partition table, we did not
generate it. This may cause problems when use these images as qemu disk
(kernel can not find rootfs), we pad enough sectors will be ok.

Signed-off-by: 李国 <uxgood.org@gmail.com>
[part_magic_* refactoring, removed genisoimage checks]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-03-31 16:20:47 +02:00
Paul Spooren
2cb6d471b7 scripts: fixup qemustart for new x86 image names
qemustart allows easy testing of created images via `qemu`. The script
automatically selects created images and can setup e.g. networks.

As the x86 target now uses the generic image.mk the profile appears also
in the image name, this is *generic*.

Add the profile name to the qemustart script so it still finds the file.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-03-21 10:36:00 +00:00
Jeffery To
f96cfe019a build: Fix directory symlinks not removed when cleaning STAGING_DIR
Currently, a symbolic link whose target is a directory will not be
removed when cleaning packages from STAGING_DIR.

In the first cleaning pass in scripts/clean-package.sh, the -f test for
a directory symlink returns false (because the link target is a
directory) and so the symlink is not removed.

In the second pass, the -d test returns true for a directory symlink,
but the symlink is not removed by rmdir because rmdir only removes
(real) directories.

This updates clean-package.sh to remove all non-directories (including
symbolic links) in the first pass.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-03-01 21:35:59 +01:00
Rosen Penev
e37e2f0cf1 scripts/arm-magic.sh: switch to /bin/sh
This does not use any special bash stuff.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-02 23:59:20 +01:00
Rosen Penev
995378a2a5 scripts/env: replace -a and -o with &&/||
The former are not well defined.

Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-02 23:56:19 +01:00
Rosen Penev
2fe5319d58 scripts/env: use explicit find location
Some find binaries do not imply the current directory.

Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-02 23:51:45 +01:00
Rosen Penev
33f87d2ef6 scripts/gen-dependencies.sh: use /bin/sh
This uses no special bash stuff.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-01 17:01:02 +01:00
Rosen Penev
dec46a9ea0 scripts/gen-dependencies.sh: replace backticks with $()
Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-01 17:00:38 +01:00
Rosen Penev
9a8ba44f9c scripts/mkits.sh: replace legacy backticks with $()
This replaces deprecated backticks by more versatile $(...) syntax.

Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
[adjust commit title and message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-12-31 11:44:04 +01:00
Rosen Penev
543d945f07 scripts/symlink-tree.sh: use /bin/sh
This uses nothing bash specific.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-12-31 11:43:15 +01:00
Adrian Schmutzler
ec0fb23a41 scripts/strip-kmod.sh: harmonize leading whitespaces
Convert leading spaces to tabs for consistency in the file.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-12-31 11:41:07 +01:00
Rosen Penev
f4b812dac1 scripts/strip-kmod.sh: use sh instead of bash
There's nothing bash specific here.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-12-31 11:40:24 +01:00
Rosen Penev
eae889b8f2 scripts/strip-kmod.sh: replace legacy backticks with $()
This replaces deprecated backticks by more versatile $(...) syntax.

Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
[slightly adjusted commit title, added commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-12-31 11:38:06 +01:00
Rosen Penev
3f65fe5d62 scripts/gen_image_generic.sh: replace deprecated backticks with $()
This replaces deprecated backticks by more versatile $(...) syntax.

Found with shellcheck.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
[adjust commit title and message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2019-12-31 01:06:20 +01:00
Rosen Penev
2e14dd23c9 scripts/gen_image_generic.sh: remove $ in arithmetic
Fixes shellcheck warning:

SC2004: $/${} is unnecessary on arithmetic variables.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-12-31 01:05:27 +01:00
Jiri Kastner
dc34c695c4 scripts/dowload.pl: add archive.apache.org to apache mirror list
apache mirrors holds only latest releases, to download
older releases, one must use archive.apache.org to get
them.

Signed-off-by: Jiri Kastner <cz172638@gmail.com>
2019-12-23 00:21:46 +01:00
Yousong Zhou
d26738bc76 scripts/dl_github_archive.py: fix python3 str, bytes confusion
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-11-04 11:11:19 +00:00
Yousong Zhou
66ebca7103 build: label kernel and rootfs ext4 volumes
UUID of ext4 volumes generated by make_ext4fs are determined by volume
label and it will all be 57f8f4bc-abf4-655f-bf67-946fc0f9f25b when label
is empty

Labeling them does not make them unique but tools like block command
from fstools have a better chance differentiating them

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-11-03 04:25:40 +00:00
Alexander Couzens
5625a02e77
ubinize-image: set image sequence from SOURCE_DATE_EPOCH
Set the image sequence number to SOURCE_DATE_EPOCH.
If not set, ubinize will use rand() as image sequence.

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
2019-11-03 04:20:21 +01:00
Paul Spooren
2ae5100d70 build: add script to sign packages
This script allows image signing indipendend of the actual build
process, to run on a master server after receiving freshly backed
images. Idea is to avoid storying private keys on third party builders
while still beeing to be able to sign packages.

Run ./scripts/sign_images.sh with the following env vars:

* TOP_DIR where to search for sysupgrade.bin images
* BUILD_KEY place of key-build{,.pub,.ucert}
* REMOVE_OTHER_SIGNATURES removes signatures added by e.g. buildbots

Only sysupgrade.bin files are touched as factory.bin signatures wouldn't
be evaluated on stock from.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2019-10-21 14:07:08 +02:00
Paul Spooren
881ed09ee6 build: create JSON files containing image info
The JSON info files contain details about the created firmware images
per device and are stored next to the created images.

The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.

An example of openwrt-ramips-rt305x-aztech_hw550-3g.json

    {
      "id": "aztech_hw550-3g",
      "image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g",
      "images": [
        {
          "name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin",
          "sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20",
          "type": "sysupgrade"
        }
      ],
      "metadata_version": 1,
      "supported_devices": [
        "aztech,hw550-3g",
        "hw550-3g"
      ],
      "target": "ramips/rt305x",
      "titles": [
        {
          "model": "HW550-3G",
          "vendor": "Aztech"
        },
        {
          "model": "ALL0239-3G",
          "vendor": "Allnet"
        }
      ],
      "version_commit": "r10920+123-0cc87b3bac",
      "version_number": "SNAPSHOT"
    }

Signed-off-by: Paul Spooren <mail@aparcar.org>
2019-09-29 13:51:28 +02:00
leo chung
b2c55d50f8 build: fix xconfig target
`make xconfig` fails with following linking error of qconf binary:

 g++ -lQt5Widgets -lQt5Gui -lQt5Core -o qconf qconf.o zconf.tab.o
 /usr/bin/ld: qconf.o: in function ConfigList::metaObject() const': qconf.cc:(.text+0x3eb): undefined reference to QObjectData::dynamicMetaObject() const'
 /usr/bin/ld: qconf.o: in function `ConfigList::qt_metacast(char const*)': link error.

which is caused by the wrong order of the linked objects/libraries so
this patch reorders the linker's arguments which makes the qconf compile
again.

Signed-off-by: leo chung <gewalalb@gmail.com>
[commit subject and message tweaks, whitespace fix]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-09-25 12:50:24 +02:00
Petr Štetiar
965f341aa9 build: fix host menu config targets using ncurses
On a recent Gentoo Linux installation, invoking `make menuconfig`, `make
kernel_menuconfig` or `make kernel_nconfig` in the build system fails,
whereas for example `make menuconfig` in the kernel tree alone works as
expected.

This is happening because STAGING_PREFIX is not defined when kernel's
{menu,n}config target calls pkg-config from the toolchain/host and thus
pkg-config returns an empty value, and the fallback values in the kernel
config script are applied but those are off and the linking fails.

Solution is to use system's pkg-config for all ncurses based menu config
targets in order to provide proper compiler/linker flags.

Ref: FS#2423
Cc: Thomas Albers <thomas.gameiro@gmail.com>
Tested-by: Stijn Tintel <stijn@linux-ipv6.be>
Tested-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-09-25 12:50:24 +02:00
Johann Neuhauser
205e0939f0 build: make device tree arg really optional in mkits.sh
If no device tree is given there is no node generated, but
the configuration does still include the name of the missing node.
This will result in a successful build fit image, but bootm does
throw a error message if we want to boot the bad configuration.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
2019-09-19 23:43:27 +02:00
Yousong Zhou
18dc533172 download.dl: fix @KERNEL mirror urls
- Prepend "/pub" for mirror.rackspace.com
 - Use https for download.xs4all.nl and mirrors.mit.edu

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-09-16 08:11:59 +00:00
Felix Fietkau
d5cd80a1e8 scripts/feeds: fix accepting "-" in feed type string
Fixes a syntax error in processing the type src-git-full

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2019-09-15 19:54:03 +02:00
Jonas Gorski
ee76bd11bb images: fix boot failures on NAND with small sub pages
SquashFS has a minimum block size of at least 1k, so we need to make
sure the last data block is also at least that big.

This is not an issue on NOR or SD CARD devices, since their rootfs
partitions go all the way to the end of the usable space.

But on NAND with ubiblock, the rootfs partition will be the exact space,
rounded up to LEB size. Unfortunately, some NAND chips with small sub
pages have a LEB size of x.5 kiB. This can cause the the last data block
to be less than 1k, which will cause the last block to be inaccessible,
causing boot failures as seen on MR24:

[    1.532960] block ubiblock0_3: created from ubi0:3(rootfs)
[    1.538457] ubiblock: device ubiblock0_3 (rootfs) set to be root filesystem
[    1.552847] SQUASHFS error: squashfs_read_data failed to read block 0x621472
[    1.559896] squashfs: SQUASHFS error: unable to read id index table
[    1.566474] VFS: Cannot open root device "(null)" or unknown-block(254,0): error -5

Since on most NOR devices, the start of the squashfs partition is not
aligned. Since the start of the rootfs_data partition there is dependend
on the SquashFS size, we cannot just always pad it, as the padding could
creep into the rootfs_data partition, breaking jffs2.

So fix this by ensuring a squashfs rootfs is always a multiple of 1k
only for UBI and NAND sysupgrade images.

Fixes #2460 without affecting NOR devices.

Tested-by: Russell Senior <russell@personaltelco.net>
Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
2019-09-14 11:43:19 +02:00
Bjørn Mork
a21b70be31 scripts/feeds: fix 'src-include' directive
Commit 775b70f8d5 renamed parse_file() parameters without
updating the recursive call. This broke parsing of any feeds.conf
using 'src-include'.

 $ scripts/feeds update -a
 Can't use string ("defaults") as a HASH ref while "strict refs" in use at scripts/feeds line 63, <$fh> line 1.

Fixes: 775b70f8d5 ("scripts/feeds: allow adding parameters to feeds")
Signed-off-by: Bjørn Mork <bjorn@mork.no>
2019-09-02 16:48:40 +02:00
Thomas Langer
035906fd05 Fix handling of BUILD_SUFFIX in remote-gdb script
When CONFIG_BUILD_SUFFIX is enabled, the target-* folders in build_dir
and staging_dir have this suffix in the name, but not the
toolchain directories. When detecting the names for "arch" and "libc",
also accept the suffix and do not use it for the toolchain path.

Signed-off-by: Thomas Langer <thomas.langer@intel.com>
2019-09-01 18:38:05 +02:00
Jo-Philipp Wich
775b70f8d5 scripts/feeds: allow adding parameters to feeds
this allows adding "--" prefixed parameters inside feeds.conf between the
target and name. The first parameter is --force which has the same effect
as using -f when installing any of the packages. This allows creating
feeds that will override base packages by default.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2019-08-28 16:10:41 +02:00
Yousong Zhou
164037983d scripts/dl_github_archive.py: fix python3 transition
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-08-20 13:34:58 +00:00
Yousong Zhou
0f3667864d download.pl: use https://source.openwrt.org
https://sources.lede-openwrt.org now redirects to there

https://downloads.openwrt.org/sources returns 404, so remove it here

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2019-08-18 15:19:22 +00:00
Jonas Gorski
7546be6007 build: allow overriding default selection state for devices
Allow overriding the default selection state for Devices, similar to
setting a default for packages.

E.g. by setting DEFAULT to n, they won't be selected by default anymore
when enabling all device in the multi device profile.

This allows preventing images being built by the default config for
known broken devices, devices without enough RAM/flash, or devices not
working with a certain kernel versions.

This does not prevent the devices from being manually selected or images
being built by the ImageBuilder. These devices often still have worth
with a reduced package-set, or as a device for regression testing, when
no better device is available.

Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
2019-08-13 22:13:18 +02:00
Petr Štetiar
c6d41c320c scripts/ubinize-image.sh: fix buildbot breakage
New Docker based buildslaves install just bare minimum of packages, thus
not having bsdmainutils package installed which provides `hexdump`
utility, leading to the following build breakage on buildbots:

 ubinize-image.sh: 12: /builder/scripts/ubinize-image.sh: hexdump: not found

So this patch simply replaces `hexdump` with `od` utility provided by
coreutils package, which should be likely available.

Co-authored-by: Yousong Zhou <yszhou4tech@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-07-30 10:16:16 +02:00
Petr Štetiar
57bb89b57f scripts/jungo-image: convert to Python 3 with 2-to-3
Let's convert the script to Python 3.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-07-26 08:09:16 +02:00
Petr Štetiar
e785da815b scripts/dl_github_archive.py: convert to Python 3 with 2-to-3
Let's convert the script to Python 3.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2019-07-26 08:09:16 +02:00