ExternalVendorCode/heads
1
0
Fork 0
mirror of https://github.com/linuxboot/heads.git synced 2024-12-18 20:47:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

CircleCI + Makefile: remove limitation to loadavg of 16 in Makefile, test CPUS=8 to maximize loadavg on CircleCI with 4 CPUs & 8GB ram

Browse Source 
See first lines of output of any make command. Change aimed to be respectful of CI resource (8GB ram 4CPUs)

With CPUS=8 AVAILABLE_MEM_GB=4, CircleCI outputs:
!!!!!! BUILD SYSTEM INFO !!!!!!
System CPUS: 36
System Available Memory: 4 GB
System Load Average: 12.99
----------------------------------------------------------------------
Used **CPUS**: 8
Used **LOADAVG**: 8
Used **AVAILABLE_MEM_GB**: 4 GB
----------------------------------------------------------------------
**MAKE_JOBS**: -j8 --max-load 8

Variables available for override (use 'make VAR_NAME=value'):
**CPUS** (default: number of processors, e.g., 'make CPUS=4')
**LOADAVG** (default: same as CPUS, e.g., 'make LOADAVG=4')
**AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4')
**MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2')
----------------------------------------------------------------------

Let's try without any limitation...

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
This commit is contained in:
Thierry Laurion 2024-04-23 14:31:54 -04:00
parent cbf984ad7c
commit 05223ca6a3
No known key found for this signature in database
GPG Key ID: 9A53E1BB3FF00461
2 changed files with 43 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.circleci/config.yml
View File

@ -11,7 +11,7 @@ commands:
type: string
steps:
- run:
name: Make Board (FULL ORDERED BUILD LOGS HERE UNTIL JOB FAILED)
name: Make Board (console logs not contiguous because multiple CPUS used to build under CircleCI, see logs in artifacts or Output build failing logs below in case of failed step)
command: |
rm -rf build/<<parameters.arch>>/<<parameters.target>>/* build/<<parameters.arch>>/log/*
nix --print-build-logs --verbose develop --ignore-environment --command make V=1 BOARD=<<parameters.target>> <<parameters.subcommand>> || touch ./tmpDir/failed_build

46
Makefile
View File

@ -25,11 +25,49 @@ INSTALL = $(pwd)/install/$(CONFIG_TARGET_ARCH)
log_dir = $(build)/log
board_build = $(build)/$(BOARD)
# Controls how many parallel jobs are invoked in subshells
CPUS ?= $(shell nproc)
MAKE_JOBS ?= -j$(CPUS) --max-load 16
WGET ?= wget
# Estimated memory required per job in GB (e.g., 1GB for gcc)
MEM_PER_JOB_GB ?= 1
# Controls how many parallel jobs are invoked in subshells
CPUS ?= $(shell getconf _NPROCESSORS_ONLN)
AVAILABLE_MEM_GB ?= $(shell awk '/MemAvailable/ {print int($$2 / 1024 / 1024)}' /proc/meminfo)
# Calculate the maximum number of jobs based on available memory
MAX_JOBS_MEM := $(shell echo $$(( $(AVAILABLE_MEM_GB) / $(MEM_PER_JOB_GB) )))
# Use the minimum of the system's CPUs and the calculated max jobs based on memory
CPUS := $(shell echo $$(($(CPUS) < $(MAX_JOBS_MEM) ? $(CPUS) : $(MAX_JOBS_MEM))))
# Load average can be adjusted to be higher than CPUS to allow for some CPU overcommit
# Multiply by 3 and then divide by 2 to achieve the effect of multiplying by 1.5 using integer arithmetic
LOADAVG ?= $(shell echo $$(( ($(CPUS) * 3) / 2 )))
# Construct MAKE_JOBS with dynamic CPU count and load average
MAKE_JOBS := -j$(CPUS) --load-average=$(LOADAVG) # Add other flags as needed
# Print out the settings and compare system values with actual ones used
$(info ----------------------------------------------------------------------)
$(info !!!!!! BUILD SYSTEM INFO !!!!!!)
$(info System CPUS: $(shell getconf _NPROCESSORS_ONLN))
$(info System Available Memory: $(AVAILABLE_MEM_GB) GB)
$(info System Load Average: $(shell uptime | awk '{print $$10}'))
$(info ----------------------------------------------------------------------)
$(info Used **CPUS**: $(CPUS))
$(info Used **LOADAVG**: $(LOADAVG))
$(info Used **AVAILABLE_MEM_GB**: $(AVAILABLE_MEM_GB) GB)
$(info ----------------------------------------------------------------------)
$(info **MAKE_JOBS**: $(MAKE_JOBS))
$(info )
$(info Variables available for override (use 'make VAR_NAME=value'):)
$(info **CPUS** (default: number of processors, e.g., 'make CPUS=4'))
$(info **LOADAVG** (default: 1.5 times CPUS, e.g., 'make LOADAVG=54'))
$(info **AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4'))
$(info **MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2'))
$(info ----------------------------------------------------------------------)
$(info !!!!!! Build starts !!!!!!)
# Timestamps should be in ISO format
DATE=`date --rfc-3339=seconds`