KNELPerf v2 wave 1: Ultix scheduler breakout + loop skeleton + reports (#826)
ci / audit (push) Failing after 26s
ci / audit (push) Failing after 26s
- scheduler/: day/night engine (conf-driven redesign), psi/perfsnap collectors, proxmox-ctl, systemd timer templates — ported from ~/projects/ultix per the breakout map in scheduler/README.md - loop/: perf-loop driver (baseline/audit/tweak/rebaseline), cpu-bench (new CPU leg), baseline-diff (tolerance compare) — smoke-tested locally - docs/ARCHITECTURE.md: full program design (loop, resource groups, VM/spindle balancing, k8s-vs-Slurm, beszel+RAPL/iDRAC telemetry spine, workload classes, solar/HA + two-site power economics) - docs/report-amt-power-telemetry.md + docs/report-moonlight-desktop.md Redmine: https://projects.knownelement.com/issues/826#note-2
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# KNELPerf scheduler — breakout from Ultix (#826)
|
||||
|
||||
The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`)
|
||||
was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains
|
||||
the live workstation deployment; this module is the going-forward home.
|
||||
|
||||
## What was ported and what changed
|
||||
|
||||
| Ultix source | KNELPerf destination | Redesign delta |
|
||||
|---|---|---|
|
||||
| `staged/ukrrs-daynight.sh` | `daynight/knelperf-daynight.sh` | Slice targets, devices, and every knob are conf-driven lists (was: hardcoded `reachableceo-*` slices, `/dev/sdb`, user 1001). Slice specs are `slice:weight:cpus:memhigh` so any host participates. |
|
||||
| `staged/etc-ukrrs-daynight.conf` | `daynight/knelperf-daynight.conf.example` | Self-documenting example; post-upgrade values live per-host, not in comments of the engine. |
|
||||
| `staged/systemd/reachableceo-{day,night}profile.*` | `daynight/systemd/knelperf-*` | Generic units; the conf — never a sed-edit of a live timer — is the only value source. |
|
||||
| `staged/reachableceo-perfsnap.sh` | `collectors/knelperf-perfsnap.sh` | Watched cgroups/interfaces are env-driven lists; no baked-in uids/NICs. Same pipe-delimited log format. |
|
||||
| `staged/ukrrs-psi-textfile.sh` | `collectors/knelperf-psi-textfile.sh` | Cgroup glob + metric prefix env-driven; emits for any slice namespace. |
|
||||
| `staged/proxmox-ctl.sh` | `proxmox/proxmox-ctl.sh` | Ported as-is (was already generic + guarded: PROTECTED vmids, graceful shutdown only, pvesh local-or-token). |
|
||||
|
||||
Not ported (stay workstation-local in ultix): `mkacct.sh` account tiers,
|
||||
`21-priority-ladder.sh` GUI-first cgroup ladder (host-specific by design),
|
||||
`gateway-ensure.sh`, RTMP stack.
|
||||
|
||||
## Lessons learned carried forward (from ultix REPORT.md / TRACKING.md)
|
||||
|
||||
1. `Slice=` is create-time only — `set-property` cannot set it; caps go in unit
|
||||
drop-ins AND live via `set-property --runtime`.
|
||||
2. `CPUWeight` on user-manager units is inert until `cpu` is in the parent's
|
||||
`cgroup.subtree_control` — enable it explicitly.
|
||||
3. IOWeight is inert under the `none` I/O scheduler (virtio) — don't bother.
|
||||
4. Guests have no cpufreq driver — governor tuning is PVE-host-side only.
|
||||
5. Reboot must land in DAY mode: slice unit files carry day defaults; the
|
||||
profile engine only does runtime mutations. Non-negotiable invariant.
|
||||
6. `Persistent=true` timers replay missed flips at boot (intentional).
|
||||
7. Duplicated truth is the failure mode: one conf file per host is the only
|
||||
place values live (the 07:00 sed-edit of a live timer was the wart that
|
||||
motivated this redesign).
|
||||
8. Measure before touching; collectors install data, they tune nothing.
|
||||
|
||||
## Fleet rollout model
|
||||
|
||||
- Workstation (ultix-streaming): stays on its live ultix units until this
|
||||
module is UAT'd, then converges onto knelperf-* units.
|
||||
- PVE nodes: daynight engine applies to host-level services + VM cgroup
|
||||
scopes (`qemu.slice` children) via the same conf mechanism; timers install
|
||||
from `daynight/systemd/`.
|
||||
- Everything deploys through AWX job templates (KNELIAC) per the 100%-IaC
|
||||
mandate; this repo supplies the scripts, not ad-hoc ssh.
|
||||
|
||||
See `docs/ARCHITECTURE.md` for the full baseline → audit → tweak → re-baseline
|
||||
program this module feeds.
|
||||
|
||||
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
|
||||
Executable
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# KNELPerf perfsnap — periodic /proc+cgroupfs snapshot collector.
|
||||
# Ported from ultix staged/reachableceo-perfsnap.sh (#826); generalizations:
|
||||
# - watched cgroups and interfaces are env/conf-driven lists (no baked-in
|
||||
# user ids or NIC names)
|
||||
# - log dir env-driven; same pipe-delimited format for awk analysis
|
||||
#
|
||||
# Env (or /etc/knelperf/perfsnap.conf):
|
||||
# KNELPERF_PERF_LOGDIR default /var/log/knelperf-perf
|
||||
# KNELPERF_SNAP_CGROUPS space-separated "label:/sys/fs/cgroup/path" list
|
||||
# KNELPERF_SNAP_IFACES space-separated interface names
|
||||
# Fields (snap.log): ts|l1|l5|l15|run|blocked|memavail_kb|psi_cpu|psi_mem|
|
||||
# psi_io|<cgroup usage_usec per label>|<iface rx tx per label>
|
||||
# psi_* = avg60 "some" percentages. *_usec = cgroup cpu.stat usage_usec
|
||||
# (cumulative; analyze deltas).
|
||||
set -u
|
||||
CONF=/etc/knelperf/perfsnap.conf
|
||||
[ -r "$CONF" ] && . "$CONF"
|
||||
LOGDIR=${KNELPERF_PERF_LOGDIR:-/var/log/knelperf-perf}
|
||||
SNAP_CGROUPS=${KNELPERF_SNAP_CGROUPS:-}
|
||||
SNAP_IFACES=${KNELPERF_SNAP_IFACES:-}
|
||||
mkdir -p "$LOGDIR"
|
||||
ts=$(date +%s)
|
||||
|
||||
read -r l1 l5 l15 _ < /proc/loadavg
|
||||
run=$(awk '/procs_running/{print $2}' /proc/stat)
|
||||
blocked=$(awk '/procs_blocked/{print $2}' /proc/stat)
|
||||
memavail=$(awk '/MemAvailable/{print $2}' /proc/meminfo)
|
||||
|
||||
psi() { awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "/proc/pressure/$1" 2>/dev/null || echo 0; }
|
||||
|
||||
line="$ts|$l1|$l5|$l15|$run|$blocked|$memavail|$(psi cpu)|$(psi memory)|$(psi io)"
|
||||
|
||||
cgusec() { awk '/^usage_usec/{print $2}' "$1/cpu.stat" 2>/dev/null || echo 0; }
|
||||
for spec in $SNAP_CGROUPS; do
|
||||
line="$line|$(cgusec "${spec#*:}")"
|
||||
done
|
||||
|
||||
nd() { awk -v i="$1" '$1==i":":{print $2" "$10}' /proc/net/dev 2>/dev/null || echo "0 0"; }
|
||||
for ifc in $SNAP_IFACES; do
|
||||
read -r rx tx <<< "$(nd "$ifc")"
|
||||
line="$line|$rx|$tx"
|
||||
done
|
||||
|
||||
echo "$line" >> "$LOGDIR/snap.log"
|
||||
|
||||
ps -eo pcpu,pid,comm --sort=-pcpu --no-headers | head -5 \
|
||||
| awk -v t="$ts" '{print t"|"$1"|"$2"|"$3}' >> "$LOGDIR/procs.log"
|
||||
|
||||
# retention: 14 days; rotate anything over 50MB
|
||||
find "$LOGDIR" -name '*.log' -mtime +14 -delete 2>/dev/null
|
||||
for f in "$LOGDIR"/*.log; do
|
||||
[ "$(wc -c <"$f" 2>/dev/null || echo 0)" -gt 52428800 ] && mv "$f" "$f.1"
|
||||
done
|
||||
exit 0
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector.
|
||||
# Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations:
|
||||
# - cgroup glob is env-driven (any slice namespace, not just ukrrs-*)
|
||||
# - metric prefix is env-driven (default knelperf_)
|
||||
set -u
|
||||
OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile}
|
||||
CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup}
|
||||
CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice}
|
||||
PREFIX=${KNELPERF_PREFIX:-knelperf}
|
||||
mkdir -p "$OUT_DIR"
|
||||
tmp=$(mktemp "$OUT_DIR/.knelperf.XXXXXX")
|
||||
|
||||
some60() { # file -> prints avg60 value or nothing
|
||||
awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "$1" 2>/dev/null
|
||||
}
|
||||
|
||||
emit() { # name cgroup_path
|
||||
local res v
|
||||
for res in cpu memory io; do
|
||||
[ -r "$2/$res.pressure" ] || continue
|
||||
v=$(some60 "$2/$res.pressure")
|
||||
[ -n "${v:-}" ] && echo "${PREFIX}_slice_pressure_some60{slice=\"$1\",res=\"$res\"} $v"
|
||||
done
|
||||
}
|
||||
|
||||
echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp"
|
||||
echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp"
|
||||
for cg in $CG_GLOB; do
|
||||
[ -d "$cg" ] && emit "${cg##*/}" "$cg"
|
||||
done
|
||||
[ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice"
|
||||
|
||||
echo "# HELP ${PREFIX}_host_pressure_some60 PSI some avg60 (percent) host-wide" >>"$tmp"
|
||||
echo "# TYPE ${PREFIX}_host_pressure_some60 gauge" >>"$tmp"
|
||||
for res in cpu memory io; do
|
||||
[ -r "/proc/pressure/$res" ] || continue
|
||||
v=$(some60 "/proc/pressure/$res")
|
||||
[ -n "${v:-}" ] && echo "${PREFIX}_host_pressure_some60{res=\"$res\"} $v"
|
||||
done
|
||||
|
||||
mv "$tmp" "$OUT_DIR/${PREFIX}_pressure.prom"
|
||||
chmod 0644 "$OUT_DIR/${PREFIX}_pressure.prom"
|
||||
@@ -0,0 +1,28 @@
|
||||
# /etc/knelperf/daynight.conf — knelperf-daynight.sh configuration.
|
||||
# Copy to /etc/knelperf/daynight.conf and edit per host. Every field is
|
||||
# optional; omit slices/sections you do not manage on this host.
|
||||
#
|
||||
# Slice spec format (space-separated list):
|
||||
# slice[:CPUWeight[:AllowedCPUs[:MemoryHigh]]]
|
||||
# Empty field = skip that property. Day AND night lists are required for
|
||||
# any slice you list (reboot-safety invariant: day values == unit-file
|
||||
# defaults).
|
||||
|
||||
# Example (workstation, 8 vCPU — mirrors the ultix values):
|
||||
# DAY_SLICES="reachableceo-batch.slice:25:0-5:12G reachableceo-gateway.slice:900::"
|
||||
# NIGHT_SLICES="reachableceo-batch.slice:400:0-6:30G reachableceo-gateway.slice:500::"
|
||||
|
||||
# Example (PVE host with a batch pool):
|
||||
# DAY_SLICES="knelperf-batch.slice:25:0-3:8G"
|
||||
# NIGHT_SLICES="knelperf-batch.slice:400:0-7:24G"
|
||||
|
||||
# Block devices for night readahead boost (partitions auto-included):
|
||||
# BLOCK_DEVS="/dev/sdb"
|
||||
|
||||
# Defaults shown in the script; override only what differs:
|
||||
# DATA_RA_DAY=256
|
||||
# DATA_RA_NIGHT=2048
|
||||
# DIRTY_DAY=1073741824
|
||||
# DIRTY_NIGHT=2147483648
|
||||
# DIRTY_BG_DAY=268435456
|
||||
# DIRTY_BG_NIGHT=536870912
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# KNELPerf day/night resource profile engine.
|
||||
#
|
||||
# Broken out of ultix staged/ukrrs-daynight.sh (2026-09-06, #826) and
|
||||
# redesigned: slice names, devices, and all numeric knobs are CONF-driven
|
||||
# (no host-specific hardcoding); slice targets are a list so any host can
|
||||
# run it. Runtime-only changes (set-property --runtime + sysctl + readahead):
|
||||
# a reboot always lands safely in day mode, because slice unit files carry
|
||||
# the day defaults (unchanged ultix invariant).
|
||||
#
|
||||
# Usage: knelperf-daynight.sh [--conf FILE] day|night
|
||||
set -euo pipefail
|
||||
|
||||
CONF=${KNELPERF_DAYNIGHT_CONF:-/etc/knelperf/daynight.conf}
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--conf) CONF=$2; shift 2 ;;
|
||||
day|night) MODE=$1; shift ;;
|
||||
*) echo "usage: $0 [--conf FILE] day|night" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ "${MODE:-}" ] || { echo "usage: $0 [--conf FILE] day|night" >&2; exit 2; }
|
||||
[ -r "$CONF" ] && . "$CONF"
|
||||
|
||||
# Targets: "slice:CPUWeight:AllowedCPUs:MemoryHigh" per mode, space-separated.
|
||||
# Empty fields mean "skip that property". See knelperf-daynight.conf.example.
|
||||
: "${DAY_SLICES:=}"; : "${NIGHT_SLICES:=}"
|
||||
# Block devices for readahead tuning (space-separated; partitions auto-appended).
|
||||
: "${BLOCK_DEVS:=}"
|
||||
: "${DATA_RA_DAY:=256}"; : "${DATA_RA_NIGHT:=2048}"
|
||||
: "${DIRTY_DAY:=1073741824}"; : "${DIRTY_NIGHT:=2147483648}"
|
||||
: "${DIRTY_BG_DAY:=268435456}"; : "${DIRTY_BG_NIGHT:=536870912}"
|
||||
|
||||
log() { echo "[knelperf-daynight] $*"; }
|
||||
ra=$DATA_RA_DAY; dirty=$DIRTY_DAY; dirtybg=$DIRTY_BG_DAY; slices=$DAY_SLICES
|
||||
[ "$MODE" = night ] && { ra=$DATA_RA_NIGHT; dirty=$DIRTY_NIGHT; dirtybg=$DIRTY_BG_NIGHT; slices=$NIGHT_SLICES; }
|
||||
|
||||
apply_slice() { # "slice:weight:cpus:memhigh"
|
||||
local spec=$1 slice weight cpus memhigh
|
||||
IFS=: read -r slice weight cpus memhigh <<< "$spec"
|
||||
[ -n "$slice" ] || return 0
|
||||
local -a props=()
|
||||
[ -n "$weight" ] && props+=(CPUWeight="$weight")
|
||||
[ -n "$cpus" ] && props+=(AllowedCPUs="$cpus")
|
||||
[ -n "$memhigh" ] && props+=(MemoryHigh="$memhigh")
|
||||
if [ ${#props[@]} -gt 0 ]; then
|
||||
systemctl set-property --runtime "$slice" "${props[@]}" \
|
||||
|| log "WARN: set-property failed for $slice (slice not loaded?)"
|
||||
fi
|
||||
}
|
||||
|
||||
for spec in $slices; do apply_slice "$spec"; done
|
||||
|
||||
if [ -n "$BLOCK_DEVS" ]; then
|
||||
for dev in $BLOCK_DEVS; do
|
||||
blockdev --setra "$ra" "$dev" 2>/dev/null || log "readahead skip: $dev"
|
||||
for part in "${dev}"[0-9]*; do
|
||||
[ -e "$part" ] && blockdev --setra "$ra" "$part" 2>/dev/null || true
|
||||
done
|
||||
done
|
||||
fi
|
||||
sysctl -q -w vm.dirty_bytes="$dirty" vm.dirty_background_bytes="$dirtybg"
|
||||
log "profile $MODE applied $(date -Is)"
|
||||
@@ -0,0 +1,24 @@
|
||||
# systemd units — day/night profile timers
|
||||
|
||||
Install per host (adjust names to your unit namespace if not knelperf-):
|
||||
|
||||
install -m 0644 knelperf-dayprofile.service /etc/systemd/system/
|
||||
install -m 0644 knelperf-dayprofile.timer /etc/systemd/system/
|
||||
install -m 0644 knelperf-nightprofile.service /etc/systemd/system/
|
||||
install -m 0644 knelperf-nightprofile.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now knelperf-dayprofile.timer knelperf-nightprofile.timer
|
||||
|
||||
Design notes (carried from ultix, keep true here):
|
||||
|
||||
- `Persistent=true` — a missed flip (host powered off) replays at next boot,
|
||||
so the host can never get stuck in the wrong profile across a reboot cycle.
|
||||
- Services are `Type=oneshot` calling the profile engine with the SAME conf
|
||||
file; the conf — not the timer — is the only place values live (fixes the
|
||||
ultix wart where the day-flip time was sed-edited on the live timer).
|
||||
- Day schedule 07:00, night 22:00 local time (batch window 22:00–07:00).
|
||||
Change the OnCalendar here when a host needs a different window; the timers
|
||||
stay trivial and identical in shape.
|
||||
|
||||
Reboot-safety invariant: slice unit files must carry the DAY values as their
|
||||
static defaults, so an unattended reboot always lands in day mode.
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=KNELPerf day resource profile (interactive-first)
|
||||
After=multi-user.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/knelperf-daynight.sh day
|
||||
# Never restart a profile flip; the timer owns scheduling
|
||||
Restart=no
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=KNELPerf day profile flip (07:00 local)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 07:00:00
|
||||
Persistent=true
|
||||
Unit=knelperf-dayprofile.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,8 @@
|
||||
[Unit]
|
||||
Description=KNELPerf night resource profile (batch window 22:00-07:00)
|
||||
After=multi-user.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/knelperf-daynight.sh night
|
||||
Restart=no
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=KNELPerf night profile flip (22:00 local)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 22:00:00
|
||||
Persistent=true
|
||||
Unit=knelperf-nightprofile.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# UKRRS Proxmox control wrapper for harness/human use. HARD RULES baked in:
|
||||
# - vmids listed in PROTECTED are never touched (list THIS VM first).
|
||||
# - graceful shutdown only; no reset, no hard stop, no bare down (house rule).
|
||||
# Env:
|
||||
# PVE_LOCAL=1 run ON the pve host itself (pvesh direct)
|
||||
# or PVE_HOST + PVE_TOKEN remote, token format: user@realm!tokenid=secret
|
||||
# PVE_NODE node name (required for vm ops)
|
||||
# PROTECTED required, comma-separated vmids
|
||||
# Usage:
|
||||
# proxmox-ctl.sh vms
|
||||
# proxmox-ctl.sh shutdown <vmid>
|
||||
# proxmox-ctl.sh start <vmid>
|
||||
# proxmox-ctl.sh snapshot <vmid> <name>
|
||||
# Token setup (on the pve host, see docs/REPORT.md appendix B):
|
||||
# pveum user add ukrrs-infra@pam
|
||||
# pveum user token add ukrrs-infra@pam harness -privsep 0 -expire 0
|
||||
# pveum acl modify /pool/<preprod-pool> -user ukrrs-infra@pam -role PVEVMUser
|
||||
set -euo pipefail
|
||||
: "${PROTECTED:?set PROTECTED=vmid1,vmid2,... (this VM must be in the list)}"
|
||||
|
||||
pvesh_() {
|
||||
if [ "${PVE_LOCAL:-0}" = 1 ]; then
|
||||
command pvesh "$@"
|
||||
else
|
||||
: "${PVE_HOST:?}" "${PVE_TOKEN:?}"
|
||||
command pvesh --host "$PVE_HOST" --api-token "$PVE_TOKEN" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
guard() {
|
||||
case ",$PROTECTED," in
|
||||
*",$1,"*) echo "REFUSED: vmid $1 is PROTECTED" >&2; exit 3 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
cmd=${1:-}; shift || true
|
||||
case "$cmd" in
|
||||
vms)
|
||||
pvesh_ get /cluster/resources --type vm ;;
|
||||
shutdown)
|
||||
[ $# = 1 ] || { echo "usage: $0 shutdown <vmid>" >&2; exit 2; }
|
||||
guard "$1"; : "${PVE_NODE:?}"
|
||||
pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/shutdown" --timeout 120 ;;
|
||||
start)
|
||||
[ $# = 1 ] || { echo "usage: $0 start <vmid>" >&2; exit 2; }
|
||||
guard "$1"; : "${PVE_NODE:?}"
|
||||
pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/start" ;;
|
||||
snapshot)
|
||||
[ $# = 2 ] || { echo "usage: $0 snapshot <vmid> <name>" >&2; exit 2; }
|
||||
guard "$1"; : "${PVE_NODE:?}"
|
||||
pvesh_ create "/nodes/$PVE_NODE/qemu/$1/snapshot" snapname="$2" ;;
|
||||
*)
|
||||
echo "usage: $0 vms|shutdown <vmid>|start <vmid>|snapshot <vmid> <name>" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user