Fleet monitoring: sysstat 28d + perfsnap/PSI/iotop timers + on-demand toolbox container (#826)
ci / audit (push) Successful in 26s

- monitoring/node-install.sh: idempotent per-node installer (sar HISTORY=28,
  three oneshot timers, per-node conf with qemu.slice/lxc cgroup globs)
- monitoring/collectors + systemd units: iotop-snap 15min, perfsnap 60s,
  psi-textfile 15s (now sources /etc/knelperf/perfsnap.conf; multi-glob)
- monitoring/toolbox: pct LXC (vmid 950, privileged, apparmor unconfined,
  stopped at rest) + knelperf-toolbox wrapper + Dockerfile variant
- monitoring/deploy-fleet.sh: workstation-side rollout driver
- piloted on pfv-tsys1: per-VM PSI gauges verified live

Redmine: https://projects.knownelement.com/issues/826
This commit is contained in:
2026-09-06 18:50:07 -05:00
parent f1729868fa
commit 3680f1dba1
16 changed files with 407 additions and 1 deletions
+2
View File
@@ -45,6 +45,7 @@ flowchart LR
| Module | What it does | Entry point |
|---|---|---|
| `monitoring/` | fleet instrumentation: sysstat 28d tuning, perfsnap/PSI/iotop timers, on-demand perf toolbox container (LXC on PVE nodes, Docker image for the rest) | `monitoring/deploy-fleet.sh` |
| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline <host>` |
| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` |
| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` |
@@ -76,6 +77,7 @@ KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh
| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths |
| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) |
| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward |
| [monitoring/README.md](monitoring/README.md) | what lands on each node, toolbox cookbook, analysis recipes, prod/CR gating |
Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar)
(solar economics + ONCOR arbitrage + ERCOT risk brief) ·
+76
View File
@@ -0,0 +1,76 @@
# 📡 Fleet monitoring & the on-demand toolbox
> **Redmine:** [#826](https://projects.knownelement.com/issues/826) ·
> **Docs:** [Discourse](https://community.turnsys.com/t/328) ·
> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine)
Continuous lightweight instrumentation for the seven PFV PVE nodes
(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is
stopped at rest and runs on demand. Ultix's instrumentation pattern
(60s perfsnap + PSI collectors, containers for deep dives) generalized
fleet-wide. AGPLv3, like the whole repo.
## What lands on a node
| Piece | Cadence | Data | Where |
|---|---|---|---|
| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) |
| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` |
| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` |
| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` |
No reboots, no package installs (sysstat + iotop ship with PVE 9),
no daemons — three oneshot systemd timers. Rollback = disable timers +
restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on
the hypervisor root.
## Deploy
From a tooling host with root SSH to the nodes:
```bash
# non-prod first, prod only after the GLPI CR exists
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
```
`--toolbox` additionally creates the per-node LXC (see below). The
installer (`monitoring/node-install.sh`) is idempotent — safe to re-run.
## The toolbox container (vmid 950, `knelperf-toolbox`)
PVE-native LXC on each node — Debian 12, privileged, apparmor
unconfined, bind mounts for host `/proc`, `/sys/fs/cgroup`, debugfs,
tracingfs, `/var/log` (ro) and up to 8 block devices. **Stopped at
rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop, sysbench,
fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
```bash
knelperf-toolbox run 'iostat -x 1 5' # starts LXC if stopped
knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06' # host sar, offline
knelperf-toolbox shell # interactive
knelperf-toolbox stop # back to zero footprint
```
Docker-capable hosts use the same manifest as an image:
`monitoring/toolbox/Dockerfile``git.knownelement.com/knel/knelperf-toolbox:v1`.
## Analysis recipes
```bash
sar -q -f /var/log/sysstat/sa06 # load vs runq
sar -d -p -f /var/log/sysstat/sa06 # per-device IO
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log # PSI trend
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log # io offenders
```
The `loop/perf-loop.sh` driver consumes these on its `audit` pass;
`baseline-diff.sh` compares before/after tweaks. Per-guest PSI
(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard
(ARCHITECTURE §3).
## Prod gating
Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout
exists and references this directory. Verification evidence goes back on
the CR; closing it is human-only.
@@ -0,0 +1,22 @@
#!/usr/bin/env bash
# KNELPerf iotop snapshot — periodic batch samples of active per-process I/O.
# Complements sar (system-wide device stats) with the WHO behind the I/O.
# Env (or /etc/knelperf/perfsnap.conf):
# KNELPERF_IOTOP_LOGDIR default /var/log/knelperf-iotop
# Output: daily file iotop-YYYYMMDD.log; per tick one header line then
# 3 batch iterations x 2s of processes with outstanding I/O (-botqq -k).
# Docs: https://community.turnsys.com/t/328
set -u
CONF=/etc/knelperf/perfsnap.conf
[ -r "$CONF" ] && . "$CONF"
LOGDIR=${KNELPERF_IOTOP_LOGDIR:-/var/log/knelperf-iotop}
mkdir -p "$LOGDIR"
F="$LOGDIR/iotop-$(date +%Y%m%d).log"
{
echo "### $(date -Iseconds)"
iotop -botqq -k -n 3 -d 2 2>/dev/null
} >> "$F"
# retention: 14 days; rotate the live file at 25MB
find "$LOGDIR" -name 'iotop-*.log*' -mtime +14 -delete 2>/dev/null
[ "$(wc -c <"$F" 2>/dev/null || echo 0)" -gt 26214400 ] && mv "$F" "$F.1"
exit 0
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# Deploy KNELPerf node monitoring (+ toolbox LXC) to PVE fleet nodes (#826).
# Runs from the workstation / a tooling host — never on a node itself.
# deploy-fleet.sh [--toolbox] host [host...]
# Prod nodes (tsys4/5/6/7) deploy ONLY after their GLPI CR exists.
# Docs: https://community.turnsys.com/t/328
set -euo pipefail
here=$(cd "$(dirname "$0")/.." && pwd)
WITH_TOOLBOX=0
if [ "${1:-}" = --toolbox ]; then WITH_TOOLBOX=1; shift; fi
[ $# -ge 1 ] || { echo "usage: $0 [--toolbox] host..." >&2; exit 1; }
for host in "$@"; do
echo "=== $host ==="
timeout 30 ssh -o ConnectTimeout=8 "root@$host" 'mkdir -p /tmp/knelperf-install'
tar -C "$here" -cf - scheduler/collectors monitoring \
| timeout 120 ssh "root@$host" 'tar -C /tmp/knelperf-install -xf -'
timeout 120 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/node-install.sh'
if [ "$WITH_TOOLBOX" = 1 ]; then
timeout 600 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/toolbox/lxc-toolbox.sh --create'
fi
timeout 30 ssh "root@$host" 'rm -rf /tmp/knelperf-install'
done
echo "fleet deploy done."
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
# KNELPerf node monitoring installer — sysstat tuning + collectors + timers.
# Idempotent; run as ROOT on a PVE node from a repo checkout:
# bash monitoring/node-install.sh
# Surface (all details in monitoring/README.md):
# - /etc/sysstat/sysstat HISTORY 7 -> 28, COMPRESSAFTER 10 -> 31
# - /usr/local/sbin/knelperf-{perfsnap,psi-textfile,iotop-snap}.sh
# - /etc/systemd/system/knelperf-*.{service,timer} (enabled timers)
# - /etc/knelperf/perfsnap.conf per-node ifaces/cgroup globs
# - logs under /var/log/knelperf-{perf,iotop}, textfiles /var/lib/knelperf/textfile
# No reboots, no package installs (PVE ships sysstat + iotop already).
# Docs: https://community.turnsys.com/t/328
set -euo pipefail
[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
here=$(cd "$(dirname "$0")/.." && pwd)
say() { echo "[knelperf-install] $*"; }
# 1) sysstat: keep 28 days of sar history (two-week program + re-baseline window)
if [ -f /etc/sysstat/sysstat ]; then
sed -i -E 's/^HISTORY=.*/HISTORY=28/; s/^COMPRESSAFTER=.*/COMPRESSAFTER=31/' /etc/sysstat/sysstat
sed -i -E 's/^ENABLED=.*/ENABLED="true"/' /etc/default/sysstat
systemctl restart sysstat 2>/dev/null || true
/usr/libexec/sysstat/debian-sa1 1 1 2>/dev/null || true
today=/var/log/sysstat/sa$(date +%d)
if [ -f "$today" ]; then say "sar ok: $today present"; else say "sar WARN: $today missing (check /etc/cron.d/sysstat)"; fi
else
say "WARN: no /etc/sysstat/sysstat on this host — sar tuning skipped"
fi
# 2) collectors + units
install -m 0755 "$here/scheduler/collectors/knelperf-perfsnap.sh" /usr/local/sbin/knelperf-perfsnap.sh
install -m 0755 "$here/scheduler/collectors/knelperf-psi-textfile.sh" /usr/local/sbin/knelperf-psi-textfile.sh
install -m 0755 "$here/monitoring/collectors/knelperf-iotop-snap.sh" /usr/local/sbin/knelperf-iotop-snap.sh
for u in perfsnap psi-textfile iotop-snap; do
install -m 0644 "$here/monitoring/systemd/knelperf-$u.service" /etc/systemd/system/knelperf-$u.service
install -m 0644 "$here/monitoring/systemd/knelperf-$u.timer" /etc/systemd/system/knelperf-$u.timer
done
# 3) per-node conf: watch PVE guest cgroups + real bridges/tailscale
ifaces=""
for i in vmbr0 datanet tailscale0; do
ip link show "$i" >/dev/null 2>&1 && ifaces="$ifaces $i"
done
mkdir -p /etc/knelperf
cat > /etc/knelperf/perfsnap.conf <<EOF
# /etc/knelperf/perfsnap.conf — generated by KNELPerf node-install.sh $(date -I)
KNELPERF_SNAP_CGROUPS="qemu:/sys/fs/cgroup/qemu.slice lxc:/sys/fs/cgroup/lxc"
KNELPERF_SNAP_IFACES="${ifaces# }"
KNELPERF_CG_GLOB="/sys/fs/cgroup/qemu.slice/*.scope /sys/fs/cgroup/lxc/*"
KNELPERF_TEXTFILE_DIR=/var/lib/knelperf/textfile
KNELPERF_PERF_LOGDIR=/var/log/knelperf-perf
KNELPERF_IOTOP_LOGDIR=/var/log/knelperf-iotop
EOF
say "conf written (ifaces: ${ifaces# })"
# 4) timers on + first tick of each collector
systemctl daemon-reload
for u in knelperf-perfsnap knelperf-psi-textfile knelperf-iotop-snap; do
systemctl enable --now "$u.timer" >/dev/null 2>&1
done
/usr/local/sbin/knelperf-perfsnap.sh
/usr/local/sbin/knelperf-psi-textfile.sh
/usr/local/sbin/knelperf-iotop-snap.sh
say "timers enabled; first ticks logged"
# 5) evidence
say "--- timers ---"
systemctl list-timers 'knelperf-*' --no-pager | sed -n '1,5p'
say "--- snap.log tail ---"
tail -1 /var/log/knelperf-perf/snap.log
say "--- textfile ---"
head -3 /var/lib/knelperf/textfile/knelperf_pressure.prom 2>/dev/null || true
say "done."
@@ -0,0 +1,7 @@
[Unit]
Description=KNELPerf iotop snapshot (3x2s active-I/O samples)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/knelperf-iotop-snap.sh
IOSchedulingClass=idle
@@ -0,0 +1,10 @@
[Unit]
Description=Run knelperf-iotop-snap every 15 minutes
[Timer]
OnCalendar=*:0/15
Persistent=true
RandomizedDelaySec=30
[Install]
WantedBy=timers.target
@@ -0,0 +1,6 @@
[Unit]
Description=KNELPerf perfsnap collector tick (/proc+cgroupfs snapshot)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/knelperf-perfsnap.sh
@@ -0,0 +1,11 @@
[Unit]
Description=Run knelperf-perfsnap every 60s
[Timer]
OnBootSec=45
OnUnitActiveSec=60
AccuracySec=5s
Persistent=true
[Install]
WantedBy=timers.target
@@ -0,0 +1,6 @@
[Unit]
Description=KNELPerf PSI textfile collector tick (pressure avg60 gauges)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/knelperf-psi-textfile.sh
@@ -0,0 +1,11 @@
[Unit]
Description=Run knelperf-psi-textfile every 15s
[Timer]
OnBootSec=20
OnUnitActiveSec=15
AccuracySec=2s
Persistent=true
[Install]
WantedBy=timers.target
+16
View File
@@ -0,0 +1,16 @@
# knelperf-toolbox — Docker image variant for docker-capable hosts
# (workstation, VMs). On PVE nodes the LXC variant (lxc-toolbox.sh) is
# canonical. Package list is shared: toolbox-pkgs.txt.
# Build: docker build -t git.knownelement.com/knel/knelperf-toolbox:v1 \
# -f monitoring/toolbox/Dockerfile monitoring/toolbox/
# Docs: https://community.turnsys.com/t/328
FROM debian:12.9-slim
COPY toolbox-pkgs.txt /tmp/toolbox-pkgs.txt
RUN apt-get update -qq && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
$(sed 's/#.*//' /tmp/toolbox-pkgs.txt | tr '\n' ' ') && \
rm -rf /var/lib/apt/lists/* /tmp/toolbox-pkgs.txt
WORKDIR /work
ENTRYPOINT ["/bin/bash"]
+27
View File
@@ -0,0 +1,27 @@
#!/bin/sh
# knelerf-toolbox — front door to the on-demand perf toolbox LXC on this node.
# usage: knelperf-toolbox run '<cmd>' (starts the LXC if stopped, runs, leaves it up)
# knelperf-toolbox shell (interactive)
# knelperf-toolbox start|stop|status
# The LXC (vmid 950) is STOPPED at rest — zero footprint; stop it when done.
# Docs: https://community.turnsys.com/t/328
VMID=${KNELPERF_TOOLBOX_VMID:-950}
ensure_running() {
pct status "$VMID" 2>/dev/null | grep -q running && return 0
pct start "$VMID"
i=0
until pct exec "$VMID" -- true 2>/dev/null; do
i=$((i + 1)); [ $i -gt 45 ] && { echo "boot timeout" >&2; return 1; }
sleep 2
done
}
case "${1:-help}" in
run) shift; ensure_running && pct exec "$VMID" -- sh -c "$*" ;;
shell) ensure_running && pct enter "$VMID" ;;
start) ensure_running ;;
stop) pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" ;;
status) pct status "$VMID" 2>/dev/null || echo "vmid $VMID not present" ;;
*) sed -n '2,6p' "$0" | sed 's/^# \?//' ;;
esac
+75
View File
@@ -0,0 +1,75 @@
#!/usr/bin/env bash
# knelperf-toolbox LXC — the on-demand perf analysis container for a PVE node.
# STOPS to zero footprint; started only when a human/agent runs a workload.
# lxc-toolbox.sh [--vmid N] {--create|--upgrade|--remove|--facts}
# Why privileged + apparmor unconfined: bpftrace/perf need host kernel BPF,
# tracefs and host /proc. Bind mounts: /host/proc, /host/cgroup, host debugfs
# + tracing, /host/var/log (ro) for offline sar analysis, block devs for
# smart/nvme. Package list lives in toolbox-pkgs.txt next to this script.
# Docs: https://community.turnsys.com/t/328
set -euo pipefail
[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
here=$(cd "$(dirname "$0")" && pwd)
VMID=950
TPL=debian-12-standard_12.12-1_amd64.tar.zst
case "${1:-}" in
--vmid) VMID=$2; shift 2 ;;
esac
case "${1:-}" in
--facts)
pct config "$VMID" 2>/dev/null || echo "vmid $VMID not present"
pct status "$VMID" 2>/dev/null || true
exit 0 ;;
--remove)
pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" 2>/dev/null || true
pct destroy "$VMID" --purge 1
rm -f /usr/local/sbin/knelperf-toolbox
echo "toolbox $VMID removed"
exit 0 ;;
--create|--upgrade) action=$1 ;;
*) sed -n 's/^# \?//p' "$0" | sed -n '2,5p'; exit 1 ;;
esac
if [ "$action" = --create ]; then
if pct status "$VMID" >/dev/null 2>&1; then
echo "vmid $VMID already present — use --upgrade" >&2; exit 1
fi
pveam list local 2>/dev/null | grep -q "$TPL" || pveam download local "$TPL"
# block-device passthrough for smartctl/nvme (first 8 devices)
dev_args=()
for d in /dev/nvme[0-9]n1 /dev/sd[a-z]; do
[ -b "$d" ] || continue
[ "${#dev_args[@]}" -ge 16 ] && break
dev_args+=("--dev$(( ${#dev_args[@]} / 2 ))" "$d")
done
pct create "$VMID" "local:vztmpl/$TPL" \
--hostname knelperf-toolbox \
--ostype debian \
--cores 2 --memory 2048 --swap 512 \
--rootfs local-lvm:8 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
--unprivileged 0 --onboot 0 \
--mp0 /proc,mp=/host/proc \
--mp1 /sys/kernel/debug,mp=/sys/kernel/debug \
--mp2 /sys/kernel/tracing,mp=/sys/kernel/tracing \
--mp3 /sys/fs/cgroup,mp=/host/cgroup \
--mp4 /var/log,mp=/host/var/log,ro=1 \
"${dev_args[@]}"
pct set "$VMID" --raw.lxc 'lxc.apparmor.profile=unconfined'
fi
# --create continues here; --upgrade rejoins an existing container
was_running=1
pct status "$VMID" 2>/dev/null | grep -q running || { was_running=0; pct start "$VMID"; }
i=0
until pct exec "$VMID" -- true 2>/dev/null; do
i=$((i + 1)); [ $i -gt 60 ] && { echo "boot timeout" >&2; exit 1; }
sleep 2
done
pkgs=$(sed 's/#.*//' "$here/toolbox-pkgs.txt" | tr '\n' ' ')
pct exec "$VMID" -- bash -c \
"DEBIAN_FRONTEND=noninteractive apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install $pkgs"
install -m 0755 "$here/knelperf-toolbox" /usr/local/sbin/knelperf-toolbox
[ "$was_running" = 0 ] && pct shutdown "$VMID" --timeout 60
echo "toolbox $VMID $action done: knelperf-toolbox wrapper installed"
pct status "$VMID"
+37
View File
@@ -0,0 +1,37 @@
# knelperf-toolbox package manifest — shared by the LXC bootstrap and the
# Docker image variant. One package (or apt argument) per line; # = comment.
# Docs: https://community.turnsys.com/t/328
bpftrace
bcc-tools
sysstat
iotop
sysbench
fio
iperf3
stress-ng
linux-perf
jq
bc
curl
procps
psmisc
util-linux
pciutils
usbutils
ethtool
net-tools
bind9-dnsutils
smartmontools
nvme-cli
hdparm
lm-sensors
numactl
hwloc-nox
strace
ltrace
file
rsync
openssh-client
vim-tiny
less
tmux
@@ -1,9 +1,11 @@
#!/usr/bin/env bash
# KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector.
# Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations:
# - cgroup glob is env-driven (any slice namespace, not just ukrrs-*)
# - cgroup globs are env-driven (space-separated LIST, any slice namespace)
# - metric prefix is env-driven (default knelperf_)
set -u
CONF=/etc/knelperf/perfsnap.conf
[ -r "$CONF" ] && . "$CONF"
OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile}
CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup}
CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice}
@@ -27,6 +29,7 @@ emit() { # name cgroup_path
echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp"
echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp"
for cg in $CG_GLOB; do
case $cg in "$CGROOT"/*) ;; *) continue ;; esac # skip unmatched glob literals
[ -d "$cg" ] && emit "${cg##*/}" "$cg"
done
[ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice"