KNELPerf/monitoring/README.md

📡 Fleet monitoring & the on-demand toolbox

Redmine: #826 · Docs: Discourse · Program: docs/ARCHITECTURE.md (§5 telemetry spine)

Continuous lightweight instrumentation for the seven PFV PVE nodes (tsys1/3/4/5/6/7/9) plus a per-node perf toolbox container that is stopped at rest and runs on demand. Ultix's instrumentation pattern (60s perfsnap + PSI collectors, containers for deep dives) generalized fleet-wide. AGPLv3, like the whole repo.

What lands on a node

Piece	Cadence	Data	Where
sysstat sadc (ships with PVE, tuned here)	10 min + daily rollup	sar: CPU, mem, disks, net, load	/var/log/sysstat/saDD (HISTORY=28)
knelperf-perfsnap.sh	60 s	load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters	/var/log/knelperf-perf/{snap,procs}.log
knelperf-psi-textfile.sh	15 s	PSI some60 per guest slice + host (Prometheus textfile format)	/var/lib/knelperf/textfile/knelperf_pressure.prom
knelperf-iotop-snap.sh	15 min	WHO does the I/O sar only aggregates (3×2 s active-only samples)	/var/log/knelperf-iotop/iotop-YYYYMMDD.log

No reboots, no package installs (sysstat + iotop ship with PVE 9), no daemons — three oneshot systemd timers. Rollback = disable timers + restore HISTORY=7. bpftrace/BCC live inside the toolbox, never on the hypervisor root.

Deploy

From a tooling host with root SSH to the nodes:

# non-prod first, prod only after the GLPI CR exists
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9

--toolbox additionally creates the per-node LXC (see below). The installer (monitoring/node-install.sh) is idempotent — safe to re-run.

The toolbox container (vmid 950, knelperf-toolbox)

PVE-native LXC on each node — Debian 12, privileged, apparmor unconfined, bind mounts for host cgroupfs and /var/log (ro), up to 8 block devices, and runtime-mounted tracefs/debugfs (pct skips virtual-fs bind sources — the wrapper mounts them on start). Stopped at rest; onboot=0. Tools: bpftrace, BCC, sysstat, iotop, sysbench, fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…

knelperf-toolbox run 'iostat -x 1 5'          # starts LXC if stopped
knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06'   # host sar, offline
knelperf-toolbox run 'smartctl -d sat -H /dev/sda'            # SAT devices need -d sat; NVMe doesn't
knelperf-toolbox shell                          # interactive
knelperf-toolbox stop                           # back to zero footprint

Docker-capable hosts use the same manifest as an image: git.knownelement.com/knel/knelperf-toolbox:v1 (digest sha256:3bb22bdb4a69707aadd0043d5151f863634a3ce5229f63d12773d5e3f90e001c, from monitoring/toolbox/Dockerfile).

Analysis recipes

sar -q -f /var/log/sysstat/sa06                          # load vs runq
sar -d -p -f /var/log/sysstat/sa06                       # per-device IO
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log   # PSI trend
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log  # io offenders

The loop/perf-loop.sh driver consumes these on its audit pass; baseline-diff.sh compares before/after tweaks. Per-guest PSI (qemu.slice/*.scope, lxc/*) feeds the placement scorecard (ARCHITECTURE §3).

Prod gating

Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout exists and references this directory. Verification evidence goes back on the CR; closing it is human-only.