ic-builder
3680f1dba1
- monitoring/node-install.sh: idempotent per-node installer (sar HISTORY=28, three oneshot timers, per-node conf with qemu.slice/lxc cgroup globs) - monitoring/collectors + systemd units: iotop-snap 15min, perfsnap 60s, psi-textfile 15s (now sources /etc/knelperf/perfsnap.conf; multi-glob) - monitoring/toolbox: pct LXC (vmid 950, privileged, apparmor unconfined, stopped at rest) + knelperf-toolbox wrapper + Dockerfile variant - monitoring/deploy-fleet.sh: workstation-side rollout driver - piloted on pfv-tsys1: per-VM PSI gauges verified live Redmine: https://projects.knownelement.com/issues/826
📡 Fleet monitoring & the on-demand toolbox
Redmine: #826 · Docs: Discourse · Program:
docs/ARCHITECTURE.md(§5 telemetry spine)
Continuous lightweight instrumentation for the seven PFV PVE nodes (tsys1/3/4/5/6/7/9) plus a per-node perf toolbox container that is stopped at rest and runs on demand. Ultix's instrumentation pattern (60s perfsnap + PSI collectors, containers for deep dives) generalized fleet-wide. AGPLv3, like the whole repo.
What lands on a node
| Piece | Cadence | Data | Where |
|---|---|---|---|
sysstat sadc (ships with PVE, tuned here) |
10 min + daily rollup | sar: CPU, mem, disks, net, load | /var/log/sysstat/saDD (HISTORY=28) |
knelperf-perfsnap.sh |
60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | /var/log/knelperf-perf/{snap,procs}.log |
knelperf-psi-textfile.sh |
15 s | PSI some60 per guest slice + host (Prometheus textfile format) | /var/lib/knelperf/textfile/knelperf_pressure.prom |
knelperf-iotop-snap.sh |
15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | /var/log/knelperf-iotop/iotop-YYYYMMDD.log |
No reboots, no package installs (sysstat + iotop ship with PVE 9),
no daemons — three oneshot systemd timers. Rollback = disable timers +
restore HISTORY=7. bpftrace/BCC live inside the toolbox, never on
the hypervisor root.
Deploy
From a tooling host with root SSH to the nodes:
# non-prod first, prod only after the GLPI CR exists
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
--toolbox additionally creates the per-node LXC (see below). The
installer (monitoring/node-install.sh) is idempotent — safe to re-run.
The toolbox container (vmid 950, knelperf-toolbox)
PVE-native LXC on each node — Debian 12, privileged, apparmor
unconfined, bind mounts for host /proc, /sys/fs/cgroup, debugfs,
tracingfs, /var/log (ro) and up to 8 block devices. Stopped at
rest; onboot=0. Tools: bpftrace, BCC, sysstat, iotop, sysbench,
fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
knelperf-toolbox run 'iostat -x 1 5' # starts LXC if stopped
knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06' # host sar, offline
knelperf-toolbox shell # interactive
knelperf-toolbox stop # back to zero footprint
Docker-capable hosts use the same manifest as an image:
monitoring/toolbox/Dockerfile → git.knownelement.com/knel/knelperf-toolbox:v1.
Analysis recipes
sar -q -f /var/log/sysstat/sa06 # load vs runq
sar -d -p -f /var/log/sysstat/sa06 # per-device IO
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log # PSI trend
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log # io offenders
The loop/perf-loop.sh driver consumes these on its audit pass;
baseline-diff.sh compares before/after tweaks. Per-guest PSI
(qemu.slice/*.scope, lxc/*) feeds the placement scorecard
(ARCHITECTURE §3).
Prod gating
Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout exists and references this directory. Verification evidence goes back on the CR; closing it is human-only.