Fleet monitoring: sysstat 28d + perfsnap/PSI/iotop timers + on-demand toolbox container (#826)

- monitoring/node-install.sh: idempotent per-node installer (sar HISTORY=28,
  three oneshot timers, per-node conf with qemu.slice/lxc cgroup globs)
- monitoring/collectors + systemd units: iotop-snap 15min, perfsnap 60s,
  psi-textfile 15s (now sources /etc/knelperf/perfsnap.conf; multi-glob)
- monitoring/toolbox: pct LXC (vmid 950, privileged, apparmor unconfined,
  stopped at rest) + knelperf-toolbox wrapper + Dockerfile variant
- monitoring/deploy-fleet.sh: workstation-side rollout driver
- piloted on pfv-tsys1: per-VM PSI gauges verified live

Redmine: https://projects.knownelement.com/issues/826

monitoring/README.md

@@ -0,0 +1,76 @@
+# 📡 Fleet monitoring & the on-demand toolbox
+
+> **Redmine:** [#826](https://projects.knownelement.com/issues/826) ·
+> **Docs:** [Discourse](https://community.turnsys.com/t/328) ·
+> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine)
+
+Continuous lightweight instrumentation for the seven PFV PVE nodes
+(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is
+stopped at rest and runs on demand. Ultix's instrumentation pattern
+(60s perfsnap + PSI collectors, containers for deep dives) generalized
+fleet-wide. AGPLv3, like the whole repo.
+
+## What lands on a node
+
+| Piece | Cadence | Data | Where |
+|---|---|---|---|
+| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) |
+| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` |
+| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` |
+| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` |
+
+No reboots, no package installs (sysstat + iotop ship with PVE 9),
+no daemons — three oneshot systemd timers. Rollback = disable timers +
+restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on
+the hypervisor root.
+
+## Deploy
+
+From a tooling host with root SSH to the nodes:
+
+```bash
+# non-prod first, prod only after the GLPI CR exists
+bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
+```
+
+`--toolbox` additionally creates the per-node LXC (see below). The
+installer (`monitoring/node-install.sh`) is idempotent — safe to re-run.
+
+## The toolbox container (vmid 950, `knelperf-toolbox`)
+
+PVE-native LXC on each node — Debian 12, privileged, apparmor
+unconfined, bind mounts for host `/proc`, `/sys/fs/cgroup`, debugfs,
+tracingfs, `/var/log` (ro) and up to 8 block devices. **Stopped at
+rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop, sysbench,
+fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
+
+```bash
+knelperf-toolbox run 'iostat -x 1 5'          # starts LXC if stopped
+knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
+knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06'   # host sar, offline
+knelperf-toolbox shell                          # interactive
+knelperf-toolbox stop                           # back to zero footprint
+```
+
+Docker-capable hosts use the same manifest as an image:
+`monitoring/toolbox/Dockerfile` → `git.knownelement.com/knel/knelperf-toolbox:v1`.
+
+## Analysis recipes
+
+```bash
+sar -q -f /var/log/sysstat/sa06                          # load vs runq
+sar -d -p -f /var/log/sysstat/sa06                       # per-device IO
+awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log   # PSI trend
+grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log  # io offenders
+```
+
+The `loop/perf-loop.sh` driver consumes these on its `audit` pass;
+`baseline-diff.sh` compares before/after tweaks. Per-guest PSI
+(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard
+(ARCHITECTURE §3).
+
+## Prod gating
+
+Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout
+exists and references this directory. Verification evidence goes back on
+the CR; closing it is human-only.