# 📡 Fleet monitoring & the on-demand toolbox

> **Redmine:** [#826](https://projects.knownelement.com/issues/826) ·
> **Docs:** [Discourse](https://community.turnsys.com/t/328) ·
> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine)

Continuous lightweight instrumentation for the seven PFV PVE nodes
(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is
stopped at rest and runs on demand. Ultix's instrumentation pattern
(60s perfsnap + PSI collectors, containers for deep dives) generalized
fleet-wide. AGPLv3, like the whole repo.

## What lands on a node

| Piece | Cadence | Data | Where |
|---|---|---|---|
| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) |
| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` |
| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` |
| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` |

No reboots, no package installs (sysstat + iotop ship with PVE 9),
no daemons — three oneshot systemd timers. Rollback = disable timers +
restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on
the hypervisor root.

## Deploy

From a tooling host with root SSH to the nodes:

```bash
# non-prod first, prod only after the GLPI CR exists
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
```

`--toolbox` additionally creates the per-node LXC (see below). The
installer (`monitoring/node-install.sh`) is idempotent — safe to re-run.

## The toolbox container (vmid 950, `knelperf-toolbox`)

PVE-native LXC on each node — Debian 12, privileged, apparmor
unconfined, bind mounts for host cgroupfs and `/var/log` (ro), up to 8
block devices, and runtime-mounted tracefs/debugfs (pct skips
virtual-fs bind sources — the wrapper mounts them on start).
**Stopped at rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop,
sysbench, fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…

```bash
knelerf-toolbox run 'iostat -x 1 5'          # starts LXC if stopped
knelerf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
knelerf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06'   # host sar, offline
knelerf-toolbox run 'smartctl -d sat -H /dev/sda'            # SAT devices need -d sat; NVMe doesn't
knelerf-toolbox shell                          # interactive
knelerf-toolbox stop                           # back to zero footprint
```

Docker-capable hosts use the same manifest as an image:
`monitoring/toolbox/Dockerfile` → `git.knownelement.com/knel/knelperf-toolbox:v1`.

## Analysis recipes

```bash
sar -q -f /var/log/sysstat/sa06                          # load vs runq
sar -d -p -f /var/log/sysstat/sa06                       # per-device IO
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log   # PSI trend
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log  # io offenders
```

The `loop/perf-loop.sh` driver consumes these on its `audit` pass;
`baseline-diff.sh` compares before/after tweaks. Per-guest PSI
(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard
(ARCHITECTURE §3).

## Prod gating

Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout
exists and references this directory. Verification evidence goes back on
the CR; closing it is human-only.