ic-builder
a7806eb3bd
ci / audit (push) Successful in 27s
- data/baselines/: cpu-bench bundles for tsys1/3/4/5/6/7/9 (2026-09-06) headliners: tsys9 1.20M k/s AES-1t > tsys3 986k > tsys4 614k > tsys1 582k > tsys5 462k (governor=none, E5620) > tsys7 401k > tsys6 176k - monitoring/README: toolbox image digest pinned (git.knownelement.com/knel/knelerf-toolbox:v1 @ sha256:3bb22bdb...) - STATUS.md: pointer to #826 program state Redmine: https://projects.knownelement.com/issues/826
79 lines
3.7 KiB
Markdown
79 lines
3.7 KiB
Markdown
# 📡 Fleet monitoring & the on-demand toolbox
|
||
|
||
> **Redmine:** [#826](https://projects.knownelement.com/issues/826) ·
|
||
> **Docs:** [Discourse](https://community.turnsys.com/t/328) ·
|
||
> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine)
|
||
|
||
Continuous lightweight instrumentation for the seven PFV PVE nodes
|
||
(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is
|
||
stopped at rest and runs on demand. Ultix's instrumentation pattern
|
||
(60s perfsnap + PSI collectors, containers for deep dives) generalized
|
||
fleet-wide. AGPLv3, like the whole repo.
|
||
|
||
## What lands on a node
|
||
|
||
| Piece | Cadence | Data | Where |
|
||
|---|---|---|---|
|
||
| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) |
|
||
| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` |
|
||
| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` |
|
||
| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` |
|
||
|
||
No reboots, no package installs (sysstat + iotop ship with PVE 9),
|
||
no daemons — three oneshot systemd timers. Rollback = disable timers +
|
||
restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on
|
||
the hypervisor root.
|
||
|
||
## Deploy
|
||
|
||
From a tooling host with root SSH to the nodes:
|
||
|
||
```bash
|
||
# non-prod first, prod only after the GLPI CR exists
|
||
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
|
||
```
|
||
|
||
`--toolbox` additionally creates the per-node LXC (see below). The
|
||
installer (`monitoring/node-install.sh`) is idempotent — safe to re-run.
|
||
|
||
## The toolbox container (vmid 950, `knelperf-toolbox`)
|
||
|
||
PVE-native LXC on each node — Debian 12, privileged, apparmor
|
||
unconfined, bind mounts for host cgroupfs and `/var/log` (ro), up to 8
|
||
block devices, and runtime-mounted tracefs/debugfs (pct skips
|
||
virtual-fs bind sources — the wrapper mounts them on start).
|
||
**Stopped at rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop,
|
||
sysbench, fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
|
||
|
||
```bash
|
||
knelperf-toolbox run 'iostat -x 1 5' # starts LXC if stopped
|
||
knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
|
||
knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06' # host sar, offline
|
||
knelperf-toolbox run 'smartctl -d sat -H /dev/sda' # SAT devices need -d sat; NVMe doesn't
|
||
knelperf-toolbox shell # interactive
|
||
knelperf-toolbox stop # back to zero footprint
|
||
```
|
||
|
||
Docker-capable hosts use the same manifest as an image:
|
||
`git.knownelement.com/knel/knelperf-toolbox:v1` (digest `sha256:3bb22bdb4a69707aadd0043d5151f863634a3ce5229f63d12773d5e3f90e001c`, from `monitoring/toolbox/Dockerfile`).
|
||
|
||
## Analysis recipes
|
||
|
||
```bash
|
||
sar -q -f /var/log/sysstat/sa06 # load vs runq
|
||
sar -d -p -f /var/log/sysstat/sa06 # per-device IO
|
||
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log # PSI trend
|
||
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log # io offenders
|
||
```
|
||
|
||
The `loop/perf-loop.sh` driver consumes these on its `audit` pass;
|
||
`baseline-diff.sh` compares before/after tweaks. Per-guest PSI
|
||
(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard
|
||
(ARCHITECTURE §3).
|
||
|
||
## Prod gating
|
||
|
||
Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout
|
||
exists and references this directory. Verification evidence goes back on
|
||
the CR; closing it is human-only.
|