ic-builder
2d2aa3ef38
- PVE 9 dropped 'pct set --raw.lxc': lxc.apparmor.profile goes straight into /etc/pve/lxc/<vmid>.conf and PVE::LXC honors it over the generated profile (verified: /proc/self/attr/current = unconfined) - pct skips virtual-fs bind sources (mp /proc,/sys/kernel/* landed empty): keep cgroupfs + /var/log binds, wrapper mounts tracefs/debugfs at start - bpftrace PROVEN against host kernel (2689 tracepoints, live syscall counts showing host pvestatd/vgs); bcc-tools -> bpfcc-tools (bookworm) - smartctl on SAT devices needs -d sat (recipes updated) Redmine: https://projects.knownelement.com/issues/826
3.6 KiB
📡 Fleet monitoring & the on-demand toolbox
Redmine: #826 · Docs: Discourse · Program:
docs/ARCHITECTURE.md(§5 telemetry spine)
Continuous lightweight instrumentation for the seven PFV PVE nodes (tsys1/3/4/5/6/7/9) plus a per-node perf toolbox container that is stopped at rest and runs on demand. Ultix's instrumentation pattern (60s perfsnap + PSI collectors, containers for deep dives) generalized fleet-wide. AGPLv3, like the whole repo.
What lands on a node
| Piece | Cadence | Data | Where |
|---|---|---|---|
sysstat sadc (ships with PVE, tuned here) |
10 min + daily rollup | sar: CPU, mem, disks, net, load | /var/log/sysstat/saDD (HISTORY=28) |
knelperf-perfsnap.sh |
60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | /var/log/knelperf-perf/{snap,procs}.log |
knelperf-psi-textfile.sh |
15 s | PSI some60 per guest slice + host (Prometheus textfile format) | /var/lib/knelperf/textfile/knelperf_pressure.prom |
knelperf-iotop-snap.sh |
15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | /var/log/knelperf-iotop/iotop-YYYYMMDD.log |
No reboots, no package installs (sysstat + iotop ship with PVE 9),
no daemons — three oneshot systemd timers. Rollback = disable timers +
restore HISTORY=7. bpftrace/BCC live inside the toolbox, never on
the hypervisor root.
Deploy
From a tooling host with root SSH to the nodes:
# non-prod first, prod only after the GLPI CR exists
bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
--toolbox additionally creates the per-node LXC (see below). The
installer (monitoring/node-install.sh) is idempotent — safe to re-run.
The toolbox container (vmid 950, knelperf-toolbox)
PVE-native LXC on each node — Debian 12, privileged, apparmor
unconfined, bind mounts for host cgroupfs and /var/log (ro), up to 8
block devices, and runtime-mounted tracefs/debugfs (pct skips
virtual-fs bind sources — the wrapper mounts them on start).
Stopped at rest; onboot=0. Tools: bpftrace, BCC, sysstat, iotop,
sysbench, fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
knelerf-toolbox run 'iostat -x 1 5' # starts LXC if stopped
knelerf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
knelerf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06' # host sar, offline
knelerf-toolbox run 'smartctl -d sat -H /dev/sda' # SAT devices need -d sat; NVMe doesn't
knelerf-toolbox shell # interactive
knelerf-toolbox stop # back to zero footprint
Docker-capable hosts use the same manifest as an image:
monitoring/toolbox/Dockerfile → git.knownelement.com/knel/knelperf-toolbox:v1.
Analysis recipes
sar -q -f /var/log/sysstat/sa06 # load vs runq
sar -d -p -f /var/log/sysstat/sa06 # per-device IO
awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log # PSI trend
grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log # io offenders
The loop/perf-loop.sh driver consumes these on its audit pass;
baseline-diff.sh compares before/after tweaks. Per-guest PSI
(qemu.slice/*.scope, lxc/*) feeds the placement scorecard
(ARCHITECTURE §3).
Prod gating
Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout exists and references this directory. Verification evidence goes back on the CR; closing it is human-only.