diff --git a/README.md b/README.md index 59b3f4b..c61e708 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,7 @@ flowchart LR | Module | What it does | Entry point | |---|---|---| +| `monitoring/` | fleet instrumentation: sysstat 28d tuning, perfsnap/PSI/iotop timers, on-demand perf toolbox container (LXC on PVE nodes, Docker image for the rest) | `monitoring/deploy-fleet.sh` | | `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline ` | | `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` | | `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` | @@ -76,6 +77,7 @@ KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh | [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths | | [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) | | [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward | +| [monitoring/README.md](monitoring/README.md) | what lands on each node, toolbox cookbook, analysis recipes, prod/CR gating | Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar) (solar economics + ONCOR arbitrage + ERCOT risk brief) · diff --git a/monitoring/README.md b/monitoring/README.md new file mode 100644 index 0000000..5dd6842 --- /dev/null +++ b/monitoring/README.md @@ -0,0 +1,76 @@ +# 📡 Fleet monitoring & the on-demand toolbox + +> **Redmine:** [#826](https://projects.knownelement.com/issues/826) · +> **Docs:** [Discourse](https://community.turnsys.com/t/328) · +> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine) + +Continuous lightweight instrumentation for the seven PFV PVE nodes +(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is +stopped at rest and runs on demand. Ultix's instrumentation pattern +(60s perfsnap + PSI collectors, containers for deep dives) generalized +fleet-wide. AGPLv3, like the whole repo. + +## What lands on a node + +| Piece | Cadence | Data | Where | +|---|---|---|---| +| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) | +| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` | +| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` | +| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` | + +No reboots, no package installs (sysstat + iotop ship with PVE 9), +no daemons — three oneshot systemd timers. Rollback = disable timers + +restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on +the hypervisor root. + +## Deploy + +From a tooling host with root SSH to the nodes: + +```bash +# non-prod first, prod only after the GLPI CR exists +bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9 +``` + +`--toolbox` additionally creates the per-node LXC (see below). The +installer (`monitoring/node-install.sh`) is idempotent — safe to re-run. + +## The toolbox container (vmid 950, `knelperf-toolbox`) + +PVE-native LXC on each node — Debian 12, privileged, apparmor +unconfined, bind mounts for host `/proc`, `/sys/fs/cgroup`, debugfs, +tracingfs, `/var/log` (ro) and up to 8 block devices. **Stopped at +rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop, sysbench, +fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc… + +```bash +knelperf-toolbox run 'iostat -x 1 5' # starts LXC if stopped +knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"' +knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06' # host sar, offline +knelperf-toolbox shell # interactive +knelperf-toolbox stop # back to zero footprint +``` + +Docker-capable hosts use the same manifest as an image: +`monitoring/toolbox/Dockerfile` → `git.knownelement.com/knel/knelperf-toolbox:v1`. + +## Analysis recipes + +```bash +sar -q -f /var/log/sysstat/sa06 # load vs runq +sar -d -p -f /var/log/sysstat/sa06 # per-device IO +awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log # PSI trend +grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log # io offenders +``` + +The `loop/perf-loop.sh` driver consumes these on its `audit` pass; +`baseline-diff.sh` compares before/after tweaks. Per-guest PSI +(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard +(ARCHITECTURE §3). + +## Prod gating + +Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout +exists and references this directory. Verification evidence goes back on +the CR; closing it is human-only. diff --git a/monitoring/collectors/knelperf-iotop-snap.sh b/monitoring/collectors/knelperf-iotop-snap.sh new file mode 100644 index 0000000..e038712 --- /dev/null +++ b/monitoring/collectors/knelperf-iotop-snap.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# KNELPerf iotop snapshot — periodic batch samples of active per-process I/O. +# Complements sar (system-wide device stats) with the WHO behind the I/O. +# Env (or /etc/knelperf/perfsnap.conf): +# KNELPERF_IOTOP_LOGDIR default /var/log/knelperf-iotop +# Output: daily file iotop-YYYYMMDD.log; per tick one header line then +# 3 batch iterations x 2s of processes with outstanding I/O (-botqq -k). +# Docs: https://community.turnsys.com/t/328 +set -u +CONF=/etc/knelperf/perfsnap.conf +[ -r "$CONF" ] && . "$CONF" +LOGDIR=${KNELPERF_IOTOP_LOGDIR:-/var/log/knelperf-iotop} +mkdir -p "$LOGDIR" +F="$LOGDIR/iotop-$(date +%Y%m%d).log" +{ + echo "### $(date -Iseconds)" + iotop -botqq -k -n 3 -d 2 2>/dev/null +} >> "$F" +# retention: 14 days; rotate the live file at 25MB +find "$LOGDIR" -name 'iotop-*.log*' -mtime +14 -delete 2>/dev/null +[ "$(wc -c <"$F" 2>/dev/null || echo 0)" -gt 26214400 ] && mv "$F" "$F.1" +exit 0 diff --git a/monitoring/deploy-fleet.sh b/monitoring/deploy-fleet.sh new file mode 100644 index 0000000..7fccbf2 --- /dev/null +++ b/monitoring/deploy-fleet.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Deploy KNELPerf node monitoring (+ toolbox LXC) to PVE fleet nodes (#826). +# Runs from the workstation / a tooling host — never on a node itself. +# deploy-fleet.sh [--toolbox] host [host...] +# Prod nodes (tsys4/5/6/7) deploy ONLY after their GLPI CR exists. +# Docs: https://community.turnsys.com/t/328 +set -euo pipefail +here=$(cd "$(dirname "$0")/.." && pwd) +WITH_TOOLBOX=0 +if [ "${1:-}" = --toolbox ]; then WITH_TOOLBOX=1; shift; fi +[ $# -ge 1 ] || { echo "usage: $0 [--toolbox] host..." >&2; exit 1; } + +for host in "$@"; do + echo "=== $host ===" + timeout 30 ssh -o ConnectTimeout=8 "root@$host" 'mkdir -p /tmp/knelperf-install' + tar -C "$here" -cf - scheduler/collectors monitoring \ + | timeout 120 ssh "root@$host" 'tar -C /tmp/knelperf-install -xf -' + timeout 120 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/node-install.sh' + if [ "$WITH_TOOLBOX" = 1 ]; then + timeout 600 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/toolbox/lxc-toolbox.sh --create' + fi + timeout 30 ssh "root@$host" 'rm -rf /tmp/knelperf-install' +done +echo "fleet deploy done." diff --git a/monitoring/node-install.sh b/monitoring/node-install.sh new file mode 100644 index 0000000..4f67e99 --- /dev/null +++ b/monitoring/node-install.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# KNELPerf node monitoring installer — sysstat tuning + collectors + timers. +# Idempotent; run as ROOT on a PVE node from a repo checkout: +# bash monitoring/node-install.sh +# Surface (all details in monitoring/README.md): +# - /etc/sysstat/sysstat HISTORY 7 -> 28, COMPRESSAFTER 10 -> 31 +# - /usr/local/sbin/knelperf-{perfsnap,psi-textfile,iotop-snap}.sh +# - /etc/systemd/system/knelperf-*.{service,timer} (enabled timers) +# - /etc/knelperf/perfsnap.conf per-node ifaces/cgroup globs +# - logs under /var/log/knelperf-{perf,iotop}, textfiles /var/lib/knelperf/textfile +# No reboots, no package installs (PVE ships sysstat + iotop already). +# Docs: https://community.turnsys.com/t/328 +set -euo pipefail +[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; } +here=$(cd "$(dirname "$0")/.." && pwd) +say() { echo "[knelperf-install] $*"; } + +# 1) sysstat: keep 28 days of sar history (two-week program + re-baseline window) +if [ -f /etc/sysstat/sysstat ]; then + sed -i -E 's/^HISTORY=.*/HISTORY=28/; s/^COMPRESSAFTER=.*/COMPRESSAFTER=31/' /etc/sysstat/sysstat + sed -i -E 's/^ENABLED=.*/ENABLED="true"/' /etc/default/sysstat + systemctl restart sysstat 2>/dev/null || true + /usr/libexec/sysstat/debian-sa1 1 1 2>/dev/null || true + today=/var/log/sysstat/sa$(date +%d) + if [ -f "$today" ]; then say "sar ok: $today present"; else say "sar WARN: $today missing (check /etc/cron.d/sysstat)"; fi +else + say "WARN: no /etc/sysstat/sysstat on this host — sar tuning skipped" +fi + +# 2) collectors + units +install -m 0755 "$here/scheduler/collectors/knelperf-perfsnap.sh" /usr/local/sbin/knelperf-perfsnap.sh +install -m 0755 "$here/scheduler/collectors/knelperf-psi-textfile.sh" /usr/local/sbin/knelperf-psi-textfile.sh +install -m 0755 "$here/monitoring/collectors/knelperf-iotop-snap.sh" /usr/local/sbin/knelperf-iotop-snap.sh +for u in perfsnap psi-textfile iotop-snap; do + install -m 0644 "$here/monitoring/systemd/knelperf-$u.service" /etc/systemd/system/knelperf-$u.service + install -m 0644 "$here/monitoring/systemd/knelperf-$u.timer" /etc/systemd/system/knelperf-$u.timer +done + +# 3) per-node conf: watch PVE guest cgroups + real bridges/tailscale +ifaces="" +for i in vmbr0 datanet tailscale0; do + ip link show "$i" >/dev/null 2>&1 && ifaces="$ifaces $i" +done +mkdir -p /etc/knelperf +cat > /etc/knelperf/perfsnap.conf </dev/null 2>&1 +done +/usr/local/sbin/knelperf-perfsnap.sh +/usr/local/sbin/knelperf-psi-textfile.sh +/usr/local/sbin/knelperf-iotop-snap.sh +say "timers enabled; first ticks logged" + +# 5) evidence +say "--- timers ---" +systemctl list-timers 'knelperf-*' --no-pager | sed -n '1,5p' +say "--- snap.log tail ---" +tail -1 /var/log/knelperf-perf/snap.log +say "--- textfile ---" +head -3 /var/lib/knelperf/textfile/knelperf_pressure.prom 2>/dev/null || true +say "done." diff --git a/monitoring/systemd/knelperf-iotop-snap.service b/monitoring/systemd/knelperf-iotop-snap.service new file mode 100644 index 0000000..f652cfa --- /dev/null +++ b/monitoring/systemd/knelperf-iotop-snap.service @@ -0,0 +1,7 @@ +[Unit] +Description=KNELPerf iotop snapshot (3x2s active-I/O samples) + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/knelperf-iotop-snap.sh +IOSchedulingClass=idle diff --git a/monitoring/systemd/knelperf-iotop-snap.timer b/monitoring/systemd/knelperf-iotop-snap.timer new file mode 100644 index 0000000..cc7f565 --- /dev/null +++ b/monitoring/systemd/knelperf-iotop-snap.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Run knelperf-iotop-snap every 15 minutes + +[Timer] +OnCalendar=*:0/15 +Persistent=true +RandomizedDelaySec=30 + +[Install] +WantedBy=timers.target diff --git a/monitoring/systemd/knelperf-perfsnap.service b/monitoring/systemd/knelperf-perfsnap.service new file mode 100644 index 0000000..050288e --- /dev/null +++ b/monitoring/systemd/knelperf-perfsnap.service @@ -0,0 +1,6 @@ +[Unit] +Description=KNELPerf perfsnap collector tick (/proc+cgroupfs snapshot) + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/knelperf-perfsnap.sh diff --git a/monitoring/systemd/knelperf-perfsnap.timer b/monitoring/systemd/knelperf-perfsnap.timer new file mode 100644 index 0000000..28c1918 --- /dev/null +++ b/monitoring/systemd/knelperf-perfsnap.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Run knelperf-perfsnap every 60s + +[Timer] +OnBootSec=45 +OnUnitActiveSec=60 +AccuracySec=5s +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/monitoring/systemd/knelperf-psi-textfile.service b/monitoring/systemd/knelperf-psi-textfile.service new file mode 100644 index 0000000..095eef7 --- /dev/null +++ b/monitoring/systemd/knelperf-psi-textfile.service @@ -0,0 +1,6 @@ +[Unit] +Description=KNELPerf PSI textfile collector tick (pressure avg60 gauges) + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/knelperf-psi-textfile.sh diff --git a/monitoring/systemd/knelperf-psi-textfile.timer b/monitoring/systemd/knelperf-psi-textfile.timer new file mode 100644 index 0000000..a1c29f7 --- /dev/null +++ b/monitoring/systemd/knelperf-psi-textfile.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Run knelperf-psi-textfile every 15s + +[Timer] +OnBootSec=20 +OnUnitActiveSec=15 +AccuracySec=2s +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/monitoring/toolbox/Dockerfile b/monitoring/toolbox/Dockerfile new file mode 100644 index 0000000..266001b --- /dev/null +++ b/monitoring/toolbox/Dockerfile @@ -0,0 +1,16 @@ +# knelperf-toolbox — Docker image variant for docker-capable hosts +# (workstation, VMs). On PVE nodes the LXC variant (lxc-toolbox.sh) is +# canonical. Package list is shared: toolbox-pkgs.txt. +# Build: docker build -t git.knownelement.com/knel/knelperf-toolbox:v1 \ +# -f monitoring/toolbox/Dockerfile monitoring/toolbox/ +# Docs: https://community.turnsys.com/t/328 +FROM debian:12.9-slim + +COPY toolbox-pkgs.txt /tmp/toolbox-pkgs.txt +RUN apt-get update -qq && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + $(sed 's/#.*//' /tmp/toolbox-pkgs.txt | tr '\n' ' ') && \ + rm -rf /var/lib/apt/lists/* /tmp/toolbox-pkgs.txt + +WORKDIR /work +ENTRYPOINT ["/bin/bash"] diff --git a/monitoring/toolbox/knelperf-toolbox b/monitoring/toolbox/knelperf-toolbox new file mode 100644 index 0000000..a85fd4b --- /dev/null +++ b/monitoring/toolbox/knelperf-toolbox @@ -0,0 +1,27 @@ +#!/bin/sh +# knelerf-toolbox — front door to the on-demand perf toolbox LXC on this node. +# usage: knelperf-toolbox run '' (starts the LXC if stopped, runs, leaves it up) +# knelperf-toolbox shell (interactive) +# knelperf-toolbox start|stop|status +# The LXC (vmid 950) is STOPPED at rest — zero footprint; stop it when done. +# Docs: https://community.turnsys.com/t/328 +VMID=${KNELPERF_TOOLBOX_VMID:-950} + +ensure_running() { + pct status "$VMID" 2>/dev/null | grep -q running && return 0 + pct start "$VMID" + i=0 + until pct exec "$VMID" -- true 2>/dev/null; do + i=$((i + 1)); [ $i -gt 45 ] && { echo "boot timeout" >&2; return 1; } + sleep 2 + done +} + +case "${1:-help}" in + run) shift; ensure_running && pct exec "$VMID" -- sh -c "$*" ;; + shell) ensure_running && pct enter "$VMID" ;; + start) ensure_running ;; + stop) pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" ;; + status) pct status "$VMID" 2>/dev/null || echo "vmid $VMID not present" ;; + *) sed -n '2,6p' "$0" | sed 's/^# \?//' ;; +esac diff --git a/monitoring/toolbox/lxc-toolbox.sh b/monitoring/toolbox/lxc-toolbox.sh new file mode 100644 index 0000000..e2abf39 --- /dev/null +++ b/monitoring/toolbox/lxc-toolbox.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# knelperf-toolbox LXC — the on-demand perf analysis container for a PVE node. +# STOPS to zero footprint; started only when a human/agent runs a workload. +# lxc-toolbox.sh [--vmid N] {--create|--upgrade|--remove|--facts} +# Why privileged + apparmor unconfined: bpftrace/perf need host kernel BPF, +# tracefs and host /proc. Bind mounts: /host/proc, /host/cgroup, host debugfs +# + tracing, /host/var/log (ro) for offline sar analysis, block devs for +# smart/nvme. Package list lives in toolbox-pkgs.txt next to this script. +# Docs: https://community.turnsys.com/t/328 +set -euo pipefail +[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; } +here=$(cd "$(dirname "$0")" && pwd) +VMID=950 +TPL=debian-12-standard_12.12-1_amd64.tar.zst +case "${1:-}" in + --vmid) VMID=$2; shift 2 ;; +esac +case "${1:-}" in + --facts) + pct config "$VMID" 2>/dev/null || echo "vmid $VMID not present" + pct status "$VMID" 2>/dev/null || true + exit 0 ;; + --remove) + pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" 2>/dev/null || true + pct destroy "$VMID" --purge 1 + rm -f /usr/local/sbin/knelperf-toolbox + echo "toolbox $VMID removed" + exit 0 ;; + --create|--upgrade) action=$1 ;; + *) sed -n 's/^# \?//p' "$0" | sed -n '2,5p'; exit 1 ;; +esac + +if [ "$action" = --create ]; then + if pct status "$VMID" >/dev/null 2>&1; then + echo "vmid $VMID already present — use --upgrade" >&2; exit 1 + fi + pveam list local 2>/dev/null | grep -q "$TPL" || pveam download local "$TPL" + # block-device passthrough for smartctl/nvme (first 8 devices) + dev_args=() + for d in /dev/nvme[0-9]n1 /dev/sd[a-z]; do + [ -b "$d" ] || continue + [ "${#dev_args[@]}" -ge 16 ] && break + dev_args+=("--dev$(( ${#dev_args[@]} / 2 ))" "$d") + done + pct create "$VMID" "local:vztmpl/$TPL" \ + --hostname knelperf-toolbox \ + --ostype debian \ + --cores 2 --memory 2048 --swap 512 \ + --rootfs local-lvm:8 \ + --net0 name=eth0,bridge=vmbr0,ip=dhcp \ + --unprivileged 0 --onboot 0 \ + --mp0 /proc,mp=/host/proc \ + --mp1 /sys/kernel/debug,mp=/sys/kernel/debug \ + --mp2 /sys/kernel/tracing,mp=/sys/kernel/tracing \ + --mp3 /sys/fs/cgroup,mp=/host/cgroup \ + --mp4 /var/log,mp=/host/var/log,ro=1 \ + "${dev_args[@]}" + pct set "$VMID" --raw.lxc 'lxc.apparmor.profile=unconfined' +fi + +# --create continues here; --upgrade rejoins an existing container +was_running=1 +pct status "$VMID" 2>/dev/null | grep -q running || { was_running=0; pct start "$VMID"; } +i=0 +until pct exec "$VMID" -- true 2>/dev/null; do + i=$((i + 1)); [ $i -gt 60 ] && { echo "boot timeout" >&2; exit 1; } + sleep 2 +done +pkgs=$(sed 's/#.*//' "$here/toolbox-pkgs.txt" | tr '\n' ' ') +pct exec "$VMID" -- bash -c \ + "DEBIAN_FRONTEND=noninteractive apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install $pkgs" +install -m 0755 "$here/knelperf-toolbox" /usr/local/sbin/knelperf-toolbox +[ "$was_running" = 0 ] && pct shutdown "$VMID" --timeout 60 +echo "toolbox $VMID $action done: knelperf-toolbox wrapper installed" +pct status "$VMID" diff --git a/monitoring/toolbox/toolbox-pkgs.txt b/monitoring/toolbox/toolbox-pkgs.txt new file mode 100644 index 0000000..68f0cd6 --- /dev/null +++ b/monitoring/toolbox/toolbox-pkgs.txt @@ -0,0 +1,37 @@ +# knelperf-toolbox package manifest — shared by the LXC bootstrap and the +# Docker image variant. One package (or apt argument) per line; # = comment. +# Docs: https://community.turnsys.com/t/328 +bpftrace +bcc-tools +sysstat +iotop +sysbench +fio +iperf3 +stress-ng +linux-perf +jq +bc +curl +procps +psmisc +util-linux +pciutils +usbutils +ethtool +net-tools +bind9-dnsutils +smartmontools +nvme-cli +hdparm +lm-sensors +numactl +hwloc-nox +strace +ltrace +file +rsync +openssh-client +vim-tiny +less +tmux diff --git a/scheduler/collectors/knelperf-psi-textfile.sh b/scheduler/collectors/knelperf-psi-textfile.sh index 716c3b3..fe70c2e 100755 --- a/scheduler/collectors/knelperf-psi-textfile.sh +++ b/scheduler/collectors/knelperf-psi-textfile.sh @@ -1,9 +1,11 @@ #!/usr/bin/env bash # KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector. # Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations: -# - cgroup glob is env-driven (any slice namespace, not just ukrrs-*) +# - cgroup globs are env-driven (space-separated LIST, any slice namespace) # - metric prefix is env-driven (default knelperf_) set -u +CONF=/etc/knelperf/perfsnap.conf +[ -r "$CONF" ] && . "$CONF" OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile} CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup} CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice} @@ -27,6 +29,7 @@ emit() { # name cgroup_path echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp" echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp" for cg in $CG_GLOB; do + case $cg in "$CGROOT"/*) ;; *) continue ;; esac # skip unmatched glob literals [ -d "$cg" ] && emit "${cg##*/}" "$cg" done [ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice"