From acd7242604cfa0f5ce78a5d257a6cc2ce9a3ecc1 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Mon, 31 Aug 2026 16:55:38 -0500 Subject: [PATCH] plan for October hardware slip: presume 8 vCPU/48G, multiqueue bounce script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hardware window #601 slipped to ~Oct 2026: interim fences and daynight values become the operating config, cpuset reservation defers with it, and the compose cgroup_parent wiring becomes the gateway starvation fix. Adds 5-activate-multiqueue.sh โ€” the one remaining VM-5111 reboot (human-timed) to bring the staged net queues live. Tracking/README/crush prefs updated. ๐Ÿ’˜ Generated with Crush Assisted-by: Crush:glm-5.2 [#602] --- 5-activate-multiqueue.sh | 10 ++++++++++ NEXT.md | 17 +++++++++++++---- README.md | 7 ++++--- TRACKING.md | 4 +++- crush.md | 5 +++++ 5 files changed, 35 insertions(+), 8 deletions(-) create mode 100755 5-activate-multiqueue.sh diff --git a/5-activate-multiqueue.sh b/5-activate-multiqueue.sh new file mode 100755 index 0000000..ce40aef --- /dev/null +++ b/5-activate-multiqueue.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Activate the staged net multiqueue (queues=4/2 already in qm config) with +# ONE graceful bounce of VM 5111. Run any time from this VM; the host-side +# work is detached, so this script returns before the VM goes down. +# WARNING: every crush session on this VM dies with the bounce (~4 min). +set -euo pipefail +echo "bouncing VM 5111 (detached host-side); ~4 min downtime" +timeout 60 ssh root@pfv-tsys5.knel.net \ + 'setsid bash -c "echo bounce-start \$(date -Is); qm shutdown 5111 --timeout 120; qm wait 5111 --timeout 180 || true; qm start 5111; echo bounce-done \$(date -Is)" /var/log/ukrrs-vm5111-bounce.log 2>&1 &' +echo "dispatched. after reboot verify: ethtool -l ens18 -> Combined 4/4, ens19 -> 2/2" diff --git a/NEXT.md b/NEXT.md index bb3ac8e..148ed10 100644 --- a/NEXT.md +++ b/NEXT.md @@ -5,8 +5,10 @@ - Host flags live: ssd=1/discard (ROTA=0), iothread, onboot=1, startup order=10, cpuunits 9000/50, ide2 gone. - Net multiqueue: queues=4/2 now IN live qm config (staged 16:40 by crush - via ssh; activates at next VM start = Friday). Guest oneshot unit - ukrrs-net-multiqueue auto-runs ethtool -L on that boot. + via ssh); activates at the next VM start โ€” ONE reboot still owed, human- + timed: run ~/optimize/5-activate-multiqueue.sh whenever convenient + (kills crush sessions, ~4 min). Guest oneshot ukrrs-net-multiqueue + auto-runs ethtool -L on that boot. - Gateway boot-race + live-restore endpoint loss: permanently fixed by ukrrs-gateway-ensure unit (enabled, both lanes). - Root cause of the 15:59 no-op: the script copy executed then predates the @@ -34,5 +36,12 @@ - mopac-demo/mcli fake containers: restart=no, exited at the first bounce, left down (test stubs; your call). - Prometheus/harness daemons not deployed here yet (that is OPT-9's wiring). -- Friday unchanged: CPU/RAM swap (plan NFS outage for -02), 2nd USB3 card, - q35 + 20 vCPU/128G reshape, GPUs. Enable the ensure unit BEFORE Friday. + +## HARDWARE SLIPPED TO ~OCT 2026 (ruling 2026-08-31) +- PRESUME 8 vCPU / 48G on this VM until the #601 window: interim fences + (accounts 3G/4.5G) and the daynight interim values ARE the operating + config; the commented post-upgrade block stays dormant. +- Gateway cpuset hard reservation (16-17) waits for the window โ†’ the + compose cgroup_parent wiring is THE starvation fix meanwhile (ticketed). +- q35 / 20 vCPU / 128G / GPUs / USB3 reseat / NFS-outage-for--02 all move + to the same October window (#601, #606). diff --git a/README.md b/README.md index 977ae03..87d0531 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ pfv-tsys5). Everything here is human-run, gated, and reversible; storage ops are manual-only by ruling. - Redmine: project 55 "Known Element Enterprises - Technology & Facility - Services" โ€” open work: #601 (Fri 2026-09-04 hardware window, urgent), - #603 root growth, #604 metrics/PSI wiring, #605 k8s + proxmox token, - #606 GPU passthrough, #607 account map/mkacct; pass record: #602. + Services" โ€” open work: #601 (hardware window ~Oct 2026; presume + 8 vCPU/48G until then), #603 root growth, #604 metrics/PSI wiring, + #605 k8s + proxmox token (blocked by #601), #606 GPU passthrough, + #607 account map/mkacct; pass record: #602. https://projects.knownelement.com/projects/55 - Discourse doc: pending (house cross-link rule: create at next doc pass). - Docs live in-repo (runbook exception per house rules): diff --git a/TRACKING.md b/TRACKING.md index 96787a7..9566c05 100644 --- a/TRACKING.md +++ b/TRACKING.md @@ -7,7 +7,7 @@ One table, updated in place at each checkpoint. States: done โœ… / doing ๐Ÿ”„ / | OPT-1 | guest | โœ… | recon: system + ukrrs fleet (REPORT ยง1) | | OPT-2 | guest | โœ… | REPORT.md + staged configs + questions-v1.md | | OPT-3 | host | โœ… | audit parsed โ†’ REPORT ยง8 (T7500, CPU2 empty, clean IOMMU, Q4000 skip) | -| OPT-4 | host | โณ | Fri 2026-09-04 #601: 2ร— X5675 + 6ร— 16GB โ†’ 24T/192G (NFS outage for -02 first; USB3 reseat; q35/20c/128G) | +| OPT-4 | host | โณ | #601 ~OCT 2026: 2ร— X5675 + 6ร— 16GB โ†’ 24T/192G (NFS outage for -02 first; USB3 reseat; q35/20c/128G). PRESUME 8 vCPU/48G until then | | OPT-5 | host | โœ… | one-shot landed + verified post-reboot; net multiqueue now in qm config (live at Friday's start) | | OPT-6 | guest | โœ… | guest prep applied + verified (bbr, THP, slices, timers, noatime, daemon.json, autologin) | | OPT-7 | guest | โณ | #603: root growth MANUAL ONLY, GROW-ROOT-RUNBOOK.md, user-executed | @@ -18,6 +18,8 @@ One table, updated in place at each checkpoint. States: done โœ… / doing ๐Ÿ”„ / Inbox (mid-task interrupts): none. Decisions log (latest wins): +- 2026-08-31 (eve): HARDWARE WINDOW โ†’ ~OCT 2026; presume 8 vCPU/48G operating values; cpuset reservation deferred โ†’ cgroup_parent wiring = the starvation fix; one VM-5111 reboot owed for multiqueue (5-activate-multiqueue.sh, human-timed; crush may roll OTHER VMs on pfv-tsys5 anytime). +- 2026-08-31 (eve): PUSH AS YOU GO standing permission (human monitors gitea/redmine/discourse, not ssh/crush). House rules updated + pushed (86d8565). - 2026-08-31 16:5x: Redmine = record: perf pass + RCA filed as #602 (project 55, tracker Support, left New for human review); open items #601 (Fri window, Urgent, due 09-04) #603 root growth #604 metrics #605 k8s+proxmox (blocked by #601) #606 GPUs (relates #601). mred = docker exec wrapper (~/.local/bin/mred โ†’ mopac-cli-mred container). - 2026-08-31 16:40: net multiqueue CLOSED: queues=4/2 written to live qm config via crush ssh (user authorized non-disruptive host mods); activates at next VM start; guest oneshot enables channels on that boot. Earlier silent no-op explained: the 15:59 run used a pre-fix script copy. - 2026-08-31 16:3x: crush CAN ssh to the host from this workspace (old "hard-banned" note obsolete); still human-gated for disruptive ops. diff --git a/crush.md b/crush.md index 6497772..2df66aa 100644 --- a/crush.md +++ b/crush.md @@ -14,6 +14,11 @@ ALLOWED (old curl ban retired). NO installs on the host (apt-get/pip/etc). - Missing CLI tool? docker pull it, standing permission โ€” run via `docker run --rm `. Work in containers. +- PUSH AS YOU GO (ruling 2026-08-31): commit+push work as it lands; the + human monitors gitea/redmine/discourse activity, not this host. +- pfv-tsys5 (ruling 2026-08-31): crush may reboot OTHER VMs there anytime + (qm); VM 5111 reboots are human-triggered (crush lives here); host never + reboots casually; hardware window ~Oct 2026 โ€” presume 8 vCPU / 48G. ## Output format (user preference, 2026-08-31) - The crush TUI sidebar truncates long chat output. NEVER hand the user a long