plan for October hardware slip: presume 8 vCPU/48G, multiqueue bounce script
Hardware window #601 slipped to ~Oct 2026: interim fences and daynight values become the operating config, cpuset reservation defers with it, and the compose cgroup_parent wiring becomes the gateway starvation fix. Adds 5-activate-multiqueue.sh — the one remaining VM-5111 reboot (human-timed) to bring the staged net queues live. Tracking/README/crush prefs updated. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 [#602]
This commit is contained in:
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Activate the staged net multiqueue (queues=4/2 already in qm config) with
|
||||
# ONE graceful bounce of VM 5111. Run any time from this VM; the host-side
|
||||
# work is detached, so this script returns before the VM goes down.
|
||||
# WARNING: every crush session on this VM dies with the bounce (~4 min).
|
||||
set -euo pipefail
|
||||
echo "bouncing VM 5111 (detached host-side); ~4 min downtime"
|
||||
timeout 60 ssh root@pfv-tsys5.knel.net \
|
||||
'setsid bash -c "echo bounce-start \$(date -Is); qm shutdown 5111 --timeout 120; qm wait 5111 --timeout 180 || true; qm start 5111; echo bounce-done \$(date -Is)" </dev/null >/var/log/ukrrs-vm5111-bounce.log 2>&1 &'
|
||||
echo "dispatched. after reboot verify: ethtool -l ens18 -> Combined 4/4, ens19 -> 2/2"
|
||||
@@ -5,8 +5,10 @@
|
||||
- Host flags live: ssd=1/discard (ROTA=0), iothread, onboot=1, startup
|
||||
order=10, cpuunits 9000/50, ide2 gone.
|
||||
- Net multiqueue: queues=4/2 now IN live qm config (staged 16:40 by crush
|
||||
via ssh; activates at next VM start = Friday). Guest oneshot unit
|
||||
ukrrs-net-multiqueue auto-runs ethtool -L on that boot.
|
||||
via ssh); activates at the next VM start — ONE reboot still owed, human-
|
||||
timed: run ~/optimize/5-activate-multiqueue.sh whenever convenient
|
||||
(kills crush sessions, ~4 min). Guest oneshot ukrrs-net-multiqueue
|
||||
auto-runs ethtool -L on that boot.
|
||||
- Gateway boot-race + live-restore endpoint loss: permanently fixed by
|
||||
ukrrs-gateway-ensure unit (enabled, both lanes).
|
||||
- Root cause of the 15:59 no-op: the script copy executed then predates the
|
||||
@@ -34,5 +36,12 @@
|
||||
- mopac-demo/mcli fake containers: restart=no, exited at the first bounce,
|
||||
left down (test stubs; your call).
|
||||
- Prometheus/harness daemons not deployed here yet (that is OPT-9's wiring).
|
||||
- Friday unchanged: CPU/RAM swap (plan NFS outage for -02), 2nd USB3 card,
|
||||
q35 + 20 vCPU/128G reshape, GPUs. Enable the ensure unit BEFORE Friday.
|
||||
|
||||
## HARDWARE SLIPPED TO ~OCT 2026 (ruling 2026-08-31)
|
||||
- PRESUME 8 vCPU / 48G on this VM until the #601 window: interim fences
|
||||
(accounts 3G/4.5G) and the daynight interim values ARE the operating
|
||||
config; the commented post-upgrade block stays dormant.
|
||||
- Gateway cpuset hard reservation (16-17) waits for the window → the
|
||||
compose cgroup_parent wiring is THE starvation fix meanwhile (ticketed).
|
||||
- q35 / 20 vCPU / 128G / GPUs / USB3 reseat / NFS-outage-for--02 all move
|
||||
to the same October window (#601, #606).
|
||||
|
||||
@@ -5,9 +5,10 @@ pfv-tsys5). Everything here is human-run, gated, and reversible; storage ops
|
||||
are manual-only by ruling.
|
||||
|
||||
- Redmine: project 55 "Known Element Enterprises - Technology & Facility
|
||||
Services" — open work: #601 (Fri 2026-09-04 hardware window, urgent),
|
||||
#603 root growth, #604 metrics/PSI wiring, #605 k8s + proxmox token,
|
||||
#606 GPU passthrough, #607 account map/mkacct; pass record: #602.
|
||||
Services" — open work: #601 (hardware window ~Oct 2026; presume
|
||||
8 vCPU/48G until then), #603 root growth, #604 metrics/PSI wiring,
|
||||
#605 k8s + proxmox token (blocked by #601), #606 GPU passthrough,
|
||||
#607 account map/mkacct; pass record: #602.
|
||||
https://projects.knownelement.com/projects/55
|
||||
- Discourse doc: pending (house cross-link rule: create at next doc pass).
|
||||
- Docs live in-repo (runbook exception per house rules):
|
||||
|
||||
+3
-1
@@ -7,7 +7,7 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 /
|
||||
| OPT-1 | guest | ✅ | recon: system + ukrrs fleet (REPORT §1) |
|
||||
| OPT-2 | guest | ✅ | REPORT.md + staged configs + questions-v1.md |
|
||||
| OPT-3 | host | ✅ | audit parsed → REPORT §8 (T7500, CPU2 empty, clean IOMMU, Q4000 skip) |
|
||||
| OPT-4 | host | ⏳ | Fri 2026-09-04 #601: 2× X5675 + 6× 16GB → 24T/192G (NFS outage for -02 first; USB3 reseat; q35/20c/128G) |
|
||||
| OPT-4 | host | ⏳ | #601 ~OCT 2026: 2× X5675 + 6× 16GB → 24T/192G (NFS outage for -02 first; USB3 reseat; q35/20c/128G). PRESUME 8 vCPU/48G until then |
|
||||
| OPT-5 | host | ✅ | one-shot landed + verified post-reboot; net multiqueue now in qm config (live at Friday's start) |
|
||||
| OPT-6 | guest | ✅ | guest prep applied + verified (bbr, THP, slices, timers, noatime, daemon.json, autologin) |
|
||||
| OPT-7 | guest | ⏳ | #603: root growth MANUAL ONLY, GROW-ROOT-RUNBOOK.md, user-executed |
|
||||
@@ -18,6 +18,8 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 /
|
||||
Inbox (mid-task interrupts): none.
|
||||
|
||||
Decisions log (latest wins):
|
||||
- 2026-08-31 (eve): HARDWARE WINDOW → ~OCT 2026; presume 8 vCPU/48G operating values; cpuset reservation deferred → cgroup_parent wiring = the starvation fix; one VM-5111 reboot owed for multiqueue (5-activate-multiqueue.sh, human-timed; crush may roll OTHER VMs on pfv-tsys5 anytime).
|
||||
- 2026-08-31 (eve): PUSH AS YOU GO standing permission (human monitors gitea/redmine/discourse, not ssh/crush). House rules updated + pushed (86d8565).
|
||||
- 2026-08-31 16:5x: Redmine = record: perf pass + RCA filed as #602 (project 55, tracker Support, left New for human review); open items #601 (Fri window, Urgent, due 09-04) #603 root growth #604 metrics #605 k8s+proxmox (blocked by #601) #606 GPUs (relates #601). mred = docker exec wrapper (~/.local/bin/mred → mopac-cli-mred container).
|
||||
- 2026-08-31 16:40: net multiqueue CLOSED: queues=4/2 written to live qm config via crush ssh (user authorized non-disruptive host mods); activates at next VM start; guest oneshot enables channels on that boot. Earlier silent no-op explained: the 15:59 run used a pre-fix script copy.
|
||||
- 2026-08-31 16:3x: crush CAN ssh to the host from this workspace (old "hard-banned" note obsolete); still human-gated for disruptive ops.
|
||||
|
||||
@@ -14,6 +14,11 @@
|
||||
ALLOWED (old curl ban retired). NO installs on the host (apt-get/pip/etc).
|
||||
- Missing CLI tool? docker pull it, standing permission — run via
|
||||
`docker run --rm <pinned image>`. Work in containers.
|
||||
- PUSH AS YOU GO (ruling 2026-08-31): commit+push work as it lands; the
|
||||
human monitors gitea/redmine/discourse activity, not this host.
|
||||
- pfv-tsys5 (ruling 2026-08-31): crush may reboot OTHER VMs there anytime
|
||||
(qm); VM 5111 reboots are human-triggered (crush lives here); host never
|
||||
reboots casually; hardware window ~Oct 2026 — presume 8 vCPU / 48G.
|
||||
|
||||
## Output format (user preference, 2026-08-31)
|
||||
- The crush TUI sidebar truncates long chat output. NEVER hand the user a long
|
||||
|
||||
Reference in New Issue
Block a user