KNEL/PFVCluster
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agents): field lessons from #684 PBS rollout + restore-drill incident

Browse Source 
Meat: https://projects.knownelement.com/issues/684
This commit is contained in:
mrcharles
2026-09-03 08:33:58 -05:00
parent 7586605b70
commit bf52f85ac5
1 changed files with 12 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+12
View File
@@ -799,3 +799,15 @@ All work is tracked in [Redmine](https://projects.knownelement.com)
use `restart` to force an immediate pull. It auto-restarts core for use `restart` to force an immediate pull. It auto-restarts core for
files in its restart list; otherwise pull ≠ reload (config lands on files in its restart list; otherwise pull ≠ reload (config lands on
disk, running core keeps the old one until a platform reload/restart). disk, running core keeps the old one until a platform reload/restart).
- **Restore drills / heavy I/O NEVER on hosts running prod VMs** (2026-09-03
incident, #684): qmrestore of a 32G disk to scratch VMID 5599 on tsys1
starved VM 100 (pfv-bms) I/O ~40 min, degraded prod HA. Drills run dev-lane
only (tsys5); prod restores are server-throttled via PBS traffic-control.
- **`pkill -f <pat>` over the ssh chokepoint self-matches** your own remote
command line and kills the session (exit 255, no output). Use the bracket
trick: `pkill -f "[q]mrestore"`.
- **pvesh backup jobs take `-vmid`** (comma list), not `-vmids`; aborted
`qmrestore` leaves a create-locked stub VM — `qm unlock <id>` then destroy.
- **PBS 4.x gotchas:** prune retention lives in `prune-job` (datastore keep-*
opts are gone); API tokens need their OWN acl entry — user-only ACL yields
"Cannot find datastore" from PVE.