KNEL/PFVCluster
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ad69416a190ef4da93bfa5e5528bf61bfc8f68ab
PFVCluster/AGENTS.md
T

858 lines
47 KiB
Markdown
Raw Blame History

# Agent Guidelines
## Agent Authority (NON-NEGOTIABLE)
**No work is permissible on any system without an approved Redmine ticket.
There are no exceptions to this rule.**
### Scope of authority
1. **Tickets govern all work.** The agent performs ONLY the work described in
the approved Redmine ticket. Anything outside that scope — no matter how
small, helpful, or "obvious" — is prohibited.
2. **No autonomous system changes.** The agent does not modify, configure,
create, delete, or grant anything on a production system unless it is
explicitly directed by an approved ticket. "Production system" means
every system in the fleet — there is no "test" exception unless the ticket
says so.
3. **Propose, never implement.** If the agent discovers additional work that
should be done — a bug, a misconfiguration, a missing dependency, an
enhancement — it does NOT implement it. Instead, it creates a Redmine
sub-ticket (status Feedback) describing the finding and surfaces it to
the user for approval.
4. **Security and access changes require extra scrutiny.** Changes to sudoers,
SSH keys, user accounts, firewall rules, authentication policy, file
permissions, or any privilege-related configuration are treated as
policy decisions, not implementation details. The agent may suggest
them but NEVER implements them without explicit user direction in the
ticket or a sub-ticket the user has approved.
5. **The user makes policy. The agent implements policy.** The agent does
not decide who gets sudo, what keys go where, what services run, or what
the access model is. The agent executes the user's decisions, exactly
as specified.
6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
require something not explicitly authorized, or if the agent is unsure
whether an action is in scope — STOP and ask the user via the ticket
or directly. Asking is always acceptable. Overstepping is never
acceptable.
7. **NEVER close a ticket without explicit user permission.** You may
SUGGEST a close when the result is clearly scoped and delivered. If
it's ambiguous whether the work is truly complete, don't suggest a
close — leave that decision to the user. This applies to ALL tickets,
no exceptions.
8. **User acceptance testing is MANDATORY before declaring work done.**
The agent performs implementation and technical validation (services
running, configs correct, APIs responding). The user performs UAT —
visually confirming dashboards render data, alerts deliver, tools are
usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
closing, and MUST NOT move to the next ticket until the user explicitly
accepts the work. "Technically wired but blank dashboard" is NOT done.
9. **NEVER access a database directly if an API exists.** APIs are the
stable contract; databases change schemas without warning. If a tool
has an API, use it — exhaust all API endpoints, check the docs, try
alternative methods. Only fall back to direct DB access as a last
resort AND with explicit user approval for that specific instance.
Soon all DB access will route through a proxy under zero trust;
building API-first habits now ensures that transition is clean.
### Access-channel policy: SSH only (NON-NEGOTIABLE)
**The qemu guest-agent is NEVER an access, execution, or key-delivery
channel.** SSH (`sshd`) is the only approved remote access path. Every
command must flow through sshd so it is captured by the standard
auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
environment — there is no back-door exception, ever.
- **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
mode). This includes using guest-agent to *deliver* an SSH key, even if
the resulting SSH login is itself audited — the delivery bypassed audit.
- **Allowed:** installing or checking qemu-guest-agent for its intended
purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
`agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
never execution.
- **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
no guest-exec mode.
- **If a system is locked out** (no SSH key, no guest-exec path): surface
it to the user. Do NOT improvise an alternate back-channel. The user
authorizes the unblock method (console login, credential, etc.).
### Ticket-closing policy (NON-NEGOTIABLE)
**NEVER close a Redmine ticket without explicit user permission.** You
may SUGGEST a close when the result is clearly scoped and delivered. If
it's ambiguous whether the work is truly complete, don't suggest a close
— leave that decision to the user. This applies to ALL tickets, no
exceptions.
### What this means in practice
- Discovered a typo in a config during approved work? **Finish the approved
work. Create a sub-ticket for the typo. Do not fix it inline.**
- Think a system should also have localuser sudo configured? **Do not add
it. Propose it in a sub-ticket.**
- Need to install a package the ticket didn't mention? **Ask first.**
- Found a security issue? **Create a ticket immediately with full details.
Do not remediate without approval.**
This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
traceable to an approved ticket. There is no "I thought it would help."
## Quick Start
**You are an AI agent working on this project. Your first actions, in order:**
> **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
> check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
> the scope-alignment gate. You MUST NOT begin any task work until the user
> (a) confirms your rule summary AND (b) names the ticket to work on. This runs
> **every session, automatically** — the user should never have to ask for it.
1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
back to the user before doing any work: Agent Authority (ticket-governed,
propose-never-implement), Access-Channel SSH-only policy, Remote access
(remote.sh chokepoint mandatory, DNS names only — never IP literals),
Questions policy (no harness question-tools), Documentation policy
(Discourse is SoR), Redmine tracking, Git policy (always commit+push,
shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
routing chokepoints (`tests/remote.sh`), the mechanically enforced
`check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
and wait for the user to confirm. This checkpoint guarantees every
session starts aligned.
5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
6. **Check current state:** `git log --oneline -10`.
7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
priority list (or the Redmine queue if no handoff exists). Do NOT scan the
full ticket queue and pick work on your own — **the user directs what gets
worked on, always.** Note any new or urgent items from the Redmine check,
then ask which ticket to work on tonight. **STOP and wait.** Do not set
`.crush/active-ticket`, do not read systems, do not run diagnostics, until
the user names the target. This gate prevents the agent from burning
context on work the user didn't ask for.
## Enforcement Model
Git hooks (`scripts/pre-commit`, `scripts/pre-push`) enforce the rules defined in
`scripts/check-rules.sh`. The rules engine checks: shellcheck (zero warnings
including info-level), Docker image pinning (no `:latest`), container naming,
required files, Discourse pointer headers, and more. Run `bash scripts/check-rules.sh`
for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
(emergencies only).
## Task Tracking
- **Redmine is the system of record for all work.**
- **NEVER close a ticket without explicit user permission.** Suggest a
close when clearly scoped/delivered; if ambiguous, don't suggest.
- **Ticket-first enforcement (mechanically enforced).** Before starting
any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
until this file exists. If no ticket exists, CREATE ONE FIRST via
redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
**Parallel sessions:** each session uses its OWN suffixed file (e.g.
`.crush/active-ticket-plant`, `.crush/active-ticket-core`) so concurrent
sessions never clobber each other's ticket; any non-empty
`.crush/active-ticket*` satisfies the gate.
- **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
session. The pre-commit hook blocks commits while any task remains unchecked.
- **Parallel sessions:** when two sessions share this repo (e.g. physical plant
vs core infra), the second session uses a gitignored scratch tracker such as
`.crush/WORKING-PLANT.md` instead of WORKING.md, so the two trackers (and
their pre-commit gates) never collide. Wipe at session close.
- Clear WORKING.md before responding to the user.
## Rolling HUD (session-scoped, NOT persisted)
The rolling HUD is a **live status display** the agent maintains throughout
the session to help the human follow along. It is NOT a system of record —
Redmine, Discourse, and git are the durable systems. The HUD exists purely
for the human's situational awareness during the session.
- **Format:** a compact block shown at the end of each significant response
(after completing a step, hitting a blocker, or pivoting). Example:
```
┌─ SESSION HUD ────────────────────────────────────────
│ Active: #343 (Monitoring coverage matrix)
│ Done: ✓ scope-alignment gate added to both AGENTS.md
│ ✓ committed + pushed (6a2550b)
│ Now: drafting coverage matrix on Discourse
│ Next: → #341 TEMPer USB (deploy on pfv-tsys1)
│ → #338 LibreNMS alerts (BLOCKED: pushover.env)
├─ USER ACTION ITEMS ──────────────────────────────────
│ • Populate ~/.creds/pushover.env (unblocks #338, #428)
│ • Populate ~/.creds/prometheus.env + grafana.env (#430)
└──────────────────────────────────────────────────────
```
- **Placement:** may be written to `.crush/hud.md` on disk to keep context
window smaller (re-read and update rather than hold in memory). Never
committed to git. Wiped at session end.
- **Sections:**
- **Active:** current ticket number + one-line description
- **Done:** ✓ items completed this session (append as work progresses)
- **Now:** what the agent is actively doing
- **Next:** the queued items (per handoff priority or user direction)
- **User action items:** things ONLY the user can do (populate creds,
physical work, manual deploys) with the tickets they unblock
- **When to show it:** after each logical unit of work, at blockers, and
when pivoting between tickets. Not every trivial response — use judgment.
- **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
work completes, update the durable systems (ticket notes, Discourse wiki,
commits). The HUD just tracks the live narrative for the human.
## Working Style
- **Stop over-thinking.** Get to code and output faster. Explore with code;
gather ground truth. Do not burn tokens reasoning about things a quick command
answers.
- **Prefer Unix utilities** (awk, sed, grep, cut, tr) for file editing and
text processing over harness edit tools when feasible — deterministic and
exact where harness editors can be whitespace-fragile.
- **Farm work out to deterministic tooling:** linters, LSPs, formatters, test
runners. If an LSP is wired up, use it; otherwise pull a Docker image and lint
inside it.
- **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
Never read 10+ files sequentially; batch into agent calls.
- **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
(status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
If a command hits the timeout, STOP and investigate root cause — never
blindly retry. A hung command is a failed command. Detect failure fast,
diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
This applies to ALL tools — bash, docker, CLIs, sub-agents.
## Field Lessons (append at session close)
Hard-won operational gotchas. One line each; keep them short and load-bearing.
- **CLI body scanner workaround:** inline `-b "body"` arguments containing
words like `systemctl` are blocked by the command scanner. Write long bodies
to `/tmp/note.md` and pass `-b "$(cat /tmp/note.md)"` (redmine-cli +
discourse-cli both).
- **python3-venv trap:** `python3 -c 'import venv'` succeeds even without the
package; the resulting venv has no pip. Guard with `dpkg -s python3-venv`,
verify `test -x venv/bin/pip`, delete pip-less venvs and recreate.
- **temperusb API:** there is no `Temper` class. Use
`TemperHandler().get_devices()[i].get_temperatures(sensors=[0,1])`
(sensor 0 = internal, 1 = external probe).
- **LibreNMS API:** device add requires `"version": "v2c"` (with the v);
`discovery.php`/`poller.php` must run as the `librenms` user; SNMP
community fields are redacted in API responses by design.
- **remote.sh modes:** `vm-copy`/`prox-copy` destination must be a FULL file
path, never a directory; `vm-file`/`prox-file` push and execute a script
via `bash -s` (use for idempotent deploy scripts).
- **Remote restart verification:** after restarting services (especially in
parallel across hosts), re-verify state several seconds later. Immediate
checks race the restart and produce false failures.
- **NUT:** `upsd.conf` LISTEN edits only take effect after
`systemctl restart nut-server`. udev rule changes need
`udevadm control --reload-rules && udevadm trigger`.
- **Tailscale app-connector sources (HA):** HA polls originate from its LAN
IP (192.168.3.12), not its Tailscale IP. Include the HA LAN IP in every
allowlist (snmpd `rocommunity` source lists, upsd LISTEN binds).
- **`git mv` fails on untracked files:** plain `mv` first, `git add` after.
- **SNMP extend with UTF-8:** net-snmp CLI prints `°C` extend output as
Hex-STRING, but pysnmp/HA receive the decoded string. Parse the decoded
lm-sensors text lines with a regex value_template.
- **pfv-bms (Home Assistant):** no SSH by design; everything goes through the
REST API (`/api/`, `/api/config`, `/api/services`) with the long-lived
token in `~/.creds/homeassistant.env`. Instance unit system is °F — set
explicit `unit_of_measurement: "°C"` on custom sensors.
- **HAOS core_ssh on 22222 is the sanctioned host shell:** `VM_IP=pfv-bms-lan.knel.net
VM_USER=root VM_PORT=22222 bash tests/remote.sh vm '...'` — use the LAN name
(knel.net zone is Tailscale-IP based; TS inbound to pfv-bms is unreliable).
Host OS has NO git/ssh/curl — run git against /mnt/data/supervisor/homeassistant
via `docker run homeassistant/amd64-addon-git_pull:<ver>`.
- **Git pull add-on (core_git_pull) wipes /config on first run** if `.git` is
absent (rm -rf + fresh clone). ALWAYS pre-seed /config as a git checkout
(deploy key ~/.creds/pfv-bms-gitpull, repo KNEL/pfv-bms) before starting it.
- **HA core boot death-spiral:** supervisor watchdog SIGTERMs slow boots
(docker logs: "finish process received signal 15" while supervisor logs
"Still waiting for Core to start"). Fix from the 22222 shell:
`ha ha options --watchdog=false`, start core, diagnose, re-enable once stable.
- **Technitium on netinfra-02 (tsys-dns container) does not reload rsync-synced
zone files** — `docker restart tsys-dns` after record adds until #469 (native
clustering) lands. netinfra-01 (the dns-cli target) is fine.
- **Supervisor add-on options via API:** docker cp a script into
hassio_supervisor, read token from /data/cli.json `access_token`, POST
/addons/<slug>/options (the /apps/ spelling 404s).
- **NEVER restart/kill BOTH members of a redundant pair simultaneously**
(founder ruling 2026-09-01 after the DNS outage): restart/redeploy ONE
node, verify service health from an independent vantage, only then the
second. Blue/green with a health gate between hops — always serial.
- **Pi-hole v6: `dnsmasq_lines` in pihole.toml are passed to FTL's embedded
dnsmasq, which rejects some valid-dnsmasq options (e.g. `no-negcache`)
with "bad option" and dnsmasq then never starts — the container looks
"Up" but serves nothing on :53. Use first-class toml settings instead;
never sed-edit pihole.toml without a single-node health-gated rollout.
- **DNS outage recovery path:** workstation resolv.conf dies with the LAN
Pi-holes; use `tailscale status` peer IPs + the chokepoint env overrides
(NETINFRA01_HOST/NETINFRA02_HOST, VM_IP) to reach hosts by Tailscale
while names are unresolvable. Fix one node, verify, then the other.
- **dhcpd + AppArmor:** on the netinfra pair, dhcpd can only read configs
under `/etc/dhcp/**` — staging a candidate config in /tmp fails dhcpd -t
with "Permission denied" even as root. Stage as /etc/dhcp/*.candidate,
test, then move into place.
## Questions (NON-NEGOTIABLE)
**NEVER use a harness "question"/"ask user" tool** (structured prompts,
modal forms, tabbed questions). Banned across every project, every harness.
They are not portable, not version-controlled, and bypass the git record.
**All questions go in the current `questions-v(N).md` file** — write the
question; the human edits the answer inline in the same file. **Version up
the filename each time answers land** (v1 → v2 → v3...): create
`questions-v2.md` with resolved Q&A marked, new questions appended. This
preserves the history of each Q&A round. Synthesize resolved Q&A into
Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
§10.
## Documentation policy (IMPORTANT)
**Discourse is the canonical source of truth for all knowledge documentation.**
Knowledge docs (architecture, runbooks, references, audits, policies) have been
migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops)
as wiki topics in the **VP TechOps** category.
All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now
**pointers** that link to their corresponding Discourse topic. **Do not update
documentation content in git** — edit the Discourse wiki topic instead. Git
edit history no longer serves as the documentation changelog; Discourse
preserves wiki edit history automatically.
Code (scripts, configs, playbooks) still lives in git as the source of truth
for executables. Only *documentation* moved to Discourse.
**Redmine wikis stay disabled — Discourse is the sole documentation space**
(founder ruling, 2026-09-02). All Redmine projects have the wiki module off
(verified 0/62, 2026-09-02); never enable it on new or existing projects.
## Top-level files
All `.md` files now point to Discourse. The key pointers:
| File | Points to | Discourse topic |
|------|-----------|-----------------|
| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) |
| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) |
| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) |
Work tracking stays in [Redmine](https://projects.knownelement.com).
## Repository Layout
```
k8s/ k3s cluster setup scripts (HA control plane over Tailscale) + docs/
proxmox/ Proxmox fleet docs (hardware audit, capacity, storage)
awx/ Ansible AWX deployment (k3s + AWX Operator)
HomeAssistant/ Redirect stub — HA tooling + reference packs moved to KNEL/pfv-bms (tools/, docs/reference-packs/) [#762]
tests/ Test suite + VM validation harness + remote.sh SSH chokepoint
scripts/ Framework: git hooks, rule engine (check-rules.sh), shared lib
docs/ Server-build docs, docmap index, and archive
archive/ Historical/superseded code (provisioning -> replaced by KNELIAC project;
KNELServerBuild -> live patterns now in KNEL/monitoring, remainder
preserved under archive/KNELServerBuild/ [#474])
vendor/ Vendored KNELShellFramework
```
- **2026-09-03 repo split (#769)** — domain tooling moved out of this catch-all
into dedicated KNEL repos (working dirs `~/projects/KNEL/<repo>`; full git
history retained here):
| body of work | repo | ticket |
|---|---|---|
| SIEM/syslog (Wazuh) | KNEL/siem | #335 |
| DNS/NTP/DHCP + switches | KNEL/netinfra | #694 |
| Fleet CA | KNEL/ca | #697 |
| Monitoring/OAM | KNEL/monitoring | #343, #773 |
| Inventory/CMDB | KNEL/inventory | #705, #774 |
| Facilities (PDU/console/UPS/sensors) | KNEL/facilities | #458 |
| Performance/tuning | KNEL/perf | #709, #732, #737 |
| Backups (PBS) | KNEL/backups (new stub) | #315, #684 |
| Outbound mail (PMG) | KNEL/mailgateway (new stub) | #696 |
| Secrets mgmt | KNEL/secrets (new stub) | #770 |
| apt-satellite (deb version control) | KNEL/apt-satellite (new stub) | #771 |
| Security Onion (ops side) | KNEL/security-onion (new stub) | #772 |
Unit tests moved with their code (netinfra, facilities). IaC consumers:
KNEL/KNELIAC. Env map + snapshot lifecycle: Discourse t/331.
- **Server provisioning moved to KNELIAC**: The
[`archive/provisioning/`](archive/provisioning/) tree is historical. Active
server provisioning lives in the **KNELIAC** project at
`/home/reachableceo/projects/KNEL/KNELIAC`.
- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
(Proxmox hosts + all VMs) or
`~/projects/KNEL/netinfra/dns-cluster-setup/remote-dns.sh`
(DNS infra hosts: netinfra-01/02, netboot; lives in KNEL/netinfra [#769]). NEVER call
`ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
rejects it. There are no exceptions.
- **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
(neither LAN nor Tailscale IPs) in any command, script, or config.
ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
`VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
DNS name does not resolve, fix it in DNS (Technitium) or consult the
[system inventory — Discourse #307](https://community.turnsys.com/t/307).
Do NOT fall back to IP literals. This rule eliminates the per-session
discovery tax of finding the right IP for each host.
- **How to access a production VM:**
1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
NOT execution) to find the Tailscale DNS name, then access via that.
## Git Policy
1. **ALWAYS commit + push. Never wait. Ever.** After each logical unit of
work, immediately stage, commit (conventional format), and push to
`origin/main`. Do NOT pause for review. Do NOT ask permission. Do NOT
stop to "let the user read it first." The user reviews rendered markdown
on Gitea AFTER it is pushed. **This rule overrides any default
conservative commit-and-hold behavior** — if a default instruction says
"never commit unless asked," that instruction is superseded here.
2. **Atomic commits.** Each commit coherent on its own.
3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
4. **All shell scripts MUST pass `shellcheck` before commit.** No exceptions.
Run it via the wrapper:
```bash
bash tests/shellcheck.sh # whole repo
bash tests/shellcheck.sh ups/*.sh # specific files
```
This invokes `koalaman/shellcheck:stable` through Docker (no native binary
needed). Fix every reported finding — including `info`-level — or add a
targeted `# shellcheck disable=SCxxxx # <reason>` directive with a
justification. A script that emits any diagnostic is a protocol violation.
Non-bash scripts (PHP with `.sh` shebang `#!/usr/bin/php`, etc.) are exempt.
## Cross-linking (NON-NEGOTIABLE)
**Every artifact cross-references its related artifacts with CLICKABLE
links.** House rule (Charles, 2026-09-02, [#743]) — becomes hard
requirement once GLPI/CMDB change control is online (ITIL/ITSM ramp):
- Redmine ticket comments link the commits they describe (full Gitea URL,
e.g. `https://git.knownelement.com/KNEL/PFVCluster/commit/<sha>`).
- Commit bodies carry the full URL of the corresponding Redmine comment
(`https://projects.knownelement.com/issues/NNN#note-M`) — the "meat"
lives in the comment; the commit body points at it.
- Discourse posts link both the Redmine ticket and the Gitea repo/commit;
Redmine tickets and repo docs link their Discourse topics.
- One click from ANY system reaches the other two. An artifact without its
links is not done.
- When GLPI is online: change requests become the entry point for infra
changes and get cross-referenced into Redmine/Discourse/Gitea the same
way; CMDB items link their tickets and vice versa.
Mechanical enforcement (rule-engine check for the ticket-comment URL in
commit bodies when `[#NNN]` is present) is tracked under [#441].
## CI/CD (fleet standard, 2026-09-04)
**Every gitea repo runs CI via Gitea Actions.** Runner:
`ukrrs-pfv-gitea-runner` (act_runner 0.2.13, compose at
`~/projects/ukrrs/ci/gitea-runner/` on the pfv workstation) — registered
**org-scoped to KNEL** with label **`ultix` ONLY**; workflows must use
`runs-on: ultix` or they queue forever.
Repo duty: carry `.gitea/workflows/ci.yml` — lint for the repo's language
(pinned container images, never `:latest`) + a secret-scan step, on `dev`
and `release` pushes. Branch model: `dev` = WIP (CI must pass), `release` =
production/deploy branch (CI is the final gate).
Full standard: [Discourse topic #333](https://community.turnsys.com/t/333).
Tracking: [#784](https://projects.knownelement.com/issues/784).
## TDD & Linting
- **Red/green TDD for all code.** Mandatory (founder 2026-08-27). Interim
relaxation applies ONLY until the Ansible/IaC transition (week of 9/1);
from then on every playbook/script ships with its failing test first.
`scripts/test.sh` is the local gate; `tests/validation` + `tests/security`
run on sectestbed targets.
- **Linters on all code, as early as possible.** shellcheck zero-warning
(including info-level) is already enforced pre-commit.
## IaC Codification (shadow tracking) — NON-NEGOTIABLE
**Every manual/direct change to a fleet system must be codified same-day as
an AWX playbook item.** The fleet converges to 100% IaC (founder mandate,
2026-08-27, #454).
- Made a manual change? Add a checklist item to #454 (or a child ticket) in
the same session — what changed, where, exact commands, and any quirks
(e.g. "needed udevadm trigger after NUT install").
- Work is NOT "done" until the manual change is codified or explicitly
ticketed for codification.
- New manual fixes during incidents: fix first, codify immediately after.
- Playbooks live in this repo, tested through the `sectestbed-*` fleet.
## DNS change discipline (NON-NEGOTIABLE)
**Any change to Technitium records/zones (dns-cli or API), dhcpd, or the
live DNS/DHCP configs MUST end with a same-session git SoR sync.**
Founder rule, 2026-09-02 — drift-check exists because git went stale; do
not reopen that gap:
1. Make the change (serial + health-gated across the redundant pair).
2. `cd ~/projects/KNEL/netinfra && bash dns/technitium/zone-snapshot.sh` — refresh the DZ
snapshots from the primary.
3. `cd ~/projects/KNEL/netinfra && bash dns/drift-check.sh` — must print `ALL IN SYNC`.
4. Commit + push the refreshed snapshots in the same session.
## Redmine Tracking Policy
**Redmine is the system of record for all work.** Do not track status,
checklists, or TODOs in repo files. Use Redmine tickets instead.
**Tracker discipline:** OAM/ops/feature/audit tickets use tracker **Support
(3)** — NOT Bug. The redmine-cli `create` defaults to Bug; always correct the
tracker after create (python escape hatch: `tracker_id=3`). [2026-08-27]
- **URL:** https://projects.knownelement.com
- **Version:** Potential to Kinetic Ready (due 2026-09-30)
- **Project:** Known Element Enterprises - Technology & Facility Services (id 55)
### Rules
1. **Every piece of work** (feature, fix, deployment, config change) gets a
Redmine ticket. If one doesn't exist, create it.
2. **Reference tickets in docs and commits** using `[#NNN]` notation.
Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
3. **When work completes**, update the ticket: set done ratio to 100%,
add a note describing what was done and where the code lives.
**NEVER close a ticket without explicit user permission.** You may
SUGGEST a close when the result is clearly scoped and delivered. If
it's ambiguous whether the work is truly complete, don't suggest a
close — leave that decision to the user.
4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
5. **Link code to tickets** — ticket descriptions and notes should reference
the relevant file paths in this repo (e.g., the ups tooling in KNEL/facilities).
### CLI access (read + write)
Tickets are managed via the `redmine-cli` container, invoked directly
with `docker run` (no wrapper script). Full command reference, patterns, and
the subtask escape hatch live in the CLI's own `AGENTS.md`
(`~/projects/ukrrs/connectors/KNEL-AIMiddleware/tooling-cli/redmine/AGENTS.md`) — read it
for anything beyond the basics.
```bash
# Connection sanity check (run first in any session):
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest whoami
# Your queue (project 55):
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55
# Show / create / update / close:
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest show 367
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest create -p 55 -s "Subject" -d "desc"
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest update 367 -n "Done: committed in abc123" --done-ratio 100
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest close 367
```
Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
**Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
`python-redmine` escape hatch inside the container (see
`tooling-cli/redmine/AGENTS.md`). Always `show` a ticket before updating it.
Credentials (`REDMINE_URL`/`REDMINE_API_KEY`) live in the centralized store
at `~/.creds/redmine.env`.
## Documentation Workflow
**Discourse is the source of truth for all knowledge docs.** After any work
session, an agent MUST:
1. **Update Discourse wiki topics** — if infrastructure facts changed (new
VM, IP change, host retired), edit the relevant wiki topic at
[community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops).
2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297))
if tickets were opened or closed.
3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'`
after any rename or restructure. Fix all references in the same commit.
4. **If a new top-level directory was created**, update:
- `AGENTS.md` → "Repository Layout" code block
- `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script)
- Create a new Discourse wiki topic for any documentation
5. **Self-audit before commit.** Code changes must be internally consistent.
Documentation changes go to Discourse, not git.
### CLI access (read + write)
Wiki topics are managed via the `discourse-cli` container, invoked directly
with `docker run` (no wrapper script). Full command reference, patterns, and
the raw-API escape hatch live in the CLI's own `AGENTS.md`
(`~/projects/ukrrs/connectors/KNEL-AIMiddleware/tooling-cli/discourse/AGENTS.md`) — read it
for anything beyond the basics.
```bash
# Connection sanity check (run first in any session):
docker run --rm --env-file ~/.creds/discourse.env \
git.knownelement.com/reachableceo/discourse-cli:latest whoami
# List VP TechOps topics:
docker run --rm --env-file ~/.creds/discourse.env \
git.knownelement.com/reachableceo/discourse-cli:latest ls -c vp-techops
# Show a topic / edit a wiki post (find post id via `show`):
docker run --rm --env-file ~/.creds/discourse.env \
git.knownelement.com/reachableceo/discourse-cli:latest show 297
docker run --rm --env-file ~/.creds/discourse.env \
git.knownelement.com/reachableceo/discourse-cli:latest update <post_id> -b "new markdown body"
```
VP TechOps = category **74**. Key topics: #296 (project overview), #297
(ops status), #298 (audit log). The API user is trust-level 4 but **not
admin** — admin-only ops (category creation, setting the wiki flag) will
403; surface those to the user rather than retrying. **Gotcha:**
`update`/`delete` take a post **id**, not a post number. Never create a new
topic for an update to existing knowledge — edit the wiki post in place.
Credentials (`DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`)
live in the centralized store at `~/.creds/discourse.env`.
## Key Scripts
| Script | Purpose |
|--------|---------|
| [`scripts/check-rules.sh`](scripts/check-rules.sh) | Rule audit engine (shellcheck, image pinning, Discourse pointers, required files) |
| [`scripts/setup-hooks.sh`](scripts/setup-hooks.sh) | Install git hooks (pre-commit, pre-push) |
| [`scripts/pve-snapshot.sh`](scripts/pve-snapshot.sh) | PVE host snapshot/backup pass — hard-protects prod before risky windows (verified 2026-09) |
| [`scripts/audit-fleet.sh`](scripts/audit-fleet.sh) | Fleet-wide audit sweep (hosts, services, drift) |
| [`scripts/garden.sh`](scripts/garden.sh) | Repo/workspace gardening loop (strays, cruft, staleness) |
| `scripts/kuma-*.py` suite | Uptime Kuma management (add-ping, fleet-sync, inventory, maintenance, notifications, regroup) |
| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
| KNEL/netinfra: `dns-cluster-setup/remote-dns.sh` | SSH chokepoint for DNS infra hosts (netinfra-01/02, netboot) |
| KNEL/netinfra: `dns/technitium/zone-snapshot.sh` | Refresh git SoR Technitium zone snapshots from the primary — **required after every DNS change** |
| KNEL/netinfra: `dns/drift-check.sh` | Live-vs-git drift audit for DHCP/DNS/NTP/zones — must be green before any DNS-adjacent push |
| KNEL/inventory: `seed/from-inventory.sh` | Regenerate the CMDB seed CSV from the Discourse #307 inventory raw (re-run when #307 changes) |
| `redmine-cli` container | Redmine CLI (ticket read/write via `docker run`; see `tooling-cli/redmine/`) |
| `discourse-cli` container | Discourse CLI (wiki topic read/write via `docker run`; see `tooling-cli/discourse/`) |
| `dns-cli` container | Technitium DNS CLI (zones, list, add, delete, search, flush; see `tooling-cli/dns/`) |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
| KNEL/netinfra: `dns-cluster-setup/setup.sh` | DNS cluster replication |
| [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
| KNEL/facilities: `powerman/setup.sh` | Configure Cyclades PDU via powerman |
| KNEL/facilities: `console/setup.sh` | Configure serial console access via ser2net + conman |
| KNEL/facilities: `ups/setup.sh` | Configure NUT (Network UPS Tools) for UPS monitoring |
| KNEL/perf: `deploy-tuning.sh` | Deploy perf tunings |
| KNEL/perf: `scripts/` | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |
## Switch Console Access (conman)
Switch configs are pulled via serial console through a conman + ser2net
stack on pfv-tsys4.
- **conmand server:** pfv-tsys4 (port 7890)
- **ser2net:** pfv-tsys4, TCP ports 2001-2006
(do NOT connect to ser2net directly — it conflicts with conman's
persistent sessions; always use the conman client)
- **Script:** KNEL/perf: `scripts/conman-console.py`
— drives console sessions read-only via PTY. No expect/tcl required.
- **Command files:** KNEL/netinfra: `switches/` — `.cmds` files
with switch-specific show commands
- **Query available consoles:** `conman -d pfv-tsys4 -q`
```bash
# Pull a switch config (example):
CONMAN_SERVER=pfv-tsys4:7890 python3 ~/projects/KNEL/perf/scripts/conman-console.py \
--console pfv-r5-core-01 --cmds ~/projects/KNEL/netinfra/switches/pfv-r5-core-01.cmds
```
| Console name | TCP port | Device |
|--------------|----------|--------|
| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
## Mandatory: Document ALL Infrastructure Changes
**This is non-negotiable. Every infrastructure change (VM config, disk
cache, network setting, service config, storage migration) MUST be
documented BEFORE moving to the next task step — not "later" or "at the
end."**
Required for EVERY infrastructure change:
1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
2. **Discourse audit log** — reply to topic
[#298](https://community.turnsys.com/t/298) with a dated entry (what
changed, why, where).
3. **Discourse relevant wiki topic** — update the architecture/reference
topic if the change affects documented infrastructure facts (storage
#300, network #299, k8s #305, etc.).
4. **Git commit** — if code/config changed in the repo, commit + push
immediately per the Git Policy above.
## Tooling
- `gh`, `docker`, `jq` available on the workstation.
- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
`koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
- For raw API calls not covered by the Redmine/Discourse CLIs, use
`python3` inside the CLI Docker containers (escape hatch pattern in
`tooling-cli/{discourse,redmine}/AGENTS.md`).
- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
the python-in-Docker escape hatch for HTTP writes.
## Key Commands Quick Reference
```bash
# Tests + validation:
bash tests/run-tests.sh # test suite
bash tests/vm-validation.sh # VM validation
bash tests/shellcheck.sh # shellcheck whole repo
bash tests/shellcheck.sh path/to/*.sh # shellcheck specific files
# Redmine + Discourse sanity checks (run first in any session):
docker run --rm --env-file ~/.creds/redmine.env \
git.knownelement.com/reachableceo/redmine-cli:latest whoami
docker run --rm --env-file ~/.creds/discourse.env \
git.knownelement.com/reachableceo/discourse-cli:latest whoami
# Proxmox host access (DNS names only — never IPs):
PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
# Production VM access (DNS names only — never IPs):
VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
# DNS infra access:
bash ~/projects/KNEL/netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
# Switch console (read-only config pull — DNS names only):
CONMAN_SERVER=pfv-tsys4:7890 python3 ~/projects/KNEL/perf/scripts/conman-console.py \
--console pfv-r5-core-01 --cmds ~/projects/KNEL/netinfra/switches/pfv-r5-core-01.cmds
```
## Credential Management
- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
- User goal: migrate all keys to Hashicorp Vault
(`vault.knownelement.com`). No vault token present yet
(`~/.vault-token` missing). Track as high-priority TODO.
## Key Docs
→ **All documentation lives on Discourse:**
[community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops)
→ All `.md` files in this repo are pointers to Discourse topics.
→ **Complete Linux System Inventory:**
[Topic #307](https://community.turnsys.com/t/307) — every Linux system
(hosts + VMs + physical), with Tailscale IPs, DNS names, SSH access
status, and tuned profiles. Reference this for monitoring coverage,
access management, and hostname consistency.
## Project Context
Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
Production lives on a Cloudron VPS in Reston VA. See the
[Operations Status topic](https://community.turnsys.com/t/297) for the ticket
index and infrastructure summary.
All work is tracked in [Redmine](https://projects.knownelement.com)
(version: Potential to Kinetic Ready, due 2026-09-30).
- **HA 2026.8 snmp platform traps:** temperature/current-classed sensors with
`state_class: measurement` + non-numeric render ('unknown') are dropped AT
ADD-TIME (ValueError) when a host is asleep — entity vanishes entirely. Fix:
no state_class (stats live in VM) + templates render EMPTY on no-match.
- **Legacy snmp extend OID encoding:** suffix = strlen + ASCII decimal per
char ("garagepdu" = 9.103.97.114.97.103.101.112.100.117 — count carefully).
- **Old APC PDUs (AOS ~2.7):** SNMPv1 only; subtree WALKS die silently, small
single-table walks + GETs work. Garage AP7830 current OID:
.1.3.6.1.4.1.318.1.1.12.2.3.1.1.2.1 (tenths of amps). Relay pattern via
tsys-librenms (the allowed poller vantage) for v1-only/ACL-locked gear.
- **Debian snmpd on monitoring VMs:** snmpd may fail to bind 127.0.0.1:161
(exit 1, "Error opening specified endpoint") when something squats
localhost — bind only the LAN/TS addresses in agentaddress.
- **iDRAC6/7 SNMP:** agents exist but hw MIB tables unpopulated on old fw;
IPMI SDR via iDRAC SSH (`racadm`, one command per exec, legacy syntax on
iDRAC6: `racadm config -g cfgOobSnmp -o ...`) is the data path.
- **jq in one-liner state checks:** use `.state` — not `.s` — a wrong key
silently returns null and looks like an entity outage (cost an hour tonight).
- **HA config flows are REST** (`POST /api/config/config_entries/flow`),
not websocket; websocket only monitors. ha-ws-call.py + REST flow are the
two levers for agent-driven HA setup.
- **mvdan/sh `source a b` silently ignores file b** — source env files
one per line.
- **HA todo lists (founder HUD):** items are hidden from `/api/states` —
fetch via `POST /api/services/todo/get_items?return_response=true`.
Renaming works ONLY with a minimal `todo/update_item` payload
(`entity_id`, `item`=<uid>, `rename`); passing `status`/`due` back earns
a 400. HUD summaries carry `[#NNN]` Redmine prefixes — sync completed
items back to Redmine when the founder asks.
- **HA package YAML: NEVER append a second top-level `sensor:` key** — YAML
duplicate keys are last-one-wins and silently shadow the whole first block
at config load (#344 outage 2026-09-02: all 22 fleet temps down after the
restart that pulled the file). Merge new sensors into the existing block.
- **HA 2026.8 snmp add-time crash, general form:** ANY unit-bearing sensor
with a non-numeric native_value at ADD (host down renders '' / literal
string "unknown"; value_template can never yield None — even `{{ none }}`
is stringified to "None") = ValueError, entity vanishes till manual
reload. `check_config` does NOT catch it. Durable pattern: unit-less raw
snmp transport + `template:` layer carrying unit/device_class under the
ORIGINAL unique_ids (KNEL/pfv-bms packages/plant_snmp.yaml header).
- **HA entity_id comes from the NAME slug, not unique_id** — if a sensor's
name and unique_id diverge, cross-references must use the name slug
(garage PDU amps lesson: semantic read a nonexistent `..._amps_raw`).
- **HA entity_id collisions spawn `_2` entities; registry remove no-ops for
LIVE entities** (they re-register on their next write). Clean `_2` by
jq-editing `.storage/core.entity_registry` with core STOPPED, backup
first. Orphaned (dead) entries remove fine via websocket
(ha-ws-call.py — pass commands as JSON files; shell quoting eats argv).
- **git_pull add-on: `ha addons start` is a no-op while it's running** —
use `restart` to force an immediate pull. It auto-restarts core for
files in its restart list; otherwise pull ≠ reload (config lands on
disk, running core keeps the old one until a platform reload/restart).
- **Restore drills / heavy I/O NEVER on hosts running prod VMs** (2026-09-03
incident, #684): qmrestore of a 32G disk to scratch VMID 5599 on tsys1
starved VM 100 (pfv-bms) I/O ~40 min, degraded prod HA. Drills run dev-lane
only (tsys5); prod restores are server-throttled via PBS traffic-control.
- **`pkill -f <pat>` over the ssh chokepoint self-matches** your own remote
command line and kills the session (exit 255, no output). Use the bracket
trick: `pkill -f "[q]mrestore"`.
- **pvesh backup jobs take `-vmid`** (comma list), not `-vmids`; aborted
`qmrestore` leaves a create-locked stub VM — `qm unlock <id>` then destroy.
- **PBS 4.x gotchas:** prune retention lives in `prune-job` (datastore keep-*
opts are gone); API tokens need their OWN acl entry — user-only ACL yields
"Cannot find datastore" from PVE.
- **PVE bridge port + hotplug NIC (#780):** a re-registered NIC (USB move
changes its enumeration path) comes back UP but is NOT re-enslaved to its
bridge — `ifup <nic>` is not enough; run `ifup <bridge>` and confirm
`bridge link` shows `state forwarding`. NFS `hard` mounts mostly self-heal,
but `nconnect=4` mounts can stay wedged on stale channels after a flap:
force-remount (`umount -f -l`, recreate the mountpoint dir if the lazy
umount removed it, fresh mount). `pvesm status` "active" can be stale —
verify with a real read, not the status line.
- **PMG pmgproxy restart hang (#696):** `systemctl restart pmgproxy` can hang
in "deactivating (stop)" on old workers ignoring TERM. If stuck >90s:
`systemctl kill -s KILL pmgproxy && systemctl start pmgproxy` — API/UI
only, mail flow unaffected. Postfix restarts don't hang.
Reference in New Issue View Git Blame Copy Permalink