docs(agents): move operational knowledge from harness memory to AGENTS.md

Make operational instructions portable across AI harnesses (Crush,
Hermes, OpenWebUI). Previously lived in .crush/memory/operational.md
which is Crush-specific.

New sections added to AGENTS.md:
- Switch Console Access (conman): server, script, console map, usage
- Mandatory Documentation Protocol: every infra change must be ticketed
  + logged in Discourse before moving on
- Tooling: available tools, shellcheck, HTTP write constraints
- Key Commands Quick Reference: common commands for all access patterns
- Credential Management: current state + Vault migration goal

Updated:
- "SSH in Crush" generalized to "SSH access" (harness-agnostic)
- Key Scripts table updated with new audit/tuned scripts

The .crush/memory/operational.md file is now redundant; all its content
lives in AGENTS.md.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-08-06 17:17:37 -05:00
parent 7fb37d5b40
commit d22db46e9f
+95 -2
View File
@@ -49,7 +49,7 @@ vendor/ Vendored KNELShellFramework
`/home/reachableceo/projects/KNELIAC`. `/home/reachableceo/projects/KNELIAC`.
- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh` - **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks. extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **SSH in Crush**: Direct ssh/scp is blocked. Use - **SSH access**: Direct ssh/scp may be blocked by some AI harnesses. Use
[`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
[`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts). [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
@@ -193,7 +193,100 @@ lives (gitignored) at `/home/reachableceo/projects/KNEL-AIMiddleware/discourse-c
| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman | | [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring | | [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings | | [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift | | [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |
## Switch Console Access (conman)
Switch configs are pulled via serial console through a conman + ser2net
stack on pfv-tsys4.
- **conmand server:** pfv-tsys4 (Tailscale `100.70.77.93`, port 7890)
- **ser2net:** pfv-tsys4, TCP ports 2001-2007 on Tailscale IP
(do NOT connect to ser2net directly — it conflicts with conman's
persistent sessions; always use the conman client)
- **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
— drives console sessions read-only via PTY. No expect/tcl required.
- **Command files:** [`netinfra/switches/`](netinfra/switches/) — `.cmds` files
with switch-specific show commands
- **Query available consoles:** `conman -d 100.70.77.93 -q`
```bash
# Pull a switch config (example):
CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
--console pfv-core-sw01 --cmds netinfra/switches/pfv-core-sw01.cmds
```
| Console name | TCP port | Device |
|--------------|----------|--------|
| pfv-core-sw01 | 2001 | Dell PowerConnect 5448 (core, rack 5) |
| pfv-tor3-mgmt | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
| pfv-tor3-stor | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
| pfv-r2-tor-top | 2005 | Rack 2 top-of-rack switch |
| subodev-torsw | 2006 | Suborbital device TOR switch |
| pfv-r2-sw | 2007 | Rack 2 old Dell switch |
## Mandatory: Document ALL Infrastructure Changes
**This is non-negotiable. Every infrastructure change (VM config, disk
cache, network setting, service config, storage migration) MUST be
documented BEFORE moving to the next task step — not "later" or "at the
end."**
Required for EVERY infrastructure change:
1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
2. **Discourse audit log** — reply to topic
[#298](https://community.turnsys.com/t/298) with a dated entry (what
changed, why, where).
3. **Discourse relevant wiki topic** — update the architecture/reference
topic if the change affects documented infrastructure facts (storage
#300, network #299, k8s #305, etc.).
4. **Git commit** — if code/config changed in the repo, commit + push
immediately per the Git Policy above.
## Tooling
- `gh`, `docker`, `jq` available on the workstation.
- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
`koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
- For raw API calls not covered by the Redmine/Discourse CLIs, use
`python3` inside the CLI Docker containers (escape hatch pattern in
`~/daytoday/redmine/AGENTS.md` and `~/daytoday/discourse/AGENTS.md`).
- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
the python-in-Docker escape hatch for HTTP writes.
## Key Commands Quick Reference
```bash
# Tests + validation:
bash tests/run-tests.sh # test suite
bash tests/vm-validation.sh # VM validation
bash tests/shellcheck.sh # shellcheck whole repo
bash tests/shellcheck.sh path/to/*.sh # shellcheck specific files
# Redmine + Discourse sanity checks (run first in any session):
~/daytoday/redmine/bin/redmine whoami
~/daytoday/discourse/bin/discourse whoami
# Proxmox host access:
PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
# DNS infra access:
bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
# Switch console (read-only config pull):
CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
--console pfv-core-sw01 --cmds netinfra/switches/pfv-core-sw01.cmds
```
## Credential Management
- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
- User goal: migrate all keys to Hashicorp Vault
(`vault.knownelement.com`). No vault token present yet
(`~/.vault-token` missing). Track as high-priority TODO.
## Key Docs ## Key Docs