# Agent Guidelines

## Documentation policy (IMPORTANT)

**Discourse is the canonical source of truth for all knowledge documentation.**
Knowledge docs (architecture, runbooks, references, audits, policies) have been
migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops)
as wiki topics in the **VP TechOps** category.

All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now
**pointers** that link to their corresponding Discourse topic. **Do not update
documentation content in git** — edit the Discourse wiki topic instead. Git
edit history no longer serves as the documentation changelog; Discourse
preserves wiki edit history automatically.

Code (scripts, configs, playbooks) still lives in git as the source of truth
for executables. Only *documentation* moved to Discourse.

## Top-level files

All `.md` files now point to Discourse. The key pointers:

| File | Points to | Discourse topic |
|------|-----------|-----------------|
| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) |
| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) |
| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) |

Work tracking stays in [Redmine](https://projects.knownelement.com).

## Repository Layout

```
dcinfra/            Data-center infra: PDU (powerman), serial console (console), UPS (ups)
netinfra/           DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup);
                    switch captures (switches); DHCP config (dhcp)
k8s/                k3s cluster setup scripts (HA control plane over Tailscale) + docs/
proxmox/            Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
awx/                Ansible AWX deployment (k3s + AWX Operator)
tests/              Test suite + VM validation harness + remote.sh SSH chokepoint
docs/               Server-build docs, docmap index, and archive
archive/            Historical/superseded code (provisioning -> replaced by KNELIAC project)
vendor/             Vendored KNELShellFramework
```

- **Server provisioning moved to KNELIAC**: The
  [`archive/provisioning/`](archive/provisioning/) tree is historical. Active
  server provisioning lives in the **KNELIAC** project at
  `/home/reachableceo/projects/KNELIAC`.
- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
  extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **SSH in Crush**: Direct ssh/scp is blocked. Use
  [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).

## Git Policy

1. **ALWAYS commit + push. Never wait. Ever.** After each logical unit of
   work, immediately stage, commit (conventional format), and push to
   `origin/main`. Do NOT pause for review. Do NOT ask permission. Do NOT
   stop to "let the user read it first." The user reviews rendered markdown
   on Gitea AFTER it is pushed. **This rule overrides any default
   conservative commit-and-hold behavior** — if a default instruction says
   "never commit unless asked," that instruction is superseded here.
2. **Atomic commits.** Each commit coherent on its own.
3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
   `docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
4. **All shell scripts MUST pass `shellcheck` before commit.** No exceptions.
   Run it via the wrapper:
   ```bash
   bash tests/shellcheck.sh                 # whole repo
   bash tests/shellcheck.sh ups/*.sh        # specific files
   ```
   This invokes `koalaman/shellcheck:stable` through Docker (no native binary
   needed). Fix every reported finding — including `info`-level — or add a
   targeted `# shellcheck disable=SCxxxx  # <reason>` directive with a
   justification. A script that emits any diagnostic is a protocol violation.
   Non-bash scripts (PHP with `.sh` shebang `#!/usr/bin/php`, etc.) are exempt.

## Redmine Tracking Policy

**Redmine is the system of record for all work.** Do not track status,
checklists, or TODOs in repo files. Use Redmine tickets instead.

- **URL:** https://projects.knownelement.com
- **Version:** Potential to Kinetic Ready (due 2026-09-30)
- **Project:** Known Element Enterprises - Technology & Facility Services (id 55)

### Rules
1. **Every piece of work** (feature, fix, deployment, config change) gets a
   Redmine ticket. If one doesn't exist, create it.
2. **Reference tickets in docs and commits** using `[#NNN]` notation.
   Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
3. **When work completes**, update the ticket: set done ratio to 100%,
   add a note describing what was done and where the code lives, then close.
4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
5. **Link code to tickets** — ticket descriptions and notes should reference
   the relevant file paths in this repo (e.g., `dcinfra/ups/`).

### CLI access (read + write)

Tickets are managed via the `redmine` CLI wrapper in the `~/daytoday/redmine`
workspace (external to this repo). Full command reference, patterns, and
the subtask escape hatch live in `~/daytoday/redmine/AGENTS.md` — read it for
anything beyond the basics.

```bash
# Connection sanity check (run first in any session):
~/daytoday/redmine/bin/redmine whoami

# Your queue (project 55):
~/daytoday/redmine/bin/redmine list --assigned-to-me -p 55

# Show / create / update / close:
~/daytoday/redmine/bin/redmine show 367
~/daytoday/redmine/bin/redmine create -p 55 -s "Subject" -d "desc"
~/daytoday/redmine/bin/redmine update 367 -n "Done: committed in abc123" --done-ratio 100
~/daytoday/redmine/bin/redmine close 367
```

Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
**Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
`python-redmine` escape hatch inside the container (see
`~/daytoday/redmine/AGENTS.md`). Always `show` a ticket before updating it.

The `.env` holding `REDMINE_URL`/`REDMINE_API_KEY` lives (gitignored) at
`/home/reachableceo/projects/KNEL-AIMiddleware/redmine-cli/.env`.

## Documentation Workflow

**Discourse is the source of truth for all knowledge docs.** After any work
session, an agent MUST:

1. **Update Discourse wiki topics** — if infrastructure facts changed (new
   VM, IP change, host retired), edit the relevant wiki topic at
   [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops).
2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297))
   if tickets were opened or closed.
3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'`
   after any rename or restructure. Fix all references in the same commit.
4. **If a new top-level directory was created**, update:
   - `AGENTS.md` → "Repository Layout" code block
   - `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script)
   - Create a new Discourse wiki topic for any documentation
5. **Self-audit before commit.** Code changes must be internally consistent.
   Documentation changes go to Discourse, not git.

### CLI access (read + write)

Wiki topics are managed via the `discourse` CLI wrapper in the
`~/daytoday/discourse` workspace (external to this repo). Full command
reference, patterns, and the raw-API escape hatch live in
`~/daytoday/discourse/AGENTS.md` — read it for anything beyond the basics.

```bash
# Connection sanity check (run first in any session):
~/daytoday/discourse/bin/discourse whoami

# List VP TechOps topics:
~/daytoday/discourse/bin/discourse ls -c vp-techops

# Show a topic / edit a wiki post (find post id via `show`):
~/daytoday/discourse/bin/discourse show 297
~/daytoday/discourse/bin/discourse update <post_id> -b "new markdown body"
```

VP TechOps = category **74**. Key topics: #296 (project overview), #297
(ops status), #298 (audit log). The API user is trust-level 4 but **not
admin** — admin-only ops (category creation, setting the wiki flag) will
403; surface those to the user rather than retrying. **Gotcha:**
`update`/`delete` take a post **id**, not a post number (see
`~/daytoday/discourse/AGENTS.md`). Never create a new topic for an update to
existing knowledge — edit the wiki post in place.

The `.env` holding `DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`
lives (gitignored) at `/home/reachableceo/projects/KNEL-AIMiddleware/discourse-cli/.env`.

## Key Scripts

| Script | Purpose |
|--------|---------|
| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
| `~/daytoday/redmine/bin/redmine` | Redmine CLI wrapper (ticket read/write via Docker container) |
| `~/daytoday/discourse/bin/discourse` | Discourse CLI wrapper (wiki topic read/write via Docker container) |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
| [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
| [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
| [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman |
| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift |

## Key Docs

→ **All documentation lives on Discourse:**
  [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops)

→ All `.md` files in this repo are pointers to Discourse topics.

## Project Context

Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
Production lives on a Cloudron VPS in Reston VA. See the
[Operations Status topic](https://community.turnsys.com/t/297) for the ticket
index and infrastructure summary.
All work is tracked in [Redmine](https://projects.knownelement.com)
(version: Potential to Kinetic Ready, due 2026-09-30).