docs: initial commit — Q2/Q3 transition planning docs

Planning documents for TSYS Group's COO→CTO handoff and AI agent identity architecture. Shared publicly as a bootstrapping reference. Includes: org prompts, transition map, agent identity bootstrap plan, TechOps/K8s/SecOps context notes. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-08-13 10:43:48 -05:00
commit 8c76cf1bab
7 changed files with 876 additions and 0 deletions
@@ -0,0 +1,311 @@
+# COO Handoff & CTO Transition — Master Map
+
+**Status:** DRAFT for review (working session artifact, not yet synthesized to Discourse/Redmine)
+**Date:** 2026-08-13
+**Deadline:** 2026-09-30 ("Potential to Kinetic Ready" + COO handoff, 48 days / "45 days")
+**Author session:** orientation/synthesis pass over Q3 prompts, all project trees, Gitea, Discourse, Redmine
+
+---
+
+## 0. Thesis (the one thing to internalize)
+
+**Revised 2026-08-13:** Timeline split confirmed with user.
+
+| Phase | Window | Focus | Who |
+|---|---|---|---|
+| **Q3 remainder** | Aug 13 → Sep 30 | TechOps finish (P1-P9) + **stand up AI agent identities** | Charles + AI agents |
+| **Q4** | Oct 1 → Dec 31 | Business ops transition + CTO come-up (80% CTO / 20% COO) | Charles + AJ + Patti + Courtney + AI agents |
+| **Jan 1 2027** | — | Full COO handoff complete. Charles = 100% CTO. | AJ + COO agent run business ops |
+
+The infrastructure (PFVCluster, KNELIAC, monitoring, k8s) is ~80% there and has a phase plan (P1-P9) with a 9/30 deadline. The "running datacenter" is commodity and will be fully delegated to AI. The **first action is standing up AI agent identities** (Cloudron + Bitwarden + system access) so agents can operate with proper attribution, RBAC, and audit trails. See `agent-identity-bootstrap.md`.
+
+**Business ops transition is deferred to Q4.** The COO Discourse category, Redmine project 53, bizopprodplan handbook refresh, ITSM tool selection, and COO Tier-1 business apps all move to Q4. AJ gets regular briefings through Q3 and takes over business ops in Q4 with Patti and Courtney as additional resources.
+
+---
+
+## 1. Current-State Map
+
+### 1.1 The TSYS Group org structure (as designed in Q3/prompt.md)
+
+```
+TSYS Group (Board of Directors)
+├── CTO  — Charles (R&D, architecture, tier-4 SME)  [you, transitioning IN fulltime Oct 1]
+└── COO  — AJ Lebsch (operations)  [transitioning IN, oversees AI agents]
+     ├── SVP KNEL  (Known Element Enterprises — owns ALL IT/business systems)
+     │    ├── VP TechOps        ← the ONLY function with real content today
+     │    ├── VP SecOps         ← does not exist as a category yet
+     │    └── VP TechCompliance ← Discourse cat 75 exists, 0 topics
+     └── SVP TCTC  (The Campus Trading Company)
+          ├── VP Finance / VP Accounting / VP Investing
+          ├── VP Treasury / VP Trading
+          └── (RedWFO = RWSCP Family Office, mission-critical, can preempt)
+```
+
+**Supporting entities** (each with its own Gitea org + Redmine project + Discourse category, mostly stubs):
+Suborbital Systems, HFNOC, HFNFC, RackRental, Starting Line Productions, Rogue Technologies, RedWFO, RWSCP, MeetMorse/MorsePod (FLO entry to CommonsNet), EzEDA, EzPodStack, AFABN, Ap4Ap, MerchantsOfHope, ThePeerNet, sol-calc, TeamRental, SideDoorGroup, YourDreamNameHere.
+
+### 1.2 Systems of Record — actual state vs intended
+
+| System | Role | Actual state |
+|---|---|---|
+| **Redmine** (projects.knownelement.com) | SoR for ALL work | **55 projects** mirror the org chart. **Only project 55 (TechnicalOperations) is active: 136 tickets, 86 open.** Project 62 (Business Services), 53 (COO), 77 (TSYS Group parent), and all entity projects are **empty/stubs.** |
+| **Discourse** (community.turnsys.com) | SoR for ALL docs | **55 categories.** Only **VP TechOps (cat 74, 13 topics)** and **Progress Reports (cat 61, 81 topics)** have real content. COO (6), VP Compliance (75), Board (76), KNEL-Bizops (72) = **0 topics.** Every business-entity category = 0-1 stub topics. |
+| **Gitea** (git.knownelement.com) | SoR for executable code | **27 orgs, ~213 repos** (87 are ExternalVendorCode mirrors). Source repos: KNEL (32), reachableceo (23), Suborbital-Systems-Public (18), RWSCP (11), + ~20 entity orgs with 1-3 repos each (mostly `-bizopprodplan` mdBook stubs). |
+| **Cloudron** (Reston VPS) | PaaS for ~57 support-stack apps | **10/57 packaged (~17%).** See §1.4. |
+| **K8s** (PFVCluster bare metal) | Scalable compute | k3s HA (3 cnode + 6 workers) but **cnodes were wiped and shut down — needs rebuild.** Zero apps deployed. |
+
+### 1.3 The 12 local projects (maturity + COO relevance)
+
+| Project | What | Maturity | Governance | COO relevance |
+|---|---|---|---|---|
+| **PFVCluster** | Proxmox fleet + OAM + k8s bootstrap | High (most mature infra) | Excellent (full rules engine) | **High** — the compute foundation |
+| **football** (KNEL-Football) | Hardened Debian live ISO for tier-0 access | ★★★★★ (788 tests, ISO built, audited) | Exceptional | High — secure access terminal |
+| **KNELIAC** | Ansible fleet config-mgmt (replaces bash) | Active (9 roles, AWX wired) | **Weak** (80-line AGENTS.md) | High — fleet baseline |
+| **KNEL-AIMiddleware** | MCP/LSP servers for AI agents (42 svcs) | 79% (33/42 prod-ready) | Moderate | **Critical** — the agent tooling backbone |
+| **EngStack** | Hardware R&D engineering workstation (45 tools) | Active (8 tools built) | Good | Med (CTO/R&D domain) |
+| **TSYS-Cloudron** | 57-app Cloudron packaging | 17% (10/57) | Strong | **High** — COO's business apps |
+| **WorkstationStack** | Local dev support stack + SelfStack | Active (7 apps in prod) | Good | Med |
+| **hermes-rceo-streaming** | Hermes AI agent deployment | Deployed | Minimal | **High** — agent runtime (security note: full host access) |
+| **netbird** | Zero-trust VPN access | Early/stub | Minimal | Med (security-critical, underbuilt) |
+| **meta (TSYSGroupAIOS)** | Canonical agent-framework template | **Production-ready, NOT pushed** | Canonical | **Critical** — must propagate to all projects |
+| dotfiles / EngineeringWorkstation / TSYS-LocalWorkstation | Personal/scratch | — | — | Low |
+
+### 1.4 Cloudron app fleet — COO-critical gap
+
+**10 packaged:** Webhook, APISIX, Healthchecks, Review Board, WireViz Web, Puter, Corteza, draw.io, Windmill, InvenTree.
+**COO Tier-1 MISSING (not started):** Grist (ops spreadsheets), PayrollEngine, KillBill (billing), Rundeck (runbook automation), Comply (compliance tracking).
+**Cloudron-blocked (need K8s):** Sentry, SigNoz, DataHub, NetBox, Fleet (all need Redis/Kafka/ClickHouse).
+
+### 1.5 Governance template (TSYSGroupAIOS / meta) — adoption status
+
+`~/daytoday/meta` is the canonical framework: `BASELINE-PROMPT.md` (14 principles) + 10-section `AGENTS.md` template + `scripts/check-rules.sh` (10 checks) + git hooks + `hooks/ticket-gate.sh`. **Self-applying (17 PASS/0 FAIL), ready to push** to `TSYSGroupCorporate/TSYSGroupAIOS` (remote configured, not yet pushed).
+
+| Project | Gap to framework |
+|---|---|
+| PFVCluster | 🟢 Minor — add BASELINE-PROMPT.md, align scripts |
+| football | 🟡 Moderate — replace custom hooks |
+| KNEL-AIMiddleware | 🟡 Moderate — drop JOURNAL.md, add pre-commit |
+| **KNELIAC** | 🔴 Critical — needs entire enforcement layer |
+| **EngStack** | 🔴 Critical — needs entire `scripts/` + governance |
+
+---
+
+## 2. The Central Gap (COO handoff) — what "done" requires on Oct 1
+
+For AJ + the COO AI agent to actually run operations, these must be true. Today **none of them are.**
+
+### 2.1 Org / role
+- [ ] COO AI agent stood up with Cloudron account, logged into all role-appropriate systems (Redmine, Discourse, Gitea, Cloudron, monitoring). **Repo exists: `KNEL/TSG-COOAndBoard-AIAgents-Public` — not yet built out.**
+- [ ] SVP KNEL agent + SVP TCTC agent (synthesize policy/strategy → COO).
+- [ ] VP TechOps / VP SecOps / VP TechCompliance agents scoped to their Redmine projects.
+- [ ] AJ's access formally documented (he "has full access" per prompt — needs an access matrix in Discourse).
+
+### 2.2 Documentation (Discourse categories to populate)
+- [ ] **COO (cat 6):** operations manual, decision authority, escalation paths, daily/weekly/monthly cadence.
+- [ ] **VP SecOps:** does not exist — create (needs admin key; current API user is trust-4, **cannot create categories** — blocker).
+- [ ] **VP Compliance (75):** 0 topics — CMMC L3, STIG, ITAR program docs.
+- [ ] **KNEL-Bizops (72):** 0 topics — the business-services knowledge base.
+- [ ] **Board (76):** 0 topics.
+- [ ] Each business entity category: operations content (currently 0-1 stubs).
+
+### 2.3 Work tracking (Redmine)
+- [ ] **Project 53 (COO) / 62 (Business Services):** define versions/milestones + seed tickets. Currently empty.
+- [ ] COO workstream tickets mirroring the operational layer (not just TechOps P1-P9).
+- [ ] AI-staff scoping: each Hermes agent scoped to its Redmine project only.
+
+### 2.4 Systems / tools the COO depends on
+- [ ] **ITSM/workflow engine** (prompt.md line 101: "We really need an ITSM tool — Discourse? Windmill? Nextcloud?"). **Unresolved decision.** Windmill is packaged (✅) — strongest candidate.
+- [ ] COO Tier-1 apps deployed: Grist, PayrollEngine, KillBill, Rundeck, Comply (all not-started).
+- [ ] Credential migration to Vault (P5, critical path) — unblocks agent secret access.
+
+### 2.5 Culture / process (the "correct" part)
+- [ ] TSYSGroupAIOS framework **pushed and adopted** into all 5 gap projects.
+- [ ] Full SDLC enforced everywhere (red/green TDD, linters, CI/CD lockstep local+hosted).
+- [ ] `sectestbed-` / `preprod-` VM workflow operational for IaC testing.
+- [ ] "Gardening" loop running to prevent doc sprawl.
+
+---
+
+## 3. CTO Transition — what changes for Charles
+
+| Stop doing (COO/founder work) | Start/continue doing (CTO work) |
+|---|---|
+| Day-to-day ops decisions | R&D architecture (EngStack, Suborbital, k8s platform) |
+| Physical infra firefighting (delegate to VP TechOps agent) | Tier-4 escalation only |
+| Direct fleet config (→ KNELIAC/AWX) | SDLC/governance ownership (the framework) |
+| Ticket-level execution on P1-P9 | Mentor/oversee AI agents; review their work |
+| Building business-ops docs | Document TSYS Group component architecture (Oct onboarding task per prompt.md) |
+
+**The CTO transition is gated on the COO handoff:** you cannot stop doing COO work until AJ + the COO agent can absorb it.
+
+---
+
+## 4. Roadmap — Revised (Q3/Q4 split)
+
+### Q3: Aug 13 → Sep 30 — TechOps + Agent Identities
+
+**Workstream A — Finish Infrastructure (P1-P9, Redmine project 55, 86 open tickets)**
+*Keep executing the existing phase plan.* Critical path: **P5 Vault → P6 IaC → P9 Compliance → P7 k8s → P8 Apps.**
+- Aug 13-17: Friday onsite physical batch. Finish P1/P2/P3.
+- Aug 18-31: P5 Vault (CRITICAL), P4 monitoring UAT, P6 IaC basics, P9 STIG/CMMC seed, P7 k8s cnode rebuild.
+- Sep 1-30: P8 Cloudron app fleet (TechOps-relevant), P7 k8s apps (cluster only), compliance hardening.
+
+**Workstream B — Stand Up AI Agent Identities (NEW, first priority)**
+*Identity-first. See `agent-identity-bootstrap.md` for full plan.*
+- Week 1: User provides prerequisites (Linux `coo` account, Bitwarden account, Cloudron invites). Agent enrolls 3 Q3 identities (vp-techops, vp-secops, vp-techcompliance) via Playwright.
+- Week 2: Deploy per-agent SSH keys, set up `coo` Linux environment, smoke test (agent creates ticket, edits Discourse, opens PR from own identity).
+- Week 3+: Agents begin operating from own identities. Enroll Q4 identities (Cloudron only, no system access yet).
+
+**Workstream C — Governance & Culture (cross-cutting)**
+- Week 1-2: Push TSYSGroupAIOS template; adopt into KNELIAC (critical) + EngStack (critical).
+- Week 2-4: Adopt into KNEL-AIMiddleware (drop JOURNAL.md), football, PFVCluster (minor).
+- Ongoing: `sectestbed-`/`preprod-` VM workflow, CI/CD lockstep, gardening loop.
+
+### Q4: Oct 1 → Dec 31 — Business Ops Transition + CTO Come-Up
+
+*Charles at 80% CTO / 20% COO. Working alongside AJ (business ops lead), Patti, Courtney.*
+
+- **October (Charles's stated focus):** Figure out the K8S workload. What apps run on K8S vs Cloudron. Create the K8S repo + Redmine project.
+- **October–November:** Build the COO business-ops layer: Discourse cat 6 content, Redmine project 53 tickets, ITSM tool selection (Windmill), COO Tier-1 apps (Grist, PayrollEngine, KillBill, Rundeck, Comply), bizopprodplan handbook refresh.
+- **November–December:** Activate Q4 AI agents (COO, SVP KNEL, SVP TCTC, financial VPs). AJ UAT on COO agent + business systems. Dry-run operational week.
+- **Jan 1 2027:** Full COO handoff. Charles = 100% CTO.
+
+---
+
+## 5. Open Decisions — Updated
+
+### Resolved by user (2026-08-13)
+- ~~Business ops timeline~~ → **Q4 (Oct–Dec), full handoff Jan 1 2027.** Q3 = TechOps only.
+- ~~Agent identity approach~~ → **Identity-first.** Stand up Cloudron + Bitwarden accounts as the very first action.
+- ~~COO/CTO split in Q4~~ → **80% CTO / 20% COO.** AJ leads business ops with Patti + Courtney.
+
+### Still open
+
+1. **Cloudron SSO** — Are Gitea/Discourse/Redmine Cloudron-managed (auto-SSO)? Or standalone? **This determines provisioning complexity.** *(blocking agent bootstrap)*
+2. **Discourse admin key** — API user is trust-4, **cannot create categories** (VP SecOps etc.). Provide admin key or create via web UI. *(blocking VP SecOps setup)*
+3. **Linux account model** — Single `coo` account (recommended Q3) vs per-agent accounts (stronger audit). *(see bootstrap spec D1)*
+4. **Agent runtime** — Crush per-agent config dirs (recommended Q3) vs Hermes vs OpenWebUI. *(see bootstrap spec D3)*
+5. **K8S scope in P7** — Is P7 (due Aug 31) just cluster bootstrap, with app deployment deferred to October (your stated K8S focus)? *Rec: yes — P7 = cluster ready, apps = October.*
+6. **ITSM tool** — Windmill (packaged) vs Discourse-only vs Nextcloud. *(deferred to Q4 but worth deciding early)*
+7. **bizopprodplan repos** — Legacy mdBook stubs (KNEL one still says "CIO Documentation"). Refresh in place during Q4 or treat as superseded by Discourse?
+
+---
+
+## 5.5 SecOps & Compliance Context (from user's VP SecOps notes)
+
+This is the security architecture that the TechOps agents operate within. Drives P5 (PKI/Vault) and P9 (Security/Compliance).
+
+### Compliance targets
+- **CMMC Level 3** is the goal (not L1 or L2)
+- **Full STIG compliance** — highest level (mission critical classified)
+- **CUI minimum everywhere** — all systems, no exceptions
+- **ITAR** — governs all technical operations
+- **Multi-tenant** — isolation between business entities (RackRental franchisees, Suborbital, TCTC, etc.)
+- **Eventually productized** as a Your Dream Name Here (YDN) offering — the compliance stack itself becomes a product
+
+### Zero trust access model
+- **NetBird** (primary zero-trust mesh) + **Tailscale** (existing, nested solution)
+- **Apple account** referenced (likely for iPad-based access)
+- All access through zero-trust — no flat network trust
+- **Remote access SCIF** capability needed
+- **Keycloak** deployed — initial setup done for NetBird integration
+- Cloudron IdP has "simple groups, lacks granular permission levels at least via GUI" — Keycloak fills the RBAC gap
+
+### FOCI concern (Foreign Ownership Control Influence)
+- **Netcup** (Cloudron VPS host) is a **German company**, even though hosting in Reston VA
+- Question: can a VPS attest back to PFV (the on-prem cluster)?
+- This affects what data/workloads can live on Cloudron vs must stay on-prem
+
+### CA / PKI (drives P5)
+- **Nitrokey HSM** as the root CA hardware
+- SSH certificates (not just keys) for day-to-day operations via Ansible
+- Custom **Ubuntu 24.04 ISO** needed as the Cloudron base image
+
+### SSH key migration (directly impacts agent identity bootstrap)
+
+| Current state | Target state |
+|---|---|
+| **Ultix-highside** (Win11 Surface, no local admin, Zoc terminal) — on-disk OpenSSH key present on every system except recent deploys | Single Bitwarden SSH key + BW agent (not yet working on Windows) |
+| **This VM's key** — on a subset of machines, being expanded | Replaced by Bitwarden key/agent |
+| **iPad secure enclave key** — used via Blink, public key on all accessible systems | **Keep** — one of two authorized broad-access keys |
+| | **iPad enclave key + Bitwarden key only.** No key material on disk. |
+
+**Agent identity impact:** Agent SSH keys (Phase 3 of bootstrap) must use the **SSH certificate** model, not raw key deployment. The CA (Nitrokey HSM) signs agent certificates. This is stronger than key-based auth and aligns with the "day-to-day operations via Ansible and SSH certificates" target.
+
+### Bitwarden structure
+- **Three BW accounts** currently exist (RCEO owns all creds/orgs/collections)
+- **Envwarden** in use
+- A **fourth BW account** (COO/AI agents) will be created per the bootstrap plan
+- AJ has **broad Bitwarden access** (business continuity)
+
+### Business continuity
+- **Patti** — iPad
+- **Remy** — iPhone
+- **Albert (AJ)** — broad Bitwarden access
+
+### Identity/IAM gaps to resolve
+1. **Cloudron IdP lacks granular RBAC** — Keycloak must fill this gap for agent scoping
+2. **SSH certificate infrastructure** — Nitrokey HSM → SSH CA → certificate signing for agents + humans
+3. **Custom Ubuntu 24.04 Cloudron ISO** — needed before app fleet deployment
+4. **NetBird/Tailscale nesting** — agents must operate within the zero-trust mesh, not bypass it
+
+---
+
+## 5.6 OAM — Environmental Monitoring (from user's notes)
+
+This is the physical/environmental monitoring layer that feeds into P4 (Monitoring & Instrumentation).
+
+| Sensor/System | Method | Purpose |
+|---|---|---|
+| **DRAC** (Dell Remote Access Controller) | IPMI/Redfish API | Out-of-band management of Dell hosts (console, power, hardware status) |
+| **SNMPd on non-PowerEdge** | SNMP polling | Hardware health on non-Dell systems (Pi, custom builds) |
+| **Sensors** (lm-sensors / TEMPer USB) | Direct probe | CPU temp, ambient temp, fan speed |
+| **Home Assistant** | HA integrations | All site environmental monitoring (temp, humidity, presence, power events) |
+| **Beszel** | Agent-based | RAM / CPU / disk metrics on all hosts |
+
+**Integration with agent identity:** Environmental monitoring systems (Home Assistant, Beszel, DRAC interfaces) are Cloudron-managed or on-prem. Agent identities need scoped access to these for the vp-techops agent (alerting, dashboard) and vp-secops agent (security event correlation).
+
+**Current state:**
+- TEMPer USB probes — Redmine #341 (Friday onsite batch)
+- Tripp Lite UPS integration — #372, #439
+- UNPoller (UniFi monitoring) — deployed, placeholder creds
+- Beszel — agent installed on hosts, dashboard accessible
+- Home Assistant — planned, not yet deployed
+- SNMPd — deployed via KNELIAC `system_config` role
+- LibreNMS — the poller/correlation layer (tsys-librenms)
+
+---
+
+## 6. Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| **Vault (P5) slips** → agents can't access secrets | High | High | P5 is critical path; prioritize immediately |
+| **Discourse admin blocker** → can't create VP SecOps category | High | Med | User creates categories via web UI (confirmed will do) |
+| **k8s cnodes not rebuilt** → P7 slips → Oct K8s focus starts late | Med | High | Rebuild cnodes in Aug onsite window |
+| **SSH cert infrastructure not ready** → agents can't use cert-based SSH | Med | High | Nitrokey HSM SSH CA setup as part of P5 |
+| **FOCI concern unresolved** → uncertain what data can live on Cloudron | Med | High | Decide data classification boundaries early in Q4 |
+| **Cloudron RBAC limits** → can't scope agent access granularly | High | Med | Keycloak fills the gap; wire before agent activation |
+| **Custom Ubuntu ISO not built** → blocks Cloudron app fleet | Med | Med | Prioritize after P1-P3 physical work |
+| **Scope creep** — P1-P9 alone is 86 open tickets | High | Med | Keep Q3 TechOps / Q4 Business split strict |
+| **Hermes full-host-access security** → COO agent blast radius | Med | Critical | Define trust model before Q4 COO agent activation |
+
+---
+
+## 7. Immediate Next Actions
+
+### Confirmed and ready to execute
+1. **Agent identity bootstrap** (see `agent-identity-bootstrap.md`) — *awaiting user prerequisites (coo account, BW account, Cloudron invites)*
+2. **Push TSYSGroupAIOS** to Gitea + create as template repo — *ready now*
+3. **Continue P1-P9 execution** — Friday onsite batch, then P5 Vault critical path
+
+### Needs user input first
+4. **Discourse VP SecOps category** — user creates via web UI (API user can't)
+5. **Cloudron invites** for the 3 Q3 agents — user generates
+6. **`coo` Linux account + Bitwarden account** — user creates
+
+### Q4 prep (not yet started)
+7. **K8S workload planning** — user's stated October focus. Need: K8S repo, Redmine project, app triage (Cloudron vs K8S)
+8. **COO business-ops layer** — Discourse cat 6 content, Redmine project 53 tickets, ITSM tool decision
+9. **Custom Ubuntu 24.04 Cloudron ISO** — dependency for app fleet
+10. **SSH certificate infrastructure** — Nitrokey HSM → SSH CA → cert signing