docs: initial commit — Q2/Q3 transition planning docs
Planning documents for TSYS Group's COO→CTO handoff and AI agent
identity architecture. Shared publicly as a bootstrapping reference.
Includes: org prompts, transition map, agent identity bootstrap plan,
TechOps/K8s/SecOps context notes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
# Gitignore for org-buildout
|
||||
# This is a docs-only repo — no secrets, no code artifacts
|
||||
|
||||
*.env
|
||||
.crush/
|
||||
*.tmp
|
||||
@@ -0,0 +1,40 @@
|
||||
# org-buildout
|
||||
|
||||
Building an AI-native organization — shared transparently as a bootstrapping reference.
|
||||
|
||||
This repository documents the full thought process behind TSYS Group's transition from a solo-founder operating model to an AI-agent-augmented organization. It captures the planning, architecture decisions, and operational design work behind:
|
||||
|
||||
- **COO → CTO transition** — delegating operations to AI agents + a human COO
|
||||
- **AI agent identity architecture** — Cloudron SSO, Bitwarden credential management, per-agent RBAC
|
||||
- **Infrastructure buildout** — Proxmox fleet, k8s cluster, monitoring, zero-trust networking
|
||||
- **Security/compliance targets** — CMMC L3, STIG, ITAR, zero trust
|
||||
|
||||
## Why public?
|
||||
|
||||
My brand (@reachableceo) is about reachability and transparency. Most organizations hide their internal operations. I'm sharing the full process — the messy early stages, the planning docs, the architecture decisions — because I believe the future of work involves AI agents as first-class organizational members, and showing how that gets built is more valuable than hiding it.
|
||||
|
||||
## Documents
|
||||
|
||||
| Document | What it covers |
|
||||
|---|---|
|
||||
| [`prompt.md`](prompt.md) | Q2/Q3 2026 planning notes — org chart, AI team roles, goals, SDLC/process requirements |
|
||||
| [`prompt-techops.md`](prompt-techops.md) | IaC testing model — sectestbed/preprod VM workflow |
|
||||
| [`prompt-k8s.md`](prompt-k8s.md) | K8s vision — VCluster, Rancher, multi-tenancy, workload catalog |
|
||||
| [`transition-map.md`](transition-map.md) | Full transition map — current state across Redmine/Discourse/Gitea/infrastructure, Q3/Q4 timeline, risk register, open decisions |
|
||||
| [`agent-identity-bootstrap.md`](agent-identity-bootstrap.md) | Agent identity provisioning plan — Cloudron enrollment, API key generation, Bitwarden credential architecture, Playwright automation design |
|
||||
|
||||
## Context
|
||||
|
||||
- **TSYS Group** is the overall organization (multiple business entities under one umbrella)
|
||||
- **Known Element Enterprises (KNEL)** owns and operates all IT/business systems
|
||||
- The environment operates in ITAR/CMMC/TS/SCI compliance space
|
||||
- The infrastructure is a Proxmox bare-metal cluster in a private residence
|
||||
- The production stack runs on Cloudron (Reston VA VPS) + on-prem k8s
|
||||
|
||||
## Status
|
||||
|
||||
This is live, evolving documentation. The transition is in progress (Q3-Q4 2026).
|
||||
|
||||
## License
|
||||
|
||||
See [LICENSE](LICENSE). All content here is shared for educational/reference purposes.
|
||||
@@ -0,0 +1,382 @@
|
||||
# AI Agent Identity Bootstrapping — Provisioning Plan
|
||||
|
||||
**Status:** DRAFT — architecture confirmed, awaiting user prerequisites
|
||||
**Date:** 2026-08-13
|
||||
**Goal:** Stand up dedicated AI agent identities with proper attribution, RBAC, and audit trails. Agents own their own credentials in Bitwarden (no shared `~/.creds/` env files).
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ CLOUDRON (tsys-cloudron.knel.net) │
|
||||
│ Identity root — IdP for ALL apps except Uptime Kuma │
|
||||
│ │
|
||||
│ Agent identities (each a separate Cloudron user): │
|
||||
│ vp-techops, vp-secops, vp-techcompliance (Q3) │
|
||||
│ coo, svp-knel, svp-tctc, vp-finance, ... (Q4) │
|
||||
└────────────┬────────────────────────────────────────────┘
|
||||
│ Cloudron SSO (auto-provisions identity)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ CLOUDRON-MANAGED APPS │
|
||||
│ Gitea · Discourse · Redmine · + all other apps │
|
||||
│ │
|
||||
│ Each agent SSOs in → generates its OWN API keys → │
|
||||
│ stores in Bitwarden. Agents are fully independent. │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ BITWARDEN (dedicated "COO" account) │
|
||||
│ Credential vault — RCEO owns all creds/orgs/collections │
|
||||
│ │
|
||||
│ Collections: vp-techops/, vp-secops/, shared/, etc. │
|
||||
│ Each item: service URL, username, password, API key, │
|
||||
│ TOTP secret, SSH private key │
|
||||
│ │
|
||||
│ bw CLI authenticates via client_id/client_secret │
|
||||
│ (machine-to-machine, non-interactive) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ COO LINUX ACCOUNT (orchestration layer) │
|
||||
│ Currently: reachableceo's workstation │
|
||||
│ Soon: dedicated hardened VM on PFVCluster │
|
||||
│ │
|
||||
│ This is where Charles/AJ interact with agents. │
|
||||
│ Agents are "personalities" (Crush/Hermes sessions) │
|
||||
│ that run here, each sourcing its own creds from BW. │
|
||||
│ │
|
||||
│ NO key material on disk. SSH via BW SSH agent. │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key principles
|
||||
1. **Cloudron is the identity root.** One invite = identity everywhere (SSO auto-provisions).
|
||||
2. **Agents own their credentials.** Each agent generates its own API keys after SSO login, stores them in Bitwarden. No shared `~/.creds/*.env` files.
|
||||
3. **Agents are fully independent.** Each agent is a separate Cloudron user with its own SSO sessions, API keys, and Bitwarden collection. They do not share sessions or credentials.
|
||||
4. **`coo` Linux account is the orchestration layer** — where humans (Charles/AJ) launch and interact with agent sessions. Not where agents "live" — agents live as Cloudron identities.
|
||||
5. **No key material on disk.** All SSH private keys, passwords, API keys live in Bitwarden. SSH via BW SSH agent. (Aligns with SecOps note: only iPad enclave key + Bitwarden key.)
|
||||
|
||||
---
|
||||
|
||||
## 2. The Credential Migration (`~/.creds/` → Bitwarden)
|
||||
|
||||
### Current state (14 credential files)
|
||||
|
||||
```
|
||||
~/.creds/
|
||||
├── beszel.env ├── librenms.env ├── redmine.env
|
||||
├── discourse.env ├── oxidized.env ├── switch-creds.env
|
||||
├── grafana.env ├── phpipam.env ├── technitium.env
|
||||
├── pushover.env ├── prometheus.env ├── unifi-creds.env
|
||||
└── uptime-kuma.env
|
||||
```
|
||||
|
||||
All CLIs invoked as: `docker run --rm --env-file ~/.creds/<service>.env <cli-image> ...`
|
||||
All using `reachableceo`'s API keys. No per-agent credential separation.
|
||||
|
||||
### Target state
|
||||
|
||||
```
|
||||
Bitwarden org: "TSYS Group AI Agents"
|
||||
├── vp-techops/ (collection)
|
||||
│ ├── Cloudron (login + TOTP)
|
||||
│ ├── Gitea (API token)
|
||||
│ ├── Discourse (API key)
|
||||
│ ├── Redmine (API key)
|
||||
│ ├── SSH key (Ed25519 private key)
|
||||
│ └── <infra creds> (LibreNMS, Grafana, etc. — scoped to TechOps)
|
||||
├── vp-secops/
|
||||
│ ├── Cloudron / Gitea / Discourse / Redmine / SSH
|
||||
│ └── <security creds>
|
||||
├── vp-techcompliance/
|
||||
│ └── ...
|
||||
├── shared/ (fleet-wide creds all agents need)
|
||||
│ ├── Technitium DNS
|
||||
│ ├── Pushover
|
||||
│ ├── Switch credentials
|
||||
│ └── Beszel
|
||||
└── coo/ (Q4)
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Migration path (3 stages)
|
||||
|
||||
**Stage 1 — Parallel operation (Q3 start).** New agent identities use BW-sourced credentials. The existing `~/.creds/` files remain for `reachableceo`'s direct use. Both work side by side. No breakage.
|
||||
|
||||
**Stage 2 — Agent adoption.** Agent-facing tooling (CLI wrappers, Crush configs) source from BW. The `~/.creds/` files are only used by Charles directly. Agents never touch `~/.creds/`.
|
||||
|
||||
**Stage 3 — Full migration.** `~/.creds/` files are deleted. Charles uses Bitwarden directly too. All credential access is through BW. (Aligns with Vault migration in P5 — Vault can later wrap BW or replace it for machine secrets.)
|
||||
|
||||
### The credential sourcing layer
|
||||
|
||||
A wrapper script that replaces the `--env-file ~/.creds/` pattern:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# bw-run.sh — run a CLI command with credentials sourced from Bitwarden
|
||||
# Usage: bw-run.sh <agent> <service> <image> [args...]
|
||||
# Example: bw-run.sh vp-techops redmine redmine-cli:latest list --assigned-to-me -p 55
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
AGENT="$1"; SERVICE="$2"; IMAGE="$3"; shift 3
|
||||
|
||||
# Ensure BW session is active
|
||||
BW_SESSION="${BW_SESSION:-$(bw unlock --raw 2>/dev/null)}" || true
|
||||
export BW_SESSION
|
||||
|
||||
# Fetch credential item from Bitwarden
|
||||
ITEM_NAME="${AGENT} ${SERVICE}"
|
||||
ENV_JSON=$(bw get item "$ITEM_NAME" | jq -r '
|
||||
.login |
|
||||
"URL=\(.uris[0].uri // "")\n" +
|
||||
"USERNAME=\(.username // "")\n" +
|
||||
"PASSWORD=\(.password // "")\n" +
|
||||
(.fields[]? | "\(.name)=\(.value)\n")
|
||||
')
|
||||
|
||||
# Write to temp env file (cleaned up on exit)
|
||||
ENVFILE=$(mktemp /tmp/bw-${AGENT}-${SERVICE}-XXXXXX.env)
|
||||
trap 'rm -f "$ENVFILE"' EXIT
|
||||
echo -e "$ENV_JSON" > "$ENVFILE"
|
||||
|
||||
# Run the CLI with BW-sourced credentials
|
||||
docker run --rm --env-file "$ENVFILE" "$IMAGE" "$@"
|
||||
```
|
||||
|
||||
This preserves the existing CLI invocation pattern (`docker run --env-file ... <image>`) while sourcing credentials from Bitwarden instead of static files. Agents call `bw-run.sh vp-techops redmine ...` instead of `docker run --env-file ~/.creds/redmine.env ...`.
|
||||
|
||||
### What needs to change in each project
|
||||
|
||||
| Project | Change | Effort |
|
||||
|---|---|---|
|
||||
| **TSYSGroupAIOS** (meta) | Add `bw-run.sh` to `scripts/`. Update `BASELINE-PROMPT.md` credential references. Document the BW credential model. | Med |
|
||||
| **PFVCluster** | Update `AGENTS.md` CLI examples to use `bw-run.sh`. Add agent-profile system. Update `scripts/check-rules.sh` to enforce no `~/.creds/` refs in new code. | Med |
|
||||
| **KNEL-AIMiddleware** | Update `tooling-cli/*/AGENTS.md` to document BW sourcing. Update `KNELCredsManager` to be the BW integration point. | Med |
|
||||
| **All CLI containers** | No image changes needed — they still read env vars. The sourcing layer (`bw-run.sh`) is outside the container. | None |
|
||||
|
||||
---
|
||||
|
||||
## 3. Identity Roster
|
||||
|
||||
### Q3 — TechOps Agents (stand up NOW)
|
||||
|
||||
| Agent | Cloudron user | Redmine scope | Gitea scope | Discourse scope |
|
||||
|---|---|---|---|---|
|
||||
| **vp-techops** | vp-techops@<cloudron-domain> | 55, 59 | KNEL, TechnicalOperations | 74, 20 |
|
||||
| **vp-secops** | vp-secops@<cloudron-domain> | 55 (security) | KNEL | 74 + new VP SecOps cat |
|
||||
| **vp-techcompliance** | vp-techcompliance@<cloudron-domain> | 55 (compliance) | KNEL | 75 |
|
||||
|
||||
### Q4 — Business Agents (enroll in Cloudron now, activate in Q4)
|
||||
|
||||
| Agent | Redmine scope |
|
||||
|---|---|
|
||||
| coo | 53, 62, 77 |
|
||||
| svp-knel | 62, 55 (read) |
|
||||
| svp-tctc | 15 (TCTC), 31 (RedWFO) |
|
||||
| vp-finance, vp-accounting, vp-investing, vp-treasury, vp-trading | 15 (TCTC) |
|
||||
|
||||
**Recommendation:** Enroll ALL identities in Cloudron during Q3 (cheap — just invite acceptance). Only provision API keys + system access for the 3 Q3 agents. Q4 agents get activated when business ops work begins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Provisioning Pipeline
|
||||
|
||||
### Phase 0 — Prerequisites (USER provides)
|
||||
|
||||
| # | Item | Detail |
|
||||
|---|---|---|
|
||||
| 1 | **Linux `coo` account** | `useradd -m -s /bin/bash coo`; add to `docker` group; `mkdir ~/.ssh ~/.creds ~/projects` |
|
||||
| 2 | **Bitwarden "COO" account** | Dedicated BW account. RCEO owns the org. Create collections per agent. |
|
||||
| 3 | **BW API credentials** | `client_id` + `client_secret` for non-interactive `bw login --apikey`. Place in a file the agent can read. |
|
||||
| 4 | **Cloudron invite manifest** | Text file, one line per identity: `agent-name | display-name | invite-link | priority` |
|
||||
| 5 | **Discourse admin key** *(for VP SecOps category)* | Current API user (trust-4) cannot create categories. User creates VP SecOps category via web UI OR provides admin key. |
|
||||
|
||||
### Phase 1 — Cloudron Identity Enrollment (Playwright automation)
|
||||
|
||||
For each identity in the manifest:
|
||||
|
||||
```
|
||||
1. Launch Playwright Docker (mcr.microsoft.com/playwright:v1.52.0-noble)
|
||||
2. Navigate to Cloudron invite link
|
||||
3. Generate strong password (bw generate -uluns --length 32)
|
||||
4. Fill password fields, submit → identity created
|
||||
5. Enable 2FA:
|
||||
a. Navigate to Cloudron account security settings
|
||||
b. Initiate TOTP enrollment
|
||||
c. Extract TOTP secret from QR code
|
||||
d. Store TOTP secret in Bitwarden
|
||||
e. Confirm with current TOTP code (bw code from stored secret)
|
||||
6. Store Cloudron credential in Bitwarden:
|
||||
- Collection: <agent-name>
|
||||
- Name: "<agent-name> Cloudron"
|
||||
- URL: https://tsys-cloudron.knel.net
|
||||
- Username: <agent-name>@<domain>
|
||||
- Password: <generated>
|
||||
- TOTP: <secret>
|
||||
```
|
||||
|
||||
**Result:** Agent identity exists in Cloudron. SSO works for all managed apps. No per-system account creation needed — SSO handles it.
|
||||
|
||||
### Phase 2 — API Key Generation (agent logs in via SSO, creates keys)
|
||||
|
||||
For each agent, for each system (Gitea, Discourse, Redmine):
|
||||
|
||||
```
|
||||
1. Launch Playwright
|
||||
2. Navigate to the app URL → Cloudron SSO redirect
|
||||
3. Complete SSO login (use agent's Cloudron creds from BW + TOTP from BW)
|
||||
4. First-time SSO → account auto-provisioned in the app
|
||||
5. Navigate to API key / access token settings:
|
||||
- Gitea: Settings → Applications → Generate New Token (scopes: api, repo, read:org)
|
||||
- Discourse: Preferences → API Keys → (may need admin to create user API key)
|
||||
- Redmine: My Account → API access key → Show
|
||||
6. Copy the generated key
|
||||
7. Store in Bitwarden:
|
||||
- Collection: <agent-name>
|
||||
- Name: "<agent-name> <SystemName>"
|
||||
- URL: <system URL>
|
||||
- Password field: <API key>
|
||||
8. Verify: use the key to call the system's API (curl or CLI)
|
||||
```
|
||||
|
||||
**After Phase 2, each agent has:**
|
||||
- Cloudron identity (login + TOTP) in BW
|
||||
- Gitea API token in BW
|
||||
- Discourse API key in BW
|
||||
- Redmine API key in BW
|
||||
- Ability to operate independently in all three systems
|
||||
|
||||
### Phase 3 — SSH Key Provisioning
|
||||
|
||||
```
|
||||
1. Generate Ed25519 keypair per agent: ssh-keygen -t ed25519 -f /tmp/<agent>-ed25519 -N ""
|
||||
2. Store PRIVATE key in Bitwarden (as "Secure Note" or BW SSH Agent item)
|
||||
3. Deploy PUBLIC key to fleet via KNELIAC:
|
||||
- Add to inventory/group_vars/all.yml managed_users
|
||||
- Run: ansible-playbook playbooks/setup_new_system.yml -t security_ssh --limit <targets>
|
||||
4. Delete private key from /tmp (lives only in BW)
|
||||
5. Configure BW SSH Agent on coo account: export SSH_AUTH_SOCK=... (bw agent)
|
||||
```
|
||||
|
||||
### Phase 4 — Agent Profile System (on `coo` Linux account)
|
||||
|
||||
Each agent has a profile that sets its identity context:
|
||||
|
||||
```bash
|
||||
# ~/agents/vp-techops/profile.sh
|
||||
export AGENT_NAME="vp-techops"
|
||||
export AGENT_DISPLAY="VP TechOps"
|
||||
export AGENT_EMAIL="vp-techops@turnsys.com"
|
||||
export AGENT_BW_COLLECTION="vp-techops"
|
||||
export GIT_AUTHOR_NAME="VP TechOps"
|
||||
export GIT_COMMITTER_NAME="VP TechOps"
|
||||
export GIT_AUTHOR_EMAIL="$AGENT_EMAIL"
|
||||
export GIT_COMMITTER_EMAIL="$AGENT_EMAIL"
|
||||
export REDMINE_PROJECT_SCOPE="55,59"
|
||||
export GITEA_ORG_SCOPE="KNEL,TechnicalOperations"
|
||||
export DISCOURSE_CAT_SCOPE="74,20"
|
||||
# SSH: BW agent provides the key for this agent
|
||||
```
|
||||
|
||||
Switching agent context:
|
||||
```bash
|
||||
source ~/agents/vp-techops/profile.sh # become vp-techops
|
||||
source ~/agents/vp-secops/profile.sh # become vp-secops
|
||||
```
|
||||
|
||||
### Phase 5 — Operational Environment
|
||||
|
||||
```
|
||||
1. Clone repos into /home/coo/projects/ (or the future hardened VM)
|
||||
2. Copy/adopt TSYSGroupAIOS framework (check-rules.sh, hooks, etc.)
|
||||
3. Install bw-run.sh credential sourcing layer
|
||||
4. Set up per-agent Crush config directories:
|
||||
~/.config/crush/vp-techops/
|
||||
~/.config/crush/vp-secops/
|
||||
5. Configure tea CLI per agent (separate logins = separate tokens)
|
||||
6. Install git hooks (ticket-gate, pre-commit, pre-push)
|
||||
7. Smoke test per agent:
|
||||
- Create a Redmine ticket (appears as agent identity)
|
||||
- Edit a Discourse topic (appears as agent identity)
|
||||
- Open a Gitea PR (appears as agent identity)
|
||||
- SSH to a host (appears in auth log as agent's key)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Playwright Automation Design
|
||||
|
||||
The Playwright script handles the browser-driven steps (Phases 1-2). Containerized — no browser on host.
|
||||
|
||||
```
|
||||
provision-agent/
|
||||
├── Dockerfile # Based on mcr.microsoft.com/playwright:v1.52.0-noble
|
||||
├── provision-agent.py # Main script
|
||||
├── bw-helper.py # Bitwarden CLI wrapper (gen password, store item, get TOTP)
|
||||
└── agents.yaml # The manifest (invite links + agent config)
|
||||
```
|
||||
|
||||
**Flow:**
|
||||
```python
|
||||
for agent in manifest:
|
||||
# Phase 1: Cloudron enrollment
|
||||
browser → invite_link
|
||||
fill password (bw generate)
|
||||
enable 2FA → extract TOTP secret → store in BW
|
||||
|
||||
# Phase 2: API keys (per system)
|
||||
for system in [gitea, discourse, redmine]:
|
||||
browser → system_url → SSO redirect → login with BW creds + TOTP
|
||||
navigate to API key page → generate → copy
|
||||
store in BW
|
||||
|
||||
# Verify
|
||||
for system in [gitea, discourse, redmine]:
|
||||
api_call(system, key_from_bw) → assert success
|
||||
```
|
||||
|
||||
**Security:** Playwright runs in Docker, no persistent browser state. Credentials are generated/stored via `bw` CLI (mounted into container). No credentials in browser memory after the script exits.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security & Compliance Posture
|
||||
|
||||
| Control | How this design satisfies it |
|
||||
|---|---|
|
||||
| **CMMC IA-2(1)** (MFA for all accounts) | Every agent identity has TOTP via Bitwarden |
|
||||
| **CMMC AU-2/AU-12** (audit events) | Every action attributable to specific agent identity (Redmine, Discourse, Gitea, SSH auth logs) |
|
||||
| **CMMC AC-2/AC-3** (least privilege) | Each agent scoped to its Redmine projects, Gitea orgs, Discourse categories, and fleet SSH access |
|
||||
| **CMMC IA-5(1)** (authenticator management) | No key material on disk. All in Bitwarden. SSH via BW agent. |
|
||||
| **Shared credential elimination** | Each agent has its own API keys and SSH keys — no more single key on "every single system" |
|
||||
| **ITAR/SCIF alignment** | Per-identity access control + audit trail for classified-adjacent systems |
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Timeline
|
||||
|
||||
| Week | What | Who |
|
||||
|---|---|---|
|
||||
| **Week 1** | User provides Phase 0 prerequisites. Agent builds Playwright automation + bw-run.sh. Executes Phase 1-2 for 3 Q3 agents. | User + Agent |
|
||||
| **Week 2** | Phase 3 (SSH keys via KNELIAC). Phase 4-5 (agent profiles, coo env, smoke tests). Enroll Q4 identities in Cloudron (Phase 1 only). | Agent |
|
||||
| **Week 3+** | Agents begin operating from own identities on P1-P9 work. Charles reviews as himself. Agents @mention Charles in Redmine/Discourse. Agents open PRs for review. | Agents + Charles |
|
||||
|
||||
---
|
||||
|
||||
## 8. What I Need From You to Start
|
||||
|
||||
1. **Create the `coo` Linux account** on this workstation.
|
||||
2. **Create the Bitwarden account** for AI agents; give me `client_id` + `client_secret`.
|
||||
3. **Generate Cloudron invite links** for the 3 Q3 agents (vp-techops, vp-secops, vp-techcompliance). Put in a manifest file:
|
||||
```
|
||||
vp-techops | VP TechOps | https://tsys-cloudron.knel.net/invitation/<token> | Q3
|
||||
vp-secops | VP SecOps | https://tsys-cloudron.knel.net/invitation/<token> | Q3
|
||||
vp-techcompliance | VP TechCompliance | https://tsys-cloudron.knel.net/invitation/<token> | Q3
|
||||
```
|
||||
4. **Resolve the Discourse admin blocker** — create the VP SecOps category via web UI (or give me an admin API key).
|
||||
5. *(Optional)* Generate invites for Q4 agents too (coo, svp-knel, svp-tctc) — Phase 1 only, no system access until Q4.
|
||||
|
||||
Once I have items 1-3, I build the Playwright automation and execute Phases 1-2.
|
||||
@@ -0,0 +1,31 @@
|
||||
Prompt - k8s
|
||||
|
||||
(See VpTechOps and VpSecOps discourse categories introduction posts for vital organizational context )
|
||||
|
||||
## vcluster
|
||||
|
||||
I want to utilize VCluster to slice up compute in a secure and scaleable manner. I have a variety of workloads , tenants and security requirements.
|
||||
|
||||
## rancher
|
||||
|
||||
I need humans , AI agents to be able to work together on the clusters in a RBAC, OIDC SSO 2fa enforced , fully logged and audited way. Via cli , clickops , API . Rancher Platform seems to be the best way to do that.
|
||||
|
||||
|
||||
- auth to OIDC/cloudron/keycloak.
|
||||
- multi tenancy
|
||||
- different security levels
|
||||
- ITAR
|
||||
- SCIF
|
||||
|
||||
## Workloads
|
||||
|
||||
- RackRental (on demand network labs SAAS). Via container lab. This will have sub tenants who are franchisees with a revenue share agreement. They’ll provide compute / power / bandwidth , hook up to NetBird. This is the lowest confidentialliry environment, but the highest risk (essentially unknown counter parties (both the franchiseees (who probably aren’t hostile as they want to make money , but they may not be overly sophisticated operators ) and the customers (who we must presume are actively hostile and using the most advanced offensive cyber capabilities). As we are running essentially a horizontally scaled distributed cloud , it will be attacked by transnational criminal organizations looking to utilize it for large scale criminal operations.
|
||||
- Suborbital Systems. Engineering workloads. Ranging from FLO (MorsePod , consumer electronics product which will be the entry point to the CommonsNet) to proprietary
|
||||
- HFNOC
|
||||
- HFNFC
|
||||
- RWSCP
|
||||
- TCTC (The Campus Trading Company).
|
||||
- and RedWFO (RWSCP Family Office) . Workloads of these two entities are mission critical Highly sensitive and proprietary and can pre empt other workloads.
|
||||
- Starting Line Productions .
|
||||
- Rogue Technologies . They run a franchisee instance of RackRental and STLP. They provide compute / storage / network . They also have engineering workloads for customers.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
|
||||
- When working on IAC, utilize the corresponding sectestbed- vm. The intent of those is to be snapshotted to the base state (joined to tailscale, registered in beszel , ssh keys in place). Over time the base state will change (ie domain joined , baseline STIG remediations applied etc) and the delta of tested code will be smaller. However , at any time , a new functional area vm can need to be stood up , and it will start in that ultra basic base state and have the roles applied.
|
||||
|
||||
- Preprod- vm machines are for testing upgrades to new versions of vendor software. They’ll always have a snapshot of current prod. This is… complicated and will need work stream specific discussion on how to snapshot , rollback etc. also , compliance mitigations may need to go through some combination of sectestbed- and preprod- testing. How compliance and upgrade work streams progress is somewhat TBD , but they’ll need to be somewhat in lockstep from time to time. Redmine gantt and overall dependency relationship modeling will be very heavily used here.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Q2 2026 Body of Work
|
||||
|
||||
I need to have all COO functions handed off to AI and Albert J Lebsch (aka AJ) (TSYS Group COO) (he has been on standby for a few months as I built out the core IT physical infrastructure) by end of September 2026.
|
||||
I MUST be fully in CTO role by October 1st 2026.
|
||||
|
||||
AJ has full access to all business/IT systems of TSYS Group. He will be overseeing the AI agents.
|
||||
|
||||
Here is the AI team roles/agents that need to be stood up with Cloudron accounts and logged into all role appropriate systems:
|
||||
|
||||
## COO
|
||||
|
||||
COO will be the main agent that AJ interacts with day to day. It will serve as his assistant and help him orchestrate all business operations. AJ reports directly to the board of directors, as do I as CTO. We are at the same level of the organization. He handles operations, i handle R&D. I've been handling operations as the founder and am now fully delegating.
|
||||
|
||||
### SVP KNEL
|
||||
|
||||
SVP KNEL is senior vice president of Known Element Enterprises (KNEL). KNEL runs and owns all IT/business systems and services for TSYS Group. TSYS Group is the name of the overall organization. I need to document in Discourse all of the TSYS Group components and will do so in October when I on-board as CTO fulltime.
|
||||
|
||||
Anyway, here are the reports to SVP KNEL. The idea is that SVP KNEL/SVP TCTC will synthesize policy level and strategic information to COO. COO/AJ will make the necessary decisions and also inform/consult the board of directors as necessary.
|
||||
|
||||
#### VPTechops
|
||||
#### VPSecOps
|
||||
#### VPTechCompliance
|
||||
|
||||
### SVP TCTC
|
||||
|
||||
- VPFinance
|
||||
- VPAccounting
|
||||
- VPInvesting
|
||||
- VPTreasury
|
||||
- VPTrading
|
||||
|
||||
# Overall goals for rest of August :
|
||||
|
||||
1. Have PFV Proxmox / Network deployment , configuration, tuning completed. Including the deployment of Proxmox onto PFV-tsys2 , and addressing the onsite physical wiring issues. This is the foundation for the heavy compute resources that Known Element Enterprises will provide to TSYS Group . I believe this sub project deliverable is at about 80% done.
|
||||
|
||||
2. Have all “pets” support servers in place (built out through a combination of interactive AI sessions doing work via tickets, using git heavily , documenting in discourse , (in sandbox / preprod ) using Ansible via AWX for the prod tier (and also in the non prod after the core functionally has been validated via iteration) and fully operational. This is :
|
||||
tsys-ca
|
||||
PFV-netinfra-01 and 02
|
||||
|
||||
I believe this is about 80% done. The main remaining deliverable is the (specific) core functionality as IAC portion (and also setting up test environment on preprod-* to test changes (including an isolated proxmox network for DHCP)) and overall (and any service specific) hardending testing on sectestbed-*.
|
||||
|
||||
3. Have full up / down alerting , instrumentation coverage of all systems and services. Across
|
||||
4.
|
||||
|
||||
## Overall goals for September:
|
||||
|
||||
Have all applications and services deployed on Cloudron/K8S
|
||||
|
||||
## K8S
|
||||
|
||||
See k8s.md for my core notes on k8s for TSYS Group. It needs to be fleshed out in discourse discussion and then turned into redmine tickets for reduction to practice.
|
||||
|
||||
## Meta Considerations
|
||||
|
||||
- full SDLC
|
||||
- I’m architect / manager / tier 4 technical subject matter expert
|
||||
- We’ve been too fast / loose with building out production infrastructure. My fault. Fixing that now
|
||||
|
||||
We are moving into full production / revenue generation operations at PFV. As such, we must adopt proper, best common practices and a full software development/test/release life cycle processes.
|
||||
|
||||
Here are my notes on what that looks like for TSYS Group operating with a mix (90% AI agents, 10% humans). This needs to be documented in discourse and reduced to practice via Redmine tickets and code.
|
||||
|
||||
- Use atomic commits, conventional style commit messages. Commit often. Push when you have completed a logical unit of work. Develop in main/master. utilize the tea cli tool for making pull requests from main/master branch to release branch.
|
||||
I’m open to per project contextualized discussion about branching approaches as needed for that project. Generally , my advice is work smart off main/master only.
|
||||
If you want to utilize branches on the workstation to move quickly and have checkpoints , go for it. Often times it’s faster to use branches to be able to more easily iterate or explore a few ideas or not have to mess around with stashing etc. Generally I dont want multiple branches in git remote, and certainly not without good reason and human approval.
|
||||
|
||||
- Ci/cd. Generally I have a strong preference for the local workstation to be able todo whatever ci/cd process is needed as well as (be able to) utilize our gitea ci / cd infrastructure. They should be maintained in lockstep for all projects. The further left we can move ci/ cd , the better. We have a mix of developers and agents. Some running inside the hosted security / convenience boundary , some running on their own beefy workstations. Use docker / kubernetes for everything. It can be a cluster of 1 or 100 just the same. Don’t presume any level of scale. Containers are containers , k8s is k8s.
|
||||
|
||||
- Approval gates , peer review etc
|
||||
|
||||
- Redmine is our single system of record for ALL project work. You have access to the redmine-cli tool. Use it. See below for discussion of discourse. Same cultural rules apply here.
|
||||
|
||||
- Discourse is our single system of record for documentation. It’s all markdown. Gitea has been where docs are , alongside code. This isn’t what we do on the engineering side of the house , and I want to stop doing in operations and move to the same model we use in engineering. No gitea .md files , no gitea issues (see previous point about redmine), no gitea wikis. Up to now , git has been your only “api”. You now have the discourse-cli tool , use it.
|
||||
|
||||
- Portability and reproducibility by anyone , not needing awx as a prerequisite requirement (an optional nice to have but not mandatory at least for bootstrap )
|
||||
|
||||
- Use status.md instead of storing todos in context / harness todo tool . It’s fine to use the harness todo tool for tracking current work . (See earlier points about token efficiency). Remember , you fully own status.md. I only consume it. Also, see earlier point about heavily using redmine. Status.md isn’t the system of record. It’s a durable git tracked scratchpad used for token efficency purposes. High fidelity tactical notes as you work for input to commit log / PR / redmine updates.
|
||||
|
||||
- Stop thinking so much on your own. You are not alone. You are my employee. You can ask me for guidance. I want you to work on getting to code/output much faster. Explore with code. Gather ground truth. You are burning tokens way too much for things that can be answered very quickly/easily with code. Also, you need to be very comfortable to ask me questions. Use a questions-v(x).md file that i can review/edit inline. Gathering questions in a git tracked way is imperative. Questions/answers/decisions and context around them is often more important than the code itself. Make sure to capture and synthesize to Discourse/Redmine. Dont over think/self debate/ruminate. Just gather data either from me , or via code or both. The token burn is totally unacceptable. I have a quota and cant keep burning through it.
|
||||
|
||||
- Unless I specifically tell you todo so , never interrupt what you are working on to pivot the whatever thing I tossed to you in another conversation turn. Track it in an inbox section in status.md. If it’s materially different , spin up a redmine ticket.
|
||||
|
||||
- Utilize belt and suspenders approach to protocol enforcement. AGENTS.md , git pre commit / pre push hooks.
|
||||
|
||||
- Run a routine “gardening” loop. I’ve noticed that you tend to sprawl .md files all over the place. You are very disciplined with code , not so much with docs.
|
||||
|
||||
- Keep docs / code / tests in sync at all times. Oh, git .md files can and should exist and be stubs to the relevant discourse urls.
|
||||
|
||||
- Utilize red / green TDD for all code.
|
||||
|
||||
- Utilize linters on all code as soon as possible. Again , be token efficient. Don’t burn tokens parsing huge blocks of code. Farm that out to deterministic tooling. Crush may have LSP wired up for the languages you are working in (both code and docs ). (This includes prose lining , but LSP doesn’t exist for that as far as I know) (so farm out to docker image). If an LSP isn’t wired up, pull a docker image and use it to lint the code.
|
||||
|
||||
- Speaking of docker do all of your development work in a container. Either a custom one , or off the shelf or a mix. You may docker pulll whatever you need without asking me first. Be smart about naming containers. Don’t ever use the default naming method docker uses. Always name it with a project prefix. Use docker compose and have hook scripts to bring up / down. Scan my ~/projects directory for a number of examples of lifecycle scripts we’ve worked on. While your at it , look at the AGENTS.md files as well. Generally my projects follow similar patterns , and I adopt same approaches with a bit of mix and match.
|
||||
|
||||
- Speaking of farming out . Utilize sub agents as you would a sub contractor. Not as staff augmentation , but as a way to get a well defined deliverable in parallel while you do other work.
|
||||
|
||||
- full SDLC support (new trackers / statuses etc )
|
||||
- Sub projects (IAC, networking , datacenter etc )
|
||||
- Maybe split along CMMC boundaries / IT functional areas and levels (admin / engineer / architect )?
|
||||
- Need to think about AI staff (individual Hermes agents ) with cloudron accounts and being scoped to their respective Redmine projects only.
|
||||
- We really need an ITSM tool / system / workflow engine (discourse ? Windmill ? Nextcloud ?)
|
||||
-
|
||||
@@ -0,0 +1,311 @@
|
||||
# COO Handoff & CTO Transition — Master Map
|
||||
|
||||
**Status:** DRAFT for review (working session artifact, not yet synthesized to Discourse/Redmine)
|
||||
**Date:** 2026-08-13
|
||||
**Deadline:** 2026-09-30 ("Potential to Kinetic Ready" + COO handoff, 48 days / "45 days")
|
||||
**Author session:** orientation/synthesis pass over Q3 prompts, all project trees, Gitea, Discourse, Redmine
|
||||
|
||||
---
|
||||
|
||||
## 0. Thesis (the one thing to internalize)
|
||||
|
||||
**Revised 2026-08-13:** Timeline split confirmed with user.
|
||||
|
||||
| Phase | Window | Focus | Who |
|
||||
|---|---|---|---|
|
||||
| **Q3 remainder** | Aug 13 → Sep 30 | TechOps finish (P1-P9) + **stand up AI agent identities** | Charles + AI agents |
|
||||
| **Q4** | Oct 1 → Dec 31 | Business ops transition + CTO come-up (80% CTO / 20% COO) | Charles + AJ + Patti + Courtney + AI agents |
|
||||
| **Jan 1 2027** | — | Full COO handoff complete. Charles = 100% CTO. | AJ + COO agent run business ops |
|
||||
|
||||
The infrastructure (PFVCluster, KNELIAC, monitoring, k8s) is ~80% there and has a phase plan (P1-P9) with a 9/30 deadline. The "running datacenter" is commodity and will be fully delegated to AI. The **first action is standing up AI agent identities** (Cloudron + Bitwarden + system access) so agents can operate with proper attribution, RBAC, and audit trails. See `agent-identity-bootstrap.md`.
|
||||
|
||||
**Business ops transition is deferred to Q4.** The COO Discourse category, Redmine project 53, bizopprodplan handbook refresh, ITSM tool selection, and COO Tier-1 business apps all move to Q4. AJ gets regular briefings through Q3 and takes over business ops in Q4 with Patti and Courtney as additional resources.
|
||||
|
||||
---
|
||||
|
||||
## 1. Current-State Map
|
||||
|
||||
### 1.1 The TSYS Group org structure (as designed in Q3/prompt.md)
|
||||
|
||||
```
|
||||
TSYS Group (Board of Directors)
|
||||
├── CTO — Charles (R&D, architecture, tier-4 SME) [you, transitioning IN fulltime Oct 1]
|
||||
└── COO — AJ Lebsch (operations) [transitioning IN, oversees AI agents]
|
||||
├── SVP KNEL (Known Element Enterprises — owns ALL IT/business systems)
|
||||
│ ├── VP TechOps ← the ONLY function with real content today
|
||||
│ ├── VP SecOps ← does not exist as a category yet
|
||||
│ └── VP TechCompliance ← Discourse cat 75 exists, 0 topics
|
||||
└── SVP TCTC (The Campus Trading Company)
|
||||
├── VP Finance / VP Accounting / VP Investing
|
||||
├── VP Treasury / VP Trading
|
||||
└── (RedWFO = RWSCP Family Office, mission-critical, can preempt)
|
||||
```
|
||||
|
||||
**Supporting entities** (each with its own Gitea org + Redmine project + Discourse category, mostly stubs):
|
||||
Suborbital Systems, HFNOC, HFNFC, RackRental, Starting Line Productions, Rogue Technologies, RedWFO, RWSCP, MeetMorse/MorsePod (FLO entry to CommonsNet), EzEDA, EzPodStack, AFABN, Ap4Ap, MerchantsOfHope, ThePeerNet, sol-calc, TeamRental, SideDoorGroup, YourDreamNameHere.
|
||||
|
||||
### 1.2 Systems of Record — actual state vs intended
|
||||
|
||||
| System | Role | Actual state |
|
||||
|---|---|---|
|
||||
| **Redmine** (projects.knownelement.com) | SoR for ALL work | **55 projects** mirror the org chart. **Only project 55 (TechnicalOperations) is active: 136 tickets, 86 open.** Project 62 (Business Services), 53 (COO), 77 (TSYS Group parent), and all entity projects are **empty/stubs.** |
|
||||
| **Discourse** (community.turnsys.com) | SoR for ALL docs | **55 categories.** Only **VP TechOps (cat 74, 13 topics)** and **Progress Reports (cat 61, 81 topics)** have real content. COO (6), VP Compliance (75), Board (76), KNEL-Bizops (72) = **0 topics.** Every business-entity category = 0-1 stub topics. |
|
||||
| **Gitea** (git.knownelement.com) | SoR for executable code | **27 orgs, ~213 repos** (87 are ExternalVendorCode mirrors). Source repos: KNEL (32), reachableceo (23), Suborbital-Systems-Public (18), RWSCP (11), + ~20 entity orgs with 1-3 repos each (mostly `-bizopprodplan` mdBook stubs). |
|
||||
| **Cloudron** (Reston VPS) | PaaS for ~57 support-stack apps | **10/57 packaged (~17%).** See §1.4. |
|
||||
| **K8s** (PFVCluster bare metal) | Scalable compute | k3s HA (3 cnode + 6 workers) but **cnodes were wiped and shut down — needs rebuild.** Zero apps deployed. |
|
||||
|
||||
### 1.3 The 12 local projects (maturity + COO relevance)
|
||||
|
||||
| Project | What | Maturity | Governance | COO relevance |
|
||||
|---|---|---|---|---|
|
||||
| **PFVCluster** | Proxmox fleet + OAM + k8s bootstrap | High (most mature infra) | Excellent (full rules engine) | **High** — the compute foundation |
|
||||
| **football** (KNEL-Football) | Hardened Debian live ISO for tier-0 access | ★★★★★ (788 tests, ISO built, audited) | Exceptional | High — secure access terminal |
|
||||
| **KNELIAC** | Ansible fleet config-mgmt (replaces bash) | Active (9 roles, AWX wired) | **Weak** (80-line AGENTS.md) | High — fleet baseline |
|
||||
| **KNEL-AIMiddleware** | MCP/LSP servers for AI agents (42 svcs) | 79% (33/42 prod-ready) | Moderate | **Critical** — the agent tooling backbone |
|
||||
| **EngStack** | Hardware R&D engineering workstation (45 tools) | Active (8 tools built) | Good | Med (CTO/R&D domain) |
|
||||
| **TSYS-Cloudron** | 57-app Cloudron packaging | 17% (10/57) | Strong | **High** — COO's business apps |
|
||||
| **WorkstationStack** | Local dev support stack + SelfStack | Active (7 apps in prod) | Good | Med |
|
||||
| **hermes-rceo-streaming** | Hermes AI agent deployment | Deployed | Minimal | **High** — agent runtime (security note: full host access) |
|
||||
| **netbird** | Zero-trust VPN access | Early/stub | Minimal | Med (security-critical, underbuilt) |
|
||||
| **meta (TSYSGroupAIOS)** | Canonical agent-framework template | **Production-ready, NOT pushed** | Canonical | **Critical** — must propagate to all projects |
|
||||
| dotfiles / EngineeringWorkstation / TSYS-LocalWorkstation | Personal/scratch | — | — | Low |
|
||||
|
||||
### 1.4 Cloudron app fleet — COO-critical gap
|
||||
|
||||
**10 packaged:** Webhook, APISIX, Healthchecks, Review Board, WireViz Web, Puter, Corteza, draw.io, Windmill, InvenTree.
|
||||
**COO Tier-1 MISSING (not started):** Grist (ops spreadsheets), PayrollEngine, KillBill (billing), Rundeck (runbook automation), Comply (compliance tracking).
|
||||
**Cloudron-blocked (need K8s):** Sentry, SigNoz, DataHub, NetBox, Fleet (all need Redis/Kafka/ClickHouse).
|
||||
|
||||
### 1.5 Governance template (TSYSGroupAIOS / meta) — adoption status
|
||||
|
||||
`~/daytoday/meta` is the canonical framework: `BASELINE-PROMPT.md` (14 principles) + 10-section `AGENTS.md` template + `scripts/check-rules.sh` (10 checks) + git hooks + `hooks/ticket-gate.sh`. **Self-applying (17 PASS/0 FAIL), ready to push** to `TSYSGroupCorporate/TSYSGroupAIOS` (remote configured, not yet pushed).
|
||||
|
||||
| Project | Gap to framework |
|
||||
|---|---|
|
||||
| PFVCluster | 🟢 Minor — add BASELINE-PROMPT.md, align scripts |
|
||||
| football | 🟡 Moderate — replace custom hooks |
|
||||
| KNEL-AIMiddleware | 🟡 Moderate — drop JOURNAL.md, add pre-commit |
|
||||
| **KNELIAC** | 🔴 Critical — needs entire enforcement layer |
|
||||
| **EngStack** | 🔴 Critical — needs entire `scripts/` + governance |
|
||||
|
||||
---
|
||||
|
||||
## 2. The Central Gap (COO handoff) — what "done" requires on Oct 1
|
||||
|
||||
For AJ + the COO AI agent to actually run operations, these must be true. Today **none of them are.**
|
||||
|
||||
### 2.1 Org / role
|
||||
- [ ] COO AI agent stood up with Cloudron account, logged into all role-appropriate systems (Redmine, Discourse, Gitea, Cloudron, monitoring). **Repo exists: `KNEL/TSG-COOAndBoard-AIAgents-Public` — not yet built out.**
|
||||
- [ ] SVP KNEL agent + SVP TCTC agent (synthesize policy/strategy → COO).
|
||||
- [ ] VP TechOps / VP SecOps / VP TechCompliance agents scoped to their Redmine projects.
|
||||
- [ ] AJ's access formally documented (he "has full access" per prompt — needs an access matrix in Discourse).
|
||||
|
||||
### 2.2 Documentation (Discourse categories to populate)
|
||||
- [ ] **COO (cat 6):** operations manual, decision authority, escalation paths, daily/weekly/monthly cadence.
|
||||
- [ ] **VP SecOps:** does not exist — create (needs admin key; current API user is trust-4, **cannot create categories** — blocker).
|
||||
- [ ] **VP Compliance (75):** 0 topics — CMMC L3, STIG, ITAR program docs.
|
||||
- [ ] **KNEL-Bizops (72):** 0 topics — the business-services knowledge base.
|
||||
- [ ] **Board (76):** 0 topics.
|
||||
- [ ] Each business entity category: operations content (currently 0-1 stubs).
|
||||
|
||||
### 2.3 Work tracking (Redmine)
|
||||
- [ ] **Project 53 (COO) / 62 (Business Services):** define versions/milestones + seed tickets. Currently empty.
|
||||
- [ ] COO workstream tickets mirroring the operational layer (not just TechOps P1-P9).
|
||||
- [ ] AI-staff scoping: each Hermes agent scoped to its Redmine project only.
|
||||
|
||||
### 2.4 Systems / tools the COO depends on
|
||||
- [ ] **ITSM/workflow engine** (prompt.md line 101: "We really need an ITSM tool — Discourse? Windmill? Nextcloud?"). **Unresolved decision.** Windmill is packaged (✅) — strongest candidate.
|
||||
- [ ] COO Tier-1 apps deployed: Grist, PayrollEngine, KillBill, Rundeck, Comply (all not-started).
|
||||
- [ ] Credential migration to Vault (P5, critical path) — unblocks agent secret access.
|
||||
|
||||
### 2.5 Culture / process (the "correct" part)
|
||||
- [ ] TSYSGroupAIOS framework **pushed and adopted** into all 5 gap projects.
|
||||
- [ ] Full SDLC enforced everywhere (red/green TDD, linters, CI/CD lockstep local+hosted).
|
||||
- [ ] `sectestbed-` / `preprod-` VM workflow operational for IaC testing.
|
||||
- [ ] "Gardening" loop running to prevent doc sprawl.
|
||||
|
||||
---
|
||||
|
||||
## 3. CTO Transition — what changes for Charles
|
||||
|
||||
| Stop doing (COO/founder work) | Start/continue doing (CTO work) |
|
||||
|---|---|
|
||||
| Day-to-day ops decisions | R&D architecture (EngStack, Suborbital, k8s platform) |
|
||||
| Physical infra firefighting (delegate to VP TechOps agent) | Tier-4 escalation only |
|
||||
| Direct fleet config (→ KNELIAC/AWX) | SDLC/governance ownership (the framework) |
|
||||
| Ticket-level execution on P1-P9 | Mentor/oversee AI agents; review their work |
|
||||
| Building business-ops docs | Document TSYS Group component architecture (Oct onboarding task per prompt.md) |
|
||||
|
||||
**The CTO transition is gated on the COO handoff:** you cannot stop doing COO work until AJ + the COO agent can absorb it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Roadmap — Revised (Q3/Q4 split)
|
||||
|
||||
### Q3: Aug 13 → Sep 30 — TechOps + Agent Identities
|
||||
|
||||
**Workstream A — Finish Infrastructure (P1-P9, Redmine project 55, 86 open tickets)**
|
||||
*Keep executing the existing phase plan.* Critical path: **P5 Vault → P6 IaC → P9 Compliance → P7 k8s → P8 Apps.**
|
||||
- Aug 13-17: Friday onsite physical batch. Finish P1/P2/P3.
|
||||
- Aug 18-31: P5 Vault (CRITICAL), P4 monitoring UAT, P6 IaC basics, P9 STIG/CMMC seed, P7 k8s cnode rebuild.
|
||||
- Sep 1-30: P8 Cloudron app fleet (TechOps-relevant), P7 k8s apps (cluster only), compliance hardening.
|
||||
|
||||
**Workstream B — Stand Up AI Agent Identities (NEW, first priority)**
|
||||
*Identity-first. See `agent-identity-bootstrap.md` for full plan.*
|
||||
- Week 1: User provides prerequisites (Linux `coo` account, Bitwarden account, Cloudron invites). Agent enrolls 3 Q3 identities (vp-techops, vp-secops, vp-techcompliance) via Playwright.
|
||||
- Week 2: Deploy per-agent SSH keys, set up `coo` Linux environment, smoke test (agent creates ticket, edits Discourse, opens PR from own identity).
|
||||
- Week 3+: Agents begin operating from own identities. Enroll Q4 identities (Cloudron only, no system access yet).
|
||||
|
||||
**Workstream C — Governance & Culture (cross-cutting)**
|
||||
- Week 1-2: Push TSYSGroupAIOS template; adopt into KNELIAC (critical) + EngStack (critical).
|
||||
- Week 2-4: Adopt into KNEL-AIMiddleware (drop JOURNAL.md), football, PFVCluster (minor).
|
||||
- Ongoing: `sectestbed-`/`preprod-` VM workflow, CI/CD lockstep, gardening loop.
|
||||
|
||||
### Q4: Oct 1 → Dec 31 — Business Ops Transition + CTO Come-Up
|
||||
|
||||
*Charles at 80% CTO / 20% COO. Working alongside AJ (business ops lead), Patti, Courtney.*
|
||||
|
||||
- **October (Charles's stated focus):** Figure out the K8S workload. What apps run on K8S vs Cloudron. Create the K8S repo + Redmine project.
|
||||
- **October–November:** Build the COO business-ops layer: Discourse cat 6 content, Redmine project 53 tickets, ITSM tool selection (Windmill), COO Tier-1 apps (Grist, PayrollEngine, KillBill, Rundeck, Comply), bizopprodplan handbook refresh.
|
||||
- **November–December:** Activate Q4 AI agents (COO, SVP KNEL, SVP TCTC, financial VPs). AJ UAT on COO agent + business systems. Dry-run operational week.
|
||||
- **Jan 1 2027:** Full COO handoff. Charles = 100% CTO.
|
||||
|
||||
---
|
||||
|
||||
## 5. Open Decisions — Updated
|
||||
|
||||
### Resolved by user (2026-08-13)
|
||||
- ~~Business ops timeline~~ → **Q4 (Oct–Dec), full handoff Jan 1 2027.** Q3 = TechOps only.
|
||||
- ~~Agent identity approach~~ → **Identity-first.** Stand up Cloudron + Bitwarden accounts as the very first action.
|
||||
- ~~COO/CTO split in Q4~~ → **80% CTO / 20% COO.** AJ leads business ops with Patti + Courtney.
|
||||
|
||||
### Still open
|
||||
|
||||
1. **Cloudron SSO** — Are Gitea/Discourse/Redmine Cloudron-managed (auto-SSO)? Or standalone? **This determines provisioning complexity.** *(blocking agent bootstrap)*
|
||||
2. **Discourse admin key** — API user is trust-4, **cannot create categories** (VP SecOps etc.). Provide admin key or create via web UI. *(blocking VP SecOps setup)*
|
||||
3. **Linux account model** — Single `coo` account (recommended Q3) vs per-agent accounts (stronger audit). *(see bootstrap spec D1)*
|
||||
4. **Agent runtime** — Crush per-agent config dirs (recommended Q3) vs Hermes vs OpenWebUI. *(see bootstrap spec D3)*
|
||||
5. **K8S scope in P7** — Is P7 (due Aug 31) just cluster bootstrap, with app deployment deferred to October (your stated K8S focus)? *Rec: yes — P7 = cluster ready, apps = October.*
|
||||
6. **ITSM tool** — Windmill (packaged) vs Discourse-only vs Nextcloud. *(deferred to Q4 but worth deciding early)*
|
||||
7. **bizopprodplan repos** — Legacy mdBook stubs (KNEL one still says "CIO Documentation"). Refresh in place during Q4 or treat as superseded by Discourse?
|
||||
|
||||
---
|
||||
|
||||
## 5.5 SecOps & Compliance Context (from user's VP SecOps notes)
|
||||
|
||||
This is the security architecture that the TechOps agents operate within. Drives P5 (PKI/Vault) and P9 (Security/Compliance).
|
||||
|
||||
### Compliance targets
|
||||
- **CMMC Level 3** is the goal (not L1 or L2)
|
||||
- **Full STIG compliance** — highest level (mission critical classified)
|
||||
- **CUI minimum everywhere** — all systems, no exceptions
|
||||
- **ITAR** — governs all technical operations
|
||||
- **Multi-tenant** — isolation between business entities (RackRental franchisees, Suborbital, TCTC, etc.)
|
||||
- **Eventually productized** as a Your Dream Name Here (YDN) offering — the compliance stack itself becomes a product
|
||||
|
||||
### Zero trust access model
|
||||
- **NetBird** (primary zero-trust mesh) + **Tailscale** (existing, nested solution)
|
||||
- **Apple account** referenced (likely for iPad-based access)
|
||||
- All access through zero-trust — no flat network trust
|
||||
- **Remote access SCIF** capability needed
|
||||
- **Keycloak** deployed — initial setup done for NetBird integration
|
||||
- Cloudron IdP has "simple groups, lacks granular permission levels at least via GUI" — Keycloak fills the RBAC gap
|
||||
|
||||
### FOCI concern (Foreign Ownership Control Influence)
|
||||
- **Netcup** (Cloudron VPS host) is a **German company**, even though hosting in Reston VA
|
||||
- Question: can a VPS attest back to PFV (the on-prem cluster)?
|
||||
- This affects what data/workloads can live on Cloudron vs must stay on-prem
|
||||
|
||||
### CA / PKI (drives P5)
|
||||
- **Nitrokey HSM** as the root CA hardware
|
||||
- SSH certificates (not just keys) for day-to-day operations via Ansible
|
||||
- Custom **Ubuntu 24.04 ISO** needed as the Cloudron base image
|
||||
|
||||
### SSH key migration (directly impacts agent identity bootstrap)
|
||||
|
||||
| Current state | Target state |
|
||||
|---|---|
|
||||
| **Ultix-highside** (Win11 Surface, no local admin, Zoc terminal) — on-disk OpenSSH key present on every system except recent deploys | Single Bitwarden SSH key + BW agent (not yet working on Windows) |
|
||||
| **This VM's key** — on a subset of machines, being expanded | Replaced by Bitwarden key/agent |
|
||||
| **iPad secure enclave key** — used via Blink, public key on all accessible systems | **Keep** — one of two authorized broad-access keys |
|
||||
| | **iPad enclave key + Bitwarden key only.** No key material on disk. |
|
||||
|
||||
**Agent identity impact:** Agent SSH keys (Phase 3 of bootstrap) must use the **SSH certificate** model, not raw key deployment. The CA (Nitrokey HSM) signs agent certificates. This is stronger than key-based auth and aligns with the "day-to-day operations via Ansible and SSH certificates" target.
|
||||
|
||||
### Bitwarden structure
|
||||
- **Three BW accounts** currently exist (RCEO owns all creds/orgs/collections)
|
||||
- **Envwarden** in use
|
||||
- A **fourth BW account** (COO/AI agents) will be created per the bootstrap plan
|
||||
- AJ has **broad Bitwarden access** (business continuity)
|
||||
|
||||
### Business continuity
|
||||
- **Patti** — iPad
|
||||
- **Remy** — iPhone
|
||||
- **Albert (AJ)** — broad Bitwarden access
|
||||
|
||||
### Identity/IAM gaps to resolve
|
||||
1. **Cloudron IdP lacks granular RBAC** — Keycloak must fill this gap for agent scoping
|
||||
2. **SSH certificate infrastructure** — Nitrokey HSM → SSH CA → certificate signing for agents + humans
|
||||
3. **Custom Ubuntu 24.04 Cloudron ISO** — needed before app fleet deployment
|
||||
4. **NetBird/Tailscale nesting** — agents must operate within the zero-trust mesh, not bypass it
|
||||
|
||||
---
|
||||
|
||||
## 5.6 OAM — Environmental Monitoring (from user's notes)
|
||||
|
||||
This is the physical/environmental monitoring layer that feeds into P4 (Monitoring & Instrumentation).
|
||||
|
||||
| Sensor/System | Method | Purpose |
|
||||
|---|---|---|
|
||||
| **DRAC** (Dell Remote Access Controller) | IPMI/Redfish API | Out-of-band management of Dell hosts (console, power, hardware status) |
|
||||
| **SNMPd on non-PowerEdge** | SNMP polling | Hardware health on non-Dell systems (Pi, custom builds) |
|
||||
| **Sensors** (lm-sensors / TEMPer USB) | Direct probe | CPU temp, ambient temp, fan speed |
|
||||
| **Home Assistant** | HA integrations | All site environmental monitoring (temp, humidity, presence, power events) |
|
||||
| **Beszel** | Agent-based | RAM / CPU / disk metrics on all hosts |
|
||||
|
||||
**Integration with agent identity:** Environmental monitoring systems (Home Assistant, Beszel, DRAC interfaces) are Cloudron-managed or on-prem. Agent identities need scoped access to these for the vp-techops agent (alerting, dashboard) and vp-secops agent (security event correlation).
|
||||
|
||||
**Current state:**
|
||||
- TEMPer USB probes — Redmine #341 (Friday onsite batch)
|
||||
- Tripp Lite UPS integration — #372, #439
|
||||
- UNPoller (UniFi monitoring) — deployed, placeholder creds
|
||||
- Beszel — agent installed on hosts, dashboard accessible
|
||||
- Home Assistant — planned, not yet deployed
|
||||
- SNMPd — deployed via KNELIAC `system_config` role
|
||||
- LibreNMS — the poller/correlation layer (tsys-librenms)
|
||||
|
||||
---
|
||||
|
||||
## 6. Risk Register
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| **Vault (P5) slips** → agents can't access secrets | High | High | P5 is critical path; prioritize immediately |
|
||||
| **Discourse admin blocker** → can't create VP SecOps category | High | Med | User creates categories via web UI (confirmed will do) |
|
||||
| **k8s cnodes not rebuilt** → P7 slips → Oct K8s focus starts late | Med | High | Rebuild cnodes in Aug onsite window |
|
||||
| **SSH cert infrastructure not ready** → agents can't use cert-based SSH | Med | High | Nitrokey HSM SSH CA setup as part of P5 |
|
||||
| **FOCI concern unresolved** → uncertain what data can live on Cloudron | Med | High | Decide data classification boundaries early in Q4 |
|
||||
| **Cloudron RBAC limits** → can't scope agent access granularly | High | Med | Keycloak fills the gap; wire before agent activation |
|
||||
| **Custom Ubuntu ISO not built** → blocks Cloudron app fleet | Med | Med | Prioritize after P1-P3 physical work |
|
||||
| **Scope creep** — P1-P9 alone is 86 open tickets | High | Med | Keep Q3 TechOps / Q4 Business split strict |
|
||||
| **Hermes full-host-access security** → COO agent blast radius | Med | Critical | Define trust model before Q4 COO agent activation |
|
||||
|
||||
---
|
||||
|
||||
## 7. Immediate Next Actions
|
||||
|
||||
### Confirmed and ready to execute
|
||||
1. **Agent identity bootstrap** (see `agent-identity-bootstrap.md`) — *awaiting user prerequisites (coo account, BW account, Cloudron invites)*
|
||||
2. **Push TSYSGroupAIOS** to Gitea + create as template repo — *ready now*
|
||||
3. **Continue P1-P9 execution** — Friday onsite batch, then P5 Vault critical path
|
||||
|
||||
### Needs user input first
|
||||
4. **Discourse VP SecOps category** — user creates via web UI (API user can't)
|
||||
5. **Cloudron invites** for the 3 Q3 agents — user generates
|
||||
6. **`coo` Linux account + Bitwarden account** — user creates
|
||||
|
||||
### Q4 prep (not yet started)
|
||||
7. **K8S workload planning** — user's stated October focus. Need: K8S repo, Redmine project, app triage (Cloudron vs K8S)
|
||||
8. **COO business-ops layer** — Discourse cat 6 content, Redmine project 53 tickets, ITSM tool decision
|
||||
9. **Custom Ubuntu 24.04 Cloudron ISO** — dependency for app fleet
|
||||
10. **SSH certificate infrastructure** — Nitrokey HSM → SSH CA → cert signing
|
||||
Reference in New Issue
Block a user