MOPAC/deploy/runbook.md

Runbook — MOPAC 9-account deploy (Redmine 494, 2 hosts)

Everything below is one Charles window. The PMO runtime cannot ssh/sudo; this runbook is the whole deployment. Mechanical steps only (no decisions). Estimated total: ~10 min mechanical (build + stage + install + start + verify), plus ~1 min/account for the secrets bootstrap if not prefilled.

Fleet (authority: deploy/accounts.tsv):

#	account	host	vertical	redmine project	events	serve
0	reachableceo	ultix-streaming	rceo	mopac-rceo	4100	8090
1	TSGBOD	ultix-streaming	bod	mopac-bod	4101	8091
2	TSGCOO	ultix-streaming	coo	mopac-coo	4102	8092
3	TSGCTO	ultix-streaming	cto	mopac-cto	4103	8093
4	TSGCCO	ultix-streaming	cco	mopac-cco	4104	8094
5	reachableceo-offstage	ultix-offstage	rceo-offstage	mopac-rceo-offstage	4105	8095
6	COSRCEO-Personal	ultix-offstage	cos-rceo-personal	mopac-cos-rceo-personal	4106	8096
7	COSRCEO-Biz	ultix-offstage	cos-rceo-biz	mopac-cos-rceo-biz	4107	8097
8	COSWFO	ultix-offstage	cos-wfo	mopac-cos-wfo	4108	8098

Port scheme: events = 4100 + index, serve = 8090 + index, index is global 0-8 — so the three daemons of any account never collide with another account's on the same host (loop has no port; state dirs are per-account under ~/.mopac/state/). Nothing listens on 0.0.0.0 roots, no systemd units, no root anywhere.

No-root model: each instance lives entirely in the target account's ~/.mopac/ (binary, config, secrets, state, reports). The install script runs AS the target user. On ultix-streaming Charles is reachableceo with passwordless sudo; on ultix-offstage he ssh's as reachableceo-offstage (also passwordless sudo) — sudo is used ONLY to switch identity (sudo -u <account>), never to install system-wide anything.

---

Step 0 — one-time bootstrap (before the window, ~5 min)

These are Redmine/Gitea side, not host side:

1. Redmine projects: create (or rename) one project per vertical with EXACTLY the identifiers above (mopac-rceo, mopac-bod, ...), or edit the redmine_project column in deploy/accounts.tsv to the real identifiers BEFORE step 2 (it feeds the generated scope_query).
2. API keys (per account, from the account's cloudron user in Bitwarden): Redmine API key, LiteLLM virtual key, three webhook secrets, serve vkey. These go into each account's ~/.mopac/env in step 3.
3. Optional, for [quota] later: the shared redis container per the README "Runbook: shared quota state" section. The generated configs ship with [quota]/[resources] commented out — flipping them on is a per-account one-liner, no redeploy needed.

Step 1 — build once (~2-3 min, any docker host, e.g. your workstation)

git clone https://git.knownelement.com/ukrrs/MOPAC.git   # or git pull
cd MOPAC
make release        # docker builder: CGO_ENABLED=0 GOOS=linux GOARCH=amd64

Output: bin/harness-linux-amd64 (static, stripped). Sanity-check it:

file bin/harness-linux-amd64          # must say: statically linked
./bin/harness-linux-amd64 help | head -5

Step 2 — stage the bundle per host (~1 min per host)

tar czf /tmp/mopac-deploy.tgz deploy bin/harness-linux-amd64
scp /tmp/mopac-deploy.tgz reachableceo@ultix-streaming:/tmp/
scp /tmp/mopac-deploy.tgz reachableceo-offstage@ultix-offstage:/tmp/

Step 3 — install per account (~30s per account, idempotent)

ultix-streaming (5 accounts)

ssh reachableceo@ultix-streaming
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle

# yourself, no sudo:
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo

# the other four, as each user (sudo = identity switch only):
for acct in TSGBOD TSGCOO TSGCTO TSGCCO; do
  sudo -u "$acct" -H sh -c \
    "mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
done

ultix-offstage (4 accounts)

ssh reachableceo-offstage@ultix-offstage
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo-offstage

for acct in COSRCEO-Personal COSRCEO-Biz COSWFO; do
  sudo -u "$acct" -H sh -c \
    "mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
done

(Variant: if direct ssh to each COS account is set up, scp + run the installer per account without sudo — same result.)

Installer contract: creates ~/.mopac/{bin,state/loop,state/events,reports,work}, installs the binary, renders ~/.mopac/harness.toml, creates ~/.mopac/env (0600) and ~/.mopac/bin/mopac-{start,stop}. Refuses to overwrite an existing harness.toml or env — hand-edits survive; re-running is the upgrade path (binary + helpers refresh, config kept).

Step 4 — secrets bootstrap (~1 min per account)

As each account (same sudo -u <acct> -H sh -c pattern, or ssh), fill ~/.mopac/env from Bitwarden — the installer printed exactly this:

vi ~/.mopac/env     # 0600 already; values: HARNESS_REDMINE_KEY,
                    # HARNESS_LITELLM_KEY, HARNESS_{REDMINE,DISCOURSE,GITEA}_WEBHOOK_SECRET,
                    # HARNESS_SERVE_VKEY

harness.toml stays secret-free (env refs only). If an env value stays empty: loop turns fail fast/clean on the missing key, events refuses to start until at least one webhook secret resolves — visible in ~/.mopac/state/events.log.

Step 5 — verify BEFORE starting (~10s per account)

Still as the target account:

# config + routing smoke, NO secrets, NO LLM call — must exit 0 and print
# the PLAN (dry-run) block:
~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml

# first real scan of the account's Redmine scope (needs HARNESS_REDMINE_KEY
# in ~/.mopac/env); prints what would dispatch, writes nothing:
. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ~/.mopac/harness.toml

Step 6 — start (~15s per account)

~/.mopac/bin/mopac-start    # starts loop + events + serve (nohup, pid files)

Then verify the two listeners (healthz; curl or any GET client):

. ~/.mopac/env
curl -fsS http://127.0.0.1:4100/healthz   # events, this account's port from the table
curl -fsS http://127.0.0.1:8090/healthz   # serve,  this account's port from the table
curl -fsS -H "Authorization: Bearer $HARNESS_SERVE_VKEY" http://127.0.0.1:8090/v1/models
tail -n 5 ~/.mopac/state/loop.log

Expected: {"status":"ok"} from both healthz endpoints, a JSON model list from /v1/models (the mopac-* catalog), loop log showing scan lines with the account's scope.

Reboot persistence — pick ONE (no systemd: that needs root, by design)

Option A, cron @reboot (recommended; per account):

( crontab -l 2>/dev/null | grep -v mopac-start; \
  echo '@reboot $HOME/.mopac/bin/mopac-start >> $HOME/.mopac/state/cron.log 2>&1' ) | crontab -

Option B, manual: re-run ~/.mopac/bin/mopac-start after a reboot (idempotent — safe to run anytime; it skips live daemons).

Step 7 — rollback (per account, ~30s)

Stop the daemons and remove the install; Redmine/Gitea data is untouched (the harness only reads issues and writes notes/REPORTs you can see):

~/.mopac/bin/mopac-stop
rm -rf ~/.mopac                     # full removal, or keep evidence:
# mv ~/.mopac ~/.mopac.disabled-$(date +%Y%m%d-%H%M%S)
crontab -l | grep -v mopac-start | crontab -    # if option A was used

mopac-stop is SIGTERM (clean stop); a stuck daemon: kill -9 $(cat ~/.mopac/state/<name>.pid). State/logs/reports live only under ~/.mopac/ — removing it removes every trace of the instance.

---

Time budget (the ~10-minute window)

Step	ultix-streaming (5 accts)	ultix-offstage (4 accts)
stage + install (2+3)	~3 min	~2.5 min
secrets bootstrap (4)	~5 min (unless prefilled)	~4 min
verify + start + healthz (5+6)	~2 min	~1.5 min
total	~10 min (mechanical: ~5)	~8 min (mechanical: ~4)

Build (step 1) and bootstrap (step 0) happen before the window.

Assumptions (one-line fixes, no redeploy)

• Redmine project identifiers are as listed; wrong ones → edit scope_query in the account's harness.toml (installer never overwrites it) or edit accounts.tsv + re-render.
• Quota grouping assumes one z.ai Max plan per host (zai-max-1/zai-max-2); when the real key-to-account mapping is confirmed, edit [quota] account per harness.toml before flipping enabled = true.
• Host short names are ultix-streaming/ultix-offstage (the installer prints an advisory note, never blocks, on a mismatch).
• [redmine.status_map] is intentionally empty in generated configs: set the Released→Done pair per project workflow when adopting the loop for real work (see the comment block in the generated file).