mopac-pmo/archive/REPORT-20260829-0900-deploy.md

# REPORT — Multi-account deploy packaging + runbook (Redmine 494)

- **When**: 2026-08-29 ~05:45-06:00 CST
- **Repo**: `projects/meta/MOPAC/harness` @ `b7799ea` (pushed to origin main)
- **Spec**: `docs/SPEC-20260829-charles-brief.md` — account list (9 Linux
  accounts / 2 hosts), "no root on target accounts" PMO constraint
- **Gates**: `./dev.sh check` clean (all 11 Go packages ok, untouched);
  `make deploy-test` **13/13 ok**; `make release` verified static +
  end-to-end smoke (install COSWFO in a fake HOME -> `once --dry-run
  --demo` exit 0 via the actual release binary)

## 1. What Charles executes (the whole deployment)

Authority: `deploy/runbook.md` + `deploy/accounts.tsv` in the repo.

1. **Build once** (workstation, ~2-3 min): `make release` ->
   `bin/harness-linux-amd64` (digest-pinned docker builder, CGO off,
   linux/amd64, static + stripped — `file` says "statically linked").
2. **Stage per host** (~1 min each): `tar czf /tmp/mopac-deploy.tgz
   deploy bin/harness-linux-amd64` + one `scp` per host.
3. **Install per account** (~30s each, idempotent): ssh in, then either
   direct (`install-account.sh reachableceo`) or via identity switch
   (`sudo -u TSGBOD -H sh -c '... && install-account.sh TSGBOD'` — sudo
   is identity-switch only, nothing system-wide). The installer creates
   `~/.mopac/{bin,state/loop,state/events,reports,work}`, installs the
   binary, renders `harness.toml` from the template + TSV row, writes the
   0600 `~/.mopac/env` secrets template, generates `mopac-start`/`mopac-stop`.
4. **Secrets bootstrap** (~1 min/account): fill `~/.mopac/env` from
   Bitwarden (Redmine key, LiteLLM vkey, 3 webhook secrets, serve vkey).
   Configs stay secret-free (env: refs only).
5. **Verify then start** (~25s/account): see §3.
6. **Reboot persistence**: cron `@reboot` line (documented verbatim) or
   manual re-run of the idempotent `mopac-start`. **No systemd** — that
   needs root, which the accounts don't have; both options documented.

## 2. Concurrency guard (why 9 daemons per host can't collide)

Port scheme from `accounts.tsv` (index is global 0-8, listed order):
`events = 4100 + index`, `serve = 8090 + index`.

| account | host | events | serve |     | account | host | events | serve |
|---|---|---|---|---|---|---|---|---|
| reachableceo | streaming | 4100 | 8090 | | reachableceo-offstage | offstage | 4105 | 8095 |
| TSGBOD | streaming | 4101 | 8091 | | COSRCEO-Personal | offstage | 4106 | 8096 |
| TSGCOO | streaming | 4102 | 8092 | | COSRCEO-Biz | offstage | 4107 | 8097 |
| TSGCTO | streaming | 4103 | 8093 | | COSWFO | offstage | 4108 | 8098 |
| TSGCCO | streaming | 4104 | 8094 | | | | | |

The loop daemon has no port; its state (`state/loop`, `state/events`) is
per-account under `~/.mopac/`. The installer hard-asserts the scheme per
row (dies if `events != 4100+index`), and `deploy/tests.sh` re-asserts
fleet-wide (host,port) uniqueness.

## 3. Verification steps (per account, runbook step 5+6)

1. `~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml`
   — no secrets, no LLM call; must exit 0 printing the PLAN with the
   account's vertical (this exact path is what the release smoke ran).
2. `. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ...`
   — first real Redmine scan of the account's scope (needs the Redmine
   key): prints what would dispatch, writes nothing.
3. `mopac-start`, then healthz on the account's two ports
   (`curl http://127.0.0.1:<events>/healthz`, `<serve>/healthz`) and an
   authenticated `GET /v1/models` through the serve vkey; `tail
   ~/.mopac/state/loop.log` for scan lines.

## 4. Rollback (per account, ~30s)

`~/.mopac/bin/mopac-stop` (SIGTERM, clean) -> `rm -rf ~/.mopac` (or `mv`
aside to keep evidence) -> drop the cron `@reboot` line if used. Every
trace of an instance lives under `~/.mopac/`; Redmine/Gitea data is
untouched by removal.

## 5. Time estimate (the Charles window)

| | streaming (5 accts) | offstage (4 accts) |
|---|---|---|
| stage + install | ~3 min | ~2.5 min |
| secrets bootstrap | ~5 min (unless prefilled) | ~4 min |
| verify + start + healthz | ~2 min | ~1.5 min |
| **total** | **~10 min** (mechanical only: ~5) | **~8 min** (mechanical: ~4) |

Build + Redmine project bootstrap happen before the window.

## 6. Tests (deploy/tests.sh, 13 assertions, all green)

TSV: 9 rows, spec-exact account/host sets, unique accounts + (host,port)
pairs, `events=4100+idx`/`serve=8090+idx` on every row. Installer: render
all 9 accounts into fake HOMEs (no leftover `@PLACEHOLDER@`s, correct
vertical/ports/project/absolute paths, env mode 0600, helpers executable
with the right ports); idempotent re-run (exit 0, config+env
byte-identical); hand-edited `harness.toml` survives re-runs; unknown
account exits non-zero; staged-bundle binary lookup works; rendered TOML
actually loads (`once --dry-run --demo`, exit 0). Plus the live
release-binary smoke described above.

## 7. Assumptions flagged in the runbook (one-line fixes, no redeploy)

- **Redmine project identifiers** `mopac-<vertical>` x9 must exist (or
  edit the TSV column before staging / the `scope_query` after install —
  installer never overwrites an existing config).
- **Quota grouping** assumes one z.ai Max plan per host
  (`zai-max-1`/`zai-max-2`); `[quota]` ships commented with values
  pre-filled — confirm grouping, then flip `enabled = true` per account.
- `[redmine.status_map]` ships empty (workflow names are per-project;
  the generated file documents the Released->Done pair to set).
- Host short names assumed `ultix-streaming`/`ultix-offstage`; the
  installer prints an advisory on mismatch, never blocks.

## 8. Open items

- Actual ssh/scp execution is Charles's window (PMO runtime cannot
  ssh/sudo) — everything is scripted, tested locally, and pushed.
- Webhook registration in Redmine/Discourse/Gitea per account (URLs +
  secrets) is post-install config, not packaging; noted in runbook step 0.
- LiteLLM virtual keys per account assumed to exist on 192.168.3.78:4001.

— PMO worker, MOPAC harness self-host loop