REPORT — Multi-account deploy packaging + runbook (Redmine 494)

When: 2026-08-29 ~05:45-06:00 CST
Repo: projects/meta/MOPAC/harness @ b7799ea (pushed to origin main)
Spec: docs/SPEC-20260829-charles-brief.md — account list (9 Linux accounts / 2 hosts), "no root on target accounts" PMO constraint
Gates: ./dev.sh check clean (all 11 Go packages ok, untouched); make deploy-test 13/13 ok; make release verified static + end-to-end smoke (install COSWFO in a fake HOME -> once --dry-run --demo exit 0 via the actual release binary)

1. What Charles executes (the whole deployment)

Authority: deploy/runbook.md + deploy/accounts.tsv in the repo.

Build once (workstation, ~2-3 min): make release -> bin/harness-linux-amd64 (digest-pinned docker builder, CGO off, linux/amd64, static + stripped — file says "statically linked").
Stage per host (~1 min each): tar czf /tmp/mopac-deploy.tgz deploy bin/harness-linux-amd64 + one scp per host.
Install per account (~30s each, idempotent): ssh in, then either direct (install-account.sh reachableceo) or via identity switch (sudo -u TSGBOD -H sh -c '... && install-account.sh TSGBOD' — sudo is identity-switch only, nothing system-wide). The installer creates ~/.mopac/{bin,state/loop,state/events,reports,work}, installs the binary, renders harness.toml from the template + TSV row, writes the 0600 ~/.mopac/env secrets template, generates mopac-start/mopac-stop.
Secrets bootstrap (~1 min/account): fill ~/.mopac/env from Bitwarden (Redmine key, LiteLLM vkey, 3 webhook secrets, serve vkey). Configs stay secret-free (env: refs only).
Verify then start (~25s/account): see §3.
Reboot persistence: cron @reboot line (documented verbatim) or manual re-run of the idempotent mopac-start. No systemd — that needs root, which the accounts don't have; both options documented.

2. Concurrency guard (why 9 daemons per host can't collide)

Port scheme from accounts.tsv (index is global 0-8, listed order): events = 4100 + index, serve = 8090 + index.

account	host	events	serve	account	host	events	serve
reachableceo	streaming	4100	8090	reachableceo-offstage	offstage	4105	8095
TSGBOD	streaming	4101	8091	COSRCEO-Personal	offstage	4106	8096
TSGCOO	streaming	4102	8092	COSRCEO-Biz	offstage	4107	8097
TSGCTO	streaming	4103	8093	COSWFO	offstage	4108	8098
TSGCCO	streaming	4104	8094

The loop daemon has no port; its state (state/loop, state/events) is per-account under ~/.mopac/. The installer hard-asserts the scheme per row (dies if events != 4100+index), and deploy/tests.sh re-asserts fleet-wide (host,port) uniqueness.

3. Verification steps (per account, runbook step 5+6)

~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml — no secrets, no LLM call; must exit 0 printing the PLAN with the account's vertical (this exact path is what the release smoke ran).
. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ... — first real Redmine scan of the account's scope (needs the Redmine key): prints what would dispatch, writes nothing.
mopac-start, then healthz on the account's two ports (curl http://127.0.0.1:<events>/healthz, <serve>/healthz) and an authenticated GET /v1/models through the serve vkey; tail ~/.mopac/state/loop.log for scan lines.

4. Rollback (per account, ~30s)

~/.mopac/bin/mopac-stop (SIGTERM, clean) -> rm -rf ~/.mopac (or mv aside to keep evidence) -> drop the cron @reboot line if used. Every trace of an instance lives under ~/.mopac/; Redmine/Gitea data is untouched by removal.

5. Time estimate (the Charles window)

	streaming (5 accts)	offstage (4 accts)
stage + install	~3 min	~2.5 min
secrets bootstrap	~5 min (unless prefilled)	~4 min
verify + start + healthz	~2 min	~1.5 min
total	~10 min (mechanical only: ~5)	~8 min (mechanical: ~4)

Build + Redmine project bootstrap happen before the window.

6. Tests (deploy/tests.sh, 13 assertions, all green)

TSV: 9 rows, spec-exact account/host sets, unique accounts + (host,port) pairs, events=4100+idx/serve=8090+idx on every row. Installer: render all 9 accounts into fake HOMEs (no leftover @PLACEHOLDER@s, correct vertical/ports/project/absolute paths, env mode 0600, helpers executable with the right ports); idempotent re-run (exit 0, config+env byte-identical); hand-edited harness.toml survives re-runs; unknown account exits non-zero; staged-bundle binary lookup works; rendered TOML actually loads (once --dry-run --demo, exit 0). Plus the live release-binary smoke described above.

7. Assumptions flagged in the runbook (one-line fixes, no redeploy)

Redmine project identifiers mopac-<vertical> x9 must exist (or edit the TSV column before staging / the scope_query after install — installer never overwrites an existing config).
Quota grouping assumes one z.ai Max plan per host (zai-max-1/zai-max-2); [quota] ships commented with values pre-filled — confirm grouping, then flip enabled = true per account.
[redmine.status_map] ships empty (workflow names are per-project; the generated file documents the Released->Done pair to set).
Host short names assumed ultix-streaming/ultix-offstage; the installer prints an advisory on mismatch, never blocks.

8. Open items

Actual ssh/scp execution is Charles's window (PMO runtime cannot ssh/sudo) — everything is scripted, tested locally, and pushed.
Webhook registration in Redmine/Discourse/Gitea per account (URLs + secrets) is post-install config, not packaging; noted in runbook step 0.
LiteLLM virtual keys per account assumed to exist on 192.168.3.78:4001.

— PMO worker, MOPAC harness self-host loop

6.0 KiB Raw Blame History