ops(archive): quota, deploy, pdf TASK/REPORT pairs verified+archived
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# REPORT — Multi-account deploy packaging + runbook (Redmine 494)
|
||||
|
||||
- **When**: 2026-08-29 ~05:45-06:00 CST
|
||||
- **Repo**: `projects/meta/MOPAC/harness` @ `b7799ea` (pushed to origin main)
|
||||
- **Spec**: `docs/SPEC-20260829-charles-brief.md` — account list (9 Linux
|
||||
accounts / 2 hosts), "no root on target accounts" PMO constraint
|
||||
- **Gates**: `./dev.sh check` clean (all 11 Go packages ok, untouched);
|
||||
`make deploy-test` **13/13 ok**; `make release` verified static +
|
||||
end-to-end smoke (install COSWFO in a fake HOME -> `once --dry-run
|
||||
--demo` exit 0 via the actual release binary)
|
||||
|
||||
## 1. What Charles executes (the whole deployment)
|
||||
|
||||
Authority: `deploy/runbook.md` + `deploy/accounts.tsv` in the repo.
|
||||
|
||||
1. **Build once** (workstation, ~2-3 min): `make release` ->
|
||||
`bin/harness-linux-amd64` (digest-pinned docker builder, CGO off,
|
||||
linux/amd64, static + stripped — `file` says "statically linked").
|
||||
2. **Stage per host** (~1 min each): `tar czf /tmp/mopac-deploy.tgz
|
||||
deploy bin/harness-linux-amd64` + one `scp` per host.
|
||||
3. **Install per account** (~30s each, idempotent): ssh in, then either
|
||||
direct (`install-account.sh reachableceo`) or via identity switch
|
||||
(`sudo -u TSGBOD -H sh -c '... && install-account.sh TSGBOD'` — sudo
|
||||
is identity-switch only, nothing system-wide). The installer creates
|
||||
`~/.mopac/{bin,state/loop,state/events,reports,work}`, installs the
|
||||
binary, renders `harness.toml` from the template + TSV row, writes the
|
||||
0600 `~/.mopac/env` secrets template, generates `mopac-start`/`mopac-stop`.
|
||||
4. **Secrets bootstrap** (~1 min/account): fill `~/.mopac/env` from
|
||||
Bitwarden (Redmine key, LiteLLM vkey, 3 webhook secrets, serve vkey).
|
||||
Configs stay secret-free (env: refs only).
|
||||
5. **Verify then start** (~25s/account): see §3.
|
||||
6. **Reboot persistence**: cron `@reboot` line (documented verbatim) or
|
||||
manual re-run of the idempotent `mopac-start`. **No systemd** — that
|
||||
needs root, which the accounts don't have; both options documented.
|
||||
|
||||
## 2. Concurrency guard (why 9 daemons per host can't collide)
|
||||
|
||||
Port scheme from `accounts.tsv` (index is global 0-8, listed order):
|
||||
`events = 4100 + index`, `serve = 8090 + index`.
|
||||
|
||||
| account | host | events | serve | | account | host | events | serve |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| reachableceo | streaming | 4100 | 8090 | | reachableceo-offstage | offstage | 4105 | 8095 |
|
||||
| TSGBOD | streaming | 4101 | 8091 | | COSRCEO-Personal | offstage | 4106 | 8096 |
|
||||
| TSGCOO | streaming | 4102 | 8092 | | COSRCEO-Biz | offstage | 4107 | 8097 |
|
||||
| TSGCTO | streaming | 4103 | 8093 | | COSWFO | offstage | 4108 | 8098 |
|
||||
| TSGCCO | streaming | 4104 | 8094 | | | | | |
|
||||
|
||||
The loop daemon has no port; its state (`state/loop`, `state/events`) is
|
||||
per-account under `~/.mopac/`. The installer hard-asserts the scheme per
|
||||
row (dies if `events != 4100+index`), and `deploy/tests.sh` re-asserts
|
||||
fleet-wide (host,port) uniqueness.
|
||||
|
||||
## 3. Verification steps (per account, runbook step 5+6)
|
||||
|
||||
1. `~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml`
|
||||
— no secrets, no LLM call; must exit 0 printing the PLAN with the
|
||||
account's vertical (this exact path is what the release smoke ran).
|
||||
2. `. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ...`
|
||||
— first real Redmine scan of the account's scope (needs the Redmine
|
||||
key): prints what would dispatch, writes nothing.
|
||||
3. `mopac-start`, then healthz on the account's two ports
|
||||
(`curl http://127.0.0.1:<events>/healthz`, `<serve>/healthz`) and an
|
||||
authenticated `GET /v1/models` through the serve vkey; `tail
|
||||
~/.mopac/state/loop.log` for scan lines.
|
||||
|
||||
## 4. Rollback (per account, ~30s)
|
||||
|
||||
`~/.mopac/bin/mopac-stop` (SIGTERM, clean) -> `rm -rf ~/.mopac` (or `mv`
|
||||
aside to keep evidence) -> drop the cron `@reboot` line if used. Every
|
||||
trace of an instance lives under `~/.mopac/`; Redmine/Gitea data is
|
||||
untouched by removal.
|
||||
|
||||
## 5. Time estimate (the Charles window)
|
||||
|
||||
| | streaming (5 accts) | offstage (4 accts) |
|
||||
|---|---|---|
|
||||
| stage + install | ~3 min | ~2.5 min |
|
||||
| secrets bootstrap | ~5 min (unless prefilled) | ~4 min |
|
||||
| verify + start + healthz | ~2 min | ~1.5 min |
|
||||
| **total** | **~10 min** (mechanical only: ~5) | **~8 min** (mechanical: ~4) |
|
||||
|
||||
Build + Redmine project bootstrap happen before the window.
|
||||
|
||||
## 6. Tests (deploy/tests.sh, 13 assertions, all green)
|
||||
|
||||
TSV: 9 rows, spec-exact account/host sets, unique accounts + (host,port)
|
||||
pairs, `events=4100+idx`/`serve=8090+idx` on every row. Installer: render
|
||||
all 9 accounts into fake HOMEs (no leftover `@PLACEHOLDER@`s, correct
|
||||
vertical/ports/project/absolute paths, env mode 0600, helpers executable
|
||||
with the right ports); idempotent re-run (exit 0, config+env
|
||||
byte-identical); hand-edited `harness.toml` survives re-runs; unknown
|
||||
account exits non-zero; staged-bundle binary lookup works; rendered TOML
|
||||
actually loads (`once --dry-run --demo`, exit 0). Plus the live
|
||||
release-binary smoke described above.
|
||||
|
||||
## 7. Assumptions flagged in the runbook (one-line fixes, no redeploy)
|
||||
|
||||
- **Redmine project identifiers** `mopac-<vertical>` x9 must exist (or
|
||||
edit the TSV column before staging / the `scope_query` after install —
|
||||
installer never overwrites an existing config).
|
||||
- **Quota grouping** assumes one z.ai Max plan per host
|
||||
(`zai-max-1`/`zai-max-2`); `[quota]` ships commented with values
|
||||
pre-filled — confirm grouping, then flip `enabled = true` per account.
|
||||
- `[redmine.status_map]` ships empty (workflow names are per-project;
|
||||
the generated file documents the Released->Done pair to set).
|
||||
- Host short names assumed `ultix-streaming`/`ultix-offstage`; the
|
||||
installer prints an advisory on mismatch, never blocks.
|
||||
|
||||
## 8. Open items
|
||||
|
||||
- Actual ssh/scp execution is Charles's window (PMO runtime cannot
|
||||
ssh/sudo) — everything is scripted, tested locally, and pushed.
|
||||
- Webhook registration in Redmine/Discourse/Gitea per account (URLs +
|
||||
secrets) is post-install config, not packaging; noted in runbook step 0.
|
||||
- LiteLLM virtual keys per account assumed to exist on 192.168.3.78:4001.
|
||||
|
||||
— PMO worker, MOPAC harness self-host loop
|
||||
Reference in New Issue
Block a user