deploy: 9-account/2-host packaging + static release build (Redmine 494)
Everything needed for one ~10-minute Charles window on ultix-streaming
and ultix-offstage, with zero root/systemd on the target accounts:
- deploy/accounts.tsv: the fleet authority (account, host, vertical,
Redmine project, quota group, ports). Port scheme events=4100+index,
serve=8090+index with a global 0-8 index so per-account daemons on
one host never collide; loop state is per-account under ~/.mopac.
- deploy/install-account.sh <account>: idempotent installer run AS the
target user; renders harness.toml from deploy/harness.toml.in, writes
a 0600 env-secrets template, refuses to overwrite existing config or
secrets (re-run = the upgrade path), generates mopac-start/stop.
- deploy/runbook.md: exact Charles sequence (build, stage, install,
secrets bootstrap, verify, start via nohup or cron @reboot, rollback)
with the account-port table, per-host time estimates and assumptions.
- Makefile: release target (digest-pinned docker builder, CGO off,
linux/amd64 static, stripped) plus check/deploy-test entrypoints.
- deploy/tests.sh: 13 packaging tests (TSV scheme, template substitution
for all 9 accounts, idempotence, refuse-to-overwrite, binary lookup,
rendered config loads via a dry-run); README deploy section.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -0,0 +1,222 @@
|
||||
# Runbook — MOPAC 9-account deploy (Redmine 494, 2 hosts)
|
||||
|
||||
Everything below is one Charles window. The PMO runtime cannot ssh/sudo;
|
||||
this runbook is the whole deployment. Mechanical steps only (no decisions).
|
||||
Estimated total: **~10 min mechanical** (build + stage + install + start +
|
||||
verify), plus **~1 min/account** for the secrets bootstrap if not prefilled.
|
||||
|
||||
Fleet (authority: `deploy/accounts.tsv`):
|
||||
|
||||
| # | account | host | vertical | redmine project | events | serve |
|
||||
|---|---------|------|----------|-----------------|--------|-------|
|
||||
| 0 | reachableceo | ultix-streaming | rceo | mopac-rceo | 4100 | 8090 |
|
||||
| 1 | TSGBOD | ultix-streaming | bod | mopac-bod | 4101 | 8091 |
|
||||
| 2 | TSGCOO | ultix-streaming | coo | mopac-coo | 4102 | 8092 |
|
||||
| 3 | TSGCTO | ultix-streaming | cto | mopac-cto | 4103 | 8093 |
|
||||
| 4 | TSGCCO | ultix-streaming | cco | mopac-cco | 4104 | 8094 |
|
||||
| 5 | reachableceo-offstage | ultix-offstage | rceo-offstage | mopac-rceo-offstage | 4105 | 8095 |
|
||||
| 6 | COSRCEO-Personal | ultix-offstage | cos-rceo-personal | mopac-cos-rceo-personal | 4106 | 8096 |
|
||||
| 7 | COSRCEO-Biz | ultix-offstage | cos-rceo-biz | mopac-cos-rceo-biz | 4107 | 8097 |
|
||||
| 8 | COSWFO | ultix-offstage | cos-wfo | mopac-cos-wfo | 4108 | 8098 |
|
||||
|
||||
Port scheme: `events = 4100 + index`, `serve = 8090 + index`, index is
|
||||
global 0-8 — so the three daemons of any account never collide with
|
||||
another account's on the same host (loop has no port; state dirs are
|
||||
per-account under `~/.mopac/state/`). Nothing listens on 0.0.0.0 roots,
|
||||
no systemd units, no root anywhere.
|
||||
|
||||
**No-root model**: each instance lives entirely in the target account's
|
||||
`~/.mopac/` (binary, config, secrets, state, reports). The install script
|
||||
runs AS the target user. On ultix-streaming Charles is `reachableceo`
|
||||
with passwordless sudo; on ultix-offstage he ssh's as
|
||||
`reachableceo-offstage` (also passwordless sudo) — sudo is used ONLY to
|
||||
switch identity (`sudo -u <account>`), never to install system-wide
|
||||
anything.
|
||||
|
||||
---
|
||||
|
||||
## Step 0 — one-time bootstrap (before the window, ~5 min)
|
||||
|
||||
These are Redmine/Gitea side, not host side:
|
||||
|
||||
1. **Redmine projects**: create (or rename) one project per vertical with
|
||||
EXACTLY the identifiers above (`mopac-rceo`, `mopac-bod`, ...), or edit
|
||||
the `redmine_project` column in `deploy/accounts.tsv` to the real
|
||||
identifiers BEFORE step 2 (it feeds the generated `scope_query`).
|
||||
2. **API keys** (per account, from the account's cloudron user in
|
||||
Bitwarden): Redmine API key, LiteLLM virtual key, three webhook
|
||||
secrets, serve vkey. These go into each account's `~/.mopac/env` in
|
||||
step 3.
|
||||
3. Optional, for `[quota]` later: the shared redis container per the
|
||||
README "Runbook: shared quota state" section. The generated configs
|
||||
ship with `[quota]`/`[resources]` commented out — flipping them on is
|
||||
a per-account one-liner, no redeploy needed.
|
||||
|
||||
## Step 1 — build once (~2-3 min, any docker host, e.g. your workstation)
|
||||
|
||||
```sh
|
||||
git clone https://git.knownelement.com/ukrrs/MOPAC.git # or git pull
|
||||
cd MOPAC
|
||||
make release # docker builder: CGO_ENABLED=0 GOOS=linux GOARCH=amd64
|
||||
```
|
||||
|
||||
Output: `bin/harness-linux-amd64` (static, stripped). Sanity-check it:
|
||||
|
||||
```sh
|
||||
file bin/harness-linux-amd64 # must say: statically linked
|
||||
./bin/harness-linux-amd64 help | head -5
|
||||
```
|
||||
|
||||
## Step 2 — stage the bundle per host (~1 min per host)
|
||||
|
||||
```sh
|
||||
tar czf /tmp/mopac-deploy.tgz deploy bin/harness-linux-amd64
|
||||
scp /tmp/mopac-deploy.tgz reachableceo@ultix-streaming:/tmp/
|
||||
scp /tmp/mopac-deploy.tgz reachableceo-offstage@ultix-offstage:/tmp/
|
||||
```
|
||||
|
||||
## Step 3 — install per account (~30s per account, idempotent)
|
||||
|
||||
### ultix-streaming (5 accounts)
|
||||
|
||||
```sh
|
||||
ssh reachableceo@ultix-streaming
|
||||
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle
|
||||
|
||||
# yourself, no sudo:
|
||||
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo
|
||||
|
||||
# the other four, as each user (sudo = identity switch only):
|
||||
for acct in TSGBOD TSGCOO TSGCTO TSGCCO; do
|
||||
sudo -u "$acct" -H sh -c \
|
||||
"mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
|
||||
done
|
||||
```
|
||||
|
||||
### ultix-offstage (4 accounts)
|
||||
|
||||
```sh
|
||||
ssh reachableceo-offstage@ultix-offstage
|
||||
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle
|
||||
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo-offstage
|
||||
|
||||
for acct in COSRCEO-Personal COSRCEO-Biz COSWFO; do
|
||||
sudo -u "$acct" -H sh -c \
|
||||
"mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
|
||||
done
|
||||
```
|
||||
|
||||
(Variant: if direct ssh to each COS account is set up, `scp` + run the
|
||||
installer per account without sudo — same result.)
|
||||
|
||||
Installer contract: creates `~/.mopac/{bin,state/loop,state/events,reports,work}`,
|
||||
installs the binary, renders `~/.mopac/harness.toml`, creates `~/.mopac/env`
|
||||
(0600) and `~/.mopac/bin/mopac-{start,stop}`. **Refuses to overwrite an
|
||||
existing `harness.toml` or `env`** — hand-edits survive; re-running is the
|
||||
upgrade path (binary + helpers refresh, config kept).
|
||||
|
||||
## Step 4 — secrets bootstrap (~1 min per account)
|
||||
|
||||
As each account (same `sudo -u <acct> -H sh -c` pattern, or ssh), fill
|
||||
`~/.mopac/env` from Bitwarden — the installer printed exactly this:
|
||||
|
||||
```sh
|
||||
vi ~/.mopac/env # 0600 already; values: HARNESS_REDMINE_KEY,
|
||||
# HARNESS_LITELLM_KEY, HARNESS_{REDMINE,DISCOURSE,GITEA}_WEBHOOK_SECRET,
|
||||
# HARNESS_SERVE_VKEY
|
||||
```
|
||||
|
||||
`harness.toml` stays secret-free (env refs only). If an env value stays
|
||||
empty: loop turns fail fast/clean on the missing key, `events` refuses to
|
||||
start until at least one webhook secret resolves — visible in
|
||||
`~/.mopac/state/events.log`.
|
||||
|
||||
## Step 5 — verify BEFORE starting (~10s per account)
|
||||
|
||||
Still as the target account:
|
||||
|
||||
```sh
|
||||
# config + routing smoke, NO secrets, NO LLM call — must exit 0 and print
|
||||
# the PLAN (dry-run) block:
|
||||
~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml
|
||||
|
||||
# first real scan of the account's Redmine scope (needs HARNESS_REDMINE_KEY
|
||||
# in ~/.mopac/env); prints what would dispatch, writes nothing:
|
||||
. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ~/.mopac/harness.toml
|
||||
```
|
||||
|
||||
## Step 6 — start (~15s per account)
|
||||
|
||||
```sh
|
||||
~/.mopac/bin/mopac-start # starts loop + events + serve (nohup, pid files)
|
||||
```
|
||||
|
||||
Then verify the two listeners (healthz; curl or any GET client):
|
||||
|
||||
```sh
|
||||
. ~/.mopac/env
|
||||
curl -fsS http://127.0.0.1:4100/healthz # events, this account's port from the table
|
||||
curl -fsS http://127.0.0.1:8090/healthz # serve, this account's port from the table
|
||||
curl -fsS -H "Authorization: Bearer $HARNESS_SERVE_VKEY" http://127.0.0.1:8090/v1/models
|
||||
tail -n 5 ~/.mopac/state/loop.log
|
||||
```
|
||||
|
||||
Expected: `{"status":"ok"}` from both healthz endpoints, a JSON model
|
||||
list from `/v1/models` (the `mopac-*` catalog), loop log showing scan
|
||||
lines with the account's scope.
|
||||
|
||||
### Reboot persistence — pick ONE (no systemd: that needs root, by design)
|
||||
|
||||
**Option A, cron `@reboot`** (recommended; per account):
|
||||
|
||||
```sh
|
||||
( crontab -l 2>/dev/null | grep -v mopac-start; \
|
||||
echo '@reboot $HOME/.mopac/bin/mopac-start >> $HOME/.mopac/state/cron.log 2>&1' ) | crontab -
|
||||
```
|
||||
|
||||
**Option B, manual**: re-run `~/.mopac/bin/mopac-start` after a reboot
|
||||
(idempotent — safe to run anytime; it skips live daemons).
|
||||
|
||||
## Step 7 — rollback (per account, ~30s)
|
||||
|
||||
Stop the daemons and remove the install; Redmine/Gitea data is untouched
|
||||
(the harness only reads issues and writes notes/REPORTs you can see):
|
||||
|
||||
```sh
|
||||
~/.mopac/bin/mopac-stop
|
||||
rm -rf ~/.mopac # full removal, or keep evidence:
|
||||
# mv ~/.mopac ~/.mopac.disabled-$(date +%Y%m%d-%H%M%S)
|
||||
crontab -l | grep -v mopac-start | crontab - # if option A was used
|
||||
```
|
||||
|
||||
`mopac-stop` is SIGTERM (clean stop); a stuck daemon: `kill -9 $(cat
|
||||
~/.mopac/state/<name>.pid)`. State/logs/reports live only under
|
||||
`~/.mopac/` — removing it removes every trace of the instance.
|
||||
|
||||
---
|
||||
|
||||
## Time budget (the ~10-minute window)
|
||||
|
||||
| Step | ultix-streaming (5 accts) | ultix-offstage (4 accts) |
|
||||
|------|---------------------------|--------------------------|
|
||||
| stage + install (2+3) | ~3 min | ~2.5 min |
|
||||
| secrets bootstrap (4) | ~5 min (unless prefilled) | ~4 min |
|
||||
| verify + start + healthz (5+6) | ~2 min | ~1.5 min |
|
||||
| **total** | **~10 min** (mechanical: ~5) | **~8 min** (mechanical: ~4) |
|
||||
|
||||
Build (step 1) and bootstrap (step 0) happen before the window.
|
||||
|
||||
## Assumptions (one-line fixes, no redeploy)
|
||||
|
||||
- **Redmine project identifiers** are as listed; wrong ones → edit
|
||||
`scope_query` in the account's `harness.toml` (installer never
|
||||
overwrites it) or edit `accounts.tsv` + re-render.
|
||||
- **Quota grouping** assumes one z.ai Max plan per host
|
||||
(`zai-max-1`/`zai-max-2`); when the real key-to-account mapping is
|
||||
confirmed, edit `[quota] account` per `harness.toml` before flipping
|
||||
`enabled = true`.
|
||||
- Host short names are `ultix-streaming`/`ultix-offstage` (the installer
|
||||
prints an advisory note, never blocks, on a mismatch).
|
||||
- `[redmine.status_map]` is intentionally empty in generated configs:
|
||||
set the Released→Done pair per project workflow when adopting the loop
|
||||
for real work (see the comment block in the generated file).
|
||||
Reference in New Issue
Block a user