# Runbook — MOPAC 9-account deploy (Redmine 494, 2 hosts)

Everything below is one Charles window. The PMO runtime cannot ssh/sudo;
this runbook is the whole deployment. Mechanical steps only (no decisions).
Estimated total: **~10 min mechanical** (build + stage + install + start +
verify), plus **~1 min/account** for the secrets bootstrap if not prefilled.

Fleet (authority: `deploy/accounts.tsv`):

| # | account | host | vertical | redmine project | events | serve |
|---|---------|------|----------|-----------------|--------|-------|
| 0 | reachableceo | ultix-streaming | rceo | mopac-rceo | 4100 | 8090 |
| 1 | TSGBOD | ultix-streaming | bod | mopac-bod | 4101 | 8091 |
| 2 | TSGCOO | ultix-streaming | coo | mopac-coo | 4102 | 8092 |
| 3 | TSGCTO | ultix-streaming | cto | mopac-cto | 4103 | 8093 |
| 4 | TSGCCO | ultix-streaming | cco | mopac-cco | 4104 | 8094 |
| 5 | reachableceo-offstage | ultix-offstage | rceo-offstage | mopac-rceo-offstage | 4105 | 8095 |
| 6 | COSRCEO-Personal | ultix-offstage | cos-rceo-personal | mopac-cos-rceo-personal | 4106 | 8096 |
| 7 | COSRCEO-Biz | ultix-offstage | cos-rceo-biz | mopac-cos-rceo-biz | 4107 | 8097 |
| 8 | COSWFO | ultix-offstage | cos-wfo | mopac-cos-wfo | 4108 | 8098 |

Port scheme: `events = 4100 + index`, `serve = 8090 + index`, index is
global 0-8 — so the three daemons of any account never collide with
another account's on the same host (loop has no port; state dirs are
per-account under `~/.mopac/state/`). Nothing listens on 0.0.0.0 roots,
no systemd units, no root anywhere.

**No-root model**: each instance lives entirely in the target account's
`~/.mopac/` (binary, config, secrets, state, reports). The install script
runs AS the target user. On ultix-streaming Charles is `reachableceo`
with passwordless sudo; on ultix-offstage he ssh's as
`reachableceo-offstage` (also passwordless sudo) — sudo is used ONLY to
switch identity (`sudo -u <account>`), never to install system-wide
anything.

---

## Step 0 — one-time bootstrap (before the window, ~5 min)

These are Redmine/Gitea side, not host side:

1. **Redmine projects**: create (or rename) one project per vertical with
   EXACTLY the identifiers above (`mopac-rceo`, `mopac-bod`, ...), or edit
   the `redmine_project` column in `deploy/accounts.tsv` to the real
   identifiers BEFORE step 2 (it feeds the generated `scope_query`).
2. **API keys** (per account, from the account's cloudron user in
   Bitwarden): Redmine API key, LiteLLM virtual key, three webhook
   secrets, serve vkey. These go into each account's `~/.mopac/env` in
   step 3.
3. Optional, for `[quota]` later: the shared redis container per the
   README "Runbook: shared quota state" section. The generated configs
   ship with `[quota]`/`[resources]` commented out — flipping them on is
   a per-account one-liner, no redeploy needed.

## Step 1 — build once (~2-3 min, any docker host, e.g. your workstation)

```sh
git clone https://git.knownelement.com/ukrrs/MOPAC.git   # or git pull
cd MOPAC
make release        # docker builder: CGO_ENABLED=0 GOOS=linux GOARCH=amd64
```

Output: `bin/harness-linux-amd64` (static, stripped). Sanity-check it:

```sh
file bin/harness-linux-amd64          # must say: statically linked
./bin/harness-linux-amd64 help | head -5
```

## Step 2 — stage the bundle per host (~1 min per host)

```sh
tar czf /tmp/mopac-deploy.tgz deploy bin/harness-linux-amd64
scp /tmp/mopac-deploy.tgz reachableceo@ultix-streaming:/tmp/
scp /tmp/mopac-deploy.tgz reachableceo-offstage@ultix-offstage:/tmp/
```

## Step 3 — install per account (~30s per account, idempotent)

### ultix-streaming (5 accounts)

```sh
ssh reachableceo@ultix-streaming
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle

# yourself, no sudo:
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo

# the other four, as each user (sudo = identity switch only):
for acct in TSGBOD TSGCOO TSGCTO TSGCCO; do
  sudo -u "$acct" -H sh -c \
    "mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
done
```

### ultix-offstage (4 accounts)

```sh
ssh reachableceo-offstage@ultix-offstage
mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle
~/mopac-deploy-bundle/deploy/install-account.sh reachableceo-offstage

for acct in COSRCEO-Personal COSRCEO-Biz COSWFO; do
  sudo -u "$acct" -H sh -c \
    "mkdir -p ~/mopac-deploy-bundle && tar xzf /tmp/mopac-deploy.tgz -C ~/mopac-deploy-bundle && ~/mopac-deploy-bundle/deploy/install-account.sh $acct"
done
```

(Variant: if direct ssh to each COS account is set up, `scp` + run the
installer per account without sudo — same result.)

Installer contract: creates `~/.mopac/{bin,state/loop,state/events,reports,work}`,
installs the binary, renders `~/.mopac/harness.toml`, creates `~/.mopac/env`
(0600) and `~/.mopac/bin/mopac-{start,stop}`. **Refuses to overwrite an
existing `harness.toml` or `env`** — hand-edits survive; re-running is the
upgrade path (binary + helpers refresh, config kept).

## Step 4 — secrets bootstrap (~1 min per account)

As each account (same `sudo -u <acct> -H sh -c` pattern, or ssh), fill
`~/.mopac/env` from Bitwarden — the installer printed exactly this:

```sh
vi ~/.mopac/env     # 0600 already; values: HARNESS_REDMINE_KEY,
                    # HARNESS_LITELLM_KEY, HARNESS_{REDMINE,DISCOURSE,GITEA}_WEBHOOK_SECRET,
                    # HARNESS_SERVE_VKEY
```

`harness.toml` stays secret-free (env refs only). If an env value stays
empty: loop turns fail fast/clean on the missing key, `events` refuses to
start until at least one webhook secret resolves — visible in
`~/.mopac/state/events.log`.

## Step 5 — verify BEFORE starting (~10s per account)

Still as the target account:

```sh
# config + routing smoke, NO secrets, NO LLM call — must exit 0 and print
# the PLAN (dry-run) block:
~/.mopac/bin/harness once --dry-run --demo -config ~/.mopac/harness.toml

# first real scan of the account's Redmine scope (needs HARNESS_REDMINE_KEY
# in ~/.mopac/env); prints what would dispatch, writes nothing:
. ~/.mopac/env && ~/.mopac/bin/harness loop --once --dry-run -config ~/.mopac/harness.toml
```

## Step 6 — start (~15s per account)

```sh
~/.mopac/bin/mopac-start    # starts loop + events + serve (nohup, pid files)
```

Then verify the two listeners (healthz; curl or any GET client):

```sh
. ~/.mopac/env
curl -fsS http://127.0.0.1:4100/healthz   # events, this account's port from the table
curl -fsS http://127.0.0.1:8090/healthz   # serve,  this account's port from the table
curl -fsS -H "Authorization: Bearer $HARNESS_SERVE_VKEY" http://127.0.0.1:8090/v1/models
tail -n 5 ~/.mopac/state/loop.log
```

Expected: `{"status":"ok"}` from both healthz endpoints, a JSON model
list from `/v1/models` (the `mopac-*` catalog), loop log showing scan
lines with the account's scope.

### Reboot persistence — pick ONE (no systemd: that needs root, by design)

**Option A, cron `@reboot`** (recommended; per account):

```sh
( crontab -l 2>/dev/null | grep -v mopac-start; \
  echo '@reboot $HOME/.mopac/bin/mopac-start >> $HOME/.mopac/state/cron.log 2>&1' ) | crontab -
```

**Option B, manual**: re-run `~/.mopac/bin/mopac-start` after a reboot
(idempotent — safe to run anytime; it skips live daemons).

## Step 7 — rollback (per account, ~30s)

Stop the daemons and remove the install; Redmine/Gitea data is untouched
(the harness only reads issues and writes notes/REPORTs you can see):

```sh
~/.mopac/bin/mopac-stop
rm -rf ~/.mopac                     # full removal, or keep evidence:
# mv ~/.mopac ~/.mopac.disabled-$(date +%Y%m%d-%H%M%S)
crontab -l | grep -v mopac-start | crontab -    # if option A was used
```

`mopac-stop` is SIGTERM (clean stop); a stuck daemon: `kill -9 $(cat
~/.mopac/state/<name>.pid)`. State/logs/reports live only under
`~/.mopac/` — removing it removes every trace of the instance.

---

## Time budget (the ~10-minute window)

| Step | ultix-streaming (5 accts) | ultix-offstage (4 accts) |
|------|---------------------------|--------------------------|
| stage + install (2+3) | ~3 min | ~2.5 min |
| secrets bootstrap (4) | ~5 min (unless prefilled) | ~4 min |
| verify + start + healthz (5+6) | ~2 min | ~1.5 min |
| **total** | **~10 min** (mechanical: ~5) | **~8 min** (mechanical: ~4) |

Build (step 1) and bootstrap (step 0) happen before the window.

## Assumptions (one-line fixes, no redeploy)

- **Redmine project identifiers** are as listed; wrong ones → edit
  `scope_query` in the account's `harness.toml` (installer never
  overwrites it) or edit `accounts.tsv` + re-render.
- **Quota grouping** assumes one z.ai Max plan per host
  (`zai-max-1`/`zai-max-2`); when the real key-to-account mapping is
  confirmed, edit `[quota] account` per `harness.toml` before flipping
  `enabled = true`.
- Host short names are `ultix-streaming`/`ultix-offstage` (the installer
  prints an advisory note, never blocks, on a mismatch).
- `[redmine.status_map]` is intentionally empty in generated configs:
  set the Released→Done pair per project workflow when adopting the loop
  for real work (see the comment block in the generated file).