From 47343d53a2638eb8ef445b6c6692a80d1fd6b184 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Fri, 28 Aug 2026 22:30:14 -0500 Subject: [PATCH] Add Docker-routed dev tooling, smoke test, example config, and docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Route every compile/vet/test path through a digest-pinned golang:1.26 builder container (dev.sh + Makefile) so the host never runs a Go toolchain. Add an end-to-end smoke that serves keyproxy in a container with a throwaway config and drives 401/200/404/400/501/405 paths with python urllib, asserting the server log is redacted. Ship a commented keyproxy.toml.example (the real config stays gitignored along with any *.env tripwire) and rewrite the README as the full quickstart, config, HTTP, CLI, and redaction reference. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- .gitignore | 7 ++ Makefile | 19 ++++ README.md | 226 ++++++++++++++++++++++++++++++++++++++---- dev.sh | 54 ++++++++++ keyproxy.toml.example | 47 +++++++++ smoke/probe.py | 108 ++++++++++++++++++++ smoke/smoke.sh | 94 ++++++++++++++++++ 7 files changed, 538 insertions(+), 17 deletions(-) create mode 100644 .gitignore create mode 100644 Makefile create mode 100755 dev.sh create mode 100644 keyproxy.toml.example create mode 100644 smoke/probe.py create mode 100755 smoke/smoke.sh diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3197a0a --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +# keyproxy.toml holds the ref map (paths + key names) — copy to +# keyproxy.toml and fill in. keyproxy.toml itself is gitignored. +# Any *.env file landing in this repo is ignored as a tripwire. +keyproxy.toml +*.env +bin/ +.smoke/ diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..52b565d --- /dev/null +++ b/Makefile @@ -0,0 +1,19 @@ +# Makefile: thin front door over dev.sh so `make build/vet/test/check` +# route through the digest-pinned Docker builder (ALL dev work in +# Docker — the host never runs a Go toolchain). +.PHONY: build vet test check smoke + +build: + @./dev.sh build + +vet: + @./dev.sh vet + +test: + @./dev.sh test + +check: + @./dev.sh check + +smoke: + @./dev.sh smoke diff --git a/README.md b/README.md index b0b6645..5ed7182 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,217 @@ # mopac-keyproxy A key-material proxy where the vault stays authoritative. Consumers receive -opaque placeholder keys; real secrets live only in Bitwarden Secrets Manager -and/or HashiCorp Vault and are resolved to real material at the wire, in -memory, never persisted, with no admin UI to attack. +opaque placeholder refs (`mpk-`); real secrets live only in the +configured backends (v0: 0600 env files and process env; phase 3: Bitwarden +Secrets Manager and HashiCorp Vault) and are resolved to real material at +the wire, in memory, never persisted, never logged, with no admin UI to +attack. -Status: 2026-08-28 — spec seed; design frozen, no code yet. +Status: 2026-08-28 — v0 LIVE: `keyproxy serve` (localhost HTTP resolve hop) ++ `keyproxy get` (exec-style CLI) on the file and env backends; +bitwarden/vault ship as explicit not-implemented stubs behind the same +interface (phase 3 drop-in). + +## Quickstart + +All dev work happens inside a Docker builder (host stays toolchain-free); +`docker pull` of the builder is pre-authorized. Commands below were verified +on 2026-08-28 from a fresh clone. + +### Build and test + +`dev.sh` routes every compile/vet/test path through the digest-pinned +builder container (or use `make build|vet|test|check`, same routing): + +```sh +./dev.sh check # = go build + go vet + go test, all inside the builder +``` + +The equivalent raw command: + +```sh +docker run --rm -v "$PWD:/h" -w /h \ + -u "$(id -u):$(id -g)" -e HOME=/tmp -e GOFLAGS=-buildvcs=false \ + golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 \ + sh -c 'go build -o bin/keyproxy ./cmd/keyproxy && go vet ./... && go test ./...' +``` + +(that digest = `golang:1.26-bookworm`.) + +Expected output (tail): + +```text +ok git.knownelement.com/ukrrs/mopac-keyproxy/internal/backend +ok git.knownelement.com/ukrrs/mopac-keyproxy/internal/config +ok git.knownelement.com/ukrrs/mopac-keyproxy/internal/server +``` + +End-to-end smoke (builds, serves in a container on port 8082 with a +throwaway config, drives 401/200/404/400/501/502 paths with python +urllib, and asserts the server log is redacted): + +```sh +./dev.sh smoke +``` + +Expected output (tail): + +```text +probe: OK +--- server log (redaction check) --- +... keyproxy: listening addr=:8082 refs=5 backends=bitwarden,env,file,vault auth=mpk-keyproxy-self=*** +... keyproxy: resolve ref=mpk-smoke=*** backend=file status=200 +``` + +### Configure + +```sh +mkdir -p ~/.config/keyproxy && umask 077 +printf 'KEYPROXY_TOKEN=%s\n' "$(od -An -N32 -tx1 /dev/urandom | tr -d ' \n')" \ + > ~/.config/keyproxy/keyproxy.env +printf 'EXAMPLE_API_KEY=%s\n' "$(od -An -N32 -tx1 /dev/urandom | tr -d ' \n')" \ + > ~/.config/keyproxy/example.env +cp keyproxy.toml.example keyproxy.toml +``` + +`keyproxy.toml` is gitignored (as is any `*.env` landing in the repo). It +holds the ref MAP — paths and key names — never material itself. + +### Serve and resolve + +```sh +./bin/keyproxy serve # binds 127.0.0.1:8082 (config: listen) +``` + +Resolve a ref over the HTTP hop (python3 urllib; curl-free hosts work +fine). The token is read from the 0600 env file, and the resolved value is +shown only as a length + mask — copy-paste safe: + +```sh +KEYPROXY_TOKEN="$(sed -n 's/^KEYPROXY_TOKEN=//p' ~/.config/keyproxy/keyproxy.env)" \ +python3 - <<'PY' +import json, os, urllib.request, urllib.error + +req = urllib.request.Request( + "http://127.0.0.1:8082/v1/resolve", + data=json.dumps({"ref": "mpk-example"}).encode(), method="POST") +req.add_header("Authorization", "Bearer " + os.environ["KEYPROXY_TOKEN"]) +try: + with urllib.request.urlopen(req, timeout=5) as r: + v = json.loads(r.read())["value"] + print(f"200: mpk-example={len(v)} chars, mpk-example=***") +except urllib.error.HTTPError as e: + print(e.code, e.read().decode().strip()) +PY +``` + +Expected output: + +```text +200: mpk-example=64 chars, mpk-example=*** +``` + +Without (or with a wrong) bearer token the response is +`401 {"error":"unauthorized"}` and the server logs only +`auth failure remote=...` — never the presented token. + +Exec-style plumbing, no HTTP hop: + +```sh +./bin/keyproxy get mpk-example > /tmp/.k && wc -c < /tmp/.k # bare value, no newline +``` + +## Config reference (`keyproxy.toml`) + +| Key | Scope | Meaning | +|---|---|---| +| `listen` | top level | bind address (default `127.0.0.1:8082`; keep it loopback unless you know why not) | +| `auth.token_ref` | `[auth]` | ref holding the bearer token for `/v1/resolve`; MUST be a `file`-backend ref, resolved once at startup, never logged | +| `refs."mpk-".backend` | ref | `file`, `env`, `bitwarden` (stub), `vault` (stub) | +| `refs."mpk-".source` | ref | file: path to a 0600 env file (`~/` expanded); env: variable name; bitwarden/vault: reserved mount/project path | +| `refs."mpk-".key` | ref | file: KEY inside the env file (required); env: forbidden (source IS the name); stubs: informational | +| `refs."mpk-".mode` | ref | file only: allowed permission mask (default `0600`; stricter files like `0400` pass, looser files are refused before being read) | + +Ref names must match `mpk-[a-z0-9][a-z0-9-]*`. Because the pattern is +enforced before any echo, refs can be named safely in logs and errors: an +mpk- ref is a placeholder, by construction not material. + +Env files are `KEY=VALUE` lines parsed in Go — NEVER sourced or exec'd +(no shell expansion, no interpolation; comments, blank lines, `export ` +prefix and surrounding quotes are handled; later duplicate keys win). + +## HTTP reference (`keyproxy serve`) + +| Route | Auth | Behavior | +|---|---|---| +| `POST /v1/resolve` | bearer token | body `{"ref":"mpk-"}` → `{"value":"..."}`; material crosses the wire here and nowhere else | +| `GET /healthz` | none | liveness | + +Status codes: 200 resolved; 401 missing/wrong token (constant-time +compare); 400 malformed body or non-`mpk-` ref (an invalid "ref" field is +never echoed — it might hold pasted material); 404 unknown ref; 405 wrong +method; 413 body over 4 KiB; 500 recovered panic (detail suppressed); 501 +backend not implemented in v0; 502 backend failure. Failure bodies carry +the ref, the backend and a fixed reason enum (`missing_key`, +`insecure_source_mode`, `unreadable_source`, `empty_value`, +`malformed_source`, `not_implemented`) — reason enums cannot embed +material by construction. + +## CLI reference + +| Command | Purpose | +|---|---| +| `keyproxy serve` | run the resolve HTTP hop until SIGINT/SIGTERM | +| `keyproxy get REF` | resolve one ref to stdout (bare value, no trailing newline — for `$(...)` plumbing) | +| `keyproxy help` | print usage | + +Flags: `-config PATH` (default `$KEYPROXY_CONFIG`, then `./keyproxy.toml`), +`-listen ADDR` (serve only, overrides config). + +Exit codes: 0 ok; 1 usage/config error; 2 resolution failure. + +## Redaction rules (enforced by tests) + +- No admin UI, no persistence, no cache files: material exists in memory + per resolve and is dropped. +- Every log line that names a ref masks any value as `=***`. +- No material in logs, error bodies, or crash paths: panics are recovered + with the panic value discarded (never stringified); auth failures log + the remote address only; parse errors carry line numbers, never line + contents. +- Resolution failures name the ref and the backend, never the value. +- 0600 enforced on secret env files (looser modes are refused before the + file is read). + +## Architecture + +```mermaid +flowchart LR + C["consumer
(harness, CLI, container)
holds mpk-... refs only"] -->|"POST /v1/resolve
bearer token"| S["keyproxy serve
auth -> ref lookup -> backend"] + S --> F["file backend
0600 env files, parsed in Go
(never sourced)"] + S --> E["env backend
process env indirection"] + S --> B["bitwarden stub
501 not_implemented"] + S --> V["vault stub
501 not_implemented"] + S -->|"200 {value} — memory only"| C +``` + +All backends sit behind one interface (`internal/backend.Backend`); phase-3 +Bitwarden Secrets Manager (plain REST, machine accounts) and Vault KV +v2 + AppRole connectors replace the stubs in that one place. ## Scope -- Placeholders in, real keys on the wire out: consumers (harness, CLIs, - containers) hold `mpk_...` opaque refs only. A leaked placeholder is - revoke-and-remap, not an incident — zero upstream exposure. -- Backends: Bitwarden Secrets Manager REST (machine accounts) and HashiCorp - Vault KV v2 + AppRole (official Go api pkg is MPL-2.0, vendored). - Stdlib-first; NO official Bitwarden SDK (its source-available license is - AGPL-incompatible). +- Placeholders in, real keys on the wire out: consumers hold `mpk-...` + opaque refs only. A leaked placeholder is revoke-and-remap, not an + incident — zero upstream exposure. +- Backends: v0 file + env; planned Bitwarden Secrets Manager REST (machine + accounts) and HashiCorp Vault KV v2 + AppRole (official Go api pkg is + MPL-2.0, AGPL-compatible, vendored). Stdlib-first; NO official + Bitwarden SDK (its source-available license is AGPL-incompatible). - Two shapes, one resolver: an HTTP hop for services, and a CLI (`keyproxy get `) for exec-style plumbing. -- Memory-only material handling: fetch-on-demand, short TTL, never persisted, - never written to logs; ref redaction everywhere. +- Memory-only material handling: fetch-on-demand, never persisted, never + written to logs; ref redaction everywhere. - Config-driven: generic, no organizational hosts/paths/defaults baked into code. @@ -29,15 +221,15 @@ Status: 2026-08-28 — spec seed; design frozen, no code yet. - Not a secrets manager: the vault stays authoritative; keyproxy never becomes a second place secrets live. - No persistence of key material to disk, cache files, or crash dumps. -- Not org-specific: TSYS policy lives outside this repo (loose-coupling - rules for the MOPAC tool family). +- Not org-specific: policy lives outside this repo (loose-coupling rules + for the MOPAC tool family). ## Today vs planned | | State | |---|---| -| Today | Spec only (this README + LICENSE). Design carried in the MOPAC harness DESIGN.md "Key proxy" section. | -| Planned | Go implementation: placeholder->material resolver, Bitwarden Secrets Manager + Vault connectors, HTTP hop + `get` CLI, ref-redaction rules. | +| Today | v0 live: `serve` + `get` on file (0600 env files) and env backends; bearer auth bootstrapped from the file backend; table-driven tests incl. redaction + fake-HTTP end-to-end; docker-routed build discipline. | +| Planned | Phase 3: Bitwarden Secrets Manager REST + Vault KV v2/AppRole connectors behind the same interface (stubs and tests already in place); short-TTL memory cache. | ## Design references diff --git a/dev.sh b/dev.sh new file mode 100755 index 0000000..b4bda55 --- /dev/null +++ b/dev.sh @@ -0,0 +1,54 @@ +#!/bin/sh +# keyproxy dev wrapper. EVERY compile/vet/test path routes through the +# digest-pinned Docker builder (DESIGN "ALL dev work in Docker" — big +# rule); the host runs containers, never toolchains. +# +# Usage: ./dev.sh {build|vet|test|check|smoke|shell} [args...] +# +# build compile ./cmd/keyproxy into bin/keyproxy +# vet go vet ./... +# test go test ./... +# check build + vet + test (the pre-push gate) +# smoke end-to-end smoke: builds, starts `keyproxy serve` in a +# container on port 8082 with a throwaway config, drives it +# from the host with python urllib (curl is banned on host), +# shows 401/200/404/501 paths + redacted server log, tears +# everything down +# shell interactive sh inside the builder +set -e + +IMAGE="golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514" # = golang:1.26-bookworm (bash present; alpine lacks it) + +run() { + docker run --rm -v "$PWD:/h" -w /h \ + -u "$(id -u):$(id -g)" -e HOME=/tmp -e GOFLAGS=-buildvcs=false \ + "$IMAGE" "$@" +} + +cmd=${1:-check} +shift || true + +case "$cmd" in +build) + run go build -o bin/keyproxy ./cmd/keyproxy + ;; +vet) + run go vet ./... + ;; +test) + run go test "$@" ./... + ;; +check) + run sh -c 'go build -o bin/keyproxy ./cmd/keyproxy && go vet ./... && go test ./...' + ;; +smoke) + ./smoke/smoke.sh + ;; +shell) + run sh + ;; +*) + echo "dev.sh: unknown command $cmd (build|vet|test|check|smoke|shell)" >&2 + exit 1 + ;; +esac diff --git a/keyproxy.toml.example b/keyproxy.toml.example new file mode 100644 index 0000000..c9ec88d --- /dev/null +++ b/keyproxy.toml.example @@ -0,0 +1,47 @@ +# keyproxy.toml.example — copy to keyproxy.toml and fill in. +# keyproxy.toml is gitignored; it holds the ref MAP (paths + key names), +# never material itself. Refs are mpk- placeholders: consumers see +# only these names; material resolves at request time, memory-only. + +# HTTP hop bind address. Keep it loopback unless you know why not. +listen = "127.0.0.1:8082" + +[auth] +# Bearer token for POST /v1/resolve, bootstrapped from the file backend +# itself: a 0600 env file holding the token. Resolved once at startup, +# never logged, never persisted by keyproxy. +token_ref = "mpk-keyproxy-self" + +# --- file backend: KEY=VALUE env files, 0600, parsed in Go (never +# sourced/exec'd). mode is the allowed permission mask (default 0600; +# stricter files like 0400 always pass, looser files are refused). +[refs."mpk-keyproxy-self"] +backend = "file" +source = "~/.config/keyproxy/keyproxy.env" +key = "KEYPROXY_TOKEN" + +[refs."mpk-example"] +backend = "file" +source = "~/.config/keyproxy/example.env" +key = "EXAMPLE_API_KEY" + +# --- env backend: process-environment indirection. source IS the +# variable name (no key field); useful for container-injected values. +[refs."mpk-example-env"] +backend = "env" +source = "EXAMPLE_API_KEY" + +# --- bitwarden backend: Bitwarden Secrets Manager REST (machine +# accounts). NOT IMPLEMENTED in v0 — resolves fail loudly with 501 +# not_implemented so phase 3 is a drop-in behind the same interface. +[refs."mpk-example-bitwarden"] +backend = "bitwarden" +source = "sm://projects/example" +key = "EXAMPLE_API_KEY" + +# --- vault backend: HashiCorp Vault KV v2 + AppRole. NOT IMPLEMENTED in +# v0 — same explicit 501 stub as bitwarden. +[refs."mpk-example-vault"] +backend = "vault" +source = "secret/data/example" +key = "EXAMPLE_API_KEY" diff --git a/smoke/probe.py b/smoke/probe.py new file mode 100644 index 0000000..0958c7c --- /dev/null +++ b/smoke/probe.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""keyproxy resolve smoke probe. + +Drives `keyproxy serve` on BASE (default http://127.0.0.1:8082) with +plain stdlib urllib: the host has no curl by policy. Checks: healthz, +401 without/with wrong token, 200 file + env resolves, 404 unknown ref, +400 invalid ref format (without echo), 501 bitwarden/vault stubs, 502 +empty-value failure. Prints one PASS/FAIL line per check; exits non-zero +on any failure. +""" + +import json +import os +import sys +import time +import urllib.error +import urllib.request + +BASE = os.environ.get("BASE_URL", "http://127.0.0.1:8082") +TOKEN = os.environ.get("KEYPROXY_TOKEN", "smoke-bearer-token-1111") + +failures = 0 + + +def req(method, path, body=None, headers=None): + r = urllib.request.Request(BASE + path, data=body, method=method, + headers=headers or {}) + try: + with urllib.request.urlopen(r, timeout=5) as resp: + return resp.status, resp.read().decode() + except urllib.error.HTTPError as e: + return e.code, e.read().decode() + except Exception as e: # transport errors surface as FAIL lines, not tracebacks + return 0, f"transport error: {e}" + + +def check(name, cond, detail=""): + global failures + print(("PASS" if cond else "FAIL"), name, ("- " + detail) if detail else "") + if not cond: + failures += 1 + + +def wait_healthz(tries=50): + for _ in range(tries): + try: + code, _ = req("GET", "/healthz") + if code == 200: + return True + except Exception: + pass + time.sleep(0.2) + return False + + +def resolve(token, ref): + body = json.dumps({"ref": ref}).encode() + headers = {"Authorization": "Bearer " + token} if token else {} + return req("POST", "/v1/resolve", body, headers) + + +def main(): + check("healthz answers (unauthenticated)", wait_healthz()) + + code, text = resolve(None, "mpk-smoke") + check("no bearer token rejected 401", code == 401, text.strip()) + + code, text = resolve("wrong-token", "mpk-smoke") + check("wrong bearer token rejected 401", code == 401, text.strip()) + + code, text = resolve(TOKEN, "mpk-smoke") + obj = json.loads(text) if text else {} + check("file ref resolves 200 with value", + code == 200 and obj.get("value") == "smoke-secret-material-2222", + text.strip()) + + code, text = resolve(TOKEN, "mpk-smoke-env") + obj = json.loads(text) if text else {} + check("env ref resolves 200 with value", + code == 200 and obj.get("value") == "smoke-env-material-4444", + text.strip()) + + code, text = resolve(TOKEN, "mpk-ghost") + check("unknown ref 404", code == 404 and "mpk-ghost" in text, text.strip()) + + code, text = resolve(TOKEN, "sk-live-pasted-secret") + check("invalid ref format 400 without echo", + code == 400 and "pasted" not in text, text.strip()) + + code, text = resolve(TOKEN, "mpk-smoke-bw") + check("bitwarden stub 501 not_implemented", + code == 501 and "not_implemented" in text and "bitwarden" in text, + text.strip()) + + code, text = resolve(TOKEN, "mpk-smoke-vault") + check("vault stub 501 not_implemented", + code == 501 and "not_implemented" in text and "vault" in text, + text.strip()) + + code, text = req("GET", "/v1/resolve") + check("GET resolve rejected 405", code == 405, text.strip()) + + print("probe:", "OK" if failures == 0 else f"{failures} FAILURE(S)") + return 0 if failures == 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/smoke/smoke.sh b/smoke/smoke.sh new file mode 100755 index 0000000..1d8bdea --- /dev/null +++ b/smoke/smoke.sh @@ -0,0 +1,94 @@ +#!/bin/sh +# keyproxy end-to-end smoke: builds in the digest-pinned builder, starts +# `keyproxy serve` in a container on host port 8082 with a throwaway +# config + throwaway 0600 env files, drives it from the host with python +# urllib (curl is banned on the host by policy), checks the 401/200/404/ +# 400/501/502 paths AND that the server log is redacted (no material, +# every ref masked as =***), then tears everything down. +set -e +cd "$(dirname "$0")/.." + +IMAGE="golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514" # = golang:1.26-bookworm +NAME="keyproxy-smoke" +PORT="${PORT:-8082}" + +./dev.sh build + +rm -rf .smoke +mkdir -p .smoke +chmod 0700 .smoke + +# Throwaway material (smoke values only; never real secrets). +cat > .smoke/keyproxy.env <<'EOF' +KEYPROXY_TOKEN=smoke-bearer-token-1111 +EOF +cat > .smoke/creds.env <<'EOF' +API_KEY=smoke-secret-material-2222 +OTHER=smoke-other-3333 +EOF +chmod 0600 .smoke/keyproxy.env .smoke/creds.env + +# In-container bind is all-interfaces ONLY because docker -p publishing +# is the boundary here; the host default (and example config) stays +# loopback. +cat > .smoke/keyproxy.toml </dev/null 2>&1 || true +docker run -d --name "$NAME" -p "$PORT:8082" \ + -v "$PWD:/h" -w /h -u "$(id -u):$(id -g)" -e HOME=/tmp \ + -e KEYPROXY_SMOKE_ENV=smoke-env-material-4444 \ + "$IMAGE" /h/bin/keyproxy serve -config /h/.smoke/keyproxy.toml + +cleanup() { + docker rm -f "$NAME" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "--- probing http://127.0.0.1:$PORT (python urllib; curl banned on host) ---" +BASE_URL="http://127.0.0.1:$PORT" KEYPROXY_TOKEN="smoke-bearer-token-1111" \ + python3 smoke/probe.py +probe_rc=$? + +echo "--- server log (redaction check) ---" +LOG="$(docker logs "$NAME" 2>&1)" +echo "$LOG" + +if echo "$LOG" | grep -q 'smoke-secret-material-2222\|smoke-env-material-4444\|smoke-bearer-token-1111'; then + echo "FAIL: server log contains material" + exit 1 +fi +if ! echo "$LOG" | grep -q 'ref=mpk-smoke=\*\*\*'; then + echo "FAIL: server log does not mask refs as =***" + exit 1 +fi + +exit "$probe_rc"