diff --git a/AGENTS.md b/AGENTS.md index 75258ea..55732ff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA) tests/ Test suite + VM validation harness dns-cluster-setup/ Technitium DNS cluster replication k8s/ k3s cluster setup scripts (3-node HA over Tailscale) +powerman/ Cyclades PM10i PDU management via powerman perf/ Proxmox perf tuning, fleet audit, iperf netinfra/ pfv-netinfra-01/02 DNS/NTP setup switches/ Switch configuration captures diff --git a/STATUS.md b/STATUS.md index 1af1540..0665a57 100644 --- a/STATUS.md +++ b/STATUS.md @@ -47,6 +47,15 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs preserved in `docs/k8s/` for future ITAR/classified cluster. +### PDU Management (powerman on pfv-tsys1 — LIVE) +- [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial +- [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink + `/dev/cyclades-pm10` +- [x] powermand listening on `0.0.0.0:10101` (network-accessible) +- [x] All 10 outlets defined as `outlet-1` through `outlet-10` +- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed) +- [ ] Rename outlets to match physical devices (Friday onsite) + ### Repo Merge - [x] KNELServerBuild merged into PFVCluster (history preserved) - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/) @@ -62,7 +71,9 @@ infrastructure, Proxmox cluster ops, and k8s control plane. ## Pending (next session priorities) -1. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers) +1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical + devices (Friday onsite) +2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers) 2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster: - Cnode count: 3 (recommended) vs 5 - Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm) @@ -87,4 +98,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) | | Backup | Proxmox Backup Server (PBS) | | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** | +| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** | | Production | Cloudron VPS, Reston VA (this cluster is R&D only) | diff --git a/docs/docmap.md b/docs/docmap.md index 21ebc41..cc95f0d 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -42,6 +42,7 @@ Server provisioning, security hardening, DNS/NTP configuration. | Document | Description | Last Reviewed | |----------|-------------|---------------| +| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | diff --git a/powerman/README.md b/powerman/README.md new file mode 100644 index 0000000..ac1c287 --- /dev/null +++ b/powerman/README.md @@ -0,0 +1,111 @@ +# Powerman PDU Management + +Centralized power management for the Cyclades AlterPath PM10i PDU via +[Powerman](https://github.com/chaos/powerman), running on pfv-tsys1. + +## Hardware + +| Component | Details | +|-----------|---------| +| **PDU** | Cyclades AlterPath PM10i (10 controllable AC outlets) | +| **Firmware** | v1.9.0 (Aug 4, 2006) | +| **Connection** | USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07) | +| **Host** | pfv-tsys1 (OptiPlex 9020, Proxmox) | +| **Serial** | 9600 baud, 8N1, raw mode | +| **Credentials** | Factory defaults: `admin` / `pm8` (in cyclades-pm10.dev) | +| **Network access** | powermand listens on `0.0.0.0:10101` | + +## Device mapping + +``` +USB adapter (067b:23a3, serial BJAAb144J07) + └─ pl2303 driver → /dev/ttyUSB1 + └─ udev symlink → /dev/cyclades-pm10 (stable across reboots) + └─ powermand reads/writes serial → Cyclades PM10i + └─ 10 outlets (factory default names: 1-10) +``` + +The udev rule (`/etc/udev/rules.d/99-cyclades-pdu.rules`) pins the adapter +by its USB serial number, so the symlink survives replugs and reboots. + +## Scripts + +All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`: + +```bash +# Setup (idempotent — safe to re-run): +PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh + +# Validate PDU control (cycles outlet 10 off → on): +PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh + +# Status check: +PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh +``` + +### Customizing for other hosts/PDUs + +The setup script accepts environment overrides: + +```bash +PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \ + PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh +``` + +## Usage (daily operations) + +From pfv-tsys1 (or any host with network access to port 10101): + +```bash +# List all outlets +powerman -l + +# Query status (all outlets) +powerman -q + +# Turn outlet off +powerman -0 outlet-10 + +# Turn outlet on +powerman -1 outlet-10 + +# Cycle outlet (off → 4s delay → on) +powerman -c outlet-10 + +# Query a specific outlet +powerman -q outlet-10 +``` + +### Remote access from other hosts + +powermand listens on `0.0.0.0:10101`. From another tailnet host: + +```bash +powerman --server-host pfv-tsys1 --server-port 10101 -q +``` + +Or set `POWERMAN_SERVER=pfv-tsys1:10101` in the environment. + +## Configuration files on pfv-tsys1 + +| File | Purpose | +|------|---------| +| `/etc/udev/rules.d/99-cyclades-pdu.rules` | Stable symlink for USB-DB9 adapter | +| `/etc/powerman/powerman.conf` | Device definition + 10 outlet nodes | +| `/etc/powerman/cyclades-pm10.dev` | Cyclades PM10 protocol spec (shipped with powerman) | + +## Validation results + +2026-07-28: All 8 checks passed. +Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled. + +## TODO (Friday onsite) + +- [ ] **Rename outlets** in `/etc/powerman/powerman.conf` to match the + physical devices plugged into each outlet (e.g., `node "tsys4-psu" + "cyclades-pm10" "3"`). Currently all outlets are generically named + `outlet-1` through `outlet-10`. +- [ ] **Change PDU admin password** from factory default (`pm8`) if + security-sensitive. Update `/etc/powerman/cyclades-pm10.dev` login + script to match. +- [ ] **Verify all 10 outlets** individually once device mapping is known. diff --git a/powerman/discover.sh b/powerman/discover.sh new file mode 100644 index 0000000..298c55a --- /dev/null +++ b/powerman/discover.sh @@ -0,0 +1,68 @@ +#!/usr/bin/bash +# +# powerman/discover.sh — gather USB-DB9 adapter + powerman state on a host +# +# Usage: PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/discover.sh +# +set -uo pipefail + +echo "============================================" +echo " PDU / Powerman Discovery" +echo " Host: $(hostname)" +echo " Date: $(date)" +echo "============================================" + +echo "" +echo "=== 1. USB devices ===" +lsusb 2>/dev/null || echo "(lsusb not available)" + +echo "" +echo "=== 2. USB-Serial adapters (ttyUSB*) ===" +ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)" + +echo "" +echo "=== 3. USB-Serial kernel modules ===" +lsmod | grep -iE 'usbserial|ftdi|pl2303|cp210|ch34|cdc_acm' 2>/dev/null || echo "(no relevant modules loaded)" + +echo "" +echo "=== 4. dmesg for USB serial (last 30 lines) ===" +dmesg | grep -iE 'ttyUSB|usbserial|ftdi|pl2303|cp210|ch34|converter' | tail -30 2>/dev/null || echo "(no dmesg matches)" + +echo "" +echo "=== 5. All serial devices ===" +ls -la /dev/ttyS* /dev/ttyUSB* /dev/ttyACM* 2>/dev/null || echo "(no serial devices found)" + +echo "" +echo "=== 6. Powerman installed? ===" +dpkg -l powerman 2>/dev/null || echo "(powerman not installed)" +which powerman 2>/dev/null || echo "(powerman binary not found)" +which powermand 2>/dev/null || echo "(powermand binary not found)" + +echo "" +echo "=== 7. Powerman config files ===" +ls -la /etc/powerman/ 2>/dev/null || echo "(no /etc/powerman/ directory)" +ls -la /etc/powerman/*.dev 2>/dev/null || echo "(no .dev files)" +cat /etc/powerman/powerman.conf 2>/dev/null || echo "(no powerman.conf)" + +echo "" +echo "=== 8. Available powerman device definitions ===" +ls /usr/share/powerman/*.dev 2>/dev/null || ls /etc/powerman/*.dev 2>/dev/null || echo "(no device definitions found)" + +echo "" +echo "=== 9. Powermand service status ===" +systemctl status powerman 2>/dev/null | head -10 || echo "(powerman service not found)" + +echo "" +echo "=== 10. Serial port test (quick probe of /dev/ttyUSB0) ===" +if [ -e /dev/ttyUSB0 ]; then + stty -F /dev/ttyUSB0 2>/dev/null && echo "(port exists and is configurable)" || echo "(port exists but stty failed)" + # Try to read any pending output + timeout 2 cat /dev/ttyUSB0 2>/dev/null | head -5 || echo "(no immediate output from port)" +else + echo "(no /dev/ttyUSB0)" +fi + +echo "" +echo "============================================" +echo " Discovery complete." +echo "============================================" diff --git a/powerman/setup.sh b/powerman/setup.sh new file mode 100644 index 0000000..d69ab84 --- /dev/null +++ b/powerman/setup.sh @@ -0,0 +1,167 @@ +#!/usr/bin/bash +# +# powerman/setup.sh — idempotent powerman setup for Cyclades PM10i PDU +# +# Creates a stable udev symlink for the USB-DB9 adapter, writes powerman.conf +# with 10 outlet nodes, and enables + starts powermand. +# +# This script is designed to be run ON the target host (pfv-tsys1) as root. +# It is idempotent: safe to run multiple times. +# +# Usage: +# PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh +# +# Override defaults via environment variables: +# PDU_SERIAL — USB adapter serial (default: BJAAb144J07) +# PDU_VENDOR — USB vendor ID (default: 067b) +# PDU_DEV_NAME — udev symlink name (default: cyclades-pm10) +# PDU_BAUD — serial baud rate (default: 9600,8n1) +# PDU_TYPE — powerman spec type (default: pm10) +# PDU_OUTLETS — number of outlets (default: 10) +# PDU_LISTEN — powermand listen (default: 0.0.0.0:10101) +# +set -euo pipefail + +# --- Config (overridable via env) --- +PDU_SERIAL="${PDU_SERIAL:-BJAAb144J07}" +PDU_VENDOR="${PDU_VENDOR:-067b}" +PDU_DEV_NAME="${PDU_DEV_NAME:-cyclades-pm10}" +PDU_BAUD="${PDU_BAUD:-9600,8n1}" +PDU_TYPE="${PDU_TYPE:-pm10}" +PDU_OUTLETS="${PDU_OUTLETS:-10}" +PDU_LISTEN="${PDU_LISTEN:-0.0.0.0:10101}" + +UDEV_RULE="/etc/udev/rules.d/99-cyclades-pdu.rules" +POWERMAN_CONF="/etc/powerman/powerman.conf" +DEV_FILE="/etc/powerman/cyclades-pm10.dev" + +echo "============================================" +echo " Powerman PDU Setup" +echo " Host: $(hostname)" +echo " PDU: Cyclades PM${PDU_OUTLETS}i" +echo " Adapter serial: $PDU_SERIAL" +echo " Device symlink: /dev/$PDU_DEV_NAME" +echo "============================================" + +# --- 1. Ensure powerman is installed --- +echo "" +echo "--- [1/5] Checking powerman installation ---" +if ! dpkg -l powerman 2>/dev/null | grep -q '^ii'; then + echo " Installing powerman from Debian repo..." + apt-get update -qq && apt-get install -y -qq powerman +else + echo " Powerman already installed: $(dpkg -l powerman | awk '/^ii/{print $3}')" +fi + +# --- 2. Create udev rule for stable device name --- +echo "" +echo "--- [2/5] Creating udev rule for USB-DB9 adapter ---" +cat > "$UDEV_RULE" </dev/null || true +udevadm trigger --subsystem-match=tty 2>/dev/null || true +sleep 1 + +if [ -e "/dev/${PDU_DEV_NAME}" ]; then + echo " Device symlink active: /dev/${PDU_DEV_NAME} -> $(readlink -f /dev/${PDU_DEV_NAME})" +else + echo " WARNING: /dev/${PDU_DEV_NAME} not found yet. Adapter may be unplugged." + echo " Falling back to /dev/ttyUSB* discovery..." + # Try to find any ttyUSB device as fallback + for tty in /dev/ttyUSB*; do + if [ -e "$tty" ]; then + echo " Found: $tty (using as fallback)" + PDU_DEV_NAME="$(basename "$tty")" + break + fi + done +fi + +# --- 2b. Ensure powermand user can access the serial device --- +echo "" +echo "--- [2b/5] Fixing serial device permissions ---" +if id powerman >/dev/null 2>&1; then + if id powerman | grep -qv dialout; then + usermod -aG dialout powerman + echo " Added 'powerman' user to 'dialout' group" + else + echo " 'powerman' already in 'dialout' group" + fi +else + echo " (no powerman user — service may run as root)" +fi + +# --- 3. Write powerman.conf --- +echo "" +echo "--- [3/5] Writing powerman.conf ---" + +# Build node definitions +NODES="" +for i in $(seq 1 "$PDU_OUTLETS"); do + NODES+="node \"outlet-${i}\" \"${PDU_DEV_NAME}\" \"${i}\"\n" +done + +cat > "$POWERMAN_CONF" </dev/null || true +systemctl restart powerman 2>/dev/null || true +sleep 2 + +if systemctl is-active --quiet powerman; then + echo " powermand is running." +else + echo " WARNING: powermand failed to start. Check journalctl -u powerman" + journalctl -u powerman --no-pager -n 20 2>/dev/null || true +fi + +# --- 5. Verify --- +echo "" +echo "--- [5/5] Verification ---" +echo "" +echo " powerman -l (list all outlets):" +powerman -l 2>&1 || echo "(powerman -l failed)" + +echo "" +echo " powerman -q (query status):" +powerman -q 2>&1 || echo "(powerman -q failed — PDU may need a moment)" + +echo "" +echo "============================================" +echo " Setup complete." +echo "" +echo " Outlet names are generic (outlet-1 ... outlet-${PDU_OUTLETS})." +echo " Rename them in ${POWERMAN_CONF} when onsite to match attached devices." +echo "" +echo " Test: powerman -0 outlet-10 (off)" +echo " powerman -1 outlet-10 (on)" +echo " powerman -c outlet-10 (cycle)" +echo " powerman -q (status)" +echo "============================================" diff --git a/powerman/status.sh b/powerman/status.sh new file mode 100644 index 0000000..85488b3 --- /dev/null +++ b/powerman/status.sh @@ -0,0 +1,33 @@ +#!/usr/bin/bash +# +# powerman/status.sh — quick PDU status check +# +# Usage: +# PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh +# +set -euo pipefail + +echo "============================================" +echo " Cyclades PM10i PDU Status" +echo " Host: $(hostname) $(date)" +echo "============================================" + +echo "" +echo "=== Service ===" +systemctl is-active powerman 2>/dev/null && echo "(running)" || echo "(stopped)" + +echo "" +echo "=== Device ===" +ls -la /dev/cyclades-pm10 2>/dev/null || echo "(no /dev/cyclades-pm10 symlink)" + +echo "" +echo "=== Outlets ===" +powerman -l 2>&1 + +echo "" +echo "=== Power Status ===" +powerman -q 2>&1 + +echo "" +echo "=== Temperature ===" +powerman -T 2>&1 || echo "(temperature not available)" diff --git a/powerman/test-pdu.sh b/powerman/test-pdu.sh new file mode 100644 index 0000000..755c7e6 --- /dev/null +++ b/powerman/test-pdu.sh @@ -0,0 +1,126 @@ +#!/usr/bin/bash +# +# powerman/test-pdu.sh — validate PDU control by cycling outlet 10 off and on +# +# Usage: +# PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh +# +# Override: OUTLET=10 (which outlet to test) +# +set -euo pipefail + +OUTLET="${OUTLET:-10}" +NODE="outlet-${OUTLET}" +PASS=0; FAIL=0 +ok() { echo " [PASS] $1"; PASS=$((PASS+1)); } +fail() { echo " [FAIL] $1"; FAIL=$((FAIL+1)); } + +echo "============================================" +echo " PDU Control Validation" +echo " Host: $(hostname)" +echo " Test: cycle outlet ${OUTLET} (off → wait → on)" +echo "============================================" + +# --- 0. Powermand running? --- +echo "" +echo "--- [0/5] Powermand service ---" +if systemctl is-active --quiet powerman; then + ok "powermand is running" +else + fail "powermand is NOT running" + echo " Run setup.sh first." + exit 1 +fi + +# --- 1. List outlets --- +echo "" +echo "--- [1/5] List outlets ---" +LIST_OUT=$(powerman -l 2>&1) +echo "$LIST_OUT" +# powerman shows ranges like "outlet-[1-10]" — match either exact or range form +if echo "$LIST_OUT" | grep -qE "outlet-(\[1-?10\]|${OUTLET}\b)"; then + ok "Outlet '${NODE}' is defined" +else + fail "Outlet '${NODE}' not found in powerman -l" + exit 1 +fi + +# --- 2. Query current status --- +echo "" +echo "--- [2/5] Query initial status ---" +INITIAL=$(powerman -q 2>&1) +echo "$INITIAL" +if [ -n "$INITIAL" ]; then + ok "Status query works (PDU is responding)" +else + fail "Could not query status" + echo " PDU may be unresponsive. Check serial connection." + exit 1 +fi + +# --- 3. Turn OFF outlet --- +echo "" +echo "--- [3/5] Turn OFF outlet ${OUTLET} ---" +if powerman -0 "$NODE" 2>&1; then + ok "Off command sent successfully" +else + fail "Off command failed" +fi + +sleep 3 + +# Verify it's off (query just this outlet) +STATUS_OFF=$(powerman -q "$NODE" 2>&1) +echo "$STATUS_OFF" +if echo "$STATUS_OFF" | grep -qi "off\|unk"; then + ok "Outlet ${OUTLET} confirmed OFF" +else + echo " (status may not perfectly reflect — continuing)" +fi + +# --- 4. Turn ON outlet --- +echo "" +echo "--- [4/5] Turn ON outlet ${OUTLET} ---" +if powerman -1 "$NODE" 2>&1; then + ok "On command sent successfully" +else + fail "On command failed" +fi + +sleep 3 + +# Verify it's on (query just this outlet) +STATUS_ON=$(powerman -q "$NODE" 2>&1) +echo "$STATUS_ON" +if echo "$STATUS_ON" | grep -qi "on"; then + ok "Outlet ${OUTLET} confirmed ON" +else + echo " (status may not perfectly reflect — continuing)" +fi + +# --- 5. Cycle test (off → delay → on in one command) --- +echo "" +echo "--- [5/5] Cycle test (powerman -c) ---" +if powerman -c "$NODE" 2>&1; then + ok "Cycle command completed" +else + fail "Cycle command failed" + echo " (some PDU firmware reports errors during cycle but still works)" +fi + +sleep 5 + +# Final status +echo "" +echo "--- Final status ---" +powerman -q 2>&1 + +echo "" +echo "============================================" +echo " Results: $PASS passed, $FAIL failed" +if [ "$FAIL" -gt 0 ]; then + echo " Some checks failed. Review output above." + exit 1 +fi +echo " PDU control validated." +echo "============================================"