feat(ups): NUT setup for APC Smart-UPS C 1500 on pfv-tsys1

Deploy Network UPS Tools to monitor the server-room UPS units and feed data to
Home Assistant for power-usage tracking.

APC Smart-UPS C 1500 is fully operational: battery charge, runtime, load,
voltage, and status all reporting via usbhid-ups + APC HID 0.100 subdriver.
upsd listens on Tailscale (100.121.189.98:3493) for HA polling; local upsmon
provides graceful hypervisor shutdown on battery-low.

Key deployment fix: the udev rule must cover SUBSYSTEM=="usb" (raw
/dev/bus/usb) in addition to hidraw, because usbhid-ups opens the raw USB
device file after dropping to the nut user via setuid(). Matching only hidraw
causes EACCES on driver start.

Tripp Lite UPS (09ae:3016) is blocked — driver matches TrippLite HID 0.85
subdriver but fails reading the 878-byte HID Report Descriptor (EAGAIN via USB
hub). Needs physical reseat to direct motherboard USB port. Driver masked to
prevent restart-loop spam.

Files: ups/discover.sh, ups/setup.sh (idempotent), ups/status.sh, ups/README.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

AGENTS.md

@@ -14,6 +14,7 @@ dns-cluster-setup/   Technitium DNS cluster replication
 k8s/                 k3s cluster setup scripts (3-node HA over Tailscale) + docs/
 powerman/            Cyclades PM10i PDU management via powerman
 console/             Serial console management (ser2net + conman) for switches
+ups/                 UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1
 perf/                Proxmox perf tuning, fleet audit, iperf
 proxmox/             Proxmox fleet docs (hardware audit, capacity, k8s host planning)
 netinfra/            pfv-netinfra-01/02 DNS/NTP/DHCP setup
@@ -96,6 +97,7 @@ vendor/              Vendored KNELShellFramework
 | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
 | [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman |
 | [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
+| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
 | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
 
 ## Key Docs

README.md

@@ -14,6 +14,7 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste
 | [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs |
 | [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 |
 | [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
+| [`ups/`](ups/) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale |
 | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
 | [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
 | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |

STATUS.md

@@ -3,7 +3,7 @@
 > **Human read-only. Agents maintain this file automatically after each work
 > session.** Do not edit by hand — the next agent run will overwrite it.
 
-> **Last updated:** 2026-07-29 by Crush (GLM-5.2)
+> **Last updated:** 2026-07-30 by Crush (GLM-5.2)
 >
 > **Capacity targets are now codified** in
 > [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
@@ -68,6 +68,20 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 - [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
 - [ ] Rename outlets to match physical devices (Friday onsite)
 
+### UPS Management (NUT on pfv-tsys1 — LIVE)
+- [x] **APC Smart-UPS C 1500** fully reporting via NUT usbhid-ups + APC HID 0.100
+- [x] upsd listening on Tailscale `100.121.189.98:3493` (port 3493)
+- [x] Local upsmon (master) configured for graceful hypervisor shutdown
+- [x] Home Assistant credentials generated (`homeassistant` user in upsd.users)
+- [x] Scripts in `ups/`: discover, setup (idempotent), status
+- [x] **Key fix:** udev rule covers BOTH `usb` and `hidraw` subsystems (driver
+  opens raw `/dev/bus/usb` after setuid drop, not `/dev/hidraw`)
+- [ ] **Tripp Lite UPS blocked:** driver matches TrippLite HID 0.85 subdriver
+  but fails reading 878-byte HID Report Descriptor (EAGAIN via USB hub).
+  Needs physical reseat to direct motherboard USB port or cable replacement.
+  Driver masked to prevent restart-loop spam.
+- [ ] **Home Assistant NUT integration:** not yet configured in HA UI (user action)
+
 ### Console Management (ser2net telnet(rfc2217) + conman on pfv-tsys4 — LIVE)
 - [x] 7 network switch/router consoles managed via ser2net + conman
 - [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
@@ -146,7 +160,7 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 - **Friday maintenance:** tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
 - **PDU:** Rename outlets in powerman.conf (Friday onsite)
 - **k8s deferred:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out
-- **UPS:** Trace USB cables on tsys1/tsys4 — confirm APC + TrippLite data cables are seated. tsys4 UPS is not enumerating at all (no HID device). tsys1 UPS may be passed through to pfv-bms (100) — verify. Goal: NUT driver + HA integration for power usage tracking + graceful shutdown
+- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port). HA NUT integration pending user config in HA UI.
 
 ## Infrastructure Summary
 
@@ -163,4 +177,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
 | **AWX** | **tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)** |
 | **Console** | **7 switch consoles via ser2net telnet(rfc2217)+conman on pfv-tsys4 (conmand:7890 on Tailscale)** |
+| **UPS** | **APC Smart-UPS C 1500 via NUT on pfv-tsys1 (upsd:3493 on Tailscale)** |
 | Production | Cloudron VPS, Reston VA (this cluster is R&D only) |

docs/docmap.md

@@ -3,7 +3,7 @@
 > **Index of all documentation in this repo.** Agents must update this file
 > whenever a doc is added, removed, or substantively changed.
 
-> **Last updated:** 2026-07-29 (DHCP migration)
+> **Last updated:** 2026-07-30 (UPS/NUT)
 
 ## Kubernetes Architecture ([`k8s/`](k8s/))
 
@@ -48,6 +48,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
 | [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
 | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
 | [`../console/README.md`](../console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
+| [`../ups/README.md`](../ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
 | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
 | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
 | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |

ups/README.md

@@ -0,0 +1,164 @@
+# UPS Management (NUT — Network UPS Tools)
+
+Centralized UPS monitoring for the server room via
+[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS
+units feed one `upsd` network server; Home Assistant polls it over Tailscale for
+real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor
+down gracefully when battery is low.
+
+> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be
+> covered. `apcupsd` only supports APC, so it would require a second daemon
+> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power
+> Device class, and Home Assistant ships a first-class NUT integration.
+
+## Hardware
+
+| UPS | Model | VID:PID | USB Serial | Status |
+|-----|-------|---------|------------|--------|
+| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** |
+| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below |
+
+## Current State (2026-07-30)
+
+### APC Smart-UPS C 1500 — OPERATIONAL
+
+Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated:
+
+```
+battery.charge: 100         battery.runtime: 1800     battery.voltage: 27.4
+ups.status: OL              ups.load: (via HA)         ups.model: Smart-UPS C 1500
+```
+
+### Tripp Lite UPS — BLOCKED (hardware issue)
+
+The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims
+the interface, and reads the HID descriptor — but **fails reading the 878-byte
+HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s).
+The driver is masked to prevent restart-loop spam.
+
+USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and
+`nut-scanner`, but the bulk control transfer for the full report descriptor
+times out. Likely causes:
+
+1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`).
+   Try plugging directly into a motherboard USB port.
+2. **USB cable** — try a high-quality data cable (not charge-only).
+3. **UPS firmware** — the USB controller may not properly implement all HID
+   endpoints.
+
+**To retry after physical reseat:**
+```bash
+# On pfv-tsys1:
+systemctl unmask nut-driver@tripp-lite-ups
+systemctl start nut-driver@tripp-lite-ups
+upsc tripp-lite-ups@localhost
+```
+
+## Architecture
+
+```
+pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
+ ├─ APC Smart-UPS C 1500      ──┐
+ └─ Tripp Lite UPS (masked)  ──┤  USB HID
+                                ▼
+                       nut-driver@apc-smartups-c1500 (usbhid-ups)
+                                ▼
+                    upsd  :3493  (LISTEN 127.0.0.1 + Tailscale)
+                                ▼            ▼
+                  upsmon (local)        Home Assistant (NUT integration)
+                  graceful shutdown      power/load/runtime sensors
+```
+
+- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses
+  templated `nut-driver@<upsname>.service` units managed by
+  `nut-driver-enumerator`.
+- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale
+  only). Clients authenticate via `upsd.users`.
+- **Monitor layer** — `upsmon` runs locally as `master` to trigger
+  `SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`.
+- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale
+  and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors.
+
+### Key deployment lesson: udev must cover raw USB devices
+
+The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files),
+**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user,
+it needs write access to those raw USB files. The udev rule must match
+`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only
+`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`.
+
+## Scripts
+
+All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`:
+
+```bash
+# Idempotent install + configure (safe to re-run):
+PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
+
+# Discover USB UPS + NUT state (read-only diagnostic):
+PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
+
+# Query UPS data + service health:
+PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
+```
+
+`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it
+can be repurposed for other hosts or UPS units. Passwords for `monuser` and
+`homeassistant` are auto-generated on first run and reused on subsequent runs
+(stored in `/etc/nut/upsd.users`).
+
+Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically
+unplugged).
+
+## Configuration files on pfv-tsys1
+
+| File | Purpose |
+|------|---------|
+| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) |
+| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver |
+| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN <tailscale>` on port 3493 |
+| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials |
+| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` |
+| `/etc/nut/nut.conf` | `MODE=netserver` |
+
+## Home Assistant integration
+
+In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**:
+
+| Field | Value |
+|-------|-------|
+| Host | `pfv-tsys1` (or `100.121.189.98`) over Tailscale |
+| Port | `3493` |
+| Username | `homeassistant` |
+| Password | *(printed by `setup.sh` / stored in `/etc/nut/upsd.users`)* |
+| UPS | `apc-smartups-c1500` |
+
+HA auto-discovers available UPS names after authentication. Useful sensors for
+power-usage tracking: `sensor.apc_smartups_c1500_ups_load` (Watts, via `ups.load`
+× VA rating), `sensor.apc_smartups_c1500_battery_runtime`,
+`sensor.apc_smartups_c1500_ups_realpower`.
+
+## Daily operations
+
+From pfv-tsys1 (or any tailnet host with NUT client installed):
+
+```bash
+# List UPS units served by upsd
+upsc -l pfv-tsys1
+
+# Full variable dump for one UPS
+upsc apc-smartups-c1500@pfv-tsys1
+
+# Just the live load + runtime
+upsc apc-smartups-c1500@pfv-tsys1 ups.load
+upsc apc-smartups-c1500@pfv-tsys1 battery.runtime
+```
+
+## Notes
+
+- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves
+  hypervisor graceful-shutdown capability and matches the `powerman/` pattern
+  (PDU managed on the host where the adapter physically lives).
+- **Per-unit power (Watts):** `usbhid-ups` reports `ups.realpower` directly on
+  the APC Smart-UPS C 1500. If absent, derive Watts as
+  `load% × VA_rating × power_factor` in HA templates.

ups/discover.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/bash
+#
+# ups/discover.sh — probe USB UPS units and NUT state on the local host
+#
+# Read-only. Prints everything needed to configure NUT. No changes made.
+#
+# Usage (run ON the target host via remote.sh):
+#   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
+#
+set -uo pipefail
+
+echo "======================================================"
+echo "  UPS / NUT Discovery on $(hostname)"
+echo "======================================================"
+
+# --- 1. USB UPS devices ------------------------------------------------------
+echo ""
+echo "--- [1/5] USB UPS devices (lsusb) ---"
+lsusb 2>/dev/null | grep -iE "UPS|American Power|Tripp|APC" || echo "  (no UPS devices found in lsusb)"
+
+echo ""
+echo "--- [2/5] UPS detail (vendor/product/serial/model) ---"
+# Common UPS vendor IDs: 051d (APC), 09ae (Tripp Lite), 0463 (Eaton),
+# 06da (MGE), 0764 (Cyber Power)
+for vid in 051d 09ae 0463 06da 0764; do
+  while read -r bus dev pid; do
+    [ -n "$bus" ] || continue
+    echo "  --- $bus:$dev ($vid:$pid) ---"
+    lsusb -v -s "${bus}:${dev}" 2>/dev/null \
+      | grep -iE "iManufacturer|iProduct|iSerial|bcdDevice" \
+      | sed 's/^/    /'
+  done < <(lsusb 2>/dev/null | awk -v v="$vid" '$0~v{split($2,a,":"); split($4,b,":"); print a[1], b[1], $6}')
+done
+
+# --- 2. sysfs paths (for udev rules) -----------------------------------------
+echo ""
+echo "--- [3/5] sysfs device paths + serials ---"
+for d in /sys/bus/usb/devices/*; do
+  man=$(cat "$d/manufacturer" 2>/dev/null)
+  prod=$(cat "$d/product" 2>/dev/null)
+  ser=$(cat "$d/serial" 2>/dev/null)
+  vid=$(cat "$d/idVendor" 2>/dev/null)
+  pid=$(cat "$d/idProduct" 2>/dev/null)
+  if echo "$man $prod" | grep -qiE "apc|tripp|power conversion|ups|eaton|mge|cyber power"; then
+    # Resolve stable ID_PATH for udev pinning
+    path=$(udevadm info -q property -p "$d" 2>/dev/null | awk -F= '/^ID_PATH=/{print $2}')
+    echo "  $d"
+    echo "    vendor=$vid product=$pid"
+    echo "    manufacturer=$man"
+    echo "    product=$prod"
+    echo "    serial=$ser"
+    echo "    ID_PATH=$path"
+  fi
+done
+
+# --- 3. HID device nodes -----------------------------------------------------
+echo ""
+echo "--- [4/5] HID device nodes ---"
+ls -la /dev/hidraw* /dev/usb/hiddev* 2>/dev/null || echo "  (no hidraw/hiddev nodes)"
+
+# --- 4. NUT install state ----------------------------------------------------
+echo ""
+echo "--- [5/5] NUT install + service state ---"
+if dpkg -l nut-server nut-client 2>/dev/null | grep -q '^ii'; then
+  echo "  NUT installed:"
+  dpkg -l nut-server nut-client 2>/dev/null | awk '/^ii/{print "    "$2" "$3}'
+else
+  echo "  NUT not installed (apt: nut-server nut-client)"
+fi
+
+echo ""
+echo "  Services:"
+for svc in nut-driver nut-server nut-monitor; do
+  printf "    %-14s " "$svc:"
+  systemctl is-active "$svc" 2>/dev/null || true
+done
+
+echo ""
+echo "  Existing config:"
+# shellcheck disable=SC2012 # ls -la is intentional for human-readable listing
+ls -la /etc/nut/ 2>/dev/null | sed 's/^/    /' || echo "    (no /etc/nut)"
+
+echo ""
+echo "======================================================"
+echo "  Discovery complete."
+echo "======================================================"

ups/setup.sh

@@ -0,0 +1,286 @@
+#!/usr/bin/bash
+#
+# ups/setup.sh — idempotent Network UPS Tools (NUT) setup on pfv-tsys1
+#
+# Installs NUT, configures two USB HID UPS units (APC + Tripp Lite) pinned by
+# USB serial, runs upsd as a network server for Home Assistant polling, and
+# runs upsmon locally so the hypervisor can shut down gracefully on battery.
+#
+# Designed to run ON the target host (pfv-tsys1) as root, idempotent.
+#
+# Usage:
+#   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
+#
+# Overrides (defaults suit pfv-tsys1):
+#   APC_SERIAL         APC UPS USB serial            (default AS1213210423)
+#  APC_VID/APC_PID     APC vendor/product ID          (default 051d / 0003)
+#   APC_NAME           NUT section name for APC       (default apc-smartups-c1500)
+#  TRIPP_SERIAL        Tripp Lite UPS USB serial      (default 2352CVLSM871900694)
+#  TRIPP_VID/TRIPP_PID Tripp Lite vendor/product      (default 09ae / 3016)
+#  TRIPP_NAME          NUT section name for Tripp     (default tripp-lite-ups)
+#  TRIPP_SUBDRIVER     Forced HID subdriver for Tripp (default "TrippLite HID 0.85")
+#  TRIPP_ENABLED       Set to 0 to disable Tripp Lite (default 1)
+#  NUT_LISTEN_IPS      space-separated upsd LISTEN IPs (default: auto Tailscale + 127.0.0.1)
+#  HA_USER             upsd username for HA           (default homeassistant)
+#  HA_PASSWORD         upsd password for HA           (default: reuse or generate)
+#  MON_USER            upsd username for local upsmon (default monuser)
+#  MON_PASSWORD        upsd password for upsmon       (default: reuse or generate)
+#
+set -euo pipefail
+
+# --- Config (overridable via env) ---
+APC_SERIAL="${APC_SERIAL:-AS1213210423}"
+APC_VID="${APC_VID:-051d}"
+APC_PID="${APC_PID:-0003}"
+APC_NAME="${APC_NAME:-apc-smartups-c1500}"
+
+TRIPP_SERIAL="${TRIPP_SERIAL:-2352CVLSM871900694}"
+TRIPP_VID="${TRIPP_VID:-09ae}"
+TRIPP_PID="${TRIPP_PID:-3016}"
+TRIPP_NAME="${TRIPP_NAME:-tripp-lite-ups}"
+TRIPP_SUBDRIVER="${TRIPP_SUBDRIVER:-TrippLite HID 0.85}"
+TRIPP_ENABLED="${TRIPP_ENABLED:-1}"
+
+HA_USER="${HA_USER:-homeassistant}"
+MON_USER="${MON_USER:-monuser}"
+NUT_PORT="${NUT_PORT:-3493}"
+UDEV_RULE="/etc/udev/rules.d/99-nut-ups.rules"
+
+UPS_CONF="/etc/nut/ups.conf"
+UPSD_CONF="/etc/nut/upsd.conf"
+UPSD_USERS="/etc/nut/upsd.users"
+UPS_CONF_MON="/etc/nut/upsmon.conf"
+NUT_CONF="/etc/nut/nut.conf"
+
+# --- Helpers ---
+gen_pw() { head -c 24 /dev/urandom | base64 | tr -d '/+=' | cut -c1-20; }
+
+# Reuse existing passwords if config already present (idempotent re-runs)
+extract_pw() {  # $1=user
+  if [ -f "$UPSD_USERS" ]; then
+    awk -v u="[$1]" '
+      $0==u {inblk=1; next}
+      /^\[/ {inblk=0}
+      inblk && $1=="password" {gsub(/"/,"",$3); print $3; exit}
+    ' "$UPSD_USERS" 2>/dev/null
+  fi
+}
+
+echo "============================================"
+echo "  NUT UPS Setup on $(hostname)"
+echo "  APC:        $APC_NAME  ($APC_VID:$APC_PID serial $APC_SERIAL)"
+if [ "$TRIPP_ENABLED" = "1" ]; then
+  echo "  Tripp Lite: $TRIPP_NAME ($TRIPP_VID:$TRIPP_PID serial $TRIPP_SERIAL)"
+else
+  echo "  Tripp Lite: DISABLED (TRIPP_ENABLED=0)"
+fi
+echo "============================================"
+
+# --- 0. Resolve / generate passwords (idempotent) ---
+MON_PASSWORD="${MON_PASSWORD:-$(extract_pw "$MON_USER")}"
+HA_PASSWORD="${HA_PASSWORD:-$(extract_pw "$HA_USER")}"
+[ -n "$MON_PASSWORD" ] || MON_PASSWORD="$(gen_pw)"
+[ -n "$HA_PASSWORD" ] || HA_PASSWORD="$(gen_pw)"
+
+# --- 0b. Auto-detect Tailscale listen IP for upsd (clients on tailnet) ---
+if [ -z "${NUT_LISTEN_IPS:-}" ]; then
+  TS_IP=$(tailscale ip -4 2>/dev/null || true)
+  if [ -n "$TS_IP" ]; then
+    NUT_LISTEN_IPS="127.0.0.1 ${TS_IP}"
+  else
+    NUT_LISTEN_IPS="127.0.0.1"
+    echo "  WARNING: No Tailscale IP detected. Listening on localhost only."
+  fi
+fi
+echo "  upsd LISTEN IPs: ${NUT_LISTEN_IPS:-<none>}"
+
+# --- 1. Install NUT ---
+echo ""
+echo "--- [1/8] Installing NUT (nut-server, nut-client) ---"
+if dpkg -l nut-server 2>/dev/null | grep -q '^ii'; then
+  echo "  NUT already installed: $(dpkg -l nut-server | awk '/^ii/{print $3}')"
+else
+  apt-get update -qq && apt-get install -y -qq nut-server nut-client
+fi
+mkdir -p /etc/nut
+
+# --- 2. udev rules: grant nut group access to BOTH raw USB + hidraw devices ---
+# CRITICAL: usbhid-ups opens /dev/bus/usb/BBB/DDD (raw USB), not /dev/hidraw.
+# The driver drops to the nut user via setuid(), so the nut group needs write
+# access to the raw USB device files. Matching on subsystem=="usb" by VID:PID
+# is required because ATTRS{serial} does not reliably traverse for usb devices.
+echo ""
+echo "--- [2/8] Writing udev rules (raw USB + hidraw, group nut) ---"
+{
+  echo "# Stable permissions for NUT USB HID UPS units"
+  echo "# Generated by ups/setup.sh — grants the 'nut' group access to both"
+  echo "# the raw USB device files (/dev/bus/usb) and hidraw devices."
+  echo "# Match BOTH subsystems: the usbhid-ups driver opens the raw USB device"
+  echo "# after dropping to the nut user via setuid()."
+  echo ""
+  echo "# APC Smart-UPS C 1500 ($APC_VID:$APC_PID)"
+  echo "SUBSYSTEM==\"usb\",    ATTR{idVendor}==\"$APC_VID\", ATTR{idProduct}==\"$APC_PID\", GROUP=\"nut\", MODE=\"0664\""
+  echo "SUBSYSTEM==\"hidraw\", ATTRS{serial}==\"$APC_SERIAL\",                   GROUP=\"nut\", MODE=\"0660\""
+  echo ""
+  echo "# Tripp Lite UPS ($TRIPP_VID:$TRIPP_PID)"
+  echo "SUBSYSTEM==\"usb\",    ATTR{idVendor}==\"$TRIPP_VID\", ATTR{idProduct}==\"$TRIPP_PID\", GROUP=\"nut\", MODE=\"0664\""
+  echo "SUBSYSTEM==\"hidraw\", ATTRS{serial}==\"$TRIPP_SERIAL\",                   GROUP=\"nut\", MODE=\"0660\""
+} > "$UDEV_RULE"
+echo "  Written: $UDEV_RULE"
+udevadm control --reload-rules 2>/dev/null || true
+udevadm trigger --subsystem-match=usb 2>/dev/null || true
+udevadm trigger --subsystem-match=hidraw 2>/dev/null || true
+sleep 1
+
+# --- 3. ups.conf ---
+echo ""
+echo "--- [3/8] Writing ups.conf ---"
+{
+  echo "# NUT UPS devices — generated by ups/setup.sh on $(date)"
+  echo ""
+  echo "maxretry = 3"
+  echo ""
+  echo "[${APC_NAME}]"
+  echo "    driver = usbhid-ups"
+  echo "    port = auto"
+  echo "    vendorid = ${APC_VID}"
+  echo "    productid = ${APC_PID}"
+  echo "    serial = ${APC_SERIAL}"
+  echo "    desc = \"APC Smart-UPS C 1500\""
+  if [ "$TRIPP_ENABLED" = "1" ]; then
+    echo ""
+    echo "[${TRIPP_NAME}]"
+    echo "    driver = usbhid-ups"
+    echo "    port = auto"
+    echo "    vendorid = ${TRIPP_VID}"
+    echo "    productid = ${TRIPP_PID}"
+    echo "    serial = ${TRIPP_SERIAL}"
+    echo "    subdriver = \"${TRIPP_SUBDRIVER}\""
+    echo "    desc = \"Tripp Lite UPS\""
+  fi
+} > "$UPS_CONF"
+echo "  Written: $UPS_CONF"
+
+# --- 4. upsd.conf: network server (localhost + Tailscale for HA) ---
+echo ""
+echo "--- [4/8] Writing upsd.conf ---"
+{
+  echo "# NUT upsd — generated by ups/setup.sh on $(date)"
+  for ip in $NUT_LISTEN_IPS; do
+    echo "LISTEN ${ip} ${NUT_PORT}"
+  done
+  echo "MAXAGE 25"
+} > "$UPSD_CONF"
+echo "  Written: $UPSD_CONF (LISTEN: $(echo "$NUT_LISTEN_IPS" | tr '\n' ' '))"
+
+# --- 5. upsd.users: monuser (master) + homeassistant (read-only monitor) ---
+echo ""
+echo "--- [5/8] Writing upsd.users ---"
+cat > "$UPSD_USERS" <<USERS
+# NUT upsd users — generated by ups/setup.sh on $(date)
+# monuser       : local upsmon (master) — graceful hypervisor shutdown
+# ${HA_USER} : Home Assistant NUT integration (read-only polling)
+
+[${MON_USER}]
+    password = "${MON_PASSWORD}"
+    upsmon master
+
+[${HA_USER}]
+    password = "${HA_PASSWORD}"
+    upsmon slave
+USERS
+echo "  Written: $UPSD_USERS"
+
+# --- 6. upsmon.conf + nut.conf ---
+echo ""
+echo "--- [6/8] Writing upsmon.conf + nut.conf ---"
+{
+  echo "# NUT upsmon (local monitor) — generated by ups/setup.sh on $(date)"
+  echo "# Monitors the APC UPS as master. On battery-low, shuts the host down."
+  echo ""
+  echo "MONITOR ${APC_NAME}@localhost 1 ${MON_USER} \"${MON_PASSWORD}\" master"
+  if [ "$TRIPP_ENABLED" = "1" ]; then
+    echo "MONITOR ${TRIPP_NAME}@localhost 1 ${MON_USER} \"${MON_PASSWORD}\" master"
+  fi
+  echo ""
+  echo "SHUTDOWNCMD \"/sbin/shutdown -h now\""
+  echo "POWERDOWNFLAG /etc/killpower"
+  echo "NOTIFYFLAG ONLINE  SYSLOG+WALL"
+  echo "NOTIFYFLAG ONBATT  SYSLOG+WALL"
+  echo "NOTIFYFLAG LOWBATT SYSLOG+WALL"
+  echo "POLLFREQ 15"
+  echo "POLLFREQALERT 5"
+  echo "HOSTSYNC 15"
+} > "$UPS_CONF_MON"
+
+cat > "$NUT_CONF" <<NUTCONF
+# NUT mode — generated by ups/setup.sh on $(date)
+# netserver = this host runs drivers + upsd; serves UPS data to clients (HA).
+MODE=netserver
+NUTCONF
+echo "  Written: $UPS_CONF_MON + $NUT_CONF"
+
+# --- 6b. Fix ownership/permissions (Debian: nut group must read config) ---
+chown root:nut "$UPS_CONF" "$UPSD_CONF" "$UPSD_USERS" "$UPS_CONF_MON" 2>/dev/null || true
+chmod 640 "$UPS_CONF" "$UPSD_CONF" "$UPSD_USERS" "$UPS_CONF_MON" 2>/dev/null || true
+chmod 644 "$NUT_CONF" 2>/dev/null || true
+
+# --- 7. Start services (Debian uses templated nut-driver@<name> units) ---
+echo ""
+echo "--- [7/8] Starting NUT services ---"
+
+# Re-read ups.conf to generate per-UPS driver instances
+systemctl restart nut-driver-enumerator 2>/dev/null || true
+sleep 2
+
+# Start per-UPS driver instances
+systemctl restart "nut-driver@${APC_NAME}" 2>/dev/null || true
+if [ "$TRIPP_ENABLED" = "1" ]; then
+  systemctl restart "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
+else
+  systemctl stop "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
+  systemctl mask "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
+fi
+sleep 3
+systemctl restart nut-server 2>/dev/null || true
+sleep 1
+systemctl restart nut-monitor 2>/dev/null || true
+
+echo ""
+echo "  Service status:"
+for svc in "nut-driver@${APC_NAME}" "nut-driver@${TRIPP_NAME}" nut-server nut-monitor; do
+  if systemctl list-unit-files "$svc" >/dev/null 2>&1; then
+    printf "    %-42s " "$svc"
+    systemctl is-active "$svc" 2>/dev/null || echo "(unknown)"
+  fi
+done
+
+# --- 8. Validate ---
+echo ""
+echo "--- [8/8] Validation ---"
+echo ""
+echo "  upsc — ${APC_NAME}:"
+upsc "${APC_NAME}@localhost" 2>&1 | head -25 || echo "    (APC UPS not responding yet)"
+if [ "$TRIPP_ENABLED" = "1" ]; then
+  echo ""
+  echo "  upsc — ${TRIPP_NAME}:"
+  upsc "${TRIPP_NAME}@localhost" 2>&1 | head -25 || echo "    (Tripp Lite UPS not responding yet)"
+fi
+
+echo ""
+echo "============================================"
+echo "  Setup complete."
+echo ""
+echo "  Home Assistant NUT integration:"
+echo "    Host:     $(echo "$NUT_LISTEN_IPS" | awk '{print $2}')   (or any LISTEN IP above)"
+echo "    Port:     ${NUT_PORT}"
+echo "    Username: ${HA_USER}"
+echo "    Password: ${HA_PASSWORD}"
+if [ "$TRIPP_ENABLED" = "1" ]; then
+  echo "    UPS names: ${APC_NAME}, ${TRIPP_NAME}"
+else
+  echo "    UPS names: ${APC_NAME}"
+fi
+echo ""
+echo "  Save the HA password now — it is stored in ${UPSD_USERS}."
+echo "============================================"

ups/status.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/bash
+#
+# ups/status.sh — query NUT UPS state + service health (read-only)
+#
+# Usage (run ON the target host via remote.sh):
+#   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
+#
+# shellcheck disable=SC2012 # ss/awk field extraction is intentional
+set -uo pipefail
+
+APC_NAME="${APC_NAME:-apc-smartups-c1500}"
+TRIPP_NAME="${TRIPP_NAME:-tripp-lite-ups}"
+
+echo "======================================================"
+echo "  NUT UPS Status on $(hostname)"
+echo "======================================================"
+
+echo ""
+echo "--- Services ---"
+for svc in "nut-driver@${APC_NAME}" "nut-driver@${TRIPP_NAME}" nut-server nut-monitor; do
+  if systemctl list-unit-files "$svc" >/dev/null 2>&1; then
+    printf "  %-42s " "$svc"
+    systemctl is-active "$svc" 2>/dev/null || echo "(unknown)"
+  fi
+done
+
+for ups in "$APC_NAME" "$TRIPP_NAME"; do
+  echo ""
+  echo "--- ${ups} ---"
+  if upsc "${ups}@localhost" >/tmp/.nutstatus.$$ 2>&1; then
+    awk -v u="$ups" '
+      BEGIN{printf "  %s\n", u}
+      /^battery\.charge:/      {printf "    battery.charge:      %s\n", $3}
+      /^battery\.runtime:/     {printf "    battery.runtime:     %ss (%.0f min)\n", $3, $3/60}
+      /^battery\.voltage:/     {printf "    battery.voltage:     %s\n", $3}
+      /^ups\.status:/          {printf "    ups.status:          %s\n", $3}
+      /^ups\.load:/            {printf "    ups.load:            %s%%\n", $3}
+      /^ups\.power:/           {printf "    ups.power:           %s\n", $3}
+      /^ups\.realpower:/       {printf "    ups.realpower:       %s W\n", $3}
+      /^input\.voltage:/       {printf "    input.voltage:       %s\n", $3}
+      /^output\.voltage:/      {printf "    output.voltage:      %s\n", $3}
+      /^ups\.model:/           {printf "    ups.model:           %s\n", $3}
+      /^ups\.serial:/          {printf "    ups.serial:          %s\n", $3}
+      /^device\.mfr:/          {printf "    device.mfr:          %s\n", $3}
+    ' /tmp/.nutstatus.$$
+    echo "    (full dump: upsc ${ups}@localhost)"
+  else
+    echo "  NOT RESPONDING:"
+    sed 's/^/    /' /tmp/.nutstatus.$$
+  fi
+  rm -f /tmp/.nutstatus.$$
+done
+
+echo ""
+echo "--- upsd LISTEN sockets ---"
+ss -ltnp 2>/dev/null | grep -E "3493|nut" | sed 's/^/  /' || echo "  (upsd not listening on 3493)"
+
+echo ""
+echo "======================================================"