PFVCluster/ups/README.md

# UPS Management (NUT — Network UPS Tools)

Centralized UPS monitoring for the server room via
[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS
units feed one `upsd` network server; Home Assistant polls it over Tailscale for
real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor
down gracefully when battery is low.

> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be
> covered. `apcupsd` only supports APC, so it would require a second daemon
> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power
> Device class, and Home Assistant ships a first-class NUT integration.

## Hardware

| UPS | Model | VID:PID | USB Serial | Status |
|-----|-------|---------|------------|--------|
| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** |
| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below |

## Current State (2026-07-30)

### APC Smart-UPS C 1500 — OPERATIONAL

Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated:

```
battery.charge: 100         battery.runtime: 1800     battery.voltage: 27.4
ups.status: OL              ups.load: (via HA)         ups.model: Smart-UPS C 1500
```

### Tripp Lite UPS — BLOCKED (hardware issue)

The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims
the interface, and reads the HID descriptor — but **fails reading the 878-byte
HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s).
The driver is masked to prevent restart-loop spam.

USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and
`nut-scanner`, but the bulk control transfer for the full report descriptor
times out. Likely causes:

1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`).
   Try plugging directly into a motherboard USB port.
2. **USB cable** — try a high-quality data cable (not charge-only).
3. **UPS firmware** — the USB controller may not properly implement all HID
   endpoints.

**To retry after physical reseat:**
```bash
# On pfv-tsys1:
systemctl unmask nut-driver@tripp-lite-ups
systemctl start nut-driver@tripp-lite-ups
upsc tripp-lite-ups@localhost
```

## Architecture

```
pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
 ├─ APC Smart-UPS C 1500      ──┐
 └─ Tripp Lite UPS (masked)  ──┤  USB HID
                                ▼
                       nut-driver@apc-smartups-c1500 (usbhid-ups)
                                ▼
          upsd  :3493  (LISTEN 127.0.0.1 + Tailscale + LAN)
                                ▼            ▼
                  upsmon (local)        Home Assistant (NUT integration)
                  graceful shutdown      via LAN 192.168.3.11 (HAOS can't
                                         route to Tailscale IPs)
```

- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses
  templated `nut-driver@<upsname>.service` units managed by
  `nut-driver-enumerator`.
- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale
  + LAN). Clients authenticate via `upsd.users`.
- **Monitor layer** — `upsmon` runs locally as `master` to trigger
  `SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`.
- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale
  and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors.

### Key deployment lesson: udev must cover raw USB devices

The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files),
**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user,
it needs write access to those raw USB files. The udev rule must match
`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only
`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`.

## Scripts

NUT host scripts run on pfv-tsys1 via `tests/remote.sh`:

```bash
# Idempotent install + configure (safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh

# Discover USB UPS + NUT state (read-only diagnostic):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh

# Query UPS data + service health:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
```

The HA integration script runs from your workstation (needs HA API access):

```bash
# Add the NUT integration to Home Assistant (idempotent):
bash ups/setup-ha-nut.sh
```

`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it
can be repurposed for other hosts or UPS units. Passwords for `monuser` and
`homeassistant` are auto-generated on first run and reused on subsequent runs
(stored in `/etc/nut/upsd.users`).

Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically
unplugged).

## Configuration files on pfv-tsys1

| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) |
| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver |
| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN <tailscale>` + `LISTEN <lan>` on port 3493 |
| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials |
| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` |
| `/etc/nut/nut.conf` | `MODE=netserver` |

## Home Assistant integration

The NUT integration is added automatically by `setup-ha-nut.sh`, which drives
HA's REST config-flow API. It is idempotent (skips if the entry exists).

```bash
# Prerequisites: create token + password files (one-time):
mkdir -p ~/.config/pfvcluster
# HA → Profile → Long-Lived Access Tokens → Create Token:
echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token
# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user):
echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password
chmod 600 ~/.config/pfvcluster/{ha-token,nut-password}

# Run:
bash ups/setup-ha-nut.sh
```

### Why LAN IP, not Tailscale

upsd listens on **both** the Tailscale IP (`100.121.189.98`) **and** the LAN IP
(`192.168.3.11`). The HA NUT integration uses the **LAN IP** because HAOS runs
Tailscale as an isolated add-on container — the HA core container cannot route
to Tailscale IPs. Since pfv-bms (HA, `192.168.3.12`) and pfv-tsys1 (`192.168.3.11`)
share the same vmbr0 bridge, LAN connectivity is instant and reliable.

### Manual UI alternative

In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**:

| Field | Value |
|-------|-------|
| Host | `192.168.3.11` (LAN — HAOS can't reach Tailscale IPs from the HA container) |
| Port | `3493` |
| Username | `homeassistant` |
| Password | *(stored in `/etc/nut/upsd.users` on pfv-tsys1)* |
| UPS | `apc-smartups-c1500` |

### Live sensors

HA exposes UPS data as sensors (prefix `sensor.apc_smartups_c1500_`):
`battery_charge`, `status` (Online/On Battery), `status_data` (OL/OB/DISCHRG).
Additional sensors (load, runtime, voltage) populate as the UPS reports them.

## Daily operations

From pfv-tsys1 (or any tailnet host with NUT client installed):

```bash
# List UPS units served by upsd
upsc -l pfv-tsys1

# Full variable dump for one UPS
upsc apc-smartups-c1500@pfv-tsys1

# Just the live load + runtime
upsc apc-smartups-c1500@pfv-tsys1 ups.load
upsc apc-smartups-c1500@pfv-tsys1 battery.runtime
```

## Notes

- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves
  hypervisor graceful-shutdown capability and matches the `powerman/` pattern
  (PDU managed on the host where the adapter physically lives).
- **Per-unit power (Watts):** `usbhid-ups` reports `ups.realpower` directly on
  the APC Smart-UPS C 1500. If absent, derive Watts as
  `load% × VA_rating × power_factor` in HA templates.