Compare commits
30
Commits
master
...
f4d96b13f0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f4d96b13f0 | ||
|
|
27f6967e1a | ||
|
|
aa8d78825a | ||
|
|
55ff572463 | ||
|
|
efe314f34e | ||
|
|
d18e467484 | ||
|
|
b06ef7dbf0 | ||
|
|
0c44be9925 | ||
|
|
bcbdeb74a2 | ||
|
|
78a9258263 | ||
|
|
c4ebb8aae7 | ||
|
|
7e13612752 | ||
|
|
83b32d5d9f | ||
|
|
807dc8ba8d | ||
|
|
239e449d45 | ||
|
|
601c70ec5e | ||
|
|
13b0976ad6 | ||
|
|
8f48730a40 | ||
|
|
c23d256ced | ||
|
|
45024f80c3 | ||
|
|
403f83a187 | ||
|
|
c8f2cbde4a | ||
|
|
5629bc9b4b | ||
|
|
4f54635b33 | ||
|
|
523a56bd52 | ||
|
|
758fc92e36 | ||
|
|
528dab5100 | ||
|
|
6266043d11 | ||
|
|
8931b653d5 | ||
|
|
09f3276b31 |
@@ -0,0 +1,16 @@
|
||||
## Release PR (dev -> release)
|
||||
|
||||
Merging this deploys to production (pfv-bms restarts if packages/** or
|
||||
configuration.yaml changed).
|
||||
|
||||
### Gate checklist (branch protection enforces the mechanical parts)
|
||||
- [ ] CI green on the head commit (yamllint + secret scan + SDLC suite)
|
||||
- [ ] Adversarial pass done: what breaks if this is wrong? what watches for it?
|
||||
- [ ] post-deploy validation (#778) will run on restart — expected entity changes noted below
|
||||
|
||||
### Reviewer (founder)
|
||||
- [ ] Diff reviewed
|
||||
- [ ] Approve to release
|
||||
|
||||
Expected entity/automation changes: <list or "none">
|
||||
Rollback: git revert <sha> on dev, re-PR, or push previous release sha directly.
|
||||
@@ -0,0 +1,28 @@
|
||||
# pfv-bms CI [#784 #778] — fleet standard per Discourse "CI/CD: Gitea Actions".
|
||||
# Runs on dev (pre-release gate) and release (final gate before live pull).
|
||||
name: ci
|
||||
on:
|
||||
push:
|
||||
branches: [dev, release]
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ultix
|
||||
container:
|
||||
image: node:20-bookworm
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: yamllint
|
||||
run: |
|
||||
apt-get update -qq >/dev/null && apt-get install -qq -y python3-pip >/dev/null && pip install --quiet --break-system-packages yamllint
|
||||
yamllint -c .yamllint .
|
||||
- name: secret scan
|
||||
run: |
|
||||
if grep -rInE "BEGIN (RSA |OPENSSH |EC |DSA )?PRIVATE KEY|aws_secret_access_key *=|AKIA[0-9A-Z]{16}" --exclude-dir=.git .; then
|
||||
echo "::error::secret material committed"; exit 1
|
||||
else
|
||||
echo "secret scan clean"
|
||||
fi
|
||||
- name: sdlc regression tests
|
||||
run: |
|
||||
chmod +x tests/test_sdlc.sh
|
||||
bash tests/test_sdlc.sh
|
||||
@@ -0,0 +1,22 @@
|
||||
# HA-config-aware yamllint [#784]: keep the checks that catch real bugs
|
||||
# (duplicates, structure), relax the stylistic ones HA templates trip.
|
||||
extends: default
|
||||
ignore: |
|
||||
docs/
|
||||
secrets.yaml
|
||||
*.db
|
||||
rules:
|
||||
line-length: disable
|
||||
comments: disable
|
||||
comments-indentation: disable
|
||||
document-start: disable
|
||||
truthy:
|
||||
check-keys: false
|
||||
indentation:
|
||||
spaces: consistent
|
||||
indent-sequences: consistent
|
||||
key-duplicates: enable
|
||||
new-line-at-end-of-file: disable
|
||||
trailing-spaces: disable
|
||||
empty-lines:
|
||||
max-end: 2
|
||||
@@ -134,16 +134,23 @@ apps.
|
||||
|
||||
## 🚀 Deployment pipeline
|
||||
|
||||
**Branch model (2026-09-04):** `dev` = WIP integration — the pipeline
|
||||
IGNORES it. `release` = what the box runs — pushing release IS the deploy.
|
||||
`master` is frozen (pre-branch-model history). A PR process with AI +
|
||||
human reviewers will gate dev → release (founder, upcoming).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
PUSH[📝 git push] --> HOOK[gitea webhook<br/>fast path]
|
||||
PUSH --> SHA[sha-watch<br/>safety net · 300 s poll]
|
||||
PUSH[📝 push to dev] --> NOOP[🚫 pipeline ignores dev]
|
||||
REL[🚀 merge dev → release<br/>and push] --> HOOK[gitea webhook<br/>fast path]
|
||||
REL --> SHA[sha-watch<br/>release sha · 300 s poll]
|
||||
HOOK & SHA --> CHECK{changed files?}
|
||||
CHECK -->|packages/**| RESTART[♻️ core restart<br/>webhook triggers need it]
|
||||
CHECK -->|configuration.yaml,<br/>packages/**| RESTART[♻️ core restart<br/>webhook triggers need it]
|
||||
CHECK -->|other YAML| RELOAD[⚡ reload only<br/>~5 min to live]
|
||||
```
|
||||
|
||||
Gate before pushing `packages/**`: `ha core check` must pass on-box.
|
||||
Gate before pushing `release` (and after any `packages/**` change on
|
||||
`dev`): `ha core check` must pass on-box.
|
||||
Measured end-to-end: 4 m 52 s commit → live.
|
||||
|
||||
## 🗺️ Repo map
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# STATUS — point-in-time agent state (sparse; history lives in Redmine)
|
||||
|
||||
Updated: 2026-09-06 ~07:10 CDT · dev @ aa8d788 · release @ 83b32d5
|
||||
|
||||
| ticket | tier | state | item |
|
||||
|---|---|---|---|
|
||||
| #811 | — | ⛔ | NeedsInput: merge PR #3, PVE tokens (#790), beta onboarding click, rotations |
|
||||
| #344 | prod | 🔄 | release PR #3 open, CI green, review gate passed (FAIL→fixed→dispositioned) |
|
||||
| #790 | dev | ✅ | shutdown chain DRY RUN, level-based re-polls; exec wiring waits on PVE tokens |
|
||||
| #628 | docs | ✅ | restore runbook + baseline backup 34a05d70 off-box on ultix |
|
||||
| #781 | beta | 🔄 | VM 53110 @ 192.168.3.240, HA up, owner created; onboarding form + config sync left |
|
||||
| #813 | — | ⏳ | CI check_config job parked — needs UI job log, recipe in ticket |
|
||||
| #345 | prod | 🔄 | rest_command verified registered; swap-back = founder re-test call |
|
||||
|
||||
## Inbox
|
||||
- 2026-09-06: tsys6 trend false-positive fix — watch trend sensors 24h post-deploy
|
||||
- 2026-09-06: stlpc-artroom dead all morning — policy ask in #811 item 8
|
||||
- 2026-09-06: runner container got external SIGTERM 11:35 UTC, restarted 11:37 — OAM check
|
||||
+266
-45
@@ -4,7 +4,7 @@
|
||||
- trigger: state
|
||||
entity_id:
|
||||
- sensor.tsys1_ups_status
|
||||
- sensor.tsys9_ups_status
|
||||
- sensor.tsys1_triplite_status
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: '{{ ''On Battery'' in states(trigger.entity_id) }}'
|
||||
@@ -16,7 +16,6 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: POWER OUTAGE
|
||||
clickAction: /lovelace-pfv-server-room/ups
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: '{{ state_attr(trigger.entity_id, ''friendly_name'') }} lost utility
|
||||
power and is ON BATTERY at {{ states(''sensor.'' ~ trigger.entity_id.split(''.'')[1]
|
||||
@@ -28,6 +27,7 @@
|
||||
- trigger: numeric_state
|
||||
entity_id:
|
||||
- sensor.tsys1_ups_battery_charge
|
||||
- sensor.tsys1_triplite_battery_charge
|
||||
below: 40
|
||||
conditions:
|
||||
- condition: template
|
||||
@@ -41,7 +41,6 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: UPS BATTERY LOW
|
||||
clickAction: /lovelace-pfv-server-room/ups
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: '{{ trigger.entity_id }} at {{ states(trigger.entity_id) }}% while
|
||||
on battery. Start graceful shutdown of VMs before hypervisors (#455 ordering).'
|
||||
@@ -52,7 +51,7 @@
|
||||
- trigger: state
|
||||
entity_id:
|
||||
- sensor.tsys1_ups_status
|
||||
- sensor.tsys9_ups_status
|
||||
- sensor.tsys1_triplite_status
|
||||
to: unavailable
|
||||
for: 00:05:00
|
||||
actions:
|
||||
@@ -63,22 +62,42 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: UPS COMMS LOST
|
||||
clickAction: /lovelace-pfv-server-room/ups
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: '{{ trigger.entity_id }} unreachable for 5 minutes (nut-server or network
|
||||
path down; tsys1 also hosts pfv-bms itself).'
|
||||
mode: single
|
||||
- id: pfv_plant_ups_runtime_low
|
||||
alias: PFV plant - UPS runtime low while on battery
|
||||
description: On battery and runtime burned below 10 min — the graceful
|
||||
shutdown window is closing. VMs before hypervisors per #455.
|
||||
description: >-
|
||||
On battery and runtime burned below 10 min — the graceful shutdown
|
||||
window is closing. VMs before hypervisors per #455 ordering. Covers
|
||||
BOTH UPSes (APC + Tripp Lite — the Tripp Lite gap closed 2026-09-06);
|
||||
the status check maps each runtime sensor to its own UPS status.
|
||||
triggers:
|
||||
- trigger: numeric_state
|
||||
entity_id: sensor.pfv_ups_runtime_minutes
|
||||
entity_id:
|
||||
- sensor.pfv_ups_runtime_minutes
|
||||
- sensor.pfv_tripp_lite_runtime_minutes
|
||||
below: 10
|
||||
# 5-min level re-check: numeric_state alone fires only on the threshold
|
||||
# CROSSING — a blip at that minute (or an HA restart mid-outage) would
|
||||
# silence the page for the whole discharge. Level-based polling turns
|
||||
# this into an escalation ladder while the outage persists.
|
||||
- trigger: time_pattern
|
||||
minutes: /5
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: "{{ states('sensor.tsys1_ups_status') not in ['Online', 'unknown', 'unavailable'] }}"
|
||||
value_template: >-
|
||||
{% set ns = namespace(hit=false) %}
|
||||
{% for rt, st in [
|
||||
('sensor.pfv_ups_runtime_minutes', 'sensor.tsys1_ups_status'),
|
||||
('sensor.pfv_tripp_lite_runtime_minutes', 'sensor.tsys1_triplite_status')] %}
|
||||
{% if is_number(states(rt)) and (states(rt) | float) < 10
|
||||
and states(st) not in ['Online', 'OL', 'unknown', 'unavailable'] %}
|
||||
{% set ns.hit = true %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.hit }}
|
||||
actions:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
@@ -87,9 +106,8 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: UPS RUNTIME LOW
|
||||
clickAction: /lovelace-pfv-server-room/ups
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: On battery {{ states('sensor.pfv_ups_runtime_minutes') }} min left — start graceful shutdown now (#455 ordering).
|
||||
message: '{{ state_attr(trigger.entity_id, ''friendly_name'') }}: {{ states(trigger.entity_id) }} min left — start graceful shutdown now (#455 ordering).'
|
||||
mode: single
|
||||
- id: pfv_plant_ups_power_restored
|
||||
alias: PFV plant - utility power restored
|
||||
@@ -97,7 +115,7 @@
|
||||
- trigger: state
|
||||
entity_id:
|
||||
- sensor.tsys1_ups_status
|
||||
- sensor.tsys9_ups_status
|
||||
- sensor.tsys1_triplite_status
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: "{{ trigger.from_state is not none and 'On Battery' in trigger.from_state.state\n
|
||||
@@ -110,7 +128,6 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: Power restored
|
||||
clickAction: /lovelace-pfv-server-room/ups
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: '{{ state_attr(trigger.entity_id, ''friendly_name'') }} is back ONLINE.'
|
||||
mode: single
|
||||
@@ -188,39 +205,61 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: HIGH TEMPERATURE
|
||||
clickAction: /lovelace-pfv-server-room/temps
|
||||
url: /lovelace-pfv-server-room/temps
|
||||
message: '{{ trigger.entity_id }} is {{ states(trigger.entity_id) }}°F (class
|
||||
threshold exceeded). Check cooling/airflow.'
|
||||
mode: single
|
||||
- id: pfv_plant_sensor_watchdog
|
||||
alias: PFV plant - sensor watchdog (unavailable sensors)
|
||||
description: HA is the environmental brain (#344) — a silent sensor is an outage.
|
||||
Re-alert suppressed by 4h cooldown via last_notified input_text if present.
|
||||
description: >
|
||||
HA is the environmental brain (#344) — a silent sensor is an outage.
|
||||
Fleet-wide sweep (every pfv_* sensor, broader than the 5-min critical
|
||||
watchdog). PUSHES ONLY when sensors are actually dead, and re-pages a
|
||||
still-dead fleet at most every 4h — the unconditional 30-min page was
|
||||
retired 2026-09-06 after live-fire alert fatigue. All-OK cycles log
|
||||
quietly; the Kuma dead-man heartbeat (#792) owns liveness signaling.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: time_pattern
|
||||
minutes: /30
|
||||
actions:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: SENSOR WATCHDOG
|
||||
clickAction: /lovelace-pfv-server-room/overview
|
||||
url: /lovelace-pfv-server-room/overview
|
||||
message: >-
|
||||
{% set fleet = states.sensor | selectattr('entity_id', 'match', 'sensor\.pfv_')
|
||||
| list %}
|
||||
{% set dead = fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list %}
|
||||
{% if dead | length > 0 %}
|
||||
- variables:
|
||||
fleet: "{{ states.sensor | selectattr('entity_id', 'match', 'sensor\\.pfv_') | list }}"
|
||||
dead: "{{ fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list }}"
|
||||
# 4h cooldown: page again only if the last page is older than 4h
|
||||
# (or we never paged). Fresh HA start (no restore) counts as never.
|
||||
# as_timestamp math — aware/naive datetime subtraction throws in Jinja.
|
||||
notify_gate: >-
|
||||
{{ states('input_datetime.pfv_plant_watchdog_last_notify') in ['unknown', 'unavailable', 'none']
|
||||
or as_timestamp(now()) - as_timestamp(states('input_datetime.pfv_plant_watchdog_last_notify')) > 14400 }}
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ dead | length > 0 and notify_gate }}"
|
||||
then:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: SENSOR WATCHDOG
|
||||
url: /lovelace-pfv-server-room/overview
|
||||
message: >-
|
||||
{{ dead | length }} of {{ fleet | length }} fleet sensors silent:
|
||||
{{ dead | map(attribute='entity_id') | join(', ') }}
|
||||
{% else %}
|
||||
all {{ fleet | length }} fleet sensors OK
|
||||
{% endif %}
|
||||
mode: single
|
||||
- action: input_datetime.set_datetime
|
||||
target:
|
||||
entity_id: input_datetime.pfv_plant_watchdog_last_notify
|
||||
data:
|
||||
datetime: "{{ now().strftime('%Y-%m-%d %H:%M:%S') }}"
|
||||
else:
|
||||
- action: logbook.log
|
||||
data:
|
||||
name: Plant watchdog
|
||||
message: >-
|
||||
Sweep: all {{ fleet | length }} fleet sensors answering
|
||||
({{ dead | length }} silent, cooldown {{ 'active' if not notify_gate else 'n/a' }}).
|
||||
- id: pfv_plant_temp_rising_fast
|
||||
alias: PFV plant - temperature rising fast (rate-of-change)
|
||||
description: Catches dead AC minutes before the 104F threshold trips.
|
||||
@@ -242,11 +281,10 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: TEMP RISING FAST
|
||||
clickAction: /lovelace-pfv-server-room/temps
|
||||
url: /lovelace-pfv-server-room/temps
|
||||
message: '{{ trigger.entity_id }} climbing ~10F/15min — check cooling
|
||||
before the 104F threshold. (rate-of-change early warning)'
|
||||
# Deploy pipeline [#734]: gitea repo webhook (master pushes) -> forced pull
|
||||
# Deploy pipeline [#734]: gitea repo webhook (release pushes) -> forced pull
|
||||
# -> minimal reload. Core restarts ONLY when configuration.yaml or a
|
||||
# packages platform-sensor file changed; everything else reloads with no
|
||||
# bounce. The git_pull add-on's auto_restart is disabled - this webhook
|
||||
@@ -254,7 +292,7 @@
|
||||
- id: pfv_deploy_git_webhook
|
||||
alias: PFV deploy - git push to pull + minimal reload
|
||||
description: >-
|
||||
Gitea webhook on master pushes. Forces a git pull now, then
|
||||
Gitea webhook on release pushes. Forces a git pull now, then
|
||||
homeassistant.reload_all unless files that require a core restart
|
||||
changed. Replaces the git_pull add-on timer auto-restart [#734].
|
||||
mode: queued
|
||||
@@ -269,9 +307,9 @@
|
||||
actions:
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ trigger.json.ref | default('') != 'refs/heads/master' }}"
|
||||
value_template: "{{ trigger.json.ref | default('') != 'refs/heads/release' }}"
|
||||
then:
|
||||
- stop: "deploy webhook: non-master ref {{ trigger.json.ref | default('') }} ignored"
|
||||
- stop: "deploy webhook: non-release ref {{ trigger.json.ref | default('') }} ignored"
|
||||
- variables:
|
||||
changed_files: >-
|
||||
{{ (trigger.json.commits | default([]) | map(attribute='added') | flatten | list)
|
||||
@@ -337,7 +375,7 @@
|
||||
data:
|
||||
title: PFV deploy complete
|
||||
message: "Pulled + reloaded YAML at {{ now().strftime('%H:%M:%S') }} - no core restart. Files: {{ changed_files | join(', ') | truncate(120) }}"
|
||||
# Deploy pipeline safety net [#734]: polls gitea master HEAD (rest sensor
|
||||
# Deploy pipeline safety net [#734]: polls gitea release HEAD (rest sensor
|
||||
# in packages/deploy_pipeline.yaml) and applies pulls reload-only. The
|
||||
# gitea webhook is the fast path once gitea's SSRF allowlist permits
|
||||
# pfv-bms.knel.net; until then this net catches every push within 5 min
|
||||
@@ -346,7 +384,7 @@
|
||||
- id: pfv_deploy_sha_watch
|
||||
alias: PFV deploy - gitea sha watch safety net
|
||||
description: >-
|
||||
Master HEAD sha changed -> pull + CORE RESTART. Interim deploy
|
||||
Release HEAD sha changed -> pull + CORE RESTART. Interim deploy
|
||||
trigger while gitea webhook delivery is blocked by its SSRF host
|
||||
allowlist [#734]. Restart (not reload) is REQUIRED: automation
|
||||
reload loses webhook-trigger registration - found live 2026-09-03
|
||||
@@ -354,12 +392,12 @@
|
||||
scan ingest, this pipeline's own fast path) silently stop firing
|
||||
while the endpoint still returns HTTP 200. Core restart
|
||||
re-registers them. Accepted trade-off: ~1 min HA outage per
|
||||
master deploy.
|
||||
release deploy.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: state
|
||||
entity_id: sensor.pfv_deploy_gitea_master_head
|
||||
entity_id: sensor.pfv_deploy_gitea_release_head
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: >-
|
||||
@@ -383,7 +421,7 @@
|
||||
data:
|
||||
title: PFV deploy (sha watch)
|
||||
message: >-
|
||||
New master HEAD {{ states('sensor.pfv_deploy_gitea_master_head')[:7] }}
|
||||
New release HEAD {{ states('sensor.pfv_deploy_gitea_release_head')[:7] }}
|
||||
pulled. Restarting HA core (webhook re-registration) - back in
|
||||
~1 min.
|
||||
- action: homeassistant.restart
|
||||
@@ -518,10 +556,193 @@
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: TEMPERATURE RECOVERED
|
||||
clickAction: /lovelace-pfv-server-room/temps
|
||||
url: /lovelace-pfv-server-room/temps
|
||||
message: >-
|
||||
{{ state_attr(trigger.entity_id, 'friendly_name') | default(trigger.entity_id) }}
|
||||
back to {{ states(trigger.entity_id) }}°F and holding — was
|
||||
{{ trigger.from_state.state }}°F when the alert cleared (tier fire line {{ fire }}°F).
|
||||
mode: single
|
||||
- id: pfv_watchdog_stale_entities
|
||||
alias: PFV watchdog - stale entity auto-heal
|
||||
description: >
|
||||
Detects silent poller wedges (2026-09-04: snmp tsys9 went 12h
|
||||
undetected; NUT APC 3h) and auto-heals config-entry platforms via
|
||||
homeassistant.reload_config_entry — no restart needed. Escalates to
|
||||
notify when reload does not recover. Auto core restart ONLY while
|
||||
input_boolean.pfv_watchdog_auto_restart is ON and entities stay
|
||||
stale — restart costs ~1 min downtime and re-registers webhooks
|
||||
(#345, #789). v1 notify repeats each cycle while stale (visible
|
||||
alarm); throttle in v2.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: time_pattern
|
||||
minutes: /5
|
||||
variables:
|
||||
checks:
|
||||
- [sensor.pfv_tsys1_rack_temperature, 20]
|
||||
- [sensor.pfv_tsys1_drive_temperature, 30]
|
||||
- [sensor.pfv_tsys3_cpu_temperature, 20]
|
||||
- [sensor.pfv_tsys4_ambient_temperature, 20]
|
||||
- [sensor.pfv_tsys6_cpu_temperature, 20]
|
||||
- [sensor.pfv_tsys6_drac_ambient, 30]
|
||||
- [sensor.pfv_tsys7_cpu_temperature, 20]
|
||||
- [sensor.pfv_tsys7_drac_inlet, 30]
|
||||
- [sensor.pfv_tsys9_cpu_temperature, 20]
|
||||
- [sensor.pfv_tsys9_drive_temperature, 30]
|
||||
- [sensor.tsys1_ups_status, 15]
|
||||
- [sensor.tsys1_triplite_status, 15]
|
||||
- [sensor.pfv_deploy_gitea_release_head, 30]
|
||||
actions:
|
||||
- variables:
|
||||
stale_list: >-
|
||||
{% set ns = namespace(stale=[]) %}
|
||||
{% for pair in checks %}
|
||||
{% set s = states[pair[0]] %}
|
||||
{% if s is not none and s.last_updated is defined
|
||||
and (now() - s.last_updated).total_seconds() > pair[1] * 60 %}
|
||||
{% set ns.stale = ns.stale + [pair[0]] %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.stale }}
|
||||
- condition: template
|
||||
value_template: "{{ stale_list | length > 0 }}"
|
||||
- repeat:
|
||||
for_each: "{{ stale_list }}"
|
||||
sequence:
|
||||
- action: homeassistant.reload_config_entry
|
||||
data:
|
||||
entity_id: "{{ repeat.item }}"
|
||||
- delay:
|
||||
hours: 0
|
||||
minutes: 2
|
||||
seconds: 0
|
||||
- variables:
|
||||
still_stale: >-
|
||||
{% set ns = namespace(stale=[]) %}
|
||||
{% for pair in checks %}
|
||||
{% set s = states[pair[0]] %}
|
||||
{% if s is not none and s.last_updated is defined
|
||||
and (now() - s.last_updated).total_seconds() > pair[1] * 60 %}
|
||||
{% set ns.stale = ns.stale + [pair[0]] %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.stale }}
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: WATCHDOG
|
||||
url: /lovelace-pfv-server-room/temps
|
||||
message: >-
|
||||
Stale after auto-reload: {{ still_stale | join(', ') }}.
|
||||
Auto-restart is {{ 'ON' if is_state('input_boolean.pfv_watchdog_auto_restart','on')
|
||||
else 'OFF (arm it on the watchdog switch if you want unattended restarts)' }}.
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ is_state('input_boolean.pfv_watchdog_auto_restart','on') }}"
|
||||
then:
|
||||
- delay:
|
||||
hours: 0
|
||||
minutes: 30
|
||||
seconds: 0
|
||||
- variables:
|
||||
hard_stale: >-
|
||||
{% set ns = namespace(stale=[]) %}
|
||||
{% for pair in checks %}
|
||||
{% set s = states[pair[0]] %}
|
||||
{% if s is not none and s.last_updated is defined
|
||||
and (now() - s.last_updated).total_seconds() > pair[1] * 60 %}
|
||||
{% set ns.stale = ns.stale + [pair[0]] %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.stale }}
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ hard_stale | length > 0 }}"
|
||||
then:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: WATCHDOG — restarting core
|
||||
message: "Still stale after reload + 30 min: {{ hard_stale | join(', ') }}. Restarting HA core now."
|
||||
- action: homeassistant.restart
|
||||
- id: pfv_post_deploy_validation
|
||||
alias: PFV post-deploy validation
|
||||
description: >
|
||||
Runs after EVERY HA start (deploy restarts included): after a settle
|
||||
delay, verifies the critical sensor set actually materialized and is
|
||||
fresh. Catches bad deploys — missing templates, broken packages,
|
||||
wedged pollers — and pages with specifics instead of waiting for a
|
||||
human to notice (#778, #777). The watchdog then handles recovery.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: event
|
||||
event_type: homeassistant_start
|
||||
actions:
|
||||
- delay:
|
||||
hours: 0
|
||||
minutes: 3
|
||||
seconds: 0
|
||||
- variables:
|
||||
required:
|
||||
- [sensor.pfv_tsys1_rack_temperature, 20]
|
||||
- [sensor.pfv_tsys9_cpu_temperature, 20]
|
||||
- [sensor.tsys1_ups_status, 15]
|
||||
- [sensor.tsys1_triplite_status, 15]
|
||||
- [sensor.pfv_ups_runtime_minutes, 30]
|
||||
- [sensor.pfv_tripp_lite_runtime_minutes, 30]
|
||||
- [sensor.pfv_deploy_gitea_release_head, 30]
|
||||
- [automation.pfv_watchdog_stale_entities, 0]
|
||||
- variables:
|
||||
bad_list: >-
|
||||
{% set ns = namespace(bad=[]) %}
|
||||
{% for pair in required %}
|
||||
{% set s = states[pair[0]] %}
|
||||
{% if s is none or s.state in ['unknown', 'unavailable', 'none'] %}
|
||||
{% set ns.bad = ns.bad + [pair[0] ~ ' (missing)'] %}
|
||||
{% elif pair[1] > 0 and s.last_updated is defined
|
||||
and (now() - s.last_updated).total_seconds() > pair[1] * 60 %}
|
||||
{% set ns.bad = ns.bad + [pair[0] ~ ' (stale)'] %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.bad }}
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ bad_list | length > 0 }}"
|
||||
then:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: POST-DEPLOY VALIDATION FAILED
|
||||
url: /lovelace-pfv-thermal
|
||||
message: >-
|
||||
After restart, these criticals are missing/stale:
|
||||
{{ bad_list | join(', ') }}. Watchdog will retry recovery;
|
||||
if this repeats, revert the last release (git revert + push).
|
||||
- action: persistent_notification.create
|
||||
data:
|
||||
title: POST-DEPLOY VALIDATION FAILED
|
||||
message: "Missing/stale: {{ bad_list | join(', ') }}"
|
||||
- id: pfv_kuma_heartbeat
|
||||
alias: PFV Kuma heartbeat (dead-man)
|
||||
description: >
|
||||
Pushes OK to the out-of-band Uptime Kuma every 5 minutes (#792).
|
||||
Kuma pages the founder directly if heartbeats stop — the alert path
|
||||
that works when HA itself is the failure.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: time_pattern
|
||||
minutes: /5
|
||||
actions:
|
||||
- action: shell_command.kuma_ha_heartbeat
|
||||
|
||||
@@ -20,6 +20,21 @@ prometheus:
|
||||
lovelace:
|
||||
mode: storage
|
||||
dashboards:
|
||||
pfv-overview:
|
||||
mode: yaml
|
||||
title: Ops Overview
|
||||
filename: dashboards/overview.yaml
|
||||
icon: mdi:view-dashboard-outline
|
||||
pfv-power:
|
||||
mode: yaml
|
||||
title: Power
|
||||
filename: dashboards/power.yaml
|
||||
icon: mdi:flash
|
||||
pfv-thermal:
|
||||
mode: yaml
|
||||
title: Thermal
|
||||
filename: dashboards/thermal.yaml
|
||||
icon: mdi:thermometer-lines
|
||||
pfv-server-room:
|
||||
mode: yaml
|
||||
title: Server Room
|
||||
@@ -34,3 +49,19 @@ lovelace:
|
||||
title: STLP
|
||||
filename: dashboards/stlp.yaml
|
||||
icon: mdi:home
|
||||
|
||||
# Watchdog arm switch (#789): auto core restart on unresolved staleness.
|
||||
# Default OFF — reload_config_entry heals config-entry wedges without it.
|
||||
input_boolean:
|
||||
pfv_watchdog_auto_restart:
|
||||
name: Watchdog auto core restart
|
||||
icon: mdi:restart-alert
|
||||
|
||||
# 30-min plant watchdog cooldown (#344, 2026-09-06): re-pages a dead sensor
|
||||
# at most every 4h instead of every cycle. Set by the automation on notify.
|
||||
input_datetime:
|
||||
pfv_plant_watchdog_last_notify:
|
||||
name: Plant watchdog last notify
|
||||
has_date: true
|
||||
has_time: true
|
||||
icon: mdi:bell-cancel
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
title: Ops Overview
|
||||
views:
|
||||
- title: Overview
|
||||
type: sections
|
||||
sections:
|
||||
- type: grid
|
||||
cards:
|
||||
- type: markdown
|
||||
content: >
|
||||
## Ops Overview
|
||||
|
||||
One screen: is anything on fire? Details live in
|
||||
[Power](/lovelace-pfv-power), [Thermal](/lovelace-pfv-thermal),
|
||||
[Server Room](/lovelace-pfv-server-room) and
|
||||
[STLP](/lovelace-pfv-stlp).
|
||||
- type: grid
|
||||
cards:
|
||||
- type: heading
|
||||
heading: Room & cooling
|
||||
- type: gauge
|
||||
entity: sensor.pfv_tsys1_rack_temperature
|
||||
name: Server room
|
||||
min: 50
|
||||
max: 110
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 85
|
||||
red: 95
|
||||
- type: entities
|
||||
entities:
|
||||
- sensor.pfv_garage_temperature
|
||||
- sensor.pfv_artroom_temperature
|
||||
- type: grid
|
||||
cards:
|
||||
- type: heading
|
||||
heading: Power
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_ups_battery_charge
|
||||
name: APC charge
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 40
|
||||
red: 20
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_triplite_battery_charge
|
||||
name: Tripp Lite charge
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 40
|
||||
red: 20
|
||||
- type: entities
|
||||
entities:
|
||||
- sensor.tsys1_ups_status
|
||||
- sensor.tsys1_triplite_status
|
||||
- sensor.pfv_ups_runtime_minutes
|
||||
- sensor.pfv_tripp_lite_runtime_minutes
|
||||
- type: grid
|
||||
cards:
|
||||
- type: heading
|
||||
heading: Alerts & self-healing
|
||||
- type: entities
|
||||
entities:
|
||||
- automation.pfv_plant_ups_on_battery
|
||||
- automation.pfv_plant_ups_battery_low
|
||||
- automation.pfv_plant_ups_communication_lost
|
||||
- automation.pfv_plant_high_temperature
|
||||
- automation.pfv_watchdog_stale_entity_auto_heal
|
||||
- input_boolean.pfv_watchdog_auto_restart
|
||||
- type: grid
|
||||
cards:
|
||||
- type: markdown
|
||||
content: >
|
||||
**Pipeline:** deploy sensor tracks `release` on KNEL/KNELBMS.
|
||||
CI runs on every dev/release push (Actions tab in gitea).
|
||||
Out-of-band paging: Uptime Kuma (Cloudron).
|
||||
@@ -0,0 +1,114 @@
|
||||
title: Power
|
||||
views:
|
||||
- title: UPS detail
|
||||
type: sections
|
||||
sections:
|
||||
- type: grid
|
||||
cards:
|
||||
- type: heading
|
||||
heading: APC Smart-UPS C 1500 (battery-side USB HID)
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_ups_battery_charge
|
||||
name: Charge
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 40
|
||||
red: 20
|
||||
- type: gauge
|
||||
entity: sensor.pfv_ups_runtime_minutes
|
||||
name: Runtime (min)
|
||||
min: 0
|
||||
max: 60
|
||||
severity:
|
||||
green: 25
|
||||
yellow: 15
|
||||
red: 5
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: APC status
|
||||
entities:
|
||||
- sensor.tsys1_ups_status
|
||||
- sensor.tsys1_ups_status_data
|
||||
- sensor.tsys1_ups_battery_charge
|
||||
- sensor.pfv_ups_runtime_minutes
|
||||
- sensor.tsys1_ups_battery_runtime
|
||||
- sensor.tsys1_ups_battery_voltage
|
||||
- sensor.tsys1_ups_nominal_battery_voltage
|
||||
- sensor.tsys1_ups_low_battery_setpoint
|
||||
- sensor.tsys1_ups_beeper_status
|
||||
- type: markdown
|
||||
content: >
|
||||
Serial AS1213210423 · mfr 2012-03 · fw UPS 02.2 · 24 V
|
||||
nominal. USB HID exposes battery-side metrics only — no
|
||||
input/output voltage or watts on this unit.
|
||||
- type: grid
|
||||
cards:
|
||||
- type: heading
|
||||
heading: Tripp Lite 1500 VA (moved from tsys9, #372)
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_triplite_battery_charge
|
||||
name: Charge
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 40
|
||||
red: 20
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_triplite_load
|
||||
name: Load %
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 70
|
||||
yellow: 85
|
||||
red: 95
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: Tripp Lite status
|
||||
entities:
|
||||
- sensor.tsys1_triplite_status
|
||||
- sensor.tsys1_triplite_status_data
|
||||
- sensor.tsys1_triplite_battery_charge
|
||||
- sensor.tsys1_triplite_apparent_power
|
||||
- sensor.tsys1_triplite_output_current
|
||||
- sensor.tsys1_triplite_load
|
||||
- sensor.pfv_tripp_lite_runtime_minutes
|
||||
- sensor.tsys1_triplite_battery_runtime
|
||||
- sensor.tsys1_triplite_input_voltage
|
||||
- sensor.tsys1_triplite_output_voltage
|
||||
- sensor.tsys1_triplite_nominal_power
|
||||
- title: Trends
|
||||
type: sections
|
||||
sections:
|
||||
- type: grid
|
||||
cards:
|
||||
- type: history-graph
|
||||
title: UPS charge (7 days)
|
||||
entities:
|
||||
- sensor.tsys1_ups_battery_charge
|
||||
- sensor.tsys1_triplite_battery_charge
|
||||
hours_to_show: 168
|
||||
- type: grid
|
||||
cards:
|
||||
- type: history-graph
|
||||
title: Tripp Lite apparent power (24h)
|
||||
entities:
|
||||
- sensor.tsys1_triplite_apparent_power
|
||||
hours_to_show: 24
|
||||
- type: history-graph
|
||||
title: UPS load % (24h)
|
||||
entities:
|
||||
- sensor.tsys1_triplite_load
|
||||
hours_to_show: 24
|
||||
- type: grid
|
||||
cards:
|
||||
- type: markdown
|
||||
content: >
|
||||
**Planned sections** (land with their data sources):
|
||||
Emporia CT rental-bank feed (#623), whole-house kWh vs
|
||||
solar production (#760), garage PDU circuits (#733).
|
||||
@@ -103,6 +103,46 @@ views:
|
||||
fw UPS 02.2 · PbAc, 24 V nominal · usbhid-ups via pfv-tsys1
|
||||
(NUT :3493). This unit reports no input/output voltage over
|
||||
USB HID — battery-side metrics only.
|
||||
- type: grid
|
||||
cards:
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_triplite_battery_charge
|
||||
name: Tripp Lite charge
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 40
|
||||
red: 20
|
||||
- type: gauge
|
||||
entity: sensor.tsys1_triplite_load
|
||||
name: Tripp Lite load %
|
||||
min: 0
|
||||
max: 100
|
||||
severity:
|
||||
green: 70
|
||||
yellow: 85
|
||||
red: 95
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: Tripp Lite status
|
||||
entities:
|
||||
- sensor.tsys1_triplite_status
|
||||
- sensor.tsys1_triplite_status_data
|
||||
- sensor.tsys1_triplite_battery_charge
|
||||
- sensor.tsys1_triplite_apparent_power
|
||||
- sensor.tsys1_triplite_output_current
|
||||
- sensor.tsys1_triplite_load
|
||||
- sensor.tsys1_triplite_input_voltage
|
||||
- sensor.tsys1_triplite_output_voltage
|
||||
- sensor.tsys1_triplite_battery_runtime
|
||||
- type: markdown
|
||||
content: >
|
||||
**Tripp Lite UPS** · usbhid-ups via pfv-tsys1 (NUT :3493);
|
||||
moved from tsys9 2026-09-04 (#372). Carries ~46% load
|
||||
(~414 VA apparent / 1500 VA nominal) — mind runtime
|
||||
before adding gear.
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
title: Thermal
|
||||
views:
|
||||
- title: Rooms
|
||||
type: sections
|
||||
sections:
|
||||
- type: grid
|
||||
cards:
|
||||
- type: gauge
|
||||
entity: sensor.pfv_tsys1_rack_temperature
|
||||
name: Server room
|
||||
min: 50
|
||||
max: 110
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 85
|
||||
red: 95
|
||||
- type: history-graph
|
||||
title: Server room (24h)
|
||||
entities:
|
||||
- sensor.pfv_tsys1_rack_temperature
|
||||
hours_to_show: 24
|
||||
- type: grid
|
||||
cards:
|
||||
- type: gauge
|
||||
entity: sensor.pfv_garage_temperature
|
||||
name: Garage
|
||||
min: 30
|
||||
max: 110
|
||||
severity:
|
||||
green: 40
|
||||
yellow: 90
|
||||
red: 100
|
||||
- type: history-graph
|
||||
title: Garage (24h)
|
||||
entities:
|
||||
- sensor.pfv_garage_temperature
|
||||
hours_to_show: 24
|
||||
- type: grid
|
||||
cards:
|
||||
- type: gauge
|
||||
entity: sensor.pfv_artroom_temperature
|
||||
name: Art room
|
||||
min: 40
|
||||
max: 110
|
||||
severity:
|
||||
green: 60
|
||||
yellow: 85
|
||||
red: 95
|
||||
- type: history-graph
|
||||
title: Art room (24h)
|
||||
entities:
|
||||
- sensor.pfv_artroom_temperature
|
||||
hours_to_show: 24
|
||||
- type: grid
|
||||
cards:
|
||||
- type: markdown
|
||||
content: >
|
||||
**Minisplit** — climate entity lands with the Carrier dongle
|
||||
commissioning (#629). Manual setpoint tonight: 62°F, with the
|
||||
oscillating fan improving distribution. Alert tiers:
|
||||
ambient 104°F · drives 140°F · NVMe 150°F · PCH 160°F ·
|
||||
GPU 185°F.
|
||||
- title: Racks & silicon
|
||||
type: sections
|
||||
sections:
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: R3 · PowerEdge (DRAC)
|
||||
entities:
|
||||
- sensor.pfv_tsys6_cpu_temperature
|
||||
- sensor.pfv_tsys6_drac_ambient
|
||||
- sensor.pfv_tsys7_cpu_temperature
|
||||
- sensor.pfv_tsys7_drac_inlet
|
||||
- sensor.pfv_tsys7_drac_exhaust
|
||||
- sensor.pfv_tsys7_drac_cpu1
|
||||
- sensor.pfv_tsys7_drac_cpu2
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: R3 · Pis + Jetson
|
||||
entities:
|
||||
- sensor.pfv_svrpi_temperature
|
||||
- sensor.pfv_subopi3_temperature
|
||||
- sensor.pfv_subopi_dev_3_temperature
|
||||
- sensor.pfv_subopi_dev_4_temperature
|
||||
- sensor.pfv_jetson_temperature
|
||||
- sensor.pfv_jetson_gpu_temperature
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: R5 · Laptops
|
||||
entities:
|
||||
- sensor.pfv_tsys3_cpu_temperature
|
||||
- sensor.pfv_tsys3_gpu_temperature
|
||||
- sensor.pfv_tsys3_nvme_temperature
|
||||
- sensor.pfv_tsys5_nvme_temperature
|
||||
- type: grid
|
||||
cards:
|
||||
- type: entities
|
||||
title: R5 · Desktops / OptiPlex-class
|
||||
entities:
|
||||
- sensor.pfv_tsys1_rack_temperature
|
||||
- sensor.pfv_tsys4_ambient_temperature
|
||||
- sensor.pfv_tsys1_cpu_temperature
|
||||
- sensor.pfv_tsys4_cpu_temperature
|
||||
- sensor.pfv_tsys9_cpu_temperature
|
||||
- sensor.pfv_tsys9_pch_temperature
|
||||
- type: grid
|
||||
cards:
|
||||
- type: history-graph
|
||||
title: CPU temperatures (24h)
|
||||
entities:
|
||||
- sensor.pfv_tsys1_cpu_temperature
|
||||
- sensor.pfv_tsys3_cpu_temperature
|
||||
- sensor.pfv_tsys6_cpu_temperature
|
||||
- sensor.pfv_tsys7_cpu_temperature
|
||||
- sensor.pfv_tsys9_cpu_temperature
|
||||
hours_to_show: 24
|
||||
@@ -0,0 +1,78 @@
|
||||
# pfv-bms restore runbook (#628)
|
||||
|
||||
Three restore layers exist. Work **inside-out**: usually only layer 1 is
|
||||
needed. Restore drills run on the **dev lane only** — never on hosts
|
||||
running prod VMs (standing rule after the 2026-09-03 PBS drill that
|
||||
I/O-starved prod HA, #684 / t/298).
|
||||
|
||||
## Layer 0 — what is NOT in git
|
||||
|
||||
| Artifact | Where it lives | Covered by |
|
||||
|----------|----------------|------------|
|
||||
| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups |
|
||||
| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups |
|
||||
| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups |
|
||||
|
||||
Git (this repo, `release` branch) is the source of truth for
|
||||
configuration **YAML only**. A fresh box restored from git still needs a
|
||||
`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes
|
||||
+ badge/webhook ids) before HA will start.
|
||||
|
||||
## Layer 1 — HA-level backup (config + settings, small, fast)
|
||||
|
||||
Taken on-box via the `ha` CLI; produces a self-contained `.tar`:
|
||||
|
||||
```bash
|
||||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
|
||||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups list' # find slug
|
||||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>' # verify size/state
|
||||
# OFF-BOX COPY (the on-box copy dies with the VM disk):
|
||||
scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/
|
||||
```
|
||||
|
||||
HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup"
|
||||
entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box
|
||||
copy remains the manual step.
|
||||
|
||||
Restore (same or fresh HAOS box):
|
||||
|
||||
```bash
|
||||
scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
|
||||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'
|
||||
```
|
||||
|
||||
Baseline on record: slug `34a05d70`, name
|
||||
`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after
|
||||
the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy:
|
||||
`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming
|
||||
(sha256 6ff2fa6bd5f3dd2f2a47…).
|
||||
|
||||
## Layer 2 — VM-level backup (prod PBS, nightly)
|
||||
|
||||
- All prod VMs — **including VM 100 (pfv-bms) on pfv-tsys1** — back up
|
||||
nightly to prod PBS, staggered 21:00–04:30 Central, throttled
|
||||
60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
|
||||
- Restore: PVE web UI or `qmrestore` — **target a scratch VM on the dev
|
||||
lane** (verify the restore boots there), never restore in place over
|
||||
the live VM without a Kuma maintenance window.
|
||||
- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on
|
||||
tsys5 are unprotected; fine for beta, not for anything promoted.
|
||||
|
||||
## Layer 3 — config-from-git (last resort, fresh HAOS)
|
||||
|
||||
1. Flash/install HAOS, complete onboarding, join Tailscale.
|
||||
2. `git clone git.knownelement.com:KNEL/KNELBMS` → copy tree to `/config`
|
||||
at the `release` HEAD the fleet expects.
|
||||
3. Re-provision `secrets.yaml` (Layer 0 table) — from keeper records.
|
||||
4. `ha core check` before starting; then watch the first boot: the
|
||||
post-deploy validation automation pages if criticals fail to
|
||||
materialize (#778).
|
||||
5. Verify the trail: deploy sensor matches `release` HEAD, Kuma
|
||||
heartbeat green, doorman roster sensors populated.
|
||||
|
||||
## Related
|
||||
|
||||
- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check`
|
||||
on-box before any release push.
|
||||
- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298
|
||||
(Sep 3 wedge), t/320 (reload vs restart deploy semantics).
|
||||
@@ -1,4 +1,4 @@
|
||||
# Deploy pipeline [#734]: gitea master-HEAD sha watcher. Feeds the
|
||||
# Deploy pipeline [#734]: gitea release-branch sha watcher. Feeds the
|
||||
# sha-watch deploy automation (reload-only apply). The gitea webhook
|
||||
# (gitea hook #10 -> pfv_deploy_git_webhook automation) is the fast path
|
||||
# with full restart mapping, but gitea cannot deliver webhooks to
|
||||
@@ -6,16 +6,23 @@
|
||||
# (founder action via Cloudron file manager); this watcher is the
|
||||
# always-works safety net.
|
||||
rest:
|
||||
- resource: https://git.knownelement.com/api/v1/repos/KNEL/pfv-bms/commits?limit=1
|
||||
- resource: https://git.knownelement.com/api/v1/repos/KNEL/KNELBMS/commits?sha=release&limit=1
|
||||
headers:
|
||||
Authorization: !secret gitea_auth_header
|
||||
User-Agent: pfv-bms-deploy-watch
|
||||
scan_interval: 300
|
||||
verify_ssl: true
|
||||
sensor:
|
||||
- name: PFV deploy gitea master head
|
||||
unique_id: pfv_deploy_gitea_master_head
|
||||
- name: PFV deploy gitea release head
|
||||
unique_id: pfv_deploy_gitea_release_head
|
||||
value_template: "{{ value_json[0].sha if value_json is list and value_json | length > 0 else (value_json | string | truncate(90)) }}"
|
||||
|
||||
# E2E self-deploy proof commit [#734] - if you can read this in /config,
|
||||
# the sha-watch deployed it autonomously.
|
||||
|
||||
# Dead-man heartbeat (#792): pushes OK to the out-of-band Uptime Kuma
|
||||
# (Cloudron, status.knownelement.com) every 5 minutes. If HA stops
|
||||
# pushing — crash, wedge, power loss, network partition — Kuma pages
|
||||
# the founder's phone directly. No HA in the alert loop.
|
||||
shell_command:
|
||||
kuma_ha_heartbeat: 'curl -sS -m 15 "{{ kuma_push_url }}"'
|
||||
|
||||
+14
-1
@@ -7,6 +7,7 @@
|
||||
# PR gate). The roster is fetched live from the gitea API — NO git
|
||||
# mechanics, NO submodule (git_pull add-on lacks submodule support),
|
||||
# NO local copy: access-roster remains the single source of truth.
|
||||
# Reader/door contract (webhook JSON + unlock endpoint): KNEL/doorman README
|
||||
#
|
||||
# UNLOCK PATH (founder-approved 2026-09-03): valid badge + the
|
||||
# doorman_unlock_enabled input_boolean ON -> rest_command fires the
|
||||
@@ -81,6 +82,18 @@ input_boolean:
|
||||
name: Doorman unlock armed
|
||||
icon: mdi:lock-open-variant
|
||||
|
||||
shell_command:
|
||||
# Unlock dispatch via curl — swapped from rest_command 2026-09-04: the
|
||||
# rest_command integration silently failed to register on this box
|
||||
# (404 on /api/services/rest_command despite valid config + secret).
|
||||
# NOT reproducible: 2026-09-06 the integration + doorman_unlock service
|
||||
# are registered again (suspect: transient state after a reload-only
|
||||
# deploy window; rest_command now also exposes a reload service).
|
||||
# curl stays as the verified-live dispatch path — swapping back to
|
||||
# rest_command (drops the token from the process list) is gated on one
|
||||
# armed badge re-test (founder call, questions-v1).
|
||||
doorman_unlock: "curl -sS -m 10 '{{ doorman_unlock_url }}'"
|
||||
|
||||
rest_command:
|
||||
# Fires the doorctl endpoint on pfvsvrpi (systemd socket-activated,
|
||||
# Tailscale-bound, token+source allowlisted; self-securing relay).
|
||||
@@ -173,7 +186,7 @@ automation:
|
||||
{{ is_state('input_boolean.doorman_unlock_enabled', 'on')
|
||||
and not (trigger.json.local_unlock | default(false) | bool(false)) }}
|
||||
then:
|
||||
- action: rest_command.doorman_unlock
|
||||
- action: shell_command.doorman_unlock
|
||||
continue_on_error: true
|
||||
- action: logbook.log
|
||||
data:
|
||||
|
||||
@@ -542,27 +542,65 @@ sensor:
|
||||
method: left
|
||||
unique_id: pfv_garage_pdu_energy
|
||||
|
||||
# --- Rate-of-change early warning (#682): +10°F/15min = AC trouble [#344] ---
|
||||
# --- Smoothed ambient feeds for the trend sensors (2026-09-06) ---
|
||||
# DRAC sensors quantize to whole °C and FLAP ~1.8°F between polls; raw-fed
|
||||
# trend sensors false-tripped TEMP RISING FAST (05:02 CDT, tsys6). Trend
|
||||
# inputs are 5-min time-weighted moving averages (legacy filter SENSOR
|
||||
# platform — a top-level `filter:` key cannot be merged from packages).
|
||||
# Real climbs survive the smoothing, quantization flapping does not.
|
||||
- platform: filter
|
||||
name: "PFV tsys1 rack temperature smoothed"
|
||||
unique_id: pfv_tsys1_rack_temperature_smoothed
|
||||
entity_id: sensor.pfv_tsys1_rack_temperature
|
||||
filters:
|
||||
- filter: time_simple_moving_average
|
||||
window: "00:05"
|
||||
precision: 2
|
||||
- platform: filter
|
||||
name: "PFV tsys6 DRAC ambient smoothed"
|
||||
unique_id: pfv_tsys6_drac_ambient_smoothed
|
||||
entity_id: sensor.pfv_tsys6_drac_ambient
|
||||
filters:
|
||||
- filter: time_simple_moving_average
|
||||
window: "00:05"
|
||||
precision: 2
|
||||
- platform: filter
|
||||
name: "PFV tsys7 DRAC inlet smoothed"
|
||||
unique_id: pfv_tsys7_drac_inlet_smoothed
|
||||
entity_id: sensor.pfv_tsys7_drac_inlet
|
||||
filters:
|
||||
- filter: time_simple_moving_average
|
||||
window: "00:05"
|
||||
precision: 2
|
||||
- platform: filter
|
||||
name: "PFV garage temperature smoothed"
|
||||
unique_id: pfv_garage_temperature_smoothed
|
||||
entity_id: sensor.pfv_garage_temperature
|
||||
filters:
|
||||
- filter: time_simple_moving_average
|
||||
window: "00:05"
|
||||
precision: 2
|
||||
|
||||
binary_sensor:
|
||||
- platform: trend
|
||||
sensors:
|
||||
server_room_temp_rising:
|
||||
entity_id: sensor.pfv_tsys1_rack_temperature
|
||||
entity_id: sensor.pfv_tsys1_rack_temperature_smoothed
|
||||
sample_duration: 900
|
||||
min_gradient: 0.011
|
||||
unique_id: trend_server_room_rising
|
||||
tsys6_ambient_rising:
|
||||
entity_id: sensor.pfv_tsys6_drac_ambient
|
||||
entity_id: sensor.pfv_tsys6_drac_ambient_smoothed
|
||||
sample_duration: 900
|
||||
min_gradient: 0.011
|
||||
unique_id: trend_tsys6_ambient_rising
|
||||
tsys7_inlet_rising:
|
||||
entity_id: sensor.pfv_tsys7_drac_inlet
|
||||
entity_id: sensor.pfv_tsys7_drac_inlet_smoothed
|
||||
sample_duration: 900
|
||||
min_gradient: 0.011
|
||||
unique_id: trend_tsys7_inlet_rising
|
||||
garage_temp_rising:
|
||||
entity_id: sensor.pfv_garage_temperature
|
||||
entity_id: sensor.pfv_garage_temperature_smoothed
|
||||
sample_duration: 900
|
||||
min_gradient: 0.011
|
||||
unique_id: trend_garage_rising
|
||||
@@ -800,6 +838,13 @@ template:
|
||||
state: >-
|
||||
{% set v = states('sensor.tsys1_ups_battery_runtime') %}
|
||||
{{ (v | float / 60) | round(0) if is_number(v) else none }}
|
||||
- name: "PFV Tripp Lite runtime minutes"
|
||||
unique_id: pfv_triplite_runtime_minutes
|
||||
device_class: duration
|
||||
unit_of_measurement: "min"
|
||||
state: >-
|
||||
{% set v = states('sensor.tsys1_triplite_battery_runtime') %}
|
||||
{{ (v | float / 60) | round(0) if is_number(v) else none }}
|
||||
- name: "PFV tsys1 drive temperature"
|
||||
unique_id: pfv_tsys1_drive_temperature
|
||||
device_class: temperature
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# ============================================================================
|
||||
# UPS graceful-shutdown chain (#790) — the #455 ordering, automated
|
||||
# ============================================================================
|
||||
# DESIGN (unchanged from the 2026-09-05 session spec):
|
||||
# DRY_RUN is the DEFAULT and the shipped mode. The chain LOGS + PAGES the
|
||||
# exact shutdown plan (#455: VMs before hypervisors, pfv-bms last of the
|
||||
# VMs — the brain stays alive as long as possible). Execution (real
|
||||
# qmshutdown / PVE API calls) is double-gated:
|
||||
# 1. input_boolean.pfv_ups_shutdown_armed — founder arm, default OFF
|
||||
# 2. input_boolean.pfv_ups_shutdown_dry_run — default ON
|
||||
# Arming for the Sept 11-12 live test needs, in order: founder review of
|
||||
# the VM ordering below (edit this file via PR), PVE API tokens provisioned
|
||||
# into on-box secrets.yaml (NeedsInput ticket), then a supervised live test.
|
||||
#
|
||||
# PLAN DATA — captured from LIVE inventory 2026-09-06 (qm list, all hosts):
|
||||
# tsys1: 100 pfv-bms, 101 tsys-ca, 102 k8s-cnode1, 103 netinfra-01,
|
||||
# 104 librenms, 105 pve-datacenter | tsys3: 313 k8s-wnode
|
||||
# tsys4: 400 PBS | tsys5: test/preprod bed (5xxx) | tsys6: 600-605
|
||||
# tsys7: 701-711, 902 | tsys9: 904, 905
|
||||
# ============================================================================
|
||||
|
||||
input_boolean:
|
||||
# NOTE: `initial:` wins over restored state at EVERY HA start — a deploy
|
||||
# mid-test silently reverts these to the safe defaults (armed OFF,
|
||||
# dry-run ON). Safe direction, but re-arm after any restart during the
|
||||
# supervised Sept 11-12 live test.
|
||||
# (no description: keys — input_boolean rejects them, 2026-09-06)
|
||||
pfv_ups_shutdown_armed:
|
||||
# Founder arm. ON + dry-run OFF = real shutdown dispatch.
|
||||
name: UPS shutdown chain ARMED
|
||||
icon: mdi:power-standby
|
||||
initial: false
|
||||
pfv_ups_shutdown_dry_run:
|
||||
# ON (default) = log/page the plan only, touch nothing.
|
||||
name: UPS shutdown DRY RUN
|
||||
icon: mdi:numeric-1-box-outline
|
||||
initial: true
|
||||
|
||||
input_datetime:
|
||||
pfv_ups_shutdown_last_page:
|
||||
name: UPS shutdown last plan page
|
||||
has_date: true
|
||||
has_time: true
|
||||
icon: mdi:clock-alert-outline
|
||||
|
||||
automation:
|
||||
- id: pfv_ups_shutdown_plan
|
||||
alias: PFV UPS shutdown - plan (dry-run by default)
|
||||
description: >-
|
||||
On battery with the shutdown window opening (runtime < 20 min or
|
||||
charge < 50%), log + page the #455 ordered plan. Re-pages at most
|
||||
every 6h per outage. When armed AND dry-run OFF (founder + supervised
|
||||
test only), the plan page says EXECUTE — real dispatch wiring lands
|
||||
with PVE API tokens (#790 NeedsInput); until then nothing but text
|
||||
can be emitted by this automation.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: numeric_state
|
||||
entity_id:
|
||||
- sensor.pfv_ups_runtime_minutes
|
||||
- sensor.pfv_tripp_lite_runtime_minutes
|
||||
below: 20
|
||||
- trigger: numeric_state
|
||||
entity_id:
|
||||
- sensor.tsys1_ups_battery_charge
|
||||
- sensor.tsys1_triplite_battery_charge
|
||||
below: 50
|
||||
# 5-min level re-check — same edge-vs-level reasoning as the
|
||||
# runtime-low alert: never let a threshold-crossing blip silence
|
||||
# the plan page for an entire discharge.
|
||||
- trigger: time_pattern
|
||||
minutes: /5
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: >-
|
||||
{% set ns = namespace(hit=false) %}
|
||||
{% for rt, st in [
|
||||
('sensor.pfv_ups_runtime_minutes', 'sensor.tsys1_ups_status'),
|
||||
('sensor.pfv_tripp_lite_runtime_minutes', 'sensor.tsys1_triplite_status')] %}
|
||||
{% if is_number(states(rt)) and (states(rt) | float) < 20
|
||||
and states(st) not in ['Online', 'OL', 'unknown', 'unavailable'] %}
|
||||
{% set ns.hit = true %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% for ch, st in [
|
||||
('sensor.tsys1_ups_battery_charge', 'sensor.tsys1_ups_status'),
|
||||
('sensor.tsys1_triplite_battery_charge', 'sensor.tsys1_triplite_status')] %}
|
||||
{% if is_number(states(ch)) and (states(ch) | float) < 50
|
||||
and states(st) not in ['Online', 'OL', 'unknown', 'unavailable'] %}
|
||||
{% set ns.hit = true %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{{ ns.hit }}
|
||||
# Status 'unavailable' suppresses (anti-noise); UPS COMMS LOST covers
|
||||
# that failure mode separately after 5 min.
|
||||
- condition: template
|
||||
value_template: >-
|
||||
{{ states('input_datetime.pfv_ups_shutdown_last_page') in ['unknown', 'unavailable', 'none']
|
||||
or as_timestamp(now()) - as_timestamp(states('input_datetime.pfv_ups_shutdown_last_page')) > 21600 }}
|
||||
actions:
|
||||
- variables:
|
||||
execute_mode: >-
|
||||
{{ is_state('input_boolean.pfv_ups_shutdown_armed', 'on')
|
||||
and not is_state('input_boolean.pfv_ups_shutdown_dry_run', 'on') }}
|
||||
plan: >-
|
||||
UPS ON BATTERY — #455 shutdown plan (VMs before hypervisors):
|
||||
1. tsys5 test bed: qmshutdown 5105 awx, 5101 siem, 51017 sec onion,
|
||||
53100-53102 preprod, 5108-5109, 513 sectestbed k8s, 5000 sandbox
|
||||
2. k8s workers: tsys9 905, tsys7 701, tsys6 601, tsys3 313
|
||||
(drain + shutdown; cnodes 102/603/705 after workers)
|
||||
3. LOB/infra VMs: tsys6 600 awx, 602/605, tsys7 702 uisp, 703, 706-711,
|
||||
902 ucs-2, 604/711 mailgw (after queue drain), tsys9 904 netinfra-02
|
||||
4. tsys1: 104 librenms, 105 pve-dc, 103 netinfra-01, 101 tsys-ca
|
||||
(ca LAST — signing path for everything above)
|
||||
5. pfv-bms VM 100 LAST of the VMs — the brain stays up until now
|
||||
6. Hypervisors: iDRAC graceful off tsys6/tsys7 (PowerEdge);
|
||||
tsys1, tsys3, tsys4, tsys5, tsys9 idle out on UPS exhaustion
|
||||
(OptiPlex/Precision, no BMC)
|
||||
mode_note: >-
|
||||
{{ 'MODE: EXECUTE (armed, live dispatch)' if execute_mode
|
||||
else 'MODE: DRY RUN (plan only — nothing will be shut down)' }}
|
||||
- action: logbook.log
|
||||
data:
|
||||
name: UPS shutdown chain
|
||||
message: "{{ mode_note }} Plan: {{ plan | truncate(250) }}"
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: "{{ 'UPS SHUTDOWN EXECUTING' if execute_mode else 'UPS SHUTDOWN PLAN (dry run)' }}"
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: >-
|
||||
{{ mode_note }} Runtime APC {{ states('sensor.pfv_ups_runtime_minutes') }}
|
||||
min / TrippLite {{ states('sensor.pfv_tripp_lite_runtime_minutes') }} min.
|
||||
Order: test bed → k8s workers → LOB VMs → tsys1 infra → CA →
|
||||
pfv-bms (100) last → hypervisors. Full plan in the logbook.
|
||||
- action: persistent_notification.create
|
||||
data:
|
||||
title: "{{ 'UPS SHUTDOWN EXECUTING' if execute_mode else 'UPS SHUTDOWN PLAN (dry run)' }}"
|
||||
message: "{{ plan }}"
|
||||
# Suppression stamp set LAST — a failed notify must not suppress the
|
||||
# next 5-min re-page cycle.
|
||||
- action: input_datetime.set_datetime
|
||||
target:
|
||||
entity_id: input_datetime.pfv_ups_shutdown_last_page
|
||||
data:
|
||||
datetime: "{{ now().strftime('%Y-%m-%d %H:%M:%S') }}"
|
||||
@@ -0,0 +1 @@
|
||||
# intentionally empty — scenes are authored in the HA UI if ever needed
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
# intentionally empty — scripts are authored in the HA UI if ever needed
|
||||
|
||||
Vendored
+7
@@ -0,0 +1,7 @@
|
||||
# CI-only stub for `homeassistant check_config` (see .gitea/workflows/ci.yml).
|
||||
# NEVER copied to the live box. Values are syntactically-valid placeholders.
|
||||
deploy_webhook_id: ci-stub-deploy-webhook
|
||||
doorman_scan_webhook_id: ci-stub-scan-webhook
|
||||
doorman_unlock_url: http://127.0.0.1:1/unlock-stub
|
||||
gitea_auth_header: token ci-stub-gitea-token
|
||||
kuma_push_url: http://127.0.0.1:1/push?status=up&msg=ci-stub
|
||||
@@ -0,0 +1,92 @@
|
||||
#!/bin/bash
|
||||
# SDLC regression tests (#778) — run in CI (see .gitea/workflows/ci.yml).
|
||||
# Each test guards a rule that exists because of a real incident or ruling.
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/.." || exit 1
|
||||
|
||||
FAIL=0
|
||||
t() {
|
||||
# $1 = label, $2 = shell command string
|
||||
if eval "$2"; then
|
||||
echo " OK $1"
|
||||
else
|
||||
echo " FAIL $1"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
echo "=== alert twins (every alert ships a recovery twin) ==="
|
||||
t "utility power restored twin" \
|
||||
"grep -q 'id: pfv_plant_ups_power_restored' automations.yaml"
|
||||
t "temperature recovered twin" \
|
||||
"grep -q 'id: pfv_plant_temperature_recovered' automations.yaml"
|
||||
|
||||
echo "=== alert deep links (taps land on dashboards, #344) ==="
|
||||
ALERTS="POWER OUTAGE|UPS BATTERY LOW|UPS COMMS LOST|UPS RUNTIME LOW|Power restored|HIGH TEMPERATURE|SENSOR WATCHDOG|TEMP RISING FAST|TEMPERATURE RECOVERED"
|
||||
DL_OK=1
|
||||
IFS='|' read -ra ALERT_ARR <<< "$ALERTS"
|
||||
for a in "${ALERT_ARR[@]}"; do
|
||||
if ! grep -A2 "title: $a$" automations.yaml | grep -q " url: "; then
|
||||
echo " (missing deep link: $a)"
|
||||
DL_OK=0
|
||||
fi
|
||||
done
|
||||
t "all alerts deep-linked" "[ \"$DL_OK\" -eq 1 ]"
|
||||
|
||||
echo "=== watchdog coverage (#789) ==="
|
||||
t "watchdog watches the deploy sensor" \
|
||||
"grep -q 'sensor.pfv_deploy_gitea_release_head, 30' automations.yaml"
|
||||
t "watchdog automation exists" \
|
||||
"grep -q 'id: pfv_watchdog_stale_entities' automations.yaml"
|
||||
|
||||
echo "=== ghost entities (dead references never return) ==="
|
||||
t "no tsys9_ups ghost refs" \
|
||||
"! grep -rq 'tsys9_ups' automations.yaml packages/ dashboards/"
|
||||
|
||||
echo "=== secrets ==="
|
||||
t "no private keys in tree" \
|
||||
"! grep -rInE 'BEGIN (RSA |OPENSSH |EC |DSA )?PRIVATE KEY' --exclude-dir=.git --exclude-dir=docs ."
|
||||
|
||||
echo "=== alert hygiene: push only when something is wrong (2026-09-06 live-fire) ==="
|
||||
# The 30-min sweep used to page every cycle even when the fleet was all-OK
|
||||
# (48 pages/day of noise). Notify must sit behind the dead>0 + 4h-cooldown gate.
|
||||
t "plant watchdog notify gated (dead-only + cooldown input)" \
|
||||
"grep -q 'pfv_plant_watchdog_last_notify' configuration.yaml && grep -q 'notify_gate' automations.yaml"
|
||||
|
||||
# tsys6 DRAC ambient quantizes to whole degrees C and flaps ~2F; the raw-fed
|
||||
# trend sensor tripped TEMP RISING FAST with zero real climb (2026-09-06 05:02 CDT).
|
||||
t "trend sensors fed from smoothed sources" \
|
||||
"grep -q 'time_simple_moving_average' packages/plant_snmp.yaml && sed -n '/^binary_sensor:/,/^utility_meter:/p' packages/plant_snmp.yaml | grep -q 'entity_id: sensor.pfv_tsys6_drac_ambient_smoothed'"
|
||||
|
||||
# Tripp Lite had runtime sensors but no runtime-low alert — half the plant's
|
||||
# shutdown window was unmonitored.
|
||||
t "runtime-low covers Tripp Lite" \
|
||||
"sed -n '/id: pfv_plant_ups_runtime_low/,/^ mode:/p' automations.yaml | grep -q 'sensor.pfv_tripp_lite_runtime_minutes'"
|
||||
t "runtime-low status check maps per-UPS" \
|
||||
"sed -n '/id: pfv_plant_ups_runtime_low/,/^ mode:/p' automations.yaml | grep -q \"sensor.pfv_tripp_lite_runtime_minutes', 'sensor.tsys1_triplite_status'\""
|
||||
|
||||
echo "=== UPS shutdown chain (#790 — dry-run is the shipped mode) ==="
|
||||
t "shutdown chain package present, dry-run default ON" \
|
||||
"grep -q 'id: pfv_ups_shutdown_plan' packages/ups_shutdown.yaml && grep -A4 'pfv_ups_shutdown_dry_run:' packages/ups_shutdown.yaml | grep -q 'initial: true'"
|
||||
t "armed switch default OFF" \
|
||||
"grep -A4 'pfv_ups_shutdown_armed:' packages/ups_shutdown.yaml | grep -q 'initial: false'"
|
||||
# Host coverage must hold in the PLAN BODY, not the header comment —
|
||||
# scope the grep to the plan template variable.
|
||||
PLAN_OK=1
|
||||
for h in tsys1 tsys3 tsys4 tsys5 tsys6 tsys7 tsys9; do
|
||||
# full-read grep (not -q): the script runs with pipefail — -q's early
|
||||
# exit SIGPIPEs sed and flips the pipeline status to 141
|
||||
sed -n '/plan: >-/,/mode_note:/p' packages/ups_shutdown.yaml | grep "$h" >/dev/null || PLAN_OK=0
|
||||
done
|
||||
t "plan covers all seven hypervisors (plan body)" "[ \$PLAN_OK -eq 1 ]"
|
||||
t "pfv-bms VM 100 ordered last of the VMs (#455)" \
|
||||
"grep -q 'pfv-bms VM 100 LAST' packages/ups_shutdown.yaml"
|
||||
|
||||
echo "=== level-based re-checks (review finding: edge-only triggers miss pages) ==="
|
||||
t "runtime-low re-polls every 5 min" \
|
||||
"sed -n '/id: pfv_plant_ups_runtime_low/,/^ mode:/p' automations.yaml | grep -q 'minutes: /5'"
|
||||
t "shutdown chain re-polls every 5 min" \
|
||||
"grep -A30 'id: pfv_ups_shutdown_plan' packages/ups_shutdown.yaml | grep -q 'minutes: /5'"
|
||||
|
||||
echo "=== SDLC SUITE: $FAIL failures ==="
|
||||
exit "$FAIL"
|
||||
@@ -0,0 +1,2 @@
|
||||
# intentional placeholder: themes dir must exist for the
|
||||
# !include_dir_merge_named in configuration.yaml [#784 gardening]
|
||||
Reference in New Issue
Block a user