fix(plant): alert hygiene — watchdog gate, trend smoothing, UPS [#344]
ci / lint (push) Successful in 1m31s

Three live-fire defects: unconditional 30-min watchdog page, tsys6 flap
false-positive, Tripp Lite runtime-low gap. TDD: 4 new regression tests.
On-box ha core check passed before commit.

Meat + verification: https://projects.knownelement.com/issues/344#note-4966
This commit is contained in:
2026-09-06 05:24:58 -05:00
parent 83b32d5d9f
commit 7e13612752
5 changed files with 127 additions and 33 deletions
+58 -24
View File
@@ -68,15 +68,25 @@
mode: single
- id: pfv_plant_ups_runtime_low
alias: PFV plant - UPS runtime low while on battery
description: On battery and runtime burned below 10 min — the graceful
shutdown window is closing. VMs before hypervisors per #455.
description: >-
On battery and runtime burned below 10 min — the graceful shutdown
window is closing. VMs before hypervisors per #455 ordering. Covers
BOTH UPSes (APC + Tripp Lite — the Tripp Lite gap closed 2026-09-06);
the status check maps each runtime sensor to its own UPS status.
triggers:
- trigger: numeric_state
entity_id: sensor.pfv_ups_runtime_minutes
entity_id:
- sensor.pfv_ups_runtime_minutes
- sensor.pfv_tripp_lite_runtime_minutes
below: 10
conditions:
- condition: template
value_template: "{{ states('sensor.tsys1_ups_status') not in ['Online', 'unknown', 'unavailable'] }}"
value_template: >-
{% set status_map = {
'sensor.pfv_ups_runtime_minutes': 'sensor.tsys1_ups_status',
'sensor.pfv_tripp_lite_runtime_minutes': 'sensor.tsys1_triplite_status'} %}
{{ states(status_map.get(trigger.entity_id, 'sensor.tsys1_ups_status'))
not in ['Online', 'OL', 'unknown', 'unavailable'] }}
actions:
- action: notify.send_message
target:
@@ -86,7 +96,7 @@
data:
title: UPS RUNTIME LOW
url: /lovelace-pfv-server-room/ups
message: On battery {{ states('sensor.pfv_ups_runtime_minutes') }} min left — start graceful shutdown now (#455 ordering).
message: '{{ state_attr(trigger.entity_id, ''friendly_name'') }}: {{ states(trigger.entity_id) }} min left — start graceful shutdown now (#455 ordering).'
mode: single
- id: pfv_plant_ups_power_restored
alias: PFV plant - utility power restored
@@ -190,31 +200,55 @@
mode: single
- id: pfv_plant_sensor_watchdog
alias: PFV plant - sensor watchdog (unavailable sensors)
description: HA is the environmental brain (#344) — a silent sensor is an outage.
Re-alert suppressed by 4h cooldown via last_notified input_text if present.
description: >
HA is the environmental brain (#344) — a silent sensor is an outage.
Fleet-wide sweep (every pfv_* sensor, broader than the 5-min critical
watchdog). PUSHES ONLY when sensors are actually dead, and re-pages a
still-dead fleet at most every 4h — the unconditional 30-min page was
retired 2026-09-06 after live-fire alert fatigue. All-OK cycles log
quietly; the Kuma dead-man heartbeat (#792) owns liveness signaling.
mode: single
max_exceeded: silent
triggers:
- trigger: time_pattern
minutes: /30
actions:
- action: notify.send_message
target:
entity_id:
- notify.ultix_sidecar_new
- notify.allthes_ipad
data:
title: SENSOR WATCHDOG
url: /lovelace-pfv-server-room/overview
message: >-
{% set fleet = states.sensor | selectattr('entity_id', 'match', 'sensor\.pfv_')
| list %}
{% set dead = fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list %}
{% if dead | length > 0 %}
- variables:
fleet: "{{ states.sensor | selectattr('entity_id', 'match', 'sensor\\.pfv_') | list }}"
dead: "{{ fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list }}"
# 4h cooldown: page again only if the last page is older than 4h
# (or we never paged). Fresh HA start (no restore) counts as never.
# as_timestamp math — aware/naive datetime subtraction throws in Jinja.
notify_gate: >-
{{ states('input_datetime.pfv_plant_watchdog_last_notify') in ['unknown', 'unavailable', 'none']
or as_timestamp(now()) - as_timestamp(states('input_datetime.pfv_plant_watchdog_last_notify')) > 14400 }}
- if:
- condition: template
value_template: "{{ dead | length > 0 and notify_gate }}"
then:
- action: notify.send_message
target:
entity_id:
- notify.ultix_sidecar_new
- notify.allthes_ipad
data:
title: SENSOR WATCHDOG
url: /lovelace-pfv-server-room/overview
message: >-
{{ dead | length }} of {{ fleet | length }} fleet sensors silent:
{{ dead | map(attribute='entity_id') | join(', ') }}
{% else %}
all {{ fleet | length }} fleet sensors OK
{% endif %}
mode: single
- action: input_datetime.set_datetime
target:
entity_id: input_datetime.pfv_plant_watchdog_last_notify
data:
datetime: "{{ now().strftime('%Y-%m-%d %H:%M:%S') }}"
else:
- action: logbook.log
data:
name: Plant watchdog
message: >-
Sweep: all {{ fleet | length }} fleet sensors answering
({{ dead | length }} silent, cooldown {{ 'active' if not notify_gate else 'n/a' }}).
- id: pfv_plant_temp_rising_fast
alias: PFV plant - temperature rising fast (rate-of-change)
description: Catches dead AC minutes before the 104F threshold trips.