fix(plant): alert hygiene — watchdog gate, trend smoothing, UPS [#344]
ci / lint (push) Successful in 1m31s
ci / lint (push) Successful in 1m31s
Three live-fire defects: unconditional 30-min watchdog page, tsys6 flap false-positive, Tripp Lite runtime-low gap. TDD: 4 new regression tests. On-box ha core check passed before commit. Meat + verification: https://projects.knownelement.com/issues/344#note-4966
This commit is contained in:
+58
-24
@@ -68,15 +68,25 @@
|
||||
mode: single
|
||||
- id: pfv_plant_ups_runtime_low
|
||||
alias: PFV plant - UPS runtime low while on battery
|
||||
description: On battery and runtime burned below 10 min — the graceful
|
||||
shutdown window is closing. VMs before hypervisors per #455.
|
||||
description: >-
|
||||
On battery and runtime burned below 10 min — the graceful shutdown
|
||||
window is closing. VMs before hypervisors per #455 ordering. Covers
|
||||
BOTH UPSes (APC + Tripp Lite — the Tripp Lite gap closed 2026-09-06);
|
||||
the status check maps each runtime sensor to its own UPS status.
|
||||
triggers:
|
||||
- trigger: numeric_state
|
||||
entity_id: sensor.pfv_ups_runtime_minutes
|
||||
entity_id:
|
||||
- sensor.pfv_ups_runtime_minutes
|
||||
- sensor.pfv_tripp_lite_runtime_minutes
|
||||
below: 10
|
||||
conditions:
|
||||
- condition: template
|
||||
value_template: "{{ states('sensor.tsys1_ups_status') not in ['Online', 'unknown', 'unavailable'] }}"
|
||||
value_template: >-
|
||||
{% set status_map = {
|
||||
'sensor.pfv_ups_runtime_minutes': 'sensor.tsys1_ups_status',
|
||||
'sensor.pfv_tripp_lite_runtime_minutes': 'sensor.tsys1_triplite_status'} %}
|
||||
{{ states(status_map.get(trigger.entity_id, 'sensor.tsys1_ups_status'))
|
||||
not in ['Online', 'OL', 'unknown', 'unavailable'] }}
|
||||
actions:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
@@ -86,7 +96,7 @@
|
||||
data:
|
||||
title: UPS RUNTIME LOW
|
||||
url: /lovelace-pfv-server-room/ups
|
||||
message: On battery {{ states('sensor.pfv_ups_runtime_minutes') }} min left — start graceful shutdown now (#455 ordering).
|
||||
message: '{{ state_attr(trigger.entity_id, ''friendly_name'') }}: {{ states(trigger.entity_id) }} min left — start graceful shutdown now (#455 ordering).'
|
||||
mode: single
|
||||
- id: pfv_plant_ups_power_restored
|
||||
alias: PFV plant - utility power restored
|
||||
@@ -190,31 +200,55 @@
|
||||
mode: single
|
||||
- id: pfv_plant_sensor_watchdog
|
||||
alias: PFV plant - sensor watchdog (unavailable sensors)
|
||||
description: HA is the environmental brain (#344) — a silent sensor is an outage.
|
||||
Re-alert suppressed by 4h cooldown via last_notified input_text if present.
|
||||
description: >
|
||||
HA is the environmental brain (#344) — a silent sensor is an outage.
|
||||
Fleet-wide sweep (every pfv_* sensor, broader than the 5-min critical
|
||||
watchdog). PUSHES ONLY when sensors are actually dead, and re-pages a
|
||||
still-dead fleet at most every 4h — the unconditional 30-min page was
|
||||
retired 2026-09-06 after live-fire alert fatigue. All-OK cycles log
|
||||
quietly; the Kuma dead-man heartbeat (#792) owns liveness signaling.
|
||||
mode: single
|
||||
max_exceeded: silent
|
||||
triggers:
|
||||
- trigger: time_pattern
|
||||
minutes: /30
|
||||
actions:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: SENSOR WATCHDOG
|
||||
url: /lovelace-pfv-server-room/overview
|
||||
message: >-
|
||||
{% set fleet = states.sensor | selectattr('entity_id', 'match', 'sensor\.pfv_')
|
||||
| list %}
|
||||
{% set dead = fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list %}
|
||||
{% if dead | length > 0 %}
|
||||
- variables:
|
||||
fleet: "{{ states.sensor | selectattr('entity_id', 'match', 'sensor\\.pfv_') | list }}"
|
||||
dead: "{{ fleet | selectattr('state', 'in', ['unavailable', 'unknown']) | list }}"
|
||||
# 4h cooldown: page again only if the last page is older than 4h
|
||||
# (or we never paged). Fresh HA start (no restore) counts as never.
|
||||
# as_timestamp math — aware/naive datetime subtraction throws in Jinja.
|
||||
notify_gate: >-
|
||||
{{ states('input_datetime.pfv_plant_watchdog_last_notify') in ['unknown', 'unavailable', 'none']
|
||||
or as_timestamp(now()) - as_timestamp(states('input_datetime.pfv_plant_watchdog_last_notify')) > 14400 }}
|
||||
- if:
|
||||
- condition: template
|
||||
value_template: "{{ dead | length > 0 and notify_gate }}"
|
||||
then:
|
||||
- action: notify.send_message
|
||||
target:
|
||||
entity_id:
|
||||
- notify.ultix_sidecar_new
|
||||
- notify.allthes_ipad
|
||||
data:
|
||||
title: SENSOR WATCHDOG
|
||||
url: /lovelace-pfv-server-room/overview
|
||||
message: >-
|
||||
{{ dead | length }} of {{ fleet | length }} fleet sensors silent:
|
||||
{{ dead | map(attribute='entity_id') | join(', ') }}
|
||||
{% else %}
|
||||
all {{ fleet | length }} fleet sensors OK
|
||||
{% endif %}
|
||||
mode: single
|
||||
- action: input_datetime.set_datetime
|
||||
target:
|
||||
entity_id: input_datetime.pfv_plant_watchdog_last_notify
|
||||
data:
|
||||
datetime: "{{ now().strftime('%Y-%m-%d %H:%M:%S') }}"
|
||||
else:
|
||||
- action: logbook.log
|
||||
data:
|
||||
name: Plant watchdog
|
||||
message: >-
|
||||
Sweep: all {{ fleet | length }} fleet sensors answering
|
||||
({{ dead | length }} silent, cooldown {{ 'active' if not notify_gate else 'n/a' }}).
|
||||
- id: pfv_plant_temp_rising_fast
|
||||
alias: PFV plant - temperature rising fast (rate-of-change)
|
||||
description: Catches dead AC minutes before the 104F threshold trips.
|
||||
|
||||
Reference in New Issue
Block a user