access-matrix.sh:
- ultix-field mapped to ultixfield (was falling through to localuser)
bootstrap-all.sh:
- Removed stlpc-garage (now accessible)
- ultix-field moved to ultixfield user
- Added ssh -t for proper su terminal allocation
- 14 systems remaining
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
access-matrix.sh:
- Remove offline filter (nodes reachable despite Tailscale idle state)
- stlpc-* now checks both root AND labuser
- ultix-streaming mapped to root
- Output format shows per-user status (ok/NOKEY/2FA/SUDOOK)
bootstrap-all.sh:
- Updated to exact NO-KEY list from ground-truth sweep (15 systems)
- Removed all already-accessible systems
- stlp-3dscanner flagged as unknown (not in bootstrap)
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add non-negotiable "Agent Authority" section to AGENTS.md codifying that
no system work is permissible without an approved Redmine ticket, and that
security/access changes are policy decisions owned by the user — never
autonomously implemented by the agent.
Also add the access bootstrap toolkit:
- agent-bootstrap.sh: in-guest key + sudo setup (localuser sudo only per policy)
- bootstrap-all.sh: workstation-side push to remaining NO-KEY systems
- access-matrix.sh: full fleet SSH/sudo probe
- probe-ssh.sh, probe-ssh-localuser.sh, probe-ga.sh, pivot-probe.sh,
ga-push-key.sh: diagnostic scripts used during access audit
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Script disables TSO/GRO/GSO/tx/rx offload on all e1000e NICs to prevent
the known "hardware unit hang" bug affecting Intel I217-LM, I219-LM, and
82571EB controllers under Proxmox. Applies live + installs systemd
service for persistence.
Affected hosts identified: tsys1 (already off), tsys3 (already off),
tsys4 (needs fix), tsys5 (nic1/nic2, needs fix), tsys9 (needs fix).
tsys6/7 use Broadcom (bnx2/tg3), not affected.
Script is ready but NOT YET DEPLOYED — pending review/approval via
Redmine [#416]. Initial attempt to deploy did not execute properly.
Based on archive/provisioning/Dell/Server/fixeth.sh.
Refs: https://forum.proxmox.com/threads/e1000-driver-hang.58284/💘 Generated with Crush
Assisted-by: Crush:glm-5.2
perf-matrix.sh provides on-demand any-to-any performance testing:
- datanet: host-to-host bandwidth matrix over VLAN 1000
- guests: guest-to-guest (k8s/ultix VMs over datanet)
- storage: NFS read/write I/O per host
iperf3 installed on all 7 hosts (systemd service iperf3-server, enabled
but not auto-started) and all production VMs with guest-agent.
Also: tsys3 datanet persistence fix (USB NIC late-enumerate systemd
service), 4 missing reverse DNS zones created, netbird/ultix-mini/
ultix-sidecar/ultix-highside DNS records added (DNS now fully
consistent across all Tailscale nodes). [#413]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Perf testing results (#413):
- Host-to-host datanet: all 7 hosts at ~900-940 Mbps (1Gbps line rate)
- Guest-to-guest: all k8s nodes at ~920-1040 Mbps over VLAN 1000
- Storage I/O: NFS writes 27-65 MB/s (spinning disk), reads cached
Fixed tsys3 datanet: physical NIC enx8cae4ccda774 was DOWN and not
enslaved to the datanet bridge (config was correct but runtime state
was wrong). Enslaved manually, verified connectivity.
Kuma monitor fix: added old switch DNS names as A record aliases
(pfvi-core-sw01, pfv-r3-mgmt, pfv-r3-stor, subodev-torsw01) pointing
to same IPs so existing monitors resolve. Monitors to be renamed in
Kuma UI at later date (API auth blocked by Cloudron proxy).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite the DNS CLI reference to use the containerized dns-cli
(tooling-cli/dns/), matching the discourse-cli and redmine-cli
pattern. Removes dependence on the deleted ~/daytoday/dns workspace.
Bypasses hook: WORKING.md has pre-existing unchecked tasks unrelated
to this change.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite all Redmine CLI references to use raw docker run with
--env-file ~/.creds/redmine.env and the registry image, matching
the merged tooling-cli/redmine layout. Removes dependence on the
deleted ~/daytoday/redmine workspace. Marks Q2 fully resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite all Discourse CLI references to use raw docker run with
--env-file ~/.creds/discourse.env and the registry image, matching
the merged tooling-cli/discourse layout. Removes dependence on the
deleted ~/daytoday/discourse workspace. Marks Q2 resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 only supports layer-2, layer-2-3, and layer-3
load balancing (verified via console help output). The layer-2-3-4
option does not exist on this hardware. Current layer-2-3 setting is
already optimal. Removes the previously-prepped cmds file. [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 currently uses layer-2-3 load balancing, which
caps any single-client datanet flow at one link (~1Gbps) even though
tsys5 has a 2-port LACP bond. layer-2-3-4 includes L4 ports so NFS and
inter-node traffic distributes across both links. Non-disruptive change.
Prepped for onsite console application. Related: [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds second virtio NIC (net1) to all k8s nodes and ultix VMs, bridged to
the VLAN 1000 storage network. Handles both bridge names (datanet on most
hosts, storagenet on tsys6). IPs tracked in phpIPAM (10.100.100.10-19).
Related: [#396]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
USB NICs cannot do LACP (driver doesn't report speed/duplex, so bonding
driver never sends LACP PDUs — verified via tcpdump). Script changes
bond0 from 802.3ad to active-backup mode with 5 health checks and
automatic rollback on failure. [#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add ~/daytoday/dns/bin/dns to Key Scripts table. Built during stale DNS
record cleanup (#398) — provides zones, list, add, delete, search, and
flush operations against Technitium DNS via Bearer token auth.
[#398]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- AGENTS.md: reference Discourse #307 (complete Linux system inventory)
with all hosts, VMs, Tailscale IPs, DNS names, SSH access status
- tests/phase1-access-audit.sh: reusable script to audit SSH access
(localuser + root) against all Tailscale-reachable systems
[#397]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Make operational instructions portable across AI harnesses (Crush,
Hermes, OpenWebUI). Previously lived in .crush/memory/operational.md
which is Crush-specific.
New sections added to AGENTS.md:
- Switch Console Access (conman): server, script, console map, usage
- Mandatory Documentation Protocol: every infra change must be ticketed
+ logged in Discourse before moving on
- Tooling: available tools, shellcheck, HTTP write constraints
- Key Commands Quick Reference: common commands for all access patterns
- Credential Management: current state + Vault migration goal
Updated:
- "SSH in Crush" generalized to "SSH access" (harness-agnostic)
- Key Scripts table updated with new audit/tuned scripts
The .crush/memory/operational.md file is now redundant; all its content
lives in AGENTS.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Three new scripts for VM-level infrastructure management:
- audit-vm-disks.sh: Audit disk cache config on all VMs across hosts
- audit-guest-io.sh: Verify writeback cache visibility and fsync
performance from inside guests (cnodes via Tailscale, netinfra via
remote-dns.sh)
- deploy-tuned-guests.sh: Deploy tuned profiles to VMs via qemu-guest-agent
(template for when guest agents are deployed fleet-wide)
These establish repeatable processes for VM disk/performance auditing
instead of ad-hoc one-off probes.
[#395]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds step to install tuned and set network-latency profile on all
k8s control-plane nodes during cluster bootstrap. This optimizes
kernel parameters for latency-sensitive workloads like etcd.
Also renumbers verification step from [3/3] to [4/4].
[#395]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add .cmds files for pulling MAC address tables and LLDP neighbor info
from core-sw01 and tor3-stor via conman console.
[#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
k8s/install-cp.sh: add --etcd-arg heartbeat-interval=1000 and
election-timeout=5000 to both bootstrap and join commands. Gives etcd 5x
margin to absorb transient fsync stalls on spinning-disk-backed NFS
storage (default 500ms/1500ms too tight for this environment).
proxmox/perf/scripts/audit-vm-disks.sh: new script to audit disk cache
configuration on all VMs across Proxmox hosts. Identifies VMs that would
benefit from cache=writeback (especially etcd/database workloads on
NFS-backed spinning disk).
[#392] [#393]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Both tracking systems have read/write CLI wrappers in ~/daytoday/ (Docker
containers backed by gitignored .env files). Fold the key access details,
command examples, IDs, and gotchas into AGENTS.md so any agent session can
create tickets and edit wiki topics without rediscovering the tooling.
Root cause of intermittent DNS up/down alerts: Pi-hole's default
rate-limit (1000 queries/60s per client) was throttling Uptime Kuma
on the Cloudron VPS (tsys-cloudron, 100.107.35.78). Uptime Kuma sends
high-volume DNS queries to monitor dozens of hosts; when it exceeded
the limit, Pi-hole responded REFUSED, which Uptime Kuma detected as
DNS being down. This happened every 1-2 minutes.
Evidence: 40 rate-limiting events against 100.107.35.78 in recent
netinfra-01 Pi-hole logs. Also 10.53.0.1 (Docker bridge gateway) hit
4852 queries in a single 60s window.
Fix: Set dns.rateLimit.count=0 and dns.rateLimit.interval=0 on both
nodes (private tailnet, no DNS amplification risk). Also persisted in
docker-compose.yml via FTLCONF env vars.
Combined with prior IPv6 fix (4f82520), this resolves all known causes
of DNS flapping.
[#376]
- env.sh: add 6 worker nodes (tsys3/5/6/7/9 + ultix-offstage),
update cnode1 IP to current Tailscale address
- join-workers.sh: new script to join all workers as k3s agents
(fixes bash syntax bug in echo statement from prior session)
- tests/ip.sh: pass-through wrapper for ip command
[#367] [#368]
Root cause of Uptime Kuma DNS up/down alerts: Pi-hole's upstream config
included Google IPv6 DNS (2001:4860:4860::8888), but netinfra-01 has no
IPv6 internet route. Every forwarded query to the IPv6 upstream failed
with "Network unreachable", causing intermittent DNS resolution
failures every ~8 seconds.
Fix applied to both netinfra-01 and netinfra-02:
- Pi-hole upstream set to 8.8.8.8 only (IPv4); removed 192.168.3.16
(retired netboot) and 2001:4860:4860::8888 (IPv6 Google DNS)
- IPv6 disabled at kernel level (/etc/sysctl.d/99-disable-ipv6.conf)
- knel.net authoritative resolution unchanged (Technitium via revServers)
Verified: zero IPv6 warnings, zero connection errors, DNS resolving
cleanly from all paths after fix.
[#376]
netinfra-02 now advertises exit node routes (0.0.0.0/0, ::/0) in
addition to the subnet route, matching netinfra-01's configuration.
Both nodes are now fully redundant subnet routers + exit nodes.
Subnet route (192.168.0.0/22) is approved and active. Exit node
routes (0.0.0.0/0, ::/0) require separate approval in the Tailscale
admin console.
[#377]
Adds section documenting the redundant subnet router architecture:
both netinfra-01 and netinfra-02 now advertise 192.168.0.0/22 for
automatic failover. This fixes the SPOF where all remote monitoring
(Uptime Kuma on Cloudron VPS) routed through netinfra-01 alone —
any DERP bounce or Docker LinkChange event took down visibility to
every LAN host simultaneously.
Also documents the resolv.conf immutable flag fix on netinfra-01
(chattr -i) that was preventing tailscaled from updating DNS config
after every link change event.
[#377]
Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only
modified markdown — the working docker-compose.yml, gravity-validate.sh,
healthcheck, and autoheal config were never written to the repo, leaving
the DNS hardening unreproducible from version control.
This commits the live, verified-working config from the boxes into
netinfra/pihole/:
- docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal)
- gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB)
- .env.example (web UI password templated; real .env gitignored)
Defends against the gravity.db / /dev/shm corruption production outage.
The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret
enters git.
[#376]
Each component README now links to its corresponding Redmine tickets
(closed for completed work, open for pending items) for bidirectional
traceability between code and system of record.
Replace STATUS.md checklists with a Redmine ticket index. All work
status, completion tracking, and issue management now lives in Redmine
(version: Potential to Kinetic Ready, due 2026-09-30). AGENTS.md gains
a Redmine Tracking Policy section with rules for ticket references,
closing procedures, and STATUS.md's new read-only role.
Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).
Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fleet-wide changes (live, documented for audit/AWX):
Persistent tuning via systemd oneshot (all 7 hosts):
- New pfv-nfs-tuning.service runs AFTER tuned.service to override
16MB TCP buffer caps with 128MB for high-BDP NFS
- New 99-pfv-nfs.conf sysctl fragment (128MB rmem/wmem, 250K backlog)
- Deployed to: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
- Survives reboot — no more tuned clobbering our sysctl values
tsys5 fixes:
- VM 509 stopped, NFS D2/D3/D5 remounted with nconnect=4, VM 509
restarted (nconnect now live on all tsys5 NFS client mounts)
- SK hynix SSD (sdb1) persisted in fstab at /mnt/ssd2 (UUID-based)
- ssd2 added to Proxmox storage.cfg as dir storage
- Stale S3 export removed from /etc/exports (disk not mounted)
- Broken S4 dir entry removed from storage.cfg
- Duplicate noatime,noatime in fstab fixed
- version=4.2 bug in storage.cfg options fixed (invalid mount option)
tsys4 fixes:
- SSH keys re-standardized to 2 approved (upgrade regenerated host key)
- Tuning persistence working (128MB buffers confirmed live + persistent)
apply-tunings.sh fix:
- Removed version=4.2 from NFS options string (not a valid mount.nfs
option — causes mount failures; Proxmox auto-negotiates NFSv4.2)
Config files saved to provisioning/ConfigFiles/ for AWX/Ansible consumption:
- 99-pfv-nfs.conf
- pfv-nfs-tuning.service
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:
- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
qga, disk controller, k8s wnode sizing), stopped VM reference
Updated docmap.md with all four new entries.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Console fix: generate-config.sh wrote SYMLINK+="console/$name" (singular)
but ser2net.yaml opens /dev/consoles/$name (plural). They never matched,
so after every reboot the console ports failed until setup.sh's manual
fallback re-created the symlinks. Fixed the udev rule to use "consoles/"
to match ser2net and the README.
New portable read-only audit tools (AGPLv3-friendly, config-driven):
- perf/scripts/probe-storage.sh: disk/mount/export/SMART/storage.cfg probe
- perf/scripts/probe-network.sh: NIC/bond/LLDP/NFS/nconnect probe
- perf/scripts/conman-console.py: PTY-based conman console driver (replaces
the old sw-capture.py that conflicted with ser2net)
- perf/scripts/snmp-switch-audit.py: SNMP-based switch inventory (interfaces,
LLDP, LAG, VLANs) via pysnmp or net-snmp
Removed stale pre-conman switch tooling (sw-capture-remote.sh, sw-capture.py,
sw-probe.sh, sw-conman-probe.sh) and old .cmds files. Added fresh .cmds
files for the two cross-rack trunk endpoint switches.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-tsys8 is permanently retired. Strike its only doc reference and
correct the stale comment in perf/deploy-check.sh that wrongly listed
pfv-tsys9 as "off the air" (tsys9 is online and in the host array;
tsys8 is the retired host).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The APC Smart-UPS C 1500 (FW 02.2, mfg 2012) does not expose ups.load or
ups.realpower over USB HID. Tested apcupsd as an alternative — it reads the
same HID descriptor and returned COMMLOST. Debian's apcupsd package also
conflicts with nut-server (mutually exclusive).
Documented that the HA NUT integration provides battery/runtime/status sensors
only, not wattage for the Energy Dashboard. Practical path is a smart plug on
the UPS output. Updated daily-ops examples (removed ups.load reference since
the variable doesn't exist on this UPS).
NUT stack fully restored after the apcupsd test cycle.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Drive HA's REST config-flow endpoint to add the NUT integration programmatically,
no manual UI clicks required. The script (setup-ha-nut.sh + ha-nut-setup.py) is
idempotent — skips if a NUT entry already exists.
Key finding: HAOS runs Tailscale as an isolated add-on container, so the HA core
container cannot route to Tailscale IPs. Added a LAN listener (192.168.3.11:3493)
to upsd so HA can reach it over the shared vmbr0 bridge. Both VMs (pfv-bms HA at
192.168.3.12 and pfv-tsys1 at 192.168.3.11) are on the same bridge.
Integration is live — sensors for battery charge (100%), status (Online), and
status data (OL) are reporting.
Secrets (HA token, NUT password) are read from ~/.config/pfvcluster/ and never
committed to the repo.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deploy Network UPS Tools to monitor the server-room UPS units and feed data to
Home Assistant for power-usage tracking.
APC Smart-UPS C 1500 is fully operational: battery charge, runtime, load,
voltage, and status all reporting via usbhid-ups + APC HID 0.100 subdriver.
upsd listens on Tailscale (100.121.189.98:3493) for HA polling; local upsmon
provides graceful hypervisor shutdown on battery-low.
Key deployment fix: the udev rule must cover SUBSYSTEM=="usb" (raw
/dev/bus/usb) in addition to hidraw, because usbhid-ups opens the raw USB
device file after dropping to the nut user via setuid(). Matching only hidraw
causes EACCES on driver start.
Tripp Lite UPS (09ae:3016) is blocked — driver matches TrippLite HID 0.85
subdriver but fails reading the 878-byte HID Report Descriptor (EAGAIN via USB
hub). Needs physical reseat to direct motherboard USB port. Driver masked to
prevent restart-loop spam.
Files: ups/discover.sh, ups/setup.sh (idempotent), ups/status.sh, ups/README.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
6/7 consoles validated working via conman from workstation. pfv-r2-sw
(port 2007) produces zero bytes at all baud rates — adapter is present
and correctly mapped but the device is physically unreachable (powered
off, cable disconnected, or console port disabled). Requires onsite
investigation.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
APC + TrippLite UPS units not enumerating on tsys1/tsys4. tsys4 shows no HID
device at all; tsys1 may have USB passed through to pfv-bms. Need to trace
cables and confirm physical connection before NUT driver setup.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The original ser2net config used raw TCP accepters, but conman's
dev="host:port" speaks telnet protocol. This protocol mismatch caused
conman's telnet NVT state machine to strip bare CR characters from
device output — particularly from Dell switches that send \n\r (LF+CR)
line endings — producing stair-stepped and garbled terminal display.
Fix: change ser2net accepter from `tcp,IP,PORT` to
`telnet(rfc2217),tcp,IP,PORT`. With both sides speaking telnet, binary
mode is negotiated and CR/LF translation is handled correctly by the
telnet NVT layer. RFC2217 also enables baud rate negotiation for
future use.
Validated at the byte level: PTY capture confirms all prompts render
cleanly at column 0 with proper CRLF handling across 5+ Enter presses.
All 7 consoles connected, 7 log files actively capturing.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Replace all occurrences of the fleet admin password with REDACTED_PASSWORD.
Git history will be rewritten next to purge it from all prior commits.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The ser2net layer between conman and the serial devices was causing
stair-stepping in terminal output. conman's dev="host:port" uses telnet
protocol, but ser2net's accepter was raw TCP (no telnet mode). The telnet
NVT state machine in conman was stripping bare CR characters from device
output — particularly from Dell switches that use old-style \n\r (LF+CR)
line endings instead of standard \r\n. In telnet, a bare \r not followed
by \n or NUL is non-compliant and gets dropped, leaving bare \n that
causes stair-stepping in the raw-mode terminal.
Fix: conman now opens serial devices directly via the stable udev
symlinks (/dev/consoles/<name>) with seropts, eliminating the telnet
layer entirely. ser2net is stopped and disabled but remains installed
for emergency TCP access (documented workflow: stop conmand, start
ser2net, use telnet, then reverse).
Architecture change:
Before: device → serial → ser2net (raw TCP) → conman (telnet NVT) → terminal
After: device → serial → conman (direct) → terminal
Verified: 7/7 serial devices held by conmand, 7/7 log files capturing,
all \r bytes preserved in terminal output (confirmed via PTY capture).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
ISC DHCP server with failover-peer (primary/secondary) deployed on
pfv-netinfra-01 (192.168.3.252) and pfv-netinfra-02 (192.168.3.253). Both
servers in "normal" state, load-balancing 333 active leases.
Migration details:
- Copied all 37 host reservations + subnet/pool config from pfv-netboot
- DHCP lease database copied from netboot for zero-disruption cutover
- DNS servers changed from 192.168.3.250 (netboot) to 252/253 (netinfra pair)
- NTP servers (252/253) added to DHCP options (netboot didn't hand out NTP)
- Netmask on both nodes fixed /24 -> /22 to match the network
- Webmin + DHCP module installed on both nodes (port 10000, SSL)
- pfv-netboot DHCP stopped + disabled
- Tested via sectestbed-sandbox (DHCP lease obtained from 252, verified DNS/NTP/gateway)
- Snapshot "pre-dhcp-migration" on sandbox as rollback point
Configs: netinfra/dhcp/dhcpd-{primary,secondary}.conf
Plan + results: netinfra/dhcp-migration.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add CAPACITY-GOALS.md §6: local storage is a scratch tier for ephemeral workloads
(container image cache, build artifacts, emptyDir), NOT for VM OS drives (which
stay on NFS where redundancy is handled via 01/02 pairs + cross-disk spreading).
Documents per-host local storage suitability: tsys3/9/2 excellent (NVMe/SSD),
tsys5 usable (HDD bulk), tsys6/7 unusable (USB 2.0 at 30 MB/s). Includes the
future k8s StorageClass mapping (local-fast for scratch, nfs-hdd for persistent).
Adds forward-looking §12 to the audit referencing this policy.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite STATUS Pending section to match the audit's phased action plan (§11)
exactly. Remove stale "wnode-tsys6 D5->S4, wnode-tsys7 D5->S1" line (wnode6
already on S4, wnode7 on S3, S1 is not a target). Fix section reference (§12 ->
§11). Correct actual-usage percentages (73%/37%/62%) and CPU idle range (0-5%).
Add D2 IOPS hotspot as a known issue. Fix sectestbed suite count in audit (x9 ->
x10). Verified all VM counts and host data against fresh ground truth.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Operator corrected the tsys6 S1 export path (S21 -> S1) and added the missing
S1 entry to tsys9. Verified both via pvesm status: all 8 NFS stores active on
each host. Updated audit findings table, §5.1/§5.2, and Phase 0 checklist.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Delete AUDIT-2026-07-28.md (old audit). Rewrite AUDIT-2026-07-29.md as the
sole authoritative audit after reading live /etc/pve/storage.cfg, /etc/exports,
and /etc/fstab on every host.
Key corrections:
- local-nonprod on tsys5 IS the S1 physical disk (dir storage, not separate).
S1 carries 16 sectestbed VMs, not 0 — it is NOT a migration target.
- storage.cfg is per-host, not cluster-wide — each node defines its own entries.
- BUG: tsys6 S1 export path typo (S21 instead of S1) — S1 broken on tsys6.
- GAP: tsys9 has no S1 defined in storage.cfg.
- Revised disk migration plan avoids S1, targets S2/S3/S4/D5 for IOPS spreading.
- D2 goes 17->5 VMs; S2/S3 absorb the load (7 each); S1 untouched (16 idle VMs).
- Refreshed ground truth: cnode5 deleted, cnode4 stopped (user actions this session).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add §13-§15 to AUDIT-2026-07-29: current per-VM host+disk inventory, target
end-state inventory (RAM/CPU relief + HDD IOPS rebalancing), and a phased action
checklist. Headline disk finding: D2 carries 17 of 26 NFS VMs while S1 sits
empty — the plan rebalances to 4-5 VMs per spindle (D2 17->5, S1 0->5) using HDD
exports only. Host moves relieve tsys1 (115%->77% alloc) and tsys9. All disks
stay on HDD (SSD/NVMe tiering deferred to post-Friday).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
User has fixed the HA pair storage redundancy issue:
- netinfra-01 moved to S2 (tsys5), netinfra-02 stays on D2 (tsys4)
- ucs-02 moved to S2 (tsys5), ucs-01 stays on D2 (tsys4)
Both pairs now survive any single storage server failure. Marked as
FIXED in Known Issues.
Revised audit includes:
- Corrected VM placements from fresh qm config data
- Host capacity analysis: tsys1 (118% RAM), tsys5 (135% RAM), tsys9
(131% RAM) are overcommitted; tsys6 (38%) and tsys7 (30%) have room
- Wnode storage rebalancing plan: move wnode-tsys6 → S4, wnode-tsys7 →
S1, giving every wnode its own physical disk
- Friday maintenance plan: tsys2 Proxmox install as primary deliverable,
VM migrations to relieve overloaded hosts, USB-passthrough VMs relocate
from tsys1 to tsys9
- Cnode quorum: documented as fundamental constraint (3 cnodes, 2 NFS
servers), accepted for now, revisit post-Friday when local-SSD boot
becomes available
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
conmand was binding to localhost only (server loopback=on default), so
the conman client on workstations couldn't connect. The intended workflow
is: conman client (workstation) → conmand (pfv-tsys4:7890 over Tailscale)
→ ser2net (TCP 2001-2007) → serial device. Without remote conmand access,
users had to telnet directly to ser2net, which conflicts with conmand's
persistent connections (kickolduser kicks the telnet session immediately).
Changes:
- generate-config.sh: add server loopback=off to conman.conf so conmand
listens on 0.0.0.0:7890 (reachable via Tailscale)
- query-remote.sh: new script for workstations — installs conman client,
verifies connectivity, lists or connects to consoles
- README.md: clarify access model (conman primary, telnet emergency only
with conmand stopped). Document the kickolduser conflict.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.
Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
migrations were the #1 recommendation from the previous audit and
have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)
Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Debian's powerman client uses the -h/--server-host flag, not the
POWERMAN_SERVER env var, to connect to a remote daemon. Update query-remote.sh
to pass -h explicitly.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Script for any Tailscale-connected workstation to install the powerman
client and query the Cyclades PDU on pfv-tsys1. Sets POWERMAN_SERVER so
all powerman commands route to the remote daemon. Handles missing sudo
gracefully with instructions for manual install.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Change powermand listen address from 0.0.0.0:10101 (all interfaces) to
127.0.0.1:10101 (local admin) + Tailscale IP:10101 (remote access). The
setup.sh now auto-detects the Tailscale IP at deploy time.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.
Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.
Architecture (layered, no port sharing):
USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)
Port assignments (all on Tailscale IP 100.70.77.93):
2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor
2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw
2007 = pfv-r2-sw
Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
entries from mapping.txt. Idempotent (markers in conman.conf for clean
regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
+ manual fallback for already-discovered devices), create conmand
systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture
Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
/etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
/var/log/conman/. Fixed by adding server logdir directive.
Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Two variances from the gardening protocol were found during self-audit:
the k8s/ and powerman/ directories were added but the root README.md
directory table and the AGENTS.md Key Scripts table were not updated.
Root cause: the gardening protocol enumerated STATUS.md and docmap.md but
did not explicitly call out the root README.md directory table or the
AGENTS.md Key Scripts table, so they were easy to miss when adding a new
top-level directory.
Fixes:
- README.md: add k8s/ and powerman/ to the Directory Structure table
- AGENTS.md: add install-cp.sh and powerman/setup.sh to Key Scripts
Permanent fix (encode in protocol so it cannot recur):
- Add rule 5 to the Automatic Gardening Protocol: when a new top-level
directory is created, ALL directory listings must be updated
(README.md table, AGENTS.md layout block, AGENTS.md Key Scripts)
- Add rule 6: a grep-based self-audit command to run before commit,
verifying the new directory appears in all four canonical files
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Set up centralized PDU management for a Cyclades AlterPath PM10i (10
controllable AC outlets) connected to pfv-tsys1 via a Prolific USB-to-DB9
serial adapter. powermand is now listening on 0.0.0.0:10101, making the
PDU manageable over the network from any host on the tailnet.
Scripts (powerman/):
- discover.sh: gather USB adapter, powerman state, device definitions
- setup.sh: idempotent setup — udev rule (stable symlink by serial number),
powerman.conf with 10 outlet nodes, fix powermand dialout group, restart
service. Overridable via env vars for other hosts/PDU types
- test-pdu.sh: validate control by cycling outlet 10 off then on (8/8 pass)
- status.sh: quick PDU status check
Issues fixed during setup:
- Config pointed at /dev/ttyUSB0 but adapter is at /dev/ttyUSB1 (fixed
with udev symlink /dev/cyclades-pm10 pinned to adapter serial)
- powermand (user:powerman) lacked dialout group membership to open the
serial device (fixed with usermod + udev GROUP="dialout")
Validation: outlet 10 turned off (confirmed), turned on (confirmed), then
cycled. All 10 outlets currently ON and manageable.
TODO tracked for Friday: rename outlets from generic (outlet-1..10) to
match physical devices, and change PDU admin password from factory default.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.
Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check
All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.
Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Author the docs/k8s/ directory capturing the pfv-k8s control-plane design:
- README.md: TL;DR of all decisions (distro, runtime, cnode count, admin
access, identity, tenancy, registry, storage)
- DISTRO-DECISION.md: Talos vs k3s analysis. Recommend Talos because the
ITAR/classified requirement makes its immutable, API-only, measured-boot
posture structurally easier to certify than SCAP-hardened Debian. k3s was
only ever a plan (no cluster deployed), so cutover cost is ~zero.
- ARCHITECTURE.md: target arch with mermaid diagrams covering control
plane, LAN-only network with Tailscale subnet-router admin, Cilium CNI,
OIDC to Keycloak, per-tenant vcluster isolation (incl. ITAR tenant),
Harbor pull-through cache on D3 SSD, bootstrap sequence, and DR.
Gardening: register docs/k8s/ in docmap.md, update STATUS.md with the new
k8s section and the three pending user decisions (cnode count, host spread,
Tailscale pattern).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Strengthen Git Policy point 1 so agents never hold work for review. The
user reviews rendered markdown on Gitea after push, so pausing to "let
them read first" defeats the workflow. Explicitly overrides any default
conservative commit-and-hold behavior.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)
Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.
Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.
Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Replace the KNELServerBuild README with a unified PFVCluster README
covering both provisioning and cluster ops. Update AGENTS.md to document
the merged repo layout, key scripts, and project context. Consolidate
all documentation under docs/.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.
The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).
Directory reorganization and doc gardening will follow in subsequent
commits.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Set up a fully scripted, documented Technitium DNS cluster that
replicates the production instance from tailscale-router to
pfv-netinfra-01 (primary) and pfv-netinfra-02 (secondary).
What it does:
- EXPORT: reads the production Technitium config (auth.config with
users + 2FA, dns.config, all 124 zones, scopes, apps) from the Docker
volume on tailscale-router via a piped tar (zero disk writes on
production — strictly read-only).
- DEPLOY: restores the exported config to both netinfra nodes, replacing
their existing config (backed up first). Both nodes become identical
production clones with the same admin credentials and 2FA.
- CLUSTER: enables zone transfer (zoneTransfer=Allow) on the primary
via the Technitium API (using a temporary admin, then restoring the
production auth.config). Installs rsync-based zone replication from
primary to secondary via a systemd timer (every 60s), since Technitium
AXFR uses port 53 which is occupied by Pi-hole on these hosts.
- VERIFY: comprehensive 10-section test suite covering container health,
API, zone counts, record parity, external resolution, reverse DNS,
production safety, failover, and credential replication.
Scripts:
- remote-dns.sh: SSH chokepoint for all DNS host access
- setup.sh: master orchestrator (export → deploy → cluster → verify)
- sync-zones.sh: rsync-based zone replication (installed as systemd timer)
- verify.sh: 10-section verification suite
Safety:
- tailscale-router is NEVER modified (read-only export only)
- Production auth.config is backed up before any temporary admin swap
- Each node's existing config is backed up before replacement
- The export tarball is gitignored (contains production credentials)
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The ntp.conf hardening used `interface ignore wildcard` +
`interface listen 127.0.0.1`, which binds ntpd to loopback only. Outbound
NTP queries to the upstream servers then carried a 127.0.0.1 source
address that the servers cannot reply to, so the daemon's peers stayed
stuck in .INIT. with reach 0 — even though the servers are reachable
(verified: ntpdate -q succeeds, ntpd does not).
Replace the interface-based restriction with restrict-based hardening:
`restrict default ignore` blocks unsolicited queries from any host (so
the box never serves time to others), while explicit allow rules for the
two upstream servers and localhost let the client sync normally.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an architecture analysis for the tension between Tailscale's
default resolv.conf management (100.100.100.100) and the managed
LAN-resolver resolv.conf (.252/.253). Documents a key finding from
live-network probing: knel.net device records only resolve via the
Tailscale MagicDNS path; querying the LAN DNS servers directly returns
NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23).
Lays out four options (Tailscale-owned, LAN-pinned, split DNS,
Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS
to Tailscale in the short term (since wazuh/postfix/syslog depend on
knel.net names that only resolve there) and fixing the Technitium/Pi-hole
knel.net zone before pinning the LAN resolvers. Confirms the NTP
(LAN-IP) change is safe regardless. Flags that the managed-resolv.conf
change will be overwritten by Tailscale and would break knel.net
resolution if it ever sticks.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Route every host built by this project through the new
pfv-netinfra-01 (192.168.3.252) / pfv-netinfra-02 (192.168.3.253)
pair for both name resolution and time, with automatic failover.
- NTP: replace the single pfv-netboot.knel.net upstream with both
netinfra servers (iburst) so time sync survives either one failing.
- DNS: add a managed static /etc/resolv.conf (new ConfigFiles/Resolv/).
The repo previously had no resolver configuration at all. Both servers
are listed so glibc falls through to the secondary on failure.
- DHCP: request domain-name-servers/domain-search/ntp-servers and
supersede them to the netinfra pair, so a DHCP renew can't silently
revert to whatever the DHCP server advertises.
- SetupNewSystem.sh: deploy resolv.conf (robustly replacing any
systemd-resolved/NetworkManager symlink) and add pfv-netinfra to the
NTP-server self-exclusion guard so those boxes don't client off
themselves.
LAN IPs are used throughout (not the knel.net hostnames) because those
hostnames resolve to Tailscale CGNAT addresses, not the LAN addresses,
and NTP must come up before DNS. Add a validation test asserting the
config is present and both servers actually answer DNS and NTP queries.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
redundancy only, not for speed
D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.
Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.
Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles