feat(netinfra): migrate DHCP from pfv-netboot to netinfra-01/02 with failover

ISC DHCP server with failover-peer (primary/secondary) deployed on
pfv-netinfra-01 (192.168.3.252) and pfv-netinfra-02 (192.168.3.253). Both
servers in "normal" state, load-balancing 333 active leases.

Migration details:
- Copied all 37 host reservations + subnet/pool config from pfv-netboot
- DHCP lease database copied from netboot for zero-disruption cutover
- DNS servers changed from 192.168.3.250 (netboot) to 252/253 (netinfra pair)
- NTP servers (252/253) added to DHCP options (netboot didn't hand out NTP)
- Netmask on both nodes fixed /24 -> /22 to match the network
- Webmin + DHCP module installed on both nodes (port 10000, SSL)
- pfv-netboot DHCP stopped + disabled
- Tested via sectestbed-sandbox (DHCP lease obtained from 252, verified DNS/NTP/gateway)
- Snapshot "pre-dhcp-migration" on sandbox as rollback point

Configs: netinfra/dhcp/dhcpd-{primary,secondary}.conf
Plan + results: netinfra/dhcp-migration.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-29 17:22:15 -05:00
parent f3821baa6c
commit 6cb59b5088
6 changed files with 547 additions and 1 deletions
+1
View File
@@ -32,6 +32,7 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
- [x] Zone replication via rsync systemd timer (every 60s)
- [x] Credentials + 2FA replicated identically to production
- [x] Both LAN IPs resolve knel.net device names + recurse externally
- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
+2 -1
View File
@@ -3,7 +3,7 @@
> **Index of all documentation in this repo.** Agents must update this file
> whenever a doc is added, removed, or substantively changed.
> **Last updated:** 2026-07-29
> **Last updated:** 2026-07-29 (DHCP migration)
## Kubernetes Architecture ([`k8s/`](k8s/))
@@ -44,6 +44,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
| [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 |
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
+125
View File
@@ -0,0 +1,125 @@
# DHCP Migration: pfv-netboot → pfv-netinfra-01/02
**Date:** 2026-07-29
**Status:****COMPLETE** — both servers in `normal` failover state, 333
active leases load-balanced. Netboot DHCP stopped + disabled.
> **Safety nets:** pfv-netboot, pfv-netinfra-01/02, sectestbed-sandbox, and
> ultix-streaming all have static IPs (reachable via Tailscale even if DHCP
> fails). sectestbed-sandbox snapshot `pre-dhcp-migration` exists as rollback.
## Migration results (2026-07-29 17:20 CDT)
| Check | Result |
|-------|--------|
| netinfra-01 (primary) | `active`, failover `normal`, 244 active leases |
| netinfra-02 (secondary) | `active`, failover `normal`, 89 active leases |
| Failover partnership | "Both servers normal" — load-balanced at split=128 |
| DHCP test (sandbox) | Lease obtained from 192.168.3.252, IP 192.168.3.50/22 |
| DNS handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
| NTP handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
| Domain | `knel.net` |
| Gateway | `192.168.3.254` |
| Webmin | Port 10000 (SSL) on both nodes; DHCP module configured |
| Netboot DHCP | Stopped + disabled (won't start on reboot) |
| Netmask fix | Both nodes corrected /24 → /22 |
## 1. Current state (discovered 2026-07-29)
| | pfv-netboot (SOURCE) | pfv-netinfra-01 (TARGET primary) | pfv-netinfra-02 (TARGET secondary) |
|---|---|---|---|
| IP | 192.168.3.250/22 | 192.168.3.252/24 | 192.168.3.253/24 |
| isc-dhcp-server | 4.4.3, **active** | NOT installed | NOT installed |
| Webmin | 2.652 (port 10000, SSL) | NOT installed | NOT installed |
| DHCP leases | **532 active** | — | — |
| DHCP config | `/etc/dhcp/dhcpd.conf` | — | — |
**Source DHCP config summary:**
- Subnet: `192.168.0.0/22` (covers .0.x/.1.x/.2.x/.3.x)
- Pool range: `192.168.0.1 — 192.168.3.200`
- Router/GW: `192.168.3.254`
- DNS: `192.168.3.250` (netboot itself — **changing to 252/253**)
- Domain: `knel.net`
- Lease times: 600s default, 7200s max
- DDNS: `ddns-update-style none` (globally disabled)
- ~37 host reservations (MAC → fixed IP)
- 3 global-scope host declarations (outside subnet)
## 2. What changes in the migrated config
| Setting | Old (netboot) | New (netinfra) | Why |
|---------|--------------|----------------|-----|
| DNS servers handed out | `192.168.3.250` | `192.168.3.252, 192.168.3.253` | Point clients at the netinfra Pi-hole pair (netboot will be decommissioned) |
| Failover | none (single server) | ISC DHCP failover-peer | Primary/secondary HA |
| DHCP interface | `eth0` | `ens18` | Different NIC name on netinfra VMs |
| Netmask on servers | /22 | **fix to /22** (currently /24) | Match the /22 network |
Everything else (subnet, pool range, reservations, lease times, domain) stays
identical. The host reservation list is copied verbatim.
## 3. Configs prepared
| File | Destination |
|------|-------------|
| [`dhcp/dhcpd-primary.conf`](dhcp/dhcpd-primary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-01 |
| [`dhcp/dhcpd-secondary.conf`](dhcp/dhcpd-secondary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-02 |
Both are identical except for the `failover peer` block (primary vs secondary).
## 4. Execution sequence
### Phase 1 — pre-flight (non-disruptive)
1. **Snapshot sectestbed-sandbox** (VMID 6000 on tsys5) — rollback point
2. **Fix netmask** on netinfra-01/02: `/24``/22` in `/etc/network/interfaces`
(then `ifdown ens18 && ifup ens18`, or `systemctl restart networking`)
3. Verify Tailscale still reachable after netmask change
### Phase 2 — install prerequisites (non-disruptive, nothing serving yet)
On **both** netinfra-01 and netinfra-02:
1. `apt-get update && apt-get install -y isc-dhcp-server`
2. Install Webmin via official setup script
3. Copy `dhcpd-{primary,secondary}.conf``/etc/dhcp/dhcpd.conf`
4. Write `/etc/default/isc-dhcp-server` with `INTERFACESv4="ens18"`
5. Syntax check: `dhcpd -t -cf /etc/dhcp/dhcpd.conf`
6. **Do NOT start the DHCP service yet**
### Phase 3 — cutover (the critical moment)
**Goal:** minimize the DHCP gap. Existing clients hold valid leases for 600
seconds — they won't notice a brief outage.
1. **Stop netboot DHCP:** `ssh netboot 'sudo systemctl stop isc-dhcp-server'`
2. **Immediately start netinfra-01:** `systemctl start isc-dhcp-server`
3. **Immediately start netinfra-02:** `systemctl start isc-dhcp-server`
4. **Verify failover partnership:** check `journalctl -u isc-dhcp-server` on both
— look for "failover peer pfv-dhcp: I move from recover to recover-done"
5. Wait ~10 seconds for lease sync
### Phase 4 — test
1. On sectestbed-sandbox: edit `/etc/network/interfaces` to DHCP
2. `dhclient -r` (release) then `dhclient` (request new lease)
3. Verify IP assigned, correct DNS, correct gateway
4. If fails: `qm rollback 6000 <snapshot>` and debug
5. If succeeds: change back to static (sandbox's normal config)
### Phase 5 — cleanup
1. **Disable netboot DHCP:** `ssh netboot 'sudo systemctl disable isc-dhcp-server'`
2. Verify Webmin DHCP module visible at `https://192.168.3.252:10000`
3. Document + commit
## 5. Failover mechanics (reference)
ISC DHCP failover uses a TCP connection on port **647** between the two servers.
- **Primary** owns the pool initially and can assign new leases immediately.
- **Secondary** enters `recover` state, syncs the lease database from primary,
then transitions to `recover-done``normal`.
- If primary dies, secondary continues serving after the MCLT timeout (600s).
- If secondary dies, primary continues serving alone.
- **split 128** = load balance 50/50 (primary handles even hashes, secondary odd).
- Host reservations (fixed-address) are NOT subject to failover — both servers
can independently assign reserved IPs.
+195
View File
@@ -0,0 +1,195 @@
# dhcpd.conf — pfv-netinfra-01 (PRIMARY)
# Migrated from pfv-netboot 2026-07-29
# Managed via Webmin DHCP module
#
# FAILOVER: this node is PRIMARY; peer is pfv-netinfra-02 (192.168.3.253)
# Global defaults
option domain-name "knel.net";
option domain-name-servers 192.168.3.252, 192.168.3.253;
option ntp-servers 192.168.3.252, 192.168.3.253;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
# ----- failover peer (PRIMARY) -----
failover peer "pfv-dhcp" {
primary;
address 192.168.3.252;
port 647;
peer address 192.168.3.253;
peer port 647;
max-response-delay 30;
max-unacked-updates 10;
mclt 600;
split 128;
load balance max seconds 3;
}
# ----- subnet (shared /22) -----
subnet 192.168.0.0 netmask 255.255.252.0 {
option routers 192.168.3.254;
option domain-name-servers 192.168.3.252, 192.168.3.253;
option ntp-servers 192.168.3.252, 192.168.3.253;
option domain-name "knel.net";
authoritative;
allow unknown-clients;
pool {
failover peer "pfv-dhcp";
range 192.168.0.1 192.168.3.200;
}
# ---- host reservations (fixed-address; not subject to failover pool) ----
host pfv-r3-mgmt {
hardware ethernet 00:14:22:69:1c:37;
fixed-address 192.168.0.7;
}
host pfv-r3-stor {
hardware ethernet 00:13:72:46:95:e4;
fixed-address 192.168.0.9;
}
host pfv-printer {
hardware ethernet 40:9f:38:b0:b5:2f;
fixed-address 192.168.1.84;
}
host pfv-r2-tor1 {
hardware ethernet 00:0d:56:41:7a:4d;
fixed-address 192.168.0.10;
}
host pfv-core-sw01 {
hardware ethernet a4:ba:db:6f:ce:28;
fixed-address 192.168.0.12;
}
host upstairs-receiver {
hardware ethernet 74:5e:1c:76:e2:60;
fixed-address 192.168.0.21;
}
host ap-TableMount {
hardware ethernet e0:63:da:36:73:39;
fixed-address 192.168.3.54;
}
host AP-WallMount {
hardware ethernet e0:63:da:33:bb:1d;
fixed-address 192.168.1.182;
}
host pfv-consrv {
hardware ethernet 00:60:2e:01:50:aa;
fixed-address 192.168.3.56;
}
host garagepdu {
hardware ethernet 00:c0:b7:7e:49:78;
fixed-address 192.168.3.18;
}
host pfv-dvr {
hardware ethernet 54:2b:57:37:a7:d9;
fixed-address 192.168.3.84;
}
host appletv-livingroom {
hardware ethernet d0:d2:b0:97:81:c2;
fixed-address 192.168.1.81;
}
host pfv-stor1 {
hardware ethernet 00:00:c0:34:0c:dc;
fixed-address 192.168.1.166;
}
host 3dscan {
hardware ethernet b8:27:eb:91:31:82;
fixed-address 192.168.0.4;
}
host pfv-jetson-nano-1 {
hardware ethernet 00:04:4b:e4:17:7b;
fixed-address 192.168.3.186;
}
host tsys7-oob {
hardware ethernet f8:bc:12:35:1e:c6;
fixed-address 192.168.3.197;
}
host tsys6-oob {
hardware ethernet a4:ba:db:0b:df:a0;
fixed-address 192.168.3.196;
}
host tsys-siem {
hardware ethernet 00:15:5d:64:e8:33;
fixed-address 192.168.3.81;
}
host brother-label-printer {
hardware ethernet 04:fe:a1:56:72:e2;
fixed-address 192.168.3.52;
}
host pfv-bms {
hardware ethernet 02:5A:39:38:3E:9F;
fixed-address 192.168.3.12;
}
host stl-canon-scanner-artroom {
hardware ethernet 74:38:b7:24:fa:4e;
fixed-address 192.168.3.142;
}
host tsys-ucs-01 {
hardware ethernet bc:24:11:86:ea:1a;
fixed-address 192.168.2.51;
}
host tsys-ucs-02 {
hardware ethernet bc:24:11:c8:da:34;
fixed-address 192.168.2.54;
}
host dell-openmanage-enterprise {
hardware ethernet bc:24:11:ac:f4:6b;
fixed-address 192.168.2.113;
}
host pfv-rrinfra-rtr {
hardware ethernet 00:1d:70:0b:4f:41;
fixed-address 192.168.3.94;
}
host pfv-tsys1 {
hardware ethernet 34:17:eb:b3:b1:2d;
fixed-address 192.168.3.11;
}
host pfv-tsys2 {
hardware ethernet 18:fd:cb:00:d2:ca;
fixed-address 192.168.2.3;
}
host pfv-tsys3 {
hardware ethernet a4:4c:c8:08:d1:b8;
fixed-address 192.168.2.5;
}
host pfv-tsys4 {
hardware ethernet 98:90:96:c4:96:9a;
fixed-address 192.168.3.191;
}
host pfv-tsys5 {
hardware ethernet 18:03:73:43:ce:de;
fixed-address 192.168.0.20;
}
host pfv-tsys6 {
hardware ethernet 00:21:9b:a2:7c:53;
fixed-address 192.168.3.169;
}
host pfv-tsys7 {
hardware ethernet f8:bc:12:34:e0:74;
fixed-address 192.168.0.250;
}
host pfv-tsys9 {
hardware ethernet a4:bb:6d:e3:56:86;
fixed-address 192.168.3.58;
}
}
# ---- host declarations outside subnet (global scope, same as netboot) ----
host subodev-torsw01 {
hardware ethernet 00:14:22:69:18:a7;
fixed-address 192.168.0.8;
}
host pfv-r1-tor-top {
hardware ethernet 00:23:ae:c1:ad:e8;
fixed-address 192.168.0.11;
}
host tailscale-router {
hardware ethernet bc:24:11:8a:69:04;
fixed-address 192.168.3.16;
}
+193
View File
@@ -0,0 +1,193 @@
# dhcpd.conf — pfv-netinfra-02 (SECONDARY)
# Migrated from pfv-netboot 2026-07-29
# Managed via Webmin DHCP module
#
# FAILOVER: this node is SECONDARY; peer is pfv-netinfra-01 (192.168.3.252)
# Global defaults
option domain-name "knel.net";
option domain-name-servers 192.168.3.252, 192.168.3.253;
option ntp-servers 192.168.3.252, 192.168.3.253;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
# ----- failover peer (SECONDARY) -----
failover peer "pfv-dhcp" {
secondary;
address 192.168.3.253;
port 647;
peer address 192.168.3.252;
peer port 647;
max-response-delay 30;
max-unacked-updates 10;
load balance max seconds 3;
}
# ----- subnet (shared /22) -----
subnet 192.168.0.0 netmask 255.255.252.0 {
option routers 192.168.3.254;
option domain-name-servers 192.168.3.252, 192.168.3.253;
option ntp-servers 192.168.3.252, 192.168.3.253;
option domain-name "knel.net";
authoritative;
allow unknown-clients;
pool {
failover peer "pfv-dhcp";
range 192.168.0.1 192.168.3.200;
}
# ---- host reservations (identical to primary) ----
host pfv-r3-mgmt {
hardware ethernet 00:14:22:69:1c:37;
fixed-address 192.168.0.7;
}
host pfv-r3-stor {
hardware ethernet 00:13:72:46:95:e4;
fixed-address 192.168.0.9;
}
host pfv-printer {
hardware ethernet 40:9f:38:b0:b5:2f;
fixed-address 192.168.1.84;
}
host pfv-r2-tor1 {
hardware ethernet 00:0d:56:41:7a:4d;
fixed-address 192.168.0.10;
}
host pfv-core-sw01 {
hardware ethernet a4:ba:db:6f:ce:28;
fixed-address 192.168.0.12;
}
host upstairs-receiver {
hardware ethernet 74:5e:1c:76:e2:60;
fixed-address 192.168.0.21;
}
host ap-TableMount {
hardware ethernet e0:63:da:36:73:39;
fixed-address 192.168.3.54;
}
host AP-WallMount {
hardware ethernet e0:63:da:33:bb:1d;
fixed-address 192.168.1.182;
}
host pfv-consrv {
hardware ethernet 00:60:2e:01:50:aa;
fixed-address 192.168.3.56;
}
host garagepdu {
hardware ethernet 00:c0:b7:7e:49:78;
fixed-address 192.168.3.18;
}
host pfv-dvr {
hardware ethernet 54:2b:57:37:a7:d9;
fixed-address 192.168.3.84;
}
host appletv-livingroom {
hardware ethernet d0:d2:b0:97:81:c2;
fixed-address 192.168.1.81;
}
host pfv-stor1 {
hardware ethernet 00:00:c0:34:0c:dc;
fixed-address 192.168.1.166;
}
host 3dscan {
hardware ethernet b8:27:eb:91:31:82;
fixed-address 192.168.0.4;
}
host pfv-jetson-nano-1 {
hardware ethernet 00:04:4b:e4:17:7b;
fixed-address 192.168.3.186;
}
host tsys7-oob {
hardware ethernet f8:bc:12:35:1e:c6;
fixed-address 192.168.3.197;
}
host tsys6-oob {
hardware ethernet a4:ba:db:0b:df:a0;
fixed-address 192.168.3.196;
}
host tsys-siem {
hardware ethernet 00:15:5d:64:e8:33;
fixed-address 192.168.3.81;
}
host brother-label-printer {
hardware ethernet 04:fe:a1:56:72:e2;
fixed-address 192.168.3.52;
}
host pfv-bms {
hardware ethernet 02:5A:39:38:3E:9F;
fixed-address 192.168.3.12;
}
host stl-canon-scanner-artroom {
hardware ethernet 74:38:b7:24:fa:4e;
fixed-address 192.168.3.142;
}
host tsys-ucs-01 {
hardware ethernet bc:24:11:86:ea:1a;
fixed-address 192.168.2.51;
}
host tsys-ucs-02 {
hardware ethernet bc:24:11:c8:da:34;
fixed-address 192.168.2.54;
}
host dell-openmanage-enterprise {
hardware ethernet bc:24:11:ac:f4:6b;
fixed-address 192.168.2.113;
}
host pfv-rrinfra-rtr {
hardware ethernet 00:1d:70:0b:4f:41;
fixed-address 192.168.3.94;
}
host pfv-tsys1 {
hardware ethernet 34:17:eb:b3:b1:2d;
fixed-address 192.168.3.11;
}
host pfv-tsys2 {
hardware ethernet 18:fd:cb:00:d2:ca;
fixed-address 192.168.2.3;
}
host pfv-tsys3 {
hardware ethernet a4:4c:c8:08:d1:b8;
fixed-address 192.168.2.5;
}
host pfv-tsys4 {
hardware ethernet 98:90:96:c4:96:9a;
fixed-address 192.168.3.191;
}
host pfv-tsys5 {
hardware ethernet 18:03:73:43:ce:de;
fixed-address 192.168.0.20;
}
host pfv-tsys6 {
hardware ethernet 00:21:9b:a2:7c:53;
fixed-address 192.168.3.169;
}
host pfv-tsys7 {
hardware ethernet f8:bc:12:34:e0:74;
fixed-address 192.168.0.250;
}
host pfv-tsys9 {
hardware ethernet a4:bb:6d:e3:56:86;
fixed-address 192.168.3.58;
}
}
# ---- host declarations outside subnet (global scope, same as netboot) ----
host subodev-torsw01 {
hardware ethernet 00:14:22:69:18:a7;
fixed-address 192.168.0.8;
}
host pfv-r1-tor-top {
hardware ethernet 00:23:ae:c1:ad:e8;
fixed-address 192.168.0.11;
}
host tailscale-router {
hardware ethernet bc:24:11:8a:69:04;
fixed-address 192.168.3.16;
}
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
# install-dhcp.sh — installs isc-dhcp-server + Webmin on a netinfra node.
# Does NOT start the DHCP service. Run on the target node itself.
set -e
echo "=== Installing isc-dhcp-server ==="
apt-get update -qq
apt-get install -y isc-dhcp-server
echo "=== Installing Webmin ==="
if ! dpkg -l | grep -q '^ii.*webmin'; then
curl -fsSL https://raw.githubusercontent.com/webmin/webmin/master/webmin-setup-repo.sh -o /tmp/webmin-setup.sh
sh /tmp/webmin-setup.sh -f
rm -f /tmp/webmin-setup.sh
apt-get install -y webmin
else
echo "Webmin already installed"
fi
echo "=== Writing /etc/default/isc-dhcp-server ==="
cat > /etc/default/isc-dhcp-server <<'EOF'
# Defaults for isc-dhcp-server (sourced by /etc/init.d/isc-dhcp-server)
INTERFACESv4="ens18"
INTERFACESv6=""
EOF
echo "=== Stopping DHCP service (should not serve yet) ==="
systemctl stop isc-dhcp-server 2>/dev/null || true
systemctl disable isc-dhcp-server 2>/dev/null || true
echo "=== Done. DHCP installed but NOT started. ==="