Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 mrcharles
|
aeb2e80ab1 | ||
|
mrcharles
|
21adb89d4e |
@@ -78,26 +78,28 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
|
||||
| Issue | Impact | Status |
|
||||
|-------|--------|--------|
|
||||
| **UCS + netinfra HA pairs both on tsys4** | tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark | **CRITICAL — needs PDM migration** |
|
||||
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost | **CRITICAL — needs PDM migration** |
|
||||
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
|
||||
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
|
||||
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
|
||||
|
||||
## Pending (next session priorities)
|
||||
|
||||
1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical
|
||||
devices (Friday onsite)
|
||||
2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
||||
2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
|
||||
- Cnode count: 3 (recommended) vs 5
|
||||
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
|
||||
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
|
||||
3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
1. **CRITICAL: Migrate HA pairs to separate storage** (PDM, 10 min):
|
||||
- netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5)
|
||||
- ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5)
|
||||
2. **CRITICAL: Fix k3s cnode quorum risk** (PDM, 5 min):
|
||||
- Move cnode1 (VMID 906) or cnode2 (VMID 705) from tsys4 to tsys5 storage
|
||||
3. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
||||
4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
|
||||
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
7. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
||||
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
|
||||
per-tenant vcluster policy templates, solar-aware scale-out
|
||||
7. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
|
||||
solar-aware scale-out
|
||||
|
||||
> **See [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md)
|
||||
> for the full fresh audit with VM inventory and action items.**
|
||||
|
||||
## Infrastructure Summary
|
||||
|
||||
|
||||
+21
-9
@@ -64,18 +64,30 @@ All ports listen on the Tailscale IP (`100.70.77.93`).
|
||||
|
||||
### Connect to a console
|
||||
|
||||
**Primary method — conman client (with logging + multiplexing):**
|
||||
|
||||
```bash
|
||||
# Via conman (with logging + multiplexing):
|
||||
conman -f pfv-core-sw01 # attach to console
|
||||
conman -q # query status of all consoles
|
||||
|
||||
# Via telnet (direct, no logging):
|
||||
telnet 100.70.77.93 2001 # pfv-core-sw01
|
||||
|
||||
# From any Tailscale-connected device:
|
||||
telnet pfv-tsys4 2001
|
||||
# From any Tailscale-connected workstation:
|
||||
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
|
||||
conman -d pfv-tsys4:7890 -q # list all consoles
|
||||
```
|
||||
|
||||
Escape sequence: `&.` to disconnect, `&?` for help.
|
||||
|
||||
**Direct telnet (emergency only — conflicts with conman):**
|
||||
|
||||
```bash
|
||||
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
||||
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
||||
ssh pfv-tsys4 'systemctl stop conmand'
|
||||
telnet pfv-tsys4 2001 # pfv-core-sw01
|
||||
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
||||
```
|
||||
|
||||
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
||||
and kick your telnet session immediately ("Connection closed by foreign host").
|
||||
The correct workflow is conman client → conmand → ser2net → device.
|
||||
|
||||
### Re-deploy after changing mapping.txt
|
||||
|
||||
```bash
|
||||
|
||||
@@ -193,6 +193,12 @@ if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
|
||||
echo " Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
|
||||
fi
|
||||
|
||||
# Ensure loopback=off so conmand is reachable over Tailscale (not localhost-only)
|
||||
if ! grep -qiE '^\s*server\s+loopback\s*=' "$CONMAN_CONF" 2>/dev/null; then
|
||||
sed -i "/^server logdir/a server loopback=off" "$CONMAN_CONF"
|
||||
echo " Added server loopback=off to $CONMAN_CONF (enables remote access)"
|
||||
fi
|
||||
|
||||
# Remove any previous auto-generated block (between markers)
|
||||
# Then append the new block
|
||||
MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/bash
|
||||
#
|
||||
# console/query-remote.sh — install conman client and connect to a console
|
||||
# on pfv-tsys4 over Tailscale.
|
||||
#
|
||||
# Usage:
|
||||
# bash console/query-remote.sh # list consoles
|
||||
# bash console/query-remote.sh pfv-core-sw01 # connect to a console
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
REMOTE_HOST="${REMOTE_HOST:-pfv-tsys4}"
|
||||
REMOTE_PORT="${REMOTE_PORT:-7890}"
|
||||
|
||||
echo "============================================"
|
||||
echo " Conman Remote Console Access"
|
||||
echo " Server: ${REMOTE_HOST}:${REMOTE_PORT} (Tailscale)"
|
||||
echo "============================================"
|
||||
|
||||
# --- 1. Install conman client if missing ---
|
||||
if ! command -v conman >/dev/null 2>&1; then
|
||||
echo ""
|
||||
echo "--- Installing conman client ---"
|
||||
if sudo -n true 2>/dev/null; then
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq conman
|
||||
else
|
||||
echo " Passwordless sudo not available. Please run:"
|
||||
echo " sudo apt-get update && sudo apt-get install -y conman"
|
||||
echo " Then re-run this script."
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo " conman client already installed."
|
||||
fi
|
||||
|
||||
# --- 2. Verify connectivity ---
|
||||
echo ""
|
||||
echo "--- Connectivity check ---"
|
||||
if timeout 3 bash -c "echo > /dev/tcp/${REMOTE_HOST}/${REMOTE_PORT}" 2>/dev/null; then
|
||||
echo " [OK] ${REMOTE_HOST}:${REMOTE_PORT} reachable"
|
||||
else
|
||||
echo " [FAIL] Cannot reach ${REMOTE_HOST}:${REMOTE_PORT}"
|
||||
echo " Is Tailscale up? Is conmand running on ${REMOTE_HOST}?"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. List or connect ---
|
||||
CONSOLE="${1:-}"
|
||||
if [ -z "$CONSOLE" ]; then
|
||||
echo ""
|
||||
echo "--- Available consoles ---"
|
||||
conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
|
||||
echo ""
|
||||
echo "To connect: bash $0 <console-name>"
|
||||
echo " e.g: bash $0 pfv-core-sw01"
|
||||
else
|
||||
echo ""
|
||||
echo "--- Connecting to: $CONSOLE ---"
|
||||
echo " Escape sequence: &. (to disconnect)"
|
||||
echo ""
|
||||
conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -f "$CONSOLE"
|
||||
fi
|
||||
@@ -23,6 +23,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
|
||||
| Document | Description | Last Reviewed |
|
||||
|----------|-------------|---------------|
|
||||
| [`proxmox/PROJECT.md`](proxmox/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/AUDIT-2026-07-28.md`](proxmox/AUDIT-2026-07-28.md) | Fresh fleet audit with current VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
|
||||
| [`proxmox/TODO.md`](proxmox/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
# Fresh Fleet Audit — 2026-07-28
|
||||
|
||||
> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**
|
||||
> The tables in PROJECT.md reflect the 2026-07-27 audit; VMs have since been
|
||||
> migrated via PDM. This file is the current ground truth.
|
||||
|
||||
**Audit time:** 2026-07-28 19:55 CDT
|
||||
**Method:** `qm list` + `qm config` on all 7 reachable hosts
|
||||
**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9
|
||||
**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
|
||||
|
||||
---
|
||||
|
||||
## 1. Host Summary
|
||||
|
||||
| Host | CPU | Threads | RAM (GB) | Local Disk | Role | Tuning |
|
||||
|------|-----|---------|----------|-----------|------|--------|
|
||||
| pfv-tsys1 | i7-4770 Haswell | 8 | 31 | HDD 932 GB | Infrastructure | Done |
|
||||
| pfv-tsys3 | Xeon E3-1535M v5 Skylake | 8 | 31 | **NVMe 477 GB** | Kubernetes | Done |
|
||||
| pfv-tsys4 | Xeon E3-1246 v3 Haswell | 8 | **15** | 6 disks (HDD+SSD) | Storage (NFS+PBS) | Blocked (NIC+RAM) |
|
||||
| pfv-tsys5 | Xeon E5620 Westmere | 8 | **94** | 6 disks (HDD+SSD) | Storage+Preprod | Blocked (cable) |
|
||||
| pfv-tsys6 | 2x Xeon E5530 Nehalem | 16 | 127 | HDD (USB 2.0!) | Kubernetes | Done |
|
||||
| pfv-tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge | 24 | 191 | HDD (USB 2.0!) | Kubernetes | Done |
|
||||
| pfv-tsys9 | i5-10500 Comet Lake | 12 | 24 | SSD 250 GB | Infrastructure | Done |
|
||||
|
||||
**Changes since last audit:**
|
||||
- tsys4 RAM still 15 GB (Friday 64 GB upgrade pending)
|
||||
- tsys5 RAM is 94 GB (was documented as 96 GB)
|
||||
- tsys6 RAM is 127 GB (was documented as 128 GB)
|
||||
- tsys7 RAM is 191 GB (was documented as 192 GB)
|
||||
|
||||
---
|
||||
|
||||
## 2. VM Fleet Inventory (Running VMs Only)
|
||||
|
||||
### tsys1 — 11 running VMs (Infrastructure)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 100 | pfv-bms (HomeAssistant) | 2×1 | 4096 | 32 GB | D2 | tsys4 |
|
||||
| 101 | tsys-ca | 2×1 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 103 | **pfv-netinfra-01** | 2×1 | 2048 | 32 GB | D5 | tsys4 |
|
||||
| 104 | tsys-librenms | 2×1 | 2048 | 50 GB | D2 | tsys4 |
|
||||
| 105 | tsys-proxmox-datacenter | 2×1 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 106 | **pfv-k8s-cnode3** | 2×1 | 4096 | 32 GB | **S3** | **tsys5** |
|
||||
| 108 | **tsys-ucs-01** | 2×2 | 8000 | 32 GB | D2 | tsys4 |
|
||||
| 109 | tailscale-router | 2×1 | 2048 | 25 GB | D2 | tsys4 |
|
||||
| 114 | kali-tsys | 2×1 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 117 | tsys-secure-workbench | 2×1 | 4000 | 32 GB | D2 | tsys4 |
|
||||
| 102 | pfv-k8s-wnode-tsys1 | 4×1 | 4096 | 32 GB | S2 | tsys5 — **STOPPED** |
|
||||
|
||||
### tsys3 — 1 running VM (Kubernetes)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 313 | **pfv-k8s-wnode-tsys3** | 8×1 | 28000 | 32 GB | D5 | tsys4 |
|
||||
|
||||
### tsys4 — 1 running VM (Storage)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 400 | pfv-proxmox-backup-server | 2×1 | 2048 | 32 GB | local-lvm | LOCAL |
|
||||
|
||||
### tsys5 — 16 running VMs (Storage + Preprod)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 509 | **pfv-k8s-wnode-tsys5** | 2×4 | 32000 | 32 GB | D2 | tsys4 |
|
||||
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
|
||||
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
|
||||
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
|
||||
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
|
||||
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
|
||||
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
|
||||
|
||||
### tsys6 — 3 running VMs (Kubernetes)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 100 | **pfv-k8s-wnode-tsys6** | 2×2 | 32000 | 32 GB | D5 | tsys4 |
|
||||
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
|
||||
| 601 | pfv-k8s-cnode4 | 4×1 | 4096 | 32 GB | D2 | tsys4 |
|
||||
|
||||
### tsys7 — 6 running VMs (Kubernetes)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 701 | **pfv-k8s-wnode-tsys7** | 4×1 | 32000 | 32 GB | D5 | tsys4 |
|
||||
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
|
||||
| 703 | rr-middleware | 2×1 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 704 | TCTC | 4×1 | 6000 | 32 GB | D2 | tsys4 |
|
||||
| 705 | **pfv-k8s-cnode2** | 4×1 | 4096 | 32 GB | D2 | tsys4 |
|
||||
| 706 | pfv-k8s-cnode5 | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
|
||||
|
||||
### tsys9 — 6 running VMs (Infrastructure)
|
||||
|
||||
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|
||||
|------|------|--------------|----------|------|---------|-----------|
|
||||
| 901 | tsys-siem | 2×1 | 8000 | 132 GB | D2 | tsys4 |
|
||||
| 902 | **tsys-ucs-02** | 2×2 | 8000 | 50 GB | D5 | tsys4 |
|
||||
| 903 | kali-rd | 2×1 | 2048 | 32 GB | D5 | tsys4 |
|
||||
| 904 | **pfv-netinfra-02** | 2×1 | 4000 | 32 GB | D2 | tsys4 |
|
||||
| 905 | **pfv-k8s-wnode-tsys9** | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
|
||||
| 906 | **pfv-k8s-cnode1** | 2×1 | 4096 | 32 GB | D5 | tsys4 |
|
||||
|
||||
---
|
||||
|
||||
## 3. Kubernetes Node Placement
|
||||
|
||||
### 3.1 Active k3s cluster (cnode1/2/3 — deployed this session)
|
||||
|
||||
| Cnode | VMID | Host | Disk | NFS Server | Quorum risk |
|
||||
|-------|------|------|------|-----------|-------------|
|
||||
| cnode1 | **906** | tsys9 | D5 | **tsys4** | **HIGH** |
|
||||
| cnode2 | **705** | tsys7 | D2 | **tsys4** | **HIGH** |
|
||||
| cnode3 | 106 | tsys1 | S3 | tsys5 | OK |
|
||||
|
||||
**CRITICAL: 2 of 3 active cnodes on tsys4.** tsys4 failure = cnode1 + cnode2 die = 1 of 3 = **QUORUM LOST**.
|
||||
|
||||
**Fix needed:** Move cnode1 or cnode2 to tsys5 storage (S2 or S3). One migration via PDM "Storage Migrate" solves this.
|
||||
|
||||
### 3.2 Inactive cnodes (cnode4/5 — exist but not in k3s cluster)
|
||||
|
||||
| Cnode | VMID | Host | Disk | NFS Server |
|
||||
|-------|------|------|------|-----------|
|
||||
| cnode4 | 601 | tsys6 | D2 | tsys4 |
|
||||
| cnode5 | 706 | tsys7 | S2 | tsys5 |
|
||||
|
||||
### 3.3 Worker nodes (wnodes)
|
||||
|
||||
| Wnode | VMID | Host | Disk | NFS Server | RAM | Status |
|
||||
|-------|------|------|------|-----------|-----|--------|
|
||||
| wnode-tsys1 | 102 | tsys1 | S2 | tsys5 | 4 GB | STOPPED |
|
||||
| wnode-tsys3 | 313 | tsys3 | D5 | tsys4 | 28 GB | Running |
|
||||
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | 32 GB | Running |
|
||||
| wnode-tsys6 | 100 | tsys6 | D5 | tsys4 | 32 GB | Running |
|
||||
| wnode-tsys7 | 701 | tsys7 | D5 | tsys4 | 32 GB | Running |
|
||||
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | 4 GB | Running |
|
||||
|
||||
**Storage: 4 wnodes on tsys4, 2 on tsys5.** One wnode per host achieved.
|
||||
|
||||
---
|
||||
|
||||
## 4. Critical HA Pairs — Storage Redundancy
|
||||
|
||||
### 4.1 netinfra pair — FAILED (both on tsys4)
|
||||
|
||||
| Role | VMID | Host | Disk | NFS Server |
|
||||
|------|------|------|------|-----------|
|
||||
| netinfra-01 | 103 | tsys1 | D5 | **tsys4** |
|
||||
| netinfra-02 | 904 | tsys9 | D2 | **tsys4** |
|
||||
|
||||
**tsys4 failure = DNS/DHCP/NTP goes fully dark.**
|
||||
|
||||
**Fix:** Migrate netinfra-02 (VMID 904) from D2 (tsys4) to S3 (tsys5) via PDM.
|
||||
|
||||
### 4.2 UCS pair — FAILED (both on tsys4)
|
||||
|
||||
| Role | VMID | Host | Disk | NFS Server |
|
||||
|------|------|------|------|-----------|
|
||||
| ucs-01 | 108 | tsys1 | D2 | **tsys4** |
|
||||
| ucs-02 | 902 | tsys9 | D5 | **tsys4** |
|
||||
|
||||
**tsys4 failure = LDAP/AD goes fully dark.**
|
||||
|
||||
**Fix:** Migrate ucs-02 (VMID 902) from D5 (tsys4) to S2 (tsys5) via PDM.
|
||||
|
||||
### 4.3 Corrected placement (after migration)
|
||||
|
||||
| VM | Host | Disk | NFS Server | Failure survival |
|
||||
|----|------|------|-----------|-----------------|
|
||||
| netinfra-01 | tsys1 | D5 | tsys4 | tsys4 dies → netinfra-02 alive on tsys5 |
|
||||
| netinfra-02 | tsys9 | **S3** | **tsys5** | tsys5 dies → netinfra-01 alive on tsys4 |
|
||||
| ucs-01 | tsys1 | D2 | tsys4 | tsys4 dies → ucs-02 alive on tsys5 |
|
||||
| ucs-02 | tsys9 | **S2** | **tsys5** | tsys5 dies → ucs-01 alive on tsys4 |
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage Concentration
|
||||
|
||||
| Storage target | # running VMs | % of fleet |
|
||||
|---------------|--------------|------------|
|
||||
| D2 (tsys4 WDC Red 3TB HDD) | 16 | 36% |
|
||||
| D5 (tsys4 Hitachi 2TB HDD) | 10 | 23% |
|
||||
| local-nonprod (tsys5 local HDD) | 12 | 27% |
|
||||
| S2 (tsys5 Seagate 1TB HDD) | 3 | 7% |
|
||||
| S3 (tsys5 Seagate 1TB HDD) | 1 | 2% |
|
||||
| T5-SSD (tsys5 Samsung SSD) | 1 | 2% |
|
||||
| local-lvm (various hosts) | 2 | 5% |
|
||||
|
||||
**26 of 44 running VMs (59%) store their disks on tsys4 NFS exports.**
|
||||
(Was 68% in the previous audit — improving but still concentrated.)
|
||||
|
||||
---
|
||||
|
||||
## 6. Pre-k8s Buildout Action Items
|
||||
|
||||
These must be done before or during k8s worker node bringup:
|
||||
|
||||
### 6.1 CRITICAL: Migrate HA pairs to separate storage (PDM, 10 min)
|
||||
|
||||
1. `netinfra-02` (VMID 904): D2 → S3 (tsys4 → tsys5)
|
||||
2. `ucs-02` (VMID 902): D5 → S2 (tsys4 → tsys5)
|
||||
|
||||
These are PDM "Storage Migrate" operations — no VM rebuild needed.
|
||||
|
||||
### 6.2 CRITICAL: Fix active cnode quorum (PDM, 5 min)
|
||||
|
||||
Move one active cnode from tsys4 to tsys5 storage:
|
||||
- Best candidate: cnode1 (VMID 906) D5 → S2 (tsys4 → tsys5)
|
||||
- Result: cnode1 on tsys5, cnode2 on tsys4, cnode3 on tsys5
|
||||
- tsys4 failure = cnode2 dies only = 2 of 3 = **quorum OK**
|
||||
|
||||
### 6.3 Join workers to k3s cluster
|
||||
|
||||
Current wnodes exist as VMs but are not joined to the k3s cluster. Need to:
|
||||
1. Install k3s agent on each wnode (using join token from cnode1)
|
||||
2. Configure `--node-ip=<tailscale-ip>` on each
|
||||
3. Label/taint per workload role
|
||||
|
||||
### 6.4 Friday hardware work (still pending)
|
||||
|
||||
- tsys4: PCIe NIC + 64 GB RAM (currently 15 GB)
|
||||
- tsys5: 2nd ethernet cable + NVMe + D3 SSD relocation
|
||||
- tsys2: Rebuild from Win10 to Proxmox
|
||||
|
||||
---
|
||||
|
||||
## 7. Changes Since Previous Audit (2026-07-27)
|
||||
|
||||
| What | Before | After |
|
||||
|------|--------|-------|
|
||||
| cnode1 VMID | 107 (tsys1, D5) | **906** (tsys9, D5) |
|
||||
| cnode2 VMID | 603 (tsys6, D2) | **705** (tsys7, D2) |
|
||||
| cnode3 storage | D2 (tsys4) | **S3** (tsys5) |
|
||||
| cnode5 VMID | 602 (tsys6) | **706** (tsys7, S2 tsys5) |
|
||||
| wnode-tsys1 | not listed | VMID 102 (S2 tsys5, stopped) |
|
||||
| wnode-tsys3 RAM | 20 GB | **28 GB** |
|
||||
| tsys-awx (600) | STOPPED | **Running** |
|
||||
| DellOpenManageEnterprise (500) | not listed | VMID 500 (D7, stopped) |
|
||||
| sectestbed-librenms (5108) | not listed | Running |
|
||||
| k3s cluster | not deployed | **3-node HA live** (cnode1/2/3) |
|
||||
| Console management | manual screen | **ser2net+conman on tsys4** |
|
||||
| PDU management | manual | **powerman on tsys1** |
|
||||
@@ -3,7 +3,7 @@
|
||||
set -uo pipefail
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts"
|
||||
|
||||
echo "=== Uploading fix script to $HOST ==="
|
||||
scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
CHECK_SH="$SCRIPT_DIR/scripts/check.sh"
|
||||
LOG_DIR="$SCRIPT_DIR/returned-logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section.
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT="/home/reachableceo/projects/perfopt/scripts/apply-tunings.sh"
|
||||
SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts/apply-tunings.sh"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
MODE=""
|
||||
EXTRA_FLAGS=""
|
||||
|
||||
+1
-1
@@ -22,7 +22,7 @@ if [ -z "$HOST" ]; then
|
||||
fi
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " finish-host.sh — $HOST (mode: $MODE)"
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
set -uo pipefail
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
set -uo pipefail
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
ALL_HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# iperf-tsys6-tsys7.sh - validate 2Gbps LACP between the two tuned hosts.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
TSYS6="10.100.100.6"
|
||||
|
||||
@@ -12,9 +12,9 @@ SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
|
||||
SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
RECV="pfv-tsys6"; SEND="pfv-tsys7"
|
||||
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
LOCAL_LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/returned-logs/iperf"
|
||||
SCRIPT="lacp-retrans-cause.sh"
|
||||
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
|
||||
LOCAL_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/scripts/${SCRIPT}"
|
||||
REMOTE_SCRIPT="/root/${SCRIPT}"
|
||||
mkdir -p "$LOCAL_LOG_DIR"
|
||||
|
||||
|
||||
@@ -19,9 +19,9 @@ SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
RECV="pfv-tsys6" # receiver (we care most about its RX split)
|
||||
SEND="pfv-tsys7" # sender (control: its TX split)
|
||||
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
LOCAL_LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/returned-logs/iperf"
|
||||
SCRIPT="lacp-rx-distribution.sh"
|
||||
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
|
||||
LOCAL_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/scripts/${SCRIPT}"
|
||||
REMOTE_SCRIPT="/root/${SCRIPT}"
|
||||
|
||||
mkdir -p "$LOCAL_LOG_DIR"
|
||||
|
||||
@@ -51,7 +51,7 @@ esac
|
||||
|
||||
BAUD=9600
|
||||
HOST=root@pfv-tsys4
|
||||
HERE=/home/reachableceo/projects/perfopt
|
||||
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
|
||||
LOCAL_DRIVER=$HERE/scripts/sw-capture.py
|
||||
LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
|
||||
LOCAL_LOG=$HERE/returned-logs/$SWITCH.log
|
||||
|
||||
Reference in New Issue
Block a user