Files

T

mrcharles 9c2882ea00 fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes

Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

2026-07-31 00:12:04 -05:00

11 KiB

Raw Blame History

VM Inventory

Every VM in the cluster, with host placement, disk location, CPU/RAM allocation, and optimization status. Data gathered 2026-07-30.

Companion docs: NETWORK-TOPOLOGY.md, STORAGE-ARCHITECTURE.md, HOST-REFERENCE.md.

1. VM placement diagram

graph TB
    subgraph tsys1["tsys1 -- Infra (57% RAM)"]
        VM100["100 pfv-bms<br>2c/4G D2"]
        VM101["101 tsys-ca<br>2c/2G D2"]
        VM103["103 netinfra-01<br>2c/4G S2"]
        VM104["104 librenms<br>2c/2G D2"]
        VM105["105 proxmox-dc<br>2c/2G D2"]
        VM106["106 cnode3<br>4c/4G D5"]
        VM108["108 ucs-01<br>2c/8G D2"]
        VM114["114 kali-tsys<br>2c/2G D2"]
        VM117["117 workbench<br>2c/4G D2"]
    end

    subgraph tsys3["tsys3 -- Compute (10% RAM)"]
        VM313["313 wnode-tsys3<br>8c/28G D5"]
    end

    subgraph tsys4["tsys4 -- Storage (15% RAM)"]
        VM400["400 PBS<br>2c/2G local-lvm"]
    end

    subgraph tsys5["tsys5 -- Storage+Sandbox (15% RAM)"]
        VM509["509 wnode-tsys5<br>2c/32G D2"]
        VM515["515 hfnoc-uisp-preprod<br>2c/2G local-nonprod"]
        VM5111["5111 ultix-streaming<br>4c/9G SSD"]
        VM5112["5112 ultix-offstage<br>4c/6G local-lvm"]
    end

    subgraph tsys6["tsys6 -- RackRental (12% RAM)"]
        VM100b["100 wnode-tsys6<br>2c/64G S4"]
        VM600["600 tsys-awx<br>2c/12G D5"]
    end

    subgraph tsys7["tsys7 -- RackRental (7% RAM)"]
        VM701["701 wnode-tsys7<br>4c/32G S3"]
        VM702["702 hfnoc-uisp<br>2c/8G S1"]
        VM703["703 rr-middleware<br>2c/2G S2"]
        VM704["704 TCTC<br>4c/6G D2"]
        VM705["705 cnode2<br>4c/4G S2"]
    end

    subgraph tsys9["tsys9 -- Infra+Compute (69% RAM)"]
        VM901["901 tsys-siem<br>2c/8G D2"]
        VM902["902 ucs-02<br>2c/8G S2"]
        VM903["903 kali-rd<br>2c/2G D5"]
        VM904["904 netinfra-02<br>2c/4G D2"]
        VM905["905 wnode-tsys9<br>4c/4G S2"]
        VM906["906 cnode1<br>2c/4G S3"]
    end

2. Complete running VM inventory (28 running VMs)

Infrastructure (netinfra, UCS, CA, LibreNMS, PDM)

VMID	Name	Host	vCPU	CPU type	RAM	Disk store	Spindle	Disk type	Cache	iothread	qga	Notes
100	pfv-bms (HomeAssistant)	tsys1	2	host	4G	D2 (raw)	D2	raw	--	--	✓	discard=on, ssd=1. HA for power/temp monitoring.
101	tsys-ca	tsys1	2	v2-AES	2G	D2 (virtio)	D2	qcow2	--	✓	--	no nested-virt flag. Certificate Authority.
103	pfv-netinfra-01	tsys1	2	v2-AES	4G	S2	S2	qcow2	--	✓	--	DNS primary (Technitium + Pi-hole).
104	tsys-librenms	tsys1	2	host	2G	D2	D2	qcow2	--	✓	✓	SNMP monitoring for switches.
105	tsys-proxmox-datacenter	tsys1	2	host	2G	D2	D2	qcow2	--	✓	✓	PDM (Proxmox Datacenter Manager).
108	tsys-ucs-01	tsys1	2	host	8G	D2	D2	qcow2	writethrough	✓	✓	Univention Corporate Server half 1.
902	tsys-ucs-02	tsys9	2	host	8G	S2	S2	qcow2	--	✓	✓	UCS half 2 (HA pair with 108).
904	pfv-netinfra-02	tsys9	2	v2-AES	4G	D2	D2	qcow2	--	✓	--	DNS secondary. HA pair with 103.

k8s control plane (cnode1/2/3)

VMID	Name	Host	vCPU	CPU type	RAM	Disk store	Spindle	Disk type	Cache	iothread	qga	Notes
106	pfv-k8s-cnode3	tsys1	4	host	4G	D5	D5	qcow2	writethrough	✓	✓	etcd member.
705	pfv-k8s-cnode2	tsys7	4	host	4G	S2	S2	qcow2	writethrough	✓	✓	etcd member.
906	pfv-k8s-cnode1	tsys9	2	host	4G	S3	S3	qcow2	--	✓	--	etcd member.

k8s workers (wnodes)

VMID	Name	Host	vCPU	CPU type	RAM	Disk store	Spindle	Disk type	Cache	iothread	qga	Notes
313	pfv-k8s-wnode-tsys3	tsys3	8	host	28G	D5	D5	qcow2	writethrough	✓	✓	Compute worker.
509	pfv-k8s-wnode-tsys5	tsys5	2	host	32G	D2 (virtio)	D2	qcow2	--	✓	--	Undersized vCPU (2).
100	pfv-k8s-wnode-tsys6	tsys6	2	host	64G	S4	S4	qcow2	--	✓	--	Undersized vCPU (2), high RAM.
701	pfv-k8s-wnode-tsys7	tsys7	4	host	32G	S3 (virtio)	S3	qcow2	--	✓	--	RackRental worker.
905	pfv-k8s-wnode-tsys9	tsys9	4	v2-AES	4G	S2	S2	qcow2	--	✓	✓	Target: grow to 6c/8G.

Application VMs

VMID	Name	Host	vCPU	CPU type	RAM	Disk store	Spindle	Disk type	Cache	iothread	qga	Notes
114	kali-tsys	tsys1	2	host	2G	D2	D2	qcow2	writethrough	✓	✓	Kali Linux pentest.
117	tsys-secure-workbench	tsys1	2	host	4G	D2	D2	qcow2	writethrough	✓	✓	Secure workbench.
400	pfv-proxmox-backup-server	tsys4	2	--	2G	local-lvm	(local)	raw	--	--	--	PBS -- all VM backups.
515	hfnoc-uisp-preprod	tsys5	2	host	2G	local-nonprod	S1	qcow2	--	✓	--	Preprod. Locked to tsys5.
5111	ultix-streaming	tsys5	4	--	9G	SSD dir	SSD	qcow2	--	--	--	288GB on Samsung 860 PRO (93% full).
5112	ultix-offstage	tsys5	4	--	6G	local-lvm	(local)	raw	--	--	--	288GB. Locked to tsys5.
600	tsys-awx	tsys6	2	host	12G	D5	D5	qcow2	writethrough	✓	✓	Ansible AWX.
702	hfnoc-uisp	tsys7	2	host	8G	S1	S1	qcow2	--	✓	✓	100GB disk. UISP network monitoring.
703	rr-middleware	tsys7	2	host	2G	S2	S2	qcow2	writethrough	✓	✓	RackRental middleware.
704	TCTC	tsys7	4	host	6G	D2 (virtio)	D2	qcow2	--	✓	--
901	tsys-siem	tsys9	2	host	8G	D2	D2	qcow2	writethrough	✓	✓	132GB disk. SIEM.
903	kali-rd	tsys9	2	host	2G	D5	D5	qcow2	writethrough	✓	✓	Kali R&D.

3. Optimization findings

3.1 Cache mode audit

cache=writethrough forces synchronous writes to the NFS server on every write operation. For NFS-backed qcow2, this is unnecessarily slow -- writeback or none is recommended (qcow2 already provides journaling integrity).

Cache mode	Count	VMs
writethrough (slow)	10	106, 108, 114, 117, 313, 600, 703, 705, 901, 903
none/default (optimal)	18	all others

Fix: Change cache=writethrough → cache=writeback or remove (defaults to writeback for qcow2). Requires VM stop/start.

3.2 CPU type audit

cpu: host passes the host's full CPU instruction set to the VM (best performance). x86-64-v2-AES is a conservative baseline (safe for migration but lacks some host instructions).

CPU type	Count	VMs
host	21	most production VMs
x86-64-v2-AES	5	101, 103, 509, 904, 905
not set	2	400, 5111, 5112

Note: Mixed CPU types prevent live migration between hosts with different CPU generations. K8s cnodes/wnodes should ideally match within each tier.

3.3 Disk controller audit

Controller	Count	Notes
scsi0 (virtio-scsi)	22	Optimal for Proxmox
virtio0	4	101, 509, 701, 704 -- also good
raw (no virtio)	2	100 (pfv-bms), 500 (stopped)

All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks.

3.4 Network audit

All running VMs use virtio-net. Firewall enabled on most (firewall=1). No legacy e1000/rtl8139 adapters.

3.5 Guest agent (qga) audit

Status	Count	VMs without qga
enabled	20	--
not set	8	101, 103, 509, 100(tsys6), 701, 704, 904, 906

Fix: Enable qm set <vmid> --agent 1 on the 8 VMs missing it. No VM stop/start required (takes effect on next guest reboot, but the setting itself is immediate).

3.6 k8s wnode sizing

VMID	Name	vCPU	RAM	Tier target	Issue
509	wnode-tsys5	2	32G	compute	Undersized vCPU (2 for a compute worker)
100	wnode-tsys6	2	64G	rackrental	Undersized vCPU (2 for 64G RAM)
701	wnode-tsys7	4	32G	rackrental	OK for idle labs
905	wnode-tsys9	4	4G	compute	Target: grow to 6c/8G after siem migrates off
313	wnode-tsys3	8	28G	compute	OK (best compute worker)

4. Stopped VMs (not running, for reference)

Sectestbed / sandboxes (all on tsys5, all stopped)

VMID	Name	RAM	Disk	Notes
5100	sectestbed-template	4G	local-lvm	Base template
5101	sectestbed-siem	10G	local-nonprod (S1)	132G disk
5102	sectestbed-proxmox-pve	4G	local-lvm
5103	sectestbed-proxmox-datacenter	4G	local-lvm
5104	sectestbed-proxmox-pbs	4G	local-lvm
5105	sectestbed-awx	4G	local-nonprod (S1)	288G disk
5106	sectestbed-k8s-cnode	4G	local-nonprod (S1)
5107	sectestbed-k8s-wnode	4G	local-nonprod (S1)
5108	sectestbed-librenms	4G	local-nonprod (S1)
5109	sectestbed-netinfra	4G	local-nonprod (S1)
51010	sectestbed-tctc	4G	local-nonprod (S1)
51011	sectestbed-cloudron	4G	local-nonprod (S1)
51012	sectestbed-hfnoc	4G	local-nonprod (S1)
51013	sectestbed-rancherplatform	4G	local-nonprod (S1)
51014	sectestbed-proxmox-mailgw	4G	local-nonprod (S1)
53100	tsys-preprod-awx	9G	local-nonprod (S1)	160G disk
53101	tsys-preprod-siem	12G	local-nonprod (S1)
53102	tsys-preprod-rancherplatform	8G	local-nonprod (S1)
53103	tsys-preprod-proxmoxmailgw	4G	local-nonprod (S1)
6000	sectestbed-sandbox	4G	local-nonprod (S1)	Has 3 state snapshots

Other stopped VMs

VMID	Name	Host	RAM	Disk	Notes
102	pfv-k8s-wnode-tsys1	tsys1	4G	(no disk)	Stopped, no boot disk
500	DellOpenManageEnterprise	tsys5	20G	D7 (gone!)	Would fail -- disk export removed
5500	RestoreTemplate	tsys5	2G	D2	Template
1000	KNELTemplate	tsys1	2G	D2	Base template
3000	KNELTemplate	tsys3	2G	D2	Base template copy
4000	KNELTemplate	tsys4	2G	(local)	Base template copy

11 KiB Raw Blame History