Files

T

mrcharles 9c2882ea00 fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes

Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

2026-07-31 00:12:04 -05:00

10 KiB

Raw Blame History

Storage Architecture

Complete storage reference for the PFVCluster. Covers physical disks, NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle VM placement. Data gathered 2026-07-30 via live host probes.

Companion docs: NETWORK-TOPOLOGY.md, HOST-REFERENCE.md, VM-INVENTORY.md. Findings/analysis: AUDIT-2026-07-30.md.

1. Storage tier definitions

Tier	Definition	Typical speed	Reliability	Use case
Tier 1	NVMe or SSD on internal SATA/SAS/PCIe	200-3000 MB/s	High	OS drives, high-IOPS workloads, build scratch
Tier 2	HDD on internal SATA/SAS	80-150 MB/s	Moderate	Production VM OS drives, general-purpose storage
Tier 3	HDD/SSD on USB adapter / "toaster"	30-60 MB/s	Low (USB)	Bulk, backup, scratch only. Never for production OS drives.

Policy (from CAPACITY-GOALS.md §6): Local disk storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS (tsys4/tsys5) where redundancy is achieved by spreading across physical disks and HA pairs. Local storage is for ephemeral data where loss on node failure is acceptable.

2. Physical disk inventory

tsys4 -- NFS storage server + PBS (7 disks)

Device	Model	Size	RPM	Interface	Mount	NFS ID	Tier	SMART	Used
sda	Hitachi HDS723020BLA642	2.0 TB	7200	SATA	/mnt/albert	(not exported)	Tier 2	PASSED, 0 reallocated	0%
sdb	HITACHI HUA723020ALA640	2.0 TB	7200	SATA	/mnt/tsys4/D5	D5	Tier 2	PASSED, 0 reallocated	13% (220 GB)
sdc	WDC WD30EFRX (Red)	3.0 TB	5400	SATA	/mnt/tsys4/D2	D2	Tier 2	PASSED, 0 reallocated	7% (170 GB)
sdd	WDC WD10EZRX	1.0 TB	--	SATA	(pve boot/swap/lvm)	--	Tier 2	PASSED, 1 reallocated (sdf has 1)	OS disk
sde	ST500LM021 (Seagate)	500 GB	7200	USB	/mnt/tsys4/D3	D3	Tier 3	PASSED	0%
sdf	WDC WD10EARS	1.0 TB	--	USB	/mnt/tsys4/D1	D1	Tier 3	PASSED, 1 reallocated sector	0%
sdg	WDC WD50NDZW (easystore)	5.0 TB	5400	USB	/mnt/backup	(not exported)	Tier 3	PASSED	0%

Changes from previous audit:

D3 changed: was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
/mnt/albert added: Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
/mnt/backup added: WD 5TB USB easystore. Bulk backup only.
D7 removed: Previously held DellOpenManageEnterprise. Now gone.

tsys5 -- NFS storage server + sandboxes (7 disks)

Device	Model	Size	RPM	Interface	Mount	NFS/Local ID	Tier	SMART	Used
sda	ST1000VN002 (IronWolf)	1.0 TB	5900	SATA	(not mounted)	was S3	Tier 2	PASSED	--
sdb	SK hynix SC300B	512 GB	SSD	SATA	(not mounted)	--	Tier 1	PASSED	NEW, not configured
sdc	HITACHI HUS724020ALA640	2.0 TB	7200	SAS	(pve boot/swap/lvm)	--	Tier 2	PASSED	OS disk
sdd	Samsung SSD 860 PRO	256 GB	SSD	SAS	/mnt/pfv-tsys5/ssd	`ssd` / `SSD` dir (local)	Tier 1	PASSED	93% full!
sde	ST1000VN002 (IronWolf)	1.0 TB	5900	SAS	/mnt/pfv-tsys5/S1	S1 + `local-nonprod`	Tier 2	PASSED	11% (96 GB)
sdf	ST1000VN002 (IronWolf)	1.0 TB	5900	SAS	/mnt/pfv-tsys5/S2	S2	Tier 2	PASSED	4% (32 GB)
nvme0n1	CT500P1SSD8 (Crucial)	500 GB	NVMe	PCIe	(not mounted)	--	Tier 1	PASSED, 0 integrity errors, 34K hrs	NEW, not configured

Changes from previous audit:

NVMe added: Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
SK hynix SSD relocated: Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
S3 unmounted: sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
S4 gone: Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken dir: S4 entry.
S1/S2 drives changed: Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
SSD nearly full: Samsung 860 PRO at 93% (206/234 GB) -- needs attention.
Boot disk changed: HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).

3. NFS export → Proxmox storage ID map

graph LR
    subgraph "tsys4 -- NFS Server"
        D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
        D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
        D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
        D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
    end

    subgraph "tsys5 -- NFS Server"
        S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
        S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
        S3stale["S3: (stale export)<br>disk not mounted"]
    end

    subgraph "All hosts -- NFS Clients"
        MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
    end

    D2 -->|NFS v4.2| MNT
    D5 -->|NFS v4.2| MNT
    D1 -->|NFS v4.2| MNT
    D3 -->|NFS v4.2| MNT
    S1 -->|NFS v4.2| MNT
    S2 -->|NFS v4.2| MNT
    S3stale -.->|broken| MNT

NFS export details

Server	Export path	NFS ID	Protocol	Options (client-side)
tsys4	/mnt/tsys4/D1	D1	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4	/mnt/tsys4/D2	D2	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4	/mnt/tsys4/D3	D3	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4	/mnt/tsys4/D5	D5	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5	/mnt/pfv-tsys5/S1	S1	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5	/mnt/pfv-tsys5/S2	S2	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5	/mnt/pfv-tsys5/S3	S3	NFSv4.2	stale (disk not mounted on server)
tsys5	/mnt/pfv-tsys5/ssd	T5-SSD	NFSv4.2	nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5	/mnt/pfv-tsys5/S4	S4	--	broken (disk gone, storage.cfg entry stale)

Note: tsys5 is the only host where NFS client mounts (D2/D3/D5 from tsys4) are missing nconnect=4. All other client hosts have nconnect on all mounts.

4. Per-spindle VM placement (running VMs only)

Shows every running VM's primary disk and the physical spindle it lives on. This is the IOPS distribution map -- the key to storage balancing.

graph TB
    subgraph "tsys4 spindles"
        D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
        D5["D5 -- Hitachi 2TB<br>4 VMs"]
        D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
        D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
        ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
    end

    subgraph "tsys5 spindles"
        S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
        S2["S2 -- IronWolf 1TB<br>5 VMs"]
        SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
        NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
        HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
    end

    D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
    D5 --- D5vms["106 313 600(d0) 903"]
    S1 --- S1vms["702 hfnoc-uisp"]
    S2 --- S2vms["103 703 705 902 905"]
    SSD --- SSDvms["5111 ultix-streaming"]

Detailed per-spindle load

Spindle	Server	NFS ID	Tier	Running VMs	VMIDs	Notes
D2 (WDC Red 3TB)	tsys4	D2	Tier 2	13	100,101,104,105,108,114,117,509,600,702,704,901,904	Worst IOPS hotspot
D5 (Hitachi 2TB)	tsys4	D5	Tier 2	4	106,313,600(disk0),903	Moderate
S1 (IronWolf 1TB)	tsys5	S1 + local-nonprod	Tier 2	1 (NFS) + 0 (local, 12 stopped)	702	Low; 12 stopped sandboxes on local-nonprod
S2 (IronWolf 1TB)	tsys5	S2	Tier 2	5	103,703,705,902,905	Moderate
ssd (Samsung 860 PRO)	tsys5	SSD dir	Tier 1	1	5111	93% full
D1 (WD 1TB USB)	tsys4	D1	Tier 3	0	--	Empty
D3 (Seagate 500GB USB)	tsys4	D3	Tier 3	0	--	Empty
/mnt/albert (Hitachi 2TB)	tsys4	(none)	Tier 2	0	--	Not in Proxmox
NVMe (CT500P1 500GB)	tsys5	(none)	Tier 1	0	--	Not configured
SK hynix SSD (512GB)	tsys5	(none)	Tier 1	0	--	Not configured

Local-only VM disks (not on NFS)

Host	Storage ID	Disk	VMs	Notes
tsys4	local-lvm	sdd (1TB internal)	400 (PBS)	Proxmox boot/OS disk
tsys5	local-lvm	sdc (2TB internal)	5102,5103,5104,5112	Mixed: stopped sandboxes + ultix-offstage
tsys5	local-nonprod (=S1)	sde (IronWolf 1TB)	5101,5105,5106-5109,515,53100-53103,6000,51010-51014	All STOPPED sandboxes. Same physical disk as S1 NFS.
tsys5	SSD dir	sdd (Samsung 860 PRO)	5111	ultix-streaming. 93% full.

5. Storage capacity summary

Spindle	Size	Used	Available	% Used
D2 (WDC Red 3TB)	2.7 TB	170 GB	2.4 TB	7%
D5 (Hitachi 2TB)	1.8 TB	220 GB	1.5 TB	13%
D1 (WD 1TB USB)	916 GB	2 MB	870 GB	0%
D3 (Seagate 500GB USB)	458 GB	2 MB	435 GB	0%
/mnt/albert (Hitachi 2TB)	1.8 TB	2 MB	1.7 TB	0%
/mnt/backup (WD 5TB USB)	4.6 TB	2 MB	4.3 TB	0%
S1 (IronWolf 1TB)	916 GB	96 GB	774 GB	11%
S2 (IronWolf 1TB)	916 GB	32 GB	838 GB	4%
SSD (Samsung 860 PRO)	234 GB	206 GB	16 GB	93%
NVMe (CT500P1)	466 GB	--	--	Not configured
SK hynix SSD	477 GB	--	--	Not configured

Total Tier 2 available (NFS production): ~5.5 TB unused across D2/D5/S1/S2 Total Tier 3 available (USB): ~6 TB unused across D1/D3/backup Total Tier 1 available (unconfigured): ~940 GB across NVMe + SK hynix SSD

10 KiB Raw Blame History