KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
290245349f67beee0a157a47380876f2ada76242
PFVCluster/proxmox/docs/STORAGE-ARCHITECTURE.md
T
mrcharles 0a8865de38 docs: full documentation suite — network, storage, hosts, VMs 
Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:

- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
  maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
  analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
  inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
  per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
  network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
  disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
  qga, disk controller, k8s wnode sizing), stopped VM reference

Updated docmap.md with all four new entries.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:10:38 -05:00

10 KiB
Raw Blame History

Storage Architecture

Complete storage reference for the PFVCluster. Covers physical disks, NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle VM placement. Data gathered 2026-07-30 via live host probes.

Companion docs: NETWORK-TOPOLOGY.md, HOST-REFERENCE.md, VM-INVENTORY.md. Findings/analysis: AUDIT-2026-07-30.md.

1. Storage tier definitions

Tier Definition Typical speed Reliability Use case
Tier 1 NVMe or SSD on internal SATA/SAS/PCIe 200-3000 MB/s High OS drives, high-IOPS workloads, build scratch
Tier 2 HDD on internal SATA/SAS 80-150 MB/s Moderate Production VM OS drives, general-purpose storage
Tier 3 HDD/SSD on USB adapter / "toaster" 30-60 MB/s Low (USB) Bulk, backup, scratch only. Never for production OS drives.

Policy (from CAPACITY-GOALS.md §6): Local disk storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS (tsys4/tsys5) where redundancy is achieved by spreading across physical disks and HA pairs. Local storage is for ephemeral data where loss on node failure is acceptable.

2. Physical disk inventory

tsys4 — NFS storage server + PBS (7 disks)

Device Model Size RPM Interface Mount NFS ID Tier SMART Used
sda Hitachi HDS723020BLA642 2.0 TB 7200 SATA /mnt/albert (not exported) Tier 2 PASSED, 0 reallocated 0%
sdb HITACHI HUA723020ALA640 2.0 TB 7200 SATA /mnt/tsys4/D5 D5 Tier 2 PASSED, 0 reallocated 13% (220 GB)
sdc WDC WD30EFRX (Red) 3.0 TB 5400 SATA /mnt/tsys4/D2 D2 Tier 2 PASSED, 0 reallocated 7% (170 GB)
sdd WDC WD10EZRX 1.0 TB — SATA (pve boot/swap/lvm) — Tier 2 PASSED, 1 reallocated (sdf has 1) OS disk
sde ST500LM021 (Seagate) 500 GB 7200 USB /mnt/tsys4/D3 D3 Tier 3 PASSED 0%
sdf WDC WD10EARS 1.0 TB — USB /mnt/tsys4/D1 D1 Tier 3 PASSED, 1 reallocated sector 0%
sdg WDC WD50NDZW (easystore) 5.0 TB 5400 USB /mnt/backup (not exported) Tier 3 PASSED 0%

Changes from previous audit:

  • D3 changed: was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
  • /mnt/albert added: Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
  • /mnt/backup added: WD 5TB USB easystore. Bulk backup only.
  • D7 removed: Previously held DellOpenManageEnterprise. Now gone.

tsys5 — NFS storage server + sandboxes (7 disks)

Device Model Size RPM Interface Mount NFS/Local ID Tier SMART Used
sda ST1000VN002 (IronWolf) 1.0 TB 5900 SATA (not mounted) was S3 Tier 2 PASSED —
sdb SK hynix SC300B 512 GB SSD SATA (not mounted) — Tier 1 PASSED NEW, not configured
sdc HITACHI HUS724020ALA640 2.0 TB 7200 SAS (pve boot/swap/lvm) — Tier 2 PASSED OS disk
sdd Samsung SSD 860 PRO 256 GB SSD SAS /mnt/pfv-tsys5/ssd ssd / SSD dir (local) Tier 1 PASSED 93% full!
sde ST1000VN002 (IronWolf) 1.0 TB 5900 SAS /mnt/pfv-tsys5/S1 S1 + local-nonprod Tier 2 PASSED 11% (96 GB)
sdf ST1000VN002 (IronWolf) 1.0 TB 5900 SAS /mnt/pfv-tsys5/S2 S2 Tier 2 PASSED 4% (32 GB)
nvme0n1 CT500P1SSD8 (Crucial) 500 GB NVMe PCIe (not mounted) — Tier 1 PASSED, 0 integrity errors, 34K hrs NEW, not configured

Changes from previous audit:

  • NVMe added: Crucial CT500P1 500GB. True NVMe — highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
  • SK hynix SSD relocated: Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
  • S3 unmounted: sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
  • S4 gone: Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken dir: S4 entry.
  • S1/S2 drives changed: Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
  • SSD nearly full: Samsung 860 PRO at 93% (206/234 GB) — needs attention.
  • Boot disk changed: HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).

3. NFS export → Proxmox storage ID map

graph LR
    subgraph "tsys4 — NFS Server"
        D2["D2: /mnt/tsys4/D2<br/>WDC Red 3TB<br/>Tier 2 · 7% used"]
        D5["D5: /mnt/tsys4/D5<br/>Hitachi 2TB<br/>Tier 2 · 13% used"]
        D1["D1: /mnt/tsys4/D1<br/>WD 1TB USB<br/>Tier 3 · 0% used"]
        D3["D3: /mnt/tsys4/D3<br/>Seagate 500GB USB<br/>Tier 3 · 0% used"]
    end

    subgraph "tsys5 — NFS Server"
        S1["S1: /mnt/pfv-tsys5/S1<br/>IronWolf 1TB<br/>Tier 2 · 11% used"]
        S2["S2: /mnt/pfv-tsys5/S2<br/>IronWolf 1TB<br/>Tier 2 · 4% used"]
        S3stale["S3: (stale export)<br/>disk not mounted"]
    end

    subgraph "All hosts — NFS Clients"
        MNT["/mnt/pve/D1 D2 D3 D5<br/>/mnt/pve/S1 S2 S3 S4<br/>/mnt/pve/T5-SSD"]
    end

    D2 -->|NFS v4.2| MNT
    D5 -->|NFS v4.2| MNT
    D1 -->|NFS v4.2| MNT
    D3 -->|NFS v4.2| MNT
    S1 -->|NFS v4.2| MNT
    S2 -->|NFS v4.2| MNT
    S3stale -.->|broken| MNT

NFS export details

Server Export path NFS ID Protocol Options (client-side)
tsys4 /mnt/tsys4/D1 D1 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D2 D2 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D3 D3 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys4 /mnt/tsys4/D5 D5 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S1 S1 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S2 S2 NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S3 S3 NFSv4.2 stale (disk not mounted on server)
tsys5 /mnt/pfv-tsys5/ssd T5-SSD NFSv4.2 nconnect=4,noatime,rsize=1048576,wsize=1048576,hard
tsys5 /mnt/pfv-tsys5/S4 S4 — broken (disk gone, storage.cfg entry stale)

Note: tsys5 is the only host where NFS client mounts (D2/D3/D5 from tsys4) are missing nconnect=4. All other client hosts have nconnect on all mounts.

4. Per-spindle VM placement (running VMs only)

Shows every running VM's primary disk and the physical spindle it lives on. This is the IOPS distribution map — the key to storage balancing.

graph TB
    subgraph "tsys4 spindles"
        D2["D2 — WDC Red 3TB<br/>★★★ 13 VMs — WORST HOTSPOT"]
        D5["D5 — Hitachi 2TB<br/>4 VMs"]
        D1["D1 — WD 1TB USB<br/>0 VMs (empty)"]
        D3["D3 — Seagate 500GB USB<br/>0 VMs (empty)"]
        ALB["/mnt/albert — Hitachi 2TB<br/>0 VMs (not in Proxmox)"]
    end

    subgraph "tsys5 spindles"
        S1["S1 — IronWolf 1TB<br/>1 NFS VM + 12 local stopped"]
        S2["S2 — IronWolf 1TB<br/>5 VMs"]
        SSD["SSD — Samsung 860 PRO<br/>1 VM (93% full!)"]
        NVME["NVMe — Crucial CT500P1<br/>NOT CONFIGURED"]
        HYNIX["SK hynix SSD 512GB<br/>NOT CONFIGURED"]
    end

    D2 --- D2vms["100 101 104 105 108<br/>114 117 509 600 702<br/>704 901 904"]
    D5 --- D5vms["106 313 600(d0) 903"]
    S1 --- S1vms["702 hfnoc-uisp"]
    S2 --- S2vms["103 703 705 902 905"]
    SSD --- SSDvms["5111 ultix-streaming"]

Detailed per-spindle load

Spindle Server NFS ID Tier Running VMs VMIDs Notes
D2 (WDC Red 3TB) tsys4 D2 Tier 2 13 100,101,104,105,108,114,117,509,600,702,704,901,904 Worst IOPS hotspot
D5 (Hitachi 2TB) tsys4 D5 Tier 2 4 106,313,600(disk0),903 Moderate
S1 (IronWolf 1TB) tsys5 S1 + local-nonprod Tier 2 1 (NFS) + 0 (local, 12 stopped) 702 Low; 12 stopped sandboxes on local-nonprod
S2 (IronWolf 1TB) tsys5 S2 Tier 2 5 103,703,705,902,905 Moderate
ssd (Samsung 860 PRO) tsys5 SSD dir Tier 1 1 5111 93% full
D1 (WD 1TB USB) tsys4 D1 Tier 3 0 — Empty
D3 (Seagate 500GB USB) tsys4 D3 Tier 3 0 — Empty
/mnt/albert (Hitachi 2TB) tsys4 (none) Tier 2 0 — Not in Proxmox
NVMe (CT500P1 500GB) tsys5 (none) Tier 1 0 — Not configured
SK hynix SSD (512GB) tsys5 (none) Tier 1 0 — Not configured

Local-only VM disks (not on NFS)

Host Storage ID Disk VMs Notes
tsys4 local-lvm sdd (1TB internal) 400 (PBS) Proxmox boot/OS disk
tsys5 local-lvm sdc (2TB internal) 5102,5103,5104,5112 Mixed: stopped sandboxes + ultix-offstage
tsys5 local-nonprod (=S1) sde (IronWolf 1TB) 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 All STOPPED sandboxes. Same physical disk as S1 NFS.
tsys5 SSD dir sdd (Samsung 860 PRO) 5111 ultix-streaming. 93% full.

5. Storage capacity summary

Spindle Size Used Available % Used
D2 (WDC Red 3TB) 2.7 TB 170 GB 2.4 TB 7%
D5 (Hitachi 2TB) 1.8 TB 220 GB 1.5 TB 13%
D1 (WD 1TB USB) 916 GB 2 MB 870 GB 0%
D3 (Seagate 500GB USB) 458 GB 2 MB 435 GB 0%
/mnt/albert (Hitachi 2TB) 1.8 TB 2 MB 1.7 TB 0%
/mnt/backup (WD 5TB USB) 4.6 TB 2 MB 4.3 TB 0%
S1 (IronWolf 1TB) 916 GB 96 GB 774 GB 11%
S2 (IronWolf 1TB) 916 GB 32 GB 838 GB 4%
SSD (Samsung 860 PRO) 234 GB 206 GB 16 GB 93%
NVMe (CT500P1) 466 GB — — Not configured
SK hynix SSD 477 GB — — Not configured

Total Tier 2 available (NFS production): ~5.5 TB unused across D2/D5/S1/S2 Total Tier 3 available (USB): ~6 TB unused across D1/D3/backup Total Tier 1 available (unconfigured): ~940 GB across NVMe + SK hynix SSD

Reference in New Issue View Git Blame Copy Permalink