# Storage Architecture > **Complete storage reference for the PFVCluster.** Covers physical disks, > NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle > VM placement. Data gathered 2026-07-30 via live host probes. > > Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), > [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). > Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). --- ## 1. Storage tier definitions | Tier | Definition | Typical speed | Reliability | Use case | |------|-----------|---------------|-------------|----------| | **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch | | **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage | | **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. | **Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS (tsys4/tsys5) where redundancy is achieved by spreading across physical disks and HA pairs. Local storage is for ephemeral data where loss on node failure is acceptable. --- ## 2. Physical disk inventory ### tsys4 -- NFS storage server + PBS (7 disks) | Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used | |--------|-------|------|-----|-----------|-------|--------|------|-------|------| | sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% | | sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) | | sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) | | sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk | | sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% | | sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% | | sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% | **Changes from previous audit:** - **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD. - **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2. - **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only. - **D7 removed:** Previously held DellOpenManageEnterprise. Now gone. ### tsys5 -- NFS storage server + sandboxes (7 disks) | Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used | |--------|-------|------|-----|-----------|-------|-------------|------|-------|------| | sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- | | sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** | | sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk | | sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** | | sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) | | sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) | | **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** | **Changes from previous audit:** - **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg. - **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured. - **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports. - **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry. - **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm). - **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention. - **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model). --- ## 3. NFS export → Proxmox storage ID map ```mermaid graph LR subgraph "tsys4 -- NFS Server" D2["D2: /mnt/tsys4/D2
WDC Red 3TB
Tier 2 | 7% used"] D5["D5: /mnt/tsys4/D5
Hitachi 2TB
Tier 2 | 13% used"] D1["D1: /mnt/tsys4/D1
WD 1TB USB
Tier 3 | 0% used"] D3["D3: /mnt/tsys4/D3
Seagate 500GB USB
Tier 3 | 0% used"] end subgraph "tsys5 -- NFS Server" S1["S1: /mnt/pfv-tsys5/S1
IronWolf 1TB
Tier 2 | 11% used"] S2["S2: /mnt/pfv-tsys5/S2
IronWolf 1TB
Tier 2 | 4% used"] S3stale["S3: (stale export)
disk not mounted"] end subgraph "All hosts -- NFS Clients" MNT["/mnt/pve/D1 D2 D3 D5
/mnt/pve/S1 S2 S3 S4
/mnt/pve/T5-SSD"] end D2 -->|NFS v4.2| MNT D5 -->|NFS v4.2| MNT D1 -->|NFS v4.2| MNT D3 -->|NFS v4.2| MNT S1 -->|NFS v4.2| MNT S2 -->|NFS v4.2| MNT S3stale -.->|broken| MNT ``` ### NFS export details | Server | Export path | NFS ID | Protocol | Options (client-side) | |--------|-------------|--------|----------|----------------------| | tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) | | tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) | > **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from > tsys4) are **missing nconnect=4**. All other client hosts have nconnect on > all mounts. --- ## 4. Per-spindle VM placement (running VMs only) > Shows every running VM's primary disk and the physical spindle it lives on. > This is the IOPS distribution map -- the key to storage balancing. ```mermaid graph TB subgraph "tsys4 spindles" D2["D2 -- WDC Red 3TB
*** 13 VMs -- WORST HOTSPOT"] D5["D5 -- Hitachi 2TB
4 VMs"] D1["D1 -- WD 1TB USB
0 VMs (empty)"] D3["D3 -- Seagate 500GB USB
0 VMs (empty)"] ALB["/mnt/albert -- Hitachi 2TB
0 VMs (not in Proxmox)"] end subgraph "tsys5 spindles" S1["S1 -- IronWolf 1TB
1 NFS VM + 12 local stopped"] S2["S2 -- IronWolf 1TB
5 VMs"] SSD["SSD -- Samsung 860 PRO
1 VM (93% full!)"] NVME["NVMe -- Crucial CT500P1
NOT CONFIGURED"] HYNIX["SK hynix SSD 512GB
NOT CONFIGURED"] end D2 --- D2vms["100 101 104 105 108
114 117 509 600 702
704 901 904"] D5 --- D5vms["106 313 600(d0) 903"] S1 --- S1vms["702 hfnoc-uisp"] S2 --- S2vms["103 703 705 902 905"] SSD --- SSDvms["5111 ultix-streaming"] ``` ### Detailed per-spindle load | Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes | |---------|--------|--------|------|-------------|-------|-------| | **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** | | D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate | | S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod | | S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate | | ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** | | D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty | | D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty | | /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox | | NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | | SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | ### Local-only VM disks (not on NFS) | Host | Storage ID | Disk | VMs | Notes | |------|-----------|------|-----|-------| | tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk | | tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage | | tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. | | tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. | --- ## 5. Storage capacity summary | Spindle | Size | Used | Available | % Used | |---------|------|------|-----------|--------| | D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% | | D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% | | D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% | | D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% | | /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% | | /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% | | S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% | | S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% | | SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** | | NVMe (CT500P1) | 466 GB | -- | -- | Not configured | | SK hynix SSD | 477 GB | -- | -- | Not configured | **Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2 **Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup **Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD