# Storage Architecture

> **Complete storage reference for the PFVCluster.** Covers physical disks,
> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
> VM placement. Data gathered 2026-07-30 via live host probes.
>
> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).

---

## 1. Storage tier definitions

| Tier | Definition | Typical speed | Reliability | Use case |
|------|-----------|---------------|-------------|----------|
| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch |
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage |
| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. |

**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk
storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
and HA pairs. Local storage is for ephemeral data where loss on node failure
is acceptable.

---

## 2. Physical disk inventory

### tsys4 -- NFS storage server + PBS (7 disks)

| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|--------|------|-------|------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% |
| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% |

**Changes from previous audit:**
- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only.
- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone.

### tsys5 -- NFS storage server + sandboxes (7 disks)

| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|-------------|------|-------|------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk |
| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |

**Changes from previous audit:**
- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry.
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention.
- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).

---

## 3. NFS export → Proxmox storage ID map

```mermaid
graph LR
    subgraph "tsys4 -- NFS Server"
        D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
        D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
        D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
        D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
    end

    subgraph "tsys5 -- NFS Server"
        S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
        S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
        S3stale["S3: (stale export)<br>disk not mounted"]
    end

    subgraph "All hosts -- NFS Clients"
        MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
    end

    D2 -->|NFS v4.2| MNT
    D5 -->|NFS v4.2| MNT
    D1 -->|NFS v4.2| MNT
    D3 -->|NFS v4.2| MNT
    S1 -->|NFS v4.2| MNT
    S2 -->|NFS v4.2| MNT
    S3stale -.->|broken| MNT
```

### NFS export details

| Server | Export path | NFS ID | Protocol | Options (client-side) |
|--------|-------------|--------|----------|----------------------|
| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) |
| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) |

> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from
> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on
> all mounts.

---

## 4. Per-spindle VM placement (running VMs only)

> Shows every running VM's primary disk and the physical spindle it lives on.
> This is the IOPS distribution map -- the key to storage balancing.

```mermaid
graph TB
    subgraph "tsys4 spindles"
        D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
        D5["D5 -- Hitachi 2TB<br>4 VMs"]
        D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
        D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
        ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
    end

    subgraph "tsys5 spindles"
        S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
        S2["S2 -- IronWolf 1TB<br>5 VMs"]
        SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
        NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
        HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
    end

    D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
    D5 --- D5vms["106 313 600(d0) 903"]
    S1 --- S1vms["702 hfnoc-uisp"]
    S2 --- S2vms["103 703 705 902 905"]
    SSD --- SSDvms["5111 ultix-streaming"]
```

### Detailed per-spindle load

| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes |
|---------|--------|--------|------|-------------|-------|-------|
| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** |
| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate |
| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |

### Local-only VM disks (not on NFS)

| Host | Storage ID | Disk | VMs | Notes |
|------|-----------|------|-----|-------|
| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk |
| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage |
| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. |
| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. |

---

## 5. Storage capacity summary

| Spindle | Size | Used | Available | % Used |
|---------|------|------|-----------|--------|
| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% |
| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% |
| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% |
| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% |
| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% |
| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% |
| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** |
| NVMe (CT500P1) | 466 GB | -- | -- | Not configured |
| SK hynix SSD | 477 GB | -- | -- | Not configured |

**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2
**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup
**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD
