KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/STORAGE-ARCHITECTURE.md
+8 -194
View File
@@ -1,196 +1,10 @@
# Storage Architecture
# proxmox/docs/STORAGE-ARCHITECTURE.md
> **Complete storage reference for the PFVCluster.** Covers physical disks,
> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
> VM placement. Data gathered 2026-07-30 via live host probes.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
---
## 1. Storage tier definitions
| Tier | Definition | Typical speed | Reliability | Use case |
|------|-----------|---------------|-------------|----------|
| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch |
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage |
| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. |
**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk
storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
and HA pairs. Local storage is for ephemeral data where loss on node failure
is acceptable.
---
## 2. Physical disk inventory
### tsys4 -- NFS storage server + PBS (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|--------|------|-------|------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% |
| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% |
**Changes from previous audit:**
- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only.
- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone.
### tsys5 -- NFS storage server + sandboxes (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|-------------|------|-------|------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk |
| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |
**Changes from previous audit:**
- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry.
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention.
- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).
---
## 3. NFS export → Proxmox storage ID map
```mermaid
graph LR
subgraph "tsys4 -- NFS Server"
D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
end
subgraph "tsys5 -- NFS Server"
S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
S3stale["S3: (stale export)<br>disk not mounted"]
end
subgraph "All hosts -- NFS Clients"
MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
end
D2 -->|NFS v4.2| MNT
D5 -->|NFS v4.2| MNT
D1 -->|NFS v4.2| MNT
D3 -->|NFS v4.2| MNT
S1 -->|NFS v4.2| MNT
S2 -->|NFS v4.2| MNT
S3stale -.->|broken| MNT
```
### NFS export details
| Server | Export path | NFS ID | Protocol | Options (client-side) |
|--------|-------------|--------|----------|----------------------|
| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) |
| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) |
> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from
> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on
> all mounts.
---
## 4. Per-spindle VM placement (running VMs only)
> Shows every running VM's primary disk and the physical spindle it lives on.
> This is the IOPS distribution map -- the key to storage balancing.
```mermaid
graph TB
subgraph "tsys4 spindles"
D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
D5["D5 -- Hitachi 2TB<br>4 VMs"]
D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
end
subgraph "tsys5 spindles"
S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
S2["S2 -- IronWolf 1TB<br>5 VMs"]
SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
end
D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
D5 --- D5vms["106 313 600(d0) 903"]
S1 --- S1vms["702 hfnoc-uisp"]
S2 --- S2vms["103 703 705 902 905"]
SSD --- SSDvms["5111 ultix-streaming"]
```
### Detailed per-spindle load
| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes |
|---------|--------|--------|------|-------------|-------|-------|
| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** |
| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate |
| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
### Local-only VM disks (not on NFS)
| Host | Storage ID | Disk | VMs | Notes |
|------|-----------|------|-----|-------|
| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk |
| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage |
| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. |
| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. |
---
## 5. Storage capacity summary
| Spindle | Size | Used | Available | % Used |
|---------|------|------|-----------|--------|
| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% |
| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% |
| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% |
| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% |
| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% |
| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% |
| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** |
| NVMe (CT500P1) | 466 GB | -- | -- | Not configured |
| SK hynix SSD | 477 GB | -- | -- | Not configured |
**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2
**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup
**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD
> **Storage reference: tiers, disk inventory, NFS exports, VM placement**
>
> **Read it here:** https://community.turnsys.com/t/300
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*