PFVCluster/TODO.md

# TODO.md — Pending User Actions

**Date:** 2026-07-27
**Items needing user input or physical action.**

---

## 1. tsys2 Windows hardware inventory (run on the Windows host)

pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
hardware data so the architecture plan can account for it.

### Option A: PowerShell (recommended — single command, copy-paste output)

Open **PowerShell as Administrator** and run:

```powershell
# Full hardware inventory in one shot
Write-Output "=== COMPUTER ==="
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List

Write-Output "`n=== CPU ==="
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List

Write-Output "`n=== MEMORY STICKS ==="
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize

Write-Output "`n=== DISKS ==="
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize

Write-Output "`n=== DISK PARTITIONS ==="
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize

Write-Output "`n=== NETWORK ADAPTERS ==="
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize

Write-Output "`n=== GPU(s) ==="
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List

Write-Output "`n=== PCIe SLOTS ==="
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize

Write-Output "`n=== USB DEVICES (storage + network only) ==="
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
```

Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
directly into the chat.

### Option B: Command Prompt (cmd.exe) fallbacks

If PowerShell is unavailable for some reason, these cmd commands give a
subset:

```cmd
:: Computer model and serial
wmic computersystem get manufacturer,model
wmic bios get serialnumber

:: CPU
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed

:: RAM (total)
wmic computersystem get totalphysicalmemory

:: RAM sticks (per-slot detail)
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator

:: Disks
wmic diskdrive get model,size,interfacetype,mediatype

:: Network adapters
wmic nic where netenabled=true get name,speed,macaddress

:: GPU
wmic path win32_videocontroller get name,adapterram,driverversion
```

### What I'm looking for

- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
  whether wnode-tsys2 can use local storage like the other wnodes.)
- **Network adapters**: How many onboard NICs? Model? (Determines whether
  tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
  onboard NIC available.)
- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
  memory bandwidth for HPC workloads — dual-channel matters.)
- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
  planning.
- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).

### STATUS: Collected 2026-07-27

**Disk inventory — RESOLVED:**
- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
- Both SSDs, no spinning rust. 1.5 TB total local SSD.

**Network adapters — RESOLVED (concerning):**
- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
- Wi-Fi: Intel 8265 (disconnected)
- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
  Unavoidable on this laptop form factor — no onboard PCIe NIC available.

**GPU — RESOLVED:**
- Intel HD Graphics 630 (integrated, 1 GB)
- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)

**PCIe slots — RESOLVED:**
- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
  user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
  by 960 PRO.

**RAM — PARTIAL:**
- Total 32 GB confirmed (34,097,573,888 bytes).
- Per-stick detail failed to run (PowerShell line-break split
  `Format-T` + `able`). Re-run the command below if bandwidth planning
  needs stick-level detail:
  ```powershell
  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  ```

**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).

---

## 2. Friday maintenance window (physical hardware)

### tsys4 — install PCIe NIC + add RAM

1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
4. Power on, then update `/etc/network/interfaces` to replace
   `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
5. Reboot to activate new NIC and NFS nconnect.
6. Run `validate-fixes.sh pfv-tsys4` to confirm.

### tsys5 — plug storage cable + install NVMe + relocate D3 SSD

1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
   "Number of ports: 2" and a real partner MAC (not all zeros).
3. Apply bond hash fix (same as tsys6/7):
   ```bash
   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   ```
4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
   - Power down tsys4
   - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
   - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
     `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
   - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
     empty, so minimal migration)
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
6. Format NVMe as local directory storage (see TODO section 3 below).
7. Reboot tsys5 to activate NFS nconnect.
8. Run `validate-fixes.sh pfv-tsys5` to confirm.

---

## 3. tsys5 NVMe format/mount decision (after Friday install)

**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
storage so it shows up as a VM image target in the Proxmox UI.

After the NVMe is physically installed and visible in Proxmox:

1. Identify the device: `lsblk` or `ls /dev/nvme*`
2. Format and add to Proxmox:
   ```bash
   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template

   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   ```

3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.

---

## 4. Post-hardware validation (run after Friday work)

1. Re-run iperf matrix: `./iperf-full-matrix.sh`
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
3. Update PROJECT.md with post-hardware iperf numbers.

---

## 5. UCS storage migration to spinning disk (do today)

UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
UCS VMs should stay on spinning disk (HDD) and be split across storage
servers for redundancy.

| VM | Current | Target | Action |
|----|---------|--------|--------|
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |

**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
   (both are NFS exports visible to tsys9, so this is a storage-only migration)
3. Verify VM 902 boots and LDAP/AD services are healthy after migration

Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
ucs-02 needs to move -- it should be on a different storage server than
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.

---

## 6. Open questions for next session

- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
  Determines whether live migration is available.
- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
- Container runtime? (containerd, cri-o?)
- Is there a local container image registry mirror?
- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
- tsys3 thermal state (laptop in rack for years) — check `sensors`.