PFVCluster/TODO.md

# TODO.md — Pending User Actions

**Date:** 2026-07-27
**Items needing user input or physical action.**

---

## 1. tsys2 Windows hardware inventory (run on the Windows host)

pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
hardware data so the architecture plan can account for it.

### Option A: PowerShell (recommended — single command, copy-paste output)

Open **PowerShell as Administrator** and run:

```powershell
# Full hardware inventory in one shot
Write-Output "=== COMPUTER ==="
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List

Write-Output "`n=== CPU ==="
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List

Write-Output "`n=== MEMORY STICKS ==="
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize

Write-Output "`n=== DISKS ==="
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize

Write-Output "`n=== DISK PARTITIONS ==="
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize

Write-Output "`n=== NETWORK ADAPTERS ==="
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize

Write-Output "`n=== GPU(s) ==="
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List

Write-Output "`n=== PCIe SLOTS ==="
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize

Write-Output "`n=== USB DEVICES (storage + network only) ==="
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
```

Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
directly into the chat.

### Option B: Command Prompt (cmd.exe) fallbacks

If PowerShell is unavailable for some reason, these cmd commands give a
subset:

```cmd
:: Computer model and serial
wmic computersystem get manufacturer,model
wmic bios get serialnumber

:: CPU
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed

:: RAM (total)
wmic computersystem get totalphysicalmemory

:: RAM sticks (per-slot detail)
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator

:: Disks
wmic diskdrive get model,size,interfacetype,mediatype

:: Network adapters
wmic nic where netenabled=true get name,speed,macaddress

:: GPU
wmic path win32_videocontroller get name,adapterram,driverversion
```

### What I'm looking for

- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
  whether wnode-tsys2 can use local storage like the other wnodes.)
- **Network adapters**: How many onboard NICs? Model? (Determines whether
  tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
  onboard NIC available.)
- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
  memory bandwidth for HPC workloads — dual-channel matters.)
- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
  planning.
- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).

### STATUS: Collected 2026-07-27

**Disk inventory — RESOLVED:**
- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
- Both SSDs, no spinning rust. 1.5 TB total local SSD.

**Network adapters — RESOLVED (concerning):**
- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
- Wi-Fi: Intel 8265 (disconnected)
- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
  Unavoidable on this laptop form factor — no onboard PCIe NIC available.

**GPU — RESOLVED:**
- Intel HD Graphics 630 (integrated, 1 GB)
- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)

**PCIe slots — RESOLVED:**
- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
  user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
  by 960 PRO.

**RAM — PARTIAL:**
- Total 32 GB confirmed (34,097,573,888 bytes).
- Per-stick detail failed to run (PowerShell line-break split
  `Format-T` + `able`). Re-run the command below if bandwidth planning
  needs stick-level detail:
  ```powershell
  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  ```

**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).

---

## 2. Friday maintenance window (physical hardware)

### tsys4 — install PCIe NIC + add RAM

1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
4. Power on, then update `/etc/network/interfaces` to replace
   `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
5. Reboot to activate new NIC and NFS nconnect.
6. Run `validate-fixes.sh pfv-tsys4` to confirm.

### tsys5 — plug storage cable + install NVMe + relocate D3 SSD

1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
   "Number of ports: 2" and a real partner MAC (not all zeros).
3. Apply bond hash fix (same as tsys6/7):
   ```bash
   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   ```
4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
   - Power down tsys4
   - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
   - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
     `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
   - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
     empty, so minimal migration)
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
6. Format NVMe as local directory storage (see TODO section 3 below).
7. Reboot tsys5 to activate NFS nconnect.
8. Run `validate-fixes.sh pfv-tsys5` to confirm.

---

## 3. tsys5 NVMe format/mount decision (after Friday install)

**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
storage so it shows up as a VM image target in the Proxmox UI.

After the NVMe is physically installed and visible in Proxmox:

1. Identify the device: `lsblk` or `ls /dev/nvme*`
2. Format and add to Proxmox:
   ```bash
   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template

   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   ```

3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.

---

## 4. Post-hardware validation (run after Friday work)

1. Re-run iperf matrix: `./iperf-full-matrix.sh`
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
3. Update PROJECT.md with post-hardware iperf numbers.

---

## 5. UCS storage migration to spinning disk (do today)

UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
UCS VMs should stay on spinning disk (HDD) and be split across storage
servers for redundancy.

| VM | Current | Target | Action |
|----|---------|--------|--------|
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |

**To migrate ucs-02 to S2:**
1. Shut down VM 902 on tsys9
2. Copy disk: `cp /mnt/pve/D5/902/vm-902-disk-0.qcow2 /mnt/pve/S2/902/`
3. Update VM 902 config: change `scsi0` from `D5:902/...` to `S2:902/...`
4. Start VM 902, verify it boots and LDAP/AD services are healthy
5. Remove old disk from D5

Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
ucs-02 needs to move -- it should be on a different storage server than
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.

---

## 6. Open questions for next session

- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
  Determines whether live migration is available.
- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
- Container runtime? (containerd, cri-o?)
- Is there a local container image registry mirror?
- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
- tsys3 thermal state (laptop in rack for years) — check `sensors`.