PFVCluster/TODO.md

TODO.md — Pending User Actions

Date: 2026-07-27 Items needing user input or physical action.

---

1. tsys2 Windows hardware inventory (run on the Windows host)

pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather hardware data so the architecture plan can account for it.

Option A: PowerShell (recommended — single command, copy-paste output)

Open PowerShell as Administrator and run:

# Full hardware inventory in one shot
Write-Output "=== COMPUTER ==="
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List

Write-Output "`n=== CPU ==="
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List

Write-Output "`n=== MEMORY STICKS ==="
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize

Write-Output "`n=== DISKS ==="
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize

Write-Output "`n=== DISK PARTITIONS ==="
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize

Write-Output "`n=== NETWORK ADAPTERS ==="
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize

Write-Output "`n=== GPU(s) ==="
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List

Write-Output "`n=== PCIe SLOTS ==="
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize

Write-Output "`n=== USB DEVICES (storage + network only) ==="
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize

Copy the full output into a file (e.g., tsys2-hardware.txt) or paste it directly into the chat.

Option B: Command Prompt (cmd.exe) fallbacks

If PowerShell is unavailable for some reason, these cmd commands give a subset:

:: Computer model and serial
wmic computersystem get manufacturer,model
wmic bios get serialnumber

:: CPU
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed

:: RAM (total)
wmic computersystem get totalphysicalmemory

:: RAM sticks (per-slot detail)
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator

:: Disks
wmic diskdrive get model,size,interfacetype,mediatype

:: Network adapters
wmic nic where netenabled=true get name,speed,macaddress

:: GPU
wmic path win32_videocontroller get name,adapterram,driverversion

What I'm looking for

• Disk inventory: Are there any SSDs/NVMe available locally? (Determines whether wnode-tsys2 can use local storage like the other wnodes.)
• Network adapters: How many onboard NICs? Model? (Determines whether tsys2 needs a USB dongle for storage network like tsys4/9, or has a real onboard NIC available.)
• RAM layout: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects memory bandwidth for HPC workloads — dual-channel matters.)
• GPU detail: Confirm the Quadro M1200 model and VRAM for passthrough planning.
• PCIe slots: Is there a free PCIe slot for adding a NIC or HBA?
• Service tag confirmation: GH1XZG2 (already on file from spreadsheet).

---

2. Friday maintenance window (physical hardware)

tsys4 — install PCIe NIC + add RAM

1. Power down tsys4 (graceful shutdown via Proxmox UI or shutdown -h now).
2. Install the PCIe NIC (Intel i350-T2 or similar 1 GbE dual-port).
3. Add RAM: 16 GB → 64 GB DDR3 ECC.
4. Power on, then update /etc/network/interfaces to replace enx8cae4ccda926 (USB dongle) with the new PCIe NIC device name.
5. Reboot to activate new NIC and NFS nconnect.
6. Run validate-fixes.sh pfv-tsys4 to confirm.

tsys5 — plug storage cable + install NVMe

1. Plug the second ethernet cable into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: cat /proc/net/bonding/bond0 — look for "Number of ports: 2" and a real partner MAC (not all zeros).
3. Apply bond hash fix (same as tsys6/7):

   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   

4. Install the PCI NVMe drive.
5. Format NVMe as local directory storage (see TODO section 3 below).
6. Reboot tsys5 to activate NFS nconnect.
7. Run validate-fixes.sh pfv-tsys5 to confirm.

---

3. tsys5 NVMe format/mount decision (after Friday install)

Recommendation: local-only, not NFS-exported. Format as Proxmox directory storage so it shows up as a VM image target in the Proxmox UI.

After the NVMe is physically installed and visible in Proxmox:

1. Identify the device: lsblk or ls /dev/nvme*

2. Format and add to Proxmox:

   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template
   
   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   

3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.

---

4. Post-hardware validation (run after Friday work)

1. Re-run iperf matrix: ./iperf-full-matrix.sh
2. Validate tsys4 and tsys5: ./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5
3. Update PROJECT.md with post-hardware iperf numbers.

---

5. Open questions for next session

• Are the hosts a Proxmox cluster (pvecm status) or standalone installs? Determines whether live migration is available.
• What k8s distribution is in use? (k3s, kubeadm, RKE2?)
• Container runtime? (containerd, cri-o?)
• Is there a local container image registry mirror?
• What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
• HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
• What uses tsys5's SDR + parallel port before planning tsys5 role changes?
• tsys3 thermal state (laptop in rack for years) — check sensors.