PFVCluster/docs/TODO.md

TODO.md — Pending User Actions

Date: 2026-07-27 Items needing user input or physical action.

---

1. tsys2 Windows hardware inventory (run on the Windows host)

pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather hardware data so the architecture plan can account for it.

Option A: PowerShell (recommended — single command, copy-paste output)

Open PowerShell as Administrator and run:

# Full hardware inventory in one shot
Write-Output "=== COMPUTER ==="
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List

Write-Output "`n=== CPU ==="
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List

Write-Output "`n=== MEMORY STICKS ==="
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize

Write-Output "`n=== DISKS ==="
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize

Write-Output "`n=== DISK PARTITIONS ==="
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize

Write-Output "`n=== NETWORK ADAPTERS ==="
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize

Write-Output "`n=== GPU(s) ==="
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List

Write-Output "`n=== PCIe SLOTS ==="
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize

Write-Output "`n=== USB DEVICES (storage + network only) ==="
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize

Copy the full output into a file (e.g., tsys2-hardware.txt) or paste it directly into the chat.

Option B: Command Prompt (cmd.exe) fallbacks

If PowerShell is unavailable for some reason, these cmd commands give a subset:

:: Computer model and serial
wmic computersystem get manufacturer,model
wmic bios get serialnumber

:: CPU
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed

:: RAM (total)
wmic computersystem get totalphysicalmemory

:: RAM sticks (per-slot detail)
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator

:: Disks
wmic diskdrive get model,size,interfacetype,mediatype

:: Network adapters
wmic nic where netenabled=true get name,speed,macaddress

:: GPU
wmic path win32_videocontroller get name,adapterram,driverversion

What I'm looking for

• Disk inventory: Are there any SSDs/NVMe available locally? (Determines whether wnode-tsys2 can use local storage like the other wnodes.)
• Network adapters: How many onboard NICs? Model? (Determines whether tsys2 needs a USB dongle for storage network like tsys4/9, or has a real onboard NIC available.)
• RAM layout: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects memory bandwidth for HPC workloads — dual-channel matters.)
• GPU detail: Confirm the Quadro M1200 model and VRAM for passthrough planning.
• PCIe slots: Is there a free PCIe slot for adding a NIC or HBA?
• Service tag confirmation: GH1XZG2 (already on file from spreadsheet).

STATUS: Collected 2026-07-27

Disk inventory — RESOLVED:

• Disk 0: Samsung SSD 960 PRO 512 GB NVMe (best local storage in fleet)
• Disk 1: Samsung SSD 850 EVO 1 TB SATA SSD
• Both SSDs, no spinning rust. 1.5 TB total local SSD.

Network adapters — RESOLVED (concerning):

• StorageNetwork: ASIX USB to Gigabit Ethernet (dongle, D4-81-D7-3E-0D-5E)
• Ethernet: Realtek USB GbE Family Controller (also USB, 18-FD-CB-00-D2-CA)
• Wi-Fi: Intel 8265 (disconnected)
• Both wired NICs are USB-attached. Same anti-pattern as tsys4/9. Unavoidable on this laptop form factor — no onboard PCIe NIC available.

GPU — RESOLVED:

• Intel HD Graphics 630 (integrated, 1 GB)
• NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)

PCIe slots — RESOLVED:

• Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not user-accessible full PCIe. Cannot add a PCIe NIC. NVMe slot occupied by 960 PRO.

RAM — PARTIAL:

• Total 32 GB confirmed (34,097,573,888 bytes).
• Per-stick detail failed to run (PowerShell line-break split Format-T + able). Re-run the command below if bandwidth planning needs stick-level detail:

  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  

Service tag — CONFIRMED: GH1XZG2 (Precision 5520).

---

2. Friday maintenance window (physical hardware)

tsys4 — install PCIe NIC + add RAM

1. Power down tsys4 (graceful shutdown via Proxmox UI or shutdown -h now).
2. Install the PCIe NIC (Intel i350-T2 or similar 1 GbE dual-port).
3. Add RAM: 16 GB → 64 GB DDR3 ECC.
4. Power on, then update /etc/network/interfaces to replace enx8cae4ccda926 (USB dongle) with the new PCIe NIC device name.
5. Reboot to activate new NIC and NFS nconnect.
6. Run validate-fixes.sh pfv-tsys4 to confirm.

tsys5 — plug storage cable + install NVMe + relocate D3 SSD

1. Plug the second ethernet cable into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: cat /proc/net/bonding/bond0 — look for "Number of ports: 2" and a real partner MAC (not all zeros).
3. Apply bond hash fix (same as tsys6/7):

   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   

4. Relocate D3 SSD from tsys4 USB to tsys5 SAS port:
   • Power down tsys4
   • Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   • Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   • On tsys5: mount as /mnt/pfv-tsys5/D3, add to /etc/exports
   • Update /etc/pve/storage.cfg cluster-wide: repoint D3 server from pfv-tsys4-nfs-stor to pfv-tsys5-nfs-stor, update export path
   • Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially empty, so minimal migration)
5. Install the PCI NVMe drive (uses a PCI slot, not a SATA/SAS port).
6. Format NVMe as local directory storage (see TODO section 3 below).
7. Reboot tsys5 to activate NFS nconnect.
8. Run validate-fixes.sh pfv-tsys5 to confirm.

---

3. tsys5 NVMe format/mount decision (after Friday install)

Recommendation: local-only, not NFS-exported. Format as Proxmox directory storage so it shows up as a VM image target in the Proxmox UI.

After the NVMe is physically installed and visible in Proxmox:

1. Identify the device: lsblk or ls /dev/nvme*

2. Format and add to Proxmox:

   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template
   
   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   

3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.

---

4. Post-hardware validation (run after Friday work)

1. Re-run iperf matrix: ./iperf-full-matrix.sh
2. Validate tsys4 and tsys5: ./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5
3. Update PROJECT.md with post-hardware iperf numbers.

---

5. UCS storage migration to spinning disk (do today)

UCS (Univention Corporate Server / open-source AD) does not need SSD. Both UCS VMs should stay on spinning disk (HDD) and be split across storage servers for redundancy.

VM	Current	Target	Action
ucs-01 (108)	D2 (tsys4 HDD)	D2 (tsys4 HDD) -- no change	Already correct
ucs-02 (902)	D5 (tsys4 HDD)	S2 (tsys5 HDD)	Move for cross-server redundancy

To migrate ucs-02 to S2 (use PDM/Proxmox UI):

1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2 (both are NFS exports visible to tsys9, so this is a storage-only migration)
3. Verify VM 902 boots and LDAP/AD services are healthy after migration

Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only ucs-02 needs to move -- it should be on a different storage server than ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.

---

6. Open questions for next session

• Are the hosts a Proxmox cluster (pvecm status) or standalone installs? Determines whether live migration is available.
• What k8s distribution is in use? (k3s, kubeadm, RKE2?)
• Container runtime? (containerd, cri-o?)
• Is there a local container image registry mirror?
• What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
• HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
• What uses tsys5's SDR + parallel port before planning tsys5 role changes?
• tsys3 thermal state (laptop in rack for years) — check sensors.