PFVCluster/TODO.md

TODO.md — Pending User Actions

Date: 2026-07-27 Items needing user input or physical action.

---

1. tsys2 Windows hardware inventory (run on the Windows host)

pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather hardware data so the architecture plan can account for it.

Option A: PowerShell (recommended — single command, copy-paste output)

Open PowerShell as Administrator and run:

# Full hardware inventory in one shot
Write-Output "=== COMPUTER ==="
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List

Write-Output "`n=== CPU ==="
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List

Write-Output "`n=== MEMORY STICKS ==="
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize

Write-Output "`n=== DISKS ==="
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize

Write-Output "`n=== DISK PARTITIONS ==="
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize

Write-Output "`n=== NETWORK ADAPTERS ==="
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize

Write-Output "`n=== GPU(s) ==="
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List

Write-Output "`n=== PCIe SLOTS ==="
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize

Write-Output "`n=== USB DEVICES (storage + network only) ==="
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize

Copy the full output into a file (e.g., tsys2-hardware.txt) or paste it directly into the chat.

Option B: Command Prompt (cmd.exe) fallbacks

If PowerShell is unavailable for some reason, these cmd commands give a subset:

:: Computer model and serial
wmic computersystem get manufacturer,model
wmic bios get serialnumber

:: CPU
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed

:: RAM (total)
wmic computersystem get totalphysicalmemory

:: RAM sticks (per-slot detail)
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator

:: Disks
wmic diskdrive get model,size,interfacetype,mediatype

:: Network adapters
wmic nic where netenabled=true get name,speed,macaddress

:: GPU
wmic path win32_videocontroller get name,adapterram,driverversion

What I'm looking for

• Disk inventory: Are there any SSDs/NVMe available locally? (Determines whether wnode-tsys2 can use local storage like the other wnodes.)
• Network adapters: How many onboard NICs? Model? (Determines whether tsys2 needs a USB dongle for storage network like tsys4/9, or has a real onboard NIC available.)
• RAM layout: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects memory bandwidth for HPC workloads — dual-channel matters.)
• GPU detail: Confirm the Quadro M1200 model and VRAM for passthrough planning.
• PCIe slots: Is there a free PCIe slot for adding a NIC or HBA?
• Service tag confirmation: GH1XZG2 (already on file from spreadsheet).

STATUS: Collected 2026-07-27

Disk inventory — RESOLVED:

• Disk 0: Samsung SSD 960 PRO 512 GB NVMe (best local storage in fleet)
• Disk 1: Samsung SSD 850 EVO 1 TB SATA SSD
• Both SSDs, no spinning rust. 1.5 TB total local SSD.

Network adapters — RESOLVED (concerning):

• StorageNetwork: ASIX USB to Gigabit Ethernet (dongle, D4-81-D7-3E-0D-5E)
• Ethernet: Realtek USB GbE Family Controller (also USB, 18-FD-CB-00-D2-CA)
• Wi-Fi: Intel 8265 (disconnected)
• Both wired NICs are USB-attached. Same anti-pattern as tsys4/9. Unavoidable on this laptop form factor — no onboard PCIe NIC available.

GPU — RESOLVED:

• Intel HD Graphics 630 (integrated, 1 GB)
• NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)

PCIe slots — RESOLVED:

• Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not user-accessible full PCIe. Cannot add a PCIe NIC. NVMe slot occupied by 960 PRO.

RAM — PARTIAL:

• Total 32 GB confirmed (34,097,573,888 bytes).
• Per-stick detail failed to run (PowerShell line-break split Format-T + able). Re-run the command below if bandwidth planning needs stick-level detail:

  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  

Service tag — CONFIRMED: GH1XZG2 (Precision 5520).

---

2. Friday maintenance window (physical hardware)

tsys4 — install PCIe NIC + add RAM

1. Power down tsys4 (graceful shutdown via Proxmox UI or shutdown -h now).
2. Install the PCIe NIC (Intel i350-T2 or similar 1 GbE dual-port).
3. Add RAM: 16 GB → 64 GB DDR3 ECC.
4. Power on, then update /etc/network/interfaces to replace enx8cae4ccda926 (USB dongle) with the new PCIe NIC device name.
5. Reboot to activate new NIC and NFS nconnect.
6. Run validate-fixes.sh pfv-tsys4 to confirm.

tsys5 — plug storage cable + install NVMe + relocate D3 SSD

1. Plug the second ethernet cable into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: cat /proc/net/bonding/bond0 — look for "Number of ports: 2" and a real partner MAC (not all zeros).
3. Apply bond hash fix (same as tsys6/7):

   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   

4. Relocate D3 SSD from tsys4 USB to tsys5 SAS port:
   • Power down tsys4
   • Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   • Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   • On tsys5: mount as /mnt/pfv-tsys5/D3, add to /etc/exports
   • Update /etc/pve/storage.cfg cluster-wide: repoint D3 server from pfv-tsys4-nfs-stor to pfv-tsys5-nfs-stor, update export path
   • Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially empty, so minimal migration)
5. Install the PCI NVMe drive (uses a PCI slot, not a SATA/SAS port).
6. Format NVMe as local directory storage (see TODO section 3 below).
7. Reboot tsys5 to activate NFS nconnect.
8. Run validate-fixes.sh pfv-tsys5 to confirm.

---

3. tsys5 NVMe format/mount decision (after Friday install)

Recommendation: local-only, not NFS-exported. Format as Proxmox directory storage so it shows up as a VM image target in the Proxmox UI.

After the NVMe is physically installed and visible in Proxmox:

1. Identify the device: lsblk or ls /dev/nvme*

2. Format and add to Proxmox:

   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template
   
   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   

3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.

---

4. Post-hardware validation (run after Friday work)

1. Re-run iperf matrix: ./iperf-full-matrix.sh
2. Validate tsys4 and tsys5: ./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5
3. Update PROJECT.md with post-hardware iperf numbers.

---

5. UCS storage migration to spinning disk (do today)

UCS (Univention Corporate Server / open-source AD) does not need SSD. Both UCS VMs should stay on spinning disk (HDD) and be split across storage servers for redundancy.

VM	Current	Target	Action
ucs-01 (108)	D2 (tsys4 HDD)	D2 (tsys4 HDD) -- no change	Already correct
ucs-02 (902)	D5 (tsys4 HDD)	S2 (tsys5 HDD)	Move for cross-server redundancy

To migrate ucs-02 to S2:

1. Shut down VM 902 on tsys9
2. Copy disk: cp /mnt/pve/D5/902/vm-902-disk-0.qcow2 /mnt/pve/S2/902/
3. Update VM 902 config: change scsi0 from D5:902/... to S2:902/...
4. Start VM 902, verify it boots and LDAP/AD services are healthy
5. Remove old disk from D5

Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only ucs-02 needs to move -- it should be on a different storage server than ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.

---

6. Open questions for next session

• Are the hosts a Proxmox cluster (pvecm status) or standalone installs? Determines whether live migration is available.
• What k8s distribution is in use? (k3s, kubeadm, RKE2?)
• Container runtime? (containerd, cri-o?)
• Is there a local container image registry mirror?
• What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
• HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
• What uses tsys5's SDR + parallel port before planning tsys5 role changes?
• tsys3 thermal state (laptop in rack for years) — check sensors.