From a1beb6cd3e91c5ad96a112855abe2d28e1343efd Mon Sep 17 00:00:00 2001 From: reachableceo Date: Fri, 31 Jul 2026 08:17:30 -0500 Subject: [PATCH] fix(tuning): persistent NFS tuning fleet-wide + apply-tunings.sh bug fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fleet-wide changes (live, documented for audit/AWX): Persistent tuning via systemd oneshot (all 7 hosts): - New pfv-nfs-tuning.service runs AFTER tuned.service to override 16MB TCP buffer caps with 128MB for high-BDP NFS - New 99-pfv-nfs.conf sysctl fragment (128MB rmem/wmem, 250K backlog) - Deployed to: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 - Survives reboot — no more tuned clobbering our sysctl values tsys5 fixes: - VM 509 stopped, NFS D2/D3/D5 remounted with nconnect=4, VM 509 restarted (nconnect now live on all tsys5 NFS client mounts) - SK hynix SSD (sdb1) persisted in fstab at /mnt/ssd2 (UUID-based) - ssd2 added to Proxmox storage.cfg as dir storage - Stale S3 export removed from /etc/exports (disk not mounted) - Broken S4 dir entry removed from storage.cfg - Duplicate noatime,noatime in fstab fixed - version=4.2 bug in storage.cfg options fixed (invalid mount option) tsys4 fixes: - SSH keys re-standardized to 2 approved (upgrade regenerated host key) - Tuning persistence working (128MB buffers confirmed live + persistent) apply-tunings.sh fix: - Removed version=4.2 from NFS options string (not a valid mount.nfs option — causes mount failures; Proxmox auto-negotiates NFSv4.2) Config files saved to provisioning/ConfigFiles/ for AWX/Ansible consumption: - 99-pfv-nfs.conf - pfv-nfs-tuning.service 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- perf/scripts/apply-tunings.sh | 4 +-- provisioning/ConfigFiles/99-pfv-nfs.conf | 23 +++++++++++++++ .../ConfigFiles/pfv-nfs-tuning.service | 28 +++++++++++++++++++ 3 files changed, 53 insertions(+), 2 deletions(-) create mode 100644 provisioning/ConfigFiles/99-pfv-nfs.conf create mode 100644 provisioning/ConfigFiles/pfv-nfs-tuning.service diff --git a/perf/scripts/apply-tunings.sh b/perf/scripts/apply-tunings.sh index f029f79..22bd605 100755 --- a/perf/scripts/apply-tunings.sh +++ b/perf/scripts/apply-tunings.sh @@ -364,7 +364,7 @@ if [ -r /etc/pve/storage.cfg ]; then if grep -A1 '^nfs:' /etc/pve/storage.cfg | grep -q 'options.*nconnect=4'; then echo " some stanzas already have nconnect=4 — check manually" else - echo " proposed change: add 'options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2' to each nfs stanza" + echo " proposed change: add 'options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2' to each nfs stanza" if [ "$ACTION" = "apply" ]; then # Apply via perl for safety (in-place edit with backup) if cp -a /etc/pve/storage.cfg "$BACKUP_DIR/etc__pve__storage.cfg"; then @@ -381,7 +381,7 @@ def fix(m): if 'options' in block: return block lines = block.rstrip('\n').split('\n') - lines.insert(1, '\toptions nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2') + lines.insert(1, '\toptions nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2') modified += 1 return '\n'.join(lines) + '\n' new_text = pattern.sub(fix, text) diff --git a/provisioning/ConfigFiles/99-pfv-nfs.conf b/provisioning/ConfigFiles/99-pfv-nfs.conf new file mode 100644 index 0000000..e3ee0b8 --- /dev/null +++ b/provisioning/ConfigFiles/99-pfv-nfs.conf @@ -0,0 +1,23 @@ +# PFV NFS tuning sysctl overrides +# +# Applied AFTER tuned via pfv-nfs-tuning.service (systemd oneshot). +# These override tuned's network-throughput/virtual-host 16MB TCP buffer +# caps with 128MB for high-BDP NFS over 1-4 GbE LACP links. +# +# Install on ALL Proxmox hosts: +# cp 99-pfv-nfs.conf /etc/sysctl.d/99-pfv-nfs.conf +# cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service +# systemctl daemon-reload && systemctl enable --now pfv-nfs-tuning.service +# +# Created: 2026-07-31 +# Deployed: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 + +net.core.rmem_max = 134217728 +net.core.wmem_max = 134217728 +net.core.rmem_default = 26214400 +net.core.wmem_default = 26214400 +net.core.netdev_max_backlog = 250000 +net.core.somaxconn = 65535 +net.ipv4.tcp_rmem = 4096 87380 134217728 +net.ipv4.tcp_wmem = 4096 65536 134217728 +net.ipv4.tcp_max_syn_backlog = 4096 diff --git a/provisioning/ConfigFiles/pfv-nfs-tuning.service b/provisioning/ConfigFiles/pfv-nfs-tuning.service new file mode 100644 index 0000000..75c66ae --- /dev/null +++ b/provisioning/ConfigFiles/pfv-nfs-tuning.service @@ -0,0 +1,28 @@ +# PFV NFS tuning service +# +# Systemd oneshot that runs AFTER tuned.service to apply TCP buffer +# overrides. The tuned daemon's profiles (network-throughput for storage +# hosts, virtual-host for compute hosts) set 16MB TCP buffer caps which +# are too small for high-BDP NFS over LACP links. This service force- +# applies 128MB buffers after tuned has finished its configuration. +# +# Install: +# cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service +# systemctl daemon-reload +# systemctl enable --now pfv-nfs-tuning.service +# +# Created: 2026-07-31 +# Deployed: all 7 Proxmox hosts (tsys1/3/4/5/6/7/9) + +[Unit] +Description=PFV NFS tuning (override tuned TCP buffer caps) +After=tuned.service +Requires=tuned.service + +[Service] +Type=oneshot +ExecStart=/sbin/sysctl -p /etc/sysctl.d/99-pfv-nfs.conf +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target