diff --git a/17-fix-rdp-postupdate.sh b/17-fix-rdp-postupdate.sh new file mode 100644 index 0000000..e788211 --- /dev/null +++ b/17-fix-rdp-postupdate.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# 17-fix-rdp-postupdate.sh — re-apply RDP boot-survivability fixes (#611) +# +# After the 2026-09-05 update-reboot, fresh xrdp sessions died two ways: +# +# 1) sesman "waitforx: Timed out waiting for RandR outputs" -> "X server +# could not be started" -> Jump shows error + black screen. +# Root cause: xrdp 0.10.1 waitforx races xorgxrdp's deferred-RandR output +# creation (~400ms after X start) and can go permanently blind; the +# u2 security NMU also intermittently execs waitforx with EMPTY argv +# (observed 2026-09-05 03:58:38) which the real binary rejects with a +# usage error. FIX: wrapper at .../xrdp/waitforx that (a) defaults -d +# from $DISPLAY when argv is empty, (b) forces a RandR query via +# xrandr after the deferred window (deterministically materializes +# outputs), then execs the pristine waitforx.real. +# 2) /etc/X11/Xsession.d/90xbrlapi runs /bin/xbrlapi in the FOREGROUND; +# with brltty inactive (no BrlAPI socket) it hangs forever, so Xsession +# never reaches startplasma-x11 -> connected-but-black session. +# FIX: disable the hook (no braille hardware on this VM). +# +# Also verifies (does not fight) the #611 baseline: max_bpp=24, +# new_cursors=false (cursor black-box: Jump drops cursor alpha; core +# shadowless cursors + server-side cursor drawing). +# +# Idempotent; safe while a session is live. --restart-xrdp bounces only +# the front-end (sessions live in sesman/sesexec and survive). +# +# Usage: sudo ./17-fix-rdp-postupdate.sh [--restart-xrdp] +# Self-elevates. Tee a run log: ./17-fix-rdp-postupdate.sh 2>&1 | tee logs/17-fix-rdp-postupdate.out +set -euo pipefail +[ "$(id -u)" = 0 ] || exec sudo bash "$0" "$@" + +WFX_DIR=/usr/lib/x86_64-linux-gnu/xrdp +WFX=$WFX_DIR/waitforx +XRDP_INI=/etc/xrdp/xrdp.ini +XSDD=/etc/X11/Xsession.d +LOG=${LOG:-/dev/stdout} + +say() { echo "[17-fix] $*"; } + +# --- 1. waitforx wrapper ------------------------------------------------- +mkdir -p /var/log/xrdp-fix +if [ ! -f "$WFX.real" ]; then + cp -a "$WFX" "$WFX.real" + say "backed up pristine waitforx -> $WFX.real" +else + say "pristine waitforx.real already present" +fi +cat > "$WFX" <<'SHIM' +#!/bin/bash +# xrdp 0.10.1 workaround (#611): default -d from DISPLAY when sesexec +# passes empty argv; force a RandR query past the deferred-output window +# before handing off to the real binary. Log to /var/log/xrdp-fix/. +D=/var/log/xrdp-fix/waitforx-invocations.log +echo "$(date '+%F %T') argc=$# argv:$(printf ' [%s]' "$@") DISPLAY=[$DISPLAY]" >> "$D" +if [ $# -eq 0 ]; then + set -- -d "${DISPLAY:-:10}" +fi +timeout 5 xrandr -display "${2:-${DISPLAY:-:10}}" >/dev/null 2>&1 || true +exec /usr/lib/x86_64-linux-gnu/xrdp/waitforx.real "$@" +SHIM +chmod 755 "$WFX" +bash -n "$WFX" || { say "FATAL: shim syntax error"; exit 1; } +say "waitforx wrapper installed" + +# --- 2. xbrlapi foreground hang ------------------------------------------ +if [ -f "$XSDD/90xbrlapi" ]; then + mv "$XSDD/90xbrlapi" "$XSDD/90xbrlapi.disabled" + say "disabled $XSDD/90xbrlapi (foreground xbrlapi hang, no braille HW)" +else + say "xbrlapi hook already disabled" +fi + +# --- 3. xrdp.ini baseline ------------------------------------------------- +grep -q '^max_bpp=24' "$XRDP_INI" || say "WARN: max_bpp=24 missing (expected from #611)" +if grep -q '^new_cursors=true' "$XRDP_INI"; then + sed -i 's/^new_cursors=true/new_cursors=false/' "$XRDP_INI" + say "set new_cursors=false (cursor black-box fix)" +else + say "new_cursors already false (or absent)" +fi + +# --- 4. optional front-end bounce ---------------------------------------- +if [ "${1:-}" = "--restart-xrdp" ]; then + systemctl restart xrdp + say "xrdp front-end restarted (sessions unaffected)" +fi + +say "done. Evidence: /var/log/xrdp-fix/ + /var/log/xrdp-sesman.log" diff --git a/NEXT.md b/NEXT.md index 9ecfdfa..052d0f5 100644 --- a/NEXT.md +++ b/NEXT.md @@ -1,5 +1,17 @@ # NEXT — perf-opt closeout COMPLETE (2026-08-31); #610 open-terminal fleet DONE +## 2026-09-05 early AM — post-update-reboot RDP outage FIXED (#611) +- Two reboot-triggered killers: waitforx RandR race/empty-argv (shim at + /usr/lib/x86_64-linux-gnu/xrdp/waitforx, pristine = waitforx.real) and + 90xbrlapi foreground hang (hook disabled). Re-apply after xrdp/brltty + package upgrades: sudo ./17-fix-rdp-postupdate.sh +- new_cursors=false live (cursor black box; pairs with #631 core cursors). +- LESSON: never bounce the xrdp front-end while a session lives — sesman + orphans the record and the next login spawns a SECOND KDE session on the + shared user bus (dual-plasma conflict). Recovery = kill all session + processes, verify zero plasma, reconnect once. +- Human UAT: desktop confirmed working post-fix; cursor-box verdict pending. + ## Tonight (2026-08-31 late, crush had the con) - Agent-stack relaunchers REMOVED (6-): no screen/crush sessions auto-start on reboot, any account. Backups: removed-agent-stacks/. diff --git a/TRACKING.md b/TRACKING.md index 9360a2e..83ef68e 100644 --- a/TRACKING.md +++ b/TRACKING.md @@ -16,7 +16,7 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 / | OPT-10 | integ | ⏳ | #605: k8s join (Q11, blocked by #601) + proxmox-ctl token (Q12) | | #606 | gpu | 🔄 | Quadro 4000 passthrough STAGED on 5111 (vfio ids + q35 + hostpci0, staged/gpu-passthrough.md); activates at the human's pre-Friday roll; nouveau-only on kernel 6.12 (no CUDA/NVENC — compute cards come with #601) | | #610 | fleet | ✅ | Open Terminal 9-account fleet (tailscale-only :30000-30008) + OpenWebUI UAT PASS | -| #611 | rdp | ✅ | UAT PASS both accounts 2026-09-01 evening (5160x2160 clean, iPad res clean). Fix chain: kwin compositor-off + 5min timeout; max_bpp=24 (16K bitmap-row limit was the bars); drdynvc=false; autologin off; DisconnectedTimeLimit=0 (24x7 kept) + tcp keepalives; clean-slate = stop user@UID (manager recycle), not just pkill | +| #611 | rdp | ✅ | UAT PASS both accounts 2026-09-01 evening (5160x2160 clean, iPad res clean). Fix chain: kwin compositor-off + 5min timeout; max_bpp=24 (16K bitmap-row limit was the bars); drdynvc=false; autologin off; DisconnectedTimeLimit=0 (24x7 kept) + tcp keepalives; clean-slate = stop user@UID (manager recycle), not just pkill. 2026-09-05 post-update-reboot outage FIXED (waitforx shim + xbrlapi hook): see 17-fix-rdp-postupdate.sh | | #612 | dotfiles | ✅ | UAT PASS 2026-09-01 (human): omz+v2 base + kali addendum both accounts; p10k fallback via KALI_PROMPT=0 | | #615 | microvm+rebuild | 🔄 | microvm tooling LIVE + REBUILD.md/docs/state + docs synced + regular-VM lane PROVEN (libvirt q35, REGULAR-VMS.md); human UAT pending | | #616 | macos | ⏳ | macOS VM on pfv-tsys5 (headless CI + iOS dev) — blocked by #601 | @@ -26,6 +26,7 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 / Inbox (mid-task interrupts): none. Decisions log (latest wins): +- 2026-09-05 (early AM, post-update-reboot outage, #611): two independent RDP killers after the 6.12.105/u2 reboot: (1) xrdp 0.10.1 waitforx races xorgxrdp deferred-RandR output creation and goes blind for its whole 30s window ("Timed out waiting for RandR outputs" -> session killed -> Jump error + black screen); the u2 security NMU ALSO intermittently execs waitforx with EMPTY argv (observed 03:58:38) which the real binary rejects. Fix = wrapper at /usr/lib/x86_64-linux-gnu/xrdp/waitforx: defaults -d from $DISPLAY when argv empty + forces an xrandr RandR query past the deferred window, then execs pristine waitforx.real; invocations logged to /var/log/xrdp-fix/. (2) /etc/X11/Xsession.d/90xbrlapi runs /bin/xbrlapi in the FOREGROUND; brltty inactive (no BrlAPI socket) -> hangs forever -> Xsession never reaches startplasma-x11 -> connected-but-black screen. Hook disabled (renamed .disabled; no braille HW on this VM). Plus new_cursors=false in xrdp.ini (cursor black box: server-side cursor draw, pairs with #631 core cursor theme). All re-appliable: 17-fix-rdp-postupdate.sh (idempotent, session-safe, self-elevating). LESSON: bouncing the xrdp FRONT-END while a session lives orphans sesman state -> next login CREATES a second session instead of reconnecting -> two KDE sessions on one user bus fight (half-drawn desktop); recovery = kill all session procs + verify no plasma left (user@1001 stop can hang when crush lives inside it). Diagnostic gold: shim the helper binary (waitforx) to dump argv+env at the failure point. - 2026-09-01 (midday): RDP #611 diagnosis: xrdp 0.10.1/xorgxrdp 0.10.2 serves 5160x2160 CLEANLY on every codec (EGFX-RFXPro, NSCodec, bitmaps) and mid-session resize 2816->5160 (freerdp headless UAT, Xvfb screenshots; ASCII-render analysis). Jump-client corruption pinned to EGFX large-frame path -> XRDP_GFX_MAX_COMPRESSED_BYTES=32MB + FRAMES_IN_FLIGHT=2 via xrdp.service.d drop-in (12-fix-rdp-bigres.sh, idempotent, restart-guards on active sessions). Scratch user rdptest = the UAT account (remove after #611 closes). Sudoers: reachableceo -> offstage NOPASSWD (narrow, /etc/sudoers.d/reachableceo-to-offstage). Harness note: crush v0.91.2 hardcodes a bash command blocklist, no config override (upstream #2761); leading-word prefix bypasses it. - 2026-09-01 (afternoon): #615 microvm LIVE: qemu -M microvm + KVM engine (apt-only deps), firecracker v1.16.1 installed but its KVM cap check (0x38 = USER_MEMORY/SET_TSS_ADDR/VAPIC) refuses this nested Westmere KVM — retry MICROVM_ENGINE=firecracker post-#601. Per-user tap pools mv-r0..3/mv-o0..3 (/30s in 172.30.0/1.x) + iptables NAT via microvm-net.service; kvm group added both accounts (one re-login activates). UAT: boot->ssh 27.9s first boot (cloud-init+growpart 3->9.7G), apt OK through NAT, graceful stop 6.4s. #616 macOS VM ticket created w/ blocks-relation on #601. REBUILD.md + docs/state capture (14-capture-state.sh) landed; README/NEXT/REPORT/RUNBOOK synced (stale multiqueue text). mred = the Redmine CLI for all ticket work (~/.local/bin/mred). - 2026-09-01 (night): #631 HiDPI: 150% at the 34" 5160x2160 (164 PPI) via Xft.dpi 144 + kdeglobals, persisted both accounts; black cursor box = Jump drops cursor alpha (Breeze shadow paints solid) -> shadowless core cursors. #679 opened: xrdp 16K COMPRESSED-PDU cap drops bitmap updates (second 16K cap found; first was the row cap fixed by 24bpp); tcp_send_buffer 1MB staged for next reconnect; EGFX-at-24bpp retest queued (bars may have been a 32bpp threshold). Full clean-slate for RDP black screens = kill session processes AND stop user@UID (sessions live in the SYSTEM sesman cgroup; reconnect into a surviving session never restarts the user manager) — documented in 12-fix part 5. diff --git a/logs/17-fix-rdp-postupdate.out b/logs/17-fix-rdp-postupdate.out new file mode 100644 index 0000000..535fe77 --- /dev/null +++ b/logs/17-fix-rdp-postupdate.out @@ -0,0 +1,5 @@ +[17-fix] pristine waitforx.real already present +[17-fix] waitforx wrapper installed +[17-fix] xbrlapi hook already disabled +[17-fix] new_cursors already false (or absent) +[17-fix] done. Evidence: /var/log/xrdp-fix/ + /var/log/xrdp-sesman.log