feat(awx): deploy AWX 24.6.1 on tsys-awx via k3s + AWX Operator

AWX (Ansible automation controller) deployed on tsys-awx.knel.net (VMID 600 on pfv-tsys6). Accessible at http://tsys-awx.knel.net:80 (admin/REDACTED_PASSWORD). Deployment details: - k3s v1.36.2 single-node (no Traefik) on Debian 13 trixie - AWX Operator 2.19.1 + AWX CR (LoadBalancer service via k3s ServiceLB) - PostgreSQL 15 bundled (8 GiB PVC on local-path provisioner) - VM disk resized 32->60 GB; swap partition converted to swapfile - Fixed kube-rbac-proxy image (gcr.io removed -> quay.io/brancz replacement) - Fixed AWX CR field names (postgres_data_volume_size -> postgres_storage_requirements) Scripts: awx/install-k3s.sh, awx/deploy-awx.sh, awx/verify-awx.sh Manifests: awx/awx-instance.yaml, awx/namespace.yaml Docs: awx/README.md All pods running: operator 2/2, postgres 1/1, web 3/3, task 4/4. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-07-29 19:12:03 -05:00
parent fa0c77fb58
commit 050e235f8b
8 changed files with 213 additions and 9 deletions
@@ -16,8 +16,9 @@ powerman/            Cyclades PM10i PDU management via powerman
 console/             Serial console management (ser2net + conman) for switches
 perf/                Proxmox perf tuning, fleet audit, iperf
 proxmox/             Proxmox fleet docs (hardware audit, capacity, k8s host planning)
-netinfra/            pfv-netinfra-01/02 DNS/NTP setup
+netinfra/            pfv-netinfra-01/02 DNS/NTP/DHCP setup
 switches/            Switch configuration captures
+awx/                 Ansible AWX deployment (k3s + AWX Operator)
 docs/                Server-build docs, docmap index, and archive
 vendor/              Vendored KNELShellFramework
 ```
@@ -16,8 +16,9 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste
 | [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
 | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
 | [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
-| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
+| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |
 | [`switches/`](switches/) | Switch configuration captures |
+| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) |
 | [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive |
 | [`vendor/`](vendor/) | Vendored KNELShellFramework |

@@ -82,6 +82,16 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 - [x] **Stair-stepping fix:** switched from ser2net→telnet→conman to
  conman direct-serial to eliminate telnet NVT CR stripping

+### Ansible AWX (tsys-awx — LIVE)
+- [x] AWX 24.6.1 deployed via AWX Operator 2.19.1 on k3s single-node
+- [x] Accessible at `http://tsys-awx.knel.net` (port 80, LoadBalancer via k3s ServiceLB)
+- [x] Admin login: `admin` / `REDACTED_PASSWORD`
+- [x] PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
+- [x] VM resized: 32→60 GB disk, swap converted to swapfile
+- [x] Scripts in `awx/`: install-k3s, deploy-awx, verify-awx
+- [ ] HTTPS/TLS not yet configured (currently HTTP on port 80)
+- [ ] Execution environments not yet customized
+
 ### Repo Merge
 - [x] KNELServerBuild merged into PFVCluster (history preserved)
 - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
@@ -142,5 +152,6 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 | Backup | Proxmox Backup Server (PBS) |
 | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
 | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
+| **AWX** | **tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)** |
 | **Console** | **7 switch consoles via conman direct-serial on pfv-tsys4 (conmand:7890 on Tailscale)** |
 | Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
@@ -0,0 +1,96 @@
+# AWX — tsys-awx.knel.net
+
+Ansible AWX deployment on a dedicated k3s single-node VM.
+
+**[→ Deployment Plan](DEPLOYMENT.md)** · **[→ Status](../../STATUS.md)**
+
+## Access
+
+| | |
+|---|---|
+| URL | `http://tsys-awx.knel.net` (LAN) or `http://100.91.39.53` (Tailscale) |
+| Port | 80 (LoadBalancer via k3s ServiceLB) |
+| User | `admin` |
+| Password | `REDACTED_PASSWORD` (fleet standard) |
+
+## Architecture
+
+```
+tsys-awx (VMID 600 on pfv-tsys6)
+├── k3s v1.36.2+k3s1 (single-node, no Traefik)
+├── AWX Operator 2.19.1 (manages the AWX lifecycle)
+└── AWX instance "tsys-awx"
+    ├── AWX web (UI/API)
+    ├── AWX task (Celery workers)
+    ├── PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
+    ├── Redis (ephemeral)
+    └── ServiceLB (binds port 80 to host IP)
+```
+
+## VM Resources
+
+| Resource | Value |
+|----------|-------|
+| Host | pfv-tsys6 (2× Xeon E5530, 126 GB RAM) |
+| vCPU | 4 (2 sockets × 2 cores) |
+| RAM | 12 GB |
+| Disk | 60 GB (D5 on tsys4 NFS) |
+| OS | Debian 13 trixie |
+| Network | 192.168.3.200 (LAN), 100.91.39.53 (Tailscale) |
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| [`install-k3s.sh`](install-k3s.sh) | Install k3s single-node (no Traefik) |
+| [`deploy-awx.sh`](deploy-awx.sh) | Deploy AWX Operator 2.19.1 + AWX instance |
+| [`verify-awx.sh`](verify-awx.sh) | Verify deployment status + access |
+| [`namespace.yaml`](namespace.yaml) | Kubernetes namespace manifest |
+| [`awx-instance.yaml`](awx-instance.yaml) | AWX Custom Resource (CR) manifest |
+
+## Deployment (from scratch)
+
+```bash
+# 1. Push scripts to VM
+for f in awx/install-k3s.sh awx/deploy-awx.sh awx/verify-awx.sh \
+         awx/namespace.yaml awx/awx-instance.yaml; do
+    VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
+        "sudo tee /tmp/awx/$(basename $f) > /dev/null" < "$f"
+done
+
+# 2. Install k3s
+VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
+    "sudo bash /tmp/awx/install-k3s.sh"
+
+# 3. Deploy AWX
+VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
+    "sudo bash /tmp/awx/deploy-awx.sh"
+
+# 4. Verify
+VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
+    "bash /tmp/awx/verify-awx.sh"
+```
+
+## Operating
+
+```bash
+# kubectl access
+sudo k3s kubectl -n awx get pods
+sudo k3s kubectl -n awx logs -f deployment/awx-operator-controller-manager
+
+# Restart AWX instance
+sudo k3s kubectl -n awx delete awx tsys-awx  # operator will recreate
+
+# Get admin password
+sudo k3s kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d
+```
+
+## Known Issues
+
+1. **kube-rbac-proxy image**: The AWX Operator 2.19.1 references
+   `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` which was removed from GCR.
+   The deploy script patches this to `quay.io/brancz/kube-rbac-proxy:v0.15.0`.
+2. **Image pulls are slow**: First deployment pulls ~2 GB of container images.
+   Postgres image alone is ~400 MB and takes several minutes.
+3. **Disk space**: AWX + k3s uses ~10 GB of the 60 GB disk. PostgreSQL data
+   grows over time — monitor `df -h /`.
@@ -30,7 +30,9 @@ spec:

  # PostgreSQL — bundled, stored on local disk via PVC (k3s local-path)
  postgres_storage_class: local-path
-  postgres_data_volume_size: 8Gi
+  postgres_storage_requirements:
+    requests:
+      storage: 8Gi
  postgres_resource_requirements:
    requests:
      memory: 1Gi
@@ -42,7 +44,3 @@ spec:
  task_resource_requirements:
    requests:
      memory: 1Gi
-
-  # Annotations for LoadBalancer (k3s ServiceLB)
-  service_annotations:
-    metallb.io/address-pool: ""
@@ -38,12 +38,28 @@ git clone --branch "${OPERATOR_VERSION}" --depth 1 \
    "https://github.com/ansible/awx-operator.git" "${OPERATOR_DIR}" 2>&1 | tail -3

 # The operator's default namespace is 'awx' — matches our setup
-# Apply the operator via kustomize
-kubectl apply -k "${OPERATOR_DIR}" 2>&1 || {
+# Apply the operator via kustomize (config/default has the full manifest set)
+kubectl apply -k "${OPERATOR_DIR}/config/default" 2>&1 || {
    echo "kustomize apply failed, trying raw manifests..."
    kubectl apply -f "https://raw.githubusercontent.com/ansible/awx-operator/${OPERATOR_VERSION}/deploy/awx-operator.yaml"
 }

+# Fix kube-rbac-proxy image (gcr.io/kubebuilder/kube-rbac-proxy was removed;
+# quay.io/brancz/kube-rbac-proxy is the maintained replacement)
+echo ""
+echo "=== Patching kube-rbac-proxy image ==="
+kubectl set image deployment/awx-operator-controller-manager -n awx \
+    kube-rbac-proxy=quay.io/brancz/kube-rbac-proxy:v0.15.0 2>&1 || true
+
+# Scale down any old replicasets that still reference the broken image
+for rs in $(kubectl -n awx get rs -l control-plane=controller-manager -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do
+    img=$(kubectl -n awx get rs "${rs}" -o jsonpath='{.spec.template.spec.containers[?(@.name=="kube-rbac-proxy")].image}' 2>/dev/null)
+    if [[ "${img}" == *"gcr.io/kubebuilder"* ]]; then
+        echo "Scaling down old RS ${rs} (has broken gcr.io image)"
+        kubectl -n awx scale rs "${rs}" --replicas=0 2>&1
+    fi
+done
+
 echo ""
 echo "Waiting for AWX Operator deployment to be ready..."
 kubectl -n awx wait --for=condition=Available deployment/awx-operator-controller-manager \
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+###############################################################################
+# verify-awx.sh — Verify AWX deployment status and access.
+#
+# Intended to run ON the target VM (tsys-awx.knel.net).
+# Usage:  bash verify-awx.sh
+###############################################################################
+set -euo pipefail
+
+export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}"
+
+echo "=========================================================="
+echo " AWX Deployment Verification — $(date)"
+echo "=========================================================="
+
+echo ""
+echo "=== 1. k3s node ==="
+kubectl get nodes
+
+echo ""
+echo "=== 2. AWX pods ==="
+kubectl -n awx get pods
+
+echo ""
+echo "=== 3. AWX CR status ==="
+kubectl -n awx get awx tsys-awx -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' 2>/dev/null || echo "AWX CR not found"
+
+echo ""
+echo "=== 4. Services ==="
+kubectl -n awx get svc
+
+echo ""
+echo "=== 5. LoadBalancer / NodePort access ==="
+LB_IP=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "")
+LB_HOST=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || echo "")
+NODE_PORT=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null || echo "")
+
+if [ -n "${LB_IP}" ]; then
+    echo "LoadBalancer IP: ${LB_IP}"
+    ACCESS_URL="http://${LB_IP}"
+elif [ -n "${LB_HOST}" ]; then
+    echo "LoadBalancer hostname: ${LB_HOST}"
+    ACCESS_URL="http://${LB_HOST}"
+elif [ -n "${NODE_PORT}" ]; then
+    echo "NodePort: ${NODE_PORT}"
+    ACCESS_URL="http://$(hostname -I | awk '{print $1}'):${NODE_PORT}"
+else
+    echo "Service not ready yet"
+    ACCESS_URL=""
+fi
+
+echo ""
+echo "=== 6. Admin password ==="
+ADMIN_PASS=$(kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+if [ -n "${ADMIN_PASS}" ]; then
+    echo "User: admin"
+    echo "Password: ${ADMIN_PASS}"
+else
+    echo "Admin password secret not found"
+fi
+
+echo ""
+echo "=== 7. HTTP check ==="
+if [ -n "${ACCESS_URL}" ]; then
+    echo "Testing ${ACCESS_URL}..."
+    HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "${ACCESS_URL}" 2>/dev/null || echo "failed")
+    echo "HTTP response: ${HTTP_CODE}"
+    if [ "${HTTP_CODE}" = "200" ] || [ "${HTTP_CODE}" = "302" ] || [ "${HTTP_CODE}" = "301" ]; then
+        echo "✓ AWX is accessible at ${ACCESS_URL}"
+    else
+        echo "✗ AWX not yet responding (HTTP ${HTTP_CODE})"
+    fi
+fi
+
+echo ""
+echo "=========================================================="
+if [ -n "${ACCESS_URL}" ]; then
+    echo " AWX Access URL: ${ACCESS_URL}"
+fi
+echo "=========================================================="
@@ -45,6 +45,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
+| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
 | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
 | [`../console/README.md`](../console/README.md) | Serial console management (conman direct-serial) for 7 network switches on pfv-tsys4 | 2026-07-29 |
 | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |