diff --git a/AGENTS.md b/AGENTS.md index b8ab23f..5833e25 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,8 +16,9 @@ powerman/ Cyclades PM10i PDU management via powerman console/ Serial console management (ser2net + conman) for switches perf/ Proxmox perf tuning, fleet audit, iperf proxmox/ Proxmox fleet docs (hardware audit, capacity, k8s host planning) -netinfra/ pfv-netinfra-01/02 DNS/NTP setup +netinfra/ pfv-netinfra-01/02 DNS/NTP/DHCP setup switches/ Switch configuration captures +awx/ Ansible AWX deployment (k3s + AWX Operator) docs/ Server-build docs, docmap index, and archive vendor/ Vendored KNELShellFramework ``` diff --git a/README.md b/README.md index cfe22eb..b3f03d6 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,9 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste | [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 | | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics | | [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning | -| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts | +| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts | | [`switches/`](switches/) | Switch configuration captures | +| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | | [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive | | [`vendor/`](vendor/) | Vendored KNELShellFramework | diff --git a/STATUS.md b/STATUS.md index ad8453e..a3505b4 100644 --- a/STATUS.md +++ b/STATUS.md @@ -82,6 +82,16 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [x] **Stair-stepping fix:** switched from ser2net→telnet→conman to conman direct-serial to eliminate telnet NVT CR stripping +### Ansible AWX (tsys-awx — LIVE) +- [x] AWX 24.6.1 deployed via AWX Operator 2.19.1 on k3s single-node +- [x] Accessible at `http://tsys-awx.knel.net` (port 80, LoadBalancer via k3s ServiceLB) +- [x] Admin login: `admin` / `REDACTED_PASSWORD` +- [x] PostgreSQL 15 (bundled, 8 GiB PVC on local-path) +- [x] VM resized: 32→60 GB disk, swap converted to swapfile +- [x] Scripts in `awx/`: install-k3s, deploy-awx, verify-awx +- [ ] HTTPS/TLS not yet configured (currently HTTP on port 80) +- [ ] Execution environments not yet customized + ### Repo Merge - [x] KNELServerBuild merged into PFVCluster (history preserved) - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/) @@ -142,5 +152,6 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Backup | Proxmox Backup Server (PBS) | | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** | | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** | +| **AWX** | **tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)** | | **Console** | **7 switch consoles via conman direct-serial on pfv-tsys4 (conmand:7890 on Tailscale)** | | Production | Cloudron VPS, Reston VA (this cluster is R&D only) | diff --git a/awx/README.md b/awx/README.md new file mode 100644 index 0000000..1c791c7 --- /dev/null +++ b/awx/README.md @@ -0,0 +1,96 @@ +# AWX — tsys-awx.knel.net + +Ansible AWX deployment on a dedicated k3s single-node VM. + +**[→ Deployment Plan](DEPLOYMENT.md)** · **[→ Status](../../STATUS.md)** + +## Access + +| | | +|---|---| +| URL | `http://tsys-awx.knel.net` (LAN) or `http://100.91.39.53` (Tailscale) | +| Port | 80 (LoadBalancer via k3s ServiceLB) | +| User | `admin` | +| Password | `REDACTED_PASSWORD` (fleet standard) | + +## Architecture + +``` +tsys-awx (VMID 600 on pfv-tsys6) +├── k3s v1.36.2+k3s1 (single-node, no Traefik) +├── AWX Operator 2.19.1 (manages the AWX lifecycle) +└── AWX instance "tsys-awx" + ├── AWX web (UI/API) + ├── AWX task (Celery workers) + ├── PostgreSQL 15 (bundled, 8 GiB PVC on local-path) + ├── Redis (ephemeral) + └── ServiceLB (binds port 80 to host IP) +``` + +## VM Resources + +| Resource | Value | +|----------|-------| +| Host | pfv-tsys6 (2× Xeon E5530, 126 GB RAM) | +| vCPU | 4 (2 sockets × 2 cores) | +| RAM | 12 GB | +| Disk | 60 GB (D5 on tsys4 NFS) | +| OS | Debian 13 trixie | +| Network | 192.168.3.200 (LAN), 100.91.39.53 (Tailscale) | + +## Files + +| File | Purpose | +|------|---------| +| [`install-k3s.sh`](install-k3s.sh) | Install k3s single-node (no Traefik) | +| [`deploy-awx.sh`](deploy-awx.sh) | Deploy AWX Operator 2.19.1 + AWX instance | +| [`verify-awx.sh`](verify-awx.sh) | Verify deployment status + access | +| [`namespace.yaml`](namespace.yaml) | Kubernetes namespace manifest | +| [`awx-instance.yaml`](awx-instance.yaml) | AWX Custom Resource (CR) manifest | + +## Deployment (from scratch) + +```bash +# 1. Push scripts to VM +for f in awx/install-k3s.sh awx/deploy-awx.sh awx/verify-awx.sh \ + awx/namespace.yaml awx/awx-instance.yaml; do + VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ + "sudo tee /tmp/awx/$(basename $f) > /dev/null" < "$f" +done + +# 2. Install k3s +VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ + "sudo bash /tmp/awx/install-k3s.sh" + +# 3. Deploy AWX +VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ + "sudo bash /tmp/awx/deploy-awx.sh" + +# 4. Verify +VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ + "bash /tmp/awx/verify-awx.sh" +``` + +## Operating + +```bash +# kubectl access +sudo k3s kubectl -n awx get pods +sudo k3s kubectl -n awx logs -f deployment/awx-operator-controller-manager + +# Restart AWX instance +sudo k3s kubectl -n awx delete awx tsys-awx # operator will recreate + +# Get admin password +sudo k3s kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d +``` + +## Known Issues + +1. **kube-rbac-proxy image**: The AWX Operator 2.19.1 references + `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` which was removed from GCR. + The deploy script patches this to `quay.io/brancz/kube-rbac-proxy:v0.15.0`. +2. **Image pulls are slow**: First deployment pulls ~2 GB of container images. + Postgres image alone is ~400 MB and takes several minutes. +3. **Disk space**: AWX + k3s uses ~10 GB of the 60 GB disk. PostgreSQL data + grows over time — monitor `df -h /`. diff --git a/awx/awx-instance.yaml b/awx/awx-instance.yaml index a9d452b..d069c01 100644 --- a/awx/awx-instance.yaml +++ b/awx/awx-instance.yaml @@ -30,7 +30,9 @@ spec: # PostgreSQL — bundled, stored on local disk via PVC (k3s local-path) postgres_storage_class: local-path - postgres_data_volume_size: 8Gi + postgres_storage_requirements: + requests: + storage: 8Gi postgres_resource_requirements: requests: memory: 1Gi @@ -42,7 +44,3 @@ spec: task_resource_requirements: requests: memory: 1Gi - - # Annotations for LoadBalancer (k3s ServiceLB) - service_annotations: - metallb.io/address-pool: "" diff --git a/awx/deploy-awx.sh b/awx/deploy-awx.sh index 34c01ea..1010e84 100644 --- a/awx/deploy-awx.sh +++ b/awx/deploy-awx.sh @@ -38,12 +38,28 @@ git clone --branch "${OPERATOR_VERSION}" --depth 1 \ "https://github.com/ansible/awx-operator.git" "${OPERATOR_DIR}" 2>&1 | tail -3 # The operator's default namespace is 'awx' — matches our setup -# Apply the operator via kustomize -kubectl apply -k "${OPERATOR_DIR}" 2>&1 || { +# Apply the operator via kustomize (config/default has the full manifest set) +kubectl apply -k "${OPERATOR_DIR}/config/default" 2>&1 || { echo "kustomize apply failed, trying raw manifests..." kubectl apply -f "https://raw.githubusercontent.com/ansible/awx-operator/${OPERATOR_VERSION}/deploy/awx-operator.yaml" } +# Fix kube-rbac-proxy image (gcr.io/kubebuilder/kube-rbac-proxy was removed; +# quay.io/brancz/kube-rbac-proxy is the maintained replacement) +echo "" +echo "=== Patching kube-rbac-proxy image ===" +kubectl set image deployment/awx-operator-controller-manager -n awx \ + kube-rbac-proxy=quay.io/brancz/kube-rbac-proxy:v0.15.0 2>&1 || true + +# Scale down any old replicasets that still reference the broken image +for rs in $(kubectl -n awx get rs -l control-plane=controller-manager -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do + img=$(kubectl -n awx get rs "${rs}" -o jsonpath='{.spec.template.spec.containers[?(@.name=="kube-rbac-proxy")].image}' 2>/dev/null) + if [[ "${img}" == *"gcr.io/kubebuilder"* ]]; then + echo "Scaling down old RS ${rs} (has broken gcr.io image)" + kubectl -n awx scale rs "${rs}" --replicas=0 2>&1 + fi +done + echo "" echo "Waiting for AWX Operator deployment to be ready..." kubectl -n awx wait --for=condition=Available deployment/awx-operator-controller-manager \ diff --git a/awx/verify-awx.sh b/awx/verify-awx.sh new file mode 100644 index 0000000..8c70b7b --- /dev/null +++ b/awx/verify-awx.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +############################################################################### +# verify-awx.sh — Verify AWX deployment status and access. +# +# Intended to run ON the target VM (tsys-awx.knel.net). +# Usage: bash verify-awx.sh +############################################################################### +set -euo pipefail + +export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" + +echo "==========================================================" +echo " AWX Deployment Verification — $(date)" +echo "==========================================================" + +echo "" +echo "=== 1. k3s node ===" +kubectl get nodes + +echo "" +echo "=== 2. AWX pods ===" +kubectl -n awx get pods + +echo "" +echo "=== 3. AWX CR status ===" +kubectl -n awx get awx tsys-awx -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' 2>/dev/null || echo "AWX CR not found" + +echo "" +echo "=== 4. Services ===" +kubectl -n awx get svc + +echo "" +echo "=== 5. LoadBalancer / NodePort access ===" +LB_IP=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "") +LB_HOST=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || echo "") +NODE_PORT=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null || echo "") + +if [ -n "${LB_IP}" ]; then + echo "LoadBalancer IP: ${LB_IP}" + ACCESS_URL="http://${LB_IP}" +elif [ -n "${LB_HOST}" ]; then + echo "LoadBalancer hostname: ${LB_HOST}" + ACCESS_URL="http://${LB_HOST}" +elif [ -n "${NODE_PORT}" ]; then + echo "NodePort: ${NODE_PORT}" + ACCESS_URL="http://$(hostname -I | awk '{print $1}'):${NODE_PORT}" +else + echo "Service not ready yet" + ACCESS_URL="" +fi + +echo "" +echo "=== 6. Admin password ===" +ADMIN_PASS=$(kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' 2>/dev/null | base64 -d 2>/dev/null || echo "") +if [ -n "${ADMIN_PASS}" ]; then + echo "User: admin" + echo "Password: ${ADMIN_PASS}" +else + echo "Admin password secret not found" +fi + +echo "" +echo "=== 7. HTTP check ===" +if [ -n "${ACCESS_URL}" ]; then + echo "Testing ${ACCESS_URL}..." + HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "${ACCESS_URL}" 2>/dev/null || echo "failed") + echo "HTTP response: ${HTTP_CODE}" + if [ "${HTTP_CODE}" = "200" ] || [ "${HTTP_CODE}" = "302" ] || [ "${HTTP_CODE}" = "301" ]; then + echo "✓ AWX is accessible at ${ACCESS_URL}" + else + echo "✗ AWX not yet responding (HTTP ${HTTP_CODE})" + fi +fi + +echo "" +echo "==========================================================" +if [ -n "${ACCESS_URL}" ]; then + echo " AWX Access URL: ${ACCESS_URL}" +fi +echo "==========================================================" diff --git a/docs/docmap.md b/docs/docmap.md index 3152229..e938652 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -45,6 +45,7 @@ Server provisioning, security hardening, DNS/NTP configuration. | Document | Description | Last Reviewed | |----------|-------------|---------------| | [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 | +| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 | | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | | [`../console/README.md`](../console/README.md) | Serial console management (conman direct-serial) for 7 network switches on pfv-tsys4 | 2026-07-29 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |