chore: ignore Python bytecode artifacts

Add __pycache__/ and *.pyc to .gitignore so bytecode produced when
validating scripts/awx_create_job_templates.py locally is not committed.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>

.gitignore

@@ -1,8 +1,13 @@
 # Ansible / AWX local artifacts
 .fact_cache/
+.venv/
 *.retry
 *.log
 
+# Python
+__pycache__/
+*.pyc
+
 # Editor / OS
 .DS_Store
 *.swp

AGENTS.md

@@ -1,5 +1,25 @@
 # Agent Guidelines
 
+## Secrets and Credentials Policy
+
+**NEVER commit secrets, passwords, tokens, API keys, SSH keys, or any
+credential to this repository — in source, in comments, in examples, in
+documentation, or in commit messages. No exceptions.**
+
+Rules:
+1. **No secrets in code.** Credentials must come from environment variables,
+   a vault, or Ansible Vault — never from a hardcoded literal.
+2. **No real secrets in examples/docs.** Use placeholders like `$AWX_PASSWORD`,
+   `***REMOVED***`, or `<your-token-here>`.
+3. **Check before committing.** If a change introduces anything that looks like
+   a secret (password assignment, `b64encode`, `AUTH =`, API key, private key
+   block), stop and refactor it to read from the environment first.
+4. **If a secret is ever introduced**, it must be purged from the entire git
+   history (not just the working tree) with `git filter-repo --replace-text`,
+   followed by a force-push. Rotating the exposed credential is mandatory.
+5. **`.gitignore`** must keep secret-bearing paths ignored (e.g. `.env`, private
+   keys). Add new entries as needed.
+
 ## Git Commit and Push Policy
 
 **ALWAYS commit and push automatically. NEVER wait to be asked.**
@@ -56,3 +76,4 @@ Assisted-by: GLM-5 via Crush <crush@charm.land>
 - **Commit immediately after each logical unit of work.**
 - **Push after every commit.**
 - **Prefer many small, focused commits over fewer large ones.**
+- **Never commit secrets** — see the Secrets and Credentials Policy above.

README.md

@@ -1,3 +1,121 @@
-# KNELConfigMgmt-Ansible
+# KNELConfigMgmt-Ansible (KNELIAC)
 
-KNEL Configuration Management Collection Ansible
+Ansible configuration-management collection for the Known Element Enterprises
+fleet. This is the Ansible port of the legacy bash provisioning system that
+lives in `PFVCluster/provisioning/`, and is the project that **AWX**
+(`tsys-awx.knel.net`) syncs and executes.
+
+**Target hosts:** Debian-family only (Debian, Ubuntu Server, Kali, Raspberry Pi OS).
+
+---
+
+## Repository layout
+
+```
+KNELIAC/
+├── ansible.cfg                # ansible settings (roles_path, inventory, become)
+├── requirements.yml           # external collections (currently none)
+├── inventory/
+│   ├── hosts.yml              # host inventory (physical_hosts / virtual_guests / raspberry_pi)
+│   └── group_vars/all.yml     # all tunable variables (mirrors hard-coded bash values)
+├── playbooks/
+│   ├── hello_world.yml        # AWX smoke-test playbook
+│   └── setup_new_system.yml   # full host build (port of SetupNewSystem.sh)
+└── roles/
+    ├── preflight              # host-class detection (physical/raspi/virt/ubuntu/kali)
+    ├── oam                    # LibreNMS / check_mk agent
+    ├── packages               # up2date, install fleet toolset, remove unwanted pkgs
+    ├── system_config          # postfix, resolv, snmp, ntp, dhcp, lldpd, cockpit, zsh, shells
+    ├── security_ssh           # sshd_config, authorized_keys, ssh-audit hardening
+    ├── security_wazuh         # wazuh-agent install + hold
+    ├── security_scap_stig     # GRUB perms, modprobe blacklist, banners, cron/at perms
+    ├── security_2fa           # TOTP 2FA for SSH / Cockpit / Webmin
+    └── security_audit         # auditd, journald, logrotate
+```
+
+## Legacy → Ansible mapping
+
+| Legacy bash                                          | KNELIAC role            |
+|------------------------------------------------------|-------------------------|
+| `SetupNewSystem.sh` (runner)                         | `playbooks/setup_new_system.yml` |
+| `Project-Includes/PreflightCheck.sh` + `pi-detect.sh`| `roles/preflight`       |
+| `Modules/OAM/oam-librenms.sh`                        | `roles/oam`             |
+| `global-installPackages` + `scripts/up2date.sh`      | `roles/packages`        |
+| `global-systemServiceConfigurationFiles` + `global-postPackageConfiguration` | `roles/system_config` |
+| `Modules/Security/secharden-ssh.sh`                  | `roles/security_ssh`    |
+| `Modules/Security/secharden-wazuh.sh`                | `roles/security_wazuh`  |
+| `Modules/Security/secharden-scap-stig.sh`            | `roles/security_scap_stig` |
+| `Modules/Security/secharden-2fa.sh`                  | `roles/security_2fa`    |
+| `Modules/Security/secharden-audit-agents.sh`         | `roles/security_audit`  |
+
+## Getting started with AWX
+
+1. **Create a Project** in AWX pointing at this git repo.
+2. **Create an Inventory** (either maintain hosts in AWX directly, or sync from
+   `inventory/hosts.yml`).
+3. **Create a Job Template**:
+   - Playbook: `playbooks/hello_world.yml`
+   - Run it against any host to confirm AWX can reach, become root, and gather
+     facts. A green run = the pipeline works.
+4. For the full build, create a second Job Template with playbook
+   `playbooks/setup_new_system.yml`. Each phase is gated by a `run_*` toggle
+   (see `inventory/group_vars/all.yml`), so you can enable phases incrementally.
+
+## Configuration
+
+All knobs live in [`inventory/group_vars/all.yml`](inventory/group_vars/all.yml)
+and can be overridden per-host (`inventory/host_vars/<host>.yml`), per-group
+(`inventory/group_vars/<group>.yml`), or directly in AWX as **extra vars** on a
+Job Template. Key variables:
+
+| Variable                | Purpose                                              |
+|-------------------------|------------------------------------------------------|
+| `dns_servers`           | Authoritative recursive DNS servers                  |
+| `ntp_servers`           | Upstream NTP sources                                 |
+| `postfix_relayhost`     | SMTP smarthost                                       |
+| `wazuh_manager`         | Wazuh server FQDN                                    |
+| `packages_install`      | Fleet toolset package list                           |
+| `packages_remove`       | Packages purged on every host                        |
+| `run_*`                 | Feature toggles to enable/skip each hardening phase  |
+
+## Host-class conditionals
+
+The `preflight` role detects host class at runtime and sets facts that
+downstream roles branch on. Inventory groups provide additional static
+classification.
+
+| Fact | Source | Controls |
+|------|--------|----------|
+| `is_physical_host` | dmidecode Dell + NOT Proxmox + NOT Pi | physical snmpd.conf, CPU governor, physical packages |
+| `is_proxmox_host` | dpkg proxmox-ve | physical packages + CPU governor; skips cockpit/tuned |
+| `is_virt_guest` | virt-what (hyperv/kvm) | VM snmpd.conf, qemu-guest-agent, tuned virtual-guest |
+| `is_raspi` | /sys/firmware/devicetree/base/model | Pi snmpd.conf, skip GRUB perms |
+| `is_kali` | ansible_distribution==Kali | skip unavailable packages |
+| `is_ubuntu` | ansible_distribution==Ubuntu | skip ssh-audit hardening drop-in |
+| `is_ntp_server` | `ntp_servers` inventory group | skip NTP client config |
+| `is_dhcp_server` | `dhcp_servers` inventory group | skip dhclient.conf deploy |
+| `is_librenms_server` | `librenms_server` inventory group | skip rsyslog forward config |
+| `is_wazuh_server` | `wazuh_server` inventory group | skip wazuh-agent install |
+| `is_dev_workstation` | `dev_workstations` inventory group | skip hardened sshd_config |
+
+## Roadmap
+
+This is the foundation for a comprehensive DISA STIG / CMMC / FedRAMP / ITAR
+compliance library. The `security_scap_stig` role is the seed for that work —
+additional STIG control roles will be added alongside it.
+
+## AWX setup
+
+AWX resources have been created at `http://tsys-awx.knel.net`:
+
+| Resource | ID | Name |
+|----------|----|------|
+| Credential | 3 | KNELIAC Git Credential (Source Control) |
+| Credential | 4 | KNELIAC Host SSH Key (Machine) |
+| Project | 10 | KNELIAC (git, auto-syncs on launch) |
+| Inventory | 2 | KNELIAC Fleet (SCM-backed, syncs from hosts.yml) |
+
+Job templates will be created after the code is pushed to git. Run:
+```bash
+python3 scripts/awx_create_job_templates.py
+```

playbooks/setup_awx_https.yml

@@ -0,0 +1,132 @@
+---
+# Put AWX behind HTTPS with a host-level nginx TLS-termination reverse proxy.
+#
+# Why a reverse proxy instead of in-pod TLS:
+#   The AWX Operator only wires nginx for HTTPS on the OpenShift Route
+#   "passthrough" code path (ingress_type: route + route_tls_termination_mechanism:
+#   passthrough), which requires the Route CRD and breaks on k3s. There is also no
+#   ingress controller on the cluster. Terminating TLS at a host nginx proxy in
+#   front of the existing LoadBalancer service is the lowest-risk option and is
+#   trivially swappable once an internal CA / ingress controller + cert-manager
+#   arrive.
+#
+# Topology after this playbook:
+#   client --https:443--> host nginx (TLS terminate) --http:80--> k3s ServiceLB
+#                                                                   |
+#                                                                   v
+#                                            tsys-awx-service (LB) -> awx-web:8052
+#
+# The cert is self-signed (CN=tsys-awx.knel.net) with SANs covering the FQDN,
+# the short hostname, and both the LAN and Tailscale IPs. Re-run this playbook
+# after rotating the cert (e.g. when the internal CA is rolled out).
+
+- name: AWX HTTPS reverse proxy
+  hosts: tsys-awx
+  gather_facts: true
+  become: true
+
+  vars:
+    # Cert subject / SANs. Extend these lists as more names are needed.
+    awx_tls_cn: tsys-awx.knel.net
+    awx_tls_sans:
+      - "DNS:tsys-awx.knel.net"
+      - "DNS:tsys-awx"
+      - "IP:192.168.3.200"
+      - "IP:100.91.39.53"
+    awx_tls_dir: /etc/nginx/ssl
+    awx_tls_cert: "{{ awx_tls_dir }}/awx-self-signed.crt"
+    awx_tls_key: "{{ awx_tls_dir }}/awx-self-signed.key"
+
+  tasks:
+    - name: Install nginx
+      ansible.builtin.apt:
+        name: nginx
+        state: present
+        update_cache: true
+
+    - name: Ensure TLS cert directory exists
+      ansible.builtin.file:
+        path: "{{ awx_tls_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Generate self-signed certificate (only if missing)  # noqa no-changed-when
+      ansible.builtin.shell: |
+        set -o pipefail
+        openssl req -x509 -newkey rsa:2048 -nodes -days 825 \
+          -keyout {{ awx_tls_key }} -out {{ awx_tls_cert }} \
+          -subj "/CN={{ awx_tls_cn }}" \
+          -addext "subjectAltName={{ awx_tls_sans | join(',') }}"
+      args:
+        creates: "{{ awx_tls_cert }}"
+
+    - name: Lock down private key permissions
+      ansible.builtin.file:
+        path: "{{ awx_tls_key }}"
+        owner: root
+        group: root
+        mode: "0600"
+
+    - name: Deploy nginx TLS proxy site config
+      ansible.builtin.copy:
+        dest: /etc/nginx/sites-available/awx-tls.conf
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Ansible-managed: TLS termination for AWX.
+          # Proxies https://tsys-awx.knel.net -> http://127.0.0.1:80 (k3s ServiceLB -> AWX).
+          # map must live in the http{} context; sites-enabled/* are included inside http.
+          map $http_upgrade $awx_connection_upgrade {
+              default upgrade;
+              ''      close;
+          }
+
+          server {
+              listen 443 ssl;
+              listen [::]:443 ssl;
+              http2 on;
+
+              server_name tsys-awx.knel.net tsys-awx 192.168.3.200 100.91.39.53;
+
+              ssl_certificate     /etc/nginx/ssl/awx-self-signed.crt;
+              ssl_certificate_key /etc/nginx/ssl/awx-self-signed.key;
+              ssl_protocols TLSv1.2 TLSv1.3;
+              ssl_ciphers HIGH:!aNULL:!MD5;
+              ssl_session_cache shared:SSL:10m;
+              ssl_session_timeout 1d;
+
+              client_max_body_size 50M;
+
+              location / {
+                  proxy_pass http://127.0.0.1:80;
+                  proxy_http_version 1.1;
+                  proxy_set_header Host $http_host;
+                  proxy_set_header X-Real-IP $remote_addr;
+                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                  proxy_set_header X-Forwarded-Proto https;
+                  proxy_set_header Upgrade $http_upgrade;
+                  proxy_set_header Connection $awx_connection_upgrade;
+                  proxy_read_timeout 300s;
+                  proxy_redirect off;
+              }
+          }
+      notify: Reload nginx
+
+    - name: Enable the AWX TLS site
+      ansible.builtin.file:
+        src: /etc/nginx/sites-available/awx-tls.conf
+        dest: /etc/nginx/sites-enabled/awx-tls.conf
+        state: link
+
+    - name: Test nginx configuration  # noqa no-changed-when
+      ansible.builtin.command: nginx -t
+
+  handlers:
+    - name: Reload nginx
+      ansible.builtin.systemd:
+        name: nginx
+        state: reloaded
+        enabled: true

scripts/awx_create_job_templates.py

@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+"""
+awx_create_job_templates.py
+
+Run this AFTER committing and pushing the KNELIAC code to the git remote.
+It will:
+  1. Trigger a project sync in AWX (pulls latest code from git)
+  2. Wait for the sync to complete
+  3. Create the two job templates (Hello World + Setup New System)
+  4. Create the inventory source (auto-syncs hosts from inventory/hosts.yml)
+
+Usage:
+  export AWX_PASSWORD='...'
+  python3 awx_create_job_templates.py
+
+Configuration (environment variables):
+  AWX_URL       AWX API base URL (default: https://tsys-awx.knel.net/api/v2)
+  AWX_USER      AWX admin username   (default: admin)
+  AWX_PASSWORD  AWX admin password   (REQUIRED - never committed to the repo)
+  AWX_PROJECT_ID, AWX_INVENTORY_ID, AWX_SCM_CRED_ID,
+  AWX_MACHINE_CRED_ID, AWX_ORG_ID  resource IDs (sensible defaults below)
+  AWX_VERIFY_TLS  "false" to skip TLS verification (for self-signed certs).
+                  Default: verify against the system trust store.
+  AWX_CA_BUNDLE   Path to a CA bundle file (e.g. internal CA) to verify against.
+"""
+
+import os
+import ssl
+import urllib.request, urllib.parse, json, base64, time, sys
+
+API = os.environ.get("AWX_URL", "https://tsys-awx.knel.net/api/v2")
+_AWX_USER = os.environ.get("AWX_USER", "admin")
+_AWX_PASS = os.environ.get("AWX_PASSWORD")
+if not _AWX_PASS:
+    sys.exit("ERROR: AWX_PASSWORD environment variable is not set. "
+             "Refusing to run — no credentials are stored in this repo.")
+AUTH = base64.b64encode(f"{_AWX_USER}:{_AWX_PASS}".encode()).decode()
+PROJECT_ID = int(os.environ.get("AWX_PROJECT_ID", "10"))
+INVENTORY_ID = int(os.environ.get("AWX_INVENTORY_ID", "2"))
+SCM_CRED_ID = int(os.environ.get("AWX_SCM_CRED_ID", "3"))
+MACHINE_CRED_ID = int(os.environ.get("AWX_MACHINE_CRED_ID", "4"))
+ORG_ID = int(os.environ.get("AWX_ORG_ID", "1"))
+
+# TLS verification context. With a self-signed cert (until the internal CA
+# bundle is distributed), set AWX_VERIFY_TLS=false. With an internal CA, point
+# AWX_CA_BUNDLE at it and leave verification enabled.
+if os.environ.get("AWX_VERIFY_TLS", "true").lower() == "false":
+    _SSL_CONTEXT = ssl._create_unverified_context()
+elif os.environ.get("AWX_CA_BUNDLE"):
+    _SSL_CONTEXT = ssl.create_default_context(cafile=os.environ["AWX_CA_BUNDLE"])
+else:
+    _SSL_CONTEXT = None  # use urllib's default (system trust store)
+
+def awx_req(method, endpoint, data=None):
+    url = f"{API}/{endpoint}"
+    body = json.dumps(data).encode() if data else None
+    req = urllib.request.Request(url, data=body, method=method)
+    req.add_header("Authorization", f"Basic {AUTH}")
+    req.add_header("Content-Type", "application/json")
+    try:
+        with urllib.request.urlopen(req, timeout=30, context=_SSL_CONTEXT) as resp:
+            raw = resp.read()
+            return json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as e:
+        err = e.read().decode()
+        print(f"  ERROR {e.code}: {err[:300]}")
+        return None
+
+def find_or_create(resource, name, payload):
+    encoded = urllib.parse.quote(name)
+    existing = awx_req("GET", f"{resource}/?name={encoded}")
+    if existing and existing.get("count", 0) > 0:
+        item = existing["results"][0]
+        print(f"  EXISTS: [{item['id']}] {name}")
+        return item
+    result = awx_req("POST", f"{resource}/", payload)
+    if result:
+        print(f"  CREATED: [{result.get('id')}] {name}")
+    return result
+
+# Step 1: Sync the project
+print("=" * 60)
+print("STEP 1: Syncing project (pulling latest code from git)")
+print("=" * 60)
+sync = awx_req("POST", f"projects/{PROJECT_ID}/update/", {})
+if not sync:
+    print("FAILED to trigger sync")
+    sys.exit(1)
+update_id = sync.get("id")
+print(f"  Sync started: project_updates/{update_id}")
+print("  Waiting...", end="", flush=True)
+for i in range(30):
+    time.sleep(3)
+    print(".", end="", flush=True)
+    status = awx_req("GET", f"project_updates/{update_id}/")
+    s = status.get("status") if status else "?"
+    if s in ("successful", "failed", "error", "canceled"):
+        print(f" {s}")
+        break
+else:
+    print(" timeout")
+    sys.exit(1)
+
+if s != "successful":
+    print(f"  Project sync failed: {s}")
+    print("  Make sure the code is committed and pushed to the git remote.")
+    sys.exit(1)
+
+# Step 2: Verify playbooks are available
+print("\n" + "=" * 60)
+print("STEP 2: Verifying playbooks")
+print("=" * 60)
+playbooks = awx_req("GET", f"projects/{PROJECT_ID}/playbooks/")
+if playbooks:
+    print(f"  Found {len(playbooks)} playbook(s):")
+    for p in playbooks:
+        print(f"    - {p}")
+else:
+    print("  No playbooks found! Push your code and re-run.")
+    sys.exit(1)
+
+# Step 3: Create inventory source
+print("\n" + "=" * 60)
+print("STEP 3: Creating inventory source")
+print("=" * 60)
+find_or_create("inventory_sources", "KNELIAC Fleet Source", {
+    "name": "KNELIAC Fleet Source",
+    "description": "Auto-synced from inventory/hosts.yml",
+    "inventory": INVENTORY_ID,
+    "source": "scm",
+    "source_project": PROJECT_ID,
+    "source_path": "inventory/hosts.yml",
+    "update_on_launch": True,
+    "overwrite": True,
+    "overwrite_vars": True,
+})
+
+# Step 4: Create job templates
+print("\n" + "=" * 60)
+print("STEP 4: Creating job templates")
+print("=" * 60)
+
+find_or_create("job_templates", "KNELIAC - Hello World", {
+    "name": "KNELIAC - Hello World",
+    "description": "AWX smoke test - verify connectivity to managed hosts",
+    "organization": ORG_ID,
+    "inventory": INVENTORY_ID,
+    "project": PROJECT_ID,
+    "playbook": "playbooks/hello_world.yml",
+    "credential": MACHINE_CRED_ID,
+    "verbosity": 1,
+    "forks": 5,
+    "ask_limit_on_launch": True,
+})
+
+find_or_create("job_templates", "KNELIAC - Setup New System", {
+    "name": "KNELIAC - Setup New System",
+    "description": "Full host provisioning (port of SetupNewSystem.sh)",
+    "organization": ORG_ID,
+    "inventory": INVENTORY_ID,
+    "project": PROJECT_ID,
+    "playbook": "playbooks/setup_new_system.yml",
+    "credential": MACHINE_CRED_ID,
+    "verbosity": 1,
+    "forks": 5,
+    "ask_limit_on_launch": True,
+    "ask_variables_on_launch": True,
+})
+
+print("\n" + "=" * 60)
+print("ALL DONE - AWX is ready!")
+print("=" * 60)
+print("""
+AWX URL: https://tsys-awx.knel.net (or https://100.91.39.53)
+Login:   $AWX_USER (password supplied via $AWX_PASSWORD env var)
+Note:    Self-signed cert for now — set AWX_VERIFY_TLS=false until the
+         internal CA bundle is distributed (then use AWX_CA_BUNDLE).
+
+Resources created:
+  Credentials:
+    [3] KNELIAC Git Credential  (Source Control - for git access)
+    [4] KNELIAC Host SSH Key    (Machine - for fleet host access)
+  Project:
+    [10] KNELIAC                 (git repo, auto-syncs on launch)
+  Inventory:
+    [2] KNELIAC Fleet           (auto-syncs hosts from inventory/hosts.yml)
+  Job Templates:
+    - KNELIAC - Hello World     (smoke test, run first!)
+    - KNELIAC - Setup New System (full provisioning)
+""")