# Project Status

> **Redmine is the system of record for all work tracking.**
> https://projects.knownelement.com — Version: **Potential to Kinetic Ready** (due 2026-09-30)
>
> This file is now a **read-only snapshot + ticket index**. Do not add
> checklists or status items here. All work is tracked as Redmine tickets.
> When work is done, update the ticket (done ratio, notes, close).

> **Last updated:** 2026-08-06 — Pi-hole hardening config committed (`netinfra/pihole/`, [#376]).

## Current State: MIXED (infra stable; k8s needs rebuild)

Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack
are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum).

## Completed Work (tickets closed @ 100%)

| Ticket | Component | Code location |
|--------|-----------|---------------|
| [#334](https://projects.knownelement.com/issues/334) | NTP synchronization (redundant stratum 3) | `netinfra/` |
| [#340](https://projects.knownelement.com/issues/340) | UPS management — NUT on pfv-tsys1 (APC Smart-UPS) | `dcinfra/ups/` |
| [#357](https://projects.knownelement.com/issues/357) | DNS cluster — Technitium primary/secondary + DHCP failover | `netinfra/dns-cluster-setup/` |
| [#358](https://projects.knownelement.com/issues/358) | Proxmox perf tuning (5 of 7 hosts) | `proxmox/perf/` |
| [#359](https://projects.knownelement.com/issues/359) | PDU management — powerman on pfv-tsys1 | `dcinfra/powerman/` |
| [#360](https://projects.knownelement.com/issues/360) | Console management — ser2net + conman on pfv-tsys4 | `dcinfra/console/` |
| [#361](https://projects.knownelement.com/issues/361) | AWX deployment — k3s + AWX Operator | `awx/` |
| [#362](https://projects.knownelement.com/issues/362) | k8s control plane build — k3s HA (initial, now needs rebuild) | `k8s/` |
| [#363](https://projects.knownelement.com/issues/363) | tsys5 storage configuration (NVMe + SSD) | `proxmox/docs/` |
| [#364](https://projects.knownelement.com/issues/364) | tsys4 PVE upgrade to 9.2.5 | `proxmox/docs/` |
| [#365](https://projects.knownelement.com/issues/365) | Repo merge — KNELServerBuild into PFVCluster | repo root |
| [#366](https://projects.knownelement.com/issues/366) | Server provisioning pipeline (SetupNewSystem.sh → KNELIAC) | `archive/provisioning/`, KNELIAC |

## Open Work (tickets in progress or queued)

| Ticket | Component | Notes |
|--------|-----------|-------|
| [#376](https://projects.knownelement.com/issues/376) | DNS cluster up/down alerts + Pi-hole hardening committed | Config now in `netinfra/pihole/`; flapping source under investigation |
| [#367](https://projects.knownelement.com/issues/367) | k8s cluster rebuild (cnode wiped, no quorum) | Blocks #368 |
| [#368](https://projects.knownelement.com/issues/368) | k8s worker nodes join cluster | Blocked by #367 |
| [#369](https://projects.knownelement.com/issues/369) | Switch config fixes (core-sw01 LAG, LACP, member list) | Onsite Friday |
| [#370](https://projects.knownelement.com/issues/370) | Vault — migrate all API keys (HIGH priority) | |
| [#371](https://projects.knownelement.com/issues/371) | AWX HTTPS/TLS configuration | Depends on #342 (PKI/SSL) |
| [#372](https://projects.knownelement.com/issues/372) | Tripp Lite UPS fix (physical reseat) | Onsite Friday |
| [#373](https://projects.knownelement.com/issues/373) | pfv-r2-sw console fix (physical, onsite) | Onsite Friday |
| [#374](https://projects.knownelement.com/issues/374) | PDU outlet renaming | |
| [#375](https://projects.knownelement.com/issues/375) | Monitoring expansion (unpoller, netdisco, weathermap, smokeping) | Relates to #337, #343 |
| [#311](https://projects.knownelement.com/issues/311) | OAM: Security Hardening (CMMC/STIG) | KNELIAC roles deployed, expansion ongoing |
| [#335](https://projects.knownelement.com/issues/335) | OAM: Centralized syslog to Wazuh | Wazuh agents deployed, fleet rollout pending |
| [#342](https://projects.knownelement.com/issues/342) | OAM: PKI/SSL — tsys-ca VM + HSM CA + ACME | Vault unconfigured, AWX HTTP-only |
| [#343](https://projects.knownelement.com/issues/343) | OAM: Monitoring coverage matrix | Coverage gaps documented |

## Known Issues (tracked in tickets)

All known infrastructure issues are captured in ticket descriptions and notes.
See [#369](https://projects.knownelement.com/issues/369) for switch issues,
[#372](https://projects.knownelement.com/issues/372) for Tripp Lite UPS,
[#373](https://projects.knownelement.com/issues/373) for dead console port.

## Infrastructure Summary

| Component | Details |
|-----------|---------|
| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) |
| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + NVMe) |
| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3) |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| NTP | pfv-netinfra-01/02 (redundant, stratum 3) |
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
| Backup | Proxmox Backup Server (PBS) |
| k8s control plane | **DOWN** — 3-node k3s HA, needs rebuild ([#367](https://projects.knownelement.com/issues/367)) |
| PDU | Cyclades PM10i via powerman on pfv-tsys1 (port 10101) |
| AWX | tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80) |
| Console | 7 switch consoles via ser2net + conman on pfv-tsys4 |
| UPS | APC Smart-UPS C 1500 via NUT on pfv-tsys1 |
| Production | Cloudron VPS, Reston, VA (this cluster is R&D only) |
| Config management | KNELIAC (Ansible) — `/home/reachableceo/projects/KNELIAC` |