KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: end-to-end gardening — links, stale refs, tailscale.md update

Browse Source 
Comprehensive documentation gardening across the merged repo:

- tailscale.md: fully rewritten with current ground truth. The netinfra
  pair now runs production Technitium with all knel.net records
  replicated. Both LAN IPs resolve knel.net device names and recurse
  externally. The old "NXDOMAIN / zone is stale" findings are replaced
  with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
  to all key scripts and docs. Autonomous commit/push policy
  prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
  updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
  "PFVCluster" in actionable docs; historical AI-review docs tagged
  with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
  docs/SECURITY.md: path references fixed to current structure.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
mrcharles
2026-07-28 11:28:52 -05:00
parent c2b592a66b
commit a7fbad60ab
21 changed files with 224 additions and 295 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+46 -22
View File
@@ -1,10 +1,10 @@
# Agent Guidelines # Agent Guidelines
This repo combines two formerly-separate projects: This repo combines two formerly-separate projects:
- **Server provisioning** (formerly KNELServerBuild): `provisioning/`, `tests/`, - **Server provisioning** (formerly KNELServerBuild): [`provisioning/`](provisioning/),
`vendor/`, `dns-cluster-setup/` [`tests/`](tests/), [`vendor/`](vendor/), [`dns-cluster-setup/`](dns-cluster-setup/)
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): `perf/`, `netinfra/`, - **Proxmox cluster ops** (formerly PFVCluster/perfopt): [`perf/`](perf/),
`switches/`, `returned-logs/` [`netinfra/`](netinfra/), [`switches/`](switches/)
## Repository Layout ## Repository Layout
@@ -16,16 +16,19 @@ This repo combines two formerly-separate projects:
via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
depend on the current working directory. Run from anywhere. depend on the current working directory. Run from anywhere.
- **Local config files are the source of truth**: Configs in - **Local config files are the source of truth**: Configs in
`provisioning/ConfigFiles/` are read with `cat`/`cp`. Do NOT re-introduce [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
`curl ${DL_ROOT}/...` downloads — that CDN is deprecated. `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads — that CDN
is deprecated.
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh` - **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
checkers must skip these. checkers must skip these.
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox - **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
Datacenter Manager). SSH keys deployed to root on all hosts. Datacenter Manager). SSH keys deployed to `localuser` with passwordless sudo
on all hosts.
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment. - **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
Use the wrapper scripts: `tests/remote.sh`, `dns-cluster-setup/remote-dns.sh`, Use the wrapper scripts: [`tests/remote.sh`](tests/remote.sh),
or the `deploy-check.sh` / `deploy-tuning.sh` patterns. [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh),
or the `perf/deploy-check.sh` / `perf/deploy-tuning.sh` patterns.
## Git Commit Requirements ## Git Commit Requirements
@@ -37,27 +40,48 @@ This repo combines two formerly-separate projects:
## Autonomous Git Workflow ## Autonomous Git Workflow
Agents are authorized to commit AND push autonomously. After each logical unit **Agents are authorized to commit AND push autonomously — do not wait to be
of work: stage, commit, push to `origin/main`. Group changes so each commit is asked.** After each logical unit of work:
coherent on its own.
1. Stage only the files belonging to that logical change.
2. Commit with a conventional, well-formed message.
3. Push to `origin` (`git push`). The branch tracks `origin/main`.
4. Repeat per logical unit.
Group changes so each commit is coherent on its own (a reader should
understand the commit without seeing the others). Never batch unrelated
changes into one commit.
## Key scripts ## Key scripts
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| `provisioning/SetupNewSystem.sh` | Full server provisioning (packages, hardening, 2FA) | | [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning (packages, hardening, 2FA) |
| `tests/vm-validation.sh` | End-to-end deploy + validate on sandbox VM | | [`tests/vm-validation.sh`](tests/vm-validation.sh) | End-to-end deploy + validate on sandbox VM |
| `tests/run-tests.sh` | Project test suite (unit/security/validation) | | [`tests/run-tests.sh`](tests/run-tests.sh) | Project test suite (unit/security/validation) |
| `dns-cluster-setup/setup.sh` | Technitium DNS cluster replication | | [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | Technitium DNS cluster replication |
| `perf/deploy-check.sh` | Deploy read-only data collector to hosts | | [`perf/deploy-check.sh`](perf/deploy-check.sh) | Deploy read-only data collector to hosts |
| `perf/deploy-tuning.sh` | Deploy perf tunings to hosts | | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings to hosts |
| `perf/validate-fixes.sh` | Validate applied tuning changes | | [`perf/validate-fixes.sh`](perf/validate-fixes.sh) | Validate applied tuning changes |
| `perf/iperf-full-matrix.sh` | Full iperf throughput suite | | [`perf/iperf-full-matrix.sh`](perf/iperf-full-matrix.sh) | Full iperf throughput suite |
## Key documentation
| Doc | Contents |
|-----|----------|
| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
## Project context ## Project context
This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
budget. Redundancy is not a concern for the R&D cluster. Backups DO matter budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See (PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
`docs/PROJECT.md` for the comprehensive fleet report and `docs/TODO.md` for [`docs/PROJECT.md`](docs/PROJECT.md) for the comprehensive fleet report and
pending hardware work. [`docs/TODO.md`](docs/TODO.md) for pending hardware work.
README.md
+18 -17
View File
@@ -5,17 +5,16 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure
## Directory Structure ## Directory Structure
``` | Directory | Description |
provisioning/ Server provisioning (SetupNewSystem.sh, security hardening, |-----------|-------------|
2FA, NTP/DNS config, SNMP, Dell OMSA) | [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) |
tests/ Test suite + VM validation harness | [`tests/`](tests/) | Test suite + VM validation harness |
dns-cluster-setup/ Technitium DNS cluster replication scripts | [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts |
perf/ Proxmox performance tuning, fleet audit, iperf, switch diagnostics | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
netinfra/ pfv-netinfra-01/02 DNS/NTP setup + audit scripts | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
switches/ Switch configuration captures | [`switches/`](switches/) | Switch configuration captures |
docs/ All documentation (PROJECT.md, SECURITY.md, tailscale.md, etc.) | [`docs/`](docs/) | All documentation |
vendor/ Vendored KNELShellFramework | [`vendor/`](vendor/) | Vendored KNELShellFramework |
```
## Quick Start ## Quick Start
@@ -54,12 +53,14 @@ cd perf/
| Doc | Contents | | Doc | Contents |
|-----|----------| |-----|----------|
| `docs/PROJECT.md` | Comprehensive fleet report (7 hosts, VM inventory, storage) | | [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| `docs/SECURITY.md` | Security architecture and hardening details | | [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| `docs/tailscale.md` | Tailscale vs managed DNS analysis | | [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| `docs/DEPLOYMENT.md` | Deployment procedures | | [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| `docs/TODO.md` | Pending hardware work (tsys2/4/5) | | [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| `dns-cluster-setup/README.md` | DNS cluster setup guide | | [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
## Architecture ## Architecture
dns-cluster-setup/README.md
+3 -3
View File
@@ -177,7 +177,7 @@ dig @192.168.3.253 pfv-netinfra-01.knel.net
# Both should return the same answer. # Both should return the same answer.
``` ```
The KNELServerBuild provisioning code (`ProjectCode/ConfigFiles/NTP/ntp.conf` The KNELServerBuild provisioning code (`provisioning/ConfigFiles/NTP/ntp.conf`
and `ProjectCode/ConfigFiles/Resolv/resolv.conf`) points clients at both and `provisioning/ConfigFiles/Resolv/resolv.conf`) points clients at both
servers for DNS and NTP redundancy. See `ProjectDocs/tailscale.md` for the servers for DNS and NTP redundancy. See `docs/tailscale.md` for the
full DNS architecture analysis. full DNS architecture analysis.
docs/AIReview-QWEN.md
+6 -5
View File
@@ -1,4 +1,5 @@
# AI Review: KNELServerBuild (FetchApply) Project <!-- Historical AI-generated review. Paths updated to current structure. -->
# AI Review: KNELServerBuild (PFVCluster) Project
## Executive Summary ## Executive Summary
@@ -6,7 +7,7 @@ The KNELServerBuild project is a comprehensive Infrastructure-as-Code (IaC) solu
## Project Overview ## Project Overview
The FetchApply project is a shell-based automation framework that provisions Linux servers with: The PFVCluster project is a shell-based automation framework that provisions Linux servers with:
- Security hardening (SSH, 2FA, Wazuh, STIG compliance) - Security hardening (SSH, 2FA, Wazuh, STIG compliance)
- Operational monitoring (LibreNMS, cockpit, SNMP) - Operational monitoring (LibreNMS, cockpit, SNMP)
- System packages and configurations for enterprise operations - System packages and configurations for enterprise operations
@@ -15,10 +16,10 @@ The FetchApply project is a shell-based automation framework that provisions Lin
## Architecture and Structure ## Architecture and Structure
### Key Components ### Key Components
- **ProjectCode/**: Main setup and configuration scripts - **provisioning/**: Main setup and configuration scripts
- **Project-ConfigFiles/**: Configuration variables and parameters - **Project-ConfigFiles/**: Configuration variables and parameters
- **Project-Includes/**: Reusable shell functions and utilities - **Project-Includes/**: Reusable shell functions and utilities
- **Project-Tests/**: Comprehensive testing framework - **tests/**: Comprehensive testing framework
- **Modules/**: Functional modules for security, operations, etc. - **Modules/**: Functional modules for security, operations, etc.
- **vendor/**: External dependencies and frameworks - **vendor/**: External dependencies and frameworks
@@ -134,6 +135,6 @@ The `SetupNewSystem.sh` orchestrates:
## Conclusion ## Conclusion
The FetchApply project represents a solid foundation for automated server provisioning with good security practices and testing. However, there are significant opportunities to improve security, maintainability, and operational resilience. Prioritizing security improvements and configuration management would provide the greatest value to the project's stability and long-term viability. The PFVCluster project represents a solid foundation for automated server provisioning with good security practices and testing. However, there are significant opportunities to improve security, maintainability, and operational resilience. Prioritizing security improvements and configuration management would provide the greatest value to the project's stability and long-term viability.
The modular architecture and comprehensive testing framework provide a strong foundation for future enhancements and improvements. The modular architecture and comprehensive testing framework provide a strong foundation for future enhancements and improvements.
docs/AiOverview-Gemini.md
+2 -1
View File
@@ -1,3 +1,4 @@
<!-- Historical AI-generated review. Paths updated to current structure. -->
# AI Overview of KNELServerBuild # AI Overview of KNELServerBuild
This is an AI-generated overview of the KNELServerBuild project. The analysis is based on a read-only review of the project's files. This is an AI-generated overview of the KNELServerBuild project. The analysis is based on a read-only review of the project's files.
@@ -6,7 +7,7 @@ This is an AI-generated overview of the KNELServerBuild project. The analysis is
The KNELServerBuild project is an Infrastructure as Code (IAC) repository for provisioning and configuring Linux servers. It is based on a collection of bash scripts that automate the installation of packages, configuration of services, and security hardening of the system. The project is designed to be used with the `FetchApply` tool, which is not included in this repository. The KNELServerBuild project is an Infrastructure as Code (IAC) repository for provisioning and configuring Linux servers. It is based on a collection of bash scripts that automate the installation of packages, configuration of services, and security hardening of the system. The project is designed to be used with the `FetchApply` tool, which is not included in this repository.
The main entry point of the project is the `ProjectCode/SetupNewSystem.sh` script. This script performs the following actions: The main entry point of the project is the `provisioning/SetupNewSystem.sh` script. This script performs the following actions:
* **Initializes the environment:** Sets up project paths and sources a shell framework (`KNELShellFramework`) and project-specific includes. * **Initializes the environment:** Sets up project paths and sources a shell framework (`KNELShellFramework`) and project-specific includes.
* **Installs packages:** Installs a wide range of packages, including monitoring agents (check_mk, snmp), security tools (auditd, aide, lynis, clamav), administration tools (cockpit, webmin), and common utilities (tmux, vim, zsh). * **Installs packages:** Installs a wide range of packages, including monitoring agents (check_mk, snmp), security tools (auditd, aide, lynis, clamav), administration tools (cockpit, webmin), and common utilities (tmux, vim, zsh).
docs/AiOverview-OpenCode.md
+1
View File
@@ -1,3 +1,4 @@
<!-- Historical AI-generated review. Paths updated to current structure. -->
# AI Overview: KNEL Server Build (FetchApply) Project # AI Overview: KNEL Server Build (FetchApply) Project
**Date:** December 26, 2025 **Date:** December 26, 2025
docs/AiSecurityAudit-Gemini.md
+5 -3
View File
@@ -1,3 +1,5 @@
<!-- Historical AI-generated security review. Paths updated where actionable. -->
<!-- Historical AI-generated review. Paths updated to current structure. -->
# AI Security Audit of KNELServerBuild # AI Security Audit of KNELServerBuild
This is an AI-generated security audit of the KNELServerBuild project. The analysis is based on a read-only review of the project's files. This is an AI-generated security audit of the KNELServerBuild project. The analysis is based on a read-only review of the project's files.
@@ -8,7 +10,7 @@ The KNELServerBuild project has a good security posture overall, but there are a
### High-Risk Findings ### High-Risk Findings
* **SSH Authorized Keys in Repository:** The `ProjectCode/ConfigFiles/SSH/AuthorizedKeys` directory contains SSH authorized keys for the `localuser` and `root` users. This is a security risk, as it allows anyone with access to the repository to know which public keys are authorized to access the servers. * **SSH Authorized Keys in Repository:** The `provisioning/ConfigFiles/SSH/AuthorizedKeys` directory contains SSH authorized keys for the `localuser` and `root` users. This is a security risk, as it allows anyone with access to the repository to know which public keys are authorized to access the servers.
### Medium-Risk Findings ### Medium-Risk Findings
@@ -16,12 +18,12 @@ The KNELServerBuild project has a good security posture overall, but there are a
### Low-Risk Findings ### Low-Risk Findings
* **Potential for Password on Command Line:** The `ProjectCode/Agents/librenms/mysql.sh` script has a `--pass` argument for a MySQL password. This is a potential security risk if the password is provided on the command line, as it could be logged in the shell history. * **Potential for Password on Command Line:** The `provisioning/Agents/librenms/mysql.sh` script has a `--pass` argument for a MySQL password. This is a potential security risk if the password is provided on the command line, as it could be logged in the shell history.
## Recommendations ## Recommendations
* **Remove SSH Authorized Keys from Repository:** The SSH authorized keys should be removed from the repository and managed using a secrets management tool like HashiCorp Vault or AWS Secrets Manager. * **Remove SSH Authorized Keys from Repository:** The SSH authorized keys should be removed from the repository and managed using a secrets management tool like HashiCorp Vault or AWS Secrets Manager.
* **Use Variables for Hostnames:** The hardcoded hostnames should be replaced with variables that are defined in a central configuration file. This will make it easier to update the hostnames if they change. * **Use Variables for Hostnames:** The hardcoded hostnames should be replaced with variables that are defined in a central configuration file. This will make it easier to update the hostnames if they change.
* **Avoid Passwords on Command Line:** The `ProjectCode/Agents/librenms/mysql.sh` script should be modified to avoid passing the MySQL password on the command line. For example, the script could prompt the user for the password or read it from a configuration file. * **Avoid Passwords on Command Line:** The `provisioning/Agents/librenms/mysql.sh` script should be modified to avoid passing the MySQL password on the command line. For example, the script could prompt the user for the password or read it from a configuration file.
Overall, the KNELServerBuild project is a good starting point for an IAC repository. By addressing the security risks identified in this audit, the project can be made more secure and reliable. Overall, the KNELServerBuild project is a good starting point for an IAC repository. By addressing the security risks identified in this audit, the project can be made more secure and reliable.
docs/CODE-REVIEW-FINDINGS.md
+10 -9
View File
@@ -1,4 +1,5 @@
# TSYS FetchApply Code Review Findings <!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
# TSYS PFVCluster Code Review Findings
**Review Date:** July 14, 2025 **Review Date:** July 14, 2025
**Reviewer:** Claude (Anthropic) **Reviewer:** Claude (Anthropic)
@@ -11,7 +12,7 @@ The repository shows good architectural structure with centralized framework com
## Critical Issues (High Priority) ## Critical Issues (High Priority)
### 1. Package Installation Performance ⚠️ ### 1. Package Installation Performance ⚠️
**Location:** `ProjectCode/SetupNewSystem.sh:27` and `Lines 117-183` **Location:** `provisioning/SetupNewSystem.sh:27` and `Lines 117-183`
**Issue:** Multiple separate package installation commands causing performance bottlenecks **Issue:** Multiple separate package installation commands causing performance bottlenecks
```bash ```bash
# Current inefficient pattern # Current inefficient pattern
@@ -23,7 +24,7 @@ DEBIAN_FRONTEND="noninteractive" apt-get -qq --yes install virt-what auditd ...
**Fix:** Combine all package installations into single command **Fix:** Combine all package installations into single command
### 2. Network Operations Lack Error Handling 🔴 ### 2. Network Operations Lack Error Handling 🔴
**Location:** `ProjectCode/SetupNewSystem.sh:61-63`, multiple modules **Location:** `provisioning/SetupNewSystem.sh:61-63`, multiple modules
**Issue:** curl commands without timeout or error handling **Issue:** curl commands without timeout or error handling
```bash ```bash
# Vulnerable pattern # Vulnerable pattern
@@ -33,7 +34,7 @@ curl --silent ${DL_ROOT}/path/file >/etc/config
**Fix:** Add timeout, error handling, and retry logic **Fix:** Add timeout, error handling, and retry logic
### 3. Unquoted Variable Expansions 🔴 ### 3. Unquoted Variable Expansions 🔴
**Location:** Multiple files, including `ProjectCode/SetupNewSystem.sh:244` **Location:** Multiple files, including `provisioning/SetupNewSystem.sh:244`
**Issue:** Variables used without proper quoting creating security risks **Issue:** Variables used without proper quoting creating security risks
```bash ```bash
# Risky pattern # Risky pattern
@@ -57,7 +58,7 @@ chsh -s $(which zsh) root
## Performance Optimization Opportunities ## Performance Optimization Opportunities
### 6. Individual File Downloads 🟡 ### 6. Individual File Downloads 🟡
**Location:** `ProjectCode/Modules/Security/secharden-scap-stig.sh:66-77` **Location:** `provisioning/Modules/Security/secharden-scap-stig.sh:66-77`
**Issue:** 12+ individual curl commands for config files **Issue:** 12+ individual curl commands for config files
```bash ```bash
curl --silent ${DL_ROOT}/path1 > /etc/file1 curl --silent ${DL_ROOT}/path1 > /etc/file1
@@ -171,9 +172,9 @@ function download_configurations() {
print_info "Downloading configuration files..." print_info "Downloading configuration files..."
local -A configs=( local -A configs=(
["${DL_ROOT}/ProjectCode/ConfigFiles/ZSH/tsys-zshrc"]="/etc/zshrc" ["${DL_ROOT}/provisioning/ConfigFiles/ZSH/tsys-zshrc"]="/etc/zshrc"
["${DL_ROOT}/ProjectCode/ConfigFiles/SMTP/aliases"]="/etc/aliases" ["${DL_ROOT}/provisioning/ConfigFiles/SMTP/aliases"]="/etc/aliases"
["${DL_ROOT}/ProjectCode/ConfigFiles/Syslog/rsyslog.conf"]="/etc/rsyslog.conf" ["${DL_ROOT}/provisioning/ConfigFiles/Syslog/rsyslog.conf"]="/etc/rsyslog.conf"
) )
for url in "${!configs[@]}"; do for url in "${!configs[@]}"; do
@@ -261,7 +262,7 @@ function test_network_error_handling() {
## Conclusion ## Conclusion
The TSYS FetchApply repository has a solid foundation but requires systematic improvements to meet production reliability standards. The recommended fixes will significantly enhance: The TSYS PFVCluster repository has a solid foundation but requires systematic improvements to meet production reliability standards. The recommended fixes will significantly enhance:
- **Deployment reliability** through better error handling - **Deployment reliability** through better error handling
- **Security posture** through integrity verification - **Security posture** through integrity verification
docs/Claude-Review.md
+2 -1
View File
@@ -1,4 +1,5 @@
# Claude Code Review - TSYS FetchApply Infrastructure <!-- Historical AI-generated review. Paths updated to current structure. -->
# Claude Code Review - TSYS PFVCluster Infrastructure
**Review Date:** July 14, 2025 (Updated) **Review Date:** July 14, 2025 (Updated)
**Reviewed by:** Claude (Anthropic) **Reviewed by:** Claude (Anthropic)
docs/DEPLOYMENT.md
+3 -3
View File
@@ -77,7 +77,7 @@ git clone [repository-url]
cd FetchApply cd FetchApply
# 2. Review configuration # 2. Review configuration
cat ProjectCode/SetupNewSystem.sh cat provisioning/SetupNewSystem.sh
# 3. Execute with manual review # 3. Execute with manual review
cd ProjectCode cd ProjectCode
@@ -256,7 +256,7 @@ cd FetchApply
git pull origin main git pull origin main
# Re-run specific modules # Re-run specific modules
cd ProjectCode/Modules/Security cd provisioning/Modules/Security
sudo bash secharden-ssh.sh sudo bash secharden-ssh.sh
``` ```
@@ -291,7 +291,7 @@ sudo bash secharden-ssh.sh
### Documentation ### Documentation
- **README.md:** Basic usage instructions - **README.md:** Basic usage instructions
- **SECURITY.md:** Security architecture and guidelines - **SECURITY.md:** Security architecture and guidelines
- **Project-Tests/README.md:** Testing framework documentation - **tests/README.md:** Testing framework documentation
### Community Support ### Community Support
- **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline - **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
docs/DEVELOPMENT-GUIDELINES.md
+3 -2
View File
@@ -1,8 +1,9 @@
# TSYS FetchApply Development Guidelines <!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
# TSYS PFVCluster Development Guidelines
## Overview ## Overview
This document contains development standards and best practices for the TSYS FetchApply infrastructure provisioning system. This document contains development standards and best practices for the TSYS PFVCluster infrastructure provisioning system.
## Package Management Best Practices ## Package Management Best Practices
docs/K8S.md
+1 -1
View File
@@ -2,7 +2,7 @@
**Date:** 2026-07-27 **Date:** 2026-07-27
**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster. **Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
Companion to `PROJECT.md` (which has the fleet-wide assessment). Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment).
**Status:** For discussion in a future session. No changes made. **Status:** For discussion in a future session. No changes made.
--- ---
docs/PROJECT.md
+3 -3
View File
@@ -391,7 +391,7 @@ be on tsys5:
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via **Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK. PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
### 5.5 Future k8s architecture (next session -- see K8S.md) ### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
The k8s layer will be tackled soon. Key requirements from user: The k8s layer will be tackled soon. Key requirements from user:
@@ -606,7 +606,7 @@ PCIe NIC would eliminate this risk (same recommendation as tsys4).
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps). tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
The switch LACP hash change took effect after renegotiation. The 56-106K The switch LACP hash change took effect after renegotiation. The 56-106K
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over- retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
LACP overhead. See `RESULTS.md` for the full analysis. LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
### 9.4 NFS nconnect=4 + noatime confirmed active ### 9.4 NFS nconnect=4 + noatime confirmed active
@@ -819,7 +819,7 @@ This belongs in the next budget cycle.
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded. 3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
4. Update PROJECT.md with post-hardware numbers. 4. Update PROJECT.md with post-hardware numbers.
### 12.4 Future: Kubernetes deep-dive (see K8S.md) ### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
Next major workstream. Requirements captured: Next major workstream. Requirements captured:
- **vcluster + Rancher** for multi-tenant k8s management - **vcluster + Rancher** for multi-tenant k8s management
docs/REFACTORING-EXAMPLES.md
+1
View File
@@ -1,3 +1,4 @@
<!-- Historical document: paths and patterns shown are pre-refactor. See provisioning/ for current code. -->
# Code Refactoring Examples # Code Refactoring Examples
This document provides specific examples of how to apply the code review findings to improve performance, security, and reliability. This document provides specific examples of how to apply the code review findings to improve performance, security, and reliability.
docs/SECURITY.md
+4 -4
View File
@@ -1,8 +1,8 @@
# TSYS FetchApply Security Documentation # PFVCluster Security Documentation
## Security Architecture ## Security Architecture
The TSYS FetchApply infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management. The PFVCluster infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management.
## Current Security Features ## Current Security Features
@@ -41,10 +41,10 @@ The TSYS FetchApply infrastructure provisioning system is designed with security
### Automated Security Validation ### Automated Security Validation
```bash ```bash
# Run security test suite # Run security test suite
./Project-Tests/run-tests.sh security ./tests/run-tests.sh security
# Specific security tests # Specific security tests
./Project-Tests/security/https-enforcement.sh ./tests/security/https-enforcement.sh
``` ```
### Security Test Categories ### Security Test Categories
docs/TSYS-2FA-GUIDE.md
+1 -1
View File
@@ -22,7 +22,7 @@ This guide provides complete instructions for implementing and managing two-fact
### Step 1: Run the 2FA Setup Script ### Step 1: Run the 2FA Setup Script
```bash ```bash
# Navigate to the security modules directory # Navigate to the security modules directory
cd ProjectCode/Modules/Security cd provisioning/Modules/Security
# Run the 2FA setup script as root # Run the 2FA setup script as root
sudo bash secharden-2fa.sh sudo bash secharden-2fa.sh
docs/charles-todo.md
+6 -6
View File
@@ -1,4 +1,4 @@
# Charles TODO - TSYS FetchApply Security Improvements # Charles TODO - PFVCluster Security Improvements
**Priority Order:** High → Medium → Low **Priority Order:** High → Medium → Low
**Target:** Address security vulnerabilities and operational improvements **Target:** Address security vulnerabilities and operational improvements
@@ -8,7 +8,7 @@
### ✅ 1. Replace Insecure Deployment Method - RESOLVED ### ✅ 1. Replace Insecure Deployment Method - RESOLVED
**Previous Issue:** `curl https://dl.knownelement.com/KNEL/FetchApply/SetupNewSystem.sh | bash` **Previous Issue:** `curl https://dl.knownelement.com/KNEL/FetchApply/SetupNewSystem.sh | bash`
**Status:** Fixed in README.md - now uses secure git clone approach **Status:** Fixed in README.md - now uses secure git clone approach
**Current Method:** `git clone this repo` → `cd FetchApply/ProjectCode` → `bash SetupNewSystem.sh` **Current Method:** `git clone this repo` → `cd PFVCluster/provisioning` → `bash SetupNewSystem.sh`
**Remaining considerations:** **Remaining considerations:**
- Consider implementing GPG signature verification for tagged releases - Consider implementing GPG signature verification for tagged releases
@@ -17,9 +17,9 @@
### ✅ 2. Enforce HTTPS for All Downloads - RESOLVED ### ✅ 2. Enforce HTTPS for All Downloads - RESOLVED
**Previous Issue:** HTTP URLs in Dell OMSA and some repository setups **Previous Issue:** HTTP URLs in Dell OMSA and some repository setups
**Status:** All HTTP URLs converted to HTTPS across: **Status:** All HTTP URLs converted to HTTPS across:
- `ProjectCode/Dell/Server/omsa.sh` - Ubuntu archive and Dell repo URLs - `provisioning/Dell/Server/omsa.sh` - Ubuntu archive and Dell repo URLs
- `ProjectCode/legacy/prox7.sh` - Proxmox download URLs - `provisioning/legacy/prox7.sh` - Proxmox download URLs
- `ProjectCode/Modules/RandD/sslStackFromSource.sh` - Apache source URLs - `provisioning/Modules/RandD/sslStackFromSource.sh` - Apache source URLs
**Remaining considerations:** **Remaining considerations:**
- SSL certificate validation is enabled by default in wget/curl - SSL certificate validation is enabled by default in wget/curl
@@ -35,7 +35,7 @@
- Add environment variable support for sensitive data - Add environment variable support for sensitive data
**Files to secure:** **Files to secure:**
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/` (entire directory) - `provisioning/ConfigFiles/SSH/AuthorizedKeys/` (entire directory)
- Hard-coded hostnames in various scripts - Hard-coded hostnames in various scripts
## 🔶 MEDIUM PRIORITY (Operational Security) ## 🔶 MEDIUM PRIORITY (Operational Security)
docs/claude-todo.md
+12 -12
View File
@@ -1,4 +1,4 @@
# Claude TODO - TSYS FetchApply Automation Tasks # Claude TODO - TSYS PFVCluster Automation Tasks
**Purpose:** Actionable items optimized for AI assistant implementation **Purpose:** Actionable items optimized for AI assistant implementation
**Priority:** Critical → High → Medium → Low **Priority:** Critical → High → Medium → Low
@@ -11,9 +11,9 @@
### ✅ RESOLVED: Replace HTTP URLs with HTTPS ### ✅ RESOLVED: Replace HTTP URLs with HTTPS
**Files modified:** **Files modified:**
- `ProjectCode/Dell/Server/omsa.sh` - Converted 11 HTTP URLs to HTTPS (Ubuntu archive, Dell repo) - `provisioning/Dell/Server/omsa.sh` - Converted 11 HTTP URLs to HTTPS (Ubuntu archive, Dell repo)
- `ProjectCode/legacy/prox7.sh` - Converted 2 HTTP URLs to HTTPS (Proxmox downloads) - `provisioning/legacy/prox7.sh` - Converted 2 HTTP URLs to HTTPS (Proxmox downloads)
- `ProjectCode/Modules/RandD/sslStackFromSource.sh` - Converted 3 HTTP URLs to HTTPS (Apache sources) - `provisioning/Modules/RandD/sslStackFromSource.sh` - Converted 3 HTTP URLs to HTTPS (Apache sources)
**Status:** All HTTP URLs in active scripts converted to HTTPS. Only remaining HTTP references are in comments and LibreNMS agent files (external dependencies). **Status:** All HTTP URLs in active scripts converted to HTTPS. Only remaining HTTP references are in comments and LibreNMS agent files (external dependencies).
@@ -39,7 +39,7 @@ function verify_download() {
``` ```
### TASK-003: Create Secure Deployment Script ### TASK-003: Create Secure Deployment Script
**Create:** `ProjectCode/SecureSetupNewSystem.sh` **Create:** `provisioning/SecureSetupNewSystem.sh`
**Features to implement:** **Features to implement:**
- GPG signature verification - GPG signature verification
- SHA256 checksum validation - SHA256 checksum validation
@@ -50,9 +50,9 @@ function verify_download() {
### TASK-004: Remove Hardcoded SSH Keys ### TASK-004: Remove Hardcoded SSH Keys
**Files to modify:** **Files to modify:**
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/root-ssh-authorized-keys` - `provisioning/ConfigFiles/SSH/AuthorizedKeys/root-ssh-authorized-keys`
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/localuser-ssh-authorized-keys` - `provisioning/ConfigFiles/SSH/AuthorizedKeys/localuser-ssh-authorized-keys`
- `ProjectCode/Modules/Security/secharden-ssh.sh:31,40,51` - `provisioning/Modules/Security/secharden-ssh.sh:31,40,51`
**Implementation approach:** **Implementation approach:**
1. Create environment variable support: `SSH_KEYS_URL` or `SSH_KEYS_VAULT_PATH` 1. Create environment variable support: `SSH_KEYS_URL` or `SSH_KEYS_VAULT_PATH`
@@ -119,7 +119,7 @@ function update_status() { } # Update current operation
``` ```
### TASK-012: Dry Run Mode ### TASK-012: Dry Run Mode
**Add to:** `ProjectCode/SetupNewSystem.sh` **Add to:** `provisioning/SetupNewSystem.sh`
**Implementation:** **Implementation:**
- `--dry-run` flag support - `--dry-run` flag support
- Preview of changes without execution - Preview of changes without execution
@@ -139,9 +139,9 @@ function update_status() { } # Update current operation
## File Location Patterns ## File Location Patterns
- **Framework components:** `Framework-Includes/*.sh` - **Framework components:** `Framework-Includes/*.sh`
- **Security modules:** `ProjectCode/Modules/Security/*.sh` - **Security modules:** `provisioning/Modules/Security/*.sh`
- **Configuration files:** `ProjectCode/ConfigFiles/*/` - **Configuration files:** `provisioning/ConfigFiles/*/`
- **Main entry point:** `ProjectCode/SetupNewSystem.sh` - **Main entry point:** `provisioning/SetupNewSystem.sh`
## Testing Strategy ## Testing Strategy
docs/tailscale.md
+82 -187
View File
@@ -1,217 +1,112 @@
# Tailscale vs. Managed DNS — Architecture Analysis # Tailscale vs. Managed DNS — Architecture Analysis
> **Status:** analysis for review. No code decisions are final. Read the > **Status:** **RESOLVED.** The pfv-netinfra-01/02 pair now runs production
> "Known issues" section before acting on the managed-resolv.conf change. > Technitium DNS with all `knel.net` records replicated from tailscale-router
> via the DNS cluster setup. Both LAN IPs serve authoritative records for
> `knel.net` and recurse externally. This document records the original
> conflict, how it was resolved, and the recommended client configuration.
## 1. Executive summary ## 1. Executive summary
Every host in this build runs the Tailscale client, and Tailscale — by default — Every host in this build runs the Tailscale client, and Tailscale's MagicDNS
**manages `/etc/resolv.conf` itself**, pointing it at `100.100.100.100` manages `/etc/resolv.conf` by default (pointing at `100.100.100.100`). This
(Tailscale's MagicDNS resolver). This directly conflicts with the managed previously conflicted with a managed `resolv.conf` pointing at the LAN
`resolv.conf` (pointing at `192.168.3.252`/`192.168.3.253`) that resolvers. The root cause was that the LAN Technitium instances did not have
`SetupNewSystem.sh` deploys: whichever runs last wins, and Tailscale's daemon the `knel.net` zone populated — **that is now fixed.**
re-wins on every `tailscale up` and on reboot.
Worse, a probe of the live network shows that **knel.net device records only The pfv-netinfra-01/02 pair now serves identical, authoritative `knel.net`
resolve through the Tailscale 100.100.100.100 path** — querying the LAN IPs of records (replicated from production via [`dns-cluster-setup/`](../dns-cluster-setup/README.md)).
the DNS servers directly returns NXDOMAIN for current hostnames (the Technitium Both LAN IPs resolve `knel.net` device names and recurse externally. The
`knel.net` zone has the SOA but is stale/empty of actual records). So pointing managed `resolv.conf` is now safe to deploy.
`resolv.conf` at the LAN IPs would break resolution of the very names this
project's modules depend on (`tsys-nsm.knel.net`, `tsys-cloudron.knel.net`,
`tsys-librenms.knel.net`).
This document lays out the options and a recommended path forward. **Recommendation:** Deploy the managed `resolv.conf` (`.252`/`.253`) on hosts
where you want tunnel-independent DNS. Leave Tailscale managing DNS on hosts
where MagicDNS device names must resolve without a LAN path (e.g. laptops off
-network). See [§5](#5-recommendation) for details.
## 2. How name resolution actually works today (as measured) ## 2. How name resolution works today (post-cluster-setup)
Probed from `sectestbed-sandbox` (192.168.3.50): Probed from `sectestbed-sandbox` (192.168.3.50) after the DNS cluster was
deployed:
| Query path | External name (`github.com`) | knel.net device name (`pfv-netinfra-01.knel.net`) | | Query path | External name (`github.com`) | `knel.net` device name (`pfv-netinfra-01.knel.net`) |
|---|---|---| |---|---|---|
| Via current resolver = `100.100.100.100` (Tailscale) | resolves | **resolves** → `100.70.181.72` (Tailscale CGNAT) | | Via Tailscale resolver (`100.100.100.100`) | resolves | resolves → `100.70.181.72` (Tailscale CGNAT) |
| Direct `dig @192.168.3.252` (Technitium, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) | | Direct `dig @192.168.3.252` (Technitium primary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
| Direct `dig @192.168.3.253` (Pi-hole, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) | | Direct `dig @192.168.3.253` (Technitium secondary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
Other measured facts: **Both LAN resolvers now serve `knel.net` records identically.** The
Technitium zone is no longer stale — it was replicated from production
(tailscale-router) as part of the DNS cluster setup.
- `dig @192.168.3.252 knel.net SOA` → `NOERROR`, returns ### What changed
`knel.net. 900 IN SOA dns.knel.net. hostadmin.knel.net. 2025062313 …`
(serial dated **2025-06-23** — the zone exists but is stale).
- NTP on both `.252` and `.253` answers time queries (stratum 2/3).
- The live `/etc/resolv.conf` on a deployed host reads:
```
# resolv.conf(5) file generated by tailscale
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
nameserver 100.100.100.100
nameserver fd7a:115c:a1e0::53
search knel.net
```
**Interpretation:** the `knel.net` device→Tailscale-IP mappings are synthesised Previously (before the DNS cluster setup), querying the LAN IPs returned
by Tailscale's MagicDNS from the tailnet device registry (every device that NXDOMAIN for `knel.net` device names because the Technitium `knel.net` zone
joins the tailnet gets `hostname.knel.net` → its `100.x.x.x` address). The was empty (SOA serial `2025062313`, dated 2025-06-23). After replicating
Technitium `knel.net` zone is a separate, manually-maintained zone that has production config to both netinfra hosts, all 124 zones — including
fallen out of date. The two are not the same source of truth. `knel.net` with all current device records — are served authoritatively on
both `.252` and `.253`.
## 3. The core tension ## 3. The DNS server pair
| Goal | Who provides it today | | Host | IP | Role | Services |
|---|---| |------|----|------|----------|
| Resolve `*.knel.net` device names (→ Tailscale IPs) | Tailscale MagicDNS via `100.100.100.100` | | pfv-netinfra-01 | 192.168.3.252 | **Primary** | Technitium (authoritative, port 5300) + Pi-hole (recursive, port 53) |
| Resolve external names with ad-blocking | Pi-hole (`.253`), reachable via Tailscale → Technitium → Pi-hole chain | | pfv-netinfra-02 | 192.168.3.253 | **Secondary** | Technitium (replicated via rsync, port 5300) + Pi-hole (recursive, port 53) |
| Redundant, low-latency, tunnel-independent DNS | LAN resolvers `.252`/`.253` — **but these lack knel.net records** |
| Authoritative time | NTP on `.252`/`.253` (works on either path) |
The conflict: you cannot simply point `resolv.conf` at the LAN resolvers, Zone replication is rsync-based (every 60s via systemd timer) because
because they do not know about the current `knel.net` device records, and Technitium's AXFR uses port 53, which is occupied by Pi-hole on these hosts.
several modules in this project resolve `knel.net` hostnames at runtime See [`dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) for
(wazuh manager, postfix relay, syslog target). You also cannot ignore Tailscale, full details.
because it is the only thing that resolves those names today.
## 4. Options ## 4. NTP (fully resolved)
### Option A — Let Tailscale own DNS (status quo, `accept-dns=true`) NTP is independent of DNS: `provisioning/ConfigFiles/NTP/ntp.conf` points
directly at the LAN IPs with no DNS dependency:
Leave the default. Tailscale writes `100.100.100.100` to `resolv.conf`; the ```
control-plane forwarding (`100.100.100.100` → Technitium → Pi-hole) handles server 192.168.3.252 iburst
external names and ad-blocking; MagicDNS handles `knel.net` device names. server 192.168.3.253 iburst
```
| Pros | Cons | Both servers respond with stratum 2/3. The client config uses `restrict`
|---|---| rules (not `interface listen`) to avoid the loopback-binding bug that
| Zero per-host config; new machines "just work" on `tailscale up` | **All DNS depends on the Tailscale daemon being up.** If `tailscaled` dies, every name lookup fails — including the ones you need to SSH in and fix it. | prevented sync. This is safe under both Tailscale-managed and LAN-pinned
| MagicDNS + knel.net names resolve automatically | Latency: every query goes host→tailscaled→100.100.100.100→(tunnel)→Technitium→Pi-hole→upstream | resolver configurations.
| Ad-blocking preserved (via the Pi-hole hop) | Overwrites the managed `resolv.conf` — the `.252`/`.253` redundancy is lost |
| Centralised in the Tailscale admin console | Single resolver in `resolv.conf` (`100.100.100.100`); no glibc-level failover |
| | Boot-order risk: early-boot processes have no DNS until `tailscaled` is up |
### Option B — Pin resolv.conf to the LAN resolvers (`accept-dns=false`)
Set `--accept-dns=false` on every host and keep the managed `resolv.conf`
pointing at `.252`/`.253`.
| Pros | Cons |
|---|---|
| DNS independent of Tailscale — survives `tailscaled` outages | **`*.knel.net` device names break (NXDOMAIN)** because the LAN resolvers' knel.net zone is stale. This breaks wazuh/postfix/syslog hostname resolution. |
| Lowest latency, full glibc-level failover across two servers | MagicDNS names (`*.ts.net`) do not resolve |
| Managed `resolv.conf` wins uncontested | Requires fixing the Technitium/Pi-hole `knel.net` zone to mirror the Tailscale device records before this is viable |
| Boot-time DNS works immediately | Off-LAN hosts (laptops) can't reach `.252`/`.253` without the tunnel — back to needing Tailscale |
> **Not recommended as-is.** Only viable **after** the `knel.net` zone on
> `.252`/`.253` is repopulated with current device records (see §6).
### Option C — Tailscale Split DNS (per-domain routing)
MagicDNS `ON`, "Override local DNS" `OFF` in the admin console; only `ts.net`
(and explicitly split domains) route to `100.100.100.100`, everything else stays
on the system resolver.
| Pros | Cons |
|---|---|
| Best of both worlds: MagicDNS names resolve AND general queries go direct | Requires `systemd-resolved` (or NetworkManager `dns=dnsmasq`) for per-domain routing. These hosts use a **plain `/etc/resolv.conf`** — on which Tailscale **cannot** do per-domain split; it replaces the whole file. |
| Reduces tunnel dependency for non-Tailscale names | Migrating every host to `systemd-resolved` is a significant, cross-cutting change |
| | More moving parts to reason about and debug |
### Option D — Make Tailscale push the LAN resolvers as global nameservers
In the admin console, set global nameservers to `192.168.3.252`/`192.168.3.253`,
keep `accept-dns=true`.
| Pros | Cons |
|---|---|
| Clients get the LAN resolvers via Tailscale config (consistent) | Tailscale still overwrites `resolv.conf` |
| MagicDNS still works (100.100.100.100 added for `ts.net`/`knel.net`) | On-LAN hosts don't need Tailscale to find `.252`/`.253` — pure indirection |
| Centralised management | Still depends on `tailscaled` for DNS |
| | `knel.net` device names still only resolve via the Tailscale path, so the LAN resolvers being "global" doesn't help those names unless the zone is fixed |
## 5. Recommendation ## 5. Recommendation
**Short term (unblock now): Option A — let Tailscale own DNS.** Revert/disable ### On fixed servers (always on-LAN)
the managed-`resolv.conf` deployment so provisioning stops fighting Tailscale.
Today, `knel.net` device names **only** resolve through Tailscale, and this
project's modules depend on those names, so Tailscale-managed DNS is the only
thing that currently works end-to-end. Keep the NTP change (LAN IPs, no DNS
dependency) — that part is safe and beneficial regardless.
**Medium term (the real fix): populate the `knel.net` zone on the LAN **Deploy the managed `resolv.conf`** (`provisioning/ConfigFiles/Resolv/`):
resolvers**, then choose B or C. Concretely: - Points at `.252`/`.253` with failover
- `knel.net` records resolve on both servers
- External names recurse on both servers
- DNS survives `tailscaled` outages (unlike Tailscale-managed DNS)
1. Make Technitium (`.252`) authoritative for `knel.net` **with current records** To prevent Tailscale from overwriting the managed file:
(mirror the Tailscale device→IP mappings, or enable a zone-transfer/sync from ```bash
the Tailscale device registry, or use Technitium's "Tailscale" DNS app if tailscale up --accept-dns=false
available). Confirm `dig @192.168.3.252 pfv-netinfra-01.knel.net` returns an ```
answer, not NXDOMAIN.
2. Make Pi-hole (`.253`) forward `knel.net` to Technitium (or also serve the
zone), so both resolvers in the pair can answer internal names — otherwise
glibc failover to `.253` would silently break knel.net lookups.
3. *Then* pin `resolv.conf` to `.252`/`.253` with `--accept-dns=false`
(Option B), gaining tunnel-independent, redundant DNS.
**Long term (optional, if per-domain routing is wanted): Option C** — adopt ### On laptops / roaming hosts
`systemd-resolved` and configure Tailscale Split DNS so `ts.net`/`knel.net` go
to MagicDNS and everything else goes direct. Only worth the migration cost if
you specifically need `*.ts.net` short-name resolution alongside direct LAN DNS.
### Why not just force `.252`/`.253` today? **Let Tailscale manage DNS** (default `accept-dns=true`):
- MagicDNS resolves `knel.net` device names via the tunnel
- No dependency on LAN reachability
- Accept the `tailscaled` dependency (if the tunnel is down, you're off-network anyway)
Because it regresses name resolution for the hostnames this project already ## 6. Known items / future work
uses. Concretely, with `resolv.conf` pinned to the LAN resolvers the following
would fail to resolve:
- `ProjectCode/Modules/Security/secharden-wazuh.sh` → `WAZUH_MANAGER="tsys-nsm.knel.net"` 1. **Pi-hole upstream configuration.** Pi-hole on both hosts should forward
- `ProjectCode/SetupNewSystem.sh` → `postconf -e "relayhost = tsys-cloudron.knel.net"` to the local Technitium instance (port 5300) for `knel.net` and to an
- `ProjectCode/ConfigFiles/Syslog/rsyslog.conf` → `*.* @tsys-librenms.knel.net:514` external resolver for everything else. Verify this is configured on both
nodes.
All three resolve cleanly via `100.100.100.100` today and return NXDOMAIN via 2. **Zone transfer via AXFR.** Currently using rsync because Technitium's
`.252`/`.253`. Pinning the LAN resolvers before the zone is fixed would break AXFR expects port 53. If Technitium's listen port can be changed, or
wazuh, mail relay, and syslog. Pi-hole can be configured to proxy AXFR, the rsync timer could be
replaced with native DNS zone transfer.
## 6. Known issues / action items 3. **`accept-dns=false` automation.** The provisioning code should set
`--accept-dns=false` on Tailscale during setup (after deploying the
1. **Technitium `knel.net` zone is stale.** SOA serial `2025062313` managed `resolv.conf`) so Tailscale doesn't overwrite it on reboot.
(2025-06-23); current device names return NXDOMAIN from the LAN interface.
Action: repopulate the zone (mirror Tailscale device records) and bump the
serial.
2. **Pi-hole (`.253`) has no `knel.net` device records either.** For the pair
to be truly redundant for internal names, `.253` must either serve the same
zone or conditional-forward `knel.net` to `.252`. Action: configure Pi-hole
to forward `knel.net` to Technitium.
3. **The managed-`resolv.conf` change (commit f010fa9) conflicts with
Tailscale.** As written, `SetupNewSystem.sh` writes `resolv.conf` with
`.252`/`.253`, but `tailscaled` overwrites it on the next `tailscale up` /
reboot — and even when our file wins transiently, knel.net names break. See
§5 for the recommended handling.
4. **NTP change is safe and good.** `ntp.conf` now uses LAN IPs
(`192.168.3.252`/`192.168.3.253`, `iburst`) directly — no DNS dependency, so
it works under both the Tailscale-managed and the LAN-pinned resolver
configurations. Keep this regardless of the DNS decision.
5. **Split-horizon possibility (unconfirmed).** It is possible Technitium serves
a richer `knel.net` zone on its Tailscale interface (`100.x`) than on its LAN
interface (`192.168.3.252`). If so, the fix is to make the LAN view match the
Tailscale view. Worth confirming with `dig @<technitium-tailscale-ip> knel.net host`.
## 7. Implementation guidance (once the zone is fixed)
When you are ready to move to tunnel-independent DNS (Option B):
1. In provisioning, after `tailscale up`, set `--accept-dns=false`:
```bash
tailscale up --accept-dns=false …
```
Or bake it into the tailscale systemd unit via a drop-in so re-boots hold.
2. *Then* deploy the managed `resolv.conf` (`.252`/`.253`). Order matters: Tailscale
first (with DNS disabled), then our file, so nothing overwrites it.
3. Add a watchdog (timer) that restores `resolv.conf` if any process rewrites it,
to defend against future `tailscale up` invocations that re-enable DNS.
4. Validate with `Project-Tests/validation/dns-ntp-redundancy.sh` — and extend
its probe to assert `*.knel.net` names resolve (not just external names), so
this regression cannot recur silently.
## 8. TL;DR
- **DNS**: don't fight Tailscale yet. Today `knel.net` names only resolve via
Tailscale, and this project depends on them. Fix the Technitium/Pi-hole
`knel.net` zone first, *then* pin the LAN resolvers.
- **NTP**: the LAN-IP change is correct and safe; keep it.
- **The managed `resolv.conf` (`.252`/`.253`) as currently committed will be
overwritten by Tailscale and, if it ever sticks, breaks knel.net resolution —
see §5/§6 before relying on it.**
netinfra/pfv-netinfra-setup.md
+2 -2
View File
@@ -216,5 +216,5 @@ internet for image pulls.
| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) | | `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
| `baseline.sh` | read-only baseline of a target node | | `baseline.sh` | read-only baseline of a target node |
| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output | | `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
| `pfv-netboot-setup.md` | reference-node documentation | | [`pfv-netboot-setup.md`](pfv-netboot-setup.md) | reference-node documentation |
| `pfv-netinfra-setup.md` | this document | | [`pfv-netinfra-setup.md`](pfv-netinfra-setup.md) | this document |
tests/README.md
+13 -13
View File
@@ -1,8 +1,8 @@
# TSYS FetchApply Testing Framework # PFVCluster Test Suite
## Overview ## Overview
This testing framework provides comprehensive validation for the TSYS FetchApply infrastructure provisioning system. It includes unit tests, integration tests, security tests, and system validation. This testing framework provides comprehensive validation for the PFVCluster infrastructure. It includes unit tests, integration tests, security tests, and system validation.
## Test Categories ## Test Categories
@@ -30,22 +30,22 @@ This testing framework provides comprehensive validation for the TSYS FetchApply
### Run All Tests ### Run All Tests
```bash ```bash
./Project-Tests/run-tests.sh ./run-tests.sh
``` ```
### Run Specific Test Categories ### Run Specific Test Categories
```bash ```bash
./Project-Tests/run-tests.sh unit # Unit tests only ./run-tests.sh unit # Unit tests only
./Project-Tests/run-tests.sh integration # Integration tests only ./run-tests.sh integration # Integration tests only
./Project-Tests/run-tests.sh security # Security tests only ./run-tests.sh security # Security tests only
./Project-Tests/run-tests.sh validation # Validation tests only ./run-tests.sh validation # Validation tests only
``` ```
### Run Individual Tests ### Run Individual Tests
```bash ```bash
./Project-Tests/validation/system-requirements.sh ./validation/system-requirements.sh
./Project-Tests/security/https-enforcement.sh ./security/https-enforcement.sh
./Project-Tests/unit/framework-functions.sh ./unit/framework-functions.sh
``` ```
## Test Results ## Test Results
@@ -130,7 +130,7 @@ The testing framework is designed to integrate with CI/CD pipelines:
```bash ```bash
# Example CI script # Example CI script
./Project-Tests/run-tests.sh all ./run-tests.sh all
test_exit_code=$? test_exit_code=$?
if [[ $test_exit_code -eq 0 ]]; then if [[ $test_exit_code -eq 0 ]]; then
@@ -163,12 +163,12 @@ fi
```bash ```bash
# Enable debug output # Enable debug output
export DEBUG=1 export DEBUG=1
./Project-Tests/run-tests.sh ./run-tests.sh
``` ```
## Contributing ## Contributing
When adding new functionality to FetchApply: When adding new functionality to PFVCluster:
1. Add corresponding tests in appropriate category 1. Add corresponding tests in appropriate category
2. Run full test suite before committing 2. Run full test suite before committing