KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: reorganize merged repo into clean directory structure

Browse Source 
Reorganize the merged KNELServerBuild + PFVCluster repo:

  provisioning/    server provisioning (was ProjectCode/ +
                   Project-Includes/ + Project-ConfigFiles/)
  tests/           test suite (was Project-Tests/)
  perf/            Proxmox perf scripts (was top-level *.sh + scripts/)
  docs/            all documentation (was ProjectDocs/ + PROJECT.md +
                   K8S.md + TODO.md)
  dns-cluster-setup/  Technitium DNS cluster (unchanged)
  netinfra/        netinfra audit scripts (unchanged)
  switches/        switch configs (unchanged)
  vendor/          vendored KNELShellFramework (unchanged)

Update all internal path references from old directory names
(ProjectCode/, Project-Includes/, Project-Tests/) to the new ones
(provisioning/, tests/) across all scripts.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
mrcharles
2026-07-28 11:24:39 -05:00
parent 132c0854d1
commit 4851517947
133 changed files with 26 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/README.md
+176
View File
@@ -0,0 +1,176 @@
# TSYS FetchApply Testing Framework
## Overview
This testing framework provides comprehensive validation for the TSYS FetchApply infrastructure provisioning system. It includes unit tests, integration tests, security tests, and system validation.
## Test Categories
### 1. Unit Tests (`unit/`)
- **Purpose:** Test individual framework functions and components
- **Scope:** Framework includes, helper functions, syntax validation
- **Example:** `framework-functions.sh` - Tests logging, pretty print, and error handling functions
### 2. Integration Tests (`integration/`)
- **Purpose:** Test complete workflows and module interactions
- **Scope:** End-to-end deployment scenarios, module integration
- **Future:** Module interaction testing, deployment workflow validation
### 3. Security Tests (`security/`)
- **Purpose:** Validate security configurations and practices
- **Scope:** HTTPS enforcement, deployment security, SSH hardening
- **Example:** `https-enforcement.sh` - Validates all URLs use HTTPS
### 4. Validation Tests (`validation/`)
- **Purpose:** System compatibility and pre-flight checks
- **Scope:** System requirements, network connectivity, permissions
- **Example:** `system-requirements.sh` - Validates minimum system requirements
## Usage
### Run All Tests
```bash
./Project-Tests/run-tests.sh
```
### Run Specific Test Categories
```bash
./Project-Tests/run-tests.sh unit # Unit tests only
./Project-Tests/run-tests.sh integration # Integration tests only
./Project-Tests/run-tests.sh security # Security tests only
./Project-Tests/run-tests.sh validation # Validation tests only
```
### Run Individual Tests
```bash
./Project-Tests/validation/system-requirements.sh
./Project-Tests/security/https-enforcement.sh
./Project-Tests/unit/framework-functions.sh
```
## Test Results
- **Console Output:** Real-time test results with color-coded status
- **JSON Reports:** Detailed test reports saved to `logs/tests/`
- **Exit Codes:** 0 for success, 1 for failures
## Configuration Validation
The validation framework performs pre-flight checks to ensure system compatibility:
### System Requirements
- **Memory:** Minimum 2GB RAM
- **Disk Space:** Minimum 10GB available
- **OS Compatibility:** Ubuntu/Debian (tested), others (may work)
### Network Connectivity
- Tests connection to required download sources
- Validates HTTPS endpoints are accessible
- Checks for firewall/proxy issues
### Command Dependencies
- Verifies required tools are installed (`curl`, `wget`, `git`, `systemctl`, `apt-get`)
- Checks for proper versions where applicable
### Permissions
- Validates write access to system directories
- Checks for required administrative privileges
## Adding New Tests
### Test File Structure
```bash
#!/bin/bash
set -euo pipefail
function test_something() {
echo "🔍 Testing something..."
if [[ condition ]]; then
echo "✅ Test passed"
return 0
else
echo "❌ Test failed"
return 1
fi
}
function main() {
echo "🧪 Running Test Suite Name"
echo "=========================="
local total_failures=0
test_something || ((total_failures++))
echo "=========================="
if [[ $total_failures -eq 0 ]]; then
echo "✅ All tests passed"
exit 0
else
echo "❌ $total_failures tests failed"
exit 1
fi
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
```
### Test Categories Guidelines
- **Unit Tests:** Focus on individual functions, fast execution
- **Integration Tests:** Test module interactions, longer execution
- **Security Tests:** Validate security configurations
- **Validation Tests:** Pre-flight system checks
## Continuous Integration
The testing framework is designed to integrate with CI/CD pipelines:
```bash
# Example CI script
./Project-Tests/run-tests.sh all
test_exit_code=$?
if [[ $test_exit_code -eq 0 ]]; then
echo "All tests passed - deployment approved"
else
echo "Tests failed - deployment blocked"
exit 1
fi
```
## Test Development Best Practices
1. **Clear Test Names:** Use descriptive function names
2. **Proper Exit Codes:** Return 0 for success, 1 for failure
3. **Informative Output:** Use emoji and clear messages
4. **Timeout Protection:** Use timeout for network operations
5. **Cleanup:** Remove temporary files and resources
6. **Error Handling:** Use `set -euo pipefail` for strict error handling
## Troubleshooting
### Common Issues
- **Permission Denied:** Run tests with appropriate privileges
- **Network Timeouts:** Check firewall and proxy settings
- **Missing Dependencies:** Install required tools before testing
- **Script Errors:** Validate syntax with `bash -n script.sh`
### Debug Mode
```bash
# Enable debug output
export DEBUG=1
./Project-Tests/run-tests.sh
```
## Contributing
When adding new functionality to FetchApply:
1. Add corresponding tests in appropriate category
2. Run full test suite before committing
3. Update documentation for new test cases
4. Ensure tests pass in clean environment
tests/remote.sh
Executable
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/bash
#
# remote.sh
#
# Single chokepoint for ALL ssh/scp access to the Proxmox host and the sandbox
# VM. Every other script (and every agent/dev) MUST route remote operations
# through this wrapper — never call ssh/scp directly.
#
# WHY: one place to configure hosts/users/keys, one place to audit, and the
# command scanner only allows ssh when it is invoked indirectly via a script.
#
# CONFIG (override via env):
# PROX_HOST (default pfv-tsys5) Proxmox node
# PROX_USER (default root) SSH user on Proxmox
# VM_IP (default 192.168.3.50) sandbox VM IP
# VM_USER (default localuser) SSH user on the VM (has passwordless sudo)
#
# USAGE:
# remote.sh prox <cmd...> run command on Proxmox
# remote.sh vm <cmd...> run command on VM as $VM_USER
# remote.sh vmroot <cmd...> run command on VM as root via sudo
# remote.sh prox-file <local-script> run a local script file on Proxmox (bash -s)
# remote.sh vm-file <local-script> run a local script file on the VM (bash -s)
# remote.sh vm-copy <local> <dest> copy a local file to the VM (~$VM_USER space)
# remote.sh prox-copy <local> <dest> copy a local file to Proxmox
#
set -uo pipefail
PROX_HOST="${PROX_HOST:-pfv-tsys5}"
PROX_USER="${PROX_USER:-root}"
VM_IP="${VM_IP:-192.168.3.50}"
VM_USER="${VM_USER:-localuser}"
VM_ID="${VM_ID:-}"
GUEST_TIMEOUT="${GUEST_TIMEOUT:-900}"
SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
die() { echo "remote.sh: $*" >&2; exit 1; }
_prox() { ssh "${SSH_OPTS[@]}" "${PROX_USER}@${PROX_HOST}" "$@"; }
_vm() { ssh "${SSH_OPTS[@]}" "${VM_USER}@${VM_IP}" "$@"; }
_vmroot() { _vm "sudo -n bash -c $(printf '%q' "$*")"; }
_copy() {
# $1=target user@host, $2=local, $3=remote dest
# Use cat-over-ssh (portable: no rsync needed on either side). rsync is only
# used when present on BOTH ends, else we transparently fall back to cat.
local target="$1" local="$2" dest="$3"
local userhost="${target%@*}@${target#*@}"
if command -v rsync >/dev/null 2>&1 \
&& ssh "${SSH_OPTS[@]}" "$userhost" 'command -v rsync' >/dev/null 2>&1; then
rsync -az -e "ssh ${SSH_OPTS[*]}" "$local" "${userhost}:${dest}"
else
ssh "${SSH_OPTS[@]}" "$userhost" "cat > '$dest'" < "$local"
fi
}
# Out-of-band VM access via the Proxmox qemu-guest-agent. This runs commands
# as root inside the VM and does NOT depend on SSH, so it works even after
# secharden-ssh replaces authorized_keys and secharden-2fa enforces
# publickey+keyboard-interactive (which blocks non-interactive SSH).
GUEST_PARSER="/root/.knel-guest-parse.py"
GUEST_PARSER_SRC="import sys, json
try:
d = json.load(sys.stdin)
except Exception:
sys.exit(3)
sys.stdout.write(d.get('out-data', '') or '')
sys.stderr.write(d.get('err-data', '') or '')
ec = d.get('exitcode', 1)
sys.exit(ec if ec is not None else 1)"
_ensure_guest_parser() {
if _prox "test -f '$GUEST_PARSER'" >/dev/null 2>&1; then return 0; fi
printf '%s\n' "$GUEST_PARSER_SRC" | _prox "cat > '$GUEST_PARSER'" >/dev/null 2>&1
}
_vm_guest() {
[ -n "$VM_ID" ] || die "vm-guest requires VM_ID"
_ensure_guest_parser
local cmdb64; cmdb64="$(printf '%s' "$*" | base64 -w0)"
_prox "qm guest exec $VM_ID --timeout ${GUEST_TIMEOUT} -- /bin/sh -c 'echo $cmdb64 | base64 -d | /bin/sh' 2>/dev/null | python3 '$GUEST_PARSER'"
}
mode="${1:-}"; shift || true
case "$mode" in
prox) [ "$#" -ge 0 ] || die "need command"; _prox "$*" ;;
vm) _vm "$*" ;;
vmroot) [ "$#" -ge 1 ] || die "need command"; _vmroot "$*" ;;
prox-file) [ -f "${1:-}" ] || die "need local script file"; _prox "bash -s" < "$1" ;;
vm-file) [ -f "${1:-}" ] || die "need local script file"; _vm "bash -s" < "$1" ;;
vm-copy) [ -f "${1:-}" ] || die "need local file"; _copy "${VM_USER}@${VM_IP}" "$1" "${2:-}" ;;
prox-copy) [ -f "${1:-}" ] || die "need local file"; _copy "${PROX_USER}@${PROX_HOST}" "$1" "${2:-}" ;;
vm-guest) [ "$#" -ge 1 ] || die "need command"; _vm_guest "$*" ;;
""|-h|--help|help) sed -n '2,40p' "${BASH_SOURCE[0]}" >&2; exit 0 ;;
*) die "unknown mode '$mode'. Run '$0 help'." ;;
esac
tests/run-tests.sh
Executable
+137
View File
@@ -0,0 +1,137 @@
#!/bin/bash
# TSYS FetchApply Testing Framework
# Main test runner script
set -euo pipefail
# Resolve repository root from this script's location (tests/ -> repo root)
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# The KNELShellFramework is vendored under vendor/
FRAMEWORK_INCLUDES="$PROJECT_ROOT/vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/Framework-Includes"
source "$FRAMEWORK_INCLUDES/Logging.sh"
source "$FRAMEWORK_INCLUDES/PrettyPrint.sh"
# The vendored PrettyPrint only defines print_info/print_error; provide the
# additional output helpers the test suite relies on.
function print_header() { echo ""; echo "=== $1 ==="; }
function print_success() { echo "✅ $1"; }
function print_warning() { echo "⚠️ $1"; }
# Test configuration
TEST_LOG_DIR="$PROJECT_ROOT/logs/tests"
TEST_RESULTS_FILE="$TEST_LOG_DIR/test-results-$(date +%Y%m%d-%H%M%S).json"
# Ensure test log directory exists
mkdir -p "$TEST_LOG_DIR"
# Test counters
declare -g TESTS_PASSED=0
declare -g TESTS_FAILED=0
declare -g TESTS_SKIPPED=0
# Test runner functions
function run_test_suite() {
local suite_name="$1"
local test_dir="$2"
print_header "Running $suite_name Tests"
if [[ ! -d "$test_dir" ]]; then
print_warning "Test directory $test_dir not found, skipping"
return 0
fi
for test_file in "$test_dir"/*.sh; do
if [[ -f "$test_file" ]]; then
run_single_test "$test_file"
fi
done
}
function run_single_test() {
local test_file="$1"
local test_name="$(basename "$test_file" .sh)"
print_info "Running test: $test_name"
if timeout 300 bash "$test_file"; then
print_success "✅ $test_name PASSED"
TESTS_PASSED=$((TESTS_PASSED + 1))
else
print_error "❌ $test_name FAILED"
TESTS_FAILED=$((TESTS_FAILED + 1))
fi
}
function generate_test_report() {
local total_tests=$((TESTS_PASSED + TESTS_FAILED + TESTS_SKIPPED))
print_header "Test Results Summary"
print_info "Total Tests: $total_tests"
print_success "Passed: $TESTS_PASSED"
print_error "Failed: $TESTS_FAILED"
print_warning "Skipped: $TESTS_SKIPPED"
# Generate JSON report
cat > "$TEST_RESULTS_FILE" <<EOF
{
"timestamp": "$(date -Iseconds)",
"total_tests": $total_tests,
"passed": $TESTS_PASSED,
"failed": $TESTS_FAILED,
"skipped": $TESTS_SKIPPED,
"success_rate": $(awk "BEGIN {printf \"%.2f\", ($TESTS_PASSED/$total_tests)*100}")
}
EOF
print_info "Test report saved to: $TEST_RESULTS_FILE"
}
# Main execution
function main() {
print_header "TSYS FetchApply Test Suite"
# Parse command line arguments
local test_type="${1:-all}"
case "$test_type" in
"unit")
run_test_suite "Unit" "$(dirname "$0")/unit"
;;
"integration")
run_test_suite "Integration" "$(dirname "$0")/integration"
;;
"security")
run_test_suite "Security" "$(dirname "$0")/security"
;;
"validation")
run_test_suite "Validation" "$(dirname "$0")/validation"
;;
"all")
run_test_suite "Unit" "$(dirname "$0")/unit"
run_test_suite "Integration" "$(dirname "$0")/integration"
run_test_suite "Security" "$(dirname "$0")/security"
run_test_suite "Validation" "$(dirname "$0")/validation"
;;
*)
print_error "Usage: $0 [unit|integration|security|validation|all]"
exit 1
;;
esac
generate_test_report
# Exit with appropriate code
if [[ $TESTS_FAILED -gt 0 ]]; then
exit 1
else
exit 0
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/security/2fa-validation.sh
Executable
+312
View File
@@ -0,0 +1,312 @@
#!/bin/bash
# Two-Factor Authentication Validation Test
# Validates 2FA configuration for SSH, Cockpit, and Webmin
set -euo pipefail
PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.."
function test_2fa_packages() {
echo "🔍 Testing 2FA package installation..."
local packages=("libpam-google-authenticator" "qrencode")
local failed=0
for package in "${packages[@]}"; do
if dpkg -s "$package" 2>/dev/null | grep -q "^Status:.*installed"; then
echo "✅ Package installed: $package"
else
echo "❌ Package missing: $package"
((++failed))
fi
done
# Check if google-authenticator command exists
if command -v google-authenticator >/dev/null 2>&1; then
echo "✅ Google Authenticator command available"
else
echo "❌ Google Authenticator command not found"
((++failed))
fi
return $failed
}
function test_ssh_2fa_config() {
echo "🔍 Testing SSH 2FA configuration..."
local ssh_config="/etc/ssh/sshd_config"
local failed=0
# Check required SSH settings
if grep -q "^ChallengeResponseAuthentication yes" "$ssh_config"; then
echo "✅ ChallengeResponseAuthentication enabled"
else
echo "❌ ChallengeResponseAuthentication not enabled"
((++failed))
fi
if grep -q "^UsePAM yes" "$ssh_config"; then
echo "✅ UsePAM enabled"
else
echo "❌ UsePAM not enabled"
((++failed))
fi
if grep -q "^AuthenticationMethods publickey,keyboard-interactive" "$ssh_config"; then
echo "✅ AuthenticationMethods configured for 2FA"
else
echo "❌ AuthenticationMethods not configured for 2FA"
((++failed))
fi
return $failed
}
function test_pam_2fa_config() {
echo "🔍 Testing PAM 2FA configuration..."
local pam_sshd="/etc/pam.d/sshd"
local failed=0
# Check if PAM includes Google Authenticator
if grep -q "pam_google_authenticator.so" "$pam_sshd"; then
echo "✅ PAM Google Authenticator module configured"
else
echo "❌ PAM Google Authenticator module not configured"
((++failed))
fi
# Check if nullok is present (allows users without 2FA setup)
if grep -q "pam_google_authenticator.so nullok" "$pam_sshd"; then
echo "✅ PAM nullok option configured (allows gradual rollout)"
else
echo "⚠️ PAM nullok option not configured (immediate enforcement)"
fi
return $failed
}
function test_cockpit_2fa_config() {
echo "🔍 Testing Cockpit 2FA configuration..."
local cockpit_config="/etc/cockpit/cockpit.conf"
local cockpit_pam="/etc/pam.d/cockpit"
local failed=0
# Check if Cockpit is installed
if ! command -v cockpit-ws >/dev/null 2>&1; then
echo "⚠️ Cockpit not installed, skipping test"
return 0
fi
# Check Cockpit configuration
if [[ -f "$cockpit_config" ]]; then
echo "✅ Cockpit configuration file exists"
else
echo "❌ Cockpit configuration file missing"
((++failed))
fi
# Check Cockpit PAM configuration
if [[ -f "$cockpit_pam" ]] && grep -q "pam_google_authenticator.so" "$cockpit_pam"; then
echo "✅ Cockpit PAM 2FA configured"
else
echo "❌ Cockpit PAM 2FA not configured"
((++failed))
fi
return $failed
}
function test_webmin_2fa_config() {
echo "🔍 Testing Webmin 2FA configuration..."
local webmin_config="/etc/webmin/miniserv.conf"
local failed=0
# Check if Webmin is installed
if [[ ! -f "$webmin_config" ]]; then
echo "⚠️ Webmin not installed, skipping test"
return 0
fi
# Check Webmin 2FA settings
if grep -q "^twofactor_provider=totp" "$webmin_config"; then
echo "✅ Webmin TOTP provider configured"
else
echo "❌ Webmin TOTP provider not configured"
((++failed))
fi
if grep -q "^twofactor=1" "$webmin_config"; then
echo "✅ Webmin 2FA enabled"
else
echo "❌ Webmin 2FA not enabled"
((++failed))
fi
return $failed
}
function test_user_2fa_setup() {
echo "🔍 Testing user 2FA setup preparation..."
local users=("localuser" "root")
local failed=0
for user in "${users[@]}"; do
if id "$user" &>/dev/null; then
local user_home; user_home="$(getent passwd "$user" | cut -d: -f6)"
# Check if setup script exists
if [[ -f "/tmp/setup-2fa-$user.sh" ]]; then
echo "✅ 2FA setup script exists for user: $user"
else
echo "❌ 2FA setup script missing for user: $user"
((++failed))
fi
# Check if instructions exist
if [[ -n "$user_home" && -f "$user_home/2fa-setup-instructions.txt" ]]; then
echo "✅ 2FA instructions exist for user: $user"
else
echo "❌ 2FA instructions missing for user: $user"
((++failed))
fi
else
echo "⚠️ User $user not found, skipping"
fi
done
return $failed
}
function test_service_status() {
echo "🔍 Testing service status..."
local failed=0
# Test SSH service
if systemctl is-active sshd >/dev/null 2>&1; then
echo "✅ SSH service is running"
else
echo "❌ SSH service is not running"
((++failed))
fi
# Test SSH configuration
if sshd -t 2>/dev/null; then
echo "✅ SSH configuration is valid"
else
echo "❌ SSH configuration is invalid"
((++failed))
fi
# Test Cockpit service if installed
if systemctl is-enabled cockpit.socket >/dev/null 2>&1; then
if systemctl is-active cockpit.socket >/dev/null 2>&1; then
echo "✅ Cockpit service is running"
else
echo "❌ Cockpit service is not running"
((++failed))
fi
fi
# Test Webmin service if installed
if systemctl is-enabled webmin >/dev/null 2>&1; then
if systemctl is-active webmin >/dev/null 2>&1; then
echo "✅ Webmin service is running"
else
echo "❌ Webmin service is not running"
((++failed))
fi
fi
return $failed
}
function test_backup_existence() {
echo "🔍 Testing backup existence..."
local backup_dir="/root/backup"
local failed=0
if [[ -d "$backup_dir" ]]; then
# Look for recent 2FA backups
local recent_backups=$(find "$backup_dir" -name "2fa-*" -type d -newer /etc/ssh/sshd_config 2>/dev/null | wc -l)
if [[ $recent_backups -gt 0 ]]; then
echo "✅ Recent 2FA backup found in $backup_dir"
else
echo "⚠️ No recent 2FA backups found"
fi
else
echo "❌ Backup directory does not exist"
((++failed))
fi
return $failed
}
function test_2fa_enforcement() {
echo "🔍 Testing 2FA enforcement level..."
local pam_sshd="/etc/pam.d/sshd"
# Check if nullok is used (allows users without 2FA)
if grep -q "pam_google_authenticator.so nullok" "$pam_sshd"; then
echo "⚠️ 2FA enforcement: GRADUAL (nullok allows users without 2FA)"
echo " Users can log in without 2FA during setup phase"
else
echo "✅ 2FA enforcement: STRICT (all users must have 2FA)"
echo " All users must have 2FA configured to log in"
fi
return 0
}
# Main test execution
function main() {
echo "🔒 Running Two-Factor Authentication Validation Tests"
echo "=================================================="
local total_failures=0
# Run all 2FA validation tests
test_2fa_packages || ((total_failures++))
test_ssh_2fa_config || ((total_failures++))
test_pam_2fa_config || ((total_failures++))
test_cockpit_2fa_config || ((total_failures++))
test_webmin_2fa_config || ((total_failures++))
test_user_2fa_setup || ((total_failures++))
test_service_status || ((total_failures++))
test_backup_existence || ((total_failures++))
test_2fa_enforcement || ((total_failures++))
echo "=================================================="
if [[ $total_failures -eq 0 ]]; then
echo "✅ All 2FA validation tests passed"
echo ""
echo "📋 Next Steps:"
echo "1. Run user setup scripts: /tmp/setup-2fa-*.sh"
echo "2. Test 2FA login from another terminal"
echo "3. Remove nullok from PAM config for strict enforcement"
exit 0
else
echo "❌ $total_failures 2FA validation tests failed"
echo ""
echo "🔧 Troubleshooting:"
echo "1. Re-run secharden-2fa.sh script"
echo "2. Check system logs: journalctl -u sshd"
echo "3. Verify package installation"
exit 1
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/security/https-enforcement.sh
Executable
+148
View File
@@ -0,0 +1,148 @@
#!/bin/bash
# HTTPS Enforcement Security Test
# Validates that all scripts use HTTPS instead of HTTP
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
FRAMEWORK_INCLUDES="$PROJECT_ROOT/vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/Framework-Includes"
function test_no_http_urls() {
echo "🔍 Checking for HTTP URLs in scripts..."
local http_violations=0
local script_dirs=("$PROJECT_ROOT/provisioning" "$FRAMEWORK_INCLUDES" "$PROJECT_ROOT/provisioning/Project-Includes")
for dir in "${script_dirs[@]}"; do
if [[ -d "$dir" ]]; then
# Find HTTP URLs in shell scripts (excluding comments)
while IFS= read -r -d '' file; do
# grep -n prefixes "linenum:", so the comment filter must allow
# for that prefix before the leading '#' of a comment line.
if grep -n "http://" "$file" | grep -vE '^[0-9]+:[[:space:]]*#' | grep -v "schema.org" | grep -v "xmlns"; then
echo "❌ HTTP URL found in: $file"
((++http_violations))
fi
done < <(find "$dir" -name "*.sh" -type f -print0)
fi
done
if [[ $http_violations -eq 0 ]]; then
echo "✅ No HTTP URLs found in active scripts"
return 0
else
echo "❌ Found $http_violations HTTP URL violations"
return 1
fi
}
function test_https_urls_valid() {
echo "🔍 Validating HTTPS URLs are accessible..."
local script_dirs=("$PROJECT_ROOT/provisioning" "$FRAMEWORK_INCLUDES" "$PROJECT_ROOT/provisioning/Project-Includes")
local https_failures=0
# Extract HTTPS URLs from scripts
for dir in "${script_dirs[@]}"; do
if [[ -d "$dir" ]]; then
while IFS= read -r -d '' file; do
# Extract HTTPS URLs from non-comment lines
grep -o "https://[^[:space:]\"']*" "$file" | grep -v "schema.org" | while read -r url; do
# Test connectivity with timeout
if timeout 30 curl -s --head --fail "$url" >/dev/null 2>&1; then
echo "✅ HTTPS URL accessible: $url"
else
echo "❌ HTTPS URL not accessible: $url"
((++https_failures))
fi
done
done < <(find "$dir" -name "*.sh" -type f -print0)
fi
done
return $https_failures
}
function test_ssl_certificate_validation() {
echo "🔍 Testing SSL certificate validation..."
local test_urls=(
"https://archive.ubuntu.com"
"https://linux.dell.com"
"https://download.proxmox.com"
)
local ssl_failures=0
for url in "${test_urls[@]}"; do
# Verify TLS is required and the certificate chain is valid. Do NOT use
# --cert-status: that requires OCSP stapling, which many valid CDNs do
# not provide, producing false negatives for otherwise-valid certs.
if curl -s --fail --ssl-reqd "$url" >/dev/null 2>&1; then
echo "✅ SSL certificate valid: $url"
else
echo "❌ SSL certificate validation failed: $url"
((++ssl_failures))
fi
done
return $ssl_failures
}
function test_deployment_security() {
echo "🔍 Testing deployment method security..."
local readme_file="$PROJECT_ROOT/README.md"
if [[ -f "$readme_file" ]]; then
# Check for insecure curl | bash patterns
if grep -q "curl.*|.*bash" "$readme_file" || grep -q "wget.*|.*bash" "$readme_file"; then
echo "❌ Insecure deployment method found in README.md"
return 1
else
echo "✅ Secure deployment method in README.md"
fi
# Check for git clone method
if grep -q "git clone" "$readme_file"; then
echo "✅ Git clone deployment method found"
return 0
else
echo "⚠️ No git clone method found in README.md"
return 1
fi
else
echo "❌ README.md not found"
return 1
fi
}
# Main test execution
function main() {
echo "🔒 Running HTTPS Enforcement Security Tests"
echo "=========================================="
local total_failures=0
# Run all security tests
test_no_http_urls || ((total_failures++))
test_https_urls_valid || ((total_failures++))
test_ssl_certificate_validation || ((total_failures++))
test_deployment_security || ((total_failures++))
echo "=========================================="
if [[ $total_failures -eq 0 ]]; then
echo "✅ All HTTPS enforcement security tests passed"
exit 0
else
echo "❌ $total_failures HTTPS enforcement security tests failed"
exit 1
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/unit/framework-functions.sh
Executable
+179
View File
@@ -0,0 +1,179 @@
#!/bin/bash
# Framework Functions Unit Tests
# Tests core framework functionality
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
# Source framework functions from the vendored KNELShellFramework
FRAMEWORK_INCLUDES="$PROJECT_ROOT/vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/Framework-Includes"
source "$FRAMEWORK_INCLUDES/Logging.sh" 2>/dev/null || echo "Warning: Logging.sh not found"
source "$FRAMEWORK_INCLUDES/PrettyPrint.sh" 2>/dev/null || echo "Warning: PrettyPrint.sh not found"
source "$FRAMEWORK_INCLUDES/ErrorHandling.sh" 2>/dev/null || echo "Warning: ErrorHandling.sh not found"
function test_logging_variables() {
echo "🔍 Testing logging variables..."
if [[ -n "${CURRENT_TIMESTAMP:-}" ]]; then
echo "✅ CURRENT_TIMESTAMP is set"
else
echo "❌ CURRENT_TIMESTAMP is not set"
return 1
fi
if [[ -n "${LOGFILENAME:-}" ]]; then
echo "✅ LOGFILENAME is set"
else
echo "❌ LOGFILENAME is not set"
return 1
fi
return 0
}
function test_pretty_print_functions() {
echo "🔍 Testing pretty print functions..."
# Test if pretty print functions exist
if command -v print_info >/dev/null 2>&1; then
print_info "Test info message" >/dev/null 2>&1 || true
echo "✅ print_info function exists"
else
echo "❌ print_info function missing"
return 1
fi
if command -v print_error >/dev/null 2>&1; then
print_error "Test error message" >/dev/null 2>&1 || true
echo "✅ print_error function exists"
else
echo "❌ print_error function missing"
return 1
fi
return 0
}
function test_error_handling() {
echo "🔍 Testing error handling..."
# Test if error handling functions exist
if command -v error_out >/dev/null 2>&1; then
echo "✅ error_out function exists"
else
echo "❌ error_out function missing"
return 1
fi
if command -v handle_failure >/dev/null 2>&1; then
echo "✅ handle_failure function exists"
else
echo "❌ handle_failure function missing"
return 1
fi
# Test bash strict mode is set
if [[ "$-" == *e* ]]; then
echo "✅ Bash strict mode (set -e) is enabled"
else
echo "❌ Bash strict mode (set -e) not enabled"
return 1
fi
if [[ "$-" == *u* ]]; then
echo "✅ Bash unset variable checking (set -u) is enabled"
else
echo "❌ Bash unset variable checking (set -u) not enabled"
return 1
fi
return 0
}
function test_framework_includes_exist() {
echo "🔍 Testing framework includes exist..."
local required_includes=(
"Logging.sh"
"PrettyPrint.sh"
"ErrorHandling.sh"
"PreflightCheck.sh"
)
local missing_files=0
for include_file in "${required_includes[@]}"; do
if [[ -f "$FRAMEWORK_INCLUDES/$include_file" ]]; then
echo "✅ Framework include exists: $include_file"
else
echo "❌ Framework include missing: $include_file"
((++missing_files))
fi
done
return $missing_files
}
function test_syntax_validation() {
echo "🔍 Testing script syntax validation..."
local syntax_errors=0
local script_dirs=(
"$FRAMEWORK_INCLUDES"
"$PROJECT_ROOT/provisioning/Project-Includes"
"$PROJECT_ROOT/ProjectCode"
)
for dir in "${script_dirs[@]}"; do
if [[ -d "$dir" ]]; then
while IFS= read -r -d '' file; do
# Skip files that aren't bash scripts despite a .sh extension (e.g. PHP agents)
local shebang
shebang="$(head -c 32 "$file" 2>/dev/null)"
case "$shebang" in
*php*|*python*|*perl*) continue ;;
esac
if bash -n "$file" 2>/dev/null; then
echo "✅ Syntax valid: $(basename "$file")"
else
echo "❌ Syntax error in: $(basename "$file")"
((++syntax_errors))
fi
done < <(find "$dir" -name "*.sh" -type f -print0)
fi
done
return $syntax_errors
}
# Main test execution
function main() {
echo "🧪 Running Framework Functions Unit Tests"
echo "========================================"
local total_failures=0
# Run all unit tests
test_framework_includes_exist || ((total_failures++))
test_logging_variables || ((total_failures++))
test_pretty_print_functions || ((total_failures++))
test_error_handling || ((total_failures++))
test_syntax_validation || ((total_failures++))
echo "========================================"
if [[ $total_failures -eq 0 ]]; then
echo "✅ All framework function unit tests passed"
exit 0
else
echo "❌ $total_failures framework function unit tests failed"
exit 1
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/unit/safe-download.sh
Executable
+297
View File
@@ -0,0 +1,297 @@
#!/bin/bash
# Safe Download Framework Unit Tests
# Tests the SafeDownload.sh framework functionality
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
# Source framework functions from the vendored KNELShellFramework
FRAMEWORK_INCLUDES="$PROJECT_ROOT/vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/Framework-Includes"
# The vendored PrettyPrint only defines print_info/print_error, but SafeDownload.sh
# calls print_success/print_warning; define lightweight shims before sourcing.
function print_success() { echo "✅ $1"; }
function print_warning() { echo "⚠️ $1"; }
source "$FRAMEWORK_INCLUDES/SafeDownload.sh"
function test_network_connectivity() {
echo "🔍 Testing network connectivity..."
if check_url_accessibility "https://github.com"; then
echo "✅ Network connectivity test passed"
return 0
else
echo "❌ Network connectivity test failed"
return 1
fi
}
function test_url_accessibility() {
echo "🔍 Testing URL accessibility..."
local test_urls=(
"https://archive.ubuntu.com"
"https://github.com"
)
local failed=0
for url in "${test_urls[@]}"; do
if check_url_accessibility "$url"; then
echo "✅ URL accessible: $url"
else
echo "❌ URL not accessible: $url"
((++failed))
fi
done
return $failed
}
function test_safe_download() {
echo "🔍 Testing safe download functionality..."
local test_url="https://raw.githubusercontent.com/torvalds/linux/master/README"
local test_dest="/tmp/test-download-$$"
local failed=0
# Test successful download
if safe_download "$test_url" "$test_dest"; then
echo "✅ Safe download successful"
# Verify file exists and has content
if [[ -f "$test_dest" && -s "$test_dest" ]]; then
echo "✅ Downloaded file exists and has content"
else
echo "❌ Downloaded file is missing or empty"
((++failed))
fi
# Cleanup
rm -f "$test_dest"
else
echo "❌ Safe download failed"
((++failed))
fi
# Test download with invalid URL
if safe_download "https://invalid.example.com/nonexistent" "/tmp/test-invalid-$$" 2>/dev/null; then
echo "❌ Invalid URL download should have failed"
((++failed))
else
echo "✅ Invalid URL download failed as expected"
fi
return $failed
}
function test_checksum_verification() {
echo "🔍 Testing checksum verification..."
local test_file="/tmp/test-checksum-$$"
local test_content="Hello, World!"
local expected_checksum="dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f"
local failed=0
# Create test file with known content
echo -n "$test_content" > "$test_file"
# Test correct checksum
if verify_checksum "$test_file" "$expected_checksum"; then
echo "✅ Correct checksum verification passed"
else
echo "❌ Correct checksum verification failed"
((++failed))
fi
# Test incorrect checksum
if verify_checksum "$test_file" "invalid_checksum" 2>/dev/null; then
echo "❌ Incorrect checksum should have failed"
((++failed))
else
echo "✅ Incorrect checksum verification failed as expected"
fi
# Test missing file
if verify_checksum "/tmp/nonexistent-file-$$" "$expected_checksum" 2>/dev/null; then
echo "❌ Missing file checksum should have failed"
((++failed))
else
echo "✅ Missing file checksum verification failed as expected"
fi
# Cleanup
rm -f "$test_file"
return $failed
}
function test_batch_download() {
echo "🔍 Testing batch download functionality..."
# Create test download map
declare -A test_downloads=(
["https://raw.githubusercontent.com/torvalds/linux/master/README"]="/tmp/batch-test-1-$$"
["https://raw.githubusercontent.com/torvalds/linux/master/COPYING"]="/tmp/batch-test-2-$$"
)
local failed=0
# Test batch download
if batch_download test_downloads; then
echo "✅ Batch download successful"
# Verify all files were downloaded
for file in "${test_downloads[@]}"; do
if [[ -f "$file" && -s "$file" ]]; then
echo "✅ Batch file downloaded: $(basename "$file")"
else
echo "❌ Batch file missing: $(basename "$file")"
((++failed))
fi
done
# Cleanup
for file in "${test_downloads[@]}"; do
rm -f "$file"
done
else
echo "❌ Batch download failed"
((++failed))
fi
return $failed
}
function test_config_backup_and_restore() {
echo "🔍 Testing config backup and restore..."
local test_config="/tmp/test-config-$$"
local original_content="Original configuration"
local failed=0
# Create original config file
echo "$original_content" > "$test_config"
# Test safe config download (this will fail with invalid URL, triggering restore)
if safe_config_download "https://invalid.example.com/config" "$test_config" ".test-backup" 2>/dev/null; then
echo "❌ Invalid config download should have failed"
((++failed))
else
echo "✅ Invalid config download failed as expected"
# Verify original file was restored
if [[ -f "$test_config" ]] && grep -q "$original_content" "$test_config"; then
echo "✅ Original config was restored after failed download"
else
echo "❌ Original config was not restored properly"
((++failed))
fi
fi
# Cleanup
rm -f "$test_config" "$test_config.test-backup"
return $failed
}
function test_download_error_handling() {
echo "🔍 Testing download error handling..."
local failed=0
# Test download with missing parameters
if safe_download "" "/tmp/test" 2>/dev/null; then
echo "❌ Download with empty URL should have failed"
((++failed))
else
echo "✅ Download with empty URL failed as expected"
fi
if safe_download "https://example.com" "" 2>/dev/null; then
echo "❌ Download with empty destination should have failed"
((++failed))
else
echo "✅ Download with empty destination failed as expected"
fi
# Test download to read-only location (should fail). Only meaningful for
# non-root users: root bypasses filesystem permissions, so the expected
# write failure never happens and the assertion is invalid.
if [[ $EUID -eq 0 ]]; then
echo "⏭️ Skipping read-only-location test (running as root; root bypasses FS perms)"
elif safe_download "https://github.com" "/test-readonly-$$" 2>/dev/null; then
echo "❌ Download to read-only location should have failed"
((++failed))
else
echo "✅ Download to read-only location failed as expected"
fi
return $failed
}
function test_download_performance() {
echo "🔍 Testing download performance..."
local test_url="https://raw.githubusercontent.com/torvalds/linux/master/README"
local test_dest="/tmp/perf-test-$$"
local start_time end_time duration
start_time=$(date +%s)
if safe_download "$test_url" "$test_dest"; then
end_time=$(date +%s)
duration=$((end_time - start_time))
echo "✅ Download completed in ${duration}s"
if [[ $duration -gt 30 ]]; then
echo "⚠️ Download took longer than expected (>30s)"
else
echo "✅ Download performance acceptable"
fi
# Cleanup
rm -f "$test_dest"
return 0
else
echo "❌ Performance test download failed"
return 1
fi
}
# Main test execution
function main() {
echo "🧪 Running Safe Download Framework Unit Tests"
echo "==========================================="
local total_failures=0
# Run all tests
test_network_connectivity || ((total_failures++))
test_url_accessibility || ((total_failures++))
test_safe_download || ((total_failures++))
test_checksum_verification || ((total_failures++))
test_batch_download || ((total_failures++))
test_config_backup_and_restore || ((total_failures++))
test_download_error_handling || ((total_failures++))
test_download_performance || ((total_failures++))
echo "==========================================="
if [[ $total_failures -eq 0 ]]; then
echo "✅ All safe download framework tests passed"
exit 0
else
echo "❌ $total_failures safe download framework tests failed"
exit 1
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/validation/dns-ntp-redundancy.sh
+202
View File
@@ -0,0 +1,202 @@
#!/bin/bash
# Redundant DNS/NTP Validation Test
# Validates that the host is configured to use the redundant pfv-netinfra-01/02
# pair for name resolution and time, and that both servers actually answer.
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
# The authoritative pair (pfv-netinfra-01 / pfv-netinfra-02).
DNS_PRIMARY="192.168.3.252"
DNS_SECONDARY="192.168.3.253"
NTP_PRIMARY="192.168.3.252"
NTP_SECONDARY="192.168.3.253"
RESOLV_CONF="/etc/resolv.conf"
NTP_CONF="/etc/ntpsec/ntp.conf"
# A name every recursive resolver must be able to resolve.
DNS_PROBE_NAME="github.com"
failed=0
have() { command -v "$1" >/dev/null 2>&1; }
# --- Configuration assertions -------------------------------------------------
function test_dns_config_present() {
echo "🔍 Checking $RESOLV_CONF ..."
local problems=0
if [[ -L "$RESOLV_CONF" ]]; then
echo "❌ $RESOLV_CONF is a symlink (would be overwritten by a resolver manager)"
((++problems))
elif [[ ! -f "$RESOLV_CONF" ]]; then
echo "❌ $RESOLV_CONF missing"
((++problems))
fi
for ns in "$DNS_PRIMARY" "$DNS_SECONDARY"; do
if grep -Eq "^[[:space:]]*nameserver[[:space:]]+$ns" "$RESOLV_CONF" 2>/dev/null; then
echo "✅ nameserver $ns configured"
else
echo "❌ nameserver $ns NOT in $RESOLV_CONF"
((++problems))
fi
done
return $problems
}
function test_ntp_config_present() {
echo "🔍 Checking $NTP_CONF ..."
if [[ ! -f "$NTP_CONF" ]]; then
echo "❌ $NTP_CONF missing (is ntpsec installed?)"
return 1
fi
local problems=0
for s in "$NTP_PRIMARY" "$NTP_SECONDARY"; do
if grep -Eq "^[[:space:]]*(server|pool)[[:space:]]+$s" "$NTP_CONF"; then
echo "✅ NTP server $s configured"
else
echo "❌ NTP server $s NOT in $NTP_CONF"
((++problems))
fi
done
return $problems
}
# --- Functional assertions: each server actually answers ----------------------
function _dns_resolves() {
# $1 = server ip. Returns 0 if it resolves DNS_PROBE_NAME.
local server="$1"
if have dig; then
dig @"$server" +short +time=4 +tries=1 "$DNS_PROBE_NAME" A 2>/dev/null | grep -Eq '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'
elif have nslookup; then
nslookup "$DNS_PROBE_NAME" "$server" 2>/dev/null | grep -Eq 'Address:[[:space:]]*[0-9]'
elif have host; then
host "$DNS_PROBE_NAME" "$server" 2>/dev/null | grep -Eq 'has address'
else
# Last resort: the resolver itself.
getent ahostsv4 "$DNS_PROBE_NAME" >/dev/null 2>&1
fi
}
function test_dns_servers_answer() {
echo "🔍 Probing DNS servers ..."
local problems=0
for ns in "$DNS_PRIMARY" "$DNS_SECONDARY"; do
if _dns_resolves "$ns"; then
echo "✅ $ns resolves $DNS_PROBE_NAME"
else
echo "❌ $ns did not resolve $DNS_PROBE_NAME"
((++problems))
fi
done
return $problems
}
function _ntp_answers() {
# $1 = server ip. Returns 0 if it responds to a time query.
local server="$1"
if have ntpdate; then
timeout 8 ntpdate -q "$server" 2>/dev/null | grep -Eq 'no-leap|leap'
elif have sntp; then
timeout 8 sntp -t 4 "$server" >/dev/null 2>&1
elif have chronyc; then
# NTS/chrony not expected here, but be tolerant.
chronyc -n -h "$server" tracking >/dev/null 2>&1
else
return 2 # cannot test
fi
}
function test_ntp_servers_answer() {
echo "🔍 Probing NTP servers ..."
local problems=0
for s in "$NTP_PRIMARY" "$NTP_SECONDARY"; do
if _ntp_answers "$s"; then
echo "✅ NTP $s responds to time query"
else
echo "❌ $s did not respond to NTP query"
((++problems))
fi
done
return $problems
}
# --- End-to-end: the host is actually USING the pair --------------------------
function test_resolver_endtoend() {
echo "🔍 End-to-end resolution via $RESOLV_CONF ..."
if getent ahostsv4 "$DNS_PROBE_NAME" >/dev/null 2>&1; then
echo "✅ Host resolves $DNS_PROBE_NAME via configured resolver"
return 0
else
echo "❌ Host cannot resolve $DNS_PROBE_NAME via configured resolver"
return 1
fi
}
function test_ntp_daemon_peers() {
echo "🔍 NTP daemon peer list ..."
local peers
if have ntpq; then
peers="$(ntpq -pn 2>/dev/null || true)"
elif have chronyc; then
peers="$(chronyc -n sources 2>/dev/null || true)"
else
echo "⚠️ No ntpq/chronyc available; skipping daemon peer check"
return 0
fi
local problems=0
for s in "$NTP_PRIMARY" "$NTP_SECONDARY"; do
if echo "$peers" | grep -Eq "^\\s*${s//./\\.}"; then
echo "✅ NTP daemon has peer $s"
else
echo "❌ NTP daemon is NOT tracking $s"
((++problems))
fi
done
# Sync status is informational only: a freshly started daemon needs several
# polls before the reach counter stabilises, so we warn rather than fail.
if echo "$peers" | grep -Eq '\*'; then
echo "✅ NTP daemon reports a synced peer"
else
echo "⚠️ NTP daemon not yet synced (normal for a few minutes after restart)"
fi
return $problems
}
# --- Main ---------------------------------------------------------------------
function main() {
echo "🛰️ Running Redundant DNS/NTP Validation Tests"
echo "================================================"
local total_failures=0
test_dns_config_present || ((++total_failures))
test_ntp_config_present || ((++total_failures))
test_dns_servers_answer || ((++total_failures))
test_ntp_servers_answer || ((++total_failures))
test_resolver_endtoend || ((++total_failures))
test_ntp_daemon_peers || ((++total_failures))
echo "================================================"
if [[ $total_failures -eq 0 ]]; then
echo "✅ All redundant DNS/NTP validation tests passed"
exit 0
else
echo "❌ $total_failures redundant DNS/NTP tests failed"
exit 1
fi
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/validation/system-requirements.sh
Executable
+142
View File
@@ -0,0 +1,142 @@
#!/bin/bash
# System Requirements Validation Test
# Validates minimum system requirements before deployment
set -euo pipefail
# Test configuration
MIN_RAM_GB=2
MIN_DISK_GB=10
REQUIRED_COMMANDS=("curl" "wget" "git" "systemctl" "apt-get")
# Test functions
function test_memory_requirements() {
local total_mem_kb=$(grep MemTotal /proc/meminfo | awk '{print $2}')
local total_mem_gb=$((total_mem_kb / 1024 / 1024))
if [[ $total_mem_gb -ge $MIN_RAM_GB ]]; then
echo "✅ Memory requirement met: ${total_mem_gb}GB >= ${MIN_RAM_GB}GB"
return 0
else
echo "❌ Memory requirement not met: ${total_mem_gb}GB < ${MIN_RAM_GB}GB"
return 1
fi
}
function test_disk_space() {
local available_gb=$(df / | tail -1 | awk '{print int($4/1024/1024)}')
if [[ $available_gb -ge $MIN_DISK_GB ]]; then
echo "✅ Disk space requirement met: ${available_gb}GB >= ${MIN_DISK_GB}GB"
return 0
else
echo "❌ Disk space requirement not met: ${available_gb}GB < ${MIN_DISK_GB}GB"
return 1
fi
}
function test_required_commands() {
local failed=0
for cmd in "${REQUIRED_COMMANDS[@]}"; do
if command -v "$cmd" >/dev/null 2>&1; then
echo "✅ Required command available: $cmd"
else
echo "❌ Required command missing: $cmd"
((++failed))
fi
done
return $failed
}
function test_os_compatibility() {
if [[ -f /etc/os-release ]]; then
local os_id=$(grep "^ID=" /etc/os-release | cut -d'=' -f2 | tr -d '"')
local os_version=$(grep "^VERSION_ID=" /etc/os-release | cut -d'=' -f2 | tr -d '"')
case "$os_id" in
ubuntu|debian)
echo "✅ OS compatibility: $os_id $os_version (supported)"
return 0
;;
*)
echo "⚠️ OS compatibility: $os_id $os_version (may work, not fully tested)"
return 0
;;
esac
else
echo "❌ Cannot determine OS version"
return 1
fi
}
function test_network_connectivity() {
local test_urls=(
"https://archive.ubuntu.com"
"https://linux.dell.com"
"https://download.proxmox.com"
"https://github.com"
)
local failed=0
for url in "${test_urls[@]}"; do
if curl -s --connect-timeout 10 --max-time 30 "$url" >/dev/null 2>&1; then
echo "✅ Network connectivity: $url"
else
echo "❌ Network connectivity failed: $url"
((++failed))
fi
done
return $failed
}
function test_permissions() {
local test_dirs=("/etc" "/usr/local/bin" "/var/log")
local failed=0
for dir in "${test_dirs[@]}"; do
if [[ -w "$dir" ]]; then
echo "✅ Write permission: $dir"
else
echo "❌ Write permission denied: $dir"
((++failed))
fi
done
return $failed
}
# Main test execution
function main() {
echo "🔍 Running System Requirements Validation"
echo "========================================"
local total_failures=0
# Run all validation tests
test_memory_requirements || ((total_failures++))
test_disk_space || ((total_failures++))
test_required_commands || ((total_failures++))
test_os_compatibility || ((total_failures++))
test_network_connectivity || ((total_failures++))
test_permissions || ((total_failures++))
echo "========================================"
if [[ $total_failures -eq 0 ]]; then
echo "✅ All system requirements validation tests passed"
exit 0
else
echo "❌ $total_failures system requirements validation tests failed"
exit 1
fi
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
tests/vm-validation.sh
Executable
+330
View File
@@ -0,0 +1,330 @@
#!/usr/bin/bash
#
# vm-validation.sh
#
# End-to-end validation driver for KNELServerBuild on a sandbox VM.
#
# This script drives a Proxmox VM through: snapshot -> deploy -> validate, with
# one-command rollback. It is designed to be re-run after code fixes are pushed.
#
# DESIGN: deployment is GIT-BASED. The VM clones (or pulls) the public repo
# itself, exactly as a real fresh server would — so the result is identical no
# matter who runs this script (no reliance on a local working copy or rsync).
# All SSH/SCP access goes through tests/remote.sh; never call ssh here.
#
# USAGE:
# # Discover the numeric VMID on Proxmox:
# ./tests/vm-validation.sh find-vmid
#
# # Full loop (snapshot + deploy + validate), auto-rollback on failure:
# VM_ID=6000 ./tests/vm-validation.sh all
#
# # Individual steps:
# VM_ID=6000 ./tests/vm-validation.sh snapshot
# VM_ID=6000 ./tests/vm-validation.sh deploy
# VM_ID=6000 ./tests/vm-validation.sh validate
# VM_ID=6000 ./tests/vm-validation.sh rollback [snapshot-name]
#
# # Clean re-deploy from scratch (delete + re-clone on VM):
# VM_ID=6000 CLEAN_CLONE=1 ./tests/vm-validation.sh deploy
#
# CONFIG (override via env, all have sensible defaults):
# PROX_HOST Proxmox node hostname (default: pfv-tsys5)
# PROX_USER SSH user on Proxmox (default: root)
# VM_NAME VM name for VMID lookup/logging (default: sectestbed-sandbox)
# VM_IP VM IP for SSH (default: 192.168.3.50)
# VM_USER SSH user on the VM (default: localuser)
# VM_ID Numeric VMID on Proxmox (REQUIRED except for find-vmid)
# REPO_URL git URL the VM clones (default: https://git.knownelement.com/KNEL/KNELServerBuild.git)
# REMOTE_REPO clone dir under ~$VM_USER (default: KNELServerBuild)
# SNAP_PREFIX snapshot name prefix (default: pre-knel-deploy)
# CLEAN_CLONE if set, delete + re-clone on VM (default: unset)
#
set -uo pipefail
# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------
PROX_HOST="${PROX_HOST:-pfv-tsys5}"
PROX_USER="${PROX_USER:-root}"
VM_NAME="${VM_NAME:-sectestbed-sandbox}"
VM_IP="${VM_IP:-192.168.3.50}"
VM_USER="${VM_USER:-localuser}"
VM_ID="${VM_ID:-}"
REPO_URL="${REPO_URL:-https://git.knownelement.com/KNEL/KNELServerBuild.git}"
REMOTE_REPO="${REMOTE_REPO:-KNELServerBuild}"
SNAP_PREFIX="${SNAP_PREFIX:-pre-knel-deploy}"
ACCESS_PUBKEY="${ACCESS_PUBKEY:-$HOME/.ssh/id_ed25519.pub}"
# Re-inject the validation pubkey after each deploy (secharden-ssh replaces
# authorized_keys with the managed production key set, locking out the
# bootstrap/dev key). Set RESTORE_ACCESS=0 to disable.
RESTORE_ACCESS="${RESTORE_ACCESS:-1}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_SRC="$(cd "$HERE/.." && pwd)"
REMOTE="$HERE/remote.sh"
STAMP="$(date +%Y%m%d-%H%M%S)"
SNAP_NAME="${SNAP_PREFIX}-${STAMP}"
LOCAL_LOG_DIR="$REPO_SRC/logs/vm-validation"
mkdir -p "$LOCAL_LOG_DIR"
LOCAL_LOG="$LOCAL_LOG_DIR/run-${STAMP}.log"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" | tee -a "$LOCAL_LOG"; }
die() { log "ERROR: $*"; exit 1; }
# All remote access funnels through remote.sh.
vm() { bash "$REMOTE" vm "$@"; } # as $VM_USER (SSH)
vmroot() { bash "$REMOTE" vmroot "$@"; } # as root via sudo (SSH)
vmfile() { bash "$REMOTE" vm-file "$@"; } # run local script on VM (SSH)
vmguest() { bash "$REMOTE" vm-guest "$@"; } # as root via guest agent (no SSH/2FA)
prox() { bash "$REMOTE" prox "$@"; } # as $PROX_USER on Proxmox
require_vm_id() {
[[ -n "$VM_ID" ]] || die "VM_ID is required for this command. Find it with: $0 find-vmid"
}
wait_for_vm_ssh() {
log "Waiting for SSH on ${VM_USER}@${VM_IP} to come up..."
for i in $(seq 1 60); do
if vm 'true' >/dev/null 2>&1; then
log "SSH is up (after ${i} tries)."
return 0
fi
sleep 5
done
die "VM did not become SSH-reachable within 5 minutes."
}
# Resolve the ABSOLUTE path of the repo clone on the VM (as $VM_USER) and echo
# it. Using an absolute path avoids the '~' -> root's home trap under sudo.
resolve_remote_repo() {
local p
p="$(vm "cd ~/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
[[ -n "$p" ]] || p="$(vmguest "cd ~${VM_USER}/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
printf '%s' "$p"
}
# Re-inject the validation pubkey into ~$VM_USER/.ssh/authorized_keys OUT OF
# BAND via the Proxmox guest agent (qm guest exec runs as root inside the VM
# and does not depend on SSH). This is necessary because secharden-ssh replaces
# authorized_keys with the managed production key set, which would otherwise
# lock out the bootstrap key used to drive validation. No-op if SSH still works.
restore_vm_access() {
[[ "$RESTORE_ACCESS" = "1" ]] || { log "RESTORE_ACCESS=0; skipping access restore."; return 0; }
[[ -f "$ACCESS_PUBKEY" ]] || { log "WARN: ACCESS_PUBKEY not found ($ACCESS_PUBKEY); cannot restore access."; return 0; }
if vm 'true' >/dev/null 2>&1; then
log "SSH access already works; no need to restore."
return 0
fi
log "SSH access lost (expected after secharden-ssh). Restoring via Proxmox guest agent..."
local payload_b64
# Leading newline guards against the managed authorized_keys lacking a
# trailing newline (which would otherwise concatenate two keys into one).
payload_b64="$(printf '\n%s' "$(cat "$ACCESS_PUBKEY")" | base64 -w0)"
prox "qm guest exec $VM_ID -- /bin/sh -c 'echo $payload_b64 | base64 -d >> /home/${VM_USER}/.ssh/authorized_keys'" \
>/dev/null 2>&1 || { log "WARN: guest-agent key append failed."; return 0; }
prox "qm guest exec $VM_ID -- /bin/sh -c 'chown ${VM_USER}:${VM_USER} /home/${VM_USER}/.ssh/authorized_keys; chmod 600 /home/${VM_USER}/.ssh/authorized_keys'" \
>/dev/null 2>&1 || true
if vm 'true' >/dev/null 2>&1; then
log "Access restored."
return 0
fi
# If SSH still fails after re-injecting the key, 2FA is almost certainly the
# cause (secharden-2fa enforces publickey+keyboard-interactive, which no
# non-interactive SSH client can satisfy). That is expected and not fatal:
# the guest agent still gives us full out-of-band access for log fetch and
# the validation suite.
if vmguest 'grep -q "^AuthenticationMethods" /etc/ssh/sshd_config' >/dev/null 2>&1; then
log "SSH requires 2FA (expected after secharden-2fa); using guest agent for further access."
else
log "WARN: access still not working after restore and 2FA not detected. Check sshd_config."
fi
}
# ---------------------------------------------------------------------------
# Commands
# ---------------------------------------------------------------------------
cmd_find_vmid() {
log "Listing VMs on Proxmox host '$PROX_HOST' matching '$VM_NAME':"
prox 'qm list' 2>&1 | tee -a "$LOCAL_LOG" \
| { IFS= read -r header; echo "$header"; grep -i "$VM_NAME" || true; }
log "Set VM_ID=<number> env var based on the row above."
}
cmd_snapshot() {
require_vm_id
log "Creating snapshot '$SNAP_NAME' of VMID $VM_ID on $PROX_HOST..."
prox "qm snapshot $VM_ID $SNAP_NAME --vmstate 1" 2>&1 | tee -a "$LOCAL_LOG" \
|| die "Snapshot creation failed."
echo "$SNAP_NAME" > "$LOCAL_LOG_DIR/.last-snapshot"
log "Snapshot '$SNAP_NAME' recorded as rollback target."
}
cmd_rollback() {
require_vm_id
local target="${1:-$(cat "$LOCAL_LOG_DIR/.last-snapshot" 2>/dev/null || true)}"
[[ -n "$target" ]] || die "No snapshot name given and no .last-snapshot on disk."
log "Rolling back VMID $VM_ID to snapshot '$target'..."
# Proxmox rollback requires the VM to be stopped.
prox "qm stop $VM_ID" 2>&1 | tee -a "$LOCAL_LOG" || true
sleep 5
prox "qm rollback $VM_ID $target" 2>&1 | tee -a "$LOCAL_LOG" \
|| die "Rollback command failed."
log "Starting VMID $VM_ID..."
prox "qm start $VM_ID" 2>&1 | tee -a "$LOCAL_LOG" || true
wait_for_vm_ssh
log "Rollback complete."
}
# Ensure the VM has git + ca-certificates (fresh-server bootstrap).
bootstrap_git_on_vm() {
log "Ensuring git is present on the VM..."
vm 'command -v git >/dev/null 2>&1 || sudo -n DEBIAN_FRONTEND=noninteractive apt-get -y -qq install git ca-certificates' \
2>&1 | tee -a "$LOCAL_LOG" || die "Failed to bootstrap git on VM."
vm 'sudo -n DEBIAN_FRONTEND=noninteractive apt-get -y -qq install ca-certificates' 2>&1 | tee -a "$LOCAL_LOG" || true
}
# Clone or pull the repo on the VM. Returns absolute path on stdout (via log).
sync_repo_on_vm() {
bootstrap_git_on_vm
if [[ -n "${CLEAN_CLONE:-}" ]]; then
log "CLEAN_CLONE set: removing existing clone on VM."
vm "rm -rf ~/${REMOTE_REPO}" 2>&1 | tee -a "$LOCAL_LOG" || true
fi
log "Ensuring repo is cloned/pulled on the VM from:"
log " $REPO_URL"
vm "
set -e
if [ -d ~/${REMOTE_REPO}/.git ]; then
cd ~/${REMOTE_REPO}
git fetch --all --prune
git reset --hard origin/HEAD 2>/dev/null || git reset --hard origin/main
git clean -xfd
else
git clone --filter=blob:none '$REPO_URL' ~/${REMOTE_REPO}
cd ~/${REMOTE_REPO}
fi
git log --oneline -1
" 2>&1 | tee -a "$LOCAL_LOG" || die "Repo sync failed on VM."
log "Repo ready on VM."
}
# The remote setup runner: a self-contained script we ship to the VM so the
# sudo'd setup runs from a known-good absolute path with full logging. Using a
# file avoids nested-quote hell across local -> ssh -> sudo -> bash -c.
deploy_runner_script() {
cat <<RUNNER
#!/usr/bin/bash
# remote-setup-runner.sh (generated by vm-validation.sh)
# Runs provisioning/SetupNewSystem.sh from the repo given by \$1, as root.
set -uo pipefail
# Ensure a sane TERM so the framework's tput-based color helpers work when run
# over a non-interactive SSH session (which has no TTY/TERM by default).
export TERM="\${TERM:-linux}"
REPO_ABS="\${1:?repo abs path required}"
REMOTE_LOG="/tmp/knel-setup.log"
echo "=== KNEL SetupNewSystem start: \$(date -Is) repo=\$REPO_ABS ===" | tee -a "\$REMOTE_LOG"
cd "\$REPO_ABS/ProjectCode" || { echo "FATAL: ProjectCode missing at \$REPO_ABS"; exit 2; }
bash SetupNewSystem.sh 2>&1 | tee -a "\$REMOTE_LOG"
rc=\${PIPESTATUS[0]}
echo "=== KNEL SetupNewSystem end: rc=\$rc \$(date -Is) ===" | tee -a "\$REMOTE_LOG"
exit \$rc
RUNNER
}
cmd_deploy() {
require_vm_id
sync_repo_on_vm
local repo_abs
repo_abs="$(resolve_remote_repo)"
[[ -n "$repo_abs" ]] || die "Could not resolve absolute repo path on VM."
log "Repo absolute path on VM: $repo_abs"
# Ship the runner script and execute it as root via sudo, passing abs path.
local runner_local="$LOCAL_LOG_DIR/remote-setup-runner.sh"
deploy_runner_script > "$runner_local"
vm "mkdir -p ~/${REMOTE_REPO}/tests/.run" 2>&1 | tee -a "$LOCAL_LOG"
bash "$REMOTE" vm-copy "$runner_local" "${REMOTE_REPO}/tests/.run/remote-setup-runner.sh" \
2>&1 | tee -a "$LOCAL_LOG" || die "Failed to ship runner script."
log "Running SetupNewSystem.sh on the VM as root (this takes several minutes)..."
# Resolve abs runner path the same way (no ~ under sudo).
local runner_abs
runner_abs="$(vm "cd ~/${REMOTE_REPO}/tests/.run && pwd")/remote-setup-runner.sh"
vmroot "bash '$runner_abs' '$repo_abs'" 2>&1 | tee -a "$LOCAL_LOG" || true
# secharden-ssh (run near the end of setup) replaces authorized_keys with the
# managed production key set, locking out the bootstrap key. Restore the
# validation key out-of-band BEFORE we try to fetch the log over SSH.
restore_vm_access
# Fetch the remote log for full fidelity (strip ANSI color codes). SSH works
# only until secharden-2fa flips 2FA on; after that, use the guest agent.
local fetch_cmd="sed -r 's/\\x1B\\[[0-9;]*[mK]//g' /tmp/knel-setup.log 2>/dev/null || cat /tmp/knel-setup.log"
if ! vm "$fetch_cmd" > "$LOCAL_LOG_DIR/setup-output-${STAMP}.log" 2>/dev/null; then
vmguest "$fetch_cmd" > "$LOCAL_LOG_DIR/setup-output-${STAMP}.log" 2>/dev/null || true
fi
# Detect the exit marker. Prefer the full fetched log, but always fall back
# to the live stream ($LOCAL_LOG) which is captured regardless of whether
# post-setup SSH/2FA let us fetch the remote log.
local rc_marker
rc_marker=$(grep -oE 'rc=[0-9]+' "$LOCAL_LOG_DIR/setup-output-${STAMP}.log" 2>/dev/null | tail -1 || true)
[[ -n "$rc_marker" ]] || rc_marker=$(grep -oE 'rc=[0-9]+' "$LOCAL_LOG" 2>/dev/null | tail -1 || true)
log "Setup run finished. Marker: ${rc_marker:-unknown}"
if [[ "${rc_marker:-}" != "rc=0" ]]; then
log "Setup did NOT complete cleanly. See: $LOCAL_LOG_DIR/setup-output-${STAMP}.log (and $LOCAL_LOG)"
return 1
fi
log "Setup completed successfully."
}
cmd_validate() {
require_vm_id
log "Running post-deploy validation suite on the VM..."
local repo_abs
repo_abs="$(resolve_remote_repo)"
[[ -n "$repo_abs" ]] || die "Could not resolve absolute repo path on VM."
# Prefer SSH; fall back to the guest agent (post-2FA SSH needs a TOTP token).
if ! vmroot "cd '$repo_abs' && bash tests/run-tests.sh all" 2>&1 | tee -a "$LOCAL_LOG"; then
vmguest "cd '$repo_abs' && bash tests/run-tests.sh all" 2>&1 | tee -a "$LOCAL_LOG" || true
fi
log "Validation run finished. Inspect output above / in $LOCAL_LOG."
}
cmd_all() {
require_vm_id
log "=== FULL VALIDATION LOOP: $VM_NAME (VMID $VM_ID) ==="
cmd_snapshot
if cmd_deploy && cmd_validate; then
log "=== ALL GREEN ==="
return 0
fi
log "=== FAILURE — auto-rolling back to '$SNAP_NAME' ==="
cmd_rollback "$SNAP_NAME"
log "Rolled back. Fix and push, then re-run: VM_ID=$VM_ID $0 deploy && VM_ID=$VM_ID $0 validate"
return 1
}
# ---------------------------------------------------------------------------
# Dispatch
# ---------------------------------------------------------------------------
subcmd="${1:-}"
case "$subcmd" in
find-vmid) cmd_find_vmid ;;
snapshot) cmd_snapshot ;;
deploy) cmd_deploy ;;
validate) cmd_validate ;;
rollback) cmd_rollback "${2:-}" ;;
all) cmd_all ;;
""|-h|--help|help)
sed -n '2,49p' "${BASH_SOURCE[0]}" >&2
exit 0
;;
*) die "Unknown command '$subcmd'. Run '$0 help'." ;;
esac