KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/server-build/DEPLOYMENT.md
+10 -336
View File
@@ -1,336 +1,10 @@
# TSYS FetchApply Deployment Guide
## Overview
This guide provides comprehensive instructions for deploying the TSYS FetchApply infrastructure provisioning system on Linux servers.
## Prerequisites
### System Requirements
- **Operating System:** Ubuntu 18.04+ or Debian 10+ (recommended)
- **RAM:** Minimum 2GB, recommended 4GB
- **Disk Space:** Minimum 10GB free space
- **Network:** Internet connectivity for package downloads
- **Privileges:** Root or sudo access required
### Required Tools
- `git` - Version control system
- `curl` - HTTP client for downloads
- `wget` - Alternative download tool
- `systemctl` - System service management
- `apt-get` - Package management (Debian/Ubuntu)
### Network Requirements
- **HTTPS access** to:
- `https://archive.ubuntu.com` (Ubuntu packages)
- `https://linux.dell.com` (Dell hardware support)
- `https://download.proxmox.com` (Proxmox packages)
- `https://github.com` (Git repositories)
## Pre-Deployment Validation
### 1. System Compatibility Check
```bash
# Clone repository
git clone [repository-url]
cd FetchApply
# Run system validation
./Project-Tests/validation/system-requirements.sh
```
### 2. Network Connectivity Test
```bash
# Test network connectivity
curl -I https://archive.ubuntu.com
curl -I https://linux.dell.com
curl -I https://download.proxmox.com
```
### 3. Permission Verification
```bash
# Verify write permissions
test -w /etc && echo "✅ /etc writable" || echo "❌ /etc not writable"
test -w /usr/local/bin && echo "✅ /usr/local/bin writable" || echo "❌ /usr/local/bin not writable"
```
## Deployment Methods
### Method 1: Standard Deployment (Recommended)
```bash
# 1. Clone repository
git clone [repository-url]
cd FetchApply
# 2. Run pre-deployment tests
./Project-Tests/run-tests.sh validation
# 3. Execute deployment
cd ProjectCode
sudo bash SetupNewSystem.sh
```
### Method 2: Dry Run Mode
```bash
# 1. Clone repository
git clone [repository-url]
cd FetchApply
# 2. Review configuration
cat provisioning/SetupNewSystem.sh
# 3. Execute with manual review
cd ProjectCode
sudo bash -x SetupNewSystem.sh # Debug mode
```
## Deployment Process
### Phase 1: Framework Initialization
1. **Environment Setup**
- Load framework variables
- Source framework includes
- Initialize logging system
2. **System Detection**
- Detect physical vs virtual hardware
- Identify operating system
- Check for existing users
### Phase 2: Base System Configuration
1. **Package Installation**
- Update package repositories
- Install essential packages
- Configure package sources
2. **User Management**
- Create required user accounts
- Configure SSH access
- Set up sudo permissions
### Phase 3: Security Hardening
1. **SSH Configuration**
- Deploy hardened SSH configuration
- Install SSH keys
- Disable password authentication
2. **System Hardening**
- Configure firewall rules
- Enable audit logging
- Install security tools
### Phase 4: Monitoring and Management
1. **Monitoring Agents**
- Deploy LibreNMS agents
- Configure SNMP
- Set up system monitoring
2. **Management Tools**
- Install Cockpit dashboard
- Configure remote access
- Set up maintenance scripts
## Post-Deployment Verification
### 1. Security Validation
```bash
# Run security tests
./Project-Tests/run-tests.sh security
# Verify SSH configuration
ssh -T [server-ip] # Should work with key authentication
```
### 2. Service Status Check
```bash
# Check critical services
sudo systemctl status ssh
sudo systemctl status auditd
sudo systemctl status snmpd
```
### 3. Network Connectivity
```bash
# Test internal services
curl -k https://localhost:9090 # Cockpit
snmpwalk -v2c -c public localhost system
```
## Troubleshooting
### Common Issues
#### 1. Permission Denied Errors
```bash
# Solution: Run with sudo
sudo bash SetupNewSystem.sh
```
#### 2. Network Connectivity Issues
```bash
# Check DNS resolution
nslookup archive.ubuntu.com
# Test direct IP access
curl -I 91.189.91.26 # Ubuntu archive IP
```
#### 3. Package Installation Failures
```bash
# Update package cache
sudo apt-get update
# Fix broken packages
sudo apt-get -f install
```
#### 4. SSH Key Issues
```bash
# Verify key permissions
ls -la ~/.ssh/
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub
```
### Debug Mode
```bash
# Enable debug logging
export DEBUG=1
bash -x SetupNewSystem.sh
```
### Log Analysis
```bash
# Check deployment logs
tail -f /var/log/fetchapply/deployment.log
# Review system logs
journalctl -u ssh
journalctl -u auditd
```
## Environment-Specific Configurations
### Physical Dell Servers
- **OMSA Installation:** Dell OpenManage Server Administrator
- **Hardware Monitoring:** iDRAC configuration
- **Performance Tuning:** CPU and memory optimizations
### Virtual Machines
- **Guest Additions:** VMware tools or VirtualBox additions
- **Resource Limits:** Memory and CPU constraints
- **Network Configuration:** Bridge vs NAT settings
### Development Environments
- **SSH Configuration:** Less restrictive settings
- **Development Tools:** Additional packages for development
- **Testing Access:** Enhanced logging and debugging
## Maintenance and Updates
### Regular Maintenance
```bash
# Update system packages
sudo apt-get update && sudo apt-get upgrade
# Update monitoring scripts
cd /usr/local/bin
sudo wget https://[repository]/scripts/up2date.sh
sudo chmod +x up2date.sh
```
### Security Updates
```bash
# Check for security updates
sudo apt-get update
sudo apt list --upgradable | grep -i security
# Apply security patches
sudo apt-get upgrade
```
### Configuration Updates
```bash
# Update FetchApply
cd FetchApply
git pull origin main
# Re-run specific modules
cd provisioning/Modules/Security
sudo bash secharden-ssh.sh
```
## Best Practices
### 1. Pre-Deployment
- Always test in non-production environment first
- Review all scripts before execution
- Validate network connectivity
- Ensure proper backup procedures
### 2. During Deployment
- Monitor deployment progress
- Check for errors and warnings
- Document any customizations
- Validate each phase completion
### 3. Post-Deployment
- Run full security test suite
- Verify all services are running
- Test remote access
- Document deployment specifics
### 4. Ongoing Operations
- Regular security updates
- Monitor system performance
- Review audit logs
- Maintain deployment documentation
## Support and Resources
### Documentation
- **README.md:** Basic usage instructions
- **SECURITY.md:** Security architecture and guidelines
- **tests/README.md:** Testing framework documentation
### Community Support
- **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
- **Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26
### Professional Support
- **Technical Support:** [Contact information to be added]
- **Consulting Services:** [Contact information to be added]
## Deployment Checklist
### Pre-Deployment
- [ ] System requirements validated
- [ ] Network connectivity tested
- [ ] Backup procedures in place
- [ ] Security review completed
### Deployment
- [ ] Repository cloned successfully
- [ ] Pre-deployment tests passed
- [ ] Deployment executed without errors
- [ ] Post-deployment verification completed
### Post-Deployment
- [ ] Security tests passed
- [ ] All services running
- [ ] Remote access verified
- [ ] Documentation updated
### Maintenance
- [ ] Update schedule established
- [ ] Monitoring configured
- [ ] Backup procedures tested
- [ ] Incident response plan activated
## Version History
- **v1.0:** Initial deployment framework
- **v1.1:** Added security hardening and secrets management
- **v1.2:** Enhanced testing framework and documentation
Last updated: July 14, 2025
# docs/server-build/DEPLOYMENT.md
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Server deployment procedures**
>
> **Read it here:** https://community.turnsys.com/t/302
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
docs/server-build/DEVELOPMENT-GUIDELINES.md
+10 -407
View File
@@ -1,407 +1,10 @@
<!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
# TSYS PFVCluster Development Guidelines
## Overview
This document contains development standards and best practices for the TSYS PFVCluster infrastructure provisioning system.
## Package Management Best Practices
### Combine apt-get Install Commands
**Rule:** Always combine multiple package installations into a single `apt-get install` command for performance.
**Rationale:** Single command execution is significantly faster than multiple separate commands due to:
- Reduced package cache processing
- Single dependency resolution
- Fewer network connections
- Optimized package download ordering
#### ✅ Correct Implementation
```bash
# Install all packages in one command
apt-get install -y package1 package2 package3 package4
# Real example from 2FA script
apt-get install -y libpam-google-authenticator qrencode
```
#### ❌ Incorrect Implementation
```bash
# Don't use separate commands for each package
apt-get install -y package1
apt-get install -y package2
apt-get install -y package3
```
#### Complex Package Installation Pattern
```bash
function install_security_packages() {
print_info "Installing security packages..."
# Update package cache once
apt-get update
# Install all packages in single command
apt-get install -y \
auditd \
fail2ban \
libpam-google-authenticator \
lynis \
rkhunter \
aide \
chkrootkit \
clamav \
clamav-daemon
print_success "Security packages installed successfully"
}
```
## Script Development Standards
### Error Handling
- Always use `set -euo pipefail` at script start
- Implement proper error trapping
- Use framework error handling functions
- Return appropriate exit codes
### Function Structure
```bash
function function_name() {
print_info "Description of what function does..."
# Local variables
local var1="value"
local var2="value"
# Function logic
if [[ condition ]]; then
print_success "Success message"
return 0
else
print_error "Error message"
return 1
fi
}
```
### Framework Integration
- Source framework includes at script start
- Use framework logging and pretty print functions
- Follow existing patterns for consistency
- Include proper PROJECT_ROOT path resolution
```bash
# Standard framework sourcing pattern
PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.."
source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh"
source "$PROJECT_ROOT/Framework-Includes/Logging.sh"
source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh"
```
## Code Quality Standards
### ShellCheck Compliance
- All scripts must pass shellcheck validation
- Address shellcheck warnings appropriately
- Use proper quoting for variables
- Handle edge cases and error conditions
### Variable Naming
- Use UPPERCASE for global constants
- Use lowercase for local variables
- Use descriptive names
- Quote all variable expansions
```bash
# Global constants
declare -g BACKUP_DIR="/root/backup"
declare -g CONFIG_FILE="/etc/ssh/sshd_config"
# Local variables
local user_name="localuser"
local temp_file="/tmp/config.tmp"
# Proper quoting
if [[ -f "$CONFIG_FILE" ]]; then
cp "$CONFIG_FILE" "$BACKUP_DIR/"
fi
```
### Function Documentation
- Include purpose description
- Document parameters if any
- Document return values
- Include usage examples for complex functions
```bash
# Configure SSH hardening settings
# Parameters: none
# Returns: 0 on success, 1 on failure
# Usage: configure_ssh_hardening
function configure_ssh_hardening() {
print_info "Configuring SSH hardening..."
# Implementation
}
```
## Testing Requirements
### Test Coverage
- Every new module must include corresponding tests
- Test both success and failure scenarios
- Validate configurations after changes
- Include integration tests for complex workflows
### Test Categories
1. **Unit Tests:** Individual function validation
2. **Integration Tests:** Module interaction testing
3. **Security Tests:** Security configuration validation
4. **Validation Tests:** System requirement checking
### Test Implementation Pattern
```bash
function test_function_name() {
echo "🔍 Testing specific functionality..."
local failed=0
# Test implementation
if [[ condition ]]; then
echo "✅ Test passed"
else
echo "❌ Test failed"
((failed++))
fi
return $failed
}
```
## Security Standards
### Configuration Backup
- Always backup configurations before modification
- Use timestamped backup directories
- Provide restore instructions
- Test backup/restore procedures
### Service Management
- Test configurations before restarting services
- Provide rollback procedures
- Validate service status after changes
- Include service dependency handling
### User Safety
- Use `nullok` for gradual 2FA rollout
- Provide clear setup instructions
- Include emergency access procedures
- Test all access methods before enforcement
## Documentation Standards
### Script Headers
```bash
#!/bin/bash
# TSYS Module Name - Brief Description
# Longer description of what this script does
# Author: TSYS Development Team
# Version: 1.0
# Last Updated: YYYY-MM-DD
set -euo pipefail
```
### Inline Documentation
- Comment complex logic
- Explain non-obvious decisions
- Document external dependencies
- Include troubleshooting notes
### User Documentation
- Create comprehensive guides for complex features
- Include step-by-step procedures
- Provide troubleshooting sections
- Include examples and use cases
## Performance Optimization
### Package Management
- Single apt-get commands (as noted above)
- Cache package lists appropriately
- Use specific package versions when stability required
- Clean up package cache when appropriate
### Network Operations
- Use connection timeouts for external requests
- Implement retry logic with backoff
- Cache downloaded resources when possible
- Validate download integrity
### File Operations
- Use efficient file processing tools
- Minimize file system operations
- Use appropriate file permissions
- Clean up temporary files
## Version Control Practices
### Commit Messages
- Use descriptive commit messages
- Include scope of changes
- Reference related issues/requirements
- Follow established commit message format
### Branch Management
- Test changes in feature branches
- Use pull requests for review
- Maintain clean commit history
- Tag releases appropriately
### Code Review Requirements
- All changes require review
- Security changes require security team review
- Test coverage must be maintained
- Documentation must be updated
## Deployment Practices
### Pre-Deployment
- Run full test suite
- Validate in test environment
- Review security implications
- Update documentation
### Deployment Process
- Use configuration validation
- Implement gradual rollout when possible
- Monitor for issues during deployment
- Have rollback procedures ready
### Post-Deployment
- Validate deployment success
- Monitor system performance
- Update operational documentation
- Gather feedback for improvements
## Example Implementation
### Complete Module Template
```bash
#!/bin/bash
# TSYS Security Module - Template
# Template for creating new security modules
# Author: TSYS Development Team
set -euo pipefail
# Source framework functions
PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.."
source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh"
source "$PROJECT_ROOT/Framework-Includes/Logging.sh"
source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh"
# Module configuration
BACKUP_DIR="/root/backup/module-$(date +%Y%m%d-%H%M%S)"
CONFIG_FILE="/etc/example.conf"
# Create backup directory
mkdir -p "$BACKUP_DIR"
print_header "TSYS Module Template"
function backup_configs() {
print_info "Creating configuration backup..."
if [[ -f "$CONFIG_FILE" ]]; then
cp "$CONFIG_FILE" "$BACKUP_DIR/"
print_success "Configuration backed up"
fi
}
function install_packages() {
print_info "Installing required packages..."
# Update package cache
apt-get update
# Install all packages in single command
apt-get install -y package1 package2 package3
print_success "Packages installed successfully"
}
function configure_module() {
print_info "Configuring module..."
# Configuration logic here
print_success "Module configured successfully"
}
function validate_configuration() {
print_info "Validating configuration..."
local failed=0
# Validation logic here
if [[ $failed -eq 0 ]]; then
print_success "Configuration validation passed"
return 0
else
print_error "Configuration validation failed"
return 1
fi
}
function main() {
# Check if running as root
if [[ $EUID -ne 0 ]]; then
print_error "This script must be run as root"
exit 1
fi
# Execute module steps
backup_configs
install_packages
configure_module
validate_configuration
print_success "Module setup completed successfully!"
}
# Run main function
main "$@"
```
## Continuous Improvement
### Regular Reviews
- Review guidelines quarterly
- Update based on lessons learned
- Incorporate new best practices
- Gather team feedback
### Tool Updates
- Keep development tools current
- Adopt new security practices
- Update testing frameworks
- Improve automation
### Knowledge Sharing
- Document lessons learned
- Share best practices
- Provide training materials
- Maintain knowledge base
---
**Last Updated:** July 14, 2025
**Version:** 1.0
**Author:** TSYS Development Team
**Note:** These guidelines are living documents and should be updated as the project evolves and new best practices are identified.
# docs/server-build/DEVELOPMENT-GUIDELINES.md
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Coding standards, commit conventions, script patterns**
>
> **Read it here:** https://community.turnsys.com/t/302
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
docs/server-build/SECURITY.md
+9 -189
View File
@@ -1,190 +1,10 @@
# PFVCluster Security Documentation
# docs/server-build/SECURITY.md
## Security Architecture
The PFVCluster infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management.
## Current Security Features
### 1. Secure Deployment Method ✅
- **Git-based deployment:** Uses `git clone` instead of `curl | bash`
- **Local execution:** Scripts run locally after inspection
- **Version control:** Full audit trail of changes
- **Code review:** Changes require explicit approval
### 2. HTTPS Enforcement ✅
- **All downloads use HTTPS:** Eliminates man-in-the-middle attacks
- **SSL certificate validation:** Automatic certificate checking
- **Secure repositories:** Ubuntu archive, Dell, Proxmox all use HTTPS
- **No HTTP fallbacks:** No insecure download methods
### 3. SSH Hardening
- **Key-only authentication:** Password login disabled
- **Secure ciphers:** Modern encryption algorithms only
- **Fail2ban protection:** Automated intrusion prevention
- **Custom SSH configuration:** Hardened sshd_config
### 4. System Security
- **Firewall configuration:** Automated iptables rules
- **Audit logging:** auditd with custom rules
- **SIEM integration:** Wazuh agent deployment
- **Compliance scanning:** SCAP-STIG automated checks
### 5. Error Handling
- **Bash strict mode:** `set -euo pipefail` prevents errors
- **Centralized logging:** All operations logged with timestamps
- **Graceful failures:** Proper cleanup on errors
- **Line-level debugging:** Error reporting with line numbers
## Security Testing
### Automated Security Validation
```bash
# Run security test suite
./tests/run-tests.sh security
# Specific security tests
./tests/security/https-enforcement.sh
```
### Security Test Categories
1. **HTTPS Enforcement:** Validates all URLs use HTTPS
2. **Deployment Security:** Checks for secure deployment methods
3. **SSL Certificate Validation:** Tests certificate authenticity
4. **Permission Validation:** Verifies proper file permissions
## Threat Model
### Mitigated Threats
- **Supply Chain Attacks:** Git-based deployment with review
- **Man-in-the-Middle:** HTTPS-only downloads
- **Privilege Escalation:** Proper permission models
- **Unauthorized Access:** SSH hardening and key management
### Remaining Risks
- **Secrets in Repository:** SSH keys stored in git (planned for removal)
- **No Integrity Verification:** Downloads lack checksum validation
- **No Backup/Recovery:** No rollback capability implemented
## Security Recommendations
### High Priority
1. **Implement Secrets Management**
- Remove SSH keys from repository
- Use Bitwarden/Vault for secret storage
- Implement key rotation procedures
2. **Add Download Integrity Verification**
- SHA256 checksum validation for all downloads
- GPG signature verification where available
- Fail-safe on integrity check failures
3. **Enhance Audit Logging**
- Centralized log collection
- Real-time security monitoring
- Automated threat detection
### Medium Priority
1. **Configuration Backup**
- System state snapshots before changes
- Rollback capability for failed deployments
- Configuration drift detection
2. **Network Security**
- VPN-based deployment (where applicable)
- Network segmentation for management
- Encrypted communication channels
## Compliance
### Security Standards
- **CIS Benchmarks:** Automated compliance checking
- **STIG Guidelines:** SCAP-based validation
- **Industry Best Practices:** Following NIST cybersecurity framework
### Audit Requirements
- **Change Tracking:** All modifications logged
- **Access Control:** Permission-based system access
- **Vulnerability Management:** Regular security assessments
## Incident Response
### Security Event Handling
1. **Detection:** Automated monitoring and alerting
2. **Containment:** Immediate isolation procedures
3. **Investigation:** Log analysis and forensics
4. **Recovery:** System restoration procedures
5. **Lessons Learned:** Process improvement
### Contact Information
- **Security Team:** [To be defined]
- **Incident Response:** [To be defined]
- **Escalation Path:** [To be defined]
## Security Development Lifecycle
### Code Review Process
1. **Static Analysis:** Automated security scanning
2. **Peer Review:** Manual code inspection
3. **Security Testing:** Automated security test suite
4. **Approval:** Security team sign-off
### Deployment Security
1. **Pre-deployment Validation:** Security test execution
2. **Secure Deployment:** Authorized personnel only
3. **Post-deployment Verification:** Security configuration validation
4. **Monitoring:** Continuous security monitoring
## Security Tools and Integrations
### Current Tools
- **Wazuh:** SIEM and security monitoring
- **Lynis:** Security auditing
- **auditd:** System call auditing
- **Fail2ban:** Intrusion prevention
### Planned Integrations
- **Vault/Bitwarden:** Secrets management
- **OSSEC:** Host-based intrusion detection
- **Nessus/OpenVAS:** Vulnerability scanning
- **ELK Stack:** Log aggregation and analysis
## Vulnerability Management
### Vulnerability Scanning
- **Regular scans:** Monthly vulnerability assessments
- **Automated patching:** Security update automation
- **Exception handling:** Risk-based patch management
- **Reporting:** Executive security dashboards
### Disclosure Process
1. **Internal Discovery:** Report to security team
2. **Assessment:** Risk and impact evaluation
3. **Remediation:** Patch development and testing
4. **Deployment:** Coordinated security updates
5. **Verification:** Post-patch validation
## Security Metrics
### Key Performance Indicators
- **Deployment Success Rate:** Percentage of successful secure deployments
- **Vulnerability Response Time:** Time to patch critical vulnerabilities
- **Security Test Coverage:** Percentage of code covered by security tests
- **Incident Response Time:** Time to detect and respond to security events
### Monitoring and Reporting
- **Real-time Dashboards:** Security status monitoring
- **Executive Reports:** Monthly security summaries
- **Compliance Reports:** Quarterly compliance assessments
- **Trend Analysis:** Security posture improvement tracking
## Contact and Support
For security-related questions or incidents:
- **Repository Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
- **Community Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26
- **Security Team:** [Contact information to be added]
## Security Updates
This document is updated as security features are implemented and threats evolve. Last updated: July 14, 2025.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd**
>
> **Read it here:** https://community.turnsys.com/t/303
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
docs/server-build/TSYS-2FA-GUIDE.md
+10 -329
View File
@@ -1,329 +1,10 @@
# TSYS Two-Factor Authentication Implementation Guide
## Overview
This guide provides complete instructions for implementing and managing two-factor authentication (2FA) on TSYS servers using Google Authenticator (TOTP).
## What This Implementation Provides
### Services Protected by 2FA
- **SSH Access:** Requires SSH key + 2FA token
- **Cockpit Web Interface:** Requires password + 2FA token
- **Webmin Administration:** Requires password + 2FA token (if installed)
### Security Features
- **Time-based One-Time Passwords (TOTP):** Standard 6-digit codes
- **Backup Codes:** Emergency access codes
- **Gradual Rollout:** Optional nullok mode for phased deployment
- **Configuration Backup:** Automatic backup of all configs
## Implementation Steps
### Step 1: Run the 2FA Setup Script
```bash
# Navigate to the security modules directory
cd provisioning/Modules/Security
# Run the 2FA setup script as root
sudo bash secharden-2fa.sh
```
### Step 2: Validate Installation
```bash
# Run 2FA validation tests
./Project-Tests/security/2fa-validation.sh
# Run specific 2FA security test
./Project-Tests/run-tests.sh security
```
### Step 3: Setup Individual Users
For each user that needs 2FA access:
```bash
# Check setup instructions
cat /home/username/2fa-setup-instructions.txt
# Run user setup script
sudo /tmp/setup-2fa-username.sh
```
### Step 4: Test 2FA Access
1. **Test SSH access** from another terminal
2. **Test Cockpit access** via web browser
3. **Test Webmin access** if installed
## User Setup Process
### Installing Authenticator Apps
Users need one of these apps on their phone:
- **Google Authenticator** (Android/iOS)
- **Authy** (Android/iOS)
- **Microsoft Authenticator** (Android/iOS)
- **1Password** (with TOTP support)
### Setting Up 2FA for a User
1. **Run setup script:**
```bash
sudo /tmp/setup-2fa-username.sh
```
2. **Follow prompts:**
- Answer "y" to update time-based token
- Scan QR code with authenticator app
- Save emergency backup codes securely
- Answer "y" to remaining security questions
3. **Test immediately:**
```bash
# Test SSH from another terminal
ssh username@server-ip
# You'll be prompted for 6-digit code
```
## Configuration Details
### SSH Configuration Changes
File: `/etc/ssh/sshd_config`
```
ChallengeResponseAuthentication yes
UsePAM yes
AuthenticationMethods publickey,keyboard-interactive
```
### PAM Configuration
File: `/etc/pam.d/sshd`
```
auth required pam_google_authenticator.so nullok
```
### Cockpit Configuration
File: `/etc/cockpit/cockpit.conf`
```
[WebService]
LoginTitle = TSYS Server Management
LoginTo = 300
RequireHost = true
[Session]
Banner = /etc/cockpit/issue.cockpit
IdleTimeout = 15
```
### Webmin Configuration
File: `/etc/webmin/miniserv.conf`
```
twofactor_provider=totp
twofactor=1
```
## Security Considerations
### Gradual vs Strict Enforcement
#### Gradual Enforcement (Default)
- Uses `nullok` option in PAM
- Users without 2FA can still log in
- Allows phased rollout
- Good for initial deployment
#### Strict Enforcement
- Remove `nullok` from PAM configuration
- All users must have 2FA configured
- Immediate security enforcement
- Risk of lockout if misconfigured
### Backup and Recovery
#### Emergency Access
- **Backup codes:** Generated during setup
- **Root access:** Can disable 2FA if needed
- **Console access:** Physical/virtual console bypasses SSH
#### Configuration Backup
- Automatic backup to `/root/backup/2fa-TIMESTAMP/`
- Includes all modified configuration files
- Can be restored if needed
## Troubleshooting
### Common Issues
#### 1. User Cannot Generate QR Code
```bash
# Ensure qrencode is installed
sudo apt-get install qrencode
# Re-run user setup
sudo /tmp/setup-2fa-username.sh
```
#### 2. SSH Connection Fails
```bash
# Check SSH service status
sudo systemctl status sshd
# Test SSH configuration
sudo sshd -t
# Check logs
sudo journalctl -u sshd -f
```
#### 3. 2FA Code Not Accepted
- **Check time synchronization** on server and phone
- **Verify app setup** - rescan QR code if needed
- **Try backup codes** if available
#### 4. Locked Out of Server
```bash
# Access via console (physical/virtual)
# Disable 2FA temporarily
sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd
sudo systemctl restart sshd
```
### Debug Commands
```bash
# Check 2FA status
./Project-Tests/security/2fa-validation.sh
# Check SSH configuration
sudo sshd -T | grep -E "(Challenge|PAM|Authentication)"
# Check PAM configuration
cat /etc/pam.d/sshd | grep google-authenticator
# Check user 2FA status
ls -la ~/.google_authenticator
```
## Management and Maintenance
### Adding New Users
1. Ensure user account exists
2. Run setup script for new user
3. Provide setup instructions
4. Test access
### Removing User 2FA
```bash
# Remove user's 2FA configuration
sudo rm /home/username/.google_authenticator
# User will need to re-setup 2FA
```
### Disabling 2FA System-Wide
```bash
# Restore original configurations
sudo cp /root/backup/2fa-*/sshd_config.bak /etc/ssh/sshd_config
sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd
sudo systemctl restart sshd
```
### Updating 2FA Configuration
```bash
# Re-run setup script
sudo bash secharden-2fa.sh
# Validate changes
./Project-Tests/security/2fa-validation.sh
```
## Best Practices
### Deployment Strategy
1. **Test in non-production** environment first
2. **Enable gradual rollout** (nullok) initially
3. **Train users** on 2FA setup process
4. **Test emergency procedures** before strict enforcement
5. **Monitor logs** for authentication issues
### Security Recommendations
- **Enforce strict mode** after successful rollout
- **Regular backup code rotation**
- **Monitor failed authentication attempts**
- **Document emergency procedures**
- **Regular security audits**
### User Training
- **Provide clear instructions**
- **Demonstrate setup process**
- **Explain backup code importance**
- **Test login process with users**
- **Establish support procedures**
## Monitoring and Logging
### Authentication Logs
```bash
# SSH authentication logs
sudo journalctl -u sshd | grep -i "authentication"
# PAM authentication logs
sudo journalctl | grep -i "pam_google_authenticator"
# Failed login attempts
sudo journalctl | grep -i "failed"
```
### Security Monitoring
- Monitor for repeated failed 2FA attempts
- Alert on successful logins without 2FA (during gradual rollout)
- Track user 2FA setup completion
- Monitor for emergency access usage
## Integration with Existing Systems
### LDAP/Active Directory
- 2FA works with existing authentication systems
- Users still need local 2FA setup
- Consider centralized 2FA solutions for large deployments
### Monitoring Systems
- LibreNMS: Will continue to work with SNMP
- Wazuh: Will log 2FA authentication events
- Cockpit: Enhanced with 2FA protection
### Backup Systems
- Ensure backup procedures account for 2FA
- Test restore procedures with 2FA enabled
- Document emergency access procedures
## Support and Resources
### Files Created by Setup
- `/tmp/setup-2fa-*.sh` - User setup scripts
- `/home/*/2fa-setup-instructions.txt` - User instructions
- `/root/backup/2fa-*/` - Configuration backups
### Validation Tools
- `./Project-Tests/security/2fa-validation.sh` - Complete 2FA validation
- `./Project-Tests/run-tests.sh security` - Security test suite
### Emergency Contacts
- System Administrator: [Contact Info]
- Security Team: [Contact Info]
- 24/7 Support: [Contact Info]
## Compliance and Audit
### Security Benefits
- Significantly reduces risk of unauthorized access
- Meets multi-factor authentication requirements
- Provides audit trail of authentication events
- Complies with security frameworks (NIST, ISO 27001)
### Audit Trail
- All authentication attempts logged
- 2FA setup events recorded
- Configuration changes tracked
- Emergency access documented
---
**Last Updated:** July 14, 2025
**Version:** 1.0
**Author:** TSYS Security Team
# docs/server-build/TSYS-2FA-GUIDE.md
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **End-user guide for 2FA setup**
>
> **Read it here:** https://community.turnsys.com/t/303
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
docs/server-build/tailscale.md
+9 -147
View File
@@ -1,148 +1,10 @@
# Tailscale vs. Managed DNS — Architecture Analysis
# docs/server-build/tailscale.md
> **Status:** **RESOLVED.** The pfv-netinfra-01/02 pair now runs production
> Technitium DNS with all `knel.net` records replicated from tailscale-router
> via the DNS cluster setup. Both LAN IPs serve authoritative records for
> `knel.net` and recurse externally. This document records the original
> conflict, how it was resolved, and the recommended client configuration.
## 1. Executive summary
Every host in this build runs the Tailscale client, and Tailscale's MagicDNS
manages `/etc/resolv.conf` by default (pointing at `100.100.100.100`). This
previously conflicted with a managed `resolv.conf` pointing at the LAN
resolvers. The root cause was that the LAN Technitium instances did not have
the `knel.net` zone populated — **that is now fixed.**
The pfv-netinfra-01/02 pair now serves identical, authoritative `knel.net`
records (replicated from production via [`dns-cluster-setup/`](../dns-cluster-setup/README.md)).
Both LAN IPs resolve `knel.net` device names and recurse externally. The
managed `resolv.conf` is now safe to deploy.
**Recommendation:** Deploy the managed `resolv.conf` (`.252`/`.253`) on hosts
where you want tunnel-independent DNS. Leave Tailscale managing DNS on hosts
where MagicDNS device names must resolve without a LAN path (e.g. laptops off
-network). See [§5](#5-recommendation) for details.
## 2. How name resolution works today (post-cluster-setup)
Probed from `sectestbed-sandbox` (192.168.3.50) after the DNS cluster was
deployed:
| Query path | External name (`github.com`) | `knel.net` device name (`pfv-netinfra-01.knel.net`) |
|---|---|---|
| Via Tailscale resolver (`100.100.100.100`) | resolves | resolves → `100.70.181.72` (Tailscale CGNAT) |
| Direct `dig @192.168.3.252` (Technitium primary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
| Direct `dig @192.168.3.253` (Technitium secondary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
**Both LAN resolvers now serve `knel.net` records identically.** The
Technitium zone is no longer stale — it was replicated from production
(tailscale-router) as part of the DNS cluster setup.
### What changed
Previously (before the DNS cluster setup), querying the LAN IPs returned
NXDOMAIN for `knel.net` device names because the Technitium `knel.net` zone
was empty (SOA serial `2025062313`, dated 2025-06-23). After replicating
production config to both netinfra hosts, all 124 zones — including
`knel.net` with all current device records — are served authoritatively on
both `.252` and `.253`.
## 3. The DNS server pair
| Host | IP | Role | Services |
|------|----|------|----------|
| pfv-netinfra-01 | 192.168.3.252 | **Primary** | Technitium (authoritative, port 5300) + Pi-hole (recursive, port 53) |
| pfv-netinfra-02 | 192.168.3.253 | **Secondary** | Technitium (replicated via rsync, port 5300) + Pi-hole (recursive, port 53) |
Zone replication is rsync-based (every 60s via systemd timer) because
Technitium's AXFR uses port 53, which is occupied by Pi-hole on these hosts.
See [`dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) for
full details.
## 4. NTP (fully resolved)
NTP is independent of DNS: `provisioning/ConfigFiles/NTP/ntp.conf` points
directly at the LAN IPs with no DNS dependency:
```
server 192.168.3.252 iburst
server 192.168.3.253 iburst
```
Both servers respond with stratum 2/3. The client config uses `restrict`
rules (not `interface listen`) to avoid the loopback-binding bug that
prevented sync. This is safe under both Tailscale-managed and LAN-pinned
resolver configurations.
## 5. Recommendation
### On fixed servers (always on-LAN)
**Deploy the managed `resolv.conf`** (`provisioning/ConfigFiles/Resolv/`):
- Points at `.252`/`.253` with failover
- `knel.net` records resolve on both servers
- External names recurse on both servers
- DNS survives `tailscaled` outages (unlike Tailscale-managed DNS)
To prevent Tailscale from overwriting the managed file:
```bash
tailscale up --accept-dns=false
```
### On laptops / roaming hosts
**Let Tailscale manage DNS** (default `accept-dns=true`):
- MagicDNS resolves `knel.net` device names via the tunnel
- No dependency on LAN reachability
- Accept the `tailscaled` dependency (if the tunnel is down, you're off-network anyway)
## 6. Subnet router HA (redundant routing)
> **Redmine:** [#377](https://projects.knownelement.com/issues/377)
Both DNS nodes advertise the LAN subnet as Tailscale subnet routes AND
serve as exit nodes, providing full HA for remote monitoring (Uptime Kuma
on the Cloudron VPS), off-site access, and internet egress:
| Node | Tailscale IP | Advertised routes | Role |
|------|-------------|-------------------|------|
| pfv-netinfra-01 | 100.70.181.72 | 192.168.0.0/22, 0.0.0.0/0, ::/0 | Primary subnet router + exit node |
| pfv-netinfra-02 | 100.71.171.20 | 192.168.0.0/22, 0.0.0.0/0, ::/0 | Secondary subnet router + exit node (failover) |
To configure on a node (both flags together):
```bash
sudo tailscale set --advertise-routes=192.168.0.0/22 --advertise-exit-node
```
**All routes must be approved** in the Tailscale admin console
(https://login.tailscale.com/admin/machines) for failover to activate.
This includes the subnet route (`192.168.0.0/22`) AND the exit node routes
(`0.0.0.0/0`, `::/0`) — they appear as separate approvals on each machine.
Tailscale uses the primary by default and fails over to the secondary if
the primary's tunnel drops.
### Known instability factors (2026-08-06, [#377])
- **DERP relay bouncing:** netinfra-01 was observed switching between DERP
relays (#9, #12, #13) every ~15 minutes, causing brief tunnel disruptions.
- **Docker LinkChange events:** container create/destroy adds/removes bridge
interfaces, triggering `LinkChange: major, rebinding` in tailscaled and
momentarily dropping all connections.
- **resolv.conf immutable flag:** netinfra-01 had the `+i` (immutable)
attribute on `/etc/resolv.conf`, preventing tailscaled from updating DNS
config after link changes. Fixed by `chattr -i /etc/resolv.conf`.
## 7. Known items / future work
1. **Pi-hole upstream configuration.** Pi-hole on both hosts should forward
to the local Technitium instance (port 5300) for `knel.net` and to an
external resolver for everything else. Verify this is configured on both
nodes.
2. **Zone transfer via AXFR.** Currently using rsync because Technitium's
AXFR expects port 53. If Technitium's listen port can be changed, or
Pi-hole can be configured to proxy AXFR, the rsync timer could be
replaced with native DNS zone transfer.
3. **`accept-dns=false` automation.** The provisioning code should set
`--accept-dns=false` on Tailscale during setup (after deploying the
managed `resolv.conf`) so Tailscale doesn't overwrite it on reboot.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Tailscale vs managed DNS analysis (resolved)**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*