Compare commits
13
Commits
main
...
8e5b9558fe
| Author | SHA1 | Date | |
|---|---|---|---|
 mrcharles
|
8e5b9558fe | ||
|
mrcharles
|
66e7843f27 | ||
|
mrcharles
|
c14b48f39e | ||
|
mrcharles
|
18d57ea4fb | ||
|
mrcharles
|
89469ff028 | ||
|
mrcharles
|
48cb6842c6 | ||
|
mrcharles
|
6814374bac | ||
|
mrcharles
|
019ab468bf | ||
|
mrcharles
|
ea2048dc87 | ||
|
mrcharles
|
db575080c9 | ||
|
mrcharles
|
11ae08f7e6 | ||
|
mrcharles
|
a791baa085 | ||
|
mrcharles
|
422999bf3c |
+24
-2
@@ -1,8 +1,27 @@
|
||||
# Crush internal state
|
||||
.crush/
|
||||
|
||||
# OS/editor
|
||||
.DS_Store
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
\#*\#
|
||||
.\#*
|
||||
|
||||
# Python bytecode
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.bak
|
||||
*.log.tmp
|
||||
|
||||
# LOGFILENAME artifacts: the framework (Logging.sh + PrettyPrint.sh) appends
|
||||
# every print_info/print_error line to LOGFILENAME, defined as
|
||||
# "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$". Running any script that sources the
|
||||
# framework therefore drops a timestamped log file next to it. Ignore these
|
||||
# everywhere in the repo.
|
||||
# framework therefore drops a timestamped log file next to it.
|
||||
*.Monday-*
|
||||
*.Tuesday-*
|
||||
*.Wednesday-*
|
||||
@@ -10,4 +29,7 @@
|
||||
*.Friday-*
|
||||
*.Saturday-*
|
||||
*.Sunday-*
|
||||
|
||||
# Sensitive exports / runtime data
|
||||
dns-cluster-setup/.export/
|
||||
returned-logs/
|
||||
|
||||
@@ -1,76 +1,63 @@
|
||||
# Agent Guidelines
|
||||
|
||||
This repo combines two formerly-separate projects:
|
||||
- **Server provisioning** (formerly KNELServerBuild): `provisioning/`, `tests/`,
|
||||
`vendor/`, `dns-cluster-setup/`
|
||||
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): `perf/`, `netinfra/`,
|
||||
`switches/`, `returned-logs/`
|
||||
|
||||
## Repository Layout
|
||||
|
||||
Knowing where things live prevents broken edits:
|
||||
|
||||
- **Vendored framework**: `KNELShellFramework` lives at
|
||||
`vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`, **not** at
|
||||
the repo root. Its includes are under `Framework-Includes/` there. Never
|
||||
assume `./Framework-Includes` exists relative to the repo root.
|
||||
- **Self-locating scripts**: All provisioning scripts derive their own
|
||||
location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must
|
||||
**never** depend on the current working directory or on `cd`/`realpath ..`
|
||||
chains. Run them from anywhere — `sudo bash SetupNewSystem.sh` works.
|
||||
`vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes
|
||||
are under `Framework-Includes/` there. Never assume `./Framework-Includes`
|
||||
exists relative to the repo root.
|
||||
- **Self-locating scripts**: All provisioning scripts derive their own location
|
||||
via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
|
||||
depend on the current working directory. Run from anywhere.
|
||||
- **Local config files are the source of truth**: Configs in
|
||||
`ProjectCode/ConfigFiles/` are read with `cat`/`cp`. Do **not** re-introduce
|
||||
`curl ${DL_ROOT}/...` downloads from `dl.knownelement.com` — that CDN is
|
||||
deprecated for this repo.
|
||||
- **Path variables**: Scripts export `PROJECT_ROOT_PATH`, `CONFIGFILES_PATH`,
|
||||
`MODULES_PATH`, `SCRIPTS_PATH`, and `AGENTS_PATH` for locating repo content.
|
||||
- **Non-bash agents**: Some files under `ProjectCode/Agents/` carry a `.sh`
|
||||
`provisioning/ConfigFiles/` are read with `cat`/`cp`. Do NOT re-introduce
|
||||
`curl ${DL_ROOT}/...` downloads — that CDN is deprecated.
|
||||
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
|
||||
extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
|
||||
checkers must skip these.
|
||||
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
|
||||
Datacenter Manager). SSH keys deployed to root on all hosts.
|
||||
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
|
||||
Use the wrapper scripts: `tests/remote.sh`, `dns-cluster-setup/remote-dns.sh`,
|
||||
or the `deploy-check.sh` / `deploy-tuning.sh` patterns.
|
||||
|
||||
## Git Commit Requirements
|
||||
|
||||
When making changes to this repository, ALWAYS:
|
||||
|
||||
1. **Commit atomically**: Each logical change should be its own commit
|
||||
2. **Use conventional commit format**:
|
||||
- `feat(scope): description` - New feature
|
||||
- `fix(scope): description` - Bug fix
|
||||
- `docs: description` - Documentation changes
|
||||
- `refactor(scope): description` - Code refactoring
|
||||
- `test(scope): description` - Test additions/changes
|
||||
- `chore: description` - Maintenance tasks
|
||||
3. **Write verbose, beautifully formatted messages**:
|
||||
- Title line (50 chars max)
|
||||
- Blank line
|
||||
- Body explaining WHAT and WHY (not how)
|
||||
- Reference related files/issues
|
||||
- Include footer with attribution
|
||||
|
||||
## Example Commit
|
||||
|
||||
```
|
||||
feat(security-hardening): implement SCAP-STIG compliance logic
|
||||
|
||||
Refactor apply script to implement comprehensive security hardening:
|
||||
|
||||
- Add GRUB bootloader permission hardening (root:root, mode 0400)
|
||||
- Disable and remove autofs service per STIG requirements
|
||||
- Deploy modprobe configurations for kernel module blacklisting
|
||||
- Create STIG-compliant network protocol blacklist
|
||||
|
||||
This ensures servers meet DoD security requirements for production
|
||||
deployment.
|
||||
|
||||
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
|
||||
|
||||
Assisted-by: GLM-5 via Crush <crush@charm.land>
|
||||
```
|
||||
1. **Commit atomically**: each logical change its own commit.
|
||||
2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`,
|
||||
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`.
|
||||
3. **Verbose messages**: title (50 chars max), blank line, body explaining
|
||||
WHAT and WHY, footer with attribution.
|
||||
|
||||
## Autonomous Git Workflow
|
||||
|
||||
**Agents are authorized to commit AND push autonomously. Do not wait to be
|
||||
asked.** After each logical unit of work:
|
||||
Agents are authorized to commit AND push autonomously. After each logical unit
|
||||
of work: stage, commit, push to `origin/main`. Group changes so each commit is
|
||||
coherent on its own.
|
||||
|
||||
1. Stage only the files belonging to that logical change.
|
||||
2. Commit with a conventional, well-formed message (see above).
|
||||
3. Push to `origin` (`git push`). The branch tracks `origin/main`.
|
||||
4. Repeat per logical unit.
|
||||
## Key scripts
|
||||
|
||||
Group changes so each commit is coherent on its own (a reader should
|
||||
understand the commit without seeing the others). Never batch unrelated
|
||||
changes into one commit.
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `provisioning/SetupNewSystem.sh` | Full server provisioning (packages, hardening, 2FA) |
|
||||
| `tests/vm-validation.sh` | End-to-end deploy + validate on sandbox VM |
|
||||
| `tests/run-tests.sh` | Project test suite (unit/security/validation) |
|
||||
| `dns-cluster-setup/setup.sh` | Technitium DNS cluster replication |
|
||||
| `perf/deploy-check.sh` | Deploy read-only data collector to hosts |
|
||||
| `perf/deploy-tuning.sh` | Deploy perf tunings to hosts |
|
||||
| `perf/validate-fixes.sh` | Validate applied tuning changes |
|
||||
| `perf/iperf-full-matrix.sh` | Full iperf throughput suite |
|
||||
|
||||
## Project context
|
||||
|
||||
This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
|
||||
budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
|
||||
(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
|
||||
`docs/PROJECT.md` for the comprehensive fleet report and `docs/TODO.md` for
|
||||
pending hardware work.
|
||||
|
||||
@@ -1,21 +1,70 @@
|
||||
# KNEL FetchApply
|
||||
# PFVCluster
|
||||
|
||||
## Repo Issues
|
||||
https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
|
||||
Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
|
||||
Combines server provisioning, Proxmox cluster operations, and DNS infrastructure.
|
||||
|
||||
## Repo Discussion
|
||||
https://community.turnsys.com/c/chieftechnologyandproductofficer/26
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
provisioning/ Server provisioning (SetupNewSystem.sh, security hardening,
|
||||
2FA, NTP/DNS config, SNMP, Dell OMSA)
|
||||
tests/ Test suite + VM validation harness
|
||||
dns-cluster-setup/ Technitium DNS cluster replication scripts
|
||||
perf/ Proxmox performance tuning, fleet audit, iperf, switch diagnostics
|
||||
netinfra/ pfv-netinfra-01/02 DNS/NTP setup + audit scripts
|
||||
switches/ Switch configuration captures
|
||||
docs/ All documentation (PROJECT.md, SECURITY.md, tailscale.md, etc.)
|
||||
vendor/ Vendored KNELShellFramework
|
||||
```
|
||||
|
||||
## Repo discription
|
||||
Known Element Enterprises (the entity serving as the TSYS Group management company) (through it’s executive leader, the COO) provides core IT/back office systems/services/support on a hands off/fully delegated authortity basis to the CCO and the orgs/members.
|
||||
## Quick Start
|
||||
|
||||
One of those functions is the provisoning of Linux servers. This repository is the Infrastructure As Code (IAC) repository for TSYS.
|
||||
### Provision a new server
|
||||
```bash
|
||||
sudo bash provisioning/SetupNewSystem.sh
|
||||
```
|
||||
Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh),
|
||||
configures NTP/DNS/SNMP/syslog/postfix.
|
||||
|
||||
In the future it will be used via FetchApply https://github.com/P5vc/fetch-apply
|
||||
### Validate provisioning on the sandbox VM
|
||||
```bash
|
||||
VM_ID=6000 ./tests/vm-validation.sh all
|
||||
```
|
||||
Snapshots, deploys, runs the test suite, auto-rolls back on failure.
|
||||
|
||||
## Usage
|
||||
### Run the test suite
|
||||
```bash
|
||||
./tests/run-tests.sh all
|
||||
```
|
||||
|
||||
git clone this repo
|
||||
cd FetchApply/ProjectCode
|
||||
bash SetupNewSystem.sh
|
||||
### Deploy DNS cluster setup
|
||||
```bash
|
||||
cd dns-cluster-setup/
|
||||
./setup.sh all
|
||||
```
|
||||
|
||||
### Deploy perf tunings to hosts
|
||||
```bash
|
||||
cd perf/
|
||||
./deploy-check.sh # read-only data collection
|
||||
./deploy-tuning.sh # apply sysctl/tuned/NFS tunings
|
||||
```
|
||||
|
||||
## Key Documentation
|
||||
|
||||
| Doc | Contents |
|
||||
|-----|----------|
|
||||
| `docs/PROJECT.md` | Comprehensive fleet report (7 hosts, VM inventory, storage) |
|
||||
| `docs/SECURITY.md` | Security architecture and hardening details |
|
||||
| `docs/tailscale.md` | Tailscale vs managed DNS analysis |
|
||||
| `docs/DEPLOYMENT.md` | Deployment procedures |
|
||||
| `docs/TODO.md` | Pending hardware work (tsys2/4/5) |
|
||||
| `dns-cluster-setup/README.md` | DNS cluster setup guide |
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Proxmox hosts**: 7 standalone PVE installs managed via PDM
|
||||
- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
|
||||
- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs)
|
||||
- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
|
||||
- **Backups**: Proxmox Backup Server (PBS)
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
#
|
||||
# WHY: one place to configure host aliases/users/keys, one place to audit,
|
||||
# and the command scanner only permits ssh when invoked indirectly via a
|
||||
# script. Mirrors the pattern of Project-Tests/remote.sh.
|
||||
# script. Mirrors the pattern of tests/remote.sh.
|
||||
#
|
||||
# HOSTS (override IPs via env if needed):
|
||||
# tsrouter tailscale-router.knel.net (PRODUCTION — READ-ONLY here)
|
||||
|
||||
+485
@@ -0,0 +1,485 @@
|
||||
# K8S.md -- Kubernetes Architecture Deep-Dive
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
|
||||
Companion to `PROJECT.md` (which has the fleet-wide assessment).
|
||||
**Status:** For discussion in a future session. No changes made.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Workload Profile](#1-workload-profile)
|
||||
2. [Current State](#2-current-state)
|
||||
3. [Target Architecture](#3-target-architecture)
|
||||
4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
|
||||
5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
|
||||
6. [Storage Class Design](#6-storage-class-design)
|
||||
7. [ETL/HPC Considerations](#7-etlhpc-considerations)
|
||||
8. [Migration Plan](#8-migration-plan)
|
||||
|
||||
---
|
||||
|
||||
## 1. Workload Profile
|
||||
|
||||
This cluster runs **R&D and RackRental (containerlab) workloads** via
|
||||
Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
|
||||
Reston, VA running Cloudron.
|
||||
|
||||
**Workload types expected:**
|
||||
|
||||
| Type | Description | Storage need | RAM need | Examples |
|
||||
|------|------------|-------------|----------|---------|
|
||||
| **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
|
||||
| **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
|
||||
| **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
|
||||
|
||||
**Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
|
||||
sequential I/O). HPC and containerlab need low-latency random I/O (local
|
||||
SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current State
|
||||
|
||||
### 2.1 pfv-k8s nodes and their storage
|
||||
|
||||
| Node | Type | Host | Storage | Disk type | Status |
|
||||
|------|------|------|---------|-----------|--------|
|
||||
| cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
|
||||
| cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
|
||||
| wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
|
||||
|
||||
### 2.2 Problems
|
||||
|
||||
1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
|
||||
etcd quorum lost.
|
||||
2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
|
||||
3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
|
||||
4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
|
||||
(349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
|
||||
5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
|
||||
6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
|
||||
|
||||
---
|
||||
|
||||
## 3. Target Architecture
|
||||
|
||||
### 3.1 Design principles
|
||||
|
||||
1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
|
||||
heavy workers.
|
||||
2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
|
||||
storage server failing.
|
||||
3. **Wnode boot disks on local storage** -- eliminates NFS latency for
|
||||
container runtime and kubelet.
|
||||
4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
|
||||
5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
|
||||
6. **One wnode per hypervisor host** -- maximize total cluster capacity.
|
||||
|
||||
### 3.2 Target node-host-storage matrix
|
||||
|
||||
| Node | Type | Host | Boot disk | Data disk | Disk type |
|
||||
|------|------|------|-----------|-----------|-----------|
|
||||
| cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
|
||||
| cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
|
||||
| cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
|
||||
| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
|
||||
| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
|
||||
| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
|
||||
| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
|
||||
| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
|
||||
|
||||
### 3.3 Storage server distribution after changes
|
||||
|
||||
| Storage server | cnodes | wnodes (boot) | wnodes (data) |
|
||||
|---------------|--------|---------------|---------------|
|
||||
| tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
|
||||
| tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
|
||||
| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
|
||||
| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
|
||||
|
||||
**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
|
||||
NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
|
||||
than NFS-HDD, and the user has chosen not to install internal drives.
|
||||
|
||||
**No single disk or server is a quorum-losing failure point.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Control Plane (Cnodes)
|
||||
|
||||
### 4.1 Cnode sizing
|
||||
|
||||
Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
|
||||
kubernetes control plane components (API server, scheduler, controller-manager).
|
||||
|
||||
### 4.2 Cnode host placement rationale
|
||||
|
||||
| Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
|
||||
|------|--------|---------------|---------------|------------------------|
|
||||
| tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
|
||||
| tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
|
||||
| tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
|
||||
|
||||
**tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
|
||||
|
||||
### 4.3 Cnode storage placement rationale
|
||||
|
||||
The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
|
||||
|
||||
| Disk | cnodes | Rationale |
|
||||
|------|--------|-----------|
|
||||
| D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
|
||||
| D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
|
||||
| S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
|
||||
| S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
|
||||
|
||||
**If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
|
||||
**If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
|
||||
**If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
|
||||
tsys5. **Only 2 of 5 -- QUORUM LOST.**
|
||||
|
||||
Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
|
||||
cnodes and only have 2 on tsys5. That loses quorum (need 3).
|
||||
|
||||
**Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
|
||||
3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
|
||||
tsys4. Same problem inverted.
|
||||
|
||||
The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
|
||||
3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
|
||||
holding 2 cnodes is safe (3 survive).
|
||||
|
||||
**Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
|
||||
other.** After Friday's hardware work:
|
||||
- tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
|
||||
- tsys5 will have bond0 fixed + NVMe -- more reliable
|
||||
|
||||
Either way, 3/2 split means one server failure could lose quorum. **To truly
|
||||
solve this, use a 3rd storage target.** Options:
|
||||
- Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
|
||||
independent failure domain)
|
||||
- Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
|
||||
no NFS dependency at all)
|
||||
|
||||
**Best option: put cnode boot disks on local-lvm where available.** This
|
||||
eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
|
||||
its own host's local disk -- no shared dependency.
|
||||
|
||||
| cnode | Host | **Recommended storage** | Type |
|
||||
|-------|------|------------------------|------|
|
||||
| cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
|
||||
| cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
|
||||
| cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
|
||||
|
||||
With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
|
||||
down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
|
||||
survives.
|
||||
|
||||
**This is the recommended approach.** Local storage for cnodes wherever
|
||||
possible. NFS only as fallback.
|
||||
|
||||
### 4.4 etcd performance on local vs NFS
|
||||
|
||||
| Storage | Typical fsync latency | etcd commit latency | Impact |
|
||||
|---------|----------------------|--------------------|--------|
|
||||
| NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
|
||||
| NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
|
||||
| Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
|
||||
| Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
|
||||
| NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
|
||||
|
||||
etcd is the heartbeat of the kubernetes control plane. Every API call, every
|
||||
pod schedule, every controller reconciliation involves an etcd write. Cutting
|
||||
etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
|
||||
responsive. **This is the single highest-impact change for k8s performance.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Worker Nodes (Wnodes)
|
||||
|
||||
### 5.1 One wnode per hypervisor host
|
||||
|
||||
| Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
|
||||
|------|-------|-----------|-----------|-----------|-----------|------|
|
||||
| tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
|
||||
| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
|
||||
| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
|
||||
| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
|
||||
| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
|
||||
|
||||
### 5.2 Why boot disks on local-lvm
|
||||
|
||||
Current: all wnodes boot from NFS. Every container image pull, every kubelet
|
||||
log write, every ephemeral volume traverses the NFS network path.
|
||||
|
||||
With local-lvm boot disks:
|
||||
- **Container image pulls** write to local disk (100-150 MB/s HDD, no network
|
||||
hop) instead of NFS-HDD (80-120 MB/s with network latency)
|
||||
- **kubelet logs** stay local (no NFS writes for log rotation)
|
||||
- **ephemeral storage** (emptyDir volumes) uses local disk by default
|
||||
- **NFS server failure does not kill the wnode** -- the VM stays running, only
|
||||
the data disk (if mounted) goes away
|
||||
|
||||
### 5.3 Wnode sizing guidance
|
||||
|
||||
| Host | Recommended wnode config | Rationale |
|
||||
|------|------------------------|-----------|
|
||||
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
|
||||
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
|
||||
| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
|
||||
| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
|
||||
| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
|
||||
| tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
|
||||
| tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
|
||||
|
||||
### 5.4 Tainting and labeling strategy
|
||||
|
||||
Label wnodes by capability so the k8s scheduler can target them:
|
||||
|
||||
```yaml
|
||||
# Heavy RAM hosts (ETL/HPC)
|
||||
wnode-tsys6: workload=heavy, ram=128g
|
||||
wnode-tsys7: workload=heavy, ram=192g
|
||||
|
||||
# NVMe host (I/O-intensive HPC)
|
||||
wnode-tsys5: workload=hpc, storage=nvme
|
||||
|
||||
# SSD boot host (low-latency)
|
||||
wnode-tsys9: workload=light, storage=ssd
|
||||
|
||||
# General workers
|
||||
wnode-tsys3: workload=general
|
||||
wnode-tsys2: workload=storage-fast, storage=nvme
|
||||
wnode-tsys1: workload=light
|
||||
```
|
||||
|
||||
Then use nodeSelector or nodeAffinity in job specs:
|
||||
|
||||
```yaml
|
||||
# Weather/GIS ETL job -- needs lots of RAM
|
||||
spec:
|
||||
nodeSelector:
|
||||
workload: heavy
|
||||
|
||||
# Firmware build -- needs fast storage
|
||||
spec:
|
||||
nodeSelector:
|
||||
storage: nvme
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Class Design
|
||||
|
||||
### 6.1 Proposed StorageClasses
|
||||
|
||||
| StorageClass | Provisioner | Where | Speed | Use case |
|
||||
|-------------|------------|-------|-------|----------|
|
||||
| `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
|
||||
| `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
|
||||
| `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
|
||||
|
||||
### 6.2 How this maps to wnode disk topology
|
||||
|
||||
Each wnode has:
|
||||
- **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
|
||||
runtime. k8s `local-fast` StorageClass provisioner points here.
|
||||
- **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
|
||||
second block device or filesystem. k8s `nfs-hdd` provisioner points here.
|
||||
|
||||
Inside k8s, pods request storage via PVC:
|
||||
|
||||
```yaml
|
||||
# ETL job: needs bulk storage for weather data
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
spec:
|
||||
storageClassName: nfs-hdd
|
||||
accessModes: [ReadWriteMany] # NFS allows RWX
|
||||
resources:
|
||||
requests:
|
||||
storage: 500Gi
|
||||
|
||||
# HPC job: needs fast scratch
|
||||
spec:
|
||||
storageClassName: local-fast
|
||||
accessModes: [ReadWriteOnce]
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
```
|
||||
|
||||
### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
|
||||
scratch space, with the exception of ultix-streaming which stays on T5-SSD.
|
||||
Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
|
||||
SIEM, etc.).
|
||||
|
||||
The SSD NFS exports:
|
||||
- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
|
||||
cache, ephemeral volumes). Currently 99% empty.
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
|
||||
occupies 83 GB. Remaining 140 GB available for k8s use.
|
||||
|
||||
**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
|
||||
This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
|
||||
tsys5 SSD exports, bulk PVCs target either server.
|
||||
|
||||
### 6.4 NFS data distribution across storage servers
|
||||
|
||||
To avoid re-creating the "everything on tsys4" problem, distribute NFS data
|
||||
disks across both servers:
|
||||
|
||||
| wnode | Boot (local) | Bulk data (NFS) | NFS server |
|
||||
|-------|-------------|-----------------|------------|
|
||||
| wnode-tsys3 | local-lvm | S3 | tsys5 |
|
||||
| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
|
||||
| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
|
||||
|
||||
This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
|
||||
|
||||
---
|
||||
|
||||
## 7. ETL/HPC Considerations
|
||||
|
||||
### 7.1 Weather/GIS ETL pipeline
|
||||
|
||||
Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
|
||||
store results.
|
||||
|
||||
| Stage | Storage class | Why |
|
||||
|-------|-------------|-----|
|
||||
| Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
|
||||
| Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
|
||||
| Store results | `nfs-hdd` | Large sequential writes. Persistent. |
|
||||
|
||||
**Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
|
||||
wnode. ETL jobs use `/scratch` for intermediate processing and write final
|
||||
output to the NFS-mounted `/data`.
|
||||
|
||||
### 7.2 HPC workloads (hardware startup)
|
||||
|
||||
Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
|
||||
|
||||
| Workload | Best wnode | Why |
|
||||
|----------|-----------|-----|
|
||||
| RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
|
||||
| Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
|
||||
| Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
|
||||
| Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
|
||||
|
||||
### 7.3 RackRental/containerlab
|
||||
|
||||
Rapid container deployment. Key need: fast container image pulls.
|
||||
|
||||
This is where **local-lvm boot disks** shine. Currently, every container image
|
||||
pull writes through NFS to a spinning disk -- slow. With local-lvm, images
|
||||
cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
|
||||
On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
|
||||
|
||||
### 7.4 Data locality for ETL
|
||||
|
||||
For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
|
||||
it on local-lvm of the heavy hosts:
|
||||
|
||||
```
|
||||
tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier
|
||||
tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier
|
||||
```
|
||||
|
||||
**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
|
||||
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
|
||||
|
||||
This avoids re-reading the same data from the same NFS export on every job
|
||||
if the data is already cached in the page cache.
|
||||
|
||||
---
|
||||
|
||||
## 8. Migration Plan
|
||||
|
||||
**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
|
||||
Datacenter Manager (PDM)** manages them collectively and supports VM migration
|
||||
between nodes. Storage migration can be done via the PDM/Proxmox UI rather
|
||||
than manual disk copies -- the destination node just needs access to the target
|
||||
storage (which all nodes have for NFS exports, and local storage can be
|
||||
migrated through the UI's "Storage Migrate" function).
|
||||
|
||||
### 8.1 Phase 1: Friday (after hardware work)
|
||||
|
||||
After tsys5 cable + NVMe and tsys4 NIC + RAM:
|
||||
|
||||
1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
|
||||
2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
|
||||
do not use for VM storage. Recreate on D2 or D5 NFS.
|
||||
3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
|
||||
|
||||
### 8.2 Phase 2: Cnode rebalance (maintenance window)
|
||||
|
||||
These changes require creating new VMs on target hosts and migrating disks.
|
||||
Plan for a maintenance window with the k8s cluster briefly down.
|
||||
|
||||
1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
|
||||
2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
|
||||
3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
|
||||
4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
|
||||
5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
|
||||
|
||||
### 8.3 Phase 3: Wnode local storage migration (maintenance window)
|
||||
|
||||
1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
|
||||
2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
|
||||
3. wnode-tsys7 stays on NFS (same reason).
|
||||
4. Recreate wnode-tsys5 with boot disk on NVMe.
|
||||
5. Add data disks (NFS) as second SCSI devices where applicable.
|
||||
|
||||
### 8.4 Phase 4: tsys2 integration (when rebuilt)
|
||||
|
||||
1. Install Proxmox on tsys2.
|
||||
2. Run `scripts/check.sh` to inventory.
|
||||
3. Run `scripts/apply-tunings.sh --apply`.
|
||||
4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
|
||||
5. Join to k8s cluster.
|
||||
|
||||
### 8.5 Phase 5: Critical VM relocation
|
||||
|
||||
1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
|
||||
2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
|
||||
3. (No change to T5-SSD -- ultix-streaming stays.)
|
||||
|
||||
---
|
||||
|
||||
## Open questions for next session
|
||||
|
||||
1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
|
||||
whether live migration is available (huge simplification) or we need manual
|
||||
disk migration. Check `pvecm status` on each host.
|
||||
2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
|
||||
nodes are joined/drain and how StorageClasses are configured.
|
||||
3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
|
||||
4. **Is there a container image registry mirror in the cluster?** Or do all
|
||||
pulls go to Docker Hub / external? A local registry on D3 SSD would speed
|
||||
up all pulls.
|
||||
5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
|
||||
whether jobs need shared (RWX) or exclusive (RWO) storage.
|
||||
6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
|
||||
how we label and taint nodes.
|
||||
+857
@@ -0,0 +1,857 @@
|
||||
# Proxmox Cluster Project Report
|
||||
|
||||
**Date:** 2026-07-27 (re-audited)
|
||||
**Prepared by:** Performance Optimization Engagement
|
||||
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
|
||||
|
||||
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
|
||||
`deploy-check.sh`. VM placements reflect live state after user's PDM
|
||||
migrations. This is ground truth.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#1-executive-summary)
|
||||
2. [Host Fleet](#2-host-fleet)
|
||||
3. [Storage Architecture](#3-storage-architecture)
|
||||
4. [VM Fleet Inventory](#4-vm-fleet-inventory)
|
||||
5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
|
||||
6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
|
||||
7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
|
||||
8. [Role Alignment Audit](#8-role-alignment-audit)
|
||||
9. [Network Findings](#9-network-findings)
|
||||
10. [Recommendations](#10-recommendations)
|
||||
11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
|
||||
12. [Open Items](#12-open-items)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
|
||||
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
|
||||
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
|
||||
are blocked on physical hardware work scheduled for Friday.
|
||||
|
||||
**Progress since initial audit:** The user has been actively rebalancing k8s
|
||||
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
|
||||
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
|
||||
needed for etcd quorum survival.
|
||||
|
||||
The VM-layer assessment reveals:
|
||||
|
||||
| # | Finding | Severity | Status |
|
||||
|---|---------|----------|--------|
|
||||
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
|
||||
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
|
||||
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
|
||||
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
|
||||
|
||||
---
|
||||
|
||||
## 2. Host Fleet
|
||||
|
||||
### 2.1 Inventory
|
||||
|
||||
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|
||||
|------|-------|-----------|-------|-----|-----------|-----------------|--------|
|
||||
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
|
||||
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
|
||||
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
|
||||
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
|
||||
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
|
||||
|
||||
### 2.2 Role taxonomy (per user directive)
|
||||
|
||||
| Role | Hosts | Workload |
|
||||
|------|-------|----------|
|
||||
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
|
||||
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
|
||||
| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
|
||||
|
||||
**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
|
||||
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
|
||||
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
|
||||
run one per hypervisor host across the fleet to maximize total cluster capacity.
|
||||
|
||||
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
|
||||
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
|
||||
running Cloudron -- not in this cluster.
|
||||
|
||||
---
|
||||
|
||||
## 3. Storage Architecture
|
||||
|
||||
### 3.1 NFS exports from tsys4 (primary storage server)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
|
||||
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
|
||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
|
||||
|
||||
Non-exported disks on tsys4:
|
||||
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
|
||||
- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
|
||||
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
|
||||
|
||||
**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
|
||||
tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
|
||||
eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
|
||||
/sdc/sdd), so tsys5 is the better target. See section 3.2.
|
||||
|
||||
### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
|
||||
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
|
||||
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
|
||||
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
|
||||
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
|
||||
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
|
||||
|
||||
**tsys5 storage controllers (plenty of free ports):**
|
||||
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
|
||||
**5 free**
|
||||
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
|
||||
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
|
||||
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
|
||||
|
||||
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
|
||||
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
|
||||
(unused, 7 GB). S4 still 99% empty (435 GB free).
|
||||
|
||||
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
|
||||
|
||||
**Friday additions:**
|
||||
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
|
||||
USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
|
||||
2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
|
||||
boot disk and HPC scratch. The fastest tier in the fleet.
|
||||
|
||||
After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
|
||||
NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
|
||||
natural home for latency-sensitive workloads and the k8s StorageClass design
|
||||
center.
|
||||
|
||||
### 3.3 Local storage tiers (per host, with utilization)
|
||||
|
||||
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|
||||
|------|-----------|-----------|-----|-------|------|------|-------------|
|
||||
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
|
||||
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
|
||||
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
|
||||
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
|
||||
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
|
||||
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
|
||||
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
|
||||
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
|
||||
|
||||
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
|
||||
|
||||
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
|
||||
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
|
||||
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
|
||||
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
|
||||
|
||||
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
|
||||
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
|
||||
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
|
||||
for VM storage.
|
||||
|
||||
Additionally, both servers have completely empty internal drive bays:
|
||||
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
|
||||
drives.
|
||||
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
|
||||
DVD-ROM on port 5). No internal drives.
|
||||
|
||||
This is a reliability risk beyond performance: the entire host OS boots
|
||||
from a consumer-grade portable USB drive not designed for 24/7 server use.
|
||||
|
||||
**tsys3 correction:** Previously documented as HDD. Actually boots from a
|
||||
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
|
||||
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
|
||||
|
||||
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
|
||||
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
|
||||
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
|
||||
storage.
|
||||
|
||||
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
|
||||
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
|
||||
NFS-over-HDD and unsuitable for wnode boot disks.
|
||||
|
||||
### 3.4 Disk speed tiers summary
|
||||
|
||||
| Tier | Where | Speed class | Best for |
|
||||
|------|-------|------------|----------|
|
||||
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
|
||||
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
|
||||
| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
|
||||
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
|
||||
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
|
||||
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
|
||||
|
||||
### 3.5 Storage tier characterization per host
|
||||
|
||||
| Host | Storage profile | Detail |
|
||||
|------|----------------|--------|
|
||||
| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
|
||||
| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
|
||||
| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
|
||||
| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
|
||||
| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
|
||||
| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
|
||||
| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
|
||||
|
||||
### 3.6 All exports are single-disk with no redundancy
|
||||
|
||||
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
|
||||
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
|
||||
on that export. This applies to **both storage servers** and to the **PBS
|
||||
backup target** (a single 4.5T SMR drive).
|
||||
|
||||
---
|
||||
|
||||
## 4. VM Fleet Inventory
|
||||
|
||||
### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
|
||||
|
||||
#### tsys1 (Infrastructure) -- 11 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys3 (Kubernetes) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys4 (Storage) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
|
||||
|
||||
#### tsys5 (Storage) -- 15 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
|
||||
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
|
||||
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
|
||||
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
|
||||
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
|
||||
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
|
||||
|
||||
#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys7 (Kubernetes) -- 4 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys9 (Infrastructure) -- 5 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
|
||||
|
||||
### 4.2 Storage concentration summary
|
||||
|
||||
| Storage target | # of running VMs | % of fleet |
|
||||
|---------------|-----------------|------------|
|
||||
| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
|
||||
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
|
||||
| local-nonprod (tsys5 local HDD) | 10 | 25% |
|
||||
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
|
||||
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
|
||||
| local-lvm (tsys4 local) | 1 | 3% |
|
||||
|
||||
**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
|
||||
If tsys4 goes offline, two-thirds of the fleet loses its storage.
|
||||
|
||||
---
|
||||
|
||||
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
|
||||
|
||||
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|
||||
|------|------|------------|---------|-----------|----------|
|
||||
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
|
||||
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
|
||||
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
|
||||
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
|
||||
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | cnodes | Quorum impact if it fails |
|
||||
|---------------|--------|--------------------------|
|
||||
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
|
||||
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
|
||||
|
||||
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
|
||||
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
|
||||
|
||||
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
|
||||
tsys9) -- good host diversity.
|
||||
|
||||
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|
||||
|------|------|------------|---------|-----------|-----|--------|----------|
|
||||
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
|
||||
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
|
||||
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
|
||||
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
|
||||
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
|
||||
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
|
||||
|
||||
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
|
||||
stopped. wnode-tsys6 restarted.
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | wnodes | Notes |
|
||||
|---------------|--------|-------|
|
||||
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
|
||||
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
|
||||
|
||||
### 5.3 Summary: k8s node storage distribution
|
||||
|
||||
| Storage server | cnodes | wnodes | Total k8s nodes |
|
||||
|---------------|--------|--------|-----------------|
|
||||
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
|
||||
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
|
||||
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
|
||||
|
||||
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
|
||||
cnodes on each storage server (60/40 or better) so either server failing
|
||||
leaves quorum intact.
|
||||
|
||||
### 5.4 Remaining cnode migration needed for etcd quorum survival
|
||||
|
||||
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
|
||||
be on tsys5:
|
||||
|
||||
| Action | Effect |
|
||||
|--------|--------|
|
||||
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
|
||||
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
|
||||
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
|
||||
|
||||
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
|
||||
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
|
||||
|
||||
### 5.5 Future k8s architecture (next session -- see K8S.md)
|
||||
|
||||
The k8s layer will be tackled soon. Key requirements from user:
|
||||
|
||||
- **Platform:** vcluster + Rancher for multi-tenant management
|
||||
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
|
||||
- **Workload isolation (vcluster per tenant):**
|
||||
- RackRental workloads (containerlab network labs)
|
||||
- Suborbital ITAR (compliance-restricted)
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
|
||||
will be brought online during peak solar production for burst capacity.
|
||||
These older cores/ram supplement the main fleet when power is abundant.
|
||||
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
|
||||
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
|
||||
leftover host capacity) to large (28-32 GB, consuming most of a host).
|
||||
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
|
||||
full final audit will be performed at that time (post-NVMe install on tsys5,
|
||||
post-D3 SSD relocation, post-tsys4 NIC+RAM).
|
||||
|
||||
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
|
||||
|
||||
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
|
||||
|
||||
| VMID | Name | Storage |
|
||||
|------|------|---------|
|
||||
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
|
||||
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
|
||||
|
||||
These are on local storage (good -- no NFS dependency) but on a single host's
|
||||
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Redundancy Analysis
|
||||
|
||||
### 6.1 -01/-02 infrastructure pair audit
|
||||
|
||||
Two -01/-02 pairs exist in the fleet:
|
||||
|
||||
**Pair 1: pfv-netinfra (network infrastructure)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 103 | tsys1 | D5 | **tsys4** |
|
||||
| -02 | 904 | tsys9 | D2 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
|
||||
disks. The -02 half should be on an S2/S3/S4 export from tsys5.
|
||||
|
||||
**Pair 2: tsys-ucs (Univention Corporate Server)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 108 | tsys1 | D2 | **tsys4** |
|
||||
| -02 | 902 | tsys9 | D5 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
|
||||
|
||||
### 6.2 Redundancy principle for paired VMs
|
||||
|
||||
For any -01/-02 pair to survive a single storage server failure:
|
||||
|
||||
```
|
||||
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
|
||||
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
|
||||
```
|
||||
|
||||
This ensures that losing either tsys4 or tsys5 takes down only one half of
|
||||
the pair. Currently, **both pairs fail this test** because both halves are on
|
||||
tsys4.
|
||||
|
||||
### 6.3 NFS server failure blast radius
|
||||
|
||||
If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
|
||||
| k8s wnodes that lose storage | 4 of 5 |
|
||||
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
|
||||
| Total VMs that lose storage | **27 of 40 (68%)** |
|
||||
|
||||
If **tsys5** goes offline:
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 0 of 5 |
|
||||
| k8s wnodes that lose storage | 1 of 5 |
|
||||
| Total VMs that lose storage | 1 of 40 (3%) |
|
||||
|
||||
**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
|
||||
Rebalancing VM storage across both servers dramatically reduces risk.
|
||||
|
||||
---
|
||||
|
||||
## 7. Local SSD/NVMe Opportunity
|
||||
|
||||
### 7.1 Available fast tiers (currently unused by k8s)
|
||||
|
||||
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|
||||
|------|--------|------|------|--------------------|-----------------------|
|
||||
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
|
||||
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
|
||||
| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
|
||||
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
|
||||
|
||||
### 7.2 Why local storage matters for k8s nodes
|
||||
|
||||
Kubernetes nodes are latency-sensitive in two specific areas:
|
||||
|
||||
1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
|
||||
On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
|
||||
client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
|
||||
latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
|
||||
This directly affects k8s API responsiveness and pod scheduling speed.
|
||||
|
||||
2. **Container image pulls:** Worker nodes pull container images frequently.
|
||||
On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
|
||||
eliminates the network hop and reduces seek time. This matters most for
|
||||
RackRental/containerlab workloads that spin up containers rapidly.
|
||||
|
||||
### 7.3 Current waste: tsys9 local SSD
|
||||
|
||||
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
|
||||
**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
|
||||
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
|
||||
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
|
||||
|
||||
### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
|
||||
|
||||
The PCI NVMe being added to tsys5 will be the fastest storage tier in the
|
||||
fleet. Two placement options:
|
||||
|
||||
**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
|
||||
VMs that might need migration. Adds the NFS/network overhead back.
|
||||
|
||||
**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
|
||||
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
|
||||
accessed from other hosts.
|
||||
|
||||
**Recommendation:** Option B (local-only). k8s worker nodes do not need
|
||||
shared storage -- pods are ephemeral and reschedule on failure. The NVMe
|
||||
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
|
||||
and used for local VM images.
|
||||
|
||||
---
|
||||
|
||||
## 8. Role Alignment Audit
|
||||
|
||||
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
|
||||
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
|
||||
**correct by design** (keeps heavy hosts free for workers).
|
||||
|
||||
### 8.1 VMs that need to move
|
||||
|
||||
| VMID | Name | Current host | Issue | Target |
|
||||
|------|------|-------------|-------|--------|
|
||||
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
|
||||
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
|
||||
|
||||
### 8.2 Host capacity for k8s nodes
|
||||
|
||||
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|
||||
|------|------|-------------------|-------------|-----------|----------|
|
||||
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
|
||||
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
|
||||
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
|
||||
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
|
||||
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
|
||||
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
|
||||
|
||||
**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
|
||||
free RAM respectively. They should be the primary targets for heavy worker
|
||||
nodes and ETL/HPC workloads.
|
||||
|
||||
### 8.3 tsys6 wnode-tsys6 is stopped
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
|
||||
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
|
||||
cluster capacity.
|
||||
|
||||
---
|
||||
|
||||
## 9. Network Findings
|
||||
|
||||
### 9.1 tsys9 storage NIC is a USB dongle (new finding)
|
||||
|
||||
Validating tsys9 revealed that its storage network interface
|
||||
(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
|
||||
anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
|
||||
management; storage uses the USB adapter.
|
||||
|
||||
**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
|
||||
wobble, ESD, and USB controller resets. For an infrastructure host with 5
|
||||
VMs, this is a reliability risk.
|
||||
|
||||
**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
|
||||
PCIe NIC would eliminate this risk (same recommendation as tsys4).
|
||||
|
||||
### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
|
||||
|
||||
| Host | Blocker | Staged fix |
|
||||
|------|---------|-----------|
|
||||
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
|
||||
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
|
||||
|
||||
### 9.3 LACP resolved on tsys6/tsys7
|
||||
|
||||
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
|
||||
The switch LACP hash change took effect after renegotiation. The 56-106K
|
||||
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
|
||||
LACP overhead. See `RESULTS.md` for the full analysis.
|
||||
|
||||
### 9.4 NFS nconnect=4 + noatime confirmed active
|
||||
|
||||
All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
|
||||
options. Each host maintains 4 TCP connections per NFS mount to each storage
|
||||
server. This was the Tier 0 tuning item from the performance optimization
|
||||
engagement and is confirmed working cluster-wide.
|
||||
|
||||
---
|
||||
|
||||
## 10. Recommendations
|
||||
|
||||
**No changes have been made. These are assessment-only recommendations.**
|
||||
See `K8S.md` for the detailed kubernetes architecture deep-dive.
|
||||
|
||||
### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
|
||||
|
||||
These are the most critical production VMs in the fleet. They must survive
|
||||
any single-point failure (host, storage server, or disk).
|
||||
|
||||
**Design principle for -01/-02 HA pairs:**
|
||||
- Different hypervisors (already satisfied: tsys1 vs tsys9)
|
||||
- Different storage servers (currently FAILED: all on tsys4)
|
||||
- Prefer SSD for latency-sensitive services
|
||||
|
||||
**Recommended placement:**
|
||||
|
||||
| VM | Host | Storage | Tier | Free space | Rationale |
|
||||
|----|------|---------|------|-----------|-----------|
|
||||
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
|
||||
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
|
||||
| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
|
||||
| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
|
||||
|
||||
**Failure survival matrix (all single-point failures):**
|
||||
|
||||
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|
||||
|---------|-------------|-------------|--------|--------|--------|
|
||||
| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
|
||||
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
|
||||
| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
|
||||
| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
|
||||
| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
|
||||
|
||||
**Why this works:** Every row has at least one netinfra and one UCS alive.
|
||||
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
|
||||
|
||||
**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
|
||||
Friday) is freed for latency-sensitive workloads that actually benefit from
|
||||
SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
|
||||
registry mirror. Not infrastructure VMs.
|
||||
|
||||
### 10.2 CRITICAL: Cnode (control plane) storage split
|
||||
|
||||
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
|
||||
quorum (3 of 5 cnodes share D2).
|
||||
|
||||
**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
|
||||
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
|
||||
|
||||
| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
|
||||
|-------|-------------|----------------|----------------|-------------------|-----------|
|
||||
| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
|
||||
| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
|
||||
| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
|
||||
| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
|
||||
| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
|
||||
|
||||
**Result after changes:**
|
||||
|
||||
| Host | cnodes | Storage server |
|
||||
|------|--------|---------------|
|
||||
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
|
||||
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
|
||||
| tsys3 | cnode5 (S3) | tsys5 |
|
||||
|
||||
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
|
||||
etcd keeps quorum (3 of 5 survive).
|
||||
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
|
||||
holds more than 2 cnodes.
|
||||
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
|
||||
at least 3 cnodes alive.
|
||||
- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
|
||||
|
||||
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
|
||||
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
|
||||
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
|
||||
|
||||
### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
|
||||
|
||||
**Target: one wnode per hypervisor host, using local storage where possible
|
||||
and NFS-HDD for bulk data.**
|
||||
|
||||
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|
||||
|-------|------|--------------------------|---------------------|-----------|
|
||||
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
|
||||
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
|
||||
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
|
||||
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
|
||||
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
|
||||
|
||||
**Storage tiering strategy per wnode:**
|
||||
|
||||
Each wnode gets two disk tiers mapped to k8s StorageClasses:
|
||||
|
||||
1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
|
||||
images, ephemeral storage. This is where local SSD/NVMe shines -- container
|
||||
image pulls and layer extraction are seek-bound and benefit enormously from
|
||||
low-latency storage.
|
||||
|
||||
2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
|
||||
areas, large files that do not fit on local storage. Mounted as a second
|
||||
disk in the VM and exposed to k8s as a StorageClass.
|
||||
|
||||
This maps to two k8s StorageClasses:
|
||||
- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
|
||||
- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
|
||||
|
||||
### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
|
||||
space and ultix-streaming (developer workstation running "cluster of 1"
|
||||
pre-production jobs). Spinning rust hosts all other infrastructure VMs**
|
||||
(UCS, netinfra, LibreNMS, SIEM, etc.).
|
||||
|
||||
SSD allocation after Friday:
|
||||
- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
|
||||
the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
|
||||
(developer workstation, runs single-node test jobs before k8s). Remaining
|
||||
140 GB available for k8s.
|
||||
|
||||
**Deep-dive on exact k8s scratch allocation is deferred to the next session**
|
||||
(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
|
||||
|
||||
### 10.5 MEDIUM: Restart wnode-tsys6
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
|
||||
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
|
||||
the boot disk to local storage simultaneously.
|
||||
|
||||
### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
|
||||
|
||||
**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
|
||||
directory storage.**
|
||||
|
||||
Use for:
|
||||
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
|
||||
- sectestbed VMs that need fast scratch space
|
||||
- Not NFS-exported (avoid adding network overhead to the fastest tier)
|
||||
|
||||
### 10.7 LOW: Add PCIe NIC to tsys9
|
||||
|
||||
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
|
||||
slots. A $150 PCIe NIC eliminates the USB reliability risk.
|
||||
|
||||
### 10.8 LOW: Standardize PVE/kernel versions
|
||||
|
||||
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
|
||||
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
|
||||
|
||||
---
|
||||
|
||||
## 11. Hardware End-of-Support Exposure
|
||||
|
||||
| Host | EOS date | Years past | Form factor |
|
||||
|------|----------|-----------|-------------|
|
||||
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
|
||||
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
|
||||
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
|
||||
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
|
||||
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
|
||||
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
|
||||
| pfv-tsys2 (5520) | TBD | -- | Laptop |
|
||||
| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
|
||||
|
||||
**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
|
||||
The two actual rack servers (R610, R620) are the most overdue for replacement.
|
||||
This belongs in the next budget cycle.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open Items
|
||||
|
||||
### 12.1 Immediate (do today via PDM)
|
||||
|
||||
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
|
||||
redundancy. UCS stays on HDD.
|
||||
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
|
||||
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
|
||||
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
|
||||
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
|
||||
|
||||
### 12.2 Friday maintenance window (user action)
|
||||
|
||||
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
|
||||
reconfigure `/etc/network/interfaces`, reboot.
|
||||
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
|
||||
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
|
||||
format NVMe as local storage, reboot.
|
||||
3. **tsys2:** Load Proxmox (replacing Windows 10).
|
||||
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
|
||||
|
||||
### 12.3 Post-Friday validation
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
|
||||
4. Update PROJECT.md with post-hardware numbers.
|
||||
|
||||
### 12.4 Future: Kubernetes deep-dive (see K8S.md)
|
||||
|
||||
Next major workstream. Requirements captured:
|
||||
- **vcluster + Rancher** for multi-tenant k8s management
|
||||
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
|
||||
- **Workload isolation** via separate vclusters:
|
||||
- RackRental (containerlab)
|
||||
- Suborbital ITAR
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
|
||||
during peak solar production for burst capacity
|
||||
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
|
||||
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
|
||||
- **Spinning rust for all infrastructure VMs**
|
||||
|
||||
### 12.5 Data gaps
|
||||
|
||||
| Gap | How to close |
|
||||
|-----|--------------|
|
||||
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
|
||||
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
|
||||
| tsys5 NVMe size and model | Confirm after Friday installation |
|
||||
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
|
||||
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
|
||||
|
||||
### 12.6 Scripts and tools available
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/check.sh` | Read-only data collector (run on hosts) |
|
||||
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
||||
| `validate-fixes.sh` | Read-only validation of all applied changes |
|
||||
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
|
||||
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
# TODO.md — Pending User Actions
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Items needing user input or physical action.**
|
||||
|
||||
---
|
||||
|
||||
## 1. tsys2 Windows hardware inventory (run on the Windows host)
|
||||
|
||||
pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
|
||||
hardware data so the architecture plan can account for it.
|
||||
|
||||
### Option A: PowerShell (recommended — single command, copy-paste output)
|
||||
|
||||
Open **PowerShell as Administrator** and run:
|
||||
|
||||
```powershell
|
||||
# Full hardware inventory in one shot
|
||||
Write-Output "=== COMPUTER ==="
|
||||
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
|
||||
|
||||
Write-Output "`n=== CPU ==="
|
||||
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
|
||||
|
||||
Write-Output "`n=== MEMORY STICKS ==="
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISKS ==="
|
||||
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISK PARTITIONS ==="
|
||||
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== NETWORK ADAPTERS ==="
|
||||
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== GPU(s) ==="
|
||||
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
|
||||
|
||||
Write-Output "`n=== PCIe SLOTS ==="
|
||||
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== USB DEVICES (storage + network only) ==="
|
||||
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
|
||||
directly into the chat.
|
||||
|
||||
### Option B: Command Prompt (cmd.exe) fallbacks
|
||||
|
||||
If PowerShell is unavailable for some reason, these cmd commands give a
|
||||
subset:
|
||||
|
||||
```cmd
|
||||
:: Computer model and serial
|
||||
wmic computersystem get manufacturer,model
|
||||
wmic bios get serialnumber
|
||||
|
||||
:: CPU
|
||||
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
|
||||
|
||||
:: RAM (total)
|
||||
wmic computersystem get totalphysicalmemory
|
||||
|
||||
:: RAM sticks (per-slot detail)
|
||||
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
|
||||
|
||||
:: Disks
|
||||
wmic diskdrive get model,size,interfacetype,mediatype
|
||||
|
||||
:: Network adapters
|
||||
wmic nic where netenabled=true get name,speed,macaddress
|
||||
|
||||
:: GPU
|
||||
wmic path win32_videocontroller get name,adapterram,driverversion
|
||||
```
|
||||
|
||||
### What I'm looking for
|
||||
|
||||
- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
|
||||
whether wnode-tsys2 can use local storage like the other wnodes.)
|
||||
- **Network adapters**: How many onboard NICs? Model? (Determines whether
|
||||
tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
|
||||
onboard NIC available.)
|
||||
- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
|
||||
memory bandwidth for HPC workloads — dual-channel matters.)
|
||||
- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
|
||||
planning.
|
||||
- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
|
||||
- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
|
||||
|
||||
### STATUS: Collected 2026-07-27
|
||||
|
||||
**Disk inventory — RESOLVED:**
|
||||
- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
|
||||
- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
|
||||
- Both SSDs, no spinning rust. 1.5 TB total local SSD.
|
||||
|
||||
**Network adapters — RESOLVED (concerning):**
|
||||
- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
|
||||
- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
|
||||
- Wi-Fi: Intel 8265 (disconnected)
|
||||
- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
|
||||
Unavoidable on this laptop form factor — no onboard PCIe NIC available.
|
||||
|
||||
**GPU — RESOLVED:**
|
||||
- Intel HD Graphics 630 (integrated, 1 GB)
|
||||
- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
|
||||
|
||||
**PCIe slots — RESOLVED:**
|
||||
- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
|
||||
user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
|
||||
by 960 PRO.
|
||||
|
||||
**RAM — PARTIAL:**
|
||||
- Total 32 GB confirmed (34,097,573,888 bytes).
|
||||
- Per-stick detail failed to run (PowerShell line-break split
|
||||
`Format-T` + `able`). Re-run the command below if bandwidth planning
|
||||
needs stick-level detail:
|
||||
```powershell
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
|
||||
|
||||
---
|
||||
|
||||
## 2. Friday maintenance window (physical hardware)
|
||||
|
||||
### tsys4 — install PCIe NIC + add RAM
|
||||
|
||||
1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
|
||||
2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
|
||||
3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
|
||||
4. Power on, then update `/etc/network/interfaces` to replace
|
||||
`enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
|
||||
5. Reboot to activate new NIC and NFS nconnect.
|
||||
6. Run `validate-fixes.sh pfv-tsys4` to confirm.
|
||||
|
||||
### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
|
||||
|
||||
1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
|
||||
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
|
||||
"Number of ports: 2" and a real partner MAC (not all zeros).
|
||||
3. Apply bond hash fix (same as tsys6/7):
|
||||
```bash
|
||||
echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
|
||||
```
|
||||
4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
|
||||
- Power down tsys4
|
||||
- Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
|
||||
- Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
|
||||
- On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
|
||||
- Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
|
||||
`pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
|
||||
- Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
|
||||
empty, so minimal migration)
|
||||
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
|
||||
6. Format NVMe as local directory storage (see TODO section 3 below).
|
||||
7. Reboot tsys5 to activate NFS nconnect.
|
||||
8. Run `validate-fixes.sh pfv-tsys5` to confirm.
|
||||
|
||||
---
|
||||
|
||||
## 3. tsys5 NVMe format/mount decision (after Friday install)
|
||||
|
||||
**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
|
||||
storage so it shows up as a VM image target in the Proxmox UI.
|
||||
|
||||
After the NVMe is physically installed and visible in Proxmox:
|
||||
|
||||
1. Identify the device: `lsblk` or `ls /dev/nvme*`
|
||||
2. Format and add to Proxmox:
|
||||
```bash
|
||||
# Option A: LVM-thin (thin provisioning, snapshots)
|
||||
pvcreate /dev/nvme0n1
|
||||
vgcreate nvme-pool /dev/nvme0n1
|
||||
lvcreate -l 100%FREE -T nvme-pool/data
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
|
||||
# ID: nvme-local
|
||||
# Volume Group: nvme-pool
|
||||
# Thin Pool: data
|
||||
# Content: Disk image, Container template
|
||||
|
||||
# Option B: Directory (simpler, no thin provisioning)
|
||||
mkfs.ext4 /dev/nvme0n1
|
||||
mkdir -p /mnt/nvme
|
||||
mount /dev/nvme0n1 /mnt/nvme
|
||||
# Add to /etc/fstab for persistence
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > Directory
|
||||
# ID: nvme-local
|
||||
# Directory: /mnt/nvme
|
||||
# Content: Disk image, Container template
|
||||
```
|
||||
|
||||
3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
|
||||
|
||||
---
|
||||
|
||||
## 4. Post-hardware validation (run after Friday work)
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Update PROJECT.md with post-hardware iperf numbers.
|
||||
|
||||
---
|
||||
|
||||
## 5. UCS storage migration to spinning disk (do today)
|
||||
|
||||
UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
|
||||
UCS VMs should stay on spinning disk (HDD) and be split across storage
|
||||
servers for redundancy.
|
||||
|
||||
| VM | Current | Target | Action |
|
||||
|----|---------|--------|--------|
|
||||
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
|
||||
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
|
||||
|
||||
**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
|
||||
1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
|
||||
2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
|
||||
(both are NFS exports visible to tsys9, so this is a storage-only migration)
|
||||
3. Verify VM 902 boots and LDAP/AD services are healthy after migration
|
||||
|
||||
Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
|
||||
ucs-02 needs to move -- it should be on a different storage server than
|
||||
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions for next session
|
||||
|
||||
- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
|
||||
Determines whether live migration is available.
|
||||
- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
|
||||
- Container runtime? (containerd, cri-o?)
|
||||
- Is there a local container image registry mirror?
|
||||
- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
|
||||
- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
|
||||
- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
|
||||
- tsys3 thermal state (laptop in rack for years) — check `sensors`.
|
||||
Executable
+297
@@ -0,0 +1,297 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# audit-netboot.sh
|
||||
# -----------------------------------------------------------------------------
|
||||
# READ-ONLY audit of the reference node "pfv-netboot".
|
||||
#
|
||||
# Purpose: gather enough information to replicate its Pi-hole, Technitium DNS,
|
||||
# and NTP services onto pfv-netinfra-01 and pfv-netinfra-02.
|
||||
#
|
||||
# Guarantees:
|
||||
# * NO writes, NO installs, NO service restarts, NO network changes.
|
||||
# * Every command below is read-only (status, cat, ls, ss, ps, sqlite3 SELECT).
|
||||
#
|
||||
# Intended to be run as the "localuser" account on pfv-netboot:
|
||||
#
|
||||
# ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh > netboot-audit.txt
|
||||
#
|
||||
# or, if sudo is needed for a few reads, the script will try `sudo -n` for
|
||||
# specific files that are normally root-readable only. It will NEVER use sudo
|
||||
# to write or modify anything.
|
||||
# =============================================================================
|
||||
|
||||
set -u
|
||||
AUDIT_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
|
||||
# Helper: print a clearly delimited section header.
|
||||
section() {
|
||||
printf '\n========================================================================\n'
|
||||
printf 'SECTION: %s\n' "$1"
|
||||
printf '========================================================================\n'
|
||||
}
|
||||
|
||||
# Helper: read a file with sudo if needed, silently skip if missing.
|
||||
read_file() {
|
||||
local path="$1"
|
||||
if [ -r "$path" ]; then
|
||||
printf '--- %s (uid-readable) ---\n' "$path"
|
||||
cat "$path" 2>/dev/null
|
||||
elif sudo -n true 2>/dev/null; then
|
||||
if sudo -n test -e "$path"; then
|
||||
printf '--- %s (via sudo -n) ---\n' "$path"
|
||||
sudo -n cat "$path" 2>/dev/null
|
||||
else
|
||||
printf '--- %s : NOT FOUND ---\n' "$path"
|
||||
fi
|
||||
else
|
||||
printf '--- %s : NOT READABLE (no passwordless sudo) ---\n' "$path"
|
||||
fi
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
list_dir() {
|
||||
local path="$1"
|
||||
printf '--- ls -la %s ---\n' "$path"
|
||||
ls -la "$path" 2>/dev/null || sudo -n ls -la "$path" 2>/dev/null || printf '(cannot list %s)\n' "$path"
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
printf '##### AUDIT START %s #####\n' "$AUDIT_DATE"
|
||||
printf 'Audit host: %s\n' "$(hostname -f 2>/dev/null || hostname)"
|
||||
printf 'Audit user: %s\n' "$(id -un 2>/dev/null)"
|
||||
printf 'Script: audit-netboot.sh (READ-ONLY)\n'
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 1. System basics
|
||||
# -----------------------------------------------------------------------------
|
||||
section "1. SYSTEM BASICS"
|
||||
echo "-- uname --"; uname -a
|
||||
echo "-- /etc/os-release --"; cat /etc/os-release 2>/dev/null
|
||||
echo "-- uptime --"; uptime
|
||||
echo "-- timezone --"; timedatectl 2>/dev/null || cat /etc/timezone 2>/dev/null || date
|
||||
echo "-- arch --"; dpkg --print-architecture 2>/dev/null || uname -m
|
||||
echo "-- memory --"; free -h 2>/dev/null
|
||||
echo "-- disk --"; df -h / 2>/dev/null
|
||||
echo "-- cpu count --"; nproc 2>/dev/null
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 2. Network configuration
|
||||
# -----------------------------------------------------------------------------
|
||||
section "2. NETWORK"
|
||||
echo "-- hostname --"; hostname; hostname -f 2>/dev/null; hostname -I 2>/dev/null
|
||||
echo "-- ip addr --"; ip -br addr 2>/dev/null; echo; ip addr 2>/dev/null
|
||||
echo "-- default route --"; ip route 2>/dev/null
|
||||
echo "-- /etc/resolv.conf --"; cat /etc/resolv.conf 2>/dev/null
|
||||
echo "-- /etc/hosts --"; cat /etc/hosts 2>/dev/null
|
||||
echo "-- listening TCP/UDP sockets --"
|
||||
ss -tlnup 2>/dev/null || sudo -n ss -tlnup 2>/dev/null
|
||||
echo "-- /etc/network/interfaces --"; cat /etc/network/interfaces 2>/dev/null
|
||||
echo "-- netplan --"; ls -la /etc/netplan/ 2>/dev/null; for f in /etc/netplan/*.yaml; do [ -e "$f" ] && { echo "--- $f ---"; cat "$f"; }; done 2>/dev/null
|
||||
echo "-- systemd-networkd --"; ls -la /etc/systemd/network/ 2>/dev/null; networkctl status 2>/dev/null | head -40
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 3. DNS / DHCP / NTP related systemd units
|
||||
# -----------------------------------------------------------------------------
|
||||
section "3. RELEVANT SYSTEMD UNITS"
|
||||
echo "-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --"
|
||||
systemctl list-units --type=service --all --no-pager 2>/dev/null \
|
||||
| grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
|
||||
echo "-- unit files (enabled state) --"
|
||||
systemctl list-unit-files --no-pager 2>/dev/null \
|
||||
| grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
|
||||
echo "-- installed packages of interest --"
|
||||
dpkg -l 2>/dev/null | grep -Ei 'pihole|dnsmasq|ftl|technitium|ntp|chrony|timesync|unbound|resolved|resolvconf' || true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 4. NTP service details
|
||||
# -----------------------------------------------------------------------------
|
||||
section "4. NTP"
|
||||
echo "-- chrony --"
|
||||
systemctl status chrony --no-pager 2>/dev/null | head -15 || echo "(no chrony unit)"
|
||||
read_file /etc/chrony/chrony.conf
|
||||
echo "-- chronyc sources/stats (if available) --"
|
||||
chronyc -n sources 2>/dev/null || sudo -n chronyc -n sources 2>/dev/null || true
|
||||
chronyc -n tracking 2>/dev/null || sudo -n chronyc -n tracking 2>/dev/null || true
|
||||
|
||||
echo "-- ntpsec / ntp classic --"
|
||||
systemctl status ntp --no-pager 2>/dev/null | head -15 || echo "(no ntp unit)"
|
||||
systemctl status ntpsec --no-pager 2>/dev/null | head -15 || true
|
||||
read_file /etc/ntp.conf
|
||||
read_file /etc/ntpsec/ntp.conf
|
||||
|
||||
echo "-- systemd-timesyncd --"
|
||||
systemctl status systemd-timesyncd --no-pager 2>/dev/null | head -15 || echo "(no timesyncd)"
|
||||
read_file /etc/systemd/timesyncd.conf
|
||||
|
||||
echo "-- openntpd --"
|
||||
systemctl status openntpd --no-pager 2>/dev/null | head -15 || true
|
||||
read_file /etc/openntpd/ntpd.conf
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 5. Docker (Pi-hole and/or Technitium may be containerized)
|
||||
# -----------------------------------------------------------------------------
|
||||
section "5. DOCKER"
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
echo "-- docker version --"; docker version 2>/dev/null || sudo -n docker version 2>/dev/null || true
|
||||
echo "-- containers (running) --"; docker ps 2>/dev/null || sudo -n docker ps 2>/dev/null || true
|
||||
echo "-- containers (all) --"; docker ps -a 2>/dev/null || sudo -n docker ps -a 2>/dev/null || true
|
||||
echo "-- images --"; docker images 2>/dev/null || sudo -n docker images 2>/dev/null || true
|
||||
echo "-- volumes --"; docker volume ls 2>/dev/null || sudo -n docker volume ls 2>/dev/null || true
|
||||
echo "-- networks --"; docker network ls 2>/dev/null || sudo -n docker network ls 2>/dev/null || true
|
||||
else
|
||||
echo "(docker not installed / not on PATH)"
|
||||
fi
|
||||
echo "-- compose files in common locations --"
|
||||
for d in /etc/docker-compose /opt/docker-compose /root/docker-compose /home/localuser/docker-compose /srv/docker-compose /opt/pihole /opt/technitium /opt; do
|
||||
if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
|
||||
echo "### $d ###"
|
||||
sudo -n ls -la "$d" 2>/dev/null || ls -la "$d" 2>/dev/null || true
|
||||
for f in docker-compose.yml docker-compose.yaml compose.yml compose.yaml; do
|
||||
if sudo -n test -f "$d/$f" 2>/dev/null; then
|
||||
read_file "$d/$f"
|
||||
elif [ -f "$d/$f" ]; then
|
||||
read_file "$d/$f"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
done
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 6. Pi-hole
|
||||
# -----------------------------------------------------------------------------
|
||||
section "6. PI-HOLE"
|
||||
if command -v pihole >/dev/null 2>&1; then
|
||||
echo "-- pihole version --"; pihole -v 2>/dev/null || sudo -n pihole -v 2>/dev/null || true
|
||||
echo "-- pihole status --"; pihole status 2>/dev/null || sudo -n pihole status 2>/dev/null || true
|
||||
else
|
||||
echo "(pihole command not on PATH)"
|
||||
fi
|
||||
|
||||
echo "-- /etc/pihole listing --"
|
||||
list_dir /etc/pihole
|
||||
|
||||
# Key Pi-hole config files
|
||||
for f in \
|
||||
/etc/pihole/setupVars.conf \
|
||||
/etc/pihole/pihole-FTL.conf \
|
||||
/etc/pihole/adlists.list \
|
||||
/etc/pihole/whitelist.txt \
|
||||
/etc/pihole/blacklist.txt \
|
||||
/etc/pihole/regex.list \
|
||||
/etc/pihole/custom.list \
|
||||
/etc/pihole/local.list \
|
||||
/etc/pihole/hostnameMappings.txt \
|
||||
/etc/pihole/dhcp.leases \
|
||||
/etc/pihole/static_ip.conf \
|
||||
/etc/pihole/GitHubVersions \
|
||||
/etc/pihole/macvendor.db ; do
|
||||
read_file "$f"
|
||||
done
|
||||
|
||||
echo "-- /etc/pihole/*.conf (all) --"
|
||||
for f in /etc/pihole/*.conf; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
|
||||
|
||||
echo "-- /etc/dnsmasq.d listing --"
|
||||
list_dir /etc/dnsmasq.d
|
||||
for f in /etc/dnsmasq.d/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
|
||||
|
||||
echo "-- /etc/dnsmasq.conf (if present) --"
|
||||
read_file /etc/dnsmasq.conf
|
||||
|
||||
echo "-- gravity.db schema + row counts (Pi-hole v5+) --"
|
||||
GRAVITY=""
|
||||
for g in /etc/pihole/gravity.db /etc/pihole/gravity.db.*; do
|
||||
if [ -e "$g" ]; then GRAVITY="$g"; break; fi
|
||||
done
|
||||
if [ -n "${GRAVITY:-}" ]; then
|
||||
echo "gravity.db = $GRAVITY"
|
||||
if command -v sqlite3 >/dev/null 2>&1; then
|
||||
sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || true
|
||||
for tbl in adlist adlist_by_group domainlist domainlist_by_group client client_by_group group info; do
|
||||
cnt=$(sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || echo "n/a")
|
||||
printf 'gravity.%s count = %s\n' "$tbl" "$cnt"
|
||||
done
|
||||
echo "-- adlist entries (url, enabled, comment) --"
|
||||
sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null \
|
||||
|| sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null || true
|
||||
echo "-- domainlist sample (first 50) --"
|
||||
sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null \
|
||||
|| sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null || true
|
||||
echo "-- client list --"
|
||||
sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null \
|
||||
|| sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null || true
|
||||
echo "-- group list --"
|
||||
sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null \
|
||||
|| sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null || true
|
||||
echo "-- info table --"
|
||||
sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null \
|
||||
|| sudo -n sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null || true
|
||||
else
|
||||
echo "(sqlite3 not installed; gravity.db present at $GRAVITY)"
|
||||
fi
|
||||
else
|
||||
echo "(no gravity.db found)"
|
||||
fi
|
||||
|
||||
echo "-- lighttpd / pihole web admin --"
|
||||
systemctl status lighttpd --no-pager 2>/dev/null | head -15 || true
|
||||
read_file /etc/lighttpd/lighttpd.conf
|
||||
for f in /etc/lighttpd/conf-enabled/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
|
||||
|
||||
echo "-- pihole-FTL service --"
|
||||
systemctl status pihole-FTL --no-pager 2>/dev/null | head -20 || true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 7. Technitium DNS
|
||||
# -----------------------------------------------------------------------------
|
||||
section "7. TECHNITIUM DNS"
|
||||
echo "-- technitium systemd unit --"
|
||||
systemctl status technitium-dns --no-pager 2>/dev/null | head -20 || echo "(no technitium-dns unit)"
|
||||
echo "-- any unit containing technitium --"
|
||||
systemctl list-units --all --no-pager 2>/dev/null | grep -i technitium || true
|
||||
echo "-- unit file path --"
|
||||
sudo -n systemctl cat technitium-dns 2>/dev/null || systemctl cat technitium-dns 2>/dev/null || true
|
||||
|
||||
# Technitium common install locations
|
||||
for d in /etc/technitium /etc/technitium/dns /opt/technitium /opt/technitium/dns /var/lib/technitium /usr/local/technitium; do
|
||||
echo "### checking $d ###"
|
||||
if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
|
||||
list_dir "$d"
|
||||
# recurse one level for config files
|
||||
for sub in "$d" "$d"/*; do
|
||||
[ -e "$sub" ] || continue
|
||||
if [ -f "$sub" ] && echo "$sub" | grep -Eq '\.(xml|json|conf|config|txt)$'; then
|
||||
read_file "$sub"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
done
|
||||
|
||||
echo "-- technitium config.xml / dnsServer.conf (search) --"
|
||||
sudo -n find /etc/technitium /opt/technitium /var/lib/technitium -maxdepth 4 \
|
||||
\( -name 'config.xml' -o -name '*.config' -o -name 'dnsServer.conf' -o -name 'blockList.txt' \) \
|
||||
-print 2>/dev/null || true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 8. Firewall / SELinux / misc
|
||||
# -----------------------------------------------------------------------------
|
||||
section "8. FIREWALL & MISC"
|
||||
echo "-- nftables --"; sudo -n nft list ruleset 2>/dev/null | head -80 || true
|
||||
echo "-- iptables --"; sudo -n iptables -S 2>/dev/null | head -40 || true
|
||||
echo "-- ufw --"; sudo -n ufw status verbose 2>/dev/null || true
|
||||
echo "-- firewalld --"; systemctl status firewalld --no-pager 2>/dev/null | head -8 || true
|
||||
echo "-- selinux/apparmor --"; getenforce 2>/dev/null || echo "(SELinux not present)"; aa-status 2>/dev/null | head -5 || true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# 9. Cron / timers that maintain these services
|
||||
# -----------------------------------------------------------------------------
|
||||
section "9. CRON & TIMERS"
|
||||
echo "-- root crontab --"; sudo -n crontab -l 2>/dev/null || echo "(n/a)"
|
||||
echo "-- localuser crontab --"; crontab -l 2>/dev/null || echo "(none)"
|
||||
echo "-- /etc/cron.d --"; ls -la /etc/cron.d 2>/dev/null
|
||||
for f in /etc/cron.d/*pihole* /etc/cron.d/*technitium* /etc/cron.d/*gravity*; do
|
||||
[ -e "$f" ] && read_file "$f"
|
||||
done 2>/dev/null
|
||||
echo "-- pihole timer --"; systemctl list-timers --all --no-pager 2>/dev/null | grep -Ei 'pihole|gravity|technitium' || true
|
||||
|
||||
printf '\n##### AUDIT END %s #####\n' "$AUDIT_DATE"
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# baseline.sh — quick read-only baseline of a target node.
|
||||
set -u
|
||||
hdr() { printf '\n=== %s ===\n' "$1"; }
|
||||
hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
|
||||
hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
|
||||
hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
|
||||
hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
|
||||
hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
|
||||
hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
|
||||
hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
|
||||
hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
|
||||
hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
|
||||
hdr "HOME"; ls -la /home/localuser 2>/dev/null | head
|
||||
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
# deep-audit-netboot.sh — READ-ONLY deep inspection. Uses `sudo` for docker/root files.
|
||||
set -u
|
||||
DG="sudo docker"
|
||||
|
||||
hdr() { printf '\n=== %s ===\n' "$1"; }
|
||||
|
||||
hdr "COMPOSE FILES: /root/pihole and /root/NTP"
|
||||
for f in /root/pihole/docker-compose.yml /root/NTP/docker-compose.yml; do
|
||||
printf '\n--- %s ---\n' "$f"
|
||||
sudo cat "$f" 2>&1
|
||||
done
|
||||
|
||||
hdr "DIR LAYOUT of compose project dirs"
|
||||
sudo ls -la /root/pihole 2>&1
|
||||
sudo ls -la /root/NTP 2>&1
|
||||
sudo ls -la /root 2>&1
|
||||
|
||||
hdr "SEARCH entire FS for any technitium / dns compose files"
|
||||
sudo find / -xdev -maxdepth 6 \( -iname 'docker-compose.y*ml' -o -iname 'compose.y*ml' \) -print 2>/dev/null \
|
||||
| grep -Ei 'dns|technitium|tsys' || true
|
||||
|
||||
hdr "ALL CONTAINERS with compose labels"
|
||||
$DG ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\tWDIR={{.Label "com.docker.compose.project.working_dir"}}\tCFG={{.Label "com.docker.compose.project.config_files"}}\tSVC={{.Label "com.docker.compose.service"}}' 2>&1
|
||||
|
||||
hdr "DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container)"
|
||||
for c in $($DG ps -aq 2>/dev/null); do
|
||||
nm=$($DG inspect --format '{{.Name}} | image={{.Config.Image}} | proj={{index .Config.Labels "com.docker.compose.project"}}' "$c" 2>/dev/null)
|
||||
printf '\n###### %s ######\n' "$nm"
|
||||
$DG inspect "$c" 2>&1
|
||||
done
|
||||
|
||||
hdr "PI-HOLE internal config (sudo docker exec, READ-ONLY)"
|
||||
echo "-- ls /etc/pihole --"; $DG exec pihole ls -la /etc/pihole 2>&1 || true
|
||||
echo "-- ls /etc/dnsmasq.d --"; $DG exec pihole ls -la /etc/dnsmasq.d 2>&1 || true
|
||||
for f in setupVars.conf pihole-FTL.conf adlists.list custom.list local.list regex.list dhcp.leases static_ip.conf; do
|
||||
echo "--- /etc/pihole/$f ---"; $DG exec pihole cat "/etc/pihole/$f" 2>&1 || true
|
||||
done
|
||||
echo "-- /etc/dnsmasq.d/* --"
|
||||
$DG exec pihole sh -c 'for f in /etc/dnsmasq.d/*; do echo "--- $f ---"; cat "$f"; done' 2>&1 || true
|
||||
echo "-- pihole version --"; $DG exec pihole pihole -v 2>&1 || true
|
||||
echo "-- gravity row counts --"
|
||||
$DG exec pihole sh -c 'for t in adlist domainlist client "group" info; do printf "%s=" "$t"; sqlite3 /etc/pihole/gravity.db "SELECT COUNT(*) FROM $t;" 2>/dev/null; done' 2>&1 || true
|
||||
echo "-- adlist addresses --"
|
||||
$DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT address,enabled,comment FROM adlist;" 2>&1 || true
|
||||
echo "-- domainlist (allow+deny, first 60) --"
|
||||
$DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT type,domain,enabled,comment FROM domainlist LIMIT 60;" 2>&1 || true
|
||||
|
||||
hdr "CHRONY/NTP container config (tsys-ntp)"
|
||||
$DG exec tsys-ntp sh -c 'echo "--- chrony.conf ---"; cat /etc/chrony/chrony.conf 2>/dev/null || cat /etc/chrony.conf 2>/dev/null || echo none; echo "--- ls /etc ---"; ls -la /etc 2>/dev/null | head -30' 2>&1 || true
|
||||
$DG exec tsys-ntp chronyc -n sources 2>&1 || true
|
||||
$DG exec tsys-ntp chronyc -n tracking 2>&1 || true
|
||||
echo "-- bare metal ntpsec.conf --"; sudo cat /etc/ntpsec/ntp.conf 2>&1
|
||||
|
||||
hdr "TECHNITIUM volumes"
|
||||
for v in dns_tsys-dns-config dns_tyss-dns-config; do
|
||||
mnt=$($DG volume inspect --format '{{.Mountpoint}}' "$v" 2>/dev/null)
|
||||
printf '\n--- volume %s -> %s ---\n' "$v" "$mnt"
|
||||
[ -n "$mnt" ] || continue
|
||||
sudo find "$mnt" -maxdepth 4 -type f 2>/dev/null | head -80
|
||||
echo "-- config dir listing --"
|
||||
sudo ls -laR "$mnt"/config 2>/dev/null | head -60 || sudo ls -laR "$mnt" 2>/dev/null | head -60 || true
|
||||
echo "-- config.xml --"
|
||||
sudo cat "$mnt"/config/config.xml 2>/dev/null | head -250 || true
|
||||
done
|
||||
|
||||
hdr "DONE"
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
|
||||
# contents, and chrony NTP details from pfv-netboot for replication.
|
||||
set -u
|
||||
hdr() { printf '\n=== %s ===\n' "$1"; }
|
||||
|
||||
hdr "TECHNITIUM dns.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM auth.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM log.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM scopes/Default.scope"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
|
||||
|
||||
hdr "TECHNITIUM zone: knel.net.zone"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
|
||||
|
||||
hdr "TECHNITIUM all zone files listing + sizes"
|
||||
sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
|
||||
|
||||
hdr "TECHNITIUM reverse zones (one sample)"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
|
||||
|
||||
hdr "TECHNITIUM top-level listing"
|
||||
sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
|
||||
|
||||
hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
|
||||
cat /root/pihole/etc-pihole/pihole.toml 2>&1
|
||||
|
||||
hdr "PI-HOLE dnsmasq.conf"
|
||||
cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
|
||||
|
||||
hdr "PI-HOLE versions file"
|
||||
cat /root/pihole/etc-pihole/versions 2>&1
|
||||
|
||||
hdr "GRAVITY.DB contents (host sqlite3 if present)"
|
||||
if command -v sqlite3 >/dev/null 2>&1; then
|
||||
G=/root/pihole/etc-pihole/gravity.db
|
||||
echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
|
||||
echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
|
||||
echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
|
||||
echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
|
||||
echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
|
||||
echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
|
||||
echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
|
||||
else
|
||||
echo "(sqlite3 not on host PATH)"
|
||||
fi
|
||||
|
||||
hdr "DONE"
|
||||
@@ -0,0 +1,720 @@
|
||||
This system is the property of Known Element Enterprises LLC.
|
||||
|
||||
Authorized uses only. All activity may be monitored and reported.
|
||||
|
||||
All activities subject to monitoring/recording/review in real time and/or at a later time.##### AUDIT START 2026-07-28T02:41:34Z #####
|
||||
Audit host: pfv-netboot.knel.net
|
||||
Audit user: localuser
|
||||
Script: audit-netboot.sh (READ-ONLY)
|
||||
|
||||
========================================================================
|
||||
SECTION: 1. SYSTEM BASICS
|
||||
========================================================================
|
||||
-- uname --
|
||||
Linux pfv-netboot 6.1.0-44-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.164-1 (2026-03-09) x86_64 GNU/Linux
|
||||
-- /etc/os-release --
|
||||
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
|
||||
NAME="Debian GNU/Linux"
|
||||
VERSION_ID="12"
|
||||
VERSION="12 (bookworm)"
|
||||
VERSION_CODENAME=bookworm
|
||||
ID=debian
|
||||
HOME_URL="https://www.debian.org/"
|
||||
SUPPORT_URL="https://www.debian.org/support"
|
||||
BUG_REPORT_URL="https://bugs.debian.org/"
|
||||
-- uptime --
|
||||
21:41:34 up 113 days, 15:32, 2 users, load average: 0.07, 0.07, 0.06
|
||||
-- timezone --
|
||||
Local time: Mon 2026-07-27 21:41:34 CDT
|
||||
Universal time: Tue 2026-07-28 02:41:34 UTC
|
||||
RTC time: Tue 2026-07-28 02:41:34
|
||||
Time zone: US/Central (CDT, -0500)
|
||||
System clock synchronized: yes
|
||||
NTP service: n/a
|
||||
RTC in local TZ: no
|
||||
-- arch --
|
||||
amd64
|
||||
-- memory --
|
||||
total used free shared buff/cache available
|
||||
Mem: 1.9Gi 940Mi 66Mi 65Mi 1.1Gi 961Mi
|
||||
Swap: 974Mi 315Mi 659Mi
|
||||
-- disk --
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/sda1 491G 18G 448G 4% /
|
||||
-- cpu count --
|
||||
2
|
||||
|
||||
========================================================================
|
||||
SECTION: 2. NETWORK
|
||||
========================================================================
|
||||
-- hostname --
|
||||
pfv-netboot
|
||||
pfv-netboot.knel.net
|
||||
192.168.3.250 172.24.0.1 172.18.0.1 172.19.0.1 172.23.0.1 172.17.0.1 172.20.0.1 172.22.0.1 100.103.64.82 2600:1700:1d1:eb00:215:5dff:fe64:e81e fd7a:115c:a1e0::2901:4052
|
||||
-- ip addr --
|
||||
lo UNKNOWN 127.0.0.1/8 ::1/128
|
||||
eth0 UP 192.168.3.250/22 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 fe80::215:5dff:fe64:e81e/64
|
||||
eth1 UP
|
||||
br-3d55385c5e7d UP 172.24.0.1/16 fe80::d883:2ff:fef8:79cd/64
|
||||
br-50e2988e78eb DOWN 172.18.0.1/16
|
||||
br-748fea589ab8 DOWN 172.19.0.1/16
|
||||
br-b52d11330b20 DOWN 172.23.0.1/16
|
||||
docker0 DOWN 172.17.0.1/16
|
||||
br-f91c61cdba28 DOWN 172.20.0.1/16
|
||||
br-25187260f675 DOWN 172.22.0.1/16
|
||||
tailscale0 UNKNOWN 100.103.64.82/32 fd7a:115c:a1e0::2901:4052/128 fe80::42db:e6ff:fe57:431c/64
|
||||
veth9ecb986@if2 UP fe80::c45b:62ff:fe19:6258/64
|
||||
|
||||
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
|
||||
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
|
||||
inet 127.0.0.1/8 scope host lo
|
||||
valid_lft forever preferred_lft forever
|
||||
inet6 ::1/128 scope host noprefixroute
|
||||
valid_lft forever preferred_lft forever
|
||||
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
|
||||
link/ether 00:15:5d:64:e8:1e brd ff:ff:ff:ff:ff:ff
|
||||
inet 192.168.3.250/22 brd 192.168.3.255 scope global eth0
|
||||
valid_lft forever preferred_lft forever
|
||||
inet6 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 scope global dynamic mngtmpaddr
|
||||
valid_lft 3391sec preferred_lft 3391sec
|
||||
inet6 fe80::215:5dff:fe64:e81e/64 scope link
|
||||
valid_lft forever preferred_lft forever
|
||||
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
|
||||
link/ether 00:15:5d:64:e8:2a brd ff:ff:ff:ff:ff:ff
|
||||
5: br-3d55385c5e7d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
|
||||
link/ether da:83:02:f8:79:cd brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.24.0.1/16 brd 172.24.255.255 scope global br-3d55385c5e7d
|
||||
valid_lft forever preferred_lft forever
|
||||
inet6 fe80::d883:2ff:fef8:79cd/64 scope link
|
||||
valid_lft forever preferred_lft forever
|
||||
6: br-50e2988e78eb: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether ca:f9:33:b2:b6:bb brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-50e2988e78eb
|
||||
valid_lft forever preferred_lft forever
|
||||
7: br-748fea589ab8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether 82:e8:2b:47:1e:91 brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.19.0.1/16 brd 172.19.255.255 scope global br-748fea589ab8
|
||||
valid_lft forever preferred_lft forever
|
||||
8: br-b52d11330b20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether de:6e:f1:ec:24:64 brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.23.0.1/16 brd 172.23.255.255 scope global br-b52d11330b20
|
||||
valid_lft forever preferred_lft forever
|
||||
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether 56:f9:55:99:fa:af brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
|
||||
valid_lft forever preferred_lft forever
|
||||
10: br-f91c61cdba28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether 8e:ed:24:fa:4b:4a brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.20.0.1/16 brd 172.20.255.255 scope global br-f91c61cdba28
|
||||
valid_lft forever preferred_lft forever
|
||||
11: br-25187260f675: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
|
||||
link/ether 82:af:8a:0a:64:01 brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.22.0.1/16 brd 172.22.255.255 scope global br-25187260f675
|
||||
valid_lft forever preferred_lft forever
|
||||
18: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
|
||||
link/none
|
||||
inet 100.103.64.82/32 scope global tailscale0
|
||||
valid_lft forever preferred_lft forever
|
||||
inet6 fd7a:115c:a1e0::2901:4052/128 scope global
|
||||
valid_lft forever preferred_lft forever
|
||||
inet6 fe80::42db:e6ff:fe57:431c/64 scope link stable-privacy
|
||||
valid_lft forever preferred_lft forever
|
||||
24: veth9ecb986@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3d55385c5e7d state UP group default
|
||||
link/ether c6:5b:62:19:62:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
|
||||
inet6 fe80::c45b:62ff:fe19:6258/64 scope link
|
||||
valid_lft forever preferred_lft forever
|
||||
-- default route --
|
||||
default via 192.168.3.254 dev eth0 onlink
|
||||
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
|
||||
172.18.0.0/16 dev br-50e2988e78eb proto kernel scope link src 172.18.0.1 linkdown
|
||||
172.19.0.0/16 dev br-748fea589ab8 proto kernel scope link src 172.19.0.1 linkdown
|
||||
172.20.0.0/16 dev br-f91c61cdba28 proto kernel scope link src 172.20.0.1 linkdown
|
||||
172.22.0.0/16 dev br-25187260f675 proto kernel scope link src 172.22.0.1 linkdown
|
||||
172.23.0.0/16 dev br-b52d11330b20 proto kernel scope link src 172.23.0.1 linkdown
|
||||
172.24.0.0/16 dev br-3d55385c5e7d proto kernel scope link src 172.24.0.1
|
||||
192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.250
|
||||
-- /etc/resolv.conf --
|
||||
# resolv.conf(5) file generated by tailscale
|
||||
# For more info, see https://tailscale.com/s/resolvconf-overwrite
|
||||
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
|
||||
|
||||
nameserver 100.100.100.100
|
||||
nameserver fd7a:115c:a1e0::53
|
||||
search knel.net
|
||||
-- /etc/hosts --
|
||||
127.0.0.1 localhost
|
||||
127.0.1.1 pfv-netboot.knel.net pfv-netboot
|
||||
|
||||
# The following lines are desirable for IPv6 capable hosts
|
||||
::1 localhost ip6-localhost ip6-loopback
|
||||
ff02::1 ip6-allnodes
|
||||
ff02::2 ip6-allrouters
|
||||
-- listening TCP/UDP sockets --
|
||||
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
|
||||
udp UNCONN 0 0 0.0.0.0:38908 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:53 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:67 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:46164 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:111 0.0.0.0:*
|
||||
udp UNCONN 0 0 100.103.64.82:123 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.24.0.1:123 0.0.0.0:*
|
||||
udp UNCONN 0 0 192.168.3.250:123 0.0.0.0:*
|
||||
udp UNCONN 0 0 127.0.0.1:123 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:123 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.17.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.17.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.18.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.18.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.19.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.19.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.20.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.20.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.22.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.22.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.23.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.23.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.24.255.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.24.0.1:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 192.168.3.255:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 192.168.3.250:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:137 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.17.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.17.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.18.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.18.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.19.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.19.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.20.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.20.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.22.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.22.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.23.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.23.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.24.255.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 172.24.0.1:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 192.168.3.255:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 192.168.3.250:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:138 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:161 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:41257 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:20000 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:41641 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:59092 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:10000 0.0.0.0:*
|
||||
udp UNCONN 0 0 127.0.0.1:786 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:40725 0.0.0.0:*
|
||||
udp UNCONN 0 0 0.0.0.0:37749 0.0.0.0:*
|
||||
udp UNCONN 0 0 [::]:53 [::]:*
|
||||
udp UNCONN 0 0 [::]:34901 [::]:*
|
||||
udp UNCONN 0 0 [::]:111 [::]:*
|
||||
udp UNCONN 0 0 [fe80::c45b:62ff:fe19:6258]%veth9ecb986:123 [::]:*
|
||||
udp UNCONN 0 0 [fe80::42db:e6ff:fe57:431c]%tailscale0:123 [::]:*
|
||||
udp UNCONN 0 0 [fd7a:115c:a1e0::2901:4052]:123 [::]:*
|
||||
udp UNCONN 0 0 [fe80::d883:2ff:fef8:79cd]%br-3d55385c5e7d:123 [::]:*
|
||||
udp UNCONN 0 0 [fe80::215:5dff:fe64:e81e]%eth0:123 [::]:*
|
||||
udp UNCONN 0 0 [2600:1700:1d1:eb00:215:5dff:fe64:e81e]:123 [::]:*
|
||||
udp UNCONN 0 0 [::1]:123 [::]:*
|
||||
udp UNCONN 0 0 [::]:123 [::]:*
|
||||
udp UNCONN 0 0 [::]:43257 [::]:*
|
||||
udp UNCONN 0 0 [::]:44469 [::]:*
|
||||
udp UNCONN 0 0 [::]:41641 [::]:*
|
||||
udp UNCONN 0 0 [::]:55132 [::]:*
|
||||
udp UNCONN 0 0 [::]:34667 [::]:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:38435 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:20000 0.0.0.0:*
|
||||
tcp LISTEN 0 5 127.0.0.1:555 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:10002 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:10003 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:10000 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 100.103.64.82:64362 0.0.0.0:*
|
||||
tcp LISTEN 0 5 127.0.0.1:61209 0.0.0.0:*
|
||||
tcp LISTEN 0 50 0.0.0.0:139 0.0.0.0:*
|
||||
tcp LISTEN 0 64 0.0.0.0:36079 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:45295 0.0.0.0:*
|
||||
tcp LISTEN 0 5 0.0.0.0:4330 0.0.0.0:*
|
||||
tcp LISTEN 0 100 0.0.0.0:25 0.0.0.0:*
|
||||
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
|
||||
tcp LISTEN 0 64 0.0.0.0:2049 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:53 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:40029 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:111 0.0.0.0:*
|
||||
tcp LISTEN 0 50 0.0.0.0:445 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 0.0.0.0:58853 0.0.0.0:*
|
||||
tcp LISTEN 0 128 0.0.0.0:44322 0.0.0.0:*
|
||||
tcp LISTEN 0 128 0.0.0.0:44323 0.0.0.0:*
|
||||
tcp LISTEN 0 5 0.0.0.0:44321 0.0.0.0:*
|
||||
tcp LISTEN 0 4096 [fd7a:115c:a1e0::2901:4052]:52472 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:45755 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:60951 [::]:*
|
||||
tcp LISTEN 0 4096 *:9090 *:*
|
||||
tcp LISTEN 0 64 [::]:37813 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:10002 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:10003 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:10000 [::]:*
|
||||
tcp LISTEN 0 4096 *:45876 *:*
|
||||
tcp LISTEN 0 50 [::]:139 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:44271 [::]:*
|
||||
tcp LISTEN 0 5 [::]:4330 [::]:*
|
||||
tcp LISTEN 0 128 [::]:22 [::]:*
|
||||
tcp LISTEN 0 64 [::]:2049 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:53 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:111 [::]:*
|
||||
tcp LISTEN 0 4096 *:6556 *:*
|
||||
tcp LISTEN 0 50 [::]:445 [::]:*
|
||||
tcp LISTEN 0 4096 [::]:60725 [::]:*
|
||||
tcp LISTEN 0 128 [::]:44322 [::]:*
|
||||
tcp LISTEN 0 128 [::]:44323 [::]:*
|
||||
tcp LISTEN 0 5 [::]:44321 [::]:*
|
||||
-- /etc/network/interfaces --
|
||||
# This file describes the network interfaces available on your system
|
||||
# and how to activate them. For more information, see interfaces(5).
|
||||
|
||||
source /etc/network/interfaces.d/*
|
||||
|
||||
# The loopback network interface
|
||||
auto lo
|
||||
iface lo inet loopback
|
||||
|
||||
# The primary network interface
|
||||
allow-hotplug eth0
|
||||
iface eth0 inet static
|
||||
address 192.168.3.250
|
||||
netmask 255.255.252.0
|
||||
gateway 192.168.3.254
|
||||
|
||||
#allow-hotplug eth1
|
||||
#iface eth1 inet static
|
||||
# address 10.100.100.4
|
||||
# netmask 255.255.255.0
|
||||
-- netplan --
|
||||
-- systemd-networkd --
|
||||
total 8
|
||||
drwxr-xr-x 2 root root 4096 Mar 6 2025 .
|
||||
drwxr-xr-x 5 root root 4096 Jul 20 17:55 ..
|
||||
● State: n/a
|
||||
Online state: unknown
|
||||
Address: 192.168.3.250 on eth0
|
||||
172.24.0.1 on br-3d55385c5e7d
|
||||
172.18.0.1 on br-50e2988e78eb
|
||||
172.19.0.1 on br-748fea589ab8
|
||||
172.23.0.1 on br-b52d11330b20
|
||||
172.17.0.1 on docker0
|
||||
172.20.0.1 on br-f91c61cdba28
|
||||
172.22.0.1 on br-25187260f675
|
||||
100.103.64.82 on tailscale0
|
||||
2600:1700:1d1:eb00:215:5dff:fe64:e81e on eth0
|
||||
fd7a:115c:a1e0::2901:4052 on tailscale0
|
||||
fe80::215:5dff:fe64:e81e on eth0
|
||||
fe80::d883:2ff:fef8:79cd on br-3d55385c5e7d
|
||||
fe80::42db:e6ff:fe57:431c on tailscale0
|
||||
fe80::c45b:62ff:fe19:6258 on veth9ecb986
|
||||
Gateway: 192.168.3.254 on eth0
|
||||
fe80::6e4b:b4ff:fe10:4061 on eth0
|
||||
|
||||
========================================================================
|
||||
SECTION: 3. RELEVANT SYSTEMD UNITS
|
||||
========================================================================
|
||||
-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --
|
||||
initrd-parse-etc.service loaded inactive dead Mountpoints Configured in the Real Root
|
||||
ntpsec-rotate-stats.service loaded inactive dead Rotate ntpd stats
|
||||
ntpsec-systemd-netif.service loaded inactive dead ntpsec-systemd-netif.service
|
||||
ntpsec.service loaded active running Network Time Service
|
||||
● systemd-resolved.service not-found inactive dead systemd-resolved.service
|
||||
● systemd-timesyncd.service not-found inactive dead systemd-timesyncd.service
|
||||
-- unit files (enabled state) --
|
||||
ntpsec-systemd-netif.path enabled enabled
|
||||
postfix-resolvconf.path disabled enabled
|
||||
ntp.service alias -
|
||||
ntpd.service alias -
|
||||
ntpsec-rotate-stats.service static -
|
||||
ntpsec-systemd-netif.service static -
|
||||
ntpsec-wait.service disabled enabled
|
||||
ntpsec.service enabled enabled
|
||||
postfix-resolvconf.service disabled enabled
|
||||
ntpsec-rotate-stats.timer enabled enabled
|
||||
ntpsec.timer masked enabled
|
||||
-- installed packages of interest --
|
||||
ii dnsmasq-base 2.90-4~deb12u2 amd64 Small caching DNS proxy and DHCP/TFTP server - executable
|
||||
ii ntpsec 1.2.2+dfsg1-1+deb12u1 amd64 Network Time Protocol daemon and utility programs
|
||||
ii ntpsec-ntpdate 1.2.2+dfsg1-1+deb12u1 amd64 client for setting system time from NTP servers
|
||||
ii ntpsec-ntpdig 1.2.2+dfsg1-1+deb12u1 amd64 ntpdig SNTP client
|
||||
ii python3-ntp 1.2.2+dfsg1-1+deb12u1 amd64 Python 3 NTP Helper Classes
|
||||
|
||||
========================================================================
|
||||
SECTION: 4. NTP
|
||||
========================================================================
|
||||
-- chrony --
|
||||
|
||||
-- chronyc sources/stats (if available) --
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
-- ntpsec / ntp classic --
|
||||
● ntpsec.service - Network Time Service
|
||||
Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
|
||||
Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
|
||||
Docs: man:ntpd(8)
|
||||
Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
|
||||
Main PID: 2124690 (ntpd)
|
||||
Tasks: 1 (limit: 2219)
|
||||
Memory: 10.7M
|
||||
CPU: 34.324s
|
||||
CGroup: /system.slice/ntpsec.service
|
||||
└─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
|
||||
● ntpsec.service - Network Time Service
|
||||
Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
|
||||
Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
|
||||
Docs: man:ntpd(8)
|
||||
Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
|
||||
Main PID: 2124690 (ntpd)
|
||||
Tasks: 1 (limit: 2219)
|
||||
Memory: 10.7M
|
||||
CPU: 34.324s
|
||||
CGroup: /system.slice/ntpsec.service
|
||||
└─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
main: line 37: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
driftfile /var/lib/ntp/ntp.drift
|
||||
leapfile /usr/share/zoneinfo/leap-seconds.list
|
||||
server pfvsvrpi.knel.net
|
||||
restrict 127.0.0.1
|
||||
restrict ::1
|
||||
|
||||
-- systemd-timesyncd --
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
-- openntpd --
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
========================================================================
|
||||
SECTION: 5. DOCKER
|
||||
========================================================================
|
||||
-- docker version --
|
||||
Client: Docker Engine - Community
|
||||
Version: 29.6.2
|
||||
API version: 1.55
|
||||
Go version: go1.26.5
|
||||
Git commit: dfc4efb
|
||||
Built: Thu Jul 16 16:12:20 2026
|
||||
OS/Arch: linux/amd64
|
||||
Context: default
|
||||
Client: Docker Engine - Community
|
||||
Version: 29.6.2
|
||||
API version: 1.55
|
||||
Go version: go1.26.5
|
||||
Git commit: dfc4efb
|
||||
Built: Thu Jul 16 16:12:20 2026
|
||||
OS/Arch: linux/amd64
|
||||
Context: default
|
||||
|
||||
Server: Docker Engine - Community
|
||||
Engine:
|
||||
Version: 29.6.2
|
||||
API version: 1.55 (minimum version 1.40)
|
||||
Go version: go1.26.5
|
||||
Git commit: 3d80467
|
||||
Built: Thu Jul 16 16:12:20 2026
|
||||
OS/Arch: linux/amd64
|
||||
Experimental: false
|
||||
containerd:
|
||||
Version: v2.2.6
|
||||
GitCommit: 11ce9d5f3c68c941867e82890e93e815c1304f1b
|
||||
runc:
|
||||
Version: 1.3.6
|
||||
GitCommit: v1.3.6-0-g491b69ba
|
||||
docker-init:
|
||||
Version: 0.19.0
|
||||
GitCommit: de40ad0
|
||||
-- containers (running) --
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
1d09249f01b0 pihole/pihole:latest "start.sh" 5 months ago Up 44 minutes (healthy) 67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp pihole
|
||||
c33ced53142e dockurr/chrony "/bin/startup" 13 months ago Up 7 days (healthy) tsys-ntp
|
||||
-- containers (all) --
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
1d09249f01b0 pihole/pihole:latest "start.sh" 5 months ago Up 44 minutes (healthy) 67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp pihole
|
||||
c33ced53142e dockurr/chrony "/bin/startup" 13 months ago Up 7 days (healthy) tsys-ntp
|
||||
-- images --
|
||||
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
|
||||
busybox:latest 6d3e4188a38a 4.28MB 0B
|
||||
dockurr/chrony:latest 3b656f2feb55 15.4MB 0B U
|
||||
pihole/pihole:latest 81365952d1f8 92.7MB 0B U
|
||||
-- volumes --
|
||||
DRIVER VOLUME NAME
|
||||
local 0cb24520936997d233409ea07681920fdb03d92967856e849474cf90446dc426
|
||||
local 0ccc1140e4b86442167f65d7caf43d8a3bc04fff2bef461f112d8b7c8e4bd4c8
|
||||
local 1d7f70d65f5d739f737c3bc0d1b0b2b5cf4f192477dcbbdb95bca344310bff84
|
||||
local 2a1726279d1d382ab0a2bcb2079f60e65fc0fd0d40880074038e0c678b2a7f56
|
||||
local 3d207dff87da7dc7d11b70f1e595cc5cce001392cb663ac4d664f5710b9879b8
|
||||
local 4f8894b9f0206c743be26e6b644e3bf017c5b6e190ac7da31e7ffb7132a1ca2f
|
||||
local 7c210964fd48cd91c8e37543d562569ec523208b5ea421f8fb47dc73a6d724ee
|
||||
local 8ccc4c07bf45d05bc922d4f619780f60cac0149b4dd7de5be7720f72f47e6c10
|
||||
local 71c98aa3d5fb266ef291ac4fc95b11fc83475b9bb0f70999e77095249ad18bf5
|
||||
local 83f977471ac2f4ad6521ad50f0311ee36547701d00d9a869903f0ba667c2000d
|
||||
local 87eced0295c12870220fc746ef685530cb5b8e20c4afafbfd573ff773459cd9f
|
||||
local 95eb8a7b389b9a04bd90f4cee485bd015b7c3fd311c1750c7596a2b2fde1605d
|
||||
local 562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41
|
||||
local 8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b
|
||||
local 201278554f24a7af6284b9564c45656e6d2dc7b9daea24fe52707d7825cb5385
|
||||
local b24c0bd3e25987537a92078d6f521f5f155e7379295d21e43c500cbabc8a02e0
|
||||
local bb353f885d620d98dbb147f0ff51f38a87725669b1a14390fb825509878a7e0b
|
||||
local bc96d56b965f1e238afd47e483b93f1ec9ed59e8458e8e851ffc9710582a5475
|
||||
local c329a5b999999c074bf767bc1f7b3647cf5d4d0624b7d45d63658cbb723e5d46
|
||||
local cf03c8ea64823ac8db5c6df1093b2d9f5146838dd8dc2986bdc485e8b59d0d24
|
||||
local d9500add842368a6d637090569e5c13cb9f068cbfeb14e02c597ae82f2807e47
|
||||
local da3164486ea59aaa9fd14113faa9104411c61ad36a9487e3ce780e86a96dac1f
|
||||
local dae4bbdcbb634b817b47558df3908906fe7635ad57377389616189bd0f2d7b64
|
||||
local dddeee12833ed02185e00732ddb8c12ef6a6b88c49e7df4e48f7a27b71182bf5
|
||||
local dns_tsys-dns-config
|
||||
local dns_tyss-dns-config
|
||||
local e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629
|
||||
local eda6668262ccd43f0e3c0b9e3c42711cc516f372b46ef174bca34d96d17740b8
|
||||
local f53cb9833442fd9499260fa8073a04a15f5c96a5576aa96d781bc24441906942
|
||||
local f142ec0c6150618365447908d5753ff5ff7defcbf27c210702a17ad205dc4aba
|
||||
local f350de2e220f4983bca2e7ca96374a597a075779224b036ccd76deb280ba8922
|
||||
-- networks --
|
||||
NETWORK ID NAME DRIVER SCOPE
|
||||
0f56376f4515 bridge bridge local
|
||||
f91c61cdba28 dns_default bridge local
|
||||
b24303598fa1 host host local
|
||||
25187260f675 msmtpd_default bridge local
|
||||
748fea589ab8 netdisco_default bridge local
|
||||
bd40993bc47f none null local
|
||||
b52d11330b20 ntp_default bridge local
|
||||
3d55385c5e7d pihole_default bridge local
|
||||
50e2988e78eb root_default bridge local
|
||||
-- compose files in common locations --
|
||||
### /opt ###
|
||||
total 16
|
||||
drwxr-xr-x 4 root root 4096 Dec 29 2025 .
|
||||
drwxr-xr-x 20 root root 4096 Jul 20 17:55 ..
|
||||
drwxr-xr-x 2 beszel beszel 4096 Apr 6 01:08 beszel-agent
|
||||
drwx--x--x 4 root root 4096 Jun 10 2025 containerd
|
||||
|
||||
========================================================================
|
||||
SECTION: 6. PI-HOLE
|
||||
========================================================================
|
||||
(pihole command not on PATH)
|
||||
-- /etc/pihole listing --
|
||||
main: line 54: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
(cannot list /etc/pihole)
|
||||
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
|
||||
-- /etc/pihole/*.conf (all) --
|
||||
-- /etc/dnsmasq.d listing --
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
main: line 54: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
(cannot list /etc/dnsmasq.d)
|
||||
|
||||
-- /etc/dnsmasq.conf (if present) --
|
||||
|
||||
-- gravity.db schema + row counts (Pi-hole v5+) --
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
(no gravity.db found)
|
||||
-- lighttpd / pihole web admin --
|
||||
|
||||
main: line 44: printf: --: invalid option
|
||||
printf: usage: printf [-v var] format [arguments]
|
||||
alias.url += ("/javascript" => "/usr/share/javascript")
|
||||
|
||||
-- pihole-FTL service --
|
||||
|
||||
========================================================================
|
||||
SECTION: 7. TECHNITIUM DNS
|
||||
========================================================================
|
||||
-- technitium systemd unit --
|
||||
-- any unit containing technitium --
|
||||
-- unit file path --
|
||||
### checking /etc/technitium ###
|
||||
### checking /etc/technitium/dns ###
|
||||
### checking /opt/technitium ###
|
||||
### checking /opt/technitium/dns ###
|
||||
### checking /var/lib/technitium ###
|
||||
### checking /usr/local/technitium ###
|
||||
-- technitium config.xml / dnsServer.conf (search) --
|
||||
|
||||
========================================================================
|
||||
SECTION: 8. FIREWALL & MISC
|
||||
========================================================================
|
||||
-- nftables --
|
||||
table ip raw {
|
||||
chain PREROUTING {
|
||||
type filter hook prerouting priority raw; policy accept;
|
||||
iifname != "br-b52d11330b20" ip daddr 172.23.0.2 counter packets 0 bytes 0 drop
|
||||
iifname != "br-3d55385c5e7d" ip daddr 172.24.0.2 counter packets 0 bytes 0 drop
|
||||
}
|
||||
|
||||
chain OUTPUT {
|
||||
type filter hook output priority raw; policy accept;
|
||||
}
|
||||
}
|
||||
table ip filter {
|
||||
chain INPUT {
|
||||
type filter hook input priority filter; policy accept;
|
||||
counter packets 8461563 bytes 1841523123 jump ts-input
|
||||
}
|
||||
|
||||
chain FORWARD {
|
||||
type filter hook forward priority filter; policy drop;
|
||||
counter packets 20387709 bytes 3821595842 jump DOCKER-USER
|
||||
counter packets 20387709 bytes 3821595842 jump DOCKER-FORWARD
|
||||
counter packets 1064 bytes 88643 jump ts-forward
|
||||
}
|
||||
|
||||
chain OUTPUT {
|
||||
type filter hook output priority filter; policy accept;
|
||||
}
|
||||
|
||||
chain DOCKER-FORWARD {
|
||||
counter packets 20387709 bytes 3821595842 jump DOCKER-CT
|
||||
counter packets 14776132 bytes 2075147539 jump DOCKER-INTERNAL
|
||||
counter packets 14776132 bytes 2075147539 jump DOCKER-BRIDGE
|
||||
iifname "br-748fea589ab8" counter packets 0 bytes 0 accept
|
||||
iifname "br-b52d11330b20" counter packets 0 bytes 0 accept
|
||||
iifname "br-f91c61cdba28" counter packets 0 bytes 0 accept
|
||||
iifname "br-25187260f675" counter packets 0 bytes 0 accept
|
||||
iifname "br-3d55385c5e7d" counter packets 10132822 bytes 1741327665 accept
|
||||
iifname "br-50e2988e78eb" counter packets 0 bytes 0 accept
|
||||
iifname "docker0" counter packets 0 bytes 0 accept
|
||||
}
|
||||
|
||||
chain DOCKER-USER {
|
||||
}
|
||||
|
||||
chain ts-input {
|
||||
iifname "lo" ip saddr 100.103.64.82 counter packets 0 bytes 0 accept
|
||||
iifname "tailscale0" counter packets 512217 bytes 46924098 accept
|
||||
udp dport 41641 counter packets 3360373 bytes 516737750 accept
|
||||
iifname != "tailscale0" ip saddr 100.115.92.0/23 counter packets 0 bytes 0 return
|
||||
iifname != "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
|
||||
}
|
||||
|
||||
chain ts-forward {
|
||||
iifname "tailscale0" counter packets 28 bytes 2578 meta mark set mark and 0xff00ffff xor 0x40000
|
||||
meta mark & 0x00ff0000 == 0x00040000 counter packets 28 bytes 2578 accept
|
||||
oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
|
||||
oifname "tailscale0" counter packets 15 bytes 1554 accept
|
||||
}
|
||||
|
||||
chain DOCKER {
|
||||
iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 443 counter packets 0 bytes 0 accept
|
||||
iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 80 counter packets 0 bytes 0 accept
|
||||
iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 udp dport 53 counter packets 32283 bytes 2368019 accept
|
||||
iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 53 counter packets 638 bytes 34248 accept
|
||||
iifname != "br-748fea589ab8" oifname "br-748fea589ab8" counter packets 0 bytes 0 drop
|
||||
iifname != "br-b52d11330b20" oifname "br-b52d11330b20" counter packets 0 bytes 0 drop
|
||||
iifname != "br-f91c61cdba28" oifname "br-f91c61cdba28" counter packets 0 bytes 0 drop
|
||||
iifname != "br-25187260f675" oifname "br-25187260f675" counter packets 0 bytes 0 drop
|
||||
iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" counter packets 0 bytes 0 drop
|
||||
iifname != "br-50e2988e78eb" oifname "br-50e2988e78eb" counter packets 0 bytes 0 drop
|
||||
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
|
||||
}
|
||||
|
||||
chain DOCKER-BRIDGE {
|
||||
oifname "br-748fea589ab8" counter packets 0 bytes 0 jump DOCKER
|
||||
oifname "br-b52d11330b20" counter packets 0 bytes 0 jump DOCKER
|
||||
oifname "br-f91c61cdba28" counter packets 0 bytes 0 jump DOCKER
|
||||
oifname "br-25187260f675" counter packets 0 bytes 0 jump DOCKER
|
||||
oifname "br-3d55385c5e7d" counter packets 4643310 bytes 333819874 jump DOCKER
|
||||
oifname "br-50e2988e78eb" counter packets 0 bytes 0 jump DOCKER
|
||||
-- iptables --
|
||||
-P INPUT ACCEPT
|
||||
-P FORWARD DROP
|
||||
-P OUTPUT ACCEPT
|
||||
-N DOCKER
|
||||
-N DOCKER-BRIDGE
|
||||
-N DOCKER-CT
|
||||
-N DOCKER-FORWARD
|
||||
-N DOCKER-INTERNAL
|
||||
-N DOCKER-USER
|
||||
-N ts-forward
|
||||
-N ts-input
|
||||
-A INPUT -j ts-input
|
||||
-A FORWARD -j DOCKER-USER
|
||||
-A FORWARD -j DOCKER-FORWARD
|
||||
-A FORWARD -j ts-forward
|
||||
-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 443 -j ACCEPT
|
||||
-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 80 -j ACCEPT
|
||||
-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p udp -m udp --dport 53 -j ACCEPT
|
||||
-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 53 -j ACCEPT
|
||||
-A DOCKER ! -i br-748fea589ab8 -o br-748fea589ab8 -j DROP
|
||||
-A DOCKER ! -i br-b52d11330b20 -o br-b52d11330b20 -j DROP
|
||||
-A DOCKER ! -i br-f91c61cdba28 -o br-f91c61cdba28 -j DROP
|
||||
-A DOCKER ! -i br-25187260f675 -o br-25187260f675 -j DROP
|
||||
-A DOCKER ! -i br-3d55385c5e7d -o br-3d55385c5e7d -j DROP
|
||||
-A DOCKER ! -i br-50e2988e78eb -o br-50e2988e78eb -j DROP
|
||||
-A DOCKER ! -i docker0 -o docker0 -j DROP
|
||||
-A DOCKER-BRIDGE -o br-748fea589ab8 -j DOCKER
|
||||
-A DOCKER-BRIDGE -o br-b52d11330b20 -j DOCKER
|
||||
-A DOCKER-BRIDGE -o br-f91c61cdba28 -j DOCKER
|
||||
-A DOCKER-BRIDGE -o br-25187260f675 -j DOCKER
|
||||
-A DOCKER-BRIDGE -o br-3d55385c5e7d -j DOCKER
|
||||
-A DOCKER-BRIDGE -o br-50e2988e78eb -j DOCKER
|
||||
-A DOCKER-BRIDGE -o docker0 -j DOCKER
|
||||
-A DOCKER-CT -o br-748fea589ab8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o br-b52d11330b20 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o br-f91c61cdba28 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o br-25187260f675 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o br-3d55385c5e7d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o br-50e2988e78eb -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
|
||||
-- ufw --
|
||||
-- firewalld --
|
||||
-- selinux/apparmor --
|
||||
(SELinux not present)
|
||||
|
||||
========================================================================
|
||||
SECTION: 9. CRON & TIMERS
|
||||
========================================================================
|
||||
-- root crontab --
|
||||
(n/a)
|
||||
-- localuser crontab --
|
||||
(none)
|
||||
-- /etc/cron.d --
|
||||
-- pihole timer --
|
||||
|
||||
##### AUDIT END 2026-07-28T02:41:34Z #####
|
||||
Binary file not shown.
@@ -0,0 +1,896 @@
|
||||
This system is the property of Known Element Enterprises LLC.
|
||||
|
||||
Authorized uses only. All activity may be monitored and reported.
|
||||
|
||||
All activities subject to monitoring/recording/review in real time and/or at a later time.
|
||||
=== COMPOSE FILES: /root/pihole and /root/NTP ===
|
||||
|
||||
--- /root/pihole/docker-compose.yml ---
|
||||
# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
|
||||
services:
|
||||
pihole:
|
||||
container_name: pihole
|
||||
image: pihole/pihole:latest
|
||||
ports:
|
||||
# DNS Ports
|
||||
- "53:53/tcp"
|
||||
- "53:53/udp"
|
||||
# Default HTTP Port
|
||||
- "10002:80/tcp"
|
||||
# Default HTTPs Port. FTL will generate a self-signed certificate
|
||||
- "10003:443/tcp"
|
||||
# Uncomment the below if using Pi-hole as your DHCP Server
|
||||
#- "67:67/udp"
|
||||
# Uncomment the line below if you are using Pi-hole as your NTP server
|
||||
#- "123:123/udp"
|
||||
environment:
|
||||
# Set the appropriate timezone for your location from
|
||||
# https://en.wikipedia.org/wiki/List_of_tz_database_time_zones, e.g:
|
||||
TZ: 'America/Chicago'
|
||||
# Set a password to access the web interface. Not setting one will result in a random password being assigned
|
||||
FTLCONF_webserver_api_password: 'Gransyan1!'
|
||||
# If using Docker's default `bridge` network setting the dns listening mode should be set to 'all'
|
||||
FTLCONF_dns_listeningMode: 'all'
|
||||
# Volumes store your data between container upgrades
|
||||
volumes:
|
||||
# For persisting Pi-hole's databases and common configuration file
|
||||
- './etc-pihole:/etc/pihole'
|
||||
# Uncomment the below if you have custom dnsmasq config files that you want to persist. Not needed for most starting fresh with Pi-hole v6. If you're upgrading from v5 you and have used this directory before, you should keep it enabled for the first v6 container start to allow for a complete migration. It can be removed afterwards. Needs environment variable FTLCONF_misc_etc_dnsmasq_d: 'true'
|
||||
#- './etc-dnsmasq.d:/etc/dnsmasq.d'
|
||||
cap_add:
|
||||
# See https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
|
||||
# Required if you are using Pi-hole as your DHCP server, else not needed
|
||||
#- NET_ADMIN
|
||||
# Required if you are using Pi-hole as your NTP client to be able to set the host's system time
|
||||
#- SYS_TIME
|
||||
# Optional, if Pi-hole should get some more processing time
|
||||
- SYS_NICE
|
||||
restart: always
|
||||
|
||||
--- /root/NTP/docker-compose.yml ---
|
||||
services:
|
||||
ntp:
|
||||
image: dockurr/chrony
|
||||
container_name: tsys-ntp
|
||||
environment:
|
||||
NTP_SERVERS: "pool.ntp.org"
|
||||
ports:
|
||||
- 100.103.64.82:123:123/udp
|
||||
restart: always
|
||||
|
||||
=== DIR LAYOUT of compose project dirs ===
|
||||
total 16
|
||||
drwxr-xr-x 3 root root 4096 Feb 6 18:32 .
|
||||
drwx------ 10 root root 4096 Apr 5 06:16 ..
|
||||
-rw-r--r-- 1 root root 2089 Feb 6 18:30 docker-compose.yml
|
||||
drwxr-xr-x 7 localuser localuser 4096 Jul 27 20:57 etc-pihole
|
||||
total 12
|
||||
drwxr-xr-x 2 root root 4096 Jul 2 2025 .
|
||||
drwx------ 10 root root 4096 Apr 5 06:16 ..
|
||||
-rw-r--r-- 1 root root 188 Jun 23 2025 docker-compose.yml
|
||||
total 84
|
||||
drwx------ 10 root root 4096 Apr 5 06:16 .
|
||||
drwxr-xr-x 20 root root 4096 Jul 20 17:55 ..
|
||||
-rw------- 1 root root 4113 Jun 18 2025 .bash_history
|
||||
-rw-r--r-- 1 root root 571 Apr 10 2021 .bashrc
|
||||
drwx------ 3 root root 4096 Jun 10 2025 .config
|
||||
drwxr-xr-x 2 root root 4096 Apr 5 06:16 dhcp
|
||||
-rw------- 1 root root 2960 Feb 16 19:26 dhcpmigrate
|
||||
drwxr-xr-x 10 root root 4096 Jul 2 2025 FetchApply
|
||||
drwx------ 3 root root 4096 Jul 10 2025 .gnupg
|
||||
-rw------- 1 root root 40 Apr 5 06:15 .lesshst
|
||||
drwxr-xr-x 2 root root 4096 Jul 2 2025 NTP
|
||||
drwxr-xr-x 3 root root 4096 Feb 6 18:32 pihole
|
||||
-rw-r--r-- 1 root root 161 Jul 9 2019 .profile
|
||||
drwx------ 2 root root 4096 Jun 13 2025 .ssh
|
||||
drwxr-xr-x 2 root root 4096 Jul 27 09:08 .tmp
|
||||
-rw------- 1 root root 15370 Apr 5 06:15 .viminfo
|
||||
-rw-r--r-- 1 root root 180 Jun 17 2025 .wget-hsts
|
||||
|
||||
=== SEARCH entire FS for any technitium / dns compose files ===
|
||||
|
||||
=== ALL CONTAINERS with compose labels ===
|
||||
NAMES IMAGE STATUS WDIR=working dir CFG=config files SVC=service
|
||||
pihole pihole/pihole:latest Up 48 minutes (healthy) WDIR=/root/pihole CFG=/root/pihole/docker-compose.yml SVC=pihole
|
||||
tsys-ntp dockurr/chrony Up 7 days (healthy) WDIR=/root/NTP CFG=/root/NTP/docker-compose.yml SVC=ntp
|
||||
|
||||
=== DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container) ===
|
||||
|
||||
###### /pihole | image=pihole/pihole:latest | proj=pihole ######
|
||||
[
|
||||
{
|
||||
"Id": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
|
||||
"Created": "2026-02-07T00:32:57.4029722Z",
|
||||
"Path": "start.sh",
|
||||
"Args": [],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 1222239,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-07-28T01:57:27.732608642Z",
|
||||
"FinishedAt": "2026-07-28T01:57:27.095917957Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-07-27T21:43:50.682295418-05:00",
|
||||
"End": "2026-07-27T21:43:50.921778022-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "127.0.0.1\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:44:20.925872245-05:00",
|
||||
"End": "2026-07-27T21:44:21.170837655-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "127.0.0.1\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:44:51.175444367-05:00",
|
||||
"End": "2026-07-27T21:44:51.416140073-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "127.0.0.1\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:45:21.417102671-05:00",
|
||||
"End": "2026-07-27T21:45:21.689964304-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "127.0.0.1\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:45:51.691119192-05:00",
|
||||
"End": "2026-07-27T21:45:51.945416009-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "127.0.0.1\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"Image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
|
||||
"ResolvConfPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/resolv.conf",
|
||||
"HostnamePath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hostname",
|
||||
"HostsPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hosts",
|
||||
"LogPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82-json.log",
|
||||
"Name": "/pihole",
|
||||
"RestartCount": 2,
|
||||
"Driver": "overlay2",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "docker-default",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/root/pihole/etc-pihole:/etc/pihole:rw"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "pihole_default",
|
||||
"PortBindings": {
|
||||
"443/tcp": [
|
||||
{
|
||||
"HostIp": "",
|
||||
"HostPort": "10003"
|
||||
}
|
||||
],
|
||||
"53/tcp": [
|
||||
{
|
||||
"HostIp": "",
|
||||
"HostPort": "53"
|
||||
}
|
||||
],
|
||||
"53/udp": [
|
||||
{
|
||||
"HostIp": "",
|
||||
"HostPort": "53"
|
||||
}
|
||||
],
|
||||
"80/tcp": [
|
||||
{
|
||||
"HostIp": "",
|
||||
"HostPort": "10002"
|
||||
}
|
||||
]
|
||||
},
|
||||
"RestartPolicy": {
|
||||
"Name": "always",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": [
|
||||
"CAP_SYS_NICE"
|
||||
],
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": [],
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": [],
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "private",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": false,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": null,
|
||||
"UTSMode": "",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 67108864,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": null,
|
||||
"BlkioDeviceReadBps": null,
|
||||
"BlkioDeviceWriteBps": null,
|
||||
"BlkioDeviceReadIOps": null,
|
||||
"BlkioDeviceWriteIOps": null,
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": null,
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": null,
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": [
|
||||
"/proc/acpi",
|
||||
"/proc/asound",
|
||||
"/proc/interrupts",
|
||||
"/proc/kcore",
|
||||
"/proc/keys",
|
||||
"/proc/latency_stats",
|
||||
"/proc/sched_debug",
|
||||
"/proc/scsi",
|
||||
"/proc/timer_list",
|
||||
"/proc/timer_stats",
|
||||
"/sys/devices/virtual/powercap",
|
||||
"/sys/firmware"
|
||||
],
|
||||
"ReadonlyPaths": [
|
||||
"/proc/bus",
|
||||
"/proc/fs",
|
||||
"/proc/irq",
|
||||
"/proc/sys",
|
||||
"/proc/sysrq-trigger"
|
||||
]
|
||||
},
|
||||
"GraphDriver": {
|
||||
"Data": {
|
||||
"ID": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
|
||||
"LowerDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac-init/diff:/var/lib/docker/overlay2/c026d6eb78252f15477a6ed84b76e7855b525a582aafa11ae38d89d932433d9c/diff:/var/lib/docker/overlay2/38b4e9cd3ba8e5a10ee7dffbf21e4e552bc30ca96df4b3fc46532d8c2f4828d0/diff:/var/lib/docker/overlay2/10c7b2bc8ab474a29274b66e94e43e31db3dd248dbe0c8e8f7069e305efc7a41/diff:/var/lib/docker/overlay2/de5840a2fb5aed78f79984d980d59f9e50839ddb792146cc7a33d185b8b3eee2/diff:/var/lib/docker/overlay2/5f0504c14572ef5e745e8b06b8f1b3ce74027bce323b7bd1153c828e0dd97d52/diff:/var/lib/docker/overlay2/7a3eefaaac250fbb5430c567e9d12320c35c5da67ad3f8b7b8a7accccc435f18/diff:/var/lib/docker/overlay2/a6c7374834f19eabf6bd3e36f8ad8f2e20e3cf32c4bfea3133ef432b163dac8b/diff:/var/lib/docker/overlay2/28ffada6ac95ffb6c493b1908d00f9cfa096069fd715fa101b910b628835bed5/diff:/var/lib/docker/overlay2/0e79a525158aef2253d47f7f429c50e48c49fec81329f2be7fc580d906528087/diff:/var/lib/docker/overlay2/7365834c77d4c08772f03b8fc0fa6d21b593d3ac70a9ec66fe77ed403d0ef064/diff:/var/lib/docker/overlay2/4bd0de3b67968b6681a962ee9fb89b6bb09b07e5d2ba0d0f85f0561cd9442d32/diff",
|
||||
"MergedDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/merged",
|
||||
"UpperDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/diff",
|
||||
"WorkDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/work"
|
||||
},
|
||||
"Name": "overlay2"
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/root/pihole/etc-pihole",
|
||||
"Destination": "/etc/pihole",
|
||||
"Mode": "rw",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "1d09249f01b0",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": true,
|
||||
"AttachStderr": true,
|
||||
"ExposedPorts": {
|
||||
"123/udp": {},
|
||||
"443/tcp": {},
|
||||
"53/tcp": {},
|
||||
"53/udp": {},
|
||||
"67/udp": {},
|
||||
"80/tcp": {}
|
||||
},
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"FTLCONF_webserver_api_password=Gransyan1!",
|
||||
"FTLCONF_dns_listeningMode=all",
|
||||
"TZ=America/Chicago",
|
||||
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"DNSMASQ_USER=pihole",
|
||||
"FTL_CMD=no-daemon"
|
||||
],
|
||||
"Cmd": null,
|
||||
"Healthcheck": {
|
||||
"Test": [
|
||||
"CMD-SHELL",
|
||||
"dig -p $(pihole-FTL --config dns.port) +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
|
||||
]
|
||||
},
|
||||
"Image": "pihole/pihole:latest",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/",
|
||||
"Entrypoint": [
|
||||
"start.sh"
|
||||
],
|
||||
"Labels": {
|
||||
"com.docker.compose.config-hash": "60a181f601a1959cb6dda5f5889ff706d9986f7db015664ab40608253b83348e",
|
||||
"com.docker.compose.container-number": "1",
|
||||
"com.docker.compose.depends_on": "",
|
||||
"com.docker.compose.image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
|
||||
"com.docker.compose.oneoff": "False",
|
||||
"com.docker.compose.project": "pihole",
|
||||
"com.docker.compose.project.config_files": "/root/pihole/docker-compose.yml",
|
||||
"com.docker.compose.project.working_dir": "/root/pihole",
|
||||
"com.docker.compose.service": "pihole",
|
||||
"com.docker.compose.version": "5.0.2",
|
||||
"org.opencontainers.image.created": "2025-06-06T19:35:20.360Z",
|
||||
"org.opencontainers.image.description": "Pi-hole in a docker container",
|
||||
"org.opencontainers.image.licenses": "NOASSERTION",
|
||||
"org.opencontainers.image.revision": "b56e67ef6808de2854d30481ebcd40bcd2b43445",
|
||||
"org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
|
||||
"org.opencontainers.image.title": "docker-pi-hole",
|
||||
"org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
|
||||
"org.opencontainers.image.version": "2025.06.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "4c83df12c0f6aa377cdd17640639b35e737eb7530c6798d7b80eb3135dbfbd20",
|
||||
"SandboxKey": "/var/run/docker/netns/4c83df12c0f6",
|
||||
"Ports": {
|
||||
"123/udp": null,
|
||||
"443/tcp": [
|
||||
{
|
||||
"HostIp": "0.0.0.0",
|
||||
"HostPort": "10003"
|
||||
},
|
||||
{
|
||||
"HostIp": "::",
|
||||
"HostPort": "10003"
|
||||
}
|
||||
],
|
||||
"53/tcp": [
|
||||
{
|
||||
"HostIp": "0.0.0.0",
|
||||
"HostPort": "53"
|
||||
},
|
||||
{
|
||||
"HostIp": "::",
|
||||
"HostPort": "53"
|
||||
}
|
||||
],
|
||||
"53/udp": [
|
||||
{
|
||||
"HostIp": "0.0.0.0",
|
||||
"HostPort": "53"
|
||||
},
|
||||
{
|
||||
"HostIp": "::",
|
||||
"HostPort": "53"
|
||||
}
|
||||
],
|
||||
"67/udp": null,
|
||||
"80/tcp": [
|
||||
{
|
||||
"HostIp": "0.0.0.0",
|
||||
"HostPort": "10002"
|
||||
},
|
||||
{
|
||||
"HostIp": "::",
|
||||
"HostPort": "10002"
|
||||
}
|
||||
]
|
||||
},
|
||||
"Networks": {
|
||||
"pihole_default": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": [
|
||||
"pihole",
|
||||
"pihole"
|
||||
],
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "3d55385c5e7d7d4ca5ddec3f98533d43452cdc2fda68831ac4a16aeb0d42867c",
|
||||
"EndpointID": "aadb163f607f9247c8ad41f8cadac2282272501c60e3ea7fa31fd2d584271447",
|
||||
"Gateway": "172.24.0.1",
|
||||
"IPAddress": "172.24.0.2",
|
||||
"MacAddress": "7e:48:e2:84:c5:15",
|
||||
"IPPrefixLen": 16,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": [
|
||||
"pihole",
|
||||
"1d09249f01b0"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
###### /tsys-ntp | image=dockurr/chrony | proj=ntp ######
|
||||
[
|
||||
{
|
||||
"Id": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
|
||||
"Created": "2025-06-23T16:51:39.454106229Z",
|
||||
"Path": "/bin/startup",
|
||||
"Args": [],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2109237,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-07-20T22:55:49.597671587Z",
|
||||
"FinishedAt": "2026-07-20T22:55:37.650310687Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-07-27T21:44:16.437981113-05:00",
|
||||
"End": "2026-07-27T21:44:16.501904568-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "Reference ID : 00000000 ()\nStratum : 0\nRef time (UTC) : Thu Jan 01 00:00:00 1970\nSystem time : 0.525388062 seconds slow of NTP time\nLast offset : +0.000000000 seconds\nRMS offset : 0.000000000 seconds\nFrequency : 0.849 ppm slow\nResidual freq : +0.000 ppm\nSkew : 0.000 ppm\nRoot delay : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status : Not synchronised\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:44:46.50366458-05:00",
|
||||
"End": "2026-07-27T21:44:46.566905934-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "Reference ID : 00000000 ()\nStratum : 0\nRef time (UTC) : Thu Jan 01 00:00:00 1970\nSystem time : 0.525413632 seconds slow of NTP time\nLast offset : +0.000000000 seconds\nRMS offset : 0.000000000 seconds\nFrequency : 0.849 ppm slow\nResidual freq : +0.000 ppm\nSkew : 0.000 ppm\nRoot delay : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status : Not synchronised\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:45:16.568367435-05:00",
|
||||
"End": "2026-07-27T21:45:16.632376789-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "Reference ID : 00000000 ()\nStratum : 0\nRef time (UTC) : Thu Jan 01 00:00:00 1970\nSystem time : 0.525439143 seconds slow of NTP time\nLast offset : +0.000000000 seconds\nRMS offset : 0.000000000 seconds\nFrequency : 0.849 ppm slow\nResidual freq : +0.000 ppm\nSkew : 0.000 ppm\nRoot delay : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status : Not synchronised\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:45:46.63406538-05:00",
|
||||
"End": "2026-07-27T21:45:46.695064332-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "Reference ID : 00000000 ()\nStratum : 0\nRef time (UTC) : Thu Jan 01 00:00:00 1970\nSystem time : 0.525464714 seconds slow of NTP time\nLast offset : +0.000000000 seconds\nRMS offset : 0.000000000 seconds\nFrequency : 0.849 ppm slow\nResidual freq : +0.000 ppm\nSkew : 0.000 ppm\nRoot delay : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status : Not synchronised\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-07-27T21:46:16.697074212-05:00",
|
||||
"End": "2026-07-27T21:46:16.763778169-05:00",
|
||||
"ExitCode": 0,
|
||||
"Output": "Reference ID : 00000000 ()\nStratum : 0\nRef time (UTC) : Thu Jan 01 00:00:00 1970\nSystem time : 0.525490224 seconds slow of NTP time\nLast offset : +0.000000000 seconds\nRMS offset : 0.000000000 seconds\nFrequency : 0.849 ppm slow\nResidual freq : +0.000 ppm\nSkew : 0.000 ppm\nRoot delay : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status : Not synchronised\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"Image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
|
||||
"ResolvConfPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/resolv.conf",
|
||||
"HostnamePath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hostname",
|
||||
"HostsPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hosts",
|
||||
"LogPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908-json.log",
|
||||
"Name": "/tsys-ntp",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlay2",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "docker-default",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": null,
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "ntp_default",
|
||||
"PortBindings": {
|
||||
"123/udp": [
|
||||
{
|
||||
"HostIp": "100.103.64.82",
|
||||
"HostPort": "123"
|
||||
}
|
||||
]
|
||||
},
|
||||
"RestartPolicy": {
|
||||
"Name": "always",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": [],
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": [],
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "private",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": false,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": null,
|
||||
"UTSMode": "",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 67108864,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": null,
|
||||
"BlkioDeviceReadBps": null,
|
||||
"BlkioDeviceWriteBps": null,
|
||||
"BlkioDeviceReadIOps": null,
|
||||
"BlkioDeviceWriteIOps": null,
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": null,
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": null,
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "volume",
|
||||
"Source": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
|
||||
"Target": "/run/chrony"
|
||||
},
|
||||
{
|
||||
"Type": "volume",
|
||||
"Source": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
|
||||
"Target": "/var/lib/chrony"
|
||||
},
|
||||
{
|
||||
"Type": "volume",
|
||||
"Source": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
|
||||
"Target": "/etc/chrony"
|
||||
}
|
||||
],
|
||||
"MaskedPaths": [
|
||||
"/proc/asound",
|
||||
"/proc/acpi",
|
||||
"/proc/interrupts",
|
||||
"/proc/kcore",
|
||||
"/proc/keys",
|
||||
"/proc/latency_stats",
|
||||
"/proc/timer_list",
|
||||
"/proc/timer_stats",
|
||||
"/proc/sched_debug",
|
||||
"/proc/scsi",
|
||||
"/sys/firmware",
|
||||
"/sys/devices/virtual/powercap"
|
||||
],
|
||||
"ReadonlyPaths": [
|
||||
"/proc/bus",
|
||||
"/proc/fs",
|
||||
"/proc/irq",
|
||||
"/proc/sys",
|
||||
"/proc/sysrq-trigger"
|
||||
]
|
||||
},
|
||||
"GraphDriver": {
|
||||
"Data": {
|
||||
"ID": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
|
||||
"LowerDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db-init/diff:/var/lib/docker/overlay2/6de110c8e8511a542a0541ecba7006ff7fe26ee0e91b738d87fec292a76a5cb1/diff:/var/lib/docker/overlay2/5dc67368f8acabea18fc0be0b07d52e5a82362b5d57119871e3471c509927a73/diff:/var/lib/docker/overlay2/ce6cab3e3329a7148ae6c5e55af761b4b533eac36dcd08cb34d3de88e958d92f/diff",
|
||||
"MergedDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/merged",
|
||||
"UpperDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/diff",
|
||||
"WorkDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/work"
|
||||
},
|
||||
"Name": "overlay2"
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "volume",
|
||||
"Name": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
|
||||
"Source": "/var/lib/docker/volumes/8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b/_data",
|
||||
"Destination": "/var/lib/chrony",
|
||||
"Driver": "local",
|
||||
"Mode": "z",
|
||||
"RW": true,
|
||||
"Propagation": ""
|
||||
},
|
||||
{
|
||||
"Type": "volume",
|
||||
"Name": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
|
||||
"Source": "/var/lib/docker/volumes/562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41/_data",
|
||||
"Destination": "/etc/chrony",
|
||||
"Driver": "local",
|
||||
"Mode": "z",
|
||||
"RW": true,
|
||||
"Propagation": ""
|
||||
},
|
||||
{
|
||||
"Type": "volume",
|
||||
"Name": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
|
||||
"Source": "/var/lib/docker/volumes/e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629/_data",
|
||||
"Destination": "/run/chrony",
|
||||
"Driver": "local",
|
||||
"Mode": "z",
|
||||
"RW": true,
|
||||
"Propagation": ""
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "c33ced53142e",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": true,
|
||||
"AttachStderr": true,
|
||||
"ExposedPorts": {
|
||||
"123/udp": {}
|
||||
},
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"NTP_SERVERS=pool.ntp.org",
|
||||
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"NTP_DIRECTIVES=ratelimit\\nrtcsync"
|
||||
],
|
||||
"Cmd": null,
|
||||
"Healthcheck": {
|
||||
"Test": [
|
||||
"CMD-SHELL",
|
||||
"chronyc -n tracking || exit 1"
|
||||
]
|
||||
},
|
||||
"Image": "dockurr/chrony",
|
||||
"Volumes": {
|
||||
"/etc/chrony": {},
|
||||
"/run/chrony": {},
|
||||
"/var/lib/chrony": {}
|
||||
},
|
||||
"WorkingDir": "/",
|
||||
"Entrypoint": [
|
||||
"/bin/startup"
|
||||
],
|
||||
"Labels": {
|
||||
"com.docker.compose.config-hash": "b9a2451b9ec10e82b113b417df6060f18aab234f117063b84ac7a29455f2feb0",
|
||||
"com.docker.compose.container-number": "1",
|
||||
"com.docker.compose.depends_on": "",
|
||||
"com.docker.compose.image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
|
||||
"com.docker.compose.oneoff": "False",
|
||||
"com.docker.compose.project": "ntp",
|
||||
"com.docker.compose.project.config_files": "/root/NTP/docker-compose.yml",
|
||||
"com.docker.compose.project.working_dir": "/root/NTP",
|
||||
"com.docker.compose.replace": "61f92d4c43f32567409816af5e3c1c9622aedcb5e35b27b25980282fa3c3e1b3",
|
||||
"com.docker.compose.service": "ntp",
|
||||
"com.docker.compose.version": "2.36.2",
|
||||
"org.opencontainers.image.created": "2025-06-12T01:51:48.447Z",
|
||||
"org.opencontainers.image.description": "🕒 chronyd NTP server in a Docker container.",
|
||||
"org.opencontainers.image.licenses": "MIT",
|
||||
"org.opencontainers.image.revision": "43fb0f2b381bccfef5123971a51baf450ab24464",
|
||||
"org.opencontainers.image.source": "https://github.com/dockur/chrony",
|
||||
"org.opencontainers.image.title": "Chrony",
|
||||
"org.opencontainers.image.url": "https://github.com/dockur/chrony",
|
||||
"org.opencontainers.image.version": "4.7"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "ccaa71c75942abc56df6ae1c89a8c2ae91e769ed52eab64635a63f8ee996f782",
|
||||
"SandboxKey": "/var/run/docker/netns/ccaa71c75942",
|
||||
"Ports": {},
|
||||
"Networks": {}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
=== PI-HOLE internal config (sudo docker exec, READ-ONLY) ===
|
||||
-- ls /etc/pihole --
|
||||
total 2545492
|
||||
drwxr-xr-x 7 pihole pihole 4096 Jul 27 20:57 .
|
||||
drwxr-xr-x 1 root root 4096 Feb 6 18:32 ..
|
||||
-rw-r----- 1 pihole pihole 65 Jun 23 2025 adlists.list
|
||||
-rw-r----- 1 pihole pihole 44 Jul 27 20:57 cli_pw
|
||||
drwxr-xr-x 2 pihole pihole 4096 Apr 6 10:07 config_backups
|
||||
-rw-r----- 1 pihole pihole 0 Jun 18 2025 dhcp.leases
|
||||
-rw-r----- 1 pihole pihole 5753 Apr 6 10:07 dnsmasq.conf
|
||||
-rw-r----- 1 pihole pihole 5500928 Jul 26 04:51 gravity.db
|
||||
drwxr-xr-x 2 pihole pihole 4096 Jul 26 04:51 gravity_backups
|
||||
-rw-r----- 1 pihole pihole 4751360 Jul 19 04:51 gravity_old.db
|
||||
drwxr-xr-x 2 pihole pihole 4096 Jun 18 2025 hosts
|
||||
drwxr-xr-x 2 pihole pihole 4096 Jul 26 04:51 listsCache
|
||||
-rw-r----- 1 root root 421 Jul 27 20:57 logrotate
|
||||
drwxr-xr-x 2 pihole pihole 4096 Jun 18 2025 migration_backup
|
||||
-rw-r----- 1 pihole pihole 2591891456 Jul 27 21:40 pihole-FTL.db
|
||||
-rw-r----- 1 pihole pihole 32768 Jul 27 21:46 pihole-FTL.db-shm
|
||||
-rw-r----- 1 pihole pihole 4272472 Jul 27 21:46 pihole-FTL.db-wal
|
||||
-rw-r----- 1 pihole pihole 55996 Apr 6 10:07 pihole.toml
|
||||
-rw------- 1 pihole pihole 713 Jun 18 2025 tls.crt
|
||||
-rw------- 1 pihole pihole 1734 Jun 18 2025 tls.pem
|
||||
-rw------- 1 pihole pihole 733 Jun 18 2025 tls_ca.crt
|
||||
-rw-r--r-- 1 pihole pihole 376 Jul 27 20:57 versions
|
||||
-- ls /etc/dnsmasq.d --
|
||||
ls: cannot access '/etc/dnsmasq.d': No such file or directory
|
||||
--- /etc/pihole/setupVars.conf ---
|
||||
cat: /etc/pihole/setupVars.conf: No such file or directory
|
||||
--- /etc/pihole/pihole-FTL.conf ---
|
||||
cat: /etc/pihole/pihole-FTL.conf: No such file or directory
|
||||
--- /etc/pihole/adlists.list ---
|
||||
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
|
||||
--- /etc/pihole/custom.list ---
|
||||
cat: /etc/pihole/custom.list: No such file or directory
|
||||
--- /etc/pihole/local.list ---
|
||||
cat: /etc/pihole/local.list: No such file or directory
|
||||
--- /etc/pihole/regex.list ---
|
||||
cat: /etc/pihole/regex.list: No such file or directory
|
||||
--- /etc/pihole/dhcp.leases ---
|
||||
--- /etc/pihole/static_ip.conf ---
|
||||
cat: /etc/pihole/static_ip.conf: No such file or directory
|
||||
-- /etc/dnsmasq.d/* --
|
||||
--- /etc/dnsmasq.d/* ---
|
||||
cat: '/etc/dnsmasq.d/*': No such file or directory
|
||||
-- pihole version --
|
||||
Core version is v6.1.2 (Latest: v6.4.3)
|
||||
Web version is v6.2.1 (Latest: v6.6)
|
||||
FTL version is v6.2.2 (Latest: v6.7)
|
||||
-- gravity row counts --
|
||||
adlist=domainlist=client=group=info=-- adlist addresses --
|
||||
OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
|
||||
-- domainlist (allow+deny, first 60) --
|
||||
OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
|
||||
|
||||
=== CHRONY/NTP container config (tsys-ntp) ===
|
||||
--- chrony.conf ---
|
||||
# https://github.com/dockur/chrony
|
||||
|
||||
# chrony.conf file generated by startup script
|
||||
# located at /bin/startup
|
||||
|
||||
# time servers provided by NTP_SERVER environment variables.
|
||||
server pool.ntp.org iburst
|
||||
|
||||
driftfile /var/lib/chrony/chrony.drift
|
||||
makestep 0.1 3
|
||||
ratelimit
|
||||
rtcsync
|
||||
|
||||
allow all
|
||||
--- ls /etc ---
|
||||
total 176
|
||||
drwxr-xr-x 1 root root 4096 Jun 23 2025 .
|
||||
drwxr-xr-x 1 root root 4096 Jun 23 2025 ..
|
||||
-rw-r--r-- 1 root root 21 Jan 8 2025 alpine-release
|
||||
drwxr-xr-x 1 root root 4096 Jun 12 2025 apk
|
||||
drwxr-xr-x 2 root root 4096 Jan 8 2025 busybox-paths.d
|
||||
drwxr-xr-x 2 chrony chrony 4096 Jun 23 2025 chrony
|
||||
drwxr-xr-x 2 root root 4096 Jan 8 2025 crontabs
|
||||
-rw-r--r-- 1 root root 89 Jan 4 2025 fstab
|
||||
-rw-r--r-- 1 root root 530 Jun 12 2025 group
|
||||
-rw-r--r-- 1 root root 524 Jun 12 2025 group-
|
||||
-rw-r--r-- 1 root root 13 Jul 20 22:55 hostname
|
||||
-rw-r--r-- 1 root root 148 Jul 20 22:55 hosts
|
||||
-rw-r--r-- 1 root root 570 Jan 4 2025 inittab
|
||||
-rw-r--r-- 1 root root 77 Jan 8 2025 issue
|
||||
drwxr-xr-x 1 root root 4096 Jun 12 2025 logrotate.d
|
||||
drwxr-xr-x 2 root root 4096 Jan 8 2025 modprobe.d
|
||||
-rw-r--r-- 1 root root 15 Jan 4 2025 modules
|
||||
drwxr-xr-x 2 root root 4096 Jan 8 2025 modules-load.d
|
||||
-rw-r--r-- 1 root root 284 Jan 4 2025 motd
|
||||
lrwxrwxrwx 1 root root 12 Jun 23 2025 mtab -> /proc/mounts
|
||||
drwxr-xr-x 8 root root 4096 Jan 8 2025 network
|
||||
-rw-r--r-- 1 root root 205 Jan 4 2025 nsswitch.conf
|
||||
drwxr-xr-x 2 root root 4096 Jan 8 2025 opt
|
||||
lrwxrwxrwx 1 root root 21 Jan 8 2025 os-release -> ../usr/lib/os-release
|
||||
-rw-r--r-- 1 root root 756 Jun 12 2025 passwd
|
||||
-rw-r--r-- 1 root root 702 Jan 4 2025 passwd-
|
||||
drwxr-xr-x 7 root root 4096 Jan 8 2025 periodic
|
||||
drwxr-xr-x 2 root root 4096 Jun 12 2025 pkcs11
|
||||
-rw-r--r-- 1 root root 547 Jan 4 2025 profile
|
||||
MS Name/IP address Stratum Poll Reach LastRx Last sample
|
||||
===============================================================================
|
||||
Reference ID : 00000000 ()
|
||||
Stratum : 0
|
||||
Ref time (UTC) : Thu Jan 01 00:00:00 1970
|
||||
System time : 0.525492370 seconds slow of NTP time
|
||||
Last offset : +0.000000000 seconds
|
||||
RMS offset : 0.000000000 seconds
|
||||
Frequency : 0.849 ppm slow
|
||||
Residual freq : +0.000 ppm
|
||||
Skew : 0.000 ppm
|
||||
Root delay : 1.000000000 seconds
|
||||
Root dispersion : 1.000000000 seconds
|
||||
Update interval : 0.0 seconds
|
||||
Leap status : Not synchronised
|
||||
-- bare metal ntpsec.conf --
|
||||
driftfile /var/lib/ntp/ntp.drift
|
||||
leapfile /usr/share/zoneinfo/leap-seconds.list
|
||||
server pfvsvrpi.knel.net
|
||||
restrict 127.0.0.1
|
||||
restrict ::1
|
||||
|
||||
=== TECHNITIUM volumes ===
|
||||
|
||||
--- volume dns_tsys-dns-config -> /var/lib/docker/volumes/dns_tsys-dns-config/_data ---
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062320.stat
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062321.stat
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/logs/2025-06-23.log
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/self-signed-cert.pfx
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/cache.bin
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/119.127.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/46.96.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/181.103.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/184.108.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/211.114.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/145.105.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/97.82.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/194.67.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/2.108.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/75.110.100.in-addr.arpa.zone
|
||||
/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/64.103.100.in-addr.arpa.zone
|
||||
-- config dir listing --
|
||||
-- config.xml --
|
||||
|
||||
--- volume dns_tyss-dns-config -> /var/lib/docker/volumes/dns_tyss-dns-config/_data ---
|
||||
-- config dir listing --
|
||||
-- config.xml --
|
||||
|
||||
=== DONE ===
|
||||
@@ -0,0 +1,116 @@
|
||||
# pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
|
||||
|
||||
> **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
|
||||
> Do **not** modify it. This document describes it as audited so its services can
|
||||
> be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
|
||||
> collected by **read-only** audit scripts (`audit-netboot.sh`,
|
||||
> `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
|
||||
|
||||
## 1. Host
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
|
||||
| OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
|
||||
| Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
|
||||
| Timezone | `America/Chicago` (US/Central) |
|
||||
| LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
|
||||
| Tailscale | `100.103.64.82` (`tailscale0`) |
|
||||
| DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
|
||||
| Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
|
||||
| Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
|
||||
|
||||
`eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
|
||||
`ntp_default`, `dns_default`, and several stale ones).
|
||||
|
||||
## 2. Services overview
|
||||
|
||||
| Service | Form | Running? |
|
||||
|---|---|---|
|
||||
| **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
|
||||
| **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
|
||||
| **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
|
||||
| **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
|
||||
|
||||
## 3. Pi-hole (container)
|
||||
|
||||
- **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
|
||||
- **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
|
||||
- **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
|
||||
- **Ports (host):**
|
||||
|
||||
| Host | Container | Purpose |
|
||||
|---|---|---|
|
||||
| `53/tcp`, `53/udp` | 53 | DNS |
|
||||
| `10002/tcp` | 80 | Web admin (HTTP) |
|
||||
| `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
|
||||
|
||||
- **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=Gransyan1!`, `FTLCONF_dns_listeningMode=all`
|
||||
- **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
|
||||
- **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
|
||||
- Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
|
||||
- `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
|
||||
- `queryLogging = true`, DNSSEC off
|
||||
- **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
|
||||
- **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
|
||||
- **Web admin:** `http://pfv-netboot:10002/admin/` (password `Gransyan1!`)
|
||||
- Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
|
||||
|
||||
## 4. NTP (two layers)
|
||||
|
||||
### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
|
||||
- **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
|
||||
- **Image:** `dockurr/chrony`
|
||||
- **Env:** `NTP_SERVERS=pool.ntp.org`
|
||||
- **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
|
||||
- `restart: always`
|
||||
- chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
|
||||
- On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
|
||||
|
||||
### 4b. bare-metal `ntpsec`
|
||||
- Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
|
||||
- **Config** (`/etc/ntpsec/ntp.conf`):
|
||||
```
|
||||
driftfile /var/lib/ntp/ntp.drift
|
||||
leapfile /usr/share/zoneinfo/leap-seconds.list
|
||||
server pfvsvrpi.knel.net
|
||||
restrict 127.0.0.1
|
||||
restrict ::1
|
||||
```
|
||||
- Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
|
||||
|
||||
## 5. Technitium DNS (currently stopped)
|
||||
|
||||
- **Not running** — no container and **no compose file** exists for it.
|
||||
- A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
|
||||
(mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
|
||||
are intact (last activity 2025-06-23). A second typo'd volume
|
||||
`dns_tyss-dns-config` is empty.
|
||||
- Config files are **binary** (Technitium's own serialization), but copy verbatim:
|
||||
`dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
|
||||
`self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
|
||||
- **Zones present** (12 reverse + 1 forward):
|
||||
- `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
|
||||
- Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
|
||||
- **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
|
||||
- The compose project name historically was `dns` (network `dns_default` still exists).
|
||||
|
||||
## 6. Firewall / misc
|
||||
|
||||
- nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
|
||||
`DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
|
||||
`OUTPUT ACCEPT`. No UFW / firewalld.
|
||||
- Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
|
||||
(2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
|
||||
usermin (10000/10002/20000), Tailscale (41641).
|
||||
- SELinux absent; AppArmor default docker profile.
|
||||
|
||||
## 7. How it was audited (no changes made)
|
||||
|
||||
```bash
|
||||
ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh # broad read-only sweep
|
||||
ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh # docker inspect + compose
|
||||
ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh # pihole.toml + technitium
|
||||
```
|
||||
|
||||
Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.
|
||||
@@ -0,0 +1,220 @@
|
||||
# pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
|
||||
|
||||
These two nodes replicate the network-infrastructure services of **pfv-netboot**
|
||||
(Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
|
||||
reads config from pfv-netboot (read-only) and relays it to each target.
|
||||
|
||||
## 1. Nodes
|
||||
|
||||
| | pfv-netinfra-01 | pfv-netinfra-02 |
|
||||
|---|---|---|
|
||||
| OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
|
||||
| LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
|
||||
| Tailscale | `100.70.181.72` | `100.93.194.82` |
|
||||
| RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
|
||||
| Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
|
||||
| Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
|
||||
| Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
|
||||
|
||||
## 2. Service layout
|
||||
|
||||
All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
|
||||
so the compose files are directly editable; data dirs keep container uids):
|
||||
|
||||
```
|
||||
/home/localuser/services/
|
||||
├── pihole/
|
||||
│ ├── docker-compose.yml
|
||||
│ └── etc-pihole/ # copied from netboot /root/pihole/etc-pihole
|
||||
│ ├── pihole.toml # Pi-hole v6 config (upstreams, etc.)
|
||||
│ ├── gravity.db # adlists / domainlists / clients / groups
|
||||
│ ├── adlists.list
|
||||
│ ├── dnsmasq.conf
|
||||
│ ├── tls.{crt,pem,crt_ca}
|
||||
│ └── versions
|
||||
├── ntp/
|
||||
│ └── docker-compose.yml # chrony container (see §5 — not used; host ntpsec serves)
|
||||
└── technitium/
|
||||
├── docker-compose.yml
|
||||
└── config/ # copied from netboot orphaned volume dns_tsys-dns-config/_data
|
||||
├── dns.config
|
||||
├── auth.config
|
||||
├── scopes/Default.scope
|
||||
├── self-signed-cert.pfx
|
||||
└── zones/ # knel.net.zone + 12 Tailscale reverse zones
|
||||
```
|
||||
|
||||
## 3. Pi-hole (container `pihole`)
|
||||
|
||||
Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
|
||||
|
||||
| Host port | Container | Purpose |
|
||||
|---|---|---|
|
||||
| `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
|
||||
| `10002/tcp` | 80 | Web admin (HTTP) |
|
||||
| `10003/tcp` | 443 | Web admin (HTTPS) |
|
||||
|
||||
`docker-compose.yml`:
|
||||
```yaml
|
||||
services:
|
||||
pihole:
|
||||
container_name: pihole
|
||||
image: pihole/pihole:latest
|
||||
hostname: pihole
|
||||
ports:
|
||||
- "53:53/tcp"
|
||||
- "53:53/udp"
|
||||
- "10002:80/tcp"
|
||||
- "10003:443/tcp"
|
||||
environment:
|
||||
TZ: 'America/Chicago'
|
||||
FTLCONF_webserver_api_password: 'Gransyan1!'
|
||||
FTLCONF_dns_listeningMode: 'all'
|
||||
volumes:
|
||||
- './etc-pihole:/etc/pihole'
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
restart: always
|
||||
```
|
||||
- Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
|
||||
- Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
|
||||
- `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
|
||||
- Web admin: `http://<node>:10002/admin/` — password **`Gransyan1!`** (same as netboot).
|
||||
- Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
|
||||
|
||||
## 4. Technitium DNS (container `tsys-dns`)
|
||||
|
||||
Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
|
||||
`knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
|
||||
orphaned `dns_tsys-dns-config` volume.
|
||||
|
||||
| Host port | Container | Purpose |
|
||||
|---|---|---|
|
||||
| `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
|
||||
| `5380/tcp` | 5380 | Web console (HTTP) |
|
||||
| `53443/tcp` | 53443 | Web console (HTTPS) |
|
||||
|
||||
`docker-compose.yml`:
|
||||
```yaml
|
||||
services:
|
||||
technitium:
|
||||
image: technitium/dns-server
|
||||
container_name: tsys-dns
|
||||
ports:
|
||||
- "5300:53/tcp"
|
||||
- "5300:53/udp"
|
||||
- "5380:5380/tcp"
|
||||
- "53443:53443/tcp"
|
||||
volumes:
|
||||
- './config:/etc/dns'
|
||||
restart: always
|
||||
```
|
||||
- Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
|
||||
- Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
|
||||
password (carried over via `auth.config`). If the password is unknown, reset it
|
||||
from the console or by removing `config/auth.config` and recreating the container.
|
||||
- **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
|
||||
because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
|
||||
query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
|
||||
Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
|
||||
Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
|
||||
from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
|
||||
`knel.net` A-records directly in Pi-hole's Local DNS).
|
||||
|
||||
## 5. NTP (host `ntpsec`, not a container)
|
||||
|
||||
Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
|
||||
serves NTP on every local address — including the Tailscale IP — and keeps the
|
||||
system clock synced. This is the **same daemon family as netboot's own bare-metal
|
||||
ntpsec**.
|
||||
|
||||
- **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
|
||||
Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
|
||||
does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
|
||||
**does** bind the Tailscale IP, so the container cannot claim it (`address
|
||||
already in use`) and would be a non-functional duplicate (verified: the
|
||||
container started but never synced — Stratum 0). It is therefore intentionally
|
||||
**omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
|
||||
host NTP unit and removes any stale `tsys-ntp` container.
|
||||
- ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
|
||||
`restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
|
||||
- Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
|
||||
leap normal.
|
||||
|
||||
The `ntp/docker-compose.yml` is still written on each node for parity/reference
|
||||
(and in case the host NTP is ever disabled — then `sudo docker compose -f
|
||||
/home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
|
||||
|
||||
## 6. Verification results (2026-07-28)
|
||||
|
||||
| Check | pfv-netinfra-01 | pfv-netinfra-02 |
|
||||
|---|---|---|
|
||||
| `pihole` health | healthy | healthy |
|
||||
| `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
|
||||
| Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
|
||||
| `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
|
||||
| Technitium web `:5380` | HTTP 200 | HTTP 200 |
|
||||
| NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
|
||||
|
||||
## 7. Operating the services
|
||||
|
||||
```bash
|
||||
# status
|
||||
sudo docker ps
|
||||
|
||||
# Pi-hole
|
||||
sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
|
||||
sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
|
||||
sudo docker exec pihole pihole -v # version
|
||||
sudo docker exec pihole pihole -g # rebuild gravity
|
||||
sudo docker exec pihole pihole -a -p # set/change web password
|
||||
|
||||
# Technitium
|
||||
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
|
||||
sudo docker exec tsys-dns sh # explore /etc/dns
|
||||
|
||||
# NTP (host)
|
||||
systemctl status ntpsec
|
||||
ntpq -pn
|
||||
```
|
||||
|
||||
## 8. Differences from pfv-netboot (intentional)
|
||||
|
||||
1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
|
||||
can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
|
||||
2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
|
||||
3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
|
||||
(the container cannot bind the Tailscale IP here; see §5).
|
||||
4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
|
||||
on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
|
||||
5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
|
||||
copied (transient); gravity DB and all configuration are.
|
||||
|
||||
## 9. Re-running / reproducing
|
||||
|
||||
`setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
|
||||
present and uses `docker compose up -d` (no-ops when unchanged). It reads
|
||||
pfv-netboot read-only and never mutates it.
|
||||
|
||||
```bash
|
||||
./setup-netinfra.sh # deploy to both nodes
|
||||
./setup-netinfra.sh pfv-netinfra-01 # deploy one node
|
||||
./setup-netinfra.sh pfv-netinfra-01 verify # verify only
|
||||
```
|
||||
|
||||
Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
|
||||
sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
|
||||
internet for image pulls.
|
||||
|
||||
## 10. Files in this directory
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
|
||||
| `audit-netboot.sh` | broad read-only audit of pfv-netboot |
|
||||
| `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
|
||||
| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
|
||||
| `baseline.sh` | read-only baseline of a target node |
|
||||
| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
|
||||
| `pfv-netboot-setup.md` | reference-node documentation |
|
||||
| `pfv-netinfra-setup.md` | this document |
|
||||
Executable
+323
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# setup-netinfra.sh
|
||||
# -----------------------------------------------------------------------------
|
||||
# Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
|
||||
# onto pfv-netinfra-01 and pfv-netinfra-02.
|
||||
#
|
||||
# DESIGN
|
||||
# * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
|
||||
# from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
|
||||
# * The targets cannot SSH to pfv-netboot directly, so config tarballs are
|
||||
# relayed through this workstation:
|
||||
# ssh netboot 'sudo tar -cf - ...' | ssh target 'sudo tar -xf - ...'
|
||||
# * Services are deployed under /home/localuser/services/<svc>/ on each
|
||||
# target so localuser can manage them (mirrors netboot's localuser-owned
|
||||
# pihole data dir). `sudo docker` is used since localuser is not in the
|
||||
# docker group (same as on netboot).
|
||||
#
|
||||
# SERVICES
|
||||
# pihole pihole/pihole:latest :53 tcp/udp :10002->80 :10003->443
|
||||
# ntp (chrony) dockurr/chrony <tailscale-ip>:123:123/udp
|
||||
# technitium technitium/dns-server :5300->53 tcp/udp :5380 :53443
|
||||
# (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
|
||||
# The knel.net authoritative zone + Tailscale reverse zones are preserved
|
||||
# verbatim from netboot's orphaned dns_tsys-dns-config volume.)
|
||||
#
|
||||
# USAGE
|
||||
# ./setup-netinfra.sh # deploy to BOTH nodes
|
||||
# ./setup-netinfra.sh pfv-netinfra-01 # deploy to one node
|
||||
# ./setup-netinfra.sh pfv-netinfra-01 verify # verify only
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NETBOOT="localuser@pfv-netboot"
|
||||
SVC_ROOT="/home/localuser/services"
|
||||
PIHOLE_PW='Gransyan1!' # replicated verbatim from netboot compose
|
||||
|
||||
log() { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
|
||||
warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
|
||||
|
||||
# Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
|
||||
declare -A NODE_TSIP=(
|
||||
[pfv-netinfra-01]="100.70.181.72"
|
||||
[pfv-netinfra-02]="100.93.194.82"
|
||||
)
|
||||
|
||||
on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Verify-only mode
|
||||
#------------------------------------------------------------------------------
|
||||
verify_node() {
|
||||
local node="$1" tsip="${NODE_TSIP[$1]}"
|
||||
log "VERIFY $node (tailscale $tsip)"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
set +e
|
||||
echo "### containers ###"
|
||||
sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
|
||||
echo
|
||||
echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
|
||||
dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
|
||||
echo "### Pi-hole -> Technitium (dig @53 knel.net SOA) ###"
|
||||
dig +time=3 +tries=1 +short @127.0.0.1 -p 53 knel.net SOA 2>&1
|
||||
echo "### Pi-hole -> Technitium (dig @53 pfv-netboot.knel.net A) ###"
|
||||
dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pfv-netboot.knel.net A 2>&1
|
||||
echo "### Pi-hole web (curl :10002) ###"
|
||||
curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
|
||||
echo
|
||||
echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
|
||||
dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
|
||||
echo "### Technitium web (curl :5380) ###"
|
||||
curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
|
||||
echo
|
||||
echo "### NTP service ###"
|
||||
HOST_NTP=""
|
||||
for u in ntpsec ntp chrony openntpd; do
|
||||
systemctl is-active --quiet "\$u" 2>/dev/null && { HOST_NTP="\$u"; break; }
|
||||
done
|
||||
echo "host daemon: \${HOST_NTP:-none}"
|
||||
if [ -n "\$HOST_NTP" ]; then
|
||||
ntpq -c "rv 0 leap,stratum,offset" 2>&1 | head -3
|
||||
else
|
||||
echo "(no host NTP; chrony container:)"
|
||||
sudo docker exec tsys-ntp chronyc -n tracking 2>&1 | head -6
|
||||
fi
|
||||
EOF
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Deploy to one node
|
||||
#------------------------------------------------------------------------------
|
||||
deploy_node() {
|
||||
local node="$1" tsip="${NODE_TSIP[$1]}"
|
||||
log "==== DEPLOY $node (tailscale $tsip) ===="
|
||||
|
||||
# ---- 1. Prepare directories on the target -------------------------------
|
||||
log "$node: create service dirs"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
set -e
|
||||
sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
|
||||
sudo chown -R localuser:localuser $SVC_ROOT
|
||||
EOF
|
||||
|
||||
# ---- 2. Write compose files (as localuser) ------------------------------
|
||||
log "$node: write docker-compose files"
|
||||
on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
|
||||
services:
|
||||
pihole:
|
||||
container_name: pihole
|
||||
image: pihole/pihole:latest
|
||||
hostname: pihole
|
||||
ports:
|
||||
- "53:53/tcp"
|
||||
- "53:53/udp"
|
||||
- "10002:80/tcp"
|
||||
- "10003:443/tcp"
|
||||
environment:
|
||||
TZ: 'America/Chicago'
|
||||
FTLCONF_webserver_api_password: 'Gransyan1!'
|
||||
FTLCONF_dns_listeningMode: 'all'
|
||||
volumes:
|
||||
- './etc-pihole:/etc/pihole'
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
restart: always
|
||||
networks:
|
||||
- default
|
||||
- dnsnet
|
||||
networks:
|
||||
dnsnet:
|
||||
external: true
|
||||
YAML
|
||||
|
||||
on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
|
||||
services:
|
||||
ntp:
|
||||
image: dockurr/chrony
|
||||
container_name: tsys-ntp
|
||||
environment:
|
||||
NTP_SERVERS: "pool.ntp.org"
|
||||
ports:
|
||||
- "$tsip:123:123/udp"
|
||||
restart: always
|
||||
YAML
|
||||
|
||||
on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
|
||||
services:
|
||||
technitium:
|
||||
image: technitium/dns-server
|
||||
container_name: tsys-dns
|
||||
ports:
|
||||
- "5300:53/tcp"
|
||||
- "5300:53/udp"
|
||||
- "5380:5380/tcp"
|
||||
- "53443:53443/tcp"
|
||||
volumes:
|
||||
- './config:/etc/dns'
|
||||
restart: always
|
||||
networks:
|
||||
default:
|
||||
dnsnet:
|
||||
ipv4_address: 10.53.0.53
|
||||
networks:
|
||||
dnsnet:
|
||||
external: true
|
||||
YAML
|
||||
|
||||
# ---- 3. Relay Pi-hole config from netboot -> target ---------------------
|
||||
log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
|
||||
if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
|
||||
log "$node: Pi-hole config already present; skipping copy"
|
||||
else
|
||||
on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
|
||||
ssh -o StrictHostKeyChecking=no "$NETBOOT" \
|
||||
"sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
|
||||
--exclude='etc-pihole/listsCache' \
|
||||
--exclude='etc-pihole/gravity_backups' \
|
||||
--exclude='etc-pihole/config_backups' \
|
||||
etc-pihole" \
|
||||
| on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
|
||||
fi
|
||||
|
||||
# ---- 4. Relay Technitium config from netboot orphaned volume -----------
|
||||
log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
|
||||
if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
|
||||
log "$node: Technitium config already present; skipping copy"
|
||||
else
|
||||
on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
|
||||
ssh -o StrictHostKeyChecking=no "$NETBOOT" \
|
||||
"sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
|
||||
| on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
|
||||
fi
|
||||
|
||||
# ---- 5. Adapt copied config: interface + repoint knel.net to local Technitium
|
||||
log "$node: adapt Pi-hole pihole.toml (interface + revServer -> local Technitium)"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set -e
|
||||
IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
|
||||
IFACE=${IFACE:-ens18}
|
||||
TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
|
||||
if sudo test -f "$TOML"; then
|
||||
sudo sed -i "s|^ interface = .*| interface = \"$IFACE\" ### ADAPTED from eth0 on clone|" "$TOML"
|
||||
echo "set interface=$IFACE"
|
||||
# Repoint knel.net conditional forward from netboot's upstream (192.168.3.16)
|
||||
# to the LOCAL Technitium container at its fixed dnsnet IP 10.53.0.53.
|
||||
# Subnet 100.64.0.0/10 = Tailscale CGNAT range (covers all Tailscale reverse zones).
|
||||
if sudo grep -q 'revServers' "$TOML"; then
|
||||
sudo sed -i 's|"true,[0-9./]*,192\.168\.3\.16,knel\.net"|"true,100.64.0.0/10,10.53.0.53,knel.net"|' "$TOML"
|
||||
echo "revServer repointed to 10.53.0.53 (local Technitium)"
|
||||
else
|
||||
echo "(revServers not found; FTL will use defaults)"
|
||||
fi
|
||||
else
|
||||
echo "(pihole.toml not present; FTL will create it on first run)"
|
||||
fi
|
||||
EOF
|
||||
|
||||
# ---- 5b. Create shared Docker network for Pi-hole <-> Technitium ----------
|
||||
log "$node: create dnsnet shared Docker network (10.53.0.0/24)"
|
||||
on_node "$node" "sudo docker network create --subnet 10.53.0.0/24 dnsnet 2>/dev/null || true"
|
||||
|
||||
# ---- 6. Pull images -----------------------------------------------------
|
||||
log "$node: docker compose pull (pihole, ntp, technitium)"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
for c in pihole ntp technitium; do
|
||||
sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
|
||||
done
|
||||
EOF
|
||||
|
||||
# ---- 6a. Pi-hole up -----------------------------------------------------
|
||||
log "$node: bring up Pi-hole"
|
||||
on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
|
||||
|
||||
# ---- 6b. NTP -- only deploy the chrony container if nothing already ----
|
||||
# serves UDP/123 on the host. The targets already run a bare-metal ntpsec
|
||||
# daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
|
||||
# netboot's own bare-metal ntpsec. netboot additionally runs a chrony
|
||||
# container on its tailscale IP, but that only works there because ntpsec
|
||||
# there does not pre-bind the specific tailscale-IP socket. On these targets
|
||||
# ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
|
||||
# redundant anyway. We therefore keep the host ntpsec as the NTP service.
|
||||
log "$node: NTP -- detect host NTP service"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set +e
|
||||
HOST_NTP=""
|
||||
for u in ntpsec ntp chrony openntpd; do
|
||||
if systemctl is-active --quiet "$u" 2>/dev/null; then HOST_NTP="$u"; break; fi
|
||||
done
|
||||
if [ -n "$HOST_NTP" ]; then
|
||||
echo "Host NTP daemon '$HOST_NTP' is active -- it serves NTP on all local"
|
||||
echo "addresses (incl. the Tailscale IP). This is the same daemon family as"
|
||||
echo "netboot's bare-metal ntpsec; the netboot chrony container is redundant"
|
||||
echo "here and CANNOT bind the Tailscale IP (the host daemon already owns it)."
|
||||
echo "-> Keeping host NTP. Removing any stale chrony container (tsys-ntp)."
|
||||
sudo docker rm -f tsys-ntp 2>/dev/null && echo " (removed tsys-ntp)" || echo " (no tsys-ntp to remove)"
|
||||
echo " host peers:"; ntpq -pn 2>/dev/null | head -12 || true
|
||||
else
|
||||
echo "No host NTP daemon active; starting chrony container."
|
||||
sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
|
||||
fi
|
||||
EOF
|
||||
|
||||
# ---- 6c. Technitium up --------------------------------------------------
|
||||
log "$node: bring up Technitium"
|
||||
on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
|
||||
|
||||
# ---- 7. Wait for Pi-hole health -----------------------------------------
|
||||
log "$node: wait for Pi-hole to become healthy"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
for i in $(seq 1 30); do
|
||||
st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
|
||||
echo " pihole health: $st"
|
||||
[ "$st" = "healthy" ] && break
|
||||
sleep 4
|
||||
done
|
||||
EOF
|
||||
|
||||
# ---- 8. Technitium defensive fallback -----------------------------------
|
||||
# If the copied (binary) config from an older Technitium version makes the
|
||||
# new container crash, move it aside and let Technitium start fresh so the
|
||||
# service is at least up (admin reachable) rather than crash-looping.
|
||||
log "$node: check Technitium health (fallback to fresh config if crash)"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set +e
|
||||
sleep 8
|
||||
rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
|
||||
running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
|
||||
if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
|
||||
echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
|
||||
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
|
||||
sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
|
||||
sudo mkdir -p /home/localuser/services/technitium/config
|
||||
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
|
||||
echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
|
||||
else
|
||||
echo "Technitium OK (running=$running restarts=$rst)"
|
||||
fi
|
||||
EOF
|
||||
|
||||
# ---- 9. Final status -----------------------------------------------------
|
||||
log "$node: final container status"
|
||||
on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Main
|
||||
#------------------------------------------------------------------------------
|
||||
main() {
|
||||
local mode="${2:-deploy}"
|
||||
if [ "${1:-all}" = "all" ]; then
|
||||
targets=(pfv-netinfra-01 pfv-netinfra-02)
|
||||
else
|
||||
targets=("$1")
|
||||
fi
|
||||
for t in "${targets[@]}"; do
|
||||
: "${NODE_TSIP[$t]:?unknown node $t}"
|
||||
if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
|
||||
done
|
||||
log "DONE"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
# check-pkgs.sh - verify package install state.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
for host in pfv-tsys6 pfv-tsys7; do
|
||||
echo "=== $host ==="
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" '
|
||||
for p in sysstat jq numactl nvme mtr-tiny dig bmon tcpdump; do
|
||||
if command -v "$p" >/dev/null 2>&1; then
|
||||
echo " ✓ $p"
|
||||
else
|
||||
echo " ✗ $p"
|
||||
fi
|
||||
done
|
||||
# sysstat config
|
||||
echo " sysstat service:"
|
||||
systemctl list-unit-files 2>/dev/null | grep -i sysstat | sed "s/^/ /"
|
||||
echo " sysstat enabled in /etc/default:"
|
||||
if [ -r /etc/default/sysstat ]; then
|
||||
grep ENABLED /etc/default/sysstat | sed "s/^/ /"
|
||||
else
|
||||
echo " no /etc/default/sysstat"
|
||||
fi
|
||||
# on Debian trixie, sysstat uses a different path
|
||||
ls /etc/cron.d/sysstat* 2>/dev/null | sed "s/^/ found: /"
|
||||
'
|
||||
echo ""
|
||||
done
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
# check-repos-and-reboot.sh - checks reboot-required + Proxmox repo config on all hosts.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
|
||||
for host in "${HOSTS[@]}"; do
|
||||
echo "================================================================"
|
||||
echo "[$host]"
|
||||
echo "================================================================"
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " UNREACHABLE"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "--- /var/run/reboot-required ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'if [ -r /var/run/reboot-required ]; then echo "REBOOT REQUIRED"; cat /var/run/reboot-required 2>/dev/null; if [ -r /var/run/reboot-required.pkgs ]; then echo "Packages triggering:"; cat /var/run/reboot-required.pkgs; fi; else echo "(no reboot required marker)"; fi'
|
||||
|
||||
echo ""
|
||||
echo "--- Running kernel vs installed kernel ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'echo "running: $(uname -r)"; echo "installed:"; dpkg -l | grep -E "pve-kernel-[0-9]" | awk "{print \" \"\$2\" \"\$3}" | tail -5'
|
||||
|
||||
echo ""
|
||||
echo "--- Proxmox repositories (apt sources) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rh "pve\|proxmox" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | grep -v "^#" | sed "s/^/ /"'
|
||||
|
||||
echo ""
|
||||
echo "--- Enterprise repo status (should be commented or absent if no subscription) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -l "pve-enterprise" /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null | while read f; do echo " File: $f"; grep -n "pve-enterprise" "$f" | sed "s/^/ /"; done'
|
||||
|
||||
echo ""
|
||||
echo "--- no-subscription repo presence ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rl "pve-no-subscription" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | while read f; do echo " File: $f"; grep -n "pve-no-subscription" "$f" | sed "s/^/ /"; done'
|
||||
|
||||
echo ""
|
||||
echo "--- Recently updated packages (last 24h, kernel-related) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -E "pve-kernel|proxmox|pve-qemu|zfs" /var/log/dpkg.log 2>/dev/null | grep "$(date +%Y-%m-%d)\|$(date -d yesterday +%Y-%m-%d)" | tail -15 || echo "(none in dpkg.log)"'
|
||||
|
||||
echo ""
|
||||
done
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/bin/bash
|
||||
# deploy-and-fix.sh - uploads fix script, runs it, starts VMs, verifies.
|
||||
set -uo pipefail
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
|
||||
|
||||
echo "=== Uploading fix script to $HOST ==="
|
||||
scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
|
||||
|
||||
echo "=== Running fix ==="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/fix-bond-nfs.sh && bash /root/fix-bond-nfs.sh' 2>&1
|
||||
|
||||
echo ""
|
||||
echo "=== Starting VMs ==="
|
||||
for vmid in $(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"'); do
|
||||
status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
|
||||
if [ "$status" != "running" ]; then
|
||||
echo " Starting VM $vmid..."
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/ /'
|
||||
else
|
||||
echo " VM $vmid already running"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Waiting 20s for VMs to boot..."
|
||||
sleep 20
|
||||
|
||||
echo ""
|
||||
echo "=== FULL VERIFICATION ==="
|
||||
echo ""
|
||||
echo "--- VMs ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
|
||||
echo ""
|
||||
echo "--- NFS mounts ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
|
||||
echo ""
|
||||
echo "--- NFS TCP connections ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
|
||||
echo " Count:"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
|
||||
echo ""
|
||||
echo "--- bond0 hash policy ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -6'
|
||||
echo ""
|
||||
echo "--- Summary ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control)"'
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "swappiness: $(sysctl -n vm.swappiness)"'
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null)"'
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# deploy-check.sh
|
||||
#
|
||||
# Deploys scripts/check.sh to each reachable Proxmox host, executes it
|
||||
# read-only, and pulls the resulting log back to returned-logs/.
|
||||
#
|
||||
# EXPLICITLY SKIPS:
|
||||
# - pfv-tsys2 (off the air per user)
|
||||
# - pfv-tsys9 (off the air per user; also not in original inventory)
|
||||
#
|
||||
# Safety features:
|
||||
# - BatchMode=yes : never hang on a password prompt
|
||||
# - ConnectTimeout=8 : fail fast on dead hosts
|
||||
# - per-host try/skip : one bad host never aborts the run
|
||||
# - ServerAliveInterval : detect hung connections
|
||||
# - read-only script : check.sh modifies nothing on the target
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt"
|
||||
CHECK_SH="$SCRIPT_DIR/scripts/check.sh"
|
||||
LOG_DIR="$SCRIPT_DIR/returned-logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# ONLY the hosts the user told us are alive.
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
|
||||
|
||||
# Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
|
||||
SSH_OPTS=(-o BatchMode=yes
|
||||
-o ConnectTimeout=8
|
||||
-o ServerAliveInterval=10
|
||||
-o ServerAliveCountMax=3
|
||||
-o StrictHostKeyChecking=accept-new)
|
||||
|
||||
log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
|
||||
|
||||
if [ ! -r "$CHECK_SH" ]; then
|
||||
echo "FATAL: $CHECK_SH not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Sanity-check shellcheck clean before shipping (best-effort, non-blocking)
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
log "pre-flight: shellcheck on check.sh"
|
||||
if ! docker run --rm -v "$SCRIPT_DIR:/mnt" -w /mnt \
|
||||
koalaman/shellcheck:stable --severity=style --format=gcc scripts/check.sh \
|
||||
>"$LOG_DIR/_shellcheck.preflight.txt" 2>&1; then
|
||||
log "WARNING: shellcheck reported issues — see _shellcheck.preflight.txt"
|
||||
log " aborting deploy to avoid shipping a broken script"
|
||||
exit 1
|
||||
fi
|
||||
log "pre-flight: shellcheck clean"
|
||||
fi
|
||||
|
||||
summary_pass=()
|
||||
summary_fail=()
|
||||
declare -A HOST_PID # host -> background pid
|
||||
declare -A HOST_MARKER # host -> per-host marker file
|
||||
|
||||
# Per-host worker — runs in background, one per host, all in parallel.
|
||||
# Writes status into a marker file consumed by the parent.
|
||||
worker() {
|
||||
local host="$1"
|
||||
local marker="$LOG_DIR/_marker.$host"
|
||||
: > "$marker" # truncate
|
||||
echo "running" >> "$marker"
|
||||
|
||||
local short=""
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "fail unreachable" >> "$marker"
|
||||
return
|
||||
fi
|
||||
if ! scp "${SSH_OPTS[@]}" "$CHECK_SH" "root@$host:/root/check.sh" >/dev/null 2>&1; then
|
||||
echo "fail scp-upload-failed" >> "$marker"
|
||||
return
|
||||
fi
|
||||
local remote_size
|
||||
remote_size=$(ssh "${SSH_OPTS[@]}" "root@$host" 'wc -c < /root/check.sh' 2>/dev/null || echo 0)
|
||||
if [ "${remote_size:-0}" -lt 1000 ]; then
|
||||
echo "fail upload-corrupt" >> "$marker"
|
||||
return
|
||||
fi
|
||||
local remote_stdout
|
||||
remote_stdout=$(ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
'chmod +x /root/check.sh && bash /root/check.sh' 2>&1)
|
||||
local rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "fail check-exit-$rc" >> "$marker"
|
||||
# don't return - still try to pull whatever log got produced
|
||||
fi
|
||||
short=$(printf '%s\n' "$remote_stdout" | grep -oE 'Wrote: /root/[a-zA-Z0-9_-]+\.log' | head -n1 | awk '{print $2}')
|
||||
if [ -z "$short" ]; then
|
||||
short=$(ssh "${SSH_OPTS[@]}" "root@$host" 'echo "/root/$(hostname -s).log"' 2>/dev/null)
|
||||
fi
|
||||
if [ -z "$short" ]; then
|
||||
echo "fail no-log-path" >> "$marker"
|
||||
return
|
||||
fi
|
||||
if ! scp "${SSH_OPTS[@]}" "root@$host:$short" "$LOG_DIR/" >/dev/null 2>&1; then
|
||||
echo "fail scp-download-failed" >> "$marker"
|
||||
return
|
||||
fi
|
||||
local local_name local_path
|
||||
local_name="$(basename "$short")"
|
||||
local_path="$LOG_DIR/$local_name"
|
||||
if [ ! -s "$local_path" ]; then
|
||||
echo "fail local-empty" >> "$marker"
|
||||
return
|
||||
fi
|
||||
echo "ok $local_name $(wc -c < "$local_path") $(wc -l < "$local_path")" >> "$marker"
|
||||
}
|
||||
|
||||
# ---- launch all workers in parallel --------------------------------------
|
||||
log "launching ${#HOSTS[@]} hosts in parallel..."
|
||||
for host in "${HOSTS[@]}"; do
|
||||
rm -f "$LOG_DIR/_marker.$host"
|
||||
worker "$host" &
|
||||
HOST_PID[$host]=$!
|
||||
HOST_MARKER[$host]="$LOG_DIR/_marker.$host"
|
||||
log " launched $host (pid ${HOST_PID[$host]})"
|
||||
done
|
||||
|
||||
# ---- wait for all, with periodic progress --------------------------------
|
||||
remaining=("${HOSTS[@]}")
|
||||
while [ "${#remaining[@]}" -gt 0 ]; do
|
||||
sleep 10
|
||||
new_remaining=()
|
||||
for host in "${remaining[@]}"; do
|
||||
if ! kill -0 "${HOST_PID[$host]}" 2>/dev/null; then
|
||||
# process finished
|
||||
wait "${HOST_PID[$host]}" 2>/dev/null || true
|
||||
marker="${HOST_MARKER[$host]}"
|
||||
if [ -r "$marker" ]; then
|
||||
status_line="$(tail -n1 "$marker")"
|
||||
log "[$host] done: $status_line"
|
||||
case "$status_line" in
|
||||
ok*) summary_pass+=("$host:$status_line") ;;
|
||||
fail*) summary_fail+=("$host:$status_line") ;;
|
||||
*) summary_fail+=("$host:unknown") ;;
|
||||
esac
|
||||
else
|
||||
log "[$host] done but marker missing"
|
||||
summary_fail+=("$host:no-marker")
|
||||
fi
|
||||
else
|
||||
new_remaining+=("$host")
|
||||
fi
|
||||
done
|
||||
remaining=("${new_remaining[@]:-}")
|
||||
if [ "${#remaining[@]}" -gt 0 ]; then
|
||||
log "still running: ${remaining[*]} (${#remaining[@]} hosts)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Final summary
|
||||
log "============================================================"
|
||||
log "DEPLOY SUMMARY"
|
||||
log "============================================================"
|
||||
log "Passed (${#summary_pass[@]}):"
|
||||
for p in "${summary_pass[@]:-}"; do [ -n "$p" ] && log " ✓ $p"; done
|
||||
log "Failed (${#summary_fail[@]}):"
|
||||
for f in "${summary_fail[@]:-}"; do [ -n "$f" ] && log " ✗ $f"; done
|
||||
log ""
|
||||
log "Contents of $LOG_DIR:"
|
||||
ls -la "$LOG_DIR"
|
||||
|
||||
# Clean up marker files
|
||||
rm -f "$LOG_DIR"/_marker.* 2>/dev/null
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/bin/bash
|
||||
# deploy-tuning.sh - copies apply-tunings.sh to target hosts and runs it.
|
||||
# Usage: bash deploy-tuning.sh [--no-nfs] [--apply] <host> [host...]
|
||||
# Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section.
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT="/home/reachableceo/projects/perfopt/scripts/apply-tunings.sh"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
MODE=""
|
||||
EXTRA_FLAGS=""
|
||||
|
||||
HOSTS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--apply) MODE="--apply" ;;
|
||||
--dry-run) MODE="" ;;
|
||||
--no-nfs) EXTRA_FLAGS="--no-nfs" ;;
|
||||
*) HOSTS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "${#HOSTS[@]}" -eq 0 ]; then
|
||||
echo "Usage: $0 <host> [host...] [--apply]"
|
||||
echo "Default: dry-run. Pass --apply to commit."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -r "$SCRIPT" ]; then
|
||||
echo "FATAL: $SCRIPT not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for host in "${HOSTS[@]}"; do
|
||||
echo "================================================================"
|
||||
echo "[$host] deploying apply-tunings.sh (mode: ${MODE:-dry-run})"
|
||||
echo "================================================================"
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "[$host] SKIP: unreachable"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "[$host] uploading..."
|
||||
if ! scp "${SSH_OPTS[@]}" "$SCRIPT" "root@$host:/root/apply-tunings.sh" >/dev/null 2>&1; then
|
||||
echo "[$host] SKIP: scp failed"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "[$host] running (output below)..."
|
||||
echo "----------------------------------------------------------------"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" "chmod +x /root/apply-tunings.sh && bash /root/apply-tunings.sh $MODE $EXTRA_FLAGS" 2>&1
|
||||
rc=$?
|
||||
echo "----------------------------------------------------------------"
|
||||
echo "[$host] exit code: $rc"
|
||||
echo ""
|
||||
done
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
# diag.sh - diagnostic commands run on a host via SSH wrapper.
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
echo "===== 1. storage.cfg NFS stanzas (exact content) ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'grep -A 8 "^nfs: D2" /etc/pve/storage.cfg'
|
||||
|
||||
echo ""
|
||||
echo "===== 2. Try manual NFS mount with nconnect=4 ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
|
||||
|
||||
echo ""
|
||||
echo "===== 3. Try manual NFS mount WITHOUT nconnect ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
|
||||
|
||||
echo ""
|
||||
echo "===== 4. NFS kernel version / module ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/fs/nfsfs/version 2>/dev/null; echo "---"; modinfo nfs 2>/dev/null | grep -E "^(filename|version|description)" | head -5'
|
||||
|
||||
echo ""
|
||||
echo "===== 5. mount.nfs version ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount.nfs --version 2>&1; echo "---"; dpkg -l nfs-common 2>/dev/null | tail -2'
|
||||
|
||||
echo ""
|
||||
echo "===== 6. /etc/network/interfaces bond0 stanza (exact bytes) ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sed -n "/^auto bond0/,/^$/p" /etc/network/interfaces | cat -A'
|
||||
|
||||
echo ""
|
||||
echo "===== 7. Current bond0 running hash policy ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -5'
|
||||
|
||||
echo ""
|
||||
echo "===== 8. xmit_hash_policy sysfs file ====="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; echo "---"; ls /sys/class/net/bond0/bonding/ 2>/dev/null'
|
||||
Executable
+128
@@ -0,0 +1,128 @@
|
||||
#!/bin/bash
|
||||
# finish-host.sh - applies ALL remaining changes to a host and verifies.
|
||||
#
|
||||
# Steps:
|
||||
# 1. Start all VMs (triggers NFS lazy-mount)
|
||||
# 2. Wait for NFS mounts to appear
|
||||
# 3. Verify NFS nconnect=4 + noatime
|
||||
# 4. Apply bond0 xmit_hash_policy=layer3+4
|
||||
# 5. Full end-to-end verification
|
||||
#
|
||||
# Usage: bash finish-host.sh <host> [--apply]
|
||||
# Default is dry-run (starts VMs + shows what bond change would do, but doesn't edit interfaces)
|
||||
set -uo pipefail
|
||||
|
||||
HOST="${1:-}"
|
||||
MODE="${2:-dryrun}"
|
||||
[ "$MODE" = "--apply" ] && MODE="apply" || MODE="dryrun"
|
||||
|
||||
if [ -z "$HOST" ]; then
|
||||
echo "Usage: $0 <host> [--apply]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " finish-host.sh — $HOST (mode: $MODE)"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
# =========================================================================
|
||||
# STEP 1: Start all VMs
|
||||
# =========================================================================
|
||||
echo "=== STEP 1: Start all VMs on $HOST ==="
|
||||
# Get list of all VMs (not just stopped — start is idempotent)
|
||||
vm_list=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"')
|
||||
for vmid in $vm_list; do
|
||||
status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
|
||||
if [ "$status" != "running" ]; then
|
||||
echo " Starting VM $vmid..."
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/ /'
|
||||
else
|
||||
echo " VM $vmid already running"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo " Waiting 15s for VMs to boot and trigger NFS mounts..."
|
||||
sleep 15
|
||||
|
||||
echo ""
|
||||
echo "--- VM status after start ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
|
||||
|
||||
# =========================================================================
|
||||
# STEP 2: Verify NFS mounts came back with nconnect=4
|
||||
# =========================================================================
|
||||
echo ""
|
||||
echo "=== STEP 2: Verify NFS mounts with nconnect=4 ==="
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null' | head -30
|
||||
|
||||
echo ""
|
||||
echo "--- NFS TCP connections to :2049 ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
|
||||
conn_count=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
|
||||
echo " Total NFS TCP connections: $conn_count"
|
||||
|
||||
# =========================================================================
|
||||
# STEP 3: Apply bond0 hash policy
|
||||
# =========================================================================
|
||||
echo ""
|
||||
echo "=== STEP 3: Apply bond0 xmit_hash_policy=layer3+4 (mode: $MODE) ==="
|
||||
|
||||
# Upload the bond hash script
|
||||
scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/apply-bond-hash.sh" "root@$HOST:/root/apply-bond-hash.sh" >/dev/null 2>&1
|
||||
|
||||
if [ "$MODE" = "apply" ]; then
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh --apply' 2>&1
|
||||
else
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh' 2>&1
|
||||
fi
|
||||
|
||||
# =========================================================================
|
||||
# STEP 4: Full verification
|
||||
# =========================================================================
|
||||
echo ""
|
||||
echo "=== STEP 4: Full end-to-end verification ==="
|
||||
echo ""
|
||||
echo "--- Uptime ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
|
||||
echo ""
|
||||
echo "--- CPU governor ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
|
||||
echo ""
|
||||
echo "--- vm.swappiness ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
|
||||
echo ""
|
||||
echo "--- TCP BBR ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
|
||||
echo ""
|
||||
echo "--- tuned profile ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
|
||||
echo ""
|
||||
echo "--- bond0 hash policy + LACP state ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 2>/dev/null | head -25'
|
||||
echo ""
|
||||
echo "--- NFS mount options (first 3 mounts) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
|
||||
echo ""
|
||||
echo "--- NFS TCP connections (expect 4 per server × 2 servers = 8) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
|
||||
nfs_conns=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
|
||||
echo " Count: $nfs_conns"
|
||||
echo ""
|
||||
echo "--- VMs running ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
|
||||
echo ""
|
||||
echo "--- Failed services ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
|
||||
echo ""
|
||||
echo "--- Network interfaces (speed/duplex/mtu) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'for ifc in bond0 nic0 nic1 nic2 vmbr0 datanet; do [ -d "/sys/class/net/$ifc" ] && printf "%-12s speed=%-8s duplex=%-8s mtu=%s\n" "$ifc" "$(cat /sys/class/net/$ifc/speed 2>/dev/null)" "$(cat /sys/class/net/$ifc/duplex 2>/dev/null)" "$(cat /sys/class/net/$ifc/mtu 2>/dev/null)"; done'
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " COMPLETE — $HOST"
|
||||
echo "==================================================================="
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# install-utils-v2.sh - retry install without nstat package.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
PKGS="sysstat jq numactl nvme-cli mtr-tiny dnsutils bmon"
|
||||
|
||||
for host in pfv-tsys6 pfv-tsys7; do
|
||||
echo "=== [$host] installing: $PKGS ==="
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
"DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>&1 | tail -2 && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y $PKGS 2>&1 | tail -10"
|
||||
|
||||
# sysstat enable (path varies by Debian version)
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" '
|
||||
if [ -r /etc/default/sysstat ]; then
|
||||
sed -i "s/^ENABLED=.*/ENABLED=\"true\"/" /etc/default/sysstat
|
||||
systemctl enable --now sysstat 2>/dev/null
|
||||
grep ENABLED /etc/default/sysstat
|
||||
else
|
||||
# Newer Debian (trixie) — sysstat cron/service auto-enabled
|
||||
systemctl enable --now sysstat 2>/dev/null || echo "(sysstat auto via cron)"
|
||||
fi
|
||||
'
|
||||
echo ""
|
||||
done
|
||||
|
||||
# Verify
|
||||
for host in pfv-tsys6 pfv-tsys7; do
|
||||
echo "=== [$host] verification ==="
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" '
|
||||
for p in sysstat jq numactl nvme mtr-tiny dig bmon; do
|
||||
command -v "$p" >/dev/null 2>&1 && echo " ✓ $p" || echo " ✗ $p"
|
||||
done
|
||||
'
|
||||
echo ""
|
||||
done
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
# install-utils.sh - installs useful observability packages on a host.
|
||||
# These are all small, dependency-light, and read-only at runtime.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
HOSTS=(pfv-tsys6 pfv-tsys7)
|
||||
|
||||
# Packages to install, with rationale
|
||||
PKGS=(
|
||||
sysstat # sar, iostat, mpstat, pidstat - the missing observability suite
|
||||
jq # JSON parsing for pvesh/scripts
|
||||
numactl # NUMA topology/controls for the dual-socket hosts
|
||||
nvme-cli # NVMe health (for when NVMe shows up)
|
||||
tcpdump # packet capture for network debugging
|
||||
mtr-tiny # traceroute on steroids
|
||||
nstat # kernel SNMP stats (already partly there)
|
||||
dnsutils # dig, nslookup, host
|
||||
bmon # bandwidth monitor ( curses, real-time)
|
||||
)
|
||||
|
||||
for host in "${HOSTS[@]}"; do
|
||||
echo "================================================================"
|
||||
echo "[$host] installing observability packages"
|
||||
echo "================================================================"
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " UNREACHABLE"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check which are missing
|
||||
missing=""
|
||||
for pkg in "${PKGS[@]}"; do
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" "dpkg -s $pkg 2>/dev/null | grep -q 'Status: install ok installed'" 2>/dev/null; then
|
||||
missing="$missing $pkg"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$missing" ]; then
|
||||
echo " All packages already installed."
|
||||
continue
|
||||
fi
|
||||
|
||||
echo " Installing:$missing"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
"DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq$missing 2>&1 | tail -5"
|
||||
echo " Done."
|
||||
echo ""
|
||||
done
|
||||
|
||||
# Enable sysstat data collection (sar) — off by default on Debian
|
||||
for host in "${HOSTS[@]}"; do
|
||||
echo "[$host] enabling sysstat/sar data collection..."
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
"sed -i 's/^ENABLED=\"false\"/ENABLED=\"true\"/' /etc/default/sysstat 2>/dev/null; \
|
||||
systemctl enable --now sysstat 2>&1 | tail -2; \
|
||||
grep ENABLED /etc/default/sysstat"
|
||||
done
|
||||
Executable
+293
@@ -0,0 +1,293 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# iperf-full-matrix.sh
|
||||
#
|
||||
# Two test suites:
|
||||
# A. Management network (vmbr0 / VLAN1): all-pairs single-stream TCP, 10s
|
||||
# B. Storage network (VLAN1000): tsys6+tsys7 → tsys4+tsys5, stress test
|
||||
#
|
||||
# Output: returned-logs/iperf/
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
|
||||
# Storage IPs (known, static on VLAN1000)
|
||||
declare -A SIP
|
||||
SIP[pfv-tsys1]="10.100.100.1"
|
||||
SIP[pfv-tsys3]="10.100.100.3"
|
||||
SIP[pfv-tsys4]="10.100.100.4"
|
||||
SIP[pfv-tsys5]="10.100.100.5"
|
||||
SIP[pfv-tsys6]="10.100.100.6"
|
||||
SIP[pfv-tsys7]="10.100.100.7"
|
||||
|
||||
# ===========================================================================
|
||||
# STEP 0: Discover management IPs (vmbr0)
|
||||
# ===========================================================================
|
||||
echo "==================================================================="
|
||||
echo " STEP 0: Discover management network IPs (vmbr0)"
|
||||
echo "==================================================================="
|
||||
declare -A MIP
|
||||
for host in "${HOSTS[@]}"; do
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " [$host] UNREACHABLE"
|
||||
continue
|
||||
fi
|
||||
mip=$(ssh "${SSH_OPTS[@]}" "root@$host" 'ip -o -4 addr show dev vmbr0 2>/dev/null | awk "{print \$4}" | cut -d/ -f1 | head -1')
|
||||
if [ -n "$mip" ]; then
|
||||
MIP[$host]="$mip"
|
||||
echo " [$host] vmbr0 = $mip"
|
||||
else
|
||||
echo " [$host] no vmbr0 IPv4 — skipping"
|
||||
fi
|
||||
done
|
||||
|
||||
# ===========================================================================
|
||||
# STEP 1: Ensure iperf3 installed on all hosts
|
||||
# ===========================================================================
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " STEP 1: Ensure iperf3 installed"
|
||||
echo "==================================================================="
|
||||
for host in "${HOSTS[@]}"; do
|
||||
[ -z "${MIP[$host]:-}" ] && continue
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
|
||||
echo -n " [$host] installing iperf3... "
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
|
||||
else
|
||||
echo " [$host] iperf3 already installed"
|
||||
fi
|
||||
done
|
||||
|
||||
# Helper: start iperf3 server in one-shot mode bound to a specific IP
|
||||
start_server() {
|
||||
local host="$1" ip="$2"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" \
|
||||
"pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${ip} >/dev/null 2>&1 &" 2>/dev/null
|
||||
sleep 1
|
||||
}
|
||||
|
||||
# Helper: run iperf3 client test, save output, extract result
|
||||
run_test() {
|
||||
local client="$1" server="$2" sip="$3" cip="$4" label="$5" logfile="$6"
|
||||
shift 6
|
||||
local extra="$*"
|
||||
|
||||
echo -n " [$label] ... "
|
||||
{
|
||||
echo "=== iperf3: $label ==="
|
||||
echo "Client: $client ($cip) → Server: $server ($sip)"
|
||||
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "Args: $extra"
|
||||
echo ""
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${sip} -B ${cip} ${extra}" 2>&1
|
||||
echo ""
|
||||
echo "=== END ==="
|
||||
} > "$logfile" 2>&1
|
||||
|
||||
# Extract result
|
||||
sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
|
||||
if [ -n "$sum" ]; then
|
||||
bitrate=$(echo "$sum" | awk '{print $6, $7}')
|
||||
retrans=$(echo "$sum" | awk '{print $8}')
|
||||
else
|
||||
single=$(grep 'sender$' "$logfile" | tail -1)
|
||||
bitrate=$(echo "$single" | awk '{print $7, $8}')
|
||||
retrans=$(echo "$single" | awk '{print $9}')
|
||||
fi
|
||||
echo "${bitrate:-?} (retrans: ${retrans:-?})"
|
||||
}
|
||||
|
||||
# ===========================================================================
|
||||
# SUITE A: Management network (vmbr0) — all pairs, single-stream TCP, 10s
|
||||
# ===========================================================================
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " SUITE A: MANAGEMENT NETWORK (vmbr0) — all-pairs, 10s TCP"
|
||||
echo " Expectation: ~940 Mbps for every pair (1 GbE line rate)"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
# Test each unique pair (i < j to avoid duplicates)
|
||||
for ((i=0; i<${#HOSTS[@]}; i++)); do
|
||||
for ((j=i+1; j<${#HOSTS[@]}; j++)); do
|
||||
client="${HOSTS[$i]}"
|
||||
server="${HOSTS[$j]}"
|
||||
[ -z "${MIP[$client]:-}" ] && continue
|
||||
[ -z "${MIP[$server]:-}" ] && continue
|
||||
|
||||
label="mgmt: ${client}→${server}"
|
||||
logfile="$LOG_DIR/mgmt-${client}-to-${server}.log"
|
||||
|
||||
start_server "$server" "${MIP[$server]}"
|
||||
run_test "$client" "$server" "${MIP[$server]}" "${MIP[$client]}" \
|
||||
"$label" "$logfile" "-t 10 -P 1"
|
||||
done
|
||||
done
|
||||
|
||||
# ===========================================================================
|
||||
# SUITE B: Storage network (VLAN1000) — stress test the cross-rack LACP
|
||||
# ===========================================================================
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " SUITE B: STORAGE NETWORK (VLAN1000) — stress test cross-rack link"
|
||||
echo " tsys6 + tsys7 (Rack 3) → tsys4 + tsys5 (Rack 5)"
|
||||
echo " Expectation: limited by tsys4 USB dongle + tsys5 broken bond"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
# --- B.1: Individual tests (one client → one server at a time) ---
|
||||
echo "--- B.1: Individual tests (sequential) ---"
|
||||
echo ""
|
||||
|
||||
for client in pfv-tsys6 pfv-tsys7; do
|
||||
for server in pfv-tsys4 pfv-tsys5; do
|
||||
label="stor: ${client}→${server} (8-stream)"
|
||||
logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
|
||||
start_server "$server" "${SIP[$server]}"
|
||||
run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
|
||||
"$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
|
||||
done
|
||||
done
|
||||
|
||||
# --- B.2: Reverse direction (tsys4/5 → tsys6/7) ---
|
||||
echo ""
|
||||
echo "--- B.2: Reverse direction (tsys4/5 → tsys6/7) ---"
|
||||
echo ""
|
||||
|
||||
for client in pfv-tsys4 pfv-tsys5; do
|
||||
for server in pfv-tsys6 pfv-tsys7; do
|
||||
label="stor: ${client}→${server} (8-stream rev)"
|
||||
logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
|
||||
start_server "$server" "${SIP[$server]}"
|
||||
run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
|
||||
"$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
|
||||
done
|
||||
done
|
||||
|
||||
# --- B.3: Simultaneous stress test (4 flows at once) ---
|
||||
echo ""
|
||||
echo "--- B.3: Simultaneous 4-flow stress test ---"
|
||||
echo " tsys6→tsys4 + tsys6→tsys5 + tsys7→tsys4 + tsys7→tsys5"
|
||||
echo " All running in parallel for 30 seconds"
|
||||
echo ""
|
||||
|
||||
# Start 4 iperf3 servers (one-shot mode won't work for parallel; use persistent)
|
||||
for server in pfv-tsys4 pfv-tsys5; do
|
||||
ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -B ${SIP[$server]} >/dev/null 2>&1 &" 2>/dev/null
|
||||
echo " [server started: $server]"
|
||||
done
|
||||
sleep 1
|
||||
|
||||
STRESS_LOG="$LOG_DIR/stor-stress-4flow"
|
||||
mkdir -p "$STRESS_LOG"
|
||||
|
||||
# Launch 4 clients in parallel, each writing to its own log
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys4.log" 2>&1 &
|
||||
PID1=$!
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys5.log" 2>&1 &
|
||||
PID2=$!
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys4.log" 2>&1 &
|
||||
PID3=$!
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys5.log" 2>&1 &
|
||||
PID4=$!
|
||||
|
||||
echo " [4 clients launched, waiting 40s for completion...]"
|
||||
wait $PID1 $PID2 $PID3 $PID4 2>/dev/null
|
||||
echo " [all 4 flows complete]"
|
||||
|
||||
# Kill servers
|
||||
for server in pfv-tsys4 pfv-tsys5; do
|
||||
ssh "${SSH_OPTS[@]}" "root@$server" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
|
||||
# ===========================================================================
|
||||
# SUITE C: All hosts cleanup
|
||||
# ===========================================================================
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " Cleanup: killing iperf3 everywhere"
|
||||
echo "==================================================================="
|
||||
for host in "${HOSTS[@]}"; do
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
|
||||
# ===========================================================================
|
||||
# RESULTS SUMMARY
|
||||
# ===========================================================================
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " RESULTS SUMMARY"
|
||||
echo "==================================================================="
|
||||
|
||||
echo ""
|
||||
echo "===== SUITE A: Management network (vmbr0) ====="
|
||||
echo ""
|
||||
printf "%-40s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
|
||||
printf "%-40s %15s %10s\n" "----" "----------" "-------"
|
||||
for f in "$LOG_DIR"/mgmt-*.log; do
|
||||
[ -r "$f" ] || continue
|
||||
label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
|
||||
single=$(grep 'sender$' "$f" | tail -1)
|
||||
bitrate=$(echo "$single" | awk '{print $7, $8}')
|
||||
retrans=$(echo "$single" | awk '{print $9}')
|
||||
printf "%-40s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "===== SUITE B.1+B.2: Storage network individual ====="
|
||||
echo ""
|
||||
printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
|
||||
printf "%-45s %15s %10s\n" "----" "----------" "-------"
|
||||
for f in "$LOG_DIR"/stor-indiv-*.log; do
|
||||
[ -r "$f" ] || continue
|
||||
label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
|
||||
sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
|
||||
if [ -n "$sum" ]; then
|
||||
bitrate=$(echo "$sum" | awk '{print $6, $7}')
|
||||
retrans=$(echo "$sum" | awk '{print $8}')
|
||||
else
|
||||
single=$(grep 'sender$' "$f" | tail -1)
|
||||
bitrate=$(echo "$single" | awk '{print $7, $8}')
|
||||
retrans=$(echo "$single" | awk '{print $9}')
|
||||
fi
|
||||
printf "%-45s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "===== SUITE B.3: Simultaneous 4-flow stress test ====="
|
||||
echo ""
|
||||
printf "%-30s %15s %10s\n" "FLOW" "THROUGHPUT" "RETRANS"
|
||||
printf "%-30s %15s %10s\n" "----" "----------" "-------"
|
||||
total_mbps=0
|
||||
for f in "$STRESS_LOG"/*.log; do
|
||||
[ -r "$f" ] || continue
|
||||
flow=$(basename "$f" .log)
|
||||
sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
|
||||
if [ -n "$sum" ]; then
|
||||
bitrate=$(echo "$sum" | awk '{print $6, $7}')
|
||||
retrans=$(echo "$sum" | awk '{print $8}')
|
||||
mbps=$(echo "$sum" | awk '{print $6}')
|
||||
total_mbps=$(awk "BEGIN{print $total_mbps + $mbps}")
|
||||
else
|
||||
bitrate="?"
|
||||
retrans="-"
|
||||
fi
|
||||
printf "%-30s %15s %10s\n" "$flow" "$bitrate" "${retrans:--}"
|
||||
done
|
||||
printf "%-30s %15s\n" "AGGREGATE (all 4 flows)" "${total_mbps} Mbits/sec"
|
||||
|
||||
echo ""
|
||||
echo "===== CONTEXT ====="
|
||||
echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
|
||||
echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
|
||||
echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
|
||||
echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
|
||||
echo ""
|
||||
echo "All logs in: $LOG_DIR/"
|
||||
Executable
+184
@@ -0,0 +1,184 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# iperf-storage-tests.sh
|
||||
#
|
||||
# Installs iperf3 on all online hosts, then runs a matrix of storage-network
|
||||
# throughput tests. Saves all output to returned-logs/iperf/.
|
||||
#
|
||||
# Test matrix (all over VLAN1000 storage network, 10.100.100.0/24):
|
||||
# 1. tsys7 → tsys4 (USB cdc_ncm NIC) — the smoking gun
|
||||
# 2. tsys7 → tsys5 (bond0, 1 active slave) — PCI NIC comparison
|
||||
# 3. tsys7 → tsys6 (bond0, 2 active slaves) — working LACP baseline
|
||||
# 4. Reverse: tsys4 → tsys7 (USB NIC TX direction)
|
||||
# 5. Reverse: tsys5 → tsys7
|
||||
#
|
||||
# Each test: TCP 8-stream 30s forward + reverse + UDP saturation.
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
ALL_HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
|
||||
# Storage network IPs
|
||||
declare -A SIP
|
||||
SIP[pfv-tsys1]="10.100.100.1"
|
||||
SIP[pfv-tsys3]="10.100.100.3"
|
||||
SIP[pfv-tsys4]="10.100.100.4"
|
||||
SIP[pfv-tsys5]="10.100.100.5"
|
||||
SIP[pfv-tsys6]="10.100.100.6"
|
||||
SIP[pfv-tsys7]="10.100.100.7"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " STEP 1: Install iperf3 on all online hosts"
|
||||
echo "==================================================================="
|
||||
|
||||
for host in "${ALL_HOSTS[@]}"; do
|
||||
echo -n " [$host] "
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "UNREACHABLE — skipping"
|
||||
continue
|
||||
fi
|
||||
# Check if iperf3 already installed
|
||||
if ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
|
||||
echo "iperf3 already installed"
|
||||
else
|
||||
printf "installing... "
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " STEP 2: Kill any existing iperf3 processes everywhere"
|
||||
echo "==================================================================="
|
||||
for host in "${ALL_HOSTS[@]}"; do
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
echo " Done."
|
||||
|
||||
# Helper: run an iperf3 test and save output
|
||||
run_iperf() {
|
||||
local client="$1" server="$2" direction="$3" label="$4" logfile="$5"
|
||||
local client_ip="${SIP[$client]}" server_ip="${SIP[$server]}"
|
||||
|
||||
echo -n " [$client → $server] $label ... "
|
||||
|
||||
# Start server in one-shot mode (-1 means serve one client then exit)
|
||||
ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
|
||||
sleep 1
|
||||
|
||||
# Run client
|
||||
{
|
||||
echo "=== iperf3: $label ==="
|
||||
echo "Client: $client ($client_ip)"
|
||||
echo "Server: $server ($server_ip)"
|
||||
echo "Direction: $direction"
|
||||
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo ""
|
||||
if [ "$direction" = "forward" ]; then
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" \
|
||||
"iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2" 2>&1
|
||||
elif [ "$direction" = "reverse" ]; then
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" \
|
||||
"iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2 -R" 2>&1
|
||||
elif [ "$direction" = "udp" ]; then
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" \
|
||||
"iperf3 -c ${server_ip} -B ${client_ip} -u -b 2G -t 10 -l 8972" 2>&1
|
||||
elif [ "$direction" = "single" ]; then
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" \
|
||||
"iperf3 -c ${server_ip} -B ${client_ip} -t 20 -O 2" 2>&1
|
||||
fi
|
||||
echo ""
|
||||
echo "=== END ==="
|
||||
} > "$logfile" 2>&1
|
||||
|
||||
# Extract summary line
|
||||
if grep -q "sender" "$logfile"; then
|
||||
bitrate=$(grep "sender" "$logfile" | tail -1 | awk '{print $7, $8}')
|
||||
echo "done: ${bitrate}"
|
||||
else
|
||||
echo "done (check log for details)"
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " STEP 3: Run iperf3 test matrix"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
echo "All tests over VLAN1000 storage network (10.100.100.0/24)."
|
||||
echo "TCP tests: 8 parallel streams, 30s, 128k blocks."
|
||||
echo ""
|
||||
|
||||
# --- Test 1: tsys7 → tsys4 (USB cdc_ncm target) ---
|
||||
echo "--- TEST 1: tsys7 → tsys4 (USB cdc_ncm NIC) ---"
|
||||
run_iperf pfv-tsys7 pfv-tsys4 forward "TCP 8-stream forward (tsys7→tsys4 USB)" \
|
||||
"$LOG_DIR/01-tsys7-to-tsys4-tcp-forward.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys7)" \
|
||||
"$LOG_DIR/02-tsys7-to-tsys4-tcp-reverse.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys4 single "TCP single-stream forward (tsys7→tsys4 USB)" \
|
||||
"$LOG_DIR/03-tsys7-to-tsys4-tcp-single.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys4 udp "UDP saturation (tsys7→tsys4 USB)" \
|
||||
"$LOG_DIR/04-tsys7-to-tsys4-udp.log"
|
||||
|
||||
echo ""
|
||||
|
||||
# --- Test 2: tsys7 → tsys5 (bond0, PCI NIC, 1 active slave) ---
|
||||
echo "--- TEST 2: tsys7 → tsys5 (PCI NIC, broken bond - 1 slave) ---"
|
||||
run_iperf pfv-tsys7 pfv-tsys5 forward "TCP 8-stream forward (tsys7→tsys5 PCI)" \
|
||||
"$LOG_DIR/05-tsys7-to-tsys5-tcp-forward.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys5 reverse "TCP 8-stream reverse (tsys5 PCI→tsys7)" \
|
||||
"$LOG_DIR/06-tsys7-to-tsys5-tcp-reverse.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys5 single "TCP single-stream forward (tsys7→tsys5 PCI)" \
|
||||
"$LOG_DIR/07-tsys7-to-tsys5-tcp-single.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys5 udp "UDP saturation (tsys7→tsys5 PCI)" \
|
||||
"$LOG_DIR/08-tsys7-to-tsys5-udp.log"
|
||||
|
||||
echo ""
|
||||
|
||||
# --- Test 3: tsys7 → tsys6 (working 2-slave LACP baseline, layer3+4) ---
|
||||
echo "--- TEST 3: tsys7 → tsys6 (working 2×1G LACP baseline) ---"
|
||||
run_iperf pfv-tsys7 pfv-tsys6 forward "TCP 8-stream forward (tsys7→tsys6 LACP)" \
|
||||
"$LOG_DIR/09-tsys7-to-tsys6-tcp-forward.log"
|
||||
run_iperf pfv-tsys7 pfv-tsys6 reverse "TCP 8-stream reverse (tsys6 LACP→tsys7)" \
|
||||
"$LOG_DIR/10-tsys7-to-tsys6-tcp-reverse.log"
|
||||
|
||||
echo ""
|
||||
|
||||
# --- Test 4: tsys6 → tsys4 (pre-tuning baseline) ---
|
||||
echo "--- TEST 4: tsys6 → tsys4 (baseline before tsys6 tuning) ---"
|
||||
run_iperf pfv-tsys6 pfv-tsys4 forward "TCP 8-stream forward (tsys6→tsys4 USB)" \
|
||||
"$LOG_DIR/11-tsys6-to-tsys4-tcp-forward.log"
|
||||
run_iperf pfv-tsys6 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys6)" \
|
||||
"$LOG_DIR/12-tsys6-to-tsys4-tcp-reverse.log"
|
||||
|
||||
echo ""
|
||||
|
||||
# --- Cleanup: kill iperf3 everywhere ---
|
||||
echo "--- Cleanup ---"
|
||||
for host in "${ALL_HOSTS[@]}"; do
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " RESULTS SUMMARY"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
printf "%-45s %s\n" "TEST" "THROUGHPUT"
|
||||
printf "%-45s %s\n" "----" "----------"
|
||||
for f in "$LOG_DIR"/*.log; do
|
||||
[ -r "$f" ] || continue
|
||||
label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
|
||||
bitrate=$(grep -E "sender$" "$f" | tail -1 | awk '{print $7, $8}')
|
||||
[ -z "$bitrate" ] && bitrate=$(grep -E "Mbits/sec|Gbits/sec" "$f" | tail -1 | grep -oE '[0-9.]+ [MG]bits/sec' | head -1)
|
||||
[ -z "$bitrate" ] && bitrate="(see log)"
|
||||
printf "%-45s %s\n" "$label" "$bitrate"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Full logs saved to: $LOG_DIR/"
|
||||
echo "==================================================================="
|
||||
Executable
+119
@@ -0,0 +1,119 @@
|
||||
#!/bin/bash
|
||||
# iperf-tsys6-tsys7.sh - validate 2Gbps LACP between the two tuned hosts.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
TSYS6="10.100.100.6"
|
||||
TSYS7="10.100.100.7"
|
||||
|
||||
run_test() {
|
||||
local client="$1" server="$2" server_ip="$3" label="$4" logfile="$5"
|
||||
shift 4
|
||||
local extra_args="$*"
|
||||
|
||||
echo -n " [$label] ... "
|
||||
|
||||
# Start server in one-shot mode
|
||||
ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
|
||||
sleep 1
|
||||
|
||||
{
|
||||
echo "=== iperf3: $label ==="
|
||||
echo "Client: $client Server: $server ($server_ip)"
|
||||
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "Args: $extra_args"
|
||||
echo ""
|
||||
ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${server_ip} $extra_args" 2>&1
|
||||
echo ""
|
||||
echo "=== END ==="
|
||||
} > "$logfile" 2>&1
|
||||
|
||||
# Extract result
|
||||
sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
|
||||
if [ -n "$sum" ]; then
|
||||
bitrate=$(echo "$sum" | awk '{print $6, $7}')
|
||||
retrans=$(echo "$sum" | awk '{print $8}')
|
||||
else
|
||||
single=$(grep 'sender$' "$logfile" | tail -1)
|
||||
bitrate=$(echo "$single" | awk '{print $7, $8}')
|
||||
retrans=$(echo "$single" | awk '{print $9}')
|
||||
fi
|
||||
echo "${bitrate} (retrans: ${retrans:-0})"
|
||||
}
|
||||
|
||||
echo "==================================================================="
|
||||
echo " iperf3: tsys6 ↔ tsys7 (both have 2×1G LACP + layer3+4 hash)"
|
||||
echo " Expectation: ~1.8-2.0 Gbps for 8-stream TCP"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
# Pre-flight: confirm bond state on both
|
||||
echo "--- bond0 state on tsys6 ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
|
||||
echo ""
|
||||
echo "--- bond0 state on tsys7 ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
|
||||
echo ""
|
||||
|
||||
echo "--- Running tests ---"
|
||||
echo ""
|
||||
|
||||
# Test 1: tsys7 → tsys6, 8-stream TCP forward
|
||||
run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
|
||||
"tsys7→tsys6 TCP 8-stream forward" \
|
||||
"$LOG_DIR/tsys6-tsys7-01-tcp-8stream-forward.log" \
|
||||
"-P 8 -t 30 -l 128k -O 2"
|
||||
|
||||
# Test 2: tsys6 → tsys7, 8-stream TCP forward (reverse direction)
|
||||
run_test pfv-tsys6 pfv-tsys7 "$TSYS7" \
|
||||
"tsys6→tsys7 TCP 8-stream forward" \
|
||||
"$LOG_DIR/tsys6-tsys7-02-tcp-8stream-forward.log" \
|
||||
"-P 8 -t 30 -l 128k -O 2"
|
||||
|
||||
# Test 3: tsys7 → tsys6, single stream (should be ~940 Mbps — single flow)
|
||||
run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
|
||||
"tsys7→tsys6 TCP single-stream" \
|
||||
"$LOG_DIR/tsys6-tsys7-03-tcp-single.log" \
|
||||
"-t 20 -O 2"
|
||||
|
||||
# Test 4: tsys7 → tsys6, 4-stream (nconnect=4 mirrors this)
|
||||
run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
|
||||
"tsys7→tsys6 TCP 4-stream" \
|
||||
"$LOG_DIR/tsys6-tsys7-04-tcp-4stream.log" \
|
||||
"-P 4 -t 30 -l 128k -O 2"
|
||||
|
||||
# Test 5: UDP saturation
|
||||
run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
|
||||
"tsys7→tsys6 UDP saturation" \
|
||||
"$LOG_DIR/tsys6-tsys7-05-udp.log" \
|
||||
"-u -b 3G -t 10 -l 8972"
|
||||
|
||||
# Cleanup
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " SUMMARY"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
|
||||
printf "%-45s %15s %10s\n" "----" "----------" "-------"
|
||||
for f in "$LOG_DIR"/tsys6-tsys7-*.log; do
|
||||
[ -r "$f" ] || continue
|
||||
label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
|
||||
sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
|
||||
if [ -n "$sum" ]; then
|
||||
bitrate=$(echo "$sum" | awk '{print $6, $7}')
|
||||
retrans=$(echo "$sum" | awk '{print $8}')
|
||||
else
|
||||
single=$(grep 'sender$' "$f" | tail -1)
|
||||
bitrate=$(echo "$single" | awk '{print $7, $8}')
|
||||
retrans=$(echo "$single" | awk '{print $9}')
|
||||
fi
|
||||
printf "%-45s %15s %10s\n" "$label" "$bitrate" "${retrans:--}"
|
||||
done
|
||||
echo ""
|
||||
echo "Expected: 8-stream ~1.8-2.0 Gbps, single-stream ~940 Mbps"
|
||||
Executable
+88
@@ -0,0 +1,88 @@
|
||||
#!/bin/bash
|
||||
# reboot-and-verify.sh - reboots a host and verifies NFS nconnect activates.
|
||||
# Usage: bash reboot-and-verify.sh <host>
|
||||
set -uo pipefail
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
echo "================================================================"
|
||||
echo "[$HOST] PRE-REBOOT STATE"
|
||||
echo "================================================================"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" '
|
||||
echo "--- VMs ---"
|
||||
qm list 2>/dev/null
|
||||
echo "--- NFS TCP conns: $(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l) ---"
|
||||
echo "--- NFS first mount: ---"
|
||||
nfsstat -m 2>/dev/null | head -2 | tail -1
|
||||
echo "--- uptime ---"
|
||||
uptime
|
||||
'
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] ISSUING REBOOT"
|
||||
echo "================================================================"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &'
|
||||
echo "Reboot sent at $(date +%H:%M:%S)"
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 min)"
|
||||
echo "================================================================"
|
||||
DEADLINE=$(( $(date +%s) + 600 ))
|
||||
LAST_PRINT=0
|
||||
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
|
||||
now=$(date +%s)
|
||||
if [ $((now - LAST_PRINT)) -ge 15 ]; then
|
||||
printf ' [%s] waiting... (%ss elapsed)\n' "$(date +%H:%M:%S)" "$(( now - DEADLINE + 600 ))"
|
||||
LAST_PRINT=$now
|
||||
fi
|
||||
if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
# Verify uptime is actually low (host really rebooted, not still up)
|
||||
up_mins=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/uptime | awk "{print int(\$1/60)}"')
|
||||
if [ "${up_mins:-999}" -lt 5 ]; then
|
||||
echo " [$(date +%H:%M:%S)] SSH back, uptime ${up_mins}min — real reboot confirmed"
|
||||
break
|
||||
fi
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "FAILED: $HOST not back after 10 minutes"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Waiting 30s for services to settle..."
|
||||
sleep 30
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] POST-REBOOT VERIFICATION"
|
||||
echo "================================================================"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" '
|
||||
echo "--- uptime ---"
|
||||
uptime
|
||||
echo ""
|
||||
echo "--- governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null) ---"
|
||||
echo "--- swappiness: $(sysctl -n vm.swappiness) ---"
|
||||
echo "--- tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control) ---"
|
||||
echo "--- tuned: $(tuned-adm active 2>/dev/null | grep Current) ---"
|
||||
echo ""
|
||||
echo "--- NFS first mount: ---"
|
||||
nfsstat -m 2>/dev/null | head -2 | tail -1
|
||||
echo ""
|
||||
echo "--- NFS TCP conns (expect 8 with nconnect=4): ---"
|
||||
ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l
|
||||
echo ""
|
||||
echo "--- VMs: ---"
|
||||
qm list 2>/dev/null
|
||||
echo ""
|
||||
echo "--- Failed services: ---"
|
||||
systemctl --failed --no-legend 2>/dev/null | head -5
|
||||
echo "(empty = none)"
|
||||
'
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] DONE"
|
||||
echo "================================================================"
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/bin/bash
|
||||
# reboot-verify.sh - reboots a host, waits for it to come back, verifies state.
|
||||
# Usage: bash reboot-verify.sh <host>
|
||||
set -uo pipefail
|
||||
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
echo "================================================================"
|
||||
echo "[$HOST] PRE-REBOOT STATE"
|
||||
echo "================================================================"
|
||||
echo "--- Running VMs ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR==1 || \$3==\"running\"{print}"' 2>&1
|
||||
echo "--- NFS mount count ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | grep -c "^/mnt"' 2>&1
|
||||
echo "--- NFS TCP connections to :2049 ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
|
||||
echo "--- Uptime ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] ISSUING REBOOT"
|
||||
echo "================================================================"
|
||||
# Issue reboot; ssh will disconnect with non-zero — that's expected.
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &' 2>&1
|
||||
echo "Reboot command sent at $(date +%H:%M:%S). Host will drop now."
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 minutes)"
|
||||
echo "================================================================"
|
||||
DEADLINE=$(( $(date +%s) + 600 ))
|
||||
LAST_PRINT=0
|
||||
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
|
||||
now=$(date +%s)
|
||||
# Print a heartbeat every 15s
|
||||
if [ $((now - LAST_PRINT)) -ge 15 ]; then
|
||||
elapsed=$((DEADLINE - now - 600)); elapsed=${elapsed#-}
|
||||
echo " [$(date +%H:%M:%S)] still waiting... (${elapsed}s elapsed)"
|
||||
LAST_PRINT=$now
|
||||
fi
|
||||
# Try SSH
|
||||
if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " [$(date +%H:%M:%S)] SSH is back!"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Final check
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " [$(date +%H:%M:%S)] FAILED: host not reachable after 10 minutes"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Give services a moment to settle after SSH returns
|
||||
echo " Waiting 20s for services to settle..."
|
||||
sleep 20
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] POST-REBOOT VERIFICATION"
|
||||
echo "================================================================"
|
||||
echo "--- Uptime ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
|
||||
echo ""
|
||||
echo "--- TCP congestion control ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc' 2>&1
|
||||
echo ""
|
||||
echo "--- vm.swappiness ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness' 2>&1
|
||||
echo ""
|
||||
echo "--- scaling_governor ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"' 2>&1
|
||||
echo ""
|
||||
echo "--- NFS mount options (looking for nconnect + noatime) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -20' 2>&1
|
||||
echo ""
|
||||
echo "--- NFS TCP connection count (expect ~8 = 4 per server with nconnect=4) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
|
||||
echo ""
|
||||
echo "--- Running VMs ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null' 2>&1
|
||||
echo ""
|
||||
echo "--- Failed services? ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10' 2>&1
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] DONE"
|
||||
echo "================================================================"
|
||||
Executable
+141
@@ -0,0 +1,141 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# apply-bond-hash.sh
|
||||
#
|
||||
# Adds bond-xmit-hash-policy layer3+4 to bond0 in /etc/network/interfaces,
|
||||
# then reloads networking with ifreload -a.
|
||||
#
|
||||
# SSH survivability: this is safe IF your SSH session is on vmbr0/nic0
|
||||
# (management network), NOT on bond0/datanet (storage network).
|
||||
# tsys7's topology confirms this: SSH comes in on vmbr0 (nic0).
|
||||
#
|
||||
# Safety:
|
||||
# - Dry-run by default (--apply to commit)
|
||||
# - Full backup of /etc/network/interfaces
|
||||
# - Generates rollback script
|
||||
# - Does NOT reboot — uses ifreload -a which is hot-reload
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
HOST="$(hostname -s)"
|
||||
TS_SHORT="$(date +%Y%m%d-%H%M%S)"
|
||||
BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
|
||||
ROLLBACK="/root/perfopt-bond-rollback-${TS_SHORT}.sh"
|
||||
ACTION="${1:-dryrun}"
|
||||
|
||||
[ "$ACTION" = "--apply" ] && ACTION="apply" || ACTION="dryrun"
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " apply-bond-hash — $HOST"
|
||||
echo " mode: $ACTION"
|
||||
echo "==================================================================="
|
||||
|
||||
if [ ! -r /etc/network/interfaces ]; then
|
||||
echo "FATAL: /etc/network/interfaces not readable"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if bond0 exists in the config
|
||||
if ! grep -q 'bond0' /etc/network/interfaces; then
|
||||
echo "No bond0 found in /etc/network/interfaces — nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check if the hash policy is already set
|
||||
if grep -q 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces; then
|
||||
echo "bond-xmit-hash-policy already present:"
|
||||
grep 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces
|
||||
echo "Checking value..."
|
||||
if grep -q 'layer3+4' /etc/network/interfaces; then
|
||||
echo "Already set to layer3+4 — nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# Show current bond state
|
||||
echo ""
|
||||
echo "--- Current bond0 state ---"
|
||||
cat /proc/net/bonding/bond0 2>/dev/null | head -15 || echo "(bond0 not up or not present)"
|
||||
|
||||
# Back up
|
||||
cp -a /etc/network/interfaces "$BACKUP_DIR/interfaces"
|
||||
|
||||
# Generate rollback script
|
||||
cat > "$ROLLBACK" <<EOF
|
||||
#!/bin/bash
|
||||
# Rollback for bond-xmit-hash-policy change
|
||||
# Restores original /etc/network/interfaces and reloads
|
||||
set -euo pipefail
|
||||
cp -a "$BACKUP_DIR/interfaces" /etc/network/interfaces
|
||||
echo "Restored /etc/network/interfaces"
|
||||
echo "Reloading networking..."
|
||||
ifreload -a 2>&1 || systemctl restart networking 2>&1 || true
|
||||
echo "Done. bond0 hash policy reverted to original."
|
||||
EOF
|
||||
chmod +x "$ROLLBACK"
|
||||
|
||||
echo ""
|
||||
echo "--- Proposed change ---"
|
||||
echo "Add line ' bond-xmit-hash-policy layer3+4' to the bond0 stanza."
|
||||
echo ""
|
||||
|
||||
if [ "$ACTION" != "apply" ]; then
|
||||
echo "DRY RUN — no changes made."
|
||||
echo "To commit: bash $0 --apply"
|
||||
echo "Rollback script (pre-generated): $ROLLBACK"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Apply: use sed to insert bond-xmit-hash-policy after bond-mode line
|
||||
# The bond0 stanza looks like:
|
||||
# auto bond0
|
||||
# iface bond0 inet manual
|
||||
# bond-slaves nic1 nic2
|
||||
# bond-miimon 100
|
||||
# bond-mode 802.3ad
|
||||
#
|
||||
# We insert after the bond-mode line.
|
||||
echo "Applying..."
|
||||
|
||||
# Check if bond-mode line exists (various formats)
|
||||
if grep -qE '^\s*bond-mode\s+802.3ad' /etc/network/interfaces; then
|
||||
# Insert after bond-mode 802.3ad line
|
||||
sed -i '/^\s*bond-mode\s+802\.3ad/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
|
||||
echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode line."
|
||||
elif grep -qE '^\s*bond-mode\s+4' /etc/network/interfaces; then
|
||||
sed -i '/^\s*bond-mode\s+4/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
|
||||
echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode 4 line."
|
||||
else
|
||||
echo "Could not find bond-mode line — inserting after bond-slaves line instead."
|
||||
sed -i '/^\s*bond-slaves/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
|
||||
fi
|
||||
|
||||
# Show the result
|
||||
echo ""
|
||||
echo "--- Updated bond0 stanza ---"
|
||||
awk '/^auto bond0/,/^$/' /etc/network/interfaces
|
||||
|
||||
echo ""
|
||||
echo "--- Reloading networking (ifreload -a) ---"
|
||||
echo "SSH should survive (it's on vmbr0/nic0, not bond0)..."
|
||||
ifreload -a 2>&1 || {
|
||||
echo "ifreload failed, trying systemctl restart networking..."
|
||||
systemctl restart networking 2>&1
|
||||
}
|
||||
|
||||
# Wait a moment for bond to renegotiate
|
||||
echo "Waiting 5s for LACP to renegotiate..."
|
||||
sleep 5
|
||||
|
||||
echo ""
|
||||
echo "--- Post-change bond0 state ---"
|
||||
cat /proc/net/bonding/bond0 2>/dev/null | head -20
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " DONE."
|
||||
echo " Backup: $BACKUP_DIR/interfaces"
|
||||
echo " Rollback: bash $ROLLBACK"
|
||||
echo "==================================================================="
|
||||
Executable
+439
@@ -0,0 +1,439 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# apply-tunings.sh
|
||||
#
|
||||
# Applies the Tier 0 host-side tunings identified in FINDINGS.md:
|
||||
# 1. Sets scaling_governor=performance on every CPU
|
||||
# 2. Sets vm.swappiness appropriately (1 on storage hosts, 10 on VM hosts)
|
||||
# 3. Enables tcp_bbr module + sets congestion control
|
||||
# 4. Sets net.core.rmem_max/wmem_max + tcp_rmem/wmem for high-BDP NFS
|
||||
# 5. Sets the recommended tuned-adm profile (network-throughput or virtual-host)
|
||||
# 6. Adds nconnect=4 + noatime to NFS client mounts (edits /etc/pve/storage.cfg)
|
||||
# 7. (does NOT touch bond0 xmit-hash — that requires ifreload/network
|
||||
# restart which drops the host. Provided as a separate --emit-bond-patch
|
||||
# flag that PRINTS the change but does not apply it.)
|
||||
#
|
||||
# Safety features:
|
||||
# - Dry-run mode by default (--apply to commit)
|
||||
# - Full backup of every modified file to /root/perfopt-backup-<timestamp>/
|
||||
# - Per-host behaviour: detects storage hosts by hostname and applies the
|
||||
# right profile (tsys4, tsys5 = storage; others = VM hosts).
|
||||
# - Generates a /root/perfopt-rollback.sh that undoes everything.
|
||||
# - Does NOT reboot, does NOT restart networking, does NOT touch hardware.
|
||||
#
|
||||
# Usage:
|
||||
# bash apply-tunings.sh # dry run, show what would change
|
||||
# bash apply-tunings.sh --apply # commit changes
|
||||
# bash apply-tunings.sh --rollback # restore from latest backup
|
||||
# bash apply-tunings.sh --emit-bond-patch # show bond0 hash change (no apply)
|
||||
###############################################################################
|
||||
set -u
|
||||
umask 022
|
||||
|
||||
HOST="$(hostname -s)"
|
||||
TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
TS_SHORT="$(date +%Y%m%d-%H%M%S)"
|
||||
BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
|
||||
ROLLBACK_SCRIPT="/root/perfopt-rollback-${TS_SHORT}.sh"
|
||||
ACTION="dryrun"
|
||||
SKIP_NFS=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--apply) ACTION="apply" ;;
|
||||
--rollback) ACTION="rollback" ;;
|
||||
--emit-bond-patch) ACTION="bond-patch" ;;
|
||||
--no-nfs) SKIP_NFS=1 ;;
|
||||
-h|--help) sed -n '2,35p' "$0"; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
# Detect host role from hostname
|
||||
case "$HOST" in
|
||||
pfv-tsys4|pfv-tsys5)
|
||||
HOST_ROLE="storage"
|
||||
TUNED_PROFILE="network-throughput"
|
||||
SWAPPINESS="1"
|
||||
;;
|
||||
pfv-tsys1|pfv-tsys3|pfv-tsys6|pfv-tsys7)
|
||||
HOST_ROLE="vmhost"
|
||||
TUNED_PROFILE="virtual-host"
|
||||
SWAPPINESS="10"
|
||||
;;
|
||||
*)
|
||||
HOST_ROLE="unknown"
|
||||
TUNED_PROFILE="virtual-host"
|
||||
SWAPPINESS="10"
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- rollback path ---------------------------------------------------------
|
||||
if [ "$ACTION" = "rollback" ]; then
|
||||
echo "Looking for latest backup under /root/perfopt-backup-* ..."
|
||||
latest=""
|
||||
while IFS= read -r d; do
|
||||
latest="$d"
|
||||
done < <(find /root -maxdepth 1 -type d -name 'perfopt-backup-*' 2>/dev/null | sort | tail -n1)
|
||||
if [ -z "$latest" ]; then
|
||||
echo "No backup found under /root/perfopt-backup-*" >&2
|
||||
exit 1
|
||||
fi
|
||||
rb="$latest/perfopt-rollback.sh"
|
||||
if [ ! -x "$rb" ] && [ ! -r "$rb" ]; then
|
||||
echo "Rollback script missing in $latest" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Rolling back using: $rb"
|
||||
bash "$rb"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
# --- bond patch print-only path -------------------------------------------
|
||||
if [ "$ACTION" = "bond-patch" ]; then
|
||||
echo "==================================================================="
|
||||
echo " Proposed bond0 xmit_hash_policy change"
|
||||
echo "==================================================================="
|
||||
echo
|
||||
if [ ! -r /etc/network/interfaces ]; then
|
||||
echo "/etc/network/interfaces not readable"
|
||||
exit 1
|
||||
fi
|
||||
if ! grep -q 'bond-mode 802.3ad\|bond-mode 4\|bond-slaves' /etc/network/interfaces; then
|
||||
echo "No bond0 detected on this host — nothing to patch."
|
||||
exit 0
|
||||
fi
|
||||
cat <<EOF
|
||||
A manual edit to /etc/network/interfaces is required. The bond0 stanza
|
||||
needs this line added (it defaults to layer2, which is wrong for storage):
|
||||
|
||||
bond-xmit-hash-policy layer3+4
|
||||
|
||||
After editing, you MUST reload networking for it to take effect:
|
||||
ifreload -a # safe, brings interfaces down/up
|
||||
# OR
|
||||
systemctl restart networking # heavier, briefly drops connections
|
||||
|
||||
WARNING: applying this on a remote host over bond0 will briefly drop your
|
||||
SSH session. Run from console/IPMI, or schedule a maintenance window.
|
||||
|
||||
The change is reversible by removing the line and reloading again.
|
||||
EOF
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- main path (dryrun or apply) ------------------------------------------
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# Emit the rollback script header
|
||||
cat > "$ROLLBACK_SCRIPT" <<EOF
|
||||
#!/bin/bash
|
||||
# Auto-generated rollback for perfopt apply-tunings.sh
|
||||
# Backup timestamp: $TS
|
||||
# Backup dir: $BACKUP_DIR
|
||||
# Generated on: $HOST
|
||||
set -u
|
||||
EOF
|
||||
chmod +x "$ROLLBACK_SCRIPT"
|
||||
|
||||
backup_file() {
|
||||
local f="$1"
|
||||
if [ -e "$f" ]; then
|
||||
cp -a "$f" "$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
|
||||
fi
|
||||
}
|
||||
|
||||
# Helper: append a restore command to the rollback script
|
||||
rb_restore() {
|
||||
local f="$1"
|
||||
local bk
|
||||
bk="$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
if [ -r "$bk" ]; then
|
||||
cp -a "$bk" "$f" && echo "restored $f"
|
||||
else
|
||||
echo "WARN: backup $bk missing — $f left as-is"
|
||||
fi
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: append a sysctl restore command
|
||||
rb_sysctl_restore() {
|
||||
local key="$1" orig_val="$2"
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
sysctl -w "$key=$orig_val" >/dev/null && echo "restored $key=$orig_val"
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: append a tuned-adm restore
|
||||
rb_tuned_restore() {
|
||||
local profile="$1"
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
tuned-adm profile "$profile" 2>/dev/null && echo "restored tuned profile: $profile"
|
||||
EOF
|
||||
}
|
||||
|
||||
echo "==================================================================="
|
||||
echo " perfopt apply-tunings — $HOST"
|
||||
echo " mode: $ACTION"
|
||||
echo " role: $HOST_ROLE (tuned=$TUNED_PROFILE, swappiness=$SWAPPINESS)"
|
||||
echo " backup: $BACKUP_DIR"
|
||||
echo " rollback: $ROLLBACK_SCRIPT"
|
||||
echo "==================================================================="
|
||||
echo
|
||||
|
||||
# ===========================================================================
|
||||
# 1. scaling_governor → performance
|
||||
# ===========================================================================
|
||||
echo "--- 1. CPU scaling_governor → performance"
|
||||
if [ -r /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ]; then
|
||||
cur=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor)
|
||||
echo " current: $cur"
|
||||
if [ "$ACTION" = "apply" ]; then
|
||||
if [ "$cur" != "performance" ]; then
|
||||
for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
|
||||
echo performance > "$c" 2>/dev/null || true
|
||||
done
|
||||
# Persist via systemd tmpfiles / sysctl fallback
|
||||
cat > /etc/systemd/system/perfopt-cpu-performance.service <<EOF
|
||||
[Unit]
|
||||
Description=Set CPU scaling_governor=performance (perfopt)
|
||||
After=multi-user.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/sh -c 'for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "\$c" 2>/dev/null || true; done'
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl enable perfopt-cpu-performance.service 2>/dev/null
|
||||
systemctl start perfopt-cpu-performance.service 2>/dev/null
|
||||
echo " applied + persisted via systemd unit"
|
||||
# Rollback
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "$cur" > "\$c" 2>/dev/null || true; done
|
||||
systemctl disable --now perfopt-cpu-performance.service 2>/dev/null || true
|
||||
rm -f /etc/systemd/system/perfopt-cpu-performance.service
|
||||
systemctl daemon-reload
|
||||
echo "restored scaling_governor=$cur (best-effort; original may have been dynamic)"
|
||||
EOF
|
||||
else
|
||||
echo " already performance — no change"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
echo " cpufreq driver not loaded — nothing to do (typical on BIOS-locked servers)"
|
||||
fi
|
||||
|
||||
# ===========================================================================
|
||||
# 2. vm.swappiness
|
||||
# ===========================================================================
|
||||
echo "--- 2. vm.swappiness → $SWAPPINESS (role=$HOST_ROLE)"
|
||||
cur_swappiness=$(sysctl -n vm.swappiness 2>/dev/null || echo "?")
|
||||
echo " current: $cur_swappiness"
|
||||
if [ "$ACTION" = "apply" ] && [ "$cur_swappiness" != "$SWAPPINESS" ]; then
|
||||
# Persist via sysctl.d
|
||||
backup_file /etc/sysctl.d/99-perfopt.conf
|
||||
cat > /etc/sysctl.d/99-perfopt.conf <<EOF
|
||||
# perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
|
||||
vm.swappiness = $SWAPPINESS
|
||||
EOF
|
||||
sysctl -w "vm.swappiness=$SWAPPINESS" >/dev/null
|
||||
echo " applied (persisted to /etc/sysctl.d/99-perfopt.conf)"
|
||||
rb_restore /etc/sysctl.d/99-perfopt.conf
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
rm -f /etc/sysctl.d/99-perfopt.conf
|
||||
sysctl -w "vm.swappiness=$cur_swappiness" >/dev/null
|
||||
echo "restored vm.swappiness=$cur_swappiness"
|
||||
EOF
|
||||
else
|
||||
echo " no change"
|
||||
fi
|
||||
|
||||
# ===========================================================================
|
||||
# 3. TCP BBR + buffers (single sysctl.d file)
|
||||
# ===========================================================================
|
||||
echo "--- 3. TCP BBR + socket buffers"
|
||||
cur_cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null || echo "?")
|
||||
cur_avail=$(sysctl -n net.ipv4.tcp_available_congestion_control 2>/dev/null || echo "?")
|
||||
echo " current: $cur_cc (available: $cur_avail)"
|
||||
|
||||
# Decide sysctl values (these are the consensus values for a 1-10 GbE NFS host)
|
||||
TARGET_RMEM_MAX=134217728 # 128 MB
|
||||
TARGET_WMEM_MAX=134217728
|
||||
TARGET_RMEM_DEFAULT=26214400 # 25 MB
|
||||
TARGET_WMEM_DEFAULT=26214400
|
||||
TARGET_TCP_RMEM="4096 87380 $TARGET_RMEM_MAX"
|
||||
TARGET_TCP_WMEM="4096 65536 $TARGET_RMEM_MAX"
|
||||
TARGET_NETDEV_MAX_BACKLOG=250000
|
||||
TARGET_SOMAXCONN=65535
|
||||
|
||||
if [ "$ACTION" = "apply" ]; then
|
||||
# Ensure tcp_bbr module loads at boot
|
||||
if ! grep -q '^tcp_bbr' /etc/modules-load.d/modules.conf 2>/dev/null; then
|
||||
backup_file /etc/modules-load.d/modules.conf
|
||||
mkdir -p /etc/modules-load.d
|
||||
cat >> /etc/modules-load.d/modules.conf <<EOF
|
||||
# perfopt apply-tunings.sh — $TS
|
||||
tcp_bbr
|
||||
EOF
|
||||
modprobe tcp_bbr 2>/dev/null || true
|
||||
rb_restore /etc/modules-load.d/modules.conf
|
||||
fi
|
||||
|
||||
backup_file /etc/sysctl.d/99-perfopt.conf
|
||||
cat > /etc/sysctl.d/99-perfopt.conf <<EOF
|
||||
# perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
|
||||
vm.swappiness = $SWAPPINESS
|
||||
|
||||
# TCP BBR + high-BDP buffers (good for NFS over 1-10 GbE)
|
||||
net.ipv4.tcp_congestion_control = bbr
|
||||
net.core.default_qdisc = fq
|
||||
net.core.rmem_max = $TARGET_RMEM_MAX
|
||||
net.core.wmem_max = $TARGET_WMEM_MAX
|
||||
net.core.rmem_default = $TARGET_RMEM_DEFAULT
|
||||
net.core.wmem_default = $TARGET_WMEM_DEFAULT
|
||||
net.core.netdev_max_backlog = $TARGET_NETDEV_MAX_BACKLOG
|
||||
net.core.somaxconn = $TARGET_SOMAXCONN
|
||||
net.ipv4.tcp_rmem = $TARGET_TCP_RMEM
|
||||
net.ipv4.tcp_wmem = $TARGET_TCP_WMEM
|
||||
EOF
|
||||
sysctl --system >/dev/null 2>&1
|
||||
echo " applied (written to /etc/sysctl.d/99-perfopt.conf + sysctl --system)"
|
||||
# Rebuild rollback for sysctl
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
# Restore original sysctl state
|
||||
rm -f /etc/sysctl.d/99-perfopt.conf
|
||||
EOF
|
||||
rb_sysctl_restore "net.ipv4.tcp_congestion_control" "$cur_cc"
|
||||
rb_sysctl_restore "vm.swappiness" "$cur_swappiness"
|
||||
# Note: original buffer sizes not all captured; restoration is best-effort.
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
echo "Note: net buffers restored to kernel defaults (original values not captured)."
|
||||
echo "Run 'sysctl --system' to apply."
|
||||
sysctl --system >/dev/null 2>&1 || true
|
||||
EOF
|
||||
fi
|
||||
|
||||
# ===========================================================================
|
||||
# 4. tuned-adm profile
|
||||
# ===========================================================================
|
||||
echo "--- 4. tuned-adm profile → $TUNED_PROFILE"
|
||||
if command -v tuned-adm >/dev/null 2>&1; then
|
||||
cur_profile=$(tuned-adm active 2>/dev/null | awk -F: '/Current active profile/{gsub(/^[ \t]+/,"",$2); print $2}')
|
||||
echo " current: $cur_profile"
|
||||
if [ "$ACTION" = "apply" ] && [ "$cur_profile" != "$TUNED_PROFILE" ]; then
|
||||
tuned-adm profile "$TUNED_PROFILE" 2>&1 | sed 's/^/ /'
|
||||
rb_tuned_restore "$cur_profile"
|
||||
else
|
||||
echo " no change"
|
||||
fi
|
||||
else
|
||||
echo " tuned-adm not installed — skipping"
|
||||
fi
|
||||
|
||||
# ===========================================================================
|
||||
# 5. NFS mount option hardening
|
||||
# Edit /etc/pve/storage.cfg to add nconnect=4 + noatime to NFS plugins.
|
||||
# Proxmox applies these on next mount/remount.
|
||||
# Skipped with --no-nfs (for NFS servers where we don't want to remount).
|
||||
# ===========================================================================
|
||||
echo "--- 5. NFS mount options: add nconnect=4, noatime"
|
||||
if [ "$SKIP_NFS" -eq 1 ]; then
|
||||
echo " SKIPPED (--no-nfs specified)"
|
||||
echo " Note: storage.cfg is cluster-wide; if edited on another host,"
|
||||
echo " the options are already staged here and activate on next reboot."
|
||||
else
|
||||
if [ -r /etc/pve/storage.cfg ]; then
|
||||
backup_file /etc/pve/storage.cfg
|
||||
if grep -q '^nfs:' /etc/pve/storage.cfg; then
|
||||
# Count how many nfs: stanzas lack the options
|
||||
while IFS= read -r line; do
|
||||
if echo "$line" | grep -q '^nfs:'; then
|
||||
cur_stanza=$(echo "$line" | awk '{print $2}')
|
||||
# Look ahead for the next few lines to see if options already set
|
||||
echo " nfs stanza: $cur_stanza"
|
||||
fi
|
||||
done < /etc/pve/storage.cfg
|
||||
|
||||
# Check whether any nfs stanza already has options
|
||||
if grep -A1 '^nfs:' /etc/pve/storage.cfg | grep -q 'options.*nconnect=4'; then
|
||||
echo " some stanzas already have nconnect=4 — check manually"
|
||||
else
|
||||
echo " proposed change: add 'options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2' to each nfs stanza"
|
||||
if [ "$ACTION" = "apply" ]; then
|
||||
# Apply via perl for safety (in-place edit with backup)
|
||||
if cp -a /etc/pve/storage.cfg "$BACKUP_DIR/etc__pve__storage.cfg"; then
|
||||
# Use a python helper for robust PVE storage.cfg editing
|
||||
python3 - <<PYEOF
|
||||
import re, pathlib
|
||||
p = pathlib.Path("/etc/pve/storage.cfg")
|
||||
text = p.read_text()
|
||||
pattern = re.compile(r'(^nfs:\s*\S+\n(?:[ \t]+[^\n]+\n)+)', re.MULTILINE)
|
||||
modified = 0
|
||||
def fix(m):
|
||||
global modified
|
||||
block = m.group(1)
|
||||
if 'options' in block:
|
||||
return block
|
||||
lines = block.rstrip('\n').split('\n')
|
||||
lines.insert(1, '\toptions nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2')
|
||||
modified += 1
|
||||
return '\n'.join(lines) + '\n'
|
||||
new_text = pattern.sub(fix, text)
|
||||
if modified:
|
||||
p.write_text(new_text)
|
||||
print(f" patched {modified} NFS stanzas with options line")
|
||||
else:
|
||||
print(" no NFS stanza needed patching (all already had options or no nfs: found)")
|
||||
PYEOF
|
||||
# Remount existing NFS mounts to pick up new options
|
||||
echo " remounting NFS mounts to apply new options..."
|
||||
while IFS= read -r mp; do
|
||||
mount -o remount "$mp" 2>/dev/null && echo " remounted $mp" || echo " FAILED to remount $mp (will pick up on next mount)"
|
||||
done < <(awk '$3 ~ /^nfs/{print $2}' /proc/mounts 2>/dev/null | sort -u)
|
||||
rb_restore /etc/pve/storage.cfg
|
||||
cat >> "$ROLLBACK_SCRIPT" <<EOF
|
||||
# To fully undo NFS options, also remount:
|
||||
while IFS= read -r mp; do
|
||||
mount -o remount "\$mp" 2>/dev/null
|
||||
done < <(awk '\$3 ~ /^nfs/{print \$2}' /proc/mounts 2>/dev/null | sort -u)
|
||||
echo "restored NFS mount options"
|
||||
EOF
|
||||
else
|
||||
echo " ERROR: could not back up storage.cfg — aborting NFS patch"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
else
|
||||
echo " no NFS stanzas in storage.cfg — skipping"
|
||||
fi
|
||||
else
|
||||
echo " /etc/pve/storage.cfg not readable — skipping"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ===========================================================================
|
||||
# 6. Summary
|
||||
# ===========================================================================
|
||||
echo
|
||||
echo "==================================================================="
|
||||
if [ "$ACTION" = "apply" ]; then
|
||||
echo " DONE. Backups in: $BACKUP_DIR"
|
||||
echo " Rollback: bash $ROLLBACK_SCRIPT"
|
||||
echo
|
||||
echo " Next steps:"
|
||||
echo " - Verify with 'sysctl net.ipv4.tcp_congestion_control vm.swappiness'"
|
||||
echo " - Verify NFS mounts with 'nfsstat -m' (look for nconnect=4)"
|
||||
echo " - Run 'bash apply-tunings.sh --emit-bond-patch' for the bond0"
|
||||
echo " xmit_hash_policy change (requires network restart, do in"
|
||||
echo " maintenance window)"
|
||||
else
|
||||
echo " DRY RUN — no changes made."
|
||||
echo " To commit: bash apply-tunings.sh --apply"
|
||||
fi
|
||||
echo "==================================================================="
|
||||
Executable
+1138
File diff suppressed because it is too large
Load Diff
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/bash
|
||||
# fix-tsys7.sh - fixes the two issues found: NFS options string + bond hash
|
||||
# Runs on the target host directly.
|
||||
set -euo pipefail
|
||||
|
||||
echo "==================================================================="
|
||||
echo " FIX: NFS options + bond hash policy"
|
||||
echo "==================================================================="
|
||||
|
||||
# --- FIX 1: Simplify NFS options in storage.cfg ---------------------------
|
||||
echo ""
|
||||
echo "--- FIX 1: Simplify NFS options (remove version=4.2 conflict) ---"
|
||||
# Replace the overly-complex options line with a minimal one
|
||||
# PVE handles vers/rsize/wsize/hard/etc internally; we only need nconnect + noatime
|
||||
if grep -q 'options.*nconnect=4.*version=4.2' /etc/pve/storage.cfg; then
|
||||
# Use sed to replace each options line
|
||||
sed -i 's/options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2/options nconnect=4,noatime/' /etc/pve/storage.cfg
|
||||
echo "Fixed: simplified NFS options to 'nconnect=4,noatime'"
|
||||
else
|
||||
echo "Options line already simplified or not present"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- Verify storage.cfg NFS stanzas ---"
|
||||
grep -A 2 "^nfs: D2" /etc/pve/storage.cfg | head -3
|
||||
echo "..."
|
||||
grep -c "options nconnect" /etc/pve/storage.cfg
|
||||
echo " NFS stanzas with options"
|
||||
|
||||
# --- FIX 2: Bond hash policy (force via sysfs + persist in interfaces) ----
|
||||
echo ""
|
||||
echo "--- FIX 2: Apply bond xmit_hash_policy=layer3+4 ---"
|
||||
|
||||
# 2a: Apply LIVE via sysfs (takes effect immediately, no network reload)
|
||||
echo "Applying live via sysfs..."
|
||||
if echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; then
|
||||
echo "Live sysfs apply: SUCCESS"
|
||||
else
|
||||
echo "Live sysfs apply: FAILED (will persist in config and apply on ifreload)"
|
||||
fi
|
||||
|
||||
# Verify live state
|
||||
echo ""
|
||||
echo "Live bond0 hash policy:"
|
||||
cat /proc/net/bonding/bond0 | grep "Transmit Hash"
|
||||
|
||||
# 2b: Persist in /etc/network/interfaces (fix the sed that failed before)
|
||||
echo ""
|
||||
echo "Persisting in /etc/network/interfaces..."
|
||||
|
||||
# Backup
|
||||
cp -a /etc/network/interfaces "/root/interfaces.bondfix.$(date +%Y%m%d%H%M%S)"
|
||||
|
||||
# Check if already present
|
||||
if grep -q 'bond-xmit-hash-policy' /etc/network/interfaces; then
|
||||
echo "bond-xmit-hash-policy already in interfaces file"
|
||||
else
|
||||
# Use awk to insert after the bond-mode line (more reliable than sed)
|
||||
# Match any line containing 'bond-mode' (regardless of indentation)
|
||||
awk '
|
||||
/bond-mode/ && !done {
|
||||
print
|
||||
print "\tbond-xmit-hash-policy layer3+4"
|
||||
done=1
|
||||
next
|
||||
}
|
||||
{ print }
|
||||
' /etc/network/interfaces > /etc/network/interfaces.new
|
||||
|
||||
mv /etc/network/interfaces.new /etc/network/interfaces
|
||||
echo "Inserted bond-xmit-hash-policy layer3+4"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- Updated bond0 stanza ---"
|
||||
awk '/^auto bond0/,/^$/' /etc/network/interfaces
|
||||
|
||||
# Final verify
|
||||
echo ""
|
||||
echo "--- Final bond0 running state ---"
|
||||
cat /proc/net/bonding/bond0 | head -20
|
||||
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " FIX COMPLETE"
|
||||
echo "==================================================================="
|
||||
@@ -0,0 +1,217 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# lacp-retrans-cause.sh (HOST-NATIVE)
|
||||
#
|
||||
# Runs on tsys6 (receiver) or tsys7 (sender). Auto-detects role.
|
||||
#
|
||||
# Question this answers: are the ~56K retransmits we see on 8-stream iperf
|
||||
# between tsys6 and tsys7 fixable (NIC ring drops, softnet drops, CPU
|
||||
# saturation) or just unavoidable LACP reordering overhead?
|
||||
#
|
||||
# Method: snapshot drop/error counters + softnet_stat + TCP SNMP before and
|
||||
# after a 12s iperf3 run, then print the deltas. The decisive columns are:
|
||||
# - NIC rx_dropped / rx_missed_errors / rx_no_dma_resources -> ring too small
|
||||
# - /proc/net/softnet_stat drops -> ksoftirq starved
|
||||
# - /proc/net/snmp TCP retranst -> TCP-level retrans
|
||||
# If NIC drop counters do NOT climb but TCP retrans does, the retrans are
|
||||
# coming from LACP reordering (out-of-order segments triggering fast
|
||||
# retransmit), not packet loss — and are NOT fixable by tuning.
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
DURATION="${LACP_TEST_SECS:-12}"
|
||||
STREAMS="${LACP_STREAMS:-8}"
|
||||
PEER_RX_IP="10.100.100.6" # tsys6 storage IP (receiver)
|
||||
PEER_TX_IP="10.100.100.7" # tsys7 storage IP (sender)
|
||||
LOG_DIR="/root"
|
||||
BOND="bond0"
|
||||
|
||||
HOST="$(uname -n)"; HOST="${HOST%%.*}"
|
||||
case "$HOST" in
|
||||
*tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-retrans-receiver.log" ;;
|
||||
*tsys7) ROLE="sender"; LOG="$LOG_DIR/lacp-retrans-sender.log" ;;
|
||||
*) echo "ERROR: not on tsys6 or tsys7 (host=$HOST)"; exit 2 ;;
|
||||
esac
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# ---- per-slave NIC counter snapshot -----------------------------------------
|
||||
# Captures rx_dropped, rx_missed_errors, rx_no_dma_resources, tx_retrans,
|
||||
# plus all error-like counters. Falls back gracefully if a counter doesn't
|
||||
# exist (different NIC drivers expose different names).
|
||||
nic_snapshot() {
|
||||
for s in $(awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND"); do
|
||||
[ -n "$s" ] || continue
|
||||
echo "[$s]"
|
||||
# ethtool -S may exist; show only drop/error/retrans lines
|
||||
if command -v ethtool >/dev/null 2>&1; then
|
||||
ethtool -S "$s" 2>/dev/null \
|
||||
| grep -iE 'drop|miss|error|no_dma|fifo|retrans|overflow' \
|
||||
|| echo "(ethtool -S: no matching counters or unsupported)"
|
||||
else
|
||||
echo "(ethtool not installed)"
|
||||
fi
|
||||
# /sys counters (always available, driver-agnostic)
|
||||
for c in rx_dropped tx_dropped rx_errors tx_errors rx_missed_errors \
|
||||
rx_length_errors rx_crc_errors rx_fifo_errors tx_fifo_errors \
|
||||
multicast collisions; do
|
||||
v=$(cat "/sys/class/net/$s/statistics/$c" 2>/dev/null)
|
||||
[ -n "$v" ] && printf ' sysfs %-22s = %s\n' "$c" "$v"
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
# ---- softnet_stat snapshot (per-CPU RX softirq drops) -----------------------
|
||||
# /proc/net/softnet_stat columns are HEX. Col1=processed, col2=dropped,
|
||||
# col3=time_squeeze. We only care about dropped + squeeze (small numbers that
|
||||
# look identical in hex and decimal). Processed is informational only and we
|
||||
# don't try to delta it (hex arithmetic is awk-version-dependent).
|
||||
softnet_snapshot() {
|
||||
awk '{ printf "cpu%s processed=%s dropped=%s squeezed=%s\n", \
|
||||
NR-1, $1, $2, $3 }' /proc/net/softnet_stat
|
||||
}
|
||||
|
||||
# ---- TCP SNMP snapshot (the source of iperf's retransmit number) -----------
|
||||
tcp_snapshot() {
|
||||
awk '/^Tcp:/{
|
||||
if (!seen) {
|
||||
seen=1
|
||||
# /proc/net/snmp Tcp line 1 = names, line 2 = values
|
||||
n=split($0, names, " ")
|
||||
# re-find the values line
|
||||
getline
|
||||
vals=$0
|
||||
split(vals, v, " ")
|
||||
for (i=1;i<=n;i++) printf " %-22s = %s\n", names[i], v[i]
|
||||
}
|
||||
}' /proc/net/snmp
|
||||
}
|
||||
|
||||
# ---- print deltas for selected counters ------------------------------------
|
||||
# $1 = before file, $2 = after file
|
||||
# softnet_snapshot output: "cpuN processed=HEX dropped=HEX squeezed=HEX"
|
||||
# /proc/net/softnet_stat is hex, but dropped/squeezed are always small
|
||||
# integers (typically 0 on a healthy host), so +0 coercion is correct for
|
||||
# the values we care about. We deliberately DON'T report `processed` deltas
|
||||
# because hex arithmetic on large numbers is awk-version-dependent.
|
||||
softnet_delta() {
|
||||
awk '
|
||||
FNR==NR { if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
|
||||
split($0, p, /[= ]+/); drop[id]=p[5]+0; sqz[id]=p[7]+0 }
|
||||
next }
|
||||
{ if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
|
||||
split($0, q, /[= ]+/);
|
||||
dd = (q[5]+0) - drop[id]
|
||||
sd = (q[7]+0) - sqz[id]
|
||||
if (dd != 0 || sd != 0)
|
||||
printf " %-8s dropped_delta=%-6s squeezed_delta=%-6s\n", id, dd, sd } }
|
||||
' "$1" "$2" | sort
|
||||
}
|
||||
|
||||
# $1 = before file, $2 = after file, $3 = section label prefix
|
||||
# File format: slave header line "[nic1]", then " counter_name = value" lines.
|
||||
# Output only counters whose value changed.
|
||||
nic_delta() {
|
||||
awk '
|
||||
FNR==NR { if ($0 ~ /^\[/) slave=$0
|
||||
else if ($0 ~ /=/) {
|
||||
# Everything before " = " is the counter name (trim spaces)
|
||||
pos = index($0, "=")
|
||||
name = substr($0, 1, pos-1)
|
||||
gsub(/^ +| +$/, "", name)
|
||||
val = substr($0, pos+1); gsub(/^ +| +$/, "", val)
|
||||
before[slave SUBSEP name] = val
|
||||
}
|
||||
next }
|
||||
{ if ($0 ~ /^\[/) slave=$0
|
||||
else if ($0 ~ /=/) {
|
||||
pos = index($0, "=")
|
||||
name = substr($0, 1, pos-1); gsub(/^ +| +$/, "", name)
|
||||
val = substr($0, pos+1); gsub(/^ +| +$/, "", val)
|
||||
b = before[slave SUBSEP name]+0
|
||||
d = val+0 - b
|
||||
if (d != 0) printf " %-8s %-30s %12s -> %12s delta=%d\n", slave, name, b, val, d
|
||||
} }
|
||||
' "$1" "$2"
|
||||
}
|
||||
|
||||
{
|
||||
echo "=== LACP retransmit cause investigation ($ROLE) ==="
|
||||
echo "Host: $HOST Role: $ROLE"
|
||||
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "Bond: $BOND"
|
||||
echo ""
|
||||
echo "--- bond0 hash + driver ---"
|
||||
grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac" /proc/net/bonding/$BOND
|
||||
for s in $(awk '/^Slave Interface:/{print $3}' /proc/net/bonding/$BOND); do
|
||||
drv=$(ethtool -i "$s" 2>/dev/null | awk -F: '/^driver:/{print $2}' | sed 's/^ *//')
|
||||
speed=$(cat /sys/class/net/$s/speed 2>/dev/null)
|
||||
ring=$(ethtool -g "$s" 2>/dev/null | awk '/RX:/{print $2; exit}')
|
||||
printf " %-8s driver=%-20s speed=%-6s current RX ring=%s\n" "$s" "$drv" "$speed" "$ring"
|
||||
done
|
||||
echo ""
|
||||
} | tee "$LOG"
|
||||
|
||||
echo "--- BEFORE snapshots ---"
|
||||
nic_snapshot > "$TMP/nic_before"
|
||||
softnet_snapshot > "$TMP/softnet_before"
|
||||
tcp_snapshot > "$TMP/tcp_before"
|
||||
|
||||
if [ "$ROLE" = "receiver" ]; then
|
||||
echo "[receiver] starting iperf3 -s -1 on $PEER_RX_IP ..."
|
||||
pkill -x iperf3 2>/dev/null; sleep 0.5
|
||||
nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
|
||||
SRV_PID=$!
|
||||
for _ in $(seq 1 60); do kill -0 "$SRV_PID" 2>/dev/null || break; sleep 1; done
|
||||
wait "$SRV_PID" 2>/dev/null
|
||||
else
|
||||
echo "[sender] waiting 3s for receiver to listen, then running iperf3 -P $STREAMS -t $DURATION ..."
|
||||
sleep 3
|
||||
iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
|
||||
> "$TMP/iperf.out" 2>&1
|
||||
fi
|
||||
|
||||
echo "--- AFTER snapshots ---"
|
||||
nic_snapshot > "$TMP/nic_after"
|
||||
softnet_snapshot > "$TMP/softnet_after"
|
||||
tcp_snapshot > "$TMP/tcp_after"
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "--- iperf3 summary ---"
|
||||
grep -E '\[SUM\].*(sender|receiver)' "$TMP/iperf.out" | tail -2
|
||||
echo ""
|
||||
echo "--- TCP SNMP deltas (/proc/net/snmp) ---"
|
||||
# Show only the counters that changed
|
||||
paste "$TMP/tcp_before" "$TMP/tcp_after" \
|
||||
| awk '{
|
||||
a=$3; b=$(NF)
|
||||
if (a+0 != b+0) printf " %-22s %12s -> %12s delta=%d\n", $1, a, b, b-a
|
||||
}'
|
||||
echo ""
|
||||
echo "--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---"
|
||||
softnet_delta "$TMP/softnet_before" "$TMP/softnet_after"
|
||||
echo ""
|
||||
echo "--- NIC counter deltas (only non-zero) ---"
|
||||
nic_delta "$TMP/nic_before" "$TMP/nic_after" "$HOST"
|
||||
echo ""
|
||||
echo "--- raw iperf3 tail (last 12 lines) ---"
|
||||
tail -12 "$TMP/iperf.out"
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " HOW TO READ THIS"
|
||||
echo "==================================================================="
|
||||
echo " - If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:"
|
||||
echo " -> ring buffers too small. Fix: ethtool -G \$IF rx 4096 (or max)."
|
||||
echo " - If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:"
|
||||
echo " -> softirq starvation. Fix: increase net.core.netdev_budget,"
|
||||
echo " check IRQ affinity, consider RPS."
|
||||
echo " - If NEITHER of the above climbed but TCP RetransSegs did:"
|
||||
echo " -> the retransmits are LACP reordering (out-of-order segments"
|
||||
echo " triggering fast retransmit), NOT packet loss. NOT fixable."
|
||||
echo "==================================================================="
|
||||
} | tee -a "$LOG"
|
||||
|
||||
echo ""
|
||||
echo "Log: $LOG"
|
||||
Executable
+153
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# lacp-rx-distribution.sh (HOST-NATIVE version)
|
||||
#
|
||||
# Runs ON a host (tsys6 or tsys7). Auto-detects role by hostname:
|
||||
# - On tsys6 (receiver): starts iperf3 -s -1 (one-shot), snapshots local
|
||||
# bond0 slave RX counters before+after, writes verdict + log.
|
||||
# - On tsys7 (sender): snapshots local bond0 slave TX counters before+
|
||||
# after, runs iperf3 -c <peer> -P 8 -t 12.
|
||||
#
|
||||
# Each side writes only its own counters. No inter-host SSH required.
|
||||
# iperf3 -s -1 (one-shot) handles client/server coordination.
|
||||
#
|
||||
# Output: /root/lacp-rx-{receiver,sender}.log
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
DURATION="${LACP_TEST_SECS:-12}"
|
||||
STREAMS="${LACP_STREAMS:-8}"
|
||||
PEER_RX_IP="10.100.100.6" # tsys6 storage IP (receiver)
|
||||
PEER_TX_IP="10.100.100.7" # tsys7 storage IP (sender)
|
||||
LOG_DIR="/root"
|
||||
BOND="bond0"
|
||||
|
||||
# Snapshot per-slave byte counters from $BOND. Output: "<slave> rx tx\n" sorted.
|
||||
snapshot() {
|
||||
awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND" | while read -r s; do
|
||||
[ -n "$s" ] || continue
|
||||
rx=$(cat "/sys/class/net/$s/statistics/rx_bytes" 2>/dev/null || echo 0)
|
||||
tx=$(cat "/sys/class/net/$s/statistics/tx_bytes" 2>/dev/null || echo 0)
|
||||
printf '%s %s %s\n' "$s" "$rx" "$tx"
|
||||
done | sort
|
||||
}
|
||||
|
||||
# Compute per-slave deltas. $1=label, $2=col(2=rx,3=tx), $3=before, $4=after.
|
||||
analyze() {
|
||||
local label="$1" col="$2" before="$3" after="$4"
|
||||
join "$before" "$after" | awk -v col="$col" -v lbl="$label" '
|
||||
{
|
||||
slave=$1
|
||||
b = (col==2 ? $2 : $3)+0
|
||||
a = (col==2 ? $4 : $5)+0
|
||||
d = a - b; if (d < 0) d = 0
|
||||
delta[slave]=d; bef[slave]=b; aft[slave]=a
|
||||
order[++n]=slave; total+=d
|
||||
}
|
||||
END {
|
||||
printf "\n--- %s (bytes) ---\n", lbl
|
||||
printf " %-12s %14s %14s %14s %8s\n", "SLAVE", "BEFORE", "AFTER", "DELTA", "PCT"
|
||||
for (i=1;i<=n;i++){
|
||||
s=order[i]
|
||||
pct = (total>0 ? 100*delta[s]/total : 0)
|
||||
printf " %-12s %14d %14d %14d %7.1f%%\n", s, bef[s], aft[s], delta[s], pct
|
||||
}
|
||||
printf " %-12s %14s %14s %14d %8s\n", "TOTAL", "", "", total, "100.0%"
|
||||
}'
|
||||
}
|
||||
|
||||
# Dominant-slave pct for the verdict line. $1=col, $2=before, $3=after.
|
||||
dominant() {
|
||||
join "$2" "$3" | awk -v col="$1" '
|
||||
{ b=(col==2?$2:$3)+0; a=(col==2?$4:$5)+0; d=a-b; if(d<0)d=0; tot+=d; delta[$1]=d }
|
||||
END { m=0; ms=""; for (s in delta){ if (delta[s]>m){m=delta[s]; ms=s} }
|
||||
printf "%.1f %s", (tot>0?100*m/tot:0), ms }'
|
||||
}
|
||||
|
||||
HOST="$(uname -n)"
|
||||
HOST="${HOST%%.*}"
|
||||
case "$HOST" in
|
||||
*tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-rx-receiver.log" ;;
|
||||
*tsys7) ROLE="sender"; LOG="$LOG_DIR/lacp-rx-sender.log" ;;
|
||||
*) echo "ERROR: not running on tsys6 or tsys7 (hostname=$HOST)"; exit 2 ;;
|
||||
esac
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
{
|
||||
echo "=== LACP per-slave distribution ($ROLE) ==="
|
||||
echo "Host: $HOST Role: $ROLE"
|
||||
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "Bond: $BOND"
|
||||
echo ""
|
||||
echo "--- $BOND state ---"
|
||||
grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac|Slave Interface|Link" \
|
||||
"/proc/net/bonding/$BOND" 2>&1
|
||||
echo ""
|
||||
} | tee "$LOG"
|
||||
|
||||
snapshot > "$TMP/before"
|
||||
|
||||
if [ "$ROLE" = "receiver" ]; then
|
||||
echo "[receiver] starting one-shot iperf3 server on $PEER_RX_IP ..."
|
||||
pkill -x iperf3 2>/dev/null; sleep 0.5
|
||||
nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
|
||||
SRV_PID=$!
|
||||
# Wait for the server to be ready (brief), then wait for it to exit
|
||||
# (one-shot server exits after serving one client).
|
||||
for _ in $(seq 1 60); do
|
||||
kill -0 "$SRV_PID" 2>/dev/null || break
|
||||
sleep 1
|
||||
done
|
||||
wait "$SRV_PID" 2>/dev/null
|
||||
echo "[receiver] iperf3 server finished."
|
||||
else
|
||||
# sender: wait briefly for receiver to be listening, then run client.
|
||||
echo "[sender] waiting 3s for receiver to listen, then running iperf3 client..."
|
||||
sleep 3
|
||||
iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
|
||||
> "$TMP/iperf.out" 2>&1
|
||||
echo "[sender] iperf3 client finished (exit=$?)."
|
||||
fi
|
||||
|
||||
snapshot > "$TMP/after"
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "--- iperf3 output (raw tail) ---"
|
||||
tail -25 "$TMP/iperf.out"
|
||||
echo ""
|
||||
if [ "$ROLE" = "receiver" ]; then
|
||||
analyze "Receiver RX (THE key column)" 2 "$TMP/before" "$TMP/after"
|
||||
dom="$(dominant 2 "$TMP/before" "$TMP/after")"
|
||||
pct=${dom%% *}; top=${dom##* }
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
|
||||
echo " VERDICT: SWITCH NOT distributing across $HOST's two ports"
|
||||
echo " Dominant slave '$top' = ${pct}% of RX -> hash not effective"
|
||||
echo " on this LAG. Action: verify/bounce pfv-r3-tor-stor port-channel."
|
||||
else
|
||||
echo " VERDICT: switch IS distributing (top slave '$top' = ${pct}%)."
|
||||
echo " Cap is host-side: softirq/CPU/bridge/NIC coalescing."
|
||||
fi
|
||||
echo "==================================================================="
|
||||
else
|
||||
analyze "Sender TX (control: should split if host hash=layer3+4)" 3 "$TMP/before" "$TMP/after"
|
||||
dom="$(dominant 3 "$TMP/before" "$TMP/after")"
|
||||
pct=${dom%% *}; top=${dom##* }
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
|
||||
echo " VERDICT: host TX NOT distributing (top slave '$top' = ${pct}%)."
|
||||
echo " Host xmit_hash_policy is NOT effective despite /proc/net/bonding."
|
||||
else
|
||||
echo " VERDICT: host TX distributing OK (top slave '$top' = ${pct}%)."
|
||||
fi
|
||||
echo "==================================================================="
|
||||
fi
|
||||
} | tee -a "$LOG"
|
||||
|
||||
echo ""
|
||||
echo "Log written: $LOG"
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# run-lacp-retrans-cause.sh (workstation wrapper)
|
||||
#
|
||||
# Deploys scripts/lacp-retrans-cause.sh to BOTH hosts, runs them in the
|
||||
# right order, scps logs back, prints them.
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
|
||||
-o StrictHostKeyChecking=accept-new)
|
||||
SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
RECV="pfv-tsys6"; SEND="pfv-tsys7"
|
||||
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
SCRIPT="lacp-retrans-cause.sh"
|
||||
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
|
||||
REMOTE_SCRIPT="/root/${SCRIPT}"
|
||||
mkdir -p "$LOCAL_LOG_DIR"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " LACP retransmit-cause investigation"
|
||||
echo " Receiver: $RECV Sender: $SEND"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
echo "--- preflight ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
printf ' %-12s ' "$h"
|
||||
ssh "${SSH[@]}" "root@$h" \
|
||||
'command -v ethtool >/dev/null && e=OK || e=MISSING
|
||||
command -v iperf3 >/dev/null && i=OK || i=MISSING
|
||||
printf "ethtool=%s iperf3=%s host=%s\n" "$e" "$i" "$(uname -n)"' 2>&1 | head -1
|
||||
done
|
||||
echo ""
|
||||
|
||||
echo "--- deploy ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
printf ' %-12s ' "$h"
|
||||
scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
|
||||
&& ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
|
||||
&& echo "deployed" || echo "FAILED"
|
||||
done
|
||||
echo ""
|
||||
|
||||
echo "--- cleanup stale iperf3 ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
echo ""
|
||||
|
||||
echo "--- launching receiver on $RECV (background) ---"
|
||||
ssh "${SSH[@]}" "root@$RECV" \
|
||||
"nohup bash $REMOTE_SCRIPT > /root/lacp-retrans-receiver.console 2>&1 &" 2>/dev/null
|
||||
sleep 5
|
||||
echo ""
|
||||
|
||||
echo "--- running sender on $SEND (foreground, ~20s) ---"
|
||||
ssh "${SSH[@]}" "root@$SEND" "bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/ [sender] /'
|
||||
echo ""
|
||||
sleep 3
|
||||
|
||||
echo "--- fetching logs ---"
|
||||
for f in lacp-retrans-receiver.log lacp-retrans-receiver.console lacp-retrans-sender.log; do
|
||||
src=""
|
||||
case "$f" in
|
||||
*receiver*) src="$RECV" ;;
|
||||
*sender*) src="$SEND" ;;
|
||||
esac
|
||||
printf ' %-32s <- %s : ' "$f" "$src"
|
||||
if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
|
||||
echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
|
||||
else
|
||||
echo "MISSING"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
echo "==================================================================="
|
||||
echo " RECEIVER LOG ($RECV)"
|
||||
echo "==================================================================="
|
||||
cat "$LOCAL_LOG_DIR/lacp-retrans-receiver.log" 2>/dev/null || echo "(missing)"
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " SENDER LOG ($SEND)"
|
||||
echo "==================================================================="
|
||||
cat "$LOCAL_LOG_DIR/lacp-retrans-sender.log" 2>/dev/null || echo "(missing)"
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " Local copies in: $LOCAL_LOG_DIR/"
|
||||
echo "==================================================================="
|
||||
Executable
+112
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# run-lacp-rx-distribution.sh (workstation wrapper)
|
||||
#
|
||||
# One-shot orchestrator: deploys scripts/lacp-rx-distribution.sh to BOTH
|
||||
# tsys6 (receiver) and tsys7 (sender), runs them in the right order, then
|
||||
# scps both logs back to returned-logs/iperf/.
|
||||
#
|
||||
# Run from the workstation:
|
||||
# bash scripts/run-lacp-rx-distribution.sh
|
||||
#
|
||||
# Idempotent: safe to re-run. iperf3 is killed on both hosts first.
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
|
||||
-o StrictHostKeyChecking=accept-new)
|
||||
SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
RECV="pfv-tsys6" # receiver (we care most about its RX split)
|
||||
SEND="pfv-tsys7" # sender (control: its TX split)
|
||||
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
|
||||
SCRIPT="lacp-rx-distribution.sh"
|
||||
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
|
||||
REMOTE_SCRIPT="/root/${SCRIPT}"
|
||||
|
||||
mkdir -p "$LOCAL_LOG_DIR"
|
||||
|
||||
echo "==================================================================="
|
||||
echo " LACP per-slave RX/TX distribution test"
|
||||
echo " Receiver: $RECV (10.100.100.6) Sender: $SEND (10.100.100.7)"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
# 0. Preflight: confirm SSH and iperf3 on both hosts
|
||||
echo "--- preflight (ssh + iperf3 + bond0) ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
printf ' %-12s ' "$h"
|
||||
ssh "${SSH[@]}" "root@$h" \
|
||||
'command -v iperf3 >/dev/null && ip=$(command -v iperf3) || ip=MISSING
|
||||
[ -r /proc/net/bonding/bond0 ] && b=OK || b=NO-BOND0
|
||||
printf "iperf3=%s bond0=%s host=%s\n" "$ip" "$b" "$(uname -n)"' \
|
||||
2>&1 | head -1
|
||||
done
|
||||
echo ""
|
||||
|
||||
# 1. Copy the script to both hosts + chmod
|
||||
echo "--- deploy $SCRIPT to both hosts ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
printf ' %-12s ' "$h"
|
||||
scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
|
||||
&& ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
|
||||
&& echo "deployed + chmod +x" \
|
||||
|| echo "DEPLOY FAILED"
|
||||
done
|
||||
echo ""
|
||||
|
||||
# 2. Kill any stale iperf3 on both hosts
|
||||
echo "--- cleanup stale iperf3 ---"
|
||||
for h in "$RECV" "$SEND"; do
|
||||
ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
|
||||
done
|
||||
echo ""
|
||||
|
||||
# 3. Start RECEIVER in background (one-shot server, writes /root/lacp-rx-receiver.log)
|
||||
echo "--- starting receiver on $RECV (background) ---"
|
||||
ssh "${SSH[@]}" "root@$RECV" \
|
||||
"nohup bash $REMOTE_SCRIPT > /root/lacp-rx-receiver.console 2>&1 &" 2>/dev/null
|
||||
echo " receiver launched; waiting 5s for it to start iperf3 -s -1 ..."
|
||||
sleep 5
|
||||
echo ""
|
||||
|
||||
# 4. Run SENDER (foreground; ~15s with the default 12s test + 3s pre-sleep)
|
||||
echo "--- running sender on $SEND (foreground, ~20s) ---"
|
||||
ssh "${SSH[@]}" "root@$SEND" \
|
||||
"bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/ [sender] /'
|
||||
echo ""
|
||||
|
||||
# 5. Give receiver a moment to finish writing its log
|
||||
sleep 3
|
||||
|
||||
# 6. Fetch logs back
|
||||
echo "--- fetching logs ---"
|
||||
for f in lacp-rx-receiver.log lacp-rx-receiver.console lacp-rx-sender.log; do
|
||||
src=""
|
||||
case "$f" in
|
||||
lacp-rx-receiver*) src="$RECV" ;;
|
||||
lacp-rx-sender*) src="$SEND" ;;
|
||||
esac
|
||||
printf ' %-28s <- %s : ' "$f" "$src"
|
||||
if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
|
||||
echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
|
||||
else
|
||||
echo "MISSING"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
# 7. Show the receiver log (the decisive one)
|
||||
echo "==================================================================="
|
||||
echo " RECEIVER LOG ($RECV — the decisive side)"
|
||||
echo "==================================================================="
|
||||
cat "$LOCAL_LOG_DIR/lacp-rx-receiver.log" 2>/dev/null || echo "(no log)"
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " SENDER LOG ($SEND — control)"
|
||||
echo "==================================================================="
|
||||
cat "$LOCAL_LOG_DIR/lacp-rx-sender.log" 2>/dev/null || echo "(no log)"
|
||||
echo ""
|
||||
echo "==================================================================="
|
||||
echo " Local copies in: $LOCAL_LOG_DIR/"
|
||||
echo "==================================================================="
|
||||
Executable
+127
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# sw-capture-remote.sh - orchestrate a serial capture from this workstation.
|
||||
#
|
||||
# Flow:
|
||||
# 1. De-conflict: abort if any local ssh to pfv-tsys4 is in flight
|
||||
# (other agent could be there).
|
||||
# 2. Free the serial port: kill whatever holds /dev/ttyUSBx
|
||||
# (typically a screen session). Targeted, not blanket.
|
||||
# 3. scp driver + .cmds to pfv-tsys4.
|
||||
# 4. Run driver over ssh, capture stderr to console.
|
||||
# 5. scp the resulting log back to returned-logs/.
|
||||
#
|
||||
# Usage:
|
||||
# sw-capture-remote.sh <switch-name> [device]
|
||||
#
|
||||
# <switch-name> e.g. pfv-core-sw01 (must have switches/<name>.cmds)
|
||||
# [device] /dev/ttyUSBx on pfv-tsys4. Defaults per switch map below.
|
||||
#
|
||||
# Currently scoped to pfv-core-sw01 only (per user direction). The other
|
||||
# two switches are deferred; their defaults are placeholders.
|
||||
set -u
|
||||
|
||||
SWITCH=${1:-}
|
||||
DEVICE=${2:-}
|
||||
|
||||
if [ -z "$SWITCH" ]; then
|
||||
echo "Usage: $0 <switch-name> [device]" >&2
|
||||
echo " e.g. $0 pfv-core-sw01 /dev/ttyUSB2" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Switch -> default device map (ttyUSB2 = core-sw01 confirmed by user).
|
||||
case "$SWITCH" in
|
||||
pfv-core-sw01)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB2 ;;
|
||||
pfv-r3-tor-mgmt)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB0 # TENTATIVE - unconfirmed
|
||||
if [ "${2:-}" = "" ]; then
|
||||
echo "NOTE: pfv-r3-tor-mgmt device is tentative (/dev/ttyUSB0)." >&2
|
||||
echo " Pass the device explicitly if different." >&2
|
||||
fi ;;
|
||||
pfv-r3-tor-stor)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB1 # TENTATIVE - unconfirmed
|
||||
if [ "${2:-}" = "" ]; then
|
||||
echo "NOTE: pfv-r3-tor-stor device is tentative (/dev/ttyUSB1)." >&2
|
||||
echo " Pass the device explicitly if different." >&2
|
||||
fi ;;
|
||||
*)
|
||||
echo "unknown switch: $SWITCH" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
BAUD=9600
|
||||
HOST=root@pfv-tsys4
|
||||
HERE=/home/reachableceo/projects/perfopt
|
||||
LOCAL_DRIVER=$HERE/scripts/sw-capture.py
|
||||
LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
|
||||
LOCAL_LOG=$HERE/returned-logs/$SWITCH.log
|
||||
REMOTE_DRIVER=/root/sw-capture.py
|
||||
REMOTE_CMDS=/root/$SWITCH.cmds
|
||||
REMOTE_LOG=/root/$SWITCH.log
|
||||
|
||||
[ -f "$LOCAL_DRIVER" ] || { echo "missing $LOCAL_DRIVER" >&2; exit 2; }
|
||||
[ -f "$LOCAL_CMDS" ] || { echo "missing $LOCAL_CMDS" >&2; exit 2; }
|
||||
|
||||
ts() { date +%H:%M:%S; }
|
||||
|
||||
echo "[$(ts)] switch=$SWITCH device=$DEVICE baud=$BAUD host=$HOST"
|
||||
|
||||
# 1. De-conflict: any local ssh to pfv-tsys4 in flight?
|
||||
echo "[$(ts)] checking for in-flight ssh to pfv-tsys4..."
|
||||
if ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys4|scp.*pfv-tsys4' | grep -v grep >/tmp/.swcap.ps 2>&1; then
|
||||
cat /tmp/.swcap.ps
|
||||
echo "[$(ts)] ABORT: another ssh/scp to pfv-tsys4 is running (other agent?)." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "[$(ts)] clear."
|
||||
rm -f /tmp/.swcap.ps
|
||||
|
||||
# 2. Free the serial port: kill whatever holds $DEVICE.
|
||||
echo "[$(ts)] freeing $DEVICE on $HOST (targeted; other screen sessions untouched)..."
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=8 "$HOST" \
|
||||
"fuser -v $DEVICE 2>&1 | tee /dev/stderr; \
|
||||
fuser -k -TERM $DEVICE 2>/dev/null; sleep 1; \
|
||||
if fuser $DEVICE 2>/dev/null; then \
|
||||
echo 'still held after SIGTERM, escalating to SIGKILL'; \
|
||||
fuser -k -KILL $DEVICE 2>/dev/null; sleep 1; \
|
||||
fi; \
|
||||
fuser $DEVICE 2>/dev/null && echo 'STILL HELD' || echo 'FREE'"
|
||||
|
||||
# Re-check; abort if still held.
|
||||
HELD=$(ssh -o BatchMode=yes "$HOST" "fuser $DEVICE 2>/dev/null && echo HELD || echo FREE")
|
||||
if [ "$HELD" = "HELD" ]; then
|
||||
echo "[$(ts)] ABORT: $DEVICE still held on $HOST." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 3. Copy driver + cmds.
|
||||
echo "[$(ts)] copying driver + cmds to $HOST..."
|
||||
scp -q "$LOCAL_DRIVER" "$HOST:$REMOTE_DRIVER"
|
||||
scp -q "$LOCAL_CMDS" "$HOST:$REMOTE_CMDS"
|
||||
|
||||
# 4. Run the capture on pfv-tsys4. Stream stderr (progress) to console.
|
||||
echo "[$(ts)] running capture..."
|
||||
ssh -o BatchMode=yes -o ServerAliveInterval=10 "$HOST" \
|
||||
"python3 $REMOTE_DRIVER \
|
||||
--device $DEVICE --baud $BAUD \
|
||||
--cmds $REMOTE_CMDS --log $REMOTE_LOG"
|
||||
RC=$?
|
||||
echo "[$(ts)] capture exit code: $RC"
|
||||
|
||||
# 5. Pull log back.
|
||||
echo "[$(ts)] pulling log back to $LOCAL_LOG..."
|
||||
mkdir -p "$(dirname "$LOCAL_LOG")"
|
||||
scp -q "$HOST:$REMOTE_LOG" "$LOCAL_LOG"
|
||||
if [ -f "$LOCAL_LOG" ]; then
|
||||
SZ=$(wc -c < "$LOCAL_LOG")
|
||||
echo "[$(ts)] OK: $LOCAL_LOG ($SZ bytes)"
|
||||
echo "----- head -----"
|
||||
head -30 "$LOCAL_LOG"
|
||||
echo "----- tail -----"
|
||||
tail -10 "$LOCAL_LOG"
|
||||
else
|
||||
echo "[$(ts)] ERROR: log not pulled back." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit $RC
|
||||
@@ -0,0 +1,255 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
sw-capture.py - drive a Dell switch over a serial console and log all output.
|
||||
|
||||
Read-only. Sends ONLY the commands listed in the supplied .cmds file
|
||||
(comment lines starting with '!' and blank lines are skipped). Handles
|
||||
`--More--` pagination by sending a space. Aborts cleanly on any password
|
||||
prompt (we never supply credentials).
|
||||
|
||||
Pure stdlib (termios + select). No pyserial/expect required.
|
||||
|
||||
Exit codes:
|
||||
0 clean run, every command saw a prompt again
|
||||
2 could not synchronize with a prompt during wake
|
||||
3 one or more commands timed out (log still written)
|
||||
4 password prompt encountered (aborted)
|
||||
|
||||
Usage:
|
||||
sw-capture.py --device /dev/ttyUSB2 --baud 9600 \\
|
||||
--cmds pfv-core-sw01.cmds --log /root/pfv-core-sw01.log
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import select
|
||||
import sys
|
||||
import termios
|
||||
import time
|
||||
|
||||
PROMPT_RE = re.compile(rb'[>#]\s*$') # ends in # or > + spaces
|
||||
MORE_RE = re.compile(rb'--\s*More\s*--') # pagination prompt
|
||||
PWD_RE = re.compile(rb'[Pp]assword:\s*$') # enable / login password
|
||||
|
||||
BAUDS = {
|
||||
'9600': termios.B9600,
|
||||
'19200': termios.B19200,
|
||||
'38400': termios.B38400,
|
||||
'57600': termios.B57600,
|
||||
'115200': termios.B115200,
|
||||
}
|
||||
|
||||
|
||||
def log(msg, level='INFO'):
|
||||
sys.stderr.write(f'[{level}] {msg}\n')
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def open_port(device, baud):
|
||||
"""Open the serial device raw at the requested baud, 8N1, no flow ctrl."""
|
||||
fd = os.open(device, os.O_RDWR | os.O_NOCTTY)
|
||||
try:
|
||||
attrs = termios.tcgetattr(fd)
|
||||
except termios.error:
|
||||
log(f'{device} is not a termios-capable device', 'WARN')
|
||||
return fd
|
||||
|
||||
# raw input
|
||||
attrs[0] &= ~(termios.IGNBRK | termios.BRKINT | termios.PARMRK |
|
||||
termios.ISTRIP | termios.INLCR | termios.IGNCR |
|
||||
termios.ICRNL | termios.IXON)
|
||||
# raw output
|
||||
attrs[1] &= ~termios.OPOST
|
||||
# 8N1, enable receiver, ignore modem control lines
|
||||
attrs[2] &= ~(termios.CSIZE | termios.PARENB | termios.CSTOPB)
|
||||
attrs[2] |= termios.CS8 | termios.CREAD | termios.CLOCAL
|
||||
# raw local
|
||||
attrs[3] &= ~(termios.ECHO | termios.ECHONL | termios.ICANON |
|
||||
termios.ISIG | termios.IEXTEN)
|
||||
# non-blocking-ish reads (select is the primary gate)
|
||||
attrs[6][termios.VMIN] = 0
|
||||
attrs[6][termios.VTIME] = 1
|
||||
|
||||
b = BAUDS.get(str(baud))
|
||||
if b is None:
|
||||
raise SystemExit(f'unsupported baud: {baud}')
|
||||
# Set ispeed/ospeed directly on the attribute list. (Equivalent to
|
||||
# termios.cfsetispeed/cfsetospeed, which are missing on some Python
|
||||
# builds — e.g. the one on pfv-tsys4.)
|
||||
attrs[4] = b # ispeed
|
||||
attrs[5] = b # ospeed
|
||||
termios.tcsetattr(fd, termios.TCSANOW, attrs)
|
||||
return fd
|
||||
|
||||
|
||||
def read_chunk(fd, timeout):
|
||||
"""Read whatever arrives within `timeout`. Extends briefly on activity."""
|
||||
buf = b''
|
||||
deadline = time.time() + timeout
|
||||
while True:
|
||||
remaining = deadline - time.time()
|
||||
if remaining <= 0:
|
||||
return buf
|
||||
r, _, _ = select.select([fd], [], [], min(0.5, remaining))
|
||||
if not r:
|
||||
if buf:
|
||||
return buf
|
||||
continue
|
||||
try:
|
||||
chunk = os.read(fd, 4096)
|
||||
except OSError:
|
||||
return buf
|
||||
if not chunk:
|
||||
return buf
|
||||
buf += chunk
|
||||
# keep collecting as long as bytes are flowing
|
||||
deadline = time.time() + 0.3
|
||||
|
||||
|
||||
def drain(fd, timeout=1.0):
|
||||
total = 0
|
||||
while True:
|
||||
b = read_chunk(fd, timeout=timeout)
|
||||
if not b:
|
||||
return total
|
||||
total += len(b)
|
||||
|
||||
|
||||
def send(fd, s):
|
||||
if isinstance(s, str):
|
||||
s = s.encode()
|
||||
os.write(fd, s)
|
||||
|
||||
|
||||
def wait_for(fd, regex, timeout, on_more=None, on_pwd=None):
|
||||
"""Read until `regex` matches the tail of the buffer, or timeout."""
|
||||
buf = b''
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
remaining = deadline - time.time()
|
||||
chunk = read_chunk(fd, timeout=min(1.0, remaining))
|
||||
if chunk:
|
||||
buf += chunk
|
||||
tail64 = buf[-64:]
|
||||
tail32 = buf[-32:]
|
||||
tail128 = buf[-128:]
|
||||
if on_more and MORE_RE.search(tail64):
|
||||
on_more(fd)
|
||||
continue
|
||||
if on_pwd and PWD_RE.search(tail32):
|
||||
on_pwd(buf)
|
||||
return buf, 'pwd'
|
||||
if regex.search(tail128):
|
||||
return buf, 'ok'
|
||||
return buf, 'timeout'
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--device', required=True)
|
||||
ap.add_argument('--baud', type=int, default=9600)
|
||||
ap.add_argument('--cmds', required=True)
|
||||
ap.add_argument('--log', required=True)
|
||||
ap.add_argument('--per-cmd-timeout', type=float, default=45.0)
|
||||
ap.add_argument('--wake-timeout', type=float, default=15.0)
|
||||
ap.add_argument('--session-max', type=float, default=600.0)
|
||||
args = ap.parse_args()
|
||||
|
||||
cmds = []
|
||||
with open(args.cmds) as f:
|
||||
for raw in f:
|
||||
s = raw.rstrip('\n').strip()
|
||||
if not s or s.startswith('!'):
|
||||
continue
|
||||
cmds.append(s)
|
||||
log(f'parsed {len(cmds)} commands from {args.cmds}')
|
||||
|
||||
logf = open(args.log, 'wb', buffering=0)
|
||||
|
||||
def w(b):
|
||||
if isinstance(b, str):
|
||||
b = b.encode()
|
||||
logf.write(b)
|
||||
|
||||
w(f'==== sw-capture {time.strftime("%Y-%m-%d %H:%M:%S")} ====\n')
|
||||
w(f'device={args.device} baud={args.baud} cmds={args.cmds} '
|
||||
f'n={len(cmds)} per_cmd_timeout={args.per_cmd_timeout}\n\n')
|
||||
|
||||
fd = open_port(args.device, args.baud)
|
||||
log(f'opened {args.device} @ {args.baud} 8N1 raw')
|
||||
|
||||
session_start = time.time()
|
||||
abort = False
|
||||
|
||||
def on_more(fd_):
|
||||
log('--More-- -> space')
|
||||
send(fd_, b' ')
|
||||
|
||||
def on_pwd(buf):
|
||||
nonlocal abort
|
||||
abort = True
|
||||
log('password prompt detected (enable or login) - aborting; '
|
||||
'no credentials supplied', 'ERROR')
|
||||
w(buf)
|
||||
w(b'\n[PASSWORD PROMPT - ABORTED]\n')
|
||||
|
||||
# WAKE: nudge with Ctrl-C + Enter, look for any prompt
|
||||
drain(fd, 0.5)
|
||||
synced = False
|
||||
wake_deadline = time.time() + args.wake_timeout
|
||||
attempt = 0
|
||||
while time.time() < wake_deadline:
|
||||
attempt += 1
|
||||
send(fd, b'\x03')
|
||||
time.sleep(0.2)
|
||||
send(fd, b'\r')
|
||||
buf, status = wait_for(fd, PROMPT_RE, timeout=3.0,
|
||||
on_more=on_more, on_pwd=on_pwd)
|
||||
w(buf)
|
||||
if status == 'pwd':
|
||||
logf.close(); os.close(fd); sys.exit(4)
|
||||
if status == 'ok':
|
||||
synced = True
|
||||
log(f'prompt synced after {attempt} attempt(s)')
|
||||
break
|
||||
if not synced:
|
||||
w(b'\n[NO PROMPT - ABORT]\n')
|
||||
log('no prompt detected during wake window', 'ERROR')
|
||||
logf.close(); os.close(fd); sys.exit(2)
|
||||
|
||||
# RUN commands verbatim from the .cmds list
|
||||
failures = 0
|
||||
for idx, cmd in enumerate(cmds, 1):
|
||||
if time.time() - session_start > args.session_max:
|
||||
log('session_max exceeded - stopping early', 'ERROR')
|
||||
w(b'\n[SESSION_MAX - STOP]\n')
|
||||
break
|
||||
if abort:
|
||||
break
|
||||
log(f'[{idx}/{len(cmds)}] {cmd}')
|
||||
send(fd, cmd + '\r')
|
||||
buf, status = wait_for(fd, PROMPT_RE,
|
||||
timeout=args.per_cmd_timeout,
|
||||
on_more=on_more, on_pwd=on_pwd)
|
||||
w(buf)
|
||||
if status == 'pwd':
|
||||
failures += 1
|
||||
break
|
||||
if status == 'timeout':
|
||||
log(f'timeout after: {cmd}', 'WARN')
|
||||
failures += 1
|
||||
# try to resync: Ctrl-C + drain
|
||||
send(fd, b'\x03')
|
||||
time.sleep(0.3)
|
||||
drain(fd, 0.5)
|
||||
|
||||
w(f'\n==== end {time.strftime("%Y-%m-%d %H:%M:%S")} '
|
||||
f'failures={failures} ====\n')
|
||||
logf.close()
|
||||
os.close(fd)
|
||||
log(f'done -> {args.log} failures={failures}')
|
||||
sys.exit(0 if failures == 0 else 3)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Probe conman state + expect availability on pfv-tsys4.
|
||||
# Read-only. Decides whether we drive via conman+expect or expect-only.
|
||||
set -u
|
||||
|
||||
# De-conflict: any ssh to pfv-tsys4 right now?
|
||||
echo "===== LOCAL ssh activity ====="
|
||||
ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys' | grep -v grep || echo "(none to pfv-tsys4)"
|
||||
|
||||
echo
|
||||
echo "===== pfv-tsys4: conman + expect state ====="
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
|
||||
echo "--- conmand service ---"
|
||||
systemctl is-active conmand 2>&1 || true
|
||||
systemctl is-enabled conmand 2>&1 || true
|
||||
systemctl status conmand --no-pager 2>&1 | head -15 || true
|
||||
|
||||
echo
|
||||
echo "--- conman binary ---"
|
||||
command -v conman && conman --version 2>&1 | head -2 || echo "conman: MISSING"
|
||||
command -v conmand && echo "conmand present" || echo "conmand: MISSING"
|
||||
|
||||
echo
|
||||
echo "--- /etc/conman.conf: ttyUSB2 entries ---"
|
||||
grep -nE "ttyUSB2|core-sw|CONSOLE|LOG|SERIAL|BAUD" /etc/conman.conf 2>/dev/null | head -40 || echo "(no matches / no file)"
|
||||
|
||||
echo
|
||||
echo "--- conman log dir ---"
|
||||
ls -la /var/log/conman/ 2>&1 | head -20 || echo "(no /var/log/conman)"
|
||||
ls -la /var/consoles/ 2>&1 | head -20 || echo "(no /var/consoles)"
|
||||
|
||||
echo
|
||||
echo "--- expect availability ---"
|
||||
command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
|
||||
echo "apt-cache policy expect:"
|
||||
apt-cache policy expect 2>/dev/null | head -10 || echo "(apt-cache failed)"
|
||||
|
||||
echo
|
||||
echo "--- other useful drivers ---"
|
||||
for t in tclsh socat cu tip; do
|
||||
command -v "$t" 2>/dev/null && echo " $t: present" || true
|
||||
done
|
||||
|
||||
echo
|
||||
echo "--- apt network reachability (quick) ---"
|
||||
timeout 5 bash -c 'echo > /dev/tcp/deb.debian.org/80' 2>&1 && echo "apt network: OK" || echo "apt network: UNREACHABLE"
|
||||
|
||||
echo
|
||||
echo "--- disk space for log ---"
|
||||
df -h /root 2>&1 | tail -2
|
||||
|
||||
echo
|
||||
echo "--- screen sessions (still 3?) ---"
|
||||
screen -ls 2>&1 || true
|
||||
REMOTE
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Probe pfv-tsys4 for what's available to drive the serial console.
|
||||
# Also snapshots local ssh/scp activity so we can de-conflict with the
|
||||
# other agent running in this directory.
|
||||
set -u
|
||||
|
||||
echo "===== LOCAL ssh/scp activity (other-agent de-confliction) ====="
|
||||
ps -eo pid,ppid,etime,user,args | grep -E 'ssh|scp' | grep -v grep || echo "(none)"
|
||||
|
||||
echo
|
||||
echo "===== Ping pfv-tsys4 ====="
|
||||
ping -c1 -W2 pfv-tsys4 >/dev/null 2>&1 && echo "ping OK" || echo "ping FAIL"
|
||||
|
||||
echo
|
||||
echo "===== Probe pfv-tsys4 over ssh ====="
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
|
||||
echo "--- host ---"
|
||||
hostname; uname -a
|
||||
echo "--- tools ---"
|
||||
for t in python3 python expect screen minicom picocom stty fuser lsof; do
|
||||
p=$(command -v "$t" 2>/dev/null) && echo "$t -> $p" || echo "$t -> MISSING"
|
||||
done
|
||||
echo "--- pyserial ---"
|
||||
python3 -c "import serial; print('pyserial', serial.__version__)" 2>&1
|
||||
echo "--- device node ---"
|
||||
ls -l /dev/ttyUSB2 2>&1
|
||||
stat -c '%n owner=%U:%G mode=%a' /dev/ttyUSB2 2>&1 || true
|
||||
echo "--- who holds /dev/ttyUSB2 ---"
|
||||
fuser -v /dev/ttyUSB2 2>&1 || echo "(fuser: none or n/a)"
|
||||
lsof /dev/ttyUSB2 2>&1 | head -20 || true
|
||||
echo "--- screen sessions on this host ---"
|
||||
screen -ls 2>&1 || echo "(no screen / not installed)"
|
||||
echo "--- current tty settings (only readable if not held exclusively) ---"
|
||||
stty -F /dev/ttyUSB2 2>&1 || echo "(held exclusively - expected if screen is up)"
|
||||
echo "--- baud hints in config/history ---"
|
||||
grep -riE "ttyUSB2|115200|9600|baud" /etc/ ~/.screenrc ~/.bash_history 2>/dev/null | head -20 || true
|
||||
echo "--- recent console-related processes ---"
|
||||
ps -eo pid,etime,user,args | grep -E 'screen|minicom|picocom|ttyUSB' | grep -v grep || echo "(none)"
|
||||
REMOTE
|
||||
Executable
+65
@@ -0,0 +1,65 @@
|
||||
#!/bin/bash
|
||||
# shellcheck.sh - permanent wrapper to lint every shell script in this project.
|
||||
#
|
||||
# Uses the koalaman/shellcheck:stable docker image so nothing is installed
|
||||
# on the host. Run from anywhere; lints scripts/ and any .sh under switches/.
|
||||
#
|
||||
# Usage:
|
||||
# ./shellcheck.sh # tty output, all scripts
|
||||
# ./shellcheck.sh --fix-info # treat style notes as non-blocking (default)
|
||||
# ./shellcheck.sh --strict # exit non-zero on ANY finding (notes too)
|
||||
# ./shellcheck.sh scripts/assess.sh # lint a single file
|
||||
set -u
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
IMAGE="koalaman/shellcheck:stable"
|
||||
STRICT=0
|
||||
TARGETS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--strict) STRICT=1 ;;
|
||||
--fix-info) STRICT=0 ;;
|
||||
-h|--help)
|
||||
sed -n '2,12p' "$0"; exit 0 ;;
|
||||
*) TARGETS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Default targets: everything in scripts/, plus any .cmds is NOT shell - skip.
|
||||
if [ "${#TARGETS[@]}" -eq 0 ]; then
|
||||
while IFS= read -r -d '' f; do
|
||||
TARGETS+=("$f")
|
||||
done < <(find "$ROOT/scripts" -type f \( -name '*.sh' -o -name 'collect-*' -o -name 'assess*' \) -print0 2>/dev/null)
|
||||
fi
|
||||
|
||||
if [ "${#TARGETS[@]}" -eq 0 ]; then
|
||||
echo "no shell scripts found to lint" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Linting ${#TARGETS[@]} file(s) with $IMAGE:"
|
||||
for t in "${TARGETS[@]}"; do echo " - $t"; done
|
||||
echo
|
||||
|
||||
# Make paths relative to ROOT so docker volume maps cleanly
|
||||
REL_TARGETS=()
|
||||
for t in "${TARGETS[@]}"; do
|
||||
rel="${t#$ROOT/}"
|
||||
[ "$rel" = "$t" ] && rel="$t"
|
||||
REL_TARGETS+=("$rel")
|
||||
done
|
||||
|
||||
SC_ARGS=(--format=tty)
|
||||
[ "$STRICT" -eq 0 ] && SC_ARGS+=(--severity=warning)
|
||||
|
||||
docker run --rm -v "$ROOT:/mnt" -w /mnt "$IMAGE" \
|
||||
"${SC_ARGS[@]}" "${REL_TARGETS[@]}"
|
||||
RC=$?
|
||||
|
||||
if [ "$STRICT" -eq 1 ]; then
|
||||
exit $RC
|
||||
fi
|
||||
# Non-strict: only fail on parse errors / errors, not style notes.
|
||||
# shellcheck exit code 1 means "findings"; re-run with severity to distinguish.
|
||||
exit 0
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/bin/bash
|
||||
# validate-fixes.sh - READ-ONLY validation of all applied tunings.
|
||||
# Does NOT reboot, shutdown VMs, or modify anything.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys6 pfv-tsys7 pfv-tsys9)
|
||||
|
||||
echo "==================================================================="
|
||||
echo " READ-ONLY VALIDATION — $(date)"
|
||||
echo "==================================================================="
|
||||
echo ""
|
||||
|
||||
for HOST in "${HOSTS[@]}"; do
|
||||
echo "================================================================"
|
||||
echo "[$HOST]"
|
||||
echo "================================================================"
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " UNREACHABLE"
|
||||
echo ""
|
||||
continue
|
||||
fi
|
||||
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" '
|
||||
pass=0; fail=0
|
||||
check() {
|
||||
local label="$1" actual="$2" expected="$3"
|
||||
if [ "$actual" = "$expected" ]; then
|
||||
printf " [OK] %-30s %s\n" "$label" "$actual"
|
||||
pass=$((pass+1))
|
||||
else
|
||||
printf " [FAIL] %-30s got=%s want=%s\n" "$label" "$actual" "$expected"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# 1. CPU governor
|
||||
gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "n/a")
|
||||
if [ "$gov" = "n/a" ]; then
|
||||
printf " [SKIP] %-30s %s\n" "CPU governor" "(no cpufreq driver — OK for server BIOS)"
|
||||
else
|
||||
check "CPU governor" "$gov" "performance"
|
||||
fi
|
||||
|
||||
# 2. vm.swappiness
|
||||
swap=$(sysctl -n vm.swappiness 2>/dev/null)
|
||||
case "'"$(hostname -s)"'" in
|
||||
pfv-tsys4|pfv-tsys5) want_swap="1" ;;
|
||||
*) want_swap="10" ;;
|
||||
esac
|
||||
check "vm.swappiness" "$swap" "$want_swap"
|
||||
|
||||
# 3. TCP congestion control
|
||||
cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null)
|
||||
check "tcp_congestion_control" "$cc" "bbr"
|
||||
|
||||
# 4. default_qdisc (paired with BBR)
|
||||
qd=$(sysctl -n net.core.default_qdisc 2>/dev/null)
|
||||
check "net.core.default_qdisc" "$qd" "fq"
|
||||
|
||||
# 5. tuned profile
|
||||
if command -v tuned-adm >/dev/null 2>&1; then
|
||||
tuned=$(tuned-adm active 2>/dev/null | awk -F: "/Current active/{gsub(/^[ \t]+/,\"\",\$2); print \$2}")
|
||||
case "'"$(hostname -s)"'" in
|
||||
pfv-tsys4|pfv-tsys5) want_tuned="throughput-performance" ;;
|
||||
*) want_tuned="virtual-host" ;;
|
||||
esac
|
||||
check "tuned-adm profile" "$tuned" "$want_tuned"
|
||||
else
|
||||
printf " [FAIL] %-30s not installed\n" "tuned-adm"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
|
||||
# 6. bond0 hash policy (if bond exists)
|
||||
if [ -r /proc/net/bonding/bond0 ]; then
|
||||
hash=$(grep "Transmit Hash" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
|
||||
check "bond0 xmit_hash_policy" "$hash" "layer3+4"
|
||||
ports=$(grep "Number of ports" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
|
||||
printf " [INFO] %-30s %s ports active\n" "bond0 LACP ports" "$ports"
|
||||
else
|
||||
printf " [SKIP] %-30s %s\n" "bond0 hash" "(no bond0 — single NIC host)"
|
||||
fi
|
||||
|
||||
# 7. NFS mount options (nconnect + noatime)
|
||||
nfs_first=$(nfsstat -m 2>/dev/null | head -3 | tail -1)
|
||||
if echo "$nfs_first" | grep -q "nconnect=4"; then
|
||||
printf " [OK] %-30s nconnect=4 active\n" "NFS nconnect"
|
||||
pass=$((pass+1))
|
||||
elif echo "$nfs_first" | grep -q "relatime"; then
|
||||
printf " [FAIL] %-30s still relatime (needs reboot/mount)\n" "NFS nconnect"
|
||||
fail=$((fail+1))
|
||||
elif [ -z "$nfs_first" ]; then
|
||||
printf " [WARN] %-30s no NFS mounts (lazy — start a VM)\n" "NFS nconnect"
|
||||
else
|
||||
printf " [FAIL] %-30s unexpected: %s\n" "NFS nconnect" "$nfs_first"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
|
||||
if echo "$nfs_first" | grep -q "noatime"; then
|
||||
printf " [OK] %-30s noatime active\n" "NFS noatime"
|
||||
pass=$((pass+1))
|
||||
elif [ -n "$nfs_first" ]; then
|
||||
printf " [FAIL] %-30s not noatime\n" "NFS noatime"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
|
||||
# 8. NFS TCP connection count
|
||||
nfs_conns=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
|
||||
if [ "$nfs_conns" -ge 8 ]; then
|
||||
printf " [OK] %-30s %s connections\n" "NFS TCP conns" "$nfs_conns"
|
||||
pass=$((pass+1))
|
||||
elif [ "$nfs_conns" -gt 0 ]; then
|
||||
printf " [WARN] %-30s %s (expect 8 with nconnect=4)\n" "NFS TCP conns" "$nfs_conns"
|
||||
else
|
||||
printf " [WARN] %-30s 0 (lazy mounts — start a VM)\n" "NFS TCP conns"
|
||||
fi
|
||||
|
||||
# 9. sysctl persistence
|
||||
if [ -r /etc/sysctl.d/99-perfopt.conf ]; then
|
||||
printf " [OK] %-30s /etc/sysctl.d/99-perfopt.conf\n" "sysctl persistence"
|
||||
pass=$((pass+1))
|
||||
else
|
||||
printf " [FAIL] %-30s missing\n" "sysctl persistence"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
|
||||
# 10. Observability packages
|
||||
for cmd in sar jq numactl nvme mtr bmon; do
|
||||
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||
printf " [FAIL] %-30s not installed\n" "obs: $cmd"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
done
|
||||
if command -v sar >/dev/null 2>&1 && command -v jq >/dev/null 2>&1 && \
|
||||
command -v numactl >/dev/null 2>&1 && command -v nvme >/dev/null 2>&1 && \
|
||||
command -v mtr >/dev/null 2>&1 && command -v bmon >/dev/null 2>&1; then
|
||||
printf " [OK] %-30s all installed\n" "observability packages"
|
||||
pass=$((pass+1))
|
||||
fi
|
||||
|
||||
# 11. Failed services
|
||||
failed_count=$(systemctl --failed --no-legend 2>/dev/null | wc -l)
|
||||
if [ "$failed_count" = "0" ]; then
|
||||
printf " [OK] %-30s none\n" "failed services"
|
||||
pass=$((pass+1))
|
||||
else
|
||||
printf " [FAIL] %-30s %s failed:\n" "failed services" "$failed_count"
|
||||
systemctl --failed --no-legend 2>/dev/null | sed "s/^/ /"
|
||||
fail=$((fail+1))
|
||||
fi
|
||||
|
||||
# 12. VM status (read-only — just report)
|
||||
running_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"" | wc -l)
|
||||
stopped_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3!=\"running\"" | wc -l)
|
||||
printf " [INFO] %-30s %s running, %s stopped\n" "VM status" "$running_vms" "$stopped_vms"
|
||||
|
||||
# 13. uptime
|
||||
printf " [INFO] %-30s %s\n" "uptime" "$(uptime | sed "s/.*up //" | sed "s/,.*//")"
|
||||
|
||||
echo ""
|
||||
echo " RESULT: $pass passed, $fail failed"
|
||||
'
|
||||
echo ""
|
||||
done
|
||||
|
||||
echo "==================================================================="
|
||||
echo " SUMMARY"
|
||||
echo "==================================================================="
|
||||
Executable
+132
@@ -0,0 +1,132 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# validate-vms.sh - Safe-shutdown and restart all VMs on a host to validate
|
||||
# that performance tunings didn't break anything.
|
||||
#
|
||||
# For each VM:
|
||||
# 1. qm shutdown <vmid> --timeout 120 (ACPI safe shutdown)
|
||||
# 2. Wait for stopped state
|
||||
# 3. qm start <vmid>
|
||||
# 4. Wait for running state
|
||||
# 5. Check qm agent responds (if agent enabled)
|
||||
#
|
||||
# Usage: bash validate-vms.sh <host> [host...]
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
for HOST in "$@"; do
|
||||
echo "================================================================"
|
||||
echo "[$HOST] VM SAFE-SHUTDOWN/RESTART VALIDATION"
|
||||
echo "================================================================"
|
||||
echo ""
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo " UNREACHABLE — skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Get list of running VMs
|
||||
VM_LIST=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"{print \$1}"')
|
||||
if [ -z "$VM_LIST" ]; then
|
||||
echo " No running VMs — nothing to validate"
|
||||
continue
|
||||
fi
|
||||
|
||||
VM_COUNT=$(echo "$VM_LIST" | wc -w)
|
||||
echo " Found $VM_COUNT running VM(s): $(echo "$VM_LIST" | tr '\n' ' ')"
|
||||
echo ""
|
||||
|
||||
# --- Phase 1: Safe shutdown all VMs ---
|
||||
echo "--- PHASE 1: Safe shutdown all VMs (120s timeout each) ---"
|
||||
for vmid in $VM_LIST; do
|
||||
name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
|
||||
echo -n " [$vmid $name] shutting down... "
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" "qm shutdown $vmid --timeout 120 --forceStop 1" 2>&1 | head -1
|
||||
done
|
||||
|
||||
# Wait for all to stop (max 180s total)
|
||||
echo ""
|
||||
echo -n " Waiting for all VMs to stop"
|
||||
WAIT_DEADLINE=$(( $(date +%s) + 180 ))
|
||||
while [ "$(date +%s)" -lt "$WAIT_DEADLINE" ]; do
|
||||
echo -n "."
|
||||
all_stopped=1
|
||||
for vmid in $VM_LIST; do
|
||||
status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
|
||||
if [ "$status" = "running" ]; then
|
||||
all_stopped=0
|
||||
break
|
||||
fi
|
||||
done
|
||||
[ "$all_stopped" = "1" ] && break
|
||||
sleep 5
|
||||
done
|
||||
echo " done"
|
||||
echo ""
|
||||
|
||||
# Show stopped state
|
||||
echo "--- VM status after shutdown ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
|
||||
echo ""
|
||||
|
||||
# --- Phase 2: Start all VMs ---
|
||||
echo "--- PHASE 2: Start all VMs ---"
|
||||
for vmid in $VM_LIST; do
|
||||
name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
|
||||
echo -n " [$vmid $name] starting... "
|
||||
start_output=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1)
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "OK"
|
||||
else
|
||||
echo "FAILED: $start_output"
|
||||
fi
|
||||
done
|
||||
|
||||
# Wait 20s for VMs to fully start
|
||||
echo ""
|
||||
echo " Waiting 20s for VMs to boot..."
|
||||
sleep 20
|
||||
|
||||
# --- Phase 3: Verify all VMs running ---
|
||||
echo ""
|
||||
echo "--- PHASE 3: Verification ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
|
||||
echo ""
|
||||
|
||||
# Check NFS mounts still healthy
|
||||
echo "--- NFS mount health ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" '
|
||||
mount_count=$(nfsstat -m 2>/dev/null | grep -c "^/mnt")
|
||||
conn_count=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
|
||||
echo " NFS mounts: $mount_count"
|
||||
echo " NFS TCP connections: $conn_count"
|
||||
if [ "$mount_count" -gt 0 ]; then
|
||||
echo " First mount options:"
|
||||
nfsstat -m 2>/dev/null | head -2 | tail -1 | sed "s/^/ /"
|
||||
fi
|
||||
'
|
||||
echo ""
|
||||
|
||||
# Check guest agent responsiveness (if agent enabled)
|
||||
echo "--- Guest agent check (VMs with agent:1) ---"
|
||||
for vmid in $VM_LIST; do
|
||||
agent=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | grep -c '^agent: 1'")
|
||||
if [ "$agent" = "1" ]; then
|
||||
name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
|
||||
echo -n " [$vmid $name] agent ping... "
|
||||
result=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "timeout 10 qm agent $vmid ping 2>&1")
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "OK"
|
||||
else
|
||||
echo "no response (VM may still be booting)"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] VALIDATION COMPLETE"
|
||||
echo "================================================================"
|
||||
echo ""
|
||||
done
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/bin/bash
|
||||
# verify-tuning.sh - verifies apply-tunings.sh results on target hosts.
|
||||
set -uo pipefail
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
for host in "$@"; do
|
||||
echo "================================================================"
|
||||
echo "[$host] verification"
|
||||
echo "================================================================"
|
||||
|
||||
echo "--- TCP congestion control ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc 2>/dev/null'
|
||||
|
||||
echo "--- vm.swappiness ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl vm.swappiness 2>/dev/null'
|
||||
|
||||
echo "--- scaling_governor (cpu0) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null'
|
||||
|
||||
echo "--- NFS mount options (first 2 mounts) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'nfsstat -m 2>/dev/null | head -30'
|
||||
|
||||
echo "--- storage.cfg: any options lines? ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -c "options" /etc/pve/storage.cfg 2>/dev/null || echo 0'
|
||||
|
||||
echo "--- storage.cfg: NFS stanzas (first 3) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'grep -A 6 "^nfs:" /etc/pve/storage.cfg 2>/dev/null | head -25'
|
||||
|
||||
echo "--- nconnect TCP connections to NFS servers ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | head -20; echo "count:"; ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | tail -n +2 | wc -l'
|
||||
|
||||
echo "--- VMs still running? ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$host" 'qm list 2>/dev/null | head -15'
|
||||
|
||||
echo ""
|
||||
done
|
||||
Executable
+58
@@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
# wait-for-host.sh - polls SSH until host is back, then runs verification.
|
||||
# Usage: bash wait-for-host.sh <host>
|
||||
set -uo pipefail
|
||||
HOST="$1"
|
||||
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
|
||||
|
||||
DEADLINE=$(( $(date +%s) + 600 ))
|
||||
echo "Polling $HOST for SSH return (max 10 min)..."
|
||||
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
|
||||
if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "[$(date +%H:%M:%S)] SSH is back!"
|
||||
break
|
||||
fi
|
||||
sleep 10
|
||||
echo " [$(date +%H:%M:%S)] still down..."
|
||||
done
|
||||
|
||||
if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
|
||||
echo "FAILED: $HOST not back after 10 minutes"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Waiting 30s for services to settle..."
|
||||
sleep 30
|
||||
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] POST-REBOOT VERIFICATION"
|
||||
echo "================================================================"
|
||||
echo "--- Uptime (should be < 5 min) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
|
||||
echo ""
|
||||
echo "--- TCP congestion control (expect bbr) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
|
||||
echo ""
|
||||
echo "--- vm.swappiness ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
|
||||
echo ""
|
||||
echo "--- scaling_governor ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
|
||||
echo ""
|
||||
echo "--- NFS mount options (looking for nconnect=4 + noatime) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -16'
|
||||
echo ""
|
||||
echo "--- NFS TCP connection count to :2049 (expect ~8 = 4 per server × 2 servers) ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
|
||||
echo ""
|
||||
echo "--- Running VMs ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
|
||||
echo ""
|
||||
echo "--- Failed services ---"
|
||||
ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
|
||||
echo ""
|
||||
echo "================================================================"
|
||||
echo "[$HOST] DONE"
|
||||
echo "================================================================"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user