feat: full re-audit of all 7 hosts with fresh ground truth
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state after the user's PDM migrations: Cnode movements since last audit: - cnode1: tsys1 -> tsys9 - cnode2: tsys6 -> tsys7 - cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5) Wnode changes: - wnode-tsys1 (102): new VM on S2, stopped - wnode-tsys3: RAM bumped 20 -> 28 GB - wnode-tsys6: now running (was stopped) - wnode-tsys9: storage moved S3 -> S2 Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5). Still need 2 more cnode moves for etcd quorum survival. Updated executive summary, k8s distribution tables, storage utilization, and open items with the fresh data. Captured future k8s requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation (RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out with PowerEdge 19xx/2950 systems. Added tsys9 to deploy-check.sh host list.
This commit is contained in:
@@ -51,22 +51,21 @@ Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
|
||||
workers). Wnodes: one per hypervisor host across the fleet. Production lives
|
||||
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
|
||||
|
||||
## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
|
||||
## Critical VM-layer findings (re-audited 2026-07-27 21:50)
|
||||
|
||||
1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
|
||||
k8s cluster dead. etcd quorum lost.
|
||||
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
|
||||
if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
|
||||
(tsys4 SSD), ucs-02 to tsys9 local SSD.
|
||||
3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
|
||||
unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
|
||||
4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
|
||||
NFS export. Should host ucs-01 (LDAP/AD latency benefit).
|
||||
5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
|
||||
used. VMs should be rebalanced to reduce tsys4 blast radius.
|
||||
6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
|
||||
Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
|
||||
hosts boots from USB 2.0 -- user accepts this risk.
|
||||
1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
|
||||
more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
|
||||
actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
|
||||
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
|
||||
netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
|
||||
3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
|
||||
(Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
|
||||
4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
|
||||
Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
|
||||
5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
|
||||
workstation). Spinning rust = all infrastructure VMs.
|
||||
6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
|
||||
VM migration between nodes is done through PDM UI, not manual disk copy.
|
||||
|
||||
## Storage network IPs (VLAN1000, 10.100.100.0/24)
|
||||
|
||||
|
||||
+152
-76
@@ -1,9 +1,13 @@
|
||||
# Proxmox Cluster Project Report
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Date:** 2026-07-27 (re-audited)
|
||||
**Prepared by:** Performance Optimization Engagement
|
||||
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
|
||||
|
||||
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
|
||||
`deploy-check.sh`. VM placements reflect live state after user's PDM
|
||||
migrations. This is ground truth.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
@@ -26,22 +30,23 @@
|
||||
## 1. Executive Summary
|
||||
|
||||
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
|
||||
running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side
|
||||
performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned
|
||||
profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked
|
||||
on physical hardware work scheduled for Friday.
|
||||
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
|
||||
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
|
||||
are blocked on physical hardware work scheduled for Friday.
|
||||
|
||||
The VM-layer assessment reveals three structural issues:
|
||||
**Progress since initial audit:** The user has been actively rebalancing k8s
|
||||
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
|
||||
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
|
||||
needed for etcd quorum survival.
|
||||
|
||||
| # | Finding | Severity | Fix Cost |
|
||||
|---|---------|----------|----------|
|
||||
| 1 | **Every pfv-k8s node stores its disk on tsys4 NFS** (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. | **CRITICAL** | $0 (rebalance) |
|
||||
| 2 | **No k8s node uses SSD or NVMe.** All boot from NFS on spinning disks, adding latency to every etcd write and container pull. | **HIGH** | $0 (move to local SSD where available) |
|
||||
| 3 | **Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only.** tsys4 failure takes down both halves of each pair. | **HIGH** | $0 (move -02 halves to tsys5 NFS) |
|
||||
The VM-layer assessment reveals:
|
||||
|
||||
Additionally, tsys9 was validated this session and revealed two new findings:
|
||||
its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has
|
||||
a 250 GB local SSD that no VM is using.
|
||||
| # | Finding | Severity | Status |
|
||||
|---|---------|----------|--------|
|
||||
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
|
||||
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
|
||||
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
|
||||
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
|
||||
|
||||
---
|
||||
|
||||
@@ -87,7 +92,7 @@ running Cloudron -- not in this cluster.
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
|
||||
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
|
||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
|
||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
|
||||
|
||||
Non-exported disks on tsys4:
|
||||
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
|
||||
@@ -103,11 +108,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% |
|
||||
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** |
|
||||
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** |
|
||||
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
|
||||
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
|
||||
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
|
||||
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
|
||||
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% |
|
||||
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
|
||||
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
|
||||
|
||||
**tsys5 storage controllers (plenty of free ports):**
|
||||
@@ -117,9 +122,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
|
||||
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
|
||||
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
|
||||
|
||||
**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
|
||||
has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to
|
||||
reduce tsys4's blast radius.
|
||||
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
|
||||
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
|
||||
(unused, 7 GB). S4 still 99% empty (435 GB free).
|
||||
|
||||
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
|
||||
|
||||
**Friday additions:**
|
||||
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
|
||||
@@ -312,57 +319,98 @@ If tsys4 goes offline, two-thirds of the fleet loses its storage.
|
||||
|
||||
---
|
||||
|
||||
## 5. Kubernetes Node Distribution
|
||||
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
|
||||
|
||||
### 5.1 pfv-k8s cnode (control plane) placement
|
||||
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | Disk Type | Host Role Compliant? |
|
||||
|------|------|------------|---------|-----------|---------------------|
|
||||
| 107 | cnode1 | tsys1 | D5 (tsys4) | HDD | **NO** (infra host) |
|
||||
| 106 | cnode3 | tsys1 | D2 (tsys4) | HDD | **NO** (infra host) |
|
||||
| 603 | cnode2 | tsys6 | D2 (tsys4) | HDD | YES |
|
||||
| 601 | cnode4 | tsys6 | D2 (tsys4) | HDD | YES |
|
||||
| 602 | cnode5 | tsys6 | D5 (tsys4) | HDD | YES |
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|
||||
|------|------|------------|---------|-----------|----------|
|
||||
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
|
||||
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
|
||||
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
|
||||
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
|
||||
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
|
||||
|
||||
**Findings:**
|
||||
- 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
|
||||
- **All 5 cnodes store disks on tsys4 NFS.** Zero use tsys5 storage.
|
||||
- 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
|
||||
- All cnodes on spinning HDDs. No SSD/NVMe use.
|
||||
- **etcd quorum risk:** if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their
|
||||
disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is
|
||||
LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes
|
||||
inoperable.
|
||||
**Storage distribution:**
|
||||
|
||||
### 5.2 pfv-k8s wnode (worker) placement
|
||||
| Storage server | cnodes | Quorum impact if it fails |
|
||||
|---------------|--------|--------------------------|
|
||||
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
|
||||
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | Disk Type | Status | Host Role Compliant? |
|
||||
|------|------|------------|---------|-----------|--------|---------------------|
|
||||
| 313 | wnode-tsys3 | tsys3 | D5 (tsys4) | HDD | running | YES |
|
||||
| 509 | wnode-tsys5 | tsys5 | D2 (tsys4) | HDD | running | **NO** (storage host) |
|
||||
| 100 | wnode-tsys6 | tsys6 | D5 (tsys4) | HDD | **STOPPED** | YES (but offline) |
|
||||
| 701 | wnode-tsys7 | tsys7 | D5 (tsys4) | HDD | running | YES |
|
||||
| 905 | wnode-tsys9 | tsys9 | S3 (tsys5) | HDD | running | **NO** (infra host) |
|
||||
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
|
||||
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
|
||||
|
||||
**Findings:**
|
||||
- 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
|
||||
- wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) --
|
||||
it depends on tsys4 for its disk despite running on tsys5.
|
||||
- wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
|
||||
- wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it
|
||||
is the ONLY k8s node using tsys5 storage (which is good for redundancy).
|
||||
- All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.
|
||||
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
|
||||
tsys9) -- good host diversity.
|
||||
|
||||
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|
||||
|------|------|------------|---------|-----------|-----|--------|----------|
|
||||
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
|
||||
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
|
||||
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
|
||||
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
|
||||
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
|
||||
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
|
||||
|
||||
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
|
||||
stopped. wnode-tsys6 restarted.
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | wnodes | Notes |
|
||||
|---------------|--------|-------|
|
||||
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
|
||||
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
|
||||
|
||||
### 5.3 Summary: k8s node storage distribution
|
||||
|
||||
| Storage server | cnodes | wnodes | Total k8s nodes |
|
||||
|---------------|--------|--------|-----------------|
|
||||
| **tsys4 NFS** | **5 (100%)** | **4 (80%)** | **9 (90%)** |
|
||||
| tsys5 NFS | 0 | 1 (20%) | 1 (10%) |
|
||||
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
|
||||
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
|
||||
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
|
||||
|
||||
**90% of the kubernetes cluster's storage depends on a single host (tsys4).**
|
||||
This is the single most critical risk in the VM layer.
|
||||
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
|
||||
cnodes on each storage server (60/40 or better) so either server failing
|
||||
leaves quorum intact.
|
||||
|
||||
### 5.4 Remaining cnode migration needed for etcd quorum survival
|
||||
|
||||
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
|
||||
be on tsys5:
|
||||
|
||||
| Action | Effect |
|
||||
|--------|--------|
|
||||
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
|
||||
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
|
||||
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
|
||||
|
||||
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
|
||||
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
|
||||
|
||||
### 5.5 Future k8s architecture (next session -- see K8S.md)
|
||||
|
||||
The k8s layer will be tackled soon. Key requirements from user:
|
||||
|
||||
- **Platform:** vcluster + Rancher for multi-tenant management
|
||||
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
|
||||
- **Workload isolation (vcluster per tenant):**
|
||||
- RackRental workloads (containerlab network labs)
|
||||
- Suborbital ITAR (compliance-restricted)
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
|
||||
will be brought online during peak solar production for burst capacity.
|
||||
These older cores/ram supplement the main fleet when power is abundant.
|
||||
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
|
||||
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
|
||||
leftover host capacity) to large (28-32 GB, consuming most of a host).
|
||||
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
|
||||
full final audit will be performed at that time (post-NVMe install on tsys5,
|
||||
post-D3 SSD relocation, post-tsys4 NIC+RAM).
|
||||
|
||||
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
|
||||
|
||||
@@ -745,37 +793,65 @@ This belongs in the next budget cycle.
|
||||
|
||||
## 12. Open Items
|
||||
|
||||
### 12.1 Friday maintenance window (user action)
|
||||
### 12.1 Immediate (do today via PDM)
|
||||
|
||||
1. **tsys5:** Plug 2nd ethernet cable into storage NIC, verify bond0 recovers,
|
||||
apply layer3+4 hash, reboot to activate NFS nconnect.
|
||||
2. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
|
||||
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
|
||||
redundancy. UCS stays on HDD.
|
||||
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
|
||||
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
|
||||
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
|
||||
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
|
||||
|
||||
### 12.2 Friday maintenance window (user action)
|
||||
|
||||
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
|
||||
reconfigure `/etc/network/interfaces`, reboot.
|
||||
3. **tsys5:** Install PCI NVMe drive, format and mount (recommend local
|
||||
directory storage for VM images).
|
||||
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
|
||||
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
|
||||
format NVMe as local storage, reboot.
|
||||
3. **tsys2:** Load Proxmox (replacing Windows 10).
|
||||
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
|
||||
|
||||
### 12.2 Post-hardware validation
|
||||
### 12.3 Post-Friday validation
|
||||
|
||||
1. Re-run `iperf-full-matrix.sh` to confirm tsys4/tsys5 network recovery.
|
||||
2. Run `validate-fixes.sh` on tsys4 and tsys5.
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
|
||||
4. Update PROJECT.md with post-hardware numbers.
|
||||
|
||||
### 12.3 Data gaps
|
||||
### 12.4 Future: Kubernetes deep-dive (see K8S.md)
|
||||
|
||||
Next major workstream. Requirements captured:
|
||||
- **vcluster + Rancher** for multi-tenant k8s management
|
||||
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
|
||||
- **Workload isolation** via separate vclusters:
|
||||
- RackRental (containerlab)
|
||||
- Suborbital ITAR
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
|
||||
during peak solar production for burst capacity
|
||||
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
|
||||
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
|
||||
- **Spinning rust for all infrastructure VMs**
|
||||
|
||||
### 12.5 Data gaps
|
||||
|
||||
| Gap | How to close |
|
||||
|-----|--------------|
|
||||
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR before planning tsys5 role changes |
|
||||
| tsys2 post-rebuild baseline | Run `check.sh` once it is converted to Proxmox |
|
||||
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
|
||||
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
|
||||
| tsys5 NVMe size and model | Confirm after Friday installation |
|
||||
| Whether hosts form a Proxmox cluster or are standalone | Check `pvecm status` on each host |
|
||||
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
|
||||
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
|
||||
|
||||
### 12.4 Scripts and tools available
|
||||
### 12.6 Scripts and tools available
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/check.sh` | Read-only data collector (run on hosts) |
|
||||
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash (live) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
||||
| `validate-fixes.sh` | Read-only validation of all applied changes |
|
||||
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
|
||||
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH |
|
||||
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ LOG_DIR="$SCRIPT_DIR/returned-logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# ONLY the hosts the user told us are alive.
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
|
||||
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
|
||||
|
||||
# Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
|
||||
SSH_OPTS=(-o BatchMode=yes
|
||||
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# baseline.sh — quick read-only baseline of a target node.
|
||||
set -u
|
||||
hdr() { printf '\n=== %s ===\n' "$1"; }
|
||||
hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
|
||||
hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
|
||||
hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
|
||||
hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
|
||||
hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
|
||||
hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
|
||||
hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
|
||||
hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
|
||||
hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
|
||||
hdr "HOME"; ls -la /home/localuser 2>/dev/null | head
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
|
||||
# contents, and chrony NTP details from pfv-netboot for replication.
|
||||
set -u
|
||||
hdr() { printf '\n=== %s ===\n' "$1"; }
|
||||
|
||||
hdr "TECHNITIUM dns.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM auth.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM log.config"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
|
||||
|
||||
hdr "TECHNITIUM scopes/Default.scope"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
|
||||
|
||||
hdr "TECHNITIUM zone: knel.net.zone"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
|
||||
|
||||
hdr "TECHNITIUM all zone files listing + sizes"
|
||||
sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
|
||||
|
||||
hdr "TECHNITIUM reverse zones (one sample)"
|
||||
sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
|
||||
|
||||
hdr "TECHNITIUM top-level listing"
|
||||
sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
|
||||
|
||||
hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
|
||||
cat /root/pihole/etc-pihole/pihole.toml 2>&1
|
||||
|
||||
hdr "PI-HOLE dnsmasq.conf"
|
||||
cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
|
||||
|
||||
hdr "PI-HOLE versions file"
|
||||
cat /root/pihole/etc-pihole/versions 2>&1
|
||||
|
||||
hdr "GRAVITY.DB contents (host sqlite3 if present)"
|
||||
if command -v sqlite3 >/dev/null 2>&1; then
|
||||
G=/root/pihole/etc-pihole/gravity.db
|
||||
echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
|
||||
echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
|
||||
echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
|
||||
echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
|
||||
echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
|
||||
echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
|
||||
echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
|
||||
else
|
||||
echo "(sqlite3 not on host PATH)"
|
||||
fi
|
||||
|
||||
hdr "DONE"
|
||||
Binary file not shown.
Executable
+283
@@ -0,0 +1,283 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# setup-netinfra.sh
|
||||
# -----------------------------------------------------------------------------
|
||||
# Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
|
||||
# onto pfv-netinfra-01 and pfv-netinfra-02.
|
||||
#
|
||||
# DESIGN
|
||||
# * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
|
||||
# from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
|
||||
# * The targets cannot SSH to pfv-netboot directly, so config tarballs are
|
||||
# relayed through this workstation:
|
||||
# ssh netboot 'sudo tar -cf - ...' | ssh target 'sudo tar -xf - ...'
|
||||
# * Services are deployed under /home/localuser/services/<svc>/ on each
|
||||
# target so localuser can manage them (mirrors netboot's localuser-owned
|
||||
# pihole data dir). `sudo docker` is used since localuser is not in the
|
||||
# docker group (same as on netboot).
|
||||
#
|
||||
# SERVICES
|
||||
# pihole pihole/pihole:latest :53 tcp/udp :10002->80 :10003->443
|
||||
# ntp (chrony) dockurr/chrony <tailscale-ip>:123:123/udp
|
||||
# technitium technitium/dns-server :5300->53 tcp/udp :5380 :53443
|
||||
# (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
|
||||
# The knel.net authoritative zone + Tailscale reverse zones are preserved
|
||||
# verbatim from netboot's orphaned dns_tsys-dns-config volume.)
|
||||
#
|
||||
# USAGE
|
||||
# ./setup-netinfra.sh # deploy to BOTH nodes
|
||||
# ./setup-netinfra.sh pfv-netinfra-01 # deploy to one node
|
||||
# ./setup-netinfra.sh pfv-netinfra-01 verify # verify only
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NETBOOT="localuser@pfv-netboot"
|
||||
SVC_ROOT="/home/localuser/services"
|
||||
PIHOLE_PW='Gransyan1!' # replicated verbatim from netboot compose
|
||||
|
||||
log() { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
|
||||
warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
|
||||
|
||||
# Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
|
||||
declare -A NODE_TSIP=(
|
||||
[pfv-netinfra-01]="100.70.181.72"
|
||||
[pfv-netinfra-02]="100.93.194.82"
|
||||
)
|
||||
|
||||
on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Verify-only mode
|
||||
#------------------------------------------------------------------------------
|
||||
verify_node() {
|
||||
local node="$1" tsip="${NODE_TSIP[$1]}"
|
||||
log "VERIFY $node (tailscale $tsip)"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
set +e
|
||||
echo "### containers ###"
|
||||
sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
|
||||
echo
|
||||
echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
|
||||
dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
|
||||
echo "### Pi-hole web (curl :10002) ###"
|
||||
curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
|
||||
echo
|
||||
echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
|
||||
dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
|
||||
echo "### Technitium web (curl :5380) ###"
|
||||
curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
|
||||
echo
|
||||
echo "### NTP on tailscale IP (ntpdig -p 1) ###"
|
||||
timeout 5 ntpdig -t1 -c1 -p1 $tsip 2>&1 || echo "(ntpdig not available or no response)"
|
||||
echo "### chrony container sources ###"
|
||||
sudo docker exec tsys-ntp chronyc -n sources 2>&1 | head -8
|
||||
EOF
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Deploy to one node
|
||||
#------------------------------------------------------------------------------
|
||||
deploy_node() {
|
||||
local node="$1" tsip="${NODE_TSIP[$1]}"
|
||||
log "==== DEPLOY $node (tailscale $tsip) ===="
|
||||
|
||||
# ---- 1. Prepare directories on the target -------------------------------
|
||||
log "$node: create service dirs"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
set -e
|
||||
sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
|
||||
sudo chown -R localuser:localuser $SVC_ROOT
|
||||
EOF
|
||||
|
||||
# ---- 2. Write compose files (as localuser) ------------------------------
|
||||
log "$node: write docker-compose files"
|
||||
on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
|
||||
services:
|
||||
pihole:
|
||||
container_name: pihole
|
||||
image: pihole/pihole:latest
|
||||
hostname: pihole
|
||||
ports:
|
||||
- "53:53/tcp"
|
||||
- "53:53/udp"
|
||||
- "10002:80/tcp"
|
||||
- "10003:443/tcp"
|
||||
environment:
|
||||
TZ: 'America/Chicago'
|
||||
FTLCONF_webserver_api_password: 'Gransyan1!'
|
||||
FTLCONF_dns_listeningMode: 'all'
|
||||
volumes:
|
||||
- './etc-pihole:/etc/pihole'
|
||||
cap_add:
|
||||
- SYS_NICE
|
||||
restart: always
|
||||
YAML
|
||||
|
||||
on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
|
||||
services:
|
||||
ntp:
|
||||
image: dockurr/chrony
|
||||
container_name: tsys-ntp
|
||||
environment:
|
||||
NTP_SERVERS: "pool.ntp.org"
|
||||
ports:
|
||||
- "$tsip:123:123/udp"
|
||||
restart: always
|
||||
YAML
|
||||
|
||||
on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
|
||||
services:
|
||||
technitium:
|
||||
image: technitium/dns-server
|
||||
container_name: tsys-dns
|
||||
ports:
|
||||
- "5300:53/tcp"
|
||||
- "5300:53/udp"
|
||||
- "5380:5380/tcp"
|
||||
- "53443:53443/tcp"
|
||||
volumes:
|
||||
- './config:/etc/dns'
|
||||
restart: always
|
||||
YAML
|
||||
|
||||
# ---- 3. Relay Pi-hole config from netboot -> target ---------------------
|
||||
log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
|
||||
if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
|
||||
log "$node: Pi-hole config already present; skipping copy"
|
||||
else
|
||||
on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
|
||||
ssh -o StrictHostKeyChecking=no "$NETBOOT" \
|
||||
"sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
|
||||
--exclude='etc-pihole/listsCache' \
|
||||
--exclude='etc-pihole/gravity_backups' \
|
||||
--exclude='etc-pihole/config_backups' \
|
||||
etc-pihole" \
|
||||
| on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
|
||||
fi
|
||||
|
||||
# ---- 4. Relay Technitium config from netboot orphaned volume -----------
|
||||
log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
|
||||
if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
|
||||
log "$node: Technitium config already present; skipping copy"
|
||||
else
|
||||
on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
|
||||
ssh -o StrictHostKeyChecking=no "$NETBOOT" \
|
||||
"sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
|
||||
| on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
|
||||
fi
|
||||
|
||||
# ---- 5. Adapt copied config: Pi-hole interface name ---------------------
|
||||
log "$node: adapt Pi-hole pihole.toml interface name to actual iface"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set -e
|
||||
IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
|
||||
IFACE=${IFACE:-ens18}
|
||||
TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
|
||||
if sudo test -f "$TOML"; then
|
||||
sudo sed -i "s|^ interface = .*| interface = \"$IFACE\" ### ADAPTED from eth0 on clone|" "$TOML"
|
||||
echo "set interface=$IFACE"
|
||||
else
|
||||
echo "(pihole.toml not present; FTL will create it on first run)"
|
||||
fi
|
||||
# Strip netboot-specific primary upstream 192.168.3.16? -> keep, it is reachable on LAN.
|
||||
EOF
|
||||
|
||||
# ---- 6. Pull images -----------------------------------------------------
|
||||
log "$node: docker compose pull (pihole, ntp, technitium)"
|
||||
on_node "$node" "bash -s" <<EOF
|
||||
for c in pihole ntp technitium; do
|
||||
sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
|
||||
done
|
||||
EOF
|
||||
|
||||
# ---- 6a. Pi-hole up -----------------------------------------------------
|
||||
log "$node: bring up Pi-hole"
|
||||
on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
|
||||
|
||||
# ---- 6b. NTP -- only deploy the chrony container if nothing already ----
|
||||
# serves UDP/123 on the host. The targets already run a bare-metal ntpsec
|
||||
# daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
|
||||
# netboot's own bare-metal ntpsec. netboot additionally runs a chrony
|
||||
# container on its tailscale IP, but that only works there because ntpsec
|
||||
# there does not pre-bind the specific tailscale-IP socket. On these targets
|
||||
# ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
|
||||
# redundant anyway. We therefore keep the host ntpsec as the NTP service.
|
||||
log "$node: NTP -- check whether host already serves UDP/123"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set +e
|
||||
HOST_NTP=$(sudo ss -lun 2>/dev/null | awk '$5 ~ /:123$/ {print}' | head -1)
|
||||
if [ -n "$HOST_NTP" ]; then
|
||||
echo "Host already serves UDP/123 ($HOST_NTP); host daemon:"
|
||||
for u in ntpsec ntp chrony openntpd systemd-timesyncd; do
|
||||
systemctl is-active "$u" 2>/dev/null | grep -q active && echo " -> $u active"
|
||||
done
|
||||
echo "Skipping chrony container (host NTP provides the service)."
|
||||
echo "Removing any stale tsys-ntp container..."
|
||||
sudo docker rm -f tsys-ntp 2>/dev/null || true
|
||||
else
|
||||
echo "Nothing serving UDP/123; starting chrony container."
|
||||
sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
|
||||
fi
|
||||
EOF
|
||||
|
||||
# ---- 6c. Technitium up --------------------------------------------------
|
||||
log "$node: bring up Technitium"
|
||||
on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
|
||||
|
||||
# ---- 7. Wait for Pi-hole health -----------------------------------------
|
||||
log "$node: wait for Pi-hole to become healthy"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
for i in $(seq 1 30); do
|
||||
st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
|
||||
echo " pihole health: $st"
|
||||
[ "$st" = "healthy" ] && break
|
||||
sleep 4
|
||||
done
|
||||
EOF
|
||||
|
||||
# ---- 8. Technitium defensive fallback -----------------------------------
|
||||
# If the copied (binary) config from an older Technitium version makes the
|
||||
# new container crash, move it aside and let Technitium start fresh so the
|
||||
# service is at least up (admin reachable) rather than crash-looping.
|
||||
log "$node: check Technitium health (fallback to fresh config if crash)"
|
||||
on_node "$node" "bash -s" <<'EOF'
|
||||
set +e
|
||||
sleep 8
|
||||
rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
|
||||
running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
|
||||
if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
|
||||
echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
|
||||
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
|
||||
sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
|
||||
sudo mkdir -p /home/localuser/services/technitium/config
|
||||
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
|
||||
echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
|
||||
else
|
||||
echo "Technitium OK (running=$running restarts=$rst)"
|
||||
fi
|
||||
EOF
|
||||
|
||||
# ---- 9. Final status -----------------------------------------------------
|
||||
log "$node: final container status"
|
||||
on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Main
|
||||
#------------------------------------------------------------------------------
|
||||
main() {
|
||||
local mode="${2:-deploy}"
|
||||
if [ "${1:-all}" = "all" ]; then
|
||||
targets=(pfv-netinfra-01 pfv-netinfra-02)
|
||||
else
|
||||
targets=("$1")
|
||||
fi
|
||||
for t in "${targets[@]}"; do
|
||||
: "${NODE_TSIP[$t]:?unknown node $t}"
|
||||
if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
|
||||
done
|
||||
log "DONE"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys1.log 911607 17324
|
||||
ok pfv-tsys1.log 951792 16094
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys3.log 725429 10091
|
||||
ok pfv-tsys3.log 750211 11660
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys4.log 603525 11030
|
||||
ok pfv-tsys4.log 617476 11115
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys5.log 1047724 17559
|
||||
ok pfv-tsys5.log 1064227 18745
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys6.log 743495 14563
|
||||
ok pfv-tsys6.log 907424 16997
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys7.log 896348 17341
|
||||
ok pfv-tsys7.log 1114764 18714
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
running
|
||||
ok pfv-tsys9.log 793781 14230
|
||||
+4031
-5261
File diff suppressed because it is too large
Load Diff
+3879
-2310
File diff suppressed because it is too large
Load Diff
+954
-869
File diff suppressed because it is too large
Load Diff
+3313
-2127
File diff suppressed because it is too large
Load Diff
+6496
-4062
File diff suppressed because it is too large
Load Diff
+5168
-3795
File diff suppressed because it is too large
Load Diff
+4676
-2914
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user