This is what the repo looked like before i refactored it

total 128
-rw-rw-r--  1 reachableceo reachableceo  6311 Jul 30 23:27 AGENTS.md
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 29 19:28 awx
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 29 19:32 console
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 28 11:33 dns-cluster-setup
drwxrwxr-x  4 reachableceo reachableceo  4096 Jul 29 05:01 docs
drwxrwxr-x  4 reachableceo reachableceo  4096 Jul 29 04:59 k8s
-rw-rw-r--  1 reachableceo reachableceo 33962 Jul 28 11:22 LICENSE
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 28 11:22 logs
drwxrwxr-x  3 reachableceo reachableceo  4096 Jul 29 19:28 netinfra
-rw-rw-r--  1 reachableceo reachableceo     0 Aug  1 15:31 nextprompt
drwxrwxr-x  3 reachableceo reachableceo  4096 Jul 28 20:01 perf
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 28 19:46 powerman
drwxrwxr-x 10 reachableceo reachableceo  4096 Jul 30 08:47 provisioning
drwxrwxr-x  3 reachableceo reachableceo  4096 Jul 29 05:01 proxmox
-rw-rw-r--  1 reachableceo reachableceo  2251 Jul 30 09:33 README.md
drwxrwxr-x  5 reachableceo reachableceo  4096 Jul 30 22:52 returned-logs
-rw-r--r--  1 reachableceo reachableceo 12514 Jul 30 23:03 STATUS.md
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 30 23:01 switches
drwxrwxr-x  5 reachableceo reachableceo  4096 Jul 30 08:41 tests
drwxrwxr-x  2 reachableceo reachableceo  4096 Jul 30 11:40 ups
drwxrwxr-x  3 reachableceo reachableceo  4096 Jul 28 11:22 vendor

Here is what it looks like now

total 112
-rw-rw-r-- 1 reachableceo reachableceo  6311 Jul 30 23:27 AGENTS.md
drwxrwxr-x 4 reachableceo reachableceo  4096 Aug  1 15:34 archive
drwxrwxr-x 2 reachableceo reachableceo  4096 Jul 29 19:28 awx
drwxrwxr-x 4 reachableceo reachableceo  4096 Aug  1 15:34 dcinfra
drwxrwxr-x 4 reachableceo reachableceo  4096 Jul 29 04:59 k8s
-rw-rw-r-- 1 reachableceo reachableceo 33962 Jul 28 11:22 LICENSE
drwxrwxr-x 2 reachableceo reachableceo  4096 Jul 28 11:22 logs
drwxrwxr-x 5 reachableceo reachableceo  4096 Aug  1 15:34 netinfra
-rw-rw-r-- 1 reachableceo reachableceo  1426 Aug  1 15:31 nextprompt
drwxrwxr-x 4 reachableceo reachableceo  4096 Aug  1 15:33 proxmox
-rw-rw-r-- 1 reachableceo reachableceo  2251 Jul 30 09:33 README.md
drwxrwxr-x 5 reachableceo reachableceo  4096 Jul 30 22:52 returned-logs
-rw-r--r-- 1 reachableceo reachableceo 12514 Jul 30 23:03 STATUS.md
drwxrwxr-x 5 reachableceo reachableceo  4096 Jul 30 08:41 tests
drwxrwxr-x 2 reachableceo reachableceo  4096 Jul 30 11:40 ups
drwxrwxr-x 3 reachableceo reachableceo  4096 Jul 28 11:22 vendor

I move the top level directories into more of a sub project/module type setup

I moved the provisioning directory to archive, because its replaced by the KNELIAC repository/project (located at: /home/reachableceo/projects/KNELIAC

As far as the servers/data center itself, I have made many changes to :

- deployed virtual machines (tsys-umbrel, tsys-sipwise)
- location of virtual machine disks
- amount of ram/cpu allocated to virtual machines
- LACP between core and r3-stor is fixed
- pfv-k8s-wnode fleet changes
- removed tctc vm 
- added all domains to uptime kuma
- added all systems/services to uptime kuma
- ssd/nvme disks are formatted and in use on pfv-tsys5

Unfortuantely during my storage moves, I accidently wiped out pfv-netinfra-02 . Can you turn it back into a redundant node? Its setup with the ssh pub key/passwordless sudo like before

Also, the default /dev/shm allocated to pihole was too small, i fixed that in compose live on the server, as it was a production outage. Can you fold that fix in? Also can you add a docker compose healthcheck for gravity.db issues? I had to manually fix gravitydb (move it out of way, bounce container, pihole auto recreates it). (actualy look at the bash history for root on pfv-netinfra-01 , youll see other fixes i had todo). and/or look at the docker compose config and poke around in the container. basically i had to prune volumes and bounce the container. i dont ever want to have dns fail again. this is enterprise IT , has to work.


In general, re-orient yourself to this repo, to the servers/viritual machines (via lldp/snmpwalk/ssh/nmap)

oh a note about the 4gbps lacp and concerns about mismatch. 

Heres my thinking (re single flow/link saturation due to mismatched lacp and/or lack of layer4 option on core switch)....

pfv-tsys6/7 will both have 1gbps to pfv-tsys4/5 right? so thats ok
the indiviual vms (doing k8s chatter, mpi interconnect, clustered whatever) will all have 1gbps (because unique ip/mac)

So, as long as, every single endpoint has 1gbps to every other endpoint over the vlan 1000 network , I am happy. 

The 4gbps was more about ensuring adequate overall capacity for all the different sessions.  I am very open to more iperf testing, fio testing on the vms and generally properly performance benchmarking/analyzing the whole proxmox/k8s cluster before any workloads are put onto them. Also cpu/ram benchmarking as well. Lets see what this cluster can do.


in general, do some gardening/true up/ground truth of the servers, this repo. 

utilize snmpwalk/nmap/lldp and your api access to :

- phpipam (mostly will be for the future, as i use dhcp/tailscale for everything to avoid ip conflicts)
- uptime kuma
- librenms
- beszel 
- technitum
- home assistant 

to get as much together as possible. let me know if anything isn't working/broke since last time. i made a bunch of changes.

I want to get you connected into promethus/promethus alert manager/grafana and have you deploy unpoller and netdisco via docker to the librenms vm (and also setup librenms with more functionality like weathermap/smokeping etc)

I want home assistant to handle all environmental monitoring and reporting for the site.

Also, any api keys must be moved into hashicorp vault as soon as possible. Track that as a very high priority todo. vault is deployed and out of box setup but nothing configured. its at https://vault.knownelement.com

Also, any work we do here, that perhaps should also be replicated in the KNELIAC repository/project, track it in a STATUS.md file in that project. We are doing a bit of fast/loose here at this time with bash scripts, in the interest of time. For the mid/long term, the work should be captured as ansible roles/playbooks.

for now, ssh to all the worker nodes (not sectestbed/preprod though) and join them to the cluster. oh the cnodes... i also accidently wiped one of them out. so that will need to be fixed. I shut the cnodes down, because one of them was 100% memory, i'm guessing cause one of the nodes got wiped out. So youll need to power them all up in proxmox.


