This commit is contained in:
2024-12-26 16:31:25 -05:00
parent 8ced82004f
commit 55768fc8b1
38 changed files with 29 additions and 207 deletions

View File

@@ -0,0 +1,50 @@
# TSYS Group - HQ data center documentation - cooling
## Introduction
Cooling is a critical component of any data center. It is often the dominate consumer of energy.
We keep our data center at about 70 degrees F.
## Make / model
We have a
* HiSense Portable Air Conditioner (standalone) the manual lists several possible models, unsure which exact one we have. It was about 700.00 at Lowes with a multiple year replacement warranty.
which is rated for:
* 15,000 BTU
It draws about 7 amps when the compressor is running.
With our heat load, the compressor does cycle on/off ,so it keeps cool pretty efficiently from an energy perspective.
## Tips/tricks
* Extended exhaust house
We moved the air conditioner to the front of the racks (cold aisle) and extended the exhaust
hose todo so.
* Heat barrier
We deployed a cardboard heat barrier above the racks, to keep hot air behind the racks. We also have a vent duct (made of cardboard) to a panel we removed above the doorway.
* Insulation
* Insulate the exhaust hose!
* Air movers
* We have a tower fan in the hot row (back), pushing the heat towards the duct.
* We have two small blowers in the cold row (front) helping "kick back" the air blowing from the HiSense.
## Instrumentation
We use:
* temper usb probe
* lm-sensors
* DRAC
all consumed via SNMP by librenms to monitor/alert on temperature.This lets us find hot/cold spots across the racks and make any necessary adjustments.

View File

@@ -0,0 +1,80 @@
pfv-servers - performance
## vm 1-3 (optiplex)
### Commands to run
* cpupower frequency-set --governor performance
### links to reference
https://itectec.com/ubuntu/ubuntu-how-to-set-performance-instead-of-powersave-as-default/
https://www.cult-of-tech.net/2018/08/linux-ubuntu-cpu-power-frequency-scaling/
https://askubuntu.com/questions/1021748/set-cpu-governor-to-performance-in-18-04
https://metebalci.com/blog/a-minimum-complete-tutorial-of-cpu-power-management-c-states-and-p-states/
## vm 4/6 (xeon poweredge)
Appears to only run at the full frequency (which is what I want)
## Keep the NIC awake
notes taken on 03/20/2021 at 18:28
vm1/2/3 use intel nic
https://downloadcenter.intel.com/download/15817 is the driver (e1000e)
### vm1
root@pfv-vm1:/usr/local/bin# ethtool -i eno1
driver: e1000e
version: 3.2.6-k
firmware-version: 0.13-4
expansion-rom-version:
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
### vm2
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
root@pfv-vmsrv-02:~# ethtool -i enp0s25
driver: e1000e
version: 3.2.6-k
firmware-version: 0.13-3
expansion-rom-version:
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
### vm3
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
ethtool -i enp0s25
driver: e1000e
version: 3.2.6-k
firmware-version: 0.13-4
expansion-rom-version:
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

View File

@@ -0,0 +1,10 @@
#!/bin/bash
#Script to set performance.
cpufreq-set -r -g performance
cpupower frequency-set --governor performance

View File

@@ -0,0 +1,59 @@
#!/bin/bash
#Setup a new server base
#curl -s http://dl.turnsys.net/newSrv.sh|/bin/bash
apt-get -y --purge remove nano
apt-get -y install ntp ntpdate
systemctl stop ntp
ntpdate 10.251.37.5
apt-get update
apt-get -y full-upgrade
apt-get -y install glances htop dstat snmpd screen lldpd lsb-release net-tools sudo gpg molly-guard lshw
rm -rf /usr/local/librenms-agent
curl -s http://dl.turnsys.net/librenms-agent/distro > /usr/local/bin/distro
chmod +x /usr/local/bin/distro
curl -s http://dl.turnsys.net/librenms.tar.gz > /usr/local/librenms.tar.gz
cd /usr/local ; tar xfs librenms.tar.gz
systemctl stop snmpd ; curl -s http://dl.turnsys.net/snmpd.conf > /etc/snmp/snmpd.conf
sed -i "s|-Lsd|-LS6d|" /lib/systemd/system/snmpd.service
systemctl daemon-reload
systemctl restart snmpd
/etc/init.d/rsyslog stop
cat <<EOF> /etc/rsyslog.conf
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
*.* @10.251.30.1:514
EOF
/etc/init.d/rsyslog start
logger "hi hi from $(hostname)"
bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait
echo "deb http://download.webmin.com/download/repository sarge contrib" > /etc/apt/sources.list.d/webmin.list
wget -q -O- http://www.webmin.com/jcameron-key.asc | sudo apt-key add
sudo apt update
sudo apt-get -y install webmin

View File

@@ -0,0 +1,36 @@
#!/bin/bash
#install dell omsa
#curl -s http://dl.turnsys.net/omsa.sh|/bin/bash
gpg --keyserver hkp://pool.sks-keyservers.net:80 --recv-key 1285491434D8786F
gpg -a --export 1285491434D8786F | apt-key add -
echo "deb http://linux.dell.com/repo/community/openmanage/930/bionic bionic main" > /etc/apt/sources.list.d/linux.dell.com.sources.list
wget http://archive.ubuntu.com/ubuntu/pool/universe/o/openwsman/libwsman-curl-client-transport1_2.6.5-0ubuntu3_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/o/openwsman/libwsman-client4_2.6.5-0ubuntu3_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/o/openwsman/libwsman1_2.6.5-0ubuntu3_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/o/openwsman/libwsman-server1_2.6.5-0ubuntu3_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/sblim-sfcc/libcimcclient0_2.2.8-0ubuntu2_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/o/openwsman/openwsman_2.6.5-0ubuntu3_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/multiverse/c/cim-schema/cim-schema_2.48.0-0ubuntu1_all.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/sblim-sfc-common/libsfcutil0_1.0.1-0ubuntu4_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/multiverse/s/sblim-sfcb/sfcb_1.4.9-0ubuntu5_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/sblim-cmpi-devel/libcmpicppimpl0_2.0.3-0ubuntu2_amd64.deb
dpkg -i libwsman-curl-client-transport1_2.6.5-0ubuntu3_amd64.deb
dpkg -i libwsman-client4_2.6.5-0ubuntu3_amd64.deb
dpkg -i libwsman1_2.6.5-0ubuntu3_amd64.deb
dpkg -i libwsman-server1_2.6.5-0ubuntu3_amd64.deb
dpkg -i libcimcclient0_2.2.8-0ubuntu2_amd64.deb
dpkg -i openwsman_2.6.5-0ubuntu3_amd64.deb
dpkg -i cim-schema_2.48.0-0ubuntu1_all.deb
dpkg -i libsfcutil0_1.0.1-0ubuntu4_amd64.deb
dpkg -i sfcb_1.4.9-0ubuntu5_amd64.deb
dpkg -i libcmpicppimpl0_2.0.3-0ubuntu2_amd64.deb
apt update
#apt -y install srvadmin-all
touch /opt/dell/srvadmin/lib64/openmanage/IGNORE_GENERATION
#logout,login, then run
# srvadmin-services.sh enable && srvadmin-services.sh start

View File

@@ -0,0 +1,12 @@
#!/bin/bash
#Make a proxmox server
rm -f /etc/apt/sources.list.d/*
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
chmod +r /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg # optional, if you have a non-default umask
apt update && apt -y full-upgrade
apt-get -y install ifupdown2 ipmitool ethtool net-tools lshw
curl -s http://dl.turnsys.net/newSrv.sh|/bin/bash

View File

@@ -0,0 +1,17 @@
# PFV Local Area Network
- [PFV Local Area Network](#pfv-local-area-network)
- [Introduction](#introduction)
- [Subnets](#subnets)
- [Diagram](#diagram)
- [Security considerations](#security-considerations)
## Introduction
## Subnets
- 10.251.0.0/16 (See phpipam for all the particulars)
## Diagram
## Security considerations

View File

@@ -0,0 +1,62 @@
# PFV WAN
- [PFV WAN](#pfv-wan)
- [Introduction](#introduction)
- [Provider](#provider)
- [IP Allocation](#ip-allocation)
- [Diagram](#diagram)
- [Security considerations](#security-considerations)
- [Availaiblity considerations](#availaiblity-considerations)
## Introduction
The HQ data center provides both corporate network and WAN services. We utilize AT&T Uverse Busienss CLass VDSL service for IP transit.
### Provider
- AT&T Uverse
- Business DSL (fiber overbuild is projected for late 2021)
- 60 down/20 up is what I see in speed tests
## IP Allocation
- Static IP setup : <https://forums.att.com/conversations/att-internet-features/how-do-i-setup-an-att-internet-static-ip/5defee02bad5f2f606ea4054>
```text
Broadband Connection Up
Broadband Network Type Lightspeed
Broadband IPv4 Address 107.140.191.0
Gateway IPv4 Address 107.140.188.1
MAC Address 84:bb:69:e1:b1:e1
Primary DNS 68.94.156.9
Secondary DNS 68.94.157.9
Primary DNS Name
Secondary DNS Name
```
```text
Address: 104.182.29.16 01101000.10110110.00011101.00010 000
Netmask: 255.255.255.248 = 29 11111111.11111111.11111111.11111 000
Wildcard: 0.0.0.7 00000000.00000000.00000000.00000 111
=>
Network: 104.182.29.16/29 01101000.10110110.00011101.00010 000 (Class A)
Broadcast: 104.182.29.23 01101000.10110110.00011101.00010 111
HostMin: 104.182.29.17 01101000.10110110.00011101.00010 001
HostMax: 104.182.29.22 01101000.10110110.00011101.00010 110
Hosts/Net: 6
```
- 104.182.29.16 (network address)
- 104.182.29.17 rtr1
- 104.182.29.18 rtr2
- 104.182.29.19 float
- 104.182.29.20 FNFMail
- 104.182.29.21 WWW testing
- 104.182.29.22 (gateway)
- 104.182.29.23 (broadcast)
## Diagram
## Security considerations
## Availaiblity considerations

View File

@@ -0,0 +1,23 @@
#!/bin/bash
#https://forum.proxmox.com/threads/e1000-driver-hang.58284/
#https://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang
#magic to detect main int
echo "Determining management interface..."
#export MAIN_INT=$(brctl show $(netstat -rn|grep 0.0.0.0|head -n1|awk '{print $NF}') | awk '{print $NF}'|tail -1|awk -F '.' '{print $1}')
export MAIN_INT=$(brctl show|grep vmbr0|awk '{print $NF}'|awk -F '.' '{print $1}')
echo "Management interface is: $MAIN_INT"
#fix the issue
echo "Fixing management interface..."
ethtool -K $MAIN_INT tso off
ethtool -K $MAIN_INT gro off
ethtool -K $MAIN_INT gso off
#https://forum.proxmox.com/threads/e1000-driver-hang.58284/
#https://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang

View File

@@ -0,0 +1,107 @@
# TSYS Group - HQ data center documentation - power
- [TSYS Group - HQ data center documentation - power](#tsys-group-hq-data-center-documentation-power)
- [Introduction](#introduction)
- [Circuits](#circuits)
- [Outlets](#outlets)
- [Surge Protectors](#surge-protectors)
- [Extension cords](#extension-cords)
- [UPS units](#ups-units)
- [Prod](#prod)
- [UPS5](#ups5)
- [UPS7](#ups7)
- [R&D](#r-d)
- [UPS1](#ups1)
- [UPS3](#ups3)
- [UPS4](#ups4)
- [UPS6](#ups6)
- [PDU](#pdu)
- [Unmanaged PDUs](#unmanaged-pdus)
- [Managed PDUs](#managed-pdus)
## Introduction
This article covers the electrical power setup for the HQ data center. We've grown it over time, bringing online more and more protected capacity as we got good deals on UPS/batteries etc and have added additional load.
## Circuits
The server room is fed by two 20amp circuits:
* Circuit 8a serving:
* dedicated air conditioner (see our cooling article for details on that)
* vm(1-3) servers
* network equipment
* overhead and led lighting
* Circuit (xx) serving:
* pfv-stor1/stor2 enclosures and drive arrays
* vm(4-6)
(future plan)
* Connect a new outlet to the 20 amp circuit currently serving front porch outlet (which shares a wall with the server room).
* This would provide sustained 15 amps for the RackRental.net rentable inventory.
## Outlets
We have upgraded the standard 15amp outlets that serve the server room, to 20amp outlets. This allows us to run a full 15amps sustained load (on 20amp circuits)
## Surge Protectors
We utilize GE surge protectors , rated for 15amps. They are about $50.00 apiece. These are placed upstream of the UPS units (between the wall outlet and the UPS extension cord).
## Extension cords
We do not have outlets close to the UPS stack. We utilize 15amp rated extension cords (from the surge protectors) to feed the UPS inputs.
## UPS units
### Prod
* UPS2
* Make/Model: Dell UPS Rack 1000W LV
* PDU served:
* UMPDU1
* Protected load:
* pfv-stor1/pfv-stor2 (Dell PowerEdge 2950s)
* backup USB drives and USB hub
* external scratch/backup arrays
* Protected Load Runtime: 12 minutes
### UPS5
* CyberPower UPS (details tbd)
* PDU served:
* UMPDU4
* BenchPDU
* Cameras
* Protected load:
* pfv-vm1/2/3
* pfv-time1
* pfv-labsw*
* pfv-core-ap01
* pfv-coresw-01
* pfv-labsw*
* Protected Load Runtime: 12 minutes
### UPS7
* PDUs served: n/a
* Monitoring server: n/a (un-monitored ups)
* Protected load: locking relay for server room
## R&D
### UPS1
### UPS3
### UPS4
### UPS6
# PDU
### Unmanaged PDUs
### Managed PDUs

View File

@@ -0,0 +1,95 @@
# TSYS Group - HQ data center documentation - security
- [TSYS Group - HQ data center documentation - security](#tsys-group-hq-data-center-documentation-security)
- [Introduction](#introduction)
- [Badge reader](#badge-reader)
- [Hardware Components](#hardware-components)
- [Software Components](#software-components)
- [Cameras](#cameras)
- [Physical Keys/Badges](#physical-keys-badges)
- [Front Door (physical key)](#front-door-physical-key)
- [Server Room (rfid badge)](#server-room-rfid-badge)
- [Keybox in server room (physical key)](#keybox-in-server-room-physical-key)
- [Gates/Machine Room/Storage](#gates-machine-room-storage)
- [Critical Physical Assets](#critical-physical-assets)
- [server room](#server-room)
- [R&D Shop](#r-d-shop)
- [Amplify Credit Union](#amplify-credit-union)
## Introduction
This article covers the physical security setup for the HQ data center.
## Badge reader
### Hardware Components
- Raspberry Pi 3
- USB relay
- automated door action
- Belkin UPS for the relay
### Software Components
Coming soon
## Cameras
Internal facing
- <http://cam2.pfv.turnsys.net/> (door/rack front cam)
- <http://cam3.pfv.turnsys.net> (rack back cam)
- <http://cam1.pfv.turnsys.net/> (external camera)
## Physical Keys/Badges
### Front Door (physical key)
Charles Wyble
Patti Wyble
Michael Almaraz
### Server Room (rfid badge)
Charles Wyble
Patti Wyble
Michael Almaraz
### Keybox in server room (physical key)
Access to this box means you would have full physical access to all TSYS assets. Access is heavily restricted and granting of access grant requires approval of CEO/CFOO
and Board of Directors.
- Charles Wyble
- Patti Wyble
- Michael Almaraz
### Gates/Machine Room/Storage
- Charles Wyble
- Patti Wyble
- Michael Almaraz
## Critical Physical Assets
### server room
- racks
- air conditioner
- UPS systems
- Digital Information Processing Equipment (servers/drives/network)
- Sentry combination safe (on site cold storage for backup hard drives)
- PKI Safe
- Firebox for important paper records (Patti durable personal/corporate PoA, legal hold records)
- File cabinet (axios customer original contracts)
### R&D Shop
- lab area (tools/prototypes under development etc)
- tool storage and tools
- component storage and components
### Amplify Credit Union
- safety deposit box (off site cold storage for backup hard drives )
- Paper records
- safety deposit box (Patti durable PoA, legal hold records)

View File

@@ -0,0 +1,12 @@
# TSYS Group Storage
## Enclosures
## Arrays
## Block Storage
## Application Object Storage
## Container Object Storage