KNEL/TSYS-AIOS-GIS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add core architecture patterns and GIS/weather components from AIOS-Public

Browse Source
This commit is contained in:
Charles N Wyble - admin
2025-10-16 13:14:30 -05:00
parent 782eec63a5
commit 5887f4e729
32 changed files with 1970 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
Docker/README.md Normal file
View File

@@ -0,0 +1,40 @@
# TSYS-AIOS-GIS Docker Documentation
This directory contains organized Docker configurations for GIS and weather data processing tools in the TSYS-AIOS-GIS project.
## Container Structure
Each container has its own subdirectory with specific configuration files:
- `TSYS-AIOS-GIS-Tools-GIS-Base/` - Base GIS environment with core geospatial libraries
- `TSYS-AIOS-GIS-Tools-GIS-Processing/` - Advanced GIS processing with Jupyter notebooks
- `TSYS-AIOS-GIS-Tools-Weather-Base/` - Base weather data processing environment
- `TSYS-AIOS-GIS-Tools-Weather-Analysis/` - Advanced weather analysis with forecasting tools
## Container Naming Convention
All containers follow the `TSYS-AIOS-GIS-Tools-` naming convention with descriptive suffixes.
## Usage
### Building and Running Individual Containers
Each container has its own subdirectory with its Dockerfile and docker-compose.yml file.
```bash
# Navigate to the specific container directory
cd /home/localuser/AIWorkspace/TSYS-AIOS-GIS/Docker/TSYS-AIOS-GIS-Tools-GIS-Base
# Build the container
./docker-compose-wrapper.sh build
# Run the container
./docker-compose-wrapper.sh up --build
# Run a specific command in the container
./docker-compose-wrapper.sh run tsys-gis-base [command]
```
### Individual Container Documentation
For specific usage information for each container, see the README files in their respective subdirectories.

78
Docker/TSYS-AIOS-GIS-Tools-GIS-Base/Dockerfile Normal file
View File

@@ -0,0 +1,78 @@
FROM debian:bookworm-slim
# Avoid prompts from apt
ENV DEBIAN_FRONTEND=noninteractive
# Install base packages for GIS tools
RUN apt-get update && apt-get install -y \
bash \
curl \
wget \
git \
python3 \
python3-pip \
build-essential \
sudo \
&& rm -rf /var/lib/apt/lists/*
# Create symbolic link for python
RUN ln -s /usr/bin/python3 /usr/bin/python
# Install GIS tools
RUN apt-get update && apt-get install -y \
gdal-bin \
libgdal-dev \
proj-bin \
proj-data \
libproj-dev \
postgis \
postgresql-client \
&& rm -rf /var/lib/apt/lists/*
# Install DuckDB with spatial extensions
RUN curl -L https://github.com/duckdb/duckdb/releases/latest/download/duckdb_cli-linux-amd64.zip \
-o /tmp/duckdb.zip && \
cd /tmp && \
unzip duckdb.zip && \
cp duckdb_*_linux_amd64 /usr/local/bin/duckdb && \
chmod +x /usr/local/bin/duckdb && \
rm -rf /tmp/duckdb*
# Install Python GIS libraries
RUN pip3 install --break-system-packages \
geopandas \
shapely \
rasterio \
folium \
plotly \
xarray \
cfgrib \
netcdf4 \
matplotlib \
seaborn \
duckdb \
dask
# Install R with spatial packages (if needed)
RUN apt-get update && apt-get install -y \
r-base \
r-base-dev \
&& rm -rf /var/lib/apt/lists/*
# Install additional tools for weather data processing
RUN apt-get update && apt-get install -y \
ftp \
&& rm -rf /var/lib/apt/lists/*
# Install R spatial packages
RUN R --slave -e "install.packages(c('raster', 'rgdal', 'ncdf4', 'rasterVis', 'leaflet'), repos='https://cran.rstudio.com/', dependencies=TRUE)"
# Add entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# Create a working directory
WORKDIR /workspace
# Use the entrypoint script to handle user creation
ENTRYPOINT ["/entrypoint.sh"]

129
Docker/TSYS-AIOS-GIS-Tools-GIS-Base/README.md Normal file
View File

@@ -0,0 +1,129 @@
# TSYS-AIOS-GIS-Tools-GIS-Base Container
This container is part of the TSYS-AIOS-GIS project and provides a base GIS and weather data processing environment.
## Overview
The TSYS-AIOS-GIS-Tools-GIS-Base container is designed for GIS data processing and weather data analysis tasks. It includes essential tools for handling geospatial data formats and weather datasets with a focus on self-hosted GIS stack capabilities.
## Tools Included
### Core Tools
- **Base OS**: Debian Bookworm slim
- **Shell**: Bash
- **Programming Languages**:
- Python 3 with geospatial libraries
- R with spatial packages
### GIS Libraries
- **GDAL/OGR**: Geospatial Data Abstraction Library for format translation and processing
- **PROJ**: Coordinate transformation software
- **PostGIS**: Client tools for spatial database operations
- **DuckDB**: With spatial extensions for efficient data processing
- **GeoPandas**: Python geospatial data handling
- **Shapely**: Python geometric operations
- **Rasterio**: Raster processing in Python
### Weather Data Processing
- **xarray**: Multi-dimensional data in Python
- **cfgrib**: GRIB format handling
- **netCDF4**: NetCDF file handling
- **MetPy**: Meteorological calculations (via Python libraries)
### Visualization
- **Folium**: Interactive maps
- **Plotly**: Time series visualization
- **Matplotlib/Seaborn**: Statistical plots
- **R visualization packages**: For statistical analysis
### Additional Tools
- **Dask**: For large data processing
- **FTP client**: For bulk data downloads
## Usage
### Building the Base Container
```bash
# From this directory
cd /home/localuser/AIWorkspace/TSYS-AIOS-GIS/Docker/TSYS-AIOS-GIS-Tools-GIS-Base
# Use the wrapper script to automatically detect and set user IDs
./docker-compose-wrapper.sh build
# Or run commands in the base container with automatic user mapping
./docker-compose-wrapper.sh run tsys-gis-base [command]
# Example: Process a shapefile with GDAL
./docker-compose-wrapper.sh run tsys-gis-base ogrinfo /workspace/path/to/shapefile.shp
# Example: Start Python with geospatial libraries
./docker-compose-wrapper.sh run tsys-gis-base python3
# Example: Start R with spatial packages
./docker-compose-wrapper.sh run tsys-gis-base R
```
### Using with docker-compose directly
```bash
# Set environment variables and run docker-compose directly
LOCAL_USER_ID=$(id -u) LOCAL_GROUP_ID=$(id -g) docker-compose up --build
# Or export variables first
export LOCAL_USER_ID=$(id -u)
export LOCAL_GROUP_ID=$(id -g)
docker-compose up
```
### Using the wrapper script
```bash
# Build and start the base GIS container with automatic user mapping
./docker-compose-wrapper.sh up --build
# Start without rebuilding
./docker-compose-wrapper.sh up
# View container status
./docker-compose-wrapper.sh ps
# Stop containers
./docker-compose-wrapper.sh down
```
## User ID Mapping (For File Permissions)
The container automatically detects and uses the host user's UID and GID to ensure proper file permissions. This means:
- Files created inside the container will have the correct ownership on the host
- No more root-owned files after container operations
- Works across different environments (development, production servers)
The container detects the user ID from the mounted workspace volume. If needed, you can override the default values by setting environment variables:
```bash
# Set specific user ID and group ID before running docker-compose
export LOCAL_USER_ID=1000
export LOCAL_GROUP_ID=1000
docker-compose up
```
Or run with inline environment variables:
```bash
LOCAL_USER_ID=1000 LOCAL_GROUP_ID=1000 docker-compose up
```
The container runs as a non-root user named `TSYS-Tools` with the detected host user's UID/GID.
## Data Processing Workflows
This container is designed to handle both:
- GIS data processing (shapefiles, GeoJSON, Parquet, etc.)
- Weather data processing (GRIB, NetCDF formats)
- ETL workflows for geospatial and meteorological datasets
- Integration with PostGIS for spatial database operations
- Output to MinIO buckets for business use
## Integration
- Can be used in CTO mode for R&D activities
- Compatible with existing documentation containers for report generation
- Designed for both workstation prototyping and server deployment

67
Docker/TSYS-AIOS-GIS-Tools-GIS-Base/docker-compose-wrapper.sh Executable file
View File

@@ -0,0 +1,67 @@
#!/bin/bash
# docker-compose-wrapper.sh - Wrapper script to detect host UID/GID and run docker-compose
set -e # Exit on any error
# Detect the UID and GID of the user that owns the workspace directory (parent directory)
WORKSPACE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
echo "Detecting user ID from workspace directory: $WORKSPACE_DIR"
if [ -d "$WORKSPACE_DIR" ]; then
DETECTED_USER_ID=$(stat -c %u "$WORKSPACE_DIR" 2>/dev/null || echo 0)
DETECTED_GROUP_ID=$(stat -c %g "$WORKSPACE_DIR" 2>/dev/null || echo 0)
# If detection failed, try current user
if [ "$DETECTED_USER_ID" = "0" ]; then
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
else
# Fallback to current user if workspace directory doesn't exist
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
echo "Detected USER_ID=$DETECTED_USER_ID and GROUP_ID=$DETECTED_GROUP_ID"
# Set environment variables for docker-compose
export LOCAL_USER_ID=$DETECTED_USER_ID
export LOCAL_GROUP_ID=$DETECTED_GROUP_ID
# Show usage information
echo ""
echo "Usage: $0 [build|up|run <service> <command>|exec <service> <command>|down|ps]"
echo ""
echo "Examples:"
echo " $0 up # Start services"
echo " $0 build # Build containers"
echo " $0 run tsys-gis-base bash # Run command in container"
echo " $0 down # Stop and remove containers"
echo ""
# Check if docker compose (new format) or docker-compose (old format) is available
if command -v docker &> /dev/null && docker compose version &> /dev/null; then
# Use new docker compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker compose up'..."
docker compose up
else
# Execute the provided docker compose command
echo "Running: docker compose $*"
docker compose "$@"
fi
elif command -v docker-compose &> /dev/null; then
# Fallback to old docker-compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker-compose up'..."
docker-compose up
else
# Execute the provided docker-compose command
echo "Running: docker-compose $*"
docker-compose "$@"
fi
else
echo "Error: Neither 'docker compose' nor 'docker-compose' command found."
echo "Please install Docker Compose to use this script."
exit 1
fi

18
Docker/TSYS-AIOS-GIS-Tools-GIS-Base/docker-compose.yml Normal file
View File

@@ -0,0 +1,18 @@
version: '3.8'
services:
tsys-gis-base:
build:
context: .
dockerfile: Dockerfile
container_name: TSYS-AIOS-GIS-Tools-GIS-Base
image: tsys-aios-gis-tools-gis-base:latest
volumes:
- ../../../:/workspace:rw
working_dir: /workspace
stdin_open: true
tty: true
environment:
- LOCAL_USER_ID=${LOCAL_USER_ID:-1000}
- LOCAL_GROUP_ID=${LOCAL_GROUP_ID:-1000}
user: "${LOCAL_USER_ID:-1000}:${LOCAL_GROUP_ID:-1000}"

49
Docker/TSYS-AIOS-GIS-Tools-GIS-Base/entrypoint.sh Normal file
View File

@@ -0,0 +1,49 @@
#!/bin/bash
# entrypoint.sh - Entrypoint script to handle user creation and permission setup at runtime
# Set default values if not provided
USER_ID=${LOCAL_USER_ID:-$(id -u 1000)}
GROUP_ID=${LOCAL_GROUP_ID:-$(id -g 1000)}
# In case the environment variables are not set properly, detect them from the workspace volume
if [ "$USER_ID" = "$(id -u 0)" ] || [ "$USER_ID" = "0" ]; then
# Detect the UID and GID of the user that owns the workspace directory
if [ -d "/workspace" ]; then
USER_ID=$(stat -c %u /workspace 2>/dev/null || echo 1000)
GROUP_ID=$(stat -c %g /workspace 2>/dev/null || echo 1000)
else
USER_ID=${LOCAL_USER_ID:-1000}
GROUP_ID=${LOCAL_GROUP_ID:-1000}
fi
fi
echo "Starting with USER_ID=$USER_ID and GROUP_ID=$GROUP_ID"
# Create the group with specified GID
groupadd -f -g $GROUP_ID -o TSYS-Tools 2>/dev/null || groupmod -g $GROUP_ID -o TSYS-Tools
# Create the user with specified UID and add to the group
useradd -f -u $USER_ID -g $GROUP_ID -m -s /bin/bash -o TSYS-Tools 2>/dev/null || usermod -u $USER_ID -g $GROUP_ID -o TSYS-Tools
# Add user to sudo group for any necessary operations
usermod -aG sudo TSYS-Tools 2>/dev/null || true
# Make sure workspace directory exists and has proper permissions
mkdir -p /workspace
chown -R $USER_ID:$GROUP_ID /workspace
# Set up proper permissions for .local (if they exist)
mkdir -p /home/TSYS-Tools/.local
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.local
# Set up proper permissions for R (if they exist)
mkdir -p /home/TSYS-Tools/R
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/R
# If there are additional arguments, run them as the created user
if [ $# -gt 0 ]; then
exec su -p TSYS-Tools -c "$*"
else
# Otherwise start an interactive bash shell as the created user
exec su -p TSYS-Tools -c "/bin/bash"
fi

32
Docker/TSYS-AIOS-GIS-Tools-GIS-Processing/Dockerfile Normal file
View File

@@ -0,0 +1,32 @@
FROM tsys-aios-gis-tools-gis-base:latest
# Avoid prompts from apt
ENV DEBIAN_FRONTEND=noninteractive
# Install additional processing tools
RUN apt-get update && apt-get install -y \
jupyter-notebook \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/*
# Install additional Python libraries for processing
RUN pip3 install --break-system-packages \
jupyter \
notebook \
ipykernel \
apache-airflow \
prefect
# Set up Jupyter
RUN jupyter notebook --generate-config --allow-root --ip=0.0.0.0 --port=8888 --no-browser --notebook-dir=/workspace --NotebookApp.token='' --NotebookApp.password='' 2>/dev/null || true
# Create a working directory
WORKDIR /workspace
# Add entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# Use the entrypoint script to handle user creation
ENTRYPOINT ["/entrypoint.sh"]

111
Docker/TSYS-AIOS-GIS-Tools-GIS-Processing/README.md Normal file
View File

@@ -0,0 +1,111 @@
# TSYS-AIOS-GIS-Tools-GIS-Processing Container
This container is part of the TSYS-AIOS-GIS project and provides advanced GIS data processing capabilities with Jupyter notebooks and workflow tools.
## Overview
The TSYS-AIOS-GIS-Tools-GIS-Processing container extends the base GIS container with advanced processing tools, Jupyter notebooks for interactive analysis, and workflow orchestration tools. This container is designed for in-depth geospatial data analysis and complex ETL workflows.
## Tools Included
### Extends from Base Container
- All tools from TSYS-AIOS-GIS-Tools-GIS-Base container
- GIS libraries, weather data processing libraries, visualization tools
### Advanced Processing Tools
- **Jupyter Notebook**: Interactive environment for data analysis
- **Node.js/npm**: JavaScript runtime and package manager
- **IPyKernel**: IPython kernel for Jupyter
### Workflow Tools
- **Apache Airflow**: Workflow orchestration platform
- **Prefect**: Modern workflow management
## Usage
### Building the Processing Container
```bash
# From this directory
cd /home/localuser/AIWorkspace/TSYS-AIOS-GIS/Docker/TSYS-AIOS-GIS-Tools-GIS-Processing
# Use the wrapper script to automatically detect and set user IDs
./docker-compose-wrapper.sh build
# Or run commands in the processing container with automatic user mapping
./docker-compose-wrapper.sh run tsys-gis-processing [command]
# Example: Start Jupyter notebook server
./docker-compose-wrapper.sh run tsys-gis-processing jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root --notebook-dir=/workspace --no-browser
# Example: Start an interactive bash session
./docker-compose-wrapper.sh run tsys-gis-processing bash
```
### Using with docker-compose directly
```bash
# Set environment variables and run docker-compose directly
LOCAL_USER_ID=$(id -u) LOCAL_GROUP_ID=$(id -g) docker-compose up --build
# Or export variables first
export LOCAL_USER_ID=$(id -u)
export LOCAL_GROUP_ID=$(id -g)
docker-compose up
```
### Using the wrapper script
```bash
# Build and start the processing container with automatic user mapping
./docker-compose-wrapper.sh up --build
# Start without rebuilding (Jupyter will be available on port 8888)
./docker-compose-wrapper.sh up
# View container status
./docker-compose-wrapper.sh ps
# Stop containers
./docker-compose-wrapper.sh down
```
## Jupyter Notebook Access
When running the container with `docker-compose up`, Jupyter notebook will be available at:
- http://localhost:8888
The notebook server is preconfigured to:
- Use the workspace directory as the notebook directory
- Allow access without authentication (in container only)
- Accept connections from any IP address
## User ID Mapping (For File Permissions)
The container automatically detects and uses the host user's UID and GID to ensure proper file permissions. This means:
- Files created inside the container will have the correct ownership on the host
- No more root-owned files after container operations
- Works across different environments (development, production servers)
The container detects the user ID from the mounted workspace volume. If needed, you can override the default values by setting environment variables:
```bash
# Set specific user ID and group ID before running docker-compose
export LOCAL_USER_ID=1000
export LOCAL_GROUP_ID=1000
docker-compose up
```
Or run with inline environment variables:
```bash
LOCAL_USER_ID=1000 LOCAL_GROUP_ID=1000 docker-compose up
```
The container runs as a non-root user named `TSYS-Tools` with the detected host user's UID/GID.
## Data Processing Workflows
This container is optimized for:
- Interactive geospatial analysis using Jupyter notebooks
- Complex ETL workflows using Apache Airflow or Prefect
- Advanced visualization and reporting
- Model development and testing
- Integration with PostGIS and other databases

67
Docker/TSYS-AIOS-GIS-Tools-GIS-Processing/docker-compose-wrapper.sh Executable file
View File

@@ -0,0 +1,67 @@
#!/bin/bash
# docker-compose-wrapper.sh - Wrapper script to detect host UID/GID and run docker-compose
set -e # Exit on any error
# Detect the UID and GID of the user that owns the workspace directory (parent directory)
WORKSPACE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
echo "Detecting user ID from workspace directory: $WORKSPACE_DIR"
if [ -d "$WORKSPACE_DIR" ]; then
DETECTED_USER_ID=$(stat -c %u "$WORKSPACE_DIR" 2>/dev/null || echo 0)
DETECTED_GROUP_ID=$(stat -c %g "$WORKSPACE_DIR" 2>/dev/null || echo 0)
# If detection failed, try current user
if [ "$DETECTED_USER_ID" = "0" ]; then
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
else
# Fallback to current user if workspace directory doesn't exist
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
echo "Detected USER_ID=$DETECTED_USER_ID and GROUP_ID=$DETECTED_GROUP_ID"
# Set environment variables for docker-compose
export LOCAL_USER_ID=$DETECTED_USER_ID
export LOCAL_GROUP_ID=$DETECTED_GROUP_ID
# Show usage information
echo ""
echo "Usage: $0 [build|up|run <service> <command>|exec <service> <command>|down|ps]"
echo ""
echo "Examples:"
echo " $0 up # Start services"
echo " $0 build # Build containers"
echo " $0 run tsys-gis-processing bash # Run command in container"
echo " $0 down # Stop and remove containers"
echo ""
# Check if docker compose (new format) or docker-compose (old format) is available
if command -v docker &> /dev/null && docker compose version &> /dev/null; then
# Use new docker compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker compose up'..."
docker compose up
else
# Execute the provided docker compose command
echo "Running: docker compose $*"
docker compose "$@"
fi
elif command -v docker-compose &> /dev/null; then
# Fallback to old docker-compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker-compose up'..."
docker-compose up
else
# Execute the provided docker-compose command
echo "Running: docker-compose $*"
docker-compose "$@"
fi
else
echo "Error: Neither 'docker compose' nor 'docker-compose' command found."
echo "Please install Docker Compose to use this script."
exit 1
fi

20
Docker/TSYS-AIOS-GIS-Tools-GIS-Processing/docker-compose.yml Normal file
View File

@@ -0,0 +1,20 @@
version: '3.8'
services:
tsys-gis-processing:
build:
context: .
dockerfile: Dockerfile
container_name: TSYS-AIOS-GIS-Tools-GIS-Processing
image: tsys-aios-gis-tools-gis-processing:latest
volumes:
- ../../../:/workspace:rw
working_dir: /workspace
stdin_open: true
tty: true
ports:
- "8888:8888"
environment:
- LOCAL_USER_ID=${LOCAL_USER_ID:-1000}
- LOCAL_GROUP_ID=${LOCAL_GROUP_ID:-1000}
user: "${LOCAL_USER_ID:-1000}:${LOCAL_GROUP_ID:-1000}"

49
Docker/TSYS-AIOS-GIS-Tools-GIS-Processing/entrypoint.sh Normal file
View File

@@ -0,0 +1,49 @@
#!/bin/bash
# entrypoint.sh - Entrypoint script to handle user creation and permission setup at runtime
# Set default values if not provided
USER_ID=${LOCAL_USER_ID:-$(id -u 1000)}
GROUP_ID=${LOCAL_GROUP_ID:-$(id -g 1000)}
# In case the environment variables are not set properly, detect them from the workspace volume
if [ "$USER_ID" = "$(id -u 0)" ] || [ "$USER_ID" = "0" ]; then
# Detect the UID and GID of the user that owns the workspace directory
if [ -d "/workspace" ]; then
USER_ID=$(stat -c %u /workspace 2>/dev/null || echo 1000)
GROUP_ID=$(stat -c %g /workspace 2>/dev/null || echo 1000)
else
USER_ID=${LOCAL_USER_ID:-1000}
GROUP_ID=${LOCAL_GROUP_ID:-1000}
fi
fi
echo "Starting with USER_ID=$USER_ID and GROUP_ID=$GROUP_ID"
# Create the group with specified GID
groupadd -f -g $GROUP_ID -o TSYS-Tools 2>/dev/null || groupmod -g $GROUP_ID -o TSYS-Tools
# Create the user with specified UID and add to the group
useradd -f -u $USER_ID -g $GROUP_ID -m -s /bin/bash -o TSYS-Tools 2>/dev/null || usermod -u $USER_ID -g $GROUP_ID -o TSYS-Tools
# Add user to sudo group for any necessary operations
usermod -aG sudo TSYS-Tools 2>/dev/null || true
# Make sure workspace directory exists and has proper permissions
mkdir -p /workspace
chown -R $USER_ID:$GROUP_ID /workspace
# Set up proper permissions for .local (if they exist)
mkdir -p /home/TSYS-Tools/.local
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.local
# Set up proper permissions for Jupyter (if they exist)
mkdir -p /home/TSYS-Tools/.jupyter
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.jupyter
# If there are additional arguments, run them as the created user
if [ $# -gt 0 ]; then
exec su -p TSYS-Tools -c "$*"
else
# Otherwise start an interactive bash shell as the created user
exec su -p TSYS-Tools -c "/bin/bash"
fi

40
Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis/Dockerfile Normal file
View File

@@ -0,0 +1,40 @@
FROM tsys-aios-gis-tools-weather-base:latest
# Avoid prompts from apt
ENV DEBIAN_FRONTEND=noninteractive
# Install additional analysis tools
RUN apt-get update && apt-get install -y \
jupyter-notebook \
nodejs \
npm \
octave \
&& rm -rf /var/lib/apt/lists/*
# Install additional Python libraries for weather analysis
RUN pip3 install --break-system-packages \
jupyter \
notebook \
ipykernel \
apache-airflow \
prefect \
folium \
plotly \
cartopy \
geoviews
# Install additional R packages for weather analysis
RUN R --slave -e "install.packages(c('lubridate', 'ggplot2', 'dplyr', 'tidyr', 'forecast', 'RColorBrewer'), repos='https://cran.rstudio.com/', dependencies=TRUE)"
# Set up Jupyter
RUN jupyter notebook --generate-config --allow-root --ip=0.0.0.0 --port=8888 --no-browser --notebook-dir=/workspace --NotebookApp.token='' --NotebookApp.password='' 2>/dev/null || true
# Create a working directory
WORKDIR /workspace
# Add entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# Use the entrypoint script to handle user creation
ENTRYPOINT ["/entrypoint.sh"]

124
Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis/README.md Normal file
View File

@@ -0,0 +1,124 @@
# TSYS-AIOS-GIS-Tools-Weather-Analysis Container
This container is part of the TSYS-AIOS-GIS project and provides advanced weather data analysis capabilities with Jupyter notebooks and specialized meteorological tools.
## Overview
The TSYS-AIOS-GIS-Tools-Weather-Analysis container extends the base weather container with advanced analysis tools, Jupyter notebooks for interactive analysis, and specialized meteorological libraries. This container is designed for in-depth weather data analysis and balloon path prediction work.
## Tools Included
### Extends from Base Container
- All tools from TSYS-AIOS-GIS-Tools-Weather-Base container
- Weather data processing libraries, APIs, bulk download tools
### Advanced Analysis Tools
- **Jupyter Notebook**: Interactive environment for weather analysis
- **Node.js/npm**: JavaScript runtime and package manager
- **IPyKernel**: IPython kernel for Jupyter
- **GNU Octave**: Numerical computations (similar to MATLAB)
### Visualization & Forecasting Libraries
- **Cartopy**: Geospatial processing and visualization
- **Geoviews**: Geospatial data visualization
- **Folium**: Interactive maps with weather data overlays
- **Plotly**: Interactive weather visualizations
- **Forecast R package**: Time series forecasting
### Additional R packages
- **Lubridate**: Time series manipulation
- **Ggplot2/Tidyr/Dplyr**: Data analysis and visualization
- **RColorBrewer**: Color palettes for weather maps
## Usage
### Building the Weather Analysis Container
```bash
# From this directory
cd /home/localuser/AIWorkspace/TSYS-AIOS-GIS/Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis
# Use the wrapper script to automatically detect and set user IDs
./docker-compose-wrapper.sh build
# Or run commands in the analysis container with automatic user mapping
./docker-compose-wrapper.sh run tsys-weather-analysis [command]
# Example: Start Jupyter notebook server
./docker-compose-wrapper.sh run tsys-weather-analysis jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root --notebook-dir=/workspace --no-browser
# Example: Start Octave for numerical computations
./docker-compose-wrapper.sh run tsys-weather-analysis octave
# Example: Start an interactive bash session
./docker-compose-wrapper.sh run tsys-weather-analysis bash
```
### Using with docker-compose directly
```bash
# Set environment variables and run docker-compose directly
LOCAL_USER_ID=$(id -u) LOCAL_GROUP_ID=$(id -g) docker-compose up --build
# Or export variables first
export LOCAL_USER_ID=$(id -u)
export LOCAL_GROUP_ID=$(id -g)
docker-compose up
```
### Using the wrapper script
```bash
# Build and start the analysis container with automatic user mapping
./docker-compose-wrapper.sh up --build
# Start without rebuilding (Jupyter will be available on port 8889)
./docker-compose-wrapper.sh up
# View container status
./docker-compose-wrapper.sh ps
# Stop containers
./docker-compose-wrapper.sh down
```
## Jupyter Notebook Access
When running the container with `docker-compose up`, Jupyter notebook will be available at:
- http://localhost:8889
The notebook server is preconfigured to:
- Use the workspace directory as the notebook directory
- Allow access without authentication (in container only)
- Accept connections from any IP address
## User ID Mapping (For File Permissions)
The container automatically detects and uses the host user's UID and GID to ensure proper file permissions. This means:
- Files created inside the container will have the correct ownership on the host
- No more root-owned files after container operations
- Works across different environments (development, production servers)
The container detects the user ID from the mounted workspace volume. If needed, you can override the default values by setting environment variables:
```bash
# Set specific user ID and group ID before running docker-compose
export LOCAL_USER_ID=1000
export LOCAL_GROUP_ID=1000
docker-compose up
```
Or run with inline environment variables:
```bash
LOCAL_USER_ID=1000 LOCAL_GROUP_ID=1000 docker-compose up
```
The container runs as a non-root user named `TSYS-Tools` with the detected host user's UID/GID.
## Weather Analysis Workflows
This container is optimized for:
- Interactive weather data analysis using Jupyter notebooks
- Balloon path prediction using weather data
- Advanced meteorological calculations
- Time series forecasting
- Weather data visualization
- Climate analysis workflows

67
Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis/docker-compose-wrapper.sh Executable file
View File

@@ -0,0 +1,67 @@
#!/bin/bash
# docker-compose-wrapper.sh - Wrapper script to detect host UID/GID and run docker-compose
set -e # Exit on any error
# Detect the UID and GID of the user that owns the workspace directory (parent directory)
WORKSPACE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
echo "Detecting user ID from workspace directory: $WORKSPACE_DIR"
if [ -d "$WORKSPACE_DIR" ]; then
DETECTED_USER_ID=$(stat -c %u "$WORKSPACE_DIR" 2>/dev/null || echo 0)
DETECTED_GROUP_ID=$(stat -c %g "$WORKSPACE_DIR" 2>/dev/null || echo 0)
# If detection failed, try current user
if [ "$DETECTED_USER_ID" = "0" ]; then
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
else
# Fallback to current user if workspace directory doesn't exist
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
echo "Detected USER_ID=$DETECTED_USER_ID and GROUP_ID=$DETECTED_GROUP_ID"
# Set environment variables for docker-compose
export LOCAL_USER_ID=$DETECTED_USER_ID
export LOCAL_GROUP_ID=$DETECTED_GROUP_ID
# Show usage information
echo ""
echo "Usage: $0 [build|up|run <service> <command>|exec <service> <command>|down|ps]"
echo ""
echo "Examples:"
echo " $0 up # Start services"
echo " $0 build # Build containers"
echo " $0 run tsys-weather-analysis bash # Run command in container"
echo " $0 down # Stop and remove containers"
echo ""
# Check if docker compose (new format) or docker-compose (old format) is available
if command -v docker &> /dev/null && docker compose version &> /dev/null; then
# Use new docker compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker compose up'..."
docker compose up
else
# Execute the provided docker compose command
echo "Running: docker compose $*"
docker compose "$@"
fi
elif command -v docker-compose &> /dev/null; then
# Fallback to old docker-compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker-compose up'..."
docker-compose up
else
# Execute the provided docker-compose command
echo "Running: docker-compose $*"
docker-compose "$@"
fi
else
echo "Error: Neither 'docker compose' nor 'docker-compose' command found."
echo "Please install Docker Compose to use this script."
exit 1
fi

20
Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis/docker-compose.yml Normal file
View File

@@ -0,0 +1,20 @@
version: '3.8'
services:
tsys-weather-analysis:
build:
context: .
dockerfile: Dockerfile
container_name: TSYS-AIOS-GIS-Tools-Weather-Analysis
image: tsys-aios-gis-tools-weather-analysis:latest
volumes:
- ../../../:/workspace:rw
working_dir: /workspace
stdin_open: true
tty: true
ports:
- "8889:8888"
environment:
- LOCAL_USER_ID=${LOCAL_USER_ID:-1000}
- LOCAL_GROUP_ID=${LOCAL_GROUP_ID:-1000}
user: "${LOCAL_USER_ID:-1000}:${LOCAL_GROUP_ID:-1000}"

49
Docker/TSYS-AIOS-GIS-Tools-Weather-Analysis/entrypoint.sh Normal file
View File

@@ -0,0 +1,49 @@
#!/bin/bash
# entrypoint.sh - Entrypoint script to handle user creation and permission setup at runtime
# Set default values if not provided
USER_ID=${LOCAL_USER_ID:-$(id -u 1000)}
GROUP_ID=${LOCAL_GROUP_ID:-$(id -g 1000)}
# In case the environment variables are not set properly, detect them from the workspace volume
if [ "$USER_ID" = "$(id -u 0)" ] || [ "$USER_ID" = "0" ]; then
# Detect the UID and GID of the user that owns the workspace directory
if [ -d "/workspace" ]; then
USER_ID=$(stat -c %u /workspace 2>/dev/null || echo 1000)
GROUP_ID=$(stat -c %g /workspace 2>/dev/null || echo 1000)
else
USER_ID=${LOCAL_USER_ID:-1000}
GROUP_ID=${LOCAL_GROUP_ID:-1000}
fi
fi
echo "Starting with USER_ID=$USER_ID and GROUP_ID=$GROUP_ID"
# Create the group with specified GID
groupadd -f -g $GROUP_ID -o TSYS-Tools 2>/dev/null || groupmod -g $GROUP_ID -o TSYS-Tools
# Create the user with specified UID and add to the group
useradd -f -u $USER_ID -g $GROUP_ID -m -s /bin/bash -o TSYS-Tools 2>/dev/null || usermod -u $USER_ID -g $GROUP_ID -o TSYS-Tools
# Add user to sudo group for any necessary operations
usermod -aG sudo TSYS-Tools 2>/dev/null || true
# Make sure workspace directory exists and has proper permissions
mkdir -p /workspace
chown -R $USER_ID:$GROUP_ID /workspace
# Set up proper permissions for .local (if they exist)
mkdir -p /home/TSYS-Tools/.local
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.local
# Set up proper permissions for Jupyter (if they exist)
mkdir -p /home/TSYS-Tools/.jupyter
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.jupyter
# If there are additional arguments, run them as the created user
if [ $# -gt 0 ]; then
exec su -p TSYS-Tools -c "$*"
else
# Otherwise start an interactive bash shell as the created user
exec su -p TSYS-Tools -c "/bin/bash"
fi

72
Docker/TSYS-AIOS-GIS-Tools-Weather-Base/Dockerfile Normal file
View File

@@ -0,0 +1,72 @@
FROM debian:bookworm-slim
# Avoid prompts from apt
ENV DEBIAN_FRONTEND=noninteractive
# Install base packages for weather tools
RUN apt-get update && apt-get install -y \
bash \
curl \
wget \
git \
python3 \
python3-pip \
build-essential \
sudo \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Create symbolic link for python
RUN ln -s /usr/bin/python3 /usr/bin/python
# Install weather data processing tools
RUN apt-get update && apt-get install -y \
ftp \
gfortran \
libgfortran5 \
&& rm -rf /var/lib/apt/lists/*
# Install Python weather libraries
RUN pip3 install --break-system-packages \
xarray \
cfgrib \
netcdf4 \
metpy \
siphon \
numpy \
pandas \
scipy \
matplotlib \
seaborn \
requests \
cdo-python
# Install Climate Data Operators (CDO)
RUN apt-get update && apt-get install -y \
cdo \
&& rm -rf /var/lib/apt/lists/*
# Install additional tools for weather data operations
RUN apt-get update && apt-get install -y \
nco \
ncl-ncarg \
&& rm -rf /var/lib/apt/lists/*
# Install R for statistical analysis
RUN apt-get update && apt-get install -y \
r-base \
r-base-dev \
&& rm -rf /var/lib/apt/lists/*
# Install R weather packages
RUN R --slave -e "install.packages(c('raster', 'rgdal', 'ncdf4', 'rasterVis', 'ncmeta'), repos='https://cran.rstudio.com/', dependencies=TRUE)"
# Add entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# Create a working directory
WORKDIR /workspace
# Use the entrypoint script to handle user creation
ENTRYPOINT ["/entrypoint.sh"]

121
Docker/TSYS-AIOS-GIS-Tools-Weather-Base/README.md Normal file
View File

@@ -0,0 +1,121 @@
# TSYS-AIOS-GIS-Tools-Weather-Base Container
This container is part of the TSYS-AIOS-GIS project and provides a base weather data processing environment.
## Overview
The TSYS-AIOS-GIS-Tools-Weather-Base container is designed for weather data processing and analysis tasks. It includes essential tools for handling weather data formats (GRIB, NetCDF), accessing weather APIs, and performing meteorological calculations.
## Tools Included
### Core Tools
- **Base OS**: Debian Bookworm slim
- **Shell**: Bash
- **Programming Languages**:
- Python 3 with meteorological libraries
- R with weather analysis packages
### Weather Data Processing Libraries
- **xarray**: Multi-dimensional data in Python
- **cfgrib**: GRIB format handling
- **netCDF4**: NetCDF file handling
- **MetPy**: Meteorological calculations
- **Siphon**: Access to weather data from various sources
- **Numpy/Pandas/Scipy**: Scientific computing libraries
### Climate Data Operators
- **CDO (Climate Data Operators)**: Tools for climate data processing
- **NCO (NetCDF Operators)**: Operators for NetCDF files
- **NCL (NCAR Command Language)**: For complex climate analysis
### Visualization & Analysis
- **Matplotlib/Seaborn**: Statistical plots
- **Requests**: HTTP library for API access
- **R packages**: raster, rgdal, ncdf4, rasterVis, ncmeta
### Additional Tools
- **FTP client**: For bulk weather data downloads
- **GFortran**: For compiling Fortran-based weather tools
## Usage
### Building the Weather Base Container
```bash
# From this directory
cd /home/localuser/AIWorkspace/TSYS-AIOS-GIS/Docker/TSYS-AIOS-GIS-Tools-Weather-Base
# Use the wrapper script to automatically detect and set user IDs
./docker-compose-wrapper.sh build
# Or run commands in the weather container with automatic user mapping
./docker-compose-wrapper.sh run tsys-weather-base [command]
# Example: Process a GRIB file with cfgrib
./docker-compose-wrapper.sh run tsys-weather-base python3 -c "import xarray as xr; ds = xr.open_dataset('file.grib', engine='cfgrib'); print(ds)"
# Example: Use CDO to process NetCDF files
./docker-compose-wrapper.sh run tsys-weather-base cdo info /workspace/weather_data.nc
# Example: Start Python with weather libraries
./docker-compose-wrapper.sh run tsys-weather-base python3
```
### Using with docker-compose directly
```bash
# Set environment variables and run docker-compose directly
LOCAL_USER_ID=$(id -u) LOCAL_GROUP_ID=$(id -g) docker-compose up --build
# Or export variables first
export LOCAL_USER_ID=$(id -u)
export LOCAL_GROUP_ID=$(id -g)
docker-compose up
```
### Using the wrapper script
```bash
# Build and start the base weather container with automatic user mapping
./docker-compose-wrapper.sh up --build
# Start without rebuilding
./docker-compose-wrapper.sh up
# View container status
./docker-compose-wrapper.sh ps
# Stop containers
./docker-compose-wrapper.sh down
```
## Weather Data Processing Workflows
This container is designed to handle:
- GRIB data format processing
- NetCDF data analysis
- NOAA and European weather API integration
- Bulk data download via HTTP/FTP
- Meteorological calculations
- Climate data processing with CDO/NCO
## User ID Mapping (For File Permissions)
The container automatically detects and uses the host user's UID and GID to ensure proper file permissions. This means:
- Files created inside the container will have the correct ownership on the host
- No more root-owned files after container operations
- Works across different environments (development, production servers)
The container detects the user ID from the mounted workspace volume. If needed, you can override the default values by setting environment variables:
```bash
# Set specific user ID and group ID before running docker-compose
export LOCAL_USER_ID=1000
export LOCAL_GROUP_ID=1000
docker-compose up
```
Or run with inline environment variables:
```bash
LOCAL_USER_ID=1000 LOCAL_GROUP_ID=1000 docker-compose up
```
The container runs as a non-root user named `TSYS-Tools` with the detected host user's UID/GID.

67
Docker/TSYS-AIOS-GIS-Tools-Weather-Base/docker-compose-wrapper.sh Executable file
View File

@@ -0,0 +1,67 @@
#!/bin/bash
# docker-compose-wrapper.sh - Wrapper script to detect host UID/GID and run docker-compose
set -e # Exit on any error
# Detect the UID and GID of the user that owns the workspace directory (parent directory)
WORKSPACE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
echo "Detecting user ID from workspace directory: $WORKSPACE_DIR"
if [ -d "$WORKSPACE_DIR" ]; then
DETECTED_USER_ID=$(stat -c %u "$WORKSPACE_DIR" 2>/dev/null || echo 0)
DETECTED_GROUP_ID=$(stat -c %g "$WORKSPACE_DIR" 2>/dev/null || echo 0)
# If detection failed, try current user
if [ "$DETECTED_USER_ID" = "0" ]; then
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
else
# Fallback to current user if workspace directory doesn't exist
DETECTED_USER_ID=$(id -u)
DETECTED_GROUP_ID=$(id -g)
fi
echo "Detected USER_ID=$DETECTED_USER_ID and GROUP_ID=$DETECTED_GROUP_ID"
# Set environment variables for docker-compose
export LOCAL_USER_ID=$DETECTED_USER_ID
export LOCAL_GROUP_ID=$DETECTED_GROUP_ID
# Show usage information
echo ""
echo "Usage: $0 [build|up|run <service> <command>|exec <service> <command>|down|ps]"
echo ""
echo "Examples:"
echo " $0 up # Start services"
echo " $0 build # Build containers"
echo " $0 run tsys-weather-base bash # Run command in container"
echo " $0 down # Stop and remove containers"
echo ""
# Check if docker compose (new format) or docker-compose (old format) is available
if command -v docker &> /dev/null && docker compose version &> /dev/null; then
# Use new docker compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker compose up'..."
docker compose up
else
# Execute the provided docker compose command
echo "Running: docker compose $*"
docker compose "$@"
fi
elif command -v docker-compose &> /dev/null; then
# Fallback to old docker-compose format
if [ $# -eq 0 ]; then
echo "No command provided. Running 'docker-compose up'..."
docker-compose up
else
# Execute the provided docker-compose command
echo "Running: docker-compose $*"
docker-compose "$@"
fi
else
echo "Error: Neither 'docker compose' nor 'docker-compose' command found."
echo "Please install Docker Compose to use this script."
exit 1
fi

18
Docker/TSYS-AIOS-GIS-Tools-Weather-Base/docker-compose.yml Normal file
View File

@@ -0,0 +1,18 @@
version: '3.8'
services:
tsys-weather-base:
build:
context: .
dockerfile: Dockerfile
container_name: TSYS-AIOS-GIS-Tools-Weather-Base
image: tsys-aios-gis-tools-weather-base:latest
volumes:
- ../../../:/workspace:rw
working_dir: /workspace
stdin_open: true
tty: true
environment:
- LOCAL_USER_ID=${LOCAL_USER_ID:-1000}
- LOCAL_GROUP_ID=${LOCAL_GROUP_ID:-1000}
user: "${LOCAL_USER_ID:-1000}:${LOCAL_GROUP_ID:-1000}"

49
Docker/TSYS-AIOS-GIS-Tools-Weather-Base/entrypoint.sh Normal file
View File

@@ -0,0 +1,49 @@
#!/bin/bash
# entrypoint.sh - Entrypoint script to handle user creation and permission setup at runtime
# Set default values if not provided
USER_ID=${LOCAL_USER_ID:-$(id -u 1000)}
GROUP_ID=${LOCAL_GROUP_ID:-$(id -g 1000)}
# In case the environment variables are not set properly, detect them from the workspace volume
if [ "$USER_ID" = "$(id -u 0)" ] || [ "$USER_ID" = "0" ]; then
# Detect the UID and GID of the user that owns the workspace directory
if [ -d "/workspace" ]; then
USER_ID=$(stat -c %u /workspace 2>/dev/null || echo 1000)
GROUP_ID=$(stat -c %g /workspace 2>/dev/null || echo 1000)
else
USER_ID=${LOCAL_USER_ID:-1000}
GROUP_ID=${LOCAL_GROUP_ID:-1000}
fi
fi
echo "Starting with USER_ID=$USER_ID and GROUP_ID=$GROUP_ID"
# Create the group with specified GID
groupadd -f -g $GROUP_ID -o TSYS-Tools 2>/dev/null || groupmod -g $GROUP_ID -o TSYS-Tools
# Create the user with specified UID and add to the group
useradd -f -u $USER_ID -g $GROUP_ID -m -s /bin/bash -o TSYS-Tools 2>/dev/null || usermod -u $USER_ID -g $GROUP_ID -o TSYS-Tools
# Add user to sudo group for any necessary operations
usermod -aG sudo TSYS-Tools 2>/dev/null || true
# Make sure workspace directory exists and has proper permissions
mkdir -p /workspace
chown -R $USER_ID:$GROUP_ID /workspace
# Set up proper permissions for .local (if they exist)
mkdir -p /home/TSYS-Tools/.local
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/.local
# Set up proper permissions for R (if they exist)
mkdir -p /home/TSYS-Tools/R
chown $USER_ID:$GROUP_ID /home/TSYS-Tools/R
# If there are additional arguments, run them as the created user
if [ $# -gt 0 ]; then
exec su -p TSYS-Tools -c "$*"
else
# Otherwise start an interactive bash shell as the created user
exec su -p TSYS-Tools -c "/bin/bash"
fi