KNEL/TSYSDevStack-SupportStack-LocalWorkstation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
25f7a6cd75035ea0d869e1cad5a28027e710d46e
TSYSDevStack-SupportStack-L…/AGENTS.md
reachableceo 25f7a6cd75 feat(demo): migrate 5 SelfStack services to demo stack (16→24 services) 
Add Reactive Resume, Metrics, Kiwix, Resume Matcher, and Apple Health
from the earlier SelfStack project. Rewrite Apple Health collector to
use InfluxDB v2 with proper error handling. Update all tests, scripts,
Homepage config, env template, and documentation for the expanded stack.

New services:
- Reactive Resume (4016) + Postgres/Minio/Chrome companions
- Metrics (4021) - GitHub metrics visualization
- Kiwix (4022) - offline wiki reader
- Resume Matcher (4023) - AI resume screening
- Apple Health (4024) - health data collector → InfluxDB v2

Also adds git policy to AGENTS.md: always commit and push automatically.

💘 Generated with Crush

Assisted-by: GLM-5.1 via Crush <crush@charm.land>
2026-05-08 12:28:56 -05:00

435 lines
18 KiB
Markdown
Raw Blame History

# TSYS Developer Support Stack - Agent Guide
## Project Overview
This repository contains a Docker Compose-based multi-service stack that provides off-the-shelf applications running locally on developer workstations. It's designed to enhance developer productivity and quality of life for the TSYS engineering team.
### Project Type
- **Infrastructure as Code**: Docker Compose with shell orchestration
- **Multi-Service Stack**: 24 services across 5 categories
- **Demo-First Architecture**: All configurations for demonstration purposes only
### Git Policy
- **ALWAYS commit** after every logical unit of work — never ask permission
- **ALWAYS push** after every commit — never ask permission
- **Commit early, commit often** — small focused commits are preferred over large ones
- **Never ask** "should I commit?" or "should I push?" — just do it
### Directory Structure
```
TSYSDevStack-SupportStack-LocalWorkstation/
├── LICENSE
├── README.md # Root-level documentation
├── AGENTS.md # This file - agent guide
├── demo/ # Demo environment (primary focus)
│ ├── AGENTS.md # Detailed development guidelines
│ ├── PRD.md # Product Requirements Document
│ ├── README.md # Demo-specific documentation
│ ├── demo.env # Environment variables (dynamic)
│ ├── docker-compose.yml # Generated compose file
│ ├── docker-compose.yml.template # Template for compose file
│ ├── scripts/ # Orchestration scripts
│ │ ├── demo-stack.sh # Deployment/management
│ │ ├── demo-test.sh # QA/testing suite
│ │ └── validate-all.sh # Pre-deployment validation
│ ├── tests/ # Test suite
│ │ ├── unit/ # Unit tests
│ │ ├── integration/ # Integration tests
│ │ └── e2e/ # End-to-end tests
│ ├── config/ # Service configurations
│ │ ├── homepage/ # Homepage dashboard config
│ │ ├── grafana/ # Grafana dashboards/datasources
│ │ ├── pihole/ # Pi-hole configuration
│ │ ├── dockhand/ # Dockhand configuration
│ │ ├── influxdb/ # InfluxDB configuration
│ │ ├── drawio/ # Draw.io configuration
│ │ ├── kroki/ # Kroki configuration
│ │ ├── atomictracker/ # Atomic Tracker configuration
│ │ ├── archivebox/ # ArchiveBox configuration
│ │ ├── tubearchivist/ # Tube Archivist configuration
│ │ ├── wakapi/ # Wakapi configuration
│ │ ├── mailhog/ # MailHog configuration
│ │ ├── applehealth/ # Apple Health configuration
│ │ ├── metrics/ # Metrics configuration
│ │ ├── reactiveresume/ # Reactive Resume configuration
│ │ ├── kiwix/ # Kiwix configuration
│ │ ├── resumematcher/ # Resume Matcher configuration
│ │ └── atuin/ # Atuin configuration
│ └── docs/ # Additional documentation
│ ├── service-guides/ # Service-specific guides
│ ├── troubleshooting/ # Troubleshooting documentation
│ └── api-docs/ # API documentation
└── prod/ # Production environment (placeholder)
```
## Essential Commands
### Deployment & Management
```bash
# Deploy the demo stack
./demo/scripts/demo-stack.sh deploy
# Stop all services
./demo/scripts/demo-stack.sh stop
# Restart all services
./demo/scripts/demo-stack.sh restart
# Show service status
./demo/scripts/demo-stack.sh status
```
### Testing & Validation
```bash
# Run comprehensive validation (before deployment)
./demo/scripts/validate-all.sh
# Run full test suite
./demo/scripts/demo-test.sh full
# Run security compliance tests only
./demo/scripts/demo-test.sh security
# Run permission validation tests only
./demo/scripts/demo-test.sh permissions
# Run network isolation tests only
./demo/scripts/demo-test.sh network
```
### Docker Operations
```bash
# View logs for all services
docker compose -f demo/docker-compose.yml logs -f
# View logs for specific service
docker compose -f demo/docker-compose.yml logs -f <service-name>
# Execute command in container
docker exec -it <container-name> <command>
# Check service health
docker compose -f demo/docker-compose.yml ps
```
### Code Quality Validation
```bash
# Validate YAML files (yamllint)
docker run --rm -v "$(pwd):/data" cytopia/yamllint demo/docker-compose.yml.template
docker run --rm -v "$(pwd):/data" cytopia/yamllint demo/config/homepage/docker.yaml
docker run --rm -v "$(pwd):/data" cytopia/yamllint demo/config/grafana/datasources.yml
docker run --rm -v "$(pwd):/data" cytopia/yamllint demo/config/grafana/dashboards.yml
# Validate shell scripts (shellcheck)
docker run --rm -v "$(pwd):/data" koalaman/shellcheck demo/scripts/*.sh
docker run --rm -v "$(pwd):/data" koalaman/shellcheck demo/tests/**/*.sh
# Validate Dockerfiles (hadolint)
docker run --rm -v "$(pwd):/workdir" hadolint/hadolint <path-to-dockerfile>
```
## Code Organization & Structure
### Service Categories
1. **Infrastructure Services** (ports 4005-4007)
- Docker Socket Proxy (4005) - Security layer for Docker API access (internal only)
- Pi-hole (4006) - DNS management with ad blocking
- Dockhand (4007) - Web-based container management
2. **Monitoring & Observability** (ports 4008-4009, 4021, 4024)
- InfluxDB (4008) - Time series database for metrics
- Grafana (4009) - Visualization platform
- Metrics (4021) - GitHub metrics visualization
- Apple Health (4024) - Health data collection
3. **Documentation & Diagramming** (ports 4010-4011, 4022)
- Draw.io (4010) - Web-based diagramming application
- Kroki (4011) - Diagrams as a service
- Kiwix (4022) - Offline wiki reader
4. **Developer Tools** (ports 4000, 4012-4018)
- Homepage (4000) - Central dashboard for service discovery
- Atomic Tracker (4012) - Habit tracking and personal dashboard
- ArchiveBox (4013) - Web archiving solution
- Tube Archivist (4014) - YouTube video archiving (requires ta-redis + ta-elasticsearch)
- Wakapi (4015) - Open-source WakaTime alternative (time tracking)
- MailHog (4017 Web, 4019 SMTP) - Web and API based SMTP testing
- Atuin (4018) - Magical shell history synchronization
5. **Productivity** (ports 4016, 4023)
- Reactive Resume (4016) - Resume builder
- Resume Matcher (4023) - AI resume screening
6. **Companion Services** (internal only, no host ports)
- ta-redis - Redis cache for Tube Archivist
- ta-elasticsearch - Elasticsearch index for Tube Archivist
### Configuration Management
- **Environment Variables**: All configuration via `demo/demo.env` (copy from `demo/demo.env.template`)
- **Template-Based**: `docker-compose.yml` generated from `docker-compose.yml.template` using `envsubst`
- **Dynamic User Detection**: UID/GID automatically detected and applied
- **Service Discovery**: Automatic via Homepage labels in docker-compose.yml
## Naming Conventions & Style Patterns
### Service Naming
- **Container Names**: `kneldevstack-supportstack-demo-<service-name>`
- **Volume Names**: `kneldevstack-supportstack-demo_<service>_data`
- **Network Name**: `kneldevstack-supportstack-demo-network`
- **Project Name**: `kneldevstack-supportstack-demo`
### Port Assignment
- **Range**: 4000-4099
- **Sequential Allocation**: Groups assigned consecutive ports
- **Documented**: All ports documented in README.md port table
### Docker Labels (Service Discovery)
```yaml
labels:
homepage.group: "Infrastructure" # Category (Infrastructure|Monitoring|Documentation|Developer Tools|Productivity)
homepage.name: "Service Display Name" # Human-readable name
homepage.icon: "icon-name" # Icon identifier
homepage.href: "http://localhost:PORT" # Access URL
homepage.description: "Brief description" # Tool tip
```
### Shell Script Patterns
- **Strict Mode**: `set -euo pipefail` at top of all scripts
- **Color Output**: Consistent color codes for logging (RED, GREEN, YELLOW, BLUE, NC)
- **Log Functions**: `log_info()`, `log_success()`, `log_warning()`, `log_error()`
- **Function Organization**: Clear function headers with purpose description
- **Error Handling**: Check prerequisites before execution
### YAML Patterns
- **Header Comments**: Include purpose and version
- **Structured Sections**: Clear separation of networks, volumes, services
- **Consistent Indentation**: 2-space indentation
- **Environment Variables**: All via `${VARIABLE_NAME}` format
- **Health Checks**: All services include healthcheck section
## Testing Approach & Patterns
### Test Organization
- **Unit Tests**: `/demo/tests/unit/` - Test individual components
- **Integration Tests**: `/demo/tests/integration/` - Test service interactions
- **E2E Tests**: `/demo/tests/e2e/` - Test complete workflows
### Test Categories (demo-test.sh)
1. **File Ownership**: Verify no root-owned files on host
2. **User Mapping**: Validate UID/GID detection and application
3. **Docker Group**: Confirm docker group access
4. **Service Health**: All services passing health checks
5. **Port Accessibility**: Verify all ports accessible from host
6. **Network Isolation**: Confirm services isolated in demo network
7. **Volume Permissions**: Validate Docker volume permissions
8. **Security Compliance**: Docker socket proxy restrictions enforced
### Validation Before Deployment
Run `./demo/scripts/validate-all.sh` to check:
- YAML syntax (yamllint)
- Shell script syntax (shellcheck)
- Docker image availability
- Port availability
- Environment variables
- Health endpoints
- Service dependencies
- Resource requirements
## Important Gotchas & Non-Obvious Patterns
### ⚠️ CRITICAL: Process Management
**NEVER interfere with existing processes**:
1. **Always check first**: Run `screen -ls` and `ps aux | grep demo-stack` before starting new work
2. **Unique identifiers**: Use unique session names with timestamps: `demo-deploy-$(date +%Y%m%d-%H%M%S)`
3. **Ask before acting**: Never kill or modify existing processes without explicit permission
4. **Respect concurrent work**: Other users/processes may be running - do not assume exclusive access
### Demo-Only Security Model
- **Hardcoded Credentials**: All passwords set to `demo_password` or similar
- **No Encryption**: No TLS, no security hardening
- **Isolated Network**: Services contained in Docker network only
- **Not for Production**: Clear separation required for production deployments
- **Documentation Required**: All security assumptions must be documented
### Zero-Tolerance Quality Standards
Before ANY file is created or modified:
1. ✅ YAML syntax validated with yamllint
2. ✅ Shell script validated with shellcheck
3. ✅ Environment variables verified
4. ✅ Port availability confirmed
5. ✅ Docker image existence verified
6. ✅ Service dependencies validated
7. ✅ Health check endpoints confirmed
8. ✅ Resource requirements assessed
**Failure on any validation must block deployment.**
### Dynamic Environment Strategy
- **User Detection**: UID/GID automatically detected and written to `demo.env`
- **Template Generation**: `docker-compose.yml` generated from template using `envsubst`
- **Variable Expansion**: All configuration via environment variables
- **Source demo.env**: Always source `demo.env` before accessing variables
### Docker Socket Proxy Requirements
- **Mandatory**: All container operations must go through `docker-socket-proxy`
- **No Direct Access**: Never mount Docker socket directly in containers (except proxy)
- **Restricted API**: Minimal permissions per service requirements
- **Group-Based**: Dynamic docker group ID assignment
### Volume vs Bind Mount Strategy
- **Prefer Volumes**: Use Docker volumes for data storage
- **Minimal Bind Mounts**: Use host bind mounts only for configuration that needs persistence
- **Dynamic Naming**: Volume names follow pattern: `kneldevstack-supportstack-demo_<service>_data`
- **Permission Mapping**: UID/GID mapped via environment variables
### Service Discovery Mechanism
- **Homepage Labels**: Services automatically discovered via Docker labels
- **No Manual Config**: Don't manually add services to Homepage configuration
- **Group-Based**: Services organized by group (Infrastructure, Monitoring, Documentation, Developer Tools, Productivity)
- **Real-Time**: Homepage updates automatically as services start/stop
### FOSS Only Policy
- **No Proprietary**: Exclusively use free/libre/open source software
- **Official Images**: Prefer official Docker images
- **Verify Licenses**: Check license compatibility before adding services
- **Document Exceptions**: Any proprietary dependencies must be documented
## Project-Specific Context
### Current State
- **Demo Environment**: Fully configured with 24 services
- **Production Environment**: Placeholder only, not yet implemented
- **Documentation**: Comprehensive (AGENTS.md, PRD.md, README.md)
- **Scripts**: Complete orchestration and testing scripts available
### Architecture Principles
1. **Demo-First**: All configurations for demonstration purposes
2. **No Persistence**: Zero data persistence between demo sessions
3. **Dynamic User Handling**: Automatic UID/GID detection
4. **Security-First**: Docker socket proxy for all container operations
5. **FOSS Only**: Exclusively open source software
6. **Inner Loop Focus**: Support daily development workflows
7. **Workstation Local**: Run locally on developer machines
### Development Workflow
1. **Research**: Verify FOSS status and official Docker image availability
2. **Plan**: Determine port assignment and service group
3. **Template Configuration**: Add to docker-compose.yml.template with variables
4. **Environment Setup**: Add service variables to demo.env
5. **Security Integration**: Configure docker-socket-proxy permissions
6. **Dynamic Testing**: Validate with demo-stack.sh and demo-test.sh
7. **Documentation Update**: Update README.md, PRD.md, and AGENTS.md
8. **Atomic Commit**: Conventional commit with detailed description
### Service Health Standards
- **All Services**: Must include healthcheck section
- **Completion**: Health checks must complete within 10 seconds
- **HTTP Endpoints**: Preferred health check method
- **Fallback**: Container status checks when HTTP not available
- **Monitoring**: All health checks visible via `docker compose ps`
### Resource Limits
- **Memory**: < 512MB per service (where applicable)
- **CPU**: < 25% per service (idle)
- **Startup Time**: < 60 seconds for full stack
- **Disk Usage**: Temporary volumes only (demo mode)
## Environment Variables (demo.env)
### Required Variables
```bash
COMPOSE_PROJECT_NAME=kneldevstack-supportstack-demo
COMPOSE_NETWORK_NAME=kneldevstack-supportstack-demo-network
# User Detection (Auto-populated by demo-stack.sh)
DEMO_UID=
DEMO_GID=
DEMO_DOCKER_GID=
# Service Ports (4000-4099 range)
HOMEPAGE_PORT=4000
DOCKER_SOCKET_PROXY_PORT=4005
PIHOLE_PORT=4006
DOCKHAND_PORT=4007
INFLUXDB_PORT=4008
GRAFANA_PORT=4009
DRAWIO_PORT=4010
KROKI_PORT=4011
ATOMIC_TRACKER_PORT=4012
ARCHIVEBOX_PORT=4013
TUBE_ARCHIVIST_PORT=4014
WAKAPI_PORT=4015
MAILHOG_PORT=4017
MAILHOG_SMTP_PORT=4019
ATUIN_PORT=4018
REACTIVE_RESUME_PORT=4016
RESUME_MINIO_PORT=4020
METRICS_PORT=4021
KIWIX_PORT=4022
RESUME_MATCHER_PORT=4023
APPLEHEALTH_PORT=4024
# Demo Credentials (NOT FOR PRODUCTION)
DEMO_ADMIN_USER=admin
DEMO_ADMIN_PASSWORD=demo_password
# Reactive Resume
RESUME_POSTGRES_DB=reactive_resume
RESUME_POSTGRES_USER=resume_user
RESUME_POSTGRES_PASSWORD=demo_password
RESUME_MINIO_USER=minio_user
RESUME_MINIO_PASSWORD=demo_password
RESUME_CHROME_TOKEN=demo_token
RESUME_ACCESS_TOKEN_SECRET=demo_secret
RESUME_REFRESH_TOKEN_SECRET=demo_secret
# Metrics
METRICS_GITHUB_TOKEN=
# Apple Health
APPLEHEALTH_INFLUXDB_BUCKET=applehealth
```
## Key Files Reference
| File | Purpose | Read Before... |
|------|---------|----------------|
| `demo/AGENTS.md` | Detailed development guidelines | Making any changes |
| `demo/PRD.md` | Product Requirements Document | Adding/removing services |
| `demo/README.md` | Demo-specific documentation | Deploying or troubleshooting |
| `demo/demo.env` | Environment configuration | Modifying service settings |
| `demo/docker-compose.yml.template` | Service definitions | Adding/modifying services |
| `demo/scripts/demo-stack.sh` | Deployment orchestration | Understanding deployment flow |
| `demo/scripts/demo-test.sh` | Test suite | Verifying changes |
| `demo/scripts/validate-all.sh` | Pre-deployment validation | Making any commits |
## Troubleshooting Quick Reference
### Common Issues
1. **Port Conflicts**: Check with `netstat -tulpn | grep :<port>`
2. **Permission Issues**: Verify UID/GID in demo.env match current user
3. **Image Pull Failures**: Run `docker pull <image>` manually
4. **Health Check Failures**: Check service logs with `docker compose logs <service>`
5. **Network Issues**: Verify network exists: `docker network ls | grep kneldevstack`
### Getting Help
1. Check troubleshooting section in demo/README.md
2. Review service logs: `docker compose -f demo/docker-compose.yml logs {service}`
3. Consult individual service documentation
4. Check health status: `docker compose -f demo/docker-compose.yml ps`
5. **CRITICAL**: Always check for existing processes first
## Related Documentation
- **demo/AGENTS.md**: Detailed development guidelines and standards
- **demo/PRD.md**: Product Requirements Document
- **demo/README.md**: Demo-specific documentation and quick start
- **demo/docs/service-guides/**: Service-specific guides
- **demo/docs/troubleshooting/**: Detailed troubleshooting procedures
- **demo/docs/api-docs/**: API documentation
---
**Last Updated**: 2026-05-08
**Version**: 2.0
Reference in New Issue View Git Blame Copy Permalink