refactor: move stack assets and wire in mailhog
This commit is contained in:
248
SupportStack/collab/BuildTheStack
Normal file
248
SupportStack/collab/BuildTheStack
Normal file
@@ -0,0 +1,248 @@
|
||||
# TSYSDevStack SupportStack Demo Builder
|
||||
|
||||
## Objective
|
||||
Create an out-of-the-box, localhost-bound only, ephemeral Docker volume-only demonstration version of the SupportStack components documented in the docs/VendorList-SupportStack.md file.
|
||||
|
||||
## MVP Test Run Objective
|
||||
Create a proof of concept with docker-socket-proxy, homepage, and wakaapi components that demonstrate proper homepage integration via Docker Compose labels. This MVP will serve as a validation of the full approach before proceeding with the complete stack implementation.
|
||||
|
||||
## Architecture Requirements
|
||||
- All Docker artifacts must be prefixed with `tsysdevstack-supportstack-demo-`
|
||||
- This includes containers, networks, volumes, and any other Docker artifacts
|
||||
- Example: `tsysdevstack-supportstack-demo-homepage`, `tsysdevstack-supportstack-demo-network`, etc.
|
||||
- Run exclusively on localhost (localhost binding only)
|
||||
- Ephemeral volumes only (no persistent storage)
|
||||
- Resource limits set for single-user demo capacity
|
||||
- No external network access (localhost bound only)
|
||||
- Components: docker-socket-proxy, portainer, homepage as foundational elements
|
||||
- All artifacts must go into artifacts/SupportStack directory to keep the directory well organized and avoid cluttering the root directory
|
||||
- Homepage container needs direct access to Docker socket for labels to auto-populate (not through proxy)
|
||||
- Docker socket proxy is for other containers that need Docker access but don't require direct socket access
|
||||
- Portainer can use docker-socket-proxy for read-only access, but homepage needs direct socket access
|
||||
- All containers need proper UID/GID mapping for security
|
||||
- Docker group GID must be mapped properly for containers using Docker socket
|
||||
- Non-Docker socket using containers should use invoking UID/GID
|
||||
|
||||
## Development Methodology
|
||||
- Strict Test Driven Development (TDD) process
|
||||
- Write test → Execute test → Test fails → Write minimal code to pass test
|
||||
- 75%+ code coverage requirement
|
||||
- 100% test pass requirement
|
||||
- Component-by-component development approach
|
||||
- Complete one component before moving to the next
|
||||
- Apply TDD for every change, no matter how surgical
|
||||
- Test changes right after implementation as atomically as possible
|
||||
- Each fix or modification should be accompanied by a specific test to verify the issue
|
||||
- Ensure all changes are validated immediately after implementation
|
||||
|
||||
## MVP Component Development Sequence (Test Run) ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
1. **MVP**: docker-socket-proxy, homepage, wakaapi (each must fully satisfy Definition of Done before proceeding) ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- docker-socket-proxy: Enable Docker socket access for containers that need it (not homepage) ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- homepage: Configure to access Docker socket directly for automatic label discovery ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- wakaapi: Integrate with homepage using proper labels ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- All services must utilize Docker Compose labels to automatically show up in homepage ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- Implement proper service discovery for homepage integration using gethomepage labels ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- Ensure all components are properly labeled with homepage integration labels ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- Implement proper startup ordering using depends_on with health checks ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- Homepage container requires direct Docker socket access for automatic service discovery ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- Docker socket proxy provides controlled access for other containers ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
- All containers must have proper UID/GID mapping for security ✅ COMPLETED ✅ MVP FULLY IMPLEMENTED AND TESTED
|
||||
|
||||
## Component Completion Validation ✅ MVP COMPLETED
|
||||
- Each component must pass health checks for 5 consecutive minutes before moving to the next ✅ MVP COMPLETED
|
||||
- All tests must pass with 100% success rate before moving to the next component ✅ MVP COMPLETED
|
||||
- Resource utilization must be within specified limits before moving to the next component ✅ MVP COMPLETED
|
||||
- Integration tests with previously completed components must pass before moving forward ✅ MVP COMPLETED
|
||||
- Homepage must automatically detect and display all services with proper labels ✅ MVP COMPLETED
|
||||
- Specific validation checkpoints after each service deployment:
|
||||
- docker-socket-proxy: Validate Docker socket access and network connectivity to Docker daemon ✅ COMPLETED
|
||||
- homepage: Validate homepage starts and can connect to Docker socket directly, verify UI is accessible ✅ COMPLETED
|
||||
- wakaapi: Validate service starts and can be integrated into homepage with proper labels ✅ COMPLETED
|
||||
- Each service must be validated in homepage dashboard after integration ✅ MVP COMPLETED
|
||||
- Detailed homepage integration validation steps:
|
||||
- Verify service appears in homepage dashboard with correct name and icon ✅ MVP COMPLETED
|
||||
- Confirm service status shows as healthy in homepage ✅ MVP COMPLETED
|
||||
- Validate service URL in homepage correctly links to the service ✅ MVP COMPLETED
|
||||
- Verify service group assignment in homepage is correct ✅ MVP COMPLETED
|
||||
- Check that any configured widgets appear properly in homepage ✅ MVP COMPLETED
|
||||
- Homepage must automatically discover services via Docker labels without manual configuration ✅ MVP COMPLETED
|
||||
- Validate Docker socket connectivity for automatic service discovery ✅ MVP COMPLETED
|
||||
- Confirm homepage can access and display service status information ✅ MVP COMPLETED
|
||||
- Update STATUS.md with validation results for each component ✅ MVP COMPLETED
|
||||
|
||||
## Technical Specifications
|
||||
- No Bitnami images allowed
|
||||
- Use official or trusted repository images only:
|
||||
- docker-socket-proxy: tecnativa/docker-socket-proxy (pinned version tag)
|
||||
- homepage: gethomepage/homepage (pinned version tag)
|
||||
- wakaapi: ghcr.io/ekkinox/wakaapi (pinned version tag)
|
||||
- Implement Docker Compose orchestration
|
||||
- Use Docker named volumes for ephemeral storage
|
||||
- Implement proper resource limits in docker-compose.yml: CPU: 0.5-1.0 cores per service, Memory: 128MB-512MB per service (variable based on service type), Disk: 1GB per service for ephemeral volumes
|
||||
- Implement comprehensive health checks for all services with appropriate intervals and timeouts
|
||||
- All services must be on a shared Docker network named: tsysdevstack_supportstack_network
|
||||
- Implement proper networking (internal only)
|
||||
- All ports bound to localhost (127.0.0.1) with specific port assignments:
|
||||
- docker-socket-proxy: Internal network only, no external ports exposed
|
||||
- homepage: Port 4000 (localhost only) - configurable via environment variable
|
||||
- wakaapi: Port 4001 (localhost only) - configurable via environment variable
|
||||
- All environment variables must be pre-set in tsysdevstack-supportstack-demo-Settings file (single settings file for simplicity in demo)
|
||||
- All docker compose files (one per component) should be prefixed with: tsysdevstack-supportstack-demo-DockerCompose-
|
||||
- All docker compose files should use environment variables for everything (variables will be set in tsysdevstack-supportstack-demo-Settings file)
|
||||
- Health checks must validate service readiness before proceeding with dependent components
|
||||
- Health check endpoints must be accessible only from internal network
|
||||
- Health check configurations must be parameterized via environment variables
|
||||
- All services must utilize Docker Compose labels to automatically show up in homepage
|
||||
- Implement proper homepage integration labels for automatic service discovery using gethomepage/homepage labels:
|
||||
- Required: homepage.group, homepage.name, homepage.icon
|
||||
- Optional: homepage.href, homepage.description, homepage.widget.type, homepage.widget.url, homepage.widget.key, homepage.widget.fields, homepage.weight
|
||||
- Homepage integration must include proper naming, icons, and status indicators
|
||||
- Use pinned image tags rather than 'latest' for all container images
|
||||
- Run containers as non-root users where possible
|
||||
- Enable read-only filesystems where appropriate
|
||||
- Implement security scanning during build process (for demo, secrets via environment variables are acceptable)
|
||||
- Define network policies for internal communication only
|
||||
- Use depends_on with health checks to ensure proper startup ordering of services
|
||||
- Use SQLite for every service that will support it to avoid heavier databases where possible
|
||||
- For services requiring databases, prefer lightweight SQLite over PostgreSQL, MySQL, or other heavy database systems
|
||||
- Only use heavier databases when SQLite is not supported or inadequate for the service requirements
|
||||
- When using SQLite, implement proper volume management for database files using Docker volumes
|
||||
- Ensure SQLite databases are properly secured with appropriate file permissions and encryption where needed
|
||||
- Avoid external database dependencies when SQLite can meet the service requirements
|
||||
- For database-backed services, configure SQLite as the default database engine in environment variables
|
||||
- When migrating from heavier databases to SQLite, ensure data integrity and performance are maintained
|
||||
- Implement proper backup strategies for SQLite databases using Docker volume snapshots
|
||||
- Homepage container requires direct Docker socket access (not through proxy) for automatic label discovery
|
||||
- Docker socket proxy provides controlled access for other containers that need Docker access
|
||||
- Portainer can use docker-socket-proxy for read-only access
|
||||
- All containers must have proper UID/GID mapping for security
|
||||
- Docker group GID must be mapped for containers using Docker socket
|
||||
- Homepage container must have Docker socket access for labels to auto-populate
|
||||
|
||||
## Stack Control
|
||||
- All control of the stack should go into a script called tsysdevstack-supportstack-demo-Control.sh
|
||||
- The script should take the following arguments: start/stop/uninstall/update/test
|
||||
- Ensure script is executable and contains error handling
|
||||
- Script must handle UID/GID mapping for non-Docker socket using containers
|
||||
- Script must map host Docker GID to containers using Docker socket
|
||||
- Script should warn about Docker socket access requirements for homepage
|
||||
|
||||
## Component Definition of Done
|
||||
- All health checks pass consistently for each component
|
||||
- docker-socket-proxy: HTTP health check on / (internal only)
|
||||
- homepage: HTTP health check on /api/health (internal only)
|
||||
- wakaapi: HTTP health check on /health (internal only)
|
||||
- Test suite passes with 100% success rate (unit, integration, e2e)
|
||||
- Code coverage of >75% for each component
|
||||
- Resource limits properly implemented and validated (CPU: 0.5-1.0 cores, Memory: 128MB-512MB, Disk: 1GB per service)
|
||||
- All services properly bound to localhost only
|
||||
- Proper error handling and logging implemented (with retry logic and exponential backoff)
|
||||
- Documentation and configuration files created
|
||||
- Component successfully starts, runs, and stops without manual intervention
|
||||
- Component properly integrates with other components without conflicts
|
||||
- Automated self-recovery mechanisms implemented for common failure scenarios
|
||||
- Performance benchmarks met for single-user demo capacity (apply reasonable defaults based on service type)
|
||||
- Security scans completed and passed (run as non-root, read-only filesystems where appropriate)
|
||||
- No hard-coded values; all configuration via environment variables
|
||||
- All dependencies properly specified and resolved using depends_on with health checks
|
||||
- Component properly labeled with homepage integration labels (homepage.group, homepage.name, homepage.icon, etc.)
|
||||
- Container uses pinned image tags rather than 'latest'
|
||||
- Services validate properly in homepage after integration
|
||||
- Homepage container has direct Docker socket access for automatic service discovery
|
||||
- Homepage automatically discovers and displays services with proper labels
|
||||
- Homepage validates Docker socket connectivity and service discovery
|
||||
- All homepage integration labels are properly applied and validated
|
||||
- Services appear in homepage with correct grouping, naming, and icons
|
||||
- Homepage container has direct Docker socket access for automatic label discovery
|
||||
- Docker socket proxy provides access for other containers that need Docker access
|
||||
- Proper UID/GID mapping implemented for all containers
|
||||
- Docker group GID properly mapped for containers using Docker socket
|
||||
- All warnings addressed and resolved during implementation
|
||||
|
||||
## Testing Requirements
|
||||
- Unit tests for each component configuration
|
||||
- Integration tests for component interactions
|
||||
- End-to-end tests for the complete stack
|
||||
- Performance tests to validate resource limits
|
||||
- Security tests for localhost binding
|
||||
- Health check tests for all services
|
||||
- Coverage report generation
|
||||
- Continuous test execution during development
|
||||
- Automated test suite execution for each component before moving to the next
|
||||
- End-to-end validation tests after each component integration
|
||||
|
||||
## Error Resolution Strategy
|
||||
- Implement autonomous error detection and resolution
|
||||
- Automatic retry mechanisms for transient failures with exponential backoff (base delay of 5s, max 5 attempts)
|
||||
- Fallback configurations for compatibility issues
|
||||
- Comprehensive logging for debugging
|
||||
- Graceful degradation for optional components
|
||||
- Automated rollback for failed deployments
|
||||
- Self-healing mechanisms for common failure scenarios
|
||||
- Automated restart policies with appropriate backoff strategies
|
||||
- Deadlock detection and resolution mechanisms
|
||||
- Resource exhaustion monitoring and mitigation
|
||||
- Automated cleanup of failed component attempts
|
||||
- Persistent state recovery mechanisms
|
||||
- Fail-safe modes for critical components
|
||||
- Circuit breaker patterns for service dependencies
|
||||
- Specific timeout values for operations:
|
||||
- Docker socket proxy connection timeout: 30 seconds
|
||||
- Homepage startup timeout: 60 seconds
|
||||
- Wakaapi initialization timeout: 45 seconds
|
||||
- Service health check timeout: 10 seconds
|
||||
- Docker Compose startup timeout: 120 seconds per service
|
||||
- If unable to resolve an issue after multiple attempts, flag it in collab/SupportStack/HUMANHELP.md and move on
|
||||
- Maintain running status reports in collab/SupportStack/STATUS.md
|
||||
- Use git commit heavily to track progress
|
||||
- Push to remote repository whenever a component is fully working/tested/validated
|
||||
- Check Docker logs for all containers during startup and health checks to identify issues
|
||||
- Monitor container logs continuously for error patterns and failure indicators
|
||||
- Implement log analysis for common failure signatures and automatic remediation
|
||||
|
||||
## Autonomous Operation Requirements
|
||||
- Project must be capable of running unattended for 1-2 days without manual intervention
|
||||
- All components must implement self-monitoring and self-healing
|
||||
- Automated monitoring of resource usage with alerts if limits exceeded
|
||||
- All failure scenarios must have automated recovery procedures
|
||||
- Consistent state maintenance across all components
|
||||
- Automated cleanup of temporary resources
|
||||
- Comprehensive logging for troubleshooting without human intervention
|
||||
- Built-in validation checks to ensure continued operation
|
||||
- Automatic restart of failed services with appropriate retry logic
|
||||
- Prevention of resource leaks and proper cleanup on shutdown
|
||||
|
||||
## Qwen Optimization
|
||||
- Structured for autonomous execution
|
||||
- Clear task decomposition
|
||||
- Explicit success/failure criteria
|
||||
- Self-contained instructions
|
||||
- Automated validation steps
|
||||
- Progress tracking mechanisms
|
||||
|
||||
## Output Deliverables
|
||||
- Directory structure in artifacts/SupportStack
|
||||
- Environment variables file: TSYSDevStack-SupportStack-Demo-Settings
|
||||
- Control script: TSYSDevStack-SupportStack-Demo-Control.sh (with start/stop/uninstall/update/test arguments)
|
||||
- Docker Compose files prefixed with: TSYSDevStack-SupportStack-Demo-DockerCompose-
|
||||
- Component configuration files
|
||||
- Test suite (unit, integration, e2e)
|
||||
- Coverage reports
|
||||
- Execution logs
|
||||
- Documentation files
|
||||
- Health check scripts and configurations
|
||||
- Component readiness and liveness check definitions
|
||||
- Automated validation scripts for component completion
|
||||
- Monitoring and alerting configurations
|
||||
|
||||
The implementation should work autonomously, handling errors and resolving configuration issues without human intervention while strictly adhering to the TDD process.
|
||||
|
||||
## Production Considerations
|
||||
- For production implementation, additional items will be addressed including:
|
||||
- Enhanced monitoring and observability with centralized logging
|
||||
- Advanced security measures (secrets management, network policies, etc.)
|
||||
- Performance benchmarks and optimization
|
||||
- Configuration management with separation of required vs optional parameters
|
||||
- Advanced documentation (architecture diagrams, troubleshooting guides, etc.)
|
||||
- Production-grade error handling and recovery procedures
|
||||
- All deferred items will be tracked in collab/SupportStack/ProdRoadmap.md
|
||||
Reference in New Issue
Block a user