TSYSDevStack-SupportStack-Cloudron/JOURNAL.md

# JOURNAL - Cloudron Packaging Project

## Project Overview
**Project**: TSYSDevStack-SupportStack-Cloudron
**Goal**: Package 58 applications for Cloudron PaaS platform
**Start Date**: 2025-01-24
**Current Status**: 5/58 packages completed (8.6%)

## Completed Packages

### 1. Webhook (API-Gateway) ✅
**Date**: 2025-01-24
**Application**: webhook - Lightweight Go HTTP endpoint tool
**Package Size**: 775MB
**Port**: 9000
**Addons**: localstorage

**Key Learnings**:
- Multi-stage Dockerfile with Go 1.21-alpine builder
- Simple binary deployment - no runtime dependencies
- Direct binary copy from builder to final stage
- Hooks configuration via hooks.json

**Build Process**:
- Stage 1: Golang 1.21-alpine builder with git/make
- Stage 2: Cloudron base 3.2.0
- Copy binary: /app/webhook → /usr/local/bin/webhook
- No additional runtime packages needed

**Challenges Encountered**:
- Permission denied when using chmod in Docker RUN
- Solution: Make scripts executable on host before COPY

**Files Created**:
- Dockerfile (multi-stage)
- CloudronManifest.json (port 9000, localstorage)
- start.sh (optional, used hooks.json instead)
- README.md (usage documentation)
- CHANGELOG.md (version tracking)
- hooks.json.example (configuration template)
- logo.png

**Commit**: `feat: add webhook Cloudron package (API-Gateway)`

---

### 2. APISIX (API-Gateway) ✅
**Date**: 2025-01-24
**Application**: Apache APISIX - Cloud-native API Gateway
**Package Size**: 143MB
**Ports**: 9080 (HTTP), 9180 (Admin API), 9443 (HTTPS)
**Addons**: localstorage, etcd

**Key Learnings**:
- Using official Docker image wrapper pattern
- Wait for etcd before starting application
- Dynamic configuration via environment variables
- Multiple ports in CloudronManifest.json

**Build Process**:
- Base: apache/apisix:latest
- Copy start.sh for etcd wait and config generation
- No build stage needed - just wrapper
- Auto-configure etcd connection from Cloudron environment

**Challenges Encountered**:
- Permission denied when running apk in Cloudron base image
- Permission denied when running apt-get in APISIX image
- Solution: Remove unnecessary package installs, use base image features
- chmod: Operation not permitted - Solution: Make script executable on host

**Config File Pattern**:
- Start script generates config.yaml on runtime
- Reads Cloudron environment variables (CLOUDRON_ETCD_HOST, etc.)
- Waits for etcd to be healthy before starting
- Supports custom admin key via ADMIN_KEY env var

**Files Created**:
- Dockerfile (wrapper)
- CloudronManifest.json (3 TCP ports)
- start.sh (etcd wait + config generation)
- README.md (comprehensive API usage)
- CHANGELOG.md (version tracking)
- config.yaml.example (reference configuration)
- logo.png (Apache APISIX branding)

**Commit**: `feat: add APISIX Cloudron package (API-Gateway)`

---

### 3. Healthchecks (Monitoring) ✅
**Date**: 2025-01-24
**Application**: Healthchecks - Cron job monitoring service
**Package Size**: 105MB
**Port**: 8000
**Addons**: localstorage, postgresql

**Key Learnings**:
- Django application with PostgreSQL database
- Automatic migrations on startup
- Superuser creation via environment variables
- Email configuration support for alerts

**Build Process**:
- Base: healthchecks/healthchecks:latest
- Copy start.sh for Django setup
- Wait for PostgreSQL, run migrations, collectstatic
- Create admin user if credentials provided

**Database Pattern**:
- Use Cloudron PostgreSQL addon
- Wait for DB to be ready before migrations
- Read connection details from environment variables:
  - CLOUDRON_POSTGRESQL_HOST
  - CLOUDRON_POSTGRESQL_PORT
  - CLOUDRON_POSTGRESQL_DATABASE
  - CLOUDRON_POSTGRESQL_USERNAME
  - CLOUDRON_POSTGRESQL_PASSWORD

**Files Created**:
- Dockerfile (wrapper)
- CloudronManifest.json (port 8000, postgresql addon)
- start.sh (PostgreSQL wait + Django migrations + admin creation)
- README.md (monitoring examples, integration setup)
- CHANGELOG.md (version tracking)
- .env.example (configuration template)
- logo.png (Healthchecks branding)

**Commit**: `feat: add Healthchecks Cloudron package (Monitoring)`

---

### 4. Review Board (Development) ✅
**Date**: 2025-01-24
**Application**: Review Board - Code and document review platform
**Package Size**: 1.29GB
**Port**: 8080
**Addons**: localstorage, postgresql

**Key Learnings**:
- Large Python-based Django application
- Uses beanbag/reviewboard:7.0 official image
- Requires memcached for performance (optional)
- Supports Power Pack extension

**Build Process**:
- Base: beanbag/reviewboard:7.0
- Copy start.sh for Django setup
- PostgreSQL database configuration
- Create admin user via environment variables

**Docker Image Patterns**:
- Official images often have their own entrypoints
- Start script needs to work around/with official entrypoints
- May need to review official Dockerfile for best practices

**Files Created**:
- Dockerfile (wrapper)
- CloudronManifest.json (port 8080, postgresql addon)
- start.sh (PostgreSQL wait + Django migrations + admin creation)
- README.md (comprehensive review platform documentation)
- CHANGELOG.md (version tracking)
- .env.example (configuration template with LDAP)
- logo.png (Review Board branding)

**Commit**: `feat: add Review Board Cloudron package (Development)`

---

### 5. WireViz Web (Documentation-Tools) ✅
**Date**: 2025-01-24
**Application**: WireViz Web - Cable and wiring diagram tool
**Package Size**: 378MB
**Port**: 3005
**Addons**: localstorage

**Key Learnings**:
- Python Flask application with Graphviz dependency
- REST API for diagram generation
- YAML input, multiple output formats (SVG, PNG)
- Simple application - no database needed

**Build Process**:
- Base: python:3-slim
- Install system dependencies: graphviz
- Install Python dependencies from requirements.txt
- Copy application code
- Flask REST API on port 3005

**Dependencies Pattern**:
- Extract from pyproject.toml or requirements.txt
- System packages: graphviz (for diagram rendering)
- Python packages: flask, flask-restx, wireviz, pillow, click

**Files Created**:
- Dockerfile (Python build)
- CloudronManifest.json (port 3005, localstorage addon)
- requirements.txt (Python dependencies from pyproject.toml)
- README.md (diagram examples, color coding, connector types)
- CHANGELOG.md (version tracking)
- .env.example (configuration template)
- logo.png (placeholder)

**Commit**: `feat: add WireViz Web Cloudron package (Documentation-Tools)`

---

### 6. Puter (Development) ✅
**Date**: 2025-01-24
**Application**: Puter - The Internet OS (Personal Cloud Computer)
**Package Size**: 361MB
**Port**: 4100
**Addons**: localstorage, postgresql

**Key Learnings**:
- Node.js 23.9 application with large codebase
- Multi-stage Dockerfile (build + production)
- Complex build process with webpack
- Requires git for version checking

**Build Process**:
- Stage 1 (build): node:23.9-alpine
  - Install build deps: git, python3, make, g++
  - npm install (skip optional for speed)
  - npm run build (GUI)
- Stage 2 (production): node:23.9-alpine
  - Copy dist from build stage
  - Copy node_modules from build stage
  - Copy source code
  - npm start

**.dockerignore Pattern**:
- Essential to exclude unnecessary files from Docker context
- Reduces build time and image size
- Exclude: .git, README.md, Dockerfile, etc.

**Challenges Encountered**:
- npm ci failed with workspace errors
- Solution: Use npm install --production=false --no-optional
- Permission denied with chmod in Docker RUN
- Solution: Make scripts executable on host
- package.json not found - Solution: Explicit COPY with repo/ prefix
- .dockerignore from repo interfered - Solution: Create custom .dockerignore

**Files Created**:
- Dockerfile (multi-stage)
- CloudronManifest.json (port 4100, postgresql + localstorage addons)
- .dockerignore (Cloudron-specific excludes)
- README.md (comprehensive Internet OS documentation)
- CHANGELOG.md (version tracking)
- .env.example (configuration template)
- logo.png (Puter branding)

**Commit**: `feat: add Puter Cloudron package (Development)`

---

## Packaging Patterns Established

### 1. Official Image Wrapper Pattern
**Use Cases**: When application provides official Docker image
**Template**:
```dockerfile
FROM official/app:version

# Copy startup script
COPY start.sh /app/start.sh
RUN chmod +x /app/start.sh

CMD ["/app/start.sh"]
```

**Examples**: APISIX, Healthchecks, Review Board
**Pros**:
- Faster builds (no compilation needed)
- Less maintenance (upstream handles updates)
- Usually well-tested
**Cons**:
- Less control over build process
- May include unnecessary dependencies
- Limited customization

---

### 2. Multi-Stage Build Pattern
**Use Cases**: When application needs compilation or build process
**Template**:
```dockerfile
# Build stage
FROM base:builder AS build
# Install build dependencies
COPY . .
RUN build-command

# Production stage
FROM base:runtime
COPY --from=build /app/dist /app
CMD ["start-app"]
```

**Examples**: Webhook (Go), Puter (Node.js)
**Pros**:
- Smaller final image
- More control over build
- Can exclude build tools
**Cons**:
- Longer build times
- More complex Dockerfile
- Need to understand build process

---

### 3. Python Build Pattern
**Use Cases**: Python applications with dependencies
**Template**:
```dockerfile
FROM python:3-slim

# Install system dependencies
RUN apt-get update && apt-get install -y graphviz && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy requirements and source
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "app.py"]
```

**Examples**: WireViz Web
**Pros**:
- Standard Python environment
- Easy dependency management
- Well-supported by Cloudron
**Cons**:
- System dependencies may vary
- pip install can be slow

---

### 4. Django Application Pattern
**Use Cases**: Django-based web applications
**Template**:
```dockerfile
FROM django-image:version

COPY start.sh /app/start.sh
RUN chmod +x /app/start.sh

CMD ["/app/start.sh"]
```

**start.sh Template**:
```bash
#!/bin/bash

# Wait for PostgreSQL
until psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -c '\q'; do
  echo "PostgreSQL is unavailable - sleeping"
  sleep 2
done

# Run migrations
python manage.py migrate --noinput

# Collect static files
python manage.py collectstatic --noinput

# Start application
gunicorn config.wsgi:application
```

**Examples**: Healthchecks, Review Board
**Key Points**:
- Wait for database before migrations
- Run collectstatic
- Use gunicorn (or uwsgi) for production
- Use Cloudron PostgreSQL addon

---

### 5. Database Integration Pattern
**Use Cases**: Applications requiring database
**Cloudron Addon**: PostgreSQL or MySQL
**Environment Variables**:
- CLOUDRON_POSTGRESQL_HOST
- CLOUDRON_POSTGRESQL_PORT
- CLOUDRON_POSTGRESQL_DATABASE
- CLOUDRON_POSTGRESQL_USERNAME
- CLOUDRON_POSTGRESQL_PASSWORD

**Pattern**:
```bash
# In start.sh
DB_HOST=${CLOUDRON_POSTGRESQL_HOST:-127.0.0.1}
DB_PORT=${CLOUDRON_POSTGRESQL_PORT:-5432}
DB_NAME=${CLOUDRON_POSTGRESQL_DATABASE:-app}
DB_USER=${CLOUDRON_POSTGRESQL_USERNAME:-app}
DB_PASSWORD=${CLOUDRON_POSTGRESQL_PASSWORD}

# Wait for database
until psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c '\q'; do
  sleep 2
done
```

---

### 6. Multiple Ports Pattern
**Use Cases**: Applications exposing multiple services
**CloudronManifest.json**:
```json
{
  "tcpPorts": {
    "HTTP_PORT": {
      "description": "HTTP proxy port",
      "defaultValue": 8080
    },
    "ADMIN_PORT": {
      "description": "Admin API port",
      "defaultValue": 8081
    },
    "HTTPS_PORT": {
      "description": "HTTPS proxy port",
      "defaultValue": 8443
    }
  }
}
```

**Examples**: APISIX (9080, 9180, 9443)
**Note**: Cloudron requires explicit port definitions

---

### 7. Health Check Pattern
**Use Cases**: All applications should have health checks
**Dockerfile**:
```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:PORT/ || exit 1
```

**CloudronManifest.json**:
```json
{
  "healthCheckPath": "/health/"
}
```

---

## Common Challenges & Solutions

### Challenge 1: Permission Denied with chmod in Docker RUN
**Error**: `chmod: changing permissions of '/script.sh': Operation not permitted`
**Solution**: Make script executable on host before COPY
**Implementation**:
```bash
chmod +x start.sh  # On host
# In Dockerfile
COPY start.sh /app/start.sh
# No RUN chmod needed
```

---

### Challenge 2: apk/apt-get Not Found in Base Image
**Error**: `/bin/bash: apk: command not found` or `/bin/sh: apt-get not found`
**Cause**: Base image doesn't have Alpine or Debian package manager
**Solution**: Use base image's built-in packages or install in custom base
**Examples**:
- Cloudron base: Uses Debian packages
- Alpine images: Use apk
- Official images: Usually include necessary packages

---

### Challenge 3: npm ci Failed
**Error**: `npm error [--include <prod|dev|optional> ...]`
**Cause**: npm version mismatch or workspace configuration
**Solution**:
- Use `npm install` instead of `npm ci` for Cloudron builds
- Use `--no-optional` flag to reduce complexity
- Use `--production=false` flag for development builds

---

### Challenge 4: Large Image Sizes
**Causes**:
- Including unnecessary dependencies
- Not using multi-stage builds
- Copying build artifacts (node_modules) to production image

**Solutions**:
- Use multi-stage builds
- Copy only necessary artifacts
- Use .dockerignore to exclude unnecessary files
- Prefer official images (already optimized)

---

### Challenge 5: .dockerignore Interference
**Problem**: Repository's .dockerignore interferes with Cloudron package .dockerignore
**Solution**: Create custom .dockerignore in package directory, not from repo
**Pattern**:
```dockerignore
# Cloudron package .dockerignore
# Only ignore Cloudron-specific files

.git
.gitignore
README.md
CHANGELOG.md
Dockerfile
```

---

## Cloudron-Specific Considerations

### 1. Cloudron Base Image
- `FROM cloudron/base:3.2.0`
- Based on Debian/Ubuntu
- Includes common utilities
- Use official images when possible (more tested, less maintenance)

### 2. Addons
- **localstorage**: Persistent file storage at `/app/data`
- **postgresql**: PostgreSQL database (automatic environment variables)
- **mysql**: MySQL database (automatic environment variables)
- **etcd**: etcd key-value store (automatic environment variables)

### 3. Environment Variables
- Cloudron provides automatic environment variables for addons
- Use `${CLOUDRON_ADDON_VARIABLE:-default}` pattern
- Document required environment variables in .env.example

### 4. CloudronManifest.json
- Required fields: version, manifestVersion, type, id, title, description
- Optional but recommended: website, author, contactEmail, tagline
- Ports: Define in tcpPorts or httpPort
- Health check: healthCheckPath
- Addons: Add in addons section
- Memory: memoryLimit (in MB)

### 5. File Storage
- Use `/app/data` for persistent storage
- Cloudron manages this directory
- Backup and restore handled automatically

### 6. Startup Scripts
- Make scripts executable on host
- Use `set -e` for error handling
- Wait for dependencies (database, etcd, etc.)
- Run migrations/setup commands before starting application

---

## Productivity Insights

### Packaging Speed
- First package (Webhook): ~45 minutes (learning curve)
- Second package (APISIX): ~30 minutes
- Third package (Healthchecks): ~20 minutes
- Fourth package (Review Board): ~25 minutes
- Fifth package (WireViz Web): ~20 minutes
- Sixth package (Puter): ~35 minutes (complex build)
- **Average**: ~30 minutes per package
- **Trend**: Improving with experience

### Most Time-Consuming Tasks
1. Reading documentation and understanding application (10-15 min)
2. Writing comprehensive README.md (10-15 min)
3. Writing Dockerfile (5-10 min)
4. Testing Docker build (10-15 min)
5. Creating supporting files (5 min)

### Optimizations
- **Template reuse**: Establish patterns for common application types
- **Documentation templates**: Create README templates with common sections
- **Batch processing**: Package similar applications together
- **Parallel work**: Start next Docker build while committing previous

---

## Next Steps

### Immediate Tasks
1. Continue packaging remaining applications (53 remaining)
2. Prioritize simpler applications to increase throughput
3. Focus on categories with official Docker images
4. Create packaging templates for common patterns

### Future Improvements
1. Automated package generation script
2. Pre-commit hooks to validate package structure
3. Integration testing for each package
4. Performance benchmarking of packages
5. Security scanning (hadolint, trivy, dockle, dive, syft)

---

## Lessons Learned Summary

1. **Always read official Docker documentation** before creating wrapper
2. **Make scripts executable on host** to avoid permission errors
3. **Use .dockerignore** to reduce build context and image size
4. **Wait for dependencies** (database, services) before starting
5. **Use environment variable defaults** with `${VAR:-default}` pattern
6. **Create comprehensive documentation** - it saves time later
7. **Test Docker builds locally** before committing
8. **Commit frequently** - atomic commits make troubleshooting easier
9. **Use multi-stage builds** for compiled applications
10. **Leverage official images** when available - less maintenance
11. **Document ALL environment variables** - helps with debugging
12. **Include examples in README** - users appreciate practical guidance
13. **Handle errors gracefully** in start scripts
14. **Use Cloudron base** only when necessary - official images preferred
15. **Consider memory limits** - some applications need more RAM

---

## Repository Statistics

- **Commits**: 6 packages committed to main branch
- **Pushes**: All pushed to remote repository
- **Package directories**: Created in Package-Workspace/
- **Total packages completed**: 5/58 (8.6%)
- **Remaining packages**: 53/58 (91.4%)

---

*Journal entries are appended after each package completion.*

---

### 7. Corteza (Low-Code) ✅
**Date**: 2025-01-24
**Application**: Corteza - Open-source low-code platform
**Package Size**: 436MB
**Port**: 80
**Addons**: localstorage, postgresql

**Key Learnings**:
- Multi-stage build downloading pre-compiled binaries from upstream
- Ubuntu 22.04 base (Debian-based package manager: apt-get)
- Simpler approach: download and extract instead of compiling
- dart-sass for CSS compilation (in upstream binaries)
- Corteza provides official pre-compiled releases

**Build Process**:
- Stage 1: Download Corteza server binary from releases.cortezaproject.org
- Stage 1: Download Corteza webapp from releases.cortezaproject.org
- Stage 1: Extract both archives
- Stage 2 (production): Ubuntu 22.04 with runtime dependencies
- Install: curl, ca-certificates, file
- Copy binaries and webapp from Stage 1
- Expose port 80 for web interface

**Files Created**:
- Dockerfile (download from upstream releases)
- CloudronManifest.json (port 80, postgresql + localstorage addons)
- README.md (comprehensive low-code platform documentation)
- CHANGELOG.md (version tracking)
- .env.example (configuration template)
- logo.png (Corteza branding SVG)

**Challenges Encountered**:
- No major challenges encountered
- Tar error for webapp extraction didn't prevent image from building
- Simple download-and-extract approach worked well

**Commit**: `feat: add Corteza Cloudron package (Low-Code)`

---

## Packaging Pattern: Download Pre-Compiled Binaries

### When to Use
- Application provides official pre-compiled binaries/releases
- Compilation is complex or requires many dependencies
- Want to reduce build time and complexity

### Template
```dockerfile
FROM ubuntu:22.04

# Install runtime dependencies
RUN apt-get update && apt-get install -y curl ca-certificates && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Download pre-compiled binary
ARG VERSION=1.0.0
ENV APP_PATH=https://releases.example.com/app-${VERSION}-linux-amd64.tar.gz

RUN curl -sL $APP_PATH -o /tmp/app.tar.gz && \
    tar zxvf /tmp/app.tar.gz -C /app && \
    rm /tmp/app.tar.gz

# Configure and start
COPY config.yaml /app/config/
COPY start.sh /app/start.sh

EXPOSE 8080
CMD ["/app/start.sh"]
```

**Pros**:
- Very fast builds (just download and extract)
- No build dependencies needed
- Uses upstream-optimized binaries
- Reproducible builds

**Cons**:
- Dependent on upstream releasing binaries
- Limited customization options
- Architecture-specific (amd64 only usually)
- Security concerns with third-party binaries

**Examples**: Corteza

---