Scaffold Cloudron packaging workspace and automation

2025-10-02 12:07:09 -05:00
parent b4121cc932
commit 482d4ff1b8
414 changed files with 6837 additions and 2 deletions
--- a/docs/APP_STATUS.md
+++ b/docs/APP_STATUS.md
@@ -0,0 +1,61 @@
+# Application Status
+
+_Updated: 2025-10-02T16:50:06Z_
+
+| Slug | Title | Version | Status | Issue |
+| --- | --- | --- | --- | --- |
+| apache-apisix | Apache APISIX | 0.1.0 | todo | https://projects.knownelement.com/issues/179 |
+| target-goalert | GoAlert | 0.1.0 | todo | https://projects.knownelement.com/issues/204 |
+| consuldemocracy | CONSUL Democracy | 0.1.0 | todo | https://projects.knownelement.com/issues/189 |
+| fleetdm-fleet | FleetDM Fleet | 0.1.0 | todo | https://projects.knownelement.com/issues/195 |
+| fonoster | Fonoster | 0.1.0 | todo | https://projects.knownelement.com/issues/227 |
+| healthchecks | Healthchecks | 0.1.0 | todo | https://projects.knownelement.com/issues/192 |
+| hyperswitch | HyperSwitch | 0.1.0 | todo | https://projects.knownelement.com/issues/209 |
+| netbox-docker | NetBox Docker | 0.1.0 | todo | https://projects.knownelement.com/issues/201 |
+| openboxes-docker | OpenBoxes Docker | 0.1.0 | todo | https://projects.knownelement.com/issues/205 |
+| openfile | OpenFile | 0.1.0 | todo | https://projects.knownelement.com/issues/316 |
+| sniperphish | SniperPhish | 0.1.0 | todo | https://projects.knownelement.com/issues/211 |
+| datahub | DataHub | 0.1.0 | todo | https://projects.knownelement.com/issues/309 |
+| easy-gate | Easy Gate | 0.1.0 | todo | https://projects.knownelement.com/issues/54 |
+| payroll-engine | Payroll Engine | 0.1.0 | todo | https://projects.knownelement.com/issues/208 |
+| huginn | Huginn | 0.1.0 | todo | https://projects.knownelement.com/issues/194 |
+| grist | Grist | 0.1.0 | todo | https://projects.knownelement.com/issues/191 |
+| docassemble | Docassemble | 0.1.0 | todo | https://projects.knownelement.com/issues/277 |
+| database-gateway | Database Gateway | 0.1.0 | todo | https://projects.knownelement.com/issues/273 |
+| rundeck | Rundeck | 0.1.0 | todo | https://projects.knownelement.com/issues/217 |
+| slurm | Slurm | 0.1.0 | todo | https://projects.knownelement.com/issues/222 |
+| rathole | rathole | 0.1.0 | todo | https://projects.knownelement.com/issues/225 |
+| jenkins | Jenkins | 0.1.0 | todo | https://projects.knownelement.com/issues/234 |
+| runme | Runme | 0.1.0 | todo | https://projects.knownelement.com/issues/322 |
+| seatunnel | SeaTunnel | 0.1.0 | todo | https://projects.knownelement.com/issues/301 |
+| docker-webhook | docker-webhook | 0.1.0 | todo | https://projects.knownelement.com/issues/271 |
+| inventree | InvenTree | 0.1.0 | todo | https://projects.knownelement.com/issues/173 |
+| tak-server | TAK Server | 0.1.0 | todo | https://projects.knownelement.com/issues/180 |
+| midday | Midday | 0.1.0 | todo | https://projects.knownelement.com/issues/178 |
+| killbill | Kill Bill | 0.1.0 | todo | https://projects.knownelement.com/issues/181 |
+| chirpstack | ChirpStack | 0.1.0 | todo | https://projects.knownelement.com/issues/184 |
+| craig | Craig (FOSS Discord Recorder) | 0.1.0 | todo | https://projects.knownelement.com/issues/185 |
+| elabftw | eLabFTW | 0.1.0 | todo | https://projects.knownelement.com/issues/188 |
+| jamovi | jamovi | 0.1.0 | todo | https://projects.knownelement.com/issues/196 |
+| kibot | KiBot | 0.1.0 | todo | https://projects.knownelement.com/issues/197 |
+| resgrid | Resgrid Core | 0.1.0 | todo | https://projects.knownelement.com/issues/214 |
+| reviewboard | Review Board | 0.1.0 | todo | https://projects.knownelement.com/issues/216 |
+| satnogs-kaitai | SatNOGS Kaitai | 0.1.0 | todo | https://projects.knownelement.com/issues/218 |
+| satnogs-webgui | SatNOGS WebGUI | 0.1.0 | todo | https://projects.knownelement.com/issues/218 |
+| sdrangel | SDRAngel | 0.1.0 | todo | https://projects.knownelement.com/issues/219 |
+| signoz | SigNoz | 0.1.0 | todo | https://projects.knownelement.com/issues/221 |
+| warp | Warp | 0.1.0 | todo | https://projects.knownelement.com/issues/228 |
+| drawio | draw.io | 0.1.0 | todo | https://projects.knownelement.com/issues/272 |
+| openblocks | OpenBlocks | 0.1.0 | todo | https://projects.knownelement.com/issues/274 |
+| wireviz-web | Wireviz Web | 0.1.0 | todo | https://projects.knownelement.com/issues/276 |
+| autobom | Autobom | 0.1.0 | todo | https://projects.knownelement.com/issues/278 |
+| plmore | PLMore | 0.1.0 | todo | https://projects.knownelement.com/issues/279 |
+| manyfold | Manyfold | 0.1.0 | todo | https://projects.knownelement.com/issues/282 |
+| langfuse | Langfuse OSS LLMOps Stack | 0.1.0 | todo | https://projects.knownelement.com/issues/283 |
+| puter | Puter | 0.1.0 | todo | https://projects.knownelement.com/issues/286 |
+| windmill | Windmill | 0.1.0 | todo | https://projects.knownelement.com/issues/285 |
+| swupdate | swupdate | 0.1.0 | todo | https://projects.knownelement.com/issues/326 |
+| mender-server | Mender Server | 0.1.0 | todo | https://projects.knownelement.com/issues/300 |
+| wireflow | Wireflow | 0.1.0 | todo | https://projects.knownelement.com/issues/50 |
+| nautilus-trader | Nautilus Trader | 0.1.0 | todo | https://projects.knownelement.com/issues/226 |
+| mirlo | Mirlo | 0.1.0 | todo | https://projects.knownelement.com/issues/TBD |
--- a/docs/CI_CD_GITEA.md
+++ b/docs/CI_CD_GITEA.md
@@ -0,0 +1,35 @@
+# Gitea CI/CD and Registry Integration
+
+This project uses the Gitea Actions runner and the built-in container registry hosted at `https://git.knownelement.com`. The workflow definition lives under `.gitea/workflows/ci.yml` and targets the Gitea Actions runtime (1.21+) alongside the built-in container registry available on current releases. The workflow is currently configured for manual `workflow_dispatch` runs so all routine testing stays on the local harness until a runner is available. citeturn0search0turn1search0
+
+## Prerequisites
+
+1. **Enable Actions** on the Gitea instance and mirror required upstream actions (`actions/checkout@v4`, optional others) via the "Actions" admin panel. citeturn0search0
+2. **Provision a runner** (e.g. `act_runner`) with Docker access so jobs can launch containers. citeturn0search5
+3. **Authenticate to the registry** by generating a Gitea access token (scope `write:package`) and logging in via Docker:
+   ```bash
+   docker login git.knownelement.com -u <username> -p <token>
+   ```
+   citeturn1search0
+
+## Workflow overview
+
+The manual `workflow_dispatch` job builds the `docker/ci-runner` image, then executes `scripts/ci_local.sh` with the requested task list (default `all`). This mirrors the local harness, so whatever succeeds locally will succeed in CI.
+
+> Re-enable push/PR triggers once a runner is available and `make ci-local` is consistently green.
+
+## Container registry usage
+
+- Tag Cloudron packages against the registry namespace, e.g. `git.knownelement.com/knel/cloudron/apache-apisix:<version>`.
+- The packager helper script reads `IMAGE_NAME`; override it when pushing to the registry:
+  ```bash
+  IMAGE_NAME=git.knownelement.com/knel/cloudron-packager BUILD=1 scripts/run_packager.sh
+  docker push git.knownelement.com/knel/cloudron-packager
+  ```
+- Cloudron’s CLI can push directly to the registry once you log in within the packager container.
+
+## Future enhancements
+
+- Add `make lint` and `make status` as required checks in Gitea branch protection.
+- Extend the workflow with matrix builds for priority apps (e.g. run smoke scripts once implemented).
+
--- a/docs/LOCAL_TESTING.md
+++ b/docs/LOCAL_TESTING.md
@@ -0,0 +1,33 @@
+# Local Test Harness
+
+All verification runs locally inside Docker containers so your workstation matches the eventual Gitea Actions runner exactly.
+
+## CI-equivalent container
+
+- `docker/ci-runner/Dockerfile` builds the image `knel/cloudron-ci`, derived from the same base the Gitea runner will use.
+- `scripts/ci_local.sh` orchestrates tasks inside that container. Run `BUILD=1 ./scripts/ci_local.sh lint` the first time to build the image.
+- Tasks:
+  - `lint` → `make lint` + `make status` + `git diff docs/APP_STATUS.md`
+  - `packager-smoke` → builds `docker/packager` and runs `cloudron --help`
+  - `all` (default) → runs both.
+
+The script mounts `/var/run/docker.sock` so Docker CLI calls inside the container reuse the host daemon.
+
+## Git hooks
+
+Use `scripts/hooks/install_hooks.sh` to install local hooks:
+
+```bash
+./scripts/hooks/install_hooks.sh
+```
+
+- `pre-commit` runs `./scripts/ci_local.sh lint`.
+- `post-commit` refreshes `docs/APP_STATUS.md` (non-fatal).
+- `pre-push` runs the packager smoke test.
+
+Set `SKIP_CI_HOOKS=1` when you need to bypass the hooks temporarily.
+
+## Alignment with Gitea Actions
+
+The `.gitea/workflows/ci.yml` workflow wraps the same `scripts/ci_local.sh` entrypoint and is currently manual-only (`workflow_dispatch`). Enable push/PR triggers once a runner is provisioned.
+
--- a/docs/PACKAGING_GUIDE.md
+++ b/docs/PACKAGING_GUIDE.md
@@ -0,0 +1,82 @@
+# Cloudron Packaging Playbook
+
+This repository standardises the workflow for building and maintaining Cloudron packages for the Known Element portfolio.
+
+## Reference workflow
+
+1. Generate a package scaffold from the shared template using `scripts/new_app.py`.
+2. Run all build, test, and release tasks inside the `docker/packager` container to avoid host pollution.
+3. Implement application-specific build steps in `apps/<slug>/Dockerfile` and configure the runtime via `start.sh`.
+4. Update `CloudronManifest.json` with accurate metadata, addons, ports, and health checks based on the upstream project.
+5. Build and test locally with the Cloudron CLI (`cloudron build`, `cloudron install --app <domain>`) inside the packaging container.
+6. Push the image to the Cloudron registry (`cloudron push`) and publish once happy with smoke-test coverage.
+
+## Cloudron packaging essentials
+
+- **Base image**: Start from `cloudron/base:<version>` and add only the dependencies the app requires.
+- **CloudronManifest**: The manifest declares metadata, exposed ports, resource limits, addons (databases, object storage, email), and health checks. Always keep the `id` stable since it uniquely identifies the package inside Cloudron ecosystems.
+- **Start script**: Use `start.sh` to render configuration from environment variables and launch the primary process under the `cloudron` user.
+- **Tests**: Provide smoke tests in `test/` that Cloudron can run via `cloudron build --test`.
+- **Updates**: Bump the manifest `version`, document the change in `changelog`, and rebuild when upstream releases.
+
+## Repository linting
+
+Run the scaffold lint checks after editing an app to catch placeholder artefacts:
+
+```bash
+python3 scripts/lint_repo.py
+```
+
+Set `CLOUDRON_BASE` to override the expected base image version when needed.
+
+Use the Makefile shortcuts for common tasks:
+
+```bash
+make scaffold   # regenerate scaffolds from catalog
+make lint       # run repo lint checks
+make status     # refresh docs/APP_STATUS.md
+```
+
+Leverage the Gitea Actions workflow (`.gitea/workflows/ci.yml`) to enforce `make lint` and status generation on every push; instructions for runner setup live in `docs/CI_CD_GITEA.md`.
+
+Run the local harness via `./scripts/ci_local.sh` (or `make ci-local`) to replicate the CI pipeline without relying on a remote runner; see `docs/LOCAL_TESTING.md` for hook integration.
+
+## Using the packager container
+
+```bash
+# Build the helper image
+BUILD=1 scripts/run_packager.sh
+
+# Launch an interactive shell inside the packaging environment
+scripts/run_packager.sh
+```
+
+The container bundles the Cloudron CLI, Docker CLI, git, curl, and other tooling for reproducible builds. The host Docker socket is mounted so you can reuse local credentials while keeping the host clean.
+
+## Adding a new application
+
+```bash
+# Create the full scaffold for all catalog entries
+python3 scripts/new_app.py
+
+# Or generate a single skeleton
+python3 scripts/new_app.py --slug apache-apisix
+```
+
+Each scaffold contains:
+
+- `Dockerfile` – build instructions
+- `start.sh` – runtime entrypoint
+- `CloudronManifest.json` – metadata and permissions
+- `test/smoke.sh` – placeholder smoke test
+- `metadata.json` – issue and upstream bookkeeping
+- `README.md` – packaging checklist
+
+Update these files with the app-specific details, add upstream artefacts under `app/`, and commit the changes.
+
+## Repository etiquette
+
+- Document design decisions and manual steps in `docs/` or the per-app README.
+- Keep automation scripts idempotent; rerunning them should not damage uncommitted work.
+- Use semantic versioning in manifest files (`MAJOR.MINOR.PATCH`).
+
--- a/docs/PRIORITY_MILESTONES.md
+++ b/docs/PRIORITY_MILESTONES.md
@@ -0,0 +1,49 @@
+# Priority Packaging Milestones
+
+_Target date references assume today is 2025-10-02._
+
+## 1. Apache APISIX
+- **Dependencies**: Requires etcd for configuration storage; default proxy port 9080 and Admin API on 9180 (traditional mode). citeturn1search1turn1search4
+- **Cloudron add-ons**: Package etcd as an internal service (local or addon) or provision external managed etcd.
+- **Key tasks**:
+  1. Implement multi-container Docker build that bundles APISIX gateway and colocated etcd with supervised lifecycle.
+  2. Expose Cloudron TCP proxy support for 9080/9443 proxy & 9180 admin, restrict Admin API via Cloudron ACL.
+  3. Template `conf/config.yaml` to align with Cloudron env vars, including dynamic Admin API keys and optional standalone mode for static config.
+  4. Provide smoke test hitting `/apisix/admin/routes` with Cloudron-issued token.
+- **Milestone window**: Design + PoC by 2025-10-16; functional package & review by 2025-11-01.
+
+## 2. NetBox (netbox-docker)
+- **Dependencies**: PostgreSQL 14+, Redis ≥4.0, persistent media storage; official guidance discourages external Redis and highlights resource needs (8 vCPU, 24 GB RAM for HA). citeturn2search0turn2search1turn2search2
+- **Cloudron add-ons**: Postgres, Redis, object storage (optional), mail outbox.
+- **Key tasks**:
+  1. Adapt upstream docker-compose into Cloudron single-image bundle with supervisord orchestrating web, worker, scheduler.
+  2. Configure manifests for dual Redis (cache + rq) using Cloudron Redis addon namespaces.
+  3. Wire migrations into `start.sh`; ensure plugin directories mounted from `/app/data` for persistence.
+  4. Expand smoke tests to include `/api/` health probe and Celery worker queue check.
+- **Milestone window**: Packaging draft by 2025-10-23; regression-tested release candidate by 2025-11-08.
+
+## 3. Jenkins
+- **Dependencies**: Requires Java 17/21 runtime; official Docker images now default to Java 21 with long-term support for Java 17+. citeturn3search0turn3search9turn3search11
+- **Cloudron add-ons**: Local storage (persistent), optional object storage for builds, outbound mail.
+- **Key tasks**:
+  1. Base image customization layering on top of `jenkins/jenkins:lts-jdk21`; harden permissions and align user IDs with Cloudron.
+  2. Inject Cloudron env integration (admin user + credentials in secrets); support optional LDAP via Cloudron directory service when available.
+  3. Provide maintenance script for plugin catalog sync and backup to `/app/data/backups`.
+  4. Smoke test to assert controller readiness on `/login` and verify Java version output.
+- **Milestone window**: Container adaptation by 2025-10-09; plugin bootstrap + documentation by 2025-10-20.
+
+## 4. DataHub
+- **Dependencies**: Core services (GMS, frontend, Kafka, MySQL, Elasticsearch, optional Neo4j) shipped via docker-compose; quickstart expects 2 CPUs, 8 GB RAM. citeturn4search0turn4search3
+- **Cloudron add-ons**: MySQL addon (primary metadata store), optional Kafka/Elasticsearch via external services or sidecar containers; large persistent volumes.
+- **Key tasks**:
+  1. Determine feasibility of multi-container package (supervisord) vs. encouraging managed dependencies; document resource footprint.
+  2. Externalize credentials & admin user provisioning through Cloudron secrets.
+  3. Provide scripted ingestion of sample metadata gated behind env flag.
+  4. Implement health probes for GMS (`/health`), frontend (`/health`) and background consumer status.
+- **Milestone window**: Architecture decision record by 2025-10-18; initial Cloudron package by 2025-11-12.
+
+## Cross-cutting Actions
+- Update Cloudron base image references repo-wide automatically when new base images (currently `cloudron/base:5.0.0` from the 8.3 release) ship. citeturn0search1
+- Use `make lint` and `make status` gates in CI to enforce placeholder removal before PRs merge.
+- Capture app-specific ADRs under `docs/apps/<slug>/ADR-0001.md` as work begins.
+