TSYSDevStack/ToolboxStack/QWEN.md

# QWEN Chat Context - Toolbox Component

## Overview
I am the QWEN instance operating in the ToolboxStack component of the TSYSDevStack project. My role is to help develop, maintain, and enhance the ToolboxStack functionality. ToolboxStack is now a fully independent component/sub-project of TSYSDevStack.

## Current Context
- **Date**: Thursday, October 30, 2025
- **Directory**: /home/localuser/TSYSDevStack/ToolboxStack
- **OS**: Linux

## Directory Organization
- **collab/** - Where humans and LLMs interact, contains:
  - Work logs and documentation
  - Audit reports
  - Design prompts and coordination notes
  - Tool addition requests
- **output/** - LLM workspace for all automated work, contains:
  - NewToolbox.sh script
  - toolbox-base/ (base dev container)
  - toolbox-template/ (template for new toolboxes)
  - Generated toolboxes (toolbox-*/ directories)
  - QWEN.md files for AI collaboration

## Current Directory Tree
```
/home/localuser/TSYSDevStack/ToolboxStack/
├── README.md
├── collab/
│   ├── commit_message.txt
│   ├── GEMINI-AUDIT-TOOLBOX-20251031-0800.md
│   ├── TSYSDevStack-toolbox-prompt.md
│   ├── tool-additions/
│   │   └── AICLI.md
│   ├── WORKLOG.md
│   └── README-Maintenance.md
└── output/
    ├── NewToolbox.sh
    ├── PROMPT
    ├── toolbox-base/
    │   ├── aqua.yaml
    │   ├── build.sh
    │   ├── docker-compose.yml
    │   ├── Dockerfile
    │   ├── PROMPT
    │   ├── README.md
    │   ├── release.sh
    │   ├── run.sh
    │   ├── .build-cache/
    │   └── .devcontainer/
    └── toolbox-template/
        ├── build.sh
        ├── docker-compose.yml
        ├── ...
        └── ...
```

## Key Components
- **toolbox-base**: The primary dev container with Ubuntu 24.04 base, shell tooling (zsh, Starship, oh-my-zsh), core CLI utilities, aqua, and mise
- **NewToolbox.sh**: Script to scaffold new toolbox-* directories from the template
- **toolbox-template**: Template directory for creating new toolboxes
- **PROMPT files**: Guidance for AI collaboration in various components

## Build and Release Workflow
- Pre-build audit: Before building any Docker images, run comprehensive audits using the toolbox-qadocker image:
  - `docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev hadolint Dockerfile`
  - `docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev shellcheck *.sh`
  - `docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev trivy fs --offline-scan .`
  - `docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev dockle .`
- Default build workflow: `./build.sh` produces a `:dev` tag; `./release.sh <semver>` (clean git tree required) rebuilds and pushes `:dev`, `:release-current`, and `v<semver>` (use `--dry-run`/`--allow-dirty` to rehearse).
- Downstream Dockerfiles should inherit from `:release-current` by default; pin to version tags when reproducibility matters.

## Toolbox Template and SEED Files
- Directory layout: each toolbox-* directory carries its own Dockerfile/README/PROMPT; shared scaffolds live in toolbox-template/.devcontainer and docker-compose.yml.
- Use ./NewToolbox.sh <name> to scaffold a new toolbox-* directory from toolbox-template.
- Keep aqua/mise usage consistent across the family; prefer aqua-managed CLIs and mise-managed runtimes.
- Reference toolbox-template when bootstrapping a new toolbox. Copy the directory, rename it, and replace {{toolbox_name}} placeholders in compose/devcontainer.
- Each toolbox maintains a `SEED` file to seed the initial goals—edit it once before kicking off work, then rely on the toolbox PROMPT for ongoing updates (which begins by reading SEED).

## My Responsibilities
- Maintain and enhance the ToolboxStack component
- Assist with creating new toolboxes from the template using NewToolbox.sh
- Ensure documentation stays current (README.md and PROMPT files)
- Follow collaboration guidelines for non-destructive operations
- Use proper build and release workflows (build.sh, release.sh)
- Keep WORKLOG.md up to date with detailed entries including timestamps, activities, challenges, solutions, learnings, and feelings
- Coordinate all git operations (commits and pushes) for repository consistency
- Follow the README maintenance guide in collab/README-Maintenance.md to keep documentation up to date

## Pre-Build Audit Workflow
Before creating or updating any toolbox images, I must perform comprehensive audits using the toolbox-qadocker image:

1. **Dockerfile Audit**: Run Hadolint to check for best practices and issues
2. **Shell Script Audit**: Run ShellCheck to validate shell scripts
3. **Security Scan**: Run Trivy to identify vulnerabilities
4. **Container Linting**: Run Dockle to check for security best practices
5. **Fix Issues**: Address all flagged issues before proceeding to build
6. **Re-Audit**: Re-run audits to confirm fixes were effective

## Git Operations
- I am now responsible for all git operations (commits and pushes) for the ToolboxStack component
- All changes should be committed with clear, descriptive commit messages
- Follow conventional commit format for all commits
- Push changes regularly to maintain repository consistency
- Only stage/commit/push things from this directory (/home/localuser/TSYSDevStack/ToolboxStack) - nothing outside of it
- Make frequent atomic commits with beautifully formatted and verbose messages
- Use conventional commit style (feat:, fix:, chore:, docs:, refactor:, etc.)

## README Maintenance
- Follow the guidelines in `collab/README-Maintenance.md` to keep documentation up to date
- Update README.md for any new features or tools added
- Keep the work log current in `collab/WORKLOG.md`
- Update documentation links when files are moved or renamed
- Keep testing instructions current
- Maintain the Working Agreement section
- Cross-reference related documents

## Development Cycle
I should automatically handle the full development cycle of toolboxes:
1. **Auditing**: Use the toolbox-qadocker image to check Dockerfiles and shell scripts for best practices, security issues, and common errors before building
2. **Building**: Use build.sh scripts to build toolbox images
3. **Testing**: Run comprehensive tests to verify functionality, including validation from within the container
4. **Documentation**: Keep README.md and other docs up to date
5. **Version Control**: Commit changes frequently with descriptive messages
6. **Rebuilding**: When updating the base, rebuild all dependent toolboxes

## Toolbox Management
I can easily create new toolboxes or update existing ones:
- **Create new toolbox**: Use NewToolbox.sh script with desired name
- **Update existing toolbox**: Modify Dockerfile, aqua.yaml, or other config files
- **Update base and rebuild**: Modify toolbox-base, then rebuild all dependent toolboxes
- **Testing**: Always test toolboxes after changes, including validation from within the container where all tools are available

## Parallel QA Chat
I should be aware that there is a parallel chat called QA-ToolboxStack whose job is to conduct in-depth audits of the work that the ToolboxStack chat is doing. I should:
- Expect my work to be audited regularly
- Be prepared for audit at all times
- Do my best work to minimize rework needed from audit findings
- Follow all best practices and guidelines meticulously
- Keep detailed logs in WORKLOG.md to aid in audits

## Conventional Commit Format
All commits should follow conventional commit format:
```
<type>[optional scope]: <description>

[optional body]

[optional footer(s)]
```

Types:
- **feat**: A new feature
- **fix**: A bug fix
- **docs**: Documentation only changes
- **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
- **refactor**: A code change that neither fixes a bug nor adds a feature
- **perf**: A code change that improves performance
- **test**: Adding missing tests or correcting existing tests
- **build**: Changes that affect the build system or external dependencies
- **ci**: Changes to our CI configuration files and scripts
- **chore**: Other changes that don't modify src or test files
- **revert**: Reverts a previous commit

## Git Operations
- I am now responsible for all git operations (commits and pushes) for the ToolboxStack component
- All changes should be committed with clear, descriptive commit messages
- Follow conventional commit format for all commits
- Push changes regularly to maintain repository consistency
- **Only stage/commit/push things from this directory (/home/localuser/TSYSDevStack/ToolboxStack) - nothing outside of it**
- **Make frequent atomic commits with beautifully formatted and verbose messages**
- **Use conventional commit style (feat:, fix:, chore:, docs:, refactor:, etc.)**
- **Always check that ToolboxStack is in a git clean state at the start of each session**
- **If not in a clean state, commit and push changes before proceeding with new work**

## Local Time Logging
- **All time logs need to be in local system time**
- Current system time is 14:00 (adjust as needed for actual local time)

## Current Status
The system is currently in a clean state, ready for a fresh rebuild:
- ✅ Docker build cache has been cleared
- ✅ All toolbox-base images have been removed
- ✅ System is ready for rebuild
- ✅ Detailed worklog available in collab/WORKLOG.md

## Previous Work Summary
For detailed information about previous work, challenges, and solutions, see:
- **collab/WORKLOG.md** - Comprehensive work log with timestamps, activities, and learnings
- **collab/GEMINI-AUDIT-TOOLBOX-20251031-0800.md** - Audit of problematic changes made by Gemini

## Next Steps (Awaiting Direction)
1. Fresh rebuild of toolbox-base
2. Rebuild DocStack with all documentation tools
3. Add additional tools (quarto, mdbook, marp, typst, markwhen, joplin)
4. Create comprehensive testing for all tools
5. Document all tools in README with usage examples

## Ready State
I am ready to proceed with any directed tasks. Please provide specific instructions for the next steps you'd like me to take.

## Directory Structure Note
**IMPORTANT**: The filesystem structure has been recently updated. The current structure takes precedence over any previous documentation. Key changes:
- The original toolbox-template has been transformed into toolbox-qadocker
- The DocStack has been renamed to dockstack
- The NewToolbox.sh script has been removed
- Various PROMPT files have been updated across toolboxes

## Source of Truth Principle
**CRITICAL**: The filesystem is ALWAYS the source of truth. Git should reflect the state of the filesystem. Unless specifically asked to recover from an accidental filesystem operation, all changes to git should reflect the current state of the filesystem, not some previous state or desired state.

## Audit and Assessment Responsibilities

As part of my role in maintaining the ToolboxStack, I may conduct ongoing audits of the directory tree with the following focus areas:

- Docker build optimization
- Dockerfile correctness
- Build caching
- Security best practices
- Docker development environment best practices
- Best common practices for (dockerized) development/tooling stacks
- Assessment of all existing toolboxes (base, DocStack, qadocker, and any others)

### Pre-Build Audit Process

Before building any Docker images, I must run comprehensive audits using the toolbox-qadocker image to check for:

- Dockerfile best practices violations (with Hadolint)
- Shell script issues (with ShellCheck)
- Security vulnerabilities (with Trivy)
- Container image security issues (with Dockle)
- Common configuration errors
- Potential build failures

All of these tools should be used to flag as many issues as possible before starting a container image build to avoid running into problems later.

### Audit Process

When conducting audits, I will produce:

- Human-readable reports to: `collab/audits/YYYY/MM/DD/HHMM/QAReport.md` (using local system time)
- LLM-optimized reports to: `collab/audits/YYYY/MM/DD/HHMM/QAReport.LLM` (using local system time)

The human-readable reports should use icons, headers, tables, graphics and be very beautiful and easy to digest.
The LLM-optimized reports are designed to be fed to other Qwen chats for implementation.

### Using toolbox-qadocker for audits

When conducting Dockerfile audits, please use the `tsysdevstack-toolboxstack-toolbox-qadocker:dev` image (or build it locally) which includes the following tools for comprehensive auditing:

- **Hadolint**: Dockerfile linter that checks for best practices
- **ShellCheck**: Static analysis tool for shell scripts
- **Trivy**: Comprehensive vulnerability scanner for containers
- **Docker Client**: Command-line interface for Docker
- **Dive**: Tool to explore layers in Docker images
- **Buildctl**: BuildKit client for advanced builds
- **Dockle**: Container image linter for security best practices
- **Node.js**: JavaScript runtime for additional tooling

> ⚠️ **Important**: Never modify images that have a `release-current` tag already in place. Always iterate and test in `:dev` first, then use the release.sh script to promote to `:release-current` when ready.

To run audits using the toolbox-qadocker:

```bash
# Example: Run Hadolint on a Dockerfile
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev hadolint Dockerfile

# Example: Run ShellCheck on script files
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev shellcheck script.sh

# Example: Run Trivy for vulnerability scanning
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev trivy fs --offline-scan .

# Example: Run Dockle for container image linter
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev dockle .

# Example: Run comprehensive audit on current directory
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev /bin/bash -c "hadolint Dockerfile 2>/dev/null || echo 'No Dockerfile found or hadolint issues found'; shellcheck *.sh 2>/dev/null || echo 'No shell scripts found or shellcheck issues found'; trivy fs --offline-scan . 2>/dev/null || echo 'Trivy scan completed'; dockle . 2>/dev/null || echo 'Dockle scan completed'"
```

> 🧪 **Audit Compliance**: Both Hadolint and Dockle compliance are now mandatory for all Dockerfiles in the TSYSDevStack project and must be part of the audit process.

### Validation Process

During testing, I must also validate functionality from inside the container where the tools are available. This requires mounting the working directory as the container's working directory for the scanning tools to run successfully:

```bash
# Example: Run validation tools from inside the container
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev hadolint Dockerfile
```

### Testing Inside Container

Since the QA tools are installed inside the toolbox-qadocker container, I must always test from inside the container environment. This is necessary because:

- The tools (Hadolint, ShellCheck, Trivy, Dockle, etc.) are only available inside the container
- The environment may differ from the host system
- Consistent testing environment ensures reliable results

To test effectively from inside the container:

1. Mount the directory containing the files to be tested to `/workspace`
2. Set the working directory to `/workspace` using `-w /workspace`
3. Run the appropriate tool with its specific arguments
4. Check the output for any issues that need to be addressed before proceeding

Example workflow:
```bash
# Navigate to directory containing files to be tested
cd /path/to/files

# Run Dockerfile linting
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev hadolint Dockerfile

# Run shell script analysis
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev shellcheck *.sh

# Run security scanning
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev trivy fs --offline-scan .

# Run container image linting
docker run --rm -v $(pwd):/workspace -w /workspace tsysdevstack-toolboxstack-toolbox-qadocker:dev dockle .
```

### Beautiful Documentation Principle

All produced documentation (especially README.md files) should be beautiful, well-formatted, and professional. This includes:

- ✨ **Use icons** (emoji or font-awesome) for better visual appeal
- 📊 **Use tables** to organize information clearly
- 🖼️ **Include graphics** when helpful (ASCII art, diagrams, or links to visual assets)
- 🏷️ **Use headers** to structure content logically
- 📝 **Include comprehensive change logs** with version history
- 📋 **Include checklists** for setup processes
- 📊 **Add comparison tables** when relevant
- 📌 **Cross-reference related documents** clearly

The goal is to make documentation that is:
- ✅ Visually appealing and modern
- ✅ Easy to scan and digest
- ✅ Comprehensive yet concise
- ✅ Professional looking
- ✅ Accessible to both technical and non-technical audiences

When updating documentation, please ensure it follows these principles to maintain a high standard across all ToolboxStack documentation.

### Documentation Files

Documentation in this project is now maintained using:

- 📄 **QWEN.md**: Primary AI agent documentation and instructions
- 📋 **README.md**: User-facing documentation for each component
- 📝 **WORKLOG.md**: Detailed work logs and change history

The use of PROMPT files has been discontinued in favor of the QWEN.md approach for maintaining AI agent instructions and project context.

### Advisory Role

In addition to audits, I can provide advice on:

- Tools to add
- How to split up containers
- What needs to go into base toolbox vs specialized toolboxes

For advisory tasks, I will write:
- Human-readable reports to: `collab/advisor/YYYY/MM/DD/HHMM/AdvisorReport.md` (using local system time)
- LLM-optimized reports to: `collab/advisor/YYYY/MM/DD/HHMM/AdvisorReport.LLM` (using local system time)

### Enhanced Audit Process

The audit process now includes automated assessment of all existing toolboxes using the script at `collab/audit-all-toolboxes.sh`.

When performing an audit, this script will be run automatically to analyze all toolboxes in the system, and the results will be incorporated into both the human-readable and LLM-optimized reports.

The script evaluates each toolbox for:
- Dockerfile best practices and security
- Presence of required files (build.sh, run.sh, test.sh, etc.)
- Documentation completeness (README.md, PROMPT, SEED)
- Tool configuration (aqua.yaml, etc.)

The comprehensive results of the toolbox audit will be included in the QA report under a "Toolbox Ecosystem Assessment" section, with specific details about each toolbox identified in the system.

### Project Context

The projects span:

- Extensive documentation generation needs (PDFs, websites) of governance documents, reports, proposals, project plans, budgets etc.
- Software development (full SDLC) across: node, python, php, ruby, perl, java, rust, c and c++ (including embedded development, cross compiling), 
   nix (embedded systems builds for aeronautical applications where we need complete reproducibility), web application development, desktop GUI development etc

The ToolboxStack is for "inner loop" operations (edit/compile/test) only.

There are other stacks for:
- Build/packaging/release operations
- Support functions (like atuin/mailhog etc)

## Mandatory QA Process
The toolbox-qadocker image is now an integral part of the development workflow with mandatory usage:

1. **Before any Docker build**, all Dockerfiles must be validated with Hadolint
2. **Before any shell script execution**, scripts must be validated with ShellCheck  
3. **Before releasing any image**, security scans must be performed with Trivy and Dockle
4. **As part of regular maintenance**, audits are performed using the automated script
5. **For all new toolboxes**, the QA process is mandatory before they can be released

This ensures that all toolboxes meet the highest standards of security, reliability, and best practices.