fix: correct databank architecture and implement proper CTO/COO structure\n\n- Remove incorrectly placed human/llm directories from databank root\n- Restructure databank with everything under databank/artifacts/ as requested\n- Implement proper CTO/COO structure under pmo/artifacts/ with complete PMO components\n- Create comprehensive collab/ directory structure for human/AI communication\n- Remove Joplin processing scripts and references as requested\n- Create proper scaffolding directories for quick domain standup\n- Update README documentation to reflect corrected architecture\n- Ensure only collab/ directories are editable by humans\n- AI agents manage databank/artifacts/ based on collab/ communications\n- Create structured intake templates and collaboration workflows\n- Maintain clear separation between readonly databank and read-write PMO\n- Implement proper single source of truth with AI-managed artifacts
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
@@ -1,153 +1,34 @@
|
||||
# Joplin Processing Pipeline
|
||||
# From Joplin Directory
|
||||
|
||||
This directory contains scripts and configurations for processing Joplin markdown exports.
|
||||
This directory is for dropping Joplin markdown exports for automatic ingestion into the databank.
|
||||
|
||||
## Structure
|
||||
## Purpose
|
||||
|
||||
```
|
||||
joplin-processing/
|
||||
├── process-joplin-export.sh # Main processing script
|
||||
├── convert-to-human-md.py # Convert Joplin to human-friendly markdown
|
||||
├── convert-to-llm-json.py # Convert Joplin to LLM-optimized JSON
|
||||
├── joplin-template-config.yaml # Template configuration
|
||||
├── processed/ # Processed files tracking
|
||||
└── README.md # This file
|
||||
```
|
||||
- **Content Ingestion**: Joplin markdown exports for processing
|
||||
- **Automatic Updates**: AI agents monitor this directory and update databank/artifacts/
|
||||
- **Seamless Integration**: Bridge between Joplin notes and databank structure
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Export**: Joplin notes exported as markdown
|
||||
2. **Place**: Drop exports in `../collab/fromjoplin/`
|
||||
3. **Trigger**: Processing script monitors directory
|
||||
4. **Convert**: Scripts convert to both human and LLM formats
|
||||
5. **Store**: Results placed in `../../artifacts/`, `../../human/`, and `../../llm/`
|
||||
6. **Track**: Processing logged in `processed/`
|
||||
1. **Export**: Export notes from Joplin as markdown
|
||||
2. **Drop**: Place exports in this directory
|
||||
3. **Process**: AI agents automatically process exports
|
||||
4. **Update**: Databank/artifacts/ updated with new content
|
||||
5. **Archive**: Processed exports moved to archive
|
||||
|
||||
## Processing Script
|
||||
## Guidelines
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# process-joplin-export.sh
|
||||
### For Humans
|
||||
- **Export as Markdown**: Use Joplin's markdown export feature
|
||||
- **Include Front Matter**: Preserve metadata and tags
|
||||
- **Organize by Topic**: Group related notes together
|
||||
- **Clear Naming**: Use descriptive filenames
|
||||
|
||||
JOPLIN_DIR="../collab/fromjoplin"
|
||||
HUMAN_DIR="../../human"
|
||||
LLM_DIR="../../llm"
|
||||
ARTIFACTS_DIR="../../artifacts"
|
||||
PROCESSED_DIR="./processed"
|
||||
|
||||
# Process new Joplin exports
|
||||
for file in "$JOPLIN_DIR"/*.md; do
|
||||
if [[ -f "$file" ]]; then
|
||||
filename=$(basename "$file")
|
||||
echo "Processing $filename..."
|
||||
|
||||
# Convert to human-friendly markdown
|
||||
python3 convert-to-human-md.py "$file" "$HUMAN_DIR/$filename"
|
||||
|
||||
# Convert to LLM-optimized JSON
|
||||
python3 convert-to-llm-json.py "$file" "$LLM_DIR/${filename%.md}.json"
|
||||
|
||||
# Store canonical version
|
||||
cp "$file" "$ARTIFACTS_DIR/$filename"
|
||||
|
||||
# Log processing
|
||||
echo "$(date): Processed $filename" >> "$PROCESSED_DIR/processing.log"
|
||||
|
||||
# Move processed file to avoid reprocessing
|
||||
mv "$file" "$PROCESSED_DIR/"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
## Conversion Scripts
|
||||
|
||||
### Human-Friendly Markdown Converter
|
||||
```python
|
||||
# convert-to-human-md.py
|
||||
import sys
|
||||
import yaml
|
||||
import json
|
||||
|
||||
def convert_joplin_to_human_md(input_file, output_file):
|
||||
"""Convert Joplin markdown to human-friendly format"""
|
||||
with open(input_file, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
# Parse front matter if present
|
||||
# Add beautiful formatting, tables, headers, etc.
|
||||
|
||||
# Write human-friendly version
|
||||
with open(output_file, 'w') as f:
|
||||
f.write(content)
|
||||
|
||||
if __name__ == "__main__":
|
||||
convert_joplin_to_human_md(sys.argv[1], sys.argv[2])
|
||||
```
|
||||
|
||||
### LLM-Optimized JSON Converter
|
||||
```python
|
||||
# convert-to-llm-json.py
|
||||
import sys
|
||||
import json
|
||||
import yaml
|
||||
from datetime import datetime
|
||||
|
||||
def convert_joplin_to_llm_json(input_file, output_file):
|
||||
"""Convert Joplin markdown to LLM-optimized JSON"""
|
||||
with open(input_file, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
# Parse and structure for LLM consumption
|
||||
# Extract key-value pairs, sections, metadata
|
||||
|
||||
structured_data = {
|
||||
"source": "joplin",
|
||||
"processed_at": datetime.now().isoformat(),
|
||||
"content": content,
|
||||
"structured": {} # Extracted structured data
|
||||
}
|
||||
|
||||
# Write LLM-optimized version
|
||||
with open(output_file, 'w') as f:
|
||||
json.dump(structured_data, f, indent=2)
|
||||
|
||||
if __name__ == "__main__":
|
||||
convert_joplin_to_llm_json(sys.argv[1], sys.argv[2])
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Template Configuration
|
||||
```yaml
|
||||
# joplin-template-config.yaml
|
||||
processing:
|
||||
input_format: "joplin_markdown"
|
||||
output_formats:
|
||||
- "human_markdown"
|
||||
- "llm_json"
|
||||
retention_days: 30
|
||||
|
||||
conversion_rules:
|
||||
human_friendly:
|
||||
add_tables: true
|
||||
add_formatting: true
|
||||
add_visual_hierarchy: true
|
||||
add_navigation: true
|
||||
|
||||
llm_optimized:
|
||||
minimize_tokens: true
|
||||
structure_data: true
|
||||
extract_metadata: true
|
||||
add_semantic_tags: true
|
||||
```
|
||||
|
||||
## Automation
|
||||
|
||||
Set up cron job or file watcher to automatically process new exports:
|
||||
|
||||
```bash
|
||||
# Run every 5 minutes
|
||||
*/5 * * * * cd /path/to/joplin-processing && ./process-joplin-export.sh
|
||||
```
|
||||
### For AI Agents
|
||||
- **Monitor Continuously**: Watch for new exports
|
||||
- **Parse Thoroughly**: Extract all relevant information
|
||||
- **Map to Structure**: Place content in appropriate databank/artifacts/ locations
|
||||
- **Update Tracking**: Maintain processing logs and archives
|
||||
- **Handle Errors**: Gracefully handle malformed exports
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user