docs: update onboarding and scripts
This commit is contained in:
cawcenter
66
SPARK_ONBOARDING.md
Normal file
66
SPARK_ONBOARDING.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Spark Platform: Zero-to-One Onboarding Guide
|
||||
|
||||
> **Current State**: Phase 8 Complete (Premium Features Active)
|
||||
> **Last Updated**: 2025-12-13
|
||||
|
||||
## 1. Project Overview
|
||||
Spark is a high-performance content scaling platform ("Titanium Pro" edition). It leverages Directus (CMS/Data) and Astro/React (Frontend) to deliver enterprise-grade mass content generation.
|
||||
|
||||
## 2. Key Architecture
|
||||
- **Frontend**: Astro (SSR) + React (Interactive Islands).
|
||||
- **Backend**: Directus Headless CMS (Postgres).
|
||||
- **Styling**: TailwindCSS (Titanium Pro Design System: Dark Mode, Glassmorphism, Neon Accents).
|
||||
- **Deployment**: Coolify (Dockerized).
|
||||
|
||||
## 3. Core Modules (Implemented)
|
||||
|
||||
### 🏭 Factory Floor (`/admin/content/factory`)
|
||||
The bread and butter of the operation.
|
||||
- **Components**: `KanbanBoard`, `BulkGrid`.
|
||||
- **Purpose**: Manage the lifecycle of thousands of articles.
|
||||
|
||||
### 🧠 Intelligence Station (`/admin/intelligence`)
|
||||
Data-driven insights engine.
|
||||
- **Geospatial Map (`/geo-targeting`)**: visualize market dominance with heatmaps using Leaflet.
|
||||
- **Trend Analysis**: Analyze keyword volatility.
|
||||
- **Avatar Metrics**: Track performance by user persona.
|
||||
|
||||
### 🧩 Assembler Engine (`/admin/assembler`)
|
||||
The heart of content generation.
|
||||
- **Template Composer**: 3-pane editor for Spintax templates.
|
||||
- **Automation Builder (`/automations/workflow`)**: **[NEW]** Visual Node-based editor (React Flow) to design content pipelines (Trigger -> Generate -> Publish).
|
||||
|
||||
### 📊 Command Center (`/admin/analytics`)
|
||||
- **Metrics Dashboard (`/metrics`)**: **[NEW]** Professional Tremor-based analytics (Area Charts, Donut Charts) tracking traffic and engagement.
|
||||
|
||||
### 🛡️ Testing Suite (`/admin/testing`)
|
||||
- **Visual Editor (`/blocks/editor`)**: Drag-and-drop page builder (Craft.js).
|
||||
- **SEO/Readability**: Real-time content scoring (Flesch-Kincaid).
|
||||
|
||||
## 4. Development Standards
|
||||
|
||||
### "Titanium Pro" Design Rules
|
||||
- **Colors**: Background `#09090b` (Zinc-950), Accents `#eab308` (Yellow-500) and `#22c55e` (Green-500).
|
||||
- **Components**: Always use **Shadcn/UI** (`/components/ui/*.tsx`).
|
||||
- **Icons**: Lucide React (`<Edit className="h-4 w-4" />`).
|
||||
- **Animations**: Use `framer-motion` for complex interactions, or CSS transitions for hover states.
|
||||
|
||||
### Code Quality
|
||||
- **Typescript**: Strict mode. No `any` unless absolutely necessary (and temporarily).
|
||||
- **Linting**: Run `npm run lint` before committing.
|
||||
- **Testing**: Use the internal `/admin/testing/suite` to validate logic changes.
|
||||
|
||||
## 5. Deployment Workflow
|
||||
1. **Commit**: `git push` triggers Coolify build.
|
||||
2. **Build Process**: Dockerfile builds frontend (SSR).
|
||||
3. **Validation**: Check `https://launch.jumpstartscaling.com` for the live site.
|
||||
4. **Staging**: Use the "Preview" button in `CollectionManager` to vie draft content at `https://launch.jumpstartscaling.com/site/[id]/preview/[post_id]`.
|
||||
|
||||
## 6. Next Actions (Phase 7: Polish)
|
||||
The logic is done. The next phase is strictly **Polish**:
|
||||
- [ ] **Accessibility Audit**: Ensure ARIA labels on all new UI.
|
||||
- [ ] **Documentation**: Expand inline JSDoc for complex engines (`engine.ts`).
|
||||
- [ ] **Performance**: Optimize React Flow rendering for large graphs.
|
||||
|
||||
---
|
||||
**Note to AI Agent**: You are working on a codebase that is feature-complete for Alpha. Your goal is stability and refinement. Do not refactor core engines without explicit instruction.
|
||||
@@ -1,417 +1,30 @@
|
||||
# Spark Platform - Management Scripts
|
||||
|
||||
This directory contains powerful Node.js utilities for managing your Spark Directus instance through the API. All scripts connect to `https://spark.jumpstartscaling.com` using admin credentials.
|
||||
|
||||
## 🔧 Available Scripts
|
||||
|
||||
### 1. Connection Test
|
||||
**File:** `test_directus_connection.js` (in project root)
|
||||
|
||||
Tests basic connectivity and admin access to Directus.
|
||||
|
||||
```bash
|
||||
node test_directus_connection.js
|
||||
```
|
||||
|
||||
**What it checks:**
|
||||
- Server availability
|
||||
- Admin authentication
|
||||
- Collections list
|
||||
- Record counts
|
||||
- Write permissions
|
||||
|
||||
---
|
||||
description: Learn how to manage deployment scripts and database migrations
|
||||
---
|
||||
|
||||
### 2. Schema Audit
|
||||
**File:** `audit_schema.js`
|
||||
# Scripts & Automation
|
||||
|
||||
Comprehensive audit of all collections, fields, and relationships.
|
||||
This directory contains utility scripts for managing the Spark Platform.
|
||||
|
||||
```bash
|
||||
node scripts/audit_schema.js
|
||||
```
|
||||
## Core Scripts
|
||||
|
||||
**Features:**
|
||||
- Lists all collections with record counts
|
||||
- Shows all fields and their interfaces
|
||||
- Identifies missing relationships
|
||||
- Detects UX issues (wrong field types, missing dropdowns)
|
||||
- Saves detailed report to `schema_audit_report.json`
|
||||
- `deploy.sh`: (Legacy) Basic shell script for deployment. We now use **Coolify**.
|
||||
- `seed.ts`: Database seeder (if applicable).
|
||||
|
||||
**Output:**
|
||||
- Field-by-field analysis
|
||||
- Relationship mapping
|
||||
- Issue summary
|
||||
- Recommendations
|
||||
## Deployment
|
||||
|
||||
---
|
||||
We use **Coolify** for all deployments. The `Dockerfile` in the root determines the build process.
|
||||
- **Frontend**: Built via `npm run build` (Astro).
|
||||
- **Backend**: Directus via Docker Image.
|
||||
|
||||
### 3. UX Improvements
|
||||
**File:** `improve_ux.js`
|
||||
## Database Migrations
|
||||
|
||||
Automatically fixes field interfaces to make Directus admin UI more user-friendly.
|
||||
Directus handles migrations internally.
|
||||
- To export schema: `npx directus schema snapshot ./schema.yaml`
|
||||
- To apply schema: `npx directus schema apply ./schema.yaml`
|
||||
|
||||
```bash
|
||||
node scripts/improve_ux.js
|
||||
```
|
||||
## ⚠️ Important Notes
|
||||
|
||||
**What it fixes:**
|
||||
- ✅ `site_id` fields → Select dropdown with site names
|
||||
- ✅ `campaign_id` fields → Select dropdown with campaign names
|
||||
- ✅ `status` fields → Dropdown with predefined choices
|
||||
- ✅ `avatar_key` fields → Avatar selection dropdown
|
||||
- ✅ JSON fields → Code editor with syntax highlighting
|
||||
- ✅ Content fields → Rich text HTML editor
|
||||
- ✅ Date fields → Proper datetime picker
|
||||
- ✅ Adds helpful descriptions and placeholders
|
||||
|
||||
**Results:**
|
||||
- Posts/Pages easily connect to Sites
|
||||
- Status uses visual labels
|
||||
- JSON editing with syntax highlighting
|
||||
- Better field validation
|
||||
|
||||
---
|
||||
|
||||
### 4. Schema Validation
|
||||
**File:** `validate_schema.js`
|
||||
|
||||
Complete validation of schema integrity, relationships, and data quality.
|
||||
|
||||
```bash
|
||||
node scripts/validate_schema.js
|
||||
```
|
||||
|
||||
**Validation checks:**
|
||||
1. **Collection Data** - Verifies all critical collections exist
|
||||
2. **Relationships** - Tests that foreign keys work correctly
|
||||
3. **Field Interfaces** - Confirms UX improvements are applied
|
||||
4. **Data Integrity** - Checks for orphaned records
|
||||
|
||||
**Output:**
|
||||
- Detailed validation report
|
||||
- Issue severity ratings (high/medium/low)
|
||||
- Saves `validation_report.json`
|
||||
|
||||
---
|
||||
|
||||
### 5. Bulk Import/Export
|
||||
**File:** `bulk_io.js`
|
||||
|
||||
Export and import any collection as JSON files.
|
||||
|
||||
```bash
|
||||
# Export all collections
|
||||
node scripts/bulk_io.js export all
|
||||
|
||||
# Export single collection
|
||||
node scripts/bulk_io.js export sites
|
||||
|
||||
# Import from file
|
||||
node scripts/bulk_io.js import sites sites_backup.json
|
||||
|
||||
# List available exports
|
||||
node scripts/bulk_io.js list
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Exports with metadata (timestamp, record count)
|
||||
- Batch import with conflict resolution
|
||||
- Automatic pagination for large collections
|
||||
- Update existing records on conflict
|
||||
- All exports saved to `./exports/` directory
|
||||
|
||||
**Use cases:**
|
||||
- Backup before major changes
|
||||
- Move data between environments
|
||||
- Share sample data
|
||||
- Restore deleted records
|
||||
|
||||
---
|
||||
|
||||
### 6. Geo Intelligence Manager
|
||||
**File:** `geo_manager.js`
|
||||
|
||||
Easy management of locations in `geo_intelligence` collection.
|
||||
|
||||
```bash
|
||||
# List all locations
|
||||
node scripts/geo_manager.js list
|
||||
|
||||
# Add a location
|
||||
node scripts/geo_manager.js add "Miami" "FL" "US" "southeast"
|
||||
|
||||
# Remove a location
|
||||
node scripts/geo_manager.js remove <id>
|
||||
|
||||
# Activate/deactivate location
|
||||
node scripts/geo_manager.js activate <id>
|
||||
node scripts/geo_manager.js deactivate <id>
|
||||
|
||||
# Update a field
|
||||
node scripts/geo_manager.js update <id> cluster "northeast"
|
||||
|
||||
# Seed with top 20 US cities
|
||||
node scripts/geo_manager.js seed-us-cities
|
||||
|
||||
# Import from CSV
|
||||
node scripts/geo_manager.js import-csv locations.csv
|
||||
```
|
||||
|
||||
**CSV Format:**
|
||||
```csv
|
||||
city,state,country,cluster,population
|
||||
Miami,FL,US,southeast,467963
|
||||
Boston,MA,US,northeast,692600
|
||||
```
|
||||
|
||||
**Built-in sample data:**
|
||||
- Top 20 US cities by population
|
||||
- Includes clusters (northeast, south, midwest, west, etc.)
|
||||
- Ready to use with `seed-us-cities` command
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start Guide
|
||||
|
||||
### First Time Setup
|
||||
|
||||
1. **Test your connection:**
|
||||
```bash
|
||||
node test_directus_connection.js
|
||||
```
|
||||
|
||||
2. **Audit current schema:**
|
||||
```bash
|
||||
node scripts/audit_schema.js
|
||||
```
|
||||
|
||||
3. **Apply UX improvements:**
|
||||
```bash
|
||||
node scripts/improve_ux.js
|
||||
```
|
||||
|
||||
4. **Validate everything works:**
|
||||
```bash
|
||||
node scripts/validate_schema.js
|
||||
```
|
||||
|
||||
5. **Backup all data:**
|
||||
```bash
|
||||
node scripts/bulk_io.js export all
|
||||
```
|
||||
|
||||
### Daily Operations
|
||||
|
||||
**Working with locations:**
|
||||
```bash
|
||||
# See all locations
|
||||
node scripts/geo_manager.js list
|
||||
|
||||
# Add custom location
|
||||
node scripts/geo_manager.js add "Portland" "OR" "US" "northwest"
|
||||
```
|
||||
|
||||
**Backing up before changes:**
|
||||
```bash
|
||||
node scripts/bulk_io.js export sites
|
||||
node scripts/bulk_io.js export generation_jobs
|
||||
```
|
||||
|
||||
**Checking system health:**
|
||||
```bash
|
||||
node scripts/validate_schema.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 What Each Script Fixed
|
||||
|
||||
### Before UX Improvements:
|
||||
- ❌ `site_id` fields showed UUID text input
|
||||
- ❌ `status` fields were plain text
|
||||
- ❌ JSON fields used tiny text box
|
||||
- ❌ Content used plain textarea
|
||||
- ❌ No field descriptions or help text
|
||||
|
||||
### After UX Improvements:
|
||||
- ✅ `site_id` shows dropdown with site names
|
||||
- ✅ `status` has predefined choices with colors
|
||||
- ✅ JSON fields have code editor with syntax highlighting
|
||||
- ✅ Content uses rich text HTML editor
|
||||
- ✅ All fields have helpful descriptions
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Confirmed Working Relationships
|
||||
|
||||
All relationships tested and verified:
|
||||
|
||||
1. **Sites → Posts** ✅
|
||||
- Posts connected to sites via `site_id`
|
||||
- Dropdown shows site names
|
||||
|
||||
2. **Sites → Pages** ✅
|
||||
- Pages connected to sites via `site_id`
|
||||
- Easy site selection
|
||||
|
||||
3. **Campaign → Generated Articles** ✅
|
||||
- Articles linked to campaigns
|
||||
- Track which campaign created each article
|
||||
|
||||
4. **Generation Jobs → Sites** ✅
|
||||
- Jobs know which site they're for
|
||||
- Filters work correctly
|
||||
|
||||
---
|
||||
|
||||
## 📁 Export Directory Structure
|
||||
|
||||
After running bulk export, you'll have:
|
||||
|
||||
```
|
||||
exports/
|
||||
├── avatar_intelligence_2025-12-13.json (10 records)
|
||||
├── avatar_variants_2025-12-13.json (30 records)
|
||||
├── campaign_masters_2025-12-13.json (2 records)
|
||||
├── cartesian_patterns_2025-12-13.json (3 records)
|
||||
├── content_fragments_2025-12-13.json (150 records)
|
||||
├── generated_articles_2025-12-13.json (0 records)
|
||||
├── generation_jobs_2025-12-13.json (30 records)
|
||||
├── geo_intelligence_2025-12-13.json (3 records)
|
||||
├── sites_2025-12-13.json (3 records)
|
||||
└── ... (all other collections)
|
||||
```
|
||||
|
||||
Each JSON file includes:
|
||||
```json
|
||||
{
|
||||
"collection": "sites",
|
||||
"exportedAt": "2025-12-13T14:30:00.000Z",
|
||||
"recordCount": 3,
|
||||
"data": [...]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Common Tasks
|
||||
|
||||
### Add Multiple Locations from CSV
|
||||
|
||||
1. Create `locations.csv`:
|
||||
```csv
|
||||
city,state,country,cluster,population
|
||||
Seattle,WA,US,northwest,753675
|
||||
Portland,OR,US,northwest,652503
|
||||
San Francisco,CA,US,west,881549
|
||||
```
|
||||
|
||||
2. Import:
|
||||
```bash
|
||||
node scripts/geo_manager.js import-csv locations.csv
|
||||
```
|
||||
|
||||
### Backup Before Major Changes
|
||||
|
||||
```bash
|
||||
# Export everything
|
||||
node scripts/bulk_io.js export all
|
||||
|
||||
# Make your changes in Directus UI...
|
||||
|
||||
# If something goes wrong, restore:
|
||||
node scripts/bulk_io.js import sites exports/sites_2025-12-13.json
|
||||
```
|
||||
|
||||
### Check What Needs Fixing
|
||||
|
||||
```bash
|
||||
# See what's wrong
|
||||
node scripts/audit_schema.js
|
||||
|
||||
# Auto-fix field interfaces
|
||||
node scripts/improve_ux.js
|
||||
|
||||
# Verify fixes worked
|
||||
node scripts/validate_schema.js
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "Authentication failed"
|
||||
- Check credentials in script files
|
||||
- Verify admin token hasn't expired
|
||||
- Test with: `node test_directus_connection.js`
|
||||
|
||||
### "Collection not found"
|
||||
- Collection may not exist yet
|
||||
- Run audit to see all collections: `node scripts/audit_schema.js`
|
||||
- Check schema is initialized
|
||||
|
||||
### Import conflicts (409 errors)
|
||||
- Script automatically tries to UPDATE existing records
|
||||
- Check the import summary for failed records
|
||||
- Review data for duplicate IDs
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Directus API Docs](https://docs.directus.io/reference/introduction.html)
|
||||
- [Spark Onboarding Guide](../spark_onboarding.md)
|
||||
- [Campaign Setup Guide](../CAMPAIGN_SETUP_GUIDE.md)
|
||||
|
||||
---
|
||||
|
||||
## ✅ Current System Status
|
||||
|
||||
**Last Validation:** 2025-12-13
|
||||
|
||||
- ✅ 11/11 critical collections exist
|
||||
- ✅ 9/11 collections have data
|
||||
- ✅ 4/4 relationships working
|
||||
- ✅ 32 field interfaces improved
|
||||
- ✅ 251 total records
|
||||
- ✅ 30 pending generation jobs
|
||||
- ✅ Zero data integrity issues
|
||||
|
||||
**Ready for production use!** 🎉
|
||||
|
||||
---
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Always backup before bulk changes:**
|
||||
```bash
|
||||
node scripts/bulk_io.js export all
|
||||
```
|
||||
|
||||
2. **Use validation after making schema changes:**
|
||||
```bash
|
||||
node scripts/validate_schema.js
|
||||
```
|
||||
|
||||
3. **Check exports directory regularly:**
|
||||
```bash
|
||||
node scripts/bulk_io.js list
|
||||
```
|
||||
|
||||
4. **Seed sample data for testing:**
|
||||
```bash
|
||||
node scripts/geo_manager.js seed-us-cities
|
||||
```
|
||||
|
||||
5. **Keep audit reports for reference:**
|
||||
- Reports saved to: `schema_audit_report.json`
|
||||
- Save with timestamps for comparison
|
||||
|
||||
---
|
||||
|
||||
**Need help?** All scripts have built-in help:
|
||||
```bash
|
||||
node scripts/[script-name].js
|
||||
# Shows available commands and examples
|
||||
```
|
||||
- **Do NOT** modify `docker-compose.yml` manually on the server. Use Coolify UI.
|
||||
- **Environment Variables**: Managed in Coolify. See `.env.example` for required keys.
|
||||
- **Onboarding**: See `SPARK_ONBOARDING.md` in the root for the full developer guide.
|
||||
|
||||
Reference in New Issue
Block a user