gatekeeper/net
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update onboarding and scripts

Browse Source
This commit is contained in:
cawcenter
2025-12-13 15:45:06 -05:00
parent 0c0cdde72c
commit f2f98e5b42
2 changed files with 85 additions and 406 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
SPARK_ONBOARDING.md Normal file
View File

@@ -0,0 +1,66 @@
# Spark Platform: Zero-to-One Onboarding Guide
> **Current State**: Phase 8 Complete (Premium Features Active)
> **Last Updated**: 2025-12-13
## 1. Project Overview
Spark is a high-performance content scaling platform ("Titanium Pro" edition). It leverages Directus (CMS/Data) and Astro/React (Frontend) to deliver enterprise-grade mass content generation.
## 2. Key Architecture
- **Frontend**: Astro (SSR) + React (Interactive Islands).
- **Backend**: Directus Headless CMS (Postgres).
- **Styling**: TailwindCSS (Titanium Pro Design System: Dark Mode, Glassmorphism, Neon Accents).
- **Deployment**: Coolify (Dockerized).
## 3. Core Modules (Implemented)
### 🏭 Factory Floor (`/admin/content/factory`)
The bread and butter of the operation.
- **Components**: `KanbanBoard`, `BulkGrid`.
- **Purpose**: Manage the lifecycle of thousands of articles.
### 🧠 Intelligence Station (`/admin/intelligence`)
Data-driven insights engine.
- **Geospatial Map (`/geo-targeting`)**: visualize market dominance with heatmaps using Leaflet.
- **Trend Analysis**: Analyze keyword volatility.
- **Avatar Metrics**: Track performance by user persona.
### 🧩 Assembler Engine (`/admin/assembler`)
The heart of content generation.
- **Template Composer**: 3-pane editor for Spintax templates.
- **Automation Builder (`/automations/workflow`)**: **[NEW]** Visual Node-based editor (React Flow) to design content pipelines (Trigger -> Generate -> Publish).
### 📊 Command Center (`/admin/analytics`)
- **Metrics Dashboard (`/metrics`)**: **[NEW]** Professional Tremor-based analytics (Area Charts, Donut Charts) tracking traffic and engagement.
### 🛡️ Testing Suite (`/admin/testing`)
- **Visual Editor (`/blocks/editor`)**: Drag-and-drop page builder (Craft.js).
- **SEO/Readability**: Real-time content scoring (Flesch-Kincaid).
## 4. Development Standards
### "Titanium Pro" Design Rules
- **Colors**: Background `#09090b` (Zinc-950), Accents `#eab308` (Yellow-500) and `#22c55e` (Green-500).
- **Components**: Always use **Shadcn/UI** (`/components/ui/*.tsx`).
- **Icons**: Lucide React (`<Edit className="h-4 w-4" />`).
- **Animations**: Use `framer-motion` for complex interactions, or CSS transitions for hover states.
### Code Quality
- **Typescript**: Strict mode. No `any` unless absolutely necessary (and temporarily).
- **Linting**: Run `npm run lint` before committing.
- **Testing**: Use the internal `/admin/testing/suite` to validate logic changes.
## 5. Deployment Workflow
1. **Commit**: `git push` triggers Coolify build.
2. **Build Process**: Dockerfile builds frontend (SSR).
3. **Validation**: Check `https://launch.jumpstartscaling.com` for the live site.
4. **Staging**: Use the "Preview" button in `CollectionManager` to vie draft content at `https://launch.jumpstartscaling.com/site/[id]/preview/[post_id]`.
## 6. Next Actions (Phase 7: Polish)
The logic is done. The next phase is strictly **Polish**:
- [ ] **Accessibility Audit**: Ensure ARIA labels on all new UI.
- [ ] **Documentation**: Expand inline JSDoc for complex engines (`engine.ts`).
- [ ] **Performance**: Optimize React Flow rendering for large graphs.
---
**Note to AI Agent**: You are working on a codebase that is feature-complete for Alpha. Your goal is stability and refinement. Do not refactor core engines without explicit instruction.

425
scripts/README.md
View File

@@ -1,417 +1,30 @@
# Spark Platform - Management Scripts ---
description: Learn how to manage deployment scripts and database migrations
This directory contains powerful Node.js utilities for managing your Spark Directus instance through the API. All scripts connect to `https://spark.jumpstartscaling.com` using admin credentials.
## 🔧 Available Scripts
### 1. Connection Test
**File:** `test_directus_connection.js` (in project root)
Tests basic connectivity and admin access to Directus.
```bash
node test_directus_connection.js
```
**What it checks:**
- Server availability
- Admin authentication
- Collections list
- Record counts
- Write permissions
--- ---
### 2. Schema Audit # Scripts & Automation
**File:** `audit_schema.js`
Comprehensive audit of all collections, fields, and relationships. This directory contains utility scripts for managing the Spark Platform.
```bash ## Core Scripts
node scripts/audit_schema.js
```
**Features:** - `deploy.sh`: (Legacy) Basic shell script for deployment. We now use **Coolify**.
- Lists all collections with record counts - `seed.ts`: Database seeder (if applicable).
- Shows all fields and their interfaces
- Identifies missing relationships
- Detects UX issues (wrong field types, missing dropdowns)
- Saves detailed report to `schema_audit_report.json`
**Output:** ## Deployment
- Field-by-field analysis
- Relationship mapping
- Issue summary
- Recommendations
--- We use **Coolify** for all deployments. The `Dockerfile` in the root determines the build process.
- **Frontend**: Built via `npm run build` (Astro).
- **Backend**: Directus via Docker Image.
### 3. UX Improvements ## Database Migrations
**File:** `improve_ux.js`
Automatically fixes field interfaces to make Directus admin UI more user-friendly. Directus handles migrations internally.
- To export schema: `npx directus schema snapshot ./schema.yaml`
- To apply schema: `npx directus schema apply ./schema.yaml`
```bash ## ⚠️ Important Notes
node scripts/improve_ux.js
```
**What it fixes:** - **Do NOT** modify `docker-compose.yml` manually on the server. Use Coolify UI.
- ✅ `site_id` fields → Select dropdown with site names - **Environment Variables**: Managed in Coolify. See `.env.example` for required keys.
- ✅ `campaign_id` fields → Select dropdown with campaign names - **Onboarding**: See `SPARK_ONBOARDING.md` in the root for the full developer guide.
- ✅ `status` fields → Dropdown with predefined choices
- ✅ `avatar_key` fields → Avatar selection dropdown
- ✅ JSON fields → Code editor with syntax highlighting
- ✅ Content fields → Rich text HTML editor
- ✅ Date fields → Proper datetime picker
- ✅ Adds helpful descriptions and placeholders
**Results:**
- Posts/Pages easily connect to Sites
- Status uses visual labels
- JSON editing with syntax highlighting
- Better field validation
---
### 4. Schema Validation
**File:** `validate_schema.js`
Complete validation of schema integrity, relationships, and data quality.
```bash
node scripts/validate_schema.js
```
**Validation checks:**
1. **Collection Data** - Verifies all critical collections exist
2. **Relationships** - Tests that foreign keys work correctly
3. **Field Interfaces** - Confirms UX improvements are applied
4. **Data Integrity** - Checks for orphaned records
**Output:**
- Detailed validation report
- Issue severity ratings (high/medium/low)
- Saves `validation_report.json`
---
### 5. Bulk Import/Export
**File:** `bulk_io.js`
Export and import any collection as JSON files.
```bash
# Export all collections
node scripts/bulk_io.js export all
# Export single collection
node scripts/bulk_io.js export sites
# Import from file
node scripts/bulk_io.js import sites sites_backup.json
# List available exports
node scripts/bulk_io.js list
```
**Features:**
- Exports with metadata (timestamp, record count)
- Batch import with conflict resolution
- Automatic pagination for large collections
- Update existing records on conflict
- All exports saved to `./exports/` directory
**Use cases:**
- Backup before major changes
- Move data between environments
- Share sample data
- Restore deleted records
---
### 6. Geo Intelligence Manager
**File:** `geo_manager.js`
Easy management of locations in `geo_intelligence` collection.
```bash
# List all locations
node scripts/geo_manager.js list
# Add a location
node scripts/geo_manager.js add "Miami" "FL" "US" "southeast"
# Remove a location
node scripts/geo_manager.js remove <id>
# Activate/deactivate location
node scripts/geo_manager.js activate <id>
node scripts/geo_manager.js deactivate <id>
# Update a field
node scripts/geo_manager.js update <id> cluster "northeast"
# Seed with top 20 US cities
node scripts/geo_manager.js seed-us-cities
# Import from CSV
node scripts/geo_manager.js import-csv locations.csv
```
**CSV Format:**
```csv
city,state,country,cluster,population
Miami,FL,US,southeast,467963
Boston,MA,US,northeast,692600
```
**Built-in sample data:**
- Top 20 US cities by population
- Includes clusters (northeast, south, midwest, west, etc.)
- Ready to use with `seed-us-cities` command
---
## 🚀 Quick Start Guide
### First Time Setup
1. **Test your connection:**
```bash
node test_directus_connection.js
```
2. **Audit current schema:**
```bash
node scripts/audit_schema.js
```
3. **Apply UX improvements:**
```bash
node scripts/improve_ux.js
```
4. **Validate everything works:**
```bash
node scripts/validate_schema.js
```
5. **Backup all data:**
```bash
node scripts/bulk_io.js export all
```
### Daily Operations
**Working with locations:**
```bash
# See all locations
node scripts/geo_manager.js list
# Add custom location
node scripts/geo_manager.js add "Portland" "OR" "US" "northwest"
```
**Backing up before changes:**
```bash
node scripts/bulk_io.js export sites
node scripts/bulk_io.js export generation_jobs
```
**Checking system health:**
```bash
node scripts/validate_schema.js
```
---
## 📊 What Each Script Fixed
### Before UX Improvements:
- ❌ `site_id` fields showed UUID text input
- ❌ `status` fields were plain text
- ❌ JSON fields used tiny text box
- ❌ Content used plain textarea
- ❌ No field descriptions or help text
### After UX Improvements:
- ✅ `site_id` shows dropdown with site names
- ✅ `status` has predefined choices with colors
- ✅ JSON fields have code editor with syntax highlighting
- ✅ Content uses rich text HTML editor
- ✅ All fields have helpful descriptions
---
## 🔗 Confirmed Working Relationships
All relationships tested and verified:
1. **Sites → Posts** ✅
- Posts connected to sites via `site_id`
- Dropdown shows site names
2. **Sites → Pages** ✅
- Pages connected to sites via `site_id`
- Easy site selection
3. **Campaign → Generated Articles** ✅
- Articles linked to campaigns
- Track which campaign created each article
4. **Generation Jobs → Sites** ✅
- Jobs know which site they're for
- Filters work correctly
---
## 📁 Export Directory Structure
After running bulk export, you'll have:
```
exports/
├── avatar_intelligence_2025-12-13.json (10 records)
├── avatar_variants_2025-12-13.json (30 records)
├── campaign_masters_2025-12-13.json (2 records)
├── cartesian_patterns_2025-12-13.json (3 records)
├── content_fragments_2025-12-13.json (150 records)
├── generated_articles_2025-12-13.json (0 records)
├── generation_jobs_2025-12-13.json (30 records)
├── geo_intelligence_2025-12-13.json (3 records)
├── sites_2025-12-13.json (3 records)
└── ... (all other collections)
```
Each JSON file includes:
```json
{
"collection": "sites",
"exportedAt": "2025-12-13T14:30:00.000Z",
"recordCount": 3,
"data": [...]
}
```
---
## 🎯 Common Tasks
### Add Multiple Locations from CSV
1. Create `locations.csv`:
```csv
city,state,country,cluster,population
Seattle,WA,US,northwest,753675
Portland,OR,US,northwest,652503
San Francisco,CA,US,west,881549
```
2. Import:
```bash
node scripts/geo_manager.js import-csv locations.csv
```
### Backup Before Major Changes
```bash
# Export everything
node scripts/bulk_io.js export all
# Make your changes in Directus UI...
# If something goes wrong, restore:
node scripts/bulk_io.js import sites exports/sites_2025-12-13.json
```
### Check What Needs Fixing
```bash
# See what's wrong
node scripts/audit_schema.js
# Auto-fix field interfaces
node scripts/improve_ux.js
# Verify fixes worked
node scripts/validate_schema.js
```
---
## 🔧 Troubleshooting
### "Authentication failed"
- Check credentials in script files
- Verify admin token hasn't expired
- Test with: `node test_directus_connection.js`
### "Collection not found"
- Collection may not exist yet
- Run audit to see all collections: `node scripts/audit_schema.js`
- Check schema is initialized
### Import conflicts (409 errors)
- Script automatically tries to UPDATE existing records
- Check the import summary for failed records
- Review data for duplicate IDs
---
## 📚 Additional Resources
- [Directus API Docs](https://docs.directus.io/reference/introduction.html)
- [Spark Onboarding Guide](../spark_onboarding.md)
- [Campaign Setup Guide](../CAMPAIGN_SETUP_GUIDE.md)
---
## ✅ Current System Status
**Last Validation:** 2025-12-13
- ✅ 11/11 critical collections exist
- ✅ 9/11 collections have data
- ✅ 4/4 relationships working
- ✅ 32 field interfaces improved
- ✅ 251 total records
- ✅ 30 pending generation jobs
- ✅ Zero data integrity issues
**Ready for production use!** 🎉
---
## 💡 Pro Tips
1. **Always backup before bulk changes:**
```bash
node scripts/bulk_io.js export all
```
2. **Use validation after making schema changes:**
```bash
node scripts/validate_schema.js
```
3. **Check exports directory regularly:**
```bash
node scripts/bulk_io.js list
```
4. **Seed sample data for testing:**
```bash
node scripts/geo_manager.js seed-us-cities
```
5. **Keep audit reports for reference:**
- Reports saved to: `schema_audit_report.json`
- Save with timestamps for comparison
---
**Need help?** All scripts have built-in help:
```bash
node scripts/[script-name].js
# Shows available commands and examples
```