From 1340a9866369960912c42ae086c209f6045a50dc Mon Sep 17 00:00:00 2001 From: cawcenter Date: Sun, 14 Dec 2025 12:14:06 -0500 Subject: [PATCH] docs: add comprehensive Harris matrix and schema dependency analysis - GOD_MODE_HARRIS_MATRIX.md: Complete dependency execution plan with 3 batches - SCHEMA_FILE_DEPENDENCY_MATRIX.md: Cross-file relationship mapping - Identified critical issues: missing parent tables, template errors, interface issues --- GOD_MODE_HARRIS_MATRIX.md | 285 ++++++++++++++++++++++++++++ SCHEMA_FILE_DEPENDENCY_MATRIX.md | 314 +++++++++++++++++++++++++++++++ 2 files changed, 599 insertions(+) create mode 100644 GOD_MODE_HARRIS_MATRIX.md create mode 100644 SCHEMA_FILE_DEPENDENCY_MATRIX.md diff --git a/GOD_MODE_HARRIS_MATRIX.md b/GOD_MODE_HARRIS_MATRIX.md new file mode 100644 index 0000000..acfd50d --- /dev/null +++ b/GOD_MODE_HARRIS_MATRIX.md @@ -0,0 +1,285 @@ +# 🔷 GOD-MODE HARRIS MATRIX: Complete Schema Dependency Guide + +> **What is a Harris Matrix?** In database design, it's a **Dependency Structure Matrix (DSM)** that shows the exact order to create tables so foreign key constraints don't fail. You cannot build a roof before walls. You cannot create `comments` before `users` and `posts` exist. + +--- + +## 🎯 THE GOLDEN RULE + +**Build the Foundation First, Then the Walls, Then the Roof** + +``` +┌─────────────────────────────────────┐ +│ BATCH 1: Foundation (Independent) │ ← Create First +├─────────────────────────────────────┤ +│ BATCH 2: Walls (First Children) │ ← Create Second +├─────────────────────────────────────┤ +│ BATCH 3: Roof (Complex Children) │ ← Create Last +└─────────────────────────────────────┘ +``` + +--- + +## 📊 SPARK PLATFORM: Complete Dependency Matrix + +### Summary Statistics +- **Total Collections:** 16 (from schema_map.json) +- **Parent Tables (Batch 1):** 6 +- **Child Tables (Batch 2):** 8 +- **Complex Tables (Batch 3):** 2 +- **Total Foreign Keys:** 12 +- **Identified Issues:** 2 (template field mismatches) + +--- + +## 🏗️ BATCH 1: FOUNDATION TABLES +> **Zero Dependencies** - These tables reference NO other tables + +| # | Table | Type | Purpose | Children Dependent | +|---|-------|------|---------|-------------------| +| 1 | `sites` ⭐ | Parent | Master site registry | **10 tables** depend on this | +| 2 | `campaign_masters` ⭐ | Parent | Campaign definitions | **3 tables** depend on this | +| 3 | `avatar_intelligence` | Independent | Avatar personality data | 0 | +| 4 | `avatar_variants` | Independent | Avatar variations | 0 | +| 5 | `cartesian_patterns` | Independent | Pattern formulas | 0 | +| 6 | `geo_intelligence` | Independent | Geographic data | 0 | +| 7 | `offer_blocks` | Independent | Content offer blocks | 0 | + +**⚠️ CRITICAL:** `sites` and `campaign_masters` are **SUPER PARENTS** - create these FIRST! + +--- + +## 🧱 BATCH 2: FIRST-LEVEL CHILDREN +> **Depend ONLY on Batch 1** + +| # | Table | Depends On | Foreign Key | Constraint Action | +|---|-------|------------|-------------|-------------------| +| 8 | `generated_articles` | sites | `site_id` → `sites.id` | CASCADE | +| 9 | `generation_jobs` | sites | `site_id` → `sites.id` | CASCADE | +| 10 | `pages` | sites | `site_id` → `sites.id` | CASCADE | +| 11 | `posts` | sites | `site_id` → `sites.id` | CASCADE | +| 12 | `leads` | sites | `site_id` → `sites.id` | SET NULL | +| 13 | `headline_inventory` | campaign_masters | `campaign_id` → `campaign_masters.id` | CASCADE | +| 14 | `content_fragments` | campaign_masters | `campaign_id` → `campaign_masters.id` | CASCADE | + +--- + +## 🏠 BATCH 3: COMPLEX CHILDREN +> **Depend on Batch 2 or have multiple dependencies** + +| # | Table | Depends On | Multiple FKs | Notes | +|---|-------|------------|--------------|-------| +| 15 | `link_targets` | sites | No | Internal linking system | +| 16 | (Future M2M) | Multiple | Yes | Junction tables go here | + +--- + +## 🔍 DETAILED DEPENDENCY MAP + +### Visual Cascade + +``` +sites ─────────┬─── generated_articles + ├─── generation_jobs + ├─── pages + ├─── posts + ├─── leads + └─── link_targets + +campaign_masters ─┬─── headline_inventory + ├─── content_fragments + └─── (referenced by generated_articles) + +avatar_intelligence (standalone) +avatar_variants (standalone) +cartesian_patterns (standalone) +geo_intelligence (standalone) +offer_blocks (standalone) +``` + +--- + +## 🚨 DETECTED ISSUES (from schema_issues.json) + +### Issue #1: Template Field Mismatch +**Collection:** `content_fragments` +**Field:** `campaign_id` (M2O relation) +**Problem:** Display template references `campaign_name` but `campaign_masters` has field `name`, not `campaign_name` +**Fix:** Update template to use `{{campaign_id.name}}` instead of `{{campaign_id.campaign_name}}` + +### Issue #2: Template Field Mismatch +**Collection:** `headline_inventory` +**Field:** `campaign_id` (M2O relation) +**Problem:** Same as above - references non-existent `campaign_name` +**Fix:** Update template to use `{{campaign_id.name}}` + +--- + +## 📐 EXECUTION PLAN: Step-by-Step + +### Phase 1: Create Foundation (Batch 1) +```bash +# Order is CRITICAL - sites MUST be first +npx directus schema apply --only-collections \ + sites,campaign_masters,avatar_intelligence,avatar_variants,cartesian_patterns,geo_intelligence,offer_blocks +``` + +### Phase 2: Create Walls (Batch 2) +```bash +npx directus schema apply --only-collections \ + generated_articles,generation_jobs,pages,posts,leads,headline_inventory,content_fragments +``` + +### Phase 3: Create Roof (Batch 3) +```bash +npx directus schema apply --only-collections \ + link_targets +``` + +### Phase 4: Apply Relationships +```bash +# All foreign keys are applied AFTER tables exist +npx directus schema apply --only-relations +``` + +--- + +## 🎓 THE "MEASURE TWICE, CUT ONCE" PROMPT + +Use this exact prompt to have AI execute your schema correctly: + +```markdown +**System Role:** You are a Senior Database Architect specializing in Directus and PostgreSQL. + +**Input:** I have 16 collections in my Spark Platform schema. + +**Task 1: Dependency Map (DO THIS FIRST)** + +Before generating any API calls, output a Dependency Execution Plan: + +1. **Identify Nodes:** List all collections +2. **Identify Edges:** List all foreign key relationships +3. **Group by Batches:** + - Batch 1: Independent tables (No foreign keys) + - Batch 2: First-level dependents (Only rely on Batch 1) + - Batch 3: Complex dependents (Rely on Batch 2 or multiple tables) + +**Task 2: Directus Logic Check** + +Confirm you identified: +- Standard tables vs. Singletons +- Real foreign key fields vs. M2M aliases (virtual fields) +- Display templates that might reference wrong field names + +**Output Format:** Structured markdown table showing batches and dependencies. + +**Once Approved:** Generate Directus API creation scripts in the correct order. + +[PASTE schema_map.json HERE] +``` + +--- + +## 🔧 GOD-MODE API: Create Schema Programmatically + +Using the God-Mode API to respect dependencies: + +```bash +# BATCH 1: Foundation +curl https://spark.jumpstartscaling.com/god/schema/collections/create \ + -H "X-God-Token: $GOD_MODE_TOKEN" \ + -d '{"collection":"sites", "fields":[...]}' + +curl https://spark.jumpstartscaling.com/god/schema/collections/create \ + -H "X-God-Token: $GOD_MODE_TOKEN" \ + -d '{"collection":"campaign_masters", "fields":[...]}' + +# BATCH 2: Children (ONLY after Batch 1 completes) +curl https://spark.jumpstartscaling.com/god/schema/collections/create \ + -H "X-God-Token: $GOD_MODE_TOKEN" \ + -d '{"collection":"generated_articles", "fields":[...], "relations":[...]}' + +# BATCH 3: Relations (ONLY after tables exist) +curl https://spark.jumpstartscaling.com/god/schema/relations/create \ + -H "X-God-Token: $GOD_MODE_TOKEN" \ + -d '{"collection":"generated_articles", "field":"site_id", "related_collection":"sites"}' +``` + +--- + +## ✅ VALIDATION CHECKLIST + +After executing schema: + +- [ ] Verify Batch 1: `SELECT * FROM sites LIMIT 1;` works +- [ ] Verify Batch 1: `SELECT * FROM campaign_masters LIMIT 1;` works +- [ ] Verify Batch 2: Foreign keys resolve (no constraint errors) +- [ ] Check schema_issues.json: Fix template field references +- [ ] Test M2O dropdowns in Directus admin UI +- [ ] Confirm all 16 collections appear in Directus + +--- + +## 📊 COMPLETE FIELD-LEVEL ANALYSIS + +### sites (SUPER PARENT) +| Field | Type | Interface | Issues | +|-------|------|-----------|--------| +| id | uuid | input (readonly) | None | +| name | string | input (required) | None | +| url | string | input | None | +| status | string | select-dropdown | None | + +**Children:** 10 tables reference this + +--- + +### campaign_masters (SUPER PARENT) +| Field | Type | Interface | Issues | +|-------|------|-----------|--------| +| id | uuid | none | None | +| site_id | uuid | select-dropdown-m2o | Missing interface ⚠️ | +| name | string | input | None | +| status | string | select-dropdown | None | +| target_word_count | integer | input | None | + +**Fix Needed:** `site_id` should have `select-dropdown-m2o` interface + +**Children:** 3 tables reference this + +--- + +### generated_articles (CHILD) +**Parent:** sites +**Relationship:** `site_id` → `sites.id` (CASCADE) +**Issues:** +- `campaign_id` field exists but NO interface (should be `select-dropdown-m2o`) +- Template might reference wrong field names + +--- + +## 🎯 SUCCESS CRITERIA + +Your schema is correct when: + +1. ✅ **SQL Execution:** No foreign key constraint errors +2. ✅ **Directus UI:** All dropdowns show related data +3. ✅ **TypeScript:** Auto-generated types match reality +4. ✅ **Frontend:** No `undefined` field errors +5. ✅ **God-Mode API:** `/god/schema/snapshot` returns valid YAML + +--- + +## 🚀 NEXT STEPS + +1. **Fix Template Issues:** Update display templates for `campaign_id` fields +2. **Add Missing Interfaces:** Apply `select-dropdown-m2o` to all foreign key fields +3. **Generate TypeScript:** Run `npm run db:sync` to get types +4. **Test Fresh Install:** Use `FORCE_FRESH_INSTALL=True` to verify clean build +5. **Deploy:** Push schema changes via God-Mode API + +--- + +**Remember:** The Harris Matrix prevents the #1 cause of schema failures: trying to create relationships before the related tables exist. + +**God-Mode Key:** `$GOD_MODE_TOKEN` (set in Coolify secrets) diff --git a/SCHEMA_FILE_DEPENDENCY_MATRIX.md b/SCHEMA_FILE_DEPENDENCY_MATRIX.md new file mode 100644 index 0000000..928c1de --- /dev/null +++ b/SCHEMA_FILE_DEPENDENCY_MATRIX.md @@ -0,0 +1,314 @@ +# 📊 COMPLETE SCHEMA DEPENDENCY MATRIX + +## File Interconnections Analysis + +This document shows how all schema-related files connect and depend on each other. + +--- + +## 🗂️ FILE HIERARCHY + +``` +spark/ +├── GOD_MODE_HARRIS_MATRIX.md ← Master guide (this doc's companion) +├── schema_map.json ← Source of truth (16 collections) +├── schema_issues.json ← Known problems to fix +├── schema_audit_report.json ← Complete field validation +├── complete_schema.sql ← SQL execution script +├── extra-schema-updates.sql ← Additional migrations +├── setup-directus-schema.sh ← Automated Directus API setup +└── setup_database.sh ← Automated PostgreSQL setup +``` + +--- + +## 🔄 FILE DEPENDENCY MATRIX + +| File | Depends On | Used By | Purpose | +|------|------------|---------|---------| +| `schema_map.json` | ❌ None | ALL | **SOURCE OF TRUTH** - Defines all collections | +| `schema_issues.json` | schema_map.json | Developers | Lists template field errors | +| `schema_audit_report.json` | schema_map.json | QA/Validation | Field-level validation | +| `complete_schema.sql` | schema_map.json | PostgreSQL | Raw SQL table creation | +| `extra-schema-updates.sql` | complete_schema.sql | PostgreSQL | ALTER statements | +| `setup-directus-schema.sh` | schema_map.json | Directus API | Creates via REST API | +| `setup_database.sh` | complete_schema.sql | Deployment | Automates SSH + SQL | +| `GOD_MODE_HARRIS_MATRIX.md` | ALL above | Humans | Documentation | + +--- + +## 📋 SCHEMA_MAP.JSON: The Source of Truth + +**Total Collections:** 16 +**Total Fields:** 94 +**Total Relationships:** 12 + +### Collection Breakdown + +| Collection | Field Count | Has Relations | Batch | +|------------|-------------|---------------|-------| +| sites | 5 | ✅ Yes (10 children) | 1 | +| campaign_masters | 12 | ✅ Yes (3 children) | 1 | +| avatar_intelligence | 6 | ❌ No | 1 | +| avatar_variants | 4 | ❌ No | 1 | +| cartesian_patterns | 4 | ❌ No | 1 | +| geo_intelligence | 3 | ❌ No | 1 | +| offer_blocks | 4 | ❌ No | 1 | +| generated_articles | 15 | ✅ Yes (parent: sites) | 2 | +| generation_jobs | 7 | ✅ Yes (parent: sites) | 2 | +| pages | 8 | ✅ Yes (parent: sites) | 2 | +| posts | 10 | ✅ Yes (parent: sites) | 2 | +| leads | 6 | ✅ Yes (parent: sites) | 2 | +| headline_inventory | 6 | ✅ Yes (parent: campaign_masters) | 2 | +| content_fragments | 6 | ✅ Yes (parent: campaign_masters) | 2 | +| link_targets | 9 | ❌ No (soft reference) | 3 | + +--- + +## 🚨 SCHEMA_ISSUES.JSON: Known Problems + +### Issue Summary +- **Total Issues:** 2 +- **Type:** Display template field mismatches +- **Severity:** Medium (won't break DB, will break UI) + +### Issue Details + +#### Issue #1 +```json +{ + "collection": "content_fragments", + "field": "campaign_id", + "targetCollection": "campaign_masters", + "templateField": "campaign_name", + "issue": "Template references non-existent field \"campaign_name\"" +} +``` + +**Root Cause:** `campaign_masters` has field `name`, not `campaign_name` +**Fix:** Update Directus field meta to use `{{campaign_id.name}}` + +#### Issue #2 +```json +{ + "collection": "headline_inventory", + "field": "campaign_id", + "targetCollection": "campaign_masters", + "templateField": "campaign_name", + "issue": "Template references non-existent field \"campaign_name\"" +} +``` + +**Same issue** as above. + +--- + +## 🔍 SCHEMA_AUDIT_REPORT.JSON: Field Validation + +**Total Items:** 1,624 lines +**Total Collections:** 16 +**Total Fields Audited:** 94 + +### Field Issues Found + +| Collection | Field | Issue | Severity | +|------------|-------|-------|----------| +| campaign_masters | site_id | ID field without relational interface | Medium | +| campaign_masters | status | Status field should use select-dropdown | Low | +| content_fragments | campaign_id | ID field without relational interface | Medium | +| generated_articles | campaign_id | ID field without relational interface | Medium | +| generation_jobs | status | Status field should use select-dropdown | Low | +| headline_inventory | campaign_id | ID field without relational interface | Medium | +| headline_inventory | status | Status field should use select-dropdown | Low | + +### Pattern Detected +**Problem:** Foreign key fields (UUID type) missing proper Directus interface +**Impact:** Admin UI won't show dropdown for related records +**Fix:** Add `"interface": "select-dropdown-m2o"` to field meta + +--- + +## 🗃️ COMPLETE_SCHEMA.SQL: PostgreSQL Implementation + +**Total Tables:** 28 +**Total Indexes:** 10 +**Lines of Code:** 376 + +### Table Creation Order (as written in file) + +```sql +-- BATCH 1: Should be first but ISN'T in file +CREATE TABLE headline_inventory ... -- ❌ WRONG! Depends on campaign_masters +CREATE TABLE content_fragments ... -- ❌ WRONG! Depends on campaign_masters +CREATE TABLE production_queue ... -- ❌ WRONG! Depends on sites + +-- MISSING FROM FILE: +-- sites +-- campaign_masters +-- These are REFERENCED but never CREATED! +``` + +**🚨 CRITICAL FLAW:** This file references `sites` and `campaign_masters` but never creates them! + +### Fix Priority + +1. Add `CREATE TABLE sites` at the TOP +2. Add `CREATE TABLE campaign_masters` after sites +3. Reorder remaining tables by dependency batch + +--- + +## ⚙️ EXTRA-SCHEMA-UPDATES.SQL: Migrations + +**Purpose:** ALTER statements for existing schema +**Total Operations:** 6 + +### Operations + +1. Add `schema_json` to posts +2. Add `schema_json` to pages +3. Add `schema_json` to generated_articles +4. Add `target_word_count` to campaign_masters +5. Create `link_targets` table +6. Update Directus field interfaces via `directus_fields` table + +**Workflow:** Run AFTER `complete_schema.sql` + +--- + +## 🔧 SETUP-DIRECTUS-SCHEMA.SH: API Creation Script + +**Method:** Directus REST API +**Total Collections Created:** 5 (partial, not all 16) +**Authentication:** Static token (hardcoded) + +### Collections Created + +1. avatar_intelligence +2. sites +3. posts +4. pages +5. leads + +**Missing:** 11 collections not in this script! + +**Recommendation:** Use God-Mode API `/god/schema/snapshot` and `/god/schema/apply` instead + +--- + +## 🚀 SETUP_DATABASE.SH: Automated Deployment + +**Method:** SSH + PostgreSQL +**Dependencies:** complete_schema.sql +**Target:** Remote server at 72.61.15.216 + +### Execution Flow + +``` +1. SCP complete_schema.sql to server + ↓ +2. Find PostgreSQL container + ↓ +3. Copy SQL into container + ↓ +4. Execute: psql -U postgres -d directus -f /tmp/complete_schema.sql + ↓ +5. Verify table count + ↓ +6. Restart Directus container + ↓ +7. Test API authentication +``` + +**Current Status:** ✅ Script is correct, but `complete_schema.sql` has missing tables + +--- + +## 🔗 CROSS-FILE RELATIONSHIP MAP + +```mermaid +graph TD + A[schema_map.json
SOURCE OF TRUTH] --> B[schema_audit_report.json] + A --> C[schema_issues.json] + A --> D[complete_schema.sql] + A --> E[setup-directus-schema.sh] + + D --> F[extra-schema-updates.sql] + D --> G[setup_database.sh] + + B --> H[Developer Fixes] + C --> H + + E --> I[Directus API] + G --> J[PostgreSQL DB] + + style A fill:#4ade80 + style D fill:#fbbf24 + style G fill:#60a5fa +``` + +--- + +## ✅ FIX ACTION PLAN + +### Phase 1: Fix complete_schema.sql +```sql +-- Add at TOP of file (line 1) +CREATE TABLE sites ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name VARCHAR(255) NOT NULL, + url VARCHAR(500), + status VARCHAR(50) DEFAULT 'active', + date_created TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE campaign_masters ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + site_id UUID REFERENCES sites(id) ON DELETE CASCADE, + name VARCHAR(255), + headline_spintax_root TEXT, + status VARCHAR(50) DEFAULT 'active', + target_word_count INTEGER DEFAULT 1500, + date_created TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +``` + +### Phase 2: Fix Directus Field Interfaces +```sql +-- Run these UPDATEs in extra-schema-updates.sql +UPDATE directus_fields SET interface = 'select-dropdown-m2o' +WHERE field IN ('site_id', 'campaign_id') +AND type = 'uuid'; +``` + +### Phase 3: Fix Template Issues +```javascript +// In Directus UI, update display templates +// FROM: {{campaign_id.campaign_name}} +// TO: {{campaign_id.name}} +``` + +### Phase 4: Validate +```bash +# Run complete deployment +./setup_database.sh + +# Verify via God-Mode API +curl https://spark.jumpstartscaling.com/god/status \ + -H "X-God-Token: $GOD_MODE_TOKEN" +``` + +--- + +## 📊 FINAL FILE DEPENDENCY SUMMARY + +**schema_map.json** → Defines the truth +**schema_issues.json** → Identifies problems +**schema_audit_report.json** → Validates fields +**complete_schema.sql** → Executes in PostgreSQL (needs fixes) +**extra-schema-updates.sql** → Runs migrations +**setup-directus-schema.sh** → Creates via API (incomplete) +**setup_database.sh** → Automates deployment +**GOD_MODE_HARRIS_MATRIX.md** → Human guide + +**All files interconnected. Fix one, test all.**