docs: add comprehensive Harris matrix and schema dependency analysis

- GOD_MODE_HARRIS_MATRIX.md: Complete dependency execution plan with 3 batches
- SCHEMA_FILE_DEPENDENCY_MATRIX.md: Cross-file relationship mapping
- Identified critical issues: missing parent tables, template errors, interface issues
This commit is contained in:
cawcenter
2025-12-14 12:14:06 -05:00
parent 658d18f1c8
commit 1340a98663
2 changed files with 599 additions and 0 deletions

285
GOD_MODE_HARRIS_MATRIX.md Normal file
View File

@@ -0,0 +1,285 @@
# 🔷 GOD-MODE HARRIS MATRIX: Complete Schema Dependency Guide
> **What is a Harris Matrix?** In database design, it's a **Dependency Structure Matrix (DSM)** that shows the exact order to create tables so foreign key constraints don't fail. You cannot build a roof before walls. You cannot create `comments` before `users` and `posts` exist.
---
## 🎯 THE GOLDEN RULE
**Build the Foundation First, Then the Walls, Then the Roof**
```
┌─────────────────────────────────────┐
│ BATCH 1: Foundation (Independent) │ ← Create First
├─────────────────────────────────────┤
│ BATCH 2: Walls (First Children) │ ← Create Second
├─────────────────────────────────────┤
│ BATCH 3: Roof (Complex Children) │ ← Create Last
└─────────────────────────────────────┘
```
---
## 📊 SPARK PLATFORM: Complete Dependency Matrix
### Summary Statistics
- **Total Collections:** 16 (from schema_map.json)
- **Parent Tables (Batch 1):** 6
- **Child Tables (Batch 2):** 8
- **Complex Tables (Batch 3):** 2
- **Total Foreign Keys:** 12
- **Identified Issues:** 2 (template field mismatches)
---
## 🏗️ BATCH 1: FOUNDATION TABLES
> **Zero Dependencies** - These tables reference NO other tables
| # | Table | Type | Purpose | Children Dependent |
|---|-------|------|---------|-------------------|
| 1 | `sites` ⭐ | Parent | Master site registry | **10 tables** depend on this |
| 2 | `campaign_masters` ⭐ | Parent | Campaign definitions | **3 tables** depend on this |
| 3 | `avatar_intelligence` | Independent | Avatar personality data | 0 |
| 4 | `avatar_variants` | Independent | Avatar variations | 0 |
| 5 | `cartesian_patterns` | Independent | Pattern formulas | 0 |
| 6 | `geo_intelligence` | Independent | Geographic data | 0 |
| 7 | `offer_blocks` | Independent | Content offer blocks | 0 |
**⚠️ CRITICAL:** `sites` and `campaign_masters` are **SUPER PARENTS** - create these FIRST!
---
## 🧱 BATCH 2: FIRST-LEVEL CHILDREN
> **Depend ONLY on Batch 1**
| # | Table | Depends On | Foreign Key | Constraint Action |
|---|-------|------------|-------------|-------------------|
| 8 | `generated_articles` | sites | `site_id``sites.id` | CASCADE |
| 9 | `generation_jobs` | sites | `site_id``sites.id` | CASCADE |
| 10 | `pages` | sites | `site_id``sites.id` | CASCADE |
| 11 | `posts` | sites | `site_id``sites.id` | CASCADE |
| 12 | `leads` | sites | `site_id``sites.id` | SET NULL |
| 13 | `headline_inventory` | campaign_masters | `campaign_id``campaign_masters.id` | CASCADE |
| 14 | `content_fragments` | campaign_masters | `campaign_id``campaign_masters.id` | CASCADE |
---
## 🏠 BATCH 3: COMPLEX CHILDREN
> **Depend on Batch 2 or have multiple dependencies**
| # | Table | Depends On | Multiple FKs | Notes |
|---|-------|------------|--------------|-------|
| 15 | `link_targets` | sites | No | Internal linking system |
| 16 | (Future M2M) | Multiple | Yes | Junction tables go here |
---
## 🔍 DETAILED DEPENDENCY MAP
### Visual Cascade
```
sites ─────────┬─── generated_articles
├─── generation_jobs
├─── pages
├─── posts
├─── leads
└─── link_targets
campaign_masters ─┬─── headline_inventory
├─── content_fragments
└─── (referenced by generated_articles)
avatar_intelligence (standalone)
avatar_variants (standalone)
cartesian_patterns (standalone)
geo_intelligence (standalone)
offer_blocks (standalone)
```
---
## 🚨 DETECTED ISSUES (from schema_issues.json)
### Issue #1: Template Field Mismatch
**Collection:** `content_fragments`
**Field:** `campaign_id` (M2O relation)
**Problem:** Display template references `campaign_name` but `campaign_masters` has field `name`, not `campaign_name`
**Fix:** Update template to use `{{campaign_id.name}}` instead of `{{campaign_id.campaign_name}}`
### Issue #2: Template Field Mismatch
**Collection:** `headline_inventory`
**Field:** `campaign_id` (M2O relation)
**Problem:** Same as above - references non-existent `campaign_name`
**Fix:** Update template to use `{{campaign_id.name}}`
---
## 📐 EXECUTION PLAN: Step-by-Step
### Phase 1: Create Foundation (Batch 1)
```bash
# Order is CRITICAL - sites MUST be first
npx directus schema apply --only-collections \
sites,campaign_masters,avatar_intelligence,avatar_variants,cartesian_patterns,geo_intelligence,offer_blocks
```
### Phase 2: Create Walls (Batch 2)
```bash
npx directus schema apply --only-collections \
generated_articles,generation_jobs,pages,posts,leads,headline_inventory,content_fragments
```
### Phase 3: Create Roof (Batch 3)
```bash
npx directus schema apply --only-collections \
link_targets
```
### Phase 4: Apply Relationships
```bash
# All foreign keys are applied AFTER tables exist
npx directus schema apply --only-relations
```
---
## 🎓 THE "MEASURE TWICE, CUT ONCE" PROMPT
Use this exact prompt to have AI execute your schema correctly:
```markdown
**System Role:** You are a Senior Database Architect specializing in Directus and PostgreSQL.
**Input:** I have 16 collections in my Spark Platform schema.
**Task 1: Dependency Map (DO THIS FIRST)**
Before generating any API calls, output a Dependency Execution Plan:
1. **Identify Nodes:** List all collections
2. **Identify Edges:** List all foreign key relationships
3. **Group by Batches:**
- Batch 1: Independent tables (No foreign keys)
- Batch 2: First-level dependents (Only rely on Batch 1)
- Batch 3: Complex dependents (Rely on Batch 2 or multiple tables)
**Task 2: Directus Logic Check**
Confirm you identified:
- Standard tables vs. Singletons
- Real foreign key fields vs. M2M aliases (virtual fields)
- Display templates that might reference wrong field names
**Output Format:** Structured markdown table showing batches and dependencies.
**Once Approved:** Generate Directus API creation scripts in the correct order.
[PASTE schema_map.json HERE]
```
---
## 🔧 GOD-MODE API: Create Schema Programmatically
Using the God-Mode API to respect dependencies:
```bash
# BATCH 1: Foundation
curl https://spark.jumpstartscaling.com/god/schema/collections/create \
-H "X-God-Token: $GOD_MODE_TOKEN" \
-d '{"collection":"sites", "fields":[...]}'
curl https://spark.jumpstartscaling.com/god/schema/collections/create \
-H "X-God-Token: $GOD_MODE_TOKEN" \
-d '{"collection":"campaign_masters", "fields":[...]}'
# BATCH 2: Children (ONLY after Batch 1 completes)
curl https://spark.jumpstartscaling.com/god/schema/collections/create \
-H "X-God-Token: $GOD_MODE_TOKEN" \
-d '{"collection":"generated_articles", "fields":[...], "relations":[...]}'
# BATCH 3: Relations (ONLY after tables exist)
curl https://spark.jumpstartscaling.com/god/schema/relations/create \
-H "X-God-Token: $GOD_MODE_TOKEN" \
-d '{"collection":"generated_articles", "field":"site_id", "related_collection":"sites"}'
```
---
## ✅ VALIDATION CHECKLIST
After executing schema:
- [ ] Verify Batch 1: `SELECT * FROM sites LIMIT 1;` works
- [ ] Verify Batch 1: `SELECT * FROM campaign_masters LIMIT 1;` works
- [ ] Verify Batch 2: Foreign keys resolve (no constraint errors)
- [ ] Check schema_issues.json: Fix template field references
- [ ] Test M2O dropdowns in Directus admin UI
- [ ] Confirm all 16 collections appear in Directus
---
## 📊 COMPLETE FIELD-LEVEL ANALYSIS
### sites (SUPER PARENT)
| Field | Type | Interface | Issues |
|-------|------|-----------|--------|
| id | uuid | input (readonly) | None |
| name | string | input (required) | None |
| url | string | input | None |
| status | string | select-dropdown | None |
**Children:** 10 tables reference this
---
### campaign_masters (SUPER PARENT)
| Field | Type | Interface | Issues |
|-------|------|-----------|--------|
| id | uuid | none | None |
| site_id | uuid | select-dropdown-m2o | Missing interface ⚠️ |
| name | string | input | None |
| status | string | select-dropdown | None |
| target_word_count | integer | input | None |
**Fix Needed:** `site_id` should have `select-dropdown-m2o` interface
**Children:** 3 tables reference this
---
### generated_articles (CHILD)
**Parent:** sites
**Relationship:** `site_id``sites.id` (CASCADE)
**Issues:**
- `campaign_id` field exists but NO interface (should be `select-dropdown-m2o`)
- Template might reference wrong field names
---
## 🎯 SUCCESS CRITERIA
Your schema is correct when:
1.**SQL Execution:** No foreign key constraint errors
2.**Directus UI:** All dropdowns show related data
3.**TypeScript:** Auto-generated types match reality
4.**Frontend:** No `undefined` field errors
5.**God-Mode API:** `/god/schema/snapshot` returns valid YAML
---
## 🚀 NEXT STEPS
1. **Fix Template Issues:** Update display templates for `campaign_id` fields
2. **Add Missing Interfaces:** Apply `select-dropdown-m2o` to all foreign key fields
3. **Generate TypeScript:** Run `npm run db:sync` to get types
4. **Test Fresh Install:** Use `FORCE_FRESH_INSTALL=True` to verify clean build
5. **Deploy:** Push schema changes via God-Mode API
---
**Remember:** The Harris Matrix prevents the #1 cause of schema failures: trying to create relationships before the related tables exist.
**God-Mode Key:** `$GOD_MODE_TOKEN` (set in Coolify secrets)

View File

@@ -0,0 +1,314 @@
# 📊 COMPLETE SCHEMA DEPENDENCY MATRIX
## File Interconnections Analysis
This document shows how all schema-related files connect and depend on each other.
---
## 🗂️ FILE HIERARCHY
```
spark/
├── GOD_MODE_HARRIS_MATRIX.md ← Master guide (this doc's companion)
├── schema_map.json ← Source of truth (16 collections)
├── schema_issues.json ← Known problems to fix
├── schema_audit_report.json ← Complete field validation
├── complete_schema.sql ← SQL execution script
├── extra-schema-updates.sql ← Additional migrations
├── setup-directus-schema.sh ← Automated Directus API setup
└── setup_database.sh ← Automated PostgreSQL setup
```
---
## 🔄 FILE DEPENDENCY MATRIX
| File | Depends On | Used By | Purpose |
|------|------------|---------|---------|
| `schema_map.json` | ❌ None | ALL | **SOURCE OF TRUTH** - Defines all collections |
| `schema_issues.json` | schema_map.json | Developers | Lists template field errors |
| `schema_audit_report.json` | schema_map.json | QA/Validation | Field-level validation |
| `complete_schema.sql` | schema_map.json | PostgreSQL | Raw SQL table creation |
| `extra-schema-updates.sql` | complete_schema.sql | PostgreSQL | ALTER statements |
| `setup-directus-schema.sh` | schema_map.json | Directus API | Creates via REST API |
| `setup_database.sh` | complete_schema.sql | Deployment | Automates SSH + SQL |
| `GOD_MODE_HARRIS_MATRIX.md` | ALL above | Humans | Documentation |
---
## 📋 SCHEMA_MAP.JSON: The Source of Truth
**Total Collections:** 16
**Total Fields:** 94
**Total Relationships:** 12
### Collection Breakdown
| Collection | Field Count | Has Relations | Batch |
|------------|-------------|---------------|-------|
| sites | 5 | ✅ Yes (10 children) | 1 |
| campaign_masters | 12 | ✅ Yes (3 children) | 1 |
| avatar_intelligence | 6 | ❌ No | 1 |
| avatar_variants | 4 | ❌ No | 1 |
| cartesian_patterns | 4 | ❌ No | 1 |
| geo_intelligence | 3 | ❌ No | 1 |
| offer_blocks | 4 | ❌ No | 1 |
| generated_articles | 15 | ✅ Yes (parent: sites) | 2 |
| generation_jobs | 7 | ✅ Yes (parent: sites) | 2 |
| pages | 8 | ✅ Yes (parent: sites) | 2 |
| posts | 10 | ✅ Yes (parent: sites) | 2 |
| leads | 6 | ✅ Yes (parent: sites) | 2 |
| headline_inventory | 6 | ✅ Yes (parent: campaign_masters) | 2 |
| content_fragments | 6 | ✅ Yes (parent: campaign_masters) | 2 |
| link_targets | 9 | ❌ No (soft reference) | 3 |
---
## 🚨 SCHEMA_ISSUES.JSON: Known Problems
### Issue Summary
- **Total Issues:** 2
- **Type:** Display template field mismatches
- **Severity:** Medium (won't break DB, will break UI)
### Issue Details
#### Issue #1
```json
{
"collection": "content_fragments",
"field": "campaign_id",
"targetCollection": "campaign_masters",
"templateField": "campaign_name",
"issue": "Template references non-existent field \"campaign_name\""
}
```
**Root Cause:** `campaign_masters` has field `name`, not `campaign_name`
**Fix:** Update Directus field meta to use `{{campaign_id.name}}`
#### Issue #2
```json
{
"collection": "headline_inventory",
"field": "campaign_id",
"targetCollection": "campaign_masters",
"templateField": "campaign_name",
"issue": "Template references non-existent field \"campaign_name\""
}
```
**Same issue** as above.
---
## 🔍 SCHEMA_AUDIT_REPORT.JSON: Field Validation
**Total Items:** 1,624 lines
**Total Collections:** 16
**Total Fields Audited:** 94
### Field Issues Found
| Collection | Field | Issue | Severity |
|------------|-------|-------|----------|
| campaign_masters | site_id | ID field without relational interface | Medium |
| campaign_masters | status | Status field should use select-dropdown | Low |
| content_fragments | campaign_id | ID field without relational interface | Medium |
| generated_articles | campaign_id | ID field without relational interface | Medium |
| generation_jobs | status | Status field should use select-dropdown | Low |
| headline_inventory | campaign_id | ID field without relational interface | Medium |
| headline_inventory | status | Status field should use select-dropdown | Low |
### Pattern Detected
**Problem:** Foreign key fields (UUID type) missing proper Directus interface
**Impact:** Admin UI won't show dropdown for related records
**Fix:** Add `"interface": "select-dropdown-m2o"` to field meta
---
## 🗃️ COMPLETE_SCHEMA.SQL: PostgreSQL Implementation
**Total Tables:** 28
**Total Indexes:** 10
**Lines of Code:** 376
### Table Creation Order (as written in file)
```sql
-- BATCH 1: Should be first but ISN'T in file
CREATE TABLE headline_inventory ... -- ❌ WRONG! Depends on campaign_masters
CREATE TABLE content_fragments ... -- ❌ WRONG! Depends on campaign_masters
CREATE TABLE production_queue ... -- ❌ WRONG! Depends on sites
-- MISSING FROM FILE:
-- sites
-- campaign_masters
-- These are REFERENCED but never CREATED!
```
**🚨 CRITICAL FLAW:** This file references `sites` and `campaign_masters` but never creates them!
### Fix Priority
1. Add `CREATE TABLE sites` at the TOP
2. Add `CREATE TABLE campaign_masters` after sites
3. Reorder remaining tables by dependency batch
---
## ⚙️ EXTRA-SCHEMA-UPDATES.SQL: Migrations
**Purpose:** ALTER statements for existing schema
**Total Operations:** 6
### Operations
1. Add `schema_json` to posts
2. Add `schema_json` to pages
3. Add `schema_json` to generated_articles
4. Add `target_word_count` to campaign_masters
5. Create `link_targets` table
6. Update Directus field interfaces via `directus_fields` table
**Workflow:** Run AFTER `complete_schema.sql`
---
## 🔧 SETUP-DIRECTUS-SCHEMA.SH: API Creation Script
**Method:** Directus REST API
**Total Collections Created:** 5 (partial, not all 16)
**Authentication:** Static token (hardcoded)
### Collections Created
1. avatar_intelligence
2. sites
3. posts
4. pages
5. leads
**Missing:** 11 collections not in this script!
**Recommendation:** Use God-Mode API `/god/schema/snapshot` and `/god/schema/apply` instead
---
## 🚀 SETUP_DATABASE.SH: Automated Deployment
**Method:** SSH + PostgreSQL
**Dependencies:** complete_schema.sql
**Target:** Remote server at 72.61.15.216
### Execution Flow
```
1. SCP complete_schema.sql to server
2. Find PostgreSQL container
3. Copy SQL into container
4. Execute: psql -U postgres -d directus -f /tmp/complete_schema.sql
5. Verify table count
6. Restart Directus container
7. Test API authentication
```
**Current Status:** ✅ Script is correct, but `complete_schema.sql` has missing tables
---
## 🔗 CROSS-FILE RELATIONSHIP MAP
```mermaid
graph TD
A[schema_map.json<br/>SOURCE OF TRUTH] --> B[schema_audit_report.json]
A --> C[schema_issues.json]
A --> D[complete_schema.sql]
A --> E[setup-directus-schema.sh]
D --> F[extra-schema-updates.sql]
D --> G[setup_database.sh]
B --> H[Developer Fixes]
C --> H
E --> I[Directus API]
G --> J[PostgreSQL DB]
style A fill:#4ade80
style D fill:#fbbf24
style G fill:#60a5fa
```
---
## ✅ FIX ACTION PLAN
### Phase 1: Fix complete_schema.sql
```sql
-- Add at TOP of file (line 1)
CREATE TABLE sites (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
url VARCHAR(500),
status VARCHAR(50) DEFAULT 'active',
date_created TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE campaign_masters (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
site_id UUID REFERENCES sites(id) ON DELETE CASCADE,
name VARCHAR(255),
headline_spintax_root TEXT,
status VARCHAR(50) DEFAULT 'active',
target_word_count INTEGER DEFAULT 1500,
date_created TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
### Phase 2: Fix Directus Field Interfaces
```sql
-- Run these UPDATEs in extra-schema-updates.sql
UPDATE directus_fields SET interface = 'select-dropdown-m2o'
WHERE field IN ('site_id', 'campaign_id')
AND type = 'uuid';
```
### Phase 3: Fix Template Issues
```javascript
// In Directus UI, update display templates
// FROM: {{campaign_id.campaign_name}}
// TO: {{campaign_id.name}}
```
### Phase 4: Validate
```bash
# Run complete deployment
./setup_database.sh
# Verify via God-Mode API
curl https://spark.jumpstartscaling.com/god/status \
-H "X-God-Token: $GOD_MODE_TOKEN"
```
---
## 📊 FINAL FILE DEPENDENCY SUMMARY
**schema_map.json** → Defines the truth
**schema_issues.json** → Identifies problems
**schema_audit_report.json** → Validates fields
**complete_schema.sql** → Executes in PostgreSQL (needs fixes)
**extra-schema-updates.sql** → Runs migrations
**setup-directus-schema.sh** → Creates via API (incomplete)
**setup_database.sh** → Automates deployment
**GOD_MODE_HARRIS_MATRIX.md** → Human guide
**All files interconnected. Fix one, test all.**