544 lines
19 KiB
Markdown
544 lines
19 KiB
Markdown
# 🔱 God Mode: Direct PostgreSQL Shim Architecture
|
|
## Implementation Plan & Task List
|
|
|
|
**Objective:** Build a frontend that connects directly to PostgreSQL without going through Directus or traditional REST APIs, using server-side rendering (SSR) and a secure "shim" layer.
|
|
|
|
**Architecture:** Astro SSR + Direct PostgreSQL Pool + Secure API Routes
|
|
|
|
---
|
|
|
|
## 📋 ARCHITECTURE OVERVIEW
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ BROWSER (Client-Side JavaScript) │
|
|
│ - React Components │
|
|
│ - TanStack Query for state management │
|
|
│ - NO database credentials, NO SQL │
|
|
└─────────────────┬───────────────────────────────────────┘
|
|
│
|
|
│ HTTP Requests
|
|
│
|
|
┌─────────────────▼───────────────────────────────────────┐
|
|
│ ASTRO SSR LAYER (Server-Side) │
|
|
│ ┌─────────────────────────────────────────────────┐ │
|
|
│ │ 1. Server-Side Pages (---) │ │
|
|
│ │ - Direct SQL queries in Astro frontmatter │ │
|
|
│ │ - Pool.query() before HTML render │ │
|
|
│ └─────────────────────────────────────────────────┘ │
|
|
│ ┌─────────────────────────────────────────────────┐ │
|
|
│ │ 2. API Routes (/api/shim/*) │ │
|
|
│ │ - Secure endpoints for client-side calls │ │
|
|
│ │ - Token validation │ │
|
|
│ │ - SQL query execution │ │
|
|
│ └─────────────────────────────────────────────────┘ │
|
|
│ ┌─────────────────────────────────────────────────┐ │
|
|
│ │ 3. Shim Layer (src/lib/shim/) │ │
|
|
│ │ - Query builders │ │
|
|
│ │ - Type-safe SQL functions │ │
|
|
│ │ - Connection pool management │ │
|
|
│ └─────────────────────────────────────────────────┘ │
|
|
└─────────────────┬───────────────────────────────────────┘
|
|
│
|
|
│ SQL Queries (via pg Pool)
|
|
│
|
|
┌─────────────────▼───────────────────────────────────────┐
|
|
│ POSTGRESQL DATABASE │
|
|
│ - sites, campaigns, generated_articles, work_log, etc. │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 📁 FILES TO CREATE (New Shim Layer)
|
|
|
|
### 1. Core Shim Infrastructure
|
|
|
|
| File Path | Purpose | Priority |
|
|
|-----------|---------|----------|
|
|
| `src/lib/shim/queries.ts` | Type-safe SQL query builders for all tables | ⭐ HIGH |
|
|
| `src/lib/shim/types.ts` | TypeScript interfaces matching DB schema | ⭐ HIGH |
|
|
| `src/lib/shim/sites.ts` | Sites-specific queries (CRUD operations) | ⭐ HIGH |
|
|
| `src/lib/shim/articles.ts` | Articles-specific queries | ⭐ HIGH |
|
|
| `src/lib/shim/campaigns.ts` | Campaign-specific queries | MEDIUM |
|
|
| `src/lib/shim/jobs.ts` | Generation jobs queries | MEDIUM |
|
|
| `src/lib/shim/utils.ts` | Helper functions (pagination, filtering) | MEDIUM |
|
|
|
|
### 2. API Routes (Secure Bridges)
|
|
|
|
| File Path | Purpose | Priority |
|
|
|-----------|---------|----------|
|
|
| `src/pages/api/shim/sites/[action].ts` | Sites CRUD endpoints | ⭐ HIGH |
|
|
| `src/pages/api/shim/articles/[action].ts` | Articles CRUD endpoints | ⭐ HIGH |
|
|
| `src/pages/api/shim/campaigns/[action].ts` | Campaigns CRUD endpoints | MEDIUM |
|
|
| `src/pages/api/shim/jobs/[action].ts` | Jobs CRUD endpoints | MEDIUM |
|
|
| `src/pages/api/shim/health.ts` | Database health check | LOW |
|
|
|
|
### 3. Example Components
|
|
|
|
| File Path | Purpose | Priority |
|
|
|-----------|---------|----------|
|
|
| `src/components/shim/SitesList.tsx` | React component using shim API | ⭐ HIGH |
|
|
| `src/components/shim/ArticleEditor.tsx` | Article CRUD using shim | ⭐ HIGH |
|
|
| `src/pages/shim/sites.astro` | Demo page: SSR sites list | ⭐ HIGH |
|
|
| `src/pages/shim/articles.astro` | Demo page: SSR articles list | MEDIUM |
|
|
|
|
---
|
|
|
|
## 📂 FILES TO USE (Existing Infrastructure)
|
|
|
|
### Already Working - Use As-Is
|
|
|
|
| File Path | What It Provides | How We'll Use It |
|
|
|-----------|------------------|------------------|
|
|
| `src/lib/db.ts` | PostgreSQL pool connection | Import `pool` for all queries |
|
|
| `src/lib/schemas.ts` | TypeScript type definitions | Reference for table structures |
|
|
| `astro.config.mjs` | Astro SSR config + Node adapter | Already configured for SSR |
|
|
| `package.json` | Dependencies (`pg`, `@types/pg`) | Already has what we need |
|
|
| `.env` | `DATABASE_URL` environment variable | Connection string source |
|
|
|
|
### Existing Patterns to Mimic
|
|
|
|
| File Path | Pattern to Copy | Why |
|
|
|-----------|-----------------|-----|
|
|
| `src/pages/api/god/[...action].ts` | Token validation, error handling | Security model |
|
|
| `src/lib/directus/client.ts` | Query abstraction pattern | Structure for shim layer |
|
|
| `src/components/admin/sites/SitesManager.tsx` | TanStack Query usage | Client-side data fetching |
|
|
|
|
---
|
|
|
|
## 📂 FILES TO MODIFY (Minimal Changes)
|
|
|
|
| File Path | Modification | Reason |
|
|
|-----------|--------------|--------|
|
|
| `src/lib/db.ts` | Add query helper functions | Convenience wrappers |
|
|
| `.env` | Verify `DATABASE_URL` is set | Required for local dev |
|
|
| `tsconfig.json` | Add shim types path alias | Easier imports |
|
|
|
|
---
|
|
|
|
## ✅ TASK LIST (Step-by-Step Implementation)
|
|
|
|
### PHASE 1: Foundation (30 min)
|
|
|
|
- [ ] **Task 1.1:** Create `src/lib/shim/types.ts`
|
|
- Define interfaces: `Site`, `Article`, `Campaign`, `Job`
|
|
- Match PostgreSQL schema from migrations
|
|
- Export as named types
|
|
|
|
- [ ] **Task 1.2:** Create `src/lib/shim/utils.ts`
|
|
- `buildWhere()` - Convert filters to SQL WHERE clauses
|
|
- `buildPagination()` - LIMIT/OFFSET helpers
|
|
- `sanitizeInput()` - Basic SQL injection prevention
|
|
|
|
- [ ] **Task 1.3:** Enhance `src/lib/db.ts`
|
|
- Add `executeQuery<T>(sql: string, params: any[]): Promise<T[]>`
|
|
- Add `executeOne<T>(sql: string, params: any[]): Promise<T | null>`
|
|
- Add error logging
|
|
|
|
### PHASE 2: Shim Query Builders (45 min)
|
|
|
|
- [ ] **Task 2.1:** Create `src/lib/shim/sites.ts`
|
|
```typescript
|
|
export async function getSites(filters?: FilterOptions): Promise<Site[]>
|
|
export async function getSiteById(id: string): Promise<Site | null>
|
|
export async function createSite(data: Partial<Site>): Promise<Site>
|
|
export async function updateSite(id: string, data: Partial<Site>): Promise<Site>
|
|
export async function deleteSite(id: string): Promise<boolean>
|
|
```
|
|
|
|
- [ ] **Task 2.2:** Create `src/lib/shim/articles.ts`
|
|
- Same CRUD pattern as sites
|
|
- Add `getArticlesByStatus(status: string)`
|
|
- Add `getArticlesBySite(siteId: string)`
|
|
|
|
- [ ] **Task 2.3:** Create `src/lib/shim/campaigns.ts`
|
|
- Campaign CRUD operations
|
|
- `getActiveCampaigns()`
|
|
|
|
- [ ] **Task 2.4:** Create `src/lib/shim/jobs.ts`
|
|
- Job queue queries
|
|
- `getJobsByStatus(status: string)`
|
|
|
|
### PHASE 3: API Routes (Secure Bridges) (60 min)
|
|
|
|
- [ ] **Task 3.1:** Create `src/pages/api/shim/sites/list.ts`
|
|
```typescript
|
|
// GET /api/shim/sites/list?limit=10&offset=0
|
|
import { pool } from '@/lib/db';
|
|
import { getSites } from '@/lib/shim/sites';
|
|
|
|
export async function GET({ request }) {
|
|
// 1. Validate token (copy from /api/god/)
|
|
// 2. Parse query params
|
|
// 3. Call getSites()
|
|
// 4. Return JSON
|
|
}
|
|
```
|
|
|
|
- [ ] **Task 3.2:** Create `src/pages/api/shim/sites/[id].ts`
|
|
- GET: Fetch single site
|
|
- PUT: Update site
|
|
- DELETE: Delete site
|
|
|
|
- [ ] **Task 3.3:** Create `src/pages/api/shim/sites/create.ts`
|
|
- POST: Create new site
|
|
- Validate required fields
|
|
- Return created site with ID
|
|
|
|
- [ ] **Task 3.4:** Repeat for articles (`/api/shim/articles/*`)
|
|
|
|
- [ ] **Task 3.5:** Add token validation middleware
|
|
- Extract from existing `/api/god/` pattern
|
|
- Check `GOD_MODE_TOKEN` or custom token
|
|
|
|
### PHASE 4: Server-Side Pages (SSR Demo) (30 min)
|
|
|
|
- [ ] **Task 4.1:** Create `src/pages/shim/sites.astro`
|
|
```astro
|
|
---
|
|
import { getSites } from '@/lib/shim/sites';
|
|
const sites = await getSites({ limit: 50 });
|
|
---
|
|
<h1>Sites ({sites.length})</h1>
|
|
<ul>
|
|
{sites.map(site => <li>{site.domain}</li>)}
|
|
</ul>
|
|
```
|
|
|
|
- [ ] **Task 4.2:** Create `src/pages/shim/articles.astro`
|
|
- Similar pattern for articles
|
|
- Show article count, recent articles
|
|
|
|
- [ ] **Task 4.3:** Create `src/pages/shim/index.astro`
|
|
- Dashboard showing counts from all tables
|
|
- Demo of multi-table queries in one page
|
|
|
|
### PHASE 5: Client-Side Components (React + TanStack Query) (45 min)
|
|
|
|
- [ ] **Task 5.1:** Create `src/components/shim/SitesList.tsx`
|
|
```typescript
|
|
import { useQuery } from '@tanstack/react-query';
|
|
|
|
export default function SitesList() {
|
|
const { data: sites } = useQuery({
|
|
queryKey: ['shim-sites'],
|
|
queryFn: () => fetch('/api/shim/sites/list').then(r => r.json())
|
|
});
|
|
|
|
return <ul>{sites?.map(...)}</ul>;
|
|
}
|
|
```
|
|
|
|
- [ ] **Task 5.2:** Create `src/components/shim/ArticleEditor.tsx`
|
|
- Form for creating/editing articles
|
|
- Uses `useMutation` for POST/PUT
|
|
- Calls `/api/shim/articles/[id]`
|
|
|
|
- [ ] **Task 5.3:** Add components to demo pages
|
|
- Wire `SitesList` to `/shim/sites.astro`
|
|
- Make it `client:load` hydrated
|
|
|
|
### PHASE 6: Testing & Security (30 min)
|
|
|
|
- [ ] **Task 6.1:** Test SSR pages locally
|
|
- Visit `/shim/sites` - should load instantly with data
|
|
- Verify no database credentials in browser
|
|
|
|
- [ ] **Task 6.2:** Test API routes
|
|
- `curl http://localhost:4321/api/shim/sites/list`
|
|
- Verify token requirement works
|
|
|
|
- [ ] **Task 6.3:** Security audit
|
|
- Ensure no SQL injection vulnerabilities
|
|
- Verify token validation on all endpoints
|
|
- Check that `DATABASE_URL` not exposed
|
|
|
|
- [ ] **Task 6.4:** Performance test
|
|
- Query 1000+ sites
|
|
- Check connection pool usage
|
|
- Monitor memory with large datasets
|
|
|
|
---
|
|
|
|
## 🔧 DETAILED EXAMPLE: Sites Table Implementation
|
|
|
|
### Step 1: Type Definition (`src/lib/shim/types.ts`)
|
|
|
|
```typescript
|
|
export interface Site {
|
|
id: string;
|
|
domain: string;
|
|
status: 'active' | 'inactive' | 'pending';
|
|
site_url: string;
|
|
site_wpjson: string;
|
|
created_at: Date;
|
|
updated_at: Date;
|
|
}
|
|
|
|
export interface FilterOptions {
|
|
limit?: number;
|
|
offset?: number;
|
|
status?: string;
|
|
search?: string;
|
|
}
|
|
```
|
|
|
|
### Step 2: Query Builder (`src/lib/shim/sites.ts`)
|
|
|
|
```typescript
|
|
import { pool } from '@/lib/db';
|
|
import type { Site, FilterOptions } from './types';
|
|
|
|
export async function getSites(options: FilterOptions = {}): Promise<Site[]> {
|
|
const { limit = 50, offset = 0, status, search } = options;
|
|
|
|
let sql = 'SELECT * FROM sites WHERE 1=1';
|
|
const params: any[] = [];
|
|
let paramIndex = 1;
|
|
|
|
if (status) {
|
|
sql += ` AND status = $${paramIndex++}`;
|
|
params.push(status);
|
|
}
|
|
|
|
if (search) {
|
|
sql += ` AND domain ILIKE $${paramIndex++}`;
|
|
params.push(`%${search}%`);
|
|
}
|
|
|
|
sql += ` ORDER BY created_at DESC LIMIT $${paramIndex++} OFFSET $${paramIndex++}`;
|
|
params.push(limit, offset);
|
|
|
|
const { rows } = await pool.query<Site>(sql, params);
|
|
return rows;
|
|
}
|
|
|
|
export async function getSiteById(id: string): Promise<Site | null> {
|
|
const { rows } = await pool.query<Site>(
|
|
'SELECT * FROM sites WHERE id = $1',
|
|
[id]
|
|
);
|
|
return rows[0] || null;
|
|
}
|
|
|
|
export async function createSite(data: Partial<Site>): Promise<Site> {
|
|
const { rows } = await pool.query<Site>(
|
|
`INSERT INTO sites (domain, status, site_url, site_wpjson)
|
|
VALUES ($1, $2, $3, $4)
|
|
RETURNING *`,
|
|
[data.domain, data.status || 'pending', data.site_url, data.site_wpjson]
|
|
);
|
|
return rows[0];
|
|
}
|
|
|
|
export async function updateSite(id: string, data: Partial<Site>): Promise<Site> {
|
|
const fields: string[] = [];
|
|
const values: any[] = [];
|
|
let paramIndex = 1;
|
|
|
|
Object.entries(data).forEach(([key, value]) => {
|
|
if (value !== undefined && key !== 'id') {
|
|
fields.push(`${key} = $${paramIndex++}`);
|
|
values.push(value);
|
|
}
|
|
});
|
|
|
|
values.push(id);
|
|
|
|
const { rows } = await pool.query<Site>(
|
|
`UPDATE sites SET ${fields.join(', ')}, updated_at = NOW()
|
|
WHERE id = $${paramIndex}
|
|
RETURNING *`,
|
|
values
|
|
);
|
|
return rows[0];
|
|
}
|
|
|
|
export async function deleteSite(id: string): Promise<boolean> {
|
|
const result = await pool.query('DELETE FROM sites WHERE id = $1', [id]);
|
|
return result.rowCount ? result.rowCount > 0 : false;
|
|
}
|
|
```
|
|
|
|
### Step 3: API Route (`src/pages/api/shim/sites/list.ts`)
|
|
|
|
```typescript
|
|
import type { APIRoute } from 'astro';
|
|
import { getSites } from '@/lib/shim/sites';
|
|
|
|
export const GET: APIRoute = async ({ request, url }) => {
|
|
try {
|
|
// 1. Token validation (optional for read operations)
|
|
const token = request.headers.get('Authorization')?.replace('Bearer ', '');
|
|
if (!token || token !== import.meta.env.GOD_MODE_TOKEN) {
|
|
return new Response(JSON.stringify({ error: 'Unauthorized' }), {
|
|
status: 401,
|
|
headers: { 'Content-Type': 'application/json' }
|
|
});
|
|
}
|
|
|
|
// 2. Parse query params
|
|
const limit = parseInt(url.searchParams.get('limit') || '50');
|
|
const offset = parseInt(url.searchParams.get('offset') || '0');
|
|
const status = url.searchParams.get('status') || undefined;
|
|
const search = url.searchParams.get('search') || undefined;
|
|
|
|
// 3. Execute query
|
|
const sites = await getSites({ limit, offset, status, search });
|
|
|
|
// 4. Return JSON
|
|
return new Response(JSON.stringify({ sites, count: sites.length }), {
|
|
status: 200,
|
|
headers: { 'Content-Type': 'application/json' }
|
|
});
|
|
|
|
} catch (error) {
|
|
console.error('API Error:', error);
|
|
return new Response(JSON.stringify({ error: 'Internal Server Error' }), {
|
|
status: 500,
|
|
headers: { 'Content-Type': 'application/json' }
|
|
});
|
|
}
|
|
};
|
|
```
|
|
|
|
### Step 4: SSR Page (`src/pages/shim/sites.astro`)
|
|
|
|
```astro
|
|
---
|
|
import AdminLayout from '@/layouts/AdminLayout.astro';
|
|
import { getSites } from '@/lib/shim/sites';
|
|
|
|
// Server-side query - runs before HTML is sent
|
|
const sites = await getSites({ limit: 100 });
|
|
---
|
|
|
|
<AdminLayout title="Sites (Direct DB)">
|
|
<h1 class="text-3xl font-bold text-white mb-4">Sites ({sites.length})</h1>
|
|
|
|
<div class="space-y-2">
|
|
{sites.map(site => (
|
|
<div class="p-4 bg-slate-800 rounded-lg">
|
|
<h3 class="text-white font-semibold">{site.domain}</h3>
|
|
<p class="text-slate-400 text-sm">{site.site_url}</p>
|
|
<span class={`text-xs px-2 py-1 rounded ${
|
|
site.status === 'active' ? 'bg-green-500/20 text-green-400' : 'bg-gray-500/20 text-gray-400'
|
|
}`}>
|
|
{site.status}
|
|
</span>
|
|
</div>
|
|
))}
|
|
</div>
|
|
</AdminLayout>
|
|
```
|
|
|
|
### Step 5: React Component (`src/components/shim/SitesList.tsx`)
|
|
|
|
```typescript
|
|
import React from 'react';
|
|
import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query';
|
|
import type { Site } from '@/lib/shim/types';
|
|
|
|
export default function SitesList() {
|
|
const queryClient = useQueryClient();
|
|
|
|
const { data, isLoading } = useQuery({
|
|
queryKey: ['shim-sites'],
|
|
queryFn: async () => {
|
|
const response = await fetch('/api/shim/sites/list', {
|
|
headers: {
|
|
'Authorization': `Bearer ${import.meta.env.PUBLIC_GOD_MODE_TOKEN}`
|
|
}
|
|
});
|
|
return response.json();
|
|
}
|
|
});
|
|
|
|
const deleteMutation = useMutation({
|
|
mutationFn: async (id: string) => {
|
|
await fetch(`/api/shim/sites/${id}`, {
|
|
method: 'DELETE',
|
|
headers: {
|
|
'Authorization': `Bearer ${import.meta.env.PUBLIC_GOD_MODE_TOKEN}`
|
|
}
|
|
});
|
|
},
|
|
onSuccess: () => {
|
|
queryClient.invalidateQueries({ queryKey: ['shim-sites'] });
|
|
}
|
|
});
|
|
|
|
if (isLoading) return <div>Loading...</div>;
|
|
|
|
return (
|
|
<div className="space-y-2">
|
|
{data?.sites?.map((site: Site) => (
|
|
<div key={site.id} className="p-4 bg-slate-800 rounded-lg flex justify-between items-center">
|
|
<div>
|
|
<h3 className="text-white font-semibold">{site.domain}</h3>
|
|
<p className="text-slate-400 text-sm">{site.site_url}</p>
|
|
</div>
|
|
<button
|
|
onClick={() => deleteMutation.mutate(site.id)}
|
|
className="px-3 py-1 bg-red-600 text-white rounded hover:bg-red-700"
|
|
>
|
|
Delete
|
|
</button>
|
|
</div>
|
|
))}
|
|
</div>
|
|
);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🔒 SECURITY CHECKLIST
|
|
|
|
- [ ] **Never** expose `DATABASE_URL` to client-side code
|
|
- [ ] **Always** validate tokens in API routes
|
|
- [ ] **Always** use parameterized queries (`$1`, `$2`) - never string concatenation
|
|
- [ ] **Never** trust user input - sanitize before queries
|
|
- [ ] **Use** connection pooling (already in `db.ts`)
|
|
- [ ] **Limit** query results (default LIMIT 50)
|
|
- [ ] **Log** all database errors server-side
|
|
- [ ] **Test** for SQL injection vulnerabilities
|
|
|
|
---
|
|
|
|
## 🚀 DEPLOYMENT CHECKLIST
|
|
|
|
- [ ] Set `DATABASE_URL` environment variable in Coolify
|
|
- [ ] Set `GOD_MODE_TOKEN` for API authentication
|
|
- [ ] Verify Dockerfile builds with `pg` dependency
|
|
- [ ] Test connection pool limits under load
|
|
- [ ] Monitor database connection count
|
|
- [ ] Set up database backups
|
|
- [ ] Configure SSL for PostgreSQL connection (production)
|
|
|
|
---
|
|
|
|
## 📊 EXPECTED PERFORMANCE
|
|
|
|
| Operation | SSR (Server-Side) | API Route (Client-Side) |
|
|
|-----------|-------------------|-------------------------|
|
|
| Get 50 sites | ~10ms | ~50ms (includes HTTP) |
|
|
| Create site | ~15ms | ~60ms |
|
|
| Update site | ~12ms | ~55ms |
|
|
| Complex join | ~50ms | ~100ms |
|
|
|
|
**Why SSR is faster:** No HTTP roundtrip, query executes before HTML is sent.
|
|
|
|
---
|
|
|
|
## 🎯 SUCCESS CRITERIA
|
|
|
|
1. ✅ Can view sites list on `/shim/sites` without any client-side API calls
|
|
2. ✅ Can create/update/delete sites via React component using `/api/shim/sites/*`
|
|
3. ✅ No database credentials visible in browser DevTools
|
|
4. ✅ All queries use parameterized SQL (no injection risk)
|
|
5. ✅ Connection pool stays under 20 connections
|
|
6. ✅ Token validation works on all API endpoints
|
|
7. ✅ SSR pages load in < 100ms with 1000+ records
|