.gitignore

@@ -2,10 +2,10 @@
 .env
 
 # Dependencies
-# Lockfiles ARE tracked — they pin the Perspective 4.5.1/4.4.1 pairing that the
-# caret ranges in ui/package.json would otherwise let drift. See docs/perspective.md.
 node_modules/
 ui/node_modules/
+package-lock.json
+ui/package-lock.json
 
 # UI build output (generated — run `cd ui && npm run build`)
 public/
@@ -28,5 +28,8 @@ Thumbs.db
 *.swp
 *.swo
 
-# Scratch data exports
+# Uploads
-/*.tsv
+uploads/*
+!uploads/.gitkeep
+
+*.tsv

CLAUDE.md

@@ -2,122 +2,207 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
-Dataflow imports CSV data, extracts structure from it with regex rules, maps the extracted
+## Overview
-values to standardized output, and serves the result over a REST API and React UI. It is a
-**simple system by design** — don't over-engineer it.
 
-**Read [docs/spec.md](docs/spec.md) for architecture, schema, data flow, the full API, and
+Dataflow is a simple data transformation tool for importing, cleaning, and standardizing data from various sources. Built with PostgreSQL and Node.js/Express, it emphasizes clarity and simplicity over complexity.
-`manage.py`.** This file covers only what you need to work in the repo without breaking
-something — the rules and non-obvious behaviours that aren't visible from the code.
 
-## Where things live
+## Core Concepts
 
-Both the API routes and the SQL are one file per resource: `api/routes/rules.js` and
+1. **Sources** - Define data sources and deduplication rules (which fields make a record unique)
-`database/rules.sql` are two halves of the same feature. Two SQL files are shared engines
+2. **Import** - Load CSV data, automatically deduplicating based on source rules
-rather than per-route — `import.sql` (CSV import, audit trail) and `transform.sql` (the
+3. **Rules** - Extract information using regex patterns (e.g., extract merchant from transaction description)
-rule/mapping engine, including the `jsonb_concat_obj` aggregate).
+4. **Mappings** - Map extracted values to standardized output (e.g., "WALMART" → {"vendor": "Walmart", "category": "Groceries"})
+5. **Transform** - Apply rules and mappings to create clean, enriched data
 
-`manage.py`'s `QUERY_FILES` list is the deploy order and the authoritative file list.
+## Architecture
 
-## Rules that matter
+### Database Schema (`database/schema.sql`)
 
-**`database/*.sql` is the source of truth for every database function. Never edit a function
+**5 simple tables:**
-directly in the database.** A live edit that isn't written back to the file is silently
+- `sources` - Source definitions with `constraint_fields` array
-reverted the next time anyone runs "Redeploy SQL functions". This has already happened once:
+- `records` - Imported data with `data` (raw) and `transformed` (enriched) JSONB columns
-five functions drifted and sat wrong in the repo for months — see the git history of the
+- `rules` - Regex extraction rules with `field`, `pattern`, `output_field`
-deleted `database/functions.sql`.
+- `mappings` - Input/output value mappings
+- `import_log` - Audit trail
 
-**Always run `npm run build` from `ui/` after any change to `ui/src/`.** The Express server
+**Key design:**
-serves the built output in `public/`; source changes are invisible until you rebuild.
+- JSONB for flexible data storage
+- Deduplication via MD5 hash of specified fields
+- Simple, flat structure (no complex relationships)
 
-**Never use `ON CONFLICT (constraint_key)`.** See deduplication below — there is no unique
+### Database Functions (`database/functions.sql`)
-constraint, and adding one would drop legitimate transactions.
 
-## The three data layers
+**4 focused functions:**
+- `import_records(source_name, data)` - Import with deduplication
+- `apply_transformations(source_name, record_ids)` - Apply rules and mappings
+- `get_unmapped_values(source_name, rule_name)` - Find values needing mappings
+- `reprocess_records(source_name)` - Re-transform all records
 
-Each row in `records` keeps its data in three JSONB columns:
+**Design principle:** Each function does ONE thing. No nested CTEs, no duplication.
 
-- `data` — raw imported values, never modified
+### API Server (`api/server.js` + `api/routes/`)
-- `transformed` — rule and mapping output only (the delta)
-- `overrides` — manual edits, highest precedence
 
-Readers merge them as `data || transformed || overrides`. Keeping them separate is what lets
+**RESTful endpoints:**
-`reprocess_records` re-run the rules without clobbering a manual edit. Anything that writes
+- `/api/sources` - CRUD sources, import CSV, trigger transformations
-overrides into `transformed` is a bug — that was the pre-May-2026 behaviour.
+- `/api/rules` - CRUD transformation rules
+- `/api/mappings` - CRUD value mappings, view unmapped values
+- `/api/records` - Query and search transformed data
 
-## Deduplication
+**Route files:**
+- `routes/sources.js` - Source management and CSV import
+- `routes/rules.js` - Rule management
+- `routes/mappings.js` - Mapping management + unmapped values
+- `routes/records.js` - Record queries and search
 
-- `constraint_key` is a JSONB object of the constraint field values — readable, no hashing
+## Common Development Tasks
-- Dedup is enforced at import time in a CTE. There is **no unique DB constraint** on it
-- **The constraint key is cross-batch re-import protection, not record uniqueness**
-- Within one import batch, all rows insert even when constraint keys collide. Banks
-  legitimately send identical-looking transactions — 11 separate Cedar Point charges on the
-  same day are 11 real rows
-- On re-import of an overlapping date range, rows whose key already exists are skipped, so
-  re-running a month-to-date export the next day doesn't double-count
-- Deleting an import log entry cascades to every record in that batch (`import_id` FK)
 
-## Error handling
+### Running the Application
 
-API routes use `try/catch` and pass errors to `next(err)`; `server.js` has a global handler.
+```bash
-Database functions return JSON with a `success` boolean.
+# Setup (first time only)
+./setup.sh
 
-## Light / dark mode
+# Start development server with auto-reload
+npm run dev
 
-Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a
+# Start production server
-`ThemeProvider` that wraps the app in `main.jsx`.
+npm start
 
-- **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
+# Test API
-- **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
+curl http://localhost:3000/health
-- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc.
+```
-- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
-- **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires
-- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
 
-## Pivot inspector panel
+### Database Changes
 
-Clicking a data cell opens a right-hand inspector panel showing the underlying transactions
+When modifying schema:
-for that cell. See [docs/perspective.md](docs/perspective.md) for the Perspective API itself.
+1. Edit `database/schema.sql`
+2. Drop and recreate schema: `psql -d dataflow -f database/schema.sql`
+3. Redeploy functions: `psql -d dataflow -f database/functions.sql`
 
-- **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views.
+For production, write migration scripts instead of dropping schema.
-- **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click.
-- **split_by filter derivation**: `detail.config.filter` from the click event may omit split_by column constraints. They are derived from `column_names` positionally (`column_names[i]` matches `config.split_by[i]`) and appended to the filter before querying.
-- **Row filtering**: a temporary `table.view({ filter, expressions })` is used so Perspective evaluates expression/computed columns correctly. Falls back to JS-side `filterRowsByConfig` on error (which skips filters for fields not in raw data).
-- The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px).
-- The transaction table is sortable (click header) and shows column totals for all-numeric columns.
 
-## Pivot layout persistence
+### Adding a New API Endpoint
 
-Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The
+1. Add route to appropriate file in `api/routes/`
-`source_name` column holds either a source name or a stack name — the FK to `sources(name)`
+2. Follow existing patterns (async/await, error handling via `next()`)
-was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use
+3. Use parameterized queries to prevent SQL injection
-`/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`,
+4. Return consistent JSON format
-`save_pivot_layout`, `delete_pivot_layout`). `localStorage` still remembers the *last active
-layout* for a view (the `psp_layout_<name>` key), but the definitions live in the DB so they
-persist across machines.
 
-## Adding features
+### Testing
 
-- One function, one job; keep functions under 100 lines
+Manual testing workflow:
-- Write clear SQL, not clever SQL
+1. Create a source: `POST /api/sources`
-- Add the SQL function to the matching `database/*.sql` file, then the route that calls it
+2. Create rules: `POST /api/rules`
-- Update `docs/spec.md` when you add or change an endpoint
+3. Import data: `POST /api/sources/:name/import`
+4. Apply transformations: `POST /api/sources/:name/transform`
+5. View results: `GET /api/records/source/:name`
+
+See `examples/GETTING_STARTED.md` for complete curl examples.
+
+## Design Principles
+
+1. **Simple over clever** - Straightforward code beats optimization
+2. **Explicit over implicit** - No magic, no hidden triggers
+3. **Clear naming** - `data` not `rec`, `transformed` not `allj`
+4. **One function, one job** - No 250-line functions
+5. **JSONB for flexibility** - Handle varying schemas without migrations
+
+## Common Patterns
+
+### Import Flow
+```
+CSV file → parse → import_records() → records table (data column)
+```
+
+### Transformation Flow
+```
+records.data → apply_transformations() →
+  - Apply each rule (regex extraction)
+  - Look up mappings
+  - Merge into records.transformed
+```
+
+### Deduplication
+- `constraint_key` is a JSONB object of the constraint field values (readable, no hashing)
+- Dedup is enforced at import time via CTE — no unique DB constraint
+- Intra-file duplicate rows are allowed (bank may send identical rows); they all insert
+- On re-import, all rows whose constraint_key already exists in the DB are skipped
+- Deleting an import log entry cascades to all records from that batch (import_id FK)
+
+### Error Handling
+- API routes use `try/catch` and pass errors to `next(err)`
+- Server.js has global error handler
+- Database functions return JSON with `success` boolean
+
+## File Structure
+
+```
+dataflow/
+├── database/
+│   ├── schema.sql         # Table definitions
+│   └── functions.sql      # Import/transform functions
+├── api/
+│   ├── server.js          # Express server
+│   └── routes/            # API endpoints
+│       ├── sources.js
+│       ├── rules.js
+│       ├── mappings.js
+│       └── records.js
+├── examples/
+│   ├── GETTING_STARTED.md # Tutorial
+│   └── bank_transactions.csv
+├── .env.example           # Config template
+├── package.json
+└── README.md
+```
+
+## Comparison to Legacy TPS System
+
+This project replaces an older system (in `/opt/tps`) that had:
+- 2,150 lines of complex SQL with heavy duplication
+- 5 nearly-identical 200+ line functions
+- Confusing names and deep nested CTEs
+- Complex trigger-based processing
+
+Dataflow achieves the same functionality with:
+- ~400 lines of simple SQL
+- 4 focused functions
+- Clear names and linear logic
+- Explicit API-triggered processing
+
+The simplification makes it easy to understand, modify, and maintain.
 
 ## Troubleshooting
 
-**Database connection fails** — check `.env` credentials, that PostgreSQL is running, and
+**Database connection fails:**
-that the search path resolves to the `dataflow` schema.
+- Check `.env` file exists and has correct credentials
+- Verify PostgreSQL is running: `psql -U postgres -l`
+- Check search path is set: Should default to `dataflow` schema
 
-**Import succeeds but transformation does nothing** — check rules exist for that source
+**Import succeeds but transformation fails:**
-(`SELECT * FROM dataflow.rules WHERE source_name = '…'`), that `field` matches an actual key
+- Check rules exist: `SELECT * FROM dataflow.rules WHERE source_name = 'xxx'`
-in `data`, and test the pattern with `GET /api/rules/preview`.
+- Verify field names match CSV columns
+- Test regex pattern manually
+- Check for SQL errors in logs
 
-**Everything is marked duplicate** — `constraint_fields` probably don't match the real field
+**All records marked as duplicates:**
-names, or the batch was already imported.
+- Verify `constraint_fields` match actual field names in data
+- Check if data was already imported
+- Use different source name for testing
 
-## History
+## Adding New Features
 
-This replaces an older system still in `/opt/tps` — 2,150 lines of SQL with five
+When adding features, follow these principles:
-nearly-identical 200-line functions and trigger-based processing. Dataflow is a clean
+- Add ONE function that does ONE thing
-rewrite, not a refactor. Some function bodies still carry `mirrors TPS …` comments pointing
+- Keep functions under 100 lines if possible
-at their counterpart there.
+- Write clear SQL, not clever SQL
+- Add API endpoint that calls the function
+- Document in README.md and update examples
+
+## Notes for Claude
+
+- This is a **simple** system by design - don't over-engineer it
+- Keep functions focused and linear
+- Use JSONB for flexibility, not as a crutch for bad design
+- When confused, read the examples/GETTING_STARTED.md walkthrough
+- The old TPS system is in `/opt/tps` - this is a clean rewrite, not a refactor

README.md

@@ -2,71 +2,198 @@
 
 A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
 
-Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will
+## What It Does
-deduplicate on import, pull structure out with regex rules, map the extracted values to clean
-output, and serve the result through a web UI and REST API.
 
-## How it works
+Dataflow helps you:
+1. **Import** CSV data with automatic deduplication
+2. **Transform** data using regex rules to extract meaningful information
+3. **Map** extracted values to standardized output
+4. **Query** the transformed data via a web UI or REST API
 
-1. **Sources** define where data comes from and which fields make a record unique
+Perfect for cleaning up messy data like bank transactions, product lists, or any repetitive data that needs normalization.
-2. **Rules** extract information with regex (`extract` or `replace` mode) —
-   e.g. pull the merchant out of a transaction description
-3. **Mappings** turn extracted values into clean output —
-   `"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
-4. **Records** are then queryable, pivotable, and exportable
 
-Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output),
+## Core Concepts
-and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never
-clobbers something you typed by hand.
 
-## Stack
+### 1. Sources
+Define where data comes from and how to deduplicate it.
 
-PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`.
+**Example:** Bank transactions deduplicated by date + amount + description
-HTTP Basic auth, configured in `.env`.
 
-## Getting started
+### 2. Rules
+Extract information using regex patterns (`extract` or `replace` modes).
 
-Requires PostgreSQL 12+, Node.js 18+, and Python 3.
+**Example:** Extract merchant name from transaction description
 
+### 3. Mappings
+Map extracted values to clean, standardized output.
+
+**Example:** "DISCOUNT DRUG MART 32" → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
+
+## Architecture
+
+- **Database:** PostgreSQL with JSONB for flexible data storage
+- **API:** Node.js/Express REST API
+- **UI:** React SPA served from `public/`
+- **Auth:** HTTP Basic auth (configured in `.env`)
+
+## Design Principles
+
+- **Simple & Clear** - Easy to understand what's happening
+- **Explicit** - No hidden magic or complex triggers
+- **Flexible** - Handle varying data formats without schema changes
+
+## Getting Started
+
+### Prerequisites
+- PostgreSQL 12+
+- Node.js 16+
+- Python 3 (for `manage.py`)
+
+### Installation
+
+1. Install Node dependencies:
 ```bash
 npm install
-python3 manage.py     # interactive setup: .env, database, schema, functions, UI, service
 ```
 
-The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api`
+2. Run the management script to configure and deploy everything:
-(port set by `API_PORT` in `.env`).
+```bash
+python3 manage.py
+```
 
-For a walkthrough that creates a source, adds rules and mappings, and imports the sample
+For development with auto-reload:
-CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**.
+```bash
+npm run dev
+```
 
-## Documentation
+The UI is available at `http://localhost:3000`. The API is at `http://localhost:3000/api`.
 
-| | |
+## Management Script (`manage.py`)
-|---|---|
-| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl |
-| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` |
-| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages |
-| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference |
 
-## Project structure
+`manage.py` is an interactive tool for configuring, deploying, and managing the service. Run it and choose from the numbered menu:
+
+```
+python3 manage.py
+```
+
+| # | Action |
+|---|--------|
+| 1 | **Database configuration** — create/update `.env`, optionally create the PostgreSQL user/database, and deploy schema + functions |
+| 2 | Redeploy schema only (`database/schema.sql`) — drops and recreates all tables |
+| 3 | Redeploy SQL functions only (`database/queries/`) |
+| 4 | Build UI (`ui/` → `public/`) |
+| 5 | Set up nginx reverse proxy (HTTP or HTTPS via certbot) |
+| 6 | Install systemd service unit (`dataflow.service`) |
+| 7 | Start / restart `dataflow.service` |
+| 8 | Stop `dataflow.service` |
+| 9 | Set login credentials (`LOGIN_USER` / `LOGIN_PASSWORD_HASH` in `.env`) |
+
+The status screen at the top of the menu shows the current state of each component (database connection, schema, UI build, service, nginx).
+
+**Typical first-time setup:** run options 1 → 4 → 9 → 6 → 7 (→ 5 if you want nginx).
+
+## API Reference
+
+All `/api` routes require HTTP Basic authentication.
+
+### Sources — `/api/sources`
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/sources` | List all sources |
+| POST | `/api/sources` | Create a source |
+| GET | `/api/sources/:name` | Get a source |
+| PUT | `/api/sources/:name` | Update a source |
+| DELETE | `/api/sources/:name` | Delete a source |
+| POST | `/api/sources/suggest` | Suggest source definition from CSV upload |
+| POST | `/api/sources/:name/import` | Import CSV data and auto-apply transformations to new records |
+| GET | `/api/sources/:name/import-log` | View import history (includes `inserted_keys` / `excluded_keys` in `info`) |
+| DELETE | `/api/sources/:name/import-log/:id` | Delete an import batch and all its records |
+| POST | `/api/sources/:name/transform` | Apply rules and mappings to any untransformed records |
+| POST | `/api/sources/:name/reprocess` | Re-transform all records |
+| GET | `/api/sources/:name/fields` | List all known field names |
+| GET | `/api/sources/:name/stats` | Get record and mapping counts |
+| POST | `/api/sources/:name/view` | Generate output view |
+| GET | `/api/sources/:name/view-data` | Query output view (paginated, sortable) |
+
+### Rules — `/api/rules`
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/rules/source/:source_name` | List rules for a source |
+| POST | `/api/rules` | Create a rule |
+| GET | `/api/rules/:id` | Get a rule |
+| PUT | `/api/rules/:id` | Update a rule |
+| DELETE | `/api/rules/:id` | Delete a rule |
+| GET | `/api/rules/preview` | Preview a pattern against real records (ad-hoc) |
+| GET | `/api/rules/:id/test` | Test a saved rule against real records |
+
+### Mappings — `/api/mappings`
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/mappings/source/:source_name` | List mappings |
+| POST | `/api/mappings` | Create a mapping |
+| POST | `/api/mappings/bulk` | Bulk create/update mappings |
+| GET | `/api/mappings/:id` | Get a mapping |
+| PUT | `/api/mappings/:id` | Update a mapping |
+| DELETE | `/api/mappings/:id` | Delete a mapping |
+| GET | `/api/mappings/source/:source_name/unmapped` | Get values with no mapping yet |
+| GET | `/api/mappings/source/:source_name/all-values` | All extracted values with counts |
+| GET | `/api/mappings/source/:source_name/counts` | Record counts for existing mappings |
+| GET | `/api/mappings/source/:source_name/export.tsv` | Export values as TSV |
+| POST | `/api/mappings/source/:source_name/import-csv` | Import mappings from TSV |
+
+### Records — `/api/records`
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/records/source/:source_name` | List records (paginated) |
+| GET | `/api/records/:id` | Get a single record |
+| POST | `/api/records/search` | Search records |
+| DELETE | `/api/records/:id` | Delete a record |
+| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
+
+## Typical Workflow
+
+```
+1. Create a source (POST /api/sources)
+2. Create transformation rules (POST /api/rules)
+3. Import CSV data (POST /api/sources/:name/import) — transformations applied automatically to new records
+4. Preview rules against real data (GET /api/rules/preview)
+5. Review unmapped values (GET /api/mappings/source/:name/unmapped)
+6. Add mappings (POST /api/mappings or bulk import via TSV)
+7. Reprocess to apply new mappings (POST /api/sources/:name/reprocess)
+8. Query results (GET /api/sources/:name/view-data)
+```
+
+See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
+
+## Project Structure
 
 ```
 dataflow/
-├── manage.py           # interactive setup / deploy / uninstall
+├── database/
-├── database/           # schema.sql + one .sql file per API route
+│   ├── schema.sql         # Table definitions
-├── api/                # Express server, routes, auth middleware
+│   └── functions.sql      # Import/transform/query functions
-├── ui/                 # React source (built to public/)
+├── api/
-├── public/             # built UI, served as static files
+│   ├── server.js          # Express server
-├── docs/
+│   ├── middleware/
-└── examples/           # sample CSV for the tutorial
+│   │   └── auth.js        # Basic auth middleware
+│   ├── lib/
+│   │   └── sql.js         # SQL literal helpers
+│   └── routes/
+│       ├── sources.js
+│       ├── rules.js
+│       ├── mappings.js
+│       └── records.js
+├── public/                # Built React UI (served as static files)
+├── examples/
+│   ├── GETTING_STARTED.md
+│   └── bank_transactions.csv
+└── .env.example
 ```
 
-Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js`
-and `database/rules.sql` are the two halves of the same feature.
-
-`database/*.sql` is the source of truth for every database function — never edit one directly
-in the database, or the next redeploy will silently revert it.
-
 ## License
 
 MIT

SPEC.md

@@ -18,7 +18,7 @@ SQL functions are the single source of truth for business logic. The API layer i
 Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does.
 
 ### One SQL file per route
-SQL is organized in `database/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `stacks.sql`, `status.sql`) plus `import.sql` and `transform.sql` for the import/transform engine. This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
+SQL is organized in `database/queries/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`). This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
 
 ### Explicit over implicit
 Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs.
@@ -34,14 +34,11 @@ Raw imported records and transformed records are stored as JSONB. This avoids sc
 manage.py              — interactive CLI for setup, deployment, and management
 database/
   schema.sql           — table definitions (run once or to reset)
-  sources.sql          — all SQL for /api/sources
+  queries/
-  rules.sql            — all SQL for /api/rules
+    sources.sql        — all SQL for /api/sources
-  mappings.sql         — all SQL for /api/mappings
+    rules.sql          — all SQL for /api/rules
-  records.sql          — all SQL for /api/records
+    mappings.sql       — all SQL for /api/mappings
-  stacks.sql           — all SQL for /api/stacks
+    records.sql        — all SQL for /api/records
-  status.sql           — all SQL for /api/status
-  import.sql           — CSV import and the import audit trail
-  transform.sql        — the rule/mapping engine
 api/
   server.js            — Express server, mounts routes, auth middleware
   middleware/
@@ -53,8 +50,6 @@ api/
     rules.js           — HTTP handlers for rule management
     mappings.js        — HTTP handlers for mapping management
     records.js         — HTTP handlers for record queries
-    stacks.js          — HTTP handlers for stack management
-    status.js          — HTTP handler for deployment status
 ui/
   src/
     api.js             — fetch wrapper, credential management
@@ -67,14 +62,8 @@ ui/
       Mappings.jsx     — mapping table with TSV import/export
       Records.jsx      — paginated, sortable view of transformed records
       Pivot.jsx        — interactive pivot table with cell inspector
-      Stacks.jsx       — multi-source union views with running balance
-      Remap.jsx        — bulk remap of an output field value across mappings
       Log.jsx          — global import log across all sources
-    components/        — Sidebar, StatusBar
-    theme.jsx          — light/dark context provider
 public/                — compiled UI (output of npm run build in ui/)
-docs/                  — this file, tutorial, UI and Perspective references
-examples/              — bank_transactions.csv, the tutorial's sample data
 ```
 
 ---
@@ -132,7 +121,7 @@ The transform is fully set-based — no row-by-row loops. All records for a sour
 
 ## SQL Functions
 
-Each route file has a matching SQL file in `database/`; `import.sql` and `transform.sql` hold the engine shared by several routes.
+Each file in `database/queries/` maps 1-to-1 with a route file.
 
 **sources.sql**
 `list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view`
@@ -146,12 +135,6 @@ Each route file has a matching SQL file in `database/`; `import.sql` and `transf
 **records.sql**
 `list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records`
 
-**stacks.sql**
-`list_stacks`, `get_stack`, `create_stack`, `update_stack`, `delete_stack`, `get_stack_view_data` (union of source views with field mapping and running balance), `list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`
-
-**status.sql**
-`get_status` — returns deployment state (schema version, function presence, service status)
-
 ---
 
 ## API
@@ -162,104 +145,43 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
 
 **Route summary:**
 
-### Sources — `api/routes/sources.js`
-
 | Method | Path | Description |
 |--------|------|-------------|
 | GET | /api/sources | List all sources |
 | POST | /api/sources | Create source |
 | GET | /api/sources/:name | Get source |
-| PUT | /api/sources/:name | Update source (constraint_fields, config, global_picklist) |
+| PUT | /api/sources/:name | Update source (constraint_fields, config) |
-| DELETE | /api/sources/:name | Delete source and all its data |
+| DELETE | /api/sources/:name | Delete source and all data |
-| POST | /api/sources/suggest | Suggest source config from an uploaded CSV |
+| POST | /api/sources/suggest | Suggest source config from CSV upload |
-| POST | /api/sources/:name/import | Import CSV; transformations are applied to the new records |
+| POST | /api/sources/:name/import | Import CSV records |
-| GET | /api/sources/import-log | Import history across all sources |
+| GET | /api/sources/:name/import-log | Import history |
-| GET | /api/sources/:name/import-log | Import history for one source |
-| DELETE | /api/sources/:name/import-log/:id | Delete an import batch and every record in it |
-| POST | /api/sources/:name/transform | Apply transformations to untransformed records only |
-| POST | /api/sources/:name/reprocess | Reapply transformations to all records |
 | GET | /api/sources/:name/stats | Record counts |
-| GET | /api/sources/:name/fields | All known field names and their origins |
+| GET | /api/sources/:name/fields | All known field names and origins |
-| GET | /api/sources/:name/override-keys | Distinct field names used in overrides for this source |
+| GET | /api/sources/:name/view-data | Paginated, sortable view data |
-| POST | /api/sources/:name/view | Generate/refresh the `dfv` view |
+| POST | /api/sources/:name/transform | Apply transformations (new records only) |
-| GET | /api/sources/:name/view-data | Paginated, sortable, filterable view data |
+| POST | /api/sources/:name/reprocess | Reapply transformations to all records |
-| GET | /api/sources/:name/layouts | List saved pivot layouts |
+| POST | /api/sources/:name/view | Generate dfv view |
-| POST | /api/sources/:name/layouts | Save a pivot layout |
-| DELETE | /api/sources/:name/layouts/:id | Delete a pivot layout |
-
-### Rules — `api/routes/rules.js`
-
-| Method | Path | Description |
-|--------|------|-------------|
 | GET | /api/rules/source/:name | List rules for a source |
-| GET | /api/rules/:id | Get a rule |
+| GET | /api/rules/preview | Preview pattern against live records |
+| GET | /api/rules/:id/test | Test saved rule against live records |
 | POST | /api/rules | Create rule |
 | PUT | /api/rules/:id | Update rule |
 | DELETE | /api/rules/:id | Delete rule |
-| GET | /api/rules/preview | Preview an ad-hoc pattern against live records |
-| GET | /api/rules/:id/test | Test a saved rule against live records |
-
-### Mappings — `api/routes/mappings.js`
-
-| Method | Path | Description |
-|--------|------|-------------|
 | GET | /api/mappings/source/:name | List mappings |
-| GET | /api/mappings/:id | Get a mapping |
+| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) |
+| GET | /api/mappings/source/:name/unmapped | Only unmapped extracted values |
+| GET | /api/mappings/source/:name/counts | Record counts per mapping |
+| GET | /api/mappings/source/:name/export.tsv | Export mappings as TSV |
+| POST | /api/mappings/source/:name/import-csv | Import/update mappings from TSV |
 | POST | /api/mappings | Create mapping |
 | POST | /api/mappings/bulk | Upsert multiple mappings |
 | PUT | /api/mappings/:id | Update mapping |
 | DELETE | /api/mappings/:id | Delete mapping |
-| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) with counts |
+| GET | /api/records/source/:name | List raw records |
-| GET | /api/mappings/source/:name/unmapped | Only values with no mapping yet |
+| GET | /api/records/:id | Get single record |
-| GET | /api/mappings/source/:name/counts | Record counts per mapping |
-| GET | /api/mappings/source/:name/export.tsv | Export extracted values as TSV |
-| POST | /api/mappings/source/:name/import-csv | Import/update mappings from an uploaded TSV |
-| GET | /api/mappings/global-values | Output values across all `global_picklist` sources (autocomplete) |
-| GET | /api/mappings/outputs | Search output field values across all mappings |
-| GET | /api/mappings/outputs/:col/:val | Mappings carrying a specific output field value |
-| POST | /api/mappings/remap-field | Replace an output field value across all mappings |
-
-### Records — `api/routes/records.js`
-
-| Method | Path | Description |
-|--------|------|-------------|
-| GET | /api/records/source/:name | List records (paginated) |
-| GET | /api/records/:id | Get a single record |
 | POST | /api/records/search | Search by JSONB containment |
 | DELETE | /api/records/:id | Delete record |
 | DELETE | /api/records/source/:name/all | Delete all records for a source |
-| PUT | /api/records/:id/overrides | Set manual overrides on a record |
-| DELETE | /api/records/:id/overrides | Clear a record's overrides |
-| POST | /api/records/bulk-overrides | Apply the same overrides to many records |
-
-### Stacks — `api/routes/stacks.js`
-
-| Method | Path | Description |
-|--------|------|-------------|
-| GET | /api/stacks | List all stacks |
-| GET | /api/stacks/:name | Get a stack with its sources |
-| POST | /api/stacks | Create stack |
-| PUT | /api/stacks/:name | Update stack |
-| DELETE | /api/stacks/:name | Delete stack |
-| PUT | /api/stacks/:name/sources/:source | Add or update a source in the stack |
-| DELETE | /api/stacks/:name/sources/:source | Remove a source from the stack |
-| PUT | /api/stacks/:name/sources/reorder | Reorder the stack's sources |
-| GET | /api/stacks/:name/view-sql | Preview the SQL that would build the view (dry run) |
-| POST | /api/stacks/:name/view | Generate/refresh the `dfv` view |
-| POST | /api/stacks/:name/exec-sql | Execute user-edited SQL for the view |
-| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
-| GET | /api/stacks/:name/balance | Current running balance from the generated view |
-| POST | /api/stacks/:name/calibrate | Set the balance offset from a known balance at a date |
-| GET | /api/stacks/:name/layouts | List saved pivot layouts |
-| POST | /api/stacks/:name/layouts | Save a pivot layout |
-| DELETE | /api/stacks/:name/layouts/:id | Delete a pivot layout |
-
-### Status — `api/routes/status.js`
-
-| Method | Path | Description |
-|--------|------|-------------|
-| GET | /api/status | Deployment status |
-| GET | /health | Health check (no auth) |
 
 ---
 
@@ -296,11 +218,11 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
 
 - **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page.
 
-- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` client/viewer/datagrid v4.5.1, viewer-d3fc v4.4.1 — installed via npm). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
+- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` v4.4.0, loaded from CDN at runtime). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
 
   **Toolbar (above the viewer):**
   - Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it.
-  - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
+  - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found in v4.4.0 — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
   - The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads.
 
   **Cell inspector (right panel):**
@@ -313,9 +235,7 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
   - `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save.
   - Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently.
 
-  See `docs/perspective.md` for the full technical reference on controlling Perspective programmatically.
+  See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically.
-
-- **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
 
 - **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column.
 
@@ -340,7 +260,7 @@ Shows current status on every screen:
 
 2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation.
 
-3. **Redeploy SQL functions** — Runs the function files in `database/` in dependency order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `import.sql`, `transform.sql`, `stacks.sql`, `status.sql`. Safe to run at any time without data loss.
+3. **Redeploy SQL functions** — Runs all four files in `database/queries/` in order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`. Safe to run at any time without data loss.
 
 4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`.
 
@@ -354,8 +274,6 @@ Shows current status on every screen:
 
 9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed.
 
-10. **Uninstall** — Reverses everything the other options install, in reverse order: stops/disables/removes the systemd unit, removes the nginx site and reloads nginx, drops the database and its user (prompts for admin credentials), then deletes `.env`, `public/`, and `node_modules`. Lists exactly what it found before doing anything and requires typing `delete` to proceed. The repository itself is left in place.
-
 **Key behaviors:**
 - All commands that will be run are printed before the user is asked to confirm.
 - Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal.
@@ -403,18 +321,13 @@ The server binds to `0.0.0.0` on `API_PORT` and serves both the API and the comp
 
 ## Deploying SQL Changes
 
-Any time SQL functions are modified, run `python3 manage.py` and choose "Redeploy SQL
+Any time SQL functions are modified:
-functions only". It runs every function file in dependency order — the list lives in
-`QUERY_FILES` in `manage.py`, which is the one place the order is defined.
-
-To deploy a single file by hand:
 ```bash
-PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -v ON_ERROR_STOP=1 -f database/rules.sql
+PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/sources.sql
+PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/rules.sql
+PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/mappings.sql
+PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/records.sql
 ```
-
+Then restart the server. Function deployment is safe to repeat — all functions use `CREATE OR REPLACE`.
-Deployment is safe to repeat — every function uses `CREATE OR REPLACE`.
-
-**The files are the source of truth.** Editing a function directly in the database, without
-writing the change back to its file, means the next redeploy silently reverts it.
 
 Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead.

api/lib/sql.js

@@ -8,7 +8,7 @@
 function lit(val) {
     if (val === null || val === undefined) return 'NULL';
     if (typeof val === 'boolean') return val ? 'TRUE' : 'FALSE';
-    if (typeof val === 'number') return String(val);
+    if (typeof val === 'number') return String(Math.trunc(val));
     if (typeof val === 'object') return `'${JSON.stringify(val).replace(/'/g, "''")}'`;
     return `'${String(val).replace(/'/g, "''")}'`;
 }

api/routes/records.js

@@ -49,56 +49,6 @@ module.exports = (pool) => {
         }
     });
 
-    // Set overrides for all selected records
-    router.post('/bulk-overrides', async (req, res, next) => {
-        try {
-            const { source_name, record_ids, overrides } = req.body;
-            if (!source_name || !Array.isArray(record_ids) || record_ids.length === 0 || !overrides || typeof overrides !== 'object')
-                return res.status(400).json({ error: 'source_name, record_ids array, and overrides object required' });
-            const idList = record_ids.map(id => parseInt(id)).join(',');
-            const result = await pool.query(
-                `SELECT bulk_set_record_overrides(${lit(source_name)}, ARRAY[${idList}]::int[], ${lit(overrides)}) as result`
-            );
-            res.json(result.rows[0].result);
-        } catch (err) {
-            next(err);
-        }
-    });
-
-    // Set overrides for a record
-    router.put('/:id/overrides', async (req, res, next) => {
-        try {
-            const { overrides } = req.body;
-            if (!overrides || typeof overrides !== 'object')
-                return res.status(400).json({ error: 'overrides object required' });
-            const result = await pool.query(
-                `SELECT set_record_overrides(${lit(parseInt(req.params.id))}, ${lit(overrides)}) as rec`
-            );
-            if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
-            res.json(result.rows[0].rec);
-        } catch (err) {
-            next(err);
-        }
-    });
-
-    // Clear overrides and reprocess that record to restore computed values
-    router.delete('/:id/overrides', async (req, res, next) => {
-        try {
-            const result = await pool.query(
-                `SELECT clear_record_overrides(${lit(parseInt(req.params.id))}) as rec`
-            );
-            if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
-            const { source_name } = result.rows[0].rec;
-            await pool.query(
-                `SELECT apply_transformations(${lit(source_name)}, ARRAY[${lit(parseInt(req.params.id))}::int], true)`
-            );
-            const updated = await pool.query(`SELECT * FROM get_record(${lit(parseInt(req.params.id))})`);
-            res.json(updated.rows[0]);
-        } catch (err) {
-            next(err);
-        }
-    });
-
     // Delete record
     router.delete('/:id', async (req, res, next) => {
         try {

api/routes/sources.js

@@ -187,11 +187,7 @@ module.exports = (pool) => {
     router.post('/:name/view', async (req, res, next) => {
         try {
             const result = await pool.query(`SELECT generate_source_view(${lit(req.params.name)}) as result`);
-            const data = result.rows[0].result;
+            res.json(result.rows[0].result);
-            if (data && data.success) {
-                await pool.query(`UPDATE dataflow.sources SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
-            }
-            res.json(data);
         } catch (err) {
             next(err);
         }
@@ -234,19 +230,6 @@ module.exports = (pool) => {
         }
     });
 
-    // Override keys — distinct field names used in overrides across all records for this source
-    router.get('/:name/override-keys', async (req, res, next) => {
-        try {
-            const result = await pool.query(
-                `SELECT DISTINCT jsonb_object_keys(overrides) AS key
-                 FROM dataflow.records
-                 WHERE source_name = ${lit(req.params.name)} AND overrides IS NOT NULL
-                 ORDER BY key`
-            );
-            res.json(result.rows.map(r => r.key));
-        } catch (err) { next(err); }
-    });
-
     // Pivot layouts
     router.get('/:name/layouts', async (req, res, next) => {
         try {

api/routes/stacks.js

@@ -1,201 +0,0 @@
-/**
- * Stacks Routes
- * Named unions of multiple sources with field mappings and running balance
- */
-
-const express = require('express');
-const { lit, arr } = require('../lib/sql');
-
-module.exports = (pool) => {
-    const router = express.Router();
-
-    // List all stacks
-    router.get('/', async (req, res, next) => {
-        try {
-            const result = await pool.query('SELECT * FROM list_stacks()');
-            res.json(result.rows);
-        } catch (err) { next(err); }
-    });
-
-    // Get single stack with sources
-    router.get('/:name', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT * FROM get_stack(${lit(req.params.name)})`);
-            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
-            res.json(result.rows[0]);
-        } catch (err) { next(err); }
-    });
-
-    // Create stack
-    router.post('/', async (req, res, next) => {
-        try {
-            const { name, label, fields, amount_field, date_field, balance_offset } = req.body;
-            if (!name) return res.status(400).json({ error: 'name is required' });
-            const result = await pool.query(
-                `SELECT * FROM create_stack(${lit(name)}, ${lit(label || null)}, ${lit(JSON.stringify(fields || []))}, ${lit(amount_field || null)}, ${lit(date_field || null)}, ${lit(balance_offset ?? 0)})`
-            );
-            res.status(201).json(result.rows[0]);
-        } catch (err) {
-            if (err.code === '23505') return res.status(409).json({ error: 'Stack already exists' });
-            next(err);
-        }
-    });
-
-    // Update stack
-    router.put('/:name', async (req, res, next) => {
-        try {
-            const { label, fields, amount_field, date_field, balance_offset } = req.body;
-            const n = v => v !== undefined ? lit(v) : 'NULL';
-            const f = v => v !== undefined ? lit(JSON.stringify(v)) : 'NULL';
-            const result = await pool.query(
-                `SELECT * FROM update_stack(${lit(req.params.name)}, ${n(label)}, ${f(fields)}, ${n(amount_field)}, ${n(date_field)}, ${n(balance_offset)})`
-            );
-            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
-            res.json(result.rows[0]);
-        } catch (err) { next(err); }
-    });
-
-    // Delete stack
-    router.delete('/:name', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT * FROM delete_stack(${lit(req.params.name)})`);
-            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
-            res.json({ success: true, deleted: req.params.name });
-        } catch (err) { next(err); }
-    });
-
-    // Add or update a source in a stack
-    router.put('/:name/sources/:source', async (req, res, next) => {
-        try {
-            const { field_map, amount_sign, balance_offset, amount_field, date_field } = req.body;
-            const n = v => v != null ? lit(v) : 'NULL';
-            const result = await pool.query(
-                `SELECT * FROM upsert_stack_source(${lit(req.params.name)}, ${lit(req.params.source)}, ${lit(JSON.stringify(field_map || {}))}, ${lit(amount_sign ?? 1)}, ${lit(balance_offset ?? 0)}, ${n(amount_field)}, ${n(date_field)})`
-            );
-            res.json(result.rows[0]);
-        } catch (err) {
-            if (err.code === '23503') return res.status(404).json({ error: 'Stack or source not found' });
-            next(err);
-        }
-    });
-
-    // Remove a source from a stack
-    router.delete('/:name/sources/:source', async (req, res, next) => {
-        try {
-            const result = await pool.query(
-                `SELECT * FROM remove_stack_source(${lit(req.params.name)}, ${lit(req.params.source)})`
-            );
-            if (!result.rows.length) return res.status(404).json({ error: 'Source not in stack' });
-            res.json({ success: true, removed: req.params.source });
-        } catch (err) { next(err); }
-    });
-
-    // Reorder sources within a stack
-    router.put('/:name/sources/reorder', async (req, res, next) => {
-        try {
-            const { source_names } = req.body;
-            if (!Array.isArray(source_names)) return res.status(400).json({ error: 'source_names array required' });
-            await pool.query(`SELECT reorder_stack_sources(${lit(req.params.name)}, ${arr(source_names)})`);
-            res.json({ success: true });
-        } catch (err) { next(err); }
-    });
-
-    // Get current running balance from the generated view
-    router.get('/:name/balance', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT get_stack_balance(${lit(req.params.name)}) AS result`);
-            res.json(result.rows[0].result);
-        } catch (err) { next(err); }
-    });
-
-    // Preview the SQL that would be generated (dry run — does not create the view)
-    router.get('/:name/view-sql', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT generate_stack_view(${lit(req.params.name)}, true) AS result`);
-            res.json(result.rows[0].result);
-        } catch (err) { next(err); }
-    });
-
-    // Generate / refresh the dfv view
-    router.post('/:name/view', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT generate_stack_view(${lit(req.params.name)}) AS result`);
-            const data = result.rows[0].result;
-            if (data && data.success) {
-                await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
-            }
-            res.json(data);
-        } catch (err) { next(err); }
-    });
-
-    // Execute custom SQL for the view (user-edited SQL)
-    router.post('/:name/exec-sql', async (req, res, next) => {
-        try {
-            const { sql } = req.body;
-            if (!sql) return res.status(400).json({ success: false, error: 'sql is required' });
-            await pool.query(`DROP VIEW IF EXISTS dfv.${req.params.name} CASCADE`);
-            await pool.query(sql);
-            await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
-            // Detect stacks whose views were dropped by CASCADE
-            const staleResult = await pool.query(`
-                SELECT array_agg(name) AS names FROM dataflow.stacks
-                WHERE name != ${lit(req.params.name)}
-                  AND view_generated_at IS NOT NULL
-                  AND NOT EXISTS (
-                      SELECT 1 FROM pg_views WHERE schemaname = 'dfv' AND viewname = name
-                  )
-            `);
-            const cascadeStale = staleResult.rows[0].names || [];
-            if (cascadeStale.length) {
-                await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NULL WHERE name = ANY($1)`, [cascadeStale]);
-            }
-            res.json({ success: true, cascade_stale: cascadeStale });
-        } catch (err) {
-            res.json({ success: false, error: err.message });
-        }
-    });
-
-    // Calibrate balance offset given a known good balance at a specific date
-    router.post('/:name/calibrate', async (req, res, next) => {
-        try {
-            const { as_of_date, known_balance, source_name } = req.body;
-            if (known_balance === undefined) {
-                return res.status(400).json({ error: 'known_balance is required' });
-            }
-            const dateExpr = as_of_date ? `${lit(as_of_date)}::date` : 'NULL';
-            const result = await pool.query(
-                `SELECT calibrate_balance(${lit(req.params.name)}, ${source_name ? lit(source_name) : 'NULL'}, ${dateExpr}, ${lit(known_balance)}::numeric) AS result`
-            );
-            res.json(result.rows[0].result);
-        } catch (err) { next(err); }
-    });
-
-    // Pivot layouts (same DB table as sources; FK was dropped to allow stack names)
-    router.get('/:name/layouts', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT * FROM list_pivot_layouts(${lit(req.params.name)})`);
-            res.json(result.rows);
-        } catch (err) { next(err); }
-    });
-
-    router.post('/:name/layouts', async (req, res, next) => {
-        try {
-            const { layout_name, config } = req.body;
-            if (!layout_name || !config) return res.status(400).json({ error: 'layout_name and config required' });
-            const result = await pool.query(
-                `SELECT * FROM save_pivot_layout(${lit(req.params.name)}, ${lit(layout_name)}, ${lit(config)})`
-            );
-            res.json(result.rows[0]);
-        } catch (err) { next(err); }
-    });
-
-    router.delete('/:name/layouts/:id', async (req, res, next) => {
-        try {
-            const result = await pool.query(`SELECT * FROM delete_pivot_layout(${lit(parseInt(req.params.id))})`);
-            if (result.rows.length === 0) return res.status(404).json({ error: 'Layout not found' });
-            res.json({ success: true });
-        } catch (err) { next(err); }
-    });
-
-    return router;
-};

api/routes/status.js

@@ -1,14 +0,0 @@
-const express = require('express');
-
-module.exports = (pool) => {
-    const router = express.Router();
-
-    router.get('/', async (req, res, next) => {
-        try {
-            const result = await pool.query('SELECT get_status() AS result');
-            res.json(result.rows[0].result);
-        } catch (err) { next(err); }
-    });
-
-    return router;
-};

api/server.js

@@ -3,7 +3,7 @@
  * Simple REST API for data transformation
  */
 
-require('dotenv').config({ quiet: true });
+require('dotenv').config();
 const express = require('express');
 const { Pool } = require('pg');
 
@@ -16,8 +16,7 @@ const pool = new Pool({
     port: process.env.DB_PORT,
     database: process.env.DB_NAME,
     user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
+    password: process.env.DB_PASSWORD
-    options: '-c search_path=dataflow,public'
 });
 
 // Middleware
@@ -32,6 +31,11 @@ app.use('/api', auth);
 const path = require('path');
 app.use(express.static(path.join(__dirname, '../public')));
 
+// Set search path for all queries
+pool.on('connect', (client) => {
+    client.query('SET search_path TO dataflow, public');
+});
+
 // Test database connection
 pool.query('SELECT NOW()', (err, res) => {
     if (err) {
@@ -50,16 +54,12 @@ const sourcesRoutes = require('./routes/sources');
 const rulesRoutes = require('./routes/rules');
 const mappingsRoutes = require('./routes/mappings');
 const recordsRoutes = require('./routes/records');
-const stacksRoutes = require('./routes/stacks');
-const statusRoutes = require('./routes/status');
 
 // Mount routes
 app.use('/api/sources', sourcesRoutes(pool));
 app.use('/api/rules', rulesRoutes(pool));
 app.use('/api/mappings', mappingsRoutes(pool));
 app.use('/api/records', recordsRoutes(pool));
-app.use('/api/stacks', stacksRoutes(pool));
-app.use('/api/status', statusRoutes(pool));
 
 // Health check
 app.get('/health', (req, res) => {

database/functions.sql

@@ -0,0 +1,536 @@
+--
+-- Dataflow Functions
+-- Simple, clear functions for import and transformation
+--
+
+SET search_path TO dataflow, public;
+
+------------------------------------------------------
+-- Function: import_records
+-- Import data with automatic deduplication
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION import_records(
+    p_source_name TEXT,
+    p_data JSONB  -- Array of records
+) RETURNS JSON AS $$
+DECLARE
+    v_constraint_fields TEXT[];
+    v_inserted          INTEGER;
+    v_duplicates        INTEGER;
+    v_log_id            INTEGER;
+BEGIN
+    SELECT constraint_fields INTO v_constraint_fields
+    FROM dataflow.sources
+    WHERE name = p_source_name;
+
+    IF v_constraint_fields IS NULL THEN
+        RETURN json_build_object(
+            'success', false,
+            'error', 'Source not found: ' || p_source_name
+        );
+    END IF;
+
+    WITH
+    -- All incoming records with their constraint keys
+    pending AS (
+        SELECT
+            rec.value      AS data,
+            rec.ordinality AS seq,
+            (SELECT jsonb_object_agg(f, rec.value->>f)
+             FROM unnest(v_constraint_fields) AS f) AS constraint_key
+        FROM jsonb_array_elements(p_data) WITH ORDINALITY AS rec
+    ),
+    -- Keys already in the database (excluded)
+    existing AS (
+        SELECT DISTINCT r.constraint_key
+        FROM dataflow.records r
+        INNER JOIN pending p ON p.constraint_key = r.constraint_key
+        WHERE r.source_name = p_source_name
+    ),
+    -- Rows whose constraint key is not yet in the database
+    new_records AS (
+        SELECT p.data, p.constraint_key, p.seq
+        FROM pending p
+        WHERE NOT EXISTS (SELECT 1 FROM existing e WHERE e.constraint_key = p.constraint_key)
+    ),
+    -- Write the log entry
+    log_entry AS (
+        INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate, info)
+        VALUES (
+            p_source_name,
+            (SELECT count(*) FROM new_records),
+            (SELECT count(*) FROM pending) - (SELECT count(*) FROM new_records),
+            jsonb_build_object(
+                'total',          jsonb_array_length(p_data),
+                'inserted_keys',  (SELECT jsonb_agg(constraint_key ORDER BY constraint_key) FROM new_records),
+                'excluded_keys',  (SELECT jsonb_agg(constraint_key) FROM existing)
+            )
+        )
+        RETURNING id, records_imported, records_duplicate
+    ),
+    -- Insert new records
+    inserted AS (
+        INSERT INTO dataflow.records (source_name, data, constraint_key, import_id)
+        SELECT p_source_name, nr.data, nr.constraint_key, (SELECT id FROM log_entry)
+        FROM new_records nr
+        ORDER BY nr.seq
+        RETURNING id
+    )
+    SELECT le.id, le.records_imported, le.records_duplicate
+    INTO   v_log_id, v_inserted, v_duplicates
+    FROM   log_entry le;
+
+    RETURN json_build_object(
+        'success',    true,
+        'imported',   v_inserted,
+        'duplicates', v_duplicates,
+        'log_id',     v_log_id
+    );
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION import_records IS 'Import records with automatic deduplication';
+
+------------------------------------------------------
+-- Function: get_import_log
+-- Return import history for a source
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION get_import_log(p_source_name TEXT)
+RETURNS TABLE (
+    id              INTEGER,
+    source_name     TEXT,
+    records_imported INTEGER,
+    records_duplicate INTEGER,
+    imported_at     TIMESTAMPTZ,
+    info            JSONB
+) AS $$
+    SELECT id, source_name, records_imported, records_duplicate, imported_at, info
+    FROM dataflow.import_log
+    WHERE source_name = p_source_name
+    ORDER BY imported_at DESC;
+$$ LANGUAGE sql;
+
+COMMENT ON FUNCTION get_import_log IS 'Return import history for a source, newest first, including inserted/excluded key lists';
+
+------------------------------------------------------
+-- Function: get_all_import_logs
+-- Return import history across all sources
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION get_all_import_logs()
+RETURNS TABLE (
+    id               INTEGER,
+    source_name      TEXT,
+    records_imported INTEGER,
+    records_duplicate INTEGER,
+    imported_at      TIMESTAMPTZ,
+    info             JSONB
+) AS $$
+    SELECT id, source_name, records_imported, records_duplicate, imported_at, info
+    FROM dataflow.import_log
+    ORDER BY imported_at DESC;
+$$ LANGUAGE sql;
+
+COMMENT ON FUNCTION get_all_import_logs IS 'Return import history across all sources, newest first';
+
+------------------------------------------------------
+-- Function: delete_import
+-- Delete all records from a specific import and remove the log entry
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION delete_import(p_log_id INTEGER)
+RETURNS JSON AS $$
+DECLARE
+    v_deleted INTEGER;
+BEGIN
+    IF NOT EXISTS (SELECT 1 FROM dataflow.import_log WHERE id = p_log_id) THEN
+        RETURN json_build_object('success', false, 'error', 'Import log entry not found');
+    END IF;
+
+    SELECT count(*) INTO v_deleted FROM dataflow.records WHERE import_id = p_log_id;
+
+    -- Cascade handles deleting records via FK ON DELETE CASCADE
+    DELETE FROM dataflow.import_log WHERE id = p_log_id;
+
+    RETURN json_build_object(
+        'success',         true,
+        'records_deleted', v_deleted,
+        'log_id',          p_log_id
+    );
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION delete_import IS 'Delete all records belonging to an import batch and remove the log entry';
+
+------------------------------------------------------
+-- Aggregate: jsonb_concat_obj
+-- Merge JSONB objects across rows (later rows win on key conflicts)
+-- Usage: jsonb_concat_obj(col ORDER BY sequence)
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION dataflow.jsonb_merge(a JSONB, b JSONB)
+RETURNS JSONB AS $$
+    SELECT COALESCE(a, '{}') || COALESCE(b, '{}')
+$$ LANGUAGE sql IMMUTABLE;
+
+DROP AGGREGATE IF EXISTS dataflow.jsonb_concat_obj(JSONB);
+CREATE AGGREGATE dataflow.jsonb_concat_obj(JSONB) (
+    sfunc    = dataflow.jsonb_merge,
+    stype    = JSONB,
+    initcond = '{}'
+);
+
+------------------------------------------------------
+-- Function: apply_transformations
+-- Apply all transformation rules to records (set-based)
+------------------------------------------------------
+DROP FUNCTION IF EXISTS apply_transformations(TEXT, INTEGER[]);
+CREATE OR REPLACE FUNCTION apply_transformations(
+    p_source_name TEXT,
+    p_record_ids INTEGER[] DEFAULT NULL,  -- NULL = all eligible records
+    p_overwrite   BOOLEAN  DEFAULT FALSE  -- FALSE = skip already-transformed, TRUE = overwrite all
+) RETURNS JSON AS $$
+WITH
+-- All records to process
+qualifying AS (
+    SELECT id, data
+    FROM dataflow.records
+    WHERE source_name = p_source_name
+      AND (p_overwrite OR transformed IS NULL)
+      AND (p_record_ids IS NULL OR id = ANY(p_record_ids))
+),
+-- Mirror TPS rx: fan out one row per regex match, drive from rules → records
+rx AS (
+    SELECT
+        q.id,
+        r.name          AS rule_name,
+        r.sequence,
+        r.output_field,
+        r.retain,
+        r.function_type,
+        COALESCE(mt.rn, rp.rn, 1) AS result_number,
+        -- extract: build map_val and retain_val per match (mirrors TPS)
+        CASE WHEN array_length(mt.mt, 1) = 1 THEN to_jsonb(mt.mt[1]) ELSE to_jsonb(mt.mt) END AS match_val,
+        to_jsonb(rp.rp) AS replace_val
+    FROM dataflow.rules r
+    INNER JOIN qualifying q ON q.data ? r.field
+    LEFT JOIN LATERAL regexp_matches(q.data ->> r.field, r.pattern, r.flags)
+        WITH ORDINALITY AS mt(mt, rn) ON r.function_type = 'extract'
+    LEFT JOIN LATERAL regexp_replace(q.data ->> r.field, r.pattern, r.replace_value, r.flags)
+        WITH ORDINALITY AS rp(rp, rn) ON r.function_type = 'replace'
+    WHERE r.source_name = p_source_name
+      AND r.enabled = true
+),
+-- Aggregate match rows back into one value per (record, rule) — mirrors TPS agg_to_target_items
+agg_matches AS (
+    SELECT
+        id,
+        rule_name,
+        sequence,
+        output_field,
+        retain,
+        function_type,
+        CASE function_type
+            WHEN 'replace' THEN jsonb_agg(replace_val) -> 0
+            ELSE
+                CASE WHEN max(result_number) = 1
+                    THEN jsonb_agg(match_val ORDER BY result_number) -> 0
+                    ELSE jsonb_agg(match_val ORDER BY result_number)
+                END
+        END AS extracted
+    FROM rx
+    GROUP BY id, rule_name, sequence, output_field, retain, function_type
+),
+-- Join with mappings to find mapped output — mirrors TPS link_map
+linked AS (
+    SELECT
+        a.id,
+        a.sequence,
+        a.output_field,
+        a.retain,
+        a.extracted,
+        m.output AS mapped
+    FROM agg_matches a
+    LEFT JOIN dataflow.mappings m ON
+        m.source_name = p_source_name
+        AND m.rule_name = a.rule_name
+        AND m.input_value = a.extracted
+    WHERE a.extracted IS NOT NULL
+),
+-- Build per-rule output JSONB:
+--   mapped  → use mapping output; also write output_field if retain = true
+--   no map  → write extracted value to output_field
+rule_output AS (
+    SELECT
+        id,
+        sequence,
+        CASE
+            WHEN mapped IS NOT NULL THEN
+                mapped ||
+                CASE WHEN retain
+                     THEN jsonb_build_object(output_field, extracted)
+                     ELSE '{}'::jsonb
+                END
+            ELSE
+                jsonb_build_object(output_field, extracted)
+        END AS output
+    FROM linked
+),
+-- Merge all rule outputs per record in sequence order — mirrors TPS agg_to_id
+record_additions AS (
+    SELECT
+        id,
+        dataflow.jsonb_concat_obj(output ORDER BY sequence) AS additions
+    FROM rule_output
+    GROUP BY id
+),
+-- Update all qualifying records; records with no rule matches get transformed = data
+updated AS (
+    UPDATE dataflow.records rec
+    SET transformed    = rec.data || COALESCE(ra.additions, '{}'::jsonb),
+        transformed_at = CURRENT_TIMESTAMP
+    FROM qualifying q
+    LEFT JOIN record_additions ra ON ra.id = q.id
+    WHERE rec.id = q.id
+    RETURNING rec.id
+)
+SELECT json_build_object('success', true, 'transformed', count(*))
+FROM updated
+$$ LANGUAGE sql;
+
+COMMENT ON FUNCTION apply_transformations IS 'Apply transformation rules and mappings to records (set-based CTE)';
+
+------------------------------------------------------
+-- Function: get_all_values
+-- All extracted values (mapped + unmapped) with counts and mapping output
+------------------------------------------------------
+DROP FUNCTION IF EXISTS get_all_values(TEXT, TEXT);
+CREATE FUNCTION get_all_values(
+    p_source_name TEXT,
+    p_rule_name TEXT DEFAULT NULL
+) RETURNS TABLE (
+    rule_name TEXT,
+    output_field TEXT,
+    source_field TEXT,
+    extracted_value JSONB,
+    record_count BIGINT,
+    sample JSONB,
+    mapping_id INTEGER,
+    output JSONB,
+    is_mapped BOOLEAN
+) AS $$
+BEGIN
+    RETURN QUERY
+    WITH extracted AS (
+        SELECT
+            r.name AS rule_name,
+            r.output_field,
+            r.field AS source_field,
+            rec.transformed->r.output_field AS extracted_value,
+            rec.data AS record_data,
+            row_number() OVER (
+                PARTITION BY r.name, rec.transformed->r.output_field
+                ORDER BY rec.id
+            ) AS rn
+        FROM dataflow.records rec
+        CROSS JOIN dataflow.rules r
+        WHERE
+            rec.source_name = p_source_name
+            AND r.source_name = p_source_name
+            AND rec.transformed IS NOT NULL
+            AND rec.transformed ? r.output_field
+            AND (p_rule_name IS NULL OR r.name = p_rule_name)
+            AND rec.data ? r.field
+    ),
+    aggregated AS (
+        SELECT
+            e.rule_name,
+            e.output_field,
+            e.source_field,
+            e.extracted_value,
+            count(*) AS record_count,
+            jsonb_agg(e.record_data ORDER BY e.rn) FILTER (WHERE e.rn <= 5) AS sample
+        FROM extracted e
+        GROUP BY e.rule_name, e.output_field, e.source_field, e.extracted_value
+    )
+    SELECT
+        a.rule_name,
+        a.output_field,
+        a.source_field,
+        a.extracted_value,
+        a.record_count,
+        a.sample,
+        m.id AS mapping_id,
+        m.output,
+        (m.id IS NOT NULL) AS is_mapped
+    FROM aggregated a
+    LEFT JOIN dataflow.mappings m ON
+        m.source_name = p_source_name
+        AND m.rule_name = a.rule_name
+        AND m.input_value = a.extracted_value
+    ORDER BY a.record_count DESC;
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION get_all_values IS 'All extracted values with record counts and mapping output (single query for All tab)';
+
+------------------------------------------------------
+-- Function: get_unmapped_values
+-- Find extracted values that need mappings
+------------------------------------------------------
+DROP FUNCTION IF EXISTS get_unmapped_values(TEXT, TEXT);
+CREATE FUNCTION get_unmapped_values(
+    p_source_name TEXT,
+    p_rule_name TEXT DEFAULT NULL
+) RETURNS TABLE (
+    rule_name TEXT,
+    output_field TEXT,
+    source_field TEXT,
+    extracted_value JSONB,
+    record_count BIGINT,
+    sample JSONB
+) AS $$
+BEGIN
+    RETURN QUERY
+    WITH extracted AS (
+        SELECT
+            r.name AS rule_name,
+            r.output_field,
+            r.field AS source_field,
+            rec.transformed->r.output_field AS extracted_value,
+            rec.data AS record_data,
+            row_number() OVER (
+                PARTITION BY r.name, rec.transformed->r.output_field
+                ORDER BY rec.id
+            ) AS rn
+        FROM
+            dataflow.records rec
+            CROSS JOIN dataflow.rules r
+        WHERE
+            rec.source_name = p_source_name
+            AND r.source_name = p_source_name
+            AND rec.transformed IS NOT NULL
+            AND rec.transformed ? r.output_field
+            AND (p_rule_name IS NULL OR r.name = p_rule_name)
+            AND rec.data ? r.field
+    )
+    SELECT
+        e.rule_name,
+        e.output_field,
+        e.source_field,
+        e.extracted_value,
+        count(*) AS record_count,
+        jsonb_agg(e.record_data ORDER BY e.rn) FILTER (WHERE e.rn <= 5) AS sample
+    FROM extracted e
+    WHERE NOT EXISTS (
+        SELECT 1 FROM dataflow.mappings m
+        WHERE m.source_name = p_source_name
+          AND m.rule_name = e.rule_name
+          AND m.input_value = e.extracted_value
+    )
+    GROUP BY e.rule_name, e.output_field, e.source_field, e.extracted_value
+    ORDER BY count(*) DESC;
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION get_unmapped_values IS 'Find extracted values that need mappings defined';
+
+------------------------------------------------------
+-- Function: reprocess_records
+-- Clear and reapply transformations
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION reprocess_records(p_source_name TEXT)
+RETURNS JSON AS $$
+    -- Overwrite all records directly — no clear step, mirrors TPS srce_map_overwrite
+    SELECT dataflow.apply_transformations(p_source_name, NULL, TRUE)
+$$ LANGUAGE sql;
+
+COMMENT ON FUNCTION reprocess_records IS 'Clear and reapply all transformations for a source';
+
+------------------------------------------------------
+-- Function: generate_source_view
+-- Build a typed flat view in dfv schema
+------------------------------------------------------
+CREATE OR REPLACE FUNCTION generate_source_view(p_source_name TEXT)
+RETURNS JSON AS $$
+DECLARE
+    v_config  JSONB;
+    v_fields  JSONB;
+    v_field   JSONB;
+    v_cols    TEXT := '';
+    v_sql     TEXT;
+    v_view    TEXT;
+BEGIN
+    SELECT config INTO v_config
+    FROM dataflow.sources
+    WHERE name = p_source_name;
+
+    IF v_config IS NULL OR NOT (v_config ? 'fields') OR jsonb_array_length(v_config->'fields') = 0 THEN
+        RETURN json_build_object('success', false, 'error', 'No schema fields defined for this source');
+    END IF;
+
+    v_fields := v_config->'fields';
+
+    FOR v_field IN SELECT * FROM jsonb_array_elements(v_fields)
+    LOOP
+        IF v_cols != '' THEN v_cols := v_cols || ', '; END IF;
+
+        IF v_field->>'expression' IS NOT NULL THEN
+            -- Computed expression: substitute {fieldname} refs with (transformed->>'fieldname')::type
+            -- e.g. "{Amount} * {sign}" → "(transformed->>'Amount')::numeric * (transformed->>'sign')::numeric"
+            DECLARE
+                v_expr    TEXT := v_field->>'expression';
+                v_ref     TEXT;
+                v_cast    TEXT := COALESCE(NULLIF(v_field->>'type', ''), 'numeric');
+            BEGIN
+                WHILE v_expr ~ '\{[^}]+\}' LOOP
+                    v_ref  := substring(v_expr FROM '\{([^}]+)\}');
+                    v_expr := replace(v_expr, '{' || v_ref || '}',
+                                      format('(transformed->>%L)::numeric', v_ref));
+                END LOOP;
+                v_cols := v_cols || format('%s AS %I', v_expr, v_field->>'name');
+            END;
+        ELSE
+            CASE v_field->>'type'
+                WHEN 'date' THEN
+                    v_cols := v_cols || format('(transformed->>%L)::date AS %I',
+                        v_field->>'name', v_field->>'name');
+                WHEN 'numeric' THEN
+                    v_cols := v_cols || format('(transformed->>%L)::numeric AS %I',
+                        v_field->>'name', v_field->>'name');
+                ELSE
+                    v_cols := v_cols || format('transformed->>%L AS %I',
+                        v_field->>'name', v_field->>'name');
+            END CASE;
+        END IF;
+    END LOOP;
+
+    CREATE SCHEMA IF NOT EXISTS dfv;
+
+    v_view := 'dfv.' || quote_ident(p_source_name);
+
+    EXECUTE format('DROP VIEW IF EXISTS %s', v_view);
+
+    v_sql := format(
+        'CREATE VIEW %s AS SELECT %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
+        v_view, v_cols, p_source_name
+    );
+
+    EXECUTE v_sql;
+
+    RETURN json_build_object('success', true, 'view', v_view, 'sql', v_sql);
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION generate_source_view IS 'Generate a typed flat view in dfv schema from source config.fields';
+
+------------------------------------------------------
+-- Summary
+------------------------------------------------------
+-- Functions: 4 simple, focused functions
+--   1. import_records - Import with deduplication
+--   2. apply_transformations - Apply rules and mappings
+--   3. get_unmapped_values - Find values needing mappings
+--   4. reprocess_records - Re-transform all records
+--
+-- Each function does ONE thing clearly
+-- No complex nested CTEs
+-- Easy to understand and debug
+------------------------------------------------------

database/import.sql

@@ -1,159 +0,0 @@
---
--- Import queries
--- CSV import and the import audit trail; SQL for the import/log endpoints in
--- api/routes/sources.js
---
-
-SET search_path TO dataflow, public;
-
--- ── Import ────────────────────────────────────────────────────────────────────
-
--- Import records, skipping any whose constraint key already exists in the table.
---
--- Dedup is enforced here, in the CTE — there is no unique constraint on
--- constraint_key and ON CONFLICT must never be used. Within one batch every row
--- inserts even if two rows share a constraint key, because banks legitimately send
--- identical-looking transactions (same date, description, amount) on the same day.
--- The key exists only to stop a re-imported overlapping date range from
--- double-counting rows already in the table.
-CREATE OR REPLACE FUNCTION import_records(
-    p_source_name TEXT,
-    p_data JSONB  -- Array of records
-) RETURNS JSON AS $$
-DECLARE
-    v_constraint_fields TEXT[];
-    v_inserted          INTEGER;
-    v_duplicates        INTEGER;
-    v_log_id            INTEGER;
-BEGIN
-    SELECT constraint_fields INTO v_constraint_fields
-    FROM dataflow.sources
-    WHERE name = p_source_name;
-
-    IF v_constraint_fields IS NULL THEN
-        RETURN json_build_object(
-            'success', false,
-            'error', 'Source not found: ' || p_source_name
-        );
-    END IF;
-
-    WITH
-    -- All incoming records with their constraint keys
-    pending AS (
-        SELECT
-            rec.value      AS data,
-            rec.ordinality AS seq,
-            (SELECT jsonb_object_agg(f, rec.value->>f)
-             FROM unnest(v_constraint_fields) AS f) AS constraint_key
-        FROM jsonb_array_elements(p_data) WITH ORDINALITY AS rec
-    ),
-    -- Keys already in the database (excluded)
-    existing AS (
-        SELECT DISTINCT r.constraint_key
-        FROM dataflow.records r
-        INNER JOIN pending p ON p.constraint_key = r.constraint_key
-        WHERE r.source_name = p_source_name
-    ),
-    -- Rows whose constraint key is not yet in the database
-    new_records AS (
-        SELECT p.data, p.constraint_key, p.seq
-        FROM pending p
-        WHERE NOT EXISTS (SELECT 1 FROM existing e WHERE e.constraint_key = p.constraint_key)
-    ),
-    -- Write the log entry
-    log_entry AS (
-        INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate, info)
-        VALUES (
-            p_source_name,
-            (SELECT count(*) FROM new_records),
-            (SELECT count(*) FROM pending) - (SELECT count(*) FROM new_records),
-            jsonb_build_object(
-                'total',          jsonb_array_length(p_data),
-                'inserted_keys',  (SELECT jsonb_agg(constraint_key ORDER BY constraint_key) FROM new_records),
-                'excluded_keys',  (SELECT jsonb_agg(constraint_key) FROM existing)
-            )
-        )
-        RETURNING id, records_imported, records_duplicate
-    ),
-    -- Insert new records
-    inserted AS (
-        INSERT INTO dataflow.records (source_name, data, constraint_key, import_id)
-        SELECT p_source_name, nr.data, nr.constraint_key, (SELECT id FROM log_entry)
-        FROM new_records nr
-        ORDER BY nr.seq
-        RETURNING id
-    )
-    SELECT le.id, le.records_imported, le.records_duplicate
-    INTO   v_log_id, v_inserted, v_duplicates
-    FROM   log_entry le;
-
-    RETURN json_build_object(
-        'success',    true,
-        'imported',   v_inserted,
-        'duplicates', v_duplicates,
-        'log_id',     v_log_id
-    );
-END;
-$$ LANGUAGE plpgsql;
-
-COMMENT ON FUNCTION import_records IS 'Import records with automatic deduplication';
-
--- ── Audit trail ───────────────────────────────────────────────────────────────
-
-CREATE OR REPLACE FUNCTION get_import_log(p_source_name TEXT)
-RETURNS TABLE (
-    id              INTEGER,
-    source_name     TEXT,
-    records_imported INTEGER,
-    records_duplicate INTEGER,
-    imported_at     TIMESTAMPTZ,
-    info            JSONB
-) AS $$
-    SELECT id, source_name, records_imported, records_duplicate, imported_at, info
-    FROM dataflow.import_log
-    WHERE source_name = p_source_name
-    ORDER BY imported_at DESC;
-$$ LANGUAGE sql;
-
-COMMENT ON FUNCTION get_import_log IS 'Return import history for a source, newest first, including inserted/excluded key lists';
-
-CREATE OR REPLACE FUNCTION get_all_import_logs()
-RETURNS TABLE (
-    id               INTEGER,
-    source_name      TEXT,
-    records_imported INTEGER,
-    records_duplicate INTEGER,
-    imported_at      TIMESTAMPTZ,
-    info             JSONB
-) AS $$
-    SELECT id, source_name, records_imported, records_duplicate, imported_at, info
-    FROM dataflow.import_log
-    ORDER BY imported_at DESC;
-$$ LANGUAGE sql;
-
-COMMENT ON FUNCTION get_all_import_logs IS 'Return import history across all sources, newest first';
-
--- Records are removed by the import_id FK's ON DELETE CASCADE
-CREATE OR REPLACE FUNCTION delete_import(p_log_id INTEGER)
-RETURNS JSON AS $$
-DECLARE
-    v_deleted INTEGER;
-BEGIN
-    IF NOT EXISTS (SELECT 1 FROM dataflow.import_log WHERE id = p_log_id) THEN
-        RETURN json_build_object('success', false, 'error', 'Import log entry not found');
-    END IF;
-
-    SELECT count(*) INTO v_deleted FROM dataflow.records WHERE import_id = p_log_id;
-
-    -- Cascade handles deleting records via FK ON DELETE CASCADE
-    DELETE FROM dataflow.import_log WHERE id = p_log_id;
-
-    RETURN json_build_object(
-        'success',         true,
-        'records_deleted', v_deleted,
-        'log_id',          p_log_id
-    );
-END;
-$$ LANGUAGE plpgsql;
-
-COMMENT ON FUNCTION delete_import IS 'Delete all records belonging to an import batch and remove the log entry';

database/migrate_input_value_jsonb.sql

@@ -0,0 +1,22 @@
+--
+-- Migration: Change mappings.input_value from TEXT to JSONB
+-- Allows multi-capture-group regex results to be used as mapping keys
+--
+
+SET search_path TO dataflow, public;
+
+-- Drop dependent constraint and index first
+ALTER TABLE dataflow.mappings DROP CONSTRAINT mappings_source_name_rule_name_input_value_key;
+DROP INDEX IF EXISTS dataflow.idx_mappings_input;
+
+-- Convert column: existing TEXT values become JSONB strings e.g. "MEIJER"
+ALTER TABLE dataflow.mappings
+    ALTER COLUMN input_value TYPE JSONB
+    USING to_jsonb(input_value);
+
+-- Recreate constraint and index
+ALTER TABLE dataflow.mappings
+    ADD CONSTRAINT mappings_source_name_rule_name_input_value_key
+    UNIQUE (source_name, rule_name, input_value);
+
+CREATE INDEX idx_mappings_input ON dataflow.mappings(source_name, rule_name, input_value);

database/migrate_tps.sql

@@ -0,0 +1,121 @@
+--
+-- TPS → Dataflow Migration
+--
+-- Migrates sources, rules, mappings, and records from the TPS system.
+-- Run against the dataflow database:
+--   PGPASSWORD=dataflow psql -U dataflow -d dataflow -h localhost -f database/migrate_tps.sql
+--
+-- Existing rows are skipped (ON CONFLICT DO NOTHING) so the script is safe to re-run.
+-- NOTE: dcard already configured in dataflow will NOT be overwritten.
+--
+
+SET search_path TO dataflow, public;
+
+CREATE EXTENSION IF NOT EXISTS dblink;
+
+-- Connection string to the TPS database
+\set tps_conn 'host=192.168.1.110 dbname=ubm user=api password=gyaswddh1983'
+
+\echo ''
+\echo '=== 1. Sources ==='
+
+INSERT INTO dataflow.sources (name, constraint_fields, config)
+SELECT
+    srce AS name,
+    -- Strip {} wrappers from constraint paths → constraint field names
+    ARRAY(
+        SELECT regexp_replace(c, '^\{|\}$', '', 'g')
+        FROM jsonb_array_elements_text(defn->'constraint') AS c
+    ) AS constraint_fields,
+    -- Build config.fields from the first schema (index 0 = "mapped" for dcard, "default" for others)
+    jsonb_build_object('fields',
+        (SELECT jsonb_agg(
+            jsonb_build_object(
+                'name', regexp_replace(col->>'path', '^\{|\}$', '', 'g'),
+                'type', COALESCE(NULLIF(col->>'type', ''), 'text')
+            ) ORDER BY ord
+        )
+        FROM jsonb_array_elements(defn->'schemas'->0->'columns')
+             WITH ORDINALITY AS t(col, ord)
+        )
+    ) AS config
+FROM dblink(:'tps_conn',
+    'SELECT srce, defn FROM tps.srce'
+) AS t(srce TEXT, defn JSONB)
+ON CONFLICT (name) DO NOTHING;
+
+SELECT name, constraint_fields, jsonb_array_length(config->'fields') AS field_count
+FROM dataflow.sources ORDER BY name;
+
+\echo ''
+\echo '=== 2. Rules ==='
+
+INSERT INTO dataflow.rules
+    (source_name, name, field, pattern, output_field, function_type, flags, replace_value, sequence, enabled, retain)
+SELECT
+    srce                                                                AS source_name,
+    target                                                              AS name,
+    -- Strip {} from the input field key
+    regexp_replace(regex->'regex'->'defn'->0->>'key', '^\{|\}$', '', 'g') AS field,
+    regex->'regex'->'defn'->0->>'regex'                                AS pattern,
+    regex->'regex'->'defn'->0->>'field'                                AS output_field,
+    COALESCE(NULLIF(regex->'regex'->>'function', ''), 'extract')       AS function_type,
+    COALESCE(regex->'regex'->'defn'->0->>'flag', '')                   AS flags,
+    ''                                                                  AS replace_value,
+    seq                                                                 AS sequence,
+    true                                                                AS enabled,
+    (regex->'regex'->'defn'->0->>'retain') = 'y'                       AS retain
+FROM dblink(:'tps_conn',
+    'SELECT srce, target, seq, regex FROM tps.map_rm'
+) AS t(srce TEXT, target TEXT, seq INT, regex JSONB)
+ON CONFLICT (source_name, name) DO NOTHING;
+
+SELECT source_name, name, field, pattern, output_field, sequence
+FROM dataflow.rules ORDER BY source_name, sequence;
+
+\echo ''
+\echo '=== 3. Mappings ==='
+
+INSERT INTO dataflow.mappings (source_name, rule_name, input_value, output)
+SELECT
+    srce        AS source_name,
+    target      AS rule_name,
+    -- retval is {"f20": "<extracted string>"} — pull out the value as JSONB
+    (SELECT value FROM jsonb_each(retval) LIMIT 1) AS input_value,
+    map         AS output
+FROM dblink(:'tps_conn',
+    'SELECT srce, target, retval, map FROM tps.map_rv'
+) AS t(srce TEXT, target TEXT, retval JSONB, map JSONB)
+ON CONFLICT (source_name, rule_name, input_value) DO NOTHING;
+
+SELECT source_name, rule_name, COUNT(*) AS mapping_count
+FROM dataflow.mappings GROUP BY source_name, rule_name ORDER BY source_name, rule_name;
+
+\echo ''
+\echo '=== 4. Records ==='
+\echo '    (13 000+ rows — may take a moment)'
+
+INSERT INTO dataflow.records (source_name, data, constraint_key, transformed, imported_at, transformed_at)
+SELECT
+    t.srce                                                  AS source_name,
+    t.rec                                                   AS data,
+    (SELECT jsonb_object_agg(f, t.rec->>f) FROM unnest(s.constraint_fields) AS f) AS constraint_key,
+    t.allj                                                  AS transformed,
+    CURRENT_TIMESTAMP                                       AS imported_at,
+    CASE WHEN t.allj IS NOT NULL THEN CURRENT_TIMESTAMP END AS transformed_at
+FROM dblink(:'tps_conn',
+    'SELECT srce, rec, allj FROM tps.trans'
+) AS t(srce TEXT, rec JSONB, allj JSONB)
+JOIN dataflow.sources s ON s.name = t.srce
+ON CONFLICT (source_name, constraint_key) DO NOTHING;
+
+SELECT source_name, COUNT(*) AS records, COUNT(transformed) AS transformed
+FROM dataflow.records GROUP BY source_name ORDER BY source_name;
+
+\echo ''
+\echo '=== Migration complete ==='
+SELECT
+    (SELECT COUNT(*) FROM dataflow.sources)  AS sources,
+    (SELECT COUNT(*) FROM dataflow.rules)    AS rules,
+    (SELECT COUNT(*) FROM dataflow.mappings) AS mappings,
+    (SELECT COUNT(*) FROM dataflow.records)  AS records;

database/queries/records.sql

@@ -39,49 +39,6 @@ RETURNS SETOF dataflow.records AS $$
     LIMIT p_limit;
 $$ LANGUAGE sql STABLE;
 
--- ── Overrides ─────────────────────────────────────────────────────────────────
-
--- Store manual overrides. Overrides stay in their own column — never merged into
--- transformed — so reprocessing rules cannot clobber a manual edit.
-DROP FUNCTION IF EXISTS set_record_overrides(INTEGER, JSONB);
-CREATE OR REPLACE FUNCTION set_record_overrides(p_id INTEGER, p_overrides JSONB)
-RETURNS JSON AS $$
-    WITH updated AS (
-        UPDATE dataflow.records
-        SET overrides = CASE WHEN p_overrides = '{}'::jsonb THEN NULL ELSE p_overrides END
-        WHERE id = p_id
-        RETURNING *
-    )
-    SELECT row_to_json(updated) FROM updated;
-$$ LANGUAGE sql;
-
--- Merge overrides into multiple records at once; returns actual updated count
-DROP FUNCTION IF EXISTS bulk_set_record_overrides(TEXT, INTEGER[], JSONB);
-CREATE OR REPLACE FUNCTION bulk_set_record_overrides(p_source_name TEXT, p_ids INTEGER[], p_overrides JSONB)
-RETURNS JSON AS $$
-    WITH updated AS (
-        UPDATE dataflow.records
-        SET overrides = COALESCE(overrides, '{}'::jsonb) || p_overrides
-        WHERE id = ANY(p_ids)
-          AND source_name = p_source_name
-        RETURNING id
-    )
-    SELECT json_build_object('updated', count(*)) FROM updated;
-$$ LANGUAGE sql;
-
--- Clear overrides; the computed values in transformed are untouched
-DROP FUNCTION IF EXISTS clear_record_overrides(INTEGER);
-CREATE OR REPLACE FUNCTION clear_record_overrides(p_id INTEGER)
-RETURNS JSON AS $$
-    WITH updated AS (
-        UPDATE dataflow.records
-        SET overrides = NULL
-        WHERE id = p_id
-        RETURNING *
-    )
-    SELECT row_to_json(updated) FROM updated;
-$$ LANGUAGE sql;
-
 -- ── Delete ────────────────────────────────────────────────────────────────────
 
 CREATE OR REPLACE FUNCTION delete_record(p_id BIGINT)

database/queries/rules.sql

@@ -86,27 +86,21 @@ CREATE OR REPLACE FUNCTION preview_rule(
     p_limit         INT  DEFAULT 20
 )
 RETURNS TABLE (id INT, raw_value TEXT, extracted_value JSONB) AS $$
--- Field is resolved from data first, then transformed (supports chained rules whose
--- input field was produced by an earlier-sequence rule rather than the raw import).
 BEGIN
     IF p_function_type = 'replace' THEN
         RETURN QUERY
         SELECT
             r.id,
-            COALESCE(r.data ->> p_field, r.transformed ->> p_field),
+            r.data ->> p_field,
-            to_jsonb(regexp_replace(
+            to_jsonb(regexp_replace(r.data ->> p_field, p_pattern, p_replace_value, p_flags))
-                COALESCE(r.data ->> p_field, r.transformed ->> p_field),
-                p_pattern, p_replace_value, p_flags
-            ))
         FROM dataflow.records r
-        WHERE source_name = p_source
+        WHERE source_name = p_source AND data ? p_field
-          AND (data ? p_field OR transformed ? p_field)
         ORDER BY r.id DESC LIMIT p_limit;
     ELSE
         RETURN QUERY
         SELECT
             r.id,
-            COALESCE(r.data ->> p_field, r.transformed ->> p_field),
+            r.data ->> p_field,
             CASE
                 WHEN agg.match_count = 0 THEN NULL
                 WHEN agg.match_count = 1 THEN agg.matches -> 0
@@ -120,14 +114,10 @@ BEGIN
                     ORDER BY rn
                 ) AS matches,
                 count(*)::int AS match_count
-            FROM regexp_matches(
+            FROM regexp_matches(r.data ->> p_field, p_pattern, p_flags)
-                COALESCE(r.data ->> p_field, r.transformed ->> p_field),
-                p_pattern, p_flags
-            )
             WITH ORDINALITY AS m(mt, rn)
         ) agg
-        WHERE r.source_name = p_source
+        WHERE r.source_name = p_source AND r.data ? p_field
-          AND (r.data ? p_field OR r.transformed ? p_field)
         ORDER BY r.id DESC LIMIT p_limit;
     END IF;
 END;

database/queries/sources.sql

@@ -40,6 +40,8 @@ RETURNS TEXT AS $$
     DELETE FROM dataflow.sources WHERE name = p_name RETURNING name;
 $$ LANGUAGE sql;
 
+-- ── Import log ────────────────────────────────────────────────────────────────
+
 -- ── Stats ─────────────────────────────────────────────────────────────────────
 
 CREATE OR REPLACE FUNCTION get_source_stats(p_source_name TEXT)
@@ -159,7 +161,6 @@ BEGIN
         RETURN json_build_object('success', false, 'error', 'No schema fields defined for this source');
     END IF;
 
-    -- Columns read from r, the merged data || transformed || overrides object
     FOR v_field IN SELECT * FROM jsonb_array_elements(v_config->'fields') LOOP
         IF v_cols != '' THEN v_cols := v_cols || ', '; END IF;
 
@@ -170,27 +171,24 @@ BEGIN
             BEGIN
                 WHILE v_expr ~ '\{[^}]+\}' LOOP
                     v_ref  := substring(v_expr FROM '\{([^}]+)\}');
-                    v_expr := replace(v_expr, '{' || v_ref || '}', format('(r->>%L)::numeric', v_ref));
+                    v_expr := replace(v_expr, '{' || v_ref || '}', format('(transformed->>%L)::numeric', v_ref));
                 END LOOP;
                 v_cols := v_cols || format('%s AS %I', v_expr, v_field->>'name');
             END;
         ELSE
             CASE v_field->>'type'
-                WHEN 'date'    THEN v_cols := v_cols || format('(r->>%L)::date    AS %I', v_field->>'name', v_field->>'name');
+                WHEN 'date'    THEN v_cols := v_cols || format('(transformed->>%L)::date    AS %I', v_field->>'name', v_field->>'name');
-                WHEN 'numeric' THEN v_cols := v_cols || format('(r->>%L)::numeric AS %I', v_field->>'name', v_field->>'name');
+                WHEN 'numeric' THEN v_cols := v_cols || format('(transformed->>%L)::numeric AS %I', v_field->>'name', v_field->>'name');
-                ELSE                v_cols := v_cols || format('r->>%L             AS %I', v_field->>'name', v_field->>'name');
+                ELSE                v_cols := v_cols || format('transformed->>%L             AS %I', v_field->>'name', v_field->>'name');
             END CASE;
         END IF;
     END LOOP;
 
     CREATE SCHEMA IF NOT EXISTS dfv;
     v_view := 'dfv.' || quote_ident(p_source_name);
-    EXECUTE format('DROP VIEW IF EXISTS %s CASCADE', v_view);
+    EXECUTE format('DROP VIEW IF EXISTS %s', v_view);
     v_sql := format(
-        'CREATE VIEW %s AS SELECT id, _overridden, %s FROM ('
+        'CREATE VIEW %s AS SELECT %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
-        || 'SELECT id, overrides IS NOT NULL AS _overridden, '
-        || 'data || COALESCE(transformed, ''{}''::jsonb) || COALESCE(overrides, ''{}''::jsonb) AS r '
-        || 'FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL) rec',
         v_view, v_cols, p_source_name
     );
     EXECUTE v_sql;

database/schema.sql

@@ -37,27 +37,26 @@ CREATE TABLE records (
     -- Data
     data JSONB NOT NULL,                    -- Original imported data
     constraint_key JSONB,                   -- Fields that uniquely identify this record (set on import)
-    transformed JSONB,                      -- Rule/mapping output fields only (delta, not raw data)
+    transformed JSONB,                      -- Data after transformations applied
-    overrides JSONB,                        -- Manual user overrides (highest precedence)
 
     -- Metadata
-    import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE,
+    import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE,  -- Which import batch this came from
     imported_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
-    transformed_at TIMESTAMPTZ
+    transformed_at TIMESTAMPTZ,
+
+
 );
 
 COMMENT ON TABLE records IS 'Imported records with raw and transformed data';
-COMMENT ON COLUMN records.data IS 'Original data as imported — never mutated after import';
+COMMENT ON COLUMN records.data IS 'Original data as imported';
 COMMENT ON COLUMN records.constraint_key IS 'JSONB object of constraint field values — uniquely identifies this record within its source';
-COMMENT ON COLUMN records.transformed IS 'Rule/mapping output fields only (delta); merge as data || transformed || overrides for final values';
+COMMENT ON COLUMN records.transformed IS 'Data after applying transformation rules';
-COMMENT ON COLUMN records.overrides IS 'Manual user overrides; highest precedence in data || transformed || overrides merge';
 
 -- Indexes
 CREATE INDEX idx_records_source ON records(source_name);
 CREATE INDEX idx_records_constraint ON records USING gin(constraint_key);
 CREATE INDEX idx_records_data ON records USING gin(data);
 CREATE INDEX idx_records_transformed ON records USING gin(transformed);
-CREATE INDEX idx_records_overrides ON records USING gin(overrides) WHERE overrides IS NOT NULL;
 
 ------------------------------------------------------
 -- Table: rules

database/stacks.sql

@@ -1,472 +0,0 @@
---
--- Stacks queries
--- All SQL for api/routes/stacks.js
---
-
-SET search_path TO dataflow, public;
-
-------------------------------------------------------
--- Tables
-------------------------------------------------------
-
-CREATE TABLE IF NOT EXISTS dataflow.stacks (
-    name            TEXT PRIMARY KEY,
-    label           TEXT,
-    -- Ordered canonical field definitions: [{name, label, type}]
-    -- type: 'text' | 'numeric' | 'date'
-    fields          JSONB NOT NULL DEFAULT '[]',
-    -- Running balance config
-    amount_field    TEXT,           -- canonical field to sum for running balance
-    date_field      TEXT,           -- canonical field to order by
-    balance_offset  NUMERIC DEFAULT 0,  -- added to running sum (calibration)
-    created_at      TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE IF NOT EXISTS dataflow.stack_sources (
-    id              SERIAL PRIMARY KEY,
-    stack_name      TEXT NOT NULL REFERENCES dataflow.stacks(name) ON DELETE CASCADE,
-    source_name     TEXT NOT NULL REFERENCES dataflow.sources(name) ON DELETE CASCADE,
-    -- Maps other canonical field names → source view column names (not amount/date — those are explicit)
-    field_map       JSONB NOT NULL DEFAULT '{}',
-    -- Which column in dfv.{source} is the amount, and its sign (+1/-1)
-    amount_field    TEXT,
-    amount_sign     INTEGER NOT NULL DEFAULT 1,
-    -- Which column in dfv.{source} is the date
-    date_field      TEXT,
-    -- Calibration offset added to this source's running balance
-    balance_offset  NUMERIC NOT NULL DEFAULT 0,
-    UNIQUE (stack_name, source_name)
-);
-
--- Migrations: add columns that may be missing from earlier deploys
-ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS balance_offset NUMERIC NOT NULL DEFAULT 0;
-ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS amount_field TEXT;
-ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS date_field TEXT;
-ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS seq INTEGER NOT NULL DEFAULT 0;
-
--- Seed seq from insertion order for existing rows
-UPDATE dataflow.stack_sources ss
-SET seq = sub.rn
-FROM (
-    SELECT id, ROW_NUMBER() OVER (PARTITION BY stack_name ORDER BY id) AS rn
-    FROM dataflow.stack_sources WHERE seq = 0
-) sub
-WHERE ss.id = sub.id AND ss.seq = 0;
-
--- Drop old signatures before recreating
-DROP FUNCTION IF EXISTS calibrate_balance(TEXT, DATE, NUMERIC);
-DROP FUNCTION IF EXISTS upsert_stack_source(TEXT, TEXT, JSONB, INTEGER, NUMERIC);
-DROP FUNCTION IF EXISTS generate_stack_view(TEXT);
-
-------------------------------------------------------
--- Function: list_stacks
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION list_stacks()
-RETURNS TABLE (
-    name            TEXT,
-    label           TEXT,
-    fields          JSONB,
-    amount_field    TEXT,
-    date_field      TEXT,
-    balance_offset  NUMERIC,
-    source_count    BIGINT,
-    created_at      TIMESTAMPTZ
-) AS $$
-    SELECT
-        s.name, s.label, s.fields,
-        s.amount_field, s.date_field, s.balance_offset,
-        count(ss.id) AS source_count,
-        s.created_at
-    FROM dataflow.stacks s
-    LEFT JOIN dataflow.stack_sources ss ON ss.stack_name = s.name
-    GROUP BY s.name, s.label, s.fields, s.amount_field, s.date_field, s.balance_offset, s.created_at
-    ORDER BY s.name;
-$$ LANGUAGE sql STABLE;
-
-------------------------------------------------------
--- Function: get_stack
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION get_stack(p_name TEXT)
-RETURNS TABLE (
-    name            TEXT,
-    label           TEXT,
-    fields          JSONB,
-    amount_field    TEXT,
-    date_field      TEXT,
-    balance_offset  NUMERIC,
-    created_at      TIMESTAMPTZ,
-    sources         JSONB
-) AS $$
-    SELECT
-        s.name, s.label, s.fields,
-        s.amount_field, s.date_field, s.balance_offset,
-        s.created_at,
-        COALESCE(jsonb_agg(
-            jsonb_build_object(
-                'id',             ss.id,
-                'source_name',    ss.source_name,
-                'field_map',      ss.field_map,
-                'amount_field',   ss.amount_field,
-                'amount_sign',    ss.amount_sign,
-                'date_field',     ss.date_field,
-                'balance_offset', ss.balance_offset,
-                'seq',            ss.seq
-            ) ORDER BY ss.seq, ss.id
-        ) FILTER (WHERE ss.id IS NOT NULL), '[]')
-    FROM dataflow.stacks s
-    LEFT JOIN dataflow.stack_sources ss ON ss.stack_name = s.name
-    WHERE s.name = p_name
-    GROUP BY s.name, s.label, s.fields, s.amount_field, s.date_field, s.balance_offset, s.created_at;
-$$ LANGUAGE sql STABLE;
-
-------------------------------------------------------
--- Function: create_stack
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION create_stack(
-    p_name          TEXT,
-    p_label         TEXT DEFAULT NULL,
-    p_fields        JSONB DEFAULT '[]',
-    p_amount_field  TEXT DEFAULT NULL,
-    p_date_field    TEXT DEFAULT NULL,
-    p_balance_offset NUMERIC DEFAULT 0
-) RETURNS dataflow.stacks AS $$
-    INSERT INTO dataflow.stacks (name, label, fields, amount_field, date_field, balance_offset)
-    VALUES (p_name, p_label, p_fields, p_amount_field, p_date_field, p_balance_offset)
-    RETURNING *;
-$$ LANGUAGE sql;
-
-------------------------------------------------------
--- Function: update_stack
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION update_stack(
-    p_name           TEXT,
-    p_label          TEXT DEFAULT NULL,
-    p_fields         JSONB DEFAULT NULL,
-    p_amount_field   TEXT DEFAULT NULL,
-    p_date_field     TEXT DEFAULT NULL,
-    p_balance_offset NUMERIC DEFAULT NULL
-) RETURNS dataflow.stacks AS $$
-    UPDATE dataflow.stacks SET
-        label          = COALESCE(p_label, label),
-        fields         = COALESCE(p_fields, fields),
-        amount_field   = COALESCE(p_amount_field, amount_field),
-        date_field     = COALESCE(p_date_field, date_field),
-        balance_offset = COALESCE(p_balance_offset, balance_offset)
-    WHERE name = p_name
-    RETURNING *;
-$$ LANGUAGE sql;
-
-------------------------------------------------------
--- Function: delete_stack
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION delete_stack(p_name TEXT)
-RETURNS TABLE (name TEXT) AS $$
-    DELETE FROM dataflow.stacks WHERE name = p_name RETURNING name;
-$$ LANGUAGE sql;
-
-------------------------------------------------------
--- Function: upsert_stack_source
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION upsert_stack_source(
-    p_stack_name      TEXT,
-    p_source_name     TEXT,
-    p_field_map       JSONB DEFAULT '{}',
-    p_amount_sign     INTEGER DEFAULT 1,
-    p_balance_offset  NUMERIC DEFAULT 0,
-    p_amount_field    TEXT DEFAULT NULL,
-    p_date_field      TEXT DEFAULT NULL
-) RETURNS dataflow.stack_sources AS $$
-    INSERT INTO dataflow.stack_sources (stack_name, source_name, field_map, amount_sign, balance_offset, amount_field, date_field, seq)
-    VALUES (
-        p_stack_name, p_source_name, p_field_map, p_amount_sign, p_balance_offset, p_amount_field, p_date_field,
-        (SELECT COALESCE(MAX(seq), 0) + 1 FROM dataflow.stack_sources WHERE stack_name = p_stack_name)
-    )
-    ON CONFLICT (stack_name, source_name) DO UPDATE SET
-        field_map       = EXCLUDED.field_map,
-        amount_sign     = EXCLUDED.amount_sign,
-        balance_offset  = EXCLUDED.balance_offset,
-        amount_field    = EXCLUDED.amount_field,
-        date_field      = EXCLUDED.date_field
-    RETURNING *;
-$$ LANGUAGE sql;
-
-------------------------------------------------------
--- Function: remove_stack_source
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION remove_stack_source(p_stack_name TEXT, p_source_name TEXT)
-RETURNS TABLE (source_name TEXT) AS $$
-    DELETE FROM dataflow.stack_sources
-    WHERE stack_name = p_stack_name AND source_name = p_source_name
-    RETURNING source_name;
-$$ LANGUAGE sql;
-
-------------------------------------------------------
--- Function: calibrate_balance
--- Queries dfv.{source} directly using per-source amount/date fields.
--- No stack view required.
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION calibrate_balance(
-    p_stack_name    TEXT,
-    p_source_name   TEXT,
-    p_as_of_date    DATE,
-    p_known_balance NUMERIC
-) RETURNS JSON AS $$
-DECLARE
-    v_src       dataflow.stack_sources%ROWTYPE;
-    v_running   NUMERIC;
-    v_sql       TEXT;
-BEGIN
-    SELECT * INTO v_src
-    FROM dataflow.stack_sources
-    WHERE stack_name = p_stack_name AND source_name = p_source_name;
-
-    IF NOT FOUND THEN
-        RETURN json_build_object('success', false, 'error', 'Source not in stack');
-    END IF;
-    IF v_src.amount_field IS NULL OR v_src.date_field IS NULL THEN
-        RETURN json_build_object('success', false, 'error', 'Set amount and date fields on this source first');
-    END IF;
-
-    BEGIN
-        IF p_as_of_date IS NULL THEN
-            v_sql := format(
-                'SELECT COALESCE(SUM(%I * %s), 0) FROM dfv.%I',
-                v_src.amount_field, v_src.amount_sign, p_source_name
-            );
-        ELSE
-            v_sql := format(
-                'SELECT COALESCE(SUM(%I * %s), 0) FROM dfv.%I WHERE %I <= %L::date',
-                v_src.amount_field, v_src.amount_sign, p_source_name, v_src.date_field, p_as_of_date
-            );
-        END IF;
-        EXECUTE v_sql INTO v_running;
-    EXCEPTION WHEN undefined_table THEN
-        RETURN json_build_object('success', false, 'error', 'Source view not found — generate the source view first');
-    END;
-
-    RETURN json_build_object(
-        'success',          true,
-        'source',           p_source_name,
-        'as_of_date',       p_as_of_date,
-        'known_balance',    p_known_balance,
-        'computed_sum',     v_running,
-        'suggested_offset', p_known_balance - v_running
-    );
-END;
-$$ LANGUAGE plpgsql STABLE;
-
-------------------------------------------------------
--- Function: generate_stack_view
--- Builds a WITH ... UNION ALL view in dfv schema from existing dfv source views.
--- Each source CTE applies amount_sign and computes a per-source running balance.
--- Outer SELECT adds net_balance across all sources.
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION generate_stack_view(p_stack_name TEXT, p_dry_run BOOLEAN DEFAULT false)
-RETURNS JSON AS $$
-DECLARE
-    v_stack      dataflow.stacks%ROWTYPE;
-    v_src        dataflow.stack_sources%ROWTYPE;
-    v_field      JSONB;
-    v_ctes       TEXT[] := '{}';
-    v_cte_names  TEXT[] := '{}';
-    v_select     TEXT;
-    v_col        TEXT;
-    v_src_field  TEXT;
-    v_amt_src    TEXT;
-    v_date_src   TEXT;
-    v_view          TEXT;
-    v_sql           TEXT;
-    v_has_bal       BOOLEAN;
-    v_canon_cols    TEXT;
-    v_src_bal_cols  TEXT;
-    v_total_offset  NUMERIC := 0;
-    v_cascade_stale TEXT[];
-BEGIN
-    SELECT * INTO v_stack FROM dataflow.stacks WHERE name = p_stack_name;
-    IF NOT FOUND THEN
-        RETURN json_build_object('success', false, 'error', 'Stack not found');
-    END IF;
-
-    v_has_bal := v_stack.amount_field IS NOT NULL AND v_stack.date_field IS NOT NULL;
-
-    -- Build one CTE per source querying dfv.{source} directly
-    FOR v_src IN
-        SELECT * FROM dataflow.stack_sources WHERE stack_name = p_stack_name ORDER BY seq, id
-    LOOP
-        v_select := format('SELECT %L AS _source, id AS _id', v_src.source_name);
-
-        FOR v_field IN SELECT * FROM jsonb_array_elements(v_stack.fields)
-        LOOP
-            v_col := v_field->>'name';
-
-            IF v_has_bal AND v_col = v_stack.amount_field THEN
-                -- Use per-source amount_field with sign applied
-                IF v_src.amount_field IS NULL THEN
-                    v_select := v_select || format(', NULL::%s AS %I', v_field->>'type', v_col);
-                ELSE
-                    v_select := v_select || format(', %I * %s AS %I', v_src.amount_field, v_src.amount_sign, v_col);
-                END IF;
-            ELSIF v_has_bal AND v_col = v_stack.date_field THEN
-                -- Use per-source date_field
-                IF v_src.date_field IS NULL THEN
-                    v_select := v_select || format(', NULL::date AS %I', v_col);
-                ELSE
-                    v_select := v_select || format(', %I AS %I', v_src.date_field, v_col);
-                END IF;
-            ELSE
-                -- Other canonical fields: use field_map or same name, NULL if column doesn't exist
-                v_src_field := COALESCE(v_src.field_map->>v_col, v_col);
-                IF EXISTS (
-                    SELECT 1 FROM information_schema.columns
-                    WHERE table_schema = 'dfv'
-                      AND table_name = v_src.source_name
-                      AND column_name = v_src_field
-                ) THEN
-                    v_select := v_select || format(', %I AS %I', v_src_field, v_col);
-                ELSE
-                    v_select := v_select || format(', NULL::text AS %I', v_col);
-                END IF;
-            END IF;
-        END LOOP;
-
-        v_select := v_select || format(' FROM dfv.%I', v_src.source_name);
-
-        v_ctes      := v_ctes      || format('%I AS (%s)', v_src.source_name, v_select);
-        v_cte_names := v_cte_names || quote_ident(v_src.source_name);
-
-        -- Accumulate carried-forward source balance column and total offset
-        IF v_has_bal THEN
-            IF v_src_bal_cols IS NOT NULL THEN v_src_bal_cols := v_src_bal_cols || ', '; END IF;
-            v_src_bal_cols := COALESCE(v_src_bal_cols, '') || format(
-                'SUM(CASE WHEN _source = %L THEN %I END) OVER (ORDER BY %I ASC, _id ASC) + %s AS %I',
-                v_src.source_name, v_stack.amount_field, v_stack.date_field,
-                v_src.balance_offset, v_src.source_name || '_balance'
-            );
-            v_total_offset := v_total_offset + v_src.balance_offset;
-        END IF;
-    END LOOP;
-
-    IF array_length(v_ctes, 1) IS NULL THEN
-        RETURN json_build_object('success', false, 'error', 'Stack has no sources');
-    END IF;
-
-    v_view := 'dfv.' || quote_ident(p_stack_name);
-
-    v_canon_cols := (
-        SELECT string_agg(quote_ident(f->>'name'), ', ')
-        FROM jsonb_array_elements(v_stack.fields) f
-    );
-
-    IF v_has_bal THEN
-        v_sql := format(
-            'CREATE VIEW %s AS '
-            'WITH %s, _stacked AS (SELECT * FROM %s) '
-            'SELECT _source, _id, %s, '
-            '%s, '
-            'SUM(%I) OVER (ORDER BY %I ASC, _id ASC) + %s AS net_balance '
-            'FROM _stacked ORDER BY %I DESC, _id DESC',
-            v_view,
-            array_to_string(v_ctes, ', '),
-            array_to_string(v_cte_names, ' UNION ALL SELECT * FROM '),
-            v_canon_cols,
-            v_src_bal_cols,
-            v_stack.amount_field,
-            v_stack.date_field,
-            v_total_offset,
-            v_stack.date_field
-        );
-    ELSE
-        v_sql := format(
-            'CREATE VIEW %s AS '
-            'WITH %s, _stacked AS (SELECT * FROM %s) '
-            'SELECT _source, _id, %s FROM _stacked',
-            v_view,
-            array_to_string(v_ctes, ', '),
-            array_to_string(v_cte_names, ' UNION ALL SELECT * FROM '),
-            v_canon_cols
-        );
-    END IF;
-
-    IF NOT p_dry_run THEN
-        CREATE SCHEMA IF NOT EXISTS dfv;
-        EXECUTE format('DROP VIEW IF EXISTS %s CASCADE', v_view);
-        EXECUTE v_sql;
-
-        -- Detect stacks whose views were dropped by CASCADE and mark them stale
-        SELECT array_agg(s.name) INTO v_cascade_stale
-        FROM dataflow.stacks s
-        WHERE s.name != p_stack_name
-          AND s.view_generated_at IS NOT NULL
-          AND NOT EXISTS (
-              SELECT 1 FROM pg_views v
-              WHERE v.schemaname = 'dfv' AND v.viewname = s.name
-          );
-
-        UPDATE dataflow.stacks SET view_generated_at = NULL
-        WHERE name = ANY(v_cascade_stale);
-    END IF;
-
-    RETURN json_build_object(
-        'success', true,
-        'view', v_view,
-        'sql', v_sql,
-        'cascade_stale', COALESCE(to_json(v_cascade_stale), '[]'::json)
-    );
-END;
-$$ LANGUAGE plpgsql;
-
-------------------------------------------------------
--- Function: get_stack_balance
--- Returns the current running balance (last row of the generated view)
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION get_stack_balance(p_stack_name TEXT)
-RETURNS JSON AS $$
-DECLARE
-    v_stack     dataflow.stacks%ROWTYPE;
-    v_balance   NUMERIC;
-    v_view      TEXT;
-    v_sql       TEXT;
-BEGIN
-    SELECT * INTO v_stack FROM dataflow.stacks WHERE name = p_stack_name;
-    IF NOT FOUND THEN
-        RETURN json_build_object('success', false, 'error', 'Stack not found');
-    END IF;
-    IF v_stack.amount_field IS NULL OR v_stack.date_field IS NULL THEN
-        RETURN json_build_object('success', false, 'error', 'amount_field and date_field must be set');
-    END IF;
-
-    v_view := 'dfv.' || quote_ident(p_stack_name);
-
-    BEGIN
-        v_sql := format(
-            'SELECT net_balance FROM %s ORDER BY %I DESC, _id DESC LIMIT 1',
-            v_view, v_stack.date_field
-        );
-        EXECUTE v_sql INTO v_balance;
-    EXCEPTION WHEN undefined_table THEN
-        RETURN json_build_object('success', false, 'error', 'View not generated yet — click Generate first');
-    END;
-
-    RETURN json_build_object('success', true, 'balance', v_balance);
-END;
-$$ LANGUAGE plpgsql STABLE;
-
-COMMENT ON FUNCTION generate_stack_view(TEXT, BOOLEAN) IS 'Generate a UNION ALL view in dfv schema combining multiple sources with optional running balance; p_dry_run=true returns SQL without executing';
-COMMENT ON FUNCTION calibrate_balance IS 'Given a known good balance at a date, compute the offset to add to balance_offset';
-COMMENT ON FUNCTION get_stack_balance IS 'Return the current running balance (last row) from the generated dfv view';
-
-------------------------------------------------------
--- Function: reorder_stack_sources
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION reorder_stack_sources(p_stack_name TEXT, p_source_names TEXT[])
-RETURNS VOID AS $$
-DECLARE
-    i INTEGER;
-BEGIN
-    FOR i IN 1..array_length(p_source_names, 1) LOOP
-        UPDATE dataflow.stack_sources
-        SET seq = i
-        WHERE stack_name = p_stack_name AND source_name = p_source_names[i];
-    END LOOP;
-END;
-$$ LANGUAGE plpgsql;

database/status.sql

@@ -1,86 +0,0 @@
---
--- Status tracking: view_generated_at on sources and stacks
--- Cleared by triggers when definitions change; set by API when views are generated.
---
-
-SET search_path TO dataflow, public;
-
--- Add view_generated_at columns
-ALTER TABLE dataflow.sources ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAMPTZ;
-ALTER TABLE dataflow.stacks  ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAMPTZ;
-
-------------------------------------------------------
--- Trigger: clear source view_generated_at when config (field definitions) changes
--- Rules and mappings affect transformed data, not view structure — no trigger needed there
-------------------------------------------------------
-DROP TRIGGER IF EXISTS trg_rules_changed ON dataflow.rules;
-DROP TRIGGER IF EXISTS trg_mappings_changed ON dataflow.mappings;
-DROP FUNCTION IF EXISTS dataflow.rules_changed();
-DROP FUNCTION IF EXISTS dataflow.mappings_changed();
-
-CREATE OR REPLACE FUNCTION dataflow.source_config_changed()
-RETURNS TRIGGER AS $$
-BEGIN
-    IF NEW.config IS DISTINCT FROM OLD.config THEN
-        NEW.view_generated_at := NULL;
-    END IF;
-    RETURN NEW;
-END;
-$$ LANGUAGE plpgsql;
-
-DROP TRIGGER IF EXISTS trg_source_config_changed ON dataflow.sources;
-CREATE TRIGGER trg_source_config_changed
-BEFORE UPDATE ON dataflow.sources
-FOR EACH ROW EXECUTE FUNCTION dataflow.source_config_changed();
-
-------------------------------------------------------
--- Trigger: clear stack view_generated_at when sources change
--- On UPDATE, skip if all view-relevant columns are unchanged (upsert no-ops should not mark stale)
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION dataflow.stack_sources_changed()
-RETURNS TRIGGER AS $$
-BEGIN
-    IF TG_OP = 'UPDATE' THEN
-        IF NEW.field_map        IS NOT DISTINCT FROM OLD.field_map        AND
-           NEW.amount_sign      IS NOT DISTINCT FROM OLD.amount_sign      AND
-           NEW.balance_offset   IS NOT DISTINCT FROM OLD.balance_offset   AND
-           NEW.amount_field     IS NOT DISTINCT FROM OLD.amount_field     AND
-           NEW.date_field       IS NOT DISTINCT FROM OLD.date_field       AND
-           NEW.seq              IS NOT DISTINCT FROM OLD.seq THEN
-            RETURN NULL;
-        END IF;
-    END IF;
-    UPDATE dataflow.stacks SET view_generated_at = NULL
-    WHERE name = COALESCE(NEW.stack_name, OLD.stack_name);
-    RETURN NULL;
-END;
-$$ LANGUAGE plpgsql;
-
-DROP TRIGGER IF EXISTS trg_stack_sources_changed ON dataflow.stack_sources;
-CREATE TRIGGER trg_stack_sources_changed
-AFTER INSERT OR UPDATE OR DELETE ON dataflow.stack_sources
-FOR EACH ROW EXECUTE FUNCTION dataflow.stack_sources_changed();
-
-------------------------------------------------------
--- Function: get_status
--- Returns sources and stacks whose view is stale (null or never generated)
-------------------------------------------------------
-CREATE OR REPLACE FUNCTION get_status()
-RETURNS JSON AS $$
-DECLARE
-    v_sources JSON;
-    v_stacks  JSON;
-BEGIN
-    SELECT COALESCE(json_agg(json_build_object('name', name, 'view_generated_at', view_generated_at) ORDER BY name), '[]'::json)
-    INTO v_sources
-    FROM dataflow.sources
-    WHERE view_generated_at IS NULL;
-
-    SELECT COALESCE(json_agg(json_build_object('name', name, 'view_generated_at', view_generated_at) ORDER BY name), '[]'::json)
-    INTO v_stacks
-    FROM dataflow.stacks
-    WHERE view_generated_at IS NULL;
-
-    RETURN json_build_object('stale_sources', v_sources, 'stale_stacks', v_stacks);
-END;
-$$ LANGUAGE plpgsql STABLE;

database/transform.sql

@@ -1,156 +0,0 @@
---
--- Transform queries
--- The rule/mapping engine; SQL for the transform endpoints in api/routes/sources.js,
--- api/routes/rules.js and api/routes/records.js
---
--- Order matters within this file: the aggregate is used by apply_transformations,
--- which in turn is called by reprocess_records.
---
-
-SET search_path TO dataflow, public;
-
--- ── Merge aggregate ───────────────────────────────────────────────────────────
-
--- Merge JSONB objects across rows (later rows win on key conflicts)
--- Usage: jsonb_concat_obj(col ORDER BY sequence)
-CREATE OR REPLACE FUNCTION dataflow.jsonb_merge(a JSONB, b JSONB)
-RETURNS JSONB AS $$
-    SELECT COALESCE(a, '{}') || COALESCE(b, '{}')
-$$ LANGUAGE sql IMMUTABLE;
-
-DROP AGGREGATE IF EXISTS dataflow.jsonb_concat_obj(JSONB);
-CREATE AGGREGATE dataflow.jsonb_concat_obj(JSONB) (
-    sfunc    = dataflow.jsonb_merge,
-    stype    = JSONB,
-    initcond = '{}'
-);
-
--- ── Apply rules and mappings ──────────────────────────────────────────────────
-
--- Writes only the rule/mapping output into records.transformed. Raw values stay in
--- data and manual edits stay in overrides; readers merge the three layers.
-DROP FUNCTION IF EXISTS apply_transformations(TEXT, INTEGER[]);
-CREATE OR REPLACE FUNCTION apply_transformations(
-    p_source_name TEXT,
-    p_record_ids INTEGER[] DEFAULT NULL,  -- NULL = all eligible records
-    p_overwrite   BOOLEAN  DEFAULT FALSE  -- FALSE = skip already-transformed, TRUE = overwrite all
-) RETURNS JSON AS $$
-WITH
--- All records to process
-qualifying AS (
-    SELECT id, data
-    FROM dataflow.records
-    WHERE source_name = p_source_name
-      AND (p_overwrite OR transformed IS NULL)
-      AND (p_record_ids IS NULL OR id = ANY(p_record_ids))
-),
--- Mirror TPS rx: fan out one row per regex match, drive from rules → records
-rx AS (
-    SELECT
-        q.id,
-        r.name          AS rule_name,
-        r.sequence,
-        r.output_field,
-        r.retain,
-        r.function_type,
-        COALESCE(mt.rn, rp.rn, 1) AS result_number,
-        -- extract: build map_val and retain_val per match (mirrors TPS)
-        CASE WHEN array_length(mt.mt, 1) = 1 THEN to_jsonb(mt.mt[1]) ELSE to_jsonb(mt.mt) END AS match_val,
-        to_jsonb(rp.rp) AS replace_val
-    FROM dataflow.rules r
-    INNER JOIN qualifying q ON q.data ? r.field
-    LEFT JOIN LATERAL regexp_matches(q.data ->> r.field, r.pattern, r.flags)
-        WITH ORDINALITY AS mt(mt, rn) ON r.function_type = 'extract'
-    LEFT JOIN LATERAL regexp_replace(q.data ->> r.field, r.pattern, r.replace_value, r.flags)
-        WITH ORDINALITY AS rp(rp, rn) ON r.function_type = 'replace'
-    WHERE r.source_name = p_source_name
-      AND r.enabled = true
-),
--- Aggregate match rows back into one value per (record, rule) — mirrors TPS agg_to_target_items
-agg_matches AS (
-    SELECT
-        id,
-        rule_name,
-        sequence,
-        output_field,
-        retain,
-        function_type,
-        CASE function_type
-            WHEN 'replace' THEN jsonb_agg(replace_val) -> 0
-            ELSE
-                CASE WHEN max(result_number) = 1
-                    THEN jsonb_agg(match_val ORDER BY result_number) -> 0
-                    ELSE jsonb_agg(match_val ORDER BY result_number)
-                END
-        END AS extracted
-    FROM rx
-    GROUP BY id, rule_name, sequence, output_field, retain, function_type
-),
--- Join with mappings to find mapped output — mirrors TPS link_map
-linked AS (
-    SELECT
-        a.id,
-        a.sequence,
-        a.output_field,
-        a.retain,
-        a.extracted,
-        m.output AS mapped
-    FROM agg_matches a
-    LEFT JOIN dataflow.mappings m ON
-        m.source_name = p_source_name
-        AND m.rule_name = a.rule_name
-        AND m.input_value = a.extracted
-    WHERE a.extracted IS NOT NULL
-),
--- Build per-rule output JSONB:
---   mapped  → use mapping output; also write output_field if retain = true
---   no map  → write extracted value to output_field
-rule_output AS (
-    SELECT
-        id,
-        sequence,
-        CASE
-            WHEN mapped IS NOT NULL THEN
-                mapped ||
-                CASE WHEN retain
-                     THEN jsonb_build_object(output_field, extracted)
-                     ELSE '{}'::jsonb
-                END
-            ELSE
-                jsonb_build_object(output_field, extracted)
-        END AS output
-    FROM linked
-),
--- Merge all rule outputs per record in sequence order — mirrors TPS agg_to_id
-record_additions AS (
-    SELECT
-        id,
-        dataflow.jsonb_concat_obj(output ORDER BY sequence) AS additions
-    FROM rule_output
-    GROUP BY id
-),
--- Update all qualifying records; records with no rule matches get an empty object
-updated AS (
-    UPDATE dataflow.records rec
-    SET transformed    = COALESCE(ra.additions, '{}'::jsonb),
-        transformed_at = CURRENT_TIMESTAMP
-    FROM qualifying q
-    LEFT JOIN record_additions ra ON ra.id = q.id
-    WHERE rec.id = q.id
-    RETURNING rec.id
-)
-SELECT json_build_object('success', true, 'transformed', count(*))
-FROM updated
-$$ LANGUAGE sql;
-
-COMMENT ON FUNCTION apply_transformations IS 'Apply transformation rules and mappings to records (set-based CTE)';
-
--- ── Reprocess ─────────────────────────────────────────────────────────────────
-
-CREATE OR REPLACE FUNCTION reprocess_records(p_source_name TEXT)
-RETURNS JSON AS $$
-    -- Overwrite all records directly — no clear step, mirrors TPS srce_map_overwrite
-    SELECT dataflow.apply_transformations(p_source_name, NULL, TRUE)
-$$ LANGUAGE sql;
-
-COMMENT ON FUNCTION reprocess_records IS 'Reapply all transformations for a source, overwriting existing values';

deploy.sh

@@ -0,0 +1,409 @@
+#!/bin/bash
+#
+# Dataflow Deploy Script
+# First run: full install (database, schema, UI, nginx, systemd)
+# Subsequent runs: update functions, UI, and restart service
+#
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$SCRIPT_DIR"
+
+# ── Helpers ───────────────────────────────────────────────────────────────────
+
+BOLD='\033[1m'
+DIM='\033[2m'
+GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
+RED='\033[0;31m'
+RESET='\033[0m'
+
+section() { echo ""; echo -e "${BOLD}── $1 ──${RESET}"; }
+step()    { printf "  %-42s" "$1..."; }
+ok()      { echo -e "${GREEN}✓${RESET}"; }
+fail()    { echo -e "${RED}✗ $1${RESET}"; exit 1; }
+info()    { echo -e "  ${DIM}$1${RESET}"; }
+warn()    { echo -e "  ${YELLOW}$1${RESET}"; }
+
+confirm() {
+    # confirm "Section name" — prints section header, asks to proceed
+    # Returns 0 to proceed, 1 to skip
+    section "$1"
+    read -p "  Proceed? [Y/n]: " _yn
+    [[ ! "$_yn" =~ ^[Nn]$ ]]
+}
+
+# ── Mode detection ────────────────────────────────────────────────────────────
+
+echo ""
+echo "  This script covers:"
+echo "    1. Database  — create user/database, deploy schema + functions"
+echo "    2. UI        — build from source"
+echo "    3. Nginx     — reverse proxy config + SSL via certbot"
+echo "    4. Service   — install and enable systemd unit"
+echo "    5. Restart   — start or restart the API server"
+echo ""
+echo "  Each step will be confirmed before running."
+echo "  Press Ctrl+C at any time to abort."
+
+section "Mode"
+if [ ! -f .env ]; then
+    echo "  No .env found — first-time install"
+    MODE=install
+else
+    echo "  .env found — update mode"
+    MODE=update
+    export $(cat .env | grep -v '^#' | xargs)
+fi
+
+# ── Phase 1: Collect all config ───────────────────────────────────────────────
+
+if [ "$MODE" = "install" ]; then
+
+    section "PostgreSQL Admin"
+    info "Needed to create the app user and database"
+    read -p "  Admin username [postgres]: " ADMIN_USER; ADMIN_USER=${ADMIN_USER:-postgres}
+    read -s -p "  Admin password: " ADMIN_PASS; echo ""
+
+    section "Application Database"
+    read -p "  Host         [localhost]: " DB_HOST;     DB_HOST=${DB_HOST:-localhost}
+    read -p "  Port         [5432]:      " DB_PORT;     DB_PORT=${DB_PORT:-5432}
+    read -p "  Database     [dataflow]:  " DB_NAME;     DB_NAME=${DB_NAME:-dataflow}
+    read -p "  App user     [dataflow]:  " DB_USER;     DB_USER=${DB_USER:-dataflow}
+    read -s -p "  App password:             " DB_PASSWORD; echo ""
+
+    section "API"
+    read -p "  Port         [3020]:      " API_PORT;    API_PORT=${API_PORT:-3020}
+    read -p "  Environment  [production]:" NODE_ENV;    NODE_ENV=${NODE_ENV:-production}
+
+    section "Nginx"
+    read -p "  Set up nginx reverse proxy? [Y/n]: " _yn
+    if [[ ! "$_yn" =~ ^[Nn]$ ]]; then
+        read -p "  Domain (e.g. dataflow.example.com): " NGINX_DOMAIN
+    fi
+
+    DO_DEPS=y; DO_DB=y; DO_SCHEMA=y; DO_FN=y; DO_BUILD=y; DO_SERVICE=y; DO_RESTART=y
+
+else
+
+    section "Database"
+    echo "  Current: ${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
+    read -p "  Change target? [y/N]: " _yn
+    if [[ "$_yn" =~ ^[Yy]$ ]]; then
+        read -p "  Host     [${DB_HOST}]: " _in; DB_HOST=${_in:-$DB_HOST}
+        read -p "  Port     [${DB_PORT}]: " _in; DB_PORT=${_in:-$DB_PORT}
+        read -p "  Database [${DB_NAME}]: " _in; DB_NAME=${_in:-$DB_NAME}
+        read -p "  User     [${DB_USER}]: " _in; DB_USER=${_in:-$DB_USER}
+        read -s -p "  Password (blank = keep): " _in; echo ""
+        if [ -n "$_in" ]; then DB_PASSWORD=$_in; fi
+        CHANGE_DB=y
+    fi
+
+    section "Select steps to run"
+    read -p "  Redeploy SQL functions? [Y/n]: " DO_FN;      DO_FN=${DO_FN:-y}
+    read -p "  Rebuild UI?             [Y/n]: " DO_BUILD;    DO_BUILD=${DO_BUILD:-y}
+    read -p "  Set up / update nginx?  [y/N]: " DO_NGINX
+    read -p "  Restart API service?    [Y/n]: " DO_RESTART;  DO_RESTART=${DO_RESTART:-y}
+
+    if [[ "$DO_NGINX" =~ ^[Yy]$ ]]; then
+        read -p "  Nginx domain: " NGINX_DOMAIN
+    fi
+fi
+
+# ── Phase 2: Plan summary ─────────────────────────────────────────────────────
+
+section "Plan"
+echo "  Mode:     $( [ "$MODE" = "install" ] && echo "First-time install" || echo "Update" )"
+echo "  Database: ${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
+echo "  API port: ${API_PORT:-3020}"
+echo ""
+echo "  Steps:"
+
+if [ "$MODE" = "install" ]; then
+    echo "    • Install Node.js dependencies"
+    echo "    • Test admin connection and create DB user/database"
+    echo "    • Deploy schema and SQL functions"
+    [ -n "$NGINX_DOMAIN" ] && echo "    • Configure nginx → $NGINX_DOMAIN" \
+                           || echo "    • Nginx — skipped (no domain provided)"
+    echo "    • Build UI"
+    echo "    • Install systemd service"
+    echo "    • Start service"
+else
+    [ "${CHANGE_DB}" = "y" ]              && echo "    • Update .env with new database target"
+    [[ ! "$DO_FN" =~ ^[Nn]$ ]]           && echo "    • Redeploy SQL functions" \
+                                          || echo "    • SQL functions — skipped"
+    [[ ! "$DO_BUILD" =~ ^[Nn]$ ]]        && echo "    • Rebuild UI" \
+                                          || echo "    • UI build — skipped"
+    [ -n "$NGINX_DOMAIN" ]               && echo "    • Configure nginx → $NGINX_DOMAIN" \
+                                          || echo "    • Nginx — skipped"
+    [[ ! "$DO_RESTART" =~ ^[Nn]$ ]]      && echo "    • Restart API service" \
+                                          || echo "    • Service restart — skipped"
+fi
+
+echo ""
+read -p "  Continue? [Y/n]: " _yn
+[[ "$_yn" =~ ^[Nn]$ ]] && echo "  Aborted." && exit 0
+
+# ── Phase 3: Execute ──────────────────────────────────────────────────────────
+
+if [ "$MODE" = "install" ]; then
+
+    # Dependencies
+    if confirm "Dependencies"; then
+        step "API (npm install)"
+        npm install --omit=dev -q && ok || fail "npm install failed"
+
+        step "UI (npm install)"
+        cd ui && npm install -q && cd .. && ok || fail "ui npm install failed"
+    else
+        info "skipped"
+    fi
+
+    # PostgreSQL
+    if confirm "PostgreSQL"; then
+        step "Testing admin connection"
+        export PGPASSWORD="$ADMIN_PASS"
+        psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c '\q' 2>/dev/null && ok \
+            || fail "Cannot connect as $ADMIN_USER"
+
+        step "Creating user '$DB_USER'"
+        if psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -tAc \
+            "SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'" 2>/dev/null | grep -q 1; then
+            echo -e "${DIM}already exists${RESET}"
+        else
+            psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres \
+                -c "CREATE USER $DB_USER WITH PASSWORD '$DB_PASSWORD';" > /dev/null && ok \
+                || fail "Could not create user"
+        fi
+
+        step "Creating database '$DB_NAME'"
+        if psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -lqt \
+            | cut -d'|' -f1 | grep -qw "$DB_NAME"; then
+            echo -e "${DIM}already exists${RESET}"
+        else
+            psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres \
+                -c "CREATE DATABASE $DB_NAME OWNER $DB_USER;" > /dev/null && ok \
+                || fail "Could not create database"
+        fi
+        unset PGPASSWORD
+
+        step "Writing .env"
+        cat > .env << ENVEOF
+# Database Configuration
+DB_HOST=$DB_HOST
+DB_PORT=$DB_PORT
+DB_NAME=$DB_NAME
+DB_USER=$DB_USER
+DB_PASSWORD=$DB_PASSWORD
+
+# API Configuration
+API_PORT=$API_PORT
+NODE_ENV=$NODE_ENV
+ENVEOF
+        ok
+
+        export PGPASSWORD="$DB_PASSWORD"
+
+        step "Deploying schema"
+        psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/schema.sql -q && ok \
+            || fail "Schema deploy failed"
+
+        step "Deploying functions"
+        psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/functions.sql -q && ok \
+            || fail "Functions deploy failed"
+    else
+        info "skipped"
+    fi
+
+else
+
+    # Update: save .env if DB target changed
+    section ".env"
+    if [ "${CHANGE_DB}" = "y" ]; then
+        step "Writing updated .env"
+        cat > .env << ENVEOF
+# Database Configuration
+DB_HOST=$DB_HOST
+DB_PORT=$DB_PORT
+DB_NAME=$DB_NAME
+DB_USER=$DB_USER
+DB_PASSWORD=$DB_PASSWORD
+
+# API Configuration
+API_PORT=${API_PORT:-3020}
+NODE_ENV=${NODE_ENV:-production}
+ENVEOF
+        ok
+    else
+        info "no changes"
+    fi
+
+    # Test connection
+    section "Database Connection"
+    step "Testing"
+    export PGPASSWORD="$DB_PASSWORD"
+    psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -c '\q' 2>/dev/null && ok \
+        || fail "Cannot connect — check credentials"
+
+    # SQL functions
+    if [[ ! "$DO_FN" =~ ^[Nn]$ ]]; then
+        if confirm "SQL Functions"; then
+            step "Deploying functions"
+            psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/functions.sql -q && ok \
+                || fail "Functions deploy failed"
+        else
+            info "skipped"
+        fi
+    else
+        section "SQL Functions"
+        info "skipped"
+    fi
+fi
+
+# ── UI ────────────────────────────────────────────────────────────────────────
+
+if [ "$MODE" = "install" ] || [[ ! "$DO_BUILD" =~ ^[Nn]$ ]]; then
+    if confirm "UI Build"; then
+        step "Building"
+        cd ui && npm run build > /dev/null 2>&1 && cd .. && ok \
+            || fail "UI build failed"
+    else
+        info "skipped"
+    fi
+else
+    section "UI Build"
+    info "skipped"
+fi
+
+# ── Nginx ─────────────────────────────────────────────────────────────────────
+
+if [ -n "$NGINX_DOMAIN" ]; then
+    if confirm "Nginx ($NGINX_DOMAIN)"; then
+        CONF_NAME=$(echo "$NGINX_DOMAIN" | cut -d. -f1)
+        CONF_PATH="/etc/nginx/sites-enabled/$CONF_NAME"
+        CERT_PATH="/etc/letsencrypt/live/$NGINX_DOMAIN/fullchain.pem"
+        TMP_CONF=$(mktemp)
+
+        if [ -f "$CERT_PATH" ]; then
+            cat > "$TMP_CONF" << NGINXEOF
+server {
+    listen 80;
+    listen [::]:80;
+    server_name $NGINX_DOMAIN;
+    location / { return 301 https://\$host\$request_uri; }
+}
+
+server {
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+    server_name $NGINX_DOMAIN;
+
+    ssl_certificate     $CERT_PATH;
+    ssl_certificate_key /etc/letsencrypt/live/$NGINX_DOMAIN/privkey.pem;
+    ssl_protocols       TLSv1.2 TLSv1.3;
+    ssl_ciphers         HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
+    ssl_prefer_server_ciphers on;
+    ssl_session_cache   shared:SSL:10m;
+
+    keepalive_timeout    70;
+    sendfile             on;
+    client_max_body_size 80m;
+
+    location / {
+        proxy_pass http://localhost:${API_PORT:-3020};
+    }
+}
+NGINXEOF
+        else
+            cat > "$TMP_CONF" << NGINXEOF
+server {
+    listen 80;
+    listen [::]:80;
+    server_name $NGINX_DOMAIN;
+    location / {
+        proxy_pass http://localhost:${API_PORT:-3020};
+    }
+}
+NGINXEOF
+        fi
+
+        step "Writing /etc/nginx/sites-enabled/$CONF_NAME"
+        sudo cp "$TMP_CONF" "$CONF_PATH" && rm "$TMP_CONF" && ok \
+            || fail "Could not write nginx config (check sudo)"
+
+        step "Testing nginx config"
+        sudo nginx -t > /dev/null 2>&1 && ok \
+            || fail "nginx config invalid — run: sudo nginx -t"
+
+        step "Reloading nginx"
+        sudo systemctl reload nginx && ok \
+            || fail "nginx reload failed"
+
+        if [ ! -f "$CERT_PATH" ]; then
+            warn "No SSL cert found for $NGINX_DOMAIN"
+            read -p "  Run certbot now? [Y/n]: " _yn
+            if [[ ! "$_yn" =~ ^[Nn]$ ]]; then
+                step "Running certbot"
+                sudo certbot --nginx -d "$NGINX_DOMAIN" --non-interactive --agree-tos \
+                    --redirect -m "admin@$NGINX_DOMAIN" > /dev/null 2>&1 && ok \
+                    || fail "certbot failed — run manually: sudo certbot --nginx -d $NGINX_DOMAIN"
+            fi
+        fi
+    else
+        info "skipped"
+    fi
+else
+    section "Nginx"
+    info "skipped"
+fi
+
+# ── Systemd service ───────────────────────────────────────────────────────────
+
+SERVICE_FILE="/etc/systemd/system/dataflow.service"
+
+section "Systemd Service"
+if [ -f "$SERVICE_FILE" ]; then
+    info "already installed"
+else
+    read -p "  Not installed. Install now? (requires sudo) [y/N]: " _yn
+    if [[ "$_yn" =~ ^[Yy]$ ]]; then
+        step "Installing service"
+        sudo cp "$SCRIPT_DIR/dataflow.service" "$SERVICE_FILE" && ok \
+            || fail "Could not install service"
+
+        step "Enabling on boot"
+        sudo systemctl daemon-reload && sudo systemctl enable dataflow > /dev/null 2>&1 && ok \
+            || fail "Could not enable service"
+    else
+        info "skipped"
+    fi
+fi
+
+# ── API Server ────────────────────────────────────────────────────────────────
+
+if [ "$MODE" = "install" ] || [[ ! "$DO_RESTART" =~ ^[Nn]$ ]]; then
+    if confirm "API Server"; then
+        if [ -f "$SERVICE_FILE" ]; then
+            step "Restarting dataflow service"
+            sudo systemctl restart dataflow && sleep 1
+            systemctl is-active --quiet dataflow && ok \
+                || fail "Service failed — check: journalctl -u dataflow -n 30"
+        else
+            info "systemd service not installed — start manually: node api/server.js"
+        fi
+    else
+        info "skipped"
+    fi
+else
+    section "API Server"
+    info "skipped"
+fi
+
+# ── Done ──────────────────────────────────────────────────────────────────────
+
+section "Done"
+echo "  API:  http://localhost:${API_PORT:-3020}"
+[ -n "$NGINX_DOMAIN" ] && echo "  Web:  https://$NGINX_DOMAIN"
+echo "  Logs: journalctl -u dataflow -f"
+echo ""

docs/perspective-pivot.md

@@ -1,79 +1,27 @@
-# Perspective
+# Perspective Pivot — Technical Reference
 
-Everything about the Perspective pivot in dataflow: which packages and versions are
+Version tested: `@perspective-dev` v4.4.0 (client, viewer, viewer-datagrid, viewer-d3fc), loaded from CDN.
-pinned and why, and a ground-truth reference for the parts of the API the official docs
-don't cover.
 
-Shared rationale across projects lives in the canonical guide at
+This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement.
-`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy
-pattern, upgrade smoke test). This file records what's specific to dataflow.
 
 ---
 
-> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
+## Loading from CDN
-> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
-> engine, separate npm scope and release schedule — don't mix the two.
-
----
-
-## Current state
-
-- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
-  This is the target loader; pf_app should adopt it.
-- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
-  Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
-  unless view sizes grow well past 100k.
-- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
-  for the org; pf_app should copy it.
-- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
-  Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
-- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
-  restore — the reference implementation; keep it.
-
-## The version pair is correct — do NOT "fix" it to 4.4.1
-
-`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
-`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
-combination that keeps both of dataflow's hard requirements:
-
-- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
-  `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
-  paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
-- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
-
-Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
-`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
-stays. Don't touch it.
-
-**What to actually do:**
-- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
-  on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
-  explicit.)
-- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
-  renders, dark/light re-themes, and save→reload→drop-column layout restore.
-- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
-  4.5.x suite becomes possible and the pair can collapse to one version.
-
----
-
-# API Reference
-
-Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at
-**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been
-re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete
-for some of these APIs — treat this as a ground-truth supplement.
-
-## Loading via npm
 
 ```js
-import perspective from '@perspective-dev/client/inline'
+const [{ default: perspective }] = await Promise.all([
-import '@perspective-dev/viewer/inline'
+  import('https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
-import '@perspective-dev/viewer-datagrid'
+  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
-import '@perspective-dev/viewer-d3fc'
+  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
-import '@perspective-dev/viewer/themes'
+  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
+])
 ```
 
-The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`.
+Stylesheet:
+```html
+<link rel="stylesheet" crossorigin="anonymous"
+  href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css" />
+```
 
 ---
 

docs/ui.md

@@ -1,25 +0,0 @@
-# Dataflow UI
-
-React + Vite + Tailwind CSS frontend for Dataflow.
-
-## Development
-
-```bash
-npm install
-npm run dev      # dev server on :5173, proxies /api to :3020
-```
-
-## Build
-
-```bash
-npm run build    # outputs to ../public/
-```
-
-The Express server serves `../public/` as static files — no separate web server needed in production.
-
-## Key packages
-
-- `react` / `react-router-dom` — SPA routing
-- `@perspective-dev/client`, `viewer`, `viewer-datagrid`, `viewer-d3fc` — pivot table (npm, inline WASM builds)
-- `tailwindcss` — utility CSS
-- `sql-formatter` — SQL display formatting

examples/GETTING_STARTED.md

@@ -4,40 +4,32 @@ This guide walks through a complete example using bank transaction data.
 
 ## Prerequisites
 
-PostgreSQL running, Node.js 18+, and Python 3.
+1. PostgreSQL database running
+2. Database created: `CREATE DATABASE dataflow;`
+3. `.env` file configured (copy from `.env.example`)
 
-## Step 1: Configure and Deploy
+## Step 1: Deploy Database Schema
 
 ```bash
 cd /opt/dataflow
-npm install
+psql -U postgres -d dataflow -f database/schema.sql
-python3 manage.py
+psql -U postgres -d dataflow -f database/functions.sql
 ```
 
-Choose option 1. It writes `.env`, creates the database and user if they don't exist,
+You should see tables created without errors.
-then deploys `database/schema.sql` and the SQL function files in dependency order.
 
 ## Step 2: Start the API Server
 
 ```bash
+npm install
 npm start
 ```
 
-The server starts on the port set by `API_PORT` in `.env` (3020 by default).
+The server should start on port 3000 (or your configured port).
-
-Every `/api` route requires HTTP Basic auth using the credentials set by `manage.py`
-option 9. The examples below omit it for readability — add `-u username:password` to each
-curl, or export it once:
-
-```bash
-alias dfcurl='curl -u username:password'
-```
-
-`GET /health` is the one route that needs no auth.
 
 Test it:
 ```bash
-curl http://localhost:3020/health
+curl http://localhost:3000/health
 # Should return: {"status":"ok","timestamp":"..."}
 ```
 
@@ -46,7 +38,7 @@ curl http://localhost:3020/health
 A source defines where data comes from and how to deduplicate it.
 
 ```bash
-curl -X POST http://localhost:3020/api/sources \
+curl -X POST http://localhost:3000/api/sources \
   -H "Content-Type: application/json" \
   -d '{
     "name": "bank_transactions",
@@ -63,7 +55,7 @@ Rules extract meaningful data using regex patterns.
 ### Rule 1: Extract merchant name (first part of description)
 
 ```bash
-curl -X POST http://localhost:3020/api/rules \
+curl -X POST http://localhost:3000/api/rules \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -78,7 +70,7 @@ curl -X POST http://localhost:3020/api/rules \
 ### Rule 2: Extract location (city + state pattern)
 
 ```bash
-curl -X POST http://localhost:3020/api/rules \
+curl -X POST http://localhost:3000/api/rules \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -95,7 +87,7 @@ curl -X POST http://localhost:3020/api/rules \
 Import the example CSV file:
 
 ```bash
-curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
+curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
   -F "file=@examples/bank_transactions.csv"
 ```
 
@@ -112,7 +104,7 @@ Response:
 ## Step 6: View Imported Records
 
 ```bash
-curl http://localhost:3020/api/records/source/bank_transactions?limit=5
+curl http://localhost:3000/api/records/source/bank_transactions?limit=5
 ```
 
 You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet!
@@ -120,7 +112,7 @@ You'll see the raw imported data. Note that `transformed` is `null` - we haven't
 ## Step 7: Apply Transformations
 
 ```bash
-curl -X POST http://localhost:3020/api/sources/bank_transactions/transform
+curl -X POST http://localhost:3000/api/sources/bank_transactions/transform
 ```
 
 Response:
@@ -133,7 +125,7 @@ Response:
 
 Now check the records again:
 ```bash
-curl http://localhost:3020/api/records/source/bank_transactions?limit=2
+curl http://localhost:3000/api/records/source/bank_transactions?limit=2
 ```
 
 You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`.
@@ -141,7 +133,7 @@ You'll see the `transformed` field now contains the original data plus extracted
 ## Step 8: View Extracted Values That Need Mapping
 
 ```bash
-curl http://localhost:3020/api/mappings/source/bank_transactions/unmapped
+curl http://localhost:3000/api/mappings/source/bank_transactions/unmapped
 ```
 
 Response shows extracted merchant names that aren't mapped yet:
@@ -159,7 +151,7 @@ Response shows extracted merchant names that aren't mapped yet:
 Map extracted values to clean, standardized output:
 
 ```bash
-curl -X POST http://localhost:3020/api/mappings \
+curl -X POST http://localhost:3000/api/mappings \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -171,7 +163,7 @@ curl -X POST http://localhost:3020/api/mappings \
     }
   }'
 
-curl -X POST http://localhost:3020/api/mappings \
+curl -X POST http://localhost:3000/api/mappings \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -183,7 +175,7 @@ curl -X POST http://localhost:3020/api/mappings \
     }
   }'
 
-curl -X POST http://localhost:3020/api/mappings \
+curl -X POST http://localhost:3000/api/mappings \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -201,13 +193,13 @@ curl -X POST http://localhost:3020/api/mappings \
 Clear and reapply transformations to pick up the new mappings:
 
 ```bash
-curl -X POST http://localhost:3020/api/sources/bank_transactions/reprocess
+curl -X POST http://localhost:3000/api/sources/bank_transactions/reprocess
 ```
 
 ## Step 11: View Final Results
 
 ```bash
-curl http://localhost:3020/api/records/source/bank_transactions?limit=5
+curl http://localhost:3000/api/records/source/bank_transactions?limit=5
 ```
 
 Now the `transformed` field contains:
@@ -242,7 +234,7 @@ Example result:
 Try importing the same file again:
 
 ```bash
-curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
+curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
   -F "file=@examples/bank_transactions.csv"
 ```
 
@@ -280,19 +272,19 @@ You've now:
 
 ```bash
 # View all sources
-curl http://localhost:3020/api/sources
+curl http://localhost:3000/api/sources
 
 # View source statistics
-curl http://localhost:3020/api/sources/bank_transactions/stats
+curl http://localhost:3000/api/sources/bank_transactions/stats
 
 # View all rules for a source
-curl http://localhost:3020/api/rules/source/bank_transactions
+curl http://localhost:3000/api/rules/source/bank_transactions
 
 # View all mappings for a source
-curl http://localhost:3020/api/mappings/source/bank_transactions
+curl http://localhost:3000/api/mappings/source/bank_transactions
 
 # Search for specific records
-curl -X POST http://localhost:3020/api/records/search \
+curl -X POST http://localhost:3000/api/records/search \
   -H "Content-Type: application/json" \
   -d '{
     "source_name": "bank_transactions",
@@ -309,11 +301,11 @@ curl -X POST http://localhost:3020/api/records/search \
 - Check logs for error messages
 
 **Import fails:**
-- Verify source exists: `curl http://localhost:3020/api/sources`
+- Verify source exists: `curl http://localhost:3000/api/sources`
 - Check CSV format matches expectations
 - Ensure constraint_fields match CSV column names
 
 **Transformations not working:**
-- Check rules exist: `curl http://localhost:3020/api/rules/source/bank_transactions`
+- Check rules exist: `curl http://localhost:3000/api/rules/source/bank_transactions`
 - Test regex pattern manually
 - Check records have the specified field

manage.py

@@ -18,20 +18,6 @@ SERVICE_FILE = Path('/etc/systemd/system/dataflow.service')
 SERVICE_SRC  = ROOT / 'dataflow.service'
 NGINX_DIR    = Path('/etc/nginx/sites-enabled')
 
-# Deployed in order — stacks.sql creates tables that reference sources, and
-# transform.sql defines an aggregate its own functions depend on
-QUERIES_DIR  = ROOT / 'database'
-QUERY_FILES  = [
-    QUERIES_DIR / 'sources.sql',
-    QUERIES_DIR / 'rules.sql',
-    QUERIES_DIR / 'mappings.sql',
-    QUERIES_DIR / 'records.sql',
-    QUERIES_DIR / 'import.sql',
-    QUERIES_DIR / 'transform.sql',
-    QUERIES_DIR / 'stacks.sql',
-    QUERIES_DIR / 'status.sql',
-]
-
 # ── Terminal helpers ──────────────────────────────────────────────────────────
 
 BOLD   = '\033[1m'
@@ -167,31 +153,23 @@ def ui_build_time():
         return datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M')
     return None
 
-def nginx_conf_path(port):
+def nginx_domain(port):
-    """Path of the nginx site proxying to our port, if any."""
+    """Find nginx site proxying to our port."""
     if not NGINX_DIR.exists():
         return None
     for f in NGINX_DIR.iterdir():
         try:
-            if f':{port}' in f.read_text():
+            text = f.read_text()
-                return f
+            if f':{port}' in text:
+                for line in text.splitlines():
+                    if 'server_name' in line:
+                        parts = line.split()
+                        if len(parts) >= 2:
+                            return parts[1].rstrip(';')
         except Exception:
             pass
     return None
 
-
-def nginx_domain(port):
-    """server_name of the nginx site proxying to our port."""
-    conf = nginx_conf_path(port)
-    if not conf:
-        return None
-    for line in conf.read_text().splitlines():
-        if 'server_name' in line:
-            parts = line.split()
-            if len(parts) >= 2:
-                return parts[1].rstrip(';')
-    return None
-
 def sudo_run(args, **kwargs):
     return subprocess.run(['sudo'] + args, **kwargs)
 
@@ -354,8 +332,13 @@ def action_configure(cfg):
 
     db_location = f'database "{new_cfg["DB_NAME"]}" on {new_cfg["DB_HOST"]}:{new_cfg["DB_PORT"]}'
     schema_file  = ROOT / 'database' / 'schema.sql'
-    queries_dir  = QUERIES_DIR
+    queries_dir  = ROOT / 'database' / 'queries'
-    query_files  = QUERY_FILES
+    query_files  = [
+        queries_dir / 'sources.sql',
+        queries_dir / 'rules.sql',
+        queries_dir / 'mappings.sql',
+        queries_dir / 'records.sql',
+    ]
 
     # Offer schema deployment
     print()
@@ -444,14 +427,19 @@ def action_deploy_schema(cfg):
 
 
 def action_deploy_functions(cfg):
-    header('Deploy SQL functions (database/*.sql)')
+    header('Deploy SQL functions (database/queries/)')
     if not cfg:
         err(f'{ENV_FILE} not found — run option 1 to configure the database connection first')
         return
 
     db_location = f'database "{cfg["DB_NAME"]}" on {cfg["DB_HOST"]}:{cfg["DB_PORT"]}'
-    queries_dir = QUERIES_DIR
+    queries_dir = ROOT / 'database' / 'queries'
-    query_files = QUERY_FILES
+    query_files = [
+        queries_dir / 'sources.sql',
+        queries_dir / 'rules.sql',
+        queries_dir / 'mappings.sql',
+        queries_dir / 'records.sql',
+    ]
 
     print(f'  Source files: {queries_dir}/')
     for f in query_files:
@@ -740,116 +728,6 @@ def action_stop_service():
     ok('dataflow.service stopped')
 
 
-def action_uninstall(cfg):
-    """Reverse everything this script installs, outside the repo itself."""
-    header('Uninstall dataflow')
-
-    port      = cfg.get('API_PORT', '3020') if cfg else '3020'
-    db_name   = cfg.get('DB_NAME', 'dataflow') if cfg else 'dataflow'
-    db_user   = cfg.get('DB_USER', 'dataflow') if cfg else 'dataflow'
-    conf_path = nginx_conf_path(port)
-
-    # Everything that exists right now, in reverse install order
-    targets = []
-    if service_installed():
-        targets.append(f'systemd service   {SERVICE_FILE}' +
-                       ('  (running)' if service_running() else ''))
-    if conf_path:
-        targets.append(f'nginx site        {conf_path}')
-    if cfg and can_connect(cfg):
-        targets.append(f'database          "{db_name}" on {cfg["DB_HOST"]}:{cfg["DB_PORT"]}  (ALL DATA)')
-        targets.append(f'database user     {db_user}')
-    if ENV_FILE.exists():
-        targets.append(f'config            {ENV_FILE}')
-    if (ROOT / 'public').exists():
-        targets.append(f'built UI          {ROOT / "public"}')
-    if (ROOT / 'node_modules').exists():
-        targets.append(f'dependencies      {ROOT / "node_modules"}')
-
-    if not targets:
-        info('Nothing installed to remove.')
-        return cfg
-
-    print('  This will permanently remove:')
-    for t in targets:
-        print(f'    {t}')
-    print()
-    info(f'The repository itself ({ROOT}) is left alone — delete it manually if you want it gone.')
-    print()
-
-    if input("  Type 'delete' to confirm: ").strip() != 'delete':
-        info('Cancelled — no changes made')
-        return cfg
-
-    # ── Service ───────────────────────────────────────────────────────────────
-    if service_installed():
-        print()
-        print('  Removing systemd service...')
-        sudo_run(['systemctl', 'stop', 'dataflow'])
-        sudo_run(['systemctl', 'disable', 'dataflow'])
-        r = sudo_run(['rm', '-f', str(SERVICE_FILE)])
-        if r.returncode != 0:
-            err(f'Could not remove {SERVICE_FILE} — check sudo permissions')
-        else:
-            sudo_run(['systemctl', 'daemon-reload'])
-            ok(f'Service stopped, disabled, and {SERVICE_FILE} removed')
-
-    # ── nginx ─────────────────────────────────────────────────────────────────
-    if conf_path:
-        print()
-        print('  Removing nginx site...')
-        r = sudo_run(['rm', '-f', str(conf_path)])
-        if r.returncode != 0:
-            err(f'Could not remove {conf_path} — check sudo permissions')
-        elif sudo_run(['nginx', '-t'], capture_output=True).returncode != 0:
-            err('nginx config test failed after removal — not reloading; check nginx manually')
-        else:
-            sudo_run(['systemctl', 'reload', 'nginx'])
-            ok(f'{conf_path} removed and nginx reloaded')
-
-    # ── Database ──────────────────────────────────────────────────────────────
-    if cfg and can_connect(cfg):
-        print()
-        print(f'  Dropping the database requires PostgreSQL admin credentials.')
-        admin = {
-            'user':     prompt('PostgreSQL admin username', 'postgres'),
-            'password': prompt('PostgreSQL admin password', secret=True),
-            'host':     cfg['DB_HOST'],
-            'port':     cfg['DB_PORT'],
-        }
-        r = psql_admin(admin, 'SELECT 1')
-        if r.returncode != 0:
-            err(f'Cannot connect as admin — database and user left in place\n{r.stderr.strip()}')
-        else:
-            r = psql_admin(admin, f'DROP DATABASE IF EXISTS {db_name}')
-            if r.returncode != 0:
-                err(f'Could not drop database "{db_name}"\n{r.stderr.strip()}')
-            else:
-                ok(f'Database "{db_name}" dropped')
-                r = psql_admin(admin, f'DROP USER IF EXISTS {db_user}')
-                if r.returncode != 0:
-                    err(f'Could not drop user {db_user}\n{r.stderr.strip()}')
-                else:
-                    ok(f'User {db_user} dropped')
-
-    # ── Generated files ───────────────────────────────────────────────────────
-    print()
-    for path, label in [(ENV_FILE, 'config'),
-                        (ROOT / 'public', 'built UI'),
-                        (ROOT / 'node_modules', 'dependencies')]:
-        if not path.exists():
-            continue
-        if path.is_dir():
-            shutil.rmtree(path, ignore_errors=True)
-        else:
-            path.unlink()
-        ok(f'Removed {label} ({path})')
-
-    print()
-    ok('Uninstall complete')
-    return None
-
-
 def action_set_login_credentials(cfg):
     header('Set login credentials (LOGIN_USER / LOGIN_PASSWORD_HASH in .env)')
 
@@ -907,14 +785,13 @@ def action_set_login_credentials(cfg):
 MENU = [
     ('Database configuration and deployment dialog  (.env)',       action_configure),
     ('Redeploy "dataflow" schema only  (database/schema.sql)',    action_deploy_schema),
-    ('Redeploy SQL functions only      (database/*.sql)',         action_deploy_functions),
+    ('Redeploy SQL functions only      (database/queries/)',      action_deploy_functions),
     ('Build UI                  (ui/ → public/)',           action_build_ui),
     ('Set up nginx reverse proxy',                         action_setup_nginx),
     ('Install dataflow systemd service unit',               action_install_service),
     ('Start / restart dataflow.service',                    action_restart_service),
     ('Stop dataflow.service',                               action_stop_service),
     ('Set login credentials',                               action_set_login_credentials),
-    ('Uninstall  (service, nginx, database, .env, build)',  action_uninstall),
 ]
 
 def main():
@@ -927,11 +804,14 @@ def main():
         show_status(cfg)
 
         db_target = f'into "{cfg["DB_NAME"]}" on {cfg["DB_HOST"]}' if cfg else '(not configured)'
-        DB_ACTIONS = {action_deploy_schema, action_deploy_functions}
+        DB_ACTIONS = {
+            'Deploy "dataflow" schema  (database/schema.sql)',
+            'Deploy SQL functions      (database/functions.sql)',
+        }
 
         print(bold('Actions'))
-        for i, (label, fn) in enumerate(MENU, 1):
+        for i, (label, _) in enumerate(MENU, 1):
-            suffix = f'  {dim(db_target)}' if fn in DB_ACTIONS else ''
+            suffix = f'  {dim(db_target)}' if label in DB_ACTIONS else ''
             print(f'  {cyan(str(i))}. {label}{suffix}')
         print(f'  {cyan("q")}. Quit')
         print()
@@ -947,12 +827,13 @@ def main():
             if 0 <= idx < len(MENU):
                 label, fn = MENU[idx]
                 import inspect
-                # cfg is reloaded from .env at the top of every loop, so a return
+                sig = inspect.signature(fn)
-                # value is only ever informational
+                if len(sig.parameters) == 0:
-                if len(inspect.signature(fn).parameters) == 0:
+                    result = fn()
-                    fn()
+                elif len(sig.parameters) == 1:
-                else:
+                    result = fn(cfg)
-                    fn(cfg)
+                if label.startswith('Configure') and result is not None:
+                    cfg = result
                 pause()
             else:
                 warn('Invalid choice — enter a number from the list above')

package.json

@@ -4,8 +4,8 @@
   "description": "Simple data transformation tool for ingesting, mapping, and transforming data",
   "main": "api/server.js",
   "scripts": {
-    "start": "node api/server.js",
+    "start": "nodemon api/server.js",
-    "dev": "nodemon api/server.js",
+    "dev": "node api/server.js",
     "test": "echo \"Tests coming soon\" && exit 0"
   },
   "keywords": [
@@ -18,11 +18,11 @@
   "license": "MIT",
   "dependencies": {
     "bcrypt": "^6.0.0",
-    "csv-parse": "^6.2.1",
+    "csv-parse": "^5.5.2",
-    "dotenv": "^17.4.2",
+    "dotenv": "^16.3.1",
-    "express": "^5.2.1",
+    "express": "^4.18.2",
-    "multer": "^2.1.1",
+    "multer": "^1.4.5-lts.1",
-    "pg": "^8.21.0"
+    "pg": "^8.11.3"
   },
   "devDependencies": {
     "nodemon": "^3.0.1"

scripts/setup-service.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+set -e
+
+SERVICE_NAME="dataflow"
+SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service"
+WORKDIR=$(pwd)
+
+echo "Creating systemd service file..."
+sudo tee "$SERVICE_FILE" > /dev/null <<EOF
+[Unit]
+Description=Dataflow API Server
+After=postgresql.service
+Wants=postgresql.service
+
+[Service]
+Type=simple
+User=www-data
+WorkingDirectory=${WORKDIR}
+Environment=NODE_ENV=production
+ExecStart=/usr/bin/node api/server.js
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+echo "Reloading systemd daemon..."
+sudo systemctl daemon-reload
+
+echo "Enabling dataflow service..."
+sudo systemctl enable "$SERVICE_NAME"
+
+echo "Starting dataflow service..."
+sudo systemctl start "$SERVICE_NAME"
+
+echo "Done! Service status:"
+sudo systemctl status "$SERVICE_NAME" --no-pager

ui/.gitignore

@@ -0,0 +1,24 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/*
+!.vscode/extensions.json
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?

ui/README.md

@@ -0,0 +1,16 @@
+# React + Vite
+
+This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
+
+Currently, two official plugins are available:
+
+- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
+- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
+
+## React Compiler
+
+The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
+
+## Expanding the ESLint configuration
+
+If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.

ui/package.json

@@ -10,26 +10,21 @@
     "preview": "vite preview"
   },
   "dependencies": {
-    "@perspective-dev/client": "^4.5.1",
+    "react": "^19.2.4",
-    "@perspective-dev/viewer": "^4.5.1",
+    "react-dom": "^19.2.4",
-    "@perspective-dev/viewer-d3fc": "^4.4.1",
+    "react-router-dom": "^7.13.2"
-    "@perspective-dev/viewer-datagrid": "^4.5.1",
-    "react": "^19.2.7",
-    "react-dom": "^19.2.7",
-    "react-router-dom": "^7.17.0",
-    "sql-formatter": "^15.8.1"
   },
   "devDependencies": {
     "@eslint/js": "^9.39.4",
-    "@tailwindcss/vite": "^4.3.1",
+    "@tailwindcss/vite": "^4.2.2",
-    "@types/react": "^19.2.17",
+    "@types/react": "^19.2.14",
     "@types/react-dom": "^19.2.3",
-    "@vitejs/plugin-react": "^6.0.2",
+    "@vitejs/plugin-react": "^6.0.1",
     "eslint": "^9.39.4",
-    "eslint-plugin-react-hooks": "^7.1.1",
+    "eslint-plugin-react-hooks": "^7.0.1",
     "eslint-plugin-react-refresh": "^0.5.2",
-    "globals": "^17.6.0",
+    "globals": "^17.4.0",
-    "tailwindcss": "^4.3.1",
+    "tailwindcss": "^4.2.2",
-    "vite": "^8.0.16"
+    "vite": "^8.0.1"
   }
 }

ui/src/App.css

@@ -0,0 +1 @@
+/* App-level styles — layout handled by Tailwind */

ui/src/App.jsx

@@ -1,8 +1,6 @@
 import { useState, useEffect } from 'react'
-import { BrowserRouter, Routes, Route, Navigate } from 'react-router-dom'
+import { BrowserRouter, Routes, Route, NavLink, Navigate } from 'react-router-dom'
 import { api, setCredentials, clearCredentials } from './api'
-import StatusBar from './components/StatusBar.jsx'
-import Sidebar from './components/Sidebar.jsx'
 import Login from './pages/Login'
 import Sources from './pages/Sources'
 import Import from './pages/Import'
@@ -12,32 +10,35 @@ import Records from './pages/Records'
 import Log from './pages/Log'
 import Pivot from './pages/Pivot'
 import Remap from './pages/Remap'
-import Stacks from './pages/Stacks'
+
+const NAV = [
+  { to: '/sources', label: 'Sources' },
+  { to: '/import', label: 'Import' },
+  { to: '/rules', label: 'Rules' },
+  { to: '/mappings', label: 'Mappings' },
+  { to: '/remap', label: 'Remap' },
+  { to: '/records', label: 'Records' },
+  { to: '/pivot', label: 'Pivot' },
+  { to: '/log', label: 'Log' },
+]
 
 export default function App() {
   const [authed, setAuthed] = useState(false)
   const [loginUser, setLoginUser] = useState('')
   const [sources, setSources] = useState([])
-  const [stacks, setStacks] = useState([])
   const [source, setSource] = useState(() => localStorage.getItem('selectedSource') || '')
-  const [selectedStack, setSelectedStack] = useState(null)
+  const [sidebarOpen, setSidebarOpen] = useState(false)
-  const [sidebarExpanded, setSidebarExpanded] = useState(() => localStorage.getItem('df_sidebar') !== 'collapsed')
-  // Sets of names whose dfv view is out of sync with current definitions
-  const [staleSources, setStaleSources] = useState(new Set())
-  const [staleStacks, setStaleStacks] = useState(new Set())
-  const [reprocessSources, setReprocessSources] = useState(new Set())
-  const [generating, setGenerating] = useState({}) // { 'source:name': true }
 
   async function handleLogin(user, pass) {
     setCredentials(user, pass)
-    const s = await api.getSources()
+    await api.getSources().then(s => {
-    sessionStorage.setItem('df_user', user)
+      sessionStorage.setItem('df_user', user)
-    sessionStorage.setItem('df_pass', pass)
+      sessionStorage.setItem('df_pass', pass)
-    setSources(s)
+      setSources(s)
-    if (!source && s.length > 0) setSource(s[0].name)
+      if (!source && s.length > 0) setSource(s[0].name)
-    setAuthed(true)
+      setAuthed(true)
-    setLoginUser(user)
+      setLoginUser(user)
-    api.getStacks().then(setStacks).catch(() => {})
+    })
   }
 
   function handleLogout() {
@@ -47,63 +48,6 @@ export default function App() {
     setAuthed(false)
     setLoginUser('')
     setSources([])
-    setStacks([])
-    setSelectedStack(null)
-    setStaleSources(new Set())
-    setStaleStacks(new Set())
-    setReprocessSources(new Set())
-  }
-
-  function refreshStacks() {
-    api.getStacks().then(setStacks).catch(() => {})
-  }
-
-  // Load initial stale state from DB once on login
-  useEffect(() => {
-    if (!authed) return
-    api.getStatus().then(s => {
-      setStaleSources(new Set((s.stale_sources || []).map(x => x.name)))
-      setStaleStacks(new Set((s.stale_stacks || []).map(x => x.name)))
-    }).catch(() => {})
-  }, [authed])
-
-  function markSourceStale(name) {
-    setStaleSources(prev => new Set([...prev, name]))
-  }
-  function markNeedsReprocess(name) {
-    setReprocessSources(prev => new Set([...prev, name]))
-  }
-  async function handleReprocessSource(name) {
-    setGenerating(g => ({ ...g, [`rp:${name}`]: true }))
-    try {
-      await api.reprocess(name)
-      setReprocessSources(prev => { const n = new Set(prev); n.delete(name); return n })
-    } catch (e) { alert(e.message) }
-    finally { setGenerating(g => { const n = { ...g }; delete n[`rp:${name}`]; return n }) }
-  }
-  function markStackStale(name) {
-    setStaleStacks(prev => new Set([...prev, name]))
-  }
-  function clearStackStale(name) {
-    setStaleStacks(prev => { const n = new Set(prev); n.delete(name); return n })
-  }
-
-  async function handleGenerateSource(name) {
-    setGenerating(g => ({ ...g, [`src:${name}`]: true }))
-    try {
-      await api.generateView(name)
-      setStaleSources(prev => { const n = new Set(prev); n.delete(name); return n })
-    } catch (e) { alert(e.message) }
-    finally { setGenerating(g => { const n = { ...g }; delete n[`src:${name}`]; return n }) }
-  }
-
-  async function handleGenerateStack(name) {
-    setGenerating(g => ({ ...g, [`stk:${name}`]: true }))
-    try {
-      await api.generateStackView(name)
-      setStaleStacks(prev => { const n = new Set(prev); n.delete(name); return n })
-    } catch (e) { alert(e.message) }
-    finally { setGenerating(g => { const n = { ...g }; delete n[`stk:${name}`]; return n }) }
   }
 
   // On mount, restore session if credentials are saved
@@ -117,89 +61,94 @@ export default function App() {
     if (source) localStorage.setItem('selectedSource', source)
   }, [source])
 
-  useEffect(() => {
-    localStorage.setItem('df_sidebar', sidebarExpanded ? 'expanded' : 'collapsed')
-  }, [sidebarExpanded])
-
   if (!authed) return <Login onLogin={handleLogin} />
 
+  const sidebar = (
+    <div className="flex flex-col h-full">
+      {/* Header */}
+      <div className="px-4 py-3 border-b border-gray-200">
+        <div className="flex items-center justify-between">
+          <span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
+          <button onClick={() => setSidebarOpen(false)} className="md:hidden text-gray-400 hover:text-gray-600 leading-none" title="Close">✕</button>
+        </div>
+        <div className="flex items-center justify-between mt-1">
+          <span className="text-xs text-gray-400">{loginUser}</span>
+          <button onClick={handleLogout} className="text-xs text-gray-400 hover:text-red-500" title="Sign out">Sign out</button>
+        </div>
+      </div>
+
+      {/* Source selector */}
+      <div className="px-3 py-3 border-b border-gray-200">
+        <div className="flex items-center justify-between mb-1">
+          <label className="text-xs text-gray-500">Source</label>
+          <NavLink to="/sources?new=1" className="text-xs text-blue-400 hover:text-blue-600 leading-none" title="New source" onClick={() => setSidebarOpen(false)}>+</NavLink>
+        </div>
+        <select
+          className="w-full text-sm border border-gray-200 rounded px-2 py-1 bg-white focus:outline-none focus:border-blue-400"
+          value={source}
+          onChange={e => setSource(e.target.value)}
+        >
+          {sources.length === 0 && <option value="">—</option>}
+          {sources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
+        </select>
+      </div>
+
+      {/* Nav */}
+      <nav className="flex-1 py-2">
+        {NAV.map(({ to, label }) => (
+          <NavLink
+            key={to}
+            to={to}
+            onClick={() => setSidebarOpen(false)}
+            className={({ isActive }) =>
+              `block px-4 py-2 text-sm ${isActive
+                ? 'bg-blue-50 text-blue-700 font-medium'
+                : 'text-gray-600 hover:bg-gray-50'}`
+            }
+          >
+            {label}
+          </NavLink>
+        ))}
+      </nav>
+    </div>
+  )
+
   return (
     <BrowserRouter>
-      <div className="flex h-screen">
+      <div className="flex h-screen bg-gray-50">
 
-        <Sidebar
+        {/* Mobile overlay */}
-          expanded={sidebarExpanded}
+        {sidebarOpen && (
-          setExpanded={setSidebarExpanded}
+          <div className="fixed inset-0 z-20 bg-black/30 md:hidden" onClick={() => setSidebarOpen(false)} />
-          loginUser={loginUser}
+        )}
-          onLogout={handleLogout}
+
-        />
+        {/* Sidebar — fixed on mobile, static on desktop */}
+        <div className={`
+          fixed inset-y-0 left-0 z-30 w-44 bg-white border-r border-gray-200 transform transition-transform duration-200
+          md:static md:translate-x-0 md:z-auto md:transition-none
+          ${sidebarOpen ? 'translate-x-0' : '-translate-x-full'}
+        `}>
+          {sidebar}
+        </div>
 
         {/* Main */}
-        <div className="flex-1 overflow-hidden flex flex-col min-w-0">
+        <div className="flex-1 overflow-auto flex flex-col min-w-0">
-          <StatusBar
+          {/* Mobile top bar */}
-            sources={sources} source={source} setSource={setSource}
+          <div className="md:hidden flex items-center px-3 py-2 bg-white border-b border-gray-200">
-            stacks={stacks} selectedStack={selectedStack} setSelectedStack={setSelectedStack}
+            <button onClick={() => setSidebarOpen(true)} className="text-gray-500 hover:text-gray-700 mr-3 text-lg leading-none">☰</button>
-          />
+            <span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
-
+          </div>
-          {(staleSources.size > 0 || staleStacks.size > 0) && (
-            <div className="bg-amber-50 border-b border-amber-200 px-4 py-1.5 text-xs text-amber-800 flex flex-wrap items-center gap-x-3 gap-y-1">
-              <span className="font-medium">View out of sync:</span>
-              {[...staleSources].map(name => (
-                <span key={name} className="flex items-center gap-1">
-                  {name}
-                  <button
-                    onClick={() => handleGenerateSource(name)}
-                    disabled={generating[`src:${name}`]}
-                    className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
-                  >
-                    {generating[`src:${name}`] ? '…' : 'Generate'}
-                  </button>
-                </span>
-              ))}
-              {staleSources.size > 0 && staleStacks.size > 0 && <span className="text-amber-400">|</span>}
-              {[...staleStacks].map(name => (
-                <span key={name} className="flex items-center gap-1">
-                  stack: {name}
-                  <button
-                    onClick={() => handleGenerateStack(name)}
-                    disabled={generating[`stk:${name}`]}
-                    className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
-                  >
-                    {generating[`stk:${name}`] ? '…' : 'Generate'}
-                  </button>
-                </span>
-              ))}
-            </div>
-          )}
-          {reprocessSources.size > 0 && (
-            <div className="bg-blue-50 border-b border-blue-200 px-4 py-1.5 text-xs text-blue-800 flex flex-wrap items-center gap-x-3 gap-y-1">
-              <span className="font-medium">Mappings updated:</span>
-              {[...reprocessSources].map(name => (
-                <span key={name} className="flex items-center gap-1">
-                  {name}
-                  <button
-                    onClick={() => handleReprocessSource(name)}
-                    disabled={generating[`rp:${name}`]}
-                    className="px-1.5 py-0.5 rounded bg-blue-200 hover:bg-blue-300 disabled:opacity-50 font-medium"
-                  >
-                    {generating[`rp:${name}`] ? '…' : 'Reprocess'}
-                  </button>
-                </span>
-              ))}
-            </div>
-          )}
 
           <div className="flex-1 overflow-auto">
             <Routes>
               <Route path="/" element={<Navigate to="/sources" replace />} />
               <Route path="/sources" element={<Sources source={source} sources={sources} setSources={setSources} setSource={setSource} />} />
               <Route path="/import" element={<Import source={source} />} />
-              <Route path="/rules" element={<Rules source={source} onStale={markSourceStale} />} />
+              <Route path="/rules" element={<Rules source={source} />} />
-              <Route path="/mappings" element={<Mappings source={source} onNeedsReprocess={markNeedsReprocess} />} />
+              <Route path="/mappings" element={<Mappings source={source} />} />
               <Route path="/remap" element={<Remap />} />
               <Route path="/records" element={<Records source={source} />} />
-              <Route path="/pivot" element={<Pivot source={source} selectedStack={selectedStack} setSelectedStack={setSelectedStack} />} />
+              <Route path="/pivot" element={<Pivot source={source} />} />
-              <Route path="/stacks" element={<Stacks sources={sources} onStackStale={markStackStale} onStackViewGenerated={clearStackStale} onStacksChange={refreshStacks} />} />
               <Route path="/log" element={<Log />} />
             </Routes>
           </div>

ui/src/api.js

@@ -108,40 +108,12 @@ export const api = {
   getMappingsByOutputField: (col, val) => request('GET', `/mappings/outputs/${encodeURIComponent(col)}/${encodeURIComponent(val)}`),
   remapOutputField: (col, from_val, to_val) => request('POST', '/mappings/remap-field', { col, from_val, to_val }),
 
-  // Pivot layouts (sources)
+  // Pivot layouts
   getPivotLayouts: (source) => request('GET', `/sources/${source}/layouts`),
   savePivotLayout: (source, layout_name, config) => request('POST', `/sources/${source}/layouts`, { layout_name, config }),
   deletePivotLayout: (source, id) => request('DELETE', `/sources/${source}/layouts/${id}`),
 
-  // Pivot layouts (stacks)
-  getStackPivotLayouts: (name) => request('GET', `/stacks/${name}/layouts`),
-  saveStackPivotLayout: (name, layout_name, config) => request('POST', `/stacks/${name}/layouts`, { layout_name, config }),
-  deleteStackPivotLayout: (name, id) => request('DELETE', `/stacks/${name}/layouts/${id}`),
-
-  // Stacks
-  getStacks: () => request('GET', '/stacks'),
-  getStack: (name) => request('GET', `/stacks/${name}`),
-  createStack: (body) => request('POST', '/stacks', body),
-  updateStack: (name, body) => request('PUT', `/stacks/${name}`, body),
-  deleteStack: (name) => request('DELETE', `/stacks/${name}`),
-  upsertStackSource: (name, source, body) => request('PUT', `/stacks/${name}/sources/${source}`, body),
-  reorderStackSources: (name, source_names) => request('PUT', `/stacks/${name}/sources/reorder`, { source_names }),
-  removeStackSource: (name, source) => request('DELETE', `/stacks/${name}/sources/${source}`),
-  previewStackSql: (name) => request('GET', `/stacks/${name}/view-sql`),
-  generateStackView: (name) => request('POST', `/stacks/${name}/view`),
-  execStackSql: (name, sql) => request('POST', `/stacks/${name}/exec-sql`, { sql }),
-  getStackBalance: (name) => request('GET', `/stacks/${name}/balance`),
-  calibrateBalance: (name, source, body) => request('POST', `/stacks/${name}/calibrate`, { ...body, source_name: source || null }),
-
-  // Status
-  getStatus: () => request('GET', '/status'),
-
   // Records
   getRecords: (source, limit = 100, offset = 0) =>
     request('GET', `/records/source/${source}?limit=${limit}&offset=${offset}`),
-  getRecord: (id) => request('GET', `/records/${id}`),
-  getOverrideKeys: (source) => request('GET', `/sources/${source}/override-keys`),
-  setBulkRecordOverrides: (source, recordIds, overrides) => request('POST', `/records/bulk-overrides`, { source_name: source, record_ids: recordIds, overrides }),
-  setRecordOverrides: (id, overrides) => request('PUT', `/records/${id}/overrides`, { overrides }),
-  clearRecordOverrides: (id) => request('DELETE', `/records/${id}/overrides`),
 }

ui/src/assets/react.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="35.93" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 228"><path fill="#00D8FF" d="M210.483 73.824a171.49 171.49 0 0 0-8.24-2.597c.465-1.9.893-3.777 1.273-5.621c6.238-30.281 2.16-54.676-11.769-62.708c-13.355-7.7-35.196.329-57.254 19.526a171.23 171.23 0 0 0-6.375 5.848a155.866 155.866 0 0 0-4.241-3.917C100.759 3.829 77.587-4.822 63.673 3.233C50.33 10.957 46.379 33.89 51.995 62.588a170.974 170.974 0 0 0 1.892 8.48c-3.28.932-6.445 1.924-9.474 2.98C17.309 83.498 0 98.307 0 113.668c0 15.865 18.582 31.778 46.812 41.427a145.52 145.52 0 0 0 6.921 2.165a167.467 167.467 0 0 0-2.01 9.138c-5.354 28.2-1.173 50.591 12.134 58.266c13.744 7.926 36.812-.22 59.273-19.855a145.567 145.567 0 0 0 5.342-4.923a168.064 168.064 0 0 0 6.92 6.314c21.758 18.722 43.246 26.282 56.54 18.586c13.731-7.949 18.194-32.003 12.4-61.268a145.016 145.016 0 0 0-1.535-6.842c1.62-.48 3.21-.974 4.76-1.488c29.348-9.723 48.443-25.443 48.443-41.52c0-15.417-17.868-30.326-45.517-39.844Zm-6.365 70.984c-1.4.463-2.836.91-4.3 1.345c-3.24-10.257-7.612-21.163-12.963-32.432c5.106-11 9.31-21.767 12.459-31.957c2.619.758 5.16 1.557 7.61 2.4c23.69 8.156 38.14 20.213 38.14 29.504c0 9.896-15.606 22.743-40.946 31.14Zm-10.514 20.834c2.562 12.94 2.927 24.64 1.23 33.787c-1.524 8.219-4.59 13.698-8.382 15.893c-8.067 4.67-25.32-1.4-43.927-17.412a156.726 156.726 0 0 1-6.437-5.87c7.214-7.889 14.423-17.06 21.459-27.246c12.376-1.098 24.068-2.894 34.671-5.345a134.17 134.17 0 0 1 1.386 6.193ZM87.276 214.515c-7.882 2.783-14.16 2.863-17.955.675c-8.075-4.657-11.432-22.636-6.853-46.752a156.923 156.923 0 0 1 1.869-8.499c10.486 2.32 22.093 3.988 34.498 4.994c7.084 9.967 14.501 19.128 21.976 27.15a134.668 134.668 0 0 1-4.877 4.492c-9.933 8.682-19.886 14.842-28.658 17.94ZM50.35 144.747c-12.483-4.267-22.792-9.812-29.858-15.863c-6.35-5.437-9.555-10.836-9.555-15.216c0-9.322 13.897-21.212 37.076-29.293c2.813-.98 5.757-1.905 8.812-2.773c3.204 10.42 7.406 21.315 12.477 32.332c-5.137 11.18-9.399 22.249-12.634 32.792a134.718 134.718 0 0 1-6.318-1.979Zm12.378-84.26c-4.811-24.587-1.616-43.134 6.425-47.789c8.564-4.958 27.502 2.111 47.463 19.835a144.318 144.318 0 0 1 3.841 3.545c-7.438 7.987-14.787 17.08-21.808 26.988c-12.04 1.116-23.565 2.908-34.161 5.309a160.342 160.342 0 0 1-1.76-7.887Zm110.427 27.268a347.8 347.8 0 0 0-7.785-12.803c8.168 1.033 15.994 2.404 23.343 4.08c-2.206 7.072-4.956 14.465-8.193 22.045a381.151 381.151 0 0 0-7.365-13.322Zm-45.032-43.861c5.044 5.465 10.096 11.566 15.065 18.186a322.04 322.04 0 0 0-30.257-.006c4.974-6.559 10.069-12.652 15.192-18.18ZM82.802 87.83a323.167 323.167 0 0 0-7.227 13.238c-3.184-7.553-5.909-14.98-8.134-22.152c7.304-1.634 15.093-2.97 23.209-3.984a321.524 321.524 0 0 0-7.848 12.897Zm8.081 65.352c-8.385-.936-16.291-2.203-23.593-3.793c2.26-7.3 5.045-14.885 8.298-22.6a321.187 321.187 0 0 0 7.257 13.246c2.594 4.48 5.28 8.868 8.038 13.147Zm37.542 31.03c-5.184-5.592-10.354-11.779-15.403-18.433c4.902.192 9.899.29 14.978.29c5.218 0 10.376-.117 15.453-.343c-4.985 6.774-10.018 12.97-15.028 18.486Zm52.198-57.817c3.422 7.8 6.306 15.345 8.596 22.52c-7.422 1.694-15.436 3.058-23.88 4.071a382.417 382.417 0 0 0 7.859-13.026a347.403 347.403 0 0 0 7.425-13.565Zm-16.898 8.101a358.557 358.557 0 0 1-12.281 19.815a329.4 329.4 0 0 1-23.444.823c-7.967 0-15.716-.248-23.178-.732a310.202 310.202 0 0 1-12.513-19.846h.001a307.41 307.41 0 0 1-10.923-20.627a310.278 310.278 0 0 1 10.89-20.637l-.001.001a307.318 307.318 0 0 1 12.413-19.761c7.613-.576 15.42-.876 23.31-.876H128c7.926 0 15.743.303 23.354.883a329.357 329.357 0 0 1 12.335 19.695a358.489 358.489 0 0 1 11.036 20.54a329.472 329.472 0 0 1-11 20.722Zm22.56-122.124c8.572 4.944 11.906 24.881 6.52 51.026c-.344 1.668-.73 3.367-1.15 5.09c-10.622-2.452-22.155-4.275-34.23-5.408c-7.034-10.017-14.323-19.124-21.64-27.008a160.789 160.789 0 0 1 5.888-5.4c18.9-16.447 36.564-22.941 44.612-18.3ZM128 90.808c12.625 0 22.86 10.235 22.86 22.86s-10.235 22.86-22.86 22.86s-22.86-10.235-22.86-22.86s10.235-22.86 22.86-22.86Z"></path></svg>

ui/src/components/Sidebar.jsx

@@ -1,183 +0,0 @@
-import { NavLink } from 'react-router-dom'
-
-const NAV = [
-  {
-    to: '/sources',
-    label: 'Sources',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <ellipse cx="10" cy="5.5" rx="7" ry="2.5"/>
-        <path d="M3 5.5v9c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5v-9"/>
-        <path d="M3 10.5c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/import',
-    label: 'Import',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <line x1="10" y1="3" x2="10" y2="14"/>
-        <polyline points="6,10 10,14 14,10"/>
-        <line x1="3" y1="18" x2="17" y2="18"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/rules',
-    label: 'Rules',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <polyline points="6,7 2,10 6,13"/>
-        <polyline points="14,7 18,10 14,13"/>
-        <line x1="12" y1="4" x2="8" y2="16"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/mappings',
-    label: 'Mappings',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <line x1="2" y1="7" x2="12" y2="7"/>
-        <polyline points="9,4 12,7 9,10"/>
-        <line x1="8" y1="13" x2="18" y2="13"/>
-        <polyline points="11,10 14,13 11,16"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/remap',
-    label: 'Remap',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <polyline points="2,8 2,4 6,4"/>
-        <path d="M2 4a8 8 0 0 1 14 2"/>
-        <polyline points="18,12 18,16 14,16"/>
-        <path d="M18 16a8 8 0 0 1-14-2"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/records',
-    label: 'Records',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <rect x="2" y="3" width="16" height="14" rx="1.5"/>
-        <line x1="2" y1="8" x2="18" y2="8"/>
-        <line x1="7" y1="8" x2="7" y2="17"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/pivot',
-    label: 'Pivot',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <rect x="2" y="2" width="7" height="7" rx="1"/>
-        <rect x="11" y="2" width="7" height="7" rx="1"/>
-        <rect x="2" y="11" width="7" height="7" rx="1"/>
-        <rect x="11" y="11" width="7" height="7" rx="1"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/stacks',
-    label: 'Stacks',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <polygon points="10,2 18,6 10,10 2,6"/>
-        <polyline points="2,10 10,14 18,10"/>
-        <polyline points="2,14 10,18 18,14"/>
-      </svg>
-    ),
-  },
-  {
-    to: '/log',
-    label: 'Log',
-    icon: (
-      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
-        <circle cx="10" cy="10" r="8"/>
-        <polyline points="10,5 10,10 14,12"/>
-      </svg>
-    ),
-  },
-]
-
-export default function Sidebar({ expanded, setExpanded, loginUser, onLogout }) {
-  return (
-    <div
-      className="bg-white border-r border-gray-200 flex flex-col shrink-0 overflow-hidden transition-all duration-150"
-      style={{ width: expanded ? 200 : 48 }}
-    >
-      {/* Header */}
-      <div className="h-12 flex items-center px-3 border-b border-gray-100 gap-2 shrink-0">
-        <button
-          onClick={() => setExpanded(e => !e)}
-          className="w-8 h-8 flex items-center justify-center rounded hover:bg-gray-100 text-gray-400 shrink-0"
-          title="Toggle sidebar"
-        >
-          <svg width="16" height="16" viewBox="0 0 16 16" fill="none">
-            <rect x="1" y="3" width="14" height="1.5" rx="0.75" fill="currentColor"/>
-            <rect x="1" y="7.25" width="14" height="1.5" rx="0.75" fill="currentColor"/>
-            <rect x="1" y="11.5" width="14" height="1.5" rx="0.75" fill="currentColor"/>
-          </svg>
-        </button>
-        <span
-          className="text-xs font-semibold text-gray-600 tracking-wide uppercase whitespace-nowrap transition-opacity duration-100"
-          style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none' }}
-        >
-          Dataflow
-        </span>
-      </div>
-
-      {/* Nav */}
-      <nav className="flex flex-col gap-0.5 p-2 flex-1">
-        {NAV.map(({ to, label, icon }) => (
-          <NavLink
-            key={to}
-            to={to}
-            title={!expanded ? label : undefined}
-            className={({ isActive }) =>
-              `flex items-center gap-3 px-2 py-2 rounded w-full transition-colors ${
-                isActive
-                  ? 'bg-blue-50 text-blue-700'
-                  : 'text-gray-500 hover:bg-gray-100 hover:text-gray-800'
-              }`
-            }
-          >
-            <span className="shrink-0">{icon}</span>
-            <span
-              className="text-sm whitespace-nowrap transition-opacity duration-100"
-              style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
-            >
-              {label}
-            </span>
-          </NavLink>
-        ))}
-      </nav>
-
-      {/* User / logout */}
-      <div className="border-t border-gray-100 px-3 py-2.5 flex items-center gap-2 shrink-0 overflow-hidden">
-        <div
-          className="w-6 h-6 rounded-full bg-gray-200 text-gray-500 flex items-center justify-center shrink-0 text-xs font-medium"
-          title={!expanded ? loginUser : undefined}
-        >
-          {loginUser ? loginUser[0].toUpperCase() : '?'}
-        </div>
-        <div
-          className="flex-1 flex items-center justify-between min-w-0 transition-opacity duration-100"
-          style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
-        >
-          <span className="text-xs text-gray-400 truncate">{loginUser}</span>
-          <button
-            onClick={onLogout}
-            className="text-xs text-gray-400 hover:text-red-500 ml-2 shrink-0"
-          >
-            Sign out
-          </button>
-        </div>
-      </div>
-    </div>
-  )
-}

ui/src/components/StatusBar.jsx

@@ -1,73 +0,0 @@
-import { NavLink } from 'react-router-dom'
-import useTheme from '../theme.jsx'
-
-export default function StatusBar({ sources = [], source, setSource, stacks = [], selectedStack, setSelectedStack }) {
-  const { dark, setDark } = useTheme()
-
-  return (
-    <div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs">
-      <span className="text-gray-400">Source</span>
-      <select
-        value={source || ''}
-        onChange={e => setSource(e.target.value)}
-        disabled={sources.length === 0}
-        className="border border-gray-200 rounded px-2 py-0.5 bg-white focus:outline-none focus:border-blue-400"
-      >
-        {sources.length === 0
-          ? <option value="">— no sources —</option>
-          : sources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
-      </select>
-      <NavLink
-        to="/sources?new=1"
-        className="text-blue-400 hover:text-blue-600 leading-none"
-        title="New source"
-      >+</NavLink>
-
-      {stacks.length > 0 && (
-        <>
-          <span className="text-gray-200">|</span>
-          <span className="text-gray-400">Stacks</span>
-          {stacks.map(s => (
-            <button
-              key={s.name}
-              onClick={() => setSelectedStack(n => n === s.name ? null : s.name)}
-              className={`rounded px-2 py-0.5 border transition-colors ${
-                selectedStack === s.name
-                  ? 'bg-purple-50 border-purple-300 text-purple-700'
-                  : 'bg-white border-gray-200 text-gray-500 hover:border-gray-400'
-              }`}
-            >
-              {s.name}
-            </button>
-          ))}
-        </>
-      )}
-
-      <div className="ml-auto">
-        <button
-          onClick={() => setDark(d => !d)}
-          className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100 text-gray-500"
-          title={dark ? 'Switch to light mode' : 'Switch to dark mode'}
-        >
-          {dark ? (
-            <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
-              <circle cx="12" cy="12" r="4"/>
-              <line x1="12" y1="2" x2="12" y2="5"/>
-              <line x1="12" y1="19" x2="12" y2="22"/>
-              <line x1="4.93" y1="4.93" x2="7.05" y2="7.05"/>
-              <line x1="16.95" y1="16.95" x2="19.07" y2="19.07"/>
-              <line x1="2" y1="12" x2="5" y2="12"/>
-              <line x1="19" y1="12" x2="22" y2="12"/>
-              <line x1="4.93" y1="19.07" x2="7.05" y2="16.95"/>
-              <line x1="16.95" y1="7.05" x2="19.07" y2="4.93"/>
-            </svg>
-          ) : (
-            <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
-              <path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
-            </svg>
-          )}
-        </button>
-      </div>
-    </div>
-  )
-}

ui/src/index.css

@@ -1,97 +1,6 @@
 @import "tailwindcss";
 
-:root, .light {
-  --bg-primary: #f3f4f6;
-  --bg-secondary: #ffffff;
-  --bg-tertiary: #f9fafb;
-  --text-primary: #1f2937;
-  --text-secondary: #374151;
-  --text-muted: #9ca3af;
-  --border-color: #e5e7eb;
-  --border-light: #f3f4f6;
-  --accent-bg: #eff6ff;
-  --accent-text: #1d4ed8;
-}
-
-/* Dark palette tuned to Perspective's "Pro Dark" theme:
-   bg #242526, tooltip #2a2c2f, gridline #3b3f46, inactive #61656e,
-   inactive border #4c505b, active #2770a9, legend #c5c9d0. */
-.dark {
-  --bg-primary: #242526;
-  --bg-secondary: #2a2c2f;
-  --bg-tertiary: #3b3f46;
-  --text-primary: #ffffff;
-  --text-secondary: #c5c9d0;
-  --text-muted: #61656e;
-  --border-color: #4c505b;
-  --border-light: #3b3f46;
-  --accent-bg: rgba(39, 113, 170, 0.32);
-  --accent-text: #4778c2;
-}
-
 body {
   margin: 0;
   font-family: system-ui, -apple-system, sans-serif;
-  background-color: var(--bg-primary);
-  color: var(--text-primary);
 }
-
-.dark .bg-white { background-color: var(--bg-secondary); }
-.dark .bg-gray-50 { background-color: var(--bg-tertiary); }
-.dark .bg-gray-100 { background-color: var(--bg-tertiary); }
-.dark .bg-gray-200 { background-color: var(--bg-tertiary); }
-.dark .bg-gray-300 { background-color: var(--bg-tertiary); }
-.dark .text-gray-300 { color: var(--text-muted); }
-.dark .text-gray-400 { color: var(--text-muted); }
-.dark .text-gray-500 { color: var(--text-muted); }
-.dark .text-gray-600 { color: var(--text-secondary); }
-.dark .text-gray-700 { color: var(--text-secondary); }
-.dark .text-gray-800 { color: var(--text-primary); }
-.dark .text-gray-900 { color: var(--text-primary); }
-.dark .bg-blue-50 { background-color: var(--accent-bg); }
-.dark .bg-blue-100 { background-color: var(--accent-bg); }
-.dark .text-blue-400 { color: var(--accent-text); }
-.dark .text-blue-600 { color: var(--accent-text); }
-.dark .text-blue-700 { color: var(--accent-text); }
-.dark .text-blue-800 { color: var(--accent-text); }
-.dark .border-blue-200 { border-color: var(--accent-text); }
-.dark .border-blue-300 { border-color: var(--accent-text); }
-.dark .hover\:bg-blue-50:hover { background-color: var(--accent-bg); }
-
-/* Status accents — desaturated to sit on Pro Dark's neutral background */
-.dark .bg-green-50 { background-color: #1a3d2c; }
-.dark .text-green-600 { color: #6ee7b7; }
-.dark .text-green-700 { color: #6ee7b7; }
-.dark .text-green-400 { color: #6ee7b7; }
-.dark .bg-amber-50 { background-color: #3a2e14; }
-.dark .text-amber-800 { color: #f5c66f; }
-.dark .border-amber-200 { border-color: #5a4a26; }
-.dark .bg-amber-200 { background-color: #5a4a26; }
-.dark .hover\:bg-amber-300:hover { background-color: #6b5830; }
-.dark .bg-red-50 { background-color: #3d1f1f; }
-.dark .text-red-500 { color: #ff9485; }
-.dark .text-red-700 { color: #ff9485; }
-.dark .border-gray-100 { border-color: var(--border-light); }
-.dark .border-gray-200 { border-color: var(--border-color); }
-.dark .border-gray-300 { border-color: var(--border-color); }
-.dark .border-blue-100 { border-color: var(--border-color); }
-.dark .border-b { border-color: var(--border-color); }
-.dark .border-t { border-color: var(--border-color); }
-.dark .border-r { border-color: var(--border-color); }
-.dark .border-l { border-color: var(--border-color); }
-.dark .hover\:bg-gray-50:hover { background-color: var(--bg-tertiary); }
-.dark .hover\:bg-gray-100:hover { background-color: var(--bg-tertiary); }
-.dark .hover\:bg-gray-200:hover { background-color: var(--bg-tertiary); }
-.dark .hover\:text-gray-500:hover { color: var(--text-secondary); }
-.dark .hover\:text-gray-600:hover { color: var(--text-secondary); }
-.dark .hover\:text-gray-700:hover { color: var(--text-primary); }
-.dark .hover\:text-gray-800:hover { color: var(--text-primary); }
-.dark .hover\:border-gray-300:hover { border-color: var(--border-color); }
-.dark .hover\:border-gray-400:hover { border-color: var(--border-color); }
-.dark .focus\:border-gray-300:focus { border-color: var(--border-color); }
-.dark .focus\:border-blue-400:focus { border-color: var(--accent-text); }
-.dark ::selection { background-color: var(--accent-bg); color: var(--text-primary); }
-.dark input { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
-.dark select { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
-.dark textarea { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
-.dark .bg-transparent { background-color: transparent; }

ui/src/main.jsx

@@ -1,13 +1,10 @@
 import { StrictMode } from 'react'
 import { createRoot } from 'react-dom/client'
-import { ThemeProvider } from './theme.jsx'
 import './index.css'
 import App from './App.jsx'
 
 createRoot(document.getElementById('root')).render(
   <StrictMode>
-    <ThemeProvider>
+    <App />
-      <App />
-    </ThemeProvider>
   </StrictMode>,
 )

ui/src/pages/Mappings.jsx

@@ -4,7 +4,6 @@ import { api, authHeaders } from '../api'
 function AutocompleteInput({ value, onChange, onEnter, suggestions = [], className, placeholder }) {
   const [open, setOpen] = useState(false)
   const [highlighted, setHighlighted] = useState(0)
-  const [dropPos, setDropPos] = useState(null)
   const inputRef = useRef()
   const listRef = useRef()
 
@@ -13,10 +12,6 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
     : suggestions
 
   function openList() {
-    if (inputRef.current) {
-      const r = inputRef.current.getBoundingClientRect()
-      setDropPos({ top: r.bottom + 2, left: r.left, minWidth: r.width })
-    }
     setOpen(true)
     setHighlighted(0)
   }
@@ -47,6 +42,7 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
     if (e.key === 'Enter') onEnter?.()
   }
 
+  // Scroll highlighted item into view
   useEffect(() => {
     if (!open || !listRef.current) return
     const item = listRef.current.children[highlighted]
@@ -64,11 +60,10 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
         onKeyDown={handleKeyDown}
         onBlur={e => { if (!listRef.current?.contains(e.relatedTarget)) setOpen(false) }}
       />
-      {open && filtered.length > 0 && dropPos && (
+      {open && filtered.length > 0 && (
         <div
           ref={listRef}
-          style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
+          className="absolute z-50 left-0 top-full mt-0.5 bg-white border border-gray-200 rounded shadow-lg max-h-48 overflow-y-auto min-w-full"
-          className="bg-white border border-gray-200 rounded shadow-lg max-h-48 overflow-y-auto"
         >
           {filtered.map((s, i) => (
             <div
@@ -109,7 +104,7 @@ function SortHeader({ col, label, sortBy, onSort, className = '' }) {
   )
 }
 
-export default function Mappings({ source, onNeedsReprocess }) {
+export default function Mappings({ source }) {
   const [rules, setRules] = useState([])
   const [selectedRule, setSelectedRule] = useState('')
   const [allValues, setAllValues] = useState([])
@@ -268,7 +263,6 @@ export default function Mappings({ source, onNeedsReprocess }) {
           valueKey(x.extracted_value) === k ? { ...x, is_mapped: true, mapping_id: created.id, output } : x
         ))
       }
-      onNeedsReprocess?.(source)
       setDrafts(d => { const n = { ...d }; delete n[k]; return n })
     } catch (err) {
       alert(err.message)
@@ -315,7 +309,6 @@ export default function Mappings({ source, onNeedsReprocess }) {
         setSaving(s => ({ ...s, [k]: false }))
       }
     }))
-    onNeedsReprocess?.(source)
     setSelected(new Set())
     setBulkDraft({})
   }
@@ -324,7 +317,6 @@ export default function Mappings({ source, onNeedsReprocess }) {
     if (!row.mapping_id) return
     try {
       await api.deleteMapping(row.mapping_id)
-      onNeedsReprocess?.(source)
       setAllValues(av => av.map(x =>
         valueKey(x.extracted_value) === valueKey(row.extracted_value)
           ? { ...x, is_mapped: false, mapping_id: null, output: null }

ui/src/pages/Pivot.jsx

@@ -1,19 +1,33 @@
 import { useEffect, useRef, useState, useCallback } from 'react'
 import { api } from '../api'
-import useTheme from '../theme.jsx'
-import perspective from '@perspective-dev/client/inline'
-import '@perspective-dev/viewer/inline'
-import '@perspective-dev/viewer-datagrid'
-import '@perspective-dev/viewer-d3fc'
-import '@perspective-dev/viewer/themes'
 
 async function fetchAllRows(source) {
   const res = await api.getViewData(source, 100000, 0)
   return res.rows || []
 }
 
+let perspectivePromise = null
+
 function loadPerspective() {
-  return Promise.resolve(perspective)
+  if (perspectivePromise) return perspectivePromise
+  perspectivePromise = (async () => {
+    if (!document.getElementById('psp-theme')) {
+      const link = document.createElement('link')
+      link.id = 'psp-theme'
+      link.rel = 'stylesheet'
+      link.crossOrigin = 'anonymous'
+      link.href = 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css'
+      document.head.appendChild(link)
+    }
+    const [{ default: perspective }] = await Promise.all([
+      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
+      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
+      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
+      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
+    ])
+    return perspective
+  })()
+  return perspectivePromise
 }
 
 function formatVal(v, decimals = 2) {
@@ -66,32 +80,19 @@ const LAYOUT_KEY = (source) => `psp_layout_${source}`
 const DEFAULT_PLUGIN_CONFIG = { edit_mode: 'SELECT_REGION' }
 
 
-export default function Pivot({ source, selectedStack, setSelectedStack }) {
+export default function Pivot({ source }) {
-  const { dark } = useTheme()
   const viewerRef = useRef()
   const workerRef = useRef()
   const tableRef = useRef()
   const allRowsRef = useRef([])
   const expandDepthRef = useRef(null)
-  const lastClickKeyRef = useRef(null)
-  const perspClickHandlerRef = useRef(null)
   const [status, setStatus] = useState('idle')
   const [error, setError] = useState('')
   const [inspectedRows, setInspectedRows] = useState(null)
   const [clickDetail, setClickDetail] = useState(null)
   const [decimals, setDecimals] = useState(2)
-  const [paneWidth, setPaneWidth] = useState(384)
-  const [sortCol, setSortCol] = useState(null)
-  const [sortDir, setSortDir] = useState('asc')
 
-  const selectedView = selectedStack ?? source
+  // Named layouts
-  const viewType = selectedStack ? 'stack' : 'source'
-
-  useEffect(() => {
-    if (viewerRef.current) viewerRef.current.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
-  }, [dark])
-
-  // Named layouts — stacks use localStorage only (no server FK to sources)
   const [layouts, setLayouts] = useState([])
   const [activeLayoutId, setActiveLayoutId] = useState(null)
   const [saveAsName, setSaveAsName] = useState('')
@@ -104,21 +105,18 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
   }
 
   const loadLayouts = useCallback(async () => {
-    if (!selectedView) return
+    if (!source) return
     try {
-      const rows = viewType === 'source'
+      const rows = await api.getPivotLayouts(source)
-        ? await api.getPivotLayouts(selectedView)
-        : await api.getStackPivotLayouts(selectedView)
       setLayouts(rows)
     } catch {}
-  }, [selectedView])
+  }, [source])
 
   useEffect(() => {
-    if (!selectedView) return
+    if (!source) return
     let cancelled = false
     setInspectedRows(null)
     setClickDetail(null)
-    lastClickKeyRef.current = null
     setActiveLayoutId(null)
     setShowSaveAs(false)
     allRowsRef.current = []
@@ -131,7 +129,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
       try {
         const [perspective, rows] = await Promise.all([
           loadPerspective(),
-          fetchAllRows(selectedView),
+          fetchAllRows(source),
         ])
         if (cancelled) return
         if (!rows.length) { setStatus('noview'); return }
@@ -144,27 +142,13 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
         if (cancelled) { worker.terminate(); return }
         workerRef.current = worker
 
-        const table = await worker.table(rows, { name: selectedView })
+        const table = await worker.table(rows, { name: source })
         if (cancelled) return
         tableRef.current = table
 
         const viewer = viewerRef.current
-        const validCols = new Set(Object.keys(rows[0] || {}))
 
-        function cleanLayout(cfg) {
+        viewer.addEventListener('perspective-click', async (e) => {
-          if (!cfg) return cfg
-          const clean = { ...cfg }
-          const exprNames = new Set(Object.keys(clean.expressions || {}))
-          const valid = (c) => validCols.has(c) || exprNames.has(c)
-          if (clean.columns) clean.columns = clean.columns.filter(c => c == null || valid(c))
-          if (clean.group_by) clean.group_by = clean.group_by.filter(valid)
-          if (clean.split_by) clean.split_by = clean.split_by.filter(valid)
-          if (clean.sort) clean.sort = clean.sort.filter(([c]) => valid(c))
-          if (clean.filter) clean.filter = clean.filter.filter(([c]) => valid(c))
-          return clean
-        }
-
-        perspClickHandlerRef.current = async (e) => {
           const detail = e.detail || {}
           const { row, column_names } = detail
           if (!row) return
@@ -176,39 +160,14 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
           const hasHierarchy = (config.group_by || []).length > 0
           if (!hasHierarchy) return
 
-          // column_names encodes the full column path: [split_val_1, ..., split_val_N, measure]
+          setClickDetail({ row, config, column_names, eventFilters })
-          // positionally matching config.split_by. Perspective may omit split_by coordinate
-          // filters from detail.config.filter, so derive any missing ones from column_names.
-          const splitByFields = config.split_by || []
-          const coveredByEvent = new Set(eventFilters.filter(([, op]) => op === '==').map(([f]) => f))
-          const derivedSplitFilters = splitByFields
-            .map((field, i) => {
-              if (coveredByEvent.has(field)) return null
-              const val = Array.isArray(column_names) && column_names[i] != null
-                ? String(column_names[i]) : null
-              return val != null ? [field, '==', val] : null
-            })
-            .filter(Boolean)
-          const allFilters = [...eventFilters, ...derivedSplitFilters]
-
-          // Same cell clicked again — toggle the pane closed.
-          // Key on row path + column names (from the raw event) rather than derived
-          // filters, which can vary between clicks on stack/expression views.
-          const clickKey = JSON.stringify({ p: row['__ROW_PATH__'], c: column_names })
-          if (lastClickKeyRef.current === clickKey) {
-            lastClickKeyRef.current = null
-            setInspectedRows(null)
-            setClickDetail(null)
-            return
-          }
-          lastClickKeyRef.current = clickKey
-
-          setClickDetail({ row, config, column_names, eventFilters: allFilters })
 
+          // Use a Perspective view with the event filters + expressions so computed
+          // columns (split_by) are evaluated and filtered correctly
           try {
             const view = await tableRef.current.view({
-              filter: allFilters,
+              filter: eventFilters,
-              expressions: config.expressions || {},
+              expressions: config.expressions || [],
             })
             const data = await view.to_json()
             await view.delete()
@@ -218,28 +177,25 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
               Object.fromEntries(Object.entries(r).filter(([k]) => !exprNames.has(k)))
             )
             setInspectedRows(cleaned)
-          } catch (err) {
+          } catch {
-            console.warn('Perspective inspector view failed, falling back to JS filter:', err)
+            setInspectedRows(filterRowsByConfig(allRowsRef.current, eventFilters))
-            setInspectedRows(filterRowsByConfig(allRowsRef.current, allFilters))
           }
-        }
+        })
-        viewer.addEventListener('perspective-click', perspClickHandlerRef.current)
 
         await viewer.load(worker)
 
         const plugin = await viewer.getPlugin()
-        const savedLayout = localStorage.getItem(LAYOUT_KEY(selectedView))
+        const savedLayout = localStorage.getItem(LAYOUT_KEY(source))
         if (savedLayout) {
-          const parsed = cleanLayout(JSON.parse(savedLayout))
+          const parsed = JSON.parse(savedLayout)
           await viewer.restore(parsed)
           await plugin.restore(parsed.plugin_config || DEFAULT_PLUGIN_CONFIG)
           if (parsed.expand_depth != null) await applyExpandDepth(viewer, parsed.expand_depth)
         } else {
-          await viewer.restore({ table: selectedView, settings: false, plugin_config: DEFAULT_PLUGIN_CONFIG })
+          await viewer.restore({ table: source, settings: false, plugin_config: DEFAULT_PLUGIN_CONFIG })
           await plugin.restore(DEFAULT_PLUGIN_CONFIG)
         }
         await viewer.flush()
-        viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
 
         setStatus('ready')
       } catch (err) {
@@ -248,14 +204,8 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
     }
 
     init()
-    return () => {
+    return () => { cancelled = true }
-      cancelled = true
+  }, [source])
-      if (perspClickHandlerRef.current && viewerRef.current) {
-        viewerRef.current.removeEventListener('perspective-click', perspClickHandlerRef.current)
-        perspClickHandlerRef.current = null
-      }
-    }
-  }, [selectedView])
 
   async function applyExpandDepth(viewer, depth) {
     if (depth == null) return
@@ -269,35 +219,15 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
   async function applyLayout(layout) {
     const viewer = viewerRef.current
     if (!viewer) return
-    try {
+    await viewer.restore(layout.config)
-      const validCols = new Set(Object.keys(allRowsRef.current[0] || {}))
+    if (layout.config.plugin_config) {
-      function cleanLayout(cfg) {
+      const plugin = await viewer.getPlugin()
-        if (!cfg) return cfg
+      await plugin.restore(layout.config.plugin_config)
-        const clean = { ...cfg }
-        const exprNames = new Set(Object.keys(clean.expressions || {}))
-        const valid = (c) => validCols.has(c) || exprNames.has(c)
-        if (clean.columns) clean.columns = clean.columns.filter(c => c == null || valid(c))
-        if (clean.group_by) clean.group_by = clean.group_by.filter(valid)
-        if (clean.split_by) clean.split_by = clean.split_by.filter(valid)
-        if (clean.sort) clean.sort = clean.sort.filter(([c]) => valid(c))
-        if (clean.filter) clean.filter = clean.filter.filter(([c]) => valid(c))
-        return clean
-      }
-      const cleaned = cleanLayout(layout.config)
-      await viewer.restore(cleaned)
-      if (cleaned.plugin_config) {
-        const plugin = await viewer.getPlugin()
-        await plugin.restore(cleaned.plugin_config)
-      }
-      await applyExpandDepth(viewer, cleaned.expand_depth ?? null)
-      setActiveLayoutId(layout.id)
-      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(cleaned))
-    } catch {
-      // Layout references columns that no longer exist — remove it
-      localStorage.removeItem(LAYOUT_KEY(selectedView))
-      setActiveLayoutId(null)
-      await viewer.restore({ table: selectedView, settings: false })
     }
+    await applyExpandDepth(viewer, layout.config.expand_depth ?? null)
+    setActiveLayoutId(layout.id)
+    // also persist to localStorage so it survives refresh
+    localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(layout.config))
   }
 
   async function captureConfig() {
@@ -308,24 +238,16 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
     return { ...viewerConfig, plugin_config: pluginConfig, expand_depth: expandDepthRef.current }
   }
 
-  const saveLayout = (name, config) => viewType === 'source'
-    ? api.savePivotLayout(selectedView, name, config)
-    : api.saveStackPivotLayout(selectedView, name, config)
-
-  const deleteLayout = (id) => viewType === 'source'
-    ? api.deletePivotLayout(selectedView, id)
-    : api.deleteStackPivotLayout(selectedView, id)
-
   async function handleSaveOver() {
     const layout = layouts.find(l => l.id === activeLayoutId)
     if (!layout) return
     const config = await captureConfig()
     if (!config) return
     try {
-      const saved = await saveLayout(layout.layout_name, config)
+      const saved = await api.savePivotLayout(source, layout.layout_name, config)
-      setActiveLayoutId(saved.id)
+      localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(config))
-      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
       await loadLayouts()
+      setActiveLayoutId(saved.id)
       flashMsg('Saved!')
     } catch (err) {
       flashMsg(err.message)
@@ -338,8 +260,8 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
     const config = await captureConfig()
     if (!config) return
     try {
-      const saved = await saveLayout(name, config)
+      const saved = await api.savePivotLayout(source, name, config)
-      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
+      localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(config))
       await loadLayouts()
       setActiveLayoutId(saved.id)
       setShowSaveAs(false)
@@ -353,7 +275,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
   async function handleDelete(layout, e) {
     e.stopPropagation()
     try {
-      await deleteLayout(layout.id)
+      await api.deletePivotLayout(source, layout.id)
       if (activeLayoutId === layout.id) setActiveLayoutId(null)
       await loadLayouts()
       flashMsg('Deleted')
@@ -365,33 +287,15 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
   function handleResetToDefault() {
     const viewer = viewerRef.current
     if (!viewer) return
-    localStorage.removeItem(LAYOUT_KEY(selectedView))
+    localStorage.removeItem(LAYOUT_KEY(source))
     setActiveLayoutId(null)
-    viewer.restore({ table: selectedView, settings: true, plugin_config: DEFAULT_PLUGIN_CONFIG })
+    viewer.restore({ table: source, settings: true, plugin_config: DEFAULT_PLUGIN_CONFIG })
   }
 
   if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
 
   const cols = inspectedRows?.length ? Object.keys(inspectedRows[0]) : []
 
-  const sortedRows = sortCol == null || !inspectedRows ? inspectedRows : [...inspectedRows].sort((a, b) => {
-    const av = a[sortCol], bv = b[sortCol]
-    if (av == null && bv == null) return 0
-    if (av == null) return 1
-    if (bv == null) return -1
-    const num = typeof av === 'number' && typeof bv === 'number'
-    const cmp = num ? av - bv : String(av).localeCompare(String(bv))
-    return sortDir === 'asc' ? cmp : -cmp
-  })
-
-  const totals = cols.reduce((acc, c) => {
-    const vals = (inspectedRows || []).map(r => r[c])
-    if (vals.length > 0 && vals.every(v => v == null || typeof v === 'number')) {
-      acc[c] = vals.reduce((s, v) => s + (v ?? 0), 0)
-    }
-    return acc
-  }, {})
-
   const groupBy = clickDetail?.config?.group_by || []
   const splitBy = clickDetail?.config?.split_by || []
   const coordFields = new Set([...groupBy, ...splitBy])
@@ -401,26 +305,23 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
       .map(([f, , v]) => [f, v])
   )
   const cellCoords = [...groupBy, ...splitBy].map(f => coordMap[f]).filter(Boolean)
-  // column_names = [split_val_1, ..., split_val_N, measure_name] — use positional split_by length
+  const splitVals = splitBy.map(f => coordMap[f]).filter(Boolean)
-  // to separate split values from measure names; fall back to coordMap when ambiguous
+  const metrics = clickDetail?.column_names || []
-  const colNames = clickDetail?.column_names || []
+  const cellKey = splitVals.length > 0 && metrics.length > 0
-  const splitVals = splitBy.map((f, i) =>
-    coordMap[f] ?? (colNames[i] != null ? String(colNames[i]) : null)
-  ).filter(Boolean)
-  const metrics = splitBy.length > 0 ? colNames.slice(splitBy.length) : colNames
-  const cellKey = metrics.length > 0
     ? [...splitVals, ...metrics].join('|')
     : null
 
   return (
     <div className="w-full h-full flex flex-col">
 
-      {/* Layouts sub-bar */}
+      {/* Layout toolbar */}
-      <div className="flex items-center gap-2 px-3 h-9 bg-white border-b border-gray-200 shrink-0 text-xs">
+      <div className="flex items-center gap-2 px-3 py-1.5 bg-white border-b border-gray-200 flex-shrink-0">
+        <span className="text-xs text-gray-400 uppercase tracking-wide mr-1">Layouts</span>
+
         {layouts.map(l => (
           <div key={l.id}
             onClick={() => applyLayout(l)}
-            className={`flex items-center gap-1 rounded px-2 py-0.5 cursor-pointer border transition-colors
+            className={`flex items-center gap-1 text-xs rounded px-2 py-0.5 cursor-pointer border transition-colors
               ${activeLayoutId === l.id
                 ? 'bg-blue-50 border-blue-300 text-blue-700'
                 : 'bg-white border-gray-200 text-gray-600 hover:border-gray-400'}`}>
@@ -433,7 +334,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
 
         {activeLayoutId !== null && !showSaveAs && (
           <button onClick={handleSaveOver}
-            className="text-blue-500 hover:text-blue-700 border border-blue-200 rounded px-2 py-0.5">
+            className="text-xs text-blue-500 hover:text-blue-700 border border-blue-200 rounded px-2 py-0.5">
             Save
           </button>
         )}
@@ -446,27 +347,30 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
               onChange={e => setSaveAsName(e.target.value)}
               onKeyDown={e => { if (e.key === 'Enter') handleSaveAs(); if (e.key === 'Escape') { setShowSaveAs(false); setSaveAsName('') } }}
               placeholder="Layout name…"
-              className="border border-gray-300 rounded px-2 py-0.5 w-36 focus:outline-none focus:border-blue-400"
+              className="text-xs border border-gray-300 rounded px-2 py-0.5 w-36 focus:outline-none focus:border-blue-400"
             />
-            <button onClick={handleSaveAs} className="text-blue-600 hover:text-blue-800 px-1">Save</button>
+            <button onClick={handleSaveAs} className="text-xs text-blue-600 hover:text-blue-800 px-1">Save</button>
-            <button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-gray-400 hover:text-gray-600 px-1">Cancel</button>
+            <button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-xs text-gray-400 hover:text-gray-600 px-1">Cancel</button>
           </div>
         ) : (
           <button
             onClick={() => setShowSaveAs(true)}
-            className="text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-0.5">
+            className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-0.5">
             + Save as…
           </button>
         )}
 
         {activeLayoutId !== null && (
-          <button onClick={handleResetToDefault} className="text-gray-300 hover:text-gray-500 ml-1">reset</button>
+          <button onClick={handleResetToDefault}
+            className="text-xs text-gray-300 hover:text-gray-500 ml-1">
+            reset
+          </button>
         )}
 
-        {layoutMsg && <span className="text-green-600 ml-1">{layoutMsg}</span>}
+        {layoutMsg && <span className="text-xs text-green-600 ml-1">{layoutMsg}</span>}
 
         <div className="ml-auto flex items-center gap-1">
-          <span className="text-gray-400">depth:</span>
+          <span className="text-xs text-gray-400">depth:</span>
           {[0, 1, 2, 3].map(d => (
             <button key={d} onClick={async () => {
               const v = viewerRef.current; if (!v) return
@@ -475,7 +379,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
               const p = await v.getPlugin()
               await p.draw(view)
               expandDepthRef.current = d
-            }} className="border border-gray-200 rounded px-1.5 py-0.5 text-gray-500 hover:border-gray-400">
+            }} className="text-xs border border-gray-200 rounded px-1.5 py-0.5 text-gray-500 hover:border-gray-400">
               {d}
             </button>
           ))}
@@ -507,40 +411,12 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
         </div>
 
         {inspectedRows && clickDetail && (
-          <div
+          <div className="w-96 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
-            style={{ width: paneWidth }}
+            <div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
-            className="relative border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0"
+              <span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">
-          >
+                {inspectedRows.length} row{inspectedRows.length !== 1 ? 's' : ''}
-            {/* Drag-to-resize handle on left edge */}
+              </span>
-            <div
+              <div className="flex items-center gap-2">
-              className="absolute left-0 top-0 bottom-0 w-1 cursor-col-resize hover:bg-blue-300 z-10"
-              onMouseDown={(e) => {
-                e.preventDefault()
-                const startX = e.clientX
-                const startW = paneWidth
-                const onMove = (me) => setPaneWidth(Math.max(240, startW + startX - me.clientX))
-                const onUp = () => {
-                  document.removeEventListener('mousemove', onMove)
-                  document.removeEventListener('mouseup', onUp)
-                }
-                document.addEventListener('mousemove', onMove)
-                document.addEventListener('mouseup', onUp)
-              }}
-            />
-
-            {/* Header: breadcrumb + row count + controls */}
-            <div className="flex items-center justify-between pl-3 pr-2 py-2 border-b border-gray-100 flex-shrink-0">
-              <div className="flex items-center gap-2 min-w-0">
-                {cellCoords.length > 0 && (
-                  <span className="text-xs text-gray-700 font-mono font-semibold truncate">
-                    {cellCoords.join(' › ')}
-                  </span>
-                )}
-                <span className="text-xs text-gray-400 flex-shrink-0">
-                  {inspectedRows.length} row{inspectedRows.length !== 1 ? 's' : ''}
-                </span>
-              </div>
-              <div className="flex items-center gap-2 flex-shrink-0">
                 <div className="flex items-center gap-0.5">
                   <button onClick={() => setDecimals(d => Math.max(0, d - 1))}
                     className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">−</button>
@@ -548,13 +424,36 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
                   <button onClick={() => setDecimals(d => Math.min(8, d + 1))}
                     className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">+</button>
                 </div>
-                <button onClick={() => { setInspectedRows(null); setClickDetail(null); lastClickKeyRef.current = null }}
+                <button onClick={() => { setInspectedRows(null); setClickDetail(null) }}
                   className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
               </div>
             </div>
-
             <div className="flex-1 overflow-y-auto">
-              {/* User-set filters (only shown when active) */}
+
+              {/* Cell coordinates */}
+              <div className="px-3 py-2 border-b border-gray-100">
+                <div className="text-xs text-gray-400 uppercase tracking-wide mb-1">
+                  {[...groupBy, ...splitBy].join(' › ') || clickDetail.column_names?.join(', ') || 'Cell'}
+                </div>
+                {cellCoords.length > 0 && (
+                  <div className="text-xs text-gray-700 font-mono font-semibold">
+                    {cellCoords.join(' › ')}
+                  </div>
+                )}
+                {Object.entries(clickDetail.row)
+                  .filter(([k, v]) => k !== '__ROW_PATH__' && v != null)
+                  .map(([k, v]) => {
+                    const isSelected = cellKey != null && k === cellKey
+                    return (
+                      <div key={k} className={`flex justify-between py-0.5 gap-2 ${isSelected ? 'font-semibold' : ''}`}>
+                        <span className={`text-xs font-mono shrink-0 ${isSelected ? 'text-gray-700' : 'text-gray-400'}`}>{k}</span>
+                        <span className={`text-xs font-mono text-right ${isSelected ? 'text-blue-600' : 'text-gray-700'}`}>{formatVal(v, decimals)}</span>
+                      </div>
+                    )
+                  })}
+              </div>
+
+              {/* User-set filters */}
               {(() => {
                 const userFilters = (clickDetail.eventFilters || []).filter(([f]) => !coordFields.has(f))
                 return userFilters.length > 0 ? (
@@ -573,20 +472,13 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
                   <table className="w-full text-xs">
                     <thead>
                       <tr className="text-left text-gray-400 border-b border-gray-100 bg-gray-50 sticky top-0">
-                        {cols.map(c => {
+                        {cols.map(c => (
-                          const active = sortCol === c
+                          <th key={c} className="px-2 py-1 font-medium whitespace-nowrap">{c}</th>
-                          return (
+                        ))}
-                            <th key={c}
-                              onClick={() => { if (active) setSortDir(d => d === 'asc' ? 'desc' : 'asc'); else { setSortCol(c); setSortDir('asc') } }}
-                              className="px-2 py-1 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600">
-                              {c}{active ? (sortDir === 'asc' ? ' ▲' : ' ▼') : ''}
-                            </th>
-                          )
-                        })}
                       </tr>
                     </thead>
                     <tbody>
-                      {sortedRows.map((row, i) => (
+                      {inspectedRows.map((row, i) => (
                         <tr key={i} className="border-t border-gray-50 hover:bg-gray-50">
                           {cols.map(c => {
                             const f = formatVal(row[c], decimals)
@@ -599,17 +491,6 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
                         </tr>
                       ))}
                     </tbody>
-                    {Object.keys(totals).length > 0 && (
-                      <tfoot>
-                        <tr className="border-t-2 border-gray-200 bg-gray-50 font-semibold text-gray-700 sticky bottom-0">
-                          {cols.map(c => (
-                            <td key={c} className="px-2 py-1 font-mono whitespace-nowrap text-right">
-                              {totals[c] != null ? formatVal(totals[c], decimals) : ''}
-                            </td>
-                          ))}
-                        </tr>
-                      </tfoot>
-                    )}
                   </table>
                 </div>
               )}

ui/src/pages/Records.jsx

@@ -1,68 +1,7 @@
 import { useState, useEffect, useRef } from 'react'
 import { api } from '../api'
 
-function AutocompleteInput({ value, onChange, onEnter, suggestions = [], className, placeholder }) {
-  const [open, setOpen] = useState(false)
-  const [highlighted, setHighlighted] = useState(0)
-  const [dropPos, setDropPos] = useState(null)
-  const inputRef = useRef()
-  const listRef = useRef()
-  const filtered = value
-    ? suggestions.filter(s => s.toLowerCase().includes(value.toLowerCase()))
-    : suggestions
-
-  function openList() {
-    if (inputRef.current) {
-      const r = inputRef.current.getBoundingClientRect()
-      setDropPos({ top: r.bottom + 2, left: r.left, minWidth: r.width })
-    }
-    setOpen(true)
-    setHighlighted(0)
-  }
-
-  function select(val) { onChange(val); setOpen(false); inputRef.current?.focus() }
-
-  function handleKeyDown(e) {
-    if (e.altKey && e.key === 'ArrowDown') { e.preventDefault(); openList(); return }
-    if (open && filtered.length > 0) {
-      if (e.key === 'Tab')       { e.preventDefault(); setHighlighted(h => (h + 1) % filtered.length); return }
-      if (e.key === 'ArrowDown') { e.preventDefault(); setHighlighted(h => Math.min(h + 1, filtered.length - 1)); return }
-      if (e.key === 'ArrowUp')   { e.preventDefault(); setHighlighted(h => Math.max(h - 1, 0)); return }
-      if (e.key === 'Enter')     { e.preventDefault(); select(filtered[highlighted]); return }
-      if (e.key === 'Escape')    { setOpen(false); return }
-    }
-    if (e.key === 'Enter') onEnter?.()
-  }
-
-  useEffect(() => {
-    if (!open || !listRef.current) return
-    listRef.current.children[highlighted]?.scrollIntoView({ block: 'nearest' })
-  }, [highlighted, open])
-
-  return (
-    <div className="relative">
-      <input ref={inputRef} className={className} value={value} placeholder={placeholder}
-        onChange={e => { onChange(e.target.value); if (e.target.value) openList() }}
-        onKeyDown={handleKeyDown}
-        onBlur={e => { if (!listRef.current?.contains(e.relatedTarget)) setOpen(false) }}
-      />
-      {open && filtered.length > 0 && dropPos && (
-        <div ref={listRef}
-          style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
-          className="bg-white border border-gray-200 rounded shadow-lg max-h-40 overflow-y-auto">
-          {filtered.map((s, i) => (
-            <div key={s}
-              className={`px-2 py-1 text-xs cursor-pointer whitespace-nowrap ${i === highlighted ? 'bg-blue-50 text-blue-700' : 'text-gray-700 hover:bg-gray-50'}`}
-              onMouseDown={e => { e.preventDefault(); select(s) }}>{s}</div>
-          ))}
-        </div>
-      )}
-    </div>
-  )
-}
-
 const DATE_RE = /^\d{4}-\d{2}-\d{2}(T[\d:.Z+-]+)?$/
-const HIDDEN_COLS = new Set(['id', '_overridden'])
 
 function formatVal(val) {
   if (val === null || val === undefined) return null
@@ -87,26 +26,9 @@ export default function Records({ source }) {
   const [loading, setLoading] = useState(false)
   const [viewError, setViewError] = useState(null)
   const [sort, setSort] = useState({ col: null, dir: 'asc' })
-  const [filters, setFilters] = useState([])  // DB sort/filter queries
+  const [filters, setFilters] = useState([])
-  const [rowFilter, setRowFilter] = useState('')  // regex filter for selecting rows
-  const [selected, setSelected] = useState(new Set())  // row IDs selected for bulk override
-  const [bulkDraft, setBulkDraft] = useState({})  // bulk override values
-  const LIMIT = 100
-
-  // Override cols — loaded from DB once per source, extended by user via +
-  const [overrideCols, setOverrideCols] = useState([])   // keys seen in overrides across all records
-  const [extraCols, setExtraCols] = useState([])         // new cols added this session via +
-  const [globalValues, setGlobalValues] = useState({})   // picklist suggestions
-
-  // Override panel
-  const [panelOpen, setPanelOpen] = useState(false)
-  const [selectedRow, setSelectedRow] = useState(null)
-  const [selectedRecord, setSelectedRecord] = useState(null)
-  const [overrideDraft, setOverrideDraft] = useState({})
-  const [panelLoading, setPanelLoading] = useState(false)
-  const [panelSaving, setPanelSaving] = useState(false)
-  const [panelMsg, setPanelMsg] = useState(null)
   const debounceRef = useRef(null)
+  const LIMIT = 100
 
   useEffect(() => {
     if (!source) return
@@ -114,34 +36,9 @@ export default function Records({ source }) {
     setSort({ col: null, dir: 'asc' })
     setFilters([])
     setViewError(null)
-    setSelectedRecord(null)
-    setSelectedRow(null)
-    setPanelOpen(false)
-    setOverrideCols([])
-    setExtraCols([])
     load(0, null, 'asc', [])
-    api.getOverrideKeys(source).then(setOverrideCols).catch(() => {})
-    api.getGlobalValues().then(setGlobalValues).catch(() => {})
-    setSelected(new Set())
-    setBulkDraft({})
-    setRowFilter('')
   }, [source])
 
-  // Auto-select all rows matching the regex filter when it changes
-  useEffect(() => {
-    if (!rowFilter) return
-    let re = null
-    try { re = new RegExp(rowFilter, 'i') } catch { return }
-    const matches = rows.filter(r => {
-      for (const col of displayCols) {
-        const val = r[col]
-        if (val != null && re.test(String(val))) return true
-      }
-      return false
-    })
-    setSelected(new Set(matches.map(r => r.id)))
-  }, [rowFilter, rows])
-
   async function load(off, col, dir, filt) {
     setLoading(true)
     try {
@@ -149,7 +46,8 @@ export default function Records({ source }) {
       const res = await api.getViewData(source, LIMIT, off, col, dir, active)
       setExists(res.exists)
       setRows(res.rows)
-      if (res.rows.length > 0) setCols(Object.keys(res.rows[0]))
+      if (res.rows.length > 0 && cols.length === 0) setCols(Object.keys(res.rows[0]))
+      else if (res.rows.length > 0) setCols(Object.keys(res.rows[0]))
     } catch (err) {
       setViewError(err.message)
     } finally {
@@ -172,14 +70,12 @@ export default function Records({ source }) {
   }
 
   function addFilter() {
-    const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
+    setFilters(f => [...f, { col: cols[0] || '', pattern: '' }])
-    setFilters(f => [...f, { col: visCols[0] || '', pattern: '' }])
   }
 
   function removeFilter(i) {
     const next = filters.filter((_, idx) => idx !== i)
     setFilters(next)
-    setSelected(new Set())
     setOffset(0)
     load(0, sort.col, sort.dir, next)
   }
@@ -187,454 +83,139 @@ export default function Records({ source }) {
   function updateFilter(i, key, val) {
     const next = filters.map((f, idx) => idx === i ? { ...f, [key]: val } : f)
     setFilters(next)
-    setSelected(new Set())
     setOffset(0)
     triggerLoad(0, sort.col, sort.dir, next)
   }
 
-  function prev() { const o = Math.max(0, offset - LIMIT); setOffset(o); setSelected(new Set()); load(o, sort.col, sort.dir, filters) }
+  function prev() { const o = Math.max(0, offset - LIMIT); setOffset(o); load(o, sort.col, sort.dir, filters) }
-  function next() { const o = offset + LIMIT; setOffset(o); setSelected(new Set()); load(o, sort.col, sort.dir, filters) }
+  function next() { const o = offset + LIMIT; setOffset(o); load(o, sort.col, sort.dir, filters) }
-
-  async function openPanel(row) {
-    setPanelOpen(true)
-    setSelectedRow(row)
-    setSelectedRecord(null)
-    setOverrideDraft({})
-    setPanelMsg(null)
-
-    const id = row.id
-    if (!id) {
-      setPanelMsg({ text: 'No record ID — regenerate the view in Sources.', ok: false })
-      return
-    }
-
-    setPanelLoading(true)
-    try {
-      const rec = await api.getRecord(id)
-      setSelectedRecord(rec)
-      setOverrideDraft(rec.overrides || {})
-    } catch (err) {
-      setPanelMsg({ text: err.message, ok: false })
-    } finally {
-      setPanelLoading(false)
-    }
-  }
-
-  function closePanel() {
-    setPanelOpen(false)
-    setSelectedRow(null)
-    setSelectedRecord(null)
-    setOverrideDraft({})
-    setPanelMsg(null)
-  }
-
-  async function handleSaveOverrides() {
-    if (!selectedRecord) return
-    setPanelSaving(true)
-    setPanelMsg(null)
-    try {
-      const toSave = { ...overrideDraft }
-      const updated = await api.setRecordOverrides(selectedRecord.id, toSave)
-      setSelectedRecord(updated)
-      setOverrideDraft(updated.overrides || {})
-      // Merge any new cols from extraCols into overrideCols
-      setOverrideCols(prev => [...new Set([...prev, ...extraCols.filter(c => c.trim())])])
-      setExtraCols([])
-      setPanelMsg({ text: 'Saved.', ok: true })
-      load(offset, sort.col, sort.dir, filters)
-    } catch (err) {
-      setPanelMsg({ text: err.message, ok: false })
-    } finally {
-      setPanelSaving(false)
-    }
-  }
-
-  async function handleClearOverrides() {
-    if (!selectedRecord) return
-    setPanelSaving(true)
-    setPanelMsg(null)
-    try {
-      const updated = await api.clearRecordOverrides(selectedRecord.id)
-      setSelectedRecord(updated)
-      setOverrideDraft({})
-      setPanelMsg({ text: 'Cleared.', ok: true })
-      load(offset, sort.col, sort.dir, filters)
-    } catch (err) {
-      setPanelMsg({ text: err.message, ok: false })
-    } finally {
-      setPanelSaving(false)
-    }
-  }
 
   if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
 
-  const displayCols = (rows.length > 0 ? Object.keys(rows[0]) : cols).filter(c => !HIDDEN_COLS.has(c))
+  const displayCols = rows.length > 0 ? Object.keys(rows[0]) : cols
-  const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
-
-  // For bulk bar: only established override keys
-  const allOverrideCols = [...new Set([...overrideCols, ...extraCols])]
-
-  const savedOverrides = selectedRecord?.overrides || {}
-  const isDirty = Object.values(overrideDraft).some(v => String(v).trim())
-    || extraCols.some(c => c.trim())
-    || Object.keys(savedOverrides).some(k => !String(overrideDraft[k] ?? '').trim())
 
   return (
-    <div className="flex h-full min-h-0 overflow-hidden">
+    <div className="p-6">
-      <div className="flex-1 overflow-auto p-6 min-w-0">
+      <div className="flex items-center justify-between mb-4">
-        <div className="flex items-center justify-between mb-4">
+        <h1 className="text-xl font-semibold text-gray-800">Records — {source}</h1>
-          <h1 className="text-xl font-semibold text-gray-800">Records — {source}</h1>
+        {exists && rows.length > 0 && (
-          {exists && rows.length > 0 && (
+          <span className="text-xs text-gray-400 font-mono">dfv.{source}</span>
-            <span className="text-xs text-gray-400 font-mono">dfv.{source}</span>
-          )}
-        </div>
-
-        {/* Filter bar */}
-        {exists !== false && visCols.length > 0 && (
-          <div className="mb-4 flex flex-wrap gap-2 items-center">
-            <span className="text-xs text-gray-400 font-medium mr-1">DB query:</span>
-            {filters.map((f, i) => (
-              <div key={i} className="flex items-center gap-1 bg-white border border-gray-200 rounded px-2 py-1">
-                <select
-                  className="text-xs text-gray-600 border-0 focus:outline-none bg-transparent"
-                  value={f.col}
-                  onChange={e => updateFilter(i, 'col', e.target.value)}
-                >
-                  {visCols.map(c => <option key={c} value={c}>{c}</option>)}
-                </select>
-                <span className="text-xs text-gray-300 mx-0.5">~*</span>
-                <input
-                  className="text-xs font-mono border-0 focus:outline-none w-36 bg-transparent"
-                  placeholder="regex…"
-                  value={f.pattern}
-                  onChange={e => updateFilter(i, 'pattern', e.target.value)}
-                />
-                <button onClick={() => removeFilter(i)} className="text-gray-300 hover:text-gray-500 ml-1 leading-none">×</button>
-              </div>
-            ))}
-            <button onClick={addFilter}
-              className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-1">
-              + filter
-            </button>
-            {filters.length > 0 && (
-              <button onClick={() => { setFilters([]); setOffset(0); load(0, sort.col, sort.dir, []) }}
-                className="text-xs text-gray-400 hover:text-red-500">clear</button>
-            )}
-          </div>
-        )}
-
-        {/* Bulk select + override bar */}
-        {exists && visCols.length > 0 && (
-          <div className="mb-4 flex flex-wrap gap-2 items-center">
-            <span className="text-xs text-gray-400 font-medium mr-1">Bulk select:</span>
-            <input
-              className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-blue-400 ${
-                rowFilter ? 'border-blue-300' : 'border-gray-200'
-              }`}
-              placeholder="regex on loaded rows…"
-              value={rowFilter}
-              onChange={e => setRowFilter(e.target.value)}
-            />
-            {rowFilter && (
-              <span className="text-xs text-gray-400">{selected.size} of {rows.length} rows selected</span>
-            )}
-            {selected.size > 0 && (
-              <div className="flex items-center gap-2 ml-4 p-2 bg-blue-50 border border-blue-200 rounded flex-wrap">
-                {allOverrideCols.map(col => (
-                  <AutocompleteInput
-                    key={col}
-                    className="border border-blue-300 rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-blue-500 bg-white"
-                    placeholder={col}
-                    value={bulkDraft[col] || ''}
-                    onChange={v => setBulkDraft(d => ({ ...d, [col]: v }))}
-                    suggestions={[...(globalValues[col] || [])].sort()}
-                  />
-                ))}
-                <button
-                  onClick={async () => {
-                    const overrides = Object.fromEntries(
-                      Object.entries(bulkDraft).filter(([, v]) => v.trim())
-                    )
-                    if (Object.keys(overrides).length === 0) return
-                    if (selected.size === 0) return
-                    setPanelSaving(true)
-                    setPanelMsg(null)
-                    try {
-                      const res = await api.setBulkRecordOverrides(source, [...selected], overrides)
-                      setSelected(new Set())
-                      setBulkDraft({})
-                      setPanelMsg({ text: `Updated ${res.updated} records.`, ok: true })
-                      load(offset, sort.col, sort.dir, filters)
-                    } catch (err) {
-                      setPanelMsg({ text: err.message, ok: false })
-                    } finally {
-                      setPanelSaving(false)
-                    }
-                  }}
-                  disabled={panelSaving || selected.size === 0 || Object.values(bulkDraft).every(v => !v.trim())}
-                  className="text-xs bg-blue-600 text-white px-3 py-1 rounded hover:bg-blue-700 disabled:opacity-40 whitespace-nowrap"
-                >
-                  Apply to {selected.size}
-                </button>
-                <button
-                  onClick={() => { setSelected(new Set()); setBulkDraft({}); setRowFilter('') }}
-                  className="text-xs text-blue-400 hover:text-blue-600"
-                >
-                  cancel
-                </button>
-              </div>
-            )}
-          </div>
-        )}
-
-        {loading && <p className="text-sm text-gray-400">Loading…</p>}
-        {!loading && viewError && <p className="text-sm text-red-500">View error: {viewError} — check field types in Sources.</p>}
-        {!loading && exists === false && (
-          <p className="text-sm text-gray-400">
-            No view generated yet. Go to <span className="font-medium text-gray-600">Sources</span>, check fields as <span className="font-medium text-gray-600">In view</span>, then click <span className="font-medium text-gray-600">Generate view</span>.
-          </p>
-        )}
-        {!loading && exists && rows.length === 0 && (
-          <p className="text-sm text-gray-400">
-            {filters.some(f => f.col && f.pattern) ? 'No records match the current filters.' : 'View exists but no transformed records yet. Import data and run a transform first.'}
-          </p>
-        )}
-
-        {!loading && exists && rows.length > 0 && (
-          <>
-            <div className="bg-white border border-gray-200 rounded overflow-auto mb-4">
-              <table className="w-full text-sm">
-                <thead>
-                  <tr className="text-left text-xs text-gray-400 border-b border-gray-100 bg-gray-50">
-                    <th className="px-2 py-2 w-8">
-                      <input
-                        type="checkbox"
-                        className="cursor-pointer"
-                        checked={rows.length > 0 && rows.every(r => selected.has(r.id))}
-                        onChange={e => {
-                          if (e.target.checked) setSelected(new Set(rows.map(r => r.id)))
-                          else setSelected(new Set())
-                        }}
-                      />
-                    </th>
-                    {displayCols.map(col => {
-                      const active = sort.col === col
-                      return (
-                        <th key={col} onClick={() => toggleSort(col)}
-                          className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600">
-                          {col}
-                          <span className="ml-1 text-gray-300">{active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}</span>
-                        </th>
-                      )
-                    })}
-                  </tr>
-                </thead>
-                <tbody>
-                  {rows.map((row, i) => {
-                    const isOverridden = row._overridden
-                    const isRowSelected = selected.has(row.id)
-                    const isPanelSelected = selectedRow?.id != null && selectedRow.id === row.id
-                    return (
-                      <tr key={i} onClick={() => openPanel(row)}
-                        className={`border-t border-gray-50 cursor-pointer transition-colors
-                          ${isPanelSelected ? 'bg-blue-50' : isRowSelected ? 'bg-blue-50' : isOverridden ? 'bg-amber-50 hover:bg-amber-100' : 'hover:bg-gray-50'}`}>
-                        <td className="px-2 py-2">
-                          <input
-                            type="checkbox"
-                            className="cursor-pointer"
-                            checked={isRowSelected}
-                            onChange={e => {
-                              e.stopPropagation()
-                              setSelected(s => { const n = new Set(s); n.has(row.id) ? n.delete(row.id) : n.add(row.id); return n })
-                            }}
-                          />
-                        </td>
-                        {displayCols.map((col, j) => {
-                          const formatted = formatVal(row[col])
-                          return (
-                            <td key={j} className="px-3 py-2 text-xs text-gray-600 whitespace-nowrap max-w-48 truncate">
-                              {formatted === null ? <span className="text-gray-300">—</span> : formatted}
-                            </td>
-                          )
-                        })}
-                      </tr>
-                    )
-                  })}
-                </tbody>
-              </table>
-            </div>
-
-            <div className="flex items-center gap-3 text-sm text-gray-500">
-              <button onClick={prev} disabled={offset === 0}
-                className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">← Prev</button>
-              <span>{offset + 1}–{offset + rows.length}</span>
-              <button onClick={next} disabled={rows.length < LIMIT}
-                className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">Next →</button>
-            </div>
-          </>
         )}
       </div>
 
-      {/* Panel */}
+      {/* Filter bar */}
-      {panelOpen && (
+      {exists !== false && displayCols.length > 0 && (
-        <div className="w-80 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
+        <div className="mb-4 flex flex-wrap gap-2 items-center">
-          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
+          {filters.map((f, i) => (
-            <span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">Record</span>
+            <div key={i} className="flex items-center gap-1 bg-white border border-gray-200 rounded px-2 py-1">
-            <button onClick={closePanel} className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
+              <select
-          </div>
+                className="text-xs text-gray-600 border-0 focus:outline-none bg-transparent"
-
+                value={f.col}
-          {panelLoading && <p className="text-xs text-gray-400 p-3">Loading…</p>}
+                onChange={e => updateFilter(i, 'col', e.target.value)}
-
+              >
-          {selectedRecord && !panelLoading && (
+                {displayCols.map(c => <option key={c} value={c}>{c}</option>)}
-            <div className="flex-1 overflow-y-auto flex flex-col min-h-0">
+              </select>
-              {panelMsg && (
+              <span className="text-xs text-gray-300 mx-0.5">~*</span>
-                <div className={`text-xs px-3 py-2 border-b border-gray-100 ${panelMsg.ok ? 'text-green-600' : 'text-red-500'}`}>
+              <input
-                  {panelMsg.text}
+                className="text-xs font-mono border-0 focus:outline-none w-36 bg-transparent"
-                </div>
+                placeholder="regex…"
-              )}
+                value={f.pattern}
-
+                onChange={e => updateFilter(i, 'pattern', e.target.value)}
-              {/* Raw fields — read only */}
+              />
-              <div className="border-b border-gray-100">
+              <button
-                <div className="px-3 py-1.5 bg-gray-50 border-b border-gray-100">
+                onClick={() => removeFilter(i)}
-                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Raw</span>
+                className="text-gray-300 hover:text-gray-500 ml-1 leading-none"
-                </div>
+              >×</button>
-                {Object.entries(selectedRecord.data || {}).map(([field, val]) => (
-                  <div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-gray-50 first:border-t-0">
-                    <span className="text-xs font-mono text-gray-400 w-28 shrink-0 truncate">{field}</span>
-                    <span className="text-xs font-mono text-gray-500 truncate">{formatVal(val) ?? <span className="text-gray-300">—</span>}</span>
-                  </div>
-                ))}
-              </div>
-
-              {/* Transformed fields — read only delta */}
-              <div className="border-b border-gray-100">
-                <div className="px-3 py-1.5 bg-gray-50 border-b border-gray-100">
-                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Transformed</span>
-                </div>
-                {Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).length === 0
-                  ? <div className="px-3 py-2 text-xs text-gray-300">No rule output yet.</div>
-                  : Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).map(([field, val]) => (
-                    <div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-gray-50 first:border-t-0">
-                      <span className="text-xs font-mono text-gray-400 w-28 shrink-0 truncate">{field}</span>
-                      <span className="text-xs font-mono text-blue-600 truncate">{formatVal(val) ?? <span className="text-gray-300">—</span>}</span>
-                    </div>
-                  ))
-                }
-              </div>
-
-              {/* Overrides — editable */}
-              <div className="flex-1 border-b border-gray-100">
-                <div className="flex items-center justify-between px-3 py-1.5 bg-gray-50 border-b border-gray-100">
-                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Overrides</span>
-                  <button
-                    onClick={() => setExtraCols(ec => [...ec, ''])}
-                    className="text-gray-400 hover:text-gray-700 font-medium text-sm leading-none"
-                    title="Add field">+</button>
-                </div>
-                <table className="w-full text-xs">
-                  <tbody>
-                    {[...new Set([
-                      ...Object.keys(selectedRecord.transformed || {}),
-                      ...Object.keys(selectedRecord.overrides || {}),
-                      ...overrideCols
-                    ])].filter(k => !HIDDEN_COLS.has(k)).map(col => {
-                      const override = overrideDraft[col] ?? ''
-                      const placeholder = formatVal(selectedRecord.transformed?.[col]) ?? ''
-                      const suggestions = [...(globalValues[col] || [])].sort()
-                      return (
-                        <tr key={col} className="border-t border-gray-50">
-                          <td className="px-3 py-1.5 w-28 shrink-0">
-                            <span className="font-mono text-gray-500 truncate block">{col}</span>
-                          </td>
-                          <td className="px-1 py-1.5">
-                            <AutocompleteInput
-                              className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
-                                override ? 'border-amber-300 bg-amber-50 text-amber-800' : 'border-gray-200 text-gray-600'
-                              }`}
-                              value={override}
-                              placeholder={placeholder}
-                              onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
-                              onEnter={handleSaveOverrides}
-                              suggestions={suggestions}
-                            />
-                          </td>
-                          <td className="pr-2 text-center w-6">
-                            {override && (
-                              <button
-                                onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
-                                className="text-gray-300 hover:text-red-400 leading-none text-base">×</button>
-                            )}
-                          </td>
-                        </tr>
-                      )
-                    })}
-                    {extraCols.map((col, i) => {
-                      const val = overrideDraft[col] ?? ''
-                      const suggestions = [...(globalValues[col] || [])].sort()
-                      return (
-                        <tr key={`extra-${i}`} className="border-t border-gray-50">
-                          <td className="px-3 py-1.5 w-28 shrink-0">
-                            <input
-                              className="w-full text-xs font-mono border border-gray-200 rounded px-1 py-0.5 focus:outline-none focus:border-blue-400"
-                              value={col}
-                              placeholder="field name"
-                              onChange={e => {
-                                const newName = e.target.value
-                                setExtraCols(ec => { const c = [...ec]; c[i] = newName; return c })
-                                if (val) setOverrideDraft(d => {
-                                  const n = { ...d }
-                                  delete n[col]
-                                  if (newName) n[newName] = val
-                                  return n
-                                })
-                              }}
-                            />
-                          </td>
-                          <td className="px-1 py-1.5">
-                            <AutocompleteInput
-                              className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
-                                val ? 'border-amber-300 bg-amber-50 text-amber-800' : 'border-gray-200 text-gray-600'
-                              }`}
-                              value={val}
-                              onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
-                              onEnter={handleSaveOverrides}
-                              suggestions={suggestions}
-                            />
-                          </td>
-                          <td className="pr-2 text-center w-6">
-                            {val && (
-                              <button
-                                onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
-                                className="text-gray-300 hover:text-red-400 leading-none text-base">×</button>
-                            )}
-                          </td>
-                        </tr>
-                      )
-                    })}
-                  </tbody>
-                </table>
-              </div>
-
-              <div className="flex gap-2 px-3 py-2 border-t border-gray-100 shrink-0">
-                <button
-                  onClick={handleSaveOverrides}
-                  disabled={panelSaving || !isDirty}
-                  className="flex-1 text-xs bg-blue-600 text-white rounded px-3 py-1.5 hover:bg-blue-700 disabled:opacity-40">
-                  {panelSaving ? 'Saving…' : 'Save'}
-                </button>
-                {selectedRecord.overrides && Object.keys(selectedRecord.overrides).length > 0 && (
-                  <button
-                    onClick={handleClearOverrides}
-                    disabled={panelSaving}
-                    className="text-xs border border-gray-200 rounded px-3 py-1.5 text-gray-500 hover:border-red-300 hover:text-red-500 disabled:opacity-40">
-                    Clear
-                  </button>
-                )}
-              </div>
             </div>
+          ))}
+          <button
+            onClick={addFilter}
+            className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-1"
+          >
+            + filter
+          </button>
+          {filters.length > 0 && (
+            <button
+              onClick={() => { setFilters([]); setOffset(0); load(0, sort.col, sort.dir, []) }}
+              className="text-xs text-gray-400 hover:text-red-500"
+            >
+              clear
+            </button>
           )}
         </div>
       )}
+
+      {loading && <p className="text-sm text-gray-400">Loading…</p>}
+
+      {!loading && viewError && (
+        <p className="text-sm text-red-500">View error: {viewError} — check field types in Sources.</p>
+      )}
+
+      {!loading && exists === false && (
+        <p className="text-sm text-gray-400">
+          No view generated yet. Go to <span className="font-medium text-gray-600">Sources</span>, check fields as <span className="font-medium text-gray-600">In view</span>, then click <span className="font-medium text-gray-600">Generate view</span>.
+        </p>
+      )}
+
+      {!loading && exists && rows.length === 0 && (
+        <p className="text-sm text-gray-400">
+          {filters.some(f => f.col && f.pattern) ? 'No records match the current filters.' : 'View exists but no transformed records yet. Import data and run a transform first.'}
+        </p>
+      )}
+
+      {!loading && exists && rows.length > 0 && (
+        <>
+          <div className="bg-white border border-gray-200 rounded overflow-auto mb-4">
+            <table className="w-full text-sm">
+              <thead>
+                <tr className="text-left text-xs text-gray-400 border-b border-gray-100 bg-gray-50">
+                  {displayCols.map(col => {
+                    const active = sort.col === col
+                    return (
+                      <th
+                        key={col}
+                        onClick={() => toggleSort(col)}
+                        className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600"
+                      >
+                        {col}
+                        <span className="ml-1 text-gray-300">
+                          {active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}
+                        </span>
+                      </th>
+                    )
+                  })}
+                </tr>
+              </thead>
+              <tbody>
+                {rows.map((row, i) => (
+                  <tr key={i} className="border-t border-gray-50 hover:bg-gray-50">
+                    {displayCols.map((col, j) => {
+                      const formatted = formatVal(row[col])
+                      return (
+                        <td key={j} className="px-3 py-2 text-xs text-gray-600 whitespace-nowrap max-w-48 truncate">
+                          {formatted === null ? <span className="text-gray-300">—</span> : formatted}
+                        </td>
+                      )
+                    })}
+                  </tr>
+                ))}
+              </tbody>
+            </table>
+          </div>
+
+          <div className="flex items-center gap-3 text-sm text-gray-500">
+            <button onClick={prev} disabled={offset === 0}
+              className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">
+              ← Prev
+            </button>
+            <span>{offset + 1}–{offset + rows.length}</span>
+            <button onClick={next} disabled={rows.length < LIMIT}
+              className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">
+              Next →
+            </button>
+          </div>
+        </>
+      )}
     </div>
   )
 }

ui/src/pages/Rules.jsx

@@ -213,7 +213,7 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
   )
 }
 
-export default function Rules({ source, onStale }) {
+export default function Rules({ source }) {
   const [rules, setRules] = useState([])
   const [creating, setCreating] = useState(false)
   const [editing, setEditing] = useState(null)
@@ -266,7 +266,6 @@ export default function Rules({ source, onStale }) {
       } else {
         await api.createRule({ ...form, source_name: source })
       }
-      onStale?.(source)
       const updated = await api.getRules(source)
       setRules(updated)
       setCreating(false)
@@ -283,7 +282,6 @@ export default function Rules({ source, onStale }) {
     if (!confirm('Delete this rule and all its mappings?')) return
     try {
       await api.deleteRule(id)
-      onStale?.(source)
       setRules(r => r.filter(x => x.id !== id))
       setTestResults(t => { const n = { ...t }; delete n[id]; return n })
     } catch (err) {
@@ -303,7 +301,6 @@ export default function Rules({ source, onStale }) {
   async function handleToggle(rule) {
     try {
       await api.updateRule(rule.id, { enabled: !rule.enabled })
-      onStale?.(source)
       setRules(r => r.map(x => x.id === rule.id ? { ...x, enabled: !x.enabled } : x))
     } catch (err) {
       alert(err.message)

ui/src/pages/Stacks.jsx

@@ -1,830 +0,0 @@
-import { useState, useEffect, useRef } from 'react'
-import { api } from '../api'
-import { format as formatSql } from 'sql-formatter'
-
-function prettySql(sql) {
-  try {
-    return formatSql(sql, { language: 'postgresql', tabWidth: 4, keywordCase: 'upper' })
-  } catch {
-    return sql
-  }
-}
-
-const FIELD_TYPES = ['text', 'numeric', 'date']
-
-// ── Calibrate modal ────────────────────────────────────────────────────────────
-
-function fmt(n) {
-  return Number(n).toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })
-}
-
-function CalibrateModal({ stack, sourceName, currentOffset, onClose, onApply }) {
-  const [asOf, setAsOf] = useState('')
-  const [known, setKnown] = useState('')
-  const [computed, setComputed] = useState(null)  // raw sum from DB (no offset)
-  const [loading, setLoading] = useState(false)
-  const [error, setError] = useState('')
-  const [applyOffset, setApplyOffset] = useState('')
-  const debounceRef = useRef(null)
-
-  const knownNum = parseFloat(known)
-  const hasKnown = known !== '' && !isNaN(knownNum)
-  const plug = hasKnown && computed !== null ? knownNum - computed : null
-
-  // Auto-fetch computed sum on mount (all transactions) and whenever date changes
-  useEffect(() => {
-    clearTimeout(debounceRef.current)
-    debounceRef.current = setTimeout(async () => {
-      setLoading(true); setError('')
-      try {
-        const r = await api.calibrateBalance(stack.name, sourceName, { as_of_date: asOf || null, known_balance: 0 })
-        if (r.success) setComputed(Number(r.computed_sum))
-        else setError(r.error)
-      } catch (e) { setError(e.message) }
-      finally { setLoading(false) }
-    }, asOf ? 400 : 0)
-    return () => clearTimeout(debounceRef.current)
-  }, [asOf])
-
-  // Keep applyOffset in sync with plug
-  useEffect(() => {
-    if (plug !== null) setApplyOffset(plug.toFixed(2))
-  }, [plug])
-
-  return (
-    <div className="fixed inset-0 bg-black/40 flex items-center justify-center z-50" onMouseDown={e => { if (e.target === e.currentTarget) onClose() }}>
-      <div className="bg-white rounded-lg shadow-xl w-[420px] p-5" onClick={e => e.stopPropagation()}>
-        <div className="flex items-center justify-between mb-4">
-          <span className="text-sm font-semibold text-gray-700">Calibrate — {sourceName}</span>
-          <button onClick={onClose} className="text-gray-400 hover:text-gray-600">✕</button>
-        </div>
-
-        {/* Date */}
-        <div className="mb-4">
-          <label className="text-xs text-gray-500 block mb-1">As-of date</label>
-          <input type="date" className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
-            value={asOf} onChange={e => setAsOf(e.target.value)} />
-        </div>
-
-        {/* Reconciliation table */}
-        <div className="bg-gray-50 rounded border border-gray-200 mb-4 text-sm">
-          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
-            <span className="text-gray-500 text-xs">Data sum at date</span>
-            <span className="font-mono text-gray-700">
-              {loading ? <span className="text-gray-300">…</span> : computed !== null ? fmt(computed) : <span className="text-gray-300">—</span>}
-            </span>
-          </div>
-          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
-            <span className="text-gray-500 text-xs">Known balance</span>
-            <input
-              type="number" step="0.01"
-              className="font-mono text-right bg-transparent border-0 focus:outline-none w-36 text-sm text-gray-700 placeholder-gray-300"
-              placeholder="enter balance"
-              value={known} onChange={e => setKnown(e.target.value)}
-            />
-          </div>
-          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
-            <span className="text-gray-500 text-xs">Current offset</span>
-            <span className="font-mono text-gray-400">{fmt(currentOffset ?? 0)}</span>
-          </div>
-          <div className="flex items-center justify-between px-3 py-2 font-medium">
-            <span className="text-gray-700 text-xs">Plug (offset needed)</span>
-            <span className={`font-mono ${plug !== null ? 'text-blue-700' : 'text-gray-300'}`}>
-              {plug !== null ? fmt(plug) : '—'}
-            </span>
-          </div>
-        </div>
-
-        {error && <p className="text-xs text-red-500 mb-3">{error}</p>}
-
-        {/* Apply */}
-        <div className="flex gap-2 items-center">
-          <input type="number" step="0.01"
-            className="flex-1 border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
-            placeholder="offset to apply"
-            value={applyOffset} onChange={e => setApplyOffset(e.target.value)} />
-          <button onClick={() => onApply(parseFloat(applyOffset))} disabled={applyOffset === '' || isNaN(parseFloat(applyOffset))}
-            className="text-sm bg-green-600 text-white px-4 py-1.5 rounded hover:bg-green-700 disabled:opacity-40">
-            Apply
-          </button>
-        </div>
-      </div>
-    </div>
-  )
-}
-
-// ── Stack panel ────────────────────────────────────────────────────────────────
-
-function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSqlGenerated }) {
-  const members = stack.sources || []
-
-  const [label, setLabel] = useState(stack.label || '')
-  const [fields, setFields] = useState(stack.fields || [])
-  const [newField, setNewField] = useState({ name: '', type: 'text' })
-  const [addingSrc, setAddingSrc] = useState('')
-
-  // Per-source config: sign, offset, amount_field, date_field, field_map
-  const [srcCfg, setSrcCfg] = useState(() =>
-    Object.fromEntries(members.map(m => [m.source_name, {
-      sign:         m.amount_sign ?? 1,
-      offset:       m.balance_offset ?? 0,
-      amount_field: m.amount_field || '',
-      date_field:   m.date_field || '',
-      field_map:    { ...(m.field_map || {}) },
-    }]))
-  )
-
-  // Available columns from each source's dfv view
-  const [srcFields, setSrcFields] = useState({})
-
-  // Drag-to-reorder state
-  const [dragIdx, setDragIdx] = useState(null)
-  const [dragOverIdx, setDragOverIdx] = useState(null)
-  const [srcDragIdx, setSrcDragIdx] = useState(null)
-  const [srcDragOverIdx, setSrcDragOverIdx] = useState(null)
-
-  // Calibrate
-  const [calibratingSource, setCalibratingSource] = useState(null)
-
-  // View / balance
-  const [viewResult, setViewResult] = useState(null)
-  const [netBalance, setNetBalance] = useState(null)
-  const [balanceError, setBalanceError] = useState('')
-
-  const [saving, setSaving] = useState(false)
-  const [mappingsDirty, setMappingsDirty] = useState(false)
-  const [error, setError] = useState('')
-
-  // Fetch source columns whenever members change
-  useEffect(() => {
-    members.forEach(m => {
-      api.getFields(m.source_name)
-        .then(f => setSrcFields(prev => ({ ...prev, [m.source_name]: f.map(x => x.key) })))
-        .catch(() => {})
-    })
-  }, [members.map(m => m.source_name).join(',')])
-
-  // Live SQL preview — debounced; syncs current UI state to DB first so preview is accurate
-  const previewTimer = useRef(null)
-  useEffect(() => {
-    clearTimeout(previewTimer.current)
-    previewTimer.current = setTimeout(async () => {
-      try {
-        for (const m of members) {
-          const cfg = srcCfg[m.source_name] || {}
-          await api.upsertStackSource(stack.name, m.source_name, {
-            field_map:      cfg.field_map    || {},
-            amount_sign:    cfg.sign         ?? 1,
-            balance_offset: cfg.offset       ?? 0,
-            amount_field:   cfg.amount_field || null,
-            date_field:     cfg.date_field   || null,
-          })
-        }
-        await api.updateStack(stack.name, {
-          fields,
-          amount_field: amountCanonical || null,
-          date_field:   dateCanonical   || null,
-        })
-        const r = await api.previewStackSql(stack.name)
-        if (r.success) onSqlGenerated?.(r.sql)
-      } catch {}
-    }, 600)
-    return () => clearTimeout(previewTimer.current)
-  }, [
-    JSON.stringify(fields),
-    JSON.stringify(srcCfg),
-    members.map(m => m.source_name).join(','),
-  ])
-
-  // Auto-detect canonical amount/date field from field types
-  const amountCanonical = fields.find(f => f.type === 'numeric')?.name || stack.amount_field
-  const dateCanonical   = fields.find(f => f.type === 'date')?.name   || stack.date_field
-
-  // ── Label ──
-  async function saveLabel() {
-    setSaving(true); setError('')
-    try { await api.updateStack(stack.name, { label }); onUpdated() }
-    catch (e) { setError(e.message) }
-    finally { setSaving(false) }
-  }
-
-  // ── Fields ──
-  async function addField() {
-    if (!newField.name) return
-    const updated = [...fields, { name: newField.name, type: newField.type }]
-    setFields(updated)
-    setNewField({ name: '', type: 'text' })
-    await api.updateStack(stack.name, { fields: updated })
-    onUpdated()
-  }
-
-  async function removeField(name) {
-    const updated = fields.filter(f => f.name !== name)
-    setFields(updated)
-    await api.updateStack(stack.name, { fields: updated })
-    onUpdated()
-  }
-
-  // ── Drag reorder ──
-  function handleDragStart(e, idx) {
-    setDragIdx(idx)
-    e.dataTransfer.effectAllowed = 'move'
-  }
-
-  function handleDragOver(e, idx) {
-    e.preventDefault()
-    setDragOverIdx(idx)
-  }
-
-  async function handleDrop(e, toIdx) {
-    e.preventDefault()
-    if (dragIdx === null || dragIdx === toIdx) { setDragIdx(null); setDragOverIdx(null); return }
-    const updated = [...fields]
-    const [moved] = updated.splice(dragIdx, 1)
-    updated.splice(toIdx, 0, moved)
-    setFields(updated)
-    setDragIdx(null); setDragOverIdx(null)
-    await api.updateStack(stack.name, { fields: updated })
-    onUpdated()
-  }
-
-  // ── Source drag-to-reorder ──
-  function handleSrcDragStart(e, idx) {
-    setSrcDragIdx(idx)
-    e.dataTransfer.effectAllowed = 'move'
-  }
-
-  function handleSrcDragOver(e, idx) {
-    e.preventDefault()
-    setSrcDragOverIdx(idx)
-  }
-
-  async function handleSrcDrop(e, toIdx) {
-    e.preventDefault()
-    if (srcDragIdx === null || srcDragIdx === toIdx) { setSrcDragIdx(null); setSrcDragOverIdx(null); return }
-    const updated = [...members]
-    const [moved] = updated.splice(srcDragIdx, 1)
-    updated.splice(toIdx, 0, moved)
-    setSrcDragIdx(null); setSrcDragOverIdx(null)
-    await api.reorderStackSources(stack.name, updated.map(m => m.source_name))
-    onUpdated()
-  }
-
-  // ── Mapping grid ──
-  function getMappingValue(srcName, canonicalName) {
-    const cfg = srcCfg[srcName] || {}
-    if (canonicalName === amountCanonical) return cfg.amount_field || ''
-    if (canonicalName === dateCanonical)   return cfg.date_field   || ''
-    return cfg.field_map?.[canonicalName] || ''
-  }
-
-  function setMappingValue(srcName, canonicalName, value) {
-    setSrcCfg(prev => {
-      const cfg = { ...prev[srcName] }
-      if (canonicalName === amountCanonical)      cfg.amount_field = value
-      else if (canonicalName === dateCanonical)   cfg.date_field   = value
-      else cfg.field_map = { ...cfg.field_map, [canonicalName]: value }
-      return { ...prev, [srcName]: cfg }
-    })
-    setMappingsDirty(true)
-  }
-
-  function setSrcSign(srcName, sign) {
-    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], sign } }))
-    setMappingsDirty(true)
-  }
-
-  function setSrcOffset(srcName, offset) {
-    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], offset } }))
-    setMappingsDirty(true)
-  }
-
-  async function saveMappings() {
-    setSaving(true); setError('')
-    try {
-      for (const m of members) {
-        const cfg = srcCfg[m.source_name] || {}
-        await api.upsertStackSource(stack.name, m.source_name, {
-          field_map:    cfg.field_map    || {},
-          amount_sign:  cfg.sign         ?? 1,
-          balance_offset: cfg.offset     ?? 0,
-          amount_field: cfg.amount_field || null,
-          date_field:   cfg.date_field   || null,
-        })
-      }
-      // Persist the auto-detected canonical field names on the stack
-      await api.updateStack(stack.name, {
-        amount_field: amountCanonical || null,
-        date_field:   dateCanonical   || null,
-      })
-      setMappingsDirty(false)
-      onStale?.(stack.name)
-      onUpdated()
-    } catch (e) { setError(e.message) }
-    finally { setSaving(false) }
-  }
-
-  // ── Sources ──
-  async function addSource() {
-    if (!addingSrc) return
-    await api.upsertStackSource(stack.name, addingSrc, { field_map: {}, amount_sign: 1 })
-    setSrcCfg(prev => ({ ...prev, [addingSrc]: { sign: 1, offset: 0, amount_field: '', date_field: '', field_map: {} } }))
-    // Load fields immediately so dropdowns are ready
-    try {
-      const f = await api.getFields(addingSrc)
-      setSrcFields(prev => ({ ...prev, [addingSrc]: f.map(x => x.key) }))
-    } catch (e) {}
-    setAddingSrc('')
-    onStale?.(stack.name)
-    onUpdated()
-  }
-
-  function handleSrcAmountField(srcName, value) {
-    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], amount_field: value } }))
-    // Update column type to numeric if a column with this name exists
-    setFields(prev => prev.map(f => f.name === value ? { ...f, type: 'numeric' } : f))
-    setMappingsDirty(true)
-    maybeAutoPopulate(srcName, value, srcCfg[srcName]?.date_field)
-  }
-
-  function handleSrcDateField(srcName, value) {
-    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], date_field: value } }))
-    setFields(prev => prev.map(f => f.name === value ? { ...f, type: 'date' } : f))
-    setMappingsDirty(true)
-    maybeAutoPopulate(srcName, srcCfg[srcName]?.amount_field, value)
-  }
-
-  function maybeAutoPopulate(srcName, amtField, dtField) {
-    if (!amtField || !dtField) return
-    if (fields.length > 0) return  // don't overwrite existing columns
-    const sourceFields = srcFields[srcName] || []
-    if (sourceFields.length === 0) return
-    const newFields = sourceFields.map(sf => ({
-      name: sf,
-      type: sf === amtField ? 'numeric' : sf === dtField ? 'date' : 'text',
-    }))
-    setFields(newFields)
-    api.updateStack(stack.name, { fields: newFields, amount_field: amtField, date_field: dtField })
-  }
-
-  async function removeSource(src) {
-    await api.removeStackSource(stack.name, src)
-    setSrcCfg(prev => { const n = { ...prev }; delete n[src]; return n })
-    onStale?.(stack.name)
-    onUpdated()
-  }
-
-  async function handleCalibrate(srcName) {
-    // Save this source's current config before opening modal
-    const cfg = srcCfg[srcName] || {}
-    await api.upsertStackSource(stack.name, srcName, {
-      field_map:     cfg.field_map    || {},
-      amount_sign:   cfg.sign         ?? 1,
-      balance_offset: cfg.offset      ?? 0,
-      amount_field:  cfg.amount_field || null,
-      date_field:    cfg.date_field   || null,
-    })
-    setCalibratingSource(srcName)
-  }
-
-  async function applyCalibration(srcName, offset) {
-    const cfg = srcCfg[srcName] || {}
-    await api.upsertStackSource(stack.name, srcName, {
-      field_map:     cfg.field_map    || {},
-      amount_sign:   cfg.sign         ?? 1,
-      balance_offset: offset,
-      amount_field:  cfg.amount_field || null,
-      date_field:    cfg.date_field   || null,
-    })
-    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], offset } }))
-    setCalibratingSource(null)
-    onStale?.(stack.name)
-    onUpdated()
-  }
-
-  // ── View ──
-  async function generateView() {
-    setViewResult(null); setNetBalance(null); setBalanceError('')
-    try {
-      const r = await api.generateStackView(stack.name)
-      setViewResult(r)
-      if (r.success) {
-        fetchBalance()
-        onViewGenerated?.(stack.name)
-        onSqlGenerated?.(r.sql || '')
-        ;(r.cascade_stale || []).forEach(n => onStale?.(n))
-      }
-    } catch (e) { setError(e.message) }
-  }
-
-  async function fetchBalance() {
-    setBalanceError('')
-    try {
-      const r = await api.getStackBalance(stack.name)
-      if (r.success) setNetBalance(r.balance)
-      else setBalanceError(r.error)
-    } catch (e) { setBalanceError(e.message) }
-  }
-
-  const availableSources = sources.filter(s => !members.find(m => m.source_name === s.name))
-  if (addingSrc === '' && availableSources.length === 1) setAddingSrc(availableSources[0].name)
-
-  return (
-    <div className="space-y-5">
-
-      {/* Label */}
-      <div className="bg-white border border-gray-200 rounded p-4">
-        <h3 className="text-sm font-semibold text-gray-700 mb-3">Configuration</h3>
-        <div className="flex gap-3 items-end">
-          <div className="flex-1">
-            <label className="text-xs text-gray-500 block mb-1">Label <span className="text-gray-400">(optional)</span></label>
-            <input className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
-              value={label} onChange={e => setLabel(e.target.value)}
-              onKeyDown={e => e.key === 'Enter' && saveLabel()} />
-          </div>
-          <button onClick={saveLabel} disabled={saving}
-            className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
-            {saving ? 'Saving…' : 'Save'}
-          </button>
-        </div>
-        {error && <p className="text-xs text-red-500 mt-2">{error}</p>}
-      </div>
-
-      {/* Sources */}
-      <div className="bg-white border border-gray-200 rounded p-4">
-        <h3 className="text-sm font-semibold text-gray-700 mb-1">Sources</h3>
-        <p className="text-xs text-gray-400 mb-3">Each source contributes rows to the combined view. Set the sign to flip the direction of amounts (e.g. credit card charges are positive in the source but should subtract from your balance). The offset adjusts the running balance — use Calibrate to compute it from a known good balance.</p>
-        <div className="space-y-2 mb-3">
-          {members.map((m, idx) => {
-            const cfg = srcCfg[m.source_name] || {}
-            const sf = srcFields[m.source_name] || []
-            const canCalibrate = !!cfg.amount_field && !!cfg.date_field
-            return (
-              <div key={m.source_name}
-                draggable
-                onDragStart={e => handleSrcDragStart(e, idx)}
-                onDragOver={e => handleSrcDragOver(e, idx)}
-                onDrop={e => handleSrcDrop(e, idx)}
-                onDragEnd={() => { setSrcDragIdx(null); setSrcDragOverIdx(null) }}
-                className={`border border-gray-100 rounded px-3 py-2 text-xs space-y-2 ${srcDragOverIdx === idx && srcDragIdx !== idx ? 'bg-blue-50' : ''}`}>
-                <div className="flex items-center gap-2">
-                  <span className="text-gray-300 cursor-grab select-none">⠿</span>
-                  <span className="font-medium text-gray-700 flex-1">{m.source_name}</span>
-                  <button onClick={() => removeSource(m.source_name)} className="text-red-300 hover:text-red-500">Remove</button>
-                </div>
-                <div className="grid grid-cols-2 gap-x-4 gap-y-1.5">
-                  <div>
-                    <label className="text-gray-400 block mb-0.5">Amount field</label>
-                    <select value={cfg.amount_field || ''}
-                      onChange={e => handleSrcAmountField(m.source_name, e.target.value)}
-                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
-                      <option value="">— select —</option>
-                      {sf.map(f => <option key={f} value={f}>{f}</option>)}
-                    </select>
-                  </div>
-                  <div>
-                    <label className="text-gray-400 block mb-0.5">Sign</label>
-                    <select value={cfg.sign ?? 1}
-                      onChange={e => { setSrcSign(m.source_name, parseInt(e.target.value)); setMappingsDirty(true) }}
-                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
-                      <option value={1}>+1 (as-is)</option>
-                      <option value={-1}>−1 (flip)</option>
-                    </select>
-                  </div>
-                  <div>
-                    <label className="text-gray-400 block mb-0.5">Date field</label>
-                    <select value={cfg.date_field || ''}
-                      onChange={e => handleSrcDateField(m.source_name, e.target.value)}
-                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
-                      <option value="">— select —</option>
-                      {sf.map(f => <option key={f} value={f}>{f}</option>)}
-                    </select>
-                  </div>
-                  <div>
-                    <label className="text-gray-400 block mb-0.5">Balance offset</label>
-                    <div className="flex items-center gap-1">
-                      <input type="number" step="0.01" value={cfg.offset ?? 0}
-                        onChange={e => { setSrcOffset(m.source_name, parseFloat(e.target.value) || 0); setMappingsDirty(true) }}
-                        className="flex-1 border border-gray-200 rounded px-1.5 py-0.5 font-mono focus:outline-none focus:border-blue-400" />
-                      <button onClick={() => handleCalibrate(m.source_name)}
-                        disabled={!canCalibrate}
-                        title={!canCalibrate ? 'Set amount and date fields first' : 'Calibrate balance'}
-                        className="text-blue-400 hover:text-blue-600 underline disabled:opacity-40 disabled:cursor-not-allowed disabled:no-underline">
-                        Calibrate
-                      </button>
-                    </div>
-                  </div>
-                </div>
-              </div>
-            )
-          })}
-          {members.length === 0 && <p className="text-xs text-gray-400">No sources added yet.</p>}
-        </div>
-        {availableSources.length > 0 && (
-          <div className="flex gap-2">
-            <select className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
-              value={addingSrc} onChange={e => setAddingSrc(e.target.value)}>
-              <option value="">— add source —</option>
-              {availableSources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
-            </select>
-            <button onClick={addSource} disabled={!addingSrc}
-              className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700 disabled:opacity-40">Add</button>
-          </div>
-        )}
-      </div>
-
-      {/* Output columns mapping grid */}
-      <div className="bg-white border border-gray-200 rounded p-4">
-        <h3 className="text-sm font-semibold text-gray-700 mb-1">Output columns</h3>
-        <p className="text-xs text-gray-400 mb-3">
-          Each row is a column in the combined view. Each source column shows which field from that source maps to it.
-          The first <span className="text-blue-500">numeric</span> field drives the running balance; the first <span className="text-green-600">date</span> field drives the ordering.
-          Both <span className="font-mono">source_balance</span> (per-source) and <span className="font-mono">net_balance</span> (combined) are always included in the generated view.
-          Drag rows to reorder.
-        </p>
-
-        {members.length === 0 ? (
-          <p className="text-xs text-gray-400 mb-3">Add sources above first.</p>
-        ) : (
-          <div className="overflow-x-auto mb-3">
-            <table className="w-full text-xs border-collapse">
-              <thead>
-                <tr className="border-b border-gray-200">
-                  <th className="w-5 pb-2"></th>
-                  <th className="text-left text-gray-400 font-normal pb-2 pr-4">Column</th>
-                  <th className="text-left text-gray-400 font-normal pb-2 pr-4">Type</th>
-                  {members.map(m => (
-                    <th key={m.source_name} className="text-left text-gray-400 font-normal pb-2 pr-3 min-w-36">{m.source_name}</th>
-                  ))}
-                  <th className="w-5 pb-2"></th>
-                </tr>
-              </thead>
-              <tbody>
-                {fields.map((f, idx) => {
-                  const isAmount = f.name === amountCanonical
-                  const isDate   = f.name === dateCanonical
-                  return (
-                    <tr key={f.name}
-                      draggable
-                      onDragStart={e => handleDragStart(e, idx)}
-                      onDragOver={e => handleDragOver(e, idx)}
-                      onDrop={e => handleDrop(e, idx)}
-                      onDragEnd={() => { setDragIdx(null); setDragOverIdx(null) }}
-                      className={`border-b border-gray-50 ${dragOverIdx === idx && dragIdx !== idx ? 'bg-blue-50' : ''}`}>
-                      <td className="py-1.5 pr-1 text-gray-300 cursor-grab select-none">⠿</td>
-                      <td className="py-1.5 pr-4 font-mono text-gray-700 whitespace-nowrap">
-                        {f.name}
-                        {isAmount && <span className="ml-1.5 text-blue-500 font-sans font-normal">amount</span>}
-                        {isDate   && <span className="ml-1.5 text-green-600 font-sans font-normal">date</span>}
-                      </td>
-                      <td className="py-1.5 pr-4 text-gray-400">{f.type}</td>
-                      {members.map(m => (
-                        <td key={m.source_name} className="py-1.5 pr-3">
-                          <div className="flex items-center gap-1">
-                            <select
-                              value={getMappingValue(m.source_name, f.name)}
-                              onChange={e => setMappingValue(m.source_name, f.name, e.target.value)}
-                              className="border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400 min-w-0 flex-1">
-                              <option value="">— same name —</option>
-                              {(srcFields[m.source_name] || []).map(sf => (
-                                <option key={sf} value={sf}>{sf}</option>
-                              ))}
-                            </select>
-                          </div>
-                        </td>
-                      ))}
-                      <td className="py-1.5">
-                        <button onClick={() => removeField(f.name)} className="text-red-300 hover:text-red-500">✕</button>
-                      </td>
-                    </tr>
-                  )
-                })}
-                {fields.length === 0 && (
-                  <tr><td colSpan={3 + members.length} className="py-3 text-gray-400 text-center">No columns defined yet — add one below.</td></tr>
-                )}
-              </tbody>
-            </table>
-          </div>
-        )}
-
-        {/* Add field */}
-        <div className="flex gap-2 mb-3">
-          <input className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
-            placeholder="column name" value={newField.name}
-            onChange={e => setNewField(f => ({ ...f, name: e.target.value }))}
-            onKeyDown={e => e.key === 'Enter' && addField()} />
-          <select className="border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
-            value={newField.type} onChange={e => setNewField(f => ({ ...f, type: e.target.value }))}>
-            {FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
-          </select>
-          <button onClick={addField} className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700">Add</button>
-        </div>
-
-        {mappingsDirty && (
-          <button onClick={saveMappings} disabled={saving}
-            className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
-            {saving ? 'Saving…' : 'Save mappings'}
-          </button>
-        )}
-      </div>
-
-      {/* Generate view + balance */}
-      <div className="bg-white border border-gray-200 rounded p-4">
-        <div className="flex items-center justify-between mb-3">
-          <h3 className="text-sm font-semibold text-gray-700">View</h3>
-          <div className="flex gap-2">
-            <button onClick={fetchBalance}
-              className="text-sm bg-gray-100 text-gray-700 px-3 py-1.5 rounded hover:bg-gray-200">
-              Refresh balance
-            </button>
-            <button onClick={generateView}
-              className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700">
-              Generate / refresh
-            </button>
-          </div>
-        </div>
-        {netBalance !== null && (
-          <div className="mb-3 flex items-center gap-3">
-            <span className="text-xs text-gray-500">Current net balance</span>
-            <span className="text-lg font-mono font-semibold text-gray-800">
-              {Number(netBalance).toLocaleString(undefined, { minimumFractionDigits: 2 })}
-            </span>
-          </div>
-        )}
-        {balanceError && <p className="text-xs text-gray-400 mb-3">{balanceError}</p>}
-        {viewResult && !viewResult.success && (
-          <p className="text-xs text-red-500">{viewResult.error}</p>
-        )}
-        {viewResult && viewResult.success && (
-          <p className="text-xs text-green-600">View created: <span className="font-mono">{viewResult.view}</span></p>
-        )}
-      </div>
-
-      {calibratingSource && (
-        <CalibrateModal
-          stack={stack}
-          sourceName={calibratingSource}
-          currentOffset={srcCfg[calibratingSource]?.offset ?? 0}
-          onClose={() => setCalibratingSource(null)}
-          onApply={offset => applyCalibration(calibratingSource, offset)}
-        />
-      )}
-    </div>
-  )
-}
-
-// ── Main page ──────────────────────────────────────────────────────────────────
-
-export default function Stacks({ sources, onStackStale, onStackViewGenerated, onStacksChange }) {
-  const [stacks, setStacks] = useState([])
-  const [selected, setSelected] = useState(null)
-  const [stackDetail, setStackDetail] = useState(null)
-  const [creating, setCreating] = useState(false)
-  const [newName, setNewName] = useState('')
-  const [error, setError] = useState('')
-  const [sqlDraft, setSqlDraft] = useState('')
-  const [sqlRunning, setSqlRunning] = useState(false)
-  const [sqlResult, setSqlResult] = useState(null)
-
-  async function load() {
-    const s = await api.getStacks()
-    setStacks(s)
-    return s
-  }
-
-  async function loadDetail(name) {
-    const s = await api.getStack(name)
-    setStackDetail(s)
-    setSelected(name)
-    localStorage.setItem('stacks_last_selected', name)
-    setSqlDraft('')
-    setSqlResult(null)
-  }
-
-  useEffect(() => {
-    load().then(s => {
-      const last = localStorage.getItem('stacks_last_selected')
-      if (last && s.find(x => x.name === last)) loadDetail(last)
-    })
-  }, [])
-  useEffect(() => { if (selected) loadDetail(selected) }, [selected])
-
-  async function createStack() {
-    if (!newName) return
-    setError('')
-    try {
-      await api.createStack({ name: newName, fields: [] })
-      setNewName(''); setCreating(false)
-      await load()
-      onStacksChange?.()
-      loadDetail(newName)
-    } catch (e) { setError(e.message) }
-  }
-
-  async function deleteStack(name) {
-    if (!confirm(`Delete stack "${name}"?`)) return
-    await api.deleteStack(name)
-    if (selected === name) { setSelected(null); setStackDetail(null); setSqlDraft(''); setSqlResult(null) }
-    load()
-    onStacksChange?.()
-  }
-
-  async function runSql() {
-    if (!sqlDraft.trim() || !selected) return
-    setSqlRunning(true); setSqlResult(null)
-    try {
-      const r = await api.execStackSql(selected, sqlDraft)
-      setSqlResult(r)
-      if (r.success) {
-        onStackViewGenerated?.(selected)
-        ;(r.cascade_stale || []).forEach(n => onStackStale?.(n))
-      }
-    } catch (e) { setSqlResult({ success: false, error: e.message }) }
-    finally { setSqlRunning(false) }
-  }
-
-  return (
-    <div className="p-6">
-      {/* Stack list — horizontal row of cards */}
-      <div className="flex items-center gap-2 mb-5 flex-wrap">
-        <h1 className="text-sm font-semibold text-gray-800 mr-1">Stacks</h1>
-        {stacks.map(s => (
-          <div key={s.name}
-            onClick={() => loadDetail(s.name)}
-            className={`flex items-center gap-2 px-3 py-1.5 rounded border cursor-pointer text-xs group transition-colors ${selected === s.name ? 'border-blue-300 bg-blue-50 text-blue-700' : 'border-gray-200 bg-white text-gray-600 hover:border-gray-300 hover:bg-gray-50'}`}>
-            <span className="font-medium">{s.label || s.name}</span>
-            <span className="text-gray-400">{s.source_count}s</span>
-            <button onClick={e => { e.stopPropagation(); deleteStack(s.name) }}
-              className="opacity-0 group-hover:opacity-100 text-red-300 hover:text-red-500 leading-none">✕</button>
-          </div>
-        ))}
-        {creating ? (
-          <div className="flex items-center gap-1">
-            <input autoFocus className="border border-blue-400 rounded px-2 py-1 text-xs focus:outline-none w-32"
-              placeholder="stack name" value={newName} onChange={e => setNewName(e.target.value)}
-              onKeyDown={e => { if (e.key === 'Enter') createStack(); if (e.key === 'Escape') setCreating(false) }} />
-            <button onClick={createStack} className="text-xs bg-blue-600 text-white px-2 py-1 rounded">Create</button>
-            <button onClick={() => setCreating(false)} className="text-xs text-gray-400 px-1">✕</button>
-            {error && <p className="text-xs text-red-500">{error}</p>}
-          </div>
-        ) : (
-          <button onClick={() => setCreating(true)} className="text-xs text-blue-500 hover:text-blue-700 px-2 py-1.5">+ New</button>
-        )}
-      </div>
-
-      {stackDetail ? (
-        <div className="flex gap-6 items-start">
-          {/* Left: config panel */}
-          <div className="flex-1 min-w-0">
-            <h2 className="text-base font-semibold text-gray-800 mb-4">
-              {stackDetail.label || stackDetail.name}
-              {stackDetail.label && <span className="text-sm text-gray-400 font-normal ml-2">{stackDetail.name}</span>}
-            </h2>
-            <StackPanel
-              key={stackDetail.name}
-              stack={stackDetail}
-              sources={sources}
-              onUpdated={() => { load(); loadDetail(stackDetail.name) }}
-              onStale={onStackStale}
-              onViewGenerated={onStackViewGenerated}
-              onSqlGenerated={sql => { setSqlDraft(prettySql(sql)); setSqlResult(null) }}
-            />
-          </div>
-
-          {/* Right: SQL panel */}
-          <div className="flex-1 min-w-0">
-            <div className="bg-white border border-gray-200 rounded p-4 sticky top-4">
-              <div className="flex items-center justify-between mb-3">
-                <h3 className="text-sm font-semibold text-gray-700">Generated SQL</h3>
-                <button
-                  onClick={runSql}
-                  disabled={!sqlDraft.trim() || sqlRunning}
-                  className="text-sm bg-blue-600 text-white px-3 py-1 rounded hover:bg-blue-700 disabled:opacity-40">
-                  {sqlRunning ? 'Running…' : 'Run'}
-                </button>
-              </div>
-              {sqlDraft ? (
-                <textarea
-                  className="w-full font-mono text-xs text-gray-700 bg-gray-50 border border-gray-200 rounded p-2 focus:outline-none focus:border-blue-400 resize-none leading-relaxed"
-                  style={{ minHeight: '60vh' }}
-                  value={sqlDraft}
-                  onChange={e => { setSqlDraft(e.target.value); setSqlResult(null) }}
-                  spellCheck={false}
-                />
-              ) : (
-                <p className="text-xs text-gray-400">Generate a view to see the SQL here.</p>
-              )}
-              {sqlResult && (
-                <p className={`text-xs mt-2 ${sqlResult.success ? 'text-green-600' : 'text-red-500'}`}>
-                  {sqlResult.success ? 'View updated successfully.' : sqlResult.error}
-                </p>
-              )}
-            </div>
-          </div>
-        </div>
-      ) : (
-        <p className="text-sm text-gray-400">Select a stack or create one.</p>
-      )}
-    </div>
-  )
-}

ui/src/theme.jsx

@@ -1,25 +0,0 @@
-import { createContext, useContext, useState, useEffect } from 'react'
-
-const ThemeContext = createContext()
-
-export function ThemeProvider({ children }) {
-  const [dark, setDark] = useState(() => {
-    const saved = localStorage.getItem('df_dark')
-    if (saved !== null) return saved === 'true'
-    return window.matchMedia('(prefers-color-scheme: dark)').matches
-  })
-
-  useEffect(() => {
-    localStorage.setItem('df_dark', dark)
-    document.documentElement.classList.toggle('dark', dark)
-  }, [dark])
-
-  return (
-    <ThemeContext.Provider value={{ dark, setDark }}>
-      {children}
-    </ThemeContext.Provider>
-  )
-}
-
-const useTheme = () => useContext(ThemeContext)
-export default useTheme

uninstall.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+#
+# Dataflow Uninstall Script
+# Removes database user, database, and optionally .env
+#
+
+echo "⚠️  Dataflow Uninstall"
+echo "====================="
+echo ""
+
+# Load .env if it exists
+if [ -f .env ]; then
+    export $(cat .env | grep -v '^#' | xargs)
+fi
+
+DB_NAME=${DB_NAME:-dataflow}
+DB_USER=${DB_USER:-dataflow}
+
+echo "⚠️  This will permanently delete:"
+echo "  - Database: $DB_NAME"
+echo "  - User: $DB_USER"
+echo ""
+read -p "Type 'delete' to confirm: " CONFIRM
+if [ "$CONFIRM" != "delete" ]; then
+    echo "Cancelled."
+    exit 0
+fi
+
+# Prompt for admin credentials
+echo ""
+echo "📋 PostgreSQL Admin Credentials"
+echo ""
+read -p "Admin username [postgres]: " ADMIN_USER
+ADMIN_USER=${ADMIN_USER:-postgres}
+read -s -p "Admin password: " ADMIN_PASS
+echo ""
+
+DB_HOST=${DB_HOST:-localhost}
+DB_PORT=${DB_PORT:-5432}
+DB_NAME=${DB_NAME:-dataflow}
+DB_USER=${DB_USER:-dataflow}
+
+# Test admin connection
+echo ""
+echo "🔍 Testing PostgreSQL admin connection..."
+export PGPASSWORD="$ADMIN_PASS"
+if ! psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c '\q' 2>/dev/null; then
+    echo "✗ Cannot connect to PostgreSQL"
+    exit 1
+fi
+echo "✓ Connected"
+
+# Drop database
+echo ""
+echo "🗄️  Dropping database..."
+psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c "DROP DATABASE IF EXISTS $DB_NAME;" 2>/dev/null || true
+echo "✓ Database dropped"
+
+# Drop user
+echo ""
+echo "👤 Dropping user..."
+psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c "DROP USER IF EXISTS $DB_USER;" 2>/dev/null || true
+echo "✓ User dropped"
+
+unset PGPASSWORD
+
+# Optionally remove .env
+echo ""
+read -p "Remove .env file? [y/N]: " REMOVE_ENV
+if [ "$REMOVE_ENV" == "y" ] || [ "$REMOVE_ENV" == "Y" ]; then
+    rm -f .env
+    echo "✓ .env removed"
+fi
+
+# Optionally remove node_modules
+echo ""
+read -p "Remove node_modules? [y/N]: " REMOVE_MODULES
+if [ "$REMOVE_MODULES" == "y" ] || [ "$REMOVE_MODULES" == "Y" ]; then
+    rm -rf node_modules
+    echo "✓ node_modules removed"
+fi
+
+echo ""
+echo "✅ Uninstall complete!"
+echo ""