Add PERSPECTIVE.md documenting the @perspective-dev version pairing

Records why the 4.5.1 viewer/client + 4.4.1 d3fc pairing is deliberate, not a skew to "fix": the /inline and /themes entrypoints exist only in 4.5.x, while viewer-d3fc caps at 4.4.1, so this is the only combination that keeps both inline WASM bundling and the d3fc charts. Verified by build failure when pinning all four to 4.4.1. Points to the canonical guide in pf_app for shared rationale. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update all docs to reflect current state
2026-06-15 23:53:31 -04:00 · 2026-06-12 23:51:00 -04:00 · 2026-06-12 23:36:27 -04:00 · 2026-06-12 23:35:15 -04:00 · 2026-06-12 23:04:25 -04:00 · 2026-06-12 23:00:23 -04:00
36 changed files with 4365 additions and 419 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -124,9 +124,15 @@ records.data → apply_transformations() →
 ### Deduplication
 - `constraint_key` is a JSONB object of the constraint field values (readable, no hashing)
- Dedup is enforced at import time via CTE — no unique DB constraint
+- Dedup is enforced at import time via CTE — NO unique DB constraint on constraint_key
- Intra-file duplicate rows are allowed (bank may send identical rows); they all insert
+- **The constraint key is for cross-batch re-import protection, NOT record uniqueness**
- On re-import, all rows whose constraint_key already exists in the DB are skipped
+- Within a single import batch, ALL rows insert regardless of duplicate constraint keys
  - Banks legitimately send multiple identical-looking transactions (same date, description, amount)
  - Example: 11 Cedar Point merchandise charges on one day — all should insert in one batch
 - On re-import of overlapping date range, rows whose constraint_key already exists in DB are skipped
  - This prevents double-counting when you re-run a month-to-date export the next day
 - NEVER use `ON CONFLICT (constraint_key)` — there is no unique constraint and it would wrongly
  drop legitimate duplicate transactions from the same batch
 - Deleting an import log entry cascades to all records from that batch (import_id FK)
 ### Error Handling
@ -134,6 +140,43 @@ records.data → apply_transformations() →
 - Server.js has global error handler
 - Database functions return JSON with `success` boolean
 ## Light / dark mode
 Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
 - **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
 - **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
 - **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc.
 - **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
 - **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires
 - **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
 ## UI (React + Vite)
 The frontend lives in `ui/src/` and is built to `public/` via `npm run build` from the `ui/` directory. **Always run `npm run build` from `ui/` after any changes to `ui/src/` files.**
 ### Pages
 - **Sources / Rules / Mappings / Records** — standard CRUD pages
 - **Pivot** (`ui/src/pages/Pivot.jsx`) — interactive pivot/crosstab powered by Perspective (`@perspective-dev` v4.5.1, installed via npm). See `docs/perspective-pivot.md` for the full Perspective API reference.
 - **Stacks** — multi-source union views with running balance
 - **Log** — import audit trail
 ### Pivot inspector panel
 Clicking a data cell opens a right-hand inspector panel showing the underlying transactions for that cell. Key behaviors:
 - **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views.
 - **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click.
 - **split_by filter derivation**: `detail.config.filter` from the click event may omit split_by column constraints. They are derived from `column_names` positionally (`column_names[i]` matches `config.split_by[i]`) and appended to the filter before querying.
 - **Row filtering**: a temporary `table.view({ filter, expressions })` is used so Perspective evaluates expression/computed columns correctly. Falls back to JS-side `filterRowsByConfig` on error (which skips filters for fields not in raw data).
 - The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px).
 - The transaction table is sortable (click header) and shows column totals for all-numeric columns.
 ### Pivot layout persistence
 Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The `source_name` column holds either a source name or a stack name — the FK to `sources(name)` was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use `/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`). `localStorage` is still used to remember the *last active layout* for a view (the `psp_layout_<name>` key), but named layout definitions live in the DB so they persist across machines.
 ## File Structure
 ```
@ -148,6 +191,14 @@ dataflow/
 │       ├── rules.js
 │       ├── mappings.js
 │       └── records.js
 ├── ui/
 │   ├── src/
 │   │   ├── pages/         # One file per page
 │   │   └── api.js         # API client
 │   └── package.json
 ├── public/                # Built UI (gitignored, generated by npm run build)
 ├── docs/
 │   └── perspective-pivot.md  # Perspective API reference
 ├── examples/
 │   ├── GETTING_STARTED.md # Tutorial
 │   └── bank_transactions.csv
--- a/PERSPECTIVE.md
+++ b/PERSPECTIVE.md
@ -0,0 +1,49 @@
 # Perspective — dataflow specifics
 Shared rationale lives in the canonical guide: **`/home/pt/pf_app/PERSPECTIVE.md`**
 (loading, version policy, Arrow constraints, deploy pattern, upgrade smoke test).
 This file records only what's specific to dataflow.
 > **Distribution:** these are the **`@perspective-dev/*`** packages (repo
 > github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
 > engine, separate npm scope and release schedule — don't mix the two.
 ---
 ## Current state
 - **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
  This is the target loader; pf_app should adopt it.
 - **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
  Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
  unless view sizes grow well past 100k.
 - **Deploy:** `deploy.sh` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
  for the org; pf_app should copy it.
 - **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
  Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
 - **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
  restore — the reference implementation; keep it.
 ## The version pair is correct — do NOT "fix" it to 4.4.1
 `ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
 `^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
 combination that keeps both of dataflow's hard requirements:
 - **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
  `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
  paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
 - **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
 Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
 `"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
 stays. Don't touch it.
 **What to actually do:**
 - Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
  on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
  explicit.)
 - Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
  renders, dark/light re-themes, and save→reload→drop-column layout restore.
 - Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
  4.5.x suite becomes possible and the pair can collapse to one version.
--- a/README.md
+++ b/README.md
@ -46,7 +46,7 @@ Map extracted values to clean, standardized output.
 ### Prerequisites
 - PostgreSQL 12+
- Node.js 16+
+- Node.js 18+
 - Python 3 (for `manage.py`)
 ### Installation
@ -66,7 +66,7 @@ For development with auto-reload:
 npm run dev
 ```
-The UI is available at `http://localhost:3000`. The API is at `http://localhost:3000/api`.
+The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`).
 ## Management Script (`manage.py`)
@ -154,6 +154,20 @@ All `/api` routes require HTTP Basic authentication.
 | DELETE | `/api/records/:id` | Delete a record |
 | DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
 ### Stacks — `/api/stacks`
 | Method | Path | Description |
 |--------|------|-------------|
 | GET | `/api/stacks` | List all stacks |
 | POST | `/api/stacks` | Create a stack |
 | GET | `/api/stacks/:name` | Get a stack |
 | PUT | `/api/stacks/:name` | Update a stack |
 | DELETE | `/api/stacks/:name` | Delete a stack |
 | GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) |
 | GET | `/api/stacks/:name/layouts` | List saved pivot layouts |
 | POST | `/api/stacks/:name/layouts` | Save a pivot layout |
 | DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout |
 ## Typical Workflow
 ```
@ -175,7 +189,13 @@ See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
 dataflow/
 ├── database/
 │   ├── schema.sql           # Table definitions
-│   └── functions.sql      # Import/transform/query functions
+│   └── queries/             # SQL functions, one file per route
 │       ├── sources.sql
 │       ├── rules.sql
 │       ├── mappings.sql
 │       ├── records.sql
 │       ├── stacks.sql
 │       └── status.sql
 ├── api/
 │   ├── server.js            # Express server
 │   ├── middleware/
@ -186,7 +206,9 @@ dataflow/
 │       ├── sources.js
 │       ├── rules.js
 │       ├── mappings.js
-│       └── records.js
+│       ├── records.js
 │       ├── stacks.js
 │       └── status.js
 ├── public/                  # Built React UI (served as static files)
 ├── examples/
 │   ├── GETTING_STARTED.md
--- a/SPEC.md
+++ b/SPEC.md
@ -50,6 +50,8 @@ api/
    rules.js           — HTTP handlers for rule management
    mappings.js        — HTTP handlers for mapping management
    records.js         — HTTP handlers for record queries
    stacks.js          — HTTP handlers for stack management
    status.js          — HTTP handler for deployment status
 ui/
  src/
    api.js             — fetch wrapper, credential management
@ -135,6 +137,12 @@ Each file in `database/queries/` maps 1-to-1 with a route file.
 **records.sql**
 `list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records`
 **stacks.sql**
 `list_stacks`, `get_stack`, `create_stack`, `update_stack`, `delete_stack`, `get_stack_view_data` (union of source views with field mapping and running balance), `list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`
 **status.sql**
 `get_status` — returns deployment state (schema version, function presence, service status)
 ---
 ## API
@ -182,6 +190,16 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
 | POST | /api/records/search | Search by JSONB containment |
 | DELETE | /api/records/:id | Delete record |
 | DELETE | /api/records/source/:name/all | Delete all records for a source |
 | GET | /api/stacks | List all stacks |
 | POST | /api/stacks | Create stack |
 | GET | /api/stacks/:name | Get stack |
 | PUT | /api/stacks/:name | Update stack |
 | DELETE | /api/stacks/:name | Delete stack |
 | GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
 | GET | /api/stacks/:name/layouts | List saved pivot layouts |
 | POST | /api/stacks/:name/layouts | Save pivot layout |
 | DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout |
 | GET | /api/status | Deployment status |
 ---
@ -218,11 +236,11 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
 - **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page.
- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` v4.4.0, loaded from CDN at runtime). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
+- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` client/viewer/datagrid v4.5.1, viewer-d3fc v4.4.1 — installed via npm). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
  **Toolbar (above the viewer):**
  - Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it.
-  - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found in v4.4.0 — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
+  - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
  - The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads.
  **Cell inspector (right panel):**
@ -237,6 +255,8 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
  See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically.
 - **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
 - **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column.
 ---
--- a/api/lib/sql.js
+++ b/api/lib/sql.js
@ -8,7 +8,7 @@
 function lit(val) {
    if (val === null || val === undefined) return 'NULL';
    if (typeof val === 'boolean') return val ? 'TRUE' : 'FALSE';
-    if (typeof val === 'number') return String(Math.trunc(val));
+    if (typeof val === 'number') return String(val);
    if (typeof val === 'object') return `'${JSON.stringify(val).replace(/'/g, "''")}'`;
    return `'${String(val).replace(/'/g, "''")}'`;
 }
--- a/api/routes/records.js
+++ b/api/routes/records.js
@ -49,6 +49,56 @@ module.exports = (pool) => {
        }
    });
    // Set overrides for all selected records
    router.post('/bulk-overrides', async (req, res, next) => {
        try {
            const { source_name, record_ids, overrides } = req.body;
            if (!source_name || !Array.isArray(record_ids) || record_ids.length === 0 || !overrides || typeof overrides !== 'object')
                return res.status(400).json({ error: 'source_name, record_ids array, and overrides object required' });
            const idList = record_ids.map(id => parseInt(id)).join(',');
            const result = await pool.query(
                `SELECT bulk_set_record_overrides(${lit(source_name)}, ARRAY[${idList}]::int[], ${lit(overrides)}) as result`
            );
            res.json(result.rows[0].result);
        } catch (err) {
            next(err);
        }
    });
    // Set overrides for a record
    router.put('/:id/overrides', async (req, res, next) => {
        try {
            const { overrides } = req.body;
            if (!overrides || typeof overrides !== 'object')
                return res.status(400).json({ error: 'overrides object required' });
            const result = await pool.query(
                `SELECT set_record_overrides(${lit(parseInt(req.params.id))}, ${lit(overrides)}) as rec`
            );
            if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
            res.json(result.rows[0].rec);
        } catch (err) {
            next(err);
        }
    });
    // Clear overrides and reprocess that record to restore computed values
    router.delete('/:id/overrides', async (req, res, next) => {
        try {
            const result = await pool.query(
                `SELECT clear_record_overrides(${lit(parseInt(req.params.id))}) as rec`
            );
            if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
            const { source_name } = result.rows[0].rec;
            await pool.query(
                `SELECT apply_transformations(${lit(source_name)}, ARRAY[${lit(parseInt(req.params.id))}::int], true)`
            );
            const updated = await pool.query(`SELECT * FROM get_record(${lit(parseInt(req.params.id))})`);
            res.json(updated.rows[0]);
        } catch (err) {
            next(err);
        }
    });
    // Delete record
    router.delete('/:id', async (req, res, next) => {
        try {
--- a/api/routes/sources.js
+++ b/api/routes/sources.js
@ -187,7 +187,11 @@ module.exports = (pool) => {
    router.post('/:name/view', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT generate_source_view(${lit(req.params.name)}) as result`);
-            res.json(result.rows[0].result);
+            const data = result.rows[0].result;
            if (data && data.success) {
                await pool.query(`UPDATE dataflow.sources SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
            }
            res.json(data);
        } catch (err) {
            next(err);
        }
@ -230,6 +234,19 @@ module.exports = (pool) => {
        }
    });
    // Override keys — distinct field names used in overrides across all records for this source
    router.get('/:name/override-keys', async (req, res, next) => {
        try {
            const result = await pool.query(
                `SELECT DISTINCT jsonb_object_keys(overrides) AS key
                 FROM dataflow.records
                 WHERE source_name = ${lit(req.params.name)} AND overrides IS NOT NULL
                 ORDER BY key`
            );
            res.json(result.rows.map(r => r.key));
        } catch (err) { next(err); }
    });
    // Pivot layouts
    router.get('/:name/layouts', async (req, res, next) => {
        try {
--- a/api/routes/stacks.js
+++ b/api/routes/stacks.js
@ -0,0 +1,201 @@
 /**
 * Stacks Routes
 * Named unions of multiple sources with field mappings and running balance
 */
 const express = require('express');
 const { lit, arr } = require('../lib/sql');
 module.exports = (pool) => {
    const router = express.Router();
    // List all stacks
    router.get('/', async (req, res, next) => {
        try {
            const result = await pool.query('SELECT * FROM list_stacks()');
            res.json(result.rows);
        } catch (err) { next(err); }
    });
    // Get single stack with sources
    router.get('/:name', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT * FROM get_stack(${lit(req.params.name)})`);
            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
            res.json(result.rows[0]);
        } catch (err) { next(err); }
    });
    // Create stack
    router.post('/', async (req, res, next) => {
        try {
            const { name, label, fields, amount_field, date_field, balance_offset } = req.body;
            if (!name) return res.status(400).json({ error: 'name is required' });
            const result = await pool.query(
                `SELECT * FROM create_stack(${lit(name)}, ${lit(label || null)}, ${lit(JSON.stringify(fields || []))}, ${lit(amount_field || null)}, ${lit(date_field || null)}, ${lit(balance_offset ?? 0)})`
            );
            res.status(201).json(result.rows[0]);
        } catch (err) {
            if (err.code === '23505') return res.status(409).json({ error: 'Stack already exists' });
            next(err);
        }
    });
    // Update stack
    router.put('/:name', async (req, res, next) => {
        try {
            const { label, fields, amount_field, date_field, balance_offset } = req.body;
            const n = v => v !== undefined ? lit(v) : 'NULL';
            const f = v => v !== undefined ? lit(JSON.stringify(v)) : 'NULL';
            const result = await pool.query(
                `SELECT * FROM update_stack(${lit(req.params.name)}, ${n(label)}, ${f(fields)}, ${n(amount_field)}, ${n(date_field)}, ${n(balance_offset)})`
            );
            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
            res.json(result.rows[0]);
        } catch (err) { next(err); }
    });
    // Delete stack
    router.delete('/:name', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT * FROM delete_stack(${lit(req.params.name)})`);
            if (!result.rows.length) return res.status(404).json({ error: 'Stack not found' });
            res.json({ success: true, deleted: req.params.name });
        } catch (err) { next(err); }
    });
    // Add or update a source in a stack
    router.put('/:name/sources/:source', async (req, res, next) => {
        try {
            const { field_map, amount_sign, balance_offset, amount_field, date_field } = req.body;
            const n = v => v != null ? lit(v) : 'NULL';
            const result = await pool.query(
                `SELECT * FROM upsert_stack_source(${lit(req.params.name)}, ${lit(req.params.source)}, ${lit(JSON.stringify(field_map || {}))}, ${lit(amount_sign ?? 1)}, ${lit(balance_offset ?? 0)}, ${n(amount_field)}, ${n(date_field)})`
            );
            res.json(result.rows[0]);
        } catch (err) {
            if (err.code === '23503') return res.status(404).json({ error: 'Stack or source not found' });
            next(err);
        }
    });
    // Remove a source from a stack
    router.delete('/:name/sources/:source', async (req, res, next) => {
        try {
            const result = await pool.query(
                `SELECT * FROM remove_stack_source(${lit(req.params.name)}, ${lit(req.params.source)})`
            );
            if (!result.rows.length) return res.status(404).json({ error: 'Source not in stack' });
            res.json({ success: true, removed: req.params.source });
        } catch (err) { next(err); }
    });
    // Reorder sources within a stack
    router.put('/:name/sources/reorder', async (req, res, next) => {
        try {
            const { source_names } = req.body;
            if (!Array.isArray(source_names)) return res.status(400).json({ error: 'source_names array required' });
            await pool.query(`SELECT reorder_stack_sources(${lit(req.params.name)}, ${arr(source_names)})`);
            res.json({ success: true });
        } catch (err) { next(err); }
    });
    // Get current running balance from the generated view
    router.get('/:name/balance', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT get_stack_balance(${lit(req.params.name)}) AS result`);
            res.json(result.rows[0].result);
        } catch (err) { next(err); }
    });
    // Preview the SQL that would be generated (dry run — does not create the view)
    router.get('/:name/view-sql', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT generate_stack_view(${lit(req.params.name)}, true) AS result`);
            res.json(result.rows[0].result);
        } catch (err) { next(err); }
    });
    // Generate / refresh the dfv view
    router.post('/:name/view', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT generate_stack_view(${lit(req.params.name)}) AS result`);
            const data = result.rows[0].result;
            if (data && data.success) {
                await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
            }
            res.json(data);
        } catch (err) { next(err); }
    });
    // Execute custom SQL for the view (user-edited SQL)
    router.post('/:name/exec-sql', async (req, res, next) => {
        try {
            const { sql } = req.body;
            if (!sql) return res.status(400).json({ success: false, error: 'sql is required' });
            await pool.query(`DROP VIEW IF EXISTS dfv.${req.params.name} CASCADE`);
            await pool.query(sql);
            await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NOW() WHERE name = ${lit(req.params.name)}`);
            // Detect stacks whose views were dropped by CASCADE
            const staleResult = await pool.query(`
                SELECT array_agg(name) AS names FROM dataflow.stacks
                WHERE name != ${lit(req.params.name)}
                  AND view_generated_at IS NOT NULL
                  AND NOT EXISTS (
                      SELECT 1 FROM pg_views WHERE schemaname = 'dfv' AND viewname = name
                  )
            `);
            const cascadeStale = staleResult.rows[0].names || [];
            if (cascadeStale.length) {
                await pool.query(`UPDATE dataflow.stacks SET view_generated_at = NULL WHERE name = ANY($1)`, [cascadeStale]);
            }
            res.json({ success: true, cascade_stale: cascadeStale });
        } catch (err) {
            res.json({ success: false, error: err.message });
        }
    });
    // Calibrate balance offset given a known good balance at a specific date
    router.post('/:name/calibrate', async (req, res, next) => {
        try {
            const { as_of_date, known_balance, source_name } = req.body;
            if (known_balance === undefined) {
                return res.status(400).json({ error: 'known_balance is required' });
            }
            const dateExpr = as_of_date ? `${lit(as_of_date)}::date` : 'NULL';
            const result = await pool.query(
                `SELECT calibrate_balance(${lit(req.params.name)}, ${source_name ? lit(source_name) : 'NULL'}, ${dateExpr}, ${lit(known_balance)}::numeric) AS result`
            );
            res.json(result.rows[0].result);
        } catch (err) { next(err); }
    });
    // Pivot layouts (same DB table as sources; FK was dropped to allow stack names)
    router.get('/:name/layouts', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT * FROM list_pivot_layouts(${lit(req.params.name)})`);
            res.json(result.rows);
        } catch (err) { next(err); }
    });
    router.post('/:name/layouts', async (req, res, next) => {
        try {
            const { layout_name, config } = req.body;
            if (!layout_name || !config) return res.status(400).json({ error: 'layout_name and config required' });
            const result = await pool.query(
                `SELECT * FROM save_pivot_layout(${lit(req.params.name)}, ${lit(layout_name)}, ${lit(config)})`
            );
            res.json(result.rows[0]);
        } catch (err) { next(err); }
    });
    router.delete('/:name/layouts/:id', async (req, res, next) => {
        try {
            const result = await pool.query(`SELECT * FROM delete_pivot_layout(${lit(parseInt(req.params.id))})`);
            if (result.rows.length === 0) return res.status(404).json({ error: 'Layout not found' });
            res.json({ success: true });
        } catch (err) { next(err); }
    });
    return router;
 };
--- a/api/routes/status.js
+++ b/api/routes/status.js
@ -0,0 +1,14 @@
 const express = require('express');
 module.exports = (pool) => {
    const router = express.Router();
    router.get('/', async (req, res, next) => {
        try {
            const result = await pool.query('SELECT get_status() AS result');
            res.json(result.rows[0].result);
        } catch (err) { next(err); }
    });
    return router;
 };
--- a/api/server.js
+++ b/api/server.js
@ -3,7 +3,7 @@
 * Simple REST API for data transformation
 */
-require('dotenv').config();
+require('dotenv').config({ quiet: true });
 const express = require('express');
 const { Pool } = require('pg');
@ -16,7 +16,8 @@ const pool = new Pool({
    port: process.env.DB_PORT,
    database: process.env.DB_NAME,
    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD
+    password: process.env.DB_PASSWORD,
    options: '-c search_path=dataflow,public'
 });
 // Middleware
@ -31,11 +32,6 @@ app.use('/api', auth);
 const path = require('path');
 app.use(express.static(path.join(__dirname, '../public')));
 // Set search path for all queries
 pool.on('connect', (client) => {
    client.query('SET search_path TO dataflow, public');
 });
 // Test database connection
 pool.query('SELECT NOW()', (err, res) => {
    if (err) {
@ -54,12 +50,16 @@ const sourcesRoutes = require('./routes/sources');
 const rulesRoutes = require('./routes/rules');
 const mappingsRoutes = require('./routes/mappings');
 const recordsRoutes = require('./routes/records');
 const stacksRoutes = require('./routes/stacks');
 const statusRoutes = require('./routes/status');
 // Mount routes
 app.use('/api/sources', sourcesRoutes(pool));
 app.use('/api/rules', rulesRoutes(pool));
 app.use('/api/mappings', mappingsRoutes(pool));
 app.use('/api/records', recordsRoutes(pool));
 app.use('/api/stacks', stacksRoutes(pool));
 app.use('/api/status', statusRoutes(pool));
 // Health check
 app.get('/health', (req, res) => {
--- a/database/functions.sql
+++ b/database/functions.sql
--- a/database/migrate_overrides_column.sql
+++ b/database/migrate_overrides_column.sql
@ -0,0 +1,40 @@
 --
 -- Migration: add overrides column to records
 --
 -- Separates the three data layers:
 --   data        — original import values, never mutated
 --   transformed — rule/mapping output fields only (delta)
 --   overrides   — manual user overrides (highest precedence)
 --
 -- Consumers merge as: data || COALESCE(transformed,'{}') || COALESCE(overrides,'{}')
 --
 -- Safe to run multiple times (IF NOT EXISTS guards).
 --
 SET search_path TO dataflow, public;
 -- 1. Add overrides column
 ALTER TABLE dataflow.records
    ADD COLUMN IF NOT EXISTS overrides JSONB;
 -- 2. Add partial GIN index (only indexes rows that have overrides)
 CREATE INDEX IF NOT EXISTS idx_records_overrides
    ON dataflow.records USING gin(overrides)
    WHERE overrides IS NOT NULL;
 -- 3. Redeploy functions (CREATE OR REPLACE — non-destructive)
 \i functions.sql
 -- 4. Reprocess all sources to strip stale data keys from transformed
 --    (apply_transformations now writes only rule additions, not data || additions)
 DO $$
 DECLARE
    src TEXT;
    result JSON;
 BEGIN
    FOR src IN SELECT name FROM dataflow.sources ORDER BY name LOOP
        SELECT dataflow.reprocess_records(src) INTO result;
        RAISE NOTICE 'Reprocessed %: %', src, result;
    END LOOP;
 END;
 $$;
--- a/database/migrate_pivot_layouts_drop_fk.sql
+++ b/database/migrate_pivot_layouts_drop_fk.sql
@ -0,0 +1,4 @@
 -- Drop the foreign key from pivot_layouts.source_name so stack view names can also
 -- be used as layout keys (stacks are not rows in the sources table).
 ALTER TABLE dataflow.pivot_layouts
    DROP CONSTRAINT pivot_layouts_source_name_fkey;
--- a/database/queries/records.sql
+++ b/database/queries/records.sql
@ -39,6 +39,41 @@ RETURNS SETOF dataflow.records AS $$
    LIMIT p_limit;
 $$ LANGUAGE sql STABLE;
 -- ── Overrides ─────────────────────────────────────────────────────────────────
 -- Store manual overrides and immediately merge into transformed
 CREATE OR REPLACE FUNCTION set_record_overrides(p_id INT, p_overrides JSONB)
 RETURNS dataflow.records AS $$
    UPDATE dataflow.records
    SET overrides  = CASE WHEN p_overrides = '{}'::jsonb THEN NULL ELSE p_overrides END,
        transformed = COALESCE(transformed, data) || COALESCE(p_overrides, '{}'::jsonb)
    WHERE id = p_id
    RETURNING *;
 $$ LANGUAGE sql;
 -- Merge overrides into multiple records at once; returns actual updated count
 CREATE OR REPLACE FUNCTION bulk_set_record_overrides(p_source_name TEXT, p_ids INT[], p_overrides JSONB)
 RETURNS BIGINT AS $$
    WITH updated AS (
        UPDATE dataflow.records
        SET overrides   = COALESCE(overrides, '{}'::jsonb) || p_overrides,
            transformed = COALESCE(transformed, data) || p_overrides
        WHERE id = ANY(p_ids)
          AND source_name = p_source_name
        RETURNING id
    )
    SELECT count(*) FROM updated;
 $$ LANGUAGE sql;
 -- Clear overrides; caller should reprocess to restore computed transformed value
 CREATE OR REPLACE FUNCTION clear_record_overrides(p_id INT)
 RETURNS dataflow.records AS $$
    UPDATE dataflow.records
    SET overrides = NULL
    WHERE id = p_id
    RETURNING *;
 $$ LANGUAGE sql;
 -- ── Delete ────────────────────────────────────────────────────────────────────
 CREATE OR REPLACE FUNCTION delete_record(p_id BIGINT)
--- a/database/queries/sources.sql
+++ b/database/queries/sources.sql
@ -188,7 +188,7 @@ BEGIN
    v_view := 'dfv.' || quote_ident(p_source_name);
    EXECUTE format('DROP VIEW IF EXISTS %s', v_view);
    v_sql := format(
-        'CREATE VIEW %s AS SELECT %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
+        'CREATE VIEW %s AS SELECT id, overrides IS NOT NULL AS _overridden, %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
        v_view, v_cols, p_source_name
    );
    EXECUTE v_sql;
--- a/database/queries/stacks.sql
+++ b/database/queries/stacks.sql
@ -0,0 +1,472 @@
 --
 -- Stacks queries
 -- All SQL for api/routes/stacks.js
 --
 SET search_path TO dataflow, public;
 ------------------------------------------------------
 -- Tables
 ------------------------------------------------------
 CREATE TABLE IF NOT EXISTS dataflow.stacks (
    name            TEXT PRIMARY KEY,
    label           TEXT,
    -- Ordered canonical field definitions: [{name, label, type}]
    -- type: 'text' | 'numeric' | 'date'
    fields          JSONB NOT NULL DEFAULT '[]',
    -- Running balance config
    amount_field    TEXT,           -- canonical field to sum for running balance
    date_field      TEXT,           -- canonical field to order by
    balance_offset  NUMERIC DEFAULT 0,  -- added to running sum (calibration)
    created_at      TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
 );
 CREATE TABLE IF NOT EXISTS dataflow.stack_sources (
    id              SERIAL PRIMARY KEY,
    stack_name      TEXT NOT NULL REFERENCES dataflow.stacks(name) ON DELETE CASCADE,
    source_name     TEXT NOT NULL REFERENCES dataflow.sources(name) ON DELETE CASCADE,
    -- Maps other canonical field names → source view column names (not amount/date — those are explicit)
    field_map       JSONB NOT NULL DEFAULT '{}',
    -- Which column in dfv.{source} is the amount, and its sign (+1/-1)
    amount_field    TEXT,
    amount_sign     INTEGER NOT NULL DEFAULT 1,
    -- Which column in dfv.{source} is the date
    date_field      TEXT,
    -- Calibration offset added to this source's running balance
    balance_offset  NUMERIC NOT NULL DEFAULT 0,
    UNIQUE (stack_name, source_name)
 );
 -- Migrations: add columns that may be missing from earlier deploys
 ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS balance_offset NUMERIC NOT NULL DEFAULT 0;
 ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS amount_field TEXT;
 ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS date_field TEXT;
 ALTER TABLE dataflow.stack_sources ADD COLUMN IF NOT EXISTS seq INTEGER NOT NULL DEFAULT 0;
 -- Seed seq from insertion order for existing rows
 UPDATE dataflow.stack_sources ss
 SET seq = sub.rn
 FROM (
    SELECT id, ROW_NUMBER() OVER (PARTITION BY stack_name ORDER BY id) AS rn
    FROM dataflow.stack_sources WHERE seq = 0
 ) sub
 WHERE ss.id = sub.id AND ss.seq = 0;
 -- Drop old signatures before recreating
 DROP FUNCTION IF EXISTS calibrate_balance(TEXT, DATE, NUMERIC);
 DROP FUNCTION IF EXISTS upsert_stack_source(TEXT, TEXT, JSONB, INTEGER, NUMERIC);
 DROP FUNCTION IF EXISTS generate_stack_view(TEXT);
 ------------------------------------------------------
 -- Function: list_stacks
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION list_stacks()
 RETURNS TABLE (
    name            TEXT,
    label           TEXT,
    fields          JSONB,
    amount_field    TEXT,
    date_field      TEXT,
    balance_offset  NUMERIC,
    source_count    BIGINT,
    created_at      TIMESTAMPTZ
 ) AS $$
    SELECT
        s.name, s.label, s.fields,
        s.amount_field, s.date_field, s.balance_offset,
        count(ss.id) AS source_count,
        s.created_at
    FROM dataflow.stacks s
    LEFT JOIN dataflow.stack_sources ss ON ss.stack_name = s.name
    GROUP BY s.name, s.label, s.fields, s.amount_field, s.date_field, s.balance_offset, s.created_at
    ORDER BY s.name;
 $$ LANGUAGE sql STABLE;
 ------------------------------------------------------
 -- Function: get_stack
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION get_stack(p_name TEXT)
 RETURNS TABLE (
    name            TEXT,
    label           TEXT,
    fields          JSONB,
    amount_field    TEXT,
    date_field      TEXT,
    balance_offset  NUMERIC,
    created_at      TIMESTAMPTZ,
    sources         JSONB
 ) AS $$
    SELECT
        s.name, s.label, s.fields,
        s.amount_field, s.date_field, s.balance_offset,
        s.created_at,
        COALESCE(jsonb_agg(
            jsonb_build_object(
                'id',             ss.id,
                'source_name',    ss.source_name,
                'field_map',      ss.field_map,
                'amount_field',   ss.amount_field,
                'amount_sign',    ss.amount_sign,
                'date_field',     ss.date_field,
                'balance_offset', ss.balance_offset,
                'seq',            ss.seq
            ) ORDER BY ss.seq, ss.id
        ) FILTER (WHERE ss.id IS NOT NULL), '[]')
    FROM dataflow.stacks s
    LEFT JOIN dataflow.stack_sources ss ON ss.stack_name = s.name
    WHERE s.name = p_name
    GROUP BY s.name, s.label, s.fields, s.amount_field, s.date_field, s.balance_offset, s.created_at;
 $$ LANGUAGE sql STABLE;
 ------------------------------------------------------
 -- Function: create_stack
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION create_stack(
    p_name          TEXT,
    p_label         TEXT DEFAULT NULL,
    p_fields        JSONB DEFAULT '[]',
    p_amount_field  TEXT DEFAULT NULL,
    p_date_field    TEXT DEFAULT NULL,
    p_balance_offset NUMERIC DEFAULT 0
 ) RETURNS dataflow.stacks AS $$
    INSERT INTO dataflow.stacks (name, label, fields, amount_field, date_field, balance_offset)
    VALUES (p_name, p_label, p_fields, p_amount_field, p_date_field, p_balance_offset)
    RETURNING *;
 $$ LANGUAGE sql;
 ------------------------------------------------------
 -- Function: update_stack
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION update_stack(
    p_name           TEXT,
    p_label          TEXT DEFAULT NULL,
    p_fields         JSONB DEFAULT NULL,
    p_amount_field   TEXT DEFAULT NULL,
    p_date_field     TEXT DEFAULT NULL,
    p_balance_offset NUMERIC DEFAULT NULL
 ) RETURNS dataflow.stacks AS $$
    UPDATE dataflow.stacks SET
        label          = COALESCE(p_label, label),
        fields         = COALESCE(p_fields, fields),
        amount_field   = COALESCE(p_amount_field, amount_field),
        date_field     = COALESCE(p_date_field, date_field),
        balance_offset = COALESCE(p_balance_offset, balance_offset)
    WHERE name = p_name
    RETURNING *;
 $$ LANGUAGE sql;
 ------------------------------------------------------
 -- Function: delete_stack
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION delete_stack(p_name TEXT)
 RETURNS TABLE (name TEXT) AS $$
    DELETE FROM dataflow.stacks WHERE name = p_name RETURNING name;
 $$ LANGUAGE sql;
 ------------------------------------------------------
 -- Function: upsert_stack_source
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION upsert_stack_source(
    p_stack_name      TEXT,
    p_source_name     TEXT,
    p_field_map       JSONB DEFAULT '{}',
    p_amount_sign     INTEGER DEFAULT 1,
    p_balance_offset  NUMERIC DEFAULT 0,
    p_amount_field    TEXT DEFAULT NULL,
    p_date_field      TEXT DEFAULT NULL
 ) RETURNS dataflow.stack_sources AS $$
    INSERT INTO dataflow.stack_sources (stack_name, source_name, field_map, amount_sign, balance_offset, amount_field, date_field, seq)
    VALUES (
        p_stack_name, p_source_name, p_field_map, p_amount_sign, p_balance_offset, p_amount_field, p_date_field,
        (SELECT COALESCE(MAX(seq), 0) + 1 FROM dataflow.stack_sources WHERE stack_name = p_stack_name)
    )
    ON CONFLICT (stack_name, source_name) DO UPDATE SET
        field_map       = EXCLUDED.field_map,
        amount_sign     = EXCLUDED.amount_sign,
        balance_offset  = EXCLUDED.balance_offset,
        amount_field    = EXCLUDED.amount_field,
        date_field      = EXCLUDED.date_field
    RETURNING *;
 $$ LANGUAGE sql;
 ------------------------------------------------------
 -- Function: remove_stack_source
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION remove_stack_source(p_stack_name TEXT, p_source_name TEXT)
 RETURNS TABLE (source_name TEXT) AS $$
    DELETE FROM dataflow.stack_sources
    WHERE stack_name = p_stack_name AND source_name = p_source_name
    RETURNING source_name;
 $$ LANGUAGE sql;
 ------------------------------------------------------
 -- Function: calibrate_balance
 -- Queries dfv.{source} directly using per-source amount/date fields.
 -- No stack view required.
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION calibrate_balance(
    p_stack_name    TEXT,
    p_source_name   TEXT,
    p_as_of_date    DATE,
    p_known_balance NUMERIC
 ) RETURNS JSON AS $$
 DECLARE
    v_src       dataflow.stack_sources%ROWTYPE;
    v_running   NUMERIC;
    v_sql       TEXT;
 BEGIN
    SELECT * INTO v_src
    FROM dataflow.stack_sources
    WHERE stack_name = p_stack_name AND source_name = p_source_name;
    IF NOT FOUND THEN
        RETURN json_build_object('success', false, 'error', 'Source not in stack');
    END IF;
    IF v_src.amount_field IS NULL OR v_src.date_field IS NULL THEN
        RETURN json_build_object('success', false, 'error', 'Set amount and date fields on this source first');
    END IF;
    BEGIN
        IF p_as_of_date IS NULL THEN
            v_sql := format(
                'SELECT COALESCE(SUM(%I * %s), 0) FROM dfv.%I',
                v_src.amount_field, v_src.amount_sign, p_source_name
            );
        ELSE
            v_sql := format(
                'SELECT COALESCE(SUM(%I * %s), 0) FROM dfv.%I WHERE %I <= %L::date',
                v_src.amount_field, v_src.amount_sign, p_source_name, v_src.date_field, p_as_of_date
            );
        END IF;
        EXECUTE v_sql INTO v_running;
    EXCEPTION WHEN undefined_table THEN
        RETURN json_build_object('success', false, 'error', 'Source view not found — generate the source view first');
    END;
    RETURN json_build_object(
        'success',          true,
        'source',           p_source_name,
        'as_of_date',       p_as_of_date,
        'known_balance',    p_known_balance,
        'computed_sum',     v_running,
        'suggested_offset', p_known_balance - v_running
    );
 END;
 $$ LANGUAGE plpgsql STABLE;
 ------------------------------------------------------
 -- Function: generate_stack_view
 -- Builds a WITH ... UNION ALL view in dfv schema from existing dfv source views.
 -- Each source CTE applies amount_sign and computes a per-source running balance.
 -- Outer SELECT adds net_balance across all sources.
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION generate_stack_view(p_stack_name TEXT, p_dry_run BOOLEAN DEFAULT false)
 RETURNS JSON AS $$
 DECLARE
    v_stack      dataflow.stacks%ROWTYPE;
    v_src        dataflow.stack_sources%ROWTYPE;
    v_field      JSONB;
    v_ctes       TEXT[] := '{}';
    v_cte_names  TEXT[] := '{}';
    v_select     TEXT;
    v_col        TEXT;
    v_src_field  TEXT;
    v_amt_src    TEXT;
    v_date_src   TEXT;
    v_view          TEXT;
    v_sql           TEXT;
    v_has_bal       BOOLEAN;
    v_canon_cols    TEXT;
    v_src_bal_cols  TEXT;
    v_total_offset  NUMERIC := 0;
    v_cascade_stale TEXT[];
 BEGIN
    SELECT * INTO v_stack FROM dataflow.stacks WHERE name = p_stack_name;
    IF NOT FOUND THEN
        RETURN json_build_object('success', false, 'error', 'Stack not found');
    END IF;
    v_has_bal := v_stack.amount_field IS NOT NULL AND v_stack.date_field IS NOT NULL;
    -- Build one CTE per source querying dfv.{source} directly
    FOR v_src IN
        SELECT * FROM dataflow.stack_sources WHERE stack_name = p_stack_name ORDER BY seq, id
    LOOP
        v_select := format('SELECT %L AS _source, id AS _id', v_src.source_name);
        FOR v_field IN SELECT * FROM jsonb_array_elements(v_stack.fields)
        LOOP
            v_col := v_field->>'name';
            IF v_has_bal AND v_col = v_stack.amount_field THEN
                -- Use per-source amount_field with sign applied
                IF v_src.amount_field IS NULL THEN
                    v_select := v_select || format(', NULL::%s AS %I', v_field->>'type', v_col);
                ELSE
                    v_select := v_select || format(', %I * %s AS %I', v_src.amount_field, v_src.amount_sign, v_col);
                END IF;
            ELSIF v_has_bal AND v_col = v_stack.date_field THEN
                -- Use per-source date_field
                IF v_src.date_field IS NULL THEN
                    v_select := v_select || format(', NULL::date AS %I', v_col);
                ELSE
                    v_select := v_select || format(', %I AS %I', v_src.date_field, v_col);
                END IF;
            ELSE
                -- Other canonical fields: use field_map or same name, NULL if column doesn't exist
                v_src_field := COALESCE(v_src.field_map->>v_col, v_col);
                IF EXISTS (
                    SELECT 1 FROM information_schema.columns
                    WHERE table_schema = 'dfv'
                      AND table_name = v_src.source_name
                      AND column_name = v_src_field
                ) THEN
                    v_select := v_select || format(', %I AS %I', v_src_field, v_col);
                ELSE
                    v_select := v_select || format(', NULL::text AS %I', v_col);
                END IF;
            END IF;
        END LOOP;
        v_select := v_select || format(' FROM dfv.%I', v_src.source_name);
        v_ctes      := v_ctes      || format('%I AS (%s)', v_src.source_name, v_select);
        v_cte_names := v_cte_names || quote_ident(v_src.source_name);
        -- Accumulate carried-forward source balance column and total offset
        IF v_has_bal THEN
            IF v_src_bal_cols IS NOT NULL THEN v_src_bal_cols := v_src_bal_cols || ', '; END IF;
            v_src_bal_cols := COALESCE(v_src_bal_cols, '') || format(
                'SUM(CASE WHEN _source = %L THEN %I END) OVER (ORDER BY %I ASC, _id ASC) + %s AS %I',
                v_src.source_name, v_stack.amount_field, v_stack.date_field,
                v_src.balance_offset, v_src.source_name || '_balance'
            );
            v_total_offset := v_total_offset + v_src.balance_offset;
        END IF;
    END LOOP;
    IF array_length(v_ctes, 1) IS NULL THEN
        RETURN json_build_object('success', false, 'error', 'Stack has no sources');
    END IF;
    v_view := 'dfv.' || quote_ident(p_stack_name);
    v_canon_cols := (
        SELECT string_agg(quote_ident(f->>'name'), ', ')
        FROM jsonb_array_elements(v_stack.fields) f
    );
    IF v_has_bal THEN
        v_sql := format(
            'CREATE VIEW %s AS '
            'WITH %s, _stacked AS (SELECT * FROM %s) '
            'SELECT _source, _id, %s, '
            '%s, '
            'SUM(%I) OVER (ORDER BY %I ASC, _id ASC) + %s AS net_balance '
            'FROM _stacked ORDER BY %I DESC, _id DESC',
            v_view,
            array_to_string(v_ctes, ', '),
            array_to_string(v_cte_names, ' UNION ALL SELECT * FROM '),
            v_canon_cols,
            v_src_bal_cols,
            v_stack.amount_field,
            v_stack.date_field,
            v_total_offset,
            v_stack.date_field
        );
    ELSE
        v_sql := format(
            'CREATE VIEW %s AS '
            'WITH %s, _stacked AS (SELECT * FROM %s) '
            'SELECT _source, _id, %s FROM _stacked',
            v_view,
            array_to_string(v_ctes, ', '),
            array_to_string(v_cte_names, ' UNION ALL SELECT * FROM '),
            v_canon_cols
        );
    END IF;
    IF NOT p_dry_run THEN
        CREATE SCHEMA IF NOT EXISTS dfv;
        EXECUTE format('DROP VIEW IF EXISTS %s CASCADE', v_view);
        EXECUTE v_sql;
        -- Detect stacks whose views were dropped by CASCADE and mark them stale
        SELECT array_agg(s.name) INTO v_cascade_stale
        FROM dataflow.stacks s
        WHERE s.name != p_stack_name
          AND s.view_generated_at IS NOT NULL
          AND NOT EXISTS (
              SELECT 1 FROM pg_views v
              WHERE v.schemaname = 'dfv' AND v.viewname = s.name
          );
        UPDATE dataflow.stacks SET view_generated_at = NULL
        WHERE name = ANY(v_cascade_stale);
    END IF;
    RETURN json_build_object(
        'success', true,
        'view', v_view,
        'sql', v_sql,
        'cascade_stale', COALESCE(to_json(v_cascade_stale), '[]'::json)
    );
 END;
 $$ LANGUAGE plpgsql;
 ------------------------------------------------------
 -- Function: get_stack_balance
 -- Returns the current running balance (last row of the generated view)
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION get_stack_balance(p_stack_name TEXT)
 RETURNS JSON AS $$
 DECLARE
    v_stack     dataflow.stacks%ROWTYPE;
    v_balance   NUMERIC;
    v_view      TEXT;
    v_sql       TEXT;
 BEGIN
    SELECT * INTO v_stack FROM dataflow.stacks WHERE name = p_stack_name;
    IF NOT FOUND THEN
        RETURN json_build_object('success', false, 'error', 'Stack not found');
    END IF;
    IF v_stack.amount_field IS NULL OR v_stack.date_field IS NULL THEN
        RETURN json_build_object('success', false, 'error', 'amount_field and date_field must be set');
    END IF;
    v_view := 'dfv.' || quote_ident(p_stack_name);
    BEGIN
        v_sql := format(
            'SELECT net_balance FROM %s ORDER BY %I DESC, _id DESC LIMIT 1',
            v_view, v_stack.date_field
        );
        EXECUTE v_sql INTO v_balance;
    EXCEPTION WHEN undefined_table THEN
        RETURN json_build_object('success', false, 'error', 'View not generated yet — click Generate first');
    END;
    RETURN json_build_object('success', true, 'balance', v_balance);
 END;
 $$ LANGUAGE plpgsql STABLE;
 COMMENT ON FUNCTION generate_stack_view(TEXT, BOOLEAN) IS 'Generate a UNION ALL view in dfv schema combining multiple sources with optional running balance; p_dry_run=true returns SQL without executing';
 COMMENT ON FUNCTION calibrate_balance IS 'Given a known good balance at a date, compute the offset to add to balance_offset';
 COMMENT ON FUNCTION get_stack_balance IS 'Return the current running balance (last row) from the generated dfv view';
 ------------------------------------------------------
 -- Function: reorder_stack_sources
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION reorder_stack_sources(p_stack_name TEXT, p_source_names TEXT[])
 RETURNS VOID AS $$
 DECLARE
    i INTEGER;
 BEGIN
    FOR i IN 1..array_length(p_source_names, 1) LOOP
        UPDATE dataflow.stack_sources
        SET seq = i
        WHERE stack_name = p_stack_name AND source_name = p_source_names[i];
    END LOOP;
 END;
 $$ LANGUAGE plpgsql;
--- a/database/queries/status.sql
+++ b/database/queries/status.sql
@ -0,0 +1,86 @@
 --
 -- Status tracking: view_generated_at on sources and stacks
 -- Cleared by triggers when definitions change; set by API when views are generated.
 --
 SET search_path TO dataflow, public;
 -- Add view_generated_at columns
 ALTER TABLE dataflow.sources ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAMPTZ;
 ALTER TABLE dataflow.stacks  ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAMPTZ;
 ------------------------------------------------------
 -- Trigger: clear source view_generated_at when config (field definitions) changes
 -- Rules and mappings affect transformed data, not view structure — no trigger needed there
 ------------------------------------------------------
 DROP TRIGGER IF EXISTS trg_rules_changed ON dataflow.rules;
 DROP TRIGGER IF EXISTS trg_mappings_changed ON dataflow.mappings;
 DROP FUNCTION IF EXISTS dataflow.rules_changed();
 DROP FUNCTION IF EXISTS dataflow.mappings_changed();
 CREATE OR REPLACE FUNCTION dataflow.source_config_changed()
 RETURNS TRIGGER AS $$
 BEGIN
    IF NEW.config IS DISTINCT FROM OLD.config THEN
        NEW.view_generated_at := NULL;
    END IF;
    RETURN NEW;
 END;
 $$ LANGUAGE plpgsql;
 DROP TRIGGER IF EXISTS trg_source_config_changed ON dataflow.sources;
 CREATE TRIGGER trg_source_config_changed
 BEFORE UPDATE ON dataflow.sources
 FOR EACH ROW EXECUTE FUNCTION dataflow.source_config_changed();
 ------------------------------------------------------
 -- Trigger: clear stack view_generated_at when sources change
 -- On UPDATE, skip if all view-relevant columns are unchanged (upsert no-ops should not mark stale)
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION dataflow.stack_sources_changed()
 RETURNS TRIGGER AS $$
 BEGIN
    IF TG_OP = 'UPDATE' THEN
        IF NEW.field_map        IS NOT DISTINCT FROM OLD.field_map        AND
           NEW.amount_sign      IS NOT DISTINCT FROM OLD.amount_sign      AND
           NEW.balance_offset   IS NOT DISTINCT FROM OLD.balance_offset   AND
           NEW.amount_field     IS NOT DISTINCT FROM OLD.amount_field     AND
           NEW.date_field       IS NOT DISTINCT FROM OLD.date_field       AND
           NEW.seq              IS NOT DISTINCT FROM OLD.seq THEN
            RETURN NULL;
        END IF;
    END IF;
    UPDATE dataflow.stacks SET view_generated_at = NULL
    WHERE name = COALESCE(NEW.stack_name, OLD.stack_name);
    RETURN NULL;
 END;
 $$ LANGUAGE plpgsql;
 DROP TRIGGER IF EXISTS trg_stack_sources_changed ON dataflow.stack_sources;
 CREATE TRIGGER trg_stack_sources_changed
 AFTER INSERT OR UPDATE OR DELETE ON dataflow.stack_sources
 FOR EACH ROW EXECUTE FUNCTION dataflow.stack_sources_changed();
 ------------------------------------------------------
 -- Function: get_status
 -- Returns sources and stacks whose view is stale (null or never generated)
 ------------------------------------------------------
 CREATE OR REPLACE FUNCTION get_status()
 RETURNS JSON AS $$
 DECLARE
    v_sources JSON;
    v_stacks  JSON;
 BEGIN
    SELECT COALESCE(json_agg(json_build_object('name', name, 'view_generated_at', view_generated_at) ORDER BY name), '[]'::json)
    INTO v_sources
    FROM dataflow.sources
    WHERE view_generated_at IS NULL;
    SELECT COALESCE(json_agg(json_build_object('name', name, 'view_generated_at', view_generated_at) ORDER BY name), '[]'::json)
    INTO v_stacks
    FROM dataflow.stacks
    WHERE view_generated_at IS NULL;
    RETURN json_build_object('stale_sources', v_sources, 'stale_stacks', v_stacks);
 END;
 $$ LANGUAGE plpgsql STABLE;
--- a/database/schema.sql
+++ b/database/schema.sql
@ -37,26 +37,27 @@ CREATE TABLE records (
    -- Data
    data JSONB NOT NULL,                    -- Original imported data
    constraint_key JSONB,                   -- Fields that uniquely identify this record (set on import)
-    transformed JSONB,                      -- Data after transformations applied
+    transformed JSONB,                      -- Rule/mapping output fields only (delta, not raw data)
    overrides JSONB,                        -- Manual user overrides (highest precedence)
    -- Metadata
-    import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE,  -- Which import batch this came from
+    import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE,
    imported_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
-    transformed_at TIMESTAMPTZ,
+    transformed_at TIMESTAMPTZ
 );
 COMMENT ON TABLE records IS 'Imported records with raw and transformed data';
-COMMENT ON COLUMN records.data IS 'Original data as imported';
+COMMENT ON COLUMN records.data IS 'Original data as imported — never mutated after import';
 COMMENT ON COLUMN records.constraint_key IS 'JSONB object of constraint field values — uniquely identifies this record within its source';
-COMMENT ON COLUMN records.transformed IS 'Data after applying transformation rules';
+COMMENT ON COLUMN records.transformed IS 'Rule/mapping output fields only (delta); merge as data || transformed || overrides for final values';
 COMMENT ON COLUMN records.overrides IS 'Manual user overrides; highest precedence in data || transformed || overrides merge';
 -- Indexes
 CREATE INDEX idx_records_source ON records(source_name);
 CREATE INDEX idx_records_constraint ON records USING gin(constraint_key);
 CREATE INDEX idx_records_data ON records USING gin(data);
 CREATE INDEX idx_records_transformed ON records USING gin(transformed);
 CREATE INDEX idx_records_overrides ON records USING gin(overrides) WHERE overrides IS NOT NULL;
 ------------------------------------------------------
 -- Table: rules
--- a/docs/perspective-pivot.md
+++ b/docs/perspective-pivot.md
@ -1,27 +1,22 @@
 # Perspective Pivot — Technical Reference
-Version tested: `@perspective-dev` v4.4.0 (client, viewer, viewer-datagrid, viewer-d3fc), loaded from CDN.
+Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply.
 This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement.
 ---
-## Loading from CDN
+## Loading via npm
 ```js
-const [{ default: perspective }] = await Promise.all([
+import perspective from '@perspective-dev/client/inline'
-  import('https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
+import '@perspective-dev/viewer/inline'
-  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
+import '@perspective-dev/viewer-datagrid'
-  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
+import '@perspective-dev/viewer-d3fc'
-  import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
+import '@perspective-dev/viewer/themes'
 ])
 ```
-Stylesheet:
+The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`.
 ```html
 <link rel="stylesheet" crossorigin="anonymous"
  href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css" />
 ```
 ---
--- a/migrate/dataflow.pg.sql
+++ b/migrate/dataflow.pg.sql
@ -0,0 +1 @@
 select id, source, constrain_key, data from dataflow.records
--- a/migrate/reimport_dcard_from_tps.sh
+++ b/migrate/reimport_dcard_from_tps.sh
@ -0,0 +1,62 @@
 #!/bin/bash
 # Reimport dcard records from ubm.tps.trans into dataflow.records
 #
 # Step 1: exports raw rec JSON from ubm
 # Step 2: wipes existing dcard data in dataflow and reloads from the export
 #
 # Usage: bash migrate/reimport_dcard_from_tps.sh
 set -e
 EXPORT_FILE="/tmp/tps_dcard_rec.csv"
 echo "==> Exporting dcard from ubm.tps.trans..."
 psql -U ptrowbridge -d ubm -p 54329 -h hptrow.me -c "\COPY (SELECT rec FROM tps.trans WHERE srce = 'dcard' ORDER BY id) TO '${EXPORT_FILE}' CSV"
 echo "    Exported $(wc -l < ${EXPORT_FILE}) rows"
 echo "==> Reimporting into dataflow.records..."
 $PG -d dataflow <<SQL
 BEGIN;
 -- Wipe existing dcard records (FK cascade deletes records too)
 DELETE FROM dataflow.import_log WHERE source_name = 'dcard';
 -- Staging table for the exported rec JSON
 CREATE TEMP TABLE _dcard_import (rec jsonb);
 \COPY _dcard_import FROM '${EXPORT_FILE}' CSV
 -- New import_log entry
 INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate)
 VALUES ('dcard', 0, 0);
 -- Insert records; constraint_key matches source constraint_fields:
 --   {"Trans. Date","Post Date",Description}
 WITH new_import AS (
    SELECT id AS import_id FROM dataflow.import_log
    WHERE source_name = 'dcard'
    ORDER BY id DESC LIMIT 1
 ),
 inserted AS (
    INSERT INTO dataflow.records (source_name, data, transformed, constraint_key, import_id)
    SELECT
        'dcard',
        s.rec,
        NULL,
        jsonb_build_object(
            'Trans. Date', s.rec->>'Trans. Date',
            'Post Date',   s.rec->>'Post Date',
            'Description', s.rec->>'Description'
        ),
        i.import_id
    FROM _dcard_import s, new_import i
    RETURNING id
 )
 UPDATE dataflow.import_log
 SET records_imported = (SELECT COUNT(*) FROM inserted)
 WHERE source_name = 'dcard'
  AND id = (SELECT id FROM dataflow.import_log WHERE source_name = 'dcard' ORDER BY id DESC LIMIT 1);
 COMMIT;
 SELECT records_imported FROM dataflow.import_log WHERE source_name = 'dcard' ORDER BY id DESC LIMIT 1;
 SQL
 echo "==> Done. Run transformations to repopulate the transformed column."
--- a/package.json
+++ b/package.json
@ -18,11 +18,11 @@
  "license": "MIT",
  "dependencies": {
    "bcrypt": "^6.0.0",
-    "csv-parse": "^5.5.2",
+    "csv-parse": "^6.2.1",
-    "dotenv": "^16.3.1",
+    "dotenv": "^17.4.2",
-    "express": "^4.18.2",
+    "express": "^5.2.1",
-    "multer": "^1.4.5-lts.1",
+    "multer": "^2.1.1",
-    "pg": "^8.11.3"
+    "pg": "^8.21.0"
  },
  "devDependencies": {
    "nodemon": "^3.0.1"
--- a/ui/README.md
+++ b/ui/README.md
@ -1,16 +1,25 @@
-# React + Vite
+# Dataflow UI
-This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
+React + Vite + Tailwind CSS frontend for Dataflow.
-Currently, two official plugins are available:
+## Development
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
+```bash
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
+npm install
 npm run dev      # dev server on :5173, proxies /api to :3020
 ```
-## React Compiler
+## Build
-The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
+```bash
 npm run build    # outputs to ../public/
 ```
-## Expanding the ESLint configuration
+The Express server serves `../public/` as static files — no separate web server needed in production.
-If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.
+## Key packages
 - `react` / `react-router-dom` — SPA routing
 - `@perspective-dev/client`, `viewer`, `viewer-datagrid`, `viewer-d3fc` — pivot table (npm, inline WASM builds)
 - `tailwindcss` — utility CSS
 - `sql-formatter` — SQL display formatting
--- a/ui/package.json
+++ b/ui/package.json
@ -10,21 +10,26 @@
    "preview": "vite preview"
  },
  "dependencies": {
-    "react": "^19.2.4",
+    "@perspective-dev/client": "^4.5.1",
-    "react-dom": "^19.2.4",
+    "@perspective-dev/viewer": "^4.5.1",
-    "react-router-dom": "^7.13.2"
+    "@perspective-dev/viewer-d3fc": "^4.4.1",
    "@perspective-dev/viewer-datagrid": "^4.5.1",
    "react": "^19.2.7",
    "react-dom": "^19.2.7",
    "react-router-dom": "^7.17.0",
    "sql-formatter": "^15.8.1"
  },
  "devDependencies": {
    "@eslint/js": "^9.39.4",
-    "@tailwindcss/vite": "^4.2.2",
+    "@tailwindcss/vite": "^4.3.1",
-    "@types/react": "^19.2.14",
+    "@types/react": "^19.2.17",
    "@types/react-dom": "^19.2.3",
-    "@vitejs/plugin-react": "^6.0.1",
+    "@vitejs/plugin-react": "^6.0.2",
    "eslint": "^9.39.4",
-    "eslint-plugin-react-hooks": "^7.0.1",
+    "eslint-plugin-react-hooks": "^7.1.1",
    "eslint-plugin-react-refresh": "^0.5.2",
-    "globals": "^17.4.0",
+    "globals": "^17.6.0",
-    "tailwindcss": "^4.2.2",
+    "tailwindcss": "^4.3.1",
-    "vite": "^8.0.1"
+    "vite": "^8.0.16"
  }
 }
--- a/ui/src/App.jsx
+++ b/ui/src/App.jsx
@ -1,6 +1,8 @@
 import { useState, useEffect } from 'react'
-import { BrowserRouter, Routes, Route, NavLink, Navigate } from 'react-router-dom'
+import { BrowserRouter, Routes, Route, Navigate } from 'react-router-dom'
 import { api, setCredentials, clearCredentials } from './api'
 import StatusBar from './components/StatusBar.jsx'
 import Sidebar from './components/Sidebar.jsx'
 import Login from './pages/Login'
 import Sources from './pages/Sources'
 import Import from './pages/Import'
@ -10,35 +12,32 @@ import Records from './pages/Records'
 import Log from './pages/Log'
 import Pivot from './pages/Pivot'
 import Remap from './pages/Remap'
-
+import Stacks from './pages/Stacks'
 const NAV = [
  { to: '/sources', label: 'Sources' },
  { to: '/import', label: 'Import' },
  { to: '/rules', label: 'Rules' },
  { to: '/mappings', label: 'Mappings' },
  { to: '/remap', label: 'Remap' },
  { to: '/records', label: 'Records' },
  { to: '/pivot', label: 'Pivot' },
  { to: '/log', label: 'Log' },
 ]
 export default function App() {
  const [authed, setAuthed] = useState(false)
  const [loginUser, setLoginUser] = useState('')
  const [sources, setSources] = useState([])
  const [stacks, setStacks] = useState([])
  const [source, setSource] = useState(() => localStorage.getItem('selectedSource') || '')
-  const [sidebarOpen, setSidebarOpen] = useState(false)
+  const [selectedStack, setSelectedStack] = useState(null)
  const [sidebarExpanded, setSidebarExpanded] = useState(() => localStorage.getItem('df_sidebar') !== 'collapsed')
  // Sets of names whose dfv view is out of sync with current definitions
  const [staleSources, setStaleSources] = useState(new Set())
  const [staleStacks, setStaleStacks] = useState(new Set())
  const [reprocessSources, setReprocessSources] = useState(new Set())
  const [generating, setGenerating] = useState({}) // { 'source:name': true }
  async function handleLogin(user, pass) {
    setCredentials(user, pass)
-    await api.getSources().then(s => {
+    const s = await api.getSources()
    sessionStorage.setItem('df_user', user)
    sessionStorage.setItem('df_pass', pass)
    setSources(s)
    if (!source && s.length > 0) setSource(s[0].name)
    setAuthed(true)
    setLoginUser(user)
-    })
+    api.getStacks().then(setStacks).catch(() => {})
  }
  function handleLogout() {
@ -48,6 +47,63 @@ export default function App() {
    setAuthed(false)
    setLoginUser('')
    setSources([])
    setStacks([])
    setSelectedStack(null)
    setStaleSources(new Set())
    setStaleStacks(new Set())
    setReprocessSources(new Set())
  }
  function refreshStacks() {
    api.getStacks().then(setStacks).catch(() => {})
  }
  // Load initial stale state from DB once on login
  useEffect(() => {
    if (!authed) return
    api.getStatus().then(s => {
      setStaleSources(new Set((s.stale_sources || []).map(x => x.name)))
      setStaleStacks(new Set((s.stale_stacks || []).map(x => x.name)))
    }).catch(() => {})
  }, [authed])
  function markSourceStale(name) {
    setStaleSources(prev => new Set([...prev, name]))
  }
  function markNeedsReprocess(name) {
    setReprocessSources(prev => new Set([...prev, name]))
  }
  async function handleReprocessSource(name) {
    setGenerating(g => ({ ...g, [`rp:${name}`]: true }))
    try {
      await api.reprocess(name)
      setReprocessSources(prev => { const n = new Set(prev); n.delete(name); return n })
    } catch (e) { alert(e.message) }
    finally { setGenerating(g => { const n = { ...g }; delete n[`rp:${name}`]; return n }) }
  }
  function markStackStale(name) {
    setStaleStacks(prev => new Set([...prev, name]))
  }
  function clearStackStale(name) {
    setStaleStacks(prev => { const n = new Set(prev); n.delete(name); return n })
  }
  async function handleGenerateSource(name) {
    setGenerating(g => ({ ...g, [`src:${name}`]: true }))
    try {
      await api.generateView(name)
      setStaleSources(prev => { const n = new Set(prev); n.delete(name); return n })
    } catch (e) { alert(e.message) }
    finally { setGenerating(g => { const n = { ...g }; delete n[`src:${name}`]; return n }) }
  }
  async function handleGenerateStack(name) {
    setGenerating(g => ({ ...g, [`stk:${name}`]: true }))
    try {
      await api.generateStackView(name)
      setStaleStacks(prev => { const n = new Set(prev); n.delete(name); return n })
    } catch (e) { alert(e.message) }
    finally { setGenerating(g => { const n = { ...g }; delete n[`stk:${name}`]; return n }) }
  }
  // On mount, restore session if credentials are saved
@ -61,94 +117,89 @@ export default function App() {
    if (source) localStorage.setItem('selectedSource', source)
  }, [source])
  useEffect(() => {
    localStorage.setItem('df_sidebar', sidebarExpanded ? 'expanded' : 'collapsed')
  }, [sidebarExpanded])
  if (!authed) return <Login onLogin={handleLogin} />
  const sidebar = (
    <div className="flex flex-col h-full">
      {/* Header */}
      <div className="px-4 py-3 border-b border-gray-200">
        <div className="flex items-center justify-between">
          <span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
          <button onClick={() => setSidebarOpen(false)} className="md:hidden text-gray-400 hover:text-gray-600 leading-none" title="Close">✕</button>
        </div>
        <div className="flex items-center justify-between mt-1">
          <span className="text-xs text-gray-400">{loginUser}</span>
          <button onClick={handleLogout} className="text-xs text-gray-400 hover:text-red-500" title="Sign out">Sign out</button>
        </div>
      </div>
      {/* Source selector */}
      <div className="px-3 py-3 border-b border-gray-200">
        <div className="flex items-center justify-between mb-1">
          <label className="text-xs text-gray-500">Source</label>
          <NavLink to="/sources?new=1" className="text-xs text-blue-400 hover:text-blue-600 leading-none" title="New source" onClick={() => setSidebarOpen(false)}>+</NavLink>
        </div>
        <select
          className="w-full text-sm border border-gray-200 rounded px-2 py-1 bg-white focus:outline-none focus:border-blue-400"
          value={source}
          onChange={e => setSource(e.target.value)}
        >
          {sources.length === 0 && <option value="">—</option>}
          {sources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
        </select>
      </div>
      {/* Nav */}
      <nav className="flex-1 py-2">
        {NAV.map(({ to, label }) => (
          <NavLink
            key={to}
            to={to}
            onClick={() => setSidebarOpen(false)}
            className={({ isActive }) =>
              `block px-4 py-2 text-sm ${isActive
                ? 'bg-blue-50 text-blue-700 font-medium'
                : 'text-gray-600 hover:bg-gray-50'}`
            }
          >
            {label}
          </NavLink>
        ))}
      </nav>
    </div>
  )
  return (
    <BrowserRouter>
-      <div className="flex h-screen bg-gray-50">
+      <div className="flex h-screen">
-        {/* Mobile overlay */}
+        <Sidebar
-        {sidebarOpen && (
+          expanded={sidebarExpanded}
-          <div className="fixed inset-0 z-20 bg-black/30 md:hidden" onClick={() => setSidebarOpen(false)} />
+          setExpanded={setSidebarExpanded}
-        )}
+          loginUser={loginUser}
-
+          onLogout={handleLogout}
-        {/* Sidebar — fixed on mobile, static on desktop */}
+        />
        <div className={`
          fixed inset-y-0 left-0 z-30 w-44 bg-white border-r border-gray-200 transform transition-transform duration-200
          md:static md:translate-x-0 md:z-auto md:transition-none
          ${sidebarOpen ? 'translate-x-0' : '-translate-x-full'}
        `}>
          {sidebar}
        </div>
        {/* Main */}
-        <div className="flex-1 overflow-auto flex flex-col min-w-0">
+        <div className="flex-1 overflow-hidden flex flex-col min-w-0">
-          {/* Mobile top bar */}
+          <StatusBar
-          <div className="md:hidden flex items-center px-3 py-2 bg-white border-b border-gray-200">
+            sources={sources} source={source} setSource={setSource}
-            <button onClick={() => setSidebarOpen(true)} className="text-gray-500 hover:text-gray-700 mr-3 text-lg leading-none">☰</button>
+            stacks={stacks} selectedStack={selectedStack} setSelectedStack={setSelectedStack}
-            <span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
+          />
          {(staleSources.size > 0 || staleStacks.size > 0) && (
            <div className="bg-amber-50 border-b border-amber-200 px-4 py-1.5 text-xs text-amber-800 flex flex-wrap items-center gap-x-3 gap-y-1">
              <span className="font-medium">View out of sync:</span>
              {[...staleSources].map(name => (
                <span key={name} className="flex items-center gap-1">
                  {name}
                  <button
                    onClick={() => handleGenerateSource(name)}
                    disabled={generating[`src:${name}`]}
                    className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
                  >
                    {generating[`src:${name}`] ? '…' : 'Generate'}
                  </button>
                </span>
              ))}
              {staleSources.size > 0 && staleStacks.size > 0 && <span className="text-amber-400">|</span>}
              {[...staleStacks].map(name => (
                <span key={name} className="flex items-center gap-1">
                  stack: {name}
                  <button
                    onClick={() => handleGenerateStack(name)}
                    disabled={generating[`stk:${name}`]}
                    className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
                  >
                    {generating[`stk:${name}`] ? '…' : 'Generate'}
                  </button>
                </span>
              ))}
            </div>
          )}
          {reprocessSources.size > 0 && (
            <div className="bg-blue-50 border-b border-blue-200 px-4 py-1.5 text-xs text-blue-800 flex flex-wrap items-center gap-x-3 gap-y-1">
              <span className="font-medium">Mappings updated:</span>
              {[...reprocessSources].map(name => (
                <span key={name} className="flex items-center gap-1">
                  {name}
                  <button
                    onClick={() => handleReprocessSource(name)}
                    disabled={generating[`rp:${name}`]}
                    className="px-1.5 py-0.5 rounded bg-blue-200 hover:bg-blue-300 disabled:opacity-50 font-medium"
                  >
                    {generating[`rp:${name}`] ? '…' : 'Reprocess'}
                  </button>
                </span>
              ))}
            </div>
          )}
          <div className="flex-1 overflow-auto">
            <Routes>
              <Route path="/" element={<Navigate to="/sources" replace />} />
              <Route path="/sources" element={<Sources source={source} sources={sources} setSources={setSources} setSource={setSource} />} />
              <Route path="/import" element={<Import source={source} />} />
-              <Route path="/rules" element={<Rules source={source} />} />
+              <Route path="/rules" element={<Rules source={source} onStale={markSourceStale} />} />
-              <Route path="/mappings" element={<Mappings source={source} />} />
+              <Route path="/mappings" element={<Mappings source={source} onNeedsReprocess={markNeedsReprocess} />} />
              <Route path="/remap" element={<Remap />} />
              <Route path="/records" element={<Records source={source} />} />
-              <Route path="/pivot" element={<Pivot source={source} />} />
+              <Route path="/pivot" element={<Pivot source={source} selectedStack={selectedStack} setSelectedStack={setSelectedStack} />} />
              <Route path="/stacks" element={<Stacks sources={sources} onStackStale={markStackStale} onStackViewGenerated={clearStackStale} onStacksChange={refreshStacks} />} />
              <Route path="/log" element={<Log />} />
            </Routes>
          </div>
--- a/ui/src/api.js
+++ b/ui/src/api.js
@ -108,12 +108,40 @@ export const api = {
  getMappingsByOutputField: (col, val) => request('GET', `/mappings/outputs/${encodeURIComponent(col)}/${encodeURIComponent(val)}`),
  remapOutputField: (col, from_val, to_val) => request('POST', '/mappings/remap-field', { col, from_val, to_val }),
-  // Pivot layouts
+  // Pivot layouts (sources)
  getPivotLayouts: (source) => request('GET', `/sources/${source}/layouts`),
  savePivotLayout: (source, layout_name, config) => request('POST', `/sources/${source}/layouts`, { layout_name, config }),
  deletePivotLayout: (source, id) => request('DELETE', `/sources/${source}/layouts/${id}`),
  // Pivot layouts (stacks)
  getStackPivotLayouts: (name) => request('GET', `/stacks/${name}/layouts`),
  saveStackPivotLayout: (name, layout_name, config) => request('POST', `/stacks/${name}/layouts`, { layout_name, config }),
  deleteStackPivotLayout: (name, id) => request('DELETE', `/stacks/${name}/layouts/${id}`),
  // Stacks
  getStacks: () => request('GET', '/stacks'),
  getStack: (name) => request('GET', `/stacks/${name}`),
  createStack: (body) => request('POST', '/stacks', body),
  updateStack: (name, body) => request('PUT', `/stacks/${name}`, body),
  deleteStack: (name) => request('DELETE', `/stacks/${name}`),
  upsertStackSource: (name, source, body) => request('PUT', `/stacks/${name}/sources/${source}`, body),
  reorderStackSources: (name, source_names) => request('PUT', `/stacks/${name}/sources/reorder`, { source_names }),
  removeStackSource: (name, source) => request('DELETE', `/stacks/${name}/sources/${source}`),
  previewStackSql: (name) => request('GET', `/stacks/${name}/view-sql`),
  generateStackView: (name) => request('POST', `/stacks/${name}/view`),
  execStackSql: (name, sql) => request('POST', `/stacks/${name}/exec-sql`, { sql }),
  getStackBalance: (name) => request('GET', `/stacks/${name}/balance`),
  calibrateBalance: (name, source, body) => request('POST', `/stacks/${name}/calibrate`, { ...body, source_name: source || null }),
  // Status
  getStatus: () => request('GET', '/status'),
  // Records
  getRecords: (source, limit = 100, offset = 0) =>
    request('GET', `/records/source/${source}?limit=${limit}&offset=${offset}`),
  getRecord: (id) => request('GET', `/records/${id}`),
  getOverrideKeys: (source) => request('GET', `/sources/${source}/override-keys`),
  setBulkRecordOverrides: (source, recordIds, overrides) => request('POST', `/records/bulk-overrides`, { source_name: source, record_ids: recordIds, overrides }),
  setRecordOverrides: (id, overrides) => request('PUT', `/records/${id}/overrides`, { overrides }),
  clearRecordOverrides: (id) => request('DELETE', `/records/${id}/overrides`),
 }
--- a/ui/src/components/Sidebar.jsx
+++ b/ui/src/components/Sidebar.jsx
@ -0,0 +1,183 @@
 import { NavLink } from 'react-router-dom'
 const NAV = [
  {
    to: '/sources',
    label: 'Sources',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <ellipse cx="10" cy="5.5" rx="7" ry="2.5"/>
        <path d="M3 5.5v9c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5v-9"/>
        <path d="M3 10.5c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5"/>
      </svg>
    ),
  },
  {
    to: '/import',
    label: 'Import',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <line x1="10" y1="3" x2="10" y2="14"/>
        <polyline points="6,10 10,14 14,10"/>
        <line x1="3" y1="18" x2="17" y2="18"/>
      </svg>
    ),
  },
  {
    to: '/rules',
    label: 'Rules',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <polyline points="6,7 2,10 6,13"/>
        <polyline points="14,7 18,10 14,13"/>
        <line x1="12" y1="4" x2="8" y2="16"/>
      </svg>
    ),
  },
  {
    to: '/mappings',
    label: 'Mappings',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <line x1="2" y1="7" x2="12" y2="7"/>
        <polyline points="9,4 12,7 9,10"/>
        <line x1="8" y1="13" x2="18" y2="13"/>
        <polyline points="11,10 14,13 11,16"/>
      </svg>
    ),
  },
  {
    to: '/remap',
    label: 'Remap',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <polyline points="2,8 2,4 6,4"/>
        <path d="M2 4a8 8 0 0 1 14 2"/>
        <polyline points="18,12 18,16 14,16"/>
        <path d="M18 16a8 8 0 0 1-14-2"/>
      </svg>
    ),
  },
  {
    to: '/records',
    label: 'Records',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <rect x="2" y="3" width="16" height="14" rx="1.5"/>
        <line x1="2" y1="8" x2="18" y2="8"/>
        <line x1="7" y1="8" x2="7" y2="17"/>
      </svg>
    ),
  },
  {
    to: '/pivot',
    label: 'Pivot',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <rect x="2" y="2" width="7" height="7" rx="1"/>
        <rect x="11" y="2" width="7" height="7" rx="1"/>
        <rect x="2" y="11" width="7" height="7" rx="1"/>
        <rect x="11" y="11" width="7" height="7" rx="1"/>
      </svg>
    ),
  },
  {
    to: '/stacks',
    label: 'Stacks',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <polygon points="10,2 18,6 10,10 2,6"/>
        <polyline points="2,10 10,14 18,10"/>
        <polyline points="2,14 10,18 18,14"/>
      </svg>
    ),
  },
  {
    to: '/log',
    label: 'Log',
    icon: (
      <svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
        <circle cx="10" cy="10" r="8"/>
        <polyline points="10,5 10,10 14,12"/>
      </svg>
    ),
  },
 ]
 export default function Sidebar({ expanded, setExpanded, loginUser, onLogout }) {
  return (
    <div
      className="bg-white border-r border-gray-200 flex flex-col shrink-0 overflow-hidden transition-all duration-150"
      style={{ width: expanded ? 200 : 48 }}
    >
      {/* Header */}
      <div className="h-12 flex items-center px-3 border-b border-gray-100 gap-2 shrink-0">
        <button
          onClick={() => setExpanded(e => !e)}
          className="w-8 h-8 flex items-center justify-center rounded hover:bg-gray-100 text-gray-400 shrink-0"
          title="Toggle sidebar"
        >
          <svg width="16" height="16" viewBox="0 0 16 16" fill="none">
            <rect x="1" y="3" width="14" height="1.5" rx="0.75" fill="currentColor"/>
            <rect x="1" y="7.25" width="14" height="1.5" rx="0.75" fill="currentColor"/>
            <rect x="1" y="11.5" width="14" height="1.5" rx="0.75" fill="currentColor"/>
          </svg>
        </button>
        <span
          className="text-xs font-semibold text-gray-600 tracking-wide uppercase whitespace-nowrap transition-opacity duration-100"
          style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none' }}
        >
          Dataflow
        </span>
      </div>
      {/* Nav */}
      <nav className="flex flex-col gap-0.5 p-2 flex-1">
        {NAV.map(({ to, label, icon }) => (
          <NavLink
            key={to}
            to={to}
            title={!expanded ? label : undefined}
            className={({ isActive }) =>
              `flex items-center gap-3 px-2 py-2 rounded w-full transition-colors ${
                isActive
                  ? 'bg-blue-50 text-blue-700'
                  : 'text-gray-500 hover:bg-gray-100 hover:text-gray-800'
              }`
            }
          >
            <span className="shrink-0">{icon}</span>
            <span
              className="text-sm whitespace-nowrap transition-opacity duration-100"
              style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
            >
              {label}
            </span>
          </NavLink>
        ))}
      </nav>
      {/* User / logout */}
      <div className="border-t border-gray-100 px-3 py-2.5 flex items-center gap-2 shrink-0 overflow-hidden">
        <div
          className="w-6 h-6 rounded-full bg-gray-200 text-gray-500 flex items-center justify-center shrink-0 text-xs font-medium"
          title={!expanded ? loginUser : undefined}
        >
          {loginUser ? loginUser[0].toUpperCase() : '?'}
        </div>
        <div
          className="flex-1 flex items-center justify-between min-w-0 transition-opacity duration-100"
          style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
        >
          <span className="text-xs text-gray-400 truncate">{loginUser}</span>
          <button
            onClick={onLogout}
            className="text-xs text-gray-400 hover:text-red-500 ml-2 shrink-0"
          >
            Sign out
          </button>
        </div>
      </div>
    </div>
  )
 }
--- a/ui/src/components/StatusBar.jsx
+++ b/ui/src/components/StatusBar.jsx
@ -0,0 +1,73 @@
 import { NavLink } from 'react-router-dom'
 import useTheme from '../theme.jsx'
 export default function StatusBar({ sources = [], source, setSource, stacks = [], selectedStack, setSelectedStack }) {
  const { dark, setDark } = useTheme()
  return (
    <div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs">
      <span className="text-gray-400">Source</span>
      <select
        value={source || ''}
        onChange={e => setSource(e.target.value)}
        disabled={sources.length === 0}
        className="border border-gray-200 rounded px-2 py-0.5 bg-white focus:outline-none focus:border-blue-400"
      >
        {sources.length === 0
          ? <option value="">— no sources —</option>
          : sources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
      </select>
      <NavLink
        to="/sources?new=1"
        className="text-blue-400 hover:text-blue-600 leading-none"
        title="New source"
      >+</NavLink>
      {stacks.length > 0 && (
        <>
          <span className="text-gray-200">|</span>
          <span className="text-gray-400">Stacks</span>
          {stacks.map(s => (
            <button
              key={s.name}
              onClick={() => setSelectedStack(n => n === s.name ? null : s.name)}
              className={`rounded px-2 py-0.5 border transition-colors ${
                selectedStack === s.name
                  ? 'bg-purple-50 border-purple-300 text-purple-700'
                  : 'bg-white border-gray-200 text-gray-500 hover:border-gray-400'
              }`}
            >
              {s.name}
            </button>
          ))}
        </>
      )}
      <div className="ml-auto">
        <button
          onClick={() => setDark(d => !d)}
          className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100 text-gray-500"
          title={dark ? 'Switch to light mode' : 'Switch to dark mode'}
        >
          {dark ? (
            <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
              <circle cx="12" cy="12" r="4"/>
              <line x1="12" y1="2" x2="12" y2="5"/>
              <line x1="12" y1="19" x2="12" y2="22"/>
              <line x1="4.93" y1="4.93" x2="7.05" y2="7.05"/>
              <line x1="16.95" y1="16.95" x2="19.07" y2="19.07"/>
              <line x1="2" y1="12" x2="5" y2="12"/>
              <line x1="19" y1="12" x2="22" y2="12"/>
              <line x1="4.93" y1="19.07" x2="7.05" y2="16.95"/>
              <line x1="16.95" y1="7.05" x2="19.07" y2="4.93"/>
            </svg>
          ) : (
            <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
              <path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
            </svg>
          )}
        </button>
      </div>
    </div>
  )
 }
--- a/ui/src/index.css
+++ b/ui/src/index.css
@ -1,6 +1,97 @@
@import "tailwindcss";
 :root, .light {
  --bg-primary: #f3f4f6;
  --bg-secondary: #ffffff;
  --bg-tertiary: #f9fafb;
  --text-primary: #1f2937;
  --text-secondary: #374151;
  --text-muted: #9ca3af;
  --border-color: #e5e7eb;
  --border-light: #f3f4f6;
  --accent-bg: #eff6ff;
  --accent-text: #1d4ed8;
 }
 /* Dark palette tuned to Perspective's "Pro Dark" theme:
   bg #242526, tooltip #2a2c2f, gridline #3b3f46, inactive #61656e,
   inactive border #4c505b, active #2770a9, legend #c5c9d0. */
 .dark {
  --bg-primary: #242526;
  --bg-secondary: #2a2c2f;
  --bg-tertiary: #3b3f46;
  --text-primary: #ffffff;
  --text-secondary: #c5c9d0;
  --text-muted: #61656e;
  --border-color: #4c505b;
  --border-light: #3b3f46;
  --accent-bg: rgba(39, 113, 170, 0.32);
  --accent-text: #4778c2;
 }
 body {
  margin: 0;
  font-family: system-ui, -apple-system, sans-serif;
  background-color: var(--bg-primary);
  color: var(--text-primary);
 }
 .dark .bg-white { background-color: var(--bg-secondary); }
 .dark .bg-gray-50 { background-color: var(--bg-tertiary); }
 .dark .bg-gray-100 { background-color: var(--bg-tertiary); }
 .dark .bg-gray-200 { background-color: var(--bg-tertiary); }
 .dark .bg-gray-300 { background-color: var(--bg-tertiary); }
 .dark .text-gray-300 { color: var(--text-muted); }
 .dark .text-gray-400 { color: var(--text-muted); }
 .dark .text-gray-500 { color: var(--text-muted); }
 .dark .text-gray-600 { color: var(--text-secondary); }
 .dark .text-gray-700 { color: var(--text-secondary); }
 .dark .text-gray-800 { color: var(--text-primary); }
 .dark .text-gray-900 { color: var(--text-primary); }
 .dark .bg-blue-50 { background-color: var(--accent-bg); }
 .dark .bg-blue-100 { background-color: var(--accent-bg); }
 .dark .text-blue-400 { color: var(--accent-text); }
 .dark .text-blue-600 { color: var(--accent-text); }
 .dark .text-blue-700 { color: var(--accent-text); }
 .dark .text-blue-800 { color: var(--accent-text); }
 .dark .border-blue-200 { border-color: var(--accent-text); }
 .dark .border-blue-300 { border-color: var(--accent-text); }
 .dark .hover\:bg-blue-50:hover { background-color: var(--accent-bg); }
 /* Status accents — desaturated to sit on Pro Dark's neutral background */
 .dark .bg-green-50 { background-color: #1a3d2c; }
 .dark .text-green-600 { color: #6ee7b7; }
 .dark .text-green-700 { color: #6ee7b7; }
 .dark .text-green-400 { color: #6ee7b7; }
 .dark .bg-amber-50 { background-color: #3a2e14; }
 .dark .text-amber-800 { color: #f5c66f; }
 .dark .border-amber-200 { border-color: #5a4a26; }
 .dark .bg-amber-200 { background-color: #5a4a26; }
 .dark .hover\:bg-amber-300:hover { background-color: #6b5830; }
 .dark .bg-red-50 { background-color: #3d1f1f; }
 .dark .text-red-500 { color: #ff9485; }
 .dark .text-red-700 { color: #ff9485; }
 .dark .border-gray-100 { border-color: var(--border-light); }
 .dark .border-gray-200 { border-color: var(--border-color); }
 .dark .border-gray-300 { border-color: var(--border-color); }
 .dark .border-blue-100 { border-color: var(--border-color); }
 .dark .border-b { border-color: var(--border-color); }
 .dark .border-t { border-color: var(--border-color); }
 .dark .border-r { border-color: var(--border-color); }
 .dark .border-l { border-color: var(--border-color); }
 .dark .hover\:bg-gray-50:hover { background-color: var(--bg-tertiary); }
 .dark .hover\:bg-gray-100:hover { background-color: var(--bg-tertiary); }
 .dark .hover\:bg-gray-200:hover { background-color: var(--bg-tertiary); }
 .dark .hover\:text-gray-500:hover { color: var(--text-secondary); }
 .dark .hover\:text-gray-600:hover { color: var(--text-secondary); }
 .dark .hover\:text-gray-700:hover { color: var(--text-primary); }
 .dark .hover\:text-gray-800:hover { color: var(--text-primary); }
 .dark .hover\:border-gray-300:hover { border-color: var(--border-color); }
 .dark .hover\:border-gray-400:hover { border-color: var(--border-color); }
 .dark .focus\:border-gray-300:focus { border-color: var(--border-color); }
 .dark .focus\:border-blue-400:focus { border-color: var(--accent-text); }
 .dark ::selection { background-color: var(--accent-bg); color: var(--text-primary); }
 .dark input { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
 .dark select { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
 .dark textarea { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
 .dark .bg-transparent { background-color: transparent; }
--- a/ui/src/main.jsx
+++ b/ui/src/main.jsx
@ -1,10 +1,13 @@
 import { StrictMode } from 'react'
 import { createRoot } from 'react-dom/client'
 import { ThemeProvider } from './theme.jsx'
 import './index.css'
 import App from './App.jsx'
 createRoot(document.getElementById('root')).render(
  <StrictMode>
    <ThemeProvider>
      <App />
    </ThemeProvider>
  </StrictMode>,
 )
--- a/ui/src/pages/Mappings.jsx
+++ b/ui/src/pages/Mappings.jsx
@ -4,6 +4,7 @@ import { api, authHeaders } from '../api'
 function AutocompleteInput({ value, onChange, onEnter, suggestions = [], className, placeholder }) {
  const [open, setOpen] = useState(false)
  const [highlighted, setHighlighted] = useState(0)
  const [dropPos, setDropPos] = useState(null)
  const inputRef = useRef()
  const listRef = useRef()
@ -12,6 +13,10 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
    : suggestions
  function openList() {
    if (inputRef.current) {
      const r = inputRef.current.getBoundingClientRect()
      setDropPos({ top: r.bottom + 2, left: r.left, minWidth: r.width })
    }
    setOpen(true)
    setHighlighted(0)
  }
@ -42,7 +47,6 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
    if (e.key === 'Enter') onEnter?.()
  }
  // Scroll highlighted item into view
  useEffect(() => {
    if (!open || !listRef.current) return
    const item = listRef.current.children[highlighted]
@ -60,10 +64,11 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
        onKeyDown={handleKeyDown}
        onBlur={e => { if (!listRef.current?.contains(e.relatedTarget)) setOpen(false) }}
      />
-      {open && filtered.length > 0 && (
+      {open && filtered.length > 0 && dropPos && (
        <div
          ref={listRef}
-          className="absolute z-50 left-0 top-full mt-0.5 bg-white border border-gray-200 rounded shadow-lg max-h-48 overflow-y-auto min-w-full"
+          style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
          className="bg-white border border-gray-200 rounded shadow-lg max-h-48 overflow-y-auto"
        >
          {filtered.map((s, i) => (
            <div
@ -104,7 +109,7 @@ function SortHeader({ col, label, sortBy, onSort, className = '' }) {
  )
 }
-export default function Mappings({ source }) {
+export default function Mappings({ source, onNeedsReprocess }) {
  const [rules, setRules] = useState([])
  const [selectedRule, setSelectedRule] = useState('')
  const [allValues, setAllValues] = useState([])
@ -263,6 +268,7 @@ export default function Mappings({ source }) {
          valueKey(x.extracted_value) === k ? { ...x, is_mapped: true, mapping_id: created.id, output } : x
        ))
      }
      onNeedsReprocess?.(source)
      setDrafts(d => { const n = { ...d }; delete n[k]; return n })
    } catch (err) {
      alert(err.message)
@ -309,6 +315,7 @@ export default function Mappings({ source }) {
        setSaving(s => ({ ...s, [k]: false }))
      }
    }))
    onNeedsReprocess?.(source)
    setSelected(new Set())
    setBulkDraft({})
  }
@ -317,6 +324,7 @@ export default function Mappings({ source }) {
    if (!row.mapping_id) return
    try {
      await api.deleteMapping(row.mapping_id)
      onNeedsReprocess?.(source)
      setAllValues(av => av.map(x =>
        valueKey(x.extracted_value) === valueKey(row.extracted_value)
          ? { ...x, is_mapped: false, mapping_id: null, output: null }
--- a/ui/src/pages/Pivot.jsx
+++ b/ui/src/pages/Pivot.jsx
@ -1,33 +1,19 @@
 import { useEffect, useRef, useState, useCallback } from 'react'
 import { api } from '../api'
 import useTheme from '../theme.jsx'
 import perspective from '@perspective-dev/client/inline'
 import '@perspective-dev/viewer/inline'
 import '@perspective-dev/viewer-datagrid'
 import '@perspective-dev/viewer-d3fc'
 import '@perspective-dev/viewer/themes'
 async function fetchAllRows(source) {
  const res = await api.getViewData(source, 100000, 0)
  return res.rows || []
 }
 let perspectivePromise = null
 function loadPerspective() {
-  if (perspectivePromise) return perspectivePromise
+  return Promise.resolve(perspective)
  perspectivePromise = (async () => {
    if (!document.getElementById('psp-theme')) {
      const link = document.createElement('link')
      link.id = 'psp-theme'
      link.rel = 'stylesheet'
      link.crossOrigin = 'anonymous'
      link.href = 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css'
      document.head.appendChild(link)
    }
    const [{ default: perspective }] = await Promise.all([
      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
      import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
    ])
    return perspective
  })()
  return perspectivePromise
 }
 function formatVal(v, decimals = 2) {
@ -80,19 +66,32 @@ const LAYOUT_KEY = (source) => `psp_layout_${source}`
 const DEFAULT_PLUGIN_CONFIG = { edit_mode: 'SELECT_REGION' }
-export default function Pivot({ source }) {
+export default function Pivot({ source, selectedStack, setSelectedStack }) {
  const { dark } = useTheme()
  const viewerRef = useRef()
  const workerRef = useRef()
  const tableRef = useRef()
  const allRowsRef = useRef([])
  const expandDepthRef = useRef(null)
  const lastClickKeyRef = useRef(null)
  const perspClickHandlerRef = useRef(null)
  const [status, setStatus] = useState('idle')
  const [error, setError] = useState('')
  const [inspectedRows, setInspectedRows] = useState(null)
  const [clickDetail, setClickDetail] = useState(null)
  const [decimals, setDecimals] = useState(2)
  const [paneWidth, setPaneWidth] = useState(384)
  const [sortCol, setSortCol] = useState(null)
  const [sortDir, setSortDir] = useState('asc')
-  // Named layouts
+  const selectedView = selectedStack ?? source
  const viewType = selectedStack ? 'stack' : 'source'
  useEffect(() => {
    if (viewerRef.current) viewerRef.current.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
  }, [dark])
  // Named layouts — stacks use localStorage only (no server FK to sources)
  const [layouts, setLayouts] = useState([])
  const [activeLayoutId, setActiveLayoutId] = useState(null)
  const [saveAsName, setSaveAsName] = useState('')
@ -105,18 +104,21 @@ export default function Pivot({ source }) {
  }
  const loadLayouts = useCallback(async () => {
-    if (!source) return
+    if (!selectedView) return
    try {
-      const rows = await api.getPivotLayouts(source)
+      const rows = viewType === 'source'
        ? await api.getPivotLayouts(selectedView)
        : await api.getStackPivotLayouts(selectedView)
      setLayouts(rows)
    } catch {}
-  }, [source])
+  }, [selectedView])
  useEffect(() => {
-    if (!source) return
+    if (!selectedView) return
    let cancelled = false
    setInspectedRows(null)
    setClickDetail(null)
    lastClickKeyRef.current = null
    setActiveLayoutId(null)
    setShowSaveAs(false)
    allRowsRef.current = []
@ -129,7 +131,7 @@ export default function Pivot({ source }) {
      try {
        const [perspective, rows] = await Promise.all([
          loadPerspective(),
-          fetchAllRows(source),
+          fetchAllRows(selectedView),
        ])
        if (cancelled) return
        if (!rows.length) { setStatus('noview'); return }
@ -142,13 +144,27 @@ export default function Pivot({ source }) {
        if (cancelled) { worker.terminate(); return }
        workerRef.current = worker
-        const table = await worker.table(rows, { name: source })
+        const table = await worker.table(rows, { name: selectedView })
        if (cancelled) return
        tableRef.current = table
        const viewer = viewerRef.current
        const validCols = new Set(Object.keys(rows[0] || {}))
-        viewer.addEventListener('perspective-click', async (e) => {
+        function cleanLayout(cfg) {
          if (!cfg) return cfg
          const clean = { ...cfg }
          const exprNames = new Set(Object.keys(clean.expressions || {}))
          const valid = (c) => validCols.has(c) || exprNames.has(c)
          if (clean.columns) clean.columns = clean.columns.filter(c => c == null || valid(c))
          if (clean.group_by) clean.group_by = clean.group_by.filter(valid)
          if (clean.split_by) clean.split_by = clean.split_by.filter(valid)
          if (clean.sort) clean.sort = clean.sort.filter(([c]) => valid(c))
          if (clean.filter) clean.filter = clean.filter.filter(([c]) => valid(c))
          return clean
        }
        perspClickHandlerRef.current = async (e) => {
          const detail = e.detail || {}
          const { row, column_names } = detail
          if (!row) return
@ -160,14 +176,39 @@ export default function Pivot({ source }) {
          const hasHierarchy = (config.group_by || []).length > 0
          if (!hasHierarchy) return
-          setClickDetail({ row, config, column_names, eventFilters })
+          // column_names encodes the full column path: [split_val_1, ..., split_val_N, measure]
          // positionally matching config.split_by. Perspective may omit split_by coordinate
          // filters from detail.config.filter, so derive any missing ones from column_names.
          const splitByFields = config.split_by || []
          const coveredByEvent = new Set(eventFilters.filter(([, op]) => op === '==').map(([f]) => f))
          const derivedSplitFilters = splitByFields
            .map((field, i) => {
              if (coveredByEvent.has(field)) return null
              const val = Array.isArray(column_names) && column_names[i] != null
                ? String(column_names[i]) : null
              return val != null ? [field, '==', val] : null
            })
            .filter(Boolean)
          const allFilters = [...eventFilters, ...derivedSplitFilters]
          // Same cell clicked again — toggle the pane closed.
          // Key on row path + column names (from the raw event) rather than derived
          // filters, which can vary between clicks on stack/expression views.
          const clickKey = JSON.stringify({ p: row['__ROW_PATH__'], c: column_names })
          if (lastClickKeyRef.current === clickKey) {
            lastClickKeyRef.current = null
            setInspectedRows(null)
            setClickDetail(null)
            return
          }
          lastClickKeyRef.current = clickKey
          setClickDetail({ row, config, column_names, eventFilters: allFilters })
          // Use a Perspective view with the event filters + expressions so computed
          // columns (split_by) are evaluated and filtered correctly
          try {
            const view = await tableRef.current.view({
-              filter: eventFilters,
+              filter: allFilters,
-              expressions: config.expressions || [],
+              expressions: config.expressions || {},
            })
            const data = await view.to_json()
            await view.delete()
@ -177,25 +218,28 @@ export default function Pivot({ source }) {
              Object.fromEntries(Object.entries(r).filter(([k]) => !exprNames.has(k)))
            )
            setInspectedRows(cleaned)
-          } catch {
+          } catch (err) {
-            setInspectedRows(filterRowsByConfig(allRowsRef.current, eventFilters))
+            console.warn('Perspective inspector view failed, falling back to JS filter:', err)
            setInspectedRows(filterRowsByConfig(allRowsRef.current, allFilters))
          }
-        })
+        }
        viewer.addEventListener('perspective-click', perspClickHandlerRef.current)
        await viewer.load(worker)
        const plugin = await viewer.getPlugin()
-        const savedLayout = localStorage.getItem(LAYOUT_KEY(source))
+        const savedLayout = localStorage.getItem(LAYOUT_KEY(selectedView))
        if (savedLayout) {
-          const parsed = JSON.parse(savedLayout)
+          const parsed = cleanLayout(JSON.parse(savedLayout))
          await viewer.restore(parsed)
          await plugin.restore(parsed.plugin_config || DEFAULT_PLUGIN_CONFIG)
          if (parsed.expand_depth != null) await applyExpandDepth(viewer, parsed.expand_depth)
        } else {
-          await viewer.restore({ table: source, settings: false, plugin_config: DEFAULT_PLUGIN_CONFIG })
+          await viewer.restore({ table: selectedView, settings: false, plugin_config: DEFAULT_PLUGIN_CONFIG })
          await plugin.restore(DEFAULT_PLUGIN_CONFIG)
        }
        await viewer.flush()
        viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
        setStatus('ready')
      } catch (err) {
@ -204,8 +248,14 @@ export default function Pivot({ source }) {
    }
    init()
-    return () => { cancelled = true }
+    return () => {
-  }, [source])
+      cancelled = true
      if (perspClickHandlerRef.current && viewerRef.current) {
        viewerRef.current.removeEventListener('perspective-click', perspClickHandlerRef.current)
        perspClickHandlerRef.current = null
      }
    }
  }, [selectedView])
  async function applyExpandDepth(viewer, depth) {
    if (depth == null) return
@ -219,15 +269,35 @@ export default function Pivot({ source }) {
  async function applyLayout(layout) {
    const viewer = viewerRef.current
    if (!viewer) return
-    await viewer.restore(layout.config)
+    try {
-    if (layout.config.plugin_config) {
+      const validCols = new Set(Object.keys(allRowsRef.current[0] || {}))
-      const plugin = await viewer.getPlugin()
+      function cleanLayout(cfg) {
-      await plugin.restore(layout.config.plugin_config)
+        if (!cfg) return cfg
        const clean = { ...cfg }
        const exprNames = new Set(Object.keys(clean.expressions || {}))
        const valid = (c) => validCols.has(c) || exprNames.has(c)
        if (clean.columns) clean.columns = clean.columns.filter(c => c == null || valid(c))
        if (clean.group_by) clean.group_by = clean.group_by.filter(valid)
        if (clean.split_by) clean.split_by = clean.split_by.filter(valid)
        if (clean.sort) clean.sort = clean.sort.filter(([c]) => valid(c))
        if (clean.filter) clean.filter = clean.filter.filter(([c]) => valid(c))
        return clean
      }
-    await applyExpandDepth(viewer, layout.config.expand_depth ?? null)
+      const cleaned = cleanLayout(layout.config)
      await viewer.restore(cleaned)
      if (cleaned.plugin_config) {
        const plugin = await viewer.getPlugin()
        await plugin.restore(cleaned.plugin_config)
      }
      await applyExpandDepth(viewer, cleaned.expand_depth ?? null)
      setActiveLayoutId(layout.id)
-    // also persist to localStorage so it survives refresh
+      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(cleaned))
-    localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(layout.config))
+    } catch {
      // Layout references columns that no longer exist — remove it
      localStorage.removeItem(LAYOUT_KEY(selectedView))
      setActiveLayoutId(null)
      await viewer.restore({ table: selectedView, settings: false })
    }
  }
  async function captureConfig() {
@ -238,16 +308,24 @@ export default function Pivot({ source }) {
    return { ...viewerConfig, plugin_config: pluginConfig, expand_depth: expandDepthRef.current }
  }
  const saveLayout = (name, config) => viewType === 'source'
    ? api.savePivotLayout(selectedView, name, config)
    : api.saveStackPivotLayout(selectedView, name, config)
  const deleteLayout = (id) => viewType === 'source'
    ? api.deletePivotLayout(selectedView, id)
    : api.deleteStackPivotLayout(selectedView, id)
  async function handleSaveOver() {
    const layout = layouts.find(l => l.id === activeLayoutId)
    if (!layout) return
    const config = await captureConfig()
    if (!config) return
    try {
-      const saved = await api.savePivotLayout(source, layout.layout_name, config)
+      const saved = await saveLayout(layout.layout_name, config)
      localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(config))
      await loadLayouts()
      setActiveLayoutId(saved.id)
      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
      await loadLayouts()
      flashMsg('Saved!')
    } catch (err) {
      flashMsg(err.message)
@ -260,8 +338,8 @@ export default function Pivot({ source }) {
    const config = await captureConfig()
    if (!config) return
    try {
-      const saved = await api.savePivotLayout(source, name, config)
+      const saved = await saveLayout(name, config)
-      localStorage.setItem(LAYOUT_KEY(source), JSON.stringify(config))
+      localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
      await loadLayouts()
      setActiveLayoutId(saved.id)
      setShowSaveAs(false)
@ -275,7 +353,7 @@ export default function Pivot({ source }) {
  async function handleDelete(layout, e) {
    e.stopPropagation()
    try {
-      await api.deletePivotLayout(source, layout.id)
+      await deleteLayout(layout.id)
      if (activeLayoutId === layout.id) setActiveLayoutId(null)
      await loadLayouts()
      flashMsg('Deleted')
@ -287,15 +365,33 @@ export default function Pivot({ source }) {
  function handleResetToDefault() {
    const viewer = viewerRef.current
    if (!viewer) return
-    localStorage.removeItem(LAYOUT_KEY(source))
+    localStorage.removeItem(LAYOUT_KEY(selectedView))
    setActiveLayoutId(null)
-    viewer.restore({ table: source, settings: true, plugin_config: DEFAULT_PLUGIN_CONFIG })
+    viewer.restore({ table: selectedView, settings: true, plugin_config: DEFAULT_PLUGIN_CONFIG })
  }
  if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
  const cols = inspectedRows?.length ? Object.keys(inspectedRows[0]) : []
  const sortedRows = sortCol == null || !inspectedRows ? inspectedRows : [...inspectedRows].sort((a, b) => {
    const av = a[sortCol], bv = b[sortCol]
    if (av == null && bv == null) return 0
    if (av == null) return 1
    if (bv == null) return -1
    const num = typeof av === 'number' && typeof bv === 'number'
    const cmp = num ? av - bv : String(av).localeCompare(String(bv))
    return sortDir === 'asc' ? cmp : -cmp
  })
  const totals = cols.reduce((acc, c) => {
    const vals = (inspectedRows || []).map(r => r[c])
    if (vals.length > 0 && vals.every(v => v == null || typeof v === 'number')) {
      acc[c] = vals.reduce((s, v) => s + (v ?? 0), 0)
    }
    return acc
  }, {})
  const groupBy = clickDetail?.config?.group_by || []
  const splitBy = clickDetail?.config?.split_by || []
  const coordFields = new Set([...groupBy, ...splitBy])
@ -305,23 +401,26 @@ export default function Pivot({ source }) {
      .map(([f, , v]) => [f, v])
  )
  const cellCoords = [...groupBy, ...splitBy].map(f => coordMap[f]).filter(Boolean)
-  const splitVals = splitBy.map(f => coordMap[f]).filter(Boolean)
+  // column_names = [split_val_1, ..., split_val_N, measure_name] — use positional split_by length
-  const metrics = clickDetail?.column_names || []
+  // to separate split values from measure names; fall back to coordMap when ambiguous
-  const cellKey = splitVals.length > 0 && metrics.length > 0
+  const colNames = clickDetail?.column_names || []
  const splitVals = splitBy.map((f, i) =>
    coordMap[f] ?? (colNames[i] != null ? String(colNames[i]) : null)
  ).filter(Boolean)
  const metrics = splitBy.length > 0 ? colNames.slice(splitBy.length) : colNames
  const cellKey = metrics.length > 0
    ? [...splitVals, ...metrics].join('|')
    : null
  return (
    <div className="w-full h-full flex flex-col">
-      {/* Layout toolbar */}
+      {/* Layouts sub-bar */}
-      <div className="flex items-center gap-2 px-3 py-1.5 bg-white border-b border-gray-200 flex-shrink-0">
+      <div className="flex items-center gap-2 px-3 h-9 bg-white border-b border-gray-200 shrink-0 text-xs">
        <span className="text-xs text-gray-400 uppercase tracking-wide mr-1">Layouts</span>
        {layouts.map(l => (
          <div key={l.id}
            onClick={() => applyLayout(l)}
-            className={`flex items-center gap-1 text-xs rounded px-2 py-0.5 cursor-pointer border transition-colors
+            className={`flex items-center gap-1 rounded px-2 py-0.5 cursor-pointer border transition-colors
              ${activeLayoutId === l.id
                ? 'bg-blue-50 border-blue-300 text-blue-700'
                : 'bg-white border-gray-200 text-gray-600 hover:border-gray-400'}`}>
@ -334,7 +433,7 @@ export default function Pivot({ source }) {
        {activeLayoutId !== null && !showSaveAs && (
          <button onClick={handleSaveOver}
-            className="text-xs text-blue-500 hover:text-blue-700 border border-blue-200 rounded px-2 py-0.5">
+            className="text-blue-500 hover:text-blue-700 border border-blue-200 rounded px-2 py-0.5">
            Save
          </button>
        )}
@ -347,30 +446,27 @@ export default function Pivot({ source }) {
              onChange={e => setSaveAsName(e.target.value)}
              onKeyDown={e => { if (e.key === 'Enter') handleSaveAs(); if (e.key === 'Escape') { setShowSaveAs(false); setSaveAsName('') } }}
              placeholder="Layout name…"
-              className="text-xs border border-gray-300 rounded px-2 py-0.5 w-36 focus:outline-none focus:border-blue-400"
+              className="border border-gray-300 rounded px-2 py-0.5 w-36 focus:outline-none focus:border-blue-400"
            />
-            <button onClick={handleSaveAs} className="text-xs text-blue-600 hover:text-blue-800 px-1">Save</button>
+            <button onClick={handleSaveAs} className="text-blue-600 hover:text-blue-800 px-1">Save</button>
-            <button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-xs text-gray-400 hover:text-gray-600 px-1">Cancel</button>
+            <button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-gray-400 hover:text-gray-600 px-1">Cancel</button>
          </div>
        ) : (
          <button
            onClick={() => setShowSaveAs(true)}
-            className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-0.5">
+            className="text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-0.5">
            + Save as…
          </button>
        )}
        {activeLayoutId !== null && (
-          <button onClick={handleResetToDefault}
+          <button onClick={handleResetToDefault} className="text-gray-300 hover:text-gray-500 ml-1">reset</button>
            className="text-xs text-gray-300 hover:text-gray-500 ml-1">
            reset
          </button>
        )}
-        {layoutMsg && <span className="text-xs text-green-600 ml-1">{layoutMsg}</span>}
+        {layoutMsg && <span className="text-green-600 ml-1">{layoutMsg}</span>}
        <div className="ml-auto flex items-center gap-1">
-          <span className="text-xs text-gray-400">depth:</span>
+          <span className="text-gray-400">depth:</span>
          {[0, 1, 2, 3].map(d => (
            <button key={d} onClick={async () => {
              const v = viewerRef.current; if (!v) return
@ -379,7 +475,7 @@ export default function Pivot({ source }) {
              const p = await v.getPlugin()
              await p.draw(view)
              expandDepthRef.current = d
-            }} className="text-xs border border-gray-200 rounded px-1.5 py-0.5 text-gray-500 hover:border-gray-400">
+            }} className="border border-gray-200 rounded px-1.5 py-0.5 text-gray-500 hover:border-gray-400">
              {d}
            </button>
          ))}
@ -411,12 +507,40 @@ export default function Pivot({ source }) {
        </div>
        {inspectedRows && clickDetail && (
-          <div className="w-96 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
+          <div
-            <div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
+            style={{ width: paneWidth }}
-              <span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">
+            className="relative border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0"
          >
            {/* Drag-to-resize handle on left edge */}
            <div
              className="absolute left-0 top-0 bottom-0 w-1 cursor-col-resize hover:bg-blue-300 z-10"
              onMouseDown={(e) => {
                e.preventDefault()
                const startX = e.clientX
                const startW = paneWidth
                const onMove = (me) => setPaneWidth(Math.max(240, startW + startX - me.clientX))
                const onUp = () => {
                  document.removeEventListener('mousemove', onMove)
                  document.removeEventListener('mouseup', onUp)
                }
                document.addEventListener('mousemove', onMove)
                document.addEventListener('mouseup', onUp)
              }}
            />
            {/* Header: breadcrumb + row count + controls */}
            <div className="flex items-center justify-between pl-3 pr-2 py-2 border-b border-gray-100 flex-shrink-0">
              <div className="flex items-center gap-2 min-w-0">
                {cellCoords.length > 0 && (
                  <span className="text-xs text-gray-700 font-mono font-semibold truncate">
                    {cellCoords.join(' › ')}
                  </span>
                )}
                <span className="text-xs text-gray-400 flex-shrink-0">
                  {inspectedRows.length} row{inspectedRows.length !== 1 ? 's' : ''}
                </span>
-              <div className="flex items-center gap-2">
+              </div>
              <div className="flex items-center gap-2 flex-shrink-0">
                <div className="flex items-center gap-0.5">
                  <button onClick={() => setDecimals(d => Math.max(0, d - 1))}
                    className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">−</button>
@ -424,36 +548,13 @@ export default function Pivot({ source }) {
                  <button onClick={() => setDecimals(d => Math.min(8, d + 1))}
                    className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">+</button>
                </div>
-                <button onClick={() => { setInspectedRows(null); setClickDetail(null) }}
+                <button onClick={() => { setInspectedRows(null); setClickDetail(null); lastClickKeyRef.current = null }}
                  className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
              </div>
            </div>
            <div className="flex-1 overflow-y-auto">
-
+              {/* User-set filters (only shown when active) */}
              {/* Cell coordinates */}
              <div className="px-3 py-2 border-b border-gray-100">
                <div className="text-xs text-gray-400 uppercase tracking-wide mb-1">
                  {[...groupBy, ...splitBy].join(' › ') || clickDetail.column_names?.join(', ') || 'Cell'}
                </div>
                {cellCoords.length > 0 && (
                  <div className="text-xs text-gray-700 font-mono font-semibold">
                    {cellCoords.join(' › ')}
                  </div>
                )}
                {Object.entries(clickDetail.row)
                  .filter(([k, v]) => k !== '__ROW_PATH__' && v != null)
                  .map(([k, v]) => {
                    const isSelected = cellKey != null && k === cellKey
                    return (
                      <div key={k} className={`flex justify-between py-0.5 gap-2 ${isSelected ? 'font-semibold' : ''}`}>
                        <span className={`text-xs font-mono shrink-0 ${isSelected ? 'text-gray-700' : 'text-gray-400'}`}>{k}</span>
                        <span className={`text-xs font-mono text-right ${isSelected ? 'text-blue-600' : 'text-gray-700'}`}>{formatVal(v, decimals)}</span>
                      </div>
                    )
                  })}
              </div>
              {/* User-set filters */}
              {(() => {
                const userFilters = (clickDetail.eventFilters || []).filter(([f]) => !coordFields.has(f))
                return userFilters.length > 0 ? (
@ -472,13 +573,20 @@ export default function Pivot({ source }) {
                  <table className="w-full text-xs">
                    <thead>
                      <tr className="text-left text-gray-400 border-b border-gray-100 bg-gray-50 sticky top-0">
-                        {cols.map(c => (
+                        {cols.map(c => {
-                          <th key={c} className="px-2 py-1 font-medium whitespace-nowrap">{c}</th>
+                          const active = sortCol === c
-                        ))}
+                          return (
                            <th key={c}
                              onClick={() => { if (active) setSortDir(d => d === 'asc' ? 'desc' : 'asc'); else { setSortCol(c); setSortDir('asc') } }}
                              className="px-2 py-1 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600">
                              {c}{active ? (sortDir === 'asc' ? ' ▲' : ' ▼') : ''}
                            </th>
                          )
                        })}
                      </tr>
                    </thead>
                    <tbody>
-                      {inspectedRows.map((row, i) => (
+                      {sortedRows.map((row, i) => (
                        <tr key={i} className="border-t border-gray-50 hover:bg-gray-50">
                          {cols.map(c => {
                            const f = formatVal(row[c], decimals)
@ -491,6 +599,17 @@ export default function Pivot({ source }) {
                        </tr>
                      ))}
                    </tbody>
                    {Object.keys(totals).length > 0 && (
                      <tfoot>
                        <tr className="border-t-2 border-gray-200 bg-gray-50 font-semibold text-gray-700 sticky bottom-0">
                          {cols.map(c => (
                            <td key={c} className="px-2 py-1 font-mono whitespace-nowrap text-right">
                              {totals[c] != null ? formatVal(totals[c], decimals) : ''}
                            </td>
                          ))}
                        </tr>
                      </tfoot>
                    )}
                  </table>
                </div>
              )}
--- a/ui/src/pages/Records.jsx
+++ b/ui/src/pages/Records.jsx
@ -1,7 +1,68 @@
 import { useState, useEffect, useRef } from 'react'
 import { api } from '../api'
 function AutocompleteInput({ value, onChange, onEnter, suggestions = [], className, placeholder }) {
  const [open, setOpen] = useState(false)
  const [highlighted, setHighlighted] = useState(0)
  const [dropPos, setDropPos] = useState(null)
  const inputRef = useRef()
  const listRef = useRef()
  const filtered = value
    ? suggestions.filter(s => s.toLowerCase().includes(value.toLowerCase()))
    : suggestions
  function openList() {
    if (inputRef.current) {
      const r = inputRef.current.getBoundingClientRect()
      setDropPos({ top: r.bottom + 2, left: r.left, minWidth: r.width })
    }
    setOpen(true)
    setHighlighted(0)
  }
  function select(val) { onChange(val); setOpen(false); inputRef.current?.focus() }
  function handleKeyDown(e) {
    if (e.altKey && e.key === 'ArrowDown') { e.preventDefault(); openList(); return }
    if (open && filtered.length > 0) {
      if (e.key === 'Tab')       { e.preventDefault(); setHighlighted(h => (h + 1) % filtered.length); return }
      if (e.key === 'ArrowDown') { e.preventDefault(); setHighlighted(h => Math.min(h + 1, filtered.length - 1)); return }
      if (e.key === 'ArrowUp')   { e.preventDefault(); setHighlighted(h => Math.max(h - 1, 0)); return }
      if (e.key === 'Enter')     { e.preventDefault(); select(filtered[highlighted]); return }
      if (e.key === 'Escape')    { setOpen(false); return }
    }
    if (e.key === 'Enter') onEnter?.()
  }
  useEffect(() => {
    if (!open || !listRef.current) return
    listRef.current.children[highlighted]?.scrollIntoView({ block: 'nearest' })
  }, [highlighted, open])
  return (
    <div className="relative">
      <input ref={inputRef} className={className} value={value} placeholder={placeholder}
        onChange={e => { onChange(e.target.value); if (e.target.value) openList() }}
        onKeyDown={handleKeyDown}
        onBlur={e => { if (!listRef.current?.contains(e.relatedTarget)) setOpen(false) }}
      />
      {open && filtered.length > 0 && dropPos && (
        <div ref={listRef}
          style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
          className="bg-white border border-gray-200 rounded shadow-lg max-h-40 overflow-y-auto">
          {filtered.map((s, i) => (
            <div key={s}
              className={`px-2 py-1 text-xs cursor-pointer whitespace-nowrap ${i === highlighted ? 'bg-blue-50 text-blue-700' : 'text-gray-700 hover:bg-gray-50'}`}
              onMouseDown={e => { e.preventDefault(); select(s) }}>{s}</div>
          ))}
        </div>
      )}
    </div>
  )
 }
 const DATE_RE = /^\d{4}-\d{2}-\d{2}(T[\d:.Z+-]+)?$/
 const HIDDEN_COLS = new Set(['id', '_overridden'])
 function formatVal(val) {
  if (val === null || val === undefined) return null
@ -26,19 +87,61 @@ export default function Records({ source }) {
  const [loading, setLoading] = useState(false)
  const [viewError, setViewError] = useState(null)
  const [sort, setSort] = useState({ col: null, dir: 'asc' })
-  const [filters, setFilters] = useState([])
+  const [filters, setFilters] = useState([])  // DB sort/filter queries
-  const debounceRef = useRef(null)
+  const [rowFilter, setRowFilter] = useState('')  // regex filter for selecting rows
  const [selected, setSelected] = useState(new Set())  // row IDs selected for bulk override
  const [bulkDraft, setBulkDraft] = useState({})  // bulk override values
  const LIMIT = 100
  // Override cols — loaded from DB once per source, extended by user via +
  const [overrideCols, setOverrideCols] = useState([])   // keys seen in overrides across all records
  const [extraCols, setExtraCols] = useState([])         // new cols added this session via +
  const [globalValues, setGlobalValues] = useState({})   // picklist suggestions
  // Override panel
  const [panelOpen, setPanelOpen] = useState(false)
  const [selectedRow, setSelectedRow] = useState(null)
  const [selectedRecord, setSelectedRecord] = useState(null)
  const [overrideDraft, setOverrideDraft] = useState({})
  const [panelLoading, setPanelLoading] = useState(false)
  const [panelSaving, setPanelSaving] = useState(false)
  const [panelMsg, setPanelMsg] = useState(null)
  const debounceRef = useRef(null)
  useEffect(() => {
    if (!source) return
    setOffset(0)
    setSort({ col: null, dir: 'asc' })
    setFilters([])
    setViewError(null)
    setSelectedRecord(null)
    setSelectedRow(null)
    setPanelOpen(false)
    setOverrideCols([])
    setExtraCols([])
    load(0, null, 'asc', [])
    api.getOverrideKeys(source).then(setOverrideCols).catch(() => {})
    api.getGlobalValues().then(setGlobalValues).catch(() => {})
    setSelected(new Set())
    setBulkDraft({})
    setRowFilter('')
  }, [source])
  // Auto-select all rows matching the regex filter when it changes
  useEffect(() => {
    if (!rowFilter) return
    let re = null
    try { re = new RegExp(rowFilter, 'i') } catch { return }
    const matches = rows.filter(r => {
      for (const col of displayCols) {
        const val = r[col]
        if (val != null && re.test(String(val))) return true
      }
      return false
    })
    setSelected(new Set(matches.map(r => r.id)))
  }, [rowFilter, rows])
  async function load(off, col, dir, filt) {
    setLoading(true)
    try {
@ -46,8 +149,7 @@ export default function Records({ source }) {
      const res = await api.getViewData(source, LIMIT, off, col, dir, active)
      setExists(res.exists)
      setRows(res.rows)
-      if (res.rows.length > 0 && cols.length === 0) setCols(Object.keys(res.rows[0]))
+      if (res.rows.length > 0) setCols(Object.keys(res.rows[0]))
      else if (res.rows.length > 0) setCols(Object.keys(res.rows[0]))
    } catch (err) {
      setViewError(err.message)
    } finally {
@ -70,12 +172,14 @@ export default function Records({ source }) {
  }
  function addFilter() {
-    setFilters(f => [...f, { col: cols[0] || '', pattern: '' }])
+    const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
    setFilters(f => [...f, { col: visCols[0] || '', pattern: '' }])
  }
  function removeFilter(i) {
    const next = filters.filter((_, idx) => idx !== i)
    setFilters(next)
    setSelected(new Set())
    setOffset(0)
    load(0, sort.col, sort.dir, next)
  }
@ -83,19 +187,101 @@ export default function Records({ source }) {
  function updateFilter(i, key, val) {
    const next = filters.map((f, idx) => idx === i ? { ...f, [key]: val } : f)
    setFilters(next)
    setSelected(new Set())
    setOffset(0)
    triggerLoad(0, sort.col, sort.dir, next)
  }
-  function prev() { const o = Math.max(0, offset - LIMIT); setOffset(o); load(o, sort.col, sort.dir, filters) }
+  function prev() { const o = Math.max(0, offset - LIMIT); setOffset(o); setSelected(new Set()); load(o, sort.col, sort.dir, filters) }
-  function next() { const o = offset + LIMIT; setOffset(o); load(o, sort.col, sort.dir, filters) }
+  function next() { const o = offset + LIMIT; setOffset(o); setSelected(new Set()); load(o, sort.col, sort.dir, filters) }
  async function openPanel(row) {
    setPanelOpen(true)
    setSelectedRow(row)
    setSelectedRecord(null)
    setOverrideDraft({})
    setPanelMsg(null)
    const id = row.id
    if (!id) {
      setPanelMsg({ text: 'No record ID — regenerate the view in Sources.', ok: false })
      return
    }
    setPanelLoading(true)
    try {
      const rec = await api.getRecord(id)
      setSelectedRecord(rec)
      setOverrideDraft(rec.overrides || {})
    } catch (err) {
      setPanelMsg({ text: err.message, ok: false })
    } finally {
      setPanelLoading(false)
    }
  }
  function closePanel() {
    setPanelOpen(false)
    setSelectedRow(null)
    setSelectedRecord(null)
    setOverrideDraft({})
    setPanelMsg(null)
  }
  async function handleSaveOverrides() {
    if (!selectedRecord) return
    setPanelSaving(true)
    setPanelMsg(null)
    try {
      const toSave = { ...overrideDraft }
      const updated = await api.setRecordOverrides(selectedRecord.id, toSave)
      setSelectedRecord(updated)
      setOverrideDraft(updated.overrides || {})
      // Merge any new cols from extraCols into overrideCols
      setOverrideCols(prev => [...new Set([...prev, ...extraCols.filter(c => c.trim())])])
      setExtraCols([])
      setPanelMsg({ text: 'Saved.', ok: true })
      load(offset, sort.col, sort.dir, filters)
    } catch (err) {
      setPanelMsg({ text: err.message, ok: false })
    } finally {
      setPanelSaving(false)
    }
  }
  async function handleClearOverrides() {
    if (!selectedRecord) return
    setPanelSaving(true)
    setPanelMsg(null)
    try {
      const updated = await api.clearRecordOverrides(selectedRecord.id)
      setSelectedRecord(updated)
      setOverrideDraft({})
      setPanelMsg({ text: 'Cleared.', ok: true })
      load(offset, sort.col, sort.dir, filters)
    } catch (err) {
      setPanelMsg({ text: err.message, ok: false })
    } finally {
      setPanelSaving(false)
    }
  }
  if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
-  const displayCols = rows.length > 0 ? Object.keys(rows[0]) : cols
+  const displayCols = (rows.length > 0 ? Object.keys(rows[0]) : cols).filter(c => !HIDDEN_COLS.has(c))
  const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
  // For bulk bar: only established override keys
  const allOverrideCols = [...new Set([...overrideCols, ...extraCols])]
  const savedOverrides = selectedRecord?.overrides || {}
  const isDirty = Object.values(overrideDraft).some(v => String(v).trim())
    || extraCols.some(c => c.trim())
    || Object.keys(savedOverrides).some(k => !String(overrideDraft[k] ?? '').trim())
  return (
-    <div className="p-6">
+    <div className="flex h-full min-h-0 overflow-hidden">
      <div className="flex-1 overflow-auto p-6 min-w-0">
        <div className="flex items-center justify-between mb-4">
          <h1 className="text-xl font-semibold text-gray-800">Records — {source}</h1>
          {exists && rows.length > 0 && (
@ -104,8 +290,9 @@ export default function Records({ source }) {
        </div>
        {/* Filter bar */}
-      {exists !== false && displayCols.length > 0 && (
+        {exists !== false && visCols.length > 0 && (
          <div className="mb-4 flex flex-wrap gap-2 items-center">
            <span className="text-xs text-gray-400 font-medium mr-1">DB query:</span>
            {filters.map((f, i) => (
              <div key={i} className="flex items-center gap-1 bg-white border border-gray-200 rounded px-2 py-1">
                <select
@ -113,7 +300,7 @@ export default function Records({ source }) {
                  value={f.col}
                  onChange={e => updateFilter(i, 'col', e.target.value)}
                >
-                {displayCols.map(c => <option key={c} value={c}>{c}</option>)}
+                  {visCols.map(c => <option key={c} value={c}>{c}</option>)}
                </select>
                <span className="text-xs text-gray-300 mx-0.5">~*</span>
                <input
@ -122,41 +309,91 @@ export default function Records({ source }) {
                  value={f.pattern}
                  onChange={e => updateFilter(i, 'pattern', e.target.value)}
                />
-              <button
+                <button onClick={() => removeFilter(i)} className="text-gray-300 hover:text-gray-500 ml-1 leading-none">×</button>
                onClick={() => removeFilter(i)}
                className="text-gray-300 hover:text-gray-500 ml-1 leading-none"
              >×</button>
              </div>
            ))}
-          <button
+            <button onClick={addFilter}
-            onClick={addFilter}
+              className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-1">
            className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-1"
          >
              + filter
            </button>
            {filters.length > 0 && (
              <button onClick={() => { setFilters([]); setOffset(0); load(0, sort.col, sort.dir, []) }}
                className="text-xs text-gray-400 hover:text-red-500">clear</button>
            )}
          </div>
        )}
        {/* Bulk select + override bar */}
        {exists && visCols.length > 0 && (
          <div className="mb-4 flex flex-wrap gap-2 items-center">
            <span className="text-xs text-gray-400 font-medium mr-1">Bulk select:</span>
            <input
              className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-blue-400 ${
                rowFilter ? 'border-blue-300' : 'border-gray-200'
              }`}
              placeholder="regex on loaded rows…"
              value={rowFilter}
              onChange={e => setRowFilter(e.target.value)}
            />
            {rowFilter && (
              <span className="text-xs text-gray-400">{selected.size} of {rows.length} rows selected</span>
            )}
            {selected.size > 0 && (
              <div className="flex items-center gap-2 ml-4 p-2 bg-blue-50 border border-blue-200 rounded flex-wrap">
                {allOverrideCols.map(col => (
                  <AutocompleteInput
                    key={col}
                    className="border border-blue-300 rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-blue-500 bg-white"
                    placeholder={col}
                    value={bulkDraft[col] || ''}
                    onChange={v => setBulkDraft(d => ({ ...d, [col]: v }))}
                    suggestions={[...(globalValues[col] || [])].sort()}
                  />
                ))}
                <button
-              onClick={() => { setFilters([]); setOffset(0); load(0, sort.col, sort.dir, []) }}
+                  onClick={async () => {
-              className="text-xs text-gray-400 hover:text-red-500"
+                    const overrides = Object.fromEntries(
                      Object.entries(bulkDraft).filter(([, v]) => v.trim())
                    )
                    if (Object.keys(overrides).length === 0) return
                    if (selected.size === 0) return
                    setPanelSaving(true)
                    setPanelMsg(null)
                    try {
                      const res = await api.setBulkRecordOverrides(source, [...selected], overrides)
                      setSelected(new Set())
                      setBulkDraft({})
                      setPanelMsg({ text: `Updated ${res.updated} records.`, ok: true })
                      load(offset, sort.col, sort.dir, filters)
                    } catch (err) {
                      setPanelMsg({ text: err.message, ok: false })
                    } finally {
                      setPanelSaving(false)
                    }
                  }}
                  disabled={panelSaving || selected.size === 0 || Object.values(bulkDraft).every(v => !v.trim())}
                  className="text-xs bg-blue-600 text-white px-3 py-1 rounded hover:bg-blue-700 disabled:opacity-40 whitespace-nowrap"
                >
-              clear
+                  Apply to {selected.size}
                </button>
                <button
                  onClick={() => { setSelected(new Set()); setBulkDraft({}); setRowFilter('') }}
                  className="text-xs text-blue-400 hover:text-blue-600"
                >
                  cancel
                </button>
              </div>
            )}
          </div>
        )}
        {loading && <p className="text-sm text-gray-400">Loading…</p>}
-
+        {!loading && viewError && <p className="text-sm text-red-500">View error: {viewError} — check field types in Sources.</p>}
      {!loading && viewError && (
        <p className="text-sm text-red-500">View error: {viewError} — check field types in Sources.</p>
      )}
        {!loading && exists === false && (
          <p className="text-sm text-gray-400">
            No view generated yet. Go to <span className="font-medium text-gray-600">Sources</span>, check fields as <span className="font-medium text-gray-600">In view</span>, then click <span className="font-medium text-gray-600">Generate view</span>.
          </p>
        )}
        {!loading && exists && rows.length === 0 && (
          <p className="text-sm text-gray-400">
            {filters.some(f => f.col && f.pattern) ? 'No records match the current filters.' : 'View exists but no transformed records yet. Import data and run a transform first.'}
@ -169,26 +406,49 @@ export default function Records({ source }) {
              <table className="w-full text-sm">
                <thead>
                  <tr className="text-left text-xs text-gray-400 border-b border-gray-100 bg-gray-50">
                    <th className="px-2 py-2 w-8">
                      <input
                        type="checkbox"
                        className="cursor-pointer"
                        checked={rows.length > 0 && rows.every(r => selected.has(r.id))}
                        onChange={e => {
                          if (e.target.checked) setSelected(new Set(rows.map(r => r.id)))
                          else setSelected(new Set())
                        }}
                      />
                    </th>
                    {displayCols.map(col => {
                      const active = sort.col === col
                      return (
-                      <th
+                        <th key={col} onClick={() => toggleSort(col)}
-                        key={col}
+                          className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600">
                        onClick={() => toggleSort(col)}
                        className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600"
                      >
                          {col}
-                        <span className="ml-1 text-gray-300">
+                          <span className="ml-1 text-gray-300">{active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}</span>
                          {active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}
                        </span>
                        </th>
                      )
                    })}
                  </tr>
                </thead>
                <tbody>
-                {rows.map((row, i) => (
+                  {rows.map((row, i) => {
-                  <tr key={i} className="border-t border-gray-50 hover:bg-gray-50">
+                    const isOverridden = row._overridden
                    const isRowSelected = selected.has(row.id)
                    const isPanelSelected = selectedRow?.id != null && selectedRow.id === row.id
                    return (
                      <tr key={i} onClick={() => openPanel(row)}
                        className={`border-t border-gray-50 cursor-pointer transition-colors
                          ${isPanelSelected ? 'bg-blue-50' : isRowSelected ? 'bg-blue-50' : isOverridden ? 'bg-amber-50 hover:bg-amber-100' : 'hover:bg-gray-50'}`}>
                        <td className="px-2 py-2">
                          <input
                            type="checkbox"
                            className="cursor-pointer"
                            checked={isRowSelected}
                            onChange={e => {
                              e.stopPropagation()
                              setSelected(s => { const n = new Set(s); n.has(row.id) ? n.delete(row.id) : n.add(row.id); return n })
                            }}
                          />
                        </td>
                        {displayCols.map((col, j) => {
                          const formatted = formatVal(row[col])
                          return (
@ -198,24 +458,183 @@ export default function Records({ source }) {
                          )
                        })}
                      </tr>
-                ))}
+                    )
                  })}
                </tbody>
              </table>
            </div>
            <div className="flex items-center gap-3 text-sm text-gray-500">
              <button onClick={prev} disabled={offset === 0}
-              className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">
+                className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">← Prev</button>
              ← Prev
            </button>
              <span>{offset + 1}–{offset + rows.length}</span>
              <button onClick={next} disabled={rows.length < LIMIT}
-              className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">
+                className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">Next →</button>
              Next →
            </button>
            </div>
          </>
        )}
      </div>
      {/* Panel */}
      {panelOpen && (
        <div className="w-80 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
            <span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">Record</span>
            <button onClick={closePanel} className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
          </div>
          {panelLoading && <p className="text-xs text-gray-400 p-3">Loading…</p>}
          {selectedRecord && !panelLoading && (
            <div className="flex-1 overflow-y-auto flex flex-col min-h-0">
              {panelMsg && (
                <div className={`text-xs px-3 py-2 border-b border-gray-100 ${panelMsg.ok ? 'text-green-600' : 'text-red-500'}`}>
                  {panelMsg.text}
                </div>
              )}
              {/* Raw fields — read only */}
              <div className="border-b border-gray-100">
                <div className="px-3 py-1.5 bg-gray-50 border-b border-gray-100">
                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Raw</span>
                </div>
                {Object.entries(selectedRecord.data || {}).map(([field, val]) => (
                  <div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-gray-50 first:border-t-0">
                    <span className="text-xs font-mono text-gray-400 w-28 shrink-0 truncate">{field}</span>
                    <span className="text-xs font-mono text-gray-500 truncate">{formatVal(val) ?? <span className="text-gray-300">—</span>}</span>
                  </div>
                ))}
              </div>
              {/* Transformed fields — read only delta */}
              <div className="border-b border-gray-100">
                <div className="px-3 py-1.5 bg-gray-50 border-b border-gray-100">
                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Transformed</span>
                </div>
                {Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).length === 0
                  ? <div className="px-3 py-2 text-xs text-gray-300">No rule output yet.</div>
                  : Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).map(([field, val]) => (
                    <div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-gray-50 first:border-t-0">
                      <span className="text-xs font-mono text-gray-400 w-28 shrink-0 truncate">{field}</span>
                      <span className="text-xs font-mono text-blue-600 truncate">{formatVal(val) ?? <span className="text-gray-300">—</span>}</span>
                    </div>
                  ))
                }
              </div>
              {/* Overrides — editable */}
              <div className="flex-1 border-b border-gray-100">
                <div className="flex items-center justify-between px-3 py-1.5 bg-gray-50 border-b border-gray-100">
                  <span className="text-xs font-medium text-gray-400 uppercase tracking-wide">Overrides</span>
                  <button
                    onClick={() => setExtraCols(ec => [...ec, ''])}
                    className="text-gray-400 hover:text-gray-700 font-medium text-sm leading-none"
                    title="Add field">+</button>
                </div>
                <table className="w-full text-xs">
                  <tbody>
                    {[...new Set([
                      ...Object.keys(selectedRecord.transformed || {}),
                      ...Object.keys(selectedRecord.overrides || {}),
                      ...overrideCols
                    ])].filter(k => !HIDDEN_COLS.has(k)).map(col => {
                      const override = overrideDraft[col] ?? ''
                      const placeholder = formatVal(selectedRecord.transformed?.[col]) ?? ''
                      const suggestions = [...(globalValues[col] || [])].sort()
                      return (
                        <tr key={col} className="border-t border-gray-50">
                          <td className="px-3 py-1.5 w-28 shrink-0">
                            <span className="font-mono text-gray-500 truncate block">{col}</span>
                          </td>
                          <td className="px-1 py-1.5">
                            <AutocompleteInput
                              className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
                                override ? 'border-amber-300 bg-amber-50 text-amber-800' : 'border-gray-200 text-gray-600'
                              }`}
                              value={override}
                              placeholder={placeholder}
                              onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
                              onEnter={handleSaveOverrides}
                              suggestions={suggestions}
                            />
                          </td>
                          <td className="pr-2 text-center w-6">
                            {override && (
                              <button
                                onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
                                className="text-gray-300 hover:text-red-400 leading-none text-base">×</button>
                            )}
                          </td>
                        </tr>
                      )
                    })}
                    {extraCols.map((col, i) => {
                      const val = overrideDraft[col] ?? ''
                      const suggestions = [...(globalValues[col] || [])].sort()
                      return (
                        <tr key={`extra-${i}`} className="border-t border-gray-50">
                          <td className="px-3 py-1.5 w-28 shrink-0">
                            <input
                              className="w-full text-xs font-mono border border-gray-200 rounded px-1 py-0.5 focus:outline-none focus:border-blue-400"
                              value={col}
                              placeholder="field name"
                              onChange={e => {
                                const newName = e.target.value
                                setExtraCols(ec => { const c = [...ec]; c[i] = newName; return c })
                                if (val) setOverrideDraft(d => {
                                  const n = { ...d }
                                  delete n[col]
                                  if (newName) n[newName] = val
                                  return n
                                })
                              }}
                            />
                          </td>
                          <td className="px-1 py-1.5">
                            <AutocompleteInput
                              className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
                                val ? 'border-amber-300 bg-amber-50 text-amber-800' : 'border-gray-200 text-gray-600'
                              }`}
                              value={val}
                              onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
                              onEnter={handleSaveOverrides}
                              suggestions={suggestions}
                            />
                          </td>
                          <td className="pr-2 text-center w-6">
                            {val && (
                              <button
                                onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
                                className="text-gray-300 hover:text-red-400 leading-none text-base">×</button>
                            )}
                          </td>
                        </tr>
                      )
                    })}
                  </tbody>
                </table>
              </div>
              <div className="flex gap-2 px-3 py-2 border-t border-gray-100 shrink-0">
                <button
                  onClick={handleSaveOverrides}
                  disabled={panelSaving || !isDirty}
                  className="flex-1 text-xs bg-blue-600 text-white rounded px-3 py-1.5 hover:bg-blue-700 disabled:opacity-40">
                  {panelSaving ? 'Saving…' : 'Save'}
                </button>
                {selectedRecord.overrides && Object.keys(selectedRecord.overrides).length > 0 && (
                  <button
                    onClick={handleClearOverrides}
                    disabled={panelSaving}
                    className="text-xs border border-gray-200 rounded px-3 py-1.5 text-gray-500 hover:border-red-300 hover:text-red-500 disabled:opacity-40">
                    Clear
                  </button>
                )}
              </div>
            </div>
          )}
        </div>
      )}
    </div>
  )
 }
--- a/ui/src/pages/Rules.jsx
+++ b/ui/src/pages/Rules.jsx
@ -213,7 +213,7 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
  )
 }
-export default function Rules({ source }) {
+export default function Rules({ source, onStale }) {
  const [rules, setRules] = useState([])
  const [creating, setCreating] = useState(false)
  const [editing, setEditing] = useState(null)
@ -266,6 +266,7 @@ export default function Rules({ source }) {
      } else {
        await api.createRule({ ...form, source_name: source })
      }
      onStale?.(source)
      const updated = await api.getRules(source)
      setRules(updated)
      setCreating(false)
@ -282,6 +283,7 @@ export default function Rules({ source }) {
    if (!confirm('Delete this rule and all its mappings?')) return
    try {
      await api.deleteRule(id)
      onStale?.(source)
      setRules(r => r.filter(x => x.id !== id))
      setTestResults(t => { const n = { ...t }; delete n[id]; return n })
    } catch (err) {
@ -301,6 +303,7 @@ export default function Rules({ source }) {
  async function handleToggle(rule) {
    try {
      await api.updateRule(rule.id, { enabled: !rule.enabled })
      onStale?.(source)
      setRules(r => r.map(x => x.id === rule.id ? { ...x, enabled: !x.enabled } : x))
    } catch (err) {
      alert(err.message)
--- a/ui/src/pages/Stacks.jsx
+++ b/ui/src/pages/Stacks.jsx
@ -0,0 +1,830 @@
 import { useState, useEffect, useRef } from 'react'
 import { api } from '../api'
 import { format as formatSql } from 'sql-formatter'
 function prettySql(sql) {
  try {
    return formatSql(sql, { language: 'postgresql', tabWidth: 4, keywordCase: 'upper' })
  } catch {
    return sql
  }
 }
 const FIELD_TYPES = ['text', 'numeric', 'date']
 // ── Calibrate modal ────────────────────────────────────────────────────────────
 function fmt(n) {
  return Number(n).toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })
 }
 function CalibrateModal({ stack, sourceName, currentOffset, onClose, onApply }) {
  const [asOf, setAsOf] = useState('')
  const [known, setKnown] = useState('')
  const [computed, setComputed] = useState(null)  // raw sum from DB (no offset)
  const [loading, setLoading] = useState(false)
  const [error, setError] = useState('')
  const [applyOffset, setApplyOffset] = useState('')
  const debounceRef = useRef(null)
  const knownNum = parseFloat(known)
  const hasKnown = known !== '' && !isNaN(knownNum)
  const plug = hasKnown && computed !== null ? knownNum - computed : null
  // Auto-fetch computed sum on mount (all transactions) and whenever date changes
  useEffect(() => {
    clearTimeout(debounceRef.current)
    debounceRef.current = setTimeout(async () => {
      setLoading(true); setError('')
      try {
        const r = await api.calibrateBalance(stack.name, sourceName, { as_of_date: asOf || null, known_balance: 0 })
        if (r.success) setComputed(Number(r.computed_sum))
        else setError(r.error)
      } catch (e) { setError(e.message) }
      finally { setLoading(false) }
    }, asOf ? 400 : 0)
    return () => clearTimeout(debounceRef.current)
  }, [asOf])
  // Keep applyOffset in sync with plug
  useEffect(() => {
    if (plug !== null) setApplyOffset(plug.toFixed(2))
  }, [plug])
  return (
    <div className="fixed inset-0 bg-black/40 flex items-center justify-center z-50" onMouseDown={e => { if (e.target === e.currentTarget) onClose() }}>
      <div className="bg-white rounded-lg shadow-xl w-[420px] p-5" onClick={e => e.stopPropagation()}>
        <div className="flex items-center justify-between mb-4">
          <span className="text-sm font-semibold text-gray-700">Calibrate — {sourceName}</span>
          <button onClick={onClose} className="text-gray-400 hover:text-gray-600">✕</button>
        </div>
        {/* Date */}
        <div className="mb-4">
          <label className="text-xs text-gray-500 block mb-1">As-of date</label>
          <input type="date" className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
            value={asOf} onChange={e => setAsOf(e.target.value)} />
        </div>
        {/* Reconciliation table */}
        <div className="bg-gray-50 rounded border border-gray-200 mb-4 text-sm">
          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
            <span className="text-gray-500 text-xs">Data sum at date</span>
            <span className="font-mono text-gray-700">
              {loading ? <span className="text-gray-300">…</span> : computed !== null ? fmt(computed) : <span className="text-gray-300">—</span>}
            </span>
          </div>
          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
            <span className="text-gray-500 text-xs">Known balance</span>
            <input
              type="number" step="0.01"
              className="font-mono text-right bg-transparent border-0 focus:outline-none w-36 text-sm text-gray-700 placeholder-gray-300"
              placeholder="enter balance"
              value={known} onChange={e => setKnown(e.target.value)}
            />
          </div>
          <div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
            <span className="text-gray-500 text-xs">Current offset</span>
            <span className="font-mono text-gray-400">{fmt(currentOffset ?? 0)}</span>
          </div>
          <div className="flex items-center justify-between px-3 py-2 font-medium">
            <span className="text-gray-700 text-xs">Plug (offset needed)</span>
            <span className={`font-mono ${plug !== null ? 'text-blue-700' : 'text-gray-300'}`}>
              {plug !== null ? fmt(plug) : '—'}
            </span>
          </div>
        </div>
        {error && <p className="text-xs text-red-500 mb-3">{error}</p>}
        {/* Apply */}
        <div className="flex gap-2 items-center">
          <input type="number" step="0.01"
            className="flex-1 border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
            placeholder="offset to apply"
            value={applyOffset} onChange={e => setApplyOffset(e.target.value)} />
          <button onClick={() => onApply(parseFloat(applyOffset))} disabled={applyOffset === '' || isNaN(parseFloat(applyOffset))}
            className="text-sm bg-green-600 text-white px-4 py-1.5 rounded hover:bg-green-700 disabled:opacity-40">
            Apply
          </button>
        </div>
      </div>
    </div>
  )
 }
 // ── Stack panel ────────────────────────────────────────────────────────────────
 function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSqlGenerated }) {
  const members = stack.sources || []
  const [label, setLabel] = useState(stack.label || '')
  const [fields, setFields] = useState(stack.fields || [])
  const [newField, setNewField] = useState({ name: '', type: 'text' })
  const [addingSrc, setAddingSrc] = useState('')
  // Per-source config: sign, offset, amount_field, date_field, field_map
  const [srcCfg, setSrcCfg] = useState(() =>
    Object.fromEntries(members.map(m => [m.source_name, {
      sign:         m.amount_sign ?? 1,
      offset:       m.balance_offset ?? 0,
      amount_field: m.amount_field || '',
      date_field:   m.date_field || '',
      field_map:    { ...(m.field_map || {}) },
    }]))
  )
  // Available columns from each source's dfv view
  const [srcFields, setSrcFields] = useState({})
  // Drag-to-reorder state
  const [dragIdx, setDragIdx] = useState(null)
  const [dragOverIdx, setDragOverIdx] = useState(null)
  const [srcDragIdx, setSrcDragIdx] = useState(null)
  const [srcDragOverIdx, setSrcDragOverIdx] = useState(null)
  // Calibrate
  const [calibratingSource, setCalibratingSource] = useState(null)
  // View / balance
  const [viewResult, setViewResult] = useState(null)
  const [netBalance, setNetBalance] = useState(null)
  const [balanceError, setBalanceError] = useState('')
  const [saving, setSaving] = useState(false)
  const [mappingsDirty, setMappingsDirty] = useState(false)
  const [error, setError] = useState('')
  // Fetch source columns whenever members change
  useEffect(() => {
    members.forEach(m => {
      api.getFields(m.source_name)
        .then(f => setSrcFields(prev => ({ ...prev, [m.source_name]: f.map(x => x.key) })))
        .catch(() => {})
    })
  }, [members.map(m => m.source_name).join(',')])
  // Live SQL preview — debounced; syncs current UI state to DB first so preview is accurate
  const previewTimer = useRef(null)
  useEffect(() => {
    clearTimeout(previewTimer.current)
    previewTimer.current = setTimeout(async () => {
      try {
        for (const m of members) {
          const cfg = srcCfg[m.source_name] || {}
          await api.upsertStackSource(stack.name, m.source_name, {
            field_map:      cfg.field_map    || {},
            amount_sign:    cfg.sign         ?? 1,
            balance_offset: cfg.offset       ?? 0,
            amount_field:   cfg.amount_field || null,
            date_field:     cfg.date_field   || null,
          })
        }
        await api.updateStack(stack.name, {
          fields,
          amount_field: amountCanonical || null,
          date_field:   dateCanonical   || null,
        })
        const r = await api.previewStackSql(stack.name)
        if (r.success) onSqlGenerated?.(r.sql)
      } catch {}
    }, 600)
    return () => clearTimeout(previewTimer.current)
  }, [
    JSON.stringify(fields),
    JSON.stringify(srcCfg),
    members.map(m => m.source_name).join(','),
  ])
  // Auto-detect canonical amount/date field from field types
  const amountCanonical = fields.find(f => f.type === 'numeric')?.name || stack.amount_field
  const dateCanonical   = fields.find(f => f.type === 'date')?.name   || stack.date_field
  // ── Label ──
  async function saveLabel() {
    setSaving(true); setError('')
    try { await api.updateStack(stack.name, { label }); onUpdated() }
    catch (e) { setError(e.message) }
    finally { setSaving(false) }
  }
  // ── Fields ──
  async function addField() {
    if (!newField.name) return
    const updated = [...fields, { name: newField.name, type: newField.type }]
    setFields(updated)
    setNewField({ name: '', type: 'text' })
    await api.updateStack(stack.name, { fields: updated })
    onUpdated()
  }
  async function removeField(name) {
    const updated = fields.filter(f => f.name !== name)
    setFields(updated)
    await api.updateStack(stack.name, { fields: updated })
    onUpdated()
  }
  // ── Drag reorder ──
  function handleDragStart(e, idx) {
    setDragIdx(idx)
    e.dataTransfer.effectAllowed = 'move'
  }
  function handleDragOver(e, idx) {
    e.preventDefault()
    setDragOverIdx(idx)
  }
  async function handleDrop(e, toIdx) {
    e.preventDefault()
    if (dragIdx === null || dragIdx === toIdx) { setDragIdx(null); setDragOverIdx(null); return }
    const updated = [...fields]
    const [moved] = updated.splice(dragIdx, 1)
    updated.splice(toIdx, 0, moved)
    setFields(updated)
    setDragIdx(null); setDragOverIdx(null)
    await api.updateStack(stack.name, { fields: updated })
    onUpdated()
  }
  // ── Source drag-to-reorder ──
  function handleSrcDragStart(e, idx) {
    setSrcDragIdx(idx)
    e.dataTransfer.effectAllowed = 'move'
  }
  function handleSrcDragOver(e, idx) {
    e.preventDefault()
    setSrcDragOverIdx(idx)
  }
  async function handleSrcDrop(e, toIdx) {
    e.preventDefault()
    if (srcDragIdx === null || srcDragIdx === toIdx) { setSrcDragIdx(null); setSrcDragOverIdx(null); return }
    const updated = [...members]
    const [moved] = updated.splice(srcDragIdx, 1)
    updated.splice(toIdx, 0, moved)
    setSrcDragIdx(null); setSrcDragOverIdx(null)
    await api.reorderStackSources(stack.name, updated.map(m => m.source_name))
    onUpdated()
  }
  // ── Mapping grid ──
  function getMappingValue(srcName, canonicalName) {
    const cfg = srcCfg[srcName] || {}
    if (canonicalName === amountCanonical) return cfg.amount_field || ''
    if (canonicalName === dateCanonical)   return cfg.date_field   || ''
    return cfg.field_map?.[canonicalName] || ''
  }
  function setMappingValue(srcName, canonicalName, value) {
    setSrcCfg(prev => {
      const cfg = { ...prev[srcName] }
      if (canonicalName === amountCanonical)      cfg.amount_field = value
      else if (canonicalName === dateCanonical)   cfg.date_field   = value
      else cfg.field_map = { ...cfg.field_map, [canonicalName]: value }
      return { ...prev, [srcName]: cfg }
    })
    setMappingsDirty(true)
  }
  function setSrcSign(srcName, sign) {
    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], sign } }))
    setMappingsDirty(true)
  }
  function setSrcOffset(srcName, offset) {
    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], offset } }))
    setMappingsDirty(true)
  }
  async function saveMappings() {
    setSaving(true); setError('')
    try {
      for (const m of members) {
        const cfg = srcCfg[m.source_name] || {}
        await api.upsertStackSource(stack.name, m.source_name, {
          field_map:    cfg.field_map    || {},
          amount_sign:  cfg.sign         ?? 1,
          balance_offset: cfg.offset     ?? 0,
          amount_field: cfg.amount_field || null,
          date_field:   cfg.date_field   || null,
        })
      }
      // Persist the auto-detected canonical field names on the stack
      await api.updateStack(stack.name, {
        amount_field: amountCanonical || null,
        date_field:   dateCanonical   || null,
      })
      setMappingsDirty(false)
      onStale?.(stack.name)
      onUpdated()
    } catch (e) { setError(e.message) }
    finally { setSaving(false) }
  }
  // ── Sources ──
  async function addSource() {
    if (!addingSrc) return
    await api.upsertStackSource(stack.name, addingSrc, { field_map: {}, amount_sign: 1 })
    setSrcCfg(prev => ({ ...prev, [addingSrc]: { sign: 1, offset: 0, amount_field: '', date_field: '', field_map: {} } }))
    // Load fields immediately so dropdowns are ready
    try {
      const f = await api.getFields(addingSrc)
      setSrcFields(prev => ({ ...prev, [addingSrc]: f.map(x => x.key) }))
    } catch (e) {}
    setAddingSrc('')
    onStale?.(stack.name)
    onUpdated()
  }
  function handleSrcAmountField(srcName, value) {
    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], amount_field: value } }))
    // Update column type to numeric if a column with this name exists
    setFields(prev => prev.map(f => f.name === value ? { ...f, type: 'numeric' } : f))
    setMappingsDirty(true)
    maybeAutoPopulate(srcName, value, srcCfg[srcName]?.date_field)
  }
  function handleSrcDateField(srcName, value) {
    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], date_field: value } }))
    setFields(prev => prev.map(f => f.name === value ? { ...f, type: 'date' } : f))
    setMappingsDirty(true)
    maybeAutoPopulate(srcName, srcCfg[srcName]?.amount_field, value)
  }
  function maybeAutoPopulate(srcName, amtField, dtField) {
    if (!amtField || !dtField) return
    if (fields.length > 0) return  // don't overwrite existing columns
    const sourceFields = srcFields[srcName] || []
    if (sourceFields.length === 0) return
    const newFields = sourceFields.map(sf => ({
      name: sf,
      type: sf === amtField ? 'numeric' : sf === dtField ? 'date' : 'text',
    }))
    setFields(newFields)
    api.updateStack(stack.name, { fields: newFields, amount_field: amtField, date_field: dtField })
  }
  async function removeSource(src) {
    await api.removeStackSource(stack.name, src)
    setSrcCfg(prev => { const n = { ...prev }; delete n[src]; return n })
    onStale?.(stack.name)
    onUpdated()
  }
  async function handleCalibrate(srcName) {
    // Save this source's current config before opening modal
    const cfg = srcCfg[srcName] || {}
    await api.upsertStackSource(stack.name, srcName, {
      field_map:     cfg.field_map    || {},
      amount_sign:   cfg.sign         ?? 1,
      balance_offset: cfg.offset      ?? 0,
      amount_field:  cfg.amount_field || null,
      date_field:    cfg.date_field   || null,
    })
    setCalibratingSource(srcName)
  }
  async function applyCalibration(srcName, offset) {
    const cfg = srcCfg[srcName] || {}
    await api.upsertStackSource(stack.name, srcName, {
      field_map:     cfg.field_map    || {},
      amount_sign:   cfg.sign         ?? 1,
      balance_offset: offset,
      amount_field:  cfg.amount_field || null,
      date_field:    cfg.date_field   || null,
    })
    setSrcCfg(prev => ({ ...prev, [srcName]: { ...prev[srcName], offset } }))
    setCalibratingSource(null)
    onStale?.(stack.name)
    onUpdated()
  }
  // ── View ──
  async function generateView() {
    setViewResult(null); setNetBalance(null); setBalanceError('')
    try {
      const r = await api.generateStackView(stack.name)
      setViewResult(r)
      if (r.success) {
        fetchBalance()
        onViewGenerated?.(stack.name)
        onSqlGenerated?.(r.sql || '')
        ;(r.cascade_stale || []).forEach(n => onStale?.(n))
      }
    } catch (e) { setError(e.message) }
  }
  async function fetchBalance() {
    setBalanceError('')
    try {
      const r = await api.getStackBalance(stack.name)
      if (r.success) setNetBalance(r.balance)
      else setBalanceError(r.error)
    } catch (e) { setBalanceError(e.message) }
  }
  const availableSources = sources.filter(s => !members.find(m => m.source_name === s.name))
  if (addingSrc === '' && availableSources.length === 1) setAddingSrc(availableSources[0].name)
  return (
    <div className="space-y-5">
      {/* Label */}
      <div className="bg-white border border-gray-200 rounded p-4">
        <h3 className="text-sm font-semibold text-gray-700 mb-3">Configuration</h3>
        <div className="flex gap-3 items-end">
          <div className="flex-1">
            <label className="text-xs text-gray-500 block mb-1">Label <span className="text-gray-400">(optional)</span></label>
            <input className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
              value={label} onChange={e => setLabel(e.target.value)}
              onKeyDown={e => e.key === 'Enter' && saveLabel()} />
          </div>
          <button onClick={saveLabel} disabled={saving}
            className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
            {saving ? 'Saving…' : 'Save'}
          </button>
        </div>
        {error && <p className="text-xs text-red-500 mt-2">{error}</p>}
      </div>
      {/* Sources */}
      <div className="bg-white border border-gray-200 rounded p-4">
        <h3 className="text-sm font-semibold text-gray-700 mb-1">Sources</h3>
        <p className="text-xs text-gray-400 mb-3">Each source contributes rows to the combined view. Set the sign to flip the direction of amounts (e.g. credit card charges are positive in the source but should subtract from your balance). The offset adjusts the running balance — use Calibrate to compute it from a known good balance.</p>
        <div className="space-y-2 mb-3">
          {members.map((m, idx) => {
            const cfg = srcCfg[m.source_name] || {}
            const sf = srcFields[m.source_name] || []
            const canCalibrate = !!cfg.amount_field && !!cfg.date_field
            return (
              <div key={m.source_name}
                draggable
                onDragStart={e => handleSrcDragStart(e, idx)}
                onDragOver={e => handleSrcDragOver(e, idx)}
                onDrop={e => handleSrcDrop(e, idx)}
                onDragEnd={() => { setSrcDragIdx(null); setSrcDragOverIdx(null) }}
                className={`border border-gray-100 rounded px-3 py-2 text-xs space-y-2 ${srcDragOverIdx === idx && srcDragIdx !== idx ? 'bg-blue-50' : ''}`}>
                <div className="flex items-center gap-2">
                  <span className="text-gray-300 cursor-grab select-none">⠿</span>
                  <span className="font-medium text-gray-700 flex-1">{m.source_name}</span>
                  <button onClick={() => removeSource(m.source_name)} className="text-red-300 hover:text-red-500">Remove</button>
                </div>
                <div className="grid grid-cols-2 gap-x-4 gap-y-1.5">
                  <div>
                    <label className="text-gray-400 block mb-0.5">Amount field</label>
                    <select value={cfg.amount_field || ''}
                      onChange={e => handleSrcAmountField(m.source_name, e.target.value)}
                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
                      <option value="">— select —</option>
                      {sf.map(f => <option key={f} value={f}>{f}</option>)}
                    </select>
                  </div>
                  <div>
                    <label className="text-gray-400 block mb-0.5">Sign</label>
                    <select value={cfg.sign ?? 1}
                      onChange={e => { setSrcSign(m.source_name, parseInt(e.target.value)); setMappingsDirty(true) }}
                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
                      <option value={1}>+1 (as-is)</option>
                      <option value={-1}>−1 (flip)</option>
                    </select>
                  </div>
                  <div>
                    <label className="text-gray-400 block mb-0.5">Date field</label>
                    <select value={cfg.date_field || ''}
                      onChange={e => handleSrcDateField(m.source_name, e.target.value)}
                      className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
                      <option value="">— select —</option>
                      {sf.map(f => <option key={f} value={f}>{f}</option>)}
                    </select>
                  </div>
                  <div>
                    <label className="text-gray-400 block mb-0.5">Balance offset</label>
                    <div className="flex items-center gap-1">
                      <input type="number" step="0.01" value={cfg.offset ?? 0}
                        onChange={e => { setSrcOffset(m.source_name, parseFloat(e.target.value) || 0); setMappingsDirty(true) }}
                        className="flex-1 border border-gray-200 rounded px-1.5 py-0.5 font-mono focus:outline-none focus:border-blue-400" />
                      <button onClick={() => handleCalibrate(m.source_name)}
                        disabled={!canCalibrate}
                        title={!canCalibrate ? 'Set amount and date fields first' : 'Calibrate balance'}
                        className="text-blue-400 hover:text-blue-600 underline disabled:opacity-40 disabled:cursor-not-allowed disabled:no-underline">
                        Calibrate
                      </button>
                    </div>
                  </div>
                </div>
              </div>
            )
          })}
          {members.length === 0 && <p className="text-xs text-gray-400">No sources added yet.</p>}
        </div>
        {availableSources.length > 0 && (
          <div className="flex gap-2">
            <select className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
              value={addingSrc} onChange={e => setAddingSrc(e.target.value)}>
              <option value="">— add source —</option>
              {availableSources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
            </select>
            <button onClick={addSource} disabled={!addingSrc}
              className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700 disabled:opacity-40">Add</button>
          </div>
        )}
      </div>
      {/* Output columns mapping grid */}
      <div className="bg-white border border-gray-200 rounded p-4">
        <h3 className="text-sm font-semibold text-gray-700 mb-1">Output columns</h3>
        <p className="text-xs text-gray-400 mb-3">
          Each row is a column in the combined view. Each source column shows which field from that source maps to it.
          The first <span className="text-blue-500">numeric</span> field drives the running balance; the first <span className="text-green-600">date</span> field drives the ordering.
          Both <span className="font-mono">source_balance</span> (per-source) and <span className="font-mono">net_balance</span> (combined) are always included in the generated view.
          Drag rows to reorder.
        </p>
        {members.length === 0 ? (
          <p className="text-xs text-gray-400 mb-3">Add sources above first.</p>
        ) : (
          <div className="overflow-x-auto mb-3">
            <table className="w-full text-xs border-collapse">
              <thead>
                <tr className="border-b border-gray-200">
                  <th className="w-5 pb-2"></th>
                  <th className="text-left text-gray-400 font-normal pb-2 pr-4">Column</th>
                  <th className="text-left text-gray-400 font-normal pb-2 pr-4">Type</th>
                  {members.map(m => (
                    <th key={m.source_name} className="text-left text-gray-400 font-normal pb-2 pr-3 min-w-36">{m.source_name}</th>
                  ))}
                  <th className="w-5 pb-2"></th>
                </tr>
              </thead>
              <tbody>
                {fields.map((f, idx) => {
                  const isAmount = f.name === amountCanonical
                  const isDate   = f.name === dateCanonical
                  return (
                    <tr key={f.name}
                      draggable
                      onDragStart={e => handleDragStart(e, idx)}
                      onDragOver={e => handleDragOver(e, idx)}
                      onDrop={e => handleDrop(e, idx)}
                      onDragEnd={() => { setDragIdx(null); setDragOverIdx(null) }}
                      className={`border-b border-gray-50 ${dragOverIdx === idx && dragIdx !== idx ? 'bg-blue-50' : ''}`}>
                      <td className="py-1.5 pr-1 text-gray-300 cursor-grab select-none">⠿</td>
                      <td className="py-1.5 pr-4 font-mono text-gray-700 whitespace-nowrap">
                        {f.name}
                        {isAmount && <span className="ml-1.5 text-blue-500 font-sans font-normal">amount</span>}
                        {isDate   && <span className="ml-1.5 text-green-600 font-sans font-normal">date</span>}
                      </td>
                      <td className="py-1.5 pr-4 text-gray-400">{f.type}</td>
                      {members.map(m => (
                        <td key={m.source_name} className="py-1.5 pr-3">
                          <div className="flex items-center gap-1">
                            <select
                              value={getMappingValue(m.source_name, f.name)}
                              onChange={e => setMappingValue(m.source_name, f.name, e.target.value)}
                              className="border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400 min-w-0 flex-1">
                              <option value="">— same name —</option>
                              {(srcFields[m.source_name] || []).map(sf => (
                                <option key={sf} value={sf}>{sf}</option>
                              ))}
                            </select>
                          </div>
                        </td>
                      ))}
                      <td className="py-1.5">
                        <button onClick={() => removeField(f.name)} className="text-red-300 hover:text-red-500">✕</button>
                      </td>
                    </tr>
                  )
                })}
                {fields.length === 0 && (
                  <tr><td colSpan={3 + members.length} className="py-3 text-gray-400 text-center">No columns defined yet — add one below.</td></tr>
                )}
              </tbody>
            </table>
          </div>
        )}
        {/* Add field */}
        <div className="flex gap-2 mb-3">
          <input className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
            placeholder="column name" value={newField.name}
            onChange={e => setNewField(f => ({ ...f, name: e.target.value }))}
            onKeyDown={e => e.key === 'Enter' && addField()} />
          <select className="border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
            value={newField.type} onChange={e => setNewField(f => ({ ...f, type: e.target.value }))}>
            {FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
          </select>
          <button onClick={addField} className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700">Add</button>
        </div>
        {mappingsDirty && (
          <button onClick={saveMappings} disabled={saving}
            className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
            {saving ? 'Saving…' : 'Save mappings'}
          </button>
        )}
      </div>
      {/* Generate view + balance */}
      <div className="bg-white border border-gray-200 rounded p-4">
        <div className="flex items-center justify-between mb-3">
          <h3 className="text-sm font-semibold text-gray-700">View</h3>
          <div className="flex gap-2">
            <button onClick={fetchBalance}
              className="text-sm bg-gray-100 text-gray-700 px-3 py-1.5 rounded hover:bg-gray-200">
              Refresh balance
            </button>
            <button onClick={generateView}
              className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700">
              Generate / refresh
            </button>
          </div>
        </div>
        {netBalance !== null && (
          <div className="mb-3 flex items-center gap-3">
            <span className="text-xs text-gray-500">Current net balance</span>
            <span className="text-lg font-mono font-semibold text-gray-800">
              {Number(netBalance).toLocaleString(undefined, { minimumFractionDigits: 2 })}
            </span>
          </div>
        )}
        {balanceError && <p className="text-xs text-gray-400 mb-3">{balanceError}</p>}
        {viewResult && !viewResult.success && (
          <p className="text-xs text-red-500">{viewResult.error}</p>
        )}
        {viewResult && viewResult.success && (
          <p className="text-xs text-green-600">View created: <span className="font-mono">{viewResult.view}</span></p>
        )}
      </div>
      {calibratingSource && (
        <CalibrateModal
          stack={stack}
          sourceName={calibratingSource}
          currentOffset={srcCfg[calibratingSource]?.offset ?? 0}
          onClose={() => setCalibratingSource(null)}
          onApply={offset => applyCalibration(calibratingSource, offset)}
        />
      )}
    </div>
  )
 }
 // ── Main page ──────────────────────────────────────────────────────────────────
 export default function Stacks({ sources, onStackStale, onStackViewGenerated, onStacksChange }) {
  const [stacks, setStacks] = useState([])
  const [selected, setSelected] = useState(null)
  const [stackDetail, setStackDetail] = useState(null)
  const [creating, setCreating] = useState(false)
  const [newName, setNewName] = useState('')
  const [error, setError] = useState('')
  const [sqlDraft, setSqlDraft] = useState('')
  const [sqlRunning, setSqlRunning] = useState(false)
  const [sqlResult, setSqlResult] = useState(null)
  async function load() {
    const s = await api.getStacks()
    setStacks(s)
    return s
  }
  async function loadDetail(name) {
    const s = await api.getStack(name)
    setStackDetail(s)
    setSelected(name)
    localStorage.setItem('stacks_last_selected', name)
    setSqlDraft('')
    setSqlResult(null)
  }
  useEffect(() => {
    load().then(s => {
      const last = localStorage.getItem('stacks_last_selected')
      if (last && s.find(x => x.name === last)) loadDetail(last)
    })
  }, [])
  useEffect(() => { if (selected) loadDetail(selected) }, [selected])
  async function createStack() {
    if (!newName) return
    setError('')
    try {
      await api.createStack({ name: newName, fields: [] })
      setNewName(''); setCreating(false)
      await load()
      onStacksChange?.()
      loadDetail(newName)
    } catch (e) { setError(e.message) }
  }
  async function deleteStack(name) {
    if (!confirm(`Delete stack "${name}"?`)) return
    await api.deleteStack(name)
    if (selected === name) { setSelected(null); setStackDetail(null); setSqlDraft(''); setSqlResult(null) }
    load()
    onStacksChange?.()
  }
  async function runSql() {
    if (!sqlDraft.trim() || !selected) return
    setSqlRunning(true); setSqlResult(null)
    try {
      const r = await api.execStackSql(selected, sqlDraft)
      setSqlResult(r)
      if (r.success) {
        onStackViewGenerated?.(selected)
        ;(r.cascade_stale || []).forEach(n => onStackStale?.(n))
      }
    } catch (e) { setSqlResult({ success: false, error: e.message }) }
    finally { setSqlRunning(false) }
  }
  return (
    <div className="p-6">
      {/* Stack list — horizontal row of cards */}
      <div className="flex items-center gap-2 mb-5 flex-wrap">
        <h1 className="text-sm font-semibold text-gray-800 mr-1">Stacks</h1>
        {stacks.map(s => (
          <div key={s.name}
            onClick={() => loadDetail(s.name)}
            className={`flex items-center gap-2 px-3 py-1.5 rounded border cursor-pointer text-xs group transition-colors ${selected === s.name ? 'border-blue-300 bg-blue-50 text-blue-700' : 'border-gray-200 bg-white text-gray-600 hover:border-gray-300 hover:bg-gray-50'}`}>
            <span className="font-medium">{s.label || s.name}</span>
            <span className="text-gray-400">{s.source_count}s</span>
            <button onClick={e => { e.stopPropagation(); deleteStack(s.name) }}
              className="opacity-0 group-hover:opacity-100 text-red-300 hover:text-red-500 leading-none">✕</button>
          </div>
        ))}
        {creating ? (
          <div className="flex items-center gap-1">
            <input autoFocus className="border border-blue-400 rounded px-2 py-1 text-xs focus:outline-none w-32"
              placeholder="stack name" value={newName} onChange={e => setNewName(e.target.value)}
              onKeyDown={e => { if (e.key === 'Enter') createStack(); if (e.key === 'Escape') setCreating(false) }} />
            <button onClick={createStack} className="text-xs bg-blue-600 text-white px-2 py-1 rounded">Create</button>
            <button onClick={() => setCreating(false)} className="text-xs text-gray-400 px-1">✕</button>
            {error && <p className="text-xs text-red-500">{error}</p>}
          </div>
        ) : (
          <button onClick={() => setCreating(true)} className="text-xs text-blue-500 hover:text-blue-700 px-2 py-1.5">+ New</button>
        )}
      </div>
      {stackDetail ? (
        <div className="flex gap-6 items-start">
          {/* Left: config panel */}
          <div className="flex-1 min-w-0">
            <h2 className="text-base font-semibold text-gray-800 mb-4">
              {stackDetail.label || stackDetail.name}
              {stackDetail.label && <span className="text-sm text-gray-400 font-normal ml-2">{stackDetail.name}</span>}
            </h2>
            <StackPanel
              key={stackDetail.name}
              stack={stackDetail}
              sources={sources}
              onUpdated={() => { load(); loadDetail(stackDetail.name) }}
              onStale={onStackStale}
              onViewGenerated={onStackViewGenerated}
              onSqlGenerated={sql => { setSqlDraft(prettySql(sql)); setSqlResult(null) }}
            />
          </div>
          {/* Right: SQL panel */}
          <div className="flex-1 min-w-0">
            <div className="bg-white border border-gray-200 rounded p-4 sticky top-4">
              <div className="flex items-center justify-between mb-3">
                <h3 className="text-sm font-semibold text-gray-700">Generated SQL</h3>
                <button
                  onClick={runSql}
                  disabled={!sqlDraft.trim() || sqlRunning}
                  className="text-sm bg-blue-600 text-white px-3 py-1 rounded hover:bg-blue-700 disabled:opacity-40">
                  {sqlRunning ? 'Running…' : 'Run'}
                </button>
              </div>
              {sqlDraft ? (
                <textarea
                  className="w-full font-mono text-xs text-gray-700 bg-gray-50 border border-gray-200 rounded p-2 focus:outline-none focus:border-blue-400 resize-none leading-relaxed"
                  style={{ minHeight: '60vh' }}
                  value={sqlDraft}
                  onChange={e => { setSqlDraft(e.target.value); setSqlResult(null) }}
                  spellCheck={false}
                />
              ) : (
                <p className="text-xs text-gray-400">Generate a view to see the SQL here.</p>
              )}
              {sqlResult && (
                <p className={`text-xs mt-2 ${sqlResult.success ? 'text-green-600' : 'text-red-500'}`}>
                  {sqlResult.success ? 'View updated successfully.' : sqlResult.error}
                </p>
              )}
            </div>
          </div>
        </div>
      ) : (
        <p className="text-sm text-gray-400">Select a stack or create one.</p>
      )}
    </div>
  )
 }
--- a/ui/src/theme.jsx
+++ b/ui/src/theme.jsx
@ -0,0 +1,25 @@
 import { createContext, useContext, useState, useEffect } from 'react'
 const ThemeContext = createContext()
 export function ThemeProvider({ children }) {
  const [dark, setDark] = useState(() => {
    const saved = localStorage.getItem('df_dark')
    if (saved !== null) return saved === 'true'
    return window.matchMedia('(prefers-color-scheme: dark)').matches
  })
  useEffect(() => {
    localStorage.setItem('df_dark', dark)
    document.documentElement.classList.toggle('dark', dark)
  }, [dark])
  return (
    <ThemeContext.Provider value={{ dark, setDark }}>
      {children}
    </ThemeContext.Provider>
  )
 }
 const useTheme = () => useContext(ThemeContext)
 export default useTheme
Author	SHA1	Message	Date
Paul Trowbridge	442c38d3c4	Add PERSPECTIVE.md documenting the @perspective-dev version pairing Records why the 4.5.1 viewer/client + 4.4.1 d3fc pairing is deliberate, not a skew to "fix": the /inline and /themes entrypoints exist only in 4.5.x, while viewer-d3fc caps at 4.4.1, so this is the only combination that keeps both inline WASM bundling and the d3fc charts. Verified by build failure when pinning all four to 4.4.1. Points to the canonical guide in pf_app for shared rationale. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 23:53:31 -04:00
Paul Trowbridge	efa65d8409	Update all docs to reflect current state - perspective-pivot.md: npm install pattern, v4.5.1/v4.4.1 versions - README.md: Node 18+, port 3020, add stacks routes, fix project structure - SPEC.md: add stacks/status routes, pages, SQL functions; update Perspective version - ui/README.md: replace Vite boilerplate with project-specific content - Remove docs/refactor-transformed-split.md (completed work) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 23:51:00 -04:00
Paul Trowbridge	0ece53e7be	Fix pg deprecation warning: set search_path via connection options Replace pool.on('connect') query with connection-level options parameter. Avoids calling client.query() during handshake, which pg will remove in v9. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 23:36:27 -04:00
Paul Trowbridge	317791341c	Bump major dependencies: express 5, csv-parse 6, dotenv 17, multer 2 All APIs compatible with existing code. Added quiet:true to dotenv config to suppress the new startup log message added in dotenv 17. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 23:35:15 -04:00
Paul Trowbridge	60924d03b5	Update patch/minor dependencies; skip major version bumps UI: vite 8.0.16, react 19.2.7, react-router-dom 7.17.0, tailwindcss 4.3.1, @vitejs/plugin-react 6.0.2, eslint-plugin-react-hooks 7.1.1, sql-formatter 15.8.1 API: pg 8.21.0 Skipped: eslint 10, express 5, csv-parse 6, dotenv 17, multer 2 (majors) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 23:04:25 -04:00
Paul Trowbridge	0c3cee4945	Migrate Perspective from CDN to npm; upgrade to 4.5.1 Replace runtime CDN imports with static ESM imports from npm packages. Uses @perspective-dev/client and viewer inline builds (WASM embedded). Bumps all packages to 4.5.1; d3fc stays at 4.4.1 (no 4.5.x release yet). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 23:00:23 -04:00
Paul Trowbridge	89a70bdf7e	Split transformed column; add override management; show all override keys in panel - transformed now stores only rule additions (not merged data+overrides) - View dynamically computes data \|\| transformed \|\| overrides at query time - New DB functions: set/clear/bulk_set_record_overrides - Records panel now includes source-wide override keys so party/reason etc. appear even on records that don't have them set yet Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-23 11:00:24 -04:00
Paul Trowbridge	1baadaca61	Lift stack state to App; merge Records panel; fix Pivot theme on load - Stack selection lifted to App.jsx: stacks fetched on login, selectedStack state shared via StatusBar (pills) and Pivot (view switching); Stacks page calls onStacksChange to keep list fresh - Pivot: derive selectedView/viewType from props, remove local stack state; toolbar replaced with dedicated layouts sub-bar (h-9, layouts only) - Records panel: merge read-only and override sections into single field list; known cols seeded from record's transformed fields; rule-derived fields (transformed minus data) will be editable in follow-up refactor - Pivot theme: setAttribute moved to after flush() so restore() can't reset it Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-03 10:35:34 -04:00
Paul Trowbridge	9e0fa4aa7e	Add collapsable sidebar with icons; move source picker to status bar - New Sidebar component (modelled on pf_app): collapses 200px→48px via hamburger toggle, persists state to df_sidebar in localStorage; each nav item has an SVG icon with label that fades out when collapsed; user avatar + sign-out at bottom - New StatusBar component: source picker + dark-mode toggle across the top of the content area - Fix Pivot theme: setAttribute('theme') moved to after flush() so viewer.restore() can no longer reset it back to light Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 23:14:49 -04:00
Paul Trowbridge	738e1919ce	Add light/dark mode with Perspective theme sync Port light/dark mode from pf_app: ThemeProvider context, CSS custom properties (Pro Dark palette), dark overrides for Tailwind classes, and Perspective viewer theme sync in Pivot. Toggle button in sidebar header. Improve toggle icons to Feather-style stroke SVGs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 22:59:24 -04:00
Paul Trowbridge	1791bf0f0a	Store stack pivot layouts in DB; drop pivot_layouts FK pivot_layouts.source_name had a FK to sources(name) preventing stack names from being used as layout keys. Dropped the FK so any view name works. - database/migrate_pivot_layouts_drop_fk.sql: drop the FK constraint - api/routes/stacks.js: add GET/POST/DELETE /:name/layouts routes - ui/src/api.js: add getStackPivotLayouts / saveStackPivotLayout / deleteStackPivotLayout - ui/src/pages/Pivot.jsx: use DB for stack layouts instead of localStorage; collapse source/stack branches into saveLayout/deleteLayout helpers - CLAUDE.md: document pivot layout persistence pattern Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 15:19:58 -04:00
Paul Trowbridge	bef3d6d89c	CLAUDE.md: add UI section covering Pivot inspector patterns Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 10:39:25 -04:00
Paul Trowbridge	b52f5c930e	Pivot inspector: toggle, resize, sort, totals, filter fix - Click cell to open inspector pane; click same cell again to close (toggle). Uses __ROW_PATH__ + column_names as key so it works on both sources and stacks. Removes event listener on view change to prevent listener accumulation. - Drag handle on left edge of inspector pane for resizing (min 240px) - Removed redundant cell-coordinates block; breadcrumb now inline in header - Sortable columns: click header to sort asc/desc with ▲/▼ indicator - Totals row: sums all-numeric columns, sticky at bottom - Derive missing split_by filters from column_names when Perspective omits them from detail.config.filter (fixes over-broad results on split_by views) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-02 10:37:47 -04:00
Paul Trowbridge	f373c85c16	Fix false-positive stale view warnings Rules/mappings changes don't affect view SQL (views read from transformed, shaped by config.fields) — remove those triggers. Replace with a BEFORE UPDATE trigger on sources that only clears view_generated_at when config actually changes. Stack sources trigger now skips no-op upserts: the live SQL preview calls upsertStackSource on every edit, which was unconditionally clearing view_generated_at even when nothing changed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-27 12:51:32 -04:00
Paul Trowbridge	e5b95e7112	Add bulk override: DB function, API route, UI select bar - bulk_set_record_overrides() DB function merges overrides into multiple records at once using a CTE with RETURNING for accurate count - POST /records/bulk-overrides calls the function (consistent with rest of API — no raw SQL in routes) - UI: regex input on loaded rows selects rows for bulk override; labeled "Bulk select:" / "DB query:" to distinguish from server-side filters Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-27 10:55:59 -04:00
Paul Trowbridge	814dcb7af1	Fix Records override save/clear: reload grid, allow blank overrides to suppress mappings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-25 12:28:35 -04:00
Paul Trowbridge	d3a423c6ad	Records override panel: read-only transformed view + Mappings-style override cols Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-25 11:13:05 -04:00
Paul Trowbridge	9ab2052f2b	Separate mapping changes from view-stale: show Reprocess prompt instead of Generate Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-25 10:28:32 -04:00
Paul Trowbridge	ca266f2839	Fix autocomplete dropdown clipped by overflow container; use fixed positioning Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-25 09:59:14 -04:00
Paul Trowbridge	5951cbbba3	Redesign Records override panel: table layout, add-new-field support Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-25 09:55:23 -04:00
Paul Trowbridge	7a7fd01285	Fix cleanLayout stripping expression columns from pivot layout save/restore Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-20 20:39:02 -04:00
Paul Trowbridge	99b7b7d721	Add migration scripts for dataflow/dcard reimport Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 21:35:28 -04:00
Paul Trowbridge	9e6d184bd8	Stacks: source ordering via seq field with drag-to-reorder - Add seq column to stack_sources; existing rows seeded by insertion order - New sources auto-assigned max(seq)+1 so they always append to the end - get_stack and generate_stack_view now order by seq instead of source_name - Add reorder_stack_sources() function and PUT /:name/sources/reorder endpoint - Source cards have drag handles matching the output columns grid behavior Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 17:11:44 -04:00
Paul Trowbridge	4e477420ad	Fix lit() truncating decimals; restore last selected stack on load - lit() was calling Math.trunc() on numbers, dropping decimals from balance_offset and any other numeric SQL params - Stacks page now saves last selected stack to localStorage and restores it on load Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 16:38:02 -04:00
Paul Trowbridge	a89bd36f40	Stacks: calibrate modal redesign, layout column cleanup, SQL preview sync - Calibrate modal now auto-fetches computed sum and shows live reconciliation table (data sum, known balance, plug) without requiring a button click - as_of_date is now optional in calibrate — omitting it sums all transactions - SQL preview syncs current UI state to DB before fetching so preview is always accurate - Pivot cleanLayout strips stale columns from saved layouts when switching stack views Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 15:36:34 -04:00
Paul Trowbridge	95e63679ef	Stacks: live SQL preview, side-by-side layout, cascade stale detection - Two-column layout: config left, SQL panel right (equal halves) - SQL panel shows formatted SQL (sql-formatter, 4-space indent) - Live preview: SQL updates 400ms after any field/source/mapping change - Run button executes edited SQL directly via new exec-sql endpoint - generate_stack_view gains p_dry_run mode for preview without executing - CASCADE drop detects dependent stacks, marks them stale in DB and status bar - net_balance moved to last column in generated view - Backfill 458 missing dcard rows and 123 missing chase rows from TPS migration bug Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 10:36:27 -04:00
Paul Trowbridge	7c63a2ac29	Status bar, stale tracking, Pivot stack selector, stack view fixes - Add get_status() SQL and /api/status route; load stale state on login - Replace polling with immediate client-side stale tracking via callbacks - Amber status bar with per-item Generate buttons for sources and stacks - Pivot: add stack selector to view any dfv.stack view via Perspective - Stack views: DROP CASCADE, add id to source views, per-source balance columns - net_balance = sum(all amounts) + total_offset guarantees chase+dcard=net per row - CLAUDE.md: document correct dedup spec (within-batch duplicates always allowed) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-19 09:43:10 -04:00
Paul Trowbridge	f941c5ae4a	Stacks: per-source amount/date fields, mapping grid UI, dfv view generation with source balance CTEs - SQL: upsert_stack_source gains amount_field/date_field params; calibrate_balance queries dfv.{source} directly (no stack view needed); generate_stack_view builds per-source CTEs with source_balance, outer net_balance; information_schema check for missing columns - API: pass amount_field/date_field through upsert route; calibrate accepts source_name - UI: mapping grid table (rows=fields, cols=sources); per-source amount/date/sign in Sources section; auto-populate output columns on first source config; horizontal stack chips above full-width config panel; calibration auto-saves before opening, editable offset input Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 21:17:12 -04:00
Paul Trowbridge	f63a0ec0e5	Stacks UI: reorder flow, balance dropdowns, current balance display, calibration editable offset Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 16:20:49 -04:00
Paul Trowbridge	ef6c6bbbb8	Add Stacks feature: multi-source union with running balance and calibration - database/queries/stacks.sql: tables, functions for create/update/delete/calibrate/generate view - api/routes/stacks.js: REST endpoints for stacks and stack sources - api/server.js: register stacks router - ui/src/api.js: stacks API methods - ui/src/App.jsx: Stacks page route and nav entry - ui/src/pages/Stacks.jsx: full UI for stack management, source mapping, calibration Note: SQL deployment pending fix for balance_offset column and calibrate_balance signature Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 15:48:42 -04:00
Paul Trowbridge	fd67bb03af	Records: fix override panel not opening Open panel immediately on row click (panelOpen state), then load full record async. Previously the panel condition depended on selectedRecord or panelLoading both of which are set after async work, so if id was missing or the API call failed the panel never appeared. Also shows a message if id is missing (view needs regeneration). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-15 21:11:28 -04:00
Paul Trowbridge	c9b830b286	Add per-record overrides that survive reprocess Schema: - records.overrides JSONB column (ALTER TABLE, already applied) - apply_transformations merges overrides on top: data \|\| rules \|\| overrides - generate_source_view always includes id and _overridden columns - set_record_overrides(id, overrides): stores and immediately merges into transformed - clear_record_overrides(id): clears overrides then reprocesses record API: - PUT /records/:id/overrides — set overrides - DELETE /records/:id/overrides — clear and reprocess UI (Records page): - Rows are clickable; overridden rows highlighted amber - Side panel shows all transformed fields as editable inputs - Overridden fields highlighted amber with pencil indicator - Save stores overrides; Clear removes them and restores computed values - id and _overridden hidden from table display Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-15 21:02:54 -04:00
		`@ -0,0 +1 @@`
							`select id, source, constrain_key, data from dataflow.records`