pt/dataflow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate documentation into docs/ and cut the duplication

Browse Source 
Architecture, file structure, the manage.py menu, and the API reference were
each documented in two or three of README.md, SPEC.md, and CLAUDE.md — the same
drift trap the SQL just had.

SPEC.md, examples/GETTING_STARTED.md, and ui/README.md move into docs/.
PERSPECTIVE.md and docs/perspective-pivot.md merge into docs/perspective.md,
version rationale first, then the API reference. README.md becomes an entry
point that links out, and CLAUDE.md keeps only working rules and non-obvious
behaviour, pointing at docs/spec.md for the rest. examples/ keeps just the
sample CSV the tutorial loads.

Corrections found while consolidating:

- the spec's API table was missing 20 routes — every override endpoint, most of
  /api/stacks, the mapping remap routes, /health. Rebuilt from the route files
- the tutorial used port 3000 (default is 3020) and never mentioned Basic auth,
  so every curl in it would have 401'd
- the tutorial and the spec each hand-listed the SQL deploy order; both now
  point at manage.py, which is where the order actually lives
- CLAUDE.md described deduplication as an MD5 hash (it is a plain JSONB object),
  claimed 5 tables and 4 functions, and told you to run a setup.sh that has not
  existed for some time

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-07-26 21:55:48 -04:00
parent 2ea2548715
commit 7dcd8c4b61
7 changed files with 320 additions and 522 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

282
CLAUDE.md
View File

@ -2,147 +2,70 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview Dataflow imports CSV data, extracts structure from it with regex rules, maps the extracted
values to standardized output, and serves the result over a REST API and React UI. It is a
**simple system by design** — don't over-engineer it.
Dataflow is a simple data transformation tool for importing, cleaning, and standardizing data from various sources. Built with PostgreSQL and Node.js/Express, it emphasizes clarity and simplicity over complexity. **Read [docs/spec.md](docs/spec.md) for architecture, schema, data flow, the full API, and
`manage.py`.** This file covers only what you need to work in the repo without breaking
something — the rules and non-obvious behaviours that aren't visible from the code.
## Core Concepts ## Where things live
1. **Sources** - Define data sources and deduplication rules (which fields make a record unique) Both the API routes and the SQL are one file per resource: `api/routes/rules.js` and
2. **Import** - Load CSV data, automatically deduplicating based on source rules `database/rules.sql` are two halves of the same feature. Two SQL files are shared engines
3. **Rules** - Extract information using regex patterns (e.g., extract merchant from transaction description) rather than per-route — `import.sql` (CSV import, audit trail) and `transform.sql` (the
4. **Mappings** - Map extracted values to standardized output (e.g., "WALMART" → {"vendor": "Walmart", "category": "Groceries"}) rule/mapping engine, including the `jsonb_concat_obj` aggregate).
5. **Transform** - Apply rules and mappings to create clean, enriched data
## Architecture `manage.py`'s `QUERY_FILES` list is the deploy order and the authoritative file list.
### Database Schema (`database/schema.sql`) ## Rules that matter
**5 simple tables:** **`database/*.sql` is the source of truth for every database function. Never edit a function
- `sources` - Source definitions with `constraint_fields` array directly in the database.** A live edit that isn't written back to the file is silently
- `records` - Imported data with `data` (raw) and `transformed` (enriched) JSONB columns reverted the next time anyone runs "Redeploy SQL functions". This has already happened once:
- `rules` - Regex extraction rules with `field`, `pattern`, `output_field` five functions drifted and sat wrong in the repo for months — see the git history of the
- `mappings` - Input/output value mappings deleted `database/functions.sql`.
- `import_log` - Audit trail
**Key design:** **Always run `npm run build` from `ui/` after any change to `ui/src/`.** The Express server
- JSONB for flexible data storage serves the built output in `public/`; source changes are invisible until you rebuild.
- Deduplication via MD5 hash of specified fields
- Simple, flat structure (no complex relationships)
### Database Functions (`database/functions.sql`) **Never use `ON CONFLICT (constraint_key)`.** See deduplication below — there is no unique
constraint, and adding one would drop legitimate transactions.
**4 focused functions:** ## The three data layers
- `import_records(source_name, data)` - Import with deduplication
- `apply_transformations(source_name, record_ids)` - Apply rules and mappings
- `get_unmapped_values(source_name, rule_name)` - Find values needing mappings
- `reprocess_records(source_name)` - Re-transform all records
**Design principle:** Each function does ONE thing. No nested CTEs, no duplication. Each row in `records` keeps its data in three JSONB columns:
### API Server (`api/server.js` + `api/routes/`) - `data` — raw imported values, never modified
- `transformed` — rule and mapping output only (the delta)
- `overrides` — manual edits, highest precedence
**RESTful endpoints:** Readers merge them as `data || transformed || overrides`. Keeping them separate is what lets
- `/api/sources` - CRUD sources, import CSV, trigger transformations `reprocess_records` re-run the rules without clobbering a manual edit. Anything that writes
- `/api/rules` - CRUD transformation rules overrides into `transformed` is a bug — that was the pre-May-2026 behaviour.
- `/api/mappings` - CRUD value mappings, view unmapped values
- `/api/records` - Query and search transformed data
**Route files:** ## Deduplication
- `routes/sources.js` - Source management and CSV import
- `routes/rules.js` - Rule management
- `routes/mappings.js` - Mapping management + unmapped values
- `routes/records.js` - Record queries and search
## Common Development Tasks - `constraint_key` is a JSONB object of the constraint field values — readable, no hashing
- Dedup is enforced at import time in a CTE. There is **no unique DB constraint** on it
- **The constraint key is cross-batch re-import protection, not record uniqueness**
- Within one import batch, all rows insert even when constraint keys collide. Banks
legitimately send identical-looking transactions — 11 separate Cedar Point charges on the
same day are 11 real rows
- On re-import of an overlapping date range, rows whose key already exists are skipped, so
re-running a month-to-date export the next day doesn't double-count
- Deleting an import log entry cascades to every record in that batch (`import_id` FK)
### Running the Application ## Error handling
```bash API routes use `try/catch` and pass errors to `next(err)`; `server.js` has a global handler.
# Setup (first time only) Database functions return JSON with a `success` boolean.
./setup.sh
# Start development server with auto-reload
npm run dev
# Start production server
npm start
# Test API
curl http://localhost:3000/health
```
### Database Changes
When modifying schema:
1. Edit `database/schema.sql`
2. Drop and recreate schema: `psql -d dataflow -f database/schema.sql`
3. Redeploy functions: `psql -d dataflow -f database/functions.sql`
For production, write migration scripts instead of dropping schema.
### Adding a New API Endpoint
1. Add route to appropriate file in `api/routes/`
2. Follow existing patterns (async/await, error handling via `next()`)
3. Use parameterized queries to prevent SQL injection
4. Return consistent JSON format
### Testing
Manual testing workflow:
1. Create a source: `POST /api/sources`
2. Create rules: `POST /api/rules`
3. Import data: `POST /api/sources/:name/import`
4. Apply transformations: `POST /api/sources/:name/transform`
5. View results: `GET /api/records/source/:name`
See `examples/GETTING_STARTED.md` for complete curl examples.
## Design Principles
1. **Simple over clever** - Straightforward code beats optimization
2. **Explicit over implicit** - No magic, no hidden triggers
3. **Clear naming** - `data` not `rec`, `transformed` not `allj`
4. **One function, one job** - No 250-line functions
5. **JSONB for flexibility** - Handle varying schemas without migrations
## Common Patterns
### Import Flow
```
CSV file → parse → import_records() → records table (data column)
```
### Transformation Flow
```
records.data → apply_transformations() →
- Apply each rule (regex extraction)
- Look up mappings
- Merge into records.transformed
```
### Deduplication
- `constraint_key` is a JSONB object of the constraint field values (readable, no hashing)
- Dedup is enforced at import time via CTE — NO unique DB constraint on constraint_key
- **The constraint key is for cross-batch re-import protection, NOT record uniqueness**
- Within a single import batch, ALL rows insert regardless of duplicate constraint keys
- Banks legitimately send multiple identical-looking transactions (same date, description, amount)
- Example: 11 Cedar Point merchandise charges on one day — all should insert in one batch
- On re-import of overlapping date range, rows whose constraint_key already exists in DB are skipped
- This prevents double-counting when you re-run a month-to-date export the next day
- NEVER use `ON CONFLICT (constraint_key)` — there is no unique constraint and it would wrongly
drop legitimate duplicate transactions from the same batch
- Deleting an import log entry cascades to all records from that batch (import_id FK)
### Error Handling
- API routes use `try/catch` and pass errors to `next(err)`
- Server.js has global error handler
- Database functions return JSON with `success` boolean
## Light / dark mode ## Light / dark mode
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`. Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a
`ThemeProvider` that wraps the app in `main.jsx`.
- **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit - **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
- **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>` - **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
@ -151,20 +74,10 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit
- **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires - **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()` - **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
## UI (React + Vite) ## Pivot inspector panel
The frontend lives in `ui/src/` and is built to `public/` via `npm run build` from the `ui/` directory. **Always run `npm run build` from `ui/` after any changes to `ui/src/` files.** Clicking a data cell opens a right-hand inspector panel showing the underlying transactions
for that cell. See [docs/perspective.md](docs/perspective.md) for the Perspective API itself.
### Pages
- **Sources / Rules / Mappings / Records** — standard CRUD pages
- **Pivot** (`ui/src/pages/Pivot.jsx`) — interactive pivot/crosstab powered by Perspective (`@perspective-dev` v4.5.1, installed via npm). See `docs/perspective-pivot.md` for the full Perspective API reference.
- **Stacks** — multi-source union views with running balance
- **Log** — import audit trail
### Pivot inspector panel
Clicking a data cell opens a right-hand inspector panel showing the underlying transactions for that cell. Key behaviors:
- **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views. - **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views.
- **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click. - **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click.
@ -173,87 +86,38 @@ Clicking a data cell opens a right-hand inspector panel showing the underlying t
- The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px). - The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px).
- The transaction table is sortable (click header) and shows column totals for all-numeric columns. - The transaction table is sortable (click header) and shows column totals for all-numeric columns.
### Pivot layout persistence ## Pivot layout persistence
Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The `source_name` column holds either a source name or a stack name — the FK to `sources(name)` was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use `/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`). `localStorage` is still used to remember the *last active layout* for a view (the `psp_layout_<name>` key), but named layout definitions live in the DB so they persist across machines. Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The
`source_name` column holds either a source name or a stack name — the FK to `sources(name)`
was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use
`/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`,
`save_pivot_layout`, `delete_pivot_layout`). `localStorage` still remembers the *last active
layout* for a view (the `psp_layout_<name>` key), but the definitions live in the DB so they
persist across machines.
## File Structure ## Adding features
``` - One function, one job; keep functions under 100 lines
dataflow/ - Write clear SQL, not clever SQL
├── database/ - Add the SQL function to the matching `database/*.sql` file, then the route that calls it
│ ├── schema.sql # Table definitions - Update `docs/spec.md` when you add or change an endpoint
│ └── functions.sql # Import/transform functions
├── api/
│ ├── server.js # Express server
│ └── routes/ # API endpoints
│ ├── sources.js
│ ├── rules.js
│ ├── mappings.js
│ └── records.js
├── ui/
│ ├── src/
│ │ ├── pages/ # One file per page
│ │ └── api.js # API client
│ └── package.json
├── public/ # Built UI (gitignored, generated by npm run build)
├── docs/
│ └── perspective-pivot.md # Perspective API reference
├── examples/
│ ├── GETTING_STARTED.md # Tutorial
│ └── bank_transactions.csv
├── .env.example # Config template
├── package.json
└── README.md
```
## Comparison to Legacy TPS System
This project replaces an older system (in `/opt/tps`) that had:
- 2,150 lines of complex SQL with heavy duplication
- 5 nearly-identical 200+ line functions
- Confusing names and deep nested CTEs
- Complex trigger-based processing
Dataflow achieves the same functionality with:
- ~400 lines of simple SQL
- 4 focused functions
- Clear names and linear logic
- Explicit API-triggered processing
The simplification makes it easy to understand, modify, and maintain.
## Troubleshooting ## Troubleshooting
**Database connection fails:** **Database connection fails** — check `.env` credentials, that PostgreSQL is running, and
- Check `.env` file exists and has correct credentials that the search path resolves to the `dataflow` schema.
- Verify PostgreSQL is running: `psql -U postgres -l`
- Check search path is set: Should default to `dataflow` schema
**Import succeeds but transformation fails:** **Import succeeds but transformation does nothing** — check rules exist for that source
- Check rules exist: `SELECT * FROM dataflow.rules WHERE source_name = 'xxx'` (`SELECT * FROM dataflow.rules WHERE source_name = '…'`), that `field` matches an actual key
- Verify field names match CSV columns in `data`, and test the pattern with `GET /api/rules/preview`.
- Test regex pattern manually
- Check for SQL errors in logs
**All records marked as duplicates:** **Everything is marked duplicate** — `constraint_fields` probably don't match the real field
- Verify `constraint_fields` match actual field names in data names, or the batch was already imported.
- Check if data was already imported
- Use different source name for testing
## Adding New Features ## History
When adding features, follow these principles: This replaces an older system still in `/opt/tps` — 2,150 lines of SQL with five
- Add ONE function that does ONE thing nearly-identical 200-line functions and trigger-based processing. Dataflow is a clean
- Keep functions under 100 lines if possible rewrite, not a refactor. Some function bodies still carry `mirrors TPS …` comments pointing
- Write clear SQL, not clever SQL at their counterpart there.
- Add API endpoint that calls the function
- Document in README.md and update examples
## Notes for Claude
- This is a **simple** system by design - don't over-engineer it
- Keep functions focused and linear
- Use JSONB for flexibility, not as a crutch for bad design
- When confused, read the examples/GETTING_STARTED.md walkthrough
- The old TPS system is in `/opt/tps` - this is a clean rewrite, not a refactor

49
PERSPECTIVE.md
View File

@ -1,49 +0,0 @@
# Perspective — dataflow specifics
Shared rationale lives in the canonical guide: **`/home/pt/pf_app/PERSPECTIVE.md`**
(loading, version policy, Arrow constraints, deploy pattern, upgrade smoke test).
This file records only what's specific to dataflow.
> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
> engine, separate npm scope and release schedule — don't mix the two.
---
## Current state
- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
This is the target loader; pf_app should adopt it.
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
unless view sizes grow well past 100k.
- **Deploy:** `deploy.sh` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
for the org; pf_app should copy it.
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
restore — the reference implementation; keep it.
## The version pair is correct — do NOT "fix" it to 4.4.1
`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
combination that keeps both of dataflow's hard requirements:
- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
stays. Don't touch it.
**What to actually do:**
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
explicit.)
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
renders, dark/light re-themes, and save→reload→drop-column layout restore.
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
4.5.x suite becomes possible and the pair can collapse to one version.

237
README.md
View File

@ -2,220 +2,71 @@
A simple data transformation tool for importing, cleaning, and standardizing data from various sources. A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
## What It Does Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will
deduplicate on import, pull structure out with regex rules, map the extracted values to clean
output, and serve the result through a web UI and REST API.
Dataflow helps you: ## How it works
1. **Import** CSV data with automatic deduplication
2. **Transform** data using regex rules to extract meaningful information
3. **Map** extracted values to standardized output
4. **Query** the transformed data via a web UI or REST API
Perfect for cleaning up messy data like bank transactions, product lists, or any repetitive data that needs normalization. 1. **Sources** define where data comes from and which fields make a record unique
2. **Rules** extract information with regex (`extract` or `replace` mode) —
e.g. pull the merchant out of a transaction description
3. **Mappings** turn extracted values into clean output —
`"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
4. **Records** are then queryable, pivotable, and exportable
## Core Concepts Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output),
and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never
clobbers something you typed by hand.
### 1. Sources ## Stack
Define where data comes from and how to deduplicate it.
**Example:** Bank transactions deduplicated by date + amount + description PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`.
HTTP Basic auth, configured in `.env`.
### 2. Rules ## Getting started
Extract information using regex patterns (`extract` or `replace` modes).
**Example:** Extract merchant name from transaction description Requires PostgreSQL 12+, Node.js 18+, and Python 3.
### 3. Mappings
Map extracted values to clean, standardized output.
**Example:** "DISCOUNT DRUG MART 32" → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
## Architecture
- **Database:** PostgreSQL with JSONB for flexible data storage
- **API:** Node.js/Express REST API
- **UI:** React SPA served from `public/`
- **Auth:** HTTP Basic auth (configured in `.env`)
## Design Principles
- **Simple & Clear** - Easy to understand what's happening
- **Explicit** - No hidden magic or complex triggers
- **Flexible** - Handle varying data formats without schema changes
## Getting Started
### Prerequisites
- PostgreSQL 12+
- Node.js 18+
- Python 3 (for `manage.py`)
### Installation
1. Install Node dependencies:
```bash ```bash
npm install npm install
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
``` ```
2. Run the management script to configure and deploy everything: The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api`
```bash (port set by `API_PORT` in `.env`).
python3 manage.py
```
For development with auto-reload: For a walkthrough that creates a source, adds rules and mappings, and imports the sample
```bash CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**.
npm run dev
```
The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`). ## Documentation
## Management Script (`manage.py`) | | |
|---|---|
| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl |
| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` |
| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages |
| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference |
`manage.py` is an interactive tool for configuring, deploying, and managing the service. Run it and choose from the numbered menu: ## Project structure
```
python3 manage.py
```
| # | Action |
|---|--------|
| 1 | **Database configuration** — create/update `.env`, optionally create the PostgreSQL user/database, and deploy schema + functions |
| 2 | Redeploy schema only (`database/schema.sql`) — drops and recreates all tables |
| 3 | Redeploy SQL functions only (`database/queries/`) |
| 4 | Build UI (`ui/` → `public/`) |
| 5 | Set up nginx reverse proxy (HTTP or HTTPS via certbot) |
| 6 | Install systemd service unit (`dataflow.service`) |
| 7 | Start / restart `dataflow.service` |
| 8 | Stop `dataflow.service` |
| 9 | Set login credentials (`LOGIN_USER` / `LOGIN_PASSWORD_HASH` in `.env`) |
The status screen at the top of the menu shows the current state of each component (database connection, schema, UI build, service, nginx).
**Typical first-time setup:** run options 1 → 4 → 9 → 6 → 7 (→ 5 if you want nginx).
## API Reference
All `/api` routes require HTTP Basic authentication.
### Sources — `/api/sources`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/sources` | List all sources |
| POST | `/api/sources` | Create a source |
| GET | `/api/sources/:name` | Get a source |
| PUT | `/api/sources/:name` | Update a source |
| DELETE | `/api/sources/:name` | Delete a source |
| POST | `/api/sources/suggest` | Suggest source definition from CSV upload |
| POST | `/api/sources/:name/import` | Import CSV data and auto-apply transformations to new records |
| GET | `/api/sources/:name/import-log` | View import history (includes `inserted_keys` / `excluded_keys` in `info`) |
| DELETE | `/api/sources/:name/import-log/:id` | Delete an import batch and all its records |
| POST | `/api/sources/:name/transform` | Apply rules and mappings to any untransformed records |
| POST | `/api/sources/:name/reprocess` | Re-transform all records |
| GET | `/api/sources/:name/fields` | List all known field names |
| GET | `/api/sources/:name/stats` | Get record and mapping counts |
| POST | `/api/sources/:name/view` | Generate output view |
| GET | `/api/sources/:name/view-data` | Query output view (paginated, sortable) |
### Rules — `/api/rules`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/rules/source/:source_name` | List rules for a source |
| POST | `/api/rules` | Create a rule |
| GET | `/api/rules/:id` | Get a rule |
| PUT | `/api/rules/:id` | Update a rule |
| DELETE | `/api/rules/:id` | Delete a rule |
| GET | `/api/rules/preview` | Preview a pattern against real records (ad-hoc) |
| GET | `/api/rules/:id/test` | Test a saved rule against real records |
### Mappings — `/api/mappings`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/mappings/source/:source_name` | List mappings |
| POST | `/api/mappings` | Create a mapping |
| POST | `/api/mappings/bulk` | Bulk create/update mappings |
| GET | `/api/mappings/:id` | Get a mapping |
| PUT | `/api/mappings/:id` | Update a mapping |
| DELETE | `/api/mappings/:id` | Delete a mapping |
| GET | `/api/mappings/source/:source_name/unmapped` | Get values with no mapping yet |
| GET | `/api/mappings/source/:source_name/all-values` | All extracted values with counts |
| GET | `/api/mappings/source/:source_name/counts` | Record counts for existing mappings |
| GET | `/api/mappings/source/:source_name/export.tsv` | Export values as TSV |
| POST | `/api/mappings/source/:source_name/import-csv` | Import mappings from TSV |
### Records — `/api/records`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/records/source/:source_name` | List records (paginated) |
| GET | `/api/records/:id` | Get a single record |
| POST | `/api/records/search` | Search records |
| DELETE | `/api/records/:id` | Delete a record |
| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
### Stacks — `/api/stacks`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/stacks` | List all stacks |
| POST | `/api/stacks` | Create a stack |
| GET | `/api/stacks/:name` | Get a stack |
| PUT | `/api/stacks/:name` | Update a stack |
| DELETE | `/api/stacks/:name` | Delete a stack |
| GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) |
| GET | `/api/stacks/:name/layouts` | List saved pivot layouts |
| POST | `/api/stacks/:name/layouts` | Save a pivot layout |
| DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout |
## Typical Workflow
```
1. Create a source (POST /api/sources)
2. Create transformation rules (POST /api/rules)
3. Import CSV data (POST /api/sources/:name/import) — transformations applied automatically to new records
4. Preview rules against real data (GET /api/rules/preview)
5. Review unmapped values (GET /api/mappings/source/:name/unmapped)
6. Add mappings (POST /api/mappings or bulk import via TSV)
7. Reprocess to apply new mappings (POST /api/sources/:name/reprocess)
8. Query results (GET /api/sources/:name/view-data)
```
See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
## Project Structure
``` ```
dataflow/ dataflow/
├── database/ ├── manage.py # interactive setup / deploy / uninstall
│ ├── schema.sql # Table definitions ├── database/ # schema.sql + one .sql file per API route
│ └── queries/ # SQL functions, one file per route ├── api/ # Express server, routes, auth middleware
│ ├── sources.sql ├── ui/ # React source (built to public/)
│ ├── rules.sql ├── public/ # built UI, served as static files
│ ├── mappings.sql ├── docs/
│ ├── records.sql └── examples/ # sample CSV for the tutorial
│ ├── stacks.sql
│ └── status.sql
├── api/
│ ├── server.js # Express server
│ ├── middleware/
│ │ └── auth.js # Basic auth middleware
│ ├── lib/
│ │ └── sql.js # SQL literal helpers
│ └── routes/
│ ├── sources.js
│ ├── rules.js
│ ├── mappings.js
│ ├── records.js
│ ├── stacks.js
│ └── status.js
├── public/ # Built React UI (served as static files)
├── examples/
│ ├── GETTING_STARTED.md
│ └── bank_transactions.csv
└── .env.example
``` ```
Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js`
and `database/rules.sql` are the two halves of the same feature.
`database/*.sql` is the source of truth for every database function — never edit one directly
in the database, or the next redeploy will silently revert it.
## License ## License
MIT MIT

70
examples/GETTING_STARTED.md → docs/getting-started.md
View File

@ -4,32 +4,40 @@ This guide walks through a complete example using bank transaction data.
## Prerequisites ## Prerequisites
1. PostgreSQL database running PostgreSQL running, Node.js 18+, and Python 3.
2. Database created: `CREATE DATABASE dataflow;`
3. `.env` file configured (copy from `.env.example`)
## Step 1: Deploy Database Schema ## Step 1: Configure and Deploy
```bash ```bash
cd /opt/dataflow cd /opt/dataflow
psql -U postgres -d dataflow -f database/schema.sql npm install
psql -U postgres -d dataflow -f database/functions.sql python3 manage.py
``` ```
You should see tables created without errors. Choose option 1. It writes `.env`, creates the database and user if they don't exist,
then deploys `database/schema.sql` and the SQL function files in dependency order.
## Step 2: Start the API Server ## Step 2: Start the API Server
```bash ```bash
npm install
npm start npm start
``` ```
The server should start on port 3000 (or your configured port). The server starts on the port set by `API_PORT` in `.env` (3020 by default).
Every `/api` route requires HTTP Basic auth using the credentials set by `manage.py`
option 9. The examples below omit it for readability — add `-u username:password` to each
curl, or export it once:
```bash
alias dfcurl='curl -u username:password'
```
`GET /health` is the one route that needs no auth.
Test it: Test it:
```bash ```bash
curl http://localhost:3000/health curl http://localhost:3020/health
# Should return: {"status":"ok","timestamp":"..."} # Should return: {"status":"ok","timestamp":"..."}
``` ```
@ -38,7 +46,7 @@ curl http://localhost:3000/health
A source defines where data comes from and how to deduplicate it. A source defines where data comes from and how to deduplicate it.
```bash ```bash
curl -X POST http://localhost:3000/api/sources \ curl -X POST http://localhost:3020/api/sources \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"name": "bank_transactions", "name": "bank_transactions",
@ -55,7 +63,7 @@ Rules extract meaningful data using regex patterns.
### Rule 1: Extract merchant name (first part of description) ### Rule 1: Extract merchant name (first part of description)
```bash ```bash
curl -X POST http://localhost:3000/api/rules \ curl -X POST http://localhost:3020/api/rules \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -70,7 +78,7 @@ curl -X POST http://localhost:3000/api/rules \
### Rule 2: Extract location (city + state pattern) ### Rule 2: Extract location (city + state pattern)
```bash ```bash
curl -X POST http://localhost:3000/api/rules \ curl -X POST http://localhost:3020/api/rules \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -87,7 +95,7 @@ curl -X POST http://localhost:3000/api/rules \
Import the example CSV file: Import the example CSV file:
```bash ```bash
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \ curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
-F "file=@examples/bank_transactions.csv" -F "file=@examples/bank_transactions.csv"
``` ```
@ -104,7 +112,7 @@ Response:
## Step 6: View Imported Records ## Step 6: View Imported Records
```bash ```bash
curl http://localhost:3000/api/records/source/bank_transactions?limit=5 curl http://localhost:3020/api/records/source/bank_transactions?limit=5
``` ```
You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet! You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet!
@ -112,7 +120,7 @@ You'll see the raw imported data. Note that `transformed` is `null` - we haven't
## Step 7: Apply Transformations ## Step 7: Apply Transformations
```bash ```bash
curl -X POST http://localhost:3000/api/sources/bank_transactions/transform curl -X POST http://localhost:3020/api/sources/bank_transactions/transform
``` ```
Response: Response:
@ -125,7 +133,7 @@ Response:
Now check the records again: Now check the records again:
```bash ```bash
curl http://localhost:3000/api/records/source/bank_transactions?limit=2 curl http://localhost:3020/api/records/source/bank_transactions?limit=2
``` ```
You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`. You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`.
@ -133,7 +141,7 @@ You'll see the `transformed` field now contains the original data plus extracted
## Step 8: View Extracted Values That Need Mapping ## Step 8: View Extracted Values That Need Mapping
```bash ```bash
curl http://localhost:3000/api/mappings/source/bank_transactions/unmapped curl http://localhost:3020/api/mappings/source/bank_transactions/unmapped
``` ```
Response shows extracted merchant names that aren't mapped yet: Response shows extracted merchant names that aren't mapped yet:
@ -151,7 +159,7 @@ Response shows extracted merchant names that aren't mapped yet:
Map extracted values to clean, standardized output: Map extracted values to clean, standardized output:
```bash ```bash
curl -X POST http://localhost:3000/api/mappings \ curl -X POST http://localhost:3020/api/mappings \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -163,7 +171,7 @@ curl -X POST http://localhost:3000/api/mappings \
} }
}' }'
curl -X POST http://localhost:3000/api/mappings \ curl -X POST http://localhost:3020/api/mappings \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -175,7 +183,7 @@ curl -X POST http://localhost:3000/api/mappings \
} }
}' }'
curl -X POST http://localhost:3000/api/mappings \ curl -X POST http://localhost:3020/api/mappings \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -193,13 +201,13 @@ curl -X POST http://localhost:3000/api/mappings \
Clear and reapply transformations to pick up the new mappings: Clear and reapply transformations to pick up the new mappings:
```bash ```bash
curl -X POST http://localhost:3000/api/sources/bank_transactions/reprocess curl -X POST http://localhost:3020/api/sources/bank_transactions/reprocess
``` ```
## Step 11: View Final Results ## Step 11: View Final Results
```bash ```bash
curl http://localhost:3000/api/records/source/bank_transactions?limit=5 curl http://localhost:3020/api/records/source/bank_transactions?limit=5
``` ```
Now the `transformed` field contains: Now the `transformed` field contains:
@ -234,7 +242,7 @@ Example result:
Try importing the same file again: Try importing the same file again:
```bash ```bash
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \ curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
-F "file=@examples/bank_transactions.csv" -F "file=@examples/bank_transactions.csv"
``` ```
@ -272,19 +280,19 @@ You've now:
```bash ```bash
# View all sources # View all sources
curl http://localhost:3000/api/sources curl http://localhost:3020/api/sources
# View source statistics # View source statistics
curl http://localhost:3000/api/sources/bank_transactions/stats curl http://localhost:3020/api/sources/bank_transactions/stats
# View all rules for a source # View all rules for a source
curl http://localhost:3000/api/rules/source/bank_transactions curl http://localhost:3020/api/rules/source/bank_transactions
# View all mappings for a source # View all mappings for a source
curl http://localhost:3000/api/mappings/source/bank_transactions curl http://localhost:3020/api/mappings/source/bank_transactions
# Search for specific records # Search for specific records
curl -X POST http://localhost:3000/api/records/search \ curl -X POST http://localhost:3020/api/records/search \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"source_name": "bank_transactions", "source_name": "bank_transactions",
@ -301,11 +309,11 @@ curl -X POST http://localhost:3000/api/records/search \
- Check logs for error messages - Check logs for error messages
**Import fails:** **Import fails:**
- Verify source exists: `curl http://localhost:3000/api/sources` - Verify source exists: `curl http://localhost:3020/api/sources`
- Check CSV format matches expectations - Check CSV format matches expectations
- Ensure constraint_fields match CSV column names - Ensure constraint_fields match CSV column names
**Transformations not working:** **Transformations not working:**
- Check rules exist: `curl http://localhost:3000/api/rules/source/bank_transactions` - Check rules exist: `curl http://localhost:3020/api/rules/source/bank_transactions`
- Test regex pattern manually - Test regex pattern manually
- Check records have the specified field - Check records have the specified field

63
docs/perspective-pivot.md → docs/perspective.md
View File

@ -1,11 +1,68 @@
# Perspective Pivot — Technical Reference # Perspective
Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply. Everything about the Perspective pivot in dataflow: which packages and versions are
pinned and why, and a ground-truth reference for the parts of the API the official docs
don't cover.
This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement. Shared rationale across projects lives in the canonical guide at
`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy
pattern, upgrade smoke test). This file records what's specific to dataflow.
--- ---
> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
> engine, separate npm scope and release schedule — don't mix the two.
---
## Current state
- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
This is the target loader; pf_app should adopt it.
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
unless view sizes grow well past 100k.
- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
for the org; pf_app should copy it.
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
restore — the reference implementation; keep it.
## The version pair is correct — do NOT "fix" it to 4.4.1
`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
combination that keeps both of dataflow's hard requirements:
- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
stays. Don't touch it.
**What to actually do:**
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
explicit.)
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
renders, dark/light re-themes, and save→reload→drop-column layout restore.
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
4.5.x suite becomes possible and the pair can collapse to one version.
---
# API Reference
Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at
**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been
re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete
for some of these APIs — treat this as a ground-truth supplement.
## Loading via npm ## Loading via npm
```js ```js

133
SPEC.md → docs/spec.md
View File

@ -18,7 +18,7 @@ SQL functions are the single source of truth for business logic. The API layer i
Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does. Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does.
### One SQL file per route ### One SQL file per route
SQL is organized in `database/queries/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`). This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation. SQL is organized in `database/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `stacks.sql`, `status.sql`) plus `import.sql` and `transform.sql` for the import/transform engine. This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
### Explicit over implicit ### Explicit over implicit
Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs. Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs.
@ -34,11 +34,14 @@ Raw imported records and transformed records are stored as JSONB. This avoids sc
manage.py — interactive CLI for setup, deployment, and management manage.py — interactive CLI for setup, deployment, and management
database/ database/
schema.sql — table definitions (run once or to reset) schema.sql — table definitions (run once or to reset)
queries/
sources.sql — all SQL for /api/sources sources.sql — all SQL for /api/sources
rules.sql — all SQL for /api/rules rules.sql — all SQL for /api/rules
mappings.sql — all SQL for /api/mappings mappings.sql — all SQL for /api/mappings
records.sql — all SQL for /api/records records.sql — all SQL for /api/records
stacks.sql — all SQL for /api/stacks
status.sql — all SQL for /api/status
import.sql — CSV import and the import audit trail
transform.sql — the rule/mapping engine
api/ api/
server.js — Express server, mounts routes, auth middleware server.js — Express server, mounts routes, auth middleware
middleware/ middleware/
@ -64,8 +67,14 @@ ui/
Mappings.jsx — mapping table with TSV import/export Mappings.jsx — mapping table with TSV import/export
Records.jsx — paginated, sortable view of transformed records Records.jsx — paginated, sortable view of transformed records
Pivot.jsx — interactive pivot table with cell inspector Pivot.jsx — interactive pivot table with cell inspector
Stacks.jsx — multi-source union views with running balance
Remap.jsx — bulk remap of an output field value across mappings
Log.jsx — global import log across all sources Log.jsx — global import log across all sources
components/ — Sidebar, StatusBar
theme.jsx — light/dark context provider
public/ — compiled UI (output of npm run build in ui/) public/ — compiled UI (output of npm run build in ui/)
docs/ — this file, tutorial, UI and Perspective references
examples/ — bank_transactions.csv, the tutorial's sample data
``` ```
--- ---
@ -123,7 +132,7 @@ The transform is fully set-based — no row-by-row loops. All records for a sour
## SQL Functions ## SQL Functions
Each file in `database/queries/` maps 1-to-1 with a route file. Each route file has a matching SQL file in `database/`; `import.sql` and `transform.sql` hold the engine shared by several routes.
**sources.sql** **sources.sql**
`list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view` `list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view`
@ -153,53 +162,104 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
**Route summary:** **Route summary:**
### Sources — `api/routes/sources.js`
| Method | Path | Description | | Method | Path | Description |
|--------|------|-------------| |--------|------|-------------|
| GET | /api/sources | List all sources | | GET | /api/sources | List all sources |
| POST | /api/sources | Create source | | POST | /api/sources | Create source |
| GET | /api/sources/:name | Get source | | GET | /api/sources/:name | Get source |
| PUT | /api/sources/:name | Update source (constraint_fields, config) | | PUT | /api/sources/:name | Update source (constraint_fields, config, global_picklist) |
| DELETE | /api/sources/:name | Delete source and all data | | DELETE | /api/sources/:name | Delete source and all its data |
| POST | /api/sources/suggest | Suggest source config from CSV upload | | POST | /api/sources/suggest | Suggest source config from an uploaded CSV |
| POST | /api/sources/:name/import | Import CSV records | | POST | /api/sources/:name/import | Import CSV; transformations are applied to the new records |
| GET | /api/sources/:name/import-log | Import history | | GET | /api/sources/import-log | Import history across all sources |
| GET | /api/sources/:name/stats | Record counts | | GET | /api/sources/:name/import-log | Import history for one source |
| GET | /api/sources/:name/fields | All known field names and origins | | DELETE | /api/sources/:name/import-log/:id | Delete an import batch and every record in it |
| GET | /api/sources/:name/view-data | Paginated, sortable view data | | POST | /api/sources/:name/transform | Apply transformations to untransformed records only |
| POST | /api/sources/:name/transform | Apply transformations (new records only) |
| POST | /api/sources/:name/reprocess | Reapply transformations to all records | | POST | /api/sources/:name/reprocess | Reapply transformations to all records |
| POST | /api/sources/:name/view | Generate dfv view | | GET | /api/sources/:name/stats | Record counts |
| GET | /api/sources/:name/fields | All known field names and their origins |
| GET | /api/sources/:name/override-keys | Distinct field names used in overrides for this source |
| POST | /api/sources/:name/view | Generate/refresh the `dfv` view |
| GET | /api/sources/:name/view-data | Paginated, sortable, filterable view data |
| GET | /api/sources/:name/layouts | List saved pivot layouts |
| POST | /api/sources/:name/layouts | Save a pivot layout |
| DELETE | /api/sources/:name/layouts/:id | Delete a pivot layout |
### Rules — `api/routes/rules.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/rules/source/:name | List rules for a source | | GET | /api/rules/source/:name | List rules for a source |
| GET | /api/rules/preview | Preview pattern against live records | | GET | /api/rules/:id | Get a rule |
| GET | /api/rules/:id/test | Test saved rule against live records |
| POST | /api/rules | Create rule | | POST | /api/rules | Create rule |
| PUT | /api/rules/:id | Update rule | | PUT | /api/rules/:id | Update rule |
| DELETE | /api/rules/:id | Delete rule | | DELETE | /api/rules/:id | Delete rule |
| GET | /api/rules/preview | Preview an ad-hoc pattern against live records |
| GET | /api/rules/:id/test | Test a saved rule against live records |
### Mappings — `api/routes/mappings.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/mappings/source/:name | List mappings | | GET | /api/mappings/source/:name | List mappings |
| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) | | GET | /api/mappings/:id | Get a mapping |
| GET | /api/mappings/source/:name/unmapped | Only unmapped extracted values |
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
| GET | /api/mappings/source/:name/export.tsv | Export mappings as TSV |
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from TSV |
| POST | /api/mappings | Create mapping | | POST | /api/mappings | Create mapping |
| POST | /api/mappings/bulk | Upsert multiple mappings | | POST | /api/mappings/bulk | Upsert multiple mappings |
| PUT | /api/mappings/:id | Update mapping | | PUT | /api/mappings/:id | Update mapping |
| DELETE | /api/mappings/:id | Delete mapping | | DELETE | /api/mappings/:id | Delete mapping |
| GET | /api/records/source/:name | List raw records | | GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) with counts |
| GET | /api/records/:id | Get single record | | GET | /api/mappings/source/:name/unmapped | Only values with no mapping yet |
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
| GET | /api/mappings/source/:name/export.tsv | Export extracted values as TSV |
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from an uploaded TSV |
| GET | /api/mappings/global-values | Output values across all `global_picklist` sources (autocomplete) |
| GET | /api/mappings/outputs | Search output field values across all mappings |
| GET | /api/mappings/outputs/:col/:val | Mappings carrying a specific output field value |
| POST | /api/mappings/remap-field | Replace an output field value across all mappings |
### Records — `api/routes/records.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/records/source/:name | List records (paginated) |
| GET | /api/records/:id | Get a single record |
| POST | /api/records/search | Search by JSONB containment | | POST | /api/records/search | Search by JSONB containment |
| DELETE | /api/records/:id | Delete record | | DELETE | /api/records/:id | Delete record |
| DELETE | /api/records/source/:name/all | Delete all records for a source | | DELETE | /api/records/source/:name/all | Delete all records for a source |
| PUT | /api/records/:id/overrides | Set manual overrides on a record |
| DELETE | /api/records/:id/overrides | Clear a record's overrides |
| POST | /api/records/bulk-overrides | Apply the same overrides to many records |
### Stacks — `api/routes/stacks.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/stacks | List all stacks | | GET | /api/stacks | List all stacks |
| GET | /api/stacks/:name | Get a stack with its sources |
| POST | /api/stacks | Create stack | | POST | /api/stacks | Create stack |
| GET | /api/stacks/:name | Get stack |
| PUT | /api/stacks/:name | Update stack | | PUT | /api/stacks/:name | Update stack |
| DELETE | /api/stacks/:name | Delete stack | | DELETE | /api/stacks/:name | Delete stack |
| PUT | /api/stacks/:name/sources/:source | Add or update a source in the stack |
| DELETE | /api/stacks/:name/sources/:source | Remove a source from the stack |
| PUT | /api/stacks/:name/sources/reorder | Reorder the stack's sources |
| GET | /api/stacks/:name/view-sql | Preview the SQL that would build the view (dry run) |
| POST | /api/stacks/:name/view | Generate/refresh the `dfv` view |
| POST | /api/stacks/:name/exec-sql | Execute user-edited SQL for the view |
| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance | | GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
| GET | /api/stacks/:name/balance | Current running balance from the generated view |
| POST | /api/stacks/:name/calibrate | Set the balance offset from a known balance at a date |
| GET | /api/stacks/:name/layouts | List saved pivot layouts | | GET | /api/stacks/:name/layouts | List saved pivot layouts |
| POST | /api/stacks/:name/layouts | Save pivot layout | | POST | /api/stacks/:name/layouts | Save a pivot layout |
| DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout | | DELETE | /api/stacks/:name/layouts/:id | Delete a pivot layout |
### Status — `api/routes/status.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/status | Deployment status | | GET | /api/status | Deployment status |
| GET | /health | Health check (no auth) |
--- ---
@ -253,7 +313,7 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
- `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save. - `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save.
- Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently. - Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently.
See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically. See `docs/perspective.md` for the full technical reference on controlling Perspective programmatically.
- **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table. - **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
@ -280,7 +340,7 @@ Shows current status on every screen:
2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation. 2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation.
3. **Redeploy SQL functions** — Runs all four files in `database/queries/` in order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`. Safe to run at any time without data loss. 3. **Redeploy SQL functions** — Runs the function files in `database/` in dependency order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `import.sql`, `transform.sql`, `stacks.sql`, `status.sql`. Safe to run at any time without data loss.
4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`. 4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`.
@ -294,6 +354,8 @@ Shows current status on every screen:
9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed. 9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed.
10. **Uninstall** — Reverses everything the other options install, in reverse order: stops/disables/removes the systemd unit, removes the nginx site and reloads nginx, drops the database and its user (prompts for admin credentials), then deletes `.env`, `public/`, and `node_modules`. Lists exactly what it found before doing anything and requires typing `delete` to proceed. The repository itself is left in place.
**Key behaviors:** **Key behaviors:**
- All commands that will be run are printed before the user is asked to confirm. - All commands that will be run are printed before the user is asked to confirm.
- Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal. - Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal.
@ -341,13 +403,18 @@ The server binds to `0.0.0.0` on `API_PORT` and serves both the API and the comp
## Deploying SQL Changes ## Deploying SQL Changes
Any time SQL functions are modified: Any time SQL functions are modified, run `python3 manage.py` and choose "Redeploy SQL
functions only". It runs every function file in dependency order — the list lives in
`QUERY_FILES` in `manage.py`, which is the one place the order is defined.
To deploy a single file by hand:
```bash ```bash
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/sources.sql PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -v ON_ERROR_STOP=1 -f database/rules.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/rules.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/mappings.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/records.sql
``` ```
Then restart the server. Function deployment is safe to repeat — all functions use `CREATE OR REPLACE`.
Deployment is safe to repeat — every function uses `CREATE OR REPLACE`.
**The files are the source of truth.** Editing a function directly in the database, without
writing the change back to its file, means the next redeploy silently reverts it.
Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead. Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead.

0
ui/README.md → docs/ui.md
View File