diff --git a/CLAUDE.md b/CLAUDE.md index 39adf2b..8ed2168 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,147 +2,70 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. -## Overview +Dataflow imports CSV data, extracts structure from it with regex rules, maps the extracted +values to standardized output, and serves the result over a REST API and React UI. It is a +**simple system by design** — don't over-engineer it. -Dataflow is a simple data transformation tool for importing, cleaning, and standardizing data from various sources. Built with PostgreSQL and Node.js/Express, it emphasizes clarity and simplicity over complexity. +**Read [docs/spec.md](docs/spec.md) for architecture, schema, data flow, the full API, and +`manage.py`.** This file covers only what you need to work in the repo without breaking +something — the rules and non-obvious behaviours that aren't visible from the code. -## Core Concepts +## Where things live -1. **Sources** - Define data sources and deduplication rules (which fields make a record unique) -2. **Import** - Load CSV data, automatically deduplicating based on source rules -3. **Rules** - Extract information using regex patterns (e.g., extract merchant from transaction description) -4. **Mappings** - Map extracted values to standardized output (e.g., "WALMART" → {"vendor": "Walmart", "category": "Groceries"}) -5. **Transform** - Apply rules and mappings to create clean, enriched data +Both the API routes and the SQL are one file per resource: `api/routes/rules.js` and +`database/rules.sql` are two halves of the same feature. Two SQL files are shared engines +rather than per-route — `import.sql` (CSV import, audit trail) and `transform.sql` (the +rule/mapping engine, including the `jsonb_concat_obj` aggregate). -## Architecture +`manage.py`'s `QUERY_FILES` list is the deploy order and the authoritative file list. -### Database Schema (`database/schema.sql`) +## Rules that matter -**5 simple tables:** -- `sources` - Source definitions with `constraint_fields` array -- `records` - Imported data with `data` (raw) and `transformed` (enriched) JSONB columns -- `rules` - Regex extraction rules with `field`, `pattern`, `output_field` -- `mappings` - Input/output value mappings -- `import_log` - Audit trail +**`database/*.sql` is the source of truth for every database function. Never edit a function +directly in the database.** A live edit that isn't written back to the file is silently +reverted the next time anyone runs "Redeploy SQL functions". This has already happened once: +five functions drifted and sat wrong in the repo for months — see the git history of the +deleted `database/functions.sql`. -**Key design:** -- JSONB for flexible data storage -- Deduplication via MD5 hash of specified fields -- Simple, flat structure (no complex relationships) +**Always run `npm run build` from `ui/` after any change to `ui/src/`.** The Express server +serves the built output in `public/`; source changes are invisible until you rebuild. -### Database Functions (`database/functions.sql`) +**Never use `ON CONFLICT (constraint_key)`.** See deduplication below — there is no unique +constraint, and adding one would drop legitimate transactions. -**4 focused functions:** -- `import_records(source_name, data)` - Import with deduplication -- `apply_transformations(source_name, record_ids)` - Apply rules and mappings -- `get_unmapped_values(source_name, rule_name)` - Find values needing mappings -- `reprocess_records(source_name)` - Re-transform all records +## The three data layers -**Design principle:** Each function does ONE thing. No nested CTEs, no duplication. +Each row in `records` keeps its data in three JSONB columns: -### API Server (`api/server.js` + `api/routes/`) +- `data` — raw imported values, never modified +- `transformed` — rule and mapping output only (the delta) +- `overrides` — manual edits, highest precedence -**RESTful endpoints:** -- `/api/sources` - CRUD sources, import CSV, trigger transformations -- `/api/rules` - CRUD transformation rules -- `/api/mappings` - CRUD value mappings, view unmapped values -- `/api/records` - Query and search transformed data +Readers merge them as `data || transformed || overrides`. Keeping them separate is what lets +`reprocess_records` re-run the rules without clobbering a manual edit. Anything that writes +overrides into `transformed` is a bug — that was the pre-May-2026 behaviour. -**Route files:** -- `routes/sources.js` - Source management and CSV import -- `routes/rules.js` - Rule management -- `routes/mappings.js` - Mapping management + unmapped values -- `routes/records.js` - Record queries and search +## Deduplication -## Common Development Tasks +- `constraint_key` is a JSONB object of the constraint field values — readable, no hashing +- Dedup is enforced at import time in a CTE. There is **no unique DB constraint** on it +- **The constraint key is cross-batch re-import protection, not record uniqueness** +- Within one import batch, all rows insert even when constraint keys collide. Banks + legitimately send identical-looking transactions — 11 separate Cedar Point charges on the + same day are 11 real rows +- On re-import of an overlapping date range, rows whose key already exists are skipped, so + re-running a month-to-date export the next day doesn't double-count +- Deleting an import log entry cascades to every record in that batch (`import_id` FK) -### Running the Application +## Error handling -```bash -# Setup (first time only) -./setup.sh - -# Start development server with auto-reload -npm run dev - -# Start production server -npm start - -# Test API -curl http://localhost:3000/health -``` - -### Database Changes - -When modifying schema: -1. Edit `database/schema.sql` -2. Drop and recreate schema: `psql -d dataflow -f database/schema.sql` -3. Redeploy functions: `psql -d dataflow -f database/functions.sql` - -For production, write migration scripts instead of dropping schema. - -### Adding a New API Endpoint - -1. Add route to appropriate file in `api/routes/` -2. Follow existing patterns (async/await, error handling via `next()`) -3. Use parameterized queries to prevent SQL injection -4. Return consistent JSON format - -### Testing - -Manual testing workflow: -1. Create a source: `POST /api/sources` -2. Create rules: `POST /api/rules` -3. Import data: `POST /api/sources/:name/import` -4. Apply transformations: `POST /api/sources/:name/transform` -5. View results: `GET /api/records/source/:name` - -See `examples/GETTING_STARTED.md` for complete curl examples. - -## Design Principles - -1. **Simple over clever** - Straightforward code beats optimization -2. **Explicit over implicit** - No magic, no hidden triggers -3. **Clear naming** - `data` not `rec`, `transformed` not `allj` -4. **One function, one job** - No 250-line functions -5. **JSONB for flexibility** - Handle varying schemas without migrations - -## Common Patterns - -### Import Flow -``` -CSV file → parse → import_records() → records table (data column) -``` - -### Transformation Flow -``` -records.data → apply_transformations() → - - Apply each rule (regex extraction) - - Look up mappings - - Merge into records.transformed -``` - -### Deduplication -- `constraint_key` is a JSONB object of the constraint field values (readable, no hashing) -- Dedup is enforced at import time via CTE — NO unique DB constraint on constraint_key -- **The constraint key is for cross-batch re-import protection, NOT record uniqueness** -- Within a single import batch, ALL rows insert regardless of duplicate constraint keys - - Banks legitimately send multiple identical-looking transactions (same date, description, amount) - - Example: 11 Cedar Point merchandise charges on one day — all should insert in one batch -- On re-import of overlapping date range, rows whose constraint_key already exists in DB are skipped - - This prevents double-counting when you re-run a month-to-date export the next day -- NEVER use `ON CONFLICT (constraint_key)` — there is no unique constraint and it would wrongly - drop legitimate duplicate transactions from the same batch -- Deleting an import log entry cascades to all records from that batch (import_id FK) - -### Error Handling -- API routes use `try/catch` and pass errors to `next(err)` -- Server.js has global error handler -- Database functions return JSON with `success` boolean +API routes use `try/catch` and pass errors to `next(err)`; `server.js` has a global handler. +Database functions return JSON with a `success` boolean. ## Light / dark mode -Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`. +Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a +`ThemeProvider` that wraps the app in `main.jsx`. - **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit - **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `` @@ -151,20 +74,10 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit - **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires - **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()` -## UI (React + Vite) +## Pivot inspector panel -The frontend lives in `ui/src/` and is built to `public/` via `npm run build` from the `ui/` directory. **Always run `npm run build` from `ui/` after any changes to `ui/src/` files.** - -### Pages - -- **Sources / Rules / Mappings / Records** — standard CRUD pages -- **Pivot** (`ui/src/pages/Pivot.jsx`) — interactive pivot/crosstab powered by Perspective (`@perspective-dev` v4.5.1, installed via npm). See `docs/perspective-pivot.md` for the full Perspective API reference. -- **Stacks** — multi-source union views with running balance -- **Log** — import audit trail - -### Pivot inspector panel - -Clicking a data cell opens a right-hand inspector panel showing the underlying transactions for that cell. Key behaviors: +Clicking a data cell opens a right-hand inspector panel showing the underlying transactions +for that cell. See [docs/perspective.md](docs/perspective.md) for the Perspective API itself. - **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views. - **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click. @@ -173,87 +86,38 @@ Clicking a data cell opens a right-hand inspector panel showing the underlying t - The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px). - The transaction table is sortable (click header) and shows column totals for all-numeric columns. -### Pivot layout persistence +## Pivot layout persistence -Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The `source_name` column holds either a source name or a stack name — the FK to `sources(name)` was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use `/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`). `localStorage` is still used to remember the *last active layout* for a view (the `psp_layout_` key), but named layout definitions live in the DB so they persist across machines. +Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The +`source_name` column holds either a source name or a stack name — the FK to `sources(name)` +was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use +`/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`, +`save_pivot_layout`, `delete_pivot_layout`). `localStorage` still remembers the *last active +layout* for a view (the `psp_layout_` key), but the definitions live in the DB so they +persist across machines. -## File Structure +## Adding features -``` -dataflow/ -├── database/ -│ ├── schema.sql # Table definitions -│ └── functions.sql # Import/transform functions -├── api/ -│ ├── server.js # Express server -│ └── routes/ # API endpoints -│ ├── sources.js -│ ├── rules.js -│ ├── mappings.js -│ └── records.js -├── ui/ -│ ├── src/ -│ │ ├── pages/ # One file per page -│ │ └── api.js # API client -│ └── package.json -├── public/ # Built UI (gitignored, generated by npm run build) -├── docs/ -│ └── perspective-pivot.md # Perspective API reference -├── examples/ -│ ├── GETTING_STARTED.md # Tutorial -│ └── bank_transactions.csv -├── .env.example # Config template -├── package.json -└── README.md -``` - -## Comparison to Legacy TPS System - -This project replaces an older system (in `/opt/tps`) that had: -- 2,150 lines of complex SQL with heavy duplication -- 5 nearly-identical 200+ line functions -- Confusing names and deep nested CTEs -- Complex trigger-based processing - -Dataflow achieves the same functionality with: -- ~400 lines of simple SQL -- 4 focused functions -- Clear names and linear logic -- Explicit API-triggered processing - -The simplification makes it easy to understand, modify, and maintain. +- One function, one job; keep functions under 100 lines +- Write clear SQL, not clever SQL +- Add the SQL function to the matching `database/*.sql` file, then the route that calls it +- Update `docs/spec.md` when you add or change an endpoint ## Troubleshooting -**Database connection fails:** -- Check `.env` file exists and has correct credentials -- Verify PostgreSQL is running: `psql -U postgres -l` -- Check search path is set: Should default to `dataflow` schema +**Database connection fails** — check `.env` credentials, that PostgreSQL is running, and +that the search path resolves to the `dataflow` schema. -**Import succeeds but transformation fails:** -- Check rules exist: `SELECT * FROM dataflow.rules WHERE source_name = 'xxx'` -- Verify field names match CSV columns -- Test regex pattern manually -- Check for SQL errors in logs +**Import succeeds but transformation does nothing** — check rules exist for that source +(`SELECT * FROM dataflow.rules WHERE source_name = '…'`), that `field` matches an actual key +in `data`, and test the pattern with `GET /api/rules/preview`. -**All records marked as duplicates:** -- Verify `constraint_fields` match actual field names in data -- Check if data was already imported -- Use different source name for testing +**Everything is marked duplicate** — `constraint_fields` probably don't match the real field +names, or the batch was already imported. -## Adding New Features +## History -When adding features, follow these principles: -- Add ONE function that does ONE thing -- Keep functions under 100 lines if possible -- Write clear SQL, not clever SQL -- Add API endpoint that calls the function -- Document in README.md and update examples - -## Notes for Claude - -- This is a **simple** system by design - don't over-engineer it -- Keep functions focused and linear -- Use JSONB for flexibility, not as a crutch for bad design -- When confused, read the examples/GETTING_STARTED.md walkthrough -- The old TPS system is in `/opt/tps` - this is a clean rewrite, not a refactor +This replaces an older system still in `/opt/tps` — 2,150 lines of SQL with five +nearly-identical 200-line functions and trigger-based processing. Dataflow is a clean +rewrite, not a refactor. Some function bodies still carry `mirrors TPS …` comments pointing +at their counterpart there. diff --git a/PERSPECTIVE.md b/PERSPECTIVE.md deleted file mode 100644 index 190e96d..0000000 --- a/PERSPECTIVE.md +++ /dev/null @@ -1,49 +0,0 @@ -# Perspective — dataflow specifics - -Shared rationale lives in the canonical guide: **`/home/pt/pf_app/PERSPECTIVE.md`** -(loading, version policy, Arrow constraints, deploy pattern, upgrade smoke test). -This file records only what's specific to dataflow. - -> **Distribution:** these are the **`@perspective-dev/*`** packages (repo -> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same -> engine, separate npm scope and release schedule — don't mix the two. - ---- - -## Current state - -- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅ - This is the target loader; pf_app should adopt it. -- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅ - Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow - unless view sizes grow well past 100k. -- **Deploy:** `deploy.sh` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern - for the org; pf_app should copy it. -- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI. - Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`). -- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before - restore — the reference implementation; keep it. - -## The version pair is correct — do NOT "fix" it to 4.4.1 - -`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at -`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only -combination that keeps both of dataflow's hard requirements: - -- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`, - `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export - paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map. -- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**. - -Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with -`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair -stays. Don't touch it. - -**What to actually do:** -- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift - on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that - explicit.) -- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart** - renders, dark/light re-themes, and save→reload→drop-column layout restore. -- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable - 4.5.x suite becomes possible and the pair can collapse to one version. diff --git a/README.md b/README.md index 68cd68d..1240b94 100644 --- a/README.md +++ b/README.md @@ -2,220 +2,71 @@ A simple data transformation tool for importing, cleaning, and standardizing data from various sources. -## What It Does +Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will +deduplicate on import, pull structure out with regex rules, map the extracted values to clean +output, and serve the result through a web UI and REST API. -Dataflow helps you: -1. **Import** CSV data with automatic deduplication -2. **Transform** data using regex rules to extract meaningful information -3. **Map** extracted values to standardized output -4. **Query** the transformed data via a web UI or REST API +## How it works -Perfect for cleaning up messy data like bank transactions, product lists, or any repetitive data that needs normalization. +1. **Sources** define where data comes from and which fields make a record unique +2. **Rules** extract information with regex (`extract` or `replace` mode) — + e.g. pull the merchant out of a transaction description +3. **Mappings** turn extracted values into clean output — + `"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}` +4. **Records** are then queryable, pivotable, and exportable -## Core Concepts +Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output), +and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never +clobbers something you typed by hand. -### 1. Sources -Define where data comes from and how to deduplicate it. +## Stack -**Example:** Bank transactions deduplicated by date + amount + description +PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`. +HTTP Basic auth, configured in `.env`. -### 2. Rules -Extract information using regex patterns (`extract` or `replace` modes). +## Getting started -**Example:** Extract merchant name from transaction description +Requires PostgreSQL 12+, Node.js 18+, and Python 3. -### 3. Mappings -Map extracted values to clean, standardized output. - -**Example:** "DISCOUNT DRUG MART 32" → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}` - -## Architecture - -- **Database:** PostgreSQL with JSONB for flexible data storage -- **API:** Node.js/Express REST API -- **UI:** React SPA served from `public/` -- **Auth:** HTTP Basic auth (configured in `.env`) - -## Design Principles - -- **Simple & Clear** - Easy to understand what's happening -- **Explicit** - No hidden magic or complex triggers -- **Flexible** - Handle varying data formats without schema changes - -## Getting Started - -### Prerequisites -- PostgreSQL 12+ -- Node.js 18+ -- Python 3 (for `manage.py`) - -### Installation - -1. Install Node dependencies: ```bash npm install +python3 manage.py # interactive setup: .env, database, schema, functions, UI, service ``` -2. Run the management script to configure and deploy everything: -```bash -python3 manage.py -``` +The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api` +(port set by `API_PORT` in `.env`). -For development with auto-reload: -```bash -npm run dev -``` +For a walkthrough that creates a source, adds rules and mappings, and imports the sample +CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**. -The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`). +## Documentation -## Management Script (`manage.py`) +| | | +|---|---| +| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl | +| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` | +| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages | +| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference | -`manage.py` is an interactive tool for configuring, deploying, and managing the service. Run it and choose from the numbered menu: - -``` -python3 manage.py -``` - -| # | Action | -|---|--------| -| 1 | **Database configuration** — create/update `.env`, optionally create the PostgreSQL user/database, and deploy schema + functions | -| 2 | Redeploy schema only (`database/schema.sql`) — drops and recreates all tables | -| 3 | Redeploy SQL functions only (`database/queries/`) | -| 4 | Build UI (`ui/` → `public/`) | -| 5 | Set up nginx reverse proxy (HTTP or HTTPS via certbot) | -| 6 | Install systemd service unit (`dataflow.service`) | -| 7 | Start / restart `dataflow.service` | -| 8 | Stop `dataflow.service` | -| 9 | Set login credentials (`LOGIN_USER` / `LOGIN_PASSWORD_HASH` in `.env`) | - -The status screen at the top of the menu shows the current state of each component (database connection, schema, UI build, service, nginx). - -**Typical first-time setup:** run options 1 → 4 → 9 → 6 → 7 (→ 5 if you want nginx). - -## API Reference - -All `/api` routes require HTTP Basic authentication. - -### Sources — `/api/sources` - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/sources` | List all sources | -| POST | `/api/sources` | Create a source | -| GET | `/api/sources/:name` | Get a source | -| PUT | `/api/sources/:name` | Update a source | -| DELETE | `/api/sources/:name` | Delete a source | -| POST | `/api/sources/suggest` | Suggest source definition from CSV upload | -| POST | `/api/sources/:name/import` | Import CSV data and auto-apply transformations to new records | -| GET | `/api/sources/:name/import-log` | View import history (includes `inserted_keys` / `excluded_keys` in `info`) | -| DELETE | `/api/sources/:name/import-log/:id` | Delete an import batch and all its records | -| POST | `/api/sources/:name/transform` | Apply rules and mappings to any untransformed records | -| POST | `/api/sources/:name/reprocess` | Re-transform all records | -| GET | `/api/sources/:name/fields` | List all known field names | -| GET | `/api/sources/:name/stats` | Get record and mapping counts | -| POST | `/api/sources/:name/view` | Generate output view | -| GET | `/api/sources/:name/view-data` | Query output view (paginated, sortable) | - -### Rules — `/api/rules` - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/rules/source/:source_name` | List rules for a source | -| POST | `/api/rules` | Create a rule | -| GET | `/api/rules/:id` | Get a rule | -| PUT | `/api/rules/:id` | Update a rule | -| DELETE | `/api/rules/:id` | Delete a rule | -| GET | `/api/rules/preview` | Preview a pattern against real records (ad-hoc) | -| GET | `/api/rules/:id/test` | Test a saved rule against real records | - -### Mappings — `/api/mappings` - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/mappings/source/:source_name` | List mappings | -| POST | `/api/mappings` | Create a mapping | -| POST | `/api/mappings/bulk` | Bulk create/update mappings | -| GET | `/api/mappings/:id` | Get a mapping | -| PUT | `/api/mappings/:id` | Update a mapping | -| DELETE | `/api/mappings/:id` | Delete a mapping | -| GET | `/api/mappings/source/:source_name/unmapped` | Get values with no mapping yet | -| GET | `/api/mappings/source/:source_name/all-values` | All extracted values with counts | -| GET | `/api/mappings/source/:source_name/counts` | Record counts for existing mappings | -| GET | `/api/mappings/source/:source_name/export.tsv` | Export values as TSV | -| POST | `/api/mappings/source/:source_name/import-csv` | Import mappings from TSV | - -### Records — `/api/records` - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/records/source/:source_name` | List records (paginated) | -| GET | `/api/records/:id` | Get a single record | -| POST | `/api/records/search` | Search records | -| DELETE | `/api/records/:id` | Delete a record | -| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source | - -### Stacks — `/api/stacks` - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/stacks` | List all stacks | -| POST | `/api/stacks` | Create a stack | -| GET | `/api/stacks/:name` | Get a stack | -| PUT | `/api/stacks/:name` | Update a stack | -| DELETE | `/api/stacks/:name` | Delete a stack | -| GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) | -| GET | `/api/stacks/:name/layouts` | List saved pivot layouts | -| POST | `/api/stacks/:name/layouts` | Save a pivot layout | -| DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout | - -## Typical Workflow - -``` -1. Create a source (POST /api/sources) -2. Create transformation rules (POST /api/rules) -3. Import CSV data (POST /api/sources/:name/import) — transformations applied automatically to new records -4. Preview rules against real data (GET /api/rules/preview) -5. Review unmapped values (GET /api/mappings/source/:name/unmapped) -6. Add mappings (POST /api/mappings or bulk import via TSV) -7. Reprocess to apply new mappings (POST /api/sources/:name/reprocess) -8. Query results (GET /api/sources/:name/view-data) -``` - -See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples. - -## Project Structure +## Project structure ``` dataflow/ -├── database/ -│ ├── schema.sql # Table definitions -│ └── queries/ # SQL functions, one file per route -│ ├── sources.sql -│ ├── rules.sql -│ ├── mappings.sql -│ ├── records.sql -│ ├── stacks.sql -│ └── status.sql -├── api/ -│ ├── server.js # Express server -│ ├── middleware/ -│ │ └── auth.js # Basic auth middleware -│ ├── lib/ -│ │ └── sql.js # SQL literal helpers -│ └── routes/ -│ ├── sources.js -│ ├── rules.js -│ ├── mappings.js -│ ├── records.js -│ ├── stacks.js -│ └── status.js -├── public/ # Built React UI (served as static files) -├── examples/ -│ ├── GETTING_STARTED.md -│ └── bank_transactions.csv -└── .env.example +├── manage.py # interactive setup / deploy / uninstall +├── database/ # schema.sql + one .sql file per API route +├── api/ # Express server, routes, auth middleware +├── ui/ # React source (built to public/) +├── public/ # built UI, served as static files +├── docs/ +└── examples/ # sample CSV for the tutorial ``` +Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js` +and `database/rules.sql` are the two halves of the same feature. + +`database/*.sql` is the source of truth for every database function — never edit one directly +in the database, or the next redeploy will silently revert it. + ## License MIT diff --git a/examples/GETTING_STARTED.md b/docs/getting-started.md similarity index 74% rename from examples/GETTING_STARTED.md rename to docs/getting-started.md index bf764d6..4034f45 100644 --- a/examples/GETTING_STARTED.md +++ b/docs/getting-started.md @@ -4,32 +4,40 @@ This guide walks through a complete example using bank transaction data. ## Prerequisites -1. PostgreSQL database running -2. Database created: `CREATE DATABASE dataflow;` -3. `.env` file configured (copy from `.env.example`) +PostgreSQL running, Node.js 18+, and Python 3. -## Step 1: Deploy Database Schema +## Step 1: Configure and Deploy ```bash cd /opt/dataflow -psql -U postgres -d dataflow -f database/schema.sql -psql -U postgres -d dataflow -f database/functions.sql +npm install +python3 manage.py ``` -You should see tables created without errors. +Choose option 1. It writes `.env`, creates the database and user if they don't exist, +then deploys `database/schema.sql` and the SQL function files in dependency order. ## Step 2: Start the API Server ```bash -npm install npm start ``` -The server should start on port 3000 (or your configured port). +The server starts on the port set by `API_PORT` in `.env` (3020 by default). + +Every `/api` route requires HTTP Basic auth using the credentials set by `manage.py` +option 9. The examples below omit it for readability — add `-u username:password` to each +curl, or export it once: + +```bash +alias dfcurl='curl -u username:password' +``` + +`GET /health` is the one route that needs no auth. Test it: ```bash -curl http://localhost:3000/health +curl http://localhost:3020/health # Should return: {"status":"ok","timestamp":"..."} ``` @@ -38,7 +46,7 @@ curl http://localhost:3000/health A source defines where data comes from and how to deduplicate it. ```bash -curl -X POST http://localhost:3000/api/sources \ +curl -X POST http://localhost:3020/api/sources \ -H "Content-Type: application/json" \ -d '{ "name": "bank_transactions", @@ -55,7 +63,7 @@ Rules extract meaningful data using regex patterns. ### Rule 1: Extract merchant name (first part of description) ```bash -curl -X POST http://localhost:3000/api/rules \ +curl -X POST http://localhost:3020/api/rules \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -70,7 +78,7 @@ curl -X POST http://localhost:3000/api/rules \ ### Rule 2: Extract location (city + state pattern) ```bash -curl -X POST http://localhost:3000/api/rules \ +curl -X POST http://localhost:3020/api/rules \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -87,7 +95,7 @@ curl -X POST http://localhost:3000/api/rules \ Import the example CSV file: ```bash -curl -X POST http://localhost:3000/api/sources/bank_transactions/import \ +curl -X POST http://localhost:3020/api/sources/bank_transactions/import \ -F "file=@examples/bank_transactions.csv" ``` @@ -104,7 +112,7 @@ Response: ## Step 6: View Imported Records ```bash -curl http://localhost:3000/api/records/source/bank_transactions?limit=5 +curl http://localhost:3020/api/records/source/bank_transactions?limit=5 ``` You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet! @@ -112,7 +120,7 @@ You'll see the raw imported data. Note that `transformed` is `null` - we haven't ## Step 7: Apply Transformations ```bash -curl -X POST http://localhost:3000/api/sources/bank_transactions/transform +curl -X POST http://localhost:3020/api/sources/bank_transactions/transform ``` Response: @@ -125,7 +133,7 @@ Response: Now check the records again: ```bash -curl http://localhost:3000/api/records/source/bank_transactions?limit=2 +curl http://localhost:3020/api/records/source/bank_transactions?limit=2 ``` You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`. @@ -133,7 +141,7 @@ You'll see the `transformed` field now contains the original data plus extracted ## Step 8: View Extracted Values That Need Mapping ```bash -curl http://localhost:3000/api/mappings/source/bank_transactions/unmapped +curl http://localhost:3020/api/mappings/source/bank_transactions/unmapped ``` Response shows extracted merchant names that aren't mapped yet: @@ -151,7 +159,7 @@ Response shows extracted merchant names that aren't mapped yet: Map extracted values to clean, standardized output: ```bash -curl -X POST http://localhost:3000/api/mappings \ +curl -X POST http://localhost:3020/api/mappings \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -163,7 +171,7 @@ curl -X POST http://localhost:3000/api/mappings \ } }' -curl -X POST http://localhost:3000/api/mappings \ +curl -X POST http://localhost:3020/api/mappings \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -175,7 +183,7 @@ curl -X POST http://localhost:3000/api/mappings \ } }' -curl -X POST http://localhost:3000/api/mappings \ +curl -X POST http://localhost:3020/api/mappings \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -193,13 +201,13 @@ curl -X POST http://localhost:3000/api/mappings \ Clear and reapply transformations to pick up the new mappings: ```bash -curl -X POST http://localhost:3000/api/sources/bank_transactions/reprocess +curl -X POST http://localhost:3020/api/sources/bank_transactions/reprocess ``` ## Step 11: View Final Results ```bash -curl http://localhost:3000/api/records/source/bank_transactions?limit=5 +curl http://localhost:3020/api/records/source/bank_transactions?limit=5 ``` Now the `transformed` field contains: @@ -234,7 +242,7 @@ Example result: Try importing the same file again: ```bash -curl -X POST http://localhost:3000/api/sources/bank_transactions/import \ +curl -X POST http://localhost:3020/api/sources/bank_transactions/import \ -F "file=@examples/bank_transactions.csv" ``` @@ -272,19 +280,19 @@ You've now: ```bash # View all sources -curl http://localhost:3000/api/sources +curl http://localhost:3020/api/sources # View source statistics -curl http://localhost:3000/api/sources/bank_transactions/stats +curl http://localhost:3020/api/sources/bank_transactions/stats # View all rules for a source -curl http://localhost:3000/api/rules/source/bank_transactions +curl http://localhost:3020/api/rules/source/bank_transactions # View all mappings for a source -curl http://localhost:3000/api/mappings/source/bank_transactions +curl http://localhost:3020/api/mappings/source/bank_transactions # Search for specific records -curl -X POST http://localhost:3000/api/records/search \ +curl -X POST http://localhost:3020/api/records/search \ -H "Content-Type: application/json" \ -d '{ "source_name": "bank_transactions", @@ -301,11 +309,11 @@ curl -X POST http://localhost:3000/api/records/search \ - Check logs for error messages **Import fails:** -- Verify source exists: `curl http://localhost:3000/api/sources` +- Verify source exists: `curl http://localhost:3020/api/sources` - Check CSV format matches expectations - Ensure constraint_fields match CSV column names **Transformations not working:** -- Check rules exist: `curl http://localhost:3000/api/rules/source/bank_transactions` +- Check rules exist: `curl http://localhost:3020/api/rules/source/bank_transactions` - Test regex pattern manually - Check records have the specified field diff --git a/docs/perspective-pivot.md b/docs/perspective.md similarity index 76% rename from docs/perspective-pivot.md rename to docs/perspective.md index 514b91b..afb7aa2 100644 --- a/docs/perspective-pivot.md +++ b/docs/perspective.md @@ -1,11 +1,68 @@ -# Perspective Pivot — Technical Reference +# Perspective -Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply. +Everything about the Perspective pivot in dataflow: which packages and versions are +pinned and why, and a ground-truth reference for the parts of the API the official docs +don't cover. -This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement. +Shared rationale across projects lives in the canonical guide at +`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy +pattern, upgrade smoke test). This file records what's specific to dataflow. --- +> **Distribution:** these are the **`@perspective-dev/*`** packages (repo +> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same +> engine, separate npm scope and release schedule — don't mix the two. + +--- + +## Current state + +- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅ + This is the target loader; pf_app should adopt it. +- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅ + Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow + unless view sizes grow well past 100k. +- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern + for the org; pf_app should copy it. +- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI. + Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`). +- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before + restore — the reference implementation; keep it. + +## The version pair is correct — do NOT "fix" it to 4.4.1 + +`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at +`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only +combination that keeps both of dataflow's hard requirements: + +- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`, + `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export + paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map. +- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**. + +Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with +`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair +stays. Don't touch it. + +**What to actually do:** +- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift + on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that + explicit.) +- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart** + renders, dark/light re-themes, and save→reload→drop-column layout restore. +- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable + 4.5.x suite becomes possible and the pair can collapse to one version. + +--- + +# API Reference + +Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at +**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been +re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete +for some of these APIs — treat this as a ground-truth supplement. + ## Loading via npm ```js diff --git a/SPEC.md b/docs/spec.md similarity index 74% rename from SPEC.md rename to docs/spec.md index fb8d33d..05684cb 100644 --- a/SPEC.md +++ b/docs/spec.md @@ -18,7 +18,7 @@ SQL functions are the single source of truth for business logic. The API layer i Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does. ### One SQL file per route -SQL is organized in `database/queries/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`). This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation. +SQL is organized in `database/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `stacks.sql`, `status.sql`) plus `import.sql` and `transform.sql` for the import/transform engine. This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation. ### Explicit over implicit Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs. @@ -34,11 +34,14 @@ Raw imported records and transformed records are stored as JSONB. This avoids sc manage.py — interactive CLI for setup, deployment, and management database/ schema.sql — table definitions (run once or to reset) - queries/ - sources.sql — all SQL for /api/sources - rules.sql — all SQL for /api/rules - mappings.sql — all SQL for /api/mappings - records.sql — all SQL for /api/records + sources.sql — all SQL for /api/sources + rules.sql — all SQL for /api/rules + mappings.sql — all SQL for /api/mappings + records.sql — all SQL for /api/records + stacks.sql — all SQL for /api/stacks + status.sql — all SQL for /api/status + import.sql — CSV import and the import audit trail + transform.sql — the rule/mapping engine api/ server.js — Express server, mounts routes, auth middleware middleware/ @@ -64,8 +67,14 @@ ui/ Mappings.jsx — mapping table with TSV import/export Records.jsx — paginated, sortable view of transformed records Pivot.jsx — interactive pivot table with cell inspector + Stacks.jsx — multi-source union views with running balance + Remap.jsx — bulk remap of an output field value across mappings Log.jsx — global import log across all sources + components/ — Sidebar, StatusBar + theme.jsx — light/dark context provider public/ — compiled UI (output of npm run build in ui/) +docs/ — this file, tutorial, UI and Perspective references +examples/ — bank_transactions.csv, the tutorial's sample data ``` --- @@ -123,7 +132,7 @@ The transform is fully set-based — no row-by-row loops. All records for a sour ## SQL Functions -Each file in `database/queries/` maps 1-to-1 with a route file. +Each route file has a matching SQL file in `database/`; `import.sql` and `transform.sql` hold the engine shared by several routes. **sources.sql** `list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view` @@ -153,53 +162,104 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea **Route summary:** +### Sources — `api/routes/sources.js` + | Method | Path | Description | |--------|------|-------------| | GET | /api/sources | List all sources | | POST | /api/sources | Create source | | GET | /api/sources/:name | Get source | -| PUT | /api/sources/:name | Update source (constraint_fields, config) | -| DELETE | /api/sources/:name | Delete source and all data | -| POST | /api/sources/suggest | Suggest source config from CSV upload | -| POST | /api/sources/:name/import | Import CSV records | -| GET | /api/sources/:name/import-log | Import history | -| GET | /api/sources/:name/stats | Record counts | -| GET | /api/sources/:name/fields | All known field names and origins | -| GET | /api/sources/:name/view-data | Paginated, sortable view data | -| POST | /api/sources/:name/transform | Apply transformations (new records only) | +| PUT | /api/sources/:name | Update source (constraint_fields, config, global_picklist) | +| DELETE | /api/sources/:name | Delete source and all its data | +| POST | /api/sources/suggest | Suggest source config from an uploaded CSV | +| POST | /api/sources/:name/import | Import CSV; transformations are applied to the new records | +| GET | /api/sources/import-log | Import history across all sources | +| GET | /api/sources/:name/import-log | Import history for one source | +| DELETE | /api/sources/:name/import-log/:id | Delete an import batch and every record in it | +| POST | /api/sources/:name/transform | Apply transformations to untransformed records only | | POST | /api/sources/:name/reprocess | Reapply transformations to all records | -| POST | /api/sources/:name/view | Generate dfv view | +| GET | /api/sources/:name/stats | Record counts | +| GET | /api/sources/:name/fields | All known field names and their origins | +| GET | /api/sources/:name/override-keys | Distinct field names used in overrides for this source | +| POST | /api/sources/:name/view | Generate/refresh the `dfv` view | +| GET | /api/sources/:name/view-data | Paginated, sortable, filterable view data | +| GET | /api/sources/:name/layouts | List saved pivot layouts | +| POST | /api/sources/:name/layouts | Save a pivot layout | +| DELETE | /api/sources/:name/layouts/:id | Delete a pivot layout | + +### Rules — `api/routes/rules.js` + +| Method | Path | Description | +|--------|------|-------------| | GET | /api/rules/source/:name | List rules for a source | -| GET | /api/rules/preview | Preview pattern against live records | -| GET | /api/rules/:id/test | Test saved rule against live records | +| GET | /api/rules/:id | Get a rule | | POST | /api/rules | Create rule | | PUT | /api/rules/:id | Update rule | | DELETE | /api/rules/:id | Delete rule | +| GET | /api/rules/preview | Preview an ad-hoc pattern against live records | +| GET | /api/rules/:id/test | Test a saved rule against live records | + +### Mappings — `api/routes/mappings.js` + +| Method | Path | Description | +|--------|------|-------------| | GET | /api/mappings/source/:name | List mappings | -| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) | -| GET | /api/mappings/source/:name/unmapped | Only unmapped extracted values | -| GET | /api/mappings/source/:name/counts | Record counts per mapping | -| GET | /api/mappings/source/:name/export.tsv | Export mappings as TSV | -| POST | /api/mappings/source/:name/import-csv | Import/update mappings from TSV | +| GET | /api/mappings/:id | Get a mapping | | POST | /api/mappings | Create mapping | | POST | /api/mappings/bulk | Upsert multiple mappings | | PUT | /api/mappings/:id | Update mapping | | DELETE | /api/mappings/:id | Delete mapping | -| GET | /api/records/source/:name | List raw records | -| GET | /api/records/:id | Get single record | +| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) with counts | +| GET | /api/mappings/source/:name/unmapped | Only values with no mapping yet | +| GET | /api/mappings/source/:name/counts | Record counts per mapping | +| GET | /api/mappings/source/:name/export.tsv | Export extracted values as TSV | +| POST | /api/mappings/source/:name/import-csv | Import/update mappings from an uploaded TSV | +| GET | /api/mappings/global-values | Output values across all `global_picklist` sources (autocomplete) | +| GET | /api/mappings/outputs | Search output field values across all mappings | +| GET | /api/mappings/outputs/:col/:val | Mappings carrying a specific output field value | +| POST | /api/mappings/remap-field | Replace an output field value across all mappings | + +### Records — `api/routes/records.js` + +| Method | Path | Description | +|--------|------|-------------| +| GET | /api/records/source/:name | List records (paginated) | +| GET | /api/records/:id | Get a single record | | POST | /api/records/search | Search by JSONB containment | | DELETE | /api/records/:id | Delete record | | DELETE | /api/records/source/:name/all | Delete all records for a source | +| PUT | /api/records/:id/overrides | Set manual overrides on a record | +| DELETE | /api/records/:id/overrides | Clear a record's overrides | +| POST | /api/records/bulk-overrides | Apply the same overrides to many records | + +### Stacks — `api/routes/stacks.js` + +| Method | Path | Description | +|--------|------|-------------| | GET | /api/stacks | List all stacks | +| GET | /api/stacks/:name | Get a stack with its sources | | POST | /api/stacks | Create stack | -| GET | /api/stacks/:name | Get stack | | PUT | /api/stacks/:name | Update stack | | DELETE | /api/stacks/:name | Delete stack | +| PUT | /api/stacks/:name/sources/:source | Add or update a source in the stack | +| DELETE | /api/stacks/:name/sources/:source | Remove a source from the stack | +| PUT | /api/stacks/:name/sources/reorder | Reorder the stack's sources | +| GET | /api/stacks/:name/view-sql | Preview the SQL that would build the view (dry run) | +| POST | /api/stacks/:name/view | Generate/refresh the `dfv` view | +| POST | /api/stacks/:name/exec-sql | Execute user-edited SQL for the view | | GET | /api/stacks/:name/view-data | Paginated stacked data with running balance | +| GET | /api/stacks/:name/balance | Current running balance from the generated view | +| POST | /api/stacks/:name/calibrate | Set the balance offset from a known balance at a date | | GET | /api/stacks/:name/layouts | List saved pivot layouts | -| POST | /api/stacks/:name/layouts | Save pivot layout | -| DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout | +| POST | /api/stacks/:name/layouts | Save a pivot layout | +| DELETE | /api/stacks/:name/layouts/:id | Delete a pivot layout | + +### Status — `api/routes/status.js` + +| Method | Path | Description | +|--------|------|-------------| | GET | /api/status | Deployment status | +| GET | /health | Health check (no auth) | --- @@ -253,7 +313,7 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s - `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save. - Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently. - See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically. + See `docs/perspective.md` for the full technical reference on controlling Perspective programmatically. - **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table. @@ -280,7 +340,7 @@ Shows current status on every screen: 2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation. -3. **Redeploy SQL functions** — Runs all four files in `database/queries/` in order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`. Safe to run at any time without data loss. +3. **Redeploy SQL functions** — Runs the function files in `database/` in dependency order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `import.sql`, `transform.sql`, `stacks.sql`, `status.sql`. Safe to run at any time without data loss. 4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`. @@ -294,6 +354,8 @@ Shows current status on every screen: 9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed. +10. **Uninstall** — Reverses everything the other options install, in reverse order: stops/disables/removes the systemd unit, removes the nginx site and reloads nginx, drops the database and its user (prompts for admin credentials), then deletes `.env`, `public/`, and `node_modules`. Lists exactly what it found before doing anything and requires typing `delete` to proceed. The repository itself is left in place. + **Key behaviors:** - All commands that will be run are printed before the user is asked to confirm. - Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal. @@ -341,13 +403,18 @@ The server binds to `0.0.0.0` on `API_PORT` and serves both the API and the comp ## Deploying SQL Changes -Any time SQL functions are modified: +Any time SQL functions are modified, run `python3 manage.py` and choose "Redeploy SQL +functions only". It runs every function file in dependency order — the list lives in +`QUERY_FILES` in `manage.py`, which is the one place the order is defined. + +To deploy a single file by hand: ```bash -PGPASSWORD= psql -h -U -d -f database/queries/sources.sql -PGPASSWORD= psql -h -U -d -f database/queries/rules.sql -PGPASSWORD= psql -h -U -d -f database/queries/mappings.sql -PGPASSWORD= psql -h -U -d -f database/queries/records.sql +PGPASSWORD= psql -h -U -d -v ON_ERROR_STOP=1 -f database/rules.sql ``` -Then restart the server. Function deployment is safe to repeat — all functions use `CREATE OR REPLACE`. + +Deployment is safe to repeat — every function uses `CREATE OR REPLACE`. + +**The files are the source of truth.** Editing a function directly in the database, without +writing the change back to its file, means the next redeploy silently reverts it. Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead. diff --git a/ui/README.md b/docs/ui.md similarity index 100% rename from ui/README.md rename to docs/ui.md