Consolidate documentation into docs/ and cut the duplication
Architecture, file structure, the manage.py menu, and the API reference were each documented in two or three of README.md, SPEC.md, and CLAUDE.md — the same drift trap the SQL just had. SPEC.md, examples/GETTING_STARTED.md, and ui/README.md move into docs/. PERSPECTIVE.md and docs/perspective-pivot.md merge into docs/perspective.md, version rationale first, then the API reference. README.md becomes an entry point that links out, and CLAUDE.md keeps only working rules and non-obvious behaviour, pointing at docs/spec.md for the rest. examples/ keeps just the sample CSV the tutorial loads. Corrections found while consolidating: - the spec's API table was missing 20 routes — every override endpoint, most of /api/stacks, the mapping remap routes, /health. Rebuilt from the route files - the tutorial used port 3000 (default is 3020) and never mentioned Basic auth, so every curl in it would have 401'd - the tutorial and the spec each hand-listed the SQL deploy order; both now point at manage.py, which is where the order actually lives - CLAUDE.md described deduplication as an MD5 hash (it is a plain JSONB object), claimed 5 tables and 4 functions, and told you to run a setup.sh that has not existed for some time Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
2ea2548715
commit
7dcd8c4b61
282
CLAUDE.md
282
CLAUDE.md
@ -2,147 +2,70 @@
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Overview
|
||||
Dataflow imports CSV data, extracts structure from it with regex rules, maps the extracted
|
||||
values to standardized output, and serves the result over a REST API and React UI. It is a
|
||||
**simple system by design** — don't over-engineer it.
|
||||
|
||||
Dataflow is a simple data transformation tool for importing, cleaning, and standardizing data from various sources. Built with PostgreSQL and Node.js/Express, it emphasizes clarity and simplicity over complexity.
|
||||
**Read [docs/spec.md](docs/spec.md) for architecture, schema, data flow, the full API, and
|
||||
`manage.py`.** This file covers only what you need to work in the repo without breaking
|
||||
something — the rules and non-obvious behaviours that aren't visible from the code.
|
||||
|
||||
## Core Concepts
|
||||
## Where things live
|
||||
|
||||
1. **Sources** - Define data sources and deduplication rules (which fields make a record unique)
|
||||
2. **Import** - Load CSV data, automatically deduplicating based on source rules
|
||||
3. **Rules** - Extract information using regex patterns (e.g., extract merchant from transaction description)
|
||||
4. **Mappings** - Map extracted values to standardized output (e.g., "WALMART" → {"vendor": "Walmart", "category": "Groceries"})
|
||||
5. **Transform** - Apply rules and mappings to create clean, enriched data
|
||||
Both the API routes and the SQL are one file per resource: `api/routes/rules.js` and
|
||||
`database/rules.sql` are two halves of the same feature. Two SQL files are shared engines
|
||||
rather than per-route — `import.sql` (CSV import, audit trail) and `transform.sql` (the
|
||||
rule/mapping engine, including the `jsonb_concat_obj` aggregate).
|
||||
|
||||
## Architecture
|
||||
`manage.py`'s `QUERY_FILES` list is the deploy order and the authoritative file list.
|
||||
|
||||
### Database Schema (`database/schema.sql`)
|
||||
## Rules that matter
|
||||
|
||||
**5 simple tables:**
|
||||
- `sources` - Source definitions with `constraint_fields` array
|
||||
- `records` - Imported data with `data` (raw) and `transformed` (enriched) JSONB columns
|
||||
- `rules` - Regex extraction rules with `field`, `pattern`, `output_field`
|
||||
- `mappings` - Input/output value mappings
|
||||
- `import_log` - Audit trail
|
||||
**`database/*.sql` is the source of truth for every database function. Never edit a function
|
||||
directly in the database.** A live edit that isn't written back to the file is silently
|
||||
reverted the next time anyone runs "Redeploy SQL functions". This has already happened once:
|
||||
five functions drifted and sat wrong in the repo for months — see the git history of the
|
||||
deleted `database/functions.sql`.
|
||||
|
||||
**Key design:**
|
||||
- JSONB for flexible data storage
|
||||
- Deduplication via MD5 hash of specified fields
|
||||
- Simple, flat structure (no complex relationships)
|
||||
**Always run `npm run build` from `ui/` after any change to `ui/src/`.** The Express server
|
||||
serves the built output in `public/`; source changes are invisible until you rebuild.
|
||||
|
||||
### Database Functions (`database/functions.sql`)
|
||||
**Never use `ON CONFLICT (constraint_key)`.** See deduplication below — there is no unique
|
||||
constraint, and adding one would drop legitimate transactions.
|
||||
|
||||
**4 focused functions:**
|
||||
- `import_records(source_name, data)` - Import with deduplication
|
||||
- `apply_transformations(source_name, record_ids)` - Apply rules and mappings
|
||||
- `get_unmapped_values(source_name, rule_name)` - Find values needing mappings
|
||||
- `reprocess_records(source_name)` - Re-transform all records
|
||||
## The three data layers
|
||||
|
||||
**Design principle:** Each function does ONE thing. No nested CTEs, no duplication.
|
||||
Each row in `records` keeps its data in three JSONB columns:
|
||||
|
||||
### API Server (`api/server.js` + `api/routes/`)
|
||||
- `data` — raw imported values, never modified
|
||||
- `transformed` — rule and mapping output only (the delta)
|
||||
- `overrides` — manual edits, highest precedence
|
||||
|
||||
**RESTful endpoints:**
|
||||
- `/api/sources` - CRUD sources, import CSV, trigger transformations
|
||||
- `/api/rules` - CRUD transformation rules
|
||||
- `/api/mappings` - CRUD value mappings, view unmapped values
|
||||
- `/api/records` - Query and search transformed data
|
||||
Readers merge them as `data || transformed || overrides`. Keeping them separate is what lets
|
||||
`reprocess_records` re-run the rules without clobbering a manual edit. Anything that writes
|
||||
overrides into `transformed` is a bug — that was the pre-May-2026 behaviour.
|
||||
|
||||
**Route files:**
|
||||
- `routes/sources.js` - Source management and CSV import
|
||||
- `routes/rules.js` - Rule management
|
||||
- `routes/mappings.js` - Mapping management + unmapped values
|
||||
- `routes/records.js` - Record queries and search
|
||||
## Deduplication
|
||||
|
||||
## Common Development Tasks
|
||||
- `constraint_key` is a JSONB object of the constraint field values — readable, no hashing
|
||||
- Dedup is enforced at import time in a CTE. There is **no unique DB constraint** on it
|
||||
- **The constraint key is cross-batch re-import protection, not record uniqueness**
|
||||
- Within one import batch, all rows insert even when constraint keys collide. Banks
|
||||
legitimately send identical-looking transactions — 11 separate Cedar Point charges on the
|
||||
same day are 11 real rows
|
||||
- On re-import of an overlapping date range, rows whose key already exists are skipped, so
|
||||
re-running a month-to-date export the next day doesn't double-count
|
||||
- Deleting an import log entry cascades to every record in that batch (`import_id` FK)
|
||||
|
||||
### Running the Application
|
||||
## Error handling
|
||||
|
||||
```bash
|
||||
# Setup (first time only)
|
||||
./setup.sh
|
||||
|
||||
# Start development server with auto-reload
|
||||
npm run dev
|
||||
|
||||
# Start production server
|
||||
npm start
|
||||
|
||||
# Test API
|
||||
curl http://localhost:3000/health
|
||||
```
|
||||
|
||||
### Database Changes
|
||||
|
||||
When modifying schema:
|
||||
1. Edit `database/schema.sql`
|
||||
2. Drop and recreate schema: `psql -d dataflow -f database/schema.sql`
|
||||
3. Redeploy functions: `psql -d dataflow -f database/functions.sql`
|
||||
|
||||
For production, write migration scripts instead of dropping schema.
|
||||
|
||||
### Adding a New API Endpoint
|
||||
|
||||
1. Add route to appropriate file in `api/routes/`
|
||||
2. Follow existing patterns (async/await, error handling via `next()`)
|
||||
3. Use parameterized queries to prevent SQL injection
|
||||
4. Return consistent JSON format
|
||||
|
||||
### Testing
|
||||
|
||||
Manual testing workflow:
|
||||
1. Create a source: `POST /api/sources`
|
||||
2. Create rules: `POST /api/rules`
|
||||
3. Import data: `POST /api/sources/:name/import`
|
||||
4. Apply transformations: `POST /api/sources/:name/transform`
|
||||
5. View results: `GET /api/records/source/:name`
|
||||
|
||||
See `examples/GETTING_STARTED.md` for complete curl examples.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Simple over clever** - Straightforward code beats optimization
|
||||
2. **Explicit over implicit** - No magic, no hidden triggers
|
||||
3. **Clear naming** - `data` not `rec`, `transformed` not `allj`
|
||||
4. **One function, one job** - No 250-line functions
|
||||
5. **JSONB for flexibility** - Handle varying schemas without migrations
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Import Flow
|
||||
```
|
||||
CSV file → parse → import_records() → records table (data column)
|
||||
```
|
||||
|
||||
### Transformation Flow
|
||||
```
|
||||
records.data → apply_transformations() →
|
||||
- Apply each rule (regex extraction)
|
||||
- Look up mappings
|
||||
- Merge into records.transformed
|
||||
```
|
||||
|
||||
### Deduplication
|
||||
- `constraint_key` is a JSONB object of the constraint field values (readable, no hashing)
|
||||
- Dedup is enforced at import time via CTE — NO unique DB constraint on constraint_key
|
||||
- **The constraint key is for cross-batch re-import protection, NOT record uniqueness**
|
||||
- Within a single import batch, ALL rows insert regardless of duplicate constraint keys
|
||||
- Banks legitimately send multiple identical-looking transactions (same date, description, amount)
|
||||
- Example: 11 Cedar Point merchandise charges on one day — all should insert in one batch
|
||||
- On re-import of overlapping date range, rows whose constraint_key already exists in DB are skipped
|
||||
- This prevents double-counting when you re-run a month-to-date export the next day
|
||||
- NEVER use `ON CONFLICT (constraint_key)` — there is no unique constraint and it would wrongly
|
||||
drop legitimate duplicate transactions from the same batch
|
||||
- Deleting an import log entry cascades to all records from that batch (import_id FK)
|
||||
|
||||
### Error Handling
|
||||
- API routes use `try/catch` and pass errors to `next(err)`
|
||||
- Server.js has global error handler
|
||||
- Database functions return JSON with `success` boolean
|
||||
API routes use `try/catch` and pass errors to `next(err)`; `server.js` has a global handler.
|
||||
Database functions return JSON with a `success` boolean.
|
||||
|
||||
## Light / dark mode
|
||||
|
||||
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
|
||||
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a
|
||||
`ThemeProvider` that wraps the app in `main.jsx`.
|
||||
|
||||
- **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
|
||||
- **Toggle:** button in the sidebar header in `App.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
|
||||
@ -151,20 +74,10 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit
|
||||
- **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires
|
||||
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
|
||||
|
||||
## UI (React + Vite)
|
||||
## Pivot inspector panel
|
||||
|
||||
The frontend lives in `ui/src/` and is built to `public/` via `npm run build` from the `ui/` directory. **Always run `npm run build` from `ui/` after any changes to `ui/src/` files.**
|
||||
|
||||
### Pages
|
||||
|
||||
- **Sources / Rules / Mappings / Records** — standard CRUD pages
|
||||
- **Pivot** (`ui/src/pages/Pivot.jsx`) — interactive pivot/crosstab powered by Perspective (`@perspective-dev` v4.5.1, installed via npm). See `docs/perspective-pivot.md` for the full Perspective API reference.
|
||||
- **Stacks** — multi-source union views with running balance
|
||||
- **Log** — import audit trail
|
||||
|
||||
### Pivot inspector panel
|
||||
|
||||
Clicking a data cell opens a right-hand inspector panel showing the underlying transactions for that cell. Key behaviors:
|
||||
Clicking a data cell opens a right-hand inspector panel showing the underlying transactions
|
||||
for that cell. See [docs/perspective.md](docs/perspective.md) for the Perspective API itself.
|
||||
|
||||
- **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views.
|
||||
- **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click.
|
||||
@ -173,87 +86,38 @@ Clicking a data cell opens a right-hand inspector panel showing the underlying t
|
||||
- The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px).
|
||||
- The transaction table is sortable (click header) and shows column totals for all-numeric columns.
|
||||
|
||||
### Pivot layout persistence
|
||||
## Pivot layout persistence
|
||||
|
||||
Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The `source_name` column holds either a source name or a stack name — the FK to `sources(name)` was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use `/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`). `localStorage` is still used to remember the *last active layout* for a view (the `psp_layout_<name>` key), but named layout definitions live in the DB so they persist across machines.
|
||||
Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The
|
||||
`source_name` column holds either a source name or a stack name — the FK to `sources(name)`
|
||||
was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use
|
||||
`/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`,
|
||||
`save_pivot_layout`, `delete_pivot_layout`). `localStorage` still remembers the *last active
|
||||
layout* for a view (the `psp_layout_<name>` key), but the definitions live in the DB so they
|
||||
persist across machines.
|
||||
|
||||
## File Structure
|
||||
## Adding features
|
||||
|
||||
```
|
||||
dataflow/
|
||||
├── database/
|
||||
│ ├── schema.sql # Table definitions
|
||||
│ └── functions.sql # Import/transform functions
|
||||
├── api/
|
||||
│ ├── server.js # Express server
|
||||
│ └── routes/ # API endpoints
|
||||
│ ├── sources.js
|
||||
│ ├── rules.js
|
||||
│ ├── mappings.js
|
||||
│ └── records.js
|
||||
├── ui/
|
||||
│ ├── src/
|
||||
│ │ ├── pages/ # One file per page
|
||||
│ │ └── api.js # API client
|
||||
│ └── package.json
|
||||
├── public/ # Built UI (gitignored, generated by npm run build)
|
||||
├── docs/
|
||||
│ └── perspective-pivot.md # Perspective API reference
|
||||
├── examples/
|
||||
│ ├── GETTING_STARTED.md # Tutorial
|
||||
│ └── bank_transactions.csv
|
||||
├── .env.example # Config template
|
||||
├── package.json
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Comparison to Legacy TPS System
|
||||
|
||||
This project replaces an older system (in `/opt/tps`) that had:
|
||||
- 2,150 lines of complex SQL with heavy duplication
|
||||
- 5 nearly-identical 200+ line functions
|
||||
- Confusing names and deep nested CTEs
|
||||
- Complex trigger-based processing
|
||||
|
||||
Dataflow achieves the same functionality with:
|
||||
- ~400 lines of simple SQL
|
||||
- 4 focused functions
|
||||
- Clear names and linear logic
|
||||
- Explicit API-triggered processing
|
||||
|
||||
The simplification makes it easy to understand, modify, and maintain.
|
||||
- One function, one job; keep functions under 100 lines
|
||||
- Write clear SQL, not clever SQL
|
||||
- Add the SQL function to the matching `database/*.sql` file, then the route that calls it
|
||||
- Update `docs/spec.md` when you add or change an endpoint
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Database connection fails:**
|
||||
- Check `.env` file exists and has correct credentials
|
||||
- Verify PostgreSQL is running: `psql -U postgres -l`
|
||||
- Check search path is set: Should default to `dataflow` schema
|
||||
**Database connection fails** — check `.env` credentials, that PostgreSQL is running, and
|
||||
that the search path resolves to the `dataflow` schema.
|
||||
|
||||
**Import succeeds but transformation fails:**
|
||||
- Check rules exist: `SELECT * FROM dataflow.rules WHERE source_name = 'xxx'`
|
||||
- Verify field names match CSV columns
|
||||
- Test regex pattern manually
|
||||
- Check for SQL errors in logs
|
||||
**Import succeeds but transformation does nothing** — check rules exist for that source
|
||||
(`SELECT * FROM dataflow.rules WHERE source_name = '…'`), that `field` matches an actual key
|
||||
in `data`, and test the pattern with `GET /api/rules/preview`.
|
||||
|
||||
**All records marked as duplicates:**
|
||||
- Verify `constraint_fields` match actual field names in data
|
||||
- Check if data was already imported
|
||||
- Use different source name for testing
|
||||
**Everything is marked duplicate** — `constraint_fields` probably don't match the real field
|
||||
names, or the batch was already imported.
|
||||
|
||||
## Adding New Features
|
||||
## History
|
||||
|
||||
When adding features, follow these principles:
|
||||
- Add ONE function that does ONE thing
|
||||
- Keep functions under 100 lines if possible
|
||||
- Write clear SQL, not clever SQL
|
||||
- Add API endpoint that calls the function
|
||||
- Document in README.md and update examples
|
||||
|
||||
## Notes for Claude
|
||||
|
||||
- This is a **simple** system by design - don't over-engineer it
|
||||
- Keep functions focused and linear
|
||||
- Use JSONB for flexibility, not as a crutch for bad design
|
||||
- When confused, read the examples/GETTING_STARTED.md walkthrough
|
||||
- The old TPS system is in `/opt/tps` - this is a clean rewrite, not a refactor
|
||||
This replaces an older system still in `/opt/tps` — 2,150 lines of SQL with five
|
||||
nearly-identical 200-line functions and trigger-based processing. Dataflow is a clean
|
||||
rewrite, not a refactor. Some function bodies still carry `mirrors TPS …` comments pointing
|
||||
at their counterpart there.
|
||||
|
||||
@ -1,49 +0,0 @@
|
||||
# Perspective — dataflow specifics
|
||||
|
||||
Shared rationale lives in the canonical guide: **`/home/pt/pf_app/PERSPECTIVE.md`**
|
||||
(loading, version policy, Arrow constraints, deploy pattern, upgrade smoke test).
|
||||
This file records only what's specific to dataflow.
|
||||
|
||||
> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
|
||||
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
|
||||
> engine, separate npm scope and release schedule — don't mix the two.
|
||||
|
||||
---
|
||||
|
||||
## Current state
|
||||
|
||||
- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
|
||||
This is the target loader; pf_app should adopt it.
|
||||
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
|
||||
Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
|
||||
unless view sizes grow well past 100k.
|
||||
- **Deploy:** `deploy.sh` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
|
||||
for the org; pf_app should copy it.
|
||||
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
|
||||
Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
|
||||
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
|
||||
restore — the reference implementation; keep it.
|
||||
|
||||
## The version pair is correct — do NOT "fix" it to 4.4.1
|
||||
|
||||
`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
|
||||
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
|
||||
combination that keeps both of dataflow's hard requirements:
|
||||
|
||||
- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
|
||||
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
|
||||
paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
|
||||
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
|
||||
|
||||
Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
|
||||
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
|
||||
stays. Don't touch it.
|
||||
|
||||
**What to actually do:**
|
||||
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
|
||||
on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
|
||||
explicit.)
|
||||
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
|
||||
renders, dark/light re-themes, and save→reload→drop-column layout restore.
|
||||
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
|
||||
4.5.x suite becomes possible and the pair can collapse to one version.
|
||||
237
README.md
237
README.md
@ -2,220 +2,71 @@
|
||||
|
||||
A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
|
||||
|
||||
## What It Does
|
||||
Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will
|
||||
deduplicate on import, pull structure out with regex rules, map the extracted values to clean
|
||||
output, and serve the result through a web UI and REST API.
|
||||
|
||||
Dataflow helps you:
|
||||
1. **Import** CSV data with automatic deduplication
|
||||
2. **Transform** data using regex rules to extract meaningful information
|
||||
3. **Map** extracted values to standardized output
|
||||
4. **Query** the transformed data via a web UI or REST API
|
||||
## How it works
|
||||
|
||||
Perfect for cleaning up messy data like bank transactions, product lists, or any repetitive data that needs normalization.
|
||||
1. **Sources** define where data comes from and which fields make a record unique
|
||||
2. **Rules** extract information with regex (`extract` or `replace` mode) —
|
||||
e.g. pull the merchant out of a transaction description
|
||||
3. **Mappings** turn extracted values into clean output —
|
||||
`"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
|
||||
4. **Records** are then queryable, pivotable, and exportable
|
||||
|
||||
## Core Concepts
|
||||
Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output),
|
||||
and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never
|
||||
clobbers something you typed by hand.
|
||||
|
||||
### 1. Sources
|
||||
Define where data comes from and how to deduplicate it.
|
||||
## Stack
|
||||
|
||||
**Example:** Bank transactions deduplicated by date + amount + description
|
||||
PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`.
|
||||
HTTP Basic auth, configured in `.env`.
|
||||
|
||||
### 2. Rules
|
||||
Extract information using regex patterns (`extract` or `replace` modes).
|
||||
## Getting started
|
||||
|
||||
**Example:** Extract merchant name from transaction description
|
||||
Requires PostgreSQL 12+, Node.js 18+, and Python 3.
|
||||
|
||||
### 3. Mappings
|
||||
Map extracted values to clean, standardized output.
|
||||
|
||||
**Example:** "DISCOUNT DRUG MART 32" → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Database:** PostgreSQL with JSONB for flexible data storage
|
||||
- **API:** Node.js/Express REST API
|
||||
- **UI:** React SPA served from `public/`
|
||||
- **Auth:** HTTP Basic auth (configured in `.env`)
|
||||
|
||||
## Design Principles
|
||||
|
||||
- **Simple & Clear** - Easy to understand what's happening
|
||||
- **Explicit** - No hidden magic or complex triggers
|
||||
- **Flexible** - Handle varying data formats without schema changes
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- PostgreSQL 12+
|
||||
- Node.js 18+
|
||||
- Python 3 (for `manage.py`)
|
||||
|
||||
### Installation
|
||||
|
||||
1. Install Node dependencies:
|
||||
```bash
|
||||
npm install
|
||||
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
|
||||
```
|
||||
|
||||
2. Run the management script to configure and deploy everything:
|
||||
```bash
|
||||
python3 manage.py
|
||||
```
|
||||
The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api`
|
||||
(port set by `API_PORT` in `.env`).
|
||||
|
||||
For development with auto-reload:
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
For a walkthrough that creates a source, adds rules and mappings, and imports the sample
|
||||
CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**.
|
||||
|
||||
The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`).
|
||||
## Documentation
|
||||
|
||||
## Management Script (`manage.py`)
|
||||
| | |
|
||||
|---|---|
|
||||
| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl |
|
||||
| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` |
|
||||
| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages |
|
||||
| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference |
|
||||
|
||||
`manage.py` is an interactive tool for configuring, deploying, and managing the service. Run it and choose from the numbered menu:
|
||||
|
||||
```
|
||||
python3 manage.py
|
||||
```
|
||||
|
||||
| # | Action |
|
||||
|---|--------|
|
||||
| 1 | **Database configuration** — create/update `.env`, optionally create the PostgreSQL user/database, and deploy schema + functions |
|
||||
| 2 | Redeploy schema only (`database/schema.sql`) — drops and recreates all tables |
|
||||
| 3 | Redeploy SQL functions only (`database/queries/`) |
|
||||
| 4 | Build UI (`ui/` → `public/`) |
|
||||
| 5 | Set up nginx reverse proxy (HTTP or HTTPS via certbot) |
|
||||
| 6 | Install systemd service unit (`dataflow.service`) |
|
||||
| 7 | Start / restart `dataflow.service` |
|
||||
| 8 | Stop `dataflow.service` |
|
||||
| 9 | Set login credentials (`LOGIN_USER` / `LOGIN_PASSWORD_HASH` in `.env`) |
|
||||
|
||||
The status screen at the top of the menu shows the current state of each component (database connection, schema, UI build, service, nginx).
|
||||
|
||||
**Typical first-time setup:** run options 1 → 4 → 9 → 6 → 7 (→ 5 if you want nginx).
|
||||
|
||||
## API Reference
|
||||
|
||||
All `/api` routes require HTTP Basic authentication.
|
||||
|
||||
### Sources — `/api/sources`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/sources` | List all sources |
|
||||
| POST | `/api/sources` | Create a source |
|
||||
| GET | `/api/sources/:name` | Get a source |
|
||||
| PUT | `/api/sources/:name` | Update a source |
|
||||
| DELETE | `/api/sources/:name` | Delete a source |
|
||||
| POST | `/api/sources/suggest` | Suggest source definition from CSV upload |
|
||||
| POST | `/api/sources/:name/import` | Import CSV data and auto-apply transformations to new records |
|
||||
| GET | `/api/sources/:name/import-log` | View import history (includes `inserted_keys` / `excluded_keys` in `info`) |
|
||||
| DELETE | `/api/sources/:name/import-log/:id` | Delete an import batch and all its records |
|
||||
| POST | `/api/sources/:name/transform` | Apply rules and mappings to any untransformed records |
|
||||
| POST | `/api/sources/:name/reprocess` | Re-transform all records |
|
||||
| GET | `/api/sources/:name/fields` | List all known field names |
|
||||
| GET | `/api/sources/:name/stats` | Get record and mapping counts |
|
||||
| POST | `/api/sources/:name/view` | Generate output view |
|
||||
| GET | `/api/sources/:name/view-data` | Query output view (paginated, sortable) |
|
||||
|
||||
### Rules — `/api/rules`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/rules/source/:source_name` | List rules for a source |
|
||||
| POST | `/api/rules` | Create a rule |
|
||||
| GET | `/api/rules/:id` | Get a rule |
|
||||
| PUT | `/api/rules/:id` | Update a rule |
|
||||
| DELETE | `/api/rules/:id` | Delete a rule |
|
||||
| GET | `/api/rules/preview` | Preview a pattern against real records (ad-hoc) |
|
||||
| GET | `/api/rules/:id/test` | Test a saved rule against real records |
|
||||
|
||||
### Mappings — `/api/mappings`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/mappings/source/:source_name` | List mappings |
|
||||
| POST | `/api/mappings` | Create a mapping |
|
||||
| POST | `/api/mappings/bulk` | Bulk create/update mappings |
|
||||
| GET | `/api/mappings/:id` | Get a mapping |
|
||||
| PUT | `/api/mappings/:id` | Update a mapping |
|
||||
| DELETE | `/api/mappings/:id` | Delete a mapping |
|
||||
| GET | `/api/mappings/source/:source_name/unmapped` | Get values with no mapping yet |
|
||||
| GET | `/api/mappings/source/:source_name/all-values` | All extracted values with counts |
|
||||
| GET | `/api/mappings/source/:source_name/counts` | Record counts for existing mappings |
|
||||
| GET | `/api/mappings/source/:source_name/export.tsv` | Export values as TSV |
|
||||
| POST | `/api/mappings/source/:source_name/import-csv` | Import mappings from TSV |
|
||||
|
||||
### Records — `/api/records`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/records/source/:source_name` | List records (paginated) |
|
||||
| GET | `/api/records/:id` | Get a single record |
|
||||
| POST | `/api/records/search` | Search records |
|
||||
| DELETE | `/api/records/:id` | Delete a record |
|
||||
| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
|
||||
|
||||
### Stacks — `/api/stacks`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/stacks` | List all stacks |
|
||||
| POST | `/api/stacks` | Create a stack |
|
||||
| GET | `/api/stacks/:name` | Get a stack |
|
||||
| PUT | `/api/stacks/:name` | Update a stack |
|
||||
| DELETE | `/api/stacks/:name` | Delete a stack |
|
||||
| GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) |
|
||||
| GET | `/api/stacks/:name/layouts` | List saved pivot layouts |
|
||||
| POST | `/api/stacks/:name/layouts` | Save a pivot layout |
|
||||
| DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout |
|
||||
|
||||
## Typical Workflow
|
||||
|
||||
```
|
||||
1. Create a source (POST /api/sources)
|
||||
2. Create transformation rules (POST /api/rules)
|
||||
3. Import CSV data (POST /api/sources/:name/import) — transformations applied automatically to new records
|
||||
4. Preview rules against real data (GET /api/rules/preview)
|
||||
5. Review unmapped values (GET /api/mappings/source/:name/unmapped)
|
||||
6. Add mappings (POST /api/mappings or bulk import via TSV)
|
||||
7. Reprocess to apply new mappings (POST /api/sources/:name/reprocess)
|
||||
8. Query results (GET /api/sources/:name/view-data)
|
||||
```
|
||||
|
||||
See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
|
||||
|
||||
## Project Structure
|
||||
## Project structure
|
||||
|
||||
```
|
||||
dataflow/
|
||||
├── database/
|
||||
│ ├── schema.sql # Table definitions
|
||||
│ └── queries/ # SQL functions, one file per route
|
||||
│ ├── sources.sql
|
||||
│ ├── rules.sql
|
||||
│ ├── mappings.sql
|
||||
│ ├── records.sql
|
||||
│ ├── stacks.sql
|
||||
│ └── status.sql
|
||||
├── api/
|
||||
│ ├── server.js # Express server
|
||||
│ ├── middleware/
|
||||
│ │ └── auth.js # Basic auth middleware
|
||||
│ ├── lib/
|
||||
│ │ └── sql.js # SQL literal helpers
|
||||
│ └── routes/
|
||||
│ ├── sources.js
|
||||
│ ├── rules.js
|
||||
│ ├── mappings.js
|
||||
│ ├── records.js
|
||||
│ ├── stacks.js
|
||||
│ └── status.js
|
||||
├── public/ # Built React UI (served as static files)
|
||||
├── examples/
|
||||
│ ├── GETTING_STARTED.md
|
||||
│ └── bank_transactions.csv
|
||||
└── .env.example
|
||||
├── manage.py # interactive setup / deploy / uninstall
|
||||
├── database/ # schema.sql + one .sql file per API route
|
||||
├── api/ # Express server, routes, auth middleware
|
||||
├── ui/ # React source (built to public/)
|
||||
├── public/ # built UI, served as static files
|
||||
├── docs/
|
||||
└── examples/ # sample CSV for the tutorial
|
||||
```
|
||||
|
||||
Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js`
|
||||
and `database/rules.sql` are the two halves of the same feature.
|
||||
|
||||
`database/*.sql` is the source of truth for every database function — never edit one directly
|
||||
in the database, or the next redeploy will silently revert it.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
@ -4,32 +4,40 @@ This guide walks through a complete example using bank transaction data.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. PostgreSQL database running
|
||||
2. Database created: `CREATE DATABASE dataflow;`
|
||||
3. `.env` file configured (copy from `.env.example`)
|
||||
PostgreSQL running, Node.js 18+, and Python 3.
|
||||
|
||||
## Step 1: Deploy Database Schema
|
||||
## Step 1: Configure and Deploy
|
||||
|
||||
```bash
|
||||
cd /opt/dataflow
|
||||
psql -U postgres -d dataflow -f database/schema.sql
|
||||
psql -U postgres -d dataflow -f database/functions.sql
|
||||
npm install
|
||||
python3 manage.py
|
||||
```
|
||||
|
||||
You should see tables created without errors.
|
||||
Choose option 1. It writes `.env`, creates the database and user if they don't exist,
|
||||
then deploys `database/schema.sql` and the SQL function files in dependency order.
|
||||
|
||||
## Step 2: Start the API Server
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm start
|
||||
```
|
||||
|
||||
The server should start on port 3000 (or your configured port).
|
||||
The server starts on the port set by `API_PORT` in `.env` (3020 by default).
|
||||
|
||||
Every `/api` route requires HTTP Basic auth using the credentials set by `manage.py`
|
||||
option 9. The examples below omit it for readability — add `-u username:password` to each
|
||||
curl, or export it once:
|
||||
|
||||
```bash
|
||||
alias dfcurl='curl -u username:password'
|
||||
```
|
||||
|
||||
`GET /health` is the one route that needs no auth.
|
||||
|
||||
Test it:
|
||||
```bash
|
||||
curl http://localhost:3000/health
|
||||
curl http://localhost:3020/health
|
||||
# Should return: {"status":"ok","timestamp":"..."}
|
||||
```
|
||||
|
||||
@ -38,7 +46,7 @@ curl http://localhost:3000/health
|
||||
A source defines where data comes from and how to deduplicate it.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sources \
|
||||
curl -X POST http://localhost:3020/api/sources \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "bank_transactions",
|
||||
@ -55,7 +63,7 @@ Rules extract meaningful data using regex patterns.
|
||||
### Rule 1: Extract merchant name (first part of description)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/rules \
|
||||
curl -X POST http://localhost:3020/api/rules \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -70,7 +78,7 @@ curl -X POST http://localhost:3000/api/rules \
|
||||
### Rule 2: Extract location (city + state pattern)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/rules \
|
||||
curl -X POST http://localhost:3020/api/rules \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -87,7 +95,7 @@ curl -X POST http://localhost:3000/api/rules \
|
||||
Import the example CSV file:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
|
||||
curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
|
||||
-F "file=@examples/bank_transactions.csv"
|
||||
```
|
||||
|
||||
@ -104,7 +112,7 @@ Response:
|
||||
## Step 6: View Imported Records
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/records/source/bank_transactions?limit=5
|
||||
curl http://localhost:3020/api/records/source/bank_transactions?limit=5
|
||||
```
|
||||
|
||||
You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet!
|
||||
@ -112,7 +120,7 @@ You'll see the raw imported data. Note that `transformed` is `null` - we haven't
|
||||
## Step 7: Apply Transformations
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sources/bank_transactions/transform
|
||||
curl -X POST http://localhost:3020/api/sources/bank_transactions/transform
|
||||
```
|
||||
|
||||
Response:
|
||||
@ -125,7 +133,7 @@ Response:
|
||||
|
||||
Now check the records again:
|
||||
```bash
|
||||
curl http://localhost:3000/api/records/source/bank_transactions?limit=2
|
||||
curl http://localhost:3020/api/records/source/bank_transactions?limit=2
|
||||
```
|
||||
|
||||
You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`.
|
||||
@ -133,7 +141,7 @@ You'll see the `transformed` field now contains the original data plus extracted
|
||||
## Step 8: View Extracted Values That Need Mapping
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/mappings/source/bank_transactions/unmapped
|
||||
curl http://localhost:3020/api/mappings/source/bank_transactions/unmapped
|
||||
```
|
||||
|
||||
Response shows extracted merchant names that aren't mapped yet:
|
||||
@ -151,7 +159,7 @@ Response shows extracted merchant names that aren't mapped yet:
|
||||
Map extracted values to clean, standardized output:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/mappings \
|
||||
curl -X POST http://localhost:3020/api/mappings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -163,7 +171,7 @@ curl -X POST http://localhost:3000/api/mappings \
|
||||
}
|
||||
}'
|
||||
|
||||
curl -X POST http://localhost:3000/api/mappings \
|
||||
curl -X POST http://localhost:3020/api/mappings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -175,7 +183,7 @@ curl -X POST http://localhost:3000/api/mappings \
|
||||
}
|
||||
}'
|
||||
|
||||
curl -X POST http://localhost:3000/api/mappings \
|
||||
curl -X POST http://localhost:3020/api/mappings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -193,13 +201,13 @@ curl -X POST http://localhost:3000/api/mappings \
|
||||
Clear and reapply transformations to pick up the new mappings:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sources/bank_transactions/reprocess
|
||||
curl -X POST http://localhost:3020/api/sources/bank_transactions/reprocess
|
||||
```
|
||||
|
||||
## Step 11: View Final Results
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/records/source/bank_transactions?limit=5
|
||||
curl http://localhost:3020/api/records/source/bank_transactions?limit=5
|
||||
```
|
||||
|
||||
Now the `transformed` field contains:
|
||||
@ -234,7 +242,7 @@ Example result:
|
||||
Try importing the same file again:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
|
||||
curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
|
||||
-F "file=@examples/bank_transactions.csv"
|
||||
```
|
||||
|
||||
@ -272,19 +280,19 @@ You've now:
|
||||
|
||||
```bash
|
||||
# View all sources
|
||||
curl http://localhost:3000/api/sources
|
||||
curl http://localhost:3020/api/sources
|
||||
|
||||
# View source statistics
|
||||
curl http://localhost:3000/api/sources/bank_transactions/stats
|
||||
curl http://localhost:3020/api/sources/bank_transactions/stats
|
||||
|
||||
# View all rules for a source
|
||||
curl http://localhost:3000/api/rules/source/bank_transactions
|
||||
curl http://localhost:3020/api/rules/source/bank_transactions
|
||||
|
||||
# View all mappings for a source
|
||||
curl http://localhost:3000/api/mappings/source/bank_transactions
|
||||
curl http://localhost:3020/api/mappings/source/bank_transactions
|
||||
|
||||
# Search for specific records
|
||||
curl -X POST http://localhost:3000/api/records/search \
|
||||
curl -X POST http://localhost:3020/api/records/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_name": "bank_transactions",
|
||||
@ -301,11 +309,11 @@ curl -X POST http://localhost:3000/api/records/search \
|
||||
- Check logs for error messages
|
||||
|
||||
**Import fails:**
|
||||
- Verify source exists: `curl http://localhost:3000/api/sources`
|
||||
- Verify source exists: `curl http://localhost:3020/api/sources`
|
||||
- Check CSV format matches expectations
|
||||
- Ensure constraint_fields match CSV column names
|
||||
|
||||
**Transformations not working:**
|
||||
- Check rules exist: `curl http://localhost:3000/api/rules/source/bank_transactions`
|
||||
- Check rules exist: `curl http://localhost:3020/api/rules/source/bank_transactions`
|
||||
- Test regex pattern manually
|
||||
- Check records have the specified field
|
||||
@ -1,11 +1,68 @@
|
||||
# Perspective Pivot — Technical Reference
|
||||
# Perspective
|
||||
|
||||
Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply.
|
||||
Everything about the Perspective pivot in dataflow: which packages and versions are
|
||||
pinned and why, and a ground-truth reference for the parts of the API the official docs
|
||||
don't cover.
|
||||
|
||||
This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement.
|
||||
Shared rationale across projects lives in the canonical guide at
|
||||
`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy
|
||||
pattern, upgrade smoke test). This file records what's specific to dataflow.
|
||||
|
||||
---
|
||||
|
||||
> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
|
||||
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
|
||||
> engine, separate npm scope and release schedule — don't mix the two.
|
||||
|
||||
---
|
||||
|
||||
## Current state
|
||||
|
||||
- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
|
||||
This is the target loader; pf_app should adopt it.
|
||||
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
|
||||
Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
|
||||
unless view sizes grow well past 100k.
|
||||
- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
|
||||
for the org; pf_app should copy it.
|
||||
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
|
||||
Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
|
||||
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
|
||||
restore — the reference implementation; keep it.
|
||||
|
||||
## The version pair is correct — do NOT "fix" it to 4.4.1
|
||||
|
||||
`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
|
||||
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
|
||||
combination that keeps both of dataflow's hard requirements:
|
||||
|
||||
- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
|
||||
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
|
||||
paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
|
||||
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
|
||||
|
||||
Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
|
||||
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
|
||||
stays. Don't touch it.
|
||||
|
||||
**What to actually do:**
|
||||
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
|
||||
on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
|
||||
explicit.)
|
||||
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
|
||||
renders, dark/light re-themes, and save→reload→drop-column layout restore.
|
||||
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
|
||||
4.5.x suite becomes possible and the pair can collapse to one version.
|
||||
|
||||
---
|
||||
|
||||
# API Reference
|
||||
|
||||
Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at
|
||||
**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been
|
||||
re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete
|
||||
for some of these APIs — treat this as a ground-truth supplement.
|
||||
|
||||
## Loading via npm
|
||||
|
||||
```js
|
||||
@ -18,7 +18,7 @@ SQL functions are the single source of truth for business logic. The API layer i
|
||||
Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does.
|
||||
|
||||
### One SQL file per route
|
||||
SQL is organized in `database/queries/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`). This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
|
||||
SQL is organized in `database/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `stacks.sql`, `status.sql`) plus `import.sql` and `transform.sql` for the import/transform engine. This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
|
||||
|
||||
### Explicit over implicit
|
||||
Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs.
|
||||
@ -34,11 +34,14 @@ Raw imported records and transformed records are stored as JSONB. This avoids sc
|
||||
manage.py — interactive CLI for setup, deployment, and management
|
||||
database/
|
||||
schema.sql — table definitions (run once or to reset)
|
||||
queries/
|
||||
sources.sql — all SQL for /api/sources
|
||||
rules.sql — all SQL for /api/rules
|
||||
mappings.sql — all SQL for /api/mappings
|
||||
records.sql — all SQL for /api/records
|
||||
stacks.sql — all SQL for /api/stacks
|
||||
status.sql — all SQL for /api/status
|
||||
import.sql — CSV import and the import audit trail
|
||||
transform.sql — the rule/mapping engine
|
||||
api/
|
||||
server.js — Express server, mounts routes, auth middleware
|
||||
middleware/
|
||||
@ -64,8 +67,14 @@ ui/
|
||||
Mappings.jsx — mapping table with TSV import/export
|
||||
Records.jsx — paginated, sortable view of transformed records
|
||||
Pivot.jsx — interactive pivot table with cell inspector
|
||||
Stacks.jsx — multi-source union views with running balance
|
||||
Remap.jsx — bulk remap of an output field value across mappings
|
||||
Log.jsx — global import log across all sources
|
||||
components/ — Sidebar, StatusBar
|
||||
theme.jsx — light/dark context provider
|
||||
public/ — compiled UI (output of npm run build in ui/)
|
||||
docs/ — this file, tutorial, UI and Perspective references
|
||||
examples/ — bank_transactions.csv, the tutorial's sample data
|
||||
```
|
||||
|
||||
---
|
||||
@ -123,7 +132,7 @@ The transform is fully set-based — no row-by-row loops. All records for a sour
|
||||
|
||||
## SQL Functions
|
||||
|
||||
Each file in `database/queries/` maps 1-to-1 with a route file.
|
||||
Each route file has a matching SQL file in `database/`; `import.sql` and `transform.sql` hold the engine shared by several routes.
|
||||
|
||||
**sources.sql**
|
||||
`list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view`
|
||||
@ -153,53 +162,104 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
|
||||
|
||||
**Route summary:**
|
||||
|
||||
### Sources — `api/routes/sources.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/sources | List all sources |
|
||||
| POST | /api/sources | Create source |
|
||||
| GET | /api/sources/:name | Get source |
|
||||
| PUT | /api/sources/:name | Update source (constraint_fields, config) |
|
||||
| DELETE | /api/sources/:name | Delete source and all data |
|
||||
| POST | /api/sources/suggest | Suggest source config from CSV upload |
|
||||
| POST | /api/sources/:name/import | Import CSV records |
|
||||
| GET | /api/sources/:name/import-log | Import history |
|
||||
| GET | /api/sources/:name/stats | Record counts |
|
||||
| GET | /api/sources/:name/fields | All known field names and origins |
|
||||
| GET | /api/sources/:name/view-data | Paginated, sortable view data |
|
||||
| POST | /api/sources/:name/transform | Apply transformations (new records only) |
|
||||
| PUT | /api/sources/:name | Update source (constraint_fields, config, global_picklist) |
|
||||
| DELETE | /api/sources/:name | Delete source and all its data |
|
||||
| POST | /api/sources/suggest | Suggest source config from an uploaded CSV |
|
||||
| POST | /api/sources/:name/import | Import CSV; transformations are applied to the new records |
|
||||
| GET | /api/sources/import-log | Import history across all sources |
|
||||
| GET | /api/sources/:name/import-log | Import history for one source |
|
||||
| DELETE | /api/sources/:name/import-log/:id | Delete an import batch and every record in it |
|
||||
| POST | /api/sources/:name/transform | Apply transformations to untransformed records only |
|
||||
| POST | /api/sources/:name/reprocess | Reapply transformations to all records |
|
||||
| POST | /api/sources/:name/view | Generate dfv view |
|
||||
| GET | /api/sources/:name/stats | Record counts |
|
||||
| GET | /api/sources/:name/fields | All known field names and their origins |
|
||||
| GET | /api/sources/:name/override-keys | Distinct field names used in overrides for this source |
|
||||
| POST | /api/sources/:name/view | Generate/refresh the `dfv` view |
|
||||
| GET | /api/sources/:name/view-data | Paginated, sortable, filterable view data |
|
||||
| GET | /api/sources/:name/layouts | List saved pivot layouts |
|
||||
| POST | /api/sources/:name/layouts | Save a pivot layout |
|
||||
| DELETE | /api/sources/:name/layouts/:id | Delete a pivot layout |
|
||||
|
||||
### Rules — `api/routes/rules.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/rules/source/:name | List rules for a source |
|
||||
| GET | /api/rules/preview | Preview pattern against live records |
|
||||
| GET | /api/rules/:id/test | Test saved rule against live records |
|
||||
| GET | /api/rules/:id | Get a rule |
|
||||
| POST | /api/rules | Create rule |
|
||||
| PUT | /api/rules/:id | Update rule |
|
||||
| DELETE | /api/rules/:id | Delete rule |
|
||||
| GET | /api/rules/preview | Preview an ad-hoc pattern against live records |
|
||||
| GET | /api/rules/:id/test | Test a saved rule against live records |
|
||||
|
||||
### Mappings — `api/routes/mappings.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/mappings/source/:name | List mappings |
|
||||
| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) |
|
||||
| GET | /api/mappings/source/:name/unmapped | Only unmapped extracted values |
|
||||
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
|
||||
| GET | /api/mappings/source/:name/export.tsv | Export mappings as TSV |
|
||||
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from TSV |
|
||||
| GET | /api/mappings/:id | Get a mapping |
|
||||
| POST | /api/mappings | Create mapping |
|
||||
| POST | /api/mappings/bulk | Upsert multiple mappings |
|
||||
| PUT | /api/mappings/:id | Update mapping |
|
||||
| DELETE | /api/mappings/:id | Delete mapping |
|
||||
| GET | /api/records/source/:name | List raw records |
|
||||
| GET | /api/records/:id | Get single record |
|
||||
| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) with counts |
|
||||
| GET | /api/mappings/source/:name/unmapped | Only values with no mapping yet |
|
||||
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
|
||||
| GET | /api/mappings/source/:name/export.tsv | Export extracted values as TSV |
|
||||
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from an uploaded TSV |
|
||||
| GET | /api/mappings/global-values | Output values across all `global_picklist` sources (autocomplete) |
|
||||
| GET | /api/mappings/outputs | Search output field values across all mappings |
|
||||
| GET | /api/mappings/outputs/:col/:val | Mappings carrying a specific output field value |
|
||||
| POST | /api/mappings/remap-field | Replace an output field value across all mappings |
|
||||
|
||||
### Records — `api/routes/records.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/records/source/:name | List records (paginated) |
|
||||
| GET | /api/records/:id | Get a single record |
|
||||
| POST | /api/records/search | Search by JSONB containment |
|
||||
| DELETE | /api/records/:id | Delete record |
|
||||
| DELETE | /api/records/source/:name/all | Delete all records for a source |
|
||||
| PUT | /api/records/:id/overrides | Set manual overrides on a record |
|
||||
| DELETE | /api/records/:id/overrides | Clear a record's overrides |
|
||||
| POST | /api/records/bulk-overrides | Apply the same overrides to many records |
|
||||
|
||||
### Stacks — `api/routes/stacks.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/stacks | List all stacks |
|
||||
| GET | /api/stacks/:name | Get a stack with its sources |
|
||||
| POST | /api/stacks | Create stack |
|
||||
| GET | /api/stacks/:name | Get stack |
|
||||
| PUT | /api/stacks/:name | Update stack |
|
||||
| DELETE | /api/stacks/:name | Delete stack |
|
||||
| PUT | /api/stacks/:name/sources/:source | Add or update a source in the stack |
|
||||
| DELETE | /api/stacks/:name/sources/:source | Remove a source from the stack |
|
||||
| PUT | /api/stacks/:name/sources/reorder | Reorder the stack's sources |
|
||||
| GET | /api/stacks/:name/view-sql | Preview the SQL that would build the view (dry run) |
|
||||
| POST | /api/stacks/:name/view | Generate/refresh the `dfv` view |
|
||||
| POST | /api/stacks/:name/exec-sql | Execute user-edited SQL for the view |
|
||||
| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
|
||||
| GET | /api/stacks/:name/balance | Current running balance from the generated view |
|
||||
| POST | /api/stacks/:name/calibrate | Set the balance offset from a known balance at a date |
|
||||
| GET | /api/stacks/:name/layouts | List saved pivot layouts |
|
||||
| POST | /api/stacks/:name/layouts | Save pivot layout |
|
||||
| DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout |
|
||||
| POST | /api/stacks/:name/layouts | Save a pivot layout |
|
||||
| DELETE | /api/stacks/:name/layouts/:id | Delete a pivot layout |
|
||||
|
||||
### Status — `api/routes/status.js`
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | /api/status | Deployment status |
|
||||
| GET | /health | Health check (no auth) |
|
||||
|
||||
---
|
||||
|
||||
@ -253,7 +313,7 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
|
||||
- `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save.
|
||||
- Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently.
|
||||
|
||||
See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically.
|
||||
See `docs/perspective.md` for the full technical reference on controlling Perspective programmatically.
|
||||
|
||||
- **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
|
||||
|
||||
@ -280,7 +340,7 @@ Shows current status on every screen:
|
||||
|
||||
2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation.
|
||||
|
||||
3. **Redeploy SQL functions** — Runs all four files in `database/queries/` in order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`. Safe to run at any time without data loss.
|
||||
3. **Redeploy SQL functions** — Runs the function files in `database/` in dependency order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `import.sql`, `transform.sql`, `stacks.sql`, `status.sql`. Safe to run at any time without data loss.
|
||||
|
||||
4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`.
|
||||
|
||||
@ -294,6 +354,8 @@ Shows current status on every screen:
|
||||
|
||||
9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed.
|
||||
|
||||
10. **Uninstall** — Reverses everything the other options install, in reverse order: stops/disables/removes the systemd unit, removes the nginx site and reloads nginx, drops the database and its user (prompts for admin credentials), then deletes `.env`, `public/`, and `node_modules`. Lists exactly what it found before doing anything and requires typing `delete` to proceed. The repository itself is left in place.
|
||||
|
||||
**Key behaviors:**
|
||||
- All commands that will be run are printed before the user is asked to confirm.
|
||||
- Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal.
|
||||
@ -341,13 +403,18 @@ The server binds to `0.0.0.0` on `API_PORT` and serves both the API and the comp
|
||||
|
||||
## Deploying SQL Changes
|
||||
|
||||
Any time SQL functions are modified:
|
||||
Any time SQL functions are modified, run `python3 manage.py` and choose "Redeploy SQL
|
||||
functions only". It runs every function file in dependency order — the list lives in
|
||||
`QUERY_FILES` in `manage.py`, which is the one place the order is defined.
|
||||
|
||||
To deploy a single file by hand:
|
||||
```bash
|
||||
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/sources.sql
|
||||
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/rules.sql
|
||||
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/mappings.sql
|
||||
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/records.sql
|
||||
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -v ON_ERROR_STOP=1 -f database/rules.sql
|
||||
```
|
||||
Then restart the server. Function deployment is safe to repeat — all functions use `CREATE OR REPLACE`.
|
||||
|
||||
Deployment is safe to repeat — every function uses `CREATE OR REPLACE`.
|
||||
|
||||
**The files are the source of truth.** Editing a function directly in the database, without
|
||||
writing the change back to its file, means the next redeploy silently reverts it.
|
||||
|
||||
Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead.
|
||||
Loading…
Reference in New Issue
Block a user