From efa65d84091de069baf6995517e2f84cd0cc70bf Mon Sep 17 00:00:00 2001 From: Paul Trowbridge Date: Fri, 12 Jun 2026 23:51:00 -0400 Subject: [PATCH] Update all docs to reflect current state - perspective-pivot.md: npm install pattern, v4.5.1/v4.4.1 versions - README.md: Node 18+, port 3020, add stacks routes, fix project structure - SPEC.md: add stacks/status routes, pages, SQL functions; update Perspective version - ui/README.md: replace Vite boilerplate with project-specific content - Remove docs/refactor-transformed-split.md (completed work) Co-Authored-By: Claude Sonnet 4.6 --- README.md | 40 ++++++++++++++----- SPEC.md | 24 +++++++++++- docs/perspective-pivot.md | 21 ++++------ docs/refactor-transformed-split.md | 62 ------------------------------ ui/README.md | 27 ++++++++----- 5 files changed, 79 insertions(+), 95 deletions(-) delete mode 100644 docs/refactor-transformed-split.md diff --git a/README.md b/README.md index 9264684..68cd68d 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Map extracted values to clean, standardized output. ### Prerequisites - PostgreSQL 12+ -- Node.js 16+ +- Node.js 18+ - Python 3 (for `manage.py`) ### Installation @@ -66,7 +66,7 @@ For development with auto-reload: npm run dev ``` -The UI is available at `http://localhost:3000`. The API is at `http://localhost:3000/api`. +The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`). ## Management Script (`manage.py`) @@ -154,6 +154,20 @@ All `/api` routes require HTTP Basic authentication. | DELETE | `/api/records/:id` | Delete a record | | DELETE | `/api/records/source/:source_name/all` | Delete all records for a source | +### Stacks — `/api/stacks` + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/stacks` | List all stacks | +| POST | `/api/stacks` | Create a stack | +| GET | `/api/stacks/:name` | Get a stack | +| PUT | `/api/stacks/:name` | Update a stack | +| DELETE | `/api/stacks/:name` | Delete a stack | +| GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) | +| GET | `/api/stacks/:name/layouts` | List saved pivot layouts | +| POST | `/api/stacks/:name/layouts` | Save a pivot layout | +| DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout | + ## Typical Workflow ``` @@ -174,20 +188,28 @@ See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples. ``` dataflow/ ├── database/ -│ ├── schema.sql # Table definitions -│ └── functions.sql # Import/transform/query functions +│ ├── schema.sql # Table definitions +│ └── queries/ # SQL functions, one file per route +│ ├── sources.sql +│ ├── rules.sql +│ ├── mappings.sql +│ ├── records.sql +│ ├── stacks.sql +│ └── status.sql ├── api/ -│ ├── server.js # Express server +│ ├── server.js # Express server │ ├── middleware/ -│ │ └── auth.js # Basic auth middleware +│ │ └── auth.js # Basic auth middleware │ ├── lib/ -│ │ └── sql.js # SQL literal helpers +│ │ └── sql.js # SQL literal helpers │ └── routes/ │ ├── sources.js │ ├── rules.js │ ├── mappings.js -│ └── records.js -├── public/ # Built React UI (served as static files) +│ ├── records.js +│ ├── stacks.js +│ └── status.js +├── public/ # Built React UI (served as static files) ├── examples/ │ ├── GETTING_STARTED.md │ └── bank_transactions.csv diff --git a/SPEC.md b/SPEC.md index 67d2920..fb8d33d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -50,6 +50,8 @@ api/ rules.js — HTTP handlers for rule management mappings.js — HTTP handlers for mapping management records.js — HTTP handlers for record queries + stacks.js — HTTP handlers for stack management + status.js — HTTP handler for deployment status ui/ src/ api.js — fetch wrapper, credential management @@ -135,6 +137,12 @@ Each file in `database/queries/` maps 1-to-1 with a route file. **records.sql** `list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records` +**stacks.sql** +`list_stacks`, `get_stack`, `create_stack`, `update_stack`, `delete_stack`, `get_stack_view_data` (union of source views with field mapping and running balance), `list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout` + +**status.sql** +`get_status` — returns deployment state (schema version, function presence, service status) + --- ## API @@ -182,6 +190,16 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea | POST | /api/records/search | Search by JSONB containment | | DELETE | /api/records/:id | Delete record | | DELETE | /api/records/source/:name/all | Delete all records for a source | +| GET | /api/stacks | List all stacks | +| POST | /api/stacks | Create stack | +| GET | /api/stacks/:name | Get stack | +| PUT | /api/stacks/:name | Update stack | +| DELETE | /api/stacks/:name | Delete stack | +| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance | +| GET | /api/stacks/:name/layouts | List saved pivot layouts | +| POST | /api/stacks/:name/layouts | Save pivot layout | +| DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout | +| GET | /api/status | Deployment status | --- @@ -218,11 +236,11 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s - **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page. -- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` v4.4.0, loaded from CDN at runtime). Loads all rows from the source view into an in-browser Perspective worker and renders a `` web component. Supports grouping, splitting, filtering, sorting, and charting interactively. +- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` client/viewer/datagrid v4.5.1, viewer-d3fc v4.4.1 — installed via npm). Loads all rows from the source view into an in-browser Perspective worker and renders a `` web component. Supports grouping, splitting, filtering, sorting, and charting interactively. **Toolbar (above the viewer):** - Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it. - - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found in v4.4.0 — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect). + - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect). - The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads. **Cell inspector (right panel):** @@ -237,6 +255,8 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically. +- **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table. + - **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column. --- diff --git a/docs/perspective-pivot.md b/docs/perspective-pivot.md index 013408b..514b91b 100644 --- a/docs/perspective-pivot.md +++ b/docs/perspective-pivot.md @@ -1,27 +1,22 @@ # Perspective Pivot — Technical Reference -Version tested: `@perspective-dev` v4.4.0 (client, viewer, viewer-datagrid, viewer-d3fc), loaded from CDN. +Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply. This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement. --- -## Loading from CDN +## Loading via npm ```js -const [{ default: perspective }] = await Promise.all([ - import('https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'), - import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'), - import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'), - import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'), -]) +import perspective from '@perspective-dev/client/inline' +import '@perspective-dev/viewer/inline' +import '@perspective-dev/viewer-datagrid' +import '@perspective-dev/viewer-d3fc' +import '@perspective-dev/viewer/themes' ``` -Stylesheet: -```html - -``` +The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`. --- diff --git a/docs/refactor-transformed-split.md b/docs/refactor-transformed-split.md deleted file mode 100644 index 561035e..0000000 --- a/docs/refactor-transformed-split.md +++ /dev/null @@ -1,62 +0,0 @@ -# Refactor: Split `transformed` into three columns - -## Goal - -Separate `records` into three clean JSONB layers with clear semantics: - -| Column | Meaning | Wins over | -|---|---|---| -| `data` | Raw import values, never mutated | — | -| `transformed` | Rule/mapping-derived fields only | `data` | -| `overrides` | Manual user overrides | `data`, `transformed` | - -Consumers merge them at read time: - -```sql -data || COALESCE(transformed, '{}'::jsonb) || COALESCE(overrides, '{}'::jsonb) -``` - -## Why - -Currently `transformed` duplicates `data` keys because `apply_transformations` was originally -written as `data || rule_additions`. This makes it impossible to tell what the rules actually -changed vs. what was carried from the original import. - -## Current State (branch: `transformed-refactor`) - -### Already done in functions.sql - -- `apply_transformations` — already stores only rule additions (`COALESCE(ra.additions, '{}')`) -- `generate_source_view` — already uses the 3-way coalesce for `dfv.*` views -- `set_record_overrides`, `clear_record_overrides`, `bulk_set_record_overrides` — exist -- API routes — `PUT /api/records/:id/overrides`, `DELETE /:id/overrides`, `POST /bulk-overrides` exist - -### Still needed - -1. **`database/schema.sql`** — add `overrides JSONB` column to `records` table and a GIN index. - Also fix the syntax error: trailing comma before `)` on line 48. - -2. **`ui/src/pages/Records.jsx`** — right panel currently iterates `selectedRecord.transformed` - for all fields. Split into three sections: - - **Original** (`data`) — read-only, muted style - - **Transformed** (`transformed`) — rule-derived delta only, highlighted - - **Overrides** (`overrides`) — editable, amber style (existing draft UI already works here) - -3. **Deploy + reprocess** (user-triggered, not automated): - - `psql -d dataflow -f database/schema.sql` (drop/recreate schema) - - `psql -d dataflow -f database/functions.sql` (redeploy functions) - - Regenerate all `dfv.*` views via the API for each source - - Run `reprocess_records` on every source to strip stale `data` keys from existing `transformed` rows - -## Rollback - -Branch `stacks` is the stable point. A pg_dump taken before deployment is the DB rollback. - -## File Checklist - -- [ ] `database/schema.sql` — add `overrides` column + index, fix syntax error -- [ ] `database/functions.sql` — no changes needed (already correct) -- [ ] `ui/src/pages/Records.jsx` — split inspector panel into 3 sections -- [ ] Build UI: `cd ui && npm run build` -- [ ] Deploy DB (user-triggered) -- [ ] Reprocess all sources (user-triggered) diff --git a/ui/README.md b/ui/README.md index a36934d..37447c2 100644 --- a/ui/README.md +++ b/ui/README.md @@ -1,16 +1,25 @@ -# React + Vite +# Dataflow UI -This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules. +React + Vite + Tailwind CSS frontend for Dataflow. -Currently, two official plugins are available: +## Development -- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs) -- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/) +```bash +npm install +npm run dev # dev server on :5173, proxies /api to :3020 +``` -## React Compiler +## Build -The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation). +```bash +npm run build # outputs to ../public/ +``` -## Expanding the ESLint configuration +The Express server serves `../public/` as static files — no separate web server needed in production. -If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project. +## Key packages + +- `react` / `react-router-dom` — SPA routing +- `@perspective-dev/client`, `viewer`, `viewer-datagrid`, `viewer-d3fc` — pivot table (npm, inline WASM builds) +- `tailwindcss` — utility CSS +- `sql-formatter` — SQL display formatting