pt/dataflow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all docs to reflect current state

Browse Source 
- perspective-pivot.md: npm install pattern, v4.5.1/v4.4.1 versions
- README.md: Node 18+, port 3020, add stacks routes, fix project structure
- SPEC.md: add stacks/status routes, pages, SQL functions; update Perspective version
- ui/README.md: replace Vite boilerplate with project-specific content
- Remove docs/refactor-transformed-split.md (completed work)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-06-12 23:51:00 -04:00
parent 0ece53e7be
commit efa65d8409
5 changed files with 79 additions and 95 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
README.md
View File

@ -46,7 +46,7 @@ Map extracted values to clean, standardized output.
### Prerequisites ### Prerequisites
- PostgreSQL 12+ - PostgreSQL 12+
- Node.js 16+ - Node.js 18+
- Python 3 (for `manage.py`) - Python 3 (for `manage.py`)
### Installation ### Installation
@ -66,7 +66,7 @@ For development with auto-reload:
npm run dev npm run dev
``` ```
The UI is available at `http://localhost:3000`. The API is at `http://localhost:3000/api`. The UI is available at `http://localhost:3020`. The API is at `http://localhost:3020/api` (port set by `API_PORT` in `.env`).
## Management Script (`manage.py`) ## Management Script (`manage.py`)
@ -154,6 +154,20 @@ All `/api` routes require HTTP Basic authentication.
| DELETE | `/api/records/:id` | Delete a record | | DELETE | `/api/records/:id` | Delete a record |
| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source | | DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
### Stacks — `/api/stacks`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/stacks` | List all stacks |
| POST | `/api/stacks` | Create a stack |
| GET | `/api/stacks/:name` | Get a stack |
| PUT | `/api/stacks/:name` | Update a stack |
| DELETE | `/api/stacks/:name` | Delete a stack |
| GET | `/api/stacks/:name/view-data` | Query stacked data (paginated) |
| GET | `/api/stacks/:name/layouts` | List saved pivot layouts |
| POST | `/api/stacks/:name/layouts` | Save a pivot layout |
| DELETE | `/api/stacks/:name/layouts/:id` | Delete a pivot layout |
## Typical Workflow ## Typical Workflow
``` ```
@ -174,20 +188,28 @@ See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
``` ```
dataflow/ dataflow/
├── database/ ├── database/
│ ├── schema.sql # Table definitions │ ├── schema.sql # Table definitions
│ └── functions.sql # Import/transform/query functions │ └── queries/ # SQL functions, one file per route
│ ├── sources.sql
│ ├── rules.sql
│ ├── mappings.sql
│ ├── records.sql
│ ├── stacks.sql
│ └── status.sql
├── api/ ├── api/
│ ├── server.js # Express server │ ├── server.js # Express server
│ ├── middleware/ │ ├── middleware/
│ │ └── auth.js # Basic auth middleware │ │ └── auth.js # Basic auth middleware
│ ├── lib/ │ ├── lib/
│ │ └── sql.js # SQL literal helpers │ │ └── sql.js # SQL literal helpers
│ └── routes/ │ └── routes/
│ ├── sources.js │ ├── sources.js
│ ├── rules.js │ ├── rules.js
│ ├── mappings.js │ ├── mappings.js
│ └── records.js │ ├── records.js
├── public/ # Built React UI (served as static files) │ ├── stacks.js
│ └── status.js
├── public/ # Built React UI (served as static files)
├── examples/ ├── examples/
│ ├── GETTING_STARTED.md │ ├── GETTING_STARTED.md
│ └── bank_transactions.csv │ └── bank_transactions.csv

24
SPEC.md
View File

@ -50,6 +50,8 @@ api/
rules.js — HTTP handlers for rule management rules.js — HTTP handlers for rule management
mappings.js — HTTP handlers for mapping management mappings.js — HTTP handlers for mapping management
records.js — HTTP handlers for record queries records.js — HTTP handlers for record queries
stacks.js — HTTP handlers for stack management
status.js — HTTP handler for deployment status
ui/ ui/
src/ src/
api.js — fetch wrapper, credential management api.js — fetch wrapper, credential management
@ -135,6 +137,12 @@ Each file in `database/queries/` maps 1-to-1 with a route file.
**records.sql** **records.sql**
`list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records` `list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records`
**stacks.sql**
`list_stacks`, `get_stack`, `create_stack`, `update_stack`, `delete_stack`, `get_stack_view_data` (union of source views with field mapping and running balance), `list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`
**status.sql**
`get_status` — returns deployment state (schema version, function presence, service status)
--- ---
## API ## API
@ -182,6 +190,16 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
| POST | /api/records/search | Search by JSONB containment | | POST | /api/records/search | Search by JSONB containment |
| DELETE | /api/records/:id | Delete record | | DELETE | /api/records/:id | Delete record |
| DELETE | /api/records/source/:name/all | Delete all records for a source | | DELETE | /api/records/source/:name/all | Delete all records for a source |
| GET | /api/stacks | List all stacks |
| POST | /api/stacks | Create stack |
| GET | /api/stacks/:name | Get stack |
| PUT | /api/stacks/:name | Update stack |
| DELETE | /api/stacks/:name | Delete stack |
| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
| GET | /api/stacks/:name/layouts | List saved pivot layouts |
| POST | /api/stacks/:name/layouts | Save pivot layout |
| DELETE | /api/stacks/:name/layouts/:id | Delete pivot layout |
| GET | /api/status | Deployment status |
--- ---
@ -218,11 +236,11 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
- **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page. - **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page.
- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` v4.4.0, loaded from CDN at runtime). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively. - **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` client/viewer/datagrid v4.5.1, viewer-d3fc v4.4.1 — installed via npm). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
**Toolbar (above the viewer):** **Toolbar (above the viewer):**
- Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it. - Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it.
- **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found in v4.4.0 — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect). - **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
- The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads. - The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads.
**Cell inspector (right panel):** **Cell inspector (right panel):**
@ -237,6 +255,8 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically. See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically.
- **Stacks** — Named unions of multiple sources. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
- **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column. - **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column.
--- ---

21
docs/perspective-pivot.md
View File

@ -1,27 +1,22 @@
# Perspective Pivot — Technical Reference # Perspective Pivot — Technical Reference
Version tested: `@perspective-dev` v4.4.0 (client, viewer, viewer-datagrid, viewer-d3fc), loaded from CDN. Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at **v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been re-verified at 4.5.1 but are believed to still apply.
This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement. This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement.
--- ---
## Loading from CDN ## Loading via npm
```js ```js
const [{ default: perspective }] = await Promise.all([ import perspective from '@perspective-dev/client/inline'
import('https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'), import '@perspective-dev/viewer/inline'
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'), import '@perspective-dev/viewer-datagrid'
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'), import '@perspective-dev/viewer-d3fc'
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'), import '@perspective-dev/viewer/themes'
])
``` ```
Stylesheet: The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`.
```html
<link rel="stylesheet" crossorigin="anonymous"
href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css" />
```
--- ---

62
docs/refactor-transformed-split.md
View File

@ -1,62 +0,0 @@
# Refactor: Split `transformed` into three columns
## Goal
Separate `records` into three clean JSONB layers with clear semantics:
| Column | Meaning | Wins over |
|---|---|---|
| `data` | Raw import values, never mutated | — |
| `transformed` | Rule/mapping-derived fields only | `data` |
| `overrides` | Manual user overrides | `data`, `transformed` |
Consumers merge them at read time:
```sql
data || COALESCE(transformed, '{}'::jsonb) || COALESCE(overrides, '{}'::jsonb)
```
## Why
Currently `transformed` duplicates `data` keys because `apply_transformations` was originally
written as `data || rule_additions`. This makes it impossible to tell what the rules actually
changed vs. what was carried from the original import.
## Current State (branch: `transformed-refactor`)
### Already done in functions.sql
- `apply_transformations` — already stores only rule additions (`COALESCE(ra.additions, '{}')`)
- `generate_source_view` — already uses the 3-way coalesce for `dfv.*` views
- `set_record_overrides`, `clear_record_overrides`, `bulk_set_record_overrides` — exist
- API routes — `PUT /api/records/:id/overrides`, `DELETE /:id/overrides`, `POST /bulk-overrides` exist
### Still needed
1. **`database/schema.sql`** — add `overrides JSONB` column to `records` table and a GIN index.
Also fix the syntax error: trailing comma before `)` on line 48.
2. **`ui/src/pages/Records.jsx`** — right panel currently iterates `selectedRecord.transformed`
for all fields. Split into three sections:
- **Original** (`data`) — read-only, muted style
- **Transformed** (`transformed`) — rule-derived delta only, highlighted
- **Overrides** (`overrides`) — editable, amber style (existing draft UI already works here)
3. **Deploy + reprocess** (user-triggered, not automated):
- `psql -d dataflow -f database/schema.sql` (drop/recreate schema)
- `psql -d dataflow -f database/functions.sql` (redeploy functions)
- Regenerate all `dfv.*` views via the API for each source
- Run `reprocess_records` on every source to strip stale `data` keys from existing `transformed` rows
## Rollback
Branch `stacks` is the stable point. A pg_dump taken before deployment is the DB rollback.
## File Checklist
- [ ] `database/schema.sql` — add `overrides` column + index, fix syntax error
- [ ] `database/functions.sql` — no changes needed (already correct)
- [ ] `ui/src/pages/Records.jsx` — split inspector panel into 3 sections
- [ ] Build UI: `cd ui && npm run build`
- [ ] Deploy DB (user-triggered)
- [ ] Reprocess all sources (user-triggered)

27
ui/README.md
View File

@ -1,16 +1,25 @@
# React + Vite # Dataflow UI
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules. React + Vite + Tailwind CSS frontend for Dataflow.
Currently, two official plugins are available: ## Development
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs) ```bash
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/) npm install
npm run dev # dev server on :5173, proxies /api to :3020
```
## React Compiler ## Build
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation). ```bash
npm run build # outputs to ../public/
```
## Expanding the ESLint configuration The Express server serves `../public/` as static files — no separate web server needed in production.
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project. ## Key packages
- `react` / `react-router-dom` — SPA routing
- `@perspective-dev/client`, `viewer`, `viewer-datagrid`, `viewer-d3fc` — pivot table (npm, inline WASM builds)
- `tailwindcss` — utility CSS
- `sql-formatter` — SQL display formatting