pt/dataflow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: pt:master
Branches Tags
pt:master
pt:ui-mobile
pt:ui-tokens
pt:ui-navigation
pt:simplefin
pt:teller-sync
pt:stacks-bulk-wip
pt:rule-chaining
..
compare: pt:stacks-bulk-wip
Branches Tags
pt:master
pt:ui-mobile
pt:ui-tokens
pt:ui-navigation
pt:simplefin
pt:teller-sync
pt:stacks-bulk-wip
pt:rule-chaining

1 Commits
master ... stacks-bul

Author SHA1 Message Date
Paul Trowbridge
0a9685777e WIP: bulk override feature (raw SQL, needs DB function) 
Saves in-progress bulk-override implementation for review.
Not merged to stacks — backend uses raw SQL instead of a DB function.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-26 11:00:05 -04:00
68 changed files with 3161 additions and 10200 deletions
Show Stats Expand all files Collapse all files

7
.env.example
View File

@ -8,10 +8,3 @@ DB_PASSWORD=your_password_here
# API Configuration
API_PORT=3000
NODE_ENV=development
# SimpleFIN (optional — only needed for API-based bank feeds)
# The access URL is the credential: claim a setup token once via
# POST /api/sources/simplefin-claim and paste the result here.
# A source picks its bridge with config.simplefin.access_url_env;
# SIMPLEFIN_ACCESS_URL is the default.
SIMPLEFIN_ACCESS_URL=https://user:pass@bridge.simplefin.org/simplefin

11
.gitignore vendored
View File

@ -2,10 +2,10 @@
.env
# Dependencies
# Lockfiles ARE tracked — they pin the Perspective 4.5.1/4.4.1 pairing that the
# caret ranges in ui/package.json would otherwise let drift. See docs/perspective.md.
node_modules/
ui/node_modules/
package-lock.json
ui/package-lock.json
# UI build output (generated — run `cd ui && npm run build`)
public/
@ -28,5 +28,8 @@ Thumbs.db
*.swp
*.swo
# Scratch data exports
/*.tsv
# Uploads
uploads/*
!uploads/.gitkeep
*.tsv

263
CLAUDE.md
View File

@ -2,122 +2,213 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Dataflow imports CSV data, extracts structure from it with regex rules, maps the extracted
values to standardized output, and serves the result over a REST API and React UI. It is a
**simple system by design** — don't over-engineer it.
## Overview
**Read [docs/spec.md](docs/spec.md) for architecture, schema, data flow, the full API, and
`manage.py`.** This file covers only what you need to work in the repo without breaking
something — the rules and non-obvious behaviours that aren't visible from the code.
Dataflow is a simple data transformation tool for importing, cleaning, and standardizing data from various sources. Built with PostgreSQL and Node.js/Express, it emphasizes clarity and simplicity over complexity.
## Where things live
## Core Concepts
Both the API routes and the SQL are one file per resource: `api/routes/rules.js` and
`database/rules.sql` are two halves of the same feature. Two SQL files are shared engines
rather than per-route — `import.sql` (CSV import, audit trail) and `transform.sql` (the
rule/mapping engine, including the `jsonb_concat_obj` aggregate).
1. **Sources** - Define data sources and deduplication rules (which fields make a record unique)
2. **Import** - Load CSV data, automatically deduplicating based on source rules
3. **Rules** - Extract information using regex patterns (e.g., extract merchant from transaction description)
4. **Mappings** - Map extracted values to standardized output (e.g., "WALMART" → {"vendor": "Walmart", "category": "Groceries"})
5. **Transform** - Apply rules and mappings to create clean, enriched data
`manage.py`'s `QUERY_FILES` list is the deploy order and the authoritative file list.
## Architecture
## Rules that matter
### Database Schema (`database/schema.sql`)
**`database/*.sql` is the source of truth for every database function. Never edit a function
directly in the database.** A live edit that isn't written back to the file is silently
reverted the next time anyone runs "Redeploy SQL functions". This has already happened once:
five functions drifted and sat wrong in the repo for months — see the git history of the
deleted `database/functions.sql`.
**5 simple tables:**
- `sources` - Source definitions with `constraint_fields` array
- `records` - Imported data with `data` (raw) and `transformed` (enriched) JSONB columns
- `rules` - Regex extraction rules with `field`, `pattern`, `output_field`
- `mappings` - Input/output value mappings
- `import_log` - Audit trail
**Always run `npm run build` from `ui/` after any change to `ui/src/`.** The Express server
serves the built output in `public/`; source changes are invisible until you rebuild.
**Key design:**
- JSONB for flexible data storage
- Deduplication via MD5 hash of specified fields
- Simple, flat structure (no complex relationships)
**Never use `ON CONFLICT (constraint_key)`.** See deduplication below — there is no unique
constraint, and adding one would drop legitimate transactions.
### Database Functions (`database/functions.sql`)
## The three data layers
**4 focused functions:**
- `import_records(source_name, data)` - Import with deduplication
- `apply_transformations(source_name, record_ids)` - Apply rules and mappings
- `get_unmapped_values(source_name, rule_name)` - Find values needing mappings
- `reprocess_records(source_name)` - Re-transform all records
Each row in `records` keeps its data in three JSONB columns:
**Design principle:** Each function does ONE thing. No nested CTEs, no duplication.
- `data` — raw imported values, never modified
- `transformed` — rule and mapping output only (the delta)
- `overrides` — manual edits, highest precedence
### API Server (`api/server.js` + `api/routes/`)
Readers merge them as `data || transformed || overrides`. Keeping them separate is what lets
`reprocess_records` re-run the rules without clobbering a manual edit. Anything that writes
overrides into `transformed` is a bug — that was the pre-May-2026 behaviour.
**RESTful endpoints:**
- `/api/sources` - CRUD sources, import CSV, trigger transformations
- `/api/rules` - CRUD transformation rules
- `/api/mappings` - CRUD value mappings, view unmapped values
- `/api/records` - Query and search transformed data
## Deduplication
**Route files:**
- `routes/sources.js` - Source management and CSV import
- `routes/rules.js` - Rule management
- `routes/mappings.js` - Mapping management + unmapped values
- `routes/records.js` - Record queries and search
- `constraint_key` is a JSONB object of the constraint field values — readable, no hashing
- Dedup is enforced at import time in a CTE. There is **no unique DB constraint** on it
- **The constraint key is cross-batch re-import protection, not record uniqueness**
- Within one import batch, all rows insert even when constraint keys collide. Banks
legitimately send identical-looking transactions — 11 separate Cedar Point charges on the
same day are 11 real rows
- On re-import of an overlapping date range, rows whose key already exists are skipped, so
re-running a month-to-date export the next day doesn't double-count
- Deleting an import log entry cascades to every record in that batch (`import_id` FK)
## Common Development Tasks
## Error handling
### Running the Application
API routes use `try/catch` and pass errors to `next(err)`; `server.js` has a global handler.
Database functions return JSON with a `success` boolean.
```bash
# Setup (first time only)
./setup.sh
## Light / dark mode
# Start development server with auto-reload
npm run dev
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a
`ThemeProvider` that wraps the app in `main.jsx`.
# Start production server
npm start
- **Storage key:** `df_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
- **Toggle:** button at the foot of the sidebar (`Sidebar.jsx`), and in `BottomNav.jsx` on mobile; the effect writes `localStorage` and toggles the `.dark` class on `<html>`
- **CSS:** `ui/src/index.css` declares semantic tokens under `@theme` (`bg-surface`, `text-ink`, `text-muted`, `border-line`, `text-danger`, …) that resolve to CSS custom properties redefined by `.dark`. **Write components against the tokens, never against literal shades like `bg-white` or `text-gray-400`** — the old per-utility `.dark .bg-white { … }` overrides are gone and must not come back
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
- **Perspective viewer:** `Pivot.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` on initial load and in a `useEffect([dark])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
# Test API
curl http://localhost:3000/health
```
## Pivot inspector panel
### Database Changes
Clicking a data cell opens a right-hand inspector panel showing the underlying transactions
for that cell. See [docs/perspective.md](docs/perspective.md) for the Perspective API itself.
When modifying schema:
1. Edit `database/schema.sql`
2. Drop and recreate schema: `psql -d dataflow -f database/schema.sql`
3. Redeploy functions: `psql -d dataflow -f database/functions.sql`
- **Toggle**: clicking the same cell again closes the panel. The toggle key is `JSON.stringify({ p: row.__ROW_PATH__, c: column_names })` — stable across source and stack views.
- **Listener cleanup**: the `perspective-click` handler is stored in `perspClickHandlerRef` and removed via `removeEventListener` on effect cleanup. Without this, switching views accumulates duplicate listeners that fire multiple times per click.
- **split_by filter derivation**: `detail.config.filter` from the click event may omit split_by column constraints. They are derived from `column_names` positionally (`column_names[i]` matches `config.split_by[i]`) and appended to the filter before querying.
- **Row filtering**: a temporary `table.view({ filter, expressions })` is used so Perspective evaluates expression/computed columns correctly. Falls back to JS-side `filterRowsByConfig` on error (which skips filters for fields not in raw data).
- The panel is resizable via a drag handle on its left edge (`paneWidth` state, min 240px).
- The transaction table is sortable (click header) and shows column totals for all-numeric columns.
For production, write migration scripts instead of dropping schema.
## Pivot layout persistence
### Adding a New API Endpoint
Named layouts are stored in `dataflow.pivot_layouts` for both sources and stacks. The
`source_name` column holds either a source name or a stack name — the FK to `sources(name)`
was dropped to allow this. Source layouts use `/api/sources/:name/layouts`; stack layouts use
`/api/stacks/:name/layouts`. Both call the same DB functions (`list_pivot_layouts`,
`save_pivot_layout`, `delete_pivot_layout`). `localStorage` still remembers the *last active
layout* for a view (the `psp_layout_<name>` key), but the definitions live in the DB so they
persist across machines.
1. Add route to appropriate file in `api/routes/`
2. Follow existing patterns (async/await, error handling via `next()`)
3. Use parameterized queries to prevent SQL injection
4. Return consistent JSON format
## Adding features
### Testing
- One function, one job; keep functions under 100 lines
- Write clear SQL, not clever SQL
- Add the SQL function to the matching `database/*.sql` file, then the route that calls it
- Update `docs/spec.md` when you add or change an endpoint
Manual testing workflow:
1. Create a source: `POST /api/sources`
2. Create rules: `POST /api/rules`
3. Import data: `POST /api/sources/:name/import`
4. Apply transformations: `POST /api/sources/:name/transform`
5. View results: `GET /api/records/source/:name`
See `examples/GETTING_STARTED.md` for complete curl examples.
## Design Principles
1. **Simple over clever** - Straightforward code beats optimization
2. **Explicit over implicit** - No magic, no hidden triggers
3. **Clear naming** - `data` not `rec`, `transformed` not `allj`
4. **One function, one job** - No 250-line functions
5. **JSONB for flexibility** - Handle varying schemas without migrations
## Common Patterns
### Import Flow
```
CSV file → parse → import_records() → records table (data column)
```
### Transformation Flow
```
records.data → apply_transformations() →
- Apply each rule (regex extraction)
- Look up mappings
- Merge into records.transformed
```
### Deduplication
- `constraint_key` is a JSONB object of the constraint field values (readable, no hashing)
- Dedup is enforced at import time via CTE — NO unique DB constraint on constraint_key
- **The constraint key is for cross-batch re-import protection, NOT record uniqueness**
- Within a single import batch, ALL rows insert regardless of duplicate constraint keys
- Banks legitimately send multiple identical-looking transactions (same date, description, amount)
- Example: 11 Cedar Point merchandise charges on one day — all should insert in one batch
- On re-import of overlapping date range, rows whose constraint_key already exists in DB are skipped
- This prevents double-counting when you re-run a month-to-date export the next day
- NEVER use `ON CONFLICT (constraint_key)` — there is no unique constraint and it would wrongly
drop legitimate duplicate transactions from the same batch
- Deleting an import log entry cascades to all records from that batch (import_id FK)
### Error Handling
- API routes use `try/catch` and pass errors to `next(err)`
- Server.js has global error handler
- Database functions return JSON with `success` boolean
## File Structure
```
dataflow/
├── database/
│ ├── schema.sql # Table definitions
│ └── functions.sql # Import/transform functions
├── api/
│ ├── server.js # Express server
│ └── routes/ # API endpoints
│ ├── sources.js
│ ├── rules.js
│ ├── mappings.js
│ └── records.js
├── examples/
│ ├── GETTING_STARTED.md # Tutorial
│ └── bank_transactions.csv
├── .env.example # Config template
├── package.json
└── README.md
```
## Comparison to Legacy TPS System
This project replaces an older system (in `/opt/tps`) that had:
- 2,150 lines of complex SQL with heavy duplication
- 5 nearly-identical 200+ line functions
- Confusing names and deep nested CTEs
- Complex trigger-based processing
Dataflow achieves the same functionality with:
- ~400 lines of simple SQL
- 4 focused functions
- Clear names and linear logic
- Explicit API-triggered processing
The simplification makes it easy to understand, modify, and maintain.
## Troubleshooting
**Database connection fails** — check `.env` credentials, that PostgreSQL is running, and
that the search path resolves to the `dataflow` schema.
**Database connection fails:**
- Check `.env` file exists and has correct credentials
- Verify PostgreSQL is running: `psql -U postgres -l`
- Check search path is set: Should default to `dataflow` schema
**Import succeeds but transformation does nothing** — check rules exist for that source
(`SELECT * FROM dataflow.rules WHERE source_name = '…'`), that `field` matches an actual key
in `data`, and test the pattern with `GET /api/rules/preview`.
**Import succeeds but transformation fails:**
- Check rules exist: `SELECT * FROM dataflow.rules WHERE source_name = 'xxx'`
- Verify field names match CSV columns
- Test regex pattern manually
- Check for SQL errors in logs
**Everything is marked duplicate** — `constraint_fields` probably don't match the real field
names, or the batch was already imported.
**All records marked as duplicates:**
- Verify `constraint_fields` match actual field names in data
- Check if data was already imported
- Use different source name for testing
## History
## Adding New Features
This replaces an older system still in `/opt/tps` — 2,150 lines of SQL with five
nearly-identical 200-line functions and trigger-based processing. Dataflow is a clean
rewrite, not a refactor. Some function bodies still carry `mirrors TPS …` comments pointing
at their counterpart there.
When adding features, follow these principles:
- Add ONE function that does ONE thing
- Keep functions under 100 lines if possible
- Write clear SQL, not clever SQL
- Add API endpoint that calls the function
- Document in README.md and update examples
## Notes for Claude
- This is a **simple** system by design - don't over-engineer it
- Keep functions focused and linear
- Use JSONB for flexibility, not as a crutch for bad design
- When confused, read the examples/GETTING_STARTED.md walkthrough
- The old TPS system is in `/opt/tps` - this is a clean rewrite, not a refactor

215
README.md
View File

@ -2,71 +2,198 @@
A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will
deduplicate on import, pull structure out with regex rules, map the extracted values to clean
output, and serve the result through a web UI and REST API.
## What It Does
## How it works
Dataflow helps you:
1. **Import** CSV data with automatic deduplication
2. **Transform** data using regex rules to extract meaningful information
3. **Map** extracted values to standardized output
4. **Query** the transformed data via a web UI or REST API
1. **Sources** define where data comes from and which fields make a record unique
2. **Rules** extract information with regex (`extract` or `replace` mode) —
e.g. pull the merchant out of a transaction description
3. **Mappings** turn extracted values into clean output —
`"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
4. **Records** are then queryable, pivotable, and exportable
Perfect for cleaning up messy data like bank transactions, product lists, or any repetitive data that needs normalization.
Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output),
and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never
clobbers something you typed by hand.
## Core Concepts
## Stack
### 1. Sources
Define where data comes from and how to deduplicate it.
PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`.
HTTP Basic auth, configured in `.env`.
**Example:** Bank transactions deduplicated by date + amount + description
## Getting started
### 2. Rules
Extract information using regex patterns (`extract` or `replace` modes).
Requires PostgreSQL 12+, Node.js 18+, and Python 3.
**Example:** Extract merchant name from transaction description
### 3. Mappings
Map extracted values to clean, standardized output.
**Example:** "DISCOUNT DRUG MART 32" → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
## Architecture
- **Database:** PostgreSQL with JSONB for flexible data storage
- **API:** Node.js/Express REST API
- **UI:** React SPA served from `public/`
- **Auth:** HTTP Basic auth (configured in `.env`)
## Design Principles
- **Simple & Clear** - Easy to understand what's happening
- **Explicit** - No hidden magic or complex triggers
- **Flexible** - Handle varying data formats without schema changes
## Getting Started
### Prerequisites
- PostgreSQL 12+
- Node.js 16+
- Python 3 (for `manage.py`)
### Installation
1. Install Node dependencies:
```bash
npm install
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
```
The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api`
(port set by `API_PORT` in `.env`).
2. Run the management script to configure and deploy everything:
```bash
python3 manage.py
```
For a walkthrough that creates a source, adds rules and mappings, and imports the sample
CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**.
For development with auto-reload:
```bash
npm run dev
```
## Documentation
The UI is available at `http://localhost:3000`. The API is at `http://localhost:3000/api`.
| | |
|---|---|
| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl |
| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` |
| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages |
| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference |
## Management Script (`manage.py`)
## Project structure
`manage.py` is an interactive tool for configuring, deploying, and managing the service. Run it and choose from the numbered menu:
```
python3 manage.py
```
| # | Action |
|---|--------|
| 1 | **Database configuration** — create/update `.env`, optionally create the PostgreSQL user/database, and deploy schema + functions |
| 2 | Redeploy schema only (`database/schema.sql`) — drops and recreates all tables |
| 3 | Redeploy SQL functions only (`database/queries/`) |
| 4 | Build UI (`ui/` → `public/`) |
| 5 | Set up nginx reverse proxy (HTTP or HTTPS via certbot) |
| 6 | Install systemd service unit (`dataflow.service`) |
| 7 | Start / restart `dataflow.service` |
| 8 | Stop `dataflow.service` |
| 9 | Set login credentials (`LOGIN_USER` / `LOGIN_PASSWORD_HASH` in `.env`) |
The status screen at the top of the menu shows the current state of each component (database connection, schema, UI build, service, nginx).
**Typical first-time setup:** run options 1 → 4 → 9 → 6 → 7 (→ 5 if you want nginx).
## API Reference
All `/api` routes require HTTP Basic authentication.
### Sources — `/api/sources`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/sources` | List all sources |
| POST | `/api/sources` | Create a source |
| GET | `/api/sources/:name` | Get a source |
| PUT | `/api/sources/:name` | Update a source |
| DELETE | `/api/sources/:name` | Delete a source |
| POST | `/api/sources/suggest` | Suggest source definition from CSV upload |
| POST | `/api/sources/:name/import` | Import CSV data and auto-apply transformations to new records |
| GET | `/api/sources/:name/import-log` | View import history (includes `inserted_keys` / `excluded_keys` in `info`) |
| DELETE | `/api/sources/:name/import-log/:id` | Delete an import batch and all its records |
| POST | `/api/sources/:name/transform` | Apply rules and mappings to any untransformed records |
| POST | `/api/sources/:name/reprocess` | Re-transform all records |
| GET | `/api/sources/:name/fields` | List all known field names |
| GET | `/api/sources/:name/stats` | Get record and mapping counts |
| POST | `/api/sources/:name/view` | Generate output view |
| GET | `/api/sources/:name/view-data` | Query output view (paginated, sortable) |
### Rules — `/api/rules`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/rules/source/:source_name` | List rules for a source |
| POST | `/api/rules` | Create a rule |
| GET | `/api/rules/:id` | Get a rule |
| PUT | `/api/rules/:id` | Update a rule |
| DELETE | `/api/rules/:id` | Delete a rule |
| GET | `/api/rules/preview` | Preview a pattern against real records (ad-hoc) |
| GET | `/api/rules/:id/test` | Test a saved rule against real records |
### Mappings — `/api/mappings`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/mappings/source/:source_name` | List mappings |
| POST | `/api/mappings` | Create a mapping |
| POST | `/api/mappings/bulk` | Bulk create/update mappings |
| GET | `/api/mappings/:id` | Get a mapping |
| PUT | `/api/mappings/:id` | Update a mapping |
| DELETE | `/api/mappings/:id` | Delete a mapping |
| GET | `/api/mappings/source/:source_name/unmapped` | Get values with no mapping yet |
| GET | `/api/mappings/source/:source_name/all-values` | All extracted values with counts |
| GET | `/api/mappings/source/:source_name/counts` | Record counts for existing mappings |
| GET | `/api/mappings/source/:source_name/export.tsv` | Export values as TSV |
| POST | `/api/mappings/source/:source_name/import-csv` | Import mappings from TSV |
### Records — `/api/records`
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/records/source/:source_name` | List records (paginated) |
| GET | `/api/records/:id` | Get a single record |
| POST | `/api/records/search` | Search records |
| DELETE | `/api/records/:id` | Delete a record |
| DELETE | `/api/records/source/:source_name/all` | Delete all records for a source |
## Typical Workflow
```
1. Create a source (POST /api/sources)
2. Create transformation rules (POST /api/rules)
3. Import CSV data (POST /api/sources/:name/import) — transformations applied automatically to new records
4. Preview rules against real data (GET /api/rules/preview)
5. Review unmapped values (GET /api/mappings/source/:name/unmapped)
6. Add mappings (POST /api/mappings or bulk import via TSV)
7. Reprocess to apply new mappings (POST /api/sources/:name/reprocess)
8. Query results (GET /api/sources/:name/view-data)
```
See `examples/GETTING_STARTED.md` for a complete walkthrough with curl examples.
## Project Structure
```
dataflow/
├── manage.py # interactive setup / deploy / uninstall
├── database/ # schema.sql + one .sql file per API route
├── api/ # Express server, routes, auth middleware
├── ui/ # React source (built to public/)
├── public/ # built UI, served as static files
├── docs/
└── examples/ # sample CSV for the tutorial
├── database/
│ ├── schema.sql # Table definitions
│ └── functions.sql # Import/transform/query functions
├── api/
│ ├── server.js # Express server
│ ├── middleware/
│ │ └── auth.js # Basic auth middleware
│ ├── lib/
│ │ └── sql.js # SQL literal helpers
│ └── routes/
│ ├── sources.js
│ ├── rules.js
│ ├── mappings.js
│ └── records.js
├── public/ # Built React UI (served as static files)
├── examples/
│ ├── GETTING_STARTED.md
│ └── bank_transactions.csv
└── .env.example
```
Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js`
and `database/rules.sql` are the two halves of the same feature.
`database/*.sql` is the source of truth for every database function — never edit one directly
in the database, or the next redeploy will silently revert it.
## License
MIT

253
docs/spec.md → SPEC.md
View File

@ -18,7 +18,7 @@ SQL functions are the single source of truth for business logic. The API layer i
Database calls in the route files use fully formed SQL strings with values interpolated directly (not parameterized). This makes every query copy-pasteable into psql for debugging. A small `lit()` helper in `api/lib/sql.js` handles quoting and escaping. This is an intentional trade-off: the tool is internal, and debuggability is worth more than the marginal injection protection parameterization provides over what `lit()` already does.
### One SQL file per route
SQL is organized in `database/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `stacks.sql`, `status.sql`) plus `import.sql` and `transform.sql` for the import/transform engine. This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
SQL is organized in `database/queries/` with one file per route (`sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`). This makes it easy to find the SQL behind any API endpoint — look at the route file to find the function name, then look at the matching query file for the implementation.
### Explicit over implicit
Nothing happens automatically. Transformations are triggered by the user. Views are generated on demand. There are no database triggers, no background workers, no scheduled jobs.
@ -34,52 +34,36 @@ Raw imported records and transformed records are stored as JSONB. This avoids sc
manage.py — interactive CLI for setup, deployment, and management
database/
schema.sql — table definitions (run once or to reset)
sources.sql — all SQL for /api/sources
rules.sql — all SQL for /api/rules
mappings.sql — all SQL for /api/mappings
records.sql — all SQL for /api/records
stacks.sql — all SQL for /api/stacks
status.sql — all SQL for /api/status
import.sql — CSV import and the import audit trail
transform.sql — the rule/mapping engine
queries/
sources.sql — all SQL for /api/sources
rules.sql — all SQL for /api/rules
mappings.sql — all SQL for /api/mappings
records.sql — all SQL for /api/records
api/
server.js — Express server, mounts routes, auth middleware
middleware/
auth.js — Basic Auth enforcement on all /api routes
lib/
sql.js — lit() and arr() helpers for SQL literal building
simplefin.js — SimpleFIN Bridge client for bank transaction pulls
routes/
sources.js — HTTP handlers for source management
rules.js — HTTP handlers for rule management
mappings.js — HTTP handlers for mapping management
records.js — HTTP handlers for record queries
stacks.js — HTTP handlers for stack management
status.js — HTTP handler for deployment status
ui/
src/
api.js — fetch wrapper, credential management
App.jsx — root: login gate, routing, stale/reprocess banners
index.css — semantic colour tokens for light and dark
App.jsx — root: login gate, sidebar, source selector, routing
pages/
Login.jsx — username/password form
SourceList.jsx — source list and the create dialog
SourceDetail.jsx — one source: connection, fields, view, maintenance
Bridge.jsx — SimpleFIN accounts, balances, and subtotals
ImportHub.jsx — all sources with sync / upload actions
Import.jsx — CSV upload, SimpleFIN sync, and import log
Sources.jsx — source CRUD, field config, view generation
Import.jsx — CSV upload and import log
Rules.jsx — rule CRUD with live pattern preview
Mappings.jsx — mapping table with TSV import/export
Records.jsx — paginated, sortable view of transformed records
Pivot.jsx — interactive pivot table with cell inspector
Stacks.jsx — multi-source union views with running balance
Remap.jsx — bulk remap of an output field value across mappings
Log.jsx — global import log across all sources
components/ — Sidebar, BottomNav, navItems, SourceTabs, Section, SampleTable
theme.jsx — light/dark context provider
public/ — compiled UI (output of npm run build in ui/)
docs/ — this file, tutorial, UI and Perspective references
examples/ — bank_transactions.csv, the tutorial's sample data
```
---
@ -117,45 +101,6 @@ CSV file → parse in Node.js → import_records(source, data)
→ apply_transformations() runs automatically on new records
```
### SimpleFIN sync (API-based bank feeds)
```
POST /api/sources/:name/sync → api/lib/simplefin.js
→ GET {access_url}/accounts?account=…&start-date=… (Basic auth)
→ drop pending, flatten transactions, fold in account context
→ import_records(source, data) — identical path to a CSV import from here on
```
An alternative to CSV upload for sources that read from a bank API. Only the
fetching differs: dedup, logging, and transformation are the same code.
- **Authentication.** A SimpleFIN access URL *is* the credential — it carries
its own username and password (`https://user:pass@bridge.simplefin.org/simplefin`).
You claim it once from a setup token (`POST /api/sources/simplefin-claim`,
which consumes the token) and store it in `.env`, one variable per bridge. It
is deliberately **not** stored in the database, which `manage.py` offers to reset.
- **Source config.** A source opts in by having `simplefin` in its `config` JSONB:
`{"simplefin": {"account_id": "ACT-…", "access_url_env": "SIMPLEFIN_ACCESS_URL",
"days": 10}}`. Only `account_id` is required. `GET /api/sources/simplefin-accounts`
lists the accounts behind a bridge so you can find the id.
- **`constraint_fields` should be `['id']`.** SimpleFIN assigns each transaction a
stable id, which makes overlapping pulls free and — unlike date + amount +
description — keeps genuinely repeated charges as separate records.
- **Pending transactions are skipped** (`?include_pending=true` overrides). A
pending transaction gets a different id once it posts, so importing it would
produce a duplicate under a different key a day or two later.
- **Bridge errors are surfaced, not swallowed.** SimpleFIN returns HTTP 200 with
an `errors` array when an institution is failing. Those errors ride along in
the sync response so a broken connection doesn't read as a successful empty
pull; the Import page shows them in orange above the counts.
- **Refresh cadence and the 90-day wall.** The bridge polls banks roughly daily
and transactions can take a few days to appear, so the pull asks for a rolling
window (`days`, default 10) rather than tracking a cursor. The window has hard
limits: SimpleFIN caps any range at 90 days and advises staying under 45, so
`MAX_DAYS` is 89 and `days=0` or anything larger clamps to it. A `start-date`
is **always** sent — omitting it does not mean "everything available", it
returns only the few most recent transactions.
- **Cron.** A daily pull is just the endpoint:
`curl -sS -u user:pass -X POST http://localhost:3000/api/sources/NAME/sync`
### Transform
```
apply_transformations(source) — pure SQL CTE
@ -176,7 +121,7 @@ The transform is fully set-based — no row-by-row loops. All records for a sour
## SQL Functions
Each route file has a matching SQL file in `database/`; `import.sql` and `transform.sql` hold the engine shared by several routes.
Each file in `database/queries/` maps 1-to-1 with a route file.
**sources.sql**
`list_sources`, `get_source`, `create_source`, `update_source`, `delete_source`, `get_import_log`, `get_source_stats`, `get_source_fields`, `get_view_data` (plpgsql — dynamic sort via EXECUTE + quote_ident), `import_records`, `jsonb_merge` + `jsonb_concat_obj` aggregate, `apply_transformations`, `reprocess_records`, `generate_source_view`
@ -190,12 +135,6 @@ Each route file has a matching SQL file in `database/`; `import.sql` and `transf
**records.sql**
`list_records`, `get_record`, `search_records` (JSONB containment on data and transformed), `delete_record`, `delete_source_records`
**stacks.sql**
`list_stacks`, `get_stack`, `create_stack`, `update_stack`, `delete_stack`, `get_stack_view_data` (union of source views with field mapping and running balance), `list_pivot_layouts`, `save_pivot_layout`, `delete_pivot_layout`
**status.sql**
`get_status` — returns deployment state (schema version, function presence, service status)
---
## API
@ -206,107 +145,43 @@ All routes are under `/api`. Every route requires HTTP Basic Auth. The `GET /hea
**Route summary:**
### Sources — `api/routes/sources.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/sources | List all sources |
| POST | /api/sources | Create source |
| GET | /api/sources/:name | Get source |
| PUT | /api/sources/:name | Update source (constraint_fields, config, global_picklist) |
| DELETE | /api/sources/:name | Delete source and all its data |
| POST | /api/sources/suggest | Suggest source config from an uploaded CSV |
| POST | /api/sources/:name/import | Import CSV; transformations are applied to the new records |
| POST | /api/sources/:name/sync | Pull transactions from SimpleFIN and import them (`?days=`, `?include_pending=`) |
| GET | /api/sources/simplefin-accounts | List accounts behind a bridge (`?access_url_env=`) |
| POST | /api/sources/simplefin-claim | Exchange a setup token for a permanent access URL |
| GET | /api/sources/import-log | Import history across all sources |
| GET | /api/sources/:name/import-log | Import history for one source |
| DELETE | /api/sources/:name/import-log/:id | Delete an import batch and every record in it |
| POST | /api/sources/:name/transform | Apply transformations to untransformed records only |
| POST | /api/sources/:name/reprocess | Reapply transformations to all records |
| PUT | /api/sources/:name | Update source (constraint_fields, config) |
| DELETE | /api/sources/:name | Delete source and all data |
| POST | /api/sources/suggest | Suggest source config from CSV upload |
| POST | /api/sources/:name/import | Import CSV records |
| GET | /api/sources/:name/import-log | Import history |
| GET | /api/sources/:name/stats | Record counts |
| GET | /api/sources/:name/fields | All known field names and their origins |
| GET | /api/sources/:name/override-keys | Distinct field names used in overrides for this source |
| POST | /api/sources/:name/view | Generate/refresh the `dfv` view |
| GET | /api/sources/:name/view-data | Paginated, sortable, filterable view data |
| GET | /api/sources/:name/layouts | List saved pivot layouts |
| POST | /api/sources/:name/layouts | Save a pivot layout |
| DELETE | /api/sources/:name/layouts/:id | Delete a pivot layout |
### Rules — `api/routes/rules.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/sources/:name/fields | All known field names and origins |
| GET | /api/sources/:name/view-data | Paginated, sortable view data |
| POST | /api/sources/:name/transform | Apply transformations (new records only) |
| POST | /api/sources/:name/reprocess | Reapply transformations to all records |
| POST | /api/sources/:name/view | Generate dfv view |
| GET | /api/rules/source/:name | List rules for a source |
| GET | /api/rules/:id | Get a rule |
| GET | /api/rules/preview | Preview pattern against live records |
| GET | /api/rules/:id/test | Test saved rule against live records |
| POST | /api/rules | Create rule |
| PUT | /api/rules/:id | Update rule |
| DELETE | /api/rules/:id | Delete rule |
| GET | /api/rules/preview | Preview an ad-hoc pattern against live records |
| GET | /api/rules/:id/test | Test a saved rule against live records |
### Mappings — `api/routes/mappings.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/mappings/source/:name | List mappings |
| GET | /api/mappings/:id | Get a mapping |
| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) |
| GET | /api/mappings/source/:name/unmapped | Only unmapped extracted values |
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
| GET | /api/mappings/source/:name/export.tsv | Export mappings as TSV |
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from TSV |
| POST | /api/mappings | Create mapping |
| POST | /api/mappings/bulk | Upsert multiple mappings |
| PUT | /api/mappings/:id | Update mapping |
| DELETE | /api/mappings/:id | Delete mapping |
| GET | /api/mappings/source/:name/all-values | All extracted values (mapped + unmapped) with counts |
| GET | /api/mappings/source/:name/unmapped | Only values with no mapping yet |
| GET | /api/mappings/source/:name/counts | Record counts per mapping |
| GET | /api/mappings/source/:name/export.tsv | Export extracted values as TSV |
| POST | /api/mappings/source/:name/import-csv | Import/update mappings from an uploaded TSV |
| GET | /api/mappings/global-values | Output values across all `global_picklist` sources (autocomplete) |
| GET | /api/mappings/outputs | Search output field values across all mappings |
| GET | /api/mappings/outputs/:col/:val | Mappings carrying a specific output field value |
| POST | /api/mappings/remap-field | Replace an output field value across all mappings |
### Records — `api/routes/records.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/records/source/:name | List records (paginated) |
| GET | /api/records/:id | Get a single record |
| GET | /api/records/source/:name | List raw records |
| GET | /api/records/:id | Get single record |
| POST | /api/records/search | Search by JSONB containment |
| DELETE | /api/records/:id | Delete record |
| DELETE | /api/records/source/:name/all | Delete all records for a source |
| PUT | /api/records/:id/overrides | Set manual overrides on a record |
| DELETE | /api/records/:id/overrides | Clear a record's overrides |
| POST | /api/records/bulk-overrides | Apply the same overrides to many records |
### Stacks — `api/routes/stacks.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/stacks | List all stacks |
| GET | /api/stacks/:name | Get a stack with its sources |
| POST | /api/stacks | Create stack |
| PUT | /api/stacks/:name | Update stack |
| DELETE | /api/stacks/:name | Delete stack |
| PUT | /api/stacks/:name/sources/:source | Add or update a source in the stack |
| DELETE | /api/stacks/:name/sources/:source | Remove a source from the stack |
| PUT | /api/stacks/:name/sources/reorder | Reorder the stack's sources |
| GET | /api/stacks/:name/view-sql | Preview the SQL that would build the view (dry run) |
| POST | /api/stacks/:name/view | Generate/refresh the `dfv` view |
| POST | /api/stacks/:name/exec-sql | Execute user-edited SQL for the view |
| GET | /api/stacks/:name/view-data | Paginated stacked data with running balance |
| GET | /api/stacks/:name/balance | Current running balance from the generated view |
| POST | /api/stacks/:name/calibrate | Set the balance offset from a known balance at a date |
| GET | /api/stacks/:name/layouts | List saved pivot layouts |
| POST | /api/stacks/:name/layouts | Save a pivot layout |
| DELETE | /api/stacks/:name/layouts/:id | Delete a pivot layout |
### Status — `api/routes/status.js`
| Method | Path | Description |
|--------|------|-------------|
| GET | /api/status | Deployment status |
| GET | /health | Health check (no auth) |
---
@ -331,43 +206,11 @@ Built with React + Vite + Tailwind CSS. Compiled output goes to `public/`. The s
3. On 401 response, credentials are cleared and the login screen is shown.
4. `localStorage` persists the selected source name across sessions.
**Navigation.** The selected source is a route parameter, not global state: `/sources`
lists sources, `/sources/:name` owns one, and Import, Rules, Mappings, Records, and Pivot
are tabs beneath it (`components/SourceTabs.jsx`). The sidebar holds only top-level
destinations — Sources, Import, Bridge, Remap, Stacks, Log — defined once in
`components/navItems.jsx` and rendered by `Sidebar.jsx` on desktop and `BottomNav.jsx`
below the `md:` breakpoint. Out-of-sync and reprocess banners render above every page from
`App.jsx`.
**Colour.** Components use semantic tokens (`bg-surface`, `text-ink`, `text-muted`,
`border-line`, `text-danger`) declared in `index.css` under `@theme`, which resolve to CSS
variables redefined by `.dark`. There are no per-utility `.dark` override rules; a new
component gets both themes for free.
**Bundle.** `Pivot` is loaded with `React.lazy`, keeping Perspective (~4.7 MB gzipped) out
of the initial download and in a chunk fetched only when a pivot is opened.
**Pages:**
- **Sources** (`SourceList.jsx`) — Lists every source with its constraint fields and a
badge for bank feeds; clicking opens it. "New source" opens the create dialog, which can
seed fields from a CSV sample or from a linked SimpleFIN account.
- **Sources** — View and edit source configuration. Shows all known field names and their origins (raw data, schema, rules, mappings). Checkboxes control which fields are constraint fields and which appear in the output view. Supports CSV upload to auto-detect fields.
- **Source detail** (`SourceDetail.jsx`) — The Setup tab, grouped into titled panels:
Connection (link/unlink a SimpleFIN account, warns when constraint fields aren't `id`),
Fields and view (all known field names and their origins, with checkboxes for constraint
fields and view columns), Sample rows, Maintenance (reprocess), and Delete source.
- **Bridge** (`Bridge.jsx`) — Every account behind the SimpleFIN credential with balances,
the source each maps to, and subtotals split into banking versus retirement (recognised by
keyword on the account and institution names). Queries SimpleFIN only when Refresh is
pressed. Balances render in accounting style with negatives in parentheses.
- **Import** (`ImportHub.jsx`) — Top-level entry point for the frequent job. Lists every
source with record counts and last import date, a Sync button for bank feeds, and an
upload link for CSV sources.
- **Source › Import tab** — Upload a CSV to import records into the selected source. Transformations run automatically on new records. Shows import log with inserted/duplicate counts, expandable key detail, checkbox selection, and delete with confirmation. Sources with `config.simplefin.account_id` also get a Sync panel — a window selector (10/30/45 days or a full backfill) and a "Sync now" button that pulls from the bank API through the same import path.
- **Import** — Upload a CSV to import records into the selected source. Transformations run automatically on new records. Shows import log with inserted/duplicate counts, expandable key detail, checkbox selection, and delete with confirmation.
- **Rules** — Create and manage regex rules. Live preview fires automatically (debounced 500ms) as pattern/field/flags are edited, showing match results against real records. Rules can be enabled/disabled by toggle.
@ -375,11 +218,11 @@ of the initial download and in a chunk fetched only when a pivot is opened.
- **Records** — Paginated table showing the `dfv.{source}` view. Server-side sorting (column validated against `information_schema.columns`, interpolated with `quote_ident`). Dates are formatted `YYYY-MM-DD` for correct lexicographic sort. Regex filters can be added per column. If the view cast fails (e.g. a field typed as `date` contains text), the error is shown inline rather than a blank page.
- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` client/viewer/datagrid v4.5.1, viewer-d3fc v4.4.1 — installed via npm). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
- **Pivot** — Interactive pivot/crosstab powered by [Perspective](https://perspective.finos.org/) (`@perspective-dev` v4.4.0, loaded from CDN at runtime). Loads all rows from the source view into an in-browser Perspective worker and renders a `<perspective-viewer>` web component. Supports grouping, splitting, filtering, sorting, and charting interactively.
**Toolbar (above the viewer):**
- Named layouts — saved per source in the `pivot_layouts` DB table. Each chip recalls the full viewer state including group_by, split_by, filters, expressions, selection mode, and expand depth. A blue **Save** button overwrites the active layout in place; **+ Save as…** saves to a new name. The × on each chip deletes it.
- **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
- **depth: 0 1 2 3** — collapses or expands all grouped rows to the specified hierarchy level. Implemented via `view.set_depth(d)` + `plugin.draw(view)` (the only working mechanism found in v4.4.0 — `plugin_config.expand_depth` and `viewer.flush()` alone have no effect).
- The Perspective built-in **selection mode button** (Read-Only / Select Row / Select Column / Select Region) defaults to **Select Region** on fresh load, set directly via `plugin.restore({ edit_mode: 'SELECT_REGION' })` after the viewer loads.
**Cell inspector (right panel):**
@ -392,10 +235,7 @@ of the initial download and in a chunk fetched only when a pivot is opened.
- `localStorage` key `psp_layout_{source}` saves the last viewer state on each named layout save.
- Named layouts store `{ ...viewer.save(), plugin_config: plugin.save(), expand_depth }` as JSONB in `pivot_layouts`. On recall, viewer config, plugin config (edit mode), and expand depth are all restored independently.
See `docs/perspective.md` for the full technical reference on controlling Perspective programmatically.
- **Stacks** — Named unions of multiple sources, each chip linking to its pivot at
`/stacks/:name/pivot`. Each stack defines a field mapping (how source fields map to common output columns), an amount field, a date field, and an optional balance offset. The view-data endpoint unions the underlying source views and computes a running balance sorted by date. The Pivot page supports stacks as well as individual sources, with layouts stored in the same `pivot_layouts` table.
See `docs/perspective-pivot.md` for the full technical reference on controlling Perspective programmatically.
- **Log** — Global import log across all sources. Same expandable key detail and delete capability as the Import page, plus a source name column.
@ -420,7 +260,7 @@ Shows current status on every screen:
2. **Redeploy schema** — Runs `database/schema.sql` against the configured database. Warns that this drops all data. Requires explicit confirmation.
3. **Redeploy SQL functions** — Runs the function files in `database/` in dependency order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`, `import.sql`, `transform.sql`, `stacks.sql`, `status.sql`. Safe to run at any time without data loss.
3. **Redeploy SQL functions** — Runs all four files in `database/queries/` in order: `sources.sql`, `rules.sql`, `mappings.sql`, `records.sql`. Safe to run at any time without data loss.
4. **Build UI** — Runs `npm run build` in `ui/`, outputting to `public/`.
@ -434,10 +274,6 @@ Shows current status on every screen:
9. **Set login credentials** — Prompts for username and password, bcrypt-hashes the password via `node -e "require('bcrypt')..."`, and writes `LOGIN_USER` and `LOGIN_PASSWORD_HASH` to `.env`. Requires Node.js and bcrypt npm package to be installed.
10. **Claim SimpleFIN setup token** — Prompts for a setup token (hidden input), exchanges it for a permanent access URL via `api/lib/simplefin.js`, and writes `SIMPLEFIN_ACCESS_URL` to `.env`. Warns and confirms before replacing an existing URL. Setup tokens are single-use, so a failed claim generally means a new token is needed. Prints only the bridge host, never the embedded credentials.
11. **Uninstall** — Reverses everything the other options install, in reverse order: stops/disables/removes the systemd unit, removes the nginx site and reloads nginx, drops the database and its user (prompts for admin credentials), then deletes `.env`, `public/`, and `node_modules`. Lists exactly what it found before doing anything and requires typing `delete` to proceed. The repository itself is left in place.
**Key behaviors:**
- All commands that will be run are printed before the user is asked to confirm.
- Actions that require sudo prompt transparently — `sudo` is not run with `-n`, so it uses cached credentials or prompts as normal.
@ -459,8 +295,6 @@ API_PORT Port the Express server listens on (default 3020)
NODE_ENV development | production
LOGIN_USER Username for Basic Auth
LOGIN_PASSWORD_HASH bcrypt hash of the password
SIMPLEFIN_ACCESS_URL Default SimpleFIN bridge URL; per-source override via config.simplefin.access_url_env
```
---
@ -487,18 +321,13 @@ The server binds to `0.0.0.0` on `API_PORT` and serves both the API and the comp
## Deploying SQL Changes
Any time SQL functions are modified, run `python3 manage.py` and choose "Redeploy SQL
functions only". It runs every function file in dependency order — the list lives in
`QUERY_FILES` in `manage.py`, which is the one place the order is defined.
To deploy a single file by hand:
Any time SQL functions are modified:
```bash
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -v ON_ERROR_STOP=1 -f database/rules.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/sources.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/rules.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/mappings.sql
PGPASSWORD=<pass> psql -h <host> -U <user> -d <db> -f database/queries/records.sql
```
Deployment is safe to repeat — every function uses `CREATE OR REPLACE`.
**The files are the source of truth.** Editing a function directly in the database, without
writing the change back to its file, means the next redeploy silently reverts it.
Then restart the server. Function deployment is safe to repeat — all functions use `CREATE OR REPLACE`.
Schema changes (`schema.sql`) drop and recreate the schema, deleting all data. In production, write migration scripts instead.

42
api/lib/fields.js
View File

@ -1,42 +0,0 @@
/**
* Field inference
* Derives a field list and column types from sample records, so a source can be
* configured from real data rather than an assumed shape. Used by the CSV
* suggest endpoint and by bank-feed sampling.
*/
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}(T[\d:.Z+-]+)?$/;
/**
* @param records array of flat objects
* @param limit how many rows to look at
* @returns { fields: [{name, type}], sampleRows }
*
* Keys are unioned across the sample, not read off the first row — API feeds
* omit optional keys entirely on records that lack them.
*/
function inferFields(records, limit = 50) {
const sampleRows = records.slice(0, limit);
const keys = [];
for (const row of sampleRows) {
for (const key of Object.keys(row)) {
if (!keys.includes(key)) keys.push(key);
}
}
const fields = keys.map(key => {
const vals = sampleRows.map(r => r[key]).filter(v => v !== '' && v != null);
let type = 'text';
if (vals.length > 0 && vals.every(v => !isNaN(parseFloat(v)) && isFinite(v) && String(v).charAt(0) !== '0')) {
type = 'numeric';
} else if (vals.length > 0 && vals.every(v => ISO_DATE_RE.test(String(v)))) {
type = 'date';
}
return { name: key, type };
});
return { fields, sampleRows };
}
module.exports = { inferFields };

219
api/lib/simplefin.js
View File

@ -1,219 +0,0 @@
/**
* SimpleFIN Bridge client
*
* Read-only access to linked bank accounts. Auth is a single "access URL" with
* credentials embedded (https://user:pass@bridge.simplefin.org/simplefin),
* obtained once by claiming a setup token. No certificates, no refresh flow —
* the URL is the credential, so it lives in .env and never in the database.
*
* Protocol: https://www.simplefin.org/protocol.html
*/
const REQUEST_TIMEOUT = 30000;
// The bridge refreshes from banks roughly daily and transactions can land a few
// days late, so pulls overlap by design and lean on constraint_key for dedupe.
const DEFAULT_DAYS = 10;
// The bridge hard-caps ranges at 90 days and reports the capping in `errors`;
// 89 stays just inside it so a routine backfill doesn't look like data loss. It
// also advises staying under 45 days, which it says on anything longer — that
// notice is passed through, since it is a real hint about future behaviour.
// Omitting start-date entirely is not "everything": it returns only the most
// recent handful of transactions, so a start-date is always sent.
const MAX_DAYS = 89;
class SimpleFinError extends Error {
constructor(message, status) {
super(message);
this.name = 'SimpleFinError';
this.status = status;
}
}
// Resolve an access URL from the environment. Sources name their own variable
// via config.simplefin.access_url_env so several bridges can coexist.
function getAccessUrl(urlEnv) {
const name = urlEnv || 'SIMPLEFIN_ACCESS_URL';
const value = process.env[name];
if (!value) {
throw new SimpleFinError(`SimpleFIN access URL not found — set ${name} in .env`, 500);
}
let parsed;
try {
parsed = new URL(value);
} catch {
throw new SimpleFinError(`${name} is not a valid URL`, 500);
}
if (!parsed.username) {
throw new SimpleFinError(`${name} has no credentials — expected https://user:pass@host/path`, 500);
}
return parsed;
}
async function get(accessUrl, path, params) {
// Credentials travel in the Authorization header, not the request line.
const target = new URL(accessUrl.pathname + path, accessUrl.origin);
for (const [k, v] of Object.entries(params || {})) {
if (v !== undefined && v !== null) target.searchParams.append(k, String(v));
}
const auth = Buffer.from(`${decodeURIComponent(accessUrl.username)}:${decodeURIComponent(accessUrl.password)}`).toString('base64');
let res;
try {
res = await fetch(target, {
headers: { Authorization: `Basic ${auth}`, Accept: 'application/json' },
signal: AbortSignal.timeout(REQUEST_TIMEOUT),
});
} catch (err) {
throw new SimpleFinError(`SimpleFIN request failed: ${err.message}`, 502);
}
if (res.status === 401 || res.status === 403) {
throw new SimpleFinError(
'SimpleFIN rejected the access URL — it may have been revoked; claim a new setup token', res.status);
}
if (!res.ok) {
throw new SimpleFinError(`SimpleFIN returned ${res.status}: ${(await res.text()).slice(0, 200)}`, res.status);
}
try {
return await res.json();
} catch {
throw new SimpleFinError('SimpleFIN returned a non-JSON response', 502);
}
}
/**
* Exchange a setup token for a permanent access URL. Run once per bridge; the
* returned URL goes in .env. The token is base64 of a one-shot claim URL and is
* consumed by this call.
*/
async function claimSetupToken(setupToken) {
let claimUrl;
try {
claimUrl = Buffer.from(String(setupToken).trim(), 'base64').toString('utf8');
new URL(claimUrl);
} catch {
throw new SimpleFinError('Setup token is not valid base64 of a claim URL', 400);
}
let res;
try {
res = await fetch(claimUrl, { method: 'POST', signal: AbortSignal.timeout(REQUEST_TIMEOUT) });
} catch (err) {
throw new SimpleFinError(`Claim request failed: ${err.message}`, 502);
}
if (!res.ok) {
throw new SimpleFinError(
`Claim returned ${res.status} — setup tokens can only be claimed once`, res.status);
}
return (await res.text()).trim();
}
// Epoch seconds → YYYY-MM-DD, so dates sort and compare as plain text the way
// CSV-imported dates already do. Pending transactions carry posted=0 rather than
// omitting it, so falsy epochs are "no date", not 1970-01-01.
function toDate(epochSeconds) {
if (!epochSeconds) return null;
return new Date(epochSeconds * 1000).toISOString().slice(0, 10);
}
// Flatten a SimpleFIN transaction into the shallow map the rule engine expects.
//
// Every scalar the bridge sends is passed through rather than whitelisted, so
// institution-specific fields (mcc, and whatever a given bank adds) show up in
// the data and in field discovery without a code change here. Only the epoch
// timestamps are reshaped, and account context is folded in so records stay
// self-describing.
function flatten(txn, account) {
const out = {};
for (const [key, value] of Object.entries(txn)) {
if (value === null || value === undefined) continue;
if (typeof value === 'object') continue; // nested objects handled below
out[key] = value;
}
// `extra` is free-form and institution-specific; hoist its scalars under a
// prefix so they can't collide with the documented fields
for (const [key, value] of Object.entries(txn.extra || {})) {
if (value !== null && value !== undefined && typeof value !== 'object') {
out[`extra_${key}`] = value;
}
}
const posted = toDate(txn.posted);
const transacted = toDate(txn.transacted_at);
delete out.posted; // replaced by the YYYY-MM-DD forms below
// A pending transaction has no posted date yet; fall back to when it was
// transacted so every record has something usable to sort and filter on
out.date = posted || transacted;
out.posted_date = posted;
out.transacted_at = transacted;
out.pending = txn.pending ? 'true' : 'false';
out.account_id = account.id;
out.account_name = account.name;
out.organization = account.org?.name || account.org?.domain;
return out;
}
async function listAccounts(urlEnv) {
const data = await get(getAccessUrl(urlEnv), '/accounts', { 'balances-only': 1 });
return {
errors: data.errors || [],
accounts: (data.accounts || []).map(a => ({
id: a.id,
name: a.name,
organization: a.org?.name || a.org?.domain,
currency: a.currency,
balance: a.balance,
available_balance: a['available-balance'],
balance_date: toDate(a['balance-date']),
})),
};
}
/**
* Fetch transactions for one account.
*
* days — how far back to ask for; 0 or more than MAX_DAYS means the full 90
* includePending — pending transactions get a new id once they post, so they are excluded by default
*/
async function fetchTransactions({ accountId, accessUrlEnv, days, includePending = false }) {
days = Number.isFinite(days) ? days : DEFAULT_DAYS;
if (days <= 0 || days > MAX_DAYS) days = MAX_DAYS;
const params = {
account: accountId,
'start-date': Math.floor(Date.now() / 1000) - days * 86400,
};
if (includePending) params.pending = 1;
const data = await get(getAccessUrl(accessUrlEnv), '/accounts', params);
// The bridge reports per-institution problems here and still returns 200 —
// one bank being down must not look like a successful empty pull.
const errors = data.errors || [];
const account = (data.accounts || []).find(a => a.id === accountId);
if (!account) {
const detail = errors.length ? ` — bridge reported: ${errors.join('; ')}` : '';
throw new SimpleFinError(`Account ${accountId} not returned by SimpleFIN${detail}`, 502);
}
const txns = account.transactions || [];
const kept = includePending ? txns : txns.filter(t => !t.pending);
return {
fetched: txns.length,
errors,
records: kept.map(t => flatten(t, account)),
};
}
module.exports = { listAccounts, fetchTransactions, claimSetupToken, SimpleFinError, DEFAULT_DAYS, MAX_DAYS };

48
api/routes/records.js
View File

@ -49,33 +49,49 @@ module.exports = (pool) => {
}
});
// Set overrides for all selected records
// Set overrides for all selected records and immediately merge into transformed
router.post('/bulk-overrides', async (req, res, next) => {
try {
const { source_name, record_ids, overrides } = req.body;
if (!source_name || !Array.isArray(record_ids) || record_ids.length === 0 || !overrides || typeof overrides !== 'object')
if (!source_name || !record_ids || !Array.isArray(record_ids) || record_ids.length === 0 || !overrides || typeof overrides !== 'object') {
return res.status(400).json({ error: 'source_name, record_ids array, and overrides object required' });
const idList = record_ids.map(id => parseInt(id)).join(',');
const result = await pool.query(
`SELECT bulk_set_record_overrides(${lit(source_name)}, ARRAY[${idList}]::int[], ${lit(overrides)}) as result`
);
res.json(result.rows[0].result);
}
const client = await pool.connect();
try {
await client.query('BEGIN');
for (const id of record_ids) {
await client.query(
`UPDATE dataflow.records
SET overrides = COALESCE(overrides, '{}'::jsonb) || $1,
transformed = transformed || $1
WHERE id = $2 AND source_name = $3`,
[overrides, id, source_name]
);
}
await client.query('COMMIT');
res.json({ updated: record_ids.length });
} catch (err) {
await client.query('ROLLBACK');
next(err);
} finally {
client.release();
}
} catch (err) {
next(err);
}
});
// Set overrides for a record
// Set overrides for a record and immediately merge into transformed
router.put('/:id/overrides', async (req, res, next) => {
try {
const { overrides } = req.body;
if (!overrides || typeof overrides !== 'object')
return res.status(400).json({ error: 'overrides object required' });
const result = await pool.query(
`SELECT set_record_overrides(${lit(parseInt(req.params.id))}, ${lit(overrides)}) as rec`
`SELECT * FROM set_record_overrides(${lit(parseInt(req.params.id))}, ${lit(overrides)})`
);
if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
res.json(result.rows[0].rec);
if (result.rows.length === 0) return res.status(404).json({ error: 'Record not found' });
res.json(result.rows[0]);
} catch (err) {
next(err);
}
@ -84,13 +100,13 @@ module.exports = (pool) => {
// Clear overrides and reprocess that record to restore computed values
router.delete('/:id/overrides', async (req, res, next) => {
try {
const result = await pool.query(
`SELECT clear_record_overrides(${lit(parseInt(req.params.id))}) as rec`
const rec = await pool.query(
`SELECT * FROM clear_record_overrides(${lit(parseInt(req.params.id))})`
);
if (!result.rows[0].rec) return res.status(404).json({ error: 'Record not found' });
const { source_name } = result.rows[0].rec;
if (rec.rows.length === 0) return res.status(404).json({ error: 'Record not found' });
// Reprocess this record so transformed reflects rules/mappings without overrides
await pool.query(
`SELECT apply_transformations(${lit(source_name)}, ARRAY[${lit(parseInt(req.params.id))}::int], true)`
`SELECT apply_transformations(${lit(rec.rows[0].source_name)}, ARRAY[${lit(parseInt(req.params.id))}::int], true)`
);
const updated = await pool.query(`SELECT * FROM get_record(${lit(parseInt(req.params.id))})`);
res.json(updated.rows[0]);

113
api/routes/sources.js
View File

@ -7,8 +7,6 @@ const express = require('express');
const multer = require('multer');
const { parse } = require('csv-parse/sync');
const { lit, arr } = require('../lib/sql');
const simplefin = require('../lib/simplefin');
const { inferFields } = require('../lib/fields');
const upload = multer({ storage: multer.memoryStorage() });
@ -25,56 +23,6 @@ module.exports = (pool) => {
}
});
// SimpleFIN helpers. Declared before /:name so they aren't shadowed by it.
// List the accounts behind a bridge — used to find the account_id for a source
router.get('/simplefin-accounts', async (req, res, next) => {
try {
res.json(await simplefin.listAccounts(req.query.access_url_env));
} catch (err) {
if (err instanceof simplefin.SimpleFinError) return res.status(err.status || 502).json({ error: err.message });
next(err);
}
});
// Sample an account's real transactions and infer its field list — the API
// equivalent of uploading a CSV to /suggest. Whatever the account actually
// returns is what gets offered, so investment or loan accounts describe
// themselves rather than being forced into a checking-account shape.
router.get('/simplefin-sample', async (req, res, next) => {
try {
const { account_id, access_url_env } = req.query;
if (!account_id) return res.status(400).json({ error: 'account_id is required' });
const { fetched, records, errors } = await simplefin.fetchTransactions({
accountId: account_id,
accessUrlEnv: access_url_env,
// The bridge advises staying under 45 days and warns at exactly
// 45, so 44 is the largest quiet sample window
days: req.query.days !== undefined ? parseInt(req.query.days) : 44,
includePending: true, // widen the sample; pending rows can carry extra keys
});
const { fields, sampleRows } = inferFields(records);
res.json({ fields, sampleRows, fetched, errors });
} catch (err) {
if (err instanceof simplefin.SimpleFinError) return res.status(err.status || 502).json({ error: err.message });
next(err);
}
});
// Exchange a one-shot setup token for the permanent access URL to put in .env
router.post('/simplefin-claim', async (req, res, next) => {
try {
const { setup_token } = req.body || {};
if (!setup_token) return res.status(400).json({ error: 'setup_token is required' });
res.json({ access_url: await simplefin.claimSetupToken(setup_token) });
} catch (err) {
if (err instanceof simplefin.SimpleFinError) return res.status(err.status || 502).json({ error: err.message });
next(err);
}
});
// List all sources
router.get('/', async (req, res, next) => {
try {
@ -104,7 +52,21 @@ module.exports = (pool) => {
const records = parse(req.file.buffer, { columns: true, skip_empty_lines: true, trim: true });
if (records.length === 0) return res.status(400).json({ error: 'CSV file is empty' });
const { fields, sampleRows } = inferFields(records);
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}(T[\d:.Z+-]+)?$/;
const sample = records[0];
const sampleRows = records.slice(0, 50);
const fields = Object.keys(sample).map(key => {
const vals = sampleRows.map(r => r[key]).filter(v => v !== '' && v != null);
let type = 'text';
if (vals.length > 0 && vals.every(v => !isNaN(parseFloat(v)) && isFinite(v) && String(v).charAt(0) !== '0')) {
type = 'numeric';
} else if (vals.length > 0 && vals.every(v => ISO_DATE_RE.test(String(v)))) {
type = 'date';
}
return { name: key, type };
});
res.json({ name: '', constraint_fields: [], fields, sampleRows });
} catch (err) {
next(err);
@ -177,51 +139,6 @@ module.exports = (pool) => {
}
});
// Pull transactions from SimpleFIN and import them, same as a CSV upload.
// Safe to re-run: overlapping transactions are skipped by constraint key.
router.post('/:name/sync', async (req, res, next) => {
try {
const sourceResult = await pool.query(`SELECT * FROM get_source(${lit(req.params.name)})`);
const source = sourceResult.rows[0];
if (!source || !source.name) return res.status(404).json({ error: 'Source not found' });
const cfg = (source.config || {}).simplefin;
if (!cfg || !cfg.account_id) {
return res.status(400).json({
error: `Source "${req.params.name}" has no simplefin.account_id in its config`
});
}
const opts = { ...req.query, ...req.body };
const { fetched, errors, records } = await simplefin.fetchTransactions({
accountId: cfg.account_id,
accessUrlEnv: cfg.access_url_env,
days: opts.days !== undefined ? parseInt(opts.days) : cfg.days,
includePending: opts.include_pending === true || opts.include_pending === 'true',
});
if (records.length === 0) {
return res.json({ success: true, fetched, errors, imported: 0, duplicates: 0 });
}
const importResult = await pool.query(
`SELECT import_records(${lit(req.params.name)}, ${lit(records)}) as result`
);
const importData = importResult.rows[0].result;
if (!importData.success) return res.json({ ...importData, fetched, errors });
const transformResult = await pool.query(
`SELECT apply_transformations(${lit(req.params.name)}) as result`
);
res.json({ ...importData, fetched, errors, transform: transformResult.rows[0].result });
} catch (err) {
if (err instanceof simplefin.SimpleFinError) return res.status(err.status || 502).json({ error: err.message });
next(err);
}
});
// Get import log
router.get('/:name/import-log', async (req, res, next) => {
try {

27
api/routes/stacks.js
View File

@ -170,32 +170,5 @@ module.exports = (pool) => {
} catch (err) { next(err); }
});
// Pivot layouts (same DB table as sources; FK was dropped to allow stack names)
router.get('/:name/layouts', async (req, res, next) => {
try {
const result = await pool.query(`SELECT * FROM list_pivot_layouts(${lit(req.params.name)})`);
res.json(result.rows);
} catch (err) { next(err); }
});
router.post('/:name/layouts', async (req, res, next) => {
try {
const { layout_name, config } = req.body;
if (!layout_name || !config) return res.status(400).json({ error: 'layout_name and config required' });
const result = await pool.query(
`SELECT * FROM save_pivot_layout(${lit(req.params.name)}, ${lit(layout_name)}, ${lit(config)})`
);
res.json(result.rows[0]);
} catch (err) { next(err); }
});
router.delete('/:name/layouts/:id', async (req, res, next) => {
try {
const result = await pool.query(`SELECT * FROM delete_pivot_layout(${lit(parseInt(req.params.id))})`);
if (result.rows.length === 0) return res.status(404).json({ error: 'Layout not found' });
res.json({ success: true });
} catch (err) { next(err); }
});
return router;
};

10
api/server.js
View File

@ -3,7 +3,7 @@
* Simple REST API for data transformation
*/
require('dotenv').config({ quiet: true });
require('dotenv').config();
const express = require('express');
const { Pool } = require('pg');
@ -16,8 +16,7 @@ const pool = new Pool({
port: process.env.DB_PORT,
database: process.env.DB_NAME,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
options: '-c search_path=dataflow,public'
password: process.env.DB_PASSWORD
});
// Middleware
@ -32,6 +31,11 @@ app.use('/api', auth);
const path = require('path');
app.use(express.static(path.join(__dirname, '../public')));
// Set search path for all queries
pool.on('connect', (client) => {
client.query('SET search_path TO dataflow, public');
});
// Test database connection
pool.query('SELECT NOW()', (err, res) => {
if (err) {

536
database/functions.sql Normal file
View File

@ -0,0 +1,536 @@
--
-- Dataflow Functions
-- Simple, clear functions for import and transformation
--
SET search_path TO dataflow, public;
------------------------------------------------------
-- Function: import_records
-- Import data with automatic deduplication
------------------------------------------------------
CREATE OR REPLACE FUNCTION import_records(
p_source_name TEXT,
p_data JSONB -- Array of records
) RETURNS JSON AS $$
DECLARE
v_constraint_fields TEXT[];
v_inserted INTEGER;
v_duplicates INTEGER;
v_log_id INTEGER;
BEGIN
SELECT constraint_fields INTO v_constraint_fields
FROM dataflow.sources
WHERE name = p_source_name;
IF v_constraint_fields IS NULL THEN
RETURN json_build_object(
'success', false,
'error', 'Source not found: ' || p_source_name
);
END IF;
WITH
-- All incoming records with their constraint keys
pending AS (
SELECT
rec.value AS data,
rec.ordinality AS seq,
(SELECT jsonb_object_agg(f, rec.value->>f)
FROM unnest(v_constraint_fields) AS f) AS constraint_key
FROM jsonb_array_elements(p_data) WITH ORDINALITY AS rec
),
-- Keys already in the database (excluded)
existing AS (
SELECT DISTINCT r.constraint_key
FROM dataflow.records r
INNER JOIN pending p ON p.constraint_key = r.constraint_key
WHERE r.source_name = p_source_name
),
-- Rows whose constraint key is not yet in the database
new_records AS (
SELECT p.data, p.constraint_key, p.seq
FROM pending p
WHERE NOT EXISTS (SELECT 1 FROM existing e WHERE e.constraint_key = p.constraint_key)
),
-- Write the log entry
log_entry AS (
INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate, info)
VALUES (
p_source_name,
(SELECT count(*) FROM new_records),
(SELECT count(*) FROM pending) - (SELECT count(*) FROM new_records),
jsonb_build_object(
'total', jsonb_array_length(p_data),
'inserted_keys', (SELECT jsonb_agg(constraint_key ORDER BY constraint_key) FROM new_records),
'excluded_keys', (SELECT jsonb_agg(constraint_key) FROM existing)
)
)
RETURNING id, records_imported, records_duplicate
),
-- Insert new records
inserted AS (
INSERT INTO dataflow.records (source_name, data, constraint_key, import_id)
SELECT p_source_name, nr.data, nr.constraint_key, (SELECT id FROM log_entry)
FROM new_records nr
ORDER BY nr.seq
RETURNING id
)
SELECT le.id, le.records_imported, le.records_duplicate
INTO v_log_id, v_inserted, v_duplicates
FROM log_entry le;
RETURN json_build_object(
'success', true,
'imported', v_inserted,
'duplicates', v_duplicates,
'log_id', v_log_id
);
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION import_records IS 'Import records with automatic deduplication';
------------------------------------------------------
-- Function: get_import_log
-- Return import history for a source
------------------------------------------------------
CREATE OR REPLACE FUNCTION get_import_log(p_source_name TEXT)
RETURNS TABLE (
id INTEGER,
source_name TEXT,
records_imported INTEGER,
records_duplicate INTEGER,
imported_at TIMESTAMPTZ,
info JSONB
) AS $$
SELECT id, source_name, records_imported, records_duplicate, imported_at, info
FROM dataflow.import_log
WHERE source_name = p_source_name
ORDER BY imported_at DESC;
$$ LANGUAGE sql;
COMMENT ON FUNCTION get_import_log IS 'Return import history for a source, newest first, including inserted/excluded key lists';
------------------------------------------------------
-- Function: get_all_import_logs
-- Return import history across all sources
------------------------------------------------------
CREATE OR REPLACE FUNCTION get_all_import_logs()
RETURNS TABLE (
id INTEGER,
source_name TEXT,
records_imported INTEGER,
records_duplicate INTEGER,
imported_at TIMESTAMPTZ,
info JSONB
) AS $$
SELECT id, source_name, records_imported, records_duplicate, imported_at, info
FROM dataflow.import_log
ORDER BY imported_at DESC;
$$ LANGUAGE sql;
COMMENT ON FUNCTION get_all_import_logs IS 'Return import history across all sources, newest first';
------------------------------------------------------
-- Function: delete_import
-- Delete all records from a specific import and remove the log entry
------------------------------------------------------
CREATE OR REPLACE FUNCTION delete_import(p_log_id INTEGER)
RETURNS JSON AS $$
DECLARE
v_deleted INTEGER;
BEGIN
IF NOT EXISTS (SELECT 1 FROM dataflow.import_log WHERE id = p_log_id) THEN
RETURN json_build_object('success', false, 'error', 'Import log entry not found');
END IF;
SELECT count(*) INTO v_deleted FROM dataflow.records WHERE import_id = p_log_id;
-- Cascade handles deleting records via FK ON DELETE CASCADE
DELETE FROM dataflow.import_log WHERE id = p_log_id;
RETURN json_build_object(
'success', true,
'records_deleted', v_deleted,
'log_id', p_log_id
);
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION delete_import IS 'Delete all records belonging to an import batch and remove the log entry';
------------------------------------------------------
-- Aggregate: jsonb_concat_obj
-- Merge JSONB objects across rows (later rows win on key conflicts)
-- Usage: jsonb_concat_obj(col ORDER BY sequence)
------------------------------------------------------
CREATE OR REPLACE FUNCTION dataflow.jsonb_merge(a JSONB, b JSONB)
RETURNS JSONB AS $$
SELECT COALESCE(a, '{}') || COALESCE(b, '{}')
$$ LANGUAGE sql IMMUTABLE;
DROP AGGREGATE IF EXISTS dataflow.jsonb_concat_obj(JSONB);
CREATE AGGREGATE dataflow.jsonb_concat_obj(JSONB) (
sfunc = dataflow.jsonb_merge,
stype = JSONB,
initcond = '{}'
);
------------------------------------------------------
-- Function: apply_transformations
-- Apply all transformation rules to records (set-based)
------------------------------------------------------
DROP FUNCTION IF EXISTS apply_transformations(TEXT, INTEGER[]);
CREATE OR REPLACE FUNCTION apply_transformations(
p_source_name TEXT,
p_record_ids INTEGER[] DEFAULT NULL, -- NULL = all eligible records
p_overwrite BOOLEAN DEFAULT FALSE -- FALSE = skip already-transformed, TRUE = overwrite all
) RETURNS JSON AS $$
WITH
-- All records to process
qualifying AS (
SELECT id, data
FROM dataflow.records
WHERE source_name = p_source_name
AND (p_overwrite OR transformed IS NULL)
AND (p_record_ids IS NULL OR id = ANY(p_record_ids))
),
-- Mirror TPS rx: fan out one row per regex match, drive from rules → records
rx AS (
SELECT
q.id,
r.name AS rule_name,
r.sequence,
r.output_field,
r.retain,
r.function_type,
COALESCE(mt.rn, rp.rn, 1) AS result_number,
-- extract: build map_val and retain_val per match (mirrors TPS)
CASE WHEN array_length(mt.mt, 1) = 1 THEN to_jsonb(mt.mt[1]) ELSE to_jsonb(mt.mt) END AS match_val,
to_jsonb(rp.rp) AS replace_val
FROM dataflow.rules r
INNER JOIN qualifying q ON q.data ? r.field
LEFT JOIN LATERAL regexp_matches(q.data ->> r.field, r.pattern, r.flags)
WITH ORDINALITY AS mt(mt, rn) ON r.function_type = 'extract'
LEFT JOIN LATERAL regexp_replace(q.data ->> r.field, r.pattern, r.replace_value, r.flags)
WITH ORDINALITY AS rp(rp, rn) ON r.function_type = 'replace'
WHERE r.source_name = p_source_name
AND r.enabled = true
),
-- Aggregate match rows back into one value per (record, rule) — mirrors TPS agg_to_target_items
agg_matches AS (
SELECT
id,
rule_name,
sequence,
output_field,
retain,
function_type,
CASE function_type
WHEN 'replace' THEN jsonb_agg(replace_val) -> 0
ELSE
CASE WHEN max(result_number) = 1
THEN jsonb_agg(match_val ORDER BY result_number) -> 0
ELSE jsonb_agg(match_val ORDER BY result_number)
END
END AS extracted
FROM rx
GROUP BY id, rule_name, sequence, output_field, retain, function_type
),
-- Join with mappings to find mapped output — mirrors TPS link_map
linked AS (
SELECT
a.id,
a.sequence,
a.output_field,
a.retain,
a.extracted,
m.output AS mapped
FROM agg_matches a
LEFT JOIN dataflow.mappings m ON
m.source_name = p_source_name
AND m.rule_name = a.rule_name
AND m.input_value = a.extracted
WHERE a.extracted IS NOT NULL
),
-- Build per-rule output JSONB:
-- mapped → use mapping output; also write output_field if retain = true
-- no map → write extracted value to output_field
rule_output AS (
SELECT
id,
sequence,
CASE
WHEN mapped IS NOT NULL THEN
mapped ||
CASE WHEN retain
THEN jsonb_build_object(output_field, extracted)
ELSE '{}'::jsonb
END
ELSE
jsonb_build_object(output_field, extracted)
END AS output
FROM linked
),
-- Merge all rule outputs per record in sequence order — mirrors TPS agg_to_id
record_additions AS (
SELECT
id,
dataflow.jsonb_concat_obj(output ORDER BY sequence) AS additions
FROM rule_output
GROUP BY id
),
-- Update all qualifying records; records with no rule matches get transformed = data
updated AS (
UPDATE dataflow.records rec
SET transformed = rec.data || COALESCE(ra.additions, '{}'::jsonb) || COALESCE(rec.overrides, '{}'::jsonb),
transformed_at = CURRENT_TIMESTAMP
FROM qualifying q
LEFT JOIN record_additions ra ON ra.id = q.id
WHERE rec.id = q.id
RETURNING rec.id
)
SELECT json_build_object('success', true, 'transformed', count(*))
FROM updated
$$ LANGUAGE sql;
COMMENT ON FUNCTION apply_transformations IS 'Apply transformation rules and mappings to records (set-based CTE)';
------------------------------------------------------
-- Function: get_all_values
-- All extracted values (mapped + unmapped) with counts and mapping output
------------------------------------------------------
DROP FUNCTION IF EXISTS get_all_values(TEXT, TEXT);
CREATE FUNCTION get_all_values(
p_source_name TEXT,
p_rule_name TEXT DEFAULT NULL
) RETURNS TABLE (
rule_name TEXT,
output_field TEXT,
source_field TEXT,
extracted_value JSONB,
record_count BIGINT,
sample JSONB,
mapping_id INTEGER,
output JSONB,
is_mapped BOOLEAN
) AS $$
BEGIN
RETURN QUERY
WITH extracted AS (
SELECT
r.name AS rule_name,
r.output_field,
r.field AS source_field,
rec.transformed->r.output_field AS extracted_value,
rec.data AS record_data,
row_number() OVER (
PARTITION BY r.name, rec.transformed->r.output_field
ORDER BY rec.id
) AS rn
FROM dataflow.records rec
CROSS JOIN dataflow.rules r
WHERE
rec.source_name = p_source_name
AND r.source_name = p_source_name
AND rec.transformed IS NOT NULL
AND rec.transformed ? r.output_field
AND (p_rule_name IS NULL OR r.name = p_rule_name)
AND rec.data ? r.field
),
aggregated AS (
SELECT
e.rule_name,
e.output_field,
e.source_field,
e.extracted_value,
count(*) AS record_count,
jsonb_agg(e.record_data ORDER BY e.rn) FILTER (WHERE e.rn <= 5) AS sample
FROM extracted e
GROUP BY e.rule_name, e.output_field, e.source_field, e.extracted_value
)
SELECT
a.rule_name,
a.output_field,
a.source_field,
a.extracted_value,
a.record_count,
a.sample,
m.id AS mapping_id,
m.output,
(m.id IS NOT NULL) AS is_mapped
FROM aggregated a
LEFT JOIN dataflow.mappings m ON
m.source_name = p_source_name
AND m.rule_name = a.rule_name
AND m.input_value = a.extracted_value
ORDER BY a.record_count DESC;
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION get_all_values IS 'All extracted values with record counts and mapping output (single query for All tab)';
------------------------------------------------------
-- Function: get_unmapped_values
-- Find extracted values that need mappings
------------------------------------------------------
DROP FUNCTION IF EXISTS get_unmapped_values(TEXT, TEXT);
CREATE FUNCTION get_unmapped_values(
p_source_name TEXT,
p_rule_name TEXT DEFAULT NULL
) RETURNS TABLE (
rule_name TEXT,
output_field TEXT,
source_field TEXT,
extracted_value JSONB,
record_count BIGINT,
sample JSONB
) AS $$
BEGIN
RETURN QUERY
WITH extracted AS (
SELECT
r.name AS rule_name,
r.output_field,
r.field AS source_field,
rec.transformed->r.output_field AS extracted_value,
rec.data AS record_data,
row_number() OVER (
PARTITION BY r.name, rec.transformed->r.output_field
ORDER BY rec.id
) AS rn
FROM
dataflow.records rec
CROSS JOIN dataflow.rules r
WHERE
rec.source_name = p_source_name
AND r.source_name = p_source_name
AND rec.transformed IS NOT NULL
AND rec.transformed ? r.output_field
AND (p_rule_name IS NULL OR r.name = p_rule_name)
AND rec.data ? r.field
)
SELECT
e.rule_name,
e.output_field,
e.source_field,
e.extracted_value,
count(*) AS record_count,
jsonb_agg(e.record_data ORDER BY e.rn) FILTER (WHERE e.rn <= 5) AS sample
FROM extracted e
WHERE NOT EXISTS (
SELECT 1 FROM dataflow.mappings m
WHERE m.source_name = p_source_name
AND m.rule_name = e.rule_name
AND m.input_value = e.extracted_value
)
GROUP BY e.rule_name, e.output_field, e.source_field, e.extracted_value
ORDER BY count(*) DESC;
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION get_unmapped_values IS 'Find extracted values that need mappings defined';
------------------------------------------------------
-- Function: reprocess_records
-- Clear and reapply transformations
------------------------------------------------------
CREATE OR REPLACE FUNCTION reprocess_records(p_source_name TEXT)
RETURNS JSON AS $$
-- Overwrite all records directly — no clear step, mirrors TPS srce_map_overwrite
SELECT dataflow.apply_transformations(p_source_name, NULL, TRUE)
$$ LANGUAGE sql;
COMMENT ON FUNCTION reprocess_records IS 'Clear and reapply all transformations for a source';
------------------------------------------------------
-- Function: generate_source_view
-- Build a typed flat view in dfv schema
------------------------------------------------------
CREATE OR REPLACE FUNCTION generate_source_view(p_source_name TEXT)
RETURNS JSON AS $$
DECLARE
v_config JSONB;
v_fields JSONB;
v_field JSONB;
v_cols TEXT := '';
v_sql TEXT;
v_view TEXT;
BEGIN
SELECT config INTO v_config
FROM dataflow.sources
WHERE name = p_source_name;
IF v_config IS NULL OR NOT (v_config ? 'fields') OR jsonb_array_length(v_config->'fields') = 0 THEN
RETURN json_build_object('success', false, 'error', 'No schema fields defined for this source');
END IF;
v_fields := v_config->'fields';
FOR v_field IN SELECT * FROM jsonb_array_elements(v_fields)
LOOP
IF v_cols != '' THEN v_cols := v_cols || ', '; END IF;
IF v_field->>'expression' IS NOT NULL THEN
-- Computed expression: substitute {fieldname} refs with (transformed->>'fieldname')::type
-- e.g. "{Amount} * {sign}" → "(transformed->>'Amount')::numeric * (transformed->>'sign')::numeric"
DECLARE
v_expr TEXT := v_field->>'expression';
v_ref TEXT;
v_cast TEXT := COALESCE(NULLIF(v_field->>'type', ''), 'numeric');
BEGIN
WHILE v_expr ~ '\{[^}]+\}' LOOP
v_ref := substring(v_expr FROM '\{([^}]+)\}');
v_expr := replace(v_expr, '{' || v_ref || '}',
format('(transformed->>%L)::numeric', v_ref));
END LOOP;
v_cols := v_cols || format('%s AS %I', v_expr, v_field->>'name');
END;
ELSE
CASE v_field->>'type'
WHEN 'date' THEN
v_cols := v_cols || format('(transformed->>%L)::date AS %I',
v_field->>'name', v_field->>'name');
WHEN 'numeric' THEN
v_cols := v_cols || format('(transformed->>%L)::numeric AS %I',
v_field->>'name', v_field->>'name');
ELSE
v_cols := v_cols || format('transformed->>%L AS %I',
v_field->>'name', v_field->>'name');
END CASE;
END IF;
END LOOP;
CREATE SCHEMA IF NOT EXISTS dfv;
v_view := 'dfv.' || quote_ident(p_source_name);
EXECUTE format('DROP VIEW IF EXISTS %s CASCADE', v_view);
v_sql := format(
'CREATE VIEW %s AS SELECT id, %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
v_view, v_cols, p_source_name
);
EXECUTE v_sql;
RETURN json_build_object('success', true, 'view', v_view, 'sql', v_sql);
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION generate_source_view IS 'Generate a typed flat view in dfv schema from source config.fields';
------------------------------------------------------
-- Summary
------------------------------------------------------
-- Functions: 4 simple, focused functions
-- 1. import_records - Import with deduplication
-- 2. apply_transformations - Apply rules and mappings
-- 3. get_unmapped_values - Find values needing mappings
-- 4. reprocess_records - Re-transform all records
--
-- Each function does ONE thing clearly
-- No complex nested CTEs
-- Easy to understand and debug
------------------------------------------------------

159
database/import.sql
View File

@ -1,159 +0,0 @@
--
-- Import queries
-- CSV import and the import audit trail; SQL for the import/log endpoints in
-- api/routes/sources.js
--
SET search_path TO dataflow, public;
-- ── Import ────────────────────────────────────────────────────────────────────
-- Import records, skipping any whose constraint key already exists in the table.
--
-- Dedup is enforced here, in the CTE — there is no unique constraint on
-- constraint_key and ON CONFLICT must never be used. Within one batch every row
-- inserts even if two rows share a constraint key, because banks legitimately send
-- identical-looking transactions (same date, description, amount) on the same day.
-- The key exists only to stop a re-imported overlapping date range from
-- double-counting rows already in the table.
CREATE OR REPLACE FUNCTION import_records(
p_source_name TEXT,
p_data JSONB -- Array of records
) RETURNS JSON AS $$
DECLARE
v_constraint_fields TEXT[];
v_inserted INTEGER;
v_duplicates INTEGER;
v_log_id INTEGER;
BEGIN
SELECT constraint_fields INTO v_constraint_fields
FROM dataflow.sources
WHERE name = p_source_name;
IF v_constraint_fields IS NULL THEN
RETURN json_build_object(
'success', false,
'error', 'Source not found: ' || p_source_name
);
END IF;
WITH
-- All incoming records with their constraint keys
pending AS (
SELECT
rec.value AS data,
rec.ordinality AS seq,
(SELECT jsonb_object_agg(f, rec.value->>f)
FROM unnest(v_constraint_fields) AS f) AS constraint_key
FROM jsonb_array_elements(p_data) WITH ORDINALITY AS rec
),
-- Keys already in the database (excluded)
existing AS (
SELECT DISTINCT r.constraint_key
FROM dataflow.records r
INNER JOIN pending p ON p.constraint_key = r.constraint_key
WHERE r.source_name = p_source_name
),
-- Rows whose constraint key is not yet in the database
new_records AS (
SELECT p.data, p.constraint_key, p.seq
FROM pending p
WHERE NOT EXISTS (SELECT 1 FROM existing e WHERE e.constraint_key = p.constraint_key)
),
-- Write the log entry
log_entry AS (
INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate, info)
VALUES (
p_source_name,
(SELECT count(*) FROM new_records),
(SELECT count(*) FROM pending) - (SELECT count(*) FROM new_records),
jsonb_build_object(
'total', jsonb_array_length(p_data),
'inserted_keys', (SELECT jsonb_agg(constraint_key ORDER BY constraint_key) FROM new_records),
'excluded_keys', (SELECT jsonb_agg(constraint_key) FROM existing)
)
)
RETURNING id, records_imported, records_duplicate
),
-- Insert new records
inserted AS (
INSERT INTO dataflow.records (source_name, data, constraint_key, import_id)
SELECT p_source_name, nr.data, nr.constraint_key, (SELECT id FROM log_entry)
FROM new_records nr
ORDER BY nr.seq
RETURNING id
)
SELECT le.id, le.records_imported, le.records_duplicate
INTO v_log_id, v_inserted, v_duplicates
FROM log_entry le;
RETURN json_build_object(
'success', true,
'imported', v_inserted,
'duplicates', v_duplicates,
'log_id', v_log_id
);
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION import_records IS 'Import records with automatic deduplication';
-- ── Audit trail ───────────────────────────────────────────────────────────────
CREATE OR REPLACE FUNCTION get_import_log(p_source_name TEXT)
RETURNS TABLE (
id INTEGER,
source_name TEXT,
records_imported INTEGER,
records_duplicate INTEGER,
imported_at TIMESTAMPTZ,
info JSONB
) AS $$
SELECT id, source_name, records_imported, records_duplicate, imported_at, info
FROM dataflow.import_log
WHERE source_name = p_source_name
ORDER BY imported_at DESC;
$$ LANGUAGE sql;
COMMENT ON FUNCTION get_import_log IS 'Return import history for a source, newest first, including inserted/excluded key lists';
CREATE OR REPLACE FUNCTION get_all_import_logs()
RETURNS TABLE (
id INTEGER,
source_name TEXT,
records_imported INTEGER,
records_duplicate INTEGER,
imported_at TIMESTAMPTZ,
info JSONB
) AS $$
SELECT id, source_name, records_imported, records_duplicate, imported_at, info
FROM dataflow.import_log
ORDER BY imported_at DESC;
$$ LANGUAGE sql;
COMMENT ON FUNCTION get_all_import_logs IS 'Return import history across all sources, newest first';
-- Records are removed by the import_id FK's ON DELETE CASCADE
CREATE OR REPLACE FUNCTION delete_import(p_log_id INTEGER)
RETURNS JSON AS $$
DECLARE
v_deleted INTEGER;
BEGIN
IF NOT EXISTS (SELECT 1 FROM dataflow.import_log WHERE id = p_log_id) THEN
RETURN json_build_object('success', false, 'error', 'Import log entry not found');
END IF;
SELECT count(*) INTO v_deleted FROM dataflow.records WHERE import_id = p_log_id;
-- Cascade handles deleting records via FK ON DELETE CASCADE
DELETE FROM dataflow.import_log WHERE id = p_log_id;
RETURN json_build_object(
'success', true,
'records_deleted', v_deleted,
'log_id', p_log_id
);
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION delete_import IS 'Delete all records belonging to an import batch and remove the log entry';

22
database/migrate_input_value_jsonb.sql Normal file
View File

@ -0,0 +1,22 @@
--
-- Migration: Change mappings.input_value from TEXT to JSONB
-- Allows multi-capture-group regex results to be used as mapping keys
--
SET search_path TO dataflow, public;
-- Drop dependent constraint and index first
ALTER TABLE dataflow.mappings DROP CONSTRAINT mappings_source_name_rule_name_input_value_key;
DROP INDEX IF EXISTS dataflow.idx_mappings_input;
-- Convert column: existing TEXT values become JSONB strings e.g. "MEIJER"
ALTER TABLE dataflow.mappings
ALTER COLUMN input_value TYPE JSONB
USING to_jsonb(input_value);
-- Recreate constraint and index
ALTER TABLE dataflow.mappings
ADD CONSTRAINT mappings_source_name_rule_name_input_value_key
UNIQUE (source_name, rule_name, input_value);
CREATE INDEX idx_mappings_input ON dataflow.mappings(source_name, rule_name, input_value);

121
database/migrate_tps.sql Normal file
View File

@ -0,0 +1,121 @@
--
-- TPS → Dataflow Migration
--
-- Migrates sources, rules, mappings, and records from the TPS system.
-- Run against the dataflow database:
-- PGPASSWORD=dataflow psql -U dataflow -d dataflow -h localhost -f database/migrate_tps.sql
--
-- Existing rows are skipped (ON CONFLICT DO NOTHING) so the script is safe to re-run.
-- NOTE: dcard already configured in dataflow will NOT be overwritten.
--
SET search_path TO dataflow, public;
CREATE EXTENSION IF NOT EXISTS dblink;
-- Connection string to the TPS database
\set tps_conn 'host=192.168.1.110 dbname=ubm user=api password=gyaswddh1983'
\echo ''
\echo '=== 1. Sources ==='
INSERT INTO dataflow.sources (name, constraint_fields, config)
SELECT
srce AS name,
-- Strip {} wrappers from constraint paths → constraint field names
ARRAY(
SELECT regexp_replace(c, '^\{|\}$', '', 'g')
FROM jsonb_array_elements_text(defn->'constraint') AS c
) AS constraint_fields,
-- Build config.fields from the first schema (index 0 = "mapped" for dcard, "default" for others)
jsonb_build_object('fields',
(SELECT jsonb_agg(
jsonb_build_object(
'name', regexp_replace(col->>'path', '^\{|\}$', '', 'g'),
'type', COALESCE(NULLIF(col->>'type', ''), 'text')
) ORDER BY ord
)
FROM jsonb_array_elements(defn->'schemas'->0->'columns')
WITH ORDINALITY AS t(col, ord)
)
) AS config
FROM dblink(:'tps_conn',
'SELECT srce, defn FROM tps.srce'
) AS t(srce TEXT, defn JSONB)
ON CONFLICT (name) DO NOTHING;
SELECT name, constraint_fields, jsonb_array_length(config->'fields') AS field_count
FROM dataflow.sources ORDER BY name;
\echo ''
\echo '=== 2. Rules ==='
INSERT INTO dataflow.rules
(source_name, name, field, pattern, output_field, function_type, flags, replace_value, sequence, enabled, retain)
SELECT
srce AS source_name,
target AS name,
-- Strip {} from the input field key
regexp_replace(regex->'regex'->'defn'->0->>'key', '^\{|\}$', '', 'g') AS field,
regex->'regex'->'defn'->0->>'regex' AS pattern,
regex->'regex'->'defn'->0->>'field' AS output_field,
COALESCE(NULLIF(regex->'regex'->>'function', ''), 'extract') AS function_type,
COALESCE(regex->'regex'->'defn'->0->>'flag', '') AS flags,
'' AS replace_value,
seq AS sequence,
true AS enabled,
(regex->'regex'->'defn'->0->>'retain') = 'y' AS retain
FROM dblink(:'tps_conn',
'SELECT srce, target, seq, regex FROM tps.map_rm'
) AS t(srce TEXT, target TEXT, seq INT, regex JSONB)
ON CONFLICT (source_name, name) DO NOTHING;
SELECT source_name, name, field, pattern, output_field, sequence
FROM dataflow.rules ORDER BY source_name, sequence;
\echo ''
\echo '=== 3. Mappings ==='
INSERT INTO dataflow.mappings (source_name, rule_name, input_value, output)
SELECT
srce AS source_name,
target AS rule_name,
-- retval is {"f20": "<extracted string>"} — pull out the value as JSONB
(SELECT value FROM jsonb_each(retval) LIMIT 1) AS input_value,
map AS output
FROM dblink(:'tps_conn',
'SELECT srce, target, retval, map FROM tps.map_rv'
) AS t(srce TEXT, target TEXT, retval JSONB, map JSONB)
ON CONFLICT (source_name, rule_name, input_value) DO NOTHING;
SELECT source_name, rule_name, COUNT(*) AS mapping_count
FROM dataflow.mappings GROUP BY source_name, rule_name ORDER BY source_name, rule_name;
\echo ''
\echo '=== 4. Records ==='
\echo ' (13 000+ rows — may take a moment)'
INSERT INTO dataflow.records (source_name, data, constraint_key, transformed, imported_at, transformed_at)
SELECT
t.srce AS source_name,
t.rec AS data,
(SELECT jsonb_object_agg(f, t.rec->>f) FROM unnest(s.constraint_fields) AS f) AS constraint_key,
t.allj AS transformed,
CURRENT_TIMESTAMP AS imported_at,
CASE WHEN t.allj IS NOT NULL THEN CURRENT_TIMESTAMP END AS transformed_at
FROM dblink(:'tps_conn',
'SELECT srce, rec, allj FROM tps.trans'
) AS t(srce TEXT, rec JSONB, allj JSONB)
JOIN dataflow.sources s ON s.name = t.srce
ON CONFLICT (source_name, constraint_key) DO NOTHING;
SELECT source_name, COUNT(*) AS records, COUNT(transformed) AS transformed
FROM dataflow.records GROUP BY source_name ORDER BY source_name;
\echo ''
\echo '=== Migration complete ==='
SELECT
(SELECT COUNT(*) FROM dataflow.sources) AS sources,
(SELECT COUNT(*) FROM dataflow.rules) AS rules,
(SELECT COUNT(*) FROM dataflow.mappings) AS mappings,
(SELECT COUNT(*) FROM dataflow.records) AS records;

0
database/mappings.sql → database/queries/mappings.sql
View File

52
database/records.sql → database/queries/records.sql
View File

@ -41,45 +41,23 @@ $$ LANGUAGE sql STABLE;
-- ── Overrides ─────────────────────────────────────────────────────────────────
-- Store manual overrides. Overrides stay in their own column — never merged into
-- transformed — so reprocessing rules cannot clobber a manual edit.
DROP FUNCTION IF EXISTS set_record_overrides(INTEGER, JSONB);
CREATE OR REPLACE FUNCTION set_record_overrides(p_id INTEGER, p_overrides JSONB)
RETURNS JSON AS $$
WITH updated AS (
UPDATE dataflow.records
SET overrides = CASE WHEN p_overrides = '{}'::jsonb THEN NULL ELSE p_overrides END
WHERE id = p_id
RETURNING *
)
SELECT row_to_json(updated) FROM updated;
-- Store manual overrides and immediately merge into transformed
CREATE OR REPLACE FUNCTION set_record_overrides(p_id INT, p_overrides JSONB)
RETURNS dataflow.records AS $$
UPDATE dataflow.records
SET overrides = CASE WHEN p_overrides = '{}'::jsonb THEN NULL ELSE p_overrides END,
transformed = COALESCE(transformed, data) || COALESCE(p_overrides, '{}'::jsonb)
WHERE id = p_id
RETURNING *;
$$ LANGUAGE sql;
-- Merge overrides into multiple records at once; returns actual updated count
DROP FUNCTION IF EXISTS bulk_set_record_overrides(TEXT, INTEGER[], JSONB);
CREATE OR REPLACE FUNCTION bulk_set_record_overrides(p_source_name TEXT, p_ids INTEGER[], p_overrides JSONB)
RETURNS JSON AS $$
WITH updated AS (
UPDATE dataflow.records
SET overrides = COALESCE(overrides, '{}'::jsonb) || p_overrides
WHERE id = ANY(p_ids)
AND source_name = p_source_name
RETURNING id
)
SELECT json_build_object('updated', count(*)) FROM updated;
$$ LANGUAGE sql;
-- Clear overrides; the computed values in transformed are untouched
DROP FUNCTION IF EXISTS clear_record_overrides(INTEGER);
CREATE OR REPLACE FUNCTION clear_record_overrides(p_id INTEGER)
RETURNS JSON AS $$
WITH updated AS (
UPDATE dataflow.records
SET overrides = NULL
WHERE id = p_id
RETURNING *
)
SELECT row_to_json(updated) FROM updated;
-- Clear overrides; caller should reprocess to restore computed transformed value
CREATE OR REPLACE FUNCTION clear_record_overrides(p_id INT)
RETURNS dataflow.records AS $$
UPDATE dataflow.records
SET overrides = NULL
WHERE id = p_id
RETURNING *;
$$ LANGUAGE sql;
-- ── Delete ────────────────────────────────────────────────────────────────────

22
database/rules.sql → database/queries/rules.sql
View File

@ -86,27 +86,21 @@ CREATE OR REPLACE FUNCTION preview_rule(
p_limit INT DEFAULT 20
)
RETURNS TABLE (id INT, raw_value TEXT, extracted_value JSONB) AS $$
-- Field is resolved from data first, then transformed (supports chained rules whose
-- input field was produced by an earlier-sequence rule rather than the raw import).
BEGIN
IF p_function_type = 'replace' THEN
RETURN QUERY
SELECT
r.id,
COALESCE(r.data ->> p_field, r.transformed ->> p_field),
to_jsonb(regexp_replace(
COALESCE(r.data ->> p_field, r.transformed ->> p_field),
p_pattern, p_replace_value, p_flags
))
r.data ->> p_field,
to_jsonb(regexp_replace(r.data ->> p_field, p_pattern, p_replace_value, p_flags))
FROM dataflow.records r
WHERE source_name = p_source
AND (data ? p_field OR transformed ? p_field)
WHERE source_name = p_source AND data ? p_field
ORDER BY r.id DESC LIMIT p_limit;
ELSE
RETURN QUERY
SELECT
r.id,
COALESCE(r.data ->> p_field, r.transformed ->> p_field),
r.data ->> p_field,
CASE
WHEN agg.match_count = 0 THEN NULL
WHEN agg.match_count = 1 THEN agg.matches -> 0
@ -120,14 +114,10 @@ BEGIN
ORDER BY rn
) AS matches,
count(*)::int AS match_count
FROM regexp_matches(
COALESCE(r.data ->> p_field, r.transformed ->> p_field),
p_pattern, p_flags
)
FROM regexp_matches(r.data ->> p_field, p_pattern, p_flags)
WITH ORDINALITY AS m(mt, rn)
) agg
WHERE r.source_name = p_source
AND (r.data ? p_field OR r.transformed ? p_field)
WHERE r.source_name = p_source AND r.data ? p_field
ORDER BY r.id DESC LIMIT p_limit;
END IF;
END;

28
database/sources.sql → database/queries/sources.sql
View File

@ -40,6 +40,8 @@ RETURNS TEXT AS $$
DELETE FROM dataflow.sources WHERE name = p_name RETURNING name;
$$ LANGUAGE sql;
-- ── Import log ────────────────────────────────────────────────────────────────
-- ── Stats ─────────────────────────────────────────────────────────────────────
CREATE OR REPLACE FUNCTION get_source_stats(p_source_name TEXT)
@ -65,16 +67,6 @@ RETURNS TABLE (key TEXT, origins TEXT[]) AS $$
SELECT jsonb_object_keys(data) AS key, 'raw' AS origin
FROM dataflow.records WHERE source_name = p_source_name
UNION ALL
-- transformed/overrides are read straight off the records so that keys with
-- no surviving rule or mapping (e.g. a manual override) still get listed
SELECT jsonb_object_keys(transformed) AS key, 'transformed' AS origin
FROM dataflow.records
WHERE source_name = p_source_name AND transformed IS NOT NULL
UNION ALL
SELECT jsonb_object_keys(overrides) AS key, 'override' AS origin
FROM dataflow.records
WHERE source_name = p_source_name AND overrides IS NOT NULL
UNION ALL
SELECT output_field AS key, 'rule: ' || name AS origin
FROM dataflow.rules WHERE source_name = p_source_name
UNION ALL
@ -169,7 +161,6 @@ BEGIN
RETURN json_build_object('success', false, 'error', 'No schema fields defined for this source');
END IF;
-- Columns read from r, the merged data || transformed || overrides object
FOR v_field IN SELECT * FROM jsonb_array_elements(v_config->'fields') LOOP
IF v_cols != '' THEN v_cols := v_cols || ', '; END IF;
@ -180,27 +171,24 @@ BEGIN
BEGIN
WHILE v_expr ~ '\{[^}]+\}' LOOP
v_ref := substring(v_expr FROM '\{([^}]+)\}');
v_expr := replace(v_expr, '{' || v_ref || '}', format('(r->>%L)::numeric', v_ref));
v_expr := replace(v_expr, '{' || v_ref || '}', format('(transformed->>%L)::numeric', v_ref));
END LOOP;
v_cols := v_cols || format('%s AS %I', v_expr, v_field->>'name');
END;
ELSE
CASE v_field->>'type'
WHEN 'date' THEN v_cols := v_cols || format('(r->>%L)::date AS %I', v_field->>'name', v_field->>'name');
WHEN 'numeric' THEN v_cols := v_cols || format('(r->>%L)::numeric AS %I', v_field->>'name', v_field->>'name');
ELSE v_cols := v_cols || format('r->>%L AS %I', v_field->>'name', v_field->>'name');
WHEN 'date' THEN v_cols := v_cols || format('(transformed->>%L)::date AS %I', v_field->>'name', v_field->>'name');
WHEN 'numeric' THEN v_cols := v_cols || format('(transformed->>%L)::numeric AS %I', v_field->>'name', v_field->>'name');
ELSE v_cols := v_cols || format('transformed->>%L AS %I', v_field->>'name', v_field->>'name');
END CASE;
END IF;
END LOOP;
CREATE SCHEMA IF NOT EXISTS dfv;
v_view := 'dfv.' || quote_ident(p_source_name);
EXECUTE format('DROP VIEW IF EXISTS %s CASCADE', v_view);
EXECUTE format('DROP VIEW IF EXISTS %s', v_view);
v_sql := format(
'CREATE VIEW %s AS SELECT id, _overridden, %s FROM ('
|| 'SELECT id, overrides IS NOT NULL AS _overridden, '
|| 'data || COALESCE(transformed, ''{}''::jsonb) || COALESCE(overrides, ''{}''::jsonb) AS r '
|| 'FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL) rec',
'CREATE VIEW %s AS SELECT id, overrides IS NOT NULL AS _overridden, %s FROM dataflow.records WHERE source_name = %L AND transformed IS NOT NULL',
v_view, v_cols, p_source_name
);
EXECUTE v_sql;

0
database/stacks.sql → database/queries/stacks.sql
View File

53
database/status.sql → database/queries/status.sql
View File

@ -10,46 +10,45 @@ ALTER TABLE dataflow.sources ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAM
ALTER TABLE dataflow.stacks ADD COLUMN IF NOT EXISTS view_generated_at TIMESTAMPTZ;
------------------------------------------------------
-- Trigger: clear source view_generated_at when config (field definitions) changes
-- Rules and mappings affect transformed data, not view structure — no trigger needed there
-- Trigger: clear source view_generated_at when rules change
------------------------------------------------------
DROP TRIGGER IF EXISTS trg_rules_changed ON dataflow.rules;
DROP TRIGGER IF EXISTS trg_mappings_changed ON dataflow.mappings;
DROP FUNCTION IF EXISTS dataflow.rules_changed();
DROP FUNCTION IF EXISTS dataflow.mappings_changed();
CREATE OR REPLACE FUNCTION dataflow.source_config_changed()
CREATE OR REPLACE FUNCTION dataflow.rules_changed()
RETURNS TRIGGER AS $$
BEGIN
IF NEW.config IS DISTINCT FROM OLD.config THEN
NEW.view_generated_at := NULL;
END IF;
RETURN NEW;
UPDATE dataflow.sources SET view_generated_at = NULL
WHERE name = COALESCE(NEW.source_name, OLD.source_name);
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_source_config_changed ON dataflow.sources;
CREATE TRIGGER trg_source_config_changed
BEFORE UPDATE ON dataflow.sources
FOR EACH ROW EXECUTE FUNCTION dataflow.source_config_changed();
DROP TRIGGER IF EXISTS trg_rules_changed ON dataflow.rules;
CREATE TRIGGER trg_rules_changed
AFTER INSERT OR UPDATE OR DELETE ON dataflow.rules
FOR EACH ROW EXECUTE FUNCTION dataflow.rules_changed();
------------------------------------------------------
-- Trigger: clear source view_generated_at when mappings change
------------------------------------------------------
CREATE OR REPLACE FUNCTION dataflow.mappings_changed()
RETURNS TRIGGER AS $$
BEGIN
UPDATE dataflow.sources SET view_generated_at = NULL
WHERE name = COALESCE(NEW.source_name, OLD.source_name);
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_mappings_changed ON dataflow.mappings;
CREATE TRIGGER trg_mappings_changed
AFTER INSERT OR UPDATE OR DELETE ON dataflow.mappings
FOR EACH ROW EXECUTE FUNCTION dataflow.mappings_changed();
------------------------------------------------------
-- Trigger: clear stack view_generated_at when sources change
-- On UPDATE, skip if all view-relevant columns are unchanged (upsert no-ops should not mark stale)
------------------------------------------------------
CREATE OR REPLACE FUNCTION dataflow.stack_sources_changed()
RETURNS TRIGGER AS $$
BEGIN
IF TG_OP = 'UPDATE' THEN
IF NEW.field_map IS NOT DISTINCT FROM OLD.field_map AND
NEW.amount_sign IS NOT DISTINCT FROM OLD.amount_sign AND
NEW.balance_offset IS NOT DISTINCT FROM OLD.balance_offset AND
NEW.amount_field IS NOT DISTINCT FROM OLD.amount_field AND
NEW.date_field IS NOT DISTINCT FROM OLD.date_field AND
NEW.seq IS NOT DISTINCT FROM OLD.seq THEN
RETURN NULL;
END IF;
END IF;
UPDATE dataflow.stacks SET view_generated_at = NULL
WHERE name = COALESCE(NEW.stack_name, OLD.stack_name);
RETURN NULL;

15
database/schema.sql
View File

@ -37,27 +37,26 @@ CREATE TABLE records (
-- Data
data JSONB NOT NULL, -- Original imported data
constraint_key JSONB, -- Fields that uniquely identify this record (set on import)
transformed JSONB, -- Rule/mapping output fields only (delta, not raw data)
overrides JSONB, -- Manual user overrides (highest precedence)
transformed JSONB, -- Data after transformations applied
-- Metadata
import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE,
import_id INTEGER REFERENCES import_log(id) ON DELETE CASCADE, -- Which import batch this came from
imported_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
transformed_at TIMESTAMPTZ
transformed_at TIMESTAMPTZ,
);
COMMENT ON TABLE records IS 'Imported records with raw and transformed data';
COMMENT ON COLUMN records.data IS 'Original data as imported — never mutated after import';
COMMENT ON COLUMN records.data IS 'Original data as imported';
COMMENT ON COLUMN records.constraint_key IS 'JSONB object of constraint field values — uniquely identifies this record within its source';
COMMENT ON COLUMN records.transformed IS 'Rule/mapping output fields only (delta); merge as data || transformed || overrides for final values';
COMMENT ON COLUMN records.overrides IS 'Manual user overrides; highest precedence in data || transformed || overrides merge';
COMMENT ON COLUMN records.transformed IS 'Data after applying transformation rules';
-- Indexes
CREATE INDEX idx_records_source ON records(source_name);
CREATE INDEX idx_records_constraint ON records USING gin(constraint_key);
CREATE INDEX idx_records_data ON records USING gin(data);
CREATE INDEX idx_records_transformed ON records USING gin(transformed);
CREATE INDEX idx_records_overrides ON records USING gin(overrides) WHERE overrides IS NOT NULL;
------------------------------------------------------
-- Table: rules

156
database/transform.sql
View File

@ -1,156 +0,0 @@
--
-- Transform queries
-- The rule/mapping engine; SQL for the transform endpoints in api/routes/sources.js,
-- api/routes/rules.js and api/routes/records.js
--
-- Order matters within this file: the aggregate is used by apply_transformations,
-- which in turn is called by reprocess_records.
--
SET search_path TO dataflow, public;
-- ── Merge aggregate ───────────────────────────────────────────────────────────
-- Merge JSONB objects across rows (later rows win on key conflicts)
-- Usage: jsonb_concat_obj(col ORDER BY sequence)
CREATE OR REPLACE FUNCTION dataflow.jsonb_merge(a JSONB, b JSONB)
RETURNS JSONB AS $$
SELECT COALESCE(a, '{}') || COALESCE(b, '{}')
$$ LANGUAGE sql IMMUTABLE;
DROP AGGREGATE IF EXISTS dataflow.jsonb_concat_obj(JSONB);
CREATE AGGREGATE dataflow.jsonb_concat_obj(JSONB) (
sfunc = dataflow.jsonb_merge,
stype = JSONB,
initcond = '{}'
);
-- ── Apply rules and mappings ──────────────────────────────────────────────────
-- Writes only the rule/mapping output into records.transformed. Raw values stay in
-- data and manual edits stay in overrides; readers merge the three layers.
DROP FUNCTION IF EXISTS apply_transformations(TEXT, INTEGER[]);
CREATE OR REPLACE FUNCTION apply_transformations(
p_source_name TEXT,
p_record_ids INTEGER[] DEFAULT NULL, -- NULL = all eligible records
p_overwrite BOOLEAN DEFAULT FALSE -- FALSE = skip already-transformed, TRUE = overwrite all
) RETURNS JSON AS $$
WITH
-- All records to process
qualifying AS (
SELECT id, data
FROM dataflow.records
WHERE source_name = p_source_name
AND (p_overwrite OR transformed IS NULL)
AND (p_record_ids IS NULL OR id = ANY(p_record_ids))
),
-- Mirror TPS rx: fan out one row per regex match, drive from rules → records
rx AS (
SELECT
q.id,
r.name AS rule_name,
r.sequence,
r.output_field,
r.retain,
r.function_type,
COALESCE(mt.rn, rp.rn, 1) AS result_number,
-- extract: build map_val and retain_val per match (mirrors TPS)
CASE WHEN array_length(mt.mt, 1) = 1 THEN to_jsonb(mt.mt[1]) ELSE to_jsonb(mt.mt) END AS match_val,
to_jsonb(rp.rp) AS replace_val
FROM dataflow.rules r
INNER JOIN qualifying q ON q.data ? r.field
LEFT JOIN LATERAL regexp_matches(q.data ->> r.field, r.pattern, r.flags)
WITH ORDINALITY AS mt(mt, rn) ON r.function_type = 'extract'
LEFT JOIN LATERAL regexp_replace(q.data ->> r.field, r.pattern, r.replace_value, r.flags)
WITH ORDINALITY AS rp(rp, rn) ON r.function_type = 'replace'
WHERE r.source_name = p_source_name
AND r.enabled = true
),
-- Aggregate match rows back into one value per (record, rule) — mirrors TPS agg_to_target_items
agg_matches AS (
SELECT
id,
rule_name,
sequence,
output_field,
retain,
function_type,
CASE function_type
WHEN 'replace' THEN jsonb_agg(replace_val) -> 0
ELSE
CASE WHEN max(result_number) = 1
THEN jsonb_agg(match_val ORDER BY result_number) -> 0
ELSE jsonb_agg(match_val ORDER BY result_number)
END
END AS extracted
FROM rx
GROUP BY id, rule_name, sequence, output_field, retain, function_type
),
-- Join with mappings to find mapped output — mirrors TPS link_map
linked AS (
SELECT
a.id,
a.sequence,
a.output_field,
a.retain,
a.extracted,
m.output AS mapped
FROM agg_matches a
LEFT JOIN dataflow.mappings m ON
m.source_name = p_source_name
AND m.rule_name = a.rule_name
AND m.input_value = a.extracted
WHERE a.extracted IS NOT NULL
),
-- Build per-rule output JSONB:
-- mapped → use mapping output; also write output_field if retain = true
-- no map → write extracted value to output_field
rule_output AS (
SELECT
id,
sequence,
CASE
WHEN mapped IS NOT NULL THEN
mapped ||
CASE WHEN retain
THEN jsonb_build_object(output_field, extracted)
ELSE '{}'::jsonb
END
ELSE
jsonb_build_object(output_field, extracted)
END AS output
FROM linked
),
-- Merge all rule outputs per record in sequence order — mirrors TPS agg_to_id
record_additions AS (
SELECT
id,
dataflow.jsonb_concat_obj(output ORDER BY sequence) AS additions
FROM rule_output
GROUP BY id
),
-- Update all qualifying records; records with no rule matches get an empty object
updated AS (
UPDATE dataflow.records rec
SET transformed = COALESCE(ra.additions, '{}'::jsonb),
transformed_at = CURRENT_TIMESTAMP
FROM qualifying q
LEFT JOIN record_additions ra ON ra.id = q.id
WHERE rec.id = q.id
RETURNING rec.id
)
SELECT json_build_object('success', true, 'transformed', count(*))
FROM updated
$$ LANGUAGE sql;
COMMENT ON FUNCTION apply_transformations IS 'Apply transformation rules and mappings to records (set-based CTE)';
-- ── Reprocess ─────────────────────────────────────────────────────────────────
CREATE OR REPLACE FUNCTION reprocess_records(p_source_name TEXT)
RETURNS JSON AS $$
-- Overwrite all records directly — no clear step, mirrors TPS srce_map_overwrite
SELECT dataflow.apply_transformations(p_source_name, NULL, TRUE)
$$ LANGUAGE sql;
COMMENT ON FUNCTION reprocess_records IS 'Reapply all transformations for a source, overwriting existing values';

409
deploy.sh Executable file
View File

@ -0,0 +1,409 @@
#!/bin/bash
#
# Dataflow Deploy Script
# First run: full install (database, schema, UI, nginx, systemd)
# Subsequent runs: update functions, UI, and restart service
#
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
# ── Helpers ───────────────────────────────────────────────────────────────────
BOLD='\033[1m'
DIM='\033[2m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
RED='\033[0;31m'
RESET='\033[0m'
section() { echo ""; echo -e "${BOLD}── $1 ──${RESET}"; }
step() { printf " %-42s" "$1..."; }
ok() { echo -e "${GREEN}✓${RESET}"; }
fail() { echo -e "${RED}✗ $1${RESET}"; exit 1; }
info() { echo -e " ${DIM}$1${RESET}"; }
warn() { echo -e " ${YELLOW}$1${RESET}"; }
confirm() {
# confirm "Section name" — prints section header, asks to proceed
# Returns 0 to proceed, 1 to skip
section "$1"
read -p " Proceed? [Y/n]: " _yn
[[ ! "$_yn" =~ ^[Nn]$ ]]
}
# ── Mode detection ────────────────────────────────────────────────────────────
echo ""
echo " This script covers:"
echo " 1. Database — create user/database, deploy schema + functions"
echo " 2. UI — build from source"
echo " 3. Nginx — reverse proxy config + SSL via certbot"
echo " 4. Service — install and enable systemd unit"
echo " 5. Restart — start or restart the API server"
echo ""
echo " Each step will be confirmed before running."
echo " Press Ctrl+C at any time to abort."
section "Mode"
if [ ! -f .env ]; then
echo " No .env found — first-time install"
MODE=install
else
echo " .env found — update mode"
MODE=update
export $(cat .env | grep -v '^#' | xargs)
fi
# ── Phase 1: Collect all config ───────────────────────────────────────────────
if [ "$MODE" = "install" ]; then
section "PostgreSQL Admin"
info "Needed to create the app user and database"
read -p " Admin username [postgres]: " ADMIN_USER; ADMIN_USER=${ADMIN_USER:-postgres}
read -s -p " Admin password: " ADMIN_PASS; echo ""
section "Application Database"
read -p " Host [localhost]: " DB_HOST; DB_HOST=${DB_HOST:-localhost}
read -p " Port [5432]: " DB_PORT; DB_PORT=${DB_PORT:-5432}
read -p " Database [dataflow]: " DB_NAME; DB_NAME=${DB_NAME:-dataflow}
read -p " App user [dataflow]: " DB_USER; DB_USER=${DB_USER:-dataflow}
read -s -p " App password: " DB_PASSWORD; echo ""
section "API"
read -p " Port [3020]: " API_PORT; API_PORT=${API_PORT:-3020}
read -p " Environment [production]:" NODE_ENV; NODE_ENV=${NODE_ENV:-production}
section "Nginx"
read -p " Set up nginx reverse proxy? [Y/n]: " _yn
if [[ ! "$_yn" =~ ^[Nn]$ ]]; then
read -p " Domain (e.g. dataflow.example.com): " NGINX_DOMAIN
fi
DO_DEPS=y; DO_DB=y; DO_SCHEMA=y; DO_FN=y; DO_BUILD=y; DO_SERVICE=y; DO_RESTART=y
else
section "Database"
echo " Current: ${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
read -p " Change target? [y/N]: " _yn
if [[ "$_yn" =~ ^[Yy]$ ]]; then
read -p " Host [${DB_HOST}]: " _in; DB_HOST=${_in:-$DB_HOST}
read -p " Port [${DB_PORT}]: " _in; DB_PORT=${_in:-$DB_PORT}
read -p " Database [${DB_NAME}]: " _in; DB_NAME=${_in:-$DB_NAME}
read -p " User [${DB_USER}]: " _in; DB_USER=${_in:-$DB_USER}
read -s -p " Password (blank = keep): " _in; echo ""
if [ -n "$_in" ]; then DB_PASSWORD=$_in; fi
CHANGE_DB=y
fi
section "Select steps to run"
read -p " Redeploy SQL functions? [Y/n]: " DO_FN; DO_FN=${DO_FN:-y}
read -p " Rebuild UI? [Y/n]: " DO_BUILD; DO_BUILD=${DO_BUILD:-y}
read -p " Set up / update nginx? [y/N]: " DO_NGINX
read -p " Restart API service? [Y/n]: " DO_RESTART; DO_RESTART=${DO_RESTART:-y}
if [[ "$DO_NGINX" =~ ^[Yy]$ ]]; then
read -p " Nginx domain: " NGINX_DOMAIN
fi
fi
# ── Phase 2: Plan summary ─────────────────────────────────────────────────────
section "Plan"
echo " Mode: $( [ "$MODE" = "install" ] && echo "First-time install" || echo "Update" )"
echo " Database: ${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
echo " API port: ${API_PORT:-3020}"
echo ""
echo " Steps:"
if [ "$MODE" = "install" ]; then
echo " • Install Node.js dependencies"
echo " • Test admin connection and create DB user/database"
echo " • Deploy schema and SQL functions"
[ -n "$NGINX_DOMAIN" ] && echo " • Configure nginx → $NGINX_DOMAIN" \
|| echo " • Nginx — skipped (no domain provided)"
echo " • Build UI"
echo " • Install systemd service"
echo " • Start service"
else
[ "${CHANGE_DB}" = "y" ] && echo " • Update .env with new database target"
[[ ! "$DO_FN" =~ ^[Nn]$ ]] && echo " • Redeploy SQL functions" \
|| echo " • SQL functions — skipped"
[[ ! "$DO_BUILD" =~ ^[Nn]$ ]] && echo " • Rebuild UI" \
|| echo " • UI build — skipped"
[ -n "$NGINX_DOMAIN" ] && echo " • Configure nginx → $NGINX_DOMAIN" \
|| echo " • Nginx — skipped"
[[ ! "$DO_RESTART" =~ ^[Nn]$ ]] && echo " • Restart API service" \
|| echo " • Service restart — skipped"
fi
echo ""
read -p " Continue? [Y/n]: " _yn
[[ "$_yn" =~ ^[Nn]$ ]] && echo " Aborted." && exit 0
# ── Phase 3: Execute ──────────────────────────────────────────────────────────
if [ "$MODE" = "install" ]; then
# Dependencies
if confirm "Dependencies"; then
step "API (npm install)"
npm install --omit=dev -q && ok || fail "npm install failed"
step "UI (npm install)"
cd ui && npm install -q && cd .. && ok || fail "ui npm install failed"
else
info "skipped"
fi
# PostgreSQL
if confirm "PostgreSQL"; then
step "Testing admin connection"
export PGPASSWORD="$ADMIN_PASS"
psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c '\q' 2>/dev/null && ok \
|| fail "Cannot connect as $ADMIN_USER"
step "Creating user '$DB_USER'"
if psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -tAc \
"SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'" 2>/dev/null | grep -q 1; then
echo -e "${DIM}already exists${RESET}"
else
psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres \
-c "CREATE USER $DB_USER WITH PASSWORD '$DB_PASSWORD';" > /dev/null && ok \
|| fail "Could not create user"
fi
step "Creating database '$DB_NAME'"
if psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -lqt \
| cut -d'|' -f1 | grep -qw "$DB_NAME"; then
echo -e "${DIM}already exists${RESET}"
else
psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres \
-c "CREATE DATABASE $DB_NAME OWNER $DB_USER;" > /dev/null && ok \
|| fail "Could not create database"
fi
unset PGPASSWORD
step "Writing .env"
cat > .env << ENVEOF
# Database Configuration
DB_HOST=$DB_HOST
DB_PORT=$DB_PORT
DB_NAME=$DB_NAME
DB_USER=$DB_USER
DB_PASSWORD=$DB_PASSWORD
# API Configuration
API_PORT=$API_PORT
NODE_ENV=$NODE_ENV
ENVEOF
ok
export PGPASSWORD="$DB_PASSWORD"
step "Deploying schema"
psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/schema.sql -q && ok \
|| fail "Schema deploy failed"
step "Deploying functions"
psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/functions.sql -q && ok \
|| fail "Functions deploy failed"
else
info "skipped"
fi
else
# Update: save .env if DB target changed
section ".env"
if [ "${CHANGE_DB}" = "y" ]; then
step "Writing updated .env"
cat > .env << ENVEOF
# Database Configuration
DB_HOST=$DB_HOST
DB_PORT=$DB_PORT
DB_NAME=$DB_NAME
DB_USER=$DB_USER
DB_PASSWORD=$DB_PASSWORD
# API Configuration
API_PORT=${API_PORT:-3020}
NODE_ENV=${NODE_ENV:-production}
ENVEOF
ok
else
info "no changes"
fi
# Test connection
section "Database Connection"
step "Testing"
export PGPASSWORD="$DB_PASSWORD"
psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -c '\q' 2>/dev/null && ok \
|| fail "Cannot connect — check credentials"
# SQL functions
if [[ ! "$DO_FN" =~ ^[Nn]$ ]]; then
if confirm "SQL Functions"; then
step "Deploying functions"
psql -U "$DB_USER" -h "$DB_HOST" -p "$DB_PORT" -d "$DB_NAME" -f database/functions.sql -q && ok \
|| fail "Functions deploy failed"
else
info "skipped"
fi
else
section "SQL Functions"
info "skipped"
fi
fi
# ── UI ────────────────────────────────────────────────────────────────────────
if [ "$MODE" = "install" ] || [[ ! "$DO_BUILD" =~ ^[Nn]$ ]]; then
if confirm "UI Build"; then
step "Building"
cd ui && npm run build > /dev/null 2>&1 && cd .. && ok \
|| fail "UI build failed"
else
info "skipped"
fi
else
section "UI Build"
info "skipped"
fi
# ── Nginx ─────────────────────────────────────────────────────────────────────
if [ -n "$NGINX_DOMAIN" ]; then
if confirm "Nginx ($NGINX_DOMAIN)"; then
CONF_NAME=$(echo "$NGINX_DOMAIN" | cut -d. -f1)
CONF_PATH="/etc/nginx/sites-enabled/$CONF_NAME"
CERT_PATH="/etc/letsencrypt/live/$NGINX_DOMAIN/fullchain.pem"
TMP_CONF=$(mktemp)
if [ -f "$CERT_PATH" ]; then
cat > "$TMP_CONF" << NGINXEOF
server {
listen 80;
listen [::]:80;
server_name $NGINX_DOMAIN;
location / { return 301 https://\$host\$request_uri; }
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name $NGINX_DOMAIN;
ssl_certificate $CERT_PATH;
ssl_certificate_key /etc/letsencrypt/live/$NGINX_DOMAIN/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
keepalive_timeout 70;
sendfile on;
client_max_body_size 80m;
location / {
proxy_pass http://localhost:${API_PORT:-3020};
}
}
NGINXEOF
else
cat > "$TMP_CONF" << NGINXEOF
server {
listen 80;
listen [::]:80;
server_name $NGINX_DOMAIN;
location / {
proxy_pass http://localhost:${API_PORT:-3020};
}
}
NGINXEOF
fi
step "Writing /etc/nginx/sites-enabled/$CONF_NAME"
sudo cp "$TMP_CONF" "$CONF_PATH" && rm "$TMP_CONF" && ok \
|| fail "Could not write nginx config (check sudo)"
step "Testing nginx config"
sudo nginx -t > /dev/null 2>&1 && ok \
|| fail "nginx config invalid — run: sudo nginx -t"
step "Reloading nginx"
sudo systemctl reload nginx && ok \
|| fail "nginx reload failed"
if [ ! -f "$CERT_PATH" ]; then
warn "No SSL cert found for $NGINX_DOMAIN"
read -p " Run certbot now? [Y/n]: " _yn
if [[ ! "$_yn" =~ ^[Nn]$ ]]; then
step "Running certbot"
sudo certbot --nginx -d "$NGINX_DOMAIN" --non-interactive --agree-tos \
--redirect -m "admin@$NGINX_DOMAIN" > /dev/null 2>&1 && ok \
|| fail "certbot failed — run manually: sudo certbot --nginx -d $NGINX_DOMAIN"
fi
fi
else
info "skipped"
fi
else
section "Nginx"
info "skipped"
fi
# ── Systemd service ───────────────────────────────────────────────────────────
SERVICE_FILE="/etc/systemd/system/dataflow.service"
section "Systemd Service"
if [ -f "$SERVICE_FILE" ]; then
info "already installed"
else
read -p " Not installed. Install now? (requires sudo) [y/N]: " _yn
if [[ "$_yn" =~ ^[Yy]$ ]]; then
step "Installing service"
sudo cp "$SCRIPT_DIR/dataflow.service" "$SERVICE_FILE" && ok \
|| fail "Could not install service"
step "Enabling on boot"
sudo systemctl daemon-reload && sudo systemctl enable dataflow > /dev/null 2>&1 && ok \
|| fail "Could not enable service"
else
info "skipped"
fi
fi
# ── API Server ────────────────────────────────────────────────────────────────
if [ "$MODE" = "install" ] || [[ ! "$DO_RESTART" =~ ^[Nn]$ ]]; then
if confirm "API Server"; then
if [ -f "$SERVICE_FILE" ]; then
step "Restarting dataflow service"
sudo systemctl restart dataflow && sleep 1
systemctl is-active --quiet dataflow && ok \
|| fail "Service failed — check: journalctl -u dataflow -n 30"
else
info "systemd service not installed — start manually: node api/server.js"
fi
else
info "skipped"
fi
else
section "API Server"
info "skipped"
fi
# ── Done ──────────────────────────────────────────────────────────────────────
section "Done"
echo " API: http://localhost:${API_PORT:-3020}"
[ -n "$NGINX_DOMAIN" ] && echo " Web: https://$NGINX_DOMAIN"
echo " Logs: journalctl -u dataflow -f"
echo ""

82
docs/perspective.md → docs/perspective-pivot.md
View File

@ -1,79 +1,27 @@
# Perspective
# Perspective Pivot — Technical Reference
Everything about the Perspective pivot in dataflow: which packages and versions are
pinned and why, and a ground-truth reference for the parts of the API the official docs
don't cover.
Version tested: `@perspective-dev` v4.4.0 (client, viewer, viewer-datagrid, viewer-d3fc), loaded from CDN.
Shared rationale across projects lives in the canonical guide at
`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy
pattern, upgrade smoke test). This file records what's specific to dataflow.
This document captures everything learned about controlling Perspective programmatically. The official docs are incomplete for some of these APIs — treat this as a ground-truth supplement.
---
> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
> engine, separate npm scope and release schedule — don't mix the two.
---
## Current state
- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
This is the target loader; pf_app should adopt it.
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
unless view sizes grow well past 100k.
- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
for the org; pf_app should copy it.
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
restore — the reference implementation; keep it.
## The version pair is correct — do NOT "fix" it to 4.4.1
`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
combination that keeps both of dataflow's hard requirements:
- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.
Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
stays. Don't touch it.
**What to actually do:**
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
explicit.)
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
renders, dark/light re-themes, and save→reload→drop-column layout restore.
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
4.5.x suite becomes possible and the pair can collapse to one version.
---
# API Reference
Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at
**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been
re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete
for some of these APIs — treat this as a ground-truth supplement.
## Loading via npm
## Loading from CDN
```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
const [{ default: perspective }] = await Promise.all([
import('https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
import('https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
])
```
The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`.
Stylesheet:
```html
<link rel="stylesheet" crossorigin="anonymous"
href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css" />
```
---

25
docs/ui.md
View File

@ -1,25 +0,0 @@
# Dataflow UI
React + Vite + Tailwind CSS frontend for Dataflow.
## Development
```bash
npm install
npm run dev # dev server on :5173, proxies /api to :3020
```
## Build
```bash
npm run build # outputs to ../public/
```
The Express server serves `../public/` as static files — no separate web server needed in production.
## Key packages
- `react` / `react-router-dom` — SPA routing
- `@perspective-dev/client`, `viewer`, `viewer-datagrid`, `viewer-d3fc` — pivot table (npm, inline WASM builds)
- `tailwindcss` — utility CSS
- `sql-formatter` — SQL display formatting

70
docs/getting-started.md → examples/GETTING_STARTED.md
View File

@ -4,40 +4,32 @@ This guide walks through a complete example using bank transaction data.
## Prerequisites
PostgreSQL running, Node.js 18+, and Python 3.
1. PostgreSQL database running
2. Database created: `CREATE DATABASE dataflow;`
3. `.env` file configured (copy from `.env.example`)
## Step 1: Configure and Deploy
## Step 1: Deploy Database Schema
```bash
cd /opt/dataflow
npm install
python3 manage.py
psql -U postgres -d dataflow -f database/schema.sql
psql -U postgres -d dataflow -f database/functions.sql
```
Choose option 1. It writes `.env`, creates the database and user if they don't exist,
then deploys `database/schema.sql` and the SQL function files in dependency order.
You should see tables created without errors.
## Step 2: Start the API Server
```bash
npm install
npm start
```
The server starts on the port set by `API_PORT` in `.env` (3020 by default).
Every `/api` route requires HTTP Basic auth using the credentials set by `manage.py`
option 9. The examples below omit it for readability — add `-u username:password` to each
curl, or export it once:
```bash
alias dfcurl='curl -u username:password'
```
`GET /health` is the one route that needs no auth.
The server should start on port 3000 (or your configured port).
Test it:
```bash
curl http://localhost:3020/health
curl http://localhost:3000/health
# Should return: {"status":"ok","timestamp":"..."}
```
@ -46,7 +38,7 @@ curl http://localhost:3020/health
A source defines where data comes from and how to deduplicate it.
```bash
curl -X POST http://localhost:3020/api/sources \
curl -X POST http://localhost:3000/api/sources \
-H "Content-Type: application/json" \
-d '{
"name": "bank_transactions",
@ -63,7 +55,7 @@ Rules extract meaningful data using regex patterns.
### Rule 1: Extract merchant name (first part of description)
```bash
curl -X POST http://localhost:3020/api/rules \
curl -X POST http://localhost:3000/api/rules \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -78,7 +70,7 @@ curl -X POST http://localhost:3020/api/rules \
### Rule 2: Extract location (city + state pattern)
```bash
curl -X POST http://localhost:3020/api/rules \
curl -X POST http://localhost:3000/api/rules \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -95,7 +87,7 @@ curl -X POST http://localhost:3020/api/rules \
Import the example CSV file:
```bash
curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
-F "file=@examples/bank_transactions.csv"
```
@ -112,7 +104,7 @@ Response:
## Step 6: View Imported Records
```bash
curl http://localhost:3020/api/records/source/bank_transactions?limit=5
curl http://localhost:3000/api/records/source/bank_transactions?limit=5
```
You'll see the raw imported data. Note that `transformed` is `null` - we haven't applied transformations yet!
@ -120,7 +112,7 @@ You'll see the raw imported data. Note that `transformed` is `null` - we haven't
## Step 7: Apply Transformations
```bash
curl -X POST http://localhost:3020/api/sources/bank_transactions/transform
curl -X POST http://localhost:3000/api/sources/bank_transactions/transform
```
Response:
@ -133,7 +125,7 @@ Response:
Now check the records again:
```bash
curl http://localhost:3020/api/records/source/bank_transactions?limit=2
curl http://localhost:3000/api/records/source/bank_transactions?limit=2
```
You'll see the `transformed` field now contains the original data plus extracted fields like `merchant` and `location`.
@ -141,7 +133,7 @@ You'll see the `transformed` field now contains the original data plus extracted
## Step 8: View Extracted Values That Need Mapping
```bash
curl http://localhost:3020/api/mappings/source/bank_transactions/unmapped
curl http://localhost:3000/api/mappings/source/bank_transactions/unmapped
```
Response shows extracted merchant names that aren't mapped yet:
@ -159,7 +151,7 @@ Response shows extracted merchant names that aren't mapped yet:
Map extracted values to clean, standardized output:
```bash
curl -X POST http://localhost:3020/api/mappings \
curl -X POST http://localhost:3000/api/mappings \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -171,7 +163,7 @@ curl -X POST http://localhost:3020/api/mappings \
}
}'
curl -X POST http://localhost:3020/api/mappings \
curl -X POST http://localhost:3000/api/mappings \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -183,7 +175,7 @@ curl -X POST http://localhost:3020/api/mappings \
}
}'
curl -X POST http://localhost:3020/api/mappings \
curl -X POST http://localhost:3000/api/mappings \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -201,13 +193,13 @@ curl -X POST http://localhost:3020/api/mappings \
Clear and reapply transformations to pick up the new mappings:
```bash
curl -X POST http://localhost:3020/api/sources/bank_transactions/reprocess
curl -X POST http://localhost:3000/api/sources/bank_transactions/reprocess
```
## Step 11: View Final Results
```bash
curl http://localhost:3020/api/records/source/bank_transactions?limit=5
curl http://localhost:3000/api/records/source/bank_transactions?limit=5
```
Now the `transformed` field contains:
@ -242,7 +234,7 @@ Example result:
Try importing the same file again:
```bash
curl -X POST http://localhost:3020/api/sources/bank_transactions/import \
curl -X POST http://localhost:3000/api/sources/bank_transactions/import \
-F "file=@examples/bank_transactions.csv"
```
@ -280,19 +272,19 @@ You've now:
```bash
# View all sources
curl http://localhost:3020/api/sources
curl http://localhost:3000/api/sources
# View source statistics
curl http://localhost:3020/api/sources/bank_transactions/stats
curl http://localhost:3000/api/sources/bank_transactions/stats
# View all rules for a source
curl http://localhost:3020/api/rules/source/bank_transactions
curl http://localhost:3000/api/rules/source/bank_transactions
# View all mappings for a source
curl http://localhost:3020/api/mappings/source/bank_transactions
curl http://localhost:3000/api/mappings/source/bank_transactions
# Search for specific records
curl -X POST http://localhost:3020/api/records/search \
curl -X POST http://localhost:3000/api/records/search \
-H "Content-Type: application/json" \
-d '{
"source_name": "bank_transactions",
@ -309,11 +301,11 @@ curl -X POST http://localhost:3020/api/records/search \
- Check logs for error messages
**Import fails:**
- Verify source exists: `curl http://localhost:3020/api/sources`
- Verify source exists: `curl http://localhost:3000/api/sources`
- Check CSV format matches expectations
- Ensure constraint_fields match CSV column names
**Transformations not working:**
- Check rules exist: `curl http://localhost:3020/api/rules/source/bank_transactions`
- Check rules exist: `curl http://localhost:3000/api/rules/source/bank_transactions`
- Test regex pattern manually
- Check records have the specified field

248
manage.py
View File

@ -18,20 +18,6 @@ SERVICE_FILE = Path('/etc/systemd/system/dataflow.service')
SERVICE_SRC = ROOT / 'dataflow.service'
NGINX_DIR = Path('/etc/nginx/sites-enabled')
# Deployed in order — stacks.sql creates tables that reference sources, and
# transform.sql defines an aggregate its own functions depend on
QUERIES_DIR = ROOT / 'database'
QUERY_FILES = [
QUERIES_DIR / 'sources.sql',
QUERIES_DIR / 'rules.sql',
QUERIES_DIR / 'mappings.sql',
QUERIES_DIR / 'records.sql',
QUERIES_DIR / 'import.sql',
QUERIES_DIR / 'transform.sql',
QUERIES_DIR / 'stacks.sql',
QUERIES_DIR / 'status.sql',
]
# ── Terminal helpers ──────────────────────────────────────────────────────────
BOLD = '\033[1m'
@ -167,31 +153,23 @@ def ui_build_time():
return datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M')
return None
def nginx_conf_path(port):
"""Path of the nginx site proxying to our port, if any."""
def nginx_domain(port):
"""Find nginx site proxying to our port."""
if not NGINX_DIR.exists():
return None
for f in NGINX_DIR.iterdir():
try:
if f':{port}' in f.read_text():
return f
text = f.read_text()
if f':{port}' in text:
for line in text.splitlines():
if 'server_name' in line:
parts = line.split()
if len(parts) >= 2:
return parts[1].rstrip(';')
except Exception:
pass
return None
def nginx_domain(port):
"""server_name of the nginx site proxying to our port."""
conf = nginx_conf_path(port)
if not conf:
return None
for line in conf.read_text().splitlines():
if 'server_name' in line:
parts = line.split()
if len(parts) >= 2:
return parts[1].rstrip(';')
return None
def sudo_run(args, **kwargs):
return subprocess.run(['sudo'] + args, **kwargs)
@ -354,8 +332,13 @@ def action_configure(cfg):
db_location = f'database "{new_cfg["DB_NAME"]}" on {new_cfg["DB_HOST"]}:{new_cfg["DB_PORT"]}'
schema_file = ROOT / 'database' / 'schema.sql'
queries_dir = QUERIES_DIR
query_files = QUERY_FILES
queries_dir = ROOT / 'database' / 'queries'
query_files = [
queries_dir / 'sources.sql',
queries_dir / 'rules.sql',
queries_dir / 'mappings.sql',
queries_dir / 'records.sql',
]
# Offer schema deployment
print()
@ -444,14 +427,19 @@ def action_deploy_schema(cfg):
def action_deploy_functions(cfg):
header('Deploy SQL functions (database/*.sql)')
header('Deploy SQL functions (database/queries/)')
if not cfg:
err(f'{ENV_FILE} not found — run option 1 to configure the database connection first')
return
db_location = f'database "{cfg["DB_NAME"]}" on {cfg["DB_HOST"]}:{cfg["DB_PORT"]}'
queries_dir = QUERIES_DIR
query_files = QUERY_FILES
queries_dir = ROOT / 'database' / 'queries'
query_files = [
queries_dir / 'sources.sql',
queries_dir / 'rules.sql',
queries_dir / 'mappings.sql',
queries_dir / 'records.sql',
]
print(f' Source files: {queries_dir}/')
for f in query_files:
@ -740,116 +728,6 @@ def action_stop_service():
ok('dataflow.service stopped')
def action_uninstall(cfg):
"""Reverse everything this script installs, outside the repo itself."""
header('Uninstall dataflow')
port = cfg.get('API_PORT', '3020') if cfg else '3020'
db_name = cfg.get('DB_NAME', 'dataflow') if cfg else 'dataflow'
db_user = cfg.get('DB_USER', 'dataflow') if cfg else 'dataflow'
conf_path = nginx_conf_path(port)
# Everything that exists right now, in reverse install order
targets = []
if service_installed():
targets.append(f'systemd service {SERVICE_FILE}' +
(' (running)' if service_running() else ''))
if conf_path:
targets.append(f'nginx site {conf_path}')
if cfg and can_connect(cfg):
targets.append(f'database "{db_name}" on {cfg["DB_HOST"]}:{cfg["DB_PORT"]} (ALL DATA)')
targets.append(f'database user {db_user}')
if ENV_FILE.exists():
targets.append(f'config {ENV_FILE}')
if (ROOT / 'public').exists():
targets.append(f'built UI {ROOT / "public"}')
if (ROOT / 'node_modules').exists():
targets.append(f'dependencies {ROOT / "node_modules"}')
if not targets:
info('Nothing installed to remove.')
return cfg
print(' This will permanently remove:')
for t in targets:
print(f' {t}')
print()
info(f'The repository itself ({ROOT}) is left alone — delete it manually if you want it gone.')
print()
if input(" Type 'delete' to confirm: ").strip() != 'delete':
info('Cancelled — no changes made')
return cfg
# ── Service ───────────────────────────────────────────────────────────────
if service_installed():
print()
print(' Removing systemd service...')
sudo_run(['systemctl', 'stop', 'dataflow'])
sudo_run(['systemctl', 'disable', 'dataflow'])
r = sudo_run(['rm', '-f', str(SERVICE_FILE)])
if r.returncode != 0:
err(f'Could not remove {SERVICE_FILE} — check sudo permissions')
else:
sudo_run(['systemctl', 'daemon-reload'])
ok(f'Service stopped, disabled, and {SERVICE_FILE} removed')
# ── nginx ─────────────────────────────────────────────────────────────────
if conf_path:
print()
print(' Removing nginx site...')
r = sudo_run(['rm', '-f', str(conf_path)])
if r.returncode != 0:
err(f'Could not remove {conf_path} — check sudo permissions')
elif sudo_run(['nginx', '-t'], capture_output=True).returncode != 0:
err('nginx config test failed after removal — not reloading; check nginx manually')
else:
sudo_run(['systemctl', 'reload', 'nginx'])
ok(f'{conf_path} removed and nginx reloaded')
# ── Database ──────────────────────────────────────────────────────────────
if cfg and can_connect(cfg):
print()
print(f' Dropping the database requires PostgreSQL admin credentials.')
admin = {
'user': prompt('PostgreSQL admin username', 'postgres'),
'password': prompt('PostgreSQL admin password', secret=True),
'host': cfg['DB_HOST'],
'port': cfg['DB_PORT'],
}
r = psql_admin(admin, 'SELECT 1')
if r.returncode != 0:
err(f'Cannot connect as admin — database and user left in place\n{r.stderr.strip()}')
else:
r = psql_admin(admin, f'DROP DATABASE IF EXISTS {db_name}')
if r.returncode != 0:
err(f'Could not drop database "{db_name}"\n{r.stderr.strip()}')
else:
ok(f'Database "{db_name}" dropped')
r = psql_admin(admin, f'DROP USER IF EXISTS {db_user}')
if r.returncode != 0:
err(f'Could not drop user {db_user}\n{r.stderr.strip()}')
else:
ok(f'User {db_user} dropped')
# ── Generated files ───────────────────────────────────────────────────────
print()
for path, label in [(ENV_FILE, 'config'),
(ROOT / 'public', 'built UI'),
(ROOT / 'node_modules', 'dependencies')]:
if not path.exists():
continue
if path.is_dir():
shutil.rmtree(path, ignore_errors=True)
else:
path.unlink()
ok(f'Removed {label} ({path})')
print()
ok('Uninstall complete')
return None
def action_set_login_credentials(cfg):
header('Set login credentials (LOGIN_USER / LOGIN_PASSWORD_HASH in .env)')
@ -902,72 +780,18 @@ def action_set_login_credentials(cfg):
info('Restart the service for changes to take effect (option 7).')
def action_claim_simplefin(cfg):
header('Claim a SimpleFIN setup token')
print(' A setup token is single-use. Claiming it returns the permanent access')
print(' URL, which is written to .env as SIMPLEFIN_ACCESS_URL.')
print()
if not ENV_FILE.exists():
err(f'{ENV_FILE} does not exist — run the database configuration dialog first')
return
if cfg and cfg.get('SIMPLEFIN_ACCESS_URL'):
warn('SIMPLEFIN_ACCESS_URL is already set — claiming again will replace it')
if not confirm('Replace the existing access URL?', default_yes=False):
info('Cancelled — no changes made')
return
token = prompt('Setup token', secret=True)
if not token:
info('Cancelled — no changes made')
return
print(' Claiming token with SimpleFIN...')
r = subprocess.run(
['node', '-e',
"require('./api/lib/simplefin.js').claimSetupToken(process.argv[1])"
".then(u=>process.stdout.write(u),e=>{process.stderr.write(e.message);process.exit(1)})",
token],
capture_output=True, text=True, cwd=ROOT
)
if r.returncode != 0 or not r.stdout:
err(f'Claim failed — the token may already have been used\n {r.stderr.strip()}')
return
access_url = r.stdout.strip()
# Update .env
env_text = ENV_FILE.read_text()
key = 'SIMPLEFIN_ACCESS_URL'
if f'{key}=' in env_text:
import re
env_text = re.sub(rf'^{key}=.*$', f'{key}={access_url}', env_text, flags=re.MULTILINE)
else:
env_text = env_text.rstrip('\n') + f'\n{key}={access_url}\n'
ENV_FILE.write_text(env_text)
# Show the host only — the URL embeds its own credentials
host = access_url.split('@')[-1] if '@' in access_url else access_url
ok(f'{key} written to {ENV_FILE}')
info(f'Bridge: {host}')
info('Restart the service for changes to take effect (option 7).')
# ── Main menu ─────────────────────────────────────────────────────────────────
MENU = [
('Database configuration and deployment dialog (.env)', action_configure),
('Redeploy "dataflow" schema only (database/schema.sql)', action_deploy_schema),
('Redeploy SQL functions only (database/*.sql)', action_deploy_functions),
('Redeploy SQL functions only (database/queries/)', action_deploy_functions),
('Build UI (ui/ → public/)', action_build_ui),
('Set up nginx reverse proxy', action_setup_nginx),
('Install dataflow systemd service unit', action_install_service),
('Start / restart dataflow.service', action_restart_service),
('Stop dataflow.service', action_stop_service),
('Set login credentials', action_set_login_credentials),
('Claim SimpleFIN setup token (.env)', action_claim_simplefin),
('Uninstall (service, nginx, database, .env, build)', action_uninstall),
]
def main():
@ -980,11 +804,14 @@ def main():
show_status(cfg)
db_target = f'into "{cfg["DB_NAME"]}" on {cfg["DB_HOST"]}' if cfg else '(not configured)'
DB_ACTIONS = {action_deploy_schema, action_deploy_functions}
DB_ACTIONS = {
'Deploy "dataflow" schema (database/schema.sql)',
'Deploy SQL functions (database/functions.sql)',
}
print(bold('Actions'))
for i, (label, fn) in enumerate(MENU, 1):
suffix = f' {dim(db_target)}' if fn in DB_ACTIONS else ''
for i, (label, _) in enumerate(MENU, 1):
suffix = f' {dim(db_target)}' if label in DB_ACTIONS else ''
print(f' {cyan(str(i))}. {label}{suffix}')
print(f' {cyan("q")}. Quit')
print()
@ -1000,12 +827,13 @@ def main():
if 0 <= idx < len(MENU):
label, fn = MENU[idx]
import inspect
# cfg is reloaded from .env at the top of every loop, so a return
# value is only ever informational
if len(inspect.signature(fn).parameters) == 0:
fn()
else:
fn(cfg)
sig = inspect.signature(fn)
if len(sig.parameters) == 0:
result = fn()
elif len(sig.parameters) == 1:
result = fn(cfg)
if label.startswith('Configure') and result is not None:
cfg = result
pause()
else:
warn('Invalid choice — enter a number from the list above')

1
migrate/dataflow.pg.sql Normal file
View File

@ -0,0 +1 @@
select id, source, constrain_key, data from dataflow.records

62
migrate/reimport_dcard_from_tps.sh Normal file
View File

@ -0,0 +1,62 @@
#!/bin/bash
# Reimport dcard records from ubm.tps.trans into dataflow.records
#
# Step 1: exports raw rec JSON from ubm
# Step 2: wipes existing dcard data in dataflow and reloads from the export
#
# Usage: bash migrate/reimport_dcard_from_tps.sh
set -e
EXPORT_FILE="/tmp/tps_dcard_rec.csv"
echo "==> Exporting dcard from ubm.tps.trans..."
psql -U ptrowbridge -d ubm -p 54329 -h hptrow.me -c "\COPY (SELECT rec FROM tps.trans WHERE srce = 'dcard' ORDER BY id) TO '${EXPORT_FILE}' CSV"
echo " Exported $(wc -l < ${EXPORT_FILE}) rows"
echo "==> Reimporting into dataflow.records..."
$PG -d dataflow <<SQL
BEGIN;
-- Wipe existing dcard records (FK cascade deletes records too)
DELETE FROM dataflow.import_log WHERE source_name = 'dcard';
-- Staging table for the exported rec JSON
CREATE TEMP TABLE _dcard_import (rec jsonb);
\COPY _dcard_import FROM '${EXPORT_FILE}' CSV
-- New import_log entry
INSERT INTO dataflow.import_log (source_name, records_imported, records_duplicate)
VALUES ('dcard', 0, 0);
-- Insert records; constraint_key matches source constraint_fields:
-- {"Trans. Date","Post Date",Description}
WITH new_import AS (
SELECT id AS import_id FROM dataflow.import_log
WHERE source_name = 'dcard'
ORDER BY id DESC LIMIT 1
),
inserted AS (
INSERT INTO dataflow.records (source_name, data, transformed, constraint_key, import_id)
SELECT
'dcard',
s.rec,
NULL,
jsonb_build_object(
'Trans. Date', s.rec->>'Trans. Date',
'Post Date', s.rec->>'Post Date',
'Description', s.rec->>'Description'
),
i.import_id
FROM _dcard_import s, new_import i
RETURNING id
)
UPDATE dataflow.import_log
SET records_imported = (SELECT COUNT(*) FROM inserted)
WHERE source_name = 'dcard'
AND id = (SELECT id FROM dataflow.import_log WHERE source_name = 'dcard' ORDER BY id DESC LIMIT 1);
COMMIT;
SELECT records_imported FROM dataflow.import_log WHERE source_name = 'dcard' ORDER BY id DESC LIMIT 1;
SQL
echo "==> Done. Run transformations to repopulate the transformed column."

1678
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

14
package.json
View File

@ -4,8 +4,8 @@
"description": "Simple data transformation tool for ingesting, mapping, and transforming data",
"main": "api/server.js",
"scripts": {
"start": "node api/server.js",
"dev": "nodemon api/server.js",
"start": "nodemon api/server.js",
"dev": "node api/server.js",
"test": "echo \"Tests coming soon\" && exit 0"
},
"keywords": [
@ -18,11 +18,11 @@
"license": "MIT",
"dependencies": {
"bcrypt": "^6.0.0",
"csv-parse": "^6.2.1",
"dotenv": "^17.4.2",
"express": "^5.2.1",
"multer": "^2.1.1",
"pg": "^8.21.0"
"csv-parse": "^5.5.2",
"dotenv": "^16.3.1",
"express": "^4.18.2",
"multer": "^1.4.5-lts.1",
"pg": "^8.11.3"
},
"devDependencies": {
"nodemon": "^3.0.1"

38
scripts/setup-service.sh Executable file
View File

@ -0,0 +1,38 @@
#!/bin/bash
set -e
SERVICE_NAME="dataflow"
SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service"
WORKDIR=$(pwd)
echo "Creating systemd service file..."
sudo tee "$SERVICE_FILE" > /dev/null <<EOF
[Unit]
Description=Dataflow API Server
After=postgresql.service
Wants=postgresql.service
[Service]
Type=simple
User=www-data
WorkingDirectory=${WORKDIR}
Environment=NODE_ENV=production
ExecStart=/usr/bin/node api/server.js
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
echo "Reloading systemd daemon..."
sudo systemctl daemon-reload
echo "Enabling dataflow service..."
sudo systemctl enable "$SERVICE_NAME"
echo "Starting dataflow service..."
sudo systemctl start "$SERVICE_NAME"
echo "Done! Service status:"
sudo systemctl status "$SERVICE_NAME" --no-pager

24
ui/.gitignore vendored Normal file
View File

@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

16
ui/README.md Normal file
View File

@ -0,0 +1,16 @@
# React + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
## React Compiler
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
## Expanding the ESLint configuration
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.

2
ui/index.html
View File

@ -4,7 +4,7 @@
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Dataflow</title>
<title>ui</title>
</head>
<body>
<div id="root"></div>

4583
ui/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

26
ui/package.json
View File

@ -10,26 +10,22 @@
"preview": "vite preview"
},
"dependencies": {
"@perspective-dev/client": "^4.5.1",
"@perspective-dev/viewer": "^4.5.1",
"@perspective-dev/viewer-d3fc": "^4.4.1",
"@perspective-dev/viewer-datagrid": "^4.5.1",
"react": "^19.2.7",
"react-dom": "^19.2.7",
"react-router-dom": "^7.17.0",
"sql-formatter": "^15.8.1"
"react": "^19.2.4",
"react-dom": "^19.2.4",
"react-router-dom": "^7.13.2",
"sql-formatter": "^15.7.3"
},
"devDependencies": {
"@eslint/js": "^9.39.4",
"@tailwindcss/vite": "^4.3.1",
"@types/react": "^19.2.17",
"@tailwindcss/vite": "^4.2.2",
"@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^6.0.2",
"@vitejs/plugin-react": "^6.0.1",
"eslint": "^9.39.4",
"eslint-plugin-react-hooks": "^7.1.1",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.5.2",
"globals": "^17.6.0",
"tailwindcss": "^4.3.1",
"vite": "^8.0.16"
"globals": "^17.4.0",
"tailwindcss": "^4.2.2",
"vite": "^8.0.1"
}
}

1
ui/src/App.css Normal file
View File

@ -0,0 +1 @@
/* App-level styles — layout handled by Tailwind */

176
ui/src/App.jsx
View File

@ -1,42 +1,35 @@
import { useState, useEffect, createElement, lazy, Suspense } from 'react'
import { BrowserRouter, Routes, Route, Navigate, useParams } from 'react-router-dom'
import { useState, useEffect } from 'react'
import { BrowserRouter, Routes, Route, NavLink, Navigate } from 'react-router-dom'
import { api, setCredentials, clearCredentials } from './api'
import Sidebar from './components/Sidebar.jsx'
import BottomNav from './components/BottomNav.jsx'
import SourceTabs from './components/SourceTabs.jsx'
import Login from './pages/Login'
import SourceList from './pages/SourceList'
import SourceDetail from './pages/SourceDetail'
import Bridge from './pages/Bridge'
import ImportHub from './pages/ImportHub'
import Sources from './pages/Sources'
import Import from './pages/Import'
import Rules from './pages/Rules'
import Mappings from './pages/Mappings'
import Records from './pages/Records'
import Log from './pages/Log'
const Pivot = lazy(() => import('./pages/Pivot'))
import Pivot from './pages/Pivot'
import Remap from './pages/Remap'
import Stacks from './pages/Stacks'
// Source-scoped pages still take a `source` prop; this reads it off the URL so
// they didn't all need rewriting when selection moved out of the status bar.
function ScopedToSource({ component, ...props }) {
const { name } = useParams()
return createElement(component, { source: name, ...props })
}
// Pivot doubles as the stack viewer; a stack in the URL takes precedence there
function StackPivot() {
const { name } = useParams()
return <Pivot source={name} selectedStack={name} setSelectedStack={() => {}} />
}
const NAV = [
{ to: '/sources', label: 'Sources' },
{ to: '/import', label: 'Import' },
{ to: '/rules', label: 'Rules' },
{ to: '/mappings', label: 'Mappings' },
{ to: '/remap', label: 'Remap' },
{ to: '/records', label: 'Records' },
{ to: '/pivot', label: 'Pivot' },
{ to: '/stacks', label: 'Stacks' },
{ to: '/log', label: 'Log' },
]
export default function App() {
const [authed, setAuthed] = useState(false)
const [loginUser, setLoginUser] = useState('')
const [sources, setSources] = useState([])
const [source, setSource] = useState(() => localStorage.getItem('selectedSource') || '')
const [sidebarExpanded, setSidebarExpanded] = useState(() => localStorage.getItem('df_sidebar') !== 'collapsed')
const [sidebarOpen, setSidebarOpen] = useState(false)
// Sets of names whose dfv view is out of sync with current definitions
const [staleSources, setStaleSources] = useState(new Set())
const [staleStacks, setStaleStacks] = useState(new Set())
@ -45,13 +38,14 @@ export default function App() {
async function handleLogin(user, pass) {
setCredentials(user, pass)
const s = await api.getSources()
sessionStorage.setItem('df_user', user)
sessionStorage.setItem('df_pass', pass)
setSources(s)
if (!source && s.length > 0) setSource(s[0].name)
setAuthed(true)
setLoginUser(user)
await api.getSources().then(s => {
sessionStorage.setItem('df_user', user)
sessionStorage.setItem('df_pass', pass)
setSources(s)
if (!source && s.length > 0) setSource(s[0].name)
setAuthed(true)
setLoginUser(user)
})
}
function handleLogout() {
@ -125,30 +119,86 @@ export default function App() {
if (source) localStorage.setItem('selectedSource', source)
}, [source])
useEffect(() => {
localStorage.setItem('df_sidebar', sidebarExpanded ? 'expanded' : 'collapsed')
}, [sidebarExpanded])
if (!authed) return <Login onLogin={handleLogin} />
const sidebar = (
<div className="flex flex-col h-full">
{/* Header */}
<div className="px-4 py-3 border-b border-gray-200">
<div className="flex items-center justify-between">
<span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
<button onClick={() => setSidebarOpen(false)} className="md:hidden text-gray-400 hover:text-gray-600 leading-none" title="Close">✕</button>
</div>
<div className="flex items-center justify-between mt-1">
<span className="text-xs text-gray-400">{loginUser}</span>
<button onClick={handleLogout} className="text-xs text-gray-400 hover:text-red-500" title="Sign out">Sign out</button>
</div>
</div>
{/* Source selector */}
<div className="px-3 py-3 border-b border-gray-200">
<div className="flex items-center justify-between mb-1">
<label className="text-xs text-gray-500">Source</label>
<NavLink to="/sources?new=1" className="text-xs text-blue-400 hover:text-blue-600 leading-none" title="New source" onClick={() => setSidebarOpen(false)}>+</NavLink>
</div>
<select
className="w-full text-sm border border-gray-200 rounded px-2 py-1 bg-white focus:outline-none focus:border-blue-400"
value={source}
onChange={e => setSource(e.target.value)}
>
{sources.length === 0 && <option value="">—</option>}
{sources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
</select>
</div>
{/* Nav */}
<nav className="flex-1 py-2">
{NAV.map(({ to, label }) => (
<NavLink
key={to}
to={to}
onClick={() => setSidebarOpen(false)}
className={({ isActive }) =>
`block px-4 py-2 text-sm ${isActive
? 'bg-blue-50 text-blue-700 font-medium'
: 'text-gray-600 hover:bg-gray-50'}`
}
>
{label}
</NavLink>
))}
</nav>
</div>
)
return (
<BrowserRouter>
<div className="flex h-screen">
<div className="flex h-screen bg-gray-50">
<div className="hidden md:flex">
<Sidebar
expanded={sidebarExpanded}
setExpanded={setSidebarExpanded}
loginUser={loginUser}
onLogout={handleLogout}
sources={sources}
/>
{/* Mobile overlay */}
{sidebarOpen && (
<div className="fixed inset-0 z-20 bg-black/30 md:hidden" onClick={() => setSidebarOpen(false)} />
)}
{/* Sidebar — fixed on mobile, static on desktop */}
<div className={`
fixed inset-y-0 left-0 z-30 w-44 bg-white border-r border-gray-200 transform transition-transform duration-200
md:static md:translate-x-0 md:z-auto md:transition-none
${sidebarOpen ? 'translate-x-0' : '-translate-x-full'}
`}>
{sidebar}
</div>
{/* Main */}
<div className="flex-1 overflow-hidden flex flex-col min-w-0">
<div className="flex-1 overflow-auto flex flex-col min-w-0">
{/* Mobile top bar */}
<div className="md:hidden flex items-center px-3 py-2 bg-white border-b border-gray-200">
<button onClick={() => setSidebarOpen(true)} className="text-gray-500 hover:text-gray-700 mr-3 text-lg leading-none">☰</button>
<span className="text-sm font-semibold text-gray-800 tracking-wide uppercase">Dataflow</span>
</div>
{(staleSources.size > 0 || staleStacks.size > 0) && (
<div className="bg-warn-soft border-b border-warn-line px-4 py-1.5 text-xs text-warn flex flex-wrap items-center gap-x-3 gap-y-1">
<div className="bg-amber-50 border-b border-amber-200 px-4 py-1.5 text-xs text-amber-800 flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="font-medium">View out of sync:</span>
{[...staleSources].map(name => (
<span key={name} className="flex items-center gap-1">
@ -156,20 +206,20 @@ export default function App() {
<button
onClick={() => handleGenerateSource(name)}
disabled={generating[`src:${name}`]}
className="px-1.5 py-0.5 rounded bg-warn-line hover:bg-warn-line disabled:opacity-50 font-medium"
className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
>
{generating[`src:${name}`] ? '…' : 'Generate'}
</button>
</span>
))}
{staleSources.size > 0 && staleStacks.size > 0 && <span className="text-warn">|</span>}
{staleSources.size > 0 && staleStacks.size > 0 && <span className="text-amber-400">|</span>}
{[...staleStacks].map(name => (
<span key={name} className="flex items-center gap-1">
stack: {name}
<button
onClick={() => handleGenerateStack(name)}
disabled={generating[`stk:${name}`]}
className="px-1.5 py-0.5 rounded bg-warn-line hover:bg-warn-line disabled:opacity-50 font-medium"
className="px-1.5 py-0.5 rounded bg-amber-200 hover:bg-amber-300 disabled:opacity-50 font-medium"
>
{generating[`stk:${name}`] ? '…' : 'Generate'}
</button>
@ -178,7 +228,7 @@ export default function App() {
</div>
)}
{reprocessSources.size > 0 && (
<div className="bg-accent-soft border-b border-accent-line px-4 py-1.5 text-xs text-accent flex flex-wrap items-center gap-x-3 gap-y-1">
<div className="bg-blue-50 border-b border-blue-200 px-4 py-1.5 text-xs text-blue-800 flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="font-medium">Mappings updated:</span>
{[...reprocessSources].map(name => (
<span key={name} className="flex items-center gap-1">
@ -195,34 +245,22 @@ export default function App() {
</div>
)}
<div className="flex-1 overflow-auto pb-14 md:pb-0">
<Suspense fallback={<div className="p-6 text-sm text-muted">Loading…</div>}>
<div className="flex-1 overflow-auto">
<Routes>
<Route path="/" element={<Navigate to="/sources" replace />} />
<Route path="/sources" element={<SourceList sources={sources} setSources={setSources} setSource={setSource} />} />
<Route path="/sources/:name" element={<SourceTabs sources={sources} />}>
<Route index element={<Navigate to="records" replace />} />
<Route path="setup" element={<SourceDetail sources={sources} setSources={setSources} />} />
<Route path="import" element={<ScopedToSource component={Import} />} />
<Route path="rules" element={<ScopedToSource component={Rules} onStale={markSourceStale} />} />
<Route path="mappings" element={<ScopedToSource component={Mappings} onNeedsReprocess={markNeedsReprocess} />} />
<Route path="records" element={<ScopedToSource component={Records} />} />
<Route path="pivot" element={<ScopedToSource component={Pivot} />} />
</Route>
<Route path="/import" element={<ImportHub sources={sources} />} />
<Route path="/bridge" element={<Bridge sources={sources} />} />
<Route path="/stacks" element={<Stacks sources={sources} onStackStale={markStackStale} onStackViewGenerated={clearStackStale} />} />
<Route path="/stacks/:name/pivot" element={<StackPivot />} />
<Route path="/sources" element={<Sources source={source} sources={sources} setSources={setSources} setSource={setSource} />} />
<Route path="/import" element={<Import source={source} />} />
<Route path="/rules" element={<Rules source={source} onStale={markSourceStale} />} />
<Route path="/mappings" element={<Mappings source={source} onNeedsReprocess={markNeedsReprocess} />} />
<Route path="/remap" element={<Remap />} />
<Route path="/records" element={<Records source={source} />} />
<Route path="/pivot" element={<Pivot source={source} />} />
<Route path="/stacks" element={<Stacks sources={sources} onStackStale={markStackStale} onStackViewGenerated={clearStackStale} />} />
<Route path="/log" element={<Log />} />
</Routes>
</Suspense>
</div>
</div>
</div>
<BottomNav />
</BrowserRouter>
)
}

19
ui/src/api.js
View File

@ -66,18 +66,6 @@ export const api = {
fd.append('file', file)
return request('POST', `/sources/${name}/import`, fd, true)
},
syncSimpleFin: (name, opts = {}) => {
const params = new URLSearchParams(opts)
return request('POST', `/sources/${name}/sync${params.toString() ? `?${params}` : ''}`)
},
getSimpleFinSample: (accountId, days) => {
const params = new URLSearchParams({ account_id: accountId })
if (days !== undefined) params.set('days', days)
return request('GET', `/sources/simplefin-sample?${params}`)
},
getSimpleFinAccounts: (accessUrlEnv) =>
request('GET', `/sources/simplefin-accounts${accessUrlEnv ? `?access_url_env=${encodeURIComponent(accessUrlEnv)}` : ''}`),
claimSimpleFinToken: (setup_token) => request('POST', '/sources/simplefin-claim', { setup_token }),
transform: (name) => request('POST', `/sources/${name}/transform`),
reprocess: (name) => request('POST', `/sources/${name}/reprocess`),
generateView: (name) => request('POST', `/sources/${name}/view`),
@ -120,16 +108,11 @@ export const api = {
getMappingsByOutputField: (col, val) => request('GET', `/mappings/outputs/${encodeURIComponent(col)}/${encodeURIComponent(val)}`),
remapOutputField: (col, from_val, to_val) => request('POST', '/mappings/remap-field', { col, from_val, to_val }),
// Pivot layouts (sources)
// Pivot layouts
getPivotLayouts: (source) => request('GET', `/sources/${source}/layouts`),
savePivotLayout: (source, layout_name, config) => request('POST', `/sources/${source}/layouts`, { layout_name, config }),
deletePivotLayout: (source, id) => request('DELETE', `/sources/${source}/layouts/${id}`),
// Pivot layouts (stacks)
getStackPivotLayouts: (name) => request('GET', `/stacks/${name}/layouts`),
saveStackPivotLayout: (name, layout_name, config) => request('POST', `/stacks/${name}/layouts`, { layout_name, config }),
deleteStackPivotLayout: (name, id) => request('DELETE', `/stacks/${name}/layouts/${id}`),
// Stacks
getStacks: () => request('GET', '/stacks'),
getStack: (name) => request('GET', `/stacks/${name}`),

BIN
ui/src/assets/hero.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

1
ui/src/assets/react.svg Normal file
View File

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="35.93" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 228"><path fill="#00D8FF" d="M210.483 73.824a171.49 171.49 0 0 0-8.24-2.597c.465-1.9.893-3.777 1.273-5.621c6.238-30.281 2.16-54.676-11.769-62.708c-13.355-7.7-35.196.329-57.254 19.526a171.23 171.23 0 0 0-6.375 5.848a155.866 155.866 0 0 0-4.241-3.917C100.759 3.829 77.587-4.822 63.673 3.233C50.33 10.957 46.379 33.89 51.995 62.588a170.974 170.974 0 0 0 1.892 8.48c-3.28.932-6.445 1.924-9.474 2.98C17.309 83.498 0 98.307 0 113.668c0 15.865 18.582 31.778 46.812 41.427a145.52 145.52 0 0 0 6.921 2.165a167.467 167.467 0 0 0-2.01 9.138c-5.354 28.2-1.173 50.591 12.134 58.266c13.744 7.926 36.812-.22 59.273-19.855a145.567 145.567 0 0 0 5.342-4.923a168.064 168.064 0 0 0 6.92 6.314c21.758 18.722 43.246 26.282 56.54 18.586c13.731-7.949 18.194-32.003 12.4-61.268a145.016 145.016 0 0 0-1.535-6.842c1.62-.48 3.21-.974 4.76-1.488c29.348-9.723 48.443-25.443 48.443-41.52c0-15.417-17.868-30.326-45.517-39.844Zm-6.365 70.984c-1.4.463-2.836.91-4.3 1.345c-3.24-10.257-7.612-21.163-12.963-32.432c5.106-11 9.31-21.767 12.459-31.957c2.619.758 5.16 1.557 7.61 2.4c23.69 8.156 38.14 20.213 38.14 29.504c0 9.896-15.606 22.743-40.946 31.14Zm-10.514 20.834c2.562 12.94 2.927 24.64 1.23 33.787c-1.524 8.219-4.59 13.698-8.382 15.893c-8.067 4.67-25.32-1.4-43.927-17.412a156.726 156.726 0 0 1-6.437-5.87c7.214-7.889 14.423-17.06 21.459-27.246c12.376-1.098 24.068-2.894 34.671-5.345a134.17 134.17 0 0 1 1.386 6.193ZM87.276 214.515c-7.882 2.783-14.16 2.863-17.955.675c-8.075-4.657-11.432-22.636-6.853-46.752a156.923 156.923 0 0 1 1.869-8.499c10.486 2.32 22.093 3.988 34.498 4.994c7.084 9.967 14.501 19.128 21.976 27.15a134.668 134.668 0 0 1-4.877 4.492c-9.933 8.682-19.886 14.842-28.658 17.94ZM50.35 144.747c-12.483-4.267-22.792-9.812-29.858-15.863c-6.35-5.437-9.555-10.836-9.555-15.216c0-9.322 13.897-21.212 37.076-29.293c2.813-.98 5.757-1.905 8.812-2.773c3.204 10.42 7.406 21.315 12.477 32.332c-5.137 11.18-9.399 22.249-12.634 32.792a134.718 134.718 0 0 1-6.318-1.979Zm12.378-84.26c-4.811-24.587-1.616-43.134 6.425-47.789c8.564-4.958 27.502 2.111 47.463 19.835a144.318 144.318 0 0 1 3.841 3.545c-7.438 7.987-14.787 17.08-21.808 26.988c-12.04 1.116-23.565 2.908-34.161 5.309a160.342 160.342 0 0 1-1.76-7.887Zm110.427 27.268a347.8 347.8 0 0 0-7.785-12.803c8.168 1.033 15.994 2.404 23.343 4.08c-2.206 7.072-4.956 14.465-8.193 22.045a381.151 381.151 0 0 0-7.365-13.322Zm-45.032-43.861c5.044 5.465 10.096 11.566 15.065 18.186a322.04 322.04 0 0 0-30.257-.006c4.974-6.559 10.069-12.652 15.192-18.18ZM82.802 87.83a323.167 323.167 0 0 0-7.227 13.238c-3.184-7.553-5.909-14.98-8.134-22.152c7.304-1.634 15.093-2.97 23.209-3.984a321.524 321.524 0 0 0-7.848 12.897Zm8.081 65.352c-8.385-.936-16.291-2.203-23.593-3.793c2.26-7.3 5.045-14.885 8.298-22.6a321.187 321.187 0 0 0 7.257 13.246c2.594 4.48 5.28 8.868 8.038 13.147Zm37.542 31.03c-5.184-5.592-10.354-11.779-15.403-18.433c4.902.192 9.899.29 14.978.29c5.218 0 10.376-.117 15.453-.343c-4.985 6.774-10.018 12.97-15.028 18.486Zm52.198-57.817c3.422 7.8 6.306 15.345 8.596 22.52c-7.422 1.694-15.436 3.058-23.88 4.071a382.417 382.417 0 0 0 7.859-13.026a347.403 347.403 0 0 0 7.425-13.565Zm-16.898 8.101a358.557 358.557 0 0 1-12.281 19.815a329.4 329.4 0 0 1-23.444.823c-7.967 0-15.716-.248-23.178-.732a310.202 310.202 0 0 1-12.513-19.846h.001a307.41 307.41 0 0 1-10.923-20.627a310.278 310.278 0 0 1 10.89-20.637l-.001.001a307.318 307.318 0 0 1 12.413-19.761c7.613-.576 15.42-.876 23.31-.876H128c7.926 0 15.743.303 23.354.883a329.357 329.357 0 0 1 12.335 19.695a358.489 358.489 0 0 1 11.036 20.54a329.472 329.472 0 0 1-11 20.722Zm22.56-122.124c8.572 4.944 11.906 24.881 6.52 51.026c-.344 1.668-.73 3.367-1.15 5.09c-10.622-2.452-22.155-4.275-34.23-5.408c-7.034-10.017-14.323-19.124-21.64-27.008a160.789 160.789 0 0 1 5.888-5.4c18.9-16.447 36.564-22.941 44.612-18.3ZM128 90.808c12.625 0 22.86 10.235 22.86 22.86s-10.235 22.86-22.86 22.86s-22.86-10.235-22.86-22.86s10.235-22.86 22.86-22.86Z"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.0 KiB

1
ui/src/assets/vite.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 8.5 KiB

40
ui/src/components/BottomNav.jsx
View File

@ -1,40 +0,0 @@
import { NavLink } from 'react-router-dom'
import useTheme from '../theme.jsx'
import { NAV } from './navItems.jsx'
// Phone-sized replacement for the sidebar: same destinations, thumb-reachable.
// Hidden from md: upward, where the sidebar takes over.
export default function BottomNav() {
const { dark, setDark } = useTheme()
return (
<nav className="md:hidden fixed bottom-0 inset-x-0 z-20 bg-surface border-t border-line flex justify-around items-stretch h-14 pb-[env(safe-area-inset-bottom)]">
{NAV.map(({ to, label, icon }) => (
<NavLink
key={to}
to={to}
className={({ isActive }) =>
`flex-1 flex flex-col items-center justify-center gap-0.5 text-[10px] ${
isActive ? 'text-accent' : 'text-muted'
}`
}
>
<span>{icon}</span>
{label}
</NavLink>
))}
<button
onClick={() => setDark(d => !d)}
className="flex-1 flex flex-col items-center justify-center gap-0.5 text-[10px] text-muted"
title={dark ? 'Light mode' : 'Dark mode'}
>
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
{dark
? <><circle cx="12" cy="12" r="4"/><line x1="12" y1="2" x2="12" y2="5"/><line x1="12" y1="19" x2="12" y2="22"/><line x1="2" y1="12" x2="5" y2="12"/><line x1="19" y1="12" x2="22" y2="12"/></>
: <path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>}
</svg>
Theme
</button>
</nav>
)
}

28
ui/src/components/SampleTable.jsx
View File

@ -1,28 +0,0 @@
// Compact preview of raw record values — used by the source detail page and by
// the create dialog when a CSV or bank feed is sampled
export default function SampleTable({ rows }) {
if (!rows || rows.length === 0) return null
const cols = Object.keys(rows[0])
return (
<div className="overflow-auto border border-line-soft rounded bg-raised max-h-36">
<table className="text-xs w-full">
<thead>
<tr className="text-left text-muted border-b border-line-soft bg-raised sticky top-0">
{cols.map(c => <th key={c} className="px-2 py-1 font-medium whitespace-nowrap">{c}</th>)}
</tr>
</thead>
<tbody>
{rows.map((row, i) => (
<tr key={i} className="border-t border-line-soft">
{cols.map(c => (
<td key={c} className="px-2 py-1 whitespace-nowrap text-ink-soft max-w-32 truncate font-mono">
{row[c] == null ? <span className="text-muted">—</span> : String(row[c])}
</td>
))}
</tr>
))}
</tbody>
</table>
</div>
)
}

13
ui/src/components/Section.jsx
View File

@ -1,13 +0,0 @@
// One titled panel per job, so setup, schema, and destructive actions read as
// separate things rather than one flat form
export default function Section({ title, description, children }) {
return (
<section className="bg-surface border border-line rounded p-4">
<h2 className="text-sm font-semibold text-ink-soft">{title}</h2>
{description
? <p className="text-xs text-muted mt-0.5 mb-3">{description}</p>
: <div className="mb-3" />}
{children}
</section>
)
}

165
ui/src/components/Sidebar.jsx
View File

@ -1,165 +0,0 @@
import { Fragment, useMemo } from 'react'
import { NavLink } from 'react-router-dom'
import useTheme from '../theme.jsx'
import { NAV } from './navItems.jsx'
// Same distinction the Import page makes: a source is either on a bank feed or
// it gets CSVs uploaded to it.
const feedIcon = (
<svg width="12" height="12" viewBox="0 0 16 16" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round">
<path d="M4 10.5a4.5 4.5 0 0 1 4.5 4.5"/>
<path d="M4 6a9 9 0 0 1 9 9"/>
<circle cx="4.2" cy="14.8" r="1.2" fill="currentColor" stroke="none"/>
</svg>
)
const csvIcon = (
<svg width="12" height="12" viewBox="0 0 16 16" fill="none" stroke="currentColor" strokeWidth="1.4" strokeLinecap="round" strokeLinejoin="round">
<path d="M4 1.5h5l3 3v10H4z"/>
<polyline points="9,1.5 9,4.5 12,4.5"/>
</svg>
)
export default function Sidebar({ expanded, setExpanded, loginUser, onLogout, sources = [] }) {
const { dark, setDark } = useTheme()
// Bank feeds first, then CSV sources, alphabetical within each group
const navSources = useMemo(() => (
sources
.map(s => ({ ...s, isFeed: !!s.config?.simplefin?.account_id }))
.sort((a, b) =>
(b.isFeed - a.isFeed) || a.name.localeCompare(b.name, undefined, { sensitivity: 'base' })
)
), [sources])
return (
<div
className="bg-surface border-r border-line flex flex-col shrink-0 overflow-hidden transition-all duration-150"
style={{ width: expanded ? 200 : 48 }}
>
{/* Header */}
<div className="h-12 flex items-center px-3 border-b border-line-soft gap-2 shrink-0">
<button
onClick={() => setExpanded(e => !e)}
className="w-8 h-8 flex items-center justify-center rounded hover:bg-raised text-muted shrink-0"
title="Toggle sidebar"
>
<svg width="16" height="16" viewBox="0 0 16 16" fill="none">
<rect x="1" y="3" width="14" height="1.5" rx="0.75" fill="currentColor"/>
<rect x="1" y="7.25" width="14" height="1.5" rx="0.75" fill="currentColor"/>
<rect x="1" y="11.5" width="14" height="1.5" rx="0.75" fill="currentColor"/>
</svg>
</button>
<span
className="text-xs font-semibold text-ink-soft tracking-wide uppercase whitespace-nowrap transition-opacity duration-100"
style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none' }}
>
Dataflow
</span>
</div>
{/* Nav */}
<nav className="flex flex-col gap-0.5 p-2 flex-1 overflow-y-auto">
{NAV.map(({ to, label, icon }) => (
<Fragment key={to}>
<NavLink
to={to}
end={to === '/sources'}
title={!expanded ? label : undefined}
className={({ isActive }) =>
`flex items-center gap-3 px-2 py-2 rounded w-full transition-colors ${
isActive
? 'bg-accent-soft text-accent'
: 'text-muted hover:bg-raised hover:text-ink'
}`
}
>
<span className="shrink-0">{icon}</span>
<span
className="text-sm whitespace-nowrap transition-opacity duration-100"
style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
>
{label}
</span>
</NavLink>
{/* Jump straight to a source. Collapsed there is no room for names,
so the shortcut list only exists when the sidebar is open. */}
{to === '/sources' && expanded && navSources.map(({ isFeed, ...s }) => {
return (
<NavLink
key={s.name}
to={`/sources/${encodeURIComponent(s.name)}`}
title={`${s.name} — ${isFeed ? 'bank feed' : 'CSV'}`}
className={({ isActive }) =>
`flex items-center gap-2 ml-4 pl-2 pr-2 py-1 rounded border-l border-line-soft transition-colors ${
isActive
? 'bg-accent-soft text-accent'
: 'text-muted hover:bg-raised hover:text-ink'
}`
}
>
<span className="shrink-0 opacity-70">{isFeed ? feedIcon : csvIcon}</span>
<span className="text-xs truncate">{s.name}</span>
</NavLink>
)
})}
</Fragment>
))}
</nav>
{/* Theme */}
<div className="border-t border-line-soft px-3 py-2 shrink-0">
<button
onClick={() => setDark(d => !d)}
title={dark ? 'Switch to light mode' : 'Switch to dark mode'}
className="flex items-center gap-2.5 w-full rounded px-1 py-1 text-muted hover:bg-raised hover:text-ink"
>
<span className="shrink-0">
{dark ? (
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<circle cx="12" cy="12" r="4"/>
<line x1="12" y1="2" x2="12" y2="5"/><line x1="12" y1="19" x2="12" y2="22"/>
<line x1="4.93" y1="4.93" x2="7.05" y2="7.05"/><line x1="16.95" y1="16.95" x2="19.07" y2="19.07"/>
<line x1="2" y1="12" x2="5" y2="12"/><line x1="19" y1="12" x2="22" y2="12"/>
<line x1="4.93" y1="19.07" x2="7.05" y2="16.95"/><line x1="16.95" y1="7.05" x2="19.07" y2="4.93"/>
</svg>
) : (
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
</svg>
)}
</span>
<span
className="text-sm whitespace-nowrap transition-opacity duration-100"
style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
>
{dark ? 'Light mode' : 'Dark mode'}
</span>
</button>
</div>
{/* User / logout */}
<div className="border-t border-line-soft px-3 py-2.5 flex items-center gap-2 shrink-0 overflow-hidden">
<div
className="w-6 h-6 rounded-full bg-raised text-muted flex items-center justify-center shrink-0 text-xs font-medium"
title={!expanded ? loginUser : undefined}
>
{loginUser ? loginUser[0].toUpperCase() : '?'}
</div>
<div
className="flex-1 flex items-center justify-between min-w-0 transition-opacity duration-100"
style={{ opacity: expanded ? 1 : 0, pointerEvents: expanded ? 'auto' : 'none', width: expanded ? 'auto' : 0, overflow: 'hidden' }}
>
<span className="text-xs text-muted truncate">{loginUser}</span>
<button
onClick={onLogout}
className="text-xs text-muted hover:text-danger ml-2 shrink-0"
>
Sign out
</button>
</div>
</div>
</div>
)
}

60
ui/src/components/SourceTabs.jsx
View File

@ -1,60 +0,0 @@
import { NavLink, Outlet, useParams, Link } from 'react-router-dom'
// Everything scoped to one source lives under /sources/:name, so the source is
// in the URL rather than in a global selector.
// Records is the tab you want nine times out of ten, so it leads and is what
// /sources/:name redirects to; Setup is the rare one and sits at the end.
const TABS = [
{ to: 'records', label: 'Records' },
{ to: 'import', label: 'Import' },
{ to: 'rules', label: 'Rules' },
{ to: 'mappings', label: 'Mappings' },
{ to: 'pivot', label: 'Pivot' },
{ to: 'setup', label: 'Setup' },
]
export default function SourceTabs({ sources }) {
const { name } = useParams()
const sourceObj = sources.find(s => s.name === name)
const base = `/sources/${encodeURIComponent(name)}`
return (
<div className="flex flex-col h-full min-h-0">
<div className="px-4 sm:px-6 pt-4 sm:pt-5 shrink-0">
<div className="flex items-center gap-3">
<Link to="/sources" className="text-xs text-muted hover:text-ink-soft">Sources</Link>
<span className="text-muted text-xs">/</span>
<h1 className="text-xl font-semibold text-ink">{name}</h1>
{sourceObj?.config?.simplefin?.account_id && (
<span className="text-xs bg-accent-soft text-accent border border-accent-line rounded px-1.5 py-0.5">
bank feed
</span>
)}
</div>
<nav className="flex gap-1 mt-3 border-b border-line overflow-x-auto whitespace-nowrap">
{TABS.map(({ to, label, end }) => (
<NavLink
key={label}
to={to ? `${base}/${to}` : base}
end={end}
className={({ isActive }) =>
`text-sm px-3 py-1.5 -mb-px border-b-2 ${
isActive
? 'border-accent text-accent font-medium'
: 'border-transparent text-muted hover:text-ink-soft'
}`
}
>
{label}
</NavLink>
))}
</nav>
</div>
<div className="flex-1 overflow-auto min-h-0">
<Outlet />
</div>
</div>
)
}

70
ui/src/components/navItems.jsx
View File

@ -1,70 +0,0 @@
// Top-level destinations, shared by the desktop sidebar and the mobile bar
export const NAV = [
{
to: '/sources',
label: 'Sources',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<ellipse cx="10" cy="5.5" rx="7" ry="2.5"/>
<path d="M3 5.5v9c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5v-9"/>
<path d="M3 10.5c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5"/>
</svg>
),
},
{
to: '/import',
label: 'Import',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<line x1="10" y1="3" x2="10" y2="14"/>
<polyline points="6,10 10,14 14,10"/>
<line x1="3" y1="18" x2="17" y2="18"/>
</svg>
),
},
{
to: '/bridge',
label: 'Bridge',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<path d="M2 13a8 8 0 0 1 16 0"/>
<line x1="2" y1="13" x2="18" y2="13"/>
<line x1="7" y1="13" x2="7" y2="9"/>
<line x1="13" y1="13" x2="13" y2="9"/>
</svg>
),
},
{
to: '/remap',
label: 'Remap',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<polyline points="2,8 2,4 6,4"/>
<path d="M2 4a8 8 0 0 1 14 2"/>
<polyline points="18,12 18,16 14,16"/>
<path d="M18 16a8 8 0 0 1-14-2"/>
</svg>
),
},
{
to: '/stacks',
label: 'Stacks',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<polygon points="10,2 18,6 10,10 2,6"/>
<polyline points="2,10 10,14 18,10"/>
<polyline points="2,14 10,18 18,14"/>
</svg>
),
},
{
to: '/log',
label: 'Log',
icon: (
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
<circle cx="10" cy="10" r="8"/>
<polyline points="10,5 10,10 14,12"/>
</svg>
),
},
]

101
ui/src/index.css
View File

@ -1,107 +1,6 @@
@import "tailwindcss";
/*
Semantic colour tokens.
Components name the *role* of a colour (bg-surface, text-muted, border-line)
rather than a literal shade, so light and dark are two sets of variable values
instead of two sets of rules. Adding a component no longer means adding a
matching `.dark` override.
The raw palette below is the only place actual colours appear.
*/
@theme {
--color-canvas: var(--bg-primary);
--color-surface: var(--bg-secondary);
--color-raised: var(--bg-tertiary);
--color-ink: var(--text-primary);
--color-ink-soft: var(--text-secondary);
--color-muted: var(--text-muted);
--color-line: var(--border-color);
--color-line-soft: var(--border-light);
--color-accent: var(--accent-text);
--color-accent-soft: var(--accent-bg);
--color-accent-line: var(--accent-line);
--color-ok: var(--ok-text);
--color-ok-soft: var(--ok-bg);
--color-warn: var(--warn-text);
--color-warn-soft: var(--warn-bg);
--color-warn-line: var(--warn-line);
--color-danger: var(--danger-text);
--color-danger-soft: var(--danger-bg);
--color-danger-line: var(--danger-line);
}
:root, .light {
--bg-primary: #f3f4f6;
--bg-secondary: #ffffff;
--bg-tertiary: #f9fafb;
--text-primary: #1f2937;
--text-secondary: #374151;
--text-muted: #9ca3af;
--border-color: #e5e7eb;
--border-light: #f3f4f6;
--accent-bg: #eff6ff;
--accent-text: #1d4ed8;
--accent-line: #bfdbfe;
--ok-text: #059669;
--ok-bg: #ecfdf5;
--warn-text: #b45309;
--warn-bg: #fffbeb;
--warn-line: #fde68a;
--danger-text: #ef4444;
--danger-bg: #fef2f2;
--danger-line: #fecaca;
}
/* Dark palette tuned to Perspective's "Pro Dark" theme:
bg #242526, tooltip #2a2c2f, gridline #3b3f46, inactive #61656e,
inactive border #4c505b, active #2770a9, legend #c5c9d0. */
.dark {
--bg-primary: #242526;
--bg-secondary: #2a2c2f;
--bg-tertiary: #3b3f46;
--text-primary: #ffffff;
--text-secondary: #c5c9d0;
--text-muted: #61656e;
--border-color: #4c505b;
--border-light: #3b3f46;
--accent-bg: rgba(39, 113, 170, 0.32);
--accent-text: #4778c2;
--accent-line: #2770a9;
/* Status accents desaturated to sit on Pro Dark's neutral background */
--ok-text: #6ee7b7;
--ok-bg: #1a3d2c;
--warn-text: #f5c66f;
--warn-bg: #3a2e14;
--warn-line: #5a4a26;
--danger-text: #ff9485;
--danger-bg: #3d1f1f;
--danger-line: #6b3030;
}
body {
margin: 0;
font-family: system-ui, -apple-system, sans-serif;
background-color: var(--bg-primary);
color: var(--text-primary);
}
/* Bare border utilities have no colour of their own */
.border, .border-t, .border-b, .border-l, .border-r { border-color: var(--border-color); }
/* Form controls don't inherit the surface token on their own */
input, select, textarea {
background-color: var(--bg-secondary);
color: var(--text-primary);
border-color: var(--border-color);
}
::selection { background-color: var(--accent-bg); color: var(--text-primary); }

5
ui/src/main.jsx
View File

@ -1,13 +1,10 @@
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import { ThemeProvider } from './theme.jsx'
import './index.css'
import App from './App.jsx'
createRoot(document.getElementById('root')).render(
<StrictMode>
<ThemeProvider>
<App />
</ThemeProvider>
<App />
</StrictMode>,
)

176
ui/src/pages/Bridge.jsx
View File

@ -1,176 +0,0 @@
import { useState } from 'react'
import { Link } from 'react-router-dom'
import { api } from '../api'
import Section from '../components/Section.jsx'
// Accounting style: aligned to 2 decimals, negatives in parentheses
function money(value) {
const n = parseFloat(value)
if (!isFinite(n)) return '—'
const abs = Math.abs(n).toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })
return n < 0 ? `(${abs})` : abs
}
// SimpleFIN doesn't report an account type, so retirement accounts are
// recognised from the institution and account names
const RETIREMENT_RE = /401\(?k\)?|403\(?b\)?|\bira\b|retirement|pension|profit sharing/i
const isRetirement = (a) => RETIREMENT_RE.test(`${a.name} ${a.organization || ''}`)
// One SimpleFIN bridge covers every linked bank account, so connection state is
// a bridge-level concern rather than something to hunt for source by source.
export default function Bridge({ sources }) {
const [accounts, setAccounts] = useState(null)
const [errors, setErrors] = useState([])
const [loading, setLoading] = useState(false)
const [error, setError] = useState('')
async function load() {
setLoading(true)
setError('')
try {
const res = await api.getSimpleFinAccounts()
setAccounts(res.accounts || [])
setErrors(res.errors || [])
} catch (err) {
setError(err.message)
} finally {
setLoading(false)
}
}
// Which source, if any, pulls from each account
const sourceFor = (accountId) =>
sources.find(s => s.config?.simplefin?.account_id === accountId)
const sum = (list) => list.reduce((t, a) => t + (parseFloat(a.balance) || 0), 0)
const banking = (accounts || []).filter(a => !isRetirement(a))
const retirement = (accounts || []).filter(isRetirement)
// Banking first, then retirement, so each subtotal sits under its own rows
const ordered = [...banking, ...retirement]
return (
<div className="p-4 sm:p-6 max-w-5xl space-y-4">
<div className="flex items-center justify-between mb-2">
<h1 className="text-xl font-semibold text-ink">Bridge</h1>
<button
onClick={load}
disabled={loading}
className="text-sm border border-line rounded px-3 py-1.5 text-ink-soft hover:bg-raised hover:border-line disabled:opacity-50"
>
{loading ? 'Refreshing…' : 'Refresh'}
</button>
</div>
{error && (
<Section title="Not connected" description="Claim a setup token with manage.py option 10, then restart the service.">
<p className="text-xs text-danger">{error}</p>
</Section>
)}
{errors.length > 0 && (
<div className="bg-warn-soft border border-warn-line rounded p-3 text-xs text-warn space-y-1">
{errors.map((e, i) => <div key={i}>{e}</div>)}
</div>
)}
{!accounts && !loading && !error && (
<p className="text-sm text-muted">Click Refresh to load balances from SimpleFIN.</p>
)}
{accounts && (
<Section
title="SimpleFIN accounts"
description="Every account behind the bridge credential, and which source pulls from it."
>
<table className="w-full text-xs hidden sm:table">
<thead>
<tr className="text-left text-muted border-b border-line-soft">
<th className="pb-1 font-medium">Account</th>
<th className="pb-1 font-medium">Institution</th>
<th className="pb-1 font-medium text-right">Balance</th>
<th className="pb-1 pl-4 font-medium">As of</th>
<th className="pb-1 font-medium">Source</th>
</tr>
</thead>
<tbody>
{ordered.map(a => {
const src = sourceFor(a.id)
return (
<tr key={a.id} className="border-t border-line-soft">
<td className="py-1.5 text-ink-soft">{a.name}</td>
<td className="py-1.5 text-muted">{a.organization}</td>
<td className="py-1.5 text-right font-mono text-ink-soft">{money(a.balance)}</td>
<td className="py-1.5 pl-4 text-muted">{a.balance_date}</td>
<td className="py-1.5">
{src
? <Link to={`/sources/${encodeURIComponent(src.name)}`} className="text-accent hover:text-accent">{src.name}</Link>
: <span className="text-muted">not linked</span>}
</td>
</tr>
)
})}
</tbody>
<tfoot>
{banking.length > 0 && (
<tr className="border-t border-line">
<td className="pt-2 text-muted" colSpan={2}>Banking and cards</td>
<td className="pt-2 text-right font-mono text-ink-soft">{money(sum(banking))}</td>
<td colSpan={2}></td>
</tr>
)}
{retirement.length > 0 && (
<tr>
<td className="pt-1 text-muted" colSpan={2}>Retirement</td>
<td className="pt-1 text-right font-mono text-ink-soft">{money(sum(retirement))}</td>
<td colSpan={2}></td>
</tr>
)}
<tr className="border-t border-line">
<td className="pt-2 text-ink-soft font-medium" colSpan={2}>Total</td>
<td className="pt-2 text-right font-mono font-medium text-ink">{money(sum(accounts))}</td>
<td colSpan={2}></td>
</tr>
</tfoot>
</table>
{/* Stacked cards for narrow screens */}
<div className="sm:hidden divide-y divide-line-soft">
{ordered.map(a => {
const src = sourceFor(a.id)
return (
<div key={a.id} className="py-2">
<div className="flex justify-between gap-2">
<span className="text-xs text-ink-soft">{a.name}</span>
<span className="text-xs font-mono text-ink">{money(a.balance)}</span>
</div>
<div className="flex justify-between gap-2 text-xs text-muted">
<span>{a.organization}</span>
<span>{a.balance_date}</span>
</div>
<div className="text-xs mt-0.5">
{src
? <Link to={`/sources/${encodeURIComponent(src.name)}`} className="text-accent">{src.name}</Link>
: <span className="text-muted">not linked</span>}
</div>
</div>
)
})}
<div className="pt-2 flex justify-between text-xs">
<span className="text-muted">Banking and cards</span>
<span className="font-mono text-ink-soft">{money(sum(banking))}</span>
</div>
<div className="pt-1 flex justify-between text-xs border-0">
<span className="text-muted">Retirement</span>
<span className="font-mono text-ink-soft">{money(sum(retirement))}</span>
</div>
<div className="pt-2 flex justify-between text-xs font-medium">
<span className="text-ink-soft">Total</span>
<span className="font-mono text-ink">{money(sum(accounts))}</span>
</div>
</div>
{accounts.length === 0 && <p className="text-xs text-muted">No accounts returned.</p>}
</Section>
)}
</div>
)
}

119
ui/src/pages/Import.jsx
View File

@ -6,7 +6,7 @@ function KeyList({ keys, label, color }) {
return (
<div className="mb-2">
<div className={`text-xs font-medium mb-1 ${color}`}>{label} ({keys.length})</div>
<div className="max-h-32 overflow-y-auto bg-raised rounded p-2 font-mono text-xs text-muted space-y-0.5">
<div className="max-h-32 overflow-y-auto bg-gray-50 rounded p-2 font-mono text-xs text-gray-500 space-y-0.5">
{keys.map((k, i) => (
<div key={i}>
{typeof k === 'object' && k !== null
@ -28,19 +28,19 @@ function LogRow({ entry, selected, onToggle }) {
return (
<>
<tr className={`border-b border-line-soft ${selected ? 'bg-danger-soft' : ''}`}>
<tr className={`border-b border-gray-50 ${selected ? 'bg-red-50' : ''}`}>
<td className="py-1.5 pr-2">
<input type="checkbox" checked={selected} onChange={onToggle} className="cursor-pointer" />
</td>
<td className="py-1.5 text-xs text-muted font-mono">{entry.id}</td>
<td className="py-1.5 text-muted">{new Date(entry.imported_at).toLocaleString()}</td>
<td className="py-1.5 text-ink">{entry.records_imported}</td>
<td className="py-1.5 text-muted">{entry.records_duplicate}</td>
<td className="py-1.5 text-xs text-gray-400 font-mono">{entry.id}</td>
<td className="py-1.5 text-gray-500">{new Date(entry.imported_at).toLocaleString()}</td>
<td className="py-1.5 text-gray-800">{entry.records_imported}</td>
<td className="py-1.5 text-gray-400">{entry.records_duplicate}</td>
<td className="py-1.5">
{hasKeys && (
<button
onClick={() => setExpanded(e => !e)}
className="text-xs text-accent hover:text-accent"
className="text-xs text-blue-400 hover:text-blue-600"
>
{expanded ? '▲ hide' : '▼ keys'}
</button>
@ -48,10 +48,10 @@ function LogRow({ entry, selected, onToggle }) {
</td>
</tr>
{expanded && (
<tr className={selected ? 'bg-danger-soft' : 'bg-raised'}>
<tr className={selected ? 'bg-red-50' : 'bg-gray-50'}>
<td colSpan={6} className="px-4 py-3">
<KeyList keys={insertedKeys} label="Inserted" color="text-ok" />
<KeyList keys={excludedKeys} label="Excluded" color="text-muted" />
<KeyList keys={insertedKeys} label="Inserted" color="text-green-600" />
<KeyList keys={excludedKeys} label="Excluded" color="text-gray-500" />
</td>
</tr>
)}
@ -67,15 +67,12 @@ export default function Import({ source }) {
const [error, setError] = useState('')
const [dragOver, setDragOver] = useState(false)
const [selected, setSelected] = useState(new Set())
const [simplefin, setSimplefin] = useState(null)
const [days, setDays] = useState('10')
const fileRef = useRef()
useEffect(() => {
if (!source) return
api.getStats(source).then(setStats).catch(() => {})
api.getImportLog(source).then(setLog).catch(() => {})
api.getSource(source).then(s => setSimplefin(s.config?.simplefin || null)).catch(() => setSimplefin(null))
setSelected(new Set())
}, [source])
@ -96,23 +93,6 @@ export default function Import({ source }) {
}
}
async function handleSync() {
if (!source) return
setLoading(true)
setError('')
setResult(null)
try {
const res = await api.syncSimpleFin(source, { days })
setResult(res)
api.getStats(source).then(setStats)
api.getImportLog(source).then(setLog)
} catch (err) {
setError(err.message)
} finally {
setLoading(false)
}
}
async function handleTransform() {
if (!source) return
setLoading(true)
@ -168,11 +148,11 @@ export default function Import({ source }) {
}
}
if (!source) return <div className="p-4 sm:p-6 text-sm text-muted">Select a source first.</div>
if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
return (
<div className="p-4 sm:p-6 max-w-2xl">
<h1 className="text-xl font-semibold text-ink mb-6">Import — {source}</h1>
<div className="p-6 max-w-2xl">
<h1 className="text-xl font-semibold text-gray-800 mb-6">Import — {source}</h1>
{/* Stats */}
{stats && (
@ -182,42 +162,18 @@ export default function Import({ source }) {
{ label: 'Transformed', value: stats.transformed_records },
{ label: 'Pending', value: stats.pending_records },
].map(({ label, value }) => (
<div key={label} className="bg-surface border border-line rounded px-4 py-3 flex-1 text-center">
<div className="text-2xl font-semibold text-ink">{value}</div>
<div className="text-xs text-muted mt-0.5">{label}</div>
<div key={label} className="bg-white border border-gray-200 rounded px-4 py-3 flex-1 text-center">
<div className="text-2xl font-semibold text-gray-800">{value}</div>
<div className="text-xs text-gray-400 mt-0.5">{label}</div>
</div>
))}
</div>
)}
{/* SimpleFIN sync — only for sources with a bridge account in their config */}
{simplefin?.account_id && (
<div className="bg-surface border border-line rounded p-4 mb-4 flex items-center gap-3">
<div className="flex-1 min-w-0">
<div className="text-sm font-medium text-ink-soft">SimpleFIN</div>
<div className="text-xs text-muted font-mono truncate">{simplefin.account_id}</div>
</div>
<select
value={days}
onChange={e => setDays(e.target.value)}
className="text-sm border border-line rounded px-2 py-1.5 bg-surface text-ink-soft"
>
<option value="10">Last 10 days</option>
<option value="30">Last 30 days</option>
<option value="45">Last 45 days</option>
<option value="89">Backfill (bridge maximum)</option>
</select>
<button onClick={handleSync} disabled={loading}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{loading ? 'Syncing…' : 'Sync now'}
</button>
</div>
)}
{/* Drop zone */}
<div
className={`border-2 border-dashed rounded-lg p-8 text-center mb-4 cursor-pointer transition-colors ${
dragOver ? 'border-accent bg-accent-soft' : 'border-line hover:border-line'
dragOver ? 'border-blue-400 bg-blue-50' : 'border-gray-200 hover:border-gray-300'
}`}
onDragOver={e => { e.preventDefault(); setDragOver(true) }}
onDragLeave={() => setDragOver(false)}
@ -232,22 +188,22 @@ export default function Import({ source }) {
onChange={e => handleImport(e.target.files[0])}
/>
{loading
? <p className="text-sm text-muted">Importing…</p>
: <p className="text-sm text-muted">Drop a CSV file here, or click to browse</p>
? <p className="text-sm text-gray-500">Importing…</p>
: <p className="text-sm text-gray-400">Drop a CSV file here, or click to browse</p>
}
</div>
{error && <p className="text-sm text-danger mb-3">{error}</p>}
{error && <p className="text-sm text-red-500 mb-3">{error}</p>}
{result && (
<div className={`border rounded p-4 mb-4 text-sm ${result.success === false ? 'bg-danger-soft border-danger-line' : 'bg-surface border-line'}`}>
<div className={`border rounded p-4 mb-4 text-sm ${result.success === false ? 'bg-red-50 border-red-200' : 'bg-white border-gray-200'}`}>
{result.success === false ? (
<>
<p className="text-danger font-medium mb-2">{result.error}</p>
<p className="text-red-600 font-medium mb-2">{result.error}</p>
{result.duplicate_rows && (
<div>
<p className="text-xs text-danger mb-1">Offending rows:</p>
<div className="max-h-48 overflow-y-auto bg-surface rounded border border-danger-line p-2 font-mono text-xs text-danger space-y-0.5">
<p className="text-xs text-red-500 mb-1">Offending rows:</p>
<div className="max-h-48 overflow-y-auto bg-white rounded border border-red-100 p-2 font-mono text-xs text-red-700 space-y-0.5">
{result.duplicate_rows.map((row, i) => (
<div key={i}>
{Object.entries(row).map(([f, v]) => `${f}: ${v}`).join(' · ')}
@ -259,29 +215,18 @@ export default function Import({ source }) {
</>
) : result.imported !== undefined ? (
<>
{result.errors?.length > 0 && (
<div className="mb-2 text-xs text-warn">
{result.errors.map((e, i) => <div key={i}>Bridge: {e}</div>)}
</div>
)}
{result.fetched !== undefined && (
<>
<span className="text-muted">{result.fetched} fetched</span>
<span className="text-muted mx-2">·</span>
</>
)}
<span className="text-ok font-medium">{result.imported} imported</span>
<span className="text-muted mx-2">·</span>
<span className="text-muted">{result.duplicates} duplicates skipped</span>
<span className="text-green-600 font-medium">{result.imported} imported</span>
<span className="text-gray-400 mx-2">·</span>
<span className="text-gray-500">{result.duplicates} duplicates skipped</span>
{result.transform && (
<>
<span className="text-muted mx-2">·</span>
<span className="text-muted">{result.transform.transformed} transformed</span>
<span className="text-gray-400 mx-2">·</span>
<span className="text-gray-500">{result.transform.transformed} transformed</span>
</>
)}
</>
) : (
<span className="text-ok font-medium">{result.transformed} records transformed</span>
<span className="text-green-600 font-medium">{result.transformed} records transformed</span>
)}
</div>
)}
@ -306,7 +251,7 @@ export default function Import({ source }) {
{log.length > 0 && (
<div>
<div className="flex items-center justify-between mb-2">
<h2 className="text-sm font-semibold text-ink-soft">Import history</h2>
<h2 className="text-sm font-semibold text-gray-700">Import history</h2>
{selected.size > 0 && (
<button
onClick={handleDeleteSelected}
@ -319,7 +264,7 @@ export default function Import({ source }) {
</div>
<table className="w-full text-sm">
<thead>
<tr className="text-left text-xs text-muted border-b border-line-soft">
<tr className="text-left text-xs text-gray-400 border-b border-gray-100">
<th className="pb-1 w-6"></th>
<th className="pb-1 font-medium w-12">ID</th>
<th className="pb-1 font-medium">Date</th>

129
ui/src/pages/ImportHub.jsx
View File

@ -1,129 +0,0 @@
import { useState, useEffect } from 'react'
import { Link } from 'react-router-dom'
import { api } from '../api'
// Importing is the frequent job; configuring a source is the rare one. This is
// the top-level entry point for the frequent one — every source in one place,
// with a sync button for anything on a bank feed.
export default function ImportHub({ sources }) {
const [stats, setStats] = useState({}) // name -> stats
const [lastImport, setLastImport] = useState({}) // name -> ISO timestamp
const [busy, setBusy] = useState('')
const [results, setResults] = useState({}) // name -> message
const [errors, setErrors] = useState({}) // name -> message
useEffect(() => {
let cancelled = false
Promise.all(sources.map(s =>
api.getStats(s.name).then(st => [s.name, st]).catch(() => [s.name, null])
)).then(pairs => {
if (!cancelled) setStats(Object.fromEntries(pairs))
})
api.getAllImportLog().then(log => {
if (cancelled) return
const latest = {}
for (const entry of log) {
if (!latest[entry.source_name] || entry.imported_at > latest[entry.source_name]) {
latest[entry.source_name] = entry.imported_at
}
}
setLastImport(latest)
}).catch(() => {})
return () => { cancelled = true }
}, [sources])
async function sync(name) {
setBusy(name)
setErrors(e => ({ ...e, [name]: '' }))
setResults(r => ({ ...r, [name]: '' }))
try {
const res = await api.syncSimpleFin(name, { days: 10 })
setResults(r => ({
...r,
[name]: `${res.imported} imported, ${res.duplicates} already had` +
(res.errors?.length ? ` — ${res.errors.join('; ')}` : ''),
}))
api.getStats(name).then(st => setStats(s => ({ ...s, [name]: st }))).catch(() => {})
api.getAllImportLog().then(log => {
const entry = log.find(l => l.source_name === name)
if (entry) setLastImport(l => ({ ...l, [name]: entry.imported_at }))
}).catch(() => {})
} catch (err) {
setErrors(e => ({ ...e, [name]: err.message }))
} finally {
setBusy('')
}
}
const feeds = sources.filter(s => s.config?.simplefin?.account_id)
const manual = sources.filter(s => !s.config?.simplefin?.account_id)
function Row({ s, isFeed }) {
const st = stats[s.name]
const when = lastImport[s.name]
return (
<div className="px-4 py-3 flex items-center gap-3 flex-wrap">
<div className="flex-1 min-w-40">
<Link to={`/sources/${encodeURIComponent(s.name)}/import`}
className="text-sm font-medium text-ink hover:text-accent">
{s.name}
</Link>
<div className="text-xs text-muted">
{st ? `${st.total_records} records` : '—'}
{st && Number(st.pending_records) > 0 && ` · ${st.pending_records} untransformed`}
{when && ` · last import ${new Date(when).toLocaleDateString()}`}
</div>
</div>
{results[s.name] && <span className="text-xs text-ok">{results[s.name]}</span>}
{errors[s.name] && <span className="text-xs text-danger">{errors[s.name]}</span>}
{isFeed ? (
<button
onClick={() => sync(s.name)}
disabled={busy === s.name}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50"
>
{busy === s.name ? 'Syncing…' : 'Sync'}
</button>
) : (
<Link to={`/sources/${encodeURIComponent(s.name)}/import`}
className="text-sm border border-line rounded px-3 py-1.5 text-ink-soft hover:bg-raised hover:border-line">
Upload CSV
</Link>
)}
</div>
)
}
return (
<div className="p-4 sm:p-6 max-w-4xl space-y-6">
<h1 className="text-xl font-semibold text-ink">Import</h1>
{feeds.length > 0 && (
<div>
<h2 className="text-sm font-semibold text-ink-soft mb-2">Bank feeds</h2>
<div className="bg-surface border border-line rounded divide-y divide-line-soft">
{feeds.map(s => <Row key={s.name} s={s} isFeed />)}
</div>
<p className="text-xs text-muted mt-1">Syncs pull the last 10 days; use a source&rsquo;s Import tab to backfill further.</p>
</div>
)}
{manual.length > 0 && (
<div>
<h2 className="text-sm font-semibold text-ink-soft mb-2">CSV sources</h2>
<div className="bg-surface border border-line rounded divide-y divide-line-soft">
{manual.map(s => <Row key={s.name} s={s} isFeed={false} />)}
</div>
</div>
)}
{sources.length === 0 && <p className="text-sm text-muted">No sources yet.</p>}
<Link to="/log" className="inline-block text-xs text-accent hover:text-accent">
Full import history →
</Link>
</div>
)
}

30
ui/src/pages/Log.jsx
View File

@ -6,7 +6,7 @@ function KeyList({ keys, label, color }) {
return (
<div className="mb-2">
<div className={`text-xs font-medium mb-1 ${color}`}>{label} ({keys.length})</div>
<div className="max-h-32 overflow-y-auto bg-raised rounded p-2 font-mono text-xs text-muted space-y-0.5">
<div className="max-h-32 overflow-y-auto bg-gray-50 rounded p-2 font-mono text-xs text-gray-500 space-y-0.5">
{keys.map((k, i) => (
<div key={i}>
{typeof k === 'object' && k !== null
@ -28,17 +28,17 @@ function LogRow({ entry }) {
return (
<>
<tr className="border-b border-line-soft hover:bg-raised">
<td className="py-1.5 text-xs text-muted font-mono pr-3">{entry.id}</td>
<td className="py-1.5 text-ink-soft pr-3">{entry.source_name}</td>
<td className="py-1.5 text-muted pr-3">{new Date(entry.imported_at).toLocaleString()}</td>
<td className="py-1.5 text-ink pr-3">{entry.records_imported}</td>
<td className="py-1.5 text-muted pr-3">{entry.records_duplicate}</td>
<tr className="border-b border-gray-50 hover:bg-gray-50">
<td className="py-1.5 text-xs text-gray-400 font-mono pr-3">{entry.id}</td>
<td className="py-1.5 text-gray-700 pr-3">{entry.source_name}</td>
<td className="py-1.5 text-gray-500 pr-3">{new Date(entry.imported_at).toLocaleString()}</td>
<td className="py-1.5 text-gray-800 pr-3">{entry.records_imported}</td>
<td className="py-1.5 text-gray-400 pr-3">{entry.records_duplicate}</td>
<td className="py-1.5">
{hasKeys && (
<button
onClick={() => setExpanded(e => !e)}
className="text-xs text-accent hover:text-accent"
className="text-xs text-blue-400 hover:text-blue-600"
>
{expanded ? '▲ hide' : '▼ keys'}
</button>
@ -46,10 +46,10 @@ function LogRow({ entry }) {
</td>
</tr>
{expanded && (
<tr className="bg-raised">
<tr className="bg-gray-50">
<td colSpan={6} className="px-4 py-3">
<KeyList keys={insertedKeys} label="Inserted" color="text-ok" />
<KeyList keys={excludedKeys} label="Excluded" color="text-muted" />
<KeyList keys={insertedKeys} label="Inserted" color="text-green-600" />
<KeyList keys={excludedKeys} label="Excluded" color="text-gray-500" />
</td>
</tr>
)}
@ -70,18 +70,18 @@ export default function Log() {
return (
<div className="p-6">
<h1 className="text-xl font-semibold text-ink mb-6">Import Log</h1>
<h1 className="text-xl font-semibold text-gray-800 mb-6">Import Log</h1>
{loading && <p className="text-sm text-muted">Loading…</p>}
{loading && <p className="text-sm text-gray-400">Loading…</p>}
{!loading && log.length === 0 && (
<p className="text-sm text-muted">No imports yet.</p>
<p className="text-sm text-gray-400">No imports yet.</p>
)}
{log.length > 0 && (
<table className="w-full text-sm">
<thead>
<tr className="text-left text-xs text-muted border-b border-line-soft">
<tr className="text-left text-xs text-gray-400 border-b border-gray-100">
<th className="pb-1 font-medium pr-3">ID</th>
<th className="pb-1 font-medium pr-3">Source</th>
<th className="pb-1 font-medium pr-3">Date</th>

16
ui/src/pages/Login.jsx
View File

@ -20,32 +20,32 @@ export default function Login({ onLogin }) {
}
return (
<div className="flex items-center justify-center h-screen bg-raised">
<div className="bg-surface border border-line rounded-lg p-8 w-80 shadow-sm">
<h1 className="text-lg font-semibold text-ink mb-6">Dataflow</h1>
<div className="flex items-center justify-center h-screen bg-gray-50">
<div className="bg-white border border-gray-200 rounded-lg p-8 w-80 shadow-sm">
<h1 className="text-lg font-semibold text-gray-800 mb-6">Dataflow</h1>
<form onSubmit={handleSubmit} className="space-y-4">
<div>
<label className="block text-xs text-muted mb-1">Username</label>
<label className="block text-xs text-gray-500 mb-1">Username</label>
<input
type="text"
autoFocus
value={user}
onChange={e => setUser(e.target.value)}
className="w-full border border-line rounded px-3 py-2 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-2 text-sm focus:outline-none focus:border-blue-400"
required
/>
</div>
<div>
<label className="block text-xs text-muted mb-1">Password</label>
<label className="block text-xs text-gray-500 mb-1">Password</label>
<input
type="password"
value={pass}
onChange={e => setPass(e.target.value)}
className="w-full border border-line rounded px-3 py-2 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-2 text-sm focus:outline-none focus:border-blue-400"
required
/>
</div>
{error && <p className="text-xs text-danger">{error}</p>}
{error && <p className="text-xs text-red-500">{error}</p>}
<button
type="submit"
disabled={loading}

80
ui/src/pages/Mappings.jsx
View File

@ -68,13 +68,13 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
<div
ref={listRef}
style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
className="bg-surface border border-line rounded shadow-lg max-h-48 overflow-y-auto"
className="bg-white border border-gray-200 rounded shadow-lg max-h-48 overflow-y-auto"
>
{filtered.map((s, i) => (
<div
key={s}
className={`px-2 py-1 text-xs cursor-pointer whitespace-nowrap ${
i === highlighted ? 'bg-accent-soft text-accent' : 'text-ink-soft hover:bg-raised'
i === highlighted ? 'bg-blue-50 text-blue-700' : 'text-gray-700 hover:bg-gray-50'
}`}
onMouseDown={e => { e.preventDefault(); select(s) }}
>
@ -100,11 +100,11 @@ function SortHeader({ col, label, sortBy, onSort, className = '' }) {
const active = sortBy?.col === col
return (
<th
className={`px-3 py-2 font-medium cursor-pointer select-none hover:text-ink-soft ${className}`}
className={`px-3 py-2 font-medium cursor-pointer select-none hover:text-gray-600 ${className}`}
onClick={() => onSort(col)}
>
{label}
<span className="ml-1 text-muted">{active ? (sortBy.dir === 'asc' ? '↑' : '↓') : '↕'}</span>
<span className="ml-1 text-gray-300">{active ? (sortBy.dir === 'asc' ? '↑' : '↓') : '↕'}</span>
</th>
)
}
@ -354,18 +354,18 @@ export default function Mappings({ source, onNeedsReprocess }) {
}
}
if (!source) return <div className="p-4 sm:p-6 text-sm text-muted">Select a source first.</div>
if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
const displayRows = sortedRows(filteredRows)
return (
<div>
{/* Sticky control bar */}
<div className="sticky top-0 z-10 bg-surface border-b border-line px-6 py-3 flex items-center gap-3 flex-wrap">
<span className="text-sm font-medium text-ink-soft">{source}</span>
<div className="sticky top-0 z-10 bg-white border-b border-gray-200 px-6 py-3 flex items-center gap-3 flex-wrap">
<span className="text-sm font-medium text-gray-700">{source}</span>
<select
className="text-sm border border-line rounded px-2 py-1.5 focus:outline-none focus:border-accent"
className="text-sm border border-gray-200 rounded px-2 py-1.5 focus:outline-none focus:border-blue-400"
value={selectedRule}
onChange={e => setSelectedRule(e.target.value)}
>
@ -374,7 +374,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
</select>
{selectedRule && (
<div className="flex bg-raised rounded p-0.5">
<div className="flex bg-gray-100 rounded p-0.5">
{[
{ key: 'all', label: `All (${allValues.length})` },
{ key: 'unmapped', label: `Unmapped (${unmappedCount})` },
@ -382,7 +382,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
].map(({ key, label }) => (
<button key={key} onClick={() => setFilter(key)}
className={`text-xs px-3 py-1 rounded transition-colors ${
filter === key ? 'bg-surface text-ink shadow-sm' : 'text-muted'
filter === key ? 'bg-white text-gray-800 shadow-sm' : 'text-gray-500'
}`}>
{label}
</button>
@ -393,15 +393,15 @@ export default function Mappings({ source, onNeedsReprocess }) {
{selectedRule && (
<div className="relative">
<input
className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-accent ${
rowFilterError ? 'border-danger-line bg-danger-soft' : rowFilter ? 'border-accent-line' : 'border-line'
className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-blue-400 ${
rowFilterError ? 'border-red-400 bg-red-50' : rowFilter ? 'border-blue-300' : 'border-gray-200'
}`}
placeholder="filter regex…"
value={rowFilter}
onChange={e => setRowFilter(e.target.value)}
/>
{rowFilter && !rowFilterError && (
<span className="absolute right-2 top-1/2 -translate-y-1/2 text-xs text-muted">
<span className="absolute right-2 top-1/2 -translate-y-1/2 text-xs text-gray-400">
{filteredRows.length}
</span>
)}
@ -435,12 +435,12 @@ export default function Mappings({ source, onNeedsReprocess }) {
alert(err.message)
}
}}
className="text-sm px-3 py-1.5 border border-line rounded hover:bg-raised text-ink-soft"
className="text-sm px-3 py-1.5 border border-gray-200 rounded hover:bg-gray-50 text-gray-600"
>
Export TSV
</button>
)}
<label className={`text-sm px-3 py-1.5 border border-line rounded cursor-pointer hover:bg-raised text-ink-soft ${importing ? 'opacity-50 pointer-events-none' : ''}`}>
<label className={`text-sm px-3 py-1.5 border border-gray-200 rounded cursor-pointer hover:bg-gray-50 text-gray-600 ${importing ? 'opacity-50 pointer-events-none' : ''}`}>
{importing ? 'Importing…' : 'Import TSV'}
<input type="file" accept=".tsv,.txt" className="hidden" onChange={handleImportCSV} />
</label>
@ -450,24 +450,24 @@ export default function Mappings({ source, onNeedsReprocess }) {
{/* Content */}
<div className="p-6">
{!selectedRule && (
<p className="text-sm text-muted">Select a rule to view mappings.</p>
<p className="text-sm text-gray-400">Select a rule to view mappings.</p>
)}
{selectedRule && loading && (
<p className="text-sm text-muted">Loading…</p>
<p className="text-sm text-gray-400">Loading…</p>
)}
{selectedRule && !loading && allValues.length === 0 && (
<p className="text-sm text-muted">No extracted values for this rule. Run a transform first.</p>
<p className="text-sm text-gray-400">No extracted values for this rule. Run a transform first.</p>
)}
{selectedRule && !loading && allValues.length > 0 && (
<div className="overflow-x-auto">
{/* Bulk assign bar */}
{selected.size > 0 && (
<div className="flex items-center gap-2 mb-2 p-2 bg-accent-soft border border-accent-line rounded flex-wrap">
<span className="text-xs text-accent font-medium whitespace-nowrap">{selected.size} selected</span>
<div className="flex items-center gap-2 mb-2 p-2 bg-blue-50 border border-blue-200 rounded flex-wrap">
<span className="text-xs text-blue-700 font-medium whitespace-nowrap">{selected.size} selected</span>
{cols.map(col => (
<AutocompleteInput
key={col}
className="border border-accent-line rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-accent bg-surface"
className="border border-blue-300 rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-blue-500 bg-white"
placeholder={col}
value={bulkDraft[col] || ''}
onChange={v => setBulkDraft(d => ({ ...d, [col]: v }))}
@ -483,15 +483,15 @@ export default function Mappings({ source, onNeedsReprocess }) {
</button>
<button
onClick={() => { setSelected(new Set()); setBulkDraft({}) }}
className="text-xs text-accent hover:text-accent"
className="text-xs text-blue-400 hover:text-blue-600"
>
cancel
</button>
</div>
)}
<table className="w-full text-xs bg-surface border border-line rounded">
<table className="w-full text-xs bg-white border border-gray-200 rounded">
<thead>
<tr className="text-left text-muted border-b border-line-soft bg-raised">
<tr className="text-left text-gray-400 border-b border-gray-100 bg-gray-50">
<th className="px-2 py-2 w-6">
<input
type="checkbox"
@ -511,7 +511,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
{extraCols.map((col, idx) => (
<th key={`extra-${idx}`} className="px-3 py-2 font-medium">
<input
className="border border-line rounded px-1 py-0.5 w-24 focus:outline-none focus:border-accent font-normal"
className="border border-gray-200 rounded px-1 py-0.5 w-24 focus:outline-none focus:border-blue-400 font-normal"
value={col}
placeholder="new key"
onChange={e => setExtraCols(ec => { const c = [...ec]; c[idx] = e.target.value; return c })}
@ -521,7 +521,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
<th className="px-2 py-2">
<button
onClick={() => setExtraCols(ec => [...ec, ''])}
className="text-muted hover:text-ink-soft font-medium"
className="text-gray-400 hover:text-gray-700 font-medium"
title="Add column"
>+</button>
</th>
@ -536,7 +536,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
const isSaving = saving[k]
const isSelected = selected.has(k)
const hasDraft = !!(drafts[k] && Object.keys(drafts[k]).length > 0)
const rowBg = isSelected ? 'bg-accent-soft' : hasDraft ? 'bg-accent-soft' : row.is_mapped ? '' : 'bg-warn-soft'
const rowBg = isSelected ? 'bg-blue-50' : hasDraft ? 'bg-blue-50' : row.is_mapped ? '' : 'bg-yellow-50'
function handleRowClick(e) {
if (e.target.closest('input,button,a,select')) return
@ -571,7 +571,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
key={k}
ref={el => rowRefs.current[k] = el}
tabIndex={0}
className={`border-t border-line-soft hover:bg-raised cursor-pointer outline-none ${rowBg}`}
className={`border-t border-gray-50 hover:bg-gray-50 cursor-pointer outline-none ${rowBg}`}
onClick={handleRowClick}
onKeyDown={handleRowKeyDown}
>
@ -586,13 +586,13 @@ export default function Mappings({ source, onNeedsReprocess }) {
}}
/>
</td>
<td className="px-3 py-1.5 font-mono text-ink whitespace-nowrap">{displayValue(row.extracted_value)}</td>
<td className="px-3 py-1.5 text-right text-muted">{row.record_count}</td>
<td className="px-3 py-1.5 font-mono text-gray-800 whitespace-nowrap">{displayValue(row.extracted_value)}</td>
<td className="px-3 py-1.5 text-right text-gray-400">{row.record_count}</td>
{cols.map(col => (
<td key={col} className="px-3 py-1.5">
<AutocompleteInput
className={`border rounded px-2 py-1 w-full min-w-24 focus:outline-none focus:border-accent ${
hasDraft ? 'border-accent-line' : row.is_mapped ? 'border-line' : 'border-warn-line'
className={`border rounded px-2 py-1 w-full min-w-24 focus:outline-none focus:border-blue-400 ${
hasDraft ? 'border-blue-300' : row.is_mapped ? 'border-gray-200' : 'border-yellow-300'
}`}
value={cellVal(col)}
onChange={v => setCellValue(row.extracted_value, col, v)}
@ -605,7 +605,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
<td className="px-3 py-1.5 whitespace-nowrap">
{samples.length > 0 && (
<button
className="text-accent hover:text-accent"
className="text-blue-400 hover:text-blue-600"
onClick={() => setSampleOpen(s => ({ ...s, [k]: !s[k] }))}
>
{sampleOpen[k] ? 'hide' : 'show'}
@ -624,7 +624,7 @@ export default function Mappings({ source, onNeedsReprocess }) {
{row.is_mapped && (
<button
onClick={() => deleteRow(row)}
className="text-danger hover:text-danger text-base leading-none"
className="text-red-400 hover:text-red-600 text-base leading-none"
title="Remove mapping"
>×</button>
)}
@ -634,21 +634,21 @@ export default function Mappings({ source, onNeedsReprocess }) {
{sampleOpen[k] && (() => {
const sampleCols = [...new Set(samples.flatMap(r => Object.keys(r)))]
return (
<tr key={`${k}-sample`} className="border-t border-line-soft bg-raised">
<tr key={`${k}-sample`} className="border-t border-gray-50 bg-gray-50">
<td colSpan={3 + cols.length + 4} className="px-3 py-2">
<table className="w-full text-xs border border-line-soft rounded bg-surface">
<table className="w-full text-xs border border-gray-100 rounded bg-white">
<thead>
<tr className="bg-raised border-b border-line-soft">
<tr className="bg-gray-50 border-b border-gray-100">
{sampleCols.map(c => (
<th key={c} className="px-2 py-1 text-left font-medium text-muted whitespace-nowrap">{c}</th>
<th key={c} className="px-2 py-1 text-left font-medium text-gray-400 whitespace-nowrap">{c}</th>
))}
</tr>
</thead>
<tbody>
{samples.map((rec, i) => (
<tr key={i} className="border-t border-line-soft">
<tr key={i} className="border-t border-gray-50">
{sampleCols.map(c => (
<td key={c} className="px-2 py-1 font-mono text-ink-soft whitespace-nowrap">
<td key={c} className="px-2 py-1 font-mono text-gray-600 whitespace-nowrap">
{rec[c] != null ? String(rec[c]) : ''}
</td>
))}

372
ui/src/pages/Pivot.jsx
View File

@ -1,19 +1,33 @@
import { useEffect, useRef, useState, useCallback } from 'react'
import { api } from '../api'
import useTheme from '../theme.jsx'
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
async function fetchAllRows(source) {
const res = await api.getViewData(source, 100000, 0)
return res.rows || []
}
let perspectivePromise = null
function loadPerspective() {
return Promise.resolve(perspective)
if (perspectivePromise) return perspectivePromise
perspectivePromise = (async () => {
if (!document.getElementById('psp-theme')) {
const link = document.createElement('link')
link.id = 'psp-theme'
link.rel = 'stylesheet'
link.crossOrigin = 'anonymous'
link.href = 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer/dist/css/themes.css'
document.head.appendChild(link)
}
const [{ default: perspective }] = await Promise.all([
import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/client@4.4.0/dist/cdn/perspective.js'),
import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/cdn/perspective-viewer.js'),
import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-datagrid@4.4.0/dist/cdn/perspective-viewer-datagrid.js'),
import(/* @vite-ignore */ 'https://cdn.jsdelivr.net/npm/@perspective-dev/viewer-d3fc@4.4.0/dist/cdn/perspective-viewer-d3fc.js'),
])
return perspective
})()
return perspectivePromise
}
function formatVal(v, decimals = 2) {
@ -66,30 +80,32 @@ const LAYOUT_KEY = (source) => `psp_layout_${source}`
const DEFAULT_PLUGIN_CONFIG = { edit_mode: 'SELECT_REGION' }
export default function Pivot({ source, selectedStack, setSelectedStack }) {
const { dark } = useTheme()
export default function Pivot({ source }) {
const viewerRef = useRef()
const workerRef = useRef()
const tableRef = useRef()
const allRowsRef = useRef([])
const expandDepthRef = useRef(null)
const lastClickKeyRef = useRef(null)
const perspClickHandlerRef = useRef(null)
const [status, setStatus] = useState('idle')
const [error, setError] = useState('')
const [inspectedRows, setInspectedRows] = useState(null)
const [clickDetail, setClickDetail] = useState(null)
const [decimals, setDecimals] = useState(2)
const [paneWidth, setPaneWidth] = useState(384)
const [sortCol, setSortCol] = useState(null)
const [sortDir, setSortDir] = useState('asc')
const selectedView = selectedStack ?? source
const viewType = selectedStack ? 'stack' : 'source'
// View selector: source or a stack
const [stacks, setStacks] = useState([])
const [selectedView, setSelectedView] = useState(source) // name of active dfv view
const [viewType, setViewType] = useState('source') // 'source' | 'stack'
useEffect(() => { api.getStacks().then(setStacks).catch(() => {}) }, [])
// When sidebar source changes, reset to that source
useEffect(() => {
if (viewerRef.current) viewerRef.current.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
}, [dark])
if (viewType === 'source') setSelectedView(source)
}, [source])
function selectSource() { setViewType('source'); setSelectedView(source) }
function selectStack(name) { setViewType('stack'); setSelectedView(name) }
// Named layouts — stacks use localStorage only (no server FK to sources)
const [layouts, setLayouts] = useState([])
@ -106,19 +122,22 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
const loadLayouts = useCallback(async () => {
if (!selectedView) return
try {
const rows = viewType === 'source'
? await api.getPivotLayouts(selectedView)
: await api.getStackPivotLayouts(selectedView)
setLayouts(rows)
if (viewType === 'source') {
const rows = await api.getPivotLayouts(selectedView)
setLayouts(rows)
} else {
// Stacks: localStorage only
const stored = localStorage.getItem(`psp_layouts_stack_${selectedView}`)
setLayouts(stored ? JSON.parse(stored) : [])
}
} catch {}
}, [selectedView])
}, [selectedView, viewType])
useEffect(() => {
if (!selectedView) return
let cancelled = false
setInspectedRows(null)
setClickDetail(null)
lastClickKeyRef.current = null
setActiveLayoutId(null)
setShowSaveAs(false)
allRowsRef.current = []
@ -164,7 +183,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
return clean
}
perspClickHandlerRef.current = async (e) => {
viewer.addEventListener('perspective-click', async (e) => {
const detail = e.detail || {}
const { row, column_names } = detail
if (!row) return
@ -176,39 +195,14 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
const hasHierarchy = (config.group_by || []).length > 0
if (!hasHierarchy) return
// column_names encodes the full column path: [split_val_1, ..., split_val_N, measure]
// positionally matching config.split_by. Perspective may omit split_by coordinate
// filters from detail.config.filter, so derive any missing ones from column_names.
const splitByFields = config.split_by || []
const coveredByEvent = new Set(eventFilters.filter(([, op]) => op === '==').map(([f]) => f))
const derivedSplitFilters = splitByFields
.map((field, i) => {
if (coveredByEvent.has(field)) return null
const val = Array.isArray(column_names) && column_names[i] != null
? String(column_names[i]) : null
return val != null ? [field, '==', val] : null
})
.filter(Boolean)
const allFilters = [...eventFilters, ...derivedSplitFilters]
// Same cell clicked again — toggle the pane closed.
// Key on row path + column names (from the raw event) rather than derived
// filters, which can vary between clicks on stack/expression views.
const clickKey = JSON.stringify({ p: row['__ROW_PATH__'], c: column_names })
if (lastClickKeyRef.current === clickKey) {
lastClickKeyRef.current = null
setInspectedRows(null)
setClickDetail(null)
return
}
lastClickKeyRef.current = clickKey
setClickDetail({ row, config, column_names, eventFilters: allFilters })
setClickDetail({ row, config, column_names, eventFilters })
// Use a Perspective view with the event filters + expressions so computed
// columns (split_by) are evaluated and filtered correctly
try {
const view = await tableRef.current.view({
filter: allFilters,
expressions: config.expressions || {},
filter: eventFilters,
expressions: config.expressions || [],
})
const data = await view.to_json()
await view.delete()
@ -218,12 +212,10 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
Object.fromEntries(Object.entries(r).filter(([k]) => !exprNames.has(k)))
)
setInspectedRows(cleaned)
} catch (err) {
console.warn('Perspective inspector view failed, falling back to JS filter:', err)
setInspectedRows(filterRowsByConfig(allRowsRef.current, allFilters))
} catch {
setInspectedRows(filterRowsByConfig(allRowsRef.current, eventFilters))
}
}
viewer.addEventListener('perspective-click', perspClickHandlerRef.current)
})
await viewer.load(worker)
@ -239,7 +231,6 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
await plugin.restore(DEFAULT_PLUGIN_CONFIG)
}
await viewer.flush()
viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')
setStatus('ready')
} catch (err) {
@ -248,13 +239,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
}
init()
return () => {
cancelled = true
if (perspClickHandlerRef.current && viewerRef.current) {
viewerRef.current.removeEventListener('perspective-click', perspClickHandlerRef.current)
perspClickHandlerRef.current = null
}
}
return () => { cancelled = true }
}, [selectedView])
async function applyExpandDepth(viewer, depth) {
@ -294,6 +279,11 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(cleaned))
} catch {
// Layout references columns that no longer exist — remove it
if (viewType === 'stack') {
const updated = layouts.filter(l => l.id !== layout.id)
setLayouts(updated)
localStorage.setItem(`psp_layouts_stack_${selectedView}`, JSON.stringify(updated))
}
localStorage.removeItem(LAYOUT_KEY(selectedView))
setActiveLayoutId(null)
await viewer.restore({ table: selectedView, settings: false })
@ -308,22 +298,19 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
return { ...viewerConfig, plugin_config: pluginConfig, expand_depth: expandDepthRef.current }
}
const saveLayout = (name, config) => viewType === 'source'
? api.savePivotLayout(selectedView, name, config)
: api.saveStackPivotLayout(selectedView, name, config)
const deleteLayout = (id) => viewType === 'source'
? api.deletePivotLayout(selectedView, id)
: api.deleteStackPivotLayout(selectedView, id)
async function handleSaveOver() {
const layout = layouts.find(l => l.id === activeLayoutId)
if (!layout) return
const config = await captureConfig()
if (!config) return
try {
const saved = await saveLayout(layout.layout_name, config)
setActiveLayoutId(saved.id)
if (viewType === 'source') {
const saved = await api.savePivotLayout(selectedView, layout.layout_name, config)
setActiveLayoutId(saved.id)
} else {
const updated = layouts.map(l => l.id === activeLayoutId ? { ...l, config } : l)
localStorage.setItem(`psp_layouts_stack_${selectedView}`, JSON.stringify(updated))
}
localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
await loadLayouts()
flashMsg('Saved!')
@ -338,10 +325,18 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
const config = await captureConfig()
if (!config) return
try {
const saved = await saveLayout(name, config)
let newId
if (viewType === 'source') {
const saved = await api.savePivotLayout(selectedView, name, config)
newId = saved.id
} else {
newId = Date.now()
const updated = [...layouts, { id: newId, layout_name: name, config }]
localStorage.setItem(`psp_layouts_stack_${selectedView}`, JSON.stringify(updated))
}
localStorage.setItem(LAYOUT_KEY(selectedView), JSON.stringify(config))
await loadLayouts()
setActiveLayoutId(saved.id)
setActiveLayoutId(newId)
setShowSaveAs(false)
setSaveAsName('')
flashMsg('Saved!')
@ -353,7 +348,12 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
async function handleDelete(layout, e) {
e.stopPropagation()
try {
await deleteLayout(layout.id)
if (viewType === 'source') {
await api.deletePivotLayout(selectedView, layout.id)
} else {
const updated = layouts.filter(l => l.id !== layout.id)
localStorage.setItem(`psp_layouts_stack_${selectedView}`, JSON.stringify(updated))
}
if (activeLayoutId === layout.id) setActiveLayoutId(null)
await loadLayouts()
flashMsg('Deleted')
@ -370,28 +370,10 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
viewer.restore({ table: selectedView, settings: true, plugin_config: DEFAULT_PLUGIN_CONFIG })
}
if (!source) return <div className="p-4 sm:p-6 text-sm text-muted">Select a source first.</div>
if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
const cols = inspectedRows?.length ? Object.keys(inspectedRows[0]) : []
const sortedRows = sortCol == null || !inspectedRows ? inspectedRows : [...inspectedRows].sort((a, b) => {
const av = a[sortCol], bv = b[sortCol]
if (av == null && bv == null) return 0
if (av == null) return 1
if (bv == null) return -1
const num = typeof av === 'number' && typeof bv === 'number'
const cmp = num ? av - bv : String(av).localeCompare(String(bv))
return sortDir === 'asc' ? cmp : -cmp
})
const totals = cols.reduce((acc, c) => {
const vals = (inspectedRows || []).map(r => r[c])
if (vals.length > 0 && vals.every(v => v == null || typeof v === 'number')) {
acc[c] = vals.reduce((s, v) => s + (v ?? 0), 0)
}
return acc
}, {})
const groupBy = clickDetail?.config?.group_by || []
const splitBy = clickDetail?.config?.split_by || []
const coordFields = new Set([...groupBy, ...splitBy])
@ -401,39 +383,51 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
.map(([f, , v]) => [f, v])
)
const cellCoords = [...groupBy, ...splitBy].map(f => coordMap[f]).filter(Boolean)
// column_names = [split_val_1, ..., split_val_N, measure_name] — use positional split_by length
// to separate split values from measure names; fall back to coordMap when ambiguous
const colNames = clickDetail?.column_names || []
const splitVals = splitBy.map((f, i) =>
coordMap[f] ?? (colNames[i] != null ? String(colNames[i]) : null)
).filter(Boolean)
const metrics = splitBy.length > 0 ? colNames.slice(splitBy.length) : colNames
const cellKey = metrics.length > 0
const splitVals = splitBy.map(f => coordMap[f]).filter(Boolean)
const metrics = clickDetail?.column_names || []
const cellKey = splitVals.length > 0 && metrics.length > 0
? [...splitVals, ...metrics].join('|')
: null
return (
<div className="w-full h-full flex flex-col">
{/* Layouts sub-bar */}
<div className="flex items-center gap-2 px-3 h-9 bg-surface border-b border-line shrink-0 text-xs">
{/* Layout toolbar */}
<div className="flex items-center gap-2 px-3 py-1.5 bg-white border-b border-gray-200 flex-shrink-0">
{/* View selector */}
<div className="flex items-center gap-1 mr-2 border-r border-gray-200 pr-3">
<button
onClick={selectSource}
className={`text-xs rounded px-2 py-0.5 border transition-colors ${viewType === 'source' ? 'bg-blue-50 border-blue-300 text-blue-700' : 'bg-white border-gray-200 text-gray-500 hover:border-gray-400'}`}
>{source}</button>
{stacks.map(s => (
<button key={s.name}
onClick={() => selectStack(s.name)}
className={`text-xs rounded px-2 py-0.5 border transition-colors ${viewType === 'stack' && selectedView === s.name ? 'bg-purple-50 border-purple-300 text-purple-700' : 'bg-white border-gray-200 text-gray-500 hover:border-gray-400'}`}
>{s.name}</button>
))}
</div>
<span className="text-xs text-gray-400 uppercase tracking-wide mr-1">Layouts</span>
{layouts.map(l => (
<div key={l.id}
onClick={() => applyLayout(l)}
className={`flex items-center gap-1 rounded px-2 py-0.5 cursor-pointer border transition-colors
className={`flex items-center gap-1 text-xs rounded px-2 py-0.5 cursor-pointer border transition-colors
${activeLayoutId === l.id
? 'bg-accent-soft border-accent-line text-accent'
: 'bg-surface border-line text-ink-soft hover:border-line'}`}>
? 'bg-blue-50 border-blue-300 text-blue-700'
: 'bg-white border-gray-200 text-gray-600 hover:border-gray-400'}`}>
{l.layout_name}
<button
onClick={(e) => handleDelete(l, e)}
className="text-muted hover:text-danger leading-none ml-0.5 text-sm">×</button>
className="text-gray-300 hover:text-red-400 leading-none ml-0.5 text-sm">×</button>
</div>
))}
{activeLayoutId !== null && !showSaveAs && (
<button onClick={handleSaveOver}
className="text-accent hover:text-accent border border-accent-line rounded px-2 py-0.5">
className="text-xs text-blue-500 hover:text-blue-700 border border-blue-200 rounded px-2 py-0.5">
Save
</button>
)}
@ -446,27 +440,30 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
onChange={e => setSaveAsName(e.target.value)}
onKeyDown={e => { if (e.key === 'Enter') handleSaveAs(); if (e.key === 'Escape') { setShowSaveAs(false); setSaveAsName('') } }}
placeholder="Layout name…"
className="border border-line rounded px-2 py-0.5 w-36 focus:outline-none focus:border-accent"
className="text-xs border border-gray-300 rounded px-2 py-0.5 w-36 focus:outline-none focus:border-blue-400"
/>
<button onClick={handleSaveAs} className="text-accent hover:text-accent px-1">Save</button>
<button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-muted hover:text-ink-soft px-1">Cancel</button>
<button onClick={handleSaveAs} className="text-xs text-blue-600 hover:text-blue-800 px-1">Save</button>
<button onClick={() => { setShowSaveAs(false); setSaveAsName('') }} className="text-xs text-gray-400 hover:text-gray-600 px-1">Cancel</button>
</div>
) : (
<button
onClick={() => setShowSaveAs(true)}
className="text-muted hover:text-ink-soft border border-dashed border-line rounded px-2 py-0.5">
className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-0.5">
+ Save as…
</button>
)}
{activeLayoutId !== null && (
<button onClick={handleResetToDefault} className="text-muted hover:text-muted ml-1">reset</button>
<button onClick={handleResetToDefault}
className="text-xs text-gray-300 hover:text-gray-500 ml-1">
reset
</button>
)}
{layoutMsg && <span className="text-ok ml-1">{layoutMsg}</span>}
{layoutMsg && <span className="text-xs text-green-600 ml-1">{layoutMsg}</span>}
<div className="ml-auto flex items-center gap-1">
<span className="text-muted">depth:</span>
<span className="text-xs text-gray-400">depth:</span>
{[0, 1, 2, 3].map(d => (
<button key={d} onClick={async () => {
const v = viewerRef.current; if (!v) return
@ -475,7 +472,7 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
const p = await v.getPlugin()
await p.draw(view)
expandDepthRef.current = d
}} className="border border-line rounded px-1.5 py-0.5 text-muted hover:border-line">
}} className="text-xs border border-gray-200 rounded px-1.5 py-0.5 text-gray-500 hover:border-gray-400">
{d}
</button>
))}
@ -486,18 +483,18 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
<div className="relative flex-1 flex min-h-0">
<div className="relative flex-1">
{status === 'loading' && (
<div className="absolute inset-0 flex items-center justify-center z-10 bg-raised">
<p className="text-sm text-muted">Loading…</p>
<div className="absolute inset-0 flex items-center justify-center z-10 bg-gray-50">
<p className="text-sm text-gray-400">Loading…</p>
</div>
)}
{status === 'error' && (
<div className="absolute inset-0 flex items-center justify-center z-10 bg-raised">
<p className="text-sm text-danger">Error: {error}</p>
<div className="absolute inset-0 flex items-center justify-center z-10 bg-gray-50">
<p className="text-sm text-red-500">Error: {error}</p>
</div>
)}
{status === 'noview' && (
<div className="absolute inset-0 flex items-center justify-center z-10 bg-raised">
<p className="text-sm text-muted">No view data — generate a view and transform records first.</p>
<div className="absolute inset-0 flex items-center justify-center z-10 bg-gray-50">
<p className="text-sm text-gray-400">No view data — generate a view and transform records first.</p>
</div>
)}
<perspective-viewer
@ -507,61 +504,56 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
</div>
{inspectedRows && clickDetail && (
<div
style={{ width: paneWidth }}
className="relative border-l border-line bg-surface flex flex-col overflow-hidden flex-shrink-0"
>
{/* Drag-to-resize handle on left edge */}
<div
className="absolute left-0 top-0 bottom-0 w-1 cursor-col-resize hover:bg-blue-300 z-10"
onMouseDown={(e) => {
e.preventDefault()
const startX = e.clientX
const startW = paneWidth
const onMove = (me) => setPaneWidth(Math.max(240, startW + startX - me.clientX))
const onUp = () => {
document.removeEventListener('mousemove', onMove)
document.removeEventListener('mouseup', onUp)
}
document.addEventListener('mousemove', onMove)
document.addEventListener('mouseup', onUp)
}}
/>
{/* Header: breadcrumb + row count + controls */}
<div className="flex items-center justify-between pl-3 pr-2 py-2 border-b border-line-soft flex-shrink-0">
<div className="flex items-center gap-2 min-w-0">
{cellCoords.length > 0 && (
<span className="text-xs text-ink-soft font-mono font-semibold truncate">
{cellCoords.join(' › ')}
</span>
)}
<span className="text-xs text-muted flex-shrink-0">
{inspectedRows.length} row{inspectedRows.length !== 1 ? 's' : ''}
</span>
</div>
<div className="flex items-center gap-2 flex-shrink-0">
<div className="w-96 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
<div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
<span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">
{inspectedRows.length} row{inspectedRows.length !== 1 ? 's' : ''}
</span>
<div className="flex items-center gap-2">
<div className="flex items-center gap-0.5">
<button onClick={() => setDecimals(d => Math.max(0, d - 1))}
className="text-xs text-muted hover:text-ink-soft w-4 text-center">−</button>
<span className="text-xs text-muted w-4 text-center">{decimals}</span>
className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">−</button>
<span className="text-xs text-gray-400 w-4 text-center">{decimals}</span>
<button onClick={() => setDecimals(d => Math.min(8, d + 1))}
className="text-xs text-muted hover:text-ink-soft w-4 text-center">+</button>
className="text-xs text-gray-400 hover:text-gray-600 w-4 text-center">+</button>
</div>
<button onClick={() => { setInspectedRows(null); setClickDetail(null); lastClickKeyRef.current = null }}
className="text-muted hover:text-muted leading-none text-lg">×</button>
<button onClick={() => { setInspectedRows(null); setClickDetail(null) }}
className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
</div>
</div>
<div className="flex-1 overflow-y-auto">
{/* User-set filters (only shown when active) */}
{/* Cell coordinates */}
<div className="px-3 py-2 border-b border-gray-100">
<div className="text-xs text-gray-400 uppercase tracking-wide mb-1">
{[...groupBy, ...splitBy].join(' › ') || clickDetail.column_names?.join(', ') || 'Cell'}
</div>
{cellCoords.length > 0 && (
<div className="text-xs text-gray-700 font-mono font-semibold">
{cellCoords.join(' › ')}
</div>
)}
{Object.entries(clickDetail.row)
.filter(([k, v]) => k !== '__ROW_PATH__' && v != null)
.map(([k, v]) => {
const isSelected = cellKey != null && k === cellKey
return (
<div key={k} className={`flex justify-between py-0.5 gap-2 ${isSelected ? 'font-semibold' : ''}`}>
<span className={`text-xs font-mono shrink-0 ${isSelected ? 'text-gray-700' : 'text-gray-400'}`}>{k}</span>
<span className={`text-xs font-mono text-right ${isSelected ? 'text-blue-600' : 'text-gray-700'}`}>{formatVal(v, decimals)}</span>
</div>
)
})}
</div>
{/* User-set filters */}
{(() => {
const userFilters = (clickDetail.eventFilters || []).filter(([f]) => !coordFields.has(f))
return userFilters.length > 0 ? (
<div className="px-3 py-2 border-b border-line-soft">
<div className="text-xs text-muted uppercase tracking-wide mb-1">Filters</div>
<div className="px-3 py-2 border-b border-gray-100">
<div className="text-xs text-gray-400 uppercase tracking-wide mb-1">Filters</div>
{userFilters.map((f, i) => (
<div key={i} className="text-xs text-muted py-0.5 font-mono">{f.join(' ')}</div>
<div key={i} className="text-xs text-gray-500 py-0.5 font-mono">{f.join(' ')}</div>
))}
</div>
) : null
@ -572,44 +564,26 @@ export default function Pivot({ source, selectedStack, setSelectedStack }) {
<div className="overflow-auto">
<table className="w-full text-xs">
<thead>
<tr className="text-left text-muted border-b border-line-soft bg-raised sticky top-0">
{cols.map(c => {
const active = sortCol === c
return (
<th key={c}
onClick={() => { if (active) setSortDir(d => d === 'asc' ? 'desc' : 'asc'); else { setSortCol(c); setSortDir('asc') } }}
className="px-2 py-1 font-medium whitespace-nowrap cursor-pointer select-none hover:text-ink-soft">
{c}{active ? (sortDir === 'asc' ? ' ▲' : ' ▼') : ''}
</th>
)
})}
<tr className="text-left text-gray-400 border-b border-gray-100 bg-gray-50 sticky top-0">
{cols.map(c => (
<th key={c} className="px-2 py-1 font-medium whitespace-nowrap">{c}</th>
))}
</tr>
</thead>
<tbody>
{sortedRows.map((row, i) => (
<tr key={i} className="border-t border-line-soft hover:bg-raised">
{inspectedRows.map((row, i) => (
<tr key={i} className="border-t border-gray-50 hover:bg-gray-50">
{cols.map(c => {
const f = formatVal(row[c], decimals)
return (
<td key={c} className="px-2 py-1 font-mono whitespace-nowrap text-ink-soft max-w-40 truncate">
{f == null ? <span className="text-muted">—</span> : f}
<td key={c} className="px-2 py-1 font-mono whitespace-nowrap text-gray-700 max-w-40 truncate">
{f == null ? <span className="text-gray-300">—</span> : f}
</td>
)
})}
</tr>
))}
</tbody>
{Object.keys(totals).length > 0 && (
<tfoot>
<tr className="border-t-2 border-line bg-raised font-semibold text-ink-soft sticky bottom-0">
{cols.map(c => (
<td key={c} className="px-2 py-1 font-mono whitespace-nowrap text-right">
{totals[c] != null ? formatVal(totals[c], decimals) : ''}
</td>
))}
</tr>
</tfoot>
)}
</table>
</div>
)}

243
ui/src/pages/Records.jsx
View File

@ -49,10 +49,10 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
{open && filtered.length > 0 && dropPos && (
<div ref={listRef}
style={{ position: 'fixed', top: dropPos.top, left: dropPos.left, minWidth: dropPos.minWidth, zIndex: 9999 }}
className="bg-surface border border-line rounded shadow-lg max-h-40 overflow-y-auto">
className="bg-white border border-gray-200 rounded shadow-lg max-h-40 overflow-y-auto">
{filtered.map((s, i) => (
<div key={s}
className={`px-2 py-1 text-xs cursor-pointer whitespace-nowrap ${i === highlighted ? 'bg-accent-soft text-accent' : 'text-ink-soft hover:bg-raised'}`}
className={`px-2 py-1 text-xs cursor-pointer whitespace-nowrap ${i === highlighted ? 'bg-blue-50 text-blue-700' : 'text-gray-700 hover:bg-gray-50'}`}
onMouseDown={e => { e.preventDefault(); select(s) }}>{s}</div>
))}
</div>
@ -62,16 +62,7 @@ function AutocompleteInput({ value, onChange, onEnter, suggestions = [], classNa
}
const DATE_RE = /^\d{4}-\d{2}-\d{2}(T[\d:.Z+-]+)?$/
// Hidden everywhere a field can be edited as an override — you can't override an id
const HIDDEN_COLS = new Set(['id', '_overridden'])
// The grid and its filters keep id: it's the only handle on a specific row
const GRID_HIDDEN_COLS = new Set(['_overridden'])
// id first, then the rest in view order
function gridCols(keys) {
const visible = keys.filter(c => !GRID_HIDDEN_COLS.has(c))
return visible.includes('id') ? ['id', ...visible.filter(c => c !== 'id')] : visible
}
function formatVal(val) {
if (val === null || val === undefined) return null
@ -128,14 +119,7 @@ export default function Records({ source }) {
setPanelOpen(false)
setOverrideCols([])
setExtraCols([])
// Default to newest first on the source's first date field, if it has one
api.getSource(source)
.then(src => (src?.config?.fields || []).find(f => f.type === 'date')?.name || null)
.catch(() => null)
.then(dateCol => {
if (dateCol) setSort({ col: dateCol, dir: 'desc' })
load(0, dateCol, 'desc', [])
})
load(0, null, 'asc', [])
api.getOverrideKeys(source).then(setOverrideCols).catch(() => {})
api.getGlobalValues().then(setGlobalValues).catch(() => {})
setSelected(new Set())
@ -188,8 +172,7 @@ export default function Records({ source }) {
}
function addFilter() {
// id is filterable but a poor default — start on the first data column
const visCols = gridCols(cols).filter(c => c !== 'id')
const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
setFilters(f => [...f, { col: visCols[0] || '', pattern: '' }])
}
@ -283,12 +266,12 @@ export default function Records({ source }) {
}
}
if (!source) return <div className="p-4 sm:p-6 text-sm text-muted">Select a source first.</div>
if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
const displayCols = gridCols(rows.length > 0 ? Object.keys(rows[0]) : cols)
const visCols = gridCols(cols)
const displayCols = (rows.length > 0 ? Object.keys(rows[0]) : cols).filter(c => !HIDDEN_COLS.has(c))
const visCols = cols.filter(c => !HIDDEN_COLS.has(c))
// For bulk bar: only established override keys
// All override cols: known from DB + new ones added this session
const allOverrideCols = [...new Set([...overrideCols, ...extraCols])]
const savedOverrides = selectedRecord?.overrides || {}
@ -300,67 +283,65 @@ export default function Records({ source }) {
<div className="flex h-full min-h-0 overflow-hidden">
<div className="flex-1 overflow-auto p-6 min-w-0">
<div className="flex items-center justify-between mb-4">
<h1 className="text-xl font-semibold text-ink">Records — {source}</h1>
<h1 className="text-xl font-semibold text-gray-800">Records — {source}</h1>
{exists && rows.length > 0 && (
<span className="text-xs text-muted font-mono">dfv.{source}</span>
<span className="text-xs text-gray-400 font-mono">dfv.{source}</span>
)}
</div>
{/* Filter bar */}
{exists !== false && visCols.length > 0 && (
<div className="mb-4 flex flex-wrap gap-2 items-center">
<span className="text-xs text-muted font-medium mr-1">DB query:</span>
{filters.map((f, i) => (
<div key={i} className="flex items-center gap-1 bg-surface border border-line rounded px-2 py-1">
<div key={i} className="flex items-center gap-1 bg-white border border-gray-200 rounded px-2 py-1">
<select
className="text-xs text-ink-soft border-0 focus:outline-none bg-transparent"
className="text-xs text-gray-600 border-0 focus:outline-none bg-transparent"
value={f.col}
onChange={e => updateFilter(i, 'col', e.target.value)}
>
{visCols.map(c => <option key={c} value={c}>{c}</option>)}
</select>
<span className="text-xs text-muted mx-0.5">~*</span>
<span className="text-xs text-gray-300 mx-0.5">~*</span>
<input
className="text-xs font-mono border-0 focus:outline-none w-36 bg-transparent"
placeholder="regex…"
value={f.pattern}
onChange={e => updateFilter(i, 'pattern', e.target.value)}
/>
<button onClick={() => removeFilter(i)} className="text-muted hover:text-muted ml-1 leading-none">×</button>
<button onClick={() => removeFilter(i)} className="text-gray-300 hover:text-gray-500 ml-1 leading-none">×</button>
</div>
))}
<button onClick={addFilter}
className="text-xs text-muted hover:text-ink-soft border border-dashed border-line rounded px-2 py-1">
className="text-xs text-gray-400 hover:text-gray-600 border border-dashed border-gray-200 rounded px-2 py-1">
+ filter
</button>
{filters.length > 0 && (
<button onClick={() => { setFilters([]); setOffset(0); load(0, sort.col, sort.dir, []) }}
className="text-xs text-muted hover:text-danger">clear</button>
className="text-xs text-gray-400 hover:text-red-500">clear</button>
)}
</div>
)}
{/* Bulk select + override bar */}
{/* Row filter + Bulk override bar */}
{exists && visCols.length > 0 && (
<div className="mb-4 flex flex-wrap gap-2 items-center">
<span className="text-xs text-muted font-medium mr-1">Bulk select:</span>
<input
className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-accent ${
rowFilter ? 'border-accent-line' : 'border-line'
className={`text-xs font-mono border rounded px-2 py-1.5 w-44 focus:outline-none focus:border-blue-400 ${
rowFilter ? 'border-blue-300' : 'border-gray-200'
}`}
placeholder="regex on loaded rows…"
placeholder="filter regex…"
value={rowFilter}
onChange={e => setRowFilter(e.target.value)}
/>
{rowFilter && (
<span className="text-xs text-muted">{selected.size} of {rows.length} rows selected</span>
<span className="text-xs text-gray-400">{selected.size} selected</span>
)}
{selected.size > 0 && (
<div className="flex items-center gap-2 ml-4 p-2 bg-accent-soft border border-accent-line rounded flex-wrap">
<div className="flex items-center gap-2 ml-4 p-2 bg-blue-50 border border-blue-200 rounded flex-wrap">
{allOverrideCols.map(col => (
<AutocompleteInput
key={col}
className="border border-accent-line rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-accent bg-surface"
className="border border-blue-300 rounded px-2 py-1 text-xs min-w-24 focus:outline-none focus:border-blue-500 bg-white"
placeholder={col}
value={bulkDraft[col] || ''}
onChange={v => setBulkDraft(d => ({ ...d, [col]: v }))}
@ -395,7 +376,7 @@ export default function Records({ source }) {
</button>
<button
onClick={() => { setSelected(new Set()); setBulkDraft({}); setRowFilter('') }}
className="text-xs text-accent hover:text-accent"
className="text-xs text-blue-400 hover:text-blue-600"
>
cancel
</button>
@ -404,25 +385,25 @@ export default function Records({ source }) {
</div>
)}
{loading && <p className="text-sm text-muted">Loading…</p>}
{!loading && viewError && <p className="text-sm text-danger">View error: {viewError} — check field types in Sources.</p>}
{loading && <p className="text-sm text-gray-400">Loading…</p>}
{!loading && viewError && <p className="text-sm text-red-500">View error: {viewError} — check field types in Sources.</p>}
{!loading && exists === false && (
<p className="text-sm text-muted">
No view generated yet. Go to <span className="font-medium text-ink-soft">Sources</span>, check fields as <span className="font-medium text-ink-soft">In view</span>, then click <span className="font-medium text-ink-soft">Generate view</span>.
<p className="text-sm text-gray-400">
No view generated yet. Go to <span className="font-medium text-gray-600">Sources</span>, check fields as <span className="font-medium text-gray-600">In view</span>, then click <span className="font-medium text-gray-600">Generate view</span>.
</p>
)}
{!loading && exists && rows.length === 0 && (
<p className="text-sm text-muted">
<p className="text-sm text-gray-400">
{filters.some(f => f.col && f.pattern) ? 'No records match the current filters.' : 'View exists but no transformed records yet. Import data and run a transform first.'}
</p>
)}
{!loading && exists && rows.length > 0 && (
<>
<div className="bg-surface border border-line rounded overflow-auto mb-4">
<div className="bg-white border border-gray-200 rounded overflow-auto mb-4">
<table className="w-full text-sm">
<thead>
<tr className="text-left text-xs text-muted border-b border-line-soft bg-raised">
<tr className="text-left text-xs text-gray-400 border-b border-gray-100 bg-gray-50">
<th className="px-2 py-2 w-8">
<input
type="checkbox"
@ -438,9 +419,9 @@ export default function Records({ source }) {
const active = sort.col === col
return (
<th key={col} onClick={() => toggleSort(col)}
className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-ink-soft">
className="px-3 py-2 font-medium whitespace-nowrap cursor-pointer select-none hover:text-gray-600">
{col}
<span className="ml-1 text-muted">{active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}</span>
<span className="ml-1 text-gray-300">{active ? (sort.dir === 'asc' ? '▲' : '▼') : '⇅'}</span>
</th>
)
})}
@ -453,8 +434,8 @@ export default function Records({ source }) {
const isPanelSelected = selectedRow?.id != null && selectedRow.id === row.id
return (
<tr key={i} onClick={() => openPanel(row)}
className={`border-t border-line-soft cursor-pointer transition-colors
${isPanelSelected ? 'bg-accent-soft' : isRowSelected ? 'bg-accent-soft' : isOverridden ? 'bg-warn-soft hover:bg-warn-soft' : 'hover:bg-raised'}`}>
className={`border-t border-gray-50 cursor-pointer transition-colors
${isPanelSelected ? 'bg-blue-50' : isRowSelected ? 'bg-blue-50' : isOverridden ? 'bg-amber-50 hover:bg-amber-100' : 'hover:bg-gray-50'}`}>
<td className="px-2 py-2">
<input
type="checkbox"
@ -469,8 +450,8 @@ export default function Records({ source }) {
{displayCols.map((col, j) => {
const formatted = formatVal(row[col])
return (
<td key={j} className="px-3 py-2 text-xs text-ink-soft whitespace-nowrap max-w-48 truncate">
{formatted === null ? <span className="text-muted">—</span> : formatted}
<td key={j} className="px-3 py-2 text-xs text-gray-600 whitespace-nowrap max-w-48 truncate">
{formatted === null ? <span className="text-gray-300">—</span> : formatted}
</td>
)
})}
@ -481,12 +462,12 @@ export default function Records({ source }) {
</table>
</div>
<div className="flex items-center gap-3 text-sm text-muted">
<div className="flex items-center gap-3 text-sm text-gray-500">
<button onClick={prev} disabled={offset === 0}
className="px-3 py-1 border border-line rounded hover:bg-raised disabled:opacity-40">← Prev</button>
className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">← Prev</button>
<span>{offset + 1}–{offset + rows.length}</span>
<button onClick={next} disabled={rows.length < LIMIT}
className="px-3 py-1 border border-line rounded hover:bg-raised disabled:opacity-40">Next →</button>
className="px-3 py-1 border border-gray-200 rounded hover:bg-gray-50 disabled:opacity-40">Next →</button>
</div>
</>
)}
@ -494,123 +475,75 @@ export default function Records({ source }) {
{/* Panel */}
{panelOpen && (
<div className="w-80 border-l border-line bg-surface flex flex-col overflow-hidden flex-shrink-0">
<div className="flex items-center justify-between px-3 py-2 border-b border-line-soft">
<span className="text-xs font-semibold text-ink-soft uppercase tracking-wide">Record</span>
<button onClick={closePanel} className="text-muted hover:text-muted leading-none text-lg">×</button>
<div className="w-80 border-l border-gray-200 bg-white flex flex-col overflow-hidden flex-shrink-0">
<div className="flex items-center justify-between px-3 py-2 border-b border-gray-100">
<span className="text-xs font-semibold text-gray-600 uppercase tracking-wide">Record</span>
<button onClick={closePanel} className="text-gray-300 hover:text-gray-500 leading-none text-lg">×</button>
</div>
{panelLoading && <p className="text-xs text-muted p-3">Loading…</p>}
{panelLoading && <p className="text-xs text-gray-400 p-3">Loading…</p>}
{selectedRecord && !panelLoading && (
<div className="flex-1 overflow-y-auto flex flex-col min-h-0">
{panelMsg && (
<div className={`text-xs px-3 py-2 border-b border-line-soft ${panelMsg.ok ? 'text-ok' : 'text-danger'}`}>
<div className={`text-xs px-3 py-2 border-b border-gray-100 ${panelMsg.ok ? 'text-green-600' : 'text-red-500'}`}>
{panelMsg.text}
</div>
)}
{/* Raw fields — read only */}
<div className="border-b border-line-soft">
<div className="px-3 py-1.5 bg-raised border-b border-line-soft">
<span className="text-xs font-medium text-muted uppercase tracking-wide">Raw</span>
</div>
{Object.entries(selectedRecord.data || {}).map(([field, val]) => (
<div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-line-soft first:border-t-0">
<span className="text-xs font-mono text-muted w-28 shrink-0 truncate">{field}</span>
<span className="text-xs font-mono text-muted truncate">{formatVal(val) ?? <span className="text-muted">—</span>}</span>
{/* Read-only transformed fields */}
<div className="border-b border-gray-100">
{Object.entries(selectedRecord.transformed || {}).map(([field, val]) => (
<div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-gray-50 first:border-t-0">
<span className="text-xs font-mono text-gray-400 w-28 shrink-0 truncate">{field}</span>
<span className="text-xs font-mono text-gray-600 truncate">{formatVal(val) ?? <span className="text-gray-300">—</span>}</span>
</div>
))}
</div>
{/* Transformed fields — read only delta */}
<div className="border-b border-line-soft">
<div className="px-3 py-1.5 bg-raised border-b border-line-soft">
<span className="text-xs font-medium text-muted uppercase tracking-wide">Transformed</span>
</div>
{Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).length === 0
? <div className="px-3 py-2 text-xs text-muted">No rule output yet.</div>
: Object.entries(selectedRecord.transformed || {}).filter(([k]) => !HIDDEN_COLS.has(k)).map(([field, val]) => (
<div key={field} className="flex items-baseline gap-2 px-3 py-1 border-t border-line-soft first:border-t-0">
<span className="text-xs font-mono text-muted w-28 shrink-0 truncate">{field}</span>
<span className="text-xs font-mono text-accent truncate">{formatVal(val) ?? <span className="text-muted">—</span>}</span>
</div>
))
}
</div>
{/* Overrides — editable */}
<div className="flex-1 border-b border-line-soft">
<div className="flex items-center justify-between px-3 py-1.5 bg-raised border-b border-line-soft">
<span className="text-xs font-medium text-muted uppercase tracking-wide">Overrides</span>
{/* Override cols — Mappings-style */}
<div className="flex-1">
<div className="flex items-center justify-between px-3 py-1.5 bg-gray-50 border-b border-gray-100">
<span className="text-xs font-medium text-gray-500 uppercase tracking-wide">Override</span>
<button
onClick={() => setExtraCols(ec => [...ec, ''])}
className="text-muted hover:text-ink-soft font-medium text-sm leading-none"
title="Add field">+</button>
className="text-gray-400 hover:text-gray-700 font-medium text-sm leading-none"
title="Add column">+</button>
</div>
<table className="w-full text-xs">
<tbody>
{[...new Set([
...Object.keys(selectedRecord.transformed || {}),
...Object.keys(selectedRecord.overrides || {}),
...overrideCols
])].filter(k => !HIDDEN_COLS.has(k)).map(col => {
const override = overrideDraft[col] ?? ''
const placeholder = formatVal(selectedRecord.transformed?.[col]) ?? ''
{allOverrideCols.map((col, idx) => {
const isExtra = idx >= overrideCols.length
const suggestions = [...(globalValues[col] || [])].sort()
const val = overrideDraft[col] ?? ''
return (
<tr key={col} className="border-t border-line-soft">
<td className="px-3 py-1.5 w-28 shrink-0">
<span className="font-mono text-muted truncate block">{col}</span>
</td>
<td className="px-1 py-1.5">
<AutocompleteInput
className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
override ? 'border-warn-line bg-warn-soft text-warn' : 'border-line text-ink-soft'
}`}
value={override}
placeholder={placeholder}
onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
onEnter={handleSaveOverrides}
suggestions={suggestions}
/>
</td>
<td className="pr-2 text-center w-6">
{override && (
<button
onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
className="text-muted hover:text-danger leading-none text-base">×</button>
<tr key={col || `extra-${idx}`} className="border-t border-gray-50">
<td className="px-3 py-1 w-28 shrink-0">
{isExtra ? (
<input
className="w-full text-xs font-mono border border-gray-200 rounded px-1 py-0.5 focus:outline-none focus:border-blue-400"
value={col}
placeholder="field name"
onChange={e => {
const newName = e.target.value
setExtraCols(ec => { const c = [...ec]; c[idx - overrideCols.length] = newName; return c })
if (val) setOverrideDraft(d => {
const n = { ...d }
delete n[col]
if (newName) n[newName] = val
return n
})
}}
/>
) : (
<span className="font-mono text-gray-500 truncate block">{col}</span>
)}
</td>
</tr>
)
})}
{extraCols.map((col, i) => {
const val = overrideDraft[col] ?? ''
const suggestions = [...(globalValues[col] || [])].sort()
return (
<tr key={`extra-${i}`} className="border-t border-line-soft">
<td className="px-3 py-1.5 w-28 shrink-0">
<input
className="w-full text-xs font-mono border border-line rounded px-1 py-0.5 focus:outline-none focus:border-accent"
value={col}
placeholder="field name"
onChange={e => {
const newName = e.target.value
setExtraCols(ec => { const c = [...ec]; c[i] = newName; return c })
if (val) setOverrideDraft(d => {
const n = { ...d }
delete n[col]
if (newName) n[newName] = val
return n
})
}}
/>
</td>
<td className="px-1 py-1.5">
<td className="px-1 py-1">
<AutocompleteInput
className={`w-full text-xs font-mono px-2 py-0.5 rounded border focus:outline-none ${
val ? 'border-warn-line bg-warn-soft text-warn' : 'border-line text-ink-soft'
val ? 'border-amber-300 bg-amber-50 text-amber-800' : 'border-gray-200 text-gray-700'
}`}
value={val}
onChange={v => setOverrideDraft(d => ({ ...d, [col]: v }))}
@ -618,11 +551,11 @@ export default function Records({ source }) {
suggestions={suggestions}
/>
</td>
<td className="pr-2 text-center w-6">
<td className="pr-2 text-center">
{val && (
<button
onClick={() => setOverrideDraft(d => { const n = { ...d }; delete n[col]; return n })}
className="text-muted hover:text-danger leading-none text-base">×</button>
className="text-gray-300 hover:text-red-400 leading-none text-base">×</button>
)}
</td>
</tr>
@ -632,7 +565,7 @@ export default function Records({ source }) {
</table>
</div>
<div className="flex gap-2 px-3 py-2 border-t border-line-soft shrink-0">
<div className="flex gap-2 px-3 py-2 border-t border-gray-100">
<button
onClick={handleSaveOverrides}
disabled={panelSaving || !isDirty}
@ -643,7 +576,7 @@ export default function Records({ source }) {
<button
onClick={handleClearOverrides}
disabled={panelSaving}
className="text-xs border border-line rounded px-3 py-1.5 text-muted hover:border-danger-line hover:text-danger disabled:opacity-40">
className="text-xs border border-gray-200 rounded px-3 py-1.5 text-gray-500 hover:border-red-300 hover:text-red-500 disabled:opacity-40">
Clear
</button>
)}

62
ui/src/pages/Remap.jsx
View File

@ -73,8 +73,8 @@ export default function Remap() {
}
return (
<div className="p-4 sm:p-6 max-w-4xl">
<h1 className="text-base font-semibold text-ink mb-4">Remap Output Values</h1>
<div className="p-6 max-w-4xl">
<h1 className="text-base font-semibold text-gray-800 mb-4">Remap Output Values</h1>
{/* Search */}
<form onSubmit={handleSearch} className="flex items-center gap-2 mb-5">
@ -83,7 +83,7 @@ export default function Remap() {
value={search}
onChange={e => setSearch(e.target.value)}
placeholder="Search output values…"
className="text-sm border border-line rounded px-3 py-1.5 w-72 focus:outline-none focus:border-accent"
className="text-sm border border-gray-300 rounded px-3 py-1.5 w-72 focus:outline-none focus:border-blue-400"
/>
<button type="submit" disabled={searching}
className="text-sm bg-blue-600 text-white rounded px-3 py-1.5 hover:bg-blue-700 disabled:opacity-50">
@ -95,15 +95,15 @@ export default function Remap() {
{results !== null && (
<div className="mb-6">
{results.length === 0 ? (
<p className="text-sm text-muted">No matching output values found.</p>
<p className="text-sm text-gray-400">No matching output values found.</p>
) : (
<>
<div className="text-xs text-muted uppercase tracking-wide mb-1">
<div className="text-xs text-gray-400 uppercase tracking-wide mb-1">
{results.length} result{results.length !== 1 ? 's' : ''} — click one to remap
</div>
<table className="w-full text-sm border border-line rounded overflow-hidden">
<table className="w-full text-sm border border-gray-200 rounded overflow-hidden">
<thead>
<tr className="bg-raised text-left text-xs text-muted uppercase tracking-wide">
<tr className="bg-gray-50 text-left text-xs text-gray-400 uppercase tracking-wide">
<th className="px-3 py-2">Field</th>
<th className="px-3 py-2">Value</th>
<th className="px-3 py-2 text-right">Mappings</th>
@ -115,11 +115,11 @@ export default function Remap() {
return (
<tr key={i}
onClick={() => handleSelect(r)}
className={`border-t border-line-soft cursor-pointer transition-colors
${isActive ? 'bg-accent-soft' : 'hover:bg-raised'}`}>
<td className="px-3 py-2 font-mono text-muted">{r.col}</td>
<td className="px-3 py-2 font-mono text-ink">{r.val}</td>
<td className="px-3 py-2 text-right text-muted">{r.mapping_count}</td>
className={`border-t border-gray-100 cursor-pointer transition-colors
${isActive ? 'bg-blue-50' : 'hover:bg-gray-50'}`}>
<td className="px-3 py-2 font-mono text-gray-500">{r.col}</td>
<td className="px-3 py-2 font-mono text-gray-800">{r.val}</td>
<td className="px-3 py-2 text-right text-gray-400">{r.mapping_count}</td>
</tr>
)
})}
@ -132,25 +132,25 @@ export default function Remap() {
{/* Remap panel */}
{selected && (
<div className="border border-line rounded p-4 mb-6 bg-surface">
<div className="text-xs text-muted uppercase tracking-wide mb-3">
Remap <span className="font-mono text-ink-soft">{selected.col}</span>
<div className="border border-gray-200 rounded p-4 mb-6 bg-white">
<div className="text-xs text-gray-400 uppercase tracking-wide mb-3">
Remap <span className="font-mono text-gray-600">{selected.col}</span>
</div>
<div className="flex items-center gap-3 mb-4">
<div className="flex-1">
<div className="text-xs text-muted mb-1">From</div>
<div className="text-sm font-mono bg-raised border border-line rounded px-3 py-1.5 text-ink-soft">
<div className="text-xs text-gray-400 mb-1">From</div>
<div className="text-sm font-mono bg-gray-50 border border-gray-200 rounded px-3 py-1.5 text-gray-700">
{selected.val}
</div>
</div>
<div className="text-muted mt-4">→</div>
<div className="text-gray-300 mt-4">→</div>
<div className="flex-1">
<div className="text-xs text-muted mb-1">To</div>
<div className="text-xs text-gray-400 mb-1">To</div>
<input
value={toVal}
onChange={e => setToVal(e.target.value)}
onKeyDown={e => e.key === 'Enter' && handleApply()}
className="w-full text-sm font-mono border border-line rounded px-3 py-1.5 focus:outline-none focus:border-accent"
className="w-full text-sm font-mono border border-gray-300 rounded px-3 py-1.5 focus:outline-none focus:border-blue-400"
/>
</div>
<div className="mt-4">
@ -164,22 +164,22 @@ export default function Remap() {
</div>
{msg && (
<div className={`text-sm mb-3 ${msg.ok ? 'text-ok' : 'text-danger'}`}>
<div className={`text-sm mb-3 ${msg.ok ? 'text-green-600' : 'text-red-500'}`}>
{msg.text}
</div>
)}
{/* Affected mappings */}
{loadingMatches ? (
<p className="text-xs text-muted">Loading…</p>
<p className="text-xs text-gray-400">Loading…</p>
) : matches && matches.length > 0 && (
<div>
<div className="text-xs text-muted uppercase tracking-wide mb-1">
<div className="text-xs text-gray-400 uppercase tracking-wide mb-1">
Affected mappings
</div>
<table className="w-full text-xs border border-line-soft rounded overflow-hidden">
<table className="w-full text-xs border border-gray-100 rounded overflow-hidden">
<thead>
<tr className="bg-raised text-left text-muted">
<tr className="bg-gray-50 text-left text-gray-400">
<th className="px-2 py-1">Source</th>
<th className="px-2 py-1">Rule</th>
<th className="px-2 py-1">Input</th>
@ -188,15 +188,15 @@ export default function Remap() {
</thead>
<tbody>
{matches.map(m => (
<tr key={m.id} className="border-t border-line-soft">
<td className="px-2 py-1 font-mono text-muted">{m.source_name}</td>
<td className="px-2 py-1 font-mono text-muted">{m.rule_name}</td>
<td className="px-2 py-1 font-mono text-ink-soft">
<tr key={m.id} className="border-t border-gray-50">
<td className="px-2 py-1 font-mono text-gray-500">{m.source_name}</td>
<td className="px-2 py-1 font-mono text-gray-500">{m.rule_name}</td>
<td className="px-2 py-1 font-mono text-gray-700">
{typeof m.input_value === 'string' ? m.input_value : JSON.stringify(m.input_value)}
</td>
<td className="px-2 py-1 font-mono text-ink-soft">
<td className="px-2 py-1 font-mono text-gray-700">
{Object.entries(m.output).map(([k, v]) => (
<span key={k} className={k === selected.col ? 'text-accent font-semibold' : ''}>
<span key={k} className={k === selected.col ? 'text-blue-600 font-semibold' : ''}>
{k}: {v}{' '}
</span>
))}

106
ui/src/pages/Rules.jsx
View File

@ -7,27 +7,27 @@ function PreviewModal({ rows, onClose }) {
const matched = rows.filter(r => r.extracted_value != null).length
return (
<div className="fixed inset-0 bg-black/40 flex items-center justify-center z-50" onClick={onClose}>
<div className="bg-surface rounded-lg shadow-xl w-3/4 max-w-3xl max-h-[80vh] flex flex-col"
<div className="bg-white rounded-lg shadow-xl w-3/4 max-w-3xl max-h-[80vh] flex flex-col"
onClick={e => e.stopPropagation()}>
<div className="flex items-center justify-between px-5 py-3 border-b border-line-soft">
<span className="text-sm font-medium text-ink-soft">
Pattern results — <span className="text-muted font-normal">{matched}/{rows.length} matched</span>
<div className="flex items-center justify-between px-5 py-3 border-b border-gray-100">
<span className="text-sm font-medium text-gray-700">
Pattern results — <span className="text-gray-500 font-normal">{matched}/{rows.length} matched</span>
</span>
<button onClick={onClose} className="text-muted hover:text-ink-soft text-lg leading-none">✕</button>
<button onClick={onClose} className="text-gray-400 hover:text-gray-600 text-lg leading-none">✕</button>
</div>
<div className="overflow-auto flex-1 px-5 py-3">
<table className="w-full text-xs">
<thead>
<tr className="text-left text-muted border-b border-line-soft">
<tr className="text-left text-gray-400 border-b border-gray-100">
<th className="pb-2 font-medium w-1/2 pr-4">Raw value</th>
<th className="pb-2 font-medium">Result</th>
</tr>
</thead>
<tbody>
{rows.map((r, i) => (
<tr key={i} className="border-t border-line-soft">
<td className="py-1 font-mono text-muted pr-4 break-all">{r.raw_value}</td>
<td className={`py-1 font-mono break-all ${r.extracted_value != null ? 'text-ink' : 'text-muted'}`}>
<tr key={i} className="border-t border-gray-50">
<td className="py-1 font-mono text-gray-400 pr-4 break-all">{r.raw_value}</td>
<td className={`py-1 font-mono break-all ${r.extracted_value != null ? 'text-gray-800' : 'text-gray-300'}`}>
{r.extracted_value != null
? (Array.isArray(r.extracted_value) ? r.extracted_value.join(' · ') : String(r.extracted_value))
: '—'}
@ -67,33 +67,33 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
}, [form.field, form.pattern, form.flags, form.function_type, form.replace_value, source])
return (
<div className="bg-surface border border-line rounded p-4 mb-4">
<h2 className="text-sm font-semibold text-ink-soft mb-3">{editing ? 'Edit rule' : 'New rule'}</h2>
<div className="bg-white border border-gray-200 rounded p-4 mb-4">
<h2 className="text-sm font-semibold text-gray-700 mb-3">{editing ? 'Edit rule' : 'New rule'}</h2>
<form onSubmit={onSubmit} className="space-y-3">
<div className="grid grid-cols-2 gap-3">
<div>
<label className="text-xs text-muted block mb-1">Rule name</label>
<label className="text-xs text-gray-500 block mb-1">Rule name</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.name} onChange={e => setForm(f => ({ ...f, name: e.target.value }))}
placeholder="e.g. First 20"
/>
</div>
<div>
<label className="text-xs text-muted block mb-1">Sequence</label>
<label className="text-xs text-gray-500 block mb-1">Sequence</label>
<input
type="number"
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.sequence} onChange={e => setForm(f => ({ ...f, sequence: parseInt(e.target.value) || 0 }))}
/>
</div>
</div>
<div className="grid grid-cols-2 gap-3">
<div>
<label className="text-xs text-muted block mb-1">Input field</label>
<label className="text-xs text-gray-500 block mb-1">Input field</label>
{fields.length > 0 ? (
<select
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.field} onChange={e => setForm(f => ({ ...f, field: e.target.value }))}
>
<option value="">— select field —</option>
@ -101,34 +101,34 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
</select>
) : (
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.field} onChange={e => setForm(f => ({ ...f, field: e.target.value }))}
placeholder="e.g. description"
/>
)}
</div>
<div>
<label className="text-xs text-muted block mb-1">Output field</label>
<label className="text-xs text-gray-500 block mb-1">Output field</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.output_field} onChange={e => setForm(f => ({ ...f, output_field: e.target.value }))}
placeholder="e.g. merchant"
/>
</div>
</div>
<div>
<label className="text-xs text-muted block mb-1">Pattern (regex)</label>
<label className="text-xs text-gray-500 block mb-1">Pattern (regex)</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
value={form.pattern} onChange={e => setForm(f => ({ ...f, pattern: e.target.value }))}
placeholder="e.g. .{1,20}"
/>
</div>
<div className="grid grid-cols-2 gap-3">
<div>
<label className="text-xs text-muted block mb-1">Function</label>
<label className="text-xs text-gray-500 block mb-1">Function</label>
<select
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.function_type} onChange={e => setForm(f => ({ ...f, function_type: e.target.value }))}
>
<option value="extract">extract</option>
@ -136,16 +136,16 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
</select>
</div>
<div>
<label className="text-xs text-muted block mb-1">Flags</label>
<label className="text-xs text-gray-500 block mb-1">Flags</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
value={form.flags} onChange={e => setForm(f => ({ ...f, flags: e.target.value }))}
placeholder="e.g. i"
/>
</div>
</div>
{form.function_type === 'extract' && (
<label className="flex items-center gap-2 text-xs text-ink-soft cursor-pointer select-none">
<label className="flex items-center gap-2 text-xs text-gray-600 cursor-pointer select-none">
<input
type="checkbox"
checked={!!form.retain}
@ -156,9 +156,9 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
)}
{form.function_type === 'replace' && (
<div>
<label className="text-xs text-muted block mb-1">Replacement string</label>
<label className="text-xs text-gray-500 block mb-1">Replacement string</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-accent"
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
value={form.replace_value} onChange={e => setForm(f => ({ ...f, replace_value: e.target.value }))}
placeholder="e.g. leave blank to delete the match"
/>
@ -166,23 +166,23 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
)}
{/* Live preview */}
{(preview.length > 0 || previewing) && (
<div className="border border-line-soft rounded p-2 bg-raised">
<div className="border border-gray-100 rounded p-2 bg-gray-50">
<div className="flex items-center justify-between mb-1">
<p className="text-xs text-muted">
<p className="text-xs text-gray-400">
{previewing ? 'Testing…' : `${preview.filter(r => r.extracted_value != null).length}/${preview.length} matched`}
</p>
{!previewing && preview.length > 0 && (
<button type="button" onClick={() => setModalOpen(true)}
className="text-xs text-accent hover:text-accent">expand</button>
className="text-xs text-blue-400 hover:text-blue-600">expand</button>
)}
</div>
{!previewing && (
<table className="w-full text-xs">
<tbody>
{preview.slice(0, 5).map((r, i) => (
<tr key={i} className="border-t border-line-soft first:border-0">
<td className="py-0.5 font-mono text-muted truncate max-w-0 w-1/2 pr-3">{r.raw_value}</td>
<td className={`py-0.5 font-mono truncate ${r.extracted_value != null ? 'text-ink' : 'text-muted'}`}>
<tr key={i} className="border-t border-gray-100 first:border-0">
<td className="py-0.5 font-mono text-gray-400 truncate max-w-0 w-1/2 pr-3">{r.raw_value}</td>
<td className={`py-0.5 font-mono truncate ${r.extracted_value != null ? 'text-gray-800' : 'text-gray-300'}`}>
{r.extracted_value != null
? (Array.isArray(r.extracted_value) ? r.extracted_value.join(' · ') : String(r.extracted_value))
: '—'}
@ -197,14 +197,14 @@ function FormPanel({ form, setForm, editing, error, loading, fields, source, onS
{modalOpen && <PreviewModal rows={preview} onClose={() => setModalOpen(false)} />}
{error && <p className="text-xs text-danger">{error}</p>}
{error && <p className="text-xs text-red-500">{error}</p>}
<div className="flex gap-2">
<button type="submit" disabled={loading}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{loading ? 'Saving…' : (editing ? 'Save' : 'Create')}
</button>
<button type="button" onClick={onCancel}
className="text-sm text-muted px-3 py-1.5 rounded hover:bg-raised">
className="text-sm text-gray-500 px-3 py-1.5 rounded hover:bg-gray-100">
Cancel
</button>
</div>
@ -310,12 +310,12 @@ export default function Rules({ source, onStale }) {
}
}
if (!source) return <div className="p-4 sm:p-6 text-sm text-muted">Select a source first.</div>
if (!source) return <div className="p-6 text-sm text-gray-400">Select a source first.</div>
return (
<div className="p-4 sm:p-6 max-w-3xl">
<div className="p-6 max-w-3xl">
<div className="flex items-center justify-between mb-6">
<h1 className="text-xl font-semibold text-ink">Rules — {source}</h1>
<h1 className="text-xl font-semibold text-gray-800">Rules — {source}</h1>
<button onClick={startCreate}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700">
New rule
@ -332,17 +332,17 @@ export default function Rules({ source, onStale }) {
)}
{rules.length === 0 && !creating && (
<p className="text-sm text-muted">No rules yet. Add a regex rule to start extracting values.</p>
<p className="text-sm text-gray-400">No rules yet. Add a regex rule to start extracting values.</p>
)}
<div className="space-y-2">
{rules.map(rule => {
const isExpanded = expanded === rule.id
return (
<div key={rule.id} className="bg-surface border border-line rounded">
<div key={rule.id} className="bg-white border border-gray-200 rounded">
{/* Header — always visible, click to expand/collapse */}
<div
className="flex items-center gap-3 px-4 py-3 cursor-pointer hover:bg-raised select-none"
className="flex items-center gap-3 px-4 py-3 cursor-pointer hover:bg-gray-50 select-none"
onClick={() => {
if (isExpanded) { setExpanded(null); setEditing(null) }
else { setExpanded(rule.id); startEdit(rule) }
@ -350,33 +350,33 @@ export default function Rules({ source, onStale }) {
>
<button
onClick={e => { e.stopPropagation(); handleToggle(rule) }}
className={`w-8 h-4 rounded-full flex-shrink-0 transition-colors ${rule.enabled ? 'bg-blue-500' : 'bg-raised'}`}
className={`w-8 h-4 rounded-full flex-shrink-0 transition-colors ${rule.enabled ? 'bg-blue-500' : 'bg-gray-200'}`}
title={rule.enabled ? 'Disable' : 'Enable'}
/>
<div className="flex-1 min-w-0">
<span className="font-medium text-ink text-sm">{rule.name}</span>
<span className="text-muted text-xs ml-2">seq {rule.sequence}</span>
<span className="font-medium text-gray-800 text-sm">{rule.name}</span>
<span className="text-gray-400 text-xs ml-2">seq {rule.sequence}</span>
{!isExpanded && (
<div className="text-xs text-muted mt-0.5 truncate">
<div className="text-xs text-gray-400 mt-0.5 truncate">
<span className="font-mono">{rule.field}</span>
<span className="mx-1">→</span>
<span className="font-mono bg-raised px-1 rounded">{rule.pattern}</span>
{rule.flags && <span className="text-accent ml-1">/{rule.flags}</span>}
<span className="font-mono bg-gray-50 px-1 rounded">{rule.pattern}</span>
{rule.flags && <span className="text-blue-400 ml-1">/{rule.flags}</span>}
<span className="mx-1">→</span>
<span className="font-mono">{rule.output_field}</span>
{rule.function_type === 'replace' && <span className="ml-1 text-warn">(replace)</span>}
{rule.function_type === 'replace' && <span className="ml-1 text-orange-400">(replace)</span>}
</div>
)}
</div>
<span className="text-xs text-muted flex-shrink-0">{isExpanded ? '▲' : '▼'}</span>
<span className="text-xs text-gray-300 flex-shrink-0">{isExpanded ? '▲' : '▼'}</span>
</div>
{/* Expanded content */}
{isExpanded && (
<div className="border-t border-line-soft">
<div className="border-t border-gray-100">
<div className="px-4 pt-3 pb-1 flex justify-end">
<button onClick={e => { e.stopPropagation(); handleDelete(rule.id) }}
className="text-xs text-danger hover:text-danger">Delete</button>
className="text-xs text-red-400 hover:text-red-600">Delete</button>
</div>
<div className="px-4 pb-4">
<FormPanel

424
ui/src/pages/SourceDetail.jsx
View File

@ -1,424 +0,0 @@
import { useState, useEffect } from 'react'
import { useParams, useNavigate } from 'react-router-dom'
import { api } from '../api'
import Section from '../components/Section.jsx'
import SampleTable from '../components/SampleTable.jsx'
const FIELD_TYPES = ['text', 'numeric', 'date']
export default function SourceDetail({ sources, setSources }) {
const { name: source } = useParams()
const navigate = useNavigate()
const [constraintFields, setConstraintFields] = useState('')
const [globalPicklist, setGlobalPicklist] = useState(true)
const [schemaFields, setSchemaFields] = useState([])
const [stats, setStats] = useState(null)
const [sampleRows, setSampleRows] = useState([])
const [saving, setSaving] = useState(false)
const [reprocessing, setReprocessing] = useState(false)
const [generating, setGenerating] = useState(false)
const [result, setResult] = useState('')
const [error, setError] = useState('')
const [viewName, setViewName] = useState('')
const [availableFields, setAvailableFields] = useState([])
const [fieldSort, setFieldSort] = useState({ col: 'key', dir: 'asc' })
const [bridgeAccounts, setBridgeAccounts] = useState(null)
const [bridgeLoading, setBridgeLoading] = useState(false)
const [bridgeError, setBridgeError] = useState('')
const sourceObj = sources.find(s => s.name === source)
useEffect(() => {
if (!sourceObj) return
setConstraintFields(sourceObj.constraint_fields?.join(', ') || '')
setGlobalPicklist(sourceObj.global_picklist !== false)
setSchemaFields((sourceObj.config?.fields || []).map((f, i) => ({ seq: i + 1, ...f })))
setViewName(sourceObj.config?.fields?.length ? `dfv.${sourceObj.name}` : '')
setResult('')
setError('')
setStats(null)
setAvailableFields([])
setSampleRows([])
setBridgeAccounts(null)
setBridgeError('')
api.getStats(sourceObj.name).then(setStats).catch(() => {})
api.getFields(sourceObj.name).then(setAvailableFields).catch(() => {})
api.getRecords(sourceObj.name, 50).then(rows => setSampleRows(rows.map(r => r.data).filter(Boolean))).catch(() => {})
}, [source, sourceObj?.name])
async function handleSave(e) {
e.preventDefault()
setSaving(true)
setError('')
try {
const constraint_fields = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const fields = [...schemaFields.filter(f => f.name)].sort((a, b) => (a.seq ?? 0) - (b.seq ?? 0))
const config = { ...(sourceObj.config || {}), fields }
await api.updateSource(sourceObj.name, { constraint_fields, config, global_picklist: globalPicklist })
if (fields.length > 0) {
const res = await api.generateView(sourceObj.name)
if (res.success) setViewName(res.view)
}
const updated = await api.getSources()
setSources(updated)
setResult('Saved.')
} catch (err) {
setError(err.message)
} finally {
setSaving(false)
}
}
async function handleGenerateView() {
setGenerating(true)
setResult('')
setError('')
try {
const constraint_fields = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const fields = [...schemaFields.filter(f => f.name)].sort((a, b) => (a.seq ?? 0) - (b.seq ?? 0))
const config = { ...(sourceObj.config || {}), fields }
await api.updateSource(sourceObj.name, { constraint_fields, config, global_picklist: globalPicklist })
const res = await api.generateView(sourceObj.name)
if (res.success) {
setViewName(res.view)
setResult(`View created: ${res.view}`)
} else {
setError(res.error)
}
} catch (err) {
setError(err.message)
} finally {
setGenerating(false)
}
}
async function handleReprocess() {
if (!confirm(`Reprocess all records for "${sourceObj.name}"? This will clear and reapply all transformations.`)) return
setReprocessing(true)
setResult('')
setError('')
try {
const res = await api.reprocess(sourceObj.name)
setResult(`Reprocessed ${res.transformed} records.`)
api.getStats(sourceObj.name).then(setStats).catch(() => {})
} catch (err) {
setError(err.message)
} finally {
setReprocessing(false)
}
}
async function handleDelete() {
if (!confirm(`Delete source "${sourceObj.name}" and all its data?`)) return
try {
await api.deleteSource(sourceObj.name)
setSources(await api.getSources())
navigate('/sources')
} catch (err) {
alert(err.message)
}
}
// The bridge is an external call, so accounts are fetched on demand rather
// than on every visit to this page.
async function loadBridgeAccounts() {
setBridgeLoading(true)
setBridgeError('')
try {
const res = await api.getSimpleFinAccounts()
setBridgeAccounts(res.accounts || [])
if (res.errors?.length) setBridgeError(res.errors.join('; '))
} catch (err) {
setBridgeError(err.message)
} finally {
setBridgeLoading(false)
}
}
// Writes only config — constraint_fields and global_picklist are left NULL so
// update_source keeps whatever is already stored.
async function handleLinkAccount(accountId) {
setError('')
setResult('')
try {
const config = { ...(sourceObj.config || {}) }
if (accountId) {
config.simplefin = { ...(config.simplefin || {}), account_id: accountId }
} else {
delete config.simplefin
}
await api.updateSource(sourceObj.name, { config })
setSources(await api.getSources())
setResult(accountId ? 'Account linked.' : 'Account unlinked.')
} catch (err) {
setError(err.message)
}
}
if (!sourceObj) return <div className="p-4 sm:p-6 text-sm text-muted">Source not found.</div>
return (
<div className="p-4 sm:p-6 max-w-5xl space-y-4">
{stats && (
<div className="flex gap-4 text-xs">
<span className="text-muted"><span className="font-medium text-ink">{stats.total_records}</span> total</span>
<span className="text-muted"><span className="font-medium text-ink">{stats.transformed_records}</span> transformed</span>
<span className="text-muted"><span className="font-medium text-ink">{stats.pending_records}</span> pending</span>
</div>
)}
{/* Bank feed — link this source to a SimpleFIN account */}
<Section
title="Connection"
description="Where this source gets its data. Unlinked sources are filled by CSV upload on the Import page."
>
<div className="flex items-center gap-3 flex-wrap">
{bridgeAccounts === null ? (
<>
<span className="text-xs text-muted font-mono">
{sourceObj.config?.simplefin?.account_id || 'not linked'}
</span>
<button
onClick={loadBridgeAccounts}
disabled={bridgeLoading}
className="text-xs border border-line rounded px-2 py-1 text-ink-soft hover:bg-raised hover:border-line disabled:opacity-50"
>
{bridgeLoading ? 'Loading…' : sourceObj.config?.simplefin?.account_id ? 'Change' : 'Link SimpleFIN account'}
</button>
</>
) : (
<select
value={sourceObj.config?.simplefin?.account_id || ''}
onChange={e => handleLinkAccount(e.target.value)}
className="text-xs border border-line rounded px-2 py-1 bg-surface text-ink-soft"
>
<option value="">Not linked</option>
{bridgeAccounts.map(a => (
<option key={a.id} value={a.id}>
{a.name}{a.organization ? ` — ${a.organization}` : ''}{a.balance ? ` (${a.balance})` : ''}
</option>
))}
</select>
)}
</div>
{bridgeError && <p className="text-xs text-warn mt-1">{bridgeError}</p>}
{/* Dedupe depends on the transaction id being the constraint key */}
{sourceObj.config?.simplefin?.account_id
&& sourceObj.constraint_fields?.join(',') !== 'id' && (
<p className="text-xs text-warn mt-1">
Constraint fields are “{sourceObj.constraint_fields?.join(', ') || 'none'}” — a
bank feed should use “id” so re-syncs don’t duplicate rows.
</p>
)}
</Section>
{/* Unified field table */}
{availableFields.length > 0 && (
<Section
title="Fields and view"
description="Every field seen in this source's records. Tick which identify a row for deduplication, and which become columns in the generated view."
>
<table className="w-full text-xs">
<thead>
<tr className="text-left text-muted border-b border-line-soft">
{[
{ col: 'key', label: 'Key' },
{ col: 'origin', label: 'Origin' },
{ col: 'type', label: 'Type' },
{ col: 'constraint', label: 'Constraint', center: true },
{ col: 'inview', label: 'In view', center: true },
{ col: 'seq', label: 'Seq', center: true },
].map(({ col, label, center }) => (
<th
key={col}
onClick={() => setFieldSort(s => ({ col, dir: s.col === col && s.dir === 'asc' ? 'desc' : 'asc' }))}
className={`pb-1 font-medium cursor-pointer select-none hover:text-ink-soft ${center ? 'text-center' : ''}`}
>
{label}
<span className="ml-1 text-muted">
{fieldSort.col === col ? (fieldSort.dir === 'asc' ? '▲' : '▼') : '⇅'}
</span>
</th>
))}
</tr>
</thead>
<tbody>
{[...availableFields].sort((a, b) => {
const constraintList = constraintFields.split(',').map(s => s.trim())
const aSchema = schemaFields.find(sf => sf.name === a.key)
const bSchema = schemaFields.find(sf => sf.name === b.key)
let av, bv
if (fieldSort.col === 'key') { av = a.key; bv = b.key }
else if (fieldSort.col === 'origin') { av = a.origins.join(','); bv = b.origins.join(',') }
else if (fieldSort.col === 'type') { av = aSchema?.type || ''; bv = bSchema?.type || '' }
else if (fieldSort.col === 'constraint') { av = constraintList.includes(a.key) ? 0 : 1; bv = constraintList.includes(b.key) ? 0 : 1 }
else if (fieldSort.col === 'inview') { av = aSchema ? 0 : 1; bv = bSchema ? 0 : 1 }
else if (fieldSort.col === 'seq') { av = aSchema?.seq ?? 999; bv = bSchema?.seq ?? 999 }
if (av < bv) return fieldSort.dir === 'asc' ? -1 : 1
if (av > bv) return fieldSort.dir === 'asc' ? 1 : -1
return 0
}).map(f => {
const isRaw = f.origins.includes('raw')
const constraintChecked = constraintFields.split(',').map(s => s.trim()).includes(f.key)
const schemaEntry = schemaFields.find(sf => sf.name === f.key)
const inView = !!schemaEntry
return (
<tr key={f.key} className="border-t border-line-soft">
<td className="py-1 font-mono text-ink-soft">{f.key}</td>
<td className="py-1 text-muted">{f.origins.join(', ')}</td>
<td className="py-1">
{inView && (
<div className="flex gap-1 items-center">
<select
className="border border-line rounded px-1 py-0.5 text-xs focus:outline-none focus:border-accent"
value={schemaEntry.type}
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, type: e.target.value } : s)
)}
>
{FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
</select>
<input
className="border border-line rounded px-1 py-0.5 text-xs font-mono w-32 focus:outline-none focus:border-accent"
value={schemaEntry.expression || ''}
placeholder="{field} * {sign}"
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, expression: e.target.value || undefined } : s)
)}
/>
</div>
)}
</td>
<td className="py-1 text-center">
{isRaw && (
<input
type="checkbox"
checked={constraintChecked}
onChange={e => {
const current = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const next = e.target.checked
? [...current, f.key]
: current.filter(k => k !== f.key)
setConstraintFields(next.join(', '))
}}
/>
)}
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={inView}
onChange={e => {
if (e.target.checked) {
const nextSeq = schemaFields.length > 0
? Math.max(...schemaFields.map(s => s.seq ?? 0)) + 1
: 1
setSchemaFields(sf => [...sf, { name: f.key, type: 'text', seq: nextSeq }])
} else {
setSchemaFields(sf => sf.filter(s => s.name !== f.key))
}
}}
/>
</td>
<td className="py-1 text-center">
{inView && (
<input
type="number"
className="w-12 border border-line rounded px-1 py-0.5 text-xs text-center focus:outline-none focus:border-accent"
value={schemaEntry.seq ?? ''}
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, seq: parseInt(e.target.value) || 0 } : s)
)}
/>
)}
</td>
</tr>
)
})}
</tbody>
</table>
<div className="flex items-center gap-3 pt-3 mt-2 border-t border-line-soft flex-wrap">
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer">
<input type="checkbox" checked={globalPicklist} onChange={e => setGlobalPicklist(e.target.checked)} />
Global picklist
</label>
<form onSubmit={handleSave}>
<button type="submit" disabled={saving}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{saving ? 'Saving…' : 'Save'}
</button>
</form>
{schemaFields.length > 0 && (
<>
<button
onClick={handleGenerateView}
disabled={generating}
className="text-xs bg-green-600 text-white px-2 py-1.5 rounded hover:bg-green-700 disabled:opacity-50"
>
{generating ? 'Generating…' : 'Generate view'}
</button>
{viewName && (
<code className="text-xs bg-raised px-2 py-1 rounded text-ink-soft">{viewName}</code>
)}
</>
)}
</div>
</Section>
)}
{/* Save button when no fields loaded yet */}
{availableFields.length === 0 && (
<Section
title="Fields and view"
description="No fields yet — they are discovered from imported records. Import or sync data first."
>
<div className="flex items-center gap-3">
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer">
<input type="checkbox" checked={globalPicklist} onChange={e => setGlobalPicklist(e.target.checked)} />
Global picklist
</label>
<form onSubmit={handleSave}>
<button type="submit" disabled={saving}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{saving ? 'Saving…' : 'Save'}
</button>
</form>
</div>
</Section>
)}
{sampleRows.length > 0 && (
<Section title="Sample rows" description="The most recent imported records, as stored.">
<SampleTable rows={sampleRows} />
</Section>
)}
<Section title="Maintenance">
<div className="flex items-center gap-3">
<button
onClick={handleReprocess}
disabled={reprocessing}
className="text-sm bg-orange-500 text-white px-3 py-1.5 rounded hover:bg-orange-600 disabled:opacity-50"
>
{reprocessing ? 'Reprocessing…' : 'Reprocess all records'}
</button>
<span className="text-xs text-muted">Clears and reruns all transformation rules</span>
</div>
</Section>
{result && <p className="text-xs text-ok">{result}</p>}
{error && <p className="text-xs text-danger">{error}</p>}
<Section title="Delete source" description="Removes the source and every record, rule, and mapping belonging to it.">
<button onClick={handleDelete}
className="text-sm border border-danger-line text-danger px-3 py-1.5 rounded hover:bg-danger-soft hover:border-danger-line">
Delete source
</button>
</Section>
</div>
)
}

388
ui/src/pages/SourceList.jsx
View File

@ -1,388 +0,0 @@
import { useState, useRef } from 'react'
import { useNavigate } from 'react-router-dom'
import { api } from '../api'
import SampleTable from '../components/SampleTable.jsx'
const FIELD_TYPES = ['text', 'numeric', 'date']
// Ticked into the view by default when a bank feed sample contains them
const FEED_DEFAULT_VIEW = ['date', 'description', 'payee', 'amount']
export default function SourceList({ sources, setSources, setSource }) {
const navigate = useNavigate()
const [creating, setCreating] = useState(false)
const [form, setForm] = useState({ name: '', constraint_fields: '', fields: [], schema: [], importSample: true })
const [createError, setCreateError] = useState('')
const [createLoading, setCreateLoading] = useState(false)
const [csvFileName, setCsvFileName] = useState('')
const [bridgeAccounts, setBridgeAccounts] = useState(null)
const [bridgeLoading, setBridgeLoading] = useState(false)
const [bridgeError, setBridgeError] = useState('')
const [sampleInfo, setSampleInfo] = useState(null)
const fileRef = useRef()
async function loadBridgeAccounts() {
setBridgeLoading(true)
setBridgeError('')
try {
const res = await api.getSimpleFinAccounts()
setBridgeAccounts(res.accounts || [])
if (res.errors?.length) setBridgeError(res.errors.join('; '))
} catch (err) {
setBridgeError(err.message)
} finally {
setBridgeLoading(false)
}
}
// Writes only config — constraint_fields and global_picklist are left NULL so
// update_source keeps whatever is already stored.
async function handleSelectFeedAccount(accountId) {
if (!accountId) {
// Clearing the feed only resets fields we populated, not a loaded CSV
setForm(f => csvFileName ? { ...f, simplefin_account_id: '' } : {
...f, simplefin_account_id: '', fields: [], schema: [], constraint_fields: '', sampleRows: [],
})
setSampleInfo(null)
return
}
setForm(f => ({ ...f, simplefin_account_id: accountId }))
setBridgeLoading(true)
setBridgeError('')
try {
const res = await api.getSimpleFinSample(accountId)
const names = res.fields.map(f => f.name)
setSampleInfo({ fetched: res.fetched, fields: res.fields.length })
if (res.errors?.length) setBridgeError(res.errors.join('; '))
setForm(f => ({
...f,
fields: res.fields,
sampleRows: res.sampleRows || [],
schema: FEED_DEFAULT_VIEW.filter(n => names.includes(n)).map((name, i) => ({
name, type: res.fields.find(sf => sf.name === name).type, seq: i + 1,
})),
// Only default the constraint if the sample actually has an id
constraint_fields: f.constraint_fields || (names.includes('id') ? 'id' : ''),
}))
} catch (err) {
setBridgeError(err.message)
} finally {
setBridgeLoading(false)
}
}
async function handleSuggest(e) {
const file = e.target.files[0]
if (!file) return
setCsvFileName(file.name)
try {
const suggestion = await api.suggestSource(file)
setForm(f => ({
...f,
fields: suggestion.fields,
constraint_fields: '',
schema: suggestion.fields.map(f => ({ name: f.name, type: f.type, seq: suggestion.fields.indexOf(f) + 1 })),
sampleRows: suggestion.sampleRows || []
}))
} catch (err) {
setCreateError(err.message)
}
}
async function handleCreate(e) {
e.preventDefault()
setCreateError('')
const constraintArr = form.constraint_fields.split(',').map(s => s.trim()).filter(Boolean)
if (!form.name || constraintArr.length === 0) {
setCreateError('Name and at least one constraint field required')
return
}
setCreateLoading(true)
try {
const config = form.schema.length > 0 ? { fields: form.schema } : {}
if (form.simplefin_account_id) {
config.simplefin = { account_id: form.simplefin_account_id }
}
await api.createSource({ name: form.name, constraint_fields: constraintArr, config, global_picklist: form.global_picklist !== false })
if (form.schema.length > 0) {
await api.generateView(form.name)
}
if (form.importSample && fileRef.current?.files[0]) {
await api.importCSV(form.name, fileRef.current.files[0])
}
const updated = await api.getSources()
setSources(updated)
setSource(form.name)
setForm({ name: '', constraint_fields: '', fields: [], schema: [], importSample: true, simplefin_account_id: '' })
setCreating(false)
} catch (err) {
setCreateError(err.message)
} finally {
setCreateLoading(false)
}
}
return (
<div className="p-4 sm:p-6 max-w-5xl">
<div className="flex items-center justify-between mb-6">
<h1 className="text-xl font-semibold text-ink">Sources</h1>
{!creating && (
<button
onClick={() => { setCreating(true); setCreateError('') }}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700"
>
New source
</button>
)}
</div>
{!creating && sources.length === 0 && (
<p className="text-sm text-muted">No sources yet. Create one to get started.</p>
)}
{!creating && sources.length > 0 && (
<div className="bg-surface border border-line rounded divide-y divide-line-soft">
{sources.map(s => (
<button
key={s.name}
onClick={() => { setSource(s.name); navigate(`/sources/${encodeURIComponent(s.name)}`) }}
className="w-full text-left px-4 py-3 hover:bg-raised flex items-center gap-3"
>
<span className="text-sm font-medium text-ink flex-1">{s.name}</span>
{s.config?.simplefin?.account_id && (
<span className="text-xs bg-accent-soft text-accent border border-accent-line rounded px-1.5 py-0.5">
bank feed
</span>
)}
<span className="text-xs text-muted">
{(s.constraint_fields || []).join(', ') || 'no constraint'}
</span>
<span className="text-muted">›</span>
</button>
))}
</div>
)}
{creating && (
<div className="bg-surface border border-line rounded p-4">
<h2 className="text-sm font-semibold text-ink-soft mb-3">New source</h2>
<div className="mb-4">
<input type="file" accept=".csv" ref={fileRef} onChange={handleSuggest} className="hidden" />
<button
type="button"
onClick={() => fileRef.current?.click()}
className="text-sm border border-line rounded px-3 py-1.5 text-ink-soft hover:bg-raised hover:border-line"
>
{csvFileName || 'Choose CSV…'}
</button>
</div>
<form onSubmit={handleCreate} className="space-y-3">
<div>
<label className="text-xs text-muted block mb-1">Source name</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
value={form.name}
onChange={e => setForm(f => ({ ...f, name: e.target.value }))}
placeholder="e.g. chase, dcard"
/>
</div>
{/* Bank feed — optional; picking an account defaults the constraint
field to the transaction id, which is what dedupe needs */}
<div>
<label className="text-xs text-muted block mb-1">Bank feed (optional)</label>
<div className="flex items-center gap-3 flex-wrap">
{bridgeAccounts === null ? (
<button
type="button"
onClick={loadBridgeAccounts}
disabled={bridgeLoading}
className="text-sm border border-line rounded px-3 py-1.5 text-ink-soft hover:bg-raised hover:border-line disabled:opacity-50"
>
{bridgeLoading ? 'Loading…' : 'Link SimpleFIN account…'}
</button>
) : (
<select
value={form.simplefin_account_id || ''}
onChange={e => handleSelectFeedAccount(e.target.value)}
className="text-sm border border-line rounded px-3 py-1.5 bg-surface text-ink-soft"
>
<option value="">No bank feed — CSV import</option>
{bridgeAccounts.map(a => (
<option key={a.id} value={a.id}>
{a.name}{a.organization ? ` — ${a.organization}` : ''}{a.balance ? ` (${a.balance})` : ''}
</option>
))}
</select>
)}
</div>
{bridgeError && <p className="text-xs text-warn mt-1">{bridgeError}</p>}
{form.simplefin_account_id && sampleInfo && (
<div className="mt-2 bg-accent-soft border border-accent-line rounded p-3 text-xs text-ink-soft space-y-1">
<p>
Read {sampleInfo.fetched} transaction{sampleInfo.fetched === 1 ? '' : 's'} from this
account and found {sampleInfo.fields} field{sampleInfo.fields === 1 ? '' : 's'}.
The table below lists what this account actually returns — fields it never sends
won&rsquo;t appear.
</p>
{form.constraint_fields === 'id' && (
<p>
<span className="font-mono text-ink-soft">id</span> is checked as the constraint
field because it is SimpleFIN&rsquo;s own transaction identifier. Syncs pull an
overlapping window of days, so the same transaction arrives more than once —
matching on <span className="font-mono text-ink-soft">id</span> skips the repeats
while still keeping genuinely separate charges that share a date, amount, and
description.
</p>
)}
{sampleInfo.fetched === 0 && (
<p className="text-warn">
No transactions came back, so there was nothing to infer fields from. Sync first,
then set the fields up here.
</p>
)}
</div>
)}
</div>
{form.fields.length > 0 && (
<div className="pt-2 border-t border-line-soft space-y-2">
<table className="w-full text-xs">
<thead>
<tr className="text-left text-muted border-b border-line-soft">
<th className="pb-1 font-medium">Key</th>
<th className="pb-1 font-medium">Type</th>
<th className="pb-1 font-medium text-center">Constraint</th>
<th className="pb-1 font-medium text-center">In view</th>
<th className="pb-1 font-medium text-center">Seq</th>
</tr>
</thead>
<tbody>
{form.fields.map(f => {
const schemaEntry = form.schema.find(s => s.name === f.name)
const inView = !!schemaEntry
const currentType = schemaEntry?.type || f.type
return (
<tr key={f.name} className="border-t border-line-soft">
<td className="py-1 font-mono text-ink-soft">{f.name}</td>
<td className="py-1">
{inView && (
<select
className="border border-line rounded px-1 py-0.5 text-xs focus:outline-none focus:border-accent"
value={currentType}
onChange={e => setForm(ff => ({
...ff,
schema: ff.schema.map(s => s.name === f.name ? { ...s, type: e.target.value } : s)
}))}
>
{FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
</select>
)}
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={form.constraint_fields.split(',').map(s => s.trim()).includes(f.name)}
onChange={e => {
const current = form.constraint_fields.split(',').map(s => s.trim()).filter(Boolean)
const next = e.target.checked
? [...current, f.name]
: current.filter(n => n !== f.name)
setForm(ff => ({ ...ff, constraint_fields: next.join(', ') }))
}}
/>
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={inView}
onChange={e => {
if (e.target.checked) {
const nextSeq = form.schema.length > 0
? Math.max(...form.schema.map(s => s.seq ?? 0)) + 1
: 1
setForm(ff => ({ ...ff, schema: [...ff.schema, { name: f.name, type: f.type, seq: nextSeq }] }))
} else {
setForm(ff => ({ ...ff, schema: ff.schema.filter(s => s.name !== f.name) }))
}
}}
/>
</td>
<td className="py-1 text-center">
{inView && (
<input
type="number"
className="w-12 border border-line rounded px-1 py-0.5 text-xs text-center focus:outline-none focus:border-accent"
value={schemaEntry.seq ?? ''}
onChange={e => setForm(ff => ({
...ff,
schema: ff.schema.map(s => s.name === f.name ? { ...s, seq: parseInt(e.target.value) || 0 } : s)
}))}
/>
)}
</td>
</tr>
)
})}
</tbody>
</table>
<SampleTable rows={form.sampleRows || []} />
</div>
)}
{form.fields.length === 0 && (
<div>
<label className="text-xs text-muted block mb-1">Constraint fields (comma-separated)</label>
<input
className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
value={form.constraint_fields}
onChange={e => setForm(f => ({ ...f, constraint_fields: e.target.value }))}
placeholder="e.g. date, amount, description"
/>
</div>
)}
<div className="flex gap-4">
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer">
<input
type="checkbox"
checked={form.global_picklist !== false}
onChange={e => setForm(f => ({ ...f, global_picklist: e.target.checked }))}
/>
Global picklist
</label>
{form.fields.length > 0 && (
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer">
<input
type="checkbox"
checked={form.importSample !== false}
onChange={e => setForm(f => ({ ...f, importSample: e.target.checked }))}
/>
Import sample data
</label>
)}
</div>
{createError && <p className="text-xs text-danger">{createError}</p>}
<div className="flex gap-2">
<button type="submit" disabled={createLoading}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{createLoading ? 'Creating…' : 'Create'}
</button>
<button type="button"
onClick={() => { setCreating(false); setCreateError(''); setForm({ name: '', constraint_fields: '', fields: [], schema: [] }) }}
className="text-sm text-muted px-3 py-1.5 rounded hover:bg-raised">
Cancel
</button>
</div>
</form>
</div>
)}
</div>
)
}

590
ui/src/pages/Sources.jsx Normal file
View File

@ -0,0 +1,590 @@
import { useState, useEffect, useRef } from 'react'
import { useSearchParams } from 'react-router-dom'
import { api } from '../api'
const FIELD_TYPES = ['text', 'numeric', 'date']
function SampleTable({ rows }) {
if (!rows || rows.length === 0) return null
const cols = Object.keys(rows[0])
return (
<div className="overflow-auto border border-gray-100 rounded bg-gray-50 max-h-36">
<table className="text-xs w-full">
<thead>
<tr className="text-left text-gray-400 border-b border-gray-100 bg-gray-50 sticky top-0">
{cols.map(c => <th key={c} className="px-2 py-1 font-medium whitespace-nowrap">{c}</th>)}
</tr>
</thead>
<tbody>
{rows.map((row, i) => (
<tr key={i} className="border-t border-gray-100">
{cols.map(c => (
<td key={c} className="px-2 py-1 whitespace-nowrap text-gray-600 max-w-32 truncate font-mono">
{row[c] == null ? <span className="text-gray-300">—</span> : String(row[c])}
</td>
))}
</tr>
))}
</tbody>
</table>
</div>
)
}
export default function Sources({ source, sources, setSources, setSource }) {
const [constraintFields, setConstraintFields] = useState('')
const [globalPicklist, setGlobalPicklist] = useState(true)
const [schemaFields, setSchemaFields] = useState([])
const [stats, setStats] = useState(null)
const [sampleRows, setSampleRows] = useState([])
const [saving, setSaving] = useState(false)
const [reprocessing, setReprocessing] = useState(false)
const [generating, setGenerating] = useState(false)
const [result, setResult] = useState('')
const [error, setError] = useState('')
const [viewName, setViewName] = useState('')
const [availableFields, setAvailableFields] = useState([])
const [fieldSort, setFieldSort] = useState({ col: 'key', dir: 'asc' })
const [creating, setCreating] = useState(false)
const [form, setForm] = useState({ name: '', constraint_fields: '', fields: [], schema: [], importSample: true })
const [createError, setCreateError] = useState('')
const [createLoading, setCreateLoading] = useState(false)
const [csvFileName, setCsvFileName] = useState('')
const fileRef = useRef()
const [searchParams, setSearchParams] = useSearchParams()
const sourceObj = sources.find(s => s.name === source)
useEffect(() => {
if (searchParams.get('new') === '1') {
setCreating(true)
setSearchParams({})
}
}, [searchParams])
useEffect(() => {
if (!sourceObj) return
setConstraintFields(sourceObj.constraint_fields?.join(', ') || '')
setGlobalPicklist(sourceObj.global_picklist !== false)
setSchemaFields((sourceObj.config?.fields || []).map((f, i) => ({ seq: i + 1, ...f })))
setViewName(sourceObj.config?.fields?.length ? `dfv.${sourceObj.name}` : '')
setResult('')
setError('')
setStats(null)
setAvailableFields([])
setSampleRows([])
api.getStats(sourceObj.name).then(setStats).catch(() => {})
api.getFields(sourceObj.name).then(setAvailableFields).catch(() => {})
api.getRecords(sourceObj.name, 50).then(rows => setSampleRows(rows.map(r => r.data).filter(Boolean))).catch(() => {})
}, [source, sourceObj?.name])
async function handleSave(e) {
e.preventDefault()
setSaving(true)
setError('')
try {
const constraint_fields = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const fields = [...schemaFields.filter(f => f.name)].sort((a, b) => (a.seq ?? 0) - (b.seq ?? 0))
const config = { ...(sourceObj.config || {}), fields }
await api.updateSource(sourceObj.name, { constraint_fields, config, global_picklist: globalPicklist })
if (fields.length > 0) {
const res = await api.generateView(sourceObj.name)
if (res.success) setViewName(res.view)
}
const updated = await api.getSources()
setSources(updated)
setResult('Saved.')
} catch (err) {
setError(err.message)
} finally {
setSaving(false)
}
}
async function handleGenerateView() {
setGenerating(true)
setResult('')
setError('')
try {
const constraint_fields = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const fields = [...schemaFields.filter(f => f.name)].sort((a, b) => (a.seq ?? 0) - (b.seq ?? 0))
const config = { ...(sourceObj.config || {}), fields }
await api.updateSource(sourceObj.name, { constraint_fields, config, global_picklist: globalPicklist })
const res = await api.generateView(sourceObj.name)
if (res.success) {
setViewName(res.view)
setResult(`View created: ${res.view}`)
} else {
setError(res.error)
}
} catch (err) {
setError(err.message)
} finally {
setGenerating(false)
}
}
async function handleReprocess() {
if (!confirm(`Reprocess all records for "${sourceObj.name}"? This will clear and reapply all transformations.`)) return
setReprocessing(true)
setResult('')
setError('')
try {
const res = await api.reprocess(sourceObj.name)
setResult(`Reprocessed ${res.transformed} records.`)
api.getStats(sourceObj.name).then(setStats).catch(() => {})
} catch (err) {
setError(err.message)
} finally {
setReprocessing(false)
}
}
async function handleDelete() {
if (!confirm(`Delete source "${sourceObj.name}" and all its data?`)) return
try {
await api.deleteSource(sourceObj.name)
const updated = await api.getSources()
setSources(updated)
if (updated.length > 0) setSource(updated[0].name)
else setSource('')
} catch (err) {
alert(err.message)
}
}
async function handleSuggest(e) {
const file = e.target.files[0]
if (!file) return
setCsvFileName(file.name)
try {
const suggestion = await api.suggestSource(file)
setForm(f => ({
...f,
fields: suggestion.fields,
constraint_fields: '',
schema: suggestion.fields.map(f => ({ name: f.name, type: f.type, seq: suggestion.fields.indexOf(f) + 1 })),
sampleRows: suggestion.sampleRows || []
}))
} catch (err) {
setCreateError(err.message)
}
}
async function handleCreate(e) {
e.preventDefault()
setCreateError('')
const constraintArr = form.constraint_fields.split(',').map(s => s.trim()).filter(Boolean)
if (!form.name || constraintArr.length === 0) {
setCreateError('Name and at least one constraint field required')
return
}
setCreateLoading(true)
try {
const config = form.schema.length > 0 ? { fields: form.schema } : {}
await api.createSource({ name: form.name, constraint_fields: constraintArr, config, global_picklist: form.global_picklist !== false })
if (form.schema.length > 0) {
await api.generateView(form.name)
}
if (form.importSample && fileRef.current?.files[0]) {
await api.importCSV(form.name, fileRef.current.files[0])
}
const updated = await api.getSources()
setSources(updated)
setSource(form.name)
setForm({ name: '', constraint_fields: '', fields: [], schema: [], importSample: true })
setCreating(false)
} catch (err) {
setCreateError(err.message)
} finally {
setCreateLoading(false)
}
}
return (
<div className="p-6 max-w-5xl">
<div className="flex items-center justify-between mb-6">
<h1 className="text-xl font-semibold text-gray-800">
{sourceObj ? sourceObj.name : 'Sources'}
</h1>
<button
onClick={() => { setCreating(true); setCreateError('') }}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700"
>
New source
</button>
</div>
{/* No source selected */}
{!sourceObj && !creating && (
<p className="text-sm text-gray-400">No sources yet. Create one to get started.</p>
)}
{/* Source detail */}
{sourceObj && !creating && (
<div className="space-y-4">
{/* Stats */}
{stats && (
<div className="flex gap-4 text-xs">
<span className="text-gray-500"><span className="font-medium text-gray-800">{stats.total_records}</span> total</span>
<span className="text-gray-500"><span className="font-medium text-gray-800">{stats.transformed_records}</span> transformed</span>
<span className="text-gray-500"><span className="font-medium text-gray-800">{stats.pending_records}</span> pending</span>
</div>
)}
{/* Unified field table */}
{availableFields.length > 0 && (
<div className="pt-2 border-t border-gray-100 space-y-2">
<table className="w-full text-xs">
<thead>
<tr className="text-left text-gray-400 border-b border-gray-100">
{[
{ col: 'key', label: 'Key' },
{ col: 'origin', label: 'Origin' },
{ col: 'type', label: 'Type' },
{ col: 'constraint', label: 'Constraint', center: true },
{ col: 'inview', label: 'In view', center: true },
{ col: 'seq', label: 'Seq', center: true },
].map(({ col, label, center }) => (
<th
key={col}
onClick={() => setFieldSort(s => ({ col, dir: s.col === col && s.dir === 'asc' ? 'desc' : 'asc' }))}
className={`pb-1 font-medium cursor-pointer select-none hover:text-gray-600 ${center ? 'text-center' : ''}`}
>
{label}
<span className="ml-1 text-gray-300">
{fieldSort.col === col ? (fieldSort.dir === 'asc' ? '▲' : '▼') : '⇅'}
</span>
</th>
))}
</tr>
</thead>
<tbody>
{[...availableFields].sort((a, b) => {
const constraintList = constraintFields.split(',').map(s => s.trim())
const aSchema = schemaFields.find(sf => sf.name === a.key)
const bSchema = schemaFields.find(sf => sf.name === b.key)
let av, bv
if (fieldSort.col === 'key') { av = a.key; bv = b.key }
else if (fieldSort.col === 'origin') { av = a.origins.join(','); bv = b.origins.join(',') }
else if (fieldSort.col === 'type') { av = aSchema?.type || ''; bv = bSchema?.type || '' }
else if (fieldSort.col === 'constraint') { av = constraintList.includes(a.key) ? 0 : 1; bv = constraintList.includes(b.key) ? 0 : 1 }
else if (fieldSort.col === 'inview') { av = aSchema ? 0 : 1; bv = bSchema ? 0 : 1 }
else if (fieldSort.col === 'seq') { av = aSchema?.seq ?? 999; bv = bSchema?.seq ?? 999 }
if (av < bv) return fieldSort.dir === 'asc' ? -1 : 1
if (av > bv) return fieldSort.dir === 'asc' ? 1 : -1
return 0
}).map(f => {
const isRaw = f.origins.includes('raw')
const constraintChecked = constraintFields.split(',').map(s => s.trim()).includes(f.key)
const schemaEntry = schemaFields.find(sf => sf.name === f.key)
const inView = !!schemaEntry
return (
<tr key={f.key} className="border-t border-gray-50">
<td className="py-1 font-mono text-gray-700">{f.key}</td>
<td className="py-1 text-gray-400">{f.origins.join(', ')}</td>
<td className="py-1">
{inView && (
<div className="flex gap-1 items-center">
<select
className="border border-gray-200 rounded px-1 py-0.5 text-xs focus:outline-none focus:border-blue-400"
value={schemaEntry.type}
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, type: e.target.value } : s)
)}
>
{FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
</select>
<input
className="border border-gray-200 rounded px-1 py-0.5 text-xs font-mono w-32 focus:outline-none focus:border-blue-400"
value={schemaEntry.expression || ''}
placeholder="{field} * {sign}"
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, expression: e.target.value || undefined } : s)
)}
/>
</div>
)}
</td>
<td className="py-1 text-center">
{isRaw && (
<input
type="checkbox"
checked={constraintChecked}
onChange={e => {
const current = constraintFields.split(',').map(s => s.trim()).filter(Boolean)
const next = e.target.checked
? [...current, f.key]
: current.filter(k => k !== f.key)
setConstraintFields(next.join(', '))
}}
/>
)}
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={inView}
onChange={e => {
if (e.target.checked) {
const nextSeq = schemaFields.length > 0
? Math.max(...schemaFields.map(s => s.seq ?? 0)) + 1
: 1
setSchemaFields(sf => [...sf, { name: f.key, type: 'text', seq: nextSeq }])
} else {
setSchemaFields(sf => sf.filter(s => s.name !== f.key))
}
}}
/>
</td>
<td className="py-1 text-center">
{inView && (
<input
type="number"
className="w-12 border border-gray-200 rounded px-1 py-0.5 text-xs text-center focus:outline-none focus:border-blue-400"
value={schemaEntry.seq ?? ''}
onChange={e => setSchemaFields(sf =>
sf.map(s => s.name === f.key ? { ...s, seq: parseInt(e.target.value) || 0 } : s)
)}
/>
)}
</td>
</tr>
)
})}
</tbody>
</table>
<div className="flex items-center gap-3 pt-1 flex-wrap">
<label className="flex items-center gap-1.5 text-xs text-gray-500 cursor-pointer">
<input type="checkbox" checked={globalPicklist} onChange={e => setGlobalPicklist(e.target.checked)} />
Global picklist
</label>
<form onSubmit={handleSave}>
<button type="submit" disabled={saving}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{saving ? 'Saving…' : 'Save'}
</button>
</form>
{schemaFields.length > 0 && (
<>
<button
onClick={handleGenerateView}
disabled={generating}
className="text-xs bg-green-600 text-white px-2 py-1.5 rounded hover:bg-green-700 disabled:opacity-50"
>
{generating ? 'Generating…' : 'Generate view'}
</button>
{viewName && (
<code className="text-xs bg-gray-100 px-2 py-1 rounded text-gray-600">{viewName}</code>
)}
</>
)}
</div>
<SampleTable rows={sampleRows} />
</div>
)}
{/* Save button when no fields loaded yet */}
{availableFields.length === 0 && (
<div className="flex items-center gap-3">
<label className="flex items-center gap-1.5 text-xs text-gray-500 cursor-pointer">
<input type="checkbox" checked={globalPicklist} onChange={e => setGlobalPicklist(e.target.checked)} />
Global picklist
</label>
<form onSubmit={handleSave}>
<button type="submit" disabled={saving}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{saving ? 'Saving…' : 'Save'}
</button>
</form>
</div>
)}
{/* Reprocess */}
<div className="flex items-center gap-3 pt-2 border-t border-gray-100">
<button
onClick={handleReprocess}
disabled={reprocessing}
className="text-sm bg-orange-500 text-white px-3 py-1.5 rounded hover:bg-orange-600 disabled:opacity-50"
>
{reprocessing ? 'Reprocessing…' : 'Reprocess all records'}
</button>
<span className="text-xs text-gray-400">Clears and reruns all transformation rules</span>
</div>
{result && <p className="text-xs text-green-600">{result}</p>}
{error && <p className="text-xs text-red-500">{error}</p>}
<div className="pt-2 border-t border-gray-100">
<button onClick={handleDelete} className="text-xs text-red-400 hover:text-red-600">
Delete source
</button>
</div>
</div>
)}
{/* Create form */}
{creating && (
<div className="bg-white border border-gray-200 rounded p-4">
<h2 className="text-sm font-semibold text-gray-700 mb-3">New source</h2>
<div className="mb-4">
<input type="file" accept=".csv" ref={fileRef} onChange={handleSuggest} className="hidden" />
<button
type="button"
onClick={() => fileRef.current?.click()}
className="text-sm border border-gray-300 rounded px-3 py-1.5 text-gray-600 hover:bg-gray-50 hover:border-gray-400"
>
{csvFileName || 'Choose CSV…'}
</button>
</div>
<form onSubmit={handleCreate} className="space-y-3">
<div>
<label className="text-xs text-gray-500 block mb-1">Source name</label>
<input
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.name}
onChange={e => setForm(f => ({ ...f, name: e.target.value }))}
placeholder="e.g. chase, dcard"
/>
</div>
{form.fields.length > 0 && (
<div className="pt-2 border-t border-gray-100 space-y-2">
<table className="w-full text-xs">
<thead>
<tr className="text-left text-gray-400 border-b border-gray-100">
<th className="pb-1 font-medium">Key</th>
<th className="pb-1 font-medium">Type</th>
<th className="pb-1 font-medium text-center">Constraint</th>
<th className="pb-1 font-medium text-center">In view</th>
<th className="pb-1 font-medium text-center">Seq</th>
</tr>
</thead>
<tbody>
{form.fields.map(f => {
const schemaEntry = form.schema.find(s => s.name === f.name)
const inView = !!schemaEntry
const currentType = schemaEntry?.type || f.type
return (
<tr key={f.name} className="border-t border-gray-50">
<td className="py-1 font-mono text-gray-700">{f.name}</td>
<td className="py-1">
{inView && (
<select
className="border border-gray-200 rounded px-1 py-0.5 text-xs focus:outline-none focus:border-blue-400"
value={currentType}
onChange={e => setForm(ff => ({
...ff,
schema: ff.schema.map(s => s.name === f.name ? { ...s, type: e.target.value } : s)
}))}
>
{FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
</select>
)}
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={form.constraint_fields.split(',').map(s => s.trim()).includes(f.name)}
onChange={e => {
const current = form.constraint_fields.split(',').map(s => s.trim()).filter(Boolean)
const next = e.target.checked
? [...current, f.name]
: current.filter(n => n !== f.name)
setForm(ff => ({ ...ff, constraint_fields: next.join(', ') }))
}}
/>
</td>
<td className="py-1 text-center">
<input
type="checkbox"
checked={inView}
onChange={e => {
if (e.target.checked) {
const nextSeq = form.schema.length > 0
? Math.max(...form.schema.map(s => s.seq ?? 0)) + 1
: 1
setForm(ff => ({ ...ff, schema: [...ff.schema, { name: f.name, type: f.type, seq: nextSeq }] }))
} else {
setForm(ff => ({ ...ff, schema: ff.schema.filter(s => s.name !== f.name) }))
}
}}
/>
</td>
<td className="py-1 text-center">
{inView && (
<input
type="number"
className="w-12 border border-gray-200 rounded px-1 py-0.5 text-xs text-center focus:outline-none focus:border-blue-400"
value={schemaEntry.seq ?? ''}
onChange={e => setForm(ff => ({
...ff,
schema: ff.schema.map(s => s.name === f.name ? { ...s, seq: parseInt(e.target.value) || 0 } : s)
}))}
/>
)}
</td>
</tr>
)
})}
</tbody>
</table>
<SampleTable rows={form.sampleRows || []} />
</div>
)}
{form.fields.length === 0 && (
<div>
<label className="text-xs text-gray-500 block mb-1">Constraint fields (comma-separated)</label>
<input
className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={form.constraint_fields}
onChange={e => setForm(f => ({ ...f, constraint_fields: e.target.value }))}
placeholder="e.g. date, amount, description"
/>
</div>
)}
<div className="flex gap-4">
<label className="flex items-center gap-1.5 text-xs text-gray-500 cursor-pointer">
<input
type="checkbox"
checked={form.global_picklist !== false}
onChange={e => setForm(f => ({ ...f, global_picklist: e.target.checked }))}
/>
Global picklist
</label>
{form.fields.length > 0 && (
<label className="flex items-center gap-1.5 text-xs text-gray-500 cursor-pointer">
<input
type="checkbox"
checked={form.importSample !== false}
onChange={e => setForm(f => ({ ...f, importSample: e.target.checked }))}
/>
Import sample data
</label>
)}
</div>
{createError && <p className="text-xs text-red-500">{createError}</p>}
<div className="flex gap-2">
<button type="submit" disabled={createLoading}
className="text-sm bg-blue-600 text-white px-3 py-1.5 rounded hover:bg-blue-700 disabled:opacity-50">
{createLoading ? 'Creating…' : 'Create'}
</button>
<button type="button"
onClick={() => { setCreating(false); setCreateError(''); setForm({ name: '', constraint_fields: '', fields: [], schema: [] }) }}
className="text-sm text-gray-500 px-3 py-1.5 rounded hover:bg-gray-100">
Cancel
</button>
</div>
</form>
</div>
)}
</div>
)
}

186
ui/src/pages/Stacks.jsx
View File

@ -1,4 +1,3 @@
import { Link } from 'react-router-dom'
import { useState, useEffect, useRef } from 'react'
import { api } from '../api'
import { format as formatSql } from 'sql-formatter'
@ -54,54 +53,54 @@ function CalibrateModal({ stack, sourceName, currentOffset, onClose, onApply })
return (
<div className="fixed inset-0 bg-black/40 flex items-center justify-center z-50" onMouseDown={e => { if (e.target === e.currentTarget) onClose() }}>
<div className="bg-surface rounded-lg shadow-xl w-[420px] p-5" onClick={e => e.stopPropagation()}>
<div className="bg-white rounded-lg shadow-xl w-[420px] p-5" onClick={e => e.stopPropagation()}>
<div className="flex items-center justify-between mb-4">
<span className="text-sm font-semibold text-ink-soft">Calibrate — {sourceName}</span>
<button onClick={onClose} className="text-muted hover:text-ink-soft">✕</button>
<span className="text-sm font-semibold text-gray-700">Calibrate — {sourceName}</span>
<button onClick={onClose} className="text-gray-400 hover:text-gray-600">✕</button>
</div>
{/* Date */}
<div className="mb-4">
<label className="text-xs text-muted block mb-1">As-of date</label>
<input type="date" className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
<label className="text-xs text-gray-500 block mb-1">As-of date</label>
<input type="date" className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={asOf} onChange={e => setAsOf(e.target.value)} />
</div>
{/* Reconciliation table */}
<div className="bg-raised rounded border border-line mb-4 text-sm">
<div className="flex items-center justify-between px-3 py-2 border-b border-line">
<span className="text-muted text-xs">Data sum at date</span>
<span className="font-mono text-ink-soft">
{loading ? <span className="text-muted">…</span> : computed !== null ? fmt(computed) : <span className="text-muted">—</span>}
<div className="bg-gray-50 rounded border border-gray-200 mb-4 text-sm">
<div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
<span className="text-gray-500 text-xs">Data sum at date</span>
<span className="font-mono text-gray-700">
{loading ? <span className="text-gray-300">…</span> : computed !== null ? fmt(computed) : <span className="text-gray-300">—</span>}
</span>
</div>
<div className="flex items-center justify-between px-3 py-2 border-b border-line">
<span className="text-muted text-xs">Known balance</span>
<div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
<span className="text-gray-500 text-xs">Known balance</span>
<input
type="number" step="0.01"
className="font-mono text-right bg-transparent border-0 focus:outline-none w-36 text-sm text-ink-soft placeholder-gray-300"
className="font-mono text-right bg-transparent border-0 focus:outline-none w-36 text-sm text-gray-700 placeholder-gray-300"
placeholder="enter balance"
value={known} onChange={e => setKnown(e.target.value)}
/>
</div>
<div className="flex items-center justify-between px-3 py-2 border-b border-line">
<span className="text-muted text-xs">Current offset</span>
<span className="font-mono text-muted">{fmt(currentOffset ?? 0)}</span>
<div className="flex items-center justify-between px-3 py-2 border-b border-gray-200">
<span className="text-gray-500 text-xs">Current offset</span>
<span className="font-mono text-gray-400">{fmt(currentOffset ?? 0)}</span>
</div>
<div className="flex items-center justify-between px-3 py-2 font-medium">
<span className="text-ink-soft text-xs">Plug (offset needed)</span>
<span className={`font-mono ${plug !== null ? 'text-accent' : 'text-muted'}`}>
<span className="text-gray-700 text-xs">Plug (offset needed)</span>
<span className={`font-mono ${plug !== null ? 'text-blue-700' : 'text-gray-300'}`}>
{plug !== null ? fmt(plug) : '—'}
</span>
</div>
</div>
{error && <p className="text-xs text-danger mb-3">{error}</p>}
{error && <p className="text-xs text-red-500 mb-3">{error}</p>}
{/* Apply */}
<div className="flex gap-2 items-center">
<input type="number" step="0.01"
className="flex-1 border border-line rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-accent"
className="flex-1 border border-gray-200 rounded px-3 py-1.5 text-sm font-mono focus:outline-none focus:border-blue-400"
placeholder="offset to apply"
value={applyOffset} onChange={e => setApplyOffset(e.target.value)} />
<button onClick={() => onApply(parseFloat(applyOffset))} disabled={applyOffset === '' || isNaN(parseFloat(applyOffset))}
@ -434,12 +433,12 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
<div className="space-y-5">
{/* Label */}
<div className="bg-surface border border-line rounded p-4">
<h3 className="text-sm font-semibold text-ink-soft mb-3">Configuration</h3>
<div className="bg-white border border-gray-200 rounded p-4">
<h3 className="text-sm font-semibold text-gray-700 mb-3">Configuration</h3>
<div className="flex gap-3 items-end">
<div className="flex-1">
<label className="text-xs text-muted block mb-1">Label <span className="text-muted">(optional)</span></label>
<input className="w-full border border-line rounded px-3 py-1.5 text-sm focus:outline-none focus:border-accent"
<label className="text-xs text-gray-500 block mb-1">Label <span className="text-gray-400">(optional)</span></label>
<input className="w-full border border-gray-200 rounded px-3 py-1.5 text-sm focus:outline-none focus:border-blue-400"
value={label} onChange={e => setLabel(e.target.value)}
onKeyDown={e => e.key === 'Enter' && saveLabel()} />
</div>
@ -448,13 +447,13 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
{saving ? 'Saving…' : 'Save'}
</button>
</div>
{error && <p className="text-xs text-danger mt-2">{error}</p>}
{error && <p className="text-xs text-red-500 mt-2">{error}</p>}
</div>
{/* Sources */}
<div className="bg-surface border border-line rounded p-4">
<h3 className="text-sm font-semibold text-ink-soft mb-1">Sources</h3>
<p className="text-xs text-muted mb-3">Each source contributes rows to the combined view. Set the sign to flip the direction of amounts (e.g. credit card charges are positive in the source but should subtract from your balance). The offset adjusts the running balance — use Calibrate to compute it from a known good balance.</p>
<div className="bg-white border border-gray-200 rounded p-4">
<h3 className="text-sm font-semibold text-gray-700 mb-1">Sources</h3>
<p className="text-xs text-gray-400 mb-3">Each source contributes rows to the combined view. Set the sign to flip the direction of amounts (e.g. credit card charges are positive in the source but should subtract from your balance). The offset adjusts the running balance — use Calibrate to compute it from a known good balance.</p>
<div className="space-y-2 mb-3">
{members.map((m, idx) => {
const cfg = srcCfg[m.source_name] || {}
@ -467,50 +466,50 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
onDragOver={e => handleSrcDragOver(e, idx)}
onDrop={e => handleSrcDrop(e, idx)}
onDragEnd={() => { setSrcDragIdx(null); setSrcDragOverIdx(null) }}
className={`border border-line-soft rounded px-3 py-2 text-xs space-y-2 ${srcDragOverIdx === idx && srcDragIdx !== idx ? 'bg-accent-soft' : ''}`}>
className={`border border-gray-100 rounded px-3 py-2 text-xs space-y-2 ${srcDragOverIdx === idx && srcDragIdx !== idx ? 'bg-blue-50' : ''}`}>
<div className="flex items-center gap-2">
<span className="text-muted cursor-grab select-none">⠿</span>
<span className="font-medium text-ink-soft flex-1">{m.source_name}</span>
<button onClick={() => removeSource(m.source_name)} className="text-danger hover:text-danger">Remove</button>
<span className="text-gray-300 cursor-grab select-none">⠿</span>
<span className="font-medium text-gray-700 flex-1">{m.source_name}</span>
<button onClick={() => removeSource(m.source_name)} className="text-red-300 hover:text-red-500">Remove</button>
</div>
<div className="grid grid-cols-2 gap-x-4 gap-y-1.5">
<div>
<label className="text-muted block mb-0.5">Amount field</label>
<label className="text-gray-400 block mb-0.5">Amount field</label>
<select value={cfg.amount_field || ''}
onChange={e => handleSrcAmountField(m.source_name, e.target.value)}
className="w-full border border-line rounded px-1.5 py-0.5 focus:outline-none focus:border-accent">
className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
<option value="">— select —</option>
{sf.map(f => <option key={f} value={f}>{f}</option>)}
</select>
</div>
<div>
<label className="text-muted block mb-0.5">Sign</label>
<label className="text-gray-400 block mb-0.5">Sign</label>
<select value={cfg.sign ?? 1}
onChange={e => { setSrcSign(m.source_name, parseInt(e.target.value)); setMappingsDirty(true) }}
className="w-full border border-line rounded px-1.5 py-0.5 focus:outline-none focus:border-accent">
className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
<option value={1}>+1 (as-is)</option>
<option value={-1}>−1 (flip)</option>
</select>
</div>
<div>
<label className="text-muted block mb-0.5">Date field</label>
<label className="text-gray-400 block mb-0.5">Date field</label>
<select value={cfg.date_field || ''}
onChange={e => handleSrcDateField(m.source_name, e.target.value)}
className="w-full border border-line rounded px-1.5 py-0.5 focus:outline-none focus:border-accent">
className="w-full border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400">
<option value="">— select —</option>
{sf.map(f => <option key={f} value={f}>{f}</option>)}
</select>
</div>
<div>
<label className="text-muted block mb-0.5">Balance offset</label>
<label className="text-gray-400 block mb-0.5">Balance offset</label>
<div className="flex items-center gap-1">
<input type="number" step="0.01" value={cfg.offset ?? 0}
onChange={e => { setSrcOffset(m.source_name, parseFloat(e.target.value) || 0); setMappingsDirty(true) }}
className="flex-1 border border-line rounded px-1.5 py-0.5 font-mono focus:outline-none focus:border-accent" />
className="flex-1 border border-gray-200 rounded px-1.5 py-0.5 font-mono focus:outline-none focus:border-blue-400" />
<button onClick={() => handleCalibrate(m.source_name)}
disabled={!canCalibrate}
title={!canCalibrate ? 'Set amount and date fields first' : 'Calibrate balance'}
className="text-accent hover:text-accent underline disabled:opacity-40 disabled:cursor-not-allowed disabled:no-underline">
className="text-blue-400 hover:text-blue-600 underline disabled:opacity-40 disabled:cursor-not-allowed disabled:no-underline">
Calibrate
</button>
</div>
@ -519,43 +518,43 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
</div>
)
})}
{members.length === 0 && <p className="text-xs text-muted">No sources added yet.</p>}
{members.length === 0 && <p className="text-xs text-gray-400">No sources added yet.</p>}
</div>
{availableSources.length > 0 && (
<div className="flex gap-2">
<select className="flex-1 border border-line rounded px-2 py-1 text-sm focus:outline-none focus:border-accent"
<select className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
value={addingSrc} onChange={e => setAddingSrc(e.target.value)}>
<option value="">— add source —</option>
{availableSources.map(s => <option key={s.name} value={s.name}>{s.name}</option>)}
</select>
<button onClick={addSource} disabled={!addingSrc}
className="text-sm bg-raised px-3 py-1 rounded hover:bg-raised text-ink-soft disabled:opacity-40">Add</button>
className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700 disabled:opacity-40">Add</button>
</div>
)}
</div>
{/* Output columns mapping grid */}
<div className="bg-surface border border-line rounded p-4">
<h3 className="text-sm font-semibold text-ink-soft mb-1">Output columns</h3>
<p className="text-xs text-muted mb-3">
<div className="bg-white border border-gray-200 rounded p-4">
<h3 className="text-sm font-semibold text-gray-700 mb-1">Output columns</h3>
<p className="text-xs text-gray-400 mb-3">
Each row is a column in the combined view. Each source column shows which field from that source maps to it.
The first <span className="text-accent">numeric</span> field drives the running balance; the first <span className="text-ok">date</span> field drives the ordering.
The first <span className="text-blue-500">numeric</span> field drives the running balance; the first <span className="text-green-600">date</span> field drives the ordering.
Both <span className="font-mono">source_balance</span> (per-source) and <span className="font-mono">net_balance</span> (combined) are always included in the generated view.
Drag rows to reorder.
</p>
{members.length === 0 ? (
<p className="text-xs text-muted mb-3">Add sources above first.</p>
<p className="text-xs text-gray-400 mb-3">Add sources above first.</p>
) : (
<div className="overflow-x-auto mb-3">
<table className="w-full text-xs border-collapse">
<thead>
<tr className="border-b border-line">
<tr className="border-b border-gray-200">
<th className="w-5 pb-2"></th>
<th className="text-left text-muted font-normal pb-2 pr-4">Column</th>
<th className="text-left text-muted font-normal pb-2 pr-4">Type</th>
<th className="text-left text-gray-400 font-normal pb-2 pr-4">Column</th>
<th className="text-left text-gray-400 font-normal pb-2 pr-4">Type</th>
{members.map(m => (
<th key={m.source_name} className="text-left text-muted font-normal pb-2 pr-3 min-w-36">{m.source_name}</th>
<th key={m.source_name} className="text-left text-gray-400 font-normal pb-2 pr-3 min-w-36">{m.source_name}</th>
))}
<th className="w-5 pb-2"></th>
</tr>
@ -571,21 +570,21 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
onDragOver={e => handleDragOver(e, idx)}
onDrop={e => handleDrop(e, idx)}
onDragEnd={() => { setDragIdx(null); setDragOverIdx(null) }}
className={`border-b border-line-soft ${dragOverIdx === idx && dragIdx !== idx ? 'bg-accent-soft' : ''}`}>
<td className="py-1.5 pr-1 text-muted cursor-grab select-none">⠿</td>
<td className="py-1.5 pr-4 font-mono text-ink-soft whitespace-nowrap">
className={`border-b border-gray-50 ${dragOverIdx === idx && dragIdx !== idx ? 'bg-blue-50' : ''}`}>
<td className="py-1.5 pr-1 text-gray-300 cursor-grab select-none">⠿</td>
<td className="py-1.5 pr-4 font-mono text-gray-700 whitespace-nowrap">
{f.name}
{isAmount && <span className="ml-1.5 text-accent font-sans font-normal">amount</span>}
{isDate && <span className="ml-1.5 text-ok font-sans font-normal">date</span>}
{isAmount && <span className="ml-1.5 text-blue-500 font-sans font-normal">amount</span>}
{isDate && <span className="ml-1.5 text-green-600 font-sans font-normal">date</span>}
</td>
<td className="py-1.5 pr-4 text-muted">{f.type}</td>
<td className="py-1.5 pr-4 text-gray-400">{f.type}</td>
{members.map(m => (
<td key={m.source_name} className="py-1.5 pr-3">
<div className="flex items-center gap-1">
<select
value={getMappingValue(m.source_name, f.name)}
onChange={e => setMappingValue(m.source_name, f.name, e.target.value)}
className="border border-line rounded px-1.5 py-0.5 focus:outline-none focus:border-accent min-w-0 flex-1">
className="border border-gray-200 rounded px-1.5 py-0.5 focus:outline-none focus:border-blue-400 min-w-0 flex-1">
<option value="">— same name —</option>
{(srcFields[m.source_name] || []).map(sf => (
<option key={sf} value={sf}>{sf}</option>
@ -595,13 +594,13 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
</td>
))}
<td className="py-1.5">
<button onClick={() => removeField(f.name)} className="text-danger hover:text-danger">✕</button>
<button onClick={() => removeField(f.name)} className="text-red-300 hover:text-red-500">✕</button>
</td>
</tr>
)
})}
{fields.length === 0 && (
<tr><td colSpan={3 + members.length} className="py-3 text-muted text-center">No columns defined yet — add one below.</td></tr>
<tr><td colSpan={3 + members.length} className="py-3 text-gray-400 text-center">No columns defined yet — add one below.</td></tr>
)}
</tbody>
</table>
@ -610,15 +609,15 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
{/* Add field */}
<div className="flex gap-2 mb-3">
<input className="flex-1 border border-line rounded px-2 py-1 text-sm focus:outline-none focus:border-accent"
<input className="flex-1 border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
placeholder="column name" value={newField.name}
onChange={e => setNewField(f => ({ ...f, name: e.target.value }))}
onKeyDown={e => e.key === 'Enter' && addField()} />
<select className="border border-line rounded px-2 py-1 text-sm focus:outline-none focus:border-accent"
<select className="border border-gray-200 rounded px-2 py-1 text-sm focus:outline-none focus:border-blue-400"
value={newField.type} onChange={e => setNewField(f => ({ ...f, type: e.target.value }))}>
{FIELD_TYPES.map(t => <option key={t} value={t}>{t}</option>)}
</select>
<button onClick={addField} className="text-sm bg-raised px-3 py-1 rounded hover:bg-raised text-ink-soft">Add</button>
<button onClick={addField} className="text-sm bg-gray-100 px-3 py-1 rounded hover:bg-gray-200 text-gray-700">Add</button>
</div>
{mappingsDirty && (
@ -630,12 +629,12 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
</div>
{/* Generate view + balance */}
<div className="bg-surface border border-line rounded p-4">
<div className="bg-white border border-gray-200 rounded p-4">
<div className="flex items-center justify-between mb-3">
<h3 className="text-sm font-semibold text-ink-soft">View</h3>
<h3 className="text-sm font-semibold text-gray-700">View</h3>
<div className="flex gap-2">
<button onClick={fetchBalance}
className="text-sm bg-raised text-ink-soft px-3 py-1.5 rounded hover:bg-raised">
className="text-sm bg-gray-100 text-gray-700 px-3 py-1.5 rounded hover:bg-gray-200">
Refresh balance
</button>
<button onClick={generateView}
@ -646,18 +645,18 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
</div>
{netBalance !== null && (
<div className="mb-3 flex items-center gap-3">
<span className="text-xs text-muted">Current net balance</span>
<span className="text-lg font-mono font-semibold text-ink">
<span className="text-xs text-gray-500">Current net balance</span>
<span className="text-lg font-mono font-semibold text-gray-800">
{Number(netBalance).toLocaleString(undefined, { minimumFractionDigits: 2 })}
</span>
</div>
)}
{balanceError && <p className="text-xs text-muted mb-3">{balanceError}</p>}
{balanceError && <p className="text-xs text-gray-400 mb-3">{balanceError}</p>}
{viewResult && !viewResult.success && (
<p className="text-xs text-danger">{viewResult.error}</p>
<p className="text-xs text-red-500">{viewResult.error}</p>
)}
{viewResult && viewResult.success && (
<p className="text-xs text-ok">View created: <span className="font-mono">{viewResult.view}</span></p>
<p className="text-xs text-green-600">View created: <span className="font-mono">{viewResult.view}</span></p>
)}
</div>
@ -676,7 +675,7 @@ function StackPanel({ stack, sources, onUpdated, onStale, onViewGenerated, onSql
// ── Main page ──────────────────────────────────────────────────────────────────
export default function Stacks({ sources, onStackStale, onStackViewGenerated, onStacksChange }) {
export default function Stacks({ sources, onStackStale, onStackViewGenerated }) {
const [stacks, setStacks] = useState([])
const [selected, setSelected] = useState(null)
const [stackDetail, setStackDetail] = useState(null)
@ -717,7 +716,6 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
await api.createStack({ name: newName, fields: [] })
setNewName(''); setCreating(false)
await load()
onStacksChange?.()
loadDetail(newName)
} catch (e) { setError(e.message) }
}
@ -727,7 +725,6 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
await api.deleteStack(name)
if (selected === name) { setSelected(null); setStackDetail(null); setSqlDraft(''); setSqlResult(null) }
load()
onStacksChange?.()
}
async function runSql() {
@ -748,31 +745,28 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
<div className="p-6">
{/* Stack list — horizontal row of cards */}
<div className="flex items-center gap-2 mb-5 flex-wrap">
<h1 className="text-sm font-semibold text-ink mr-1">Stacks</h1>
<h1 className="text-sm font-semibold text-gray-800 mr-1">Stacks</h1>
{stacks.map(s => (
<div key={s.name}
onClick={() => loadDetail(s.name)}
className={`flex items-center gap-2 px-3 py-1.5 rounded border cursor-pointer text-xs group transition-colors ${selected === s.name ? 'border-accent-line bg-accent-soft text-accent' : 'border-line bg-surface text-ink-soft hover:border-line hover:bg-raised'}`}>
className={`flex items-center gap-2 px-3 py-1.5 rounded border cursor-pointer text-xs group transition-colors ${selected === s.name ? 'border-blue-300 bg-blue-50 text-blue-700' : 'border-gray-200 bg-white text-gray-600 hover:border-gray-300 hover:bg-gray-50'}`}>
<span className="font-medium">{s.label || s.name}</span>
<span className="text-muted">{s.source_count}s</span>
<Link to={`/stacks/${encodeURIComponent(s.name)}/pivot`}
onClick={e => e.stopPropagation()}
className="text-accent underline decoration-transparent hover:decoration-inherit">pivot</Link>
<span className="text-gray-400">{s.source_count}s</span>
<button onClick={e => { e.stopPropagation(); deleteStack(s.name) }}
className="opacity-0 group-hover:opacity-100 text-danger hover:text-danger leading-none ml-2 pl-2 border-l border-line">✕</button>
className="opacity-0 group-hover:opacity-100 text-red-300 hover:text-red-500 leading-none">✕</button>
</div>
))}
{creating ? (
<div className="flex items-center gap-1">
<input autoFocus className="border border-accent rounded px-2 py-1 text-xs focus:outline-none w-32"
<input autoFocus className="border border-blue-400 rounded px-2 py-1 text-xs focus:outline-none w-32"
placeholder="stack name" value={newName} onChange={e => setNewName(e.target.value)}
onKeyDown={e => { if (e.key === 'Enter') createStack(); if (e.key === 'Escape') setCreating(false) }} />
<button onClick={createStack} className="text-xs bg-blue-600 text-white px-2 py-1 rounded">Create</button>
<button onClick={() => setCreating(false)} className="text-xs text-muted px-1">✕</button>
{error && <p className="text-xs text-danger">{error}</p>}
<button onClick={() => setCreating(false)} className="text-xs text-gray-400 px-1">✕</button>
{error && <p className="text-xs text-red-500">{error}</p>}
</div>
) : (
<button onClick={() => setCreating(true)} className="text-xs text-accent hover:text-accent px-2 py-1.5">+ New</button>
<button onClick={() => setCreating(true)} className="text-xs text-blue-500 hover:text-blue-700 px-2 py-1.5">+ New</button>
)}
</div>
@ -780,9 +774,9 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
<div className="flex gap-6 items-start">
{/* Left: config panel */}
<div className="flex-1 min-w-0">
<h2 className="text-base font-semibold text-ink mb-4">
<h2 className="text-base font-semibold text-gray-800 mb-4">
{stackDetail.label || stackDetail.name}
{stackDetail.label && <span className="text-sm text-muted font-normal ml-2">{stackDetail.name}</span>}
{stackDetail.label && <span className="text-sm text-gray-400 font-normal ml-2">{stackDetail.name}</span>}
</h2>
<StackPanel
key={stackDetail.name}
@ -797,9 +791,9 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
{/* Right: SQL panel */}
<div className="flex-1 min-w-0">
<div className="bg-surface border border-line rounded p-4 sticky top-4">
<div className="bg-white border border-gray-200 rounded p-4 sticky top-4">
<div className="flex items-center justify-between mb-3">
<h3 className="text-sm font-semibold text-ink-soft">Generated SQL</h3>
<h3 className="text-sm font-semibold text-gray-700">Generated SQL</h3>
<button
onClick={runSql}
disabled={!sqlDraft.trim() || sqlRunning}
@ -809,17 +803,17 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
</div>
{sqlDraft ? (
<textarea
className="w-full font-mono text-xs text-ink-soft bg-raised border border-line rounded p-2 focus:outline-none focus:border-accent resize-none leading-relaxed"
className="w-full font-mono text-xs text-gray-700 bg-gray-50 border border-gray-200 rounded p-2 focus:outline-none focus:border-blue-400 resize-none leading-relaxed"
style={{ minHeight: '60vh' }}
value={sqlDraft}
onChange={e => { setSqlDraft(e.target.value); setSqlResult(null) }}
spellCheck={false}
/>
) : (
<p className="text-xs text-muted">Generate a view to see the SQL here.</p>
<p className="text-xs text-gray-400">Generate a view to see the SQL here.</p>
)}
{sqlResult && (
<p className={`text-xs mt-2 ${sqlResult.success ? 'text-ok' : 'text-danger'}`}>
<p className={`text-xs mt-2 ${sqlResult.success ? 'text-green-600' : 'text-red-500'}`}>
{sqlResult.success ? 'View updated successfully.' : sqlResult.error}
</p>
)}
@ -827,7 +821,7 @@ export default function Stacks({ sources, onStackStale, onStackViewGenerated, on
</div>
</div>
) : (
<p className="text-sm text-muted">Select a stack or create one.</p>
<p className="text-sm text-gray-400">Select a stack or create one.</p>
)}
</div>
)

25
ui/src/theme.jsx
View File

@ -1,25 +0,0 @@
import { createContext, useContext, useState, useEffect } from 'react'
const ThemeContext = createContext()
export function ThemeProvider({ children }) {
const [dark, setDark] = useState(() => {
const saved = localStorage.getItem('df_dark')
if (saved !== null) return saved === 'true'
return window.matchMedia('(prefers-color-scheme: dark)').matches
})
useEffect(() => {
localStorage.setItem('df_dark', dark)
document.documentElement.classList.toggle('dark', dark)
}, [dark])
return (
<ThemeContext.Provider value={{ dark, setDark }}>
{children}
</ThemeContext.Provider>
)
}
const useTheme = () => useContext(ThemeContext)
export default useTheme

85
uninstall.sh Executable file
View File

@ -0,0 +1,85 @@
#!/bin/bash
#
# Dataflow Uninstall Script
# Removes database user, database, and optionally .env
#
echo "⚠️ Dataflow Uninstall"
echo "====================="
echo ""
# Load .env if it exists
if [ -f .env ]; then
export $(cat .env | grep -v '^#' | xargs)
fi
DB_NAME=${DB_NAME:-dataflow}
DB_USER=${DB_USER:-dataflow}
echo "⚠️ This will permanently delete:"
echo " - Database: $DB_NAME"
echo " - User: $DB_USER"
echo ""
read -p "Type 'delete' to confirm: " CONFIRM
if [ "$CONFIRM" != "delete" ]; then
echo "Cancelled."
exit 0
fi
# Prompt for admin credentials
echo ""
echo "📋 PostgreSQL Admin Credentials"
echo ""
read -p "Admin username [postgres]: " ADMIN_USER
ADMIN_USER=${ADMIN_USER:-postgres}
read -s -p "Admin password: " ADMIN_PASS
echo ""
DB_HOST=${DB_HOST:-localhost}
DB_PORT=${DB_PORT:-5432}
DB_NAME=${DB_NAME:-dataflow}
DB_USER=${DB_USER:-dataflow}
# Test admin connection
echo ""
echo "🔍 Testing PostgreSQL admin connection..."
export PGPASSWORD="$ADMIN_PASS"
if ! psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c '\q' 2>/dev/null; then
echo "✗ Cannot connect to PostgreSQL"
exit 1
fi
echo "✓ Connected"
# Drop database
echo ""
echo "🗄️ Dropping database..."
psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c "DROP DATABASE IF EXISTS $DB_NAME;" 2>/dev/null || true
echo "✓ Database dropped"
# Drop user
echo ""
echo "👤 Dropping user..."
psql -U "$ADMIN_USER" -h "$DB_HOST" -p "$DB_PORT" -d postgres -c "DROP USER IF EXISTS $DB_USER;" 2>/dev/null || true
echo "✓ User dropped"
unset PGPASSWORD
# Optionally remove .env
echo ""
read -p "Remove .env file? [y/N]: " REMOVE_ENV
if [ "$REMOVE_ENV" == "y" ] || [ "$REMOVE_ENV" == "Y" ]; then
rm -f .env
echo "✓ .env removed"
fi
# Optionally remove node_modules
echo ""
read -p "Remove node_modules? [y/N]: " REMOVE_MODULES
if [ "$REMOVE_MODULES" == "y" ] || [ "$REMOVE_MODULES" == "Y" ]; then
rm -rf node_modules
echo "✓ node_modules removed"
fi
echo ""
echo "✅ Uninstall complete!"
echo ""