Paul Trowbridge
3613037ab5
Sources with a `simplefin` block in their config can pull transactions straight from the bridge instead of taking a CSV upload. Only the fetch differs — dedupe, logging, and transformation reuse the import path. The access URL is the whole credential, so it lives in .env rather than the database that manage.py offers to reset. Claiming a setup token is exposed as an endpoint because the token is single-use and easy to burn. The bridge answers 200 with a populated `errors` array when a bank is failing, which would otherwise read as a successful empty pull — those errors ride along in the sync response and show on the Import page. Pending transactions are skipped by default: they get a new id once they post, which would import the same charge twice under two keys. Sources should use ['id'] as constraint_fields — the transaction id makes overlapping pulls free while keeping genuinely repeated charges distinct. Verified against a stubbed bridge response, not a live account. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HFeU5neCKagTnmA6o9Tu |
||
|---|---|---|
| api | ||
| database | ||
| docs | ||
| examples | ||
| ui | ||
| .env.example | ||
| .gitignore | ||
| CLAUDE.md | ||
| dataflow.service | ||
| manage.py | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
Dataflow
A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will deduplicate on import, pull structure out with regex rules, map the extracted values to clean output, and serve the result through a web UI and REST API.
How it works
- Sources define where data comes from and which fields make a record unique
- Rules extract information with regex (
extractorreplacemode) — e.g. pull the merchant out of a transaction description - Mappings turn extracted values into clean output —
"DISCOUNT DRUG MART 32"→{"vendor": "Discount Drug Mart", "category": "Healthcare"} - Records are then queryable, pivotable, and exportable
Each record keeps three layers: data (raw import), transformed (rule and mapping output),
and overrides (manual edits). Reads merge them in that order, so re-running the rules never
clobbers something you typed by hand.
Stack
PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from public/.
HTTP Basic auth, configured in .env.
Getting started
Requires PostgreSQL 12+, Node.js 18+, and Python 3.
npm install
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
The UI is then at http://localhost:3020 and the API at http://localhost:3020/api
(port set by API_PORT in .env).
For a walkthrough that creates a source, adds rules and mappings, and imports the sample
CSV in examples/, see docs/getting-started.md.
Documentation
| docs/getting-started.md | Tutorial — build a working pipeline from scratch with curl |
| docs/spec.md | Full reference — architecture, schema, data flow, API, manage.py |
| docs/ui.md | Frontend: React + Vite build, key packages |
| docs/perspective.md | Pivot table: pinned versions and API reference |
Project structure
dataflow/
├── manage.py # interactive setup / deploy / uninstall
├── database/ # schema.sql + one .sql file per API route
├── api/ # Express server, routes, auth middleware
├── ui/ # React source (built to public/)
├── public/ # built UI, served as static files
├── docs/
└── examples/ # sample CSV for the tutorial
Both the API routes and the SQL are organized one file per resource, so api/routes/rules.js
and database/rules.sql are the two halves of the same feature.
database/*.sql is the source of truth for every database function — never edit one directly
in the database, or the next redeploy will silently revert it.
License
MIT