Paul Trowbridge
c4e7211e6d
Sources with a `teller` block in their config can pull transactions straight from the bank instead of taking a CSV upload. Only the fetch differs — dedupe, logging, and transformation reuse the import path. api/lib/teller.js speaks Teller's mutual-TLS protocol (client cert plus the access token as the HTTP Basic username) and flattens transactions into the shallow map the rule engine expects. Access tokens live in .env, one per enrollment, not in the database that manage.py offers to reset. Pending transactions are skipped by default: their ids change when they post, which would import the same charge twice under two keys. Sources should use ['id'] as constraint_fields — Teller's transaction id makes overlapping pulls free while keeping genuinely repeated charges distinct. Untested against the live API — Teller has no self-serve signup at the moment, so no account to verify against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HFeU5neCKagTnmA6o9Tu |
||
|---|---|---|
| api | ||
| database | ||
| docs | ||
| examples | ||
| ui | ||
| .env.example | ||
| .gitignore | ||
| CLAUDE.md | ||
| dataflow.service | ||
| manage.py | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
Dataflow
A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will deduplicate on import, pull structure out with regex rules, map the extracted values to clean output, and serve the result through a web UI and REST API.
How it works
- Sources define where data comes from and which fields make a record unique
- Rules extract information with regex (
extractorreplacemode) — e.g. pull the merchant out of a transaction description - Mappings turn extracted values into clean output —
"DISCOUNT DRUG MART 32"→{"vendor": "Discount Drug Mart", "category": "Healthcare"} - Records are then queryable, pivotable, and exportable
Each record keeps three layers: data (raw import), transformed (rule and mapping output),
and overrides (manual edits). Reads merge them in that order, so re-running the rules never
clobbers something you typed by hand.
Stack
PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from public/.
HTTP Basic auth, configured in .env.
Getting started
Requires PostgreSQL 12+, Node.js 18+, and Python 3.
npm install
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
The UI is then at http://localhost:3020 and the API at http://localhost:3020/api
(port set by API_PORT in .env).
For a walkthrough that creates a source, adds rules and mappings, and imports the sample
CSV in examples/, see docs/getting-started.md.
Documentation
| docs/getting-started.md | Tutorial — build a working pipeline from scratch with curl |
| docs/spec.md | Full reference — architecture, schema, data flow, API, manage.py |
| docs/ui.md | Frontend: React + Vite build, key packages |
| docs/perspective.md | Pivot table: pinned versions and API reference |
Project structure
dataflow/
├── manage.py # interactive setup / deploy / uninstall
├── database/ # schema.sql + one .sql file per API route
├── api/ # Express server, routes, auth middleware
├── ui/ # React source (built to public/)
├── public/ # built UI, served as static files
├── docs/
└── examples/ # sample CSV for the tutorial
Both the API routes and the SQL are organized one file per resource, so api/routes/rules.js
and database/rules.sql are the two halves of the same feature.
database/*.sql is the source of truth for every database function — never edit one directly
in the database, or the next redeploy will silently revert it.
License
MIT