pt/dataflow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
103 Commits 2 Branches 0 Tags 714 KiB
JavaScript 72.5%
PLpgSQL 16.2%
Python 10.1%
CSS 1.1%
simplefin
Go to file
Paul Trowbridge 9f164bcd34 Discover feed fields from real data; fix the sync window 
Field discovery now samples an account's actual transactions instead of
assuming a shape. flatten() passes through every scalar the bridge sends
rather than whitelisting eleven keys, so institution-specific fields turn
up on their own, and inferFields() — extracted from the CSV suggest route
so both paths share it — unions keys across the sample because API feeds
omit optional fields entirely.

Three bugs the live bridge exposed:

- posted=0 on pending transactions became 1970-01-01; falsy epochs are
  now "no date", with date falling back to transacted_at and posted_date
  kept separate.
- days=0 omitted start-date, which returns only the few most recent
  transactions rather than everything — 4 instead of 89. A start-date is
  always sent now, clamped to 89 days (the bridge hard-caps at 90).
- Sampling asked for more than 45 days, and the bridge's advisory notice
  about that surfaced in the UI as an error. Samples use 44 days; the
  threshold is exclusive.

The Sources page can now link an account: a picker in both the create
dialog and the detail panel, populated on demand, which fills the field
table from the sample and defaults the constraint field to the
transaction id with an explanation of why.

manage.py option 10 claims a setup token and writes the access URL to
.env, replacing the throwaway script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G2HFeU5neCKagTnmA6o9Tu
2026-08-02 02:50:03 -04:00
api Discover feed fields from real data; fix the sync window 2026-08-02 02:50:03 -04:00
database List override and transformed keys as source fields; show record id 2026-07-26 23:10:01 -04:00
docs Discover feed fields from real data; fix the sync window 2026-08-02 02:50:03 -04:00
examples Consolidate documentation into docs/ and cut the duplication 2026-07-26 21:55:58 -04:00
ui Discover feed fields from real data; fix the sync window 2026-08-02 02:50:03 -04:00
.env.example Add SimpleFIN Bridge sync as an alternative to CSV import 2026-08-01 12:47:04 -04:00
.gitignore Track package-lock.json for both the API and the UI 2026-07-26 21:54:10 -04:00
CLAUDE.md Consolidate documentation into docs/ and cut the duplication 2026-07-26 21:55:58 -04:00
dataflow.service Add unified deploy.sh and systemd service unit 2026-04-05 15:53:02 -04:00
manage.py Discover feed fields from real data; fix the sync window 2026-08-02 02:50:03 -04:00
package-lock.json Track package-lock.json for both the API and the UI 2026-07-26 21:54:10 -04:00
package.json Remove obsolete deploy scripts, migration helpers, and unused UI assets 2026-07-26 13:15:46 -04:00
README.md Consolidate documentation into docs/ and cut the duplication 2026-07-26 21:55:58 -04:00

README.md

Dataflow

A simple data transformation tool for importing, cleaning, and standardizing data from various sources.

Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will deduplicate on import, pull structure out with regex rules, map the extracted values to clean output, and serve the result through a web UI and REST API.

How it works

  1. Sources define where data comes from and which fields make a record unique
  2. Rules extract information with regex (extract or replace mode) — e.g. pull the merchant out of a transaction description
  3. Mappings turn extracted values into clean output — "DISCOUNT DRUG MART 32" → {"vendor": "Discount Drug Mart", "category": "Healthcare"}
  4. Records are then queryable, pivotable, and exportable

Each record keeps three layers: data (raw import), transformed (rule and mapping output), and overrides (manual edits). Reads merge them in that order, so re-running the rules never clobbers something you typed by hand.

Stack

PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from public/. HTTP Basic auth, configured in .env.

Getting started

Requires PostgreSQL 12+, Node.js 18+, and Python 3.

npm install
python3 manage.py     # interactive setup: .env, database, schema, functions, UI, service

The UI is then at http://localhost:3020 and the API at http://localhost:3020/api (port set by API_PORT in .env).

For a walkthrough that creates a source, adds rules and mappings, and imports the sample CSV in examples/, see docs/getting-started.md.

Documentation
docs/getting-started.md Tutorial — build a working pipeline from scratch with curl
docs/spec.md Full reference — architecture, schema, data flow, API, manage.py
docs/ui.md Frontend: React + Vite build, key packages
docs/perspective.md Pivot table: pinned versions and API reference

Project structure

dataflow/
├── manage.py           # interactive setup / deploy / uninstall
├── database/           # schema.sql + one .sql file per API route
├── api/                # Express server, routes, auth middleware
├── ui/                 # React source (built to public/)
├── public/             # built UI, served as static files
├── docs/
└── examples/           # sample CSV for the tutorial

Both the API routes and the SQL are organized one file per resource, so api/routes/rules.js and database/rules.sql are the two halves of the same feature.

database/*.sql is the source of truth for every database function — never edit one directly in the database, or the next redeploy will silently revert it.

License

MIT