Dataflow

A simple data transformation tool for importing, cleaning, and standardizing data from various sources.

Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will deduplicate on import, pull structure out with regex rules, map the extracted values to clean output, and serve the result through a web UI and REST API.

How it works

1. Sources define where data comes from and which fields make a record unique
2. Rules extract information with regex (extract or replace mode) — e.g. pull the merchant out of a transaction description
3. Mappings turn extracted values into clean output — "DISCOUNT DRUG MART 32" → {"vendor": "Discount Drug Mart", "category": "Healthcare"}
4. Records are then queryable, pivotable, and exportable

Each record keeps three layers: data (raw import), transformed (rule and mapping output), and overrides (manual edits). Reads merge them in that order, so re-running the rules never clobbers something you typed by hand.

Stack

PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from public/. HTTP Basic auth, configured in .env.

Getting started

Requires PostgreSQL 12+, Node.js 18+, and Python 3.

npm install
python3 manage.py     # interactive setup: .env, database, schema, functions, UI, service

The UI is then at http://localhost:3020 and the API at http://localhost:3020/api (port set by API_PORT in .env).

For a walkthrough that creates a source, adds rules and mappings, and imports the sample CSV in examples/, see docs/getting-started.md.

Documentation

docs/getting-started.md	Tutorial — build a working pipeline from scratch with curl
docs/spec.md	Full reference — architecture, schema, data flow, API, manage.py
docs/ui.md	Frontend: React + Vite build, key packages
docs/perspective.md	Pivot table: pinned versions and API reference

Project structure

dataflow/
├── manage.py           # interactive setup / deploy / uninstall
├── database/           # schema.sql + one .sql file per API route
├── api/                # Express server, routes, auth middleware
├── ui/                 # React source (built to public/)
├── public/             # built UI, served as static files
├── docs/
└── examples/           # sample CSV for the tutorial

Both the API routes and the SQL are organized one file per resource, so api/routes/rules.js and database/rules.sql are the two halves of the same feature.

database/*.sql is the source of truth for every database function — never edit one directly in the database, or the next redeploy will silently revert it.

License

MIT