Paul Trowbridge
7dcd8c4b61
Architecture, file structure, the manage.py menu, and the API reference were each documented in two or three of README.md, SPEC.md, and CLAUDE.md — the same drift trap the SQL just had. SPEC.md, examples/GETTING_STARTED.md, and ui/README.md move into docs/. PERSPECTIVE.md and docs/perspective-pivot.md merge into docs/perspective.md, version rationale first, then the API reference. README.md becomes an entry point that links out, and CLAUDE.md keeps only working rules and non-obvious behaviour, pointing at docs/spec.md for the rest. examples/ keeps just the sample CSV the tutorial loads. Corrections found while consolidating: - the spec's API table was missing 20 routes — every override endpoint, most of /api/stacks, the mapping remap routes, /health. Rebuilt from the route files - the tutorial used port 3000 (default is 3020) and never mentioned Basic auth, so every curl in it would have 401'd - the tutorial and the spec each hand-listed the SQL deploy order; both now point at manage.py, which is where the order actually lives - CLAUDE.md described deduplication as an MD5 hash (it is a plain JSONB object), claimed 5 tables and 4 functions, and told you to run a setup.sh that has not existed for some time Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
73 lines
2.8 KiB
Markdown
73 lines
2.8 KiB
Markdown
# Dataflow
|
|
|
|
A simple data transformation tool for importing, cleaning, and standardizing data from various sources.
|
|
|
|
Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will
|
|
deduplicate on import, pull structure out with regex rules, map the extracted values to clean
|
|
output, and serve the result through a web UI and REST API.
|
|
|
|
## How it works
|
|
|
|
1. **Sources** define where data comes from and which fields make a record unique
|
|
2. **Rules** extract information with regex (`extract` or `replace` mode) —
|
|
e.g. pull the merchant out of a transaction description
|
|
3. **Mappings** turn extracted values into clean output —
|
|
`"DISCOUNT DRUG MART 32"` → `{"vendor": "Discount Drug Mart", "category": "Healthcare"}`
|
|
4. **Records** are then queryable, pivotable, and exportable
|
|
|
|
Each record keeps three layers: `data` (raw import), `transformed` (rule and mapping output),
|
|
and `overrides` (manual edits). Reads merge them in that order, so re-running the rules never
|
|
clobbers something you typed by hand.
|
|
|
|
## Stack
|
|
|
|
PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from `public/`.
|
|
HTTP Basic auth, configured in `.env`.
|
|
|
|
## Getting started
|
|
|
|
Requires PostgreSQL 12+, Node.js 18+, and Python 3.
|
|
|
|
```bash
|
|
npm install
|
|
python3 manage.py # interactive setup: .env, database, schema, functions, UI, service
|
|
```
|
|
|
|
The UI is then at `http://localhost:3020` and the API at `http://localhost:3020/api`
|
|
(port set by `API_PORT` in `.env`).
|
|
|
|
For a walkthrough that creates a source, adds rules and mappings, and imports the sample
|
|
CSV in `examples/`, see **[docs/getting-started.md](docs/getting-started.md)**.
|
|
|
|
## Documentation
|
|
|
|
| | |
|
|
|---|---|
|
|
| **[docs/getting-started.md](docs/getting-started.md)** | Tutorial — build a working pipeline from scratch with curl |
|
|
| **[docs/spec.md](docs/spec.md)** | Full reference — architecture, schema, data flow, API, `manage.py` |
|
|
| **[docs/ui.md](docs/ui.md)** | Frontend: React + Vite build, key packages |
|
|
| **[docs/perspective.md](docs/perspective.md)** | Pivot table: pinned versions and API reference |
|
|
|
|
## Project structure
|
|
|
|
```
|
|
dataflow/
|
|
├── manage.py # interactive setup / deploy / uninstall
|
|
├── database/ # schema.sql + one .sql file per API route
|
|
├── api/ # Express server, routes, auth middleware
|
|
├── ui/ # React source (built to public/)
|
|
├── public/ # built UI, served as static files
|
|
├── docs/
|
|
└── examples/ # sample CSV for the tutorial
|
|
```
|
|
|
|
Both the API routes and the SQL are organized one file per resource, so `api/routes/rules.js`
|
|
and `database/rules.sql` are the two halves of the same feature.
|
|
|
|
`database/*.sql` is the source of truth for every database function — never edit one directly
|
|
in the database, or the next redeploy will silently revert it.
|
|
|
|
## License
|
|
|
|
MIT
|