Paul Trowbridge 7dcd8c4b61 Consolidate documentation into docs/ and cut the duplication

Architecture, file structure, the manage.py menu, and the API reference were
each documented in two or three of README.md, SPEC.md, and CLAUDE.md — the same
drift trap the SQL just had.

SPEC.md, examples/GETTING_STARTED.md, and ui/README.md move into docs/.
PERSPECTIVE.md and docs/perspective-pivot.md merge into docs/perspective.md,
version rationale first, then the API reference. README.md becomes an entry
point that links out, and CLAUDE.md keeps only working rules and non-obvious
behaviour, pointing at docs/spec.md for the rest. examples/ keeps just the
sample CSV the tutorial loads.

Corrections found while consolidating:

- the spec's API table was missing 20 routes — every override endpoint, most of
  /api/stacks, the mapping remap routes, /health. Rebuilt from the route files
- the tutorial used port 3000 (default is 3020) and never mentioned Basic auth,
  so every curl in it would have 401'd
- the tutorial and the spec each hand-listed the SQL deploy order; both now
  point at manage.py, which is where the order actually lives
- CLAUDE.md described deduplication as an MD5 hash (it is a plain JSONB object),
  claimed 5 tables and 4 functions, and told you to run a setup.sh that has not
  existed for some time

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-07-26 21:55:58 -04:00

2.8 KiB

Raw Blame History

Dataflow

A simple data transformation tool for importing, cleaning, and standardizing data from various sources.

Point it at a messy CSV — bank transactions, product lists, anything repetitive — and it will deduplicate on import, pull structure out with regex rules, map the extracted values to clean output, and serve the result through a web UI and REST API.

How it works

Sources define where data comes from and which fields make a record unique
Rules extract information with regex (extract or replace mode) — e.g. pull the merchant out of a transaction description
Mappings turn extracted values into clean output — "DISCOUNT DRUG MART 32" → {"vendor": "Discount Drug Mart", "category": "Healthcare"}
Records are then queryable, pivotable, and exportable

Each record keeps three layers: data (raw import), transformed (rule and mapping output), and overrides (manual edits). Reads merge them in that order, so re-running the rules never clobbers something you typed by hand.

Stack

PostgreSQL with JSONB storage, a Node.js/Express API, and a React SPA served from public/. HTTP Basic auth, configured in .env.

Getting started

Requires PostgreSQL 12+, Node.js 18+, and Python 3.

npm install
python3 manage.py     # interactive setup: .env, database, schema, functions, UI, service

The UI is then at http://localhost:3020 and the API at http://localhost:3020/api (port set by API_PORT in .env).

For a walkthrough that creates a source, adds rules and mappings, and imports the sample CSV in examples/, see docs/getting-started.md.

Documentation


docs/getting-started.md	Tutorial — build a working pipeline from scratch with curl
docs/spec.md	Full reference — architecture, schema, data flow, API, `manage.py`
docs/ui.md	Frontend: React + Vite build, key packages
docs/perspective.md	Pivot table: pinned versions and API reference

Project structure

dataflow/
├── manage.py           # interactive setup / deploy / uninstall
├── database/           # schema.sql + one .sql file per API route
├── api/                # Express server, routes, auth middleware
├── ui/                 # React source (built to public/)
├── public/             # built UI, served as static files
├── docs/
└── examples/           # sample CSV for the tutorial

Both the API routes and the SQL are organized one file per resource, so api/routes/rules.js and database/rules.sql are the two halves of the same feature.

database/*.sql is the source of truth for every database function — never edit one directly in the database, or the next redeploy will silently revert it.

License

MIT

2.8 KiB Raw Blame History