Documents manage.py menu, adds full API reference tables, fixes
incorrect route in quick example, and removes stale sections
(docs/ dir, initial development status).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add database/queries/{sources,rules,mappings,records}.sql — one file per
route, all business logic in PostgreSQL functions
- Replace parameterized queries in all four route files with lit()/jsonLit()
literal interpolation for debuggability
- Add api/lib/sql.js with lit(), jsonLit(), arr() helpers
- Fix get_view_data to use json_agg (preserves column order) with subquery
(guarantees sort order is respected before aggregation)
- Fix jsonLit() for JSONB params so plain strings become valid JSON
- Update manage.py option 3 to deploy database/queries/ instead of functions.sql
- Add SPEC.md covering architecture, philosophy, and manage.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After createMapping, the new mapping's id was not stored in allValues
state, so editing the row again fell into the create path instead of
update. Now stores created.id so subsequent saves correctly use updateMapping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Credentials are saved to sessionStorage on login and restored on mount,
so a page refresh re-authenticates silently. Closing the tab clears them.
Logout explicitly removes them.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clicking a column header reloads from page 1 with ORDER BY col ASC/DESC
NULLS LAST passed to the view query. Sort column is validated against
information_schema.columns to prevent injection. Pagination preserves
the active sort across prev/next.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Click any column header to sort asc/desc (⇅ / ▲ / ▼ indicators)
- Sort is client-side within the current page, numeric-aware
- Dates matching ISO format are displayed as e.g. "Apr 5, 26"
- Sort resets on source change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Username and 'Sign out' link now sit on their own row under the
Dataflow title, so they always have room to display regardless of
title width.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On mobile: hamburger button in top bar opens sidebar as a slide-over
with a backdrop overlay. Nav links and source selector close it on tap.
On desktop: sidebar is static as before.
Logout button now sits alongside the close button in the sidebar header
with a gap so both are always visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Express auth middleware checks Authorization: Basic header on all /api
routes using bcrypt against LOGIN_USER/LOGIN_PASSWORD_HASH in .env
- React login screen shown before app loads, stores credentials in memory,
sends them with every API request, clears and returns to login on 401
- Logout button in sidebar header
- manage.py option 9: set login credentials (bcrypt via node, writes to .env)
- manage.py status shows whether login credentials are configured
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Show exact commands that will be run before each confirm prompt
- Step 1 dialog now offers schema and function deployment after writing .env
- Steps 2/3 relabeled as 'Redeploy only' for standalone use
- Option 5 (nginx) detects existing config and warns before overwriting
- Option 1 menu label clarified as 'Database configuration and deployment dialog'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo cp creates the file as root:root 0600, making it unreadable by the
app user. Add sudo chmod 644 after writing so status detection can read
it without sudo, matching how other nginx configs are set up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx config files written by sudo are root-only (rw-------), so
nginx_domain() was silently failing to read them. Now uses 'sudo -n cat'
with fallback to direct read for world-readable files.
Also fix PermissionError on cert_path.exists() — /etc/letsencrypt/live/
requires root, so use 'sudo test -f' instead of Path.exists().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
capture_output=True on sudo_run suppresses the sudo password prompt,
causing silent auth failure and permission denied on subsequent calls.
Removed capture_output from nginx -t and systemctl enable so the
password prompt and any error output appear on screen.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every status line, action header, confirm prompt, and ok/err message now
names exactly what it refers to — schema name, database, host, file paths,
and systemd commands. Menu items include source/target context. No ambiguous
shorthand anywhere.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Menu item: 'Configure database' → 'Configure database connection'
Intro text: disambiguates that the database is what gets created, not a connection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No external dependencies — uses psql CLI via subprocess. Interactive menu
detects current state (DB connection, schema, UI build, service status) and
guides through configure, deploy, rebuild, nginx setup, and service management.
Handles both new installs and existing databases (grants access vs. creating).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Collect all config upfront then print a plan showing every step
(active or skipped) before doing any work
- Prompt "Proceed? [Y/n]" at each section for granular control
- Add nginx reverse proxy setup with certbot SSL support
- Add overall "Continue?" confirmation after plan is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Merge setup.sh and deploy.sh into single deploy.sh
- First run (no .env): creates DB user/database, deploys schema +
functions, builds UI, installs systemd service
- Subsequent runs: optionally change DB target, redeploy functions,
rebuild UI, restart service
- Add dataflow.service systemd unit for process management
- Remove setup.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace three-tab layout with single unified grid and filter buttons
(All/Unmapped/Mapped with counts) in a sticky top control bar
- Require rule selection before loading any data
- Move source label, rule selector, filter, Save All, Export/Import into
sticky header that stays visible while scrolling
- Add inline delete (×) per mapped row — reverts to unmapped rather than
removing the row from view
- Simplify component state: drop separate unmapped/mapped state, derive
everything from allValues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_all_values() SQL function returning all extracted values (mapped
+ unmapped) with real record counts and mapping output in one query
- Add /mappings/source/:source/all-values API endpoint
- Rewrite All tab to use get_all_values directly instead of merging two
separate API calls; counts now populated for all rows
- Rewrite export.tsv to use get_all_values (real counts for mapped rows)
- Fix save bug where editing one output field blanked unedited fields by
merging drafts over existing mapping output instead of replacing
- Add dirty row highlighting (blue tint) and per-rule Save All button
- Fix sort instability during editing by sorting on committed values only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the nested FOR loops (row-by-row, rule-by-rule) with a single
SQL CTE chain that processes all records × rules in one pass, mirroring
the TPS approach.
CTE chain:
qualifying → all untransformed records for the source
rx → apply each rule (extract/replace) to each record
linked → LEFT JOIN mappings to find mapped output
rule_output → build per-rule JSONB (with retain support)
record_additions → merge all rule outputs per record in sequence order
UPDATE → set transformed = data || additions
Also adds jsonb_concat_obj aggregate (jsonb merge with ORDER BY support)
needed to collapse multiple rule outputs per record into one object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors TPS's retain: y behaviour — when a mapping is applied, the extracted
value is also written to output_field so both the raw extraction and the
mapped result are available in transformed data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore export.tsv and import-csv endpoints to mappings routes
- sample column is always last in export and discarded on import
- get_unmapped_values now returns distinct source field values as sample instead of full raw records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Export button downloads unmapped + existing mappings as TSV with sample column showing distinct source field values for context
- Import button uploads filled TSV, any non-system column treated as an output key
- Exclude *.tsv files from git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch apply_transformations from regexp_match to regexp_matches with
ORDINALITY, enabling the g flag to return all occurrences as a JSONB array
- Aggregate matches directly to JSONB in lateral subquery to avoid
text[][] type errors when subscripting array_agg results
- Pass flags as proper third argument to regexp_matches/regexp_replace
instead of inline (?flags) prefix — the only way g works correctly
- Apply same fix to preview and test endpoints in rules.js
- Add migrate_tps.sql script for migrating data from TPS to Dataflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Support multi-capture-group regex: mappings.input_value changed to JSONB,
regexp_match() result stored as scalar or array JSONB in transformed column
- Computed expression fields in generated views: {fieldname} refs substituted
with (transformed->>'fieldname')::numeric for arithmetic in view columns
- Fix generate_source_view to DROP VIEW before CREATE (avoids column drop error)
- Collapsible rule cards that open directly to inline edit form
- Debounced live regex preview (extract + replace) with popout modal for 50 rows
- Records page now shows dfv.<source> view output instead of raw records
- Unified field table in Sources: single table with In view, Seq, expression columns
- Fix "Rule already exists" error when editing by passing rule.id directly to submit
- Fix Sources page clearing on F5 by watching sourceObj?.name in useEffect dep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- POST /api/sources/suggest: derive source definition from CSV upload
- GET /api/sources/:name/import-log: query import history
- GET /api/rules/:id/test: test rule pattern against real records
- rules: add function_type (extract/replace) and flags columns
- get_unmapped_values: include up to 3 sample records per value
- npm start now uses nodemon for auto-reload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
