- Merge setup.sh and deploy.sh into single deploy.sh
- First run (no .env): creates DB user/database, deploys schema +
functions, builds UI, installs systemd service
- Subsequent runs: optionally change DB target, redeploy functions,
rebuild UI, restart service
- Add dataflow.service systemd unit for process management
- Remove setup.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace three-tab layout with single unified grid and filter buttons
(All/Unmapped/Mapped with counts) in a sticky top control bar
- Require rule selection before loading any data
- Move source label, rule selector, filter, Save All, Export/Import into
sticky header that stays visible while scrolling
- Add inline delete (×) per mapped row — reverts to unmapped rather than
removing the row from view
- Simplify component state: drop separate unmapped/mapped state, derive
everything from allValues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_all_values() SQL function returning all extracted values (mapped
+ unmapped) with real record counts and mapping output in one query
- Add /mappings/source/:source/all-values API endpoint
- Rewrite All tab to use get_all_values directly instead of merging two
separate API calls; counts now populated for all rows
- Rewrite export.tsv to use get_all_values (real counts for mapped rows)
- Fix save bug where editing one output field blanked unedited fields by
merging drafts over existing mapping output instead of replacing
- Add dirty row highlighting (blue tint) and per-rule Save All button
- Fix sort instability during editing by sorting on committed values only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the nested FOR loops (row-by-row, rule-by-rule) with a single
SQL CTE chain that processes all records × rules in one pass, mirroring
the TPS approach.
CTE chain:
qualifying → all untransformed records for the source
rx → apply each rule (extract/replace) to each record
linked → LEFT JOIN mappings to find mapped output
rule_output → build per-rule JSONB (with retain support)
record_additions → merge all rule outputs per record in sequence order
UPDATE → set transformed = data || additions
Also adds jsonb_concat_obj aggregate (jsonb merge with ORDER BY support)
needed to collapse multiple rule outputs per record into one object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors TPS's retain: y behaviour — when a mapping is applied, the extracted
value is also written to output_field so both the raw extraction and the
mapped result are available in transformed data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore export.tsv and import-csv endpoints to mappings routes
- sample column is always last in export and discarded on import
- get_unmapped_values now returns distinct source field values as sample instead of full raw records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Export button downloads unmapped + existing mappings as TSV with sample column showing distinct source field values for context
- Import button uploads filled TSV, any non-system column treated as an output key
- Exclude *.tsv files from git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch apply_transformations from regexp_match to regexp_matches with
ORDINALITY, enabling the g flag to return all occurrences as a JSONB array
- Aggregate matches directly to JSONB in lateral subquery to avoid
text[][] type errors when subscripting array_agg results
- Pass flags as proper third argument to regexp_matches/regexp_replace
instead of inline (?flags) prefix — the only way g works correctly
- Apply same fix to preview and test endpoints in rules.js
- Add migrate_tps.sql script for migrating data from TPS to Dataflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Support multi-capture-group regex: mappings.input_value changed to JSONB,
regexp_match() result stored as scalar or array JSONB in transformed column
- Computed expression fields in generated views: {fieldname} refs substituted
with (transformed->>'fieldname')::numeric for arithmetic in view columns
- Fix generate_source_view to DROP VIEW before CREATE (avoids column drop error)
- Collapsible rule cards that open directly to inline edit form
- Debounced live regex preview (extract + replace) with popout modal for 50 rows
- Records page now shows dfv.<source> view output instead of raw records
- Unified field table in Sources: single table with In view, Seq, expression columns
- Fix "Rule already exists" error when editing by passing rule.id directly to submit
- Fix Sources page clearing on F5 by watching sourceObj?.name in useEffect dep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- POST /api/sources/suggest: derive source definition from CSV upload
- GET /api/sources/:name/import-log: query import history
- GET /api/rules/:id/test: test rule pattern against real records
- rules: add function_type (extract/replace) and flags columns
- get_unmapped_values: include up to 3 sample records per value
- npm start now uses nodemon for auto-reload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
