sudo cp creates the file as root:root 0600, making it unreadable by the
app user. Add sudo chmod 644 after writing so status detection can read
it without sudo, matching how other nginx configs are set up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx config files written by sudo are root-only (rw-------), so
nginx_domain() was silently failing to read them. Now uses 'sudo -n cat'
with fallback to direct read for world-readable files.
Also fix PermissionError on cert_path.exists() — /etc/letsencrypt/live/
requires root, so use 'sudo test -f' instead of Path.exists().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
capture_output=True on sudo_run suppresses the sudo password prompt,
causing silent auth failure and permission denied on subsequent calls.
Removed capture_output from nginx -t and systemctl enable so the
password prompt and any error output appear on screen.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every status line, action header, confirm prompt, and ok/err message now
names exactly what it refers to — schema name, database, host, file paths,
and systemd commands. Menu items include source/target context. No ambiguous
shorthand anywhere.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Menu item: 'Configure database' → 'Configure database connection'
Intro text: disambiguates that the database is what gets created, not a connection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No external dependencies — uses psql CLI via subprocess. Interactive menu
detects current state (DB connection, schema, UI build, service status) and
guides through configure, deploy, rebuild, nginx setup, and service management.
Handles both new installs and existing databases (grants access vs. creating).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Collect all config upfront then print a plan showing every step
(active or skipped) before doing any work
- Prompt "Proceed? [Y/n]" at each section for granular control
- Add nginx reverse proxy setup with certbot SSL support
- Add overall "Continue?" confirmation after plan is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Merge setup.sh and deploy.sh into single deploy.sh
- First run (no .env): creates DB user/database, deploys schema +
functions, builds UI, installs systemd service
- Subsequent runs: optionally change DB target, redeploy functions,
rebuild UI, restart service
- Add dataflow.service systemd unit for process management
- Remove setup.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace three-tab layout with single unified grid and filter buttons
(All/Unmapped/Mapped with counts) in a sticky top control bar
- Require rule selection before loading any data
- Move source label, rule selector, filter, Save All, Export/Import into
sticky header that stays visible while scrolling
- Add inline delete (×) per mapped row — reverts to unmapped rather than
removing the row from view
- Simplify component state: drop separate unmapped/mapped state, derive
everything from allValues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_all_values() SQL function returning all extracted values (mapped
+ unmapped) with real record counts and mapping output in one query
- Add /mappings/source/:source/all-values API endpoint
- Rewrite All tab to use get_all_values directly instead of merging two
separate API calls; counts now populated for all rows
- Rewrite export.tsv to use get_all_values (real counts for mapped rows)
- Fix save bug where editing one output field blanked unedited fields by
merging drafts over existing mapping output instead of replacing
- Add dirty row highlighting (blue tint) and per-rule Save All button
- Fix sort instability during editing by sorting on committed values only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the nested FOR loops (row-by-row, rule-by-rule) with a single
SQL CTE chain that processes all records × rules in one pass, mirroring
the TPS approach.
CTE chain:
qualifying → all untransformed records for the source
rx → apply each rule (extract/replace) to each record
linked → LEFT JOIN mappings to find mapped output
rule_output → build per-rule JSONB (with retain support)
record_additions → merge all rule outputs per record in sequence order
UPDATE → set transformed = data || additions
Also adds jsonb_concat_obj aggregate (jsonb merge with ORDER BY support)
needed to collapse multiple rule outputs per record into one object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors TPS's retain: y behaviour — when a mapping is applied, the extracted
value is also written to output_field so both the raw extraction and the
mapped result are available in transformed data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore export.tsv and import-csv endpoints to mappings routes
- sample column is always last in export and discarded on import
- get_unmapped_values now returns distinct source field values as sample instead of full raw records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Export button downloads unmapped + existing mappings as TSV with sample column showing distinct source field values for context
- Import button uploads filled TSV, any non-system column treated as an output key
- Exclude *.tsv files from git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch apply_transformations from regexp_match to regexp_matches with
ORDINALITY, enabling the g flag to return all occurrences as a JSONB array
- Aggregate matches directly to JSONB in lateral subquery to avoid
text[][] type errors when subscripting array_agg results
- Pass flags as proper third argument to regexp_matches/regexp_replace
instead of inline (?flags) prefix — the only way g works correctly
- Apply same fix to preview and test endpoints in rules.js
- Add migrate_tps.sql script for migrating data from TPS to Dataflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Support multi-capture-group regex: mappings.input_value changed to JSONB,
regexp_match() result stored as scalar or array JSONB in transformed column
- Computed expression fields in generated views: {fieldname} refs substituted
with (transformed->>'fieldname')::numeric for arithmetic in view columns
- Fix generate_source_view to DROP VIEW before CREATE (avoids column drop error)
- Collapsible rule cards that open directly to inline edit form
- Debounced live regex preview (extract + replace) with popout modal for 50 rows
- Records page now shows dfv.<source> view output instead of raw records
- Unified field table in Sources: single table with In view, Seq, expression columns
- Fix "Rule already exists" error when editing by passing rule.id directly to submit
- Fix Sources page clearing on F5 by watching sourceObj?.name in useEffect dep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- POST /api/sources/suggest: derive source definition from CSV upload
- GET /api/sources/:name/import-log: query import history
- GET /api/rules/:id/test: test rule pattern against real records
- rules: add function_type (extract/replace) and flags columns
- get_unmapped_values: include up to 3 sample records per value
- npm start now uses nodemon for auto-reload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
