- Default selection mode is now SELECT_REGION
- plugin.save()/restore() used to capture and apply edit mode
- expand_depth tracked in ref and included in layout config
- applyExpandDepth helper restores depth on layout recall and page load
- Save button overwrites active layout in place (no re-typing name)
- captureConfig() helper shared by save-over and save-as flows
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace debug test buttons with a minimal 'depth: 0 1 2 3' control
in the pivot toolbar right side.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After testing plugin_config.expand_depth (no effect) and view.set_depth
+ flush() (no effect), confirmed that view.set_depth(d) followed by
plugin.draw(view) correctly collapses/expands all rows to depth d.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Temporary UI for testing programmatic row expansion control via
plugin_config.expand_depth in Perspective viewer.restore().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- pivot_layouts table (source_name, layout_name, config JSONB)
- list/save/delete SQL functions and API routes
- Pivot toolbar above viewer: layout chips, save-as inline input,
delete per layout, reset to default
- Applying a named layout also updates localStorage working state
- Layouts reload on source change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Always display all non-null metric columns from the clicked row.
When a specific cell can be identified (split_by in use, cell mode),
highlight that row in blue/bold. Fixes row mode showing only one value.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace __ROW_PATH__ zip approach with direct application of
perspective-click event filters against raw rows. Fields not
present in the raw data (Perspective computed columns like Month,
YearDate) are skipped. Also removes debug console.log calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Show row path prominently, filter to non-null metric values,
use group_by › split_by as section header.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix stale import_records in sources.sql that referenced deleted generate_constraint_key
- Auto-transform after import, auto-generate view after create
- New source form matches existing source layout (In view, Seq, type dropdown)
- Sample data table (50 rows) shown below field config in both new and existing source views
- Import sample CSV on create (checked by default)
- Sortable column headers on field table
- Choose CSV styled as a button showing filename
- + button in sidebar opens new source form
- Records tab shows error message when view cast fails instead of blank
- Pivot page with Perspective viewer, per-source saved layouts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename dedup_key/dedup_fields → constraint_key/constraint_fields everywhere
(schema, functions, routes, UI, migration script, docs)
- Change constraint_key from MD5 TEXT hash to readable JSONB object
- Drop unique constraint on (source_name, constraint_key); dedup is now
enforced at import time via CTE, allowing intra-file duplicate rows
- Add import_id FK (ON DELETE CASCADE) so deleting a log entry removes its records
- Add info JSONB to import_log with inserted_keys and excluded_keys arrays
- Add get_import_log, get_all_import_logs, delete_import SQL functions
- Auto-apply transformations immediately after import
- Import UI: expandable key detail, checkbox selection, delete with confirm,
import ID column, transform result display
- New Log page: global import log across all sources
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add import_id column to records (links each record to its import batch)
- import_records() now stores readable dedup field values (not hashes) in
info.inserted_keys / info.excluded_keys, and stamps import_id on insert
- delete_import() simplified to delete log row; ON DELETE CASCADE removes records
- Add get_import_log() and get_all_import_logs() DB functions
- Add DELETE /api/sources/:name/import-log/:id endpoint
- Add GET /api/sources/import-log global log endpoint
- Import route now auto-applies transformations to new records after import
- Import page: show ID column, expandable key detail, checkbox delete
- New Log page: global view of all imports across sources
- Update README API reference and workflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents manage.py menu, adds full API reference tables, fixes
incorrect route in quick example, and removes stale sections
(docs/ dir, initial development status).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add database/queries/{sources,rules,mappings,records}.sql — one file per
route, all business logic in PostgreSQL functions
- Replace parameterized queries in all four route files with lit()/jsonLit()
literal interpolation for debuggability
- Add api/lib/sql.js with lit(), jsonLit(), arr() helpers
- Fix get_view_data to use json_agg (preserves column order) with subquery
(guarantees sort order is respected before aggregation)
- Fix jsonLit() for JSONB params so plain strings become valid JSON
- Update manage.py option 3 to deploy database/queries/ instead of functions.sql
- Add SPEC.md covering architecture, philosophy, and manage.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After createMapping, the new mapping's id was not stored in allValues
state, so editing the row again fell into the create path instead of
update. Now stores created.id so subsequent saves correctly use updateMapping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Credentials are saved to sessionStorage on login and restored on mount,
so a page refresh re-authenticates silently. Closing the tab clears them.
Logout explicitly removes them.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clicking a column header reloads from page 1 with ORDER BY col ASC/DESC
NULLS LAST passed to the view query. Sort column is validated against
information_schema.columns to prevent injection. Pagination preserves
the active sort across prev/next.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Click any column header to sort asc/desc (⇅ / ▲ / ▼ indicators)
- Sort is client-side within the current page, numeric-aware
- Dates matching ISO format are displayed as e.g. "Apr 5, 26"
- Sort resets on source change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Username and 'Sign out' link now sit on their own row under the
Dataflow title, so they always have room to display regardless of
title width.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On mobile: hamburger button in top bar opens sidebar as a slide-over
with a backdrop overlay. Nav links and source selector close it on tap.
On desktop: sidebar is static as before.
Logout button now sits alongside the close button in the sidebar header
with a gap so both are always visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Express auth middleware checks Authorization: Basic header on all /api
routes using bcrypt against LOGIN_USER/LOGIN_PASSWORD_HASH in .env
- React login screen shown before app loads, stores credentials in memory,
sends them with every API request, clears and returns to login on 401
- Logout button in sidebar header
- manage.py option 9: set login credentials (bcrypt via node, writes to .env)
- manage.py status shows whether login credentials are configured
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Show exact commands that will be run before each confirm prompt
- Step 1 dialog now offers schema and function deployment after writing .env
- Steps 2/3 relabeled as 'Redeploy only' for standalone use
- Option 5 (nginx) detects existing config and warns before overwriting
- Option 1 menu label clarified as 'Database configuration and deployment dialog'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo cp creates the file as root:root 0600, making it unreadable by the
app user. Add sudo chmod 644 after writing so status detection can read
it without sudo, matching how other nginx configs are set up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx config files written by sudo are root-only (rw-------), so
nginx_domain() was silently failing to read them. Now uses 'sudo -n cat'
with fallback to direct read for world-readable files.
Also fix PermissionError on cert_path.exists() — /etc/letsencrypt/live/
requires root, so use 'sudo test -f' instead of Path.exists().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
capture_output=True on sudo_run suppresses the sudo password prompt,
causing silent auth failure and permission denied on subsequent calls.
Removed capture_output from nginx -t and systemctl enable so the
password prompt and any error output appear on screen.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every status line, action header, confirm prompt, and ok/err message now
names exactly what it refers to — schema name, database, host, file paths,
and systemd commands. Menu items include source/target context. No ambiguous
shorthand anywhere.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Menu item: 'Configure database' → 'Configure database connection'
Intro text: disambiguates that the database is what gets created, not a connection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No external dependencies — uses psql CLI via subprocess. Interactive menu
detects current state (DB connection, schema, UI build, service status) and
guides through configure, deploy, rebuild, nginx setup, and service management.
Handles both new installs and existing databases (grants access vs. creating).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Collect all config upfront then print a plan showing every step
(active or skipped) before doing any work
- Prompt "Proceed? [Y/n]" at each section for granular control
- Add nginx reverse proxy setup with certbot SSL support
- Add overall "Continue?" confirmation after plan is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Merge setup.sh and deploy.sh into single deploy.sh
- First run (no .env): creates DB user/database, deploys schema +
functions, builds UI, installs systemd service
- Subsequent runs: optionally change DB target, redeploy functions,
rebuild UI, restart service
- Add dataflow.service systemd unit for process management
- Remove setup.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace three-tab layout with single unified grid and filter buttons
(All/Unmapped/Mapped with counts) in a sticky top control bar
- Require rule selection before loading any data
- Move source label, rule selector, filter, Save All, Export/Import into
sticky header that stays visible while scrolling
- Add inline delete (×) per mapped row — reverts to unmapped rather than
removing the row from view
- Simplify component state: drop separate unmapped/mapped state, derive
everything from allValues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_all_values() SQL function returning all extracted values (mapped
+ unmapped) with real record counts and mapping output in one query
- Add /mappings/source/:source/all-values API endpoint
- Rewrite All tab to use get_all_values directly instead of merging two
separate API calls; counts now populated for all rows
- Rewrite export.tsv to use get_all_values (real counts for mapped rows)
- Fix save bug where editing one output field blanked unedited fields by
merging drafts over existing mapping output instead of replacing
- Add dirty row highlighting (blue tint) and per-rule Save All button
- Fix sort instability during editing by sorting on committed values only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the nested FOR loops (row-by-row, rule-by-rule) with a single
SQL CTE chain that processes all records × rules in one pass, mirroring
the TPS approach.
CTE chain:
qualifying → all untransformed records for the source
rx → apply each rule (extract/replace) to each record
linked → LEFT JOIN mappings to find mapped output
rule_output → build per-rule JSONB (with retain support)
record_additions → merge all rule outputs per record in sequence order
UPDATE → set transformed = data || additions
Also adds jsonb_concat_obj aggregate (jsonb merge with ORDER BY support)
needed to collapse multiple rule outputs per record into one object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors TPS's retain: y behaviour — when a mapping is applied, the extracted
value is also written to output_field so both the raw extraction and the
mapped result are available in transformed data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore export.tsv and import-csv endpoints to mappings routes
- sample column is always last in export and discarded on import
- get_unmapped_values now returns distinct source field values as sample instead of full raw records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Export button downloads unmapped + existing mappings as TSV with sample column showing distinct source field values for context
- Import button uploads filled TSV, any non-system column treated as an output key
- Exclude *.tsv files from git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch apply_transformations from regexp_match to regexp_matches with
ORDINALITY, enabling the g flag to return all occurrences as a JSONB array
- Aggregate matches directly to JSONB in lateral subquery to avoid
text[][] type errors when subscripting array_agg results
- Pass flags as proper third argument to regexp_matches/regexp_replace
instead of inline (?flags) prefix — the only way g works correctly
- Apply same fix to preview and test endpoints in rules.js
- Add migrate_tps.sql script for migrating data from TPS to Dataflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
