get_source_fields only unioned schema fields, raw data keys, rule
output_field and mapping output keys. Keys that live solely in
records.transformed or records.overrides — a manual override such as
dcard's "Note", or a transformed key whose rule was since deleted — never
appeared on the source page, so there was no way to add them to the view.
Read both columns off the records directly.
Records showed no id column, leaving no handle to identify a row. Split
the hidden-column set: HIDDEN_COLS still keeps id out of the override
editor, GRID_HIDDEN_COLS hides only _overridden, and gridCols() pins id
first in the grid and the filter dropdown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BDFU3ueEbCYGE37oDgeu2j
database/functions.sql held a full pg_dump appended onto the original
hand-written file, duplicating ~50 functions that had since been split into
database/queries/. Nothing deployed it, but CLAUDE.md and the tutorial still
told you to psql it, which would have reverted the split versions.
The reverse had also happened: five functions in queries/ were behind the live
database, all of them undoing the May 2026 split of the transformed column.
preview_rule lost its data -> transformed fallback for chained rules;
set_/clear_/bulk_set_record_overrides wrote overrides back into transformed and
returned the wrong type (which would have made the redeploy error outright);
generate_source_view read only transformed instead of merging all three layers.
Those are corrected here from the live definitions.
generate_source_view additionally regains the _overridden column that queries/
had and live lacked — Records.jsx reads row._overridden to highlight manually
edited rows, so that indicator had been dead.
The seven functions that existed only in functions.sql move to two new files,
import.sql (import + audit trail) and transform.sql (the rule/mapping engine),
leaving database/ flat: schema.sql plus one file per route. The four already
applied migrate_*.sql scripts are removed.
manage.py picks up the new files in QUERY_FILES, and its DB_ACTIONS set now
keys off the action functions rather than duplicated label strings that no
longer matched any menu entry, so the "into database X" hint renders again.
uninstall.sh is folded into manage.py as menu option 10. Beyond what the script
did, it stops/disables/removes the systemd unit, removes the nginx site with an
nginx -t check before reloading, and deletes public/.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- transformed now stores only rule additions (not merged data+overrides)
- View dynamically computes data || transformed || overrides at query time
- New DB functions: set/clear/bulk_set_record_overrides
- Records panel now includes source-wide override keys so party/reason etc.
appear even on records that don't have them set yet
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pivot_layouts.source_name had a FK to sources(name) preventing stack names
from being used as layout keys. Dropped the FK so any view name works.
- database/migrate_pivot_layouts_drop_fk.sql: drop the FK constraint
- api/routes/stacks.js: add GET/POST/DELETE /:name/layouts routes
- ui/src/api.js: add getStackPivotLayouts / saveStackPivotLayout / deleteStackPivotLayout
- ui/src/pages/Pivot.jsx: use DB for stack layouts instead of localStorage;
collapse source/stack branches into saveLayout/deleteLayout helpers
- CLAUDE.md: document pivot layout persistence pattern
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rules/mappings changes don't affect view SQL (views read from
transformed, shaped by config.fields) — remove those triggers.
Replace with a BEFORE UPDATE trigger on sources that only clears
view_generated_at when config actually changes.
Stack sources trigger now skips no-op upserts: the live SQL preview
calls upsertStackSource on every edit, which was unconditionally
clearing view_generated_at even when nothing changed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bulk_set_record_overrides() DB function merges overrides into multiple
records at once using a CTE with RETURNING for accurate count
- POST /records/bulk-overrides calls the function (consistent with rest
of API — no raw SQL in routes)
- UI: regex input on loaded rows selects rows for bulk override; labeled
"Bulk select:" / "DB query:" to distinguish from server-side filters
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add seq column to stack_sources; existing rows seeded by insertion order
- New sources auto-assigned max(seq)+1 so they always append to the end
- get_stack and generate_stack_view now order by seq instead of source_name
- Add reorder_stack_sources() function and PUT /:name/sources/reorder endpoint
- Source cards have drag handles matching the output columns grid behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Calibrate modal now auto-fetches computed sum and shows live reconciliation table (data sum, known balance, plug) without requiring a button click
- as_of_date is now optional in calibrate — omitting it sums all transactions
- SQL preview syncs current UI state to DB before fetching so preview is always accurate
- Pivot cleanLayout strips stale columns from saved layouts when switching stack views
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Two-column layout: config left, SQL panel right (equal halves)
- SQL panel shows formatted SQL (sql-formatter, 4-space indent)
- Live preview: SQL updates 400ms after any field/source/mapping change
- Run button executes edited SQL directly via new exec-sql endpoint
- generate_stack_view gains p_dry_run mode for preview without executing
- CASCADE drop detects dependent stacks, marks them stale in DB and status bar
- net_balance moved to last column in generated view
- Backfill 458 missing dcard rows and 123 missing chase rows from TPS migration bug
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_status() SQL and /api/status route; load stale state on login
- Replace polling with immediate client-side stale tracking via callbacks
- Amber status bar with per-item Generate buttons for sources and stacks
- Pivot: add stack selector to view any dfv.stack view via Perspective
- Stack views: DROP CASCADE, add id to source views, per-source balance columns
- net_balance = sum(all amounts) + total_offset guarantees chase+dcard=net per row
- CLAUDE.md: document correct dedup spec (within-batch duplicates always allowed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema:
- records.overrides JSONB column (ALTER TABLE, already applied)
- apply_transformations merges overrides on top: data || rules || overrides
- generate_source_view always includes id and _overridden columns
- set_record_overrides(id, overrides): stores and immediately merges into transformed
- clear_record_overrides(id): clears overrides then reprocesses record
API:
- PUT /records/:id/overrides — set overrides
- DELETE /records/:id/overrides — clear and reprocess
UI (Records page):
- Rows are clickable; overridden rows highlighted amber
- Side panel shows all transformed fields as editable inputs
- Overridden fields highlighted amber with pencil indicator
- Save stores overrides; Clear removes them and restores computed values
- id and _overridden hidden from table display
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL: search_mapping_outputs(search) — distinct (col, val, count) groups
get_mappings_by_output_field(col, val) — individual mappings
remap_output_field(col, from, to) — bulk UPDATE via jsonb_set
- API: GET /mappings/outputs?search=, GET /mappings/outputs/:col/:val,
POST /mappings/remap-field
- UI: Remap page — search output values, click to select, edit the
replacement value, see all affected mappings, apply globally
- Nav: Remap added between Mappings and Records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- pivot_layouts table (source_name, layout_name, config JSONB)
- list/save/delete SQL functions and API routes
- Pivot toolbar above viewer: layout chips, save-as inline input,
delete per layout, reset to default
- Applying a named layout also updates localStorage working state
- Layouts reload on source change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix stale import_records in sources.sql that referenced deleted generate_constraint_key
- Auto-transform after import, auto-generate view after create
- New source form matches existing source layout (In view, Seq, type dropdown)
- Sample data table (50 rows) shown below field config in both new and existing source views
- Import sample CSV on create (checked by default)
- Sortable column headers on field table
- Choose CSV styled as a button showing filename
- + button in sidebar opens new source form
- Records tab shows error message when view cast fails instead of blank
- Pivot page with Perspective viewer, per-source saved layouts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename dedup_key/dedup_fields → constraint_key/constraint_fields everywhere
(schema, functions, routes, UI, migration script, docs)
- Change constraint_key from MD5 TEXT hash to readable JSONB object
- Drop unique constraint on (source_name, constraint_key); dedup is now
enforced at import time via CTE, allowing intra-file duplicate rows
- Add import_id FK (ON DELETE CASCADE) so deleting a log entry removes its records
- Add info JSONB to import_log with inserted_keys and excluded_keys arrays
- Add get_import_log, get_all_import_logs, delete_import SQL functions
- Auto-apply transformations immediately after import
- Import UI: expandable key detail, checkbox selection, delete with confirm,
import ID column, transform result display
- New Log page: global import log across all sources
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add import_id column to records (links each record to its import batch)
- import_records() now stores readable dedup field values (not hashes) in
info.inserted_keys / info.excluded_keys, and stamps import_id on insert
- delete_import() simplified to delete log row; ON DELETE CASCADE removes records
- Add get_import_log() and get_all_import_logs() DB functions
- Add DELETE /api/sources/:name/import-log/:id endpoint
- Add GET /api/sources/import-log global log endpoint
- Import route now auto-applies transformations to new records after import
- Import page: show ID column, expandable key detail, checkbox delete
- New Log page: global view of all imports across sources
- Update README API reference and workflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add database/queries/{sources,rules,mappings,records}.sql — one file per
route, all business logic in PostgreSQL functions
- Replace parameterized queries in all four route files with lit()/jsonLit()
literal interpolation for debuggability
- Add api/lib/sql.js with lit(), jsonLit(), arr() helpers
- Fix get_view_data to use json_agg (preserves column order) with subquery
(guarantees sort order is respected before aggregation)
- Fix jsonLit() for JSONB params so plain strings become valid JSON
- Update manage.py option 3 to deploy database/queries/ instead of functions.sql
- Add SPEC.md covering architecture, philosophy, and manage.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_all_values() SQL function returning all extracted values (mapped
+ unmapped) with real record counts and mapping output in one query
- Add /mappings/source/:source/all-values API endpoint
- Rewrite All tab to use get_all_values directly instead of merging two
separate API calls; counts now populated for all rows
- Rewrite export.tsv to use get_all_values (real counts for mapped rows)
- Fix save bug where editing one output field blanked unedited fields by
merging drafts over existing mapping output instead of replacing
- Add dirty row highlighting (blue tint) and per-rule Save All button
- Fix sort instability during editing by sorting on committed values only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the nested FOR loops (row-by-row, rule-by-rule) with a single
SQL CTE chain that processes all records × rules in one pass, mirroring
the TPS approach.
CTE chain:
qualifying → all untransformed records for the source
rx → apply each rule (extract/replace) to each record
linked → LEFT JOIN mappings to find mapped output
rule_output → build per-rule JSONB (with retain support)
record_additions → merge all rule outputs per record in sequence order
UPDATE → set transformed = data || additions
Also adds jsonb_concat_obj aggregate (jsonb merge with ORDER BY support)
needed to collapse multiple rule outputs per record into one object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors TPS's retain: y behaviour — when a mapping is applied, the extracted
value is also written to output_field so both the raw extraction and the
mapped result are available in transformed data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Restore export.tsv and import-csv endpoints to mappings routes
- sample column is always last in export and discarded on import
- get_unmapped_values now returns distinct source field values as sample instead of full raw records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch apply_transformations from regexp_match to regexp_matches with
ORDINALITY, enabling the g flag to return all occurrences as a JSONB array
- Aggregate matches directly to JSONB in lateral subquery to avoid
text[][] type errors when subscripting array_agg results
- Pass flags as proper third argument to regexp_matches/regexp_replace
instead of inline (?flags) prefix — the only way g works correctly
- Apply same fix to preview and test endpoints in rules.js
- Add migrate_tps.sql script for migrating data from TPS to Dataflow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Support multi-capture-group regex: mappings.input_value changed to JSONB,
regexp_match() result stored as scalar or array JSONB in transformed column
- Computed expression fields in generated views: {fieldname} refs substituted
with (transformed->>'fieldname')::numeric for arithmetic in view columns
- Fix generate_source_view to DROP VIEW before CREATE (avoids column drop error)
- Collapsible rule cards that open directly to inline edit form
- Debounced live regex preview (extract + replace) with popout modal for 50 rows
- Records page now shows dfv.<source> view output instead of raw records
- Unified field table in Sources: single table with In view, Seq, expression columns
- Fix "Rule already exists" error when editing by passing rule.id directly to submit
- Fix Sources page clearing on F5 by watching sourceObj?.name in useEffect dep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- POST /api/sources/suggest: derive source definition from CSV upload
- GET /api/sources/:name/import-log: query import history
- GET /api/rules/:id/test: test rule pattern against real records
- rules: add function_type (extract/replace) and flags columns
- get_unmapped_values: include up to 3 sample records per value
- npm start now uses nodemon for auto-reload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
