Field discovery now samples an account's actual transactions instead of
assuming a shape. flatten() passes through every scalar the bridge sends
rather than whitelisting eleven keys, so institution-specific fields turn
up on their own, and inferFields() — extracted from the CSV suggest route
so both paths share it — unions keys across the sample because API feeds
omit optional fields entirely.
Three bugs the live bridge exposed:
- posted=0 on pending transactions became 1970-01-01; falsy epochs are
now "no date", with date falling back to transacted_at and posted_date
kept separate.
- days=0 omitted start-date, which returns only the few most recent
transactions rather than everything — 4 instead of 89. A start-date is
always sent now, clamped to 89 days (the bridge hard-caps at 90).
- Sampling asked for more than 45 days, and the bridge's advisory notice
about that surfaced in the UI as an error. Samples use 44 days; the
threshold is exclusive.
The Sources page can now link an account: a picker in both the create
dialog and the detail panel, populated on demand, which fills the field
table from the sample and defaults the constraint field to the
transaction id with an explanation of why.
manage.py option 10 claims a setup token and writes the access URL to
.env, replacing the throwaway script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G2HFeU5neCKagTnmA6o9Tu
Sources with a `simplefin` block in their config can pull transactions
straight from the bridge instead of taking a CSV upload. Only the fetch
differs — dedupe, logging, and transformation reuse the import path.
The access URL is the whole credential, so it lives in .env rather than
the database that manage.py offers to reset. Claiming a setup token is
exposed as an endpoint because the token is single-use and easy to burn.
The bridge answers 200 with a populated `errors` array when a bank is
failing, which would otherwise read as a successful empty pull — those
errors ride along in the sync response and show on the Import page.
Pending transactions are skipped by default: they get a new id once they
post, which would import the same charge twice under two keys. Sources
should use ['id'] as constraint_fields — the transaction id makes
overlapping pulls free while keeping genuinely repeated charges distinct.
Verified against a stubbed bridge response, not a live account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G2HFeU5neCKagTnmA6o9Tu
get_source_fields only unioned schema fields, raw data keys, rule
output_field and mapping output keys. Keys that live solely in
records.transformed or records.overrides — a manual override such as
dcard's "Note", or a transformed key whose rule was since deleted — never
appeared on the source page, so there was no way to add them to the view.
Read both columns off the records directly.
Records showed no id column, leaving no handle to identify a row. Split
the hidden-column set: HIDDEN_COLS still keeps id out of the override
editor, GRID_HIDDEN_COLS hides only _overridden, and gridCols() pins id
first in the grid and the filter dropdown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BDFU3ueEbCYGE37oDgeu2j
The page loaded unsorted, so the most recent rows were rarely on screen. On
source change it now looks up the source's first date-typed field and sorts
descending on it, falling back to unsorted when a source has no date field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Vite scaffolding, unmodified since the UI was added. Everything in it that this
project produces is already covered by the root .gitignore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Architecture, file structure, the manage.py menu, and the API reference were
each documented in two or three of README.md, SPEC.md, and CLAUDE.md — the same
drift trap the SQL just had.
SPEC.md, examples/GETTING_STARTED.md, and ui/README.md move into docs/.
PERSPECTIVE.md and docs/perspective-pivot.md merge into docs/perspective.md,
version rationale first, then the API reference. README.md becomes an entry
point that links out, and CLAUDE.md keeps only working rules and non-obvious
behaviour, pointing at docs/spec.md for the rest. examples/ keeps just the
sample CSV the tutorial loads.
Corrections found while consolidating:
- the spec's API table was missing 20 routes — every override endpoint, most of
/api/stacks, the mapping remap routes, /health. Rebuilt from the route files
- the tutorial used port 3000 (default is 3020) and never mentioned Basic auth,
so every curl in it would have 401'd
- the tutorial and the spec each hand-listed the SQL deploy order; both now
point at manage.py, which is where the order actually lives
- CLAUDE.md described deduplication as an MD5 hash (it is a plain JSONB object),
claimed 5 tables and 4 functions, and told you to run a setup.sh that has not
existed for some time
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
database/functions.sql held a full pg_dump appended onto the original
hand-written file, duplicating ~50 functions that had since been split into
database/queries/. Nothing deployed it, but CLAUDE.md and the tutorial still
told you to psql it, which would have reverted the split versions.
The reverse had also happened: five functions in queries/ were behind the live
database, all of them undoing the May 2026 split of the transformed column.
preview_rule lost its data -> transformed fallback for chained rules;
set_/clear_/bulk_set_record_overrides wrote overrides back into transformed and
returned the wrong type (which would have made the redeploy error outright);
generate_source_view read only transformed instead of merging all three layers.
Those are corrected here from the live definitions.
generate_source_view additionally regains the _overridden column that queries/
had and live lacked — Records.jsx reads row._overridden to highlight manually
edited rows, so that indicator had been dead.
The seven functions that existed only in functions.sql move to two new files,
import.sql (import + audit trail) and transform.sql (the rule/mapping engine),
leaving database/ flat: schema.sql plus one file per route. The four already
applied migrate_*.sql scripts are removed.
manage.py picks up the new files in QUERY_FILES, and its DB_ACTIONS set now
keys off the action functions rather than duplicated label strings that no
longer matched any menu entry, so the "into database X" hint renders again.
uninstall.sh is folded into manage.py as menu option 10. Beyond what the script
did, it stops/disables/removes the systemd unit, removes the nginx site with an
nginx -t check before reloading, and deletes public/.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The caret ranges in ui/package.json (^4.5.1 viewer/client/datagrid, ^4.4.1
viewer-d3fc) let npm install resolve a newer Perspective on a fresh machine.
That pairing is deliberate and load-bearing — 4.4.1 lacks the /inline export
paths Pivot.jsx imports, and viewer-d3fc has no 4.5.x — so the resolved tree
needs pinning, not just the ranges.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop deploy.sh, scripts/setup-service.sh, and migrate/ — all superseded by
manage.py. Remove leftover Vite template assets (App.css, hero.png, react.svg,
vite.svg), none of which are imported.
Hoist the database/queries/ file list in manage.py to a module-level
QUERY_FILES constant so configure and deploy-functions share one definition,
and add the stacks/status query files that were missing from it.
Swap npm start/dev so start runs node and dev runs nodemon.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records why the 4.5.1 viewer/client + 4.4.1 d3fc pairing is deliberate,
not a skew to "fix": the /inline and /themes entrypoints exist only in
4.5.x, while viewer-d3fc caps at 4.4.1, so this is the only combination
that keeps both inline WASM bundling and the d3fc charts. Verified by
build failure when pinning all four to 4.4.1. Points to the canonical
guide in pf_app for shared rationale.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace pool.on('connect') query with connection-level options parameter.
Avoids calling client.query() during handshake, which pg will remove in v9.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All APIs compatible with existing code. Added quiet:true to dotenv config
to suppress the new startup log message added in dotenv 17.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace runtime CDN imports with static ESM imports from npm packages.
Uses @perspective-dev/client and viewer inline builds (WASM embedded).
Bumps all packages to 4.5.1; d3fc stays at 4.4.1 (no 4.5.x release yet).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- transformed now stores only rule additions (not merged data+overrides)
- View dynamically computes data || transformed || overrides at query time
- New DB functions: set/clear/bulk_set_record_overrides
- Records panel now includes source-wide override keys so party/reason etc.
appear even on records that don't have them set yet
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Stack selection lifted to App.jsx: stacks fetched on login, selectedStack
state shared via StatusBar (pills) and Pivot (view switching); Stacks page
calls onStacksChange to keep list fresh
- Pivot: derive selectedView/viewType from props, remove local stack state;
toolbar replaced with dedicated layouts sub-bar (h-9, layouts only)
- Records panel: merge read-only and override sections into single field list;
known cols seeded from record's transformed fields; rule-derived fields
(transformed minus data) will be editable in follow-up refactor
- Pivot theme: setAttribute moved to after flush() so restore() can't reset it
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New Sidebar component (modelled on pf_app): collapses 200px→48px via
hamburger toggle, persists state to df_sidebar in localStorage; each
nav item has an SVG icon with label that fades out when collapsed;
user avatar + sign-out at bottom
- New StatusBar component: source picker + dark-mode toggle across the
top of the content area
- Fix Pivot theme: setAttribute('theme') moved to after flush() so
viewer.restore() can no longer reset it back to light
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port light/dark mode from pf_app: ThemeProvider context, CSS custom
properties (Pro Dark palette), dark overrides for Tailwind classes, and
Perspective viewer theme sync in Pivot. Toggle button in sidebar header.
Improve toggle icons to Feather-style stroke SVGs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pivot_layouts.source_name had a FK to sources(name) preventing stack names
from being used as layout keys. Dropped the FK so any view name works.
- database/migrate_pivot_layouts_drop_fk.sql: drop the FK constraint
- api/routes/stacks.js: add GET/POST/DELETE /:name/layouts routes
- ui/src/api.js: add getStackPivotLayouts / saveStackPivotLayout / deleteStackPivotLayout
- ui/src/pages/Pivot.jsx: use DB for stack layouts instead of localStorage;
collapse source/stack branches into saveLayout/deleteLayout helpers
- CLAUDE.md: document pivot layout persistence pattern
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Click cell to open inspector pane; click same cell again to close (toggle).
Uses __ROW_PATH__ + column_names as key so it works on both sources and stacks.
Removes event listener on view change to prevent listener accumulation.
- Drag handle on left edge of inspector pane for resizing (min 240px)
- Removed redundant cell-coordinates block; breadcrumb now inline in header
- Sortable columns: click header to sort asc/desc with ▲/▼ indicator
- Totals row: sums all-numeric columns, sticky at bottom
- Derive missing split_by filters from column_names when Perspective omits
them from detail.config.filter (fixes over-broad results on split_by views)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rules/mappings changes don't affect view SQL (views read from
transformed, shaped by config.fields) — remove those triggers.
Replace with a BEFORE UPDATE trigger on sources that only clears
view_generated_at when config actually changes.
Stack sources trigger now skips no-op upserts: the live SQL preview
calls upsertStackSource on every edit, which was unconditionally
clearing view_generated_at even when nothing changed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bulk_set_record_overrides() DB function merges overrides into multiple
records at once using a CTE with RETURNING for accurate count
- POST /records/bulk-overrides calls the function (consistent with rest
of API — no raw SQL in routes)
- UI: regex input on loaded rows selects rows for bulk override; labeled
"Bulk select:" / "DB query:" to distinguish from server-side filters
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add seq column to stack_sources; existing rows seeded by insertion order
- New sources auto-assigned max(seq)+1 so they always append to the end
- get_stack and generate_stack_view now order by seq instead of source_name
- Add reorder_stack_sources() function and PUT /:name/sources/reorder endpoint
- Source cards have drag handles matching the output columns grid behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- lit() was calling Math.trunc() on numbers, dropping decimals from balance_offset and any other numeric SQL params
- Stacks page now saves last selected stack to localStorage and restores it on load
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Calibrate modal now auto-fetches computed sum and shows live reconciliation table (data sum, known balance, plug) without requiring a button click
- as_of_date is now optional in calibrate — omitting it sums all transactions
- SQL preview syncs current UI state to DB before fetching so preview is always accurate
- Pivot cleanLayout strips stale columns from saved layouts when switching stack views
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Two-column layout: config left, SQL panel right (equal halves)
- SQL panel shows formatted SQL (sql-formatter, 4-space indent)
- Live preview: SQL updates 400ms after any field/source/mapping change
- Run button executes edited SQL directly via new exec-sql endpoint
- generate_stack_view gains p_dry_run mode for preview without executing
- CASCADE drop detects dependent stacks, marks them stale in DB and status bar
- net_balance moved to last column in generated view
- Backfill 458 missing dcard rows and 123 missing chase rows from TPS migration bug
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add get_status() SQL and /api/status route; load stale state on login
- Replace polling with immediate client-side stale tracking via callbacks
- Amber status bar with per-item Generate buttons for sources and stacks
- Pivot: add stack selector to view any dfv.stack view via Perspective
- Stack views: DROP CASCADE, add id to source views, per-source balance columns
- net_balance = sum(all amounts) + total_offset guarantees chase+dcard=net per row
- CLAUDE.md: document correct dedup spec (within-batch duplicates always allowed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Open panel immediately on row click (panelOpen state), then load full
record async. Previously the panel condition depended on selectedRecord
or panelLoading both of which are set after async work, so if id was
missing or the API call failed the panel never appeared.
Also shows a message if id is missing (view needs regeneration).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema:
- records.overrides JSONB column (ALTER TABLE, already applied)
- apply_transformations merges overrides on top: data || rules || overrides
- generate_source_view always includes id and _overridden columns
- set_record_overrides(id, overrides): stores and immediately merges into transformed
- clear_record_overrides(id): clears overrides then reprocesses record
API:
- PUT /records/:id/overrides — set overrides
- DELETE /records/:id/overrides — clear and reprocess
UI (Records page):
- Rows are clickable; overridden rows highlighted amber
- Side panel shows all transformed fields as editable inputs
- Overridden fields highlighted amber with pencil indicator
- Save stores overrides; Clear removes them and restores computed values
- id and _overridden hidden from table display
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL: search_mapping_outputs(search) — distinct (col, val, count) groups
get_mappings_by_output_field(col, val) — individual mappings
remap_output_field(col, from, to) — bulk UPDATE via jsonb_set
- API: GET /mappings/outputs?search=, GET /mappings/outputs/:col/:val,
POST /mappings/remap-field
- UI: Remap page — search output values, click to select, edit the
replacement value, see all affected mappings, apply globally
- Nav: Remap added between Mappings and Records
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SPEC.md: rewrite Pivot page description to cover named layouts, depth
control, selection mode, inspector filtering, and layout persistence.
docs/perspective-pivot.md: new file documenting all discovered Perspective
v4.4.0 APIs — viewer/plugin/view methods, selection modes, set_depth
mechanism, perspective-click event shape, full state save/restore pattern,
and common pitfalls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
formatVal now rounds numeric values using toLocaleString with
configurable decimal places (default 2, range 0-8). Adds -/+ controls
in the inspector header to adjust on the fly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without group_by there are no coordinate filters, so the view query
would return the full dataset and hang. Early-return on click if
config.group_by is empty.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Default selection mode is now SELECT_REGION
- plugin.save()/restore() used to capture and apply edit mode
- expand_depth tracked in ref and included in layout config
- applyExpandDepth helper restores depth on layout recall and page load
- Save button overwrites active layout in place (no re-typing name)
- captureConfig() helper shared by save-over and save-as flows
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace debug test buttons with a minimal 'depth: 0 1 2 3' control
in the pivot toolbar right side.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
