Answers whether Excel-style column collapse — 2025 collapsed while 2026
stays expanded — can be had from our own code, given that the engine is
the one thing @perspective-dev ships as a binary.
It can, and the engine needs no change. Under split_rollup_mode 'rollup'
the engine already emits the whole column tree in pre-order: grand total,
then each branch's subtotal immediately followed by its own leaves. Every
number Excel would show is already on the client. What is missing is only
the ability to not show some of them, which is a DOM concern.
So branchCollapse.js hides the leaves of a collapsed branch from a style
listener on the regular-table, shrinks the spanning header cells to the
columns still showing under them, and makes the group headers clickable.
No fork, no engine change, ~150 lines.
Verified on an 8-column grid: collapse 2025 (8 -> 6 cells), also collapse
2026 (-> 3), expand 2025 while 2026 stays collapsed (-> 5), header row and
body stay width-aligned throughout, and the state survives a redraw the
plugin initiates itself (a sort change). Clicking the real header works,
not just the API.
Where it stops working, which is the point of the spike:
The plugin virtualises the column axis by fetching a CONTIGUOUS
[start_col, end_col) window from the view and indexing everything by that
x. Hiding cells inside that window does not make it fetch more. On the
same pivot with no filter — 101 engine columns, 20 rendered — collapsing
2025 leaves 2 of 21 rendered cells visible and the rest of the viewport
empty, because 2026's columns are outside the fetched window and nothing
tells the grid to widen it. Scroll right and it looks correct again.
So this is not a rendering bug to patch; collapse punches holes in the
axis rather than compacting it. It holds while the whole axis fits in one
fetched window (~20 columns at 1500px), and degrades past that. Fixing it
properly means the visible-index -> engine-index indirection a fork would
have to add anyway.
Not merged, and not suitable to ship as-is. Kept as the evidence for
choosing between forking viewer-datagrid and raising it upstream, where
nothing is currently in flight: zero open PRs, no branch, and no issue
asking for it, though split_rollup_mode itself landed only in PR #3211 on
2026-08-10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
The row axis has had Expand 0/1/2/3 since the pivot landed; the column
axis had nothing. With a year over month split there was no way to step
back to whole years short of dragging split_by apart in the settings
panel and putting it back afterwards.
Perspective gives the two axes nothing in common here. Rows collapse
because the GROUP BY ROLLUP view holds every level at once and
view.set_depth() hides the deeper ones. For columns there is no
equivalent: expand()/collapse() take a row index, ViewConfig has
group_by_depth but no split_by_depth, and split_rollup_mode only chooses
whether subtotal column groups are emitted — a view shape, not an
interaction. So applySplitDepth() collapses by restoring a truncated
split_by, which rebuilds the view.
Three consequences of that rebuild, each handled:
- Once collapsed, viewer.save() only reports the short split_by, so the
full hierarchy is held separately (splitFullRef) and persisted into the
layout as split_full. Without it, collapsing would be a one-way door:
reload while collapsed and the deeper levels are gone. adoptSplit() is
the single place it is set.
- perspective-config-update fires for our own restore as well as for the
user rearranging the pivot, and the two mean opposite things — one must
adopt the new hierarchy, the other must not. collapsingRef separates
them.
- Row depth lives on the discarded view, so it is re-applied afterwards.
The selection is cleared on each change: slices name the split_by
dimensions they were cut from, and the highlight is keyed on grid
coordinates. Neither survives a column axis that just changed shape.
Buttons are named for the level they show — Total, then one per split_by
column — rather than numbered like Expand, since the levels are named and
a number would say nothing about what you are collapsing to.
Whole-axis, not per-branch: Excel can collapse 2025 while 2026 stays
expanded, and this cannot. `columns` selects which measures appear, not
individual split combinations, so there is no way to hide one branch's
leaves while keeping another's.
Verified in the browser against cash/test with split_by Year x Reason:
Reason -> Total -> Year -> Reason all render the expected column sets and
the right button highlights; and a reload while collapsed to Year comes
back collapsed with Reason still offered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Two gaps in how the pivot reports a selection. Dragging across cells did
nothing, and a selection built from several ctrl-clicks was invisible on
the grid — the panel listed the slices but nothing on screen said which
cells they came from.
Drag-select
- The perspective-select handler was reading detail.selected and
detail.insertConfigs. In 5.2.0 that event carries a ViewWindow —
{ start_row, end_row, start_col, end_col }; insertConfigs only appears
on perspective-global-filter, and only in SELECT_ROW_TREE mode. So the
handler always returned early and the whole path was dead.
- It fires on every mouseover as the region grows, so the handler now
records the latest window and a window-level mouseup commits it. A
single-cell region is skipped there: perspective-click already owns
plain and modifier clicks, and handling it in both places would undo a
ctrl-click toggle. Modifier+drag adds to the selection.
- Turning a region back into slices re-derives, per cell, the same
filters Perspective attaches to a click — row dimensions from the
view's __ROW_PATH__, column dimensions from the split_by segments of
the column name. Reading the path rather than the rendered cell is
what keeps dates as epoch millis instead of whatever the grid
formatted them as. The grand-total row resolves to no dimension at
all — that would mean the whole version — and is skipped.
Highlight
- The datagrid already highlights whatever sits in its own
model._selection_state.selected_areas, and wipes that list on every
mousedown, so a multi-click selection only ever showed the last cell.
Keep a sliceKey -> rectangles map parallel to `slices` and push the
full set back after each change, which gets the native highlight for
every selected cell without any styling of our own.
- Deselecting anywhere — ctrl-click, the panel's x, Clear selection —
prunes the map by live slice key, so the grid and the panel cannot
disagree.
Also: edit_mode is forced to SELECT_REGION on restore rather than
defaulted, since a saved layout's plugin_config could previously
override it and turn selection off entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
The pivot stopped rendering with:
LinkError: WebAssembly.instantiate(): Import #8 "env" "psp_opfs_load":
function import requires a callable
Nobody changed anything. The 4.x CDN bundle resolves its server WASM with
new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)
which from .../client@4.4.0/dist/cdn/ resolves to
.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm -- with no
version. jsdelivr serves @latest, so when @perspective-dev/server@5.2.0 was
published on 2026-08-10 every page load began linking a 5.2.0 WASM against a
4.4.0 client. 4.4.1 and 4.5.2 carry the identical unversioned pattern, so no
4.x pin is safe over CDN. Beyond the outage, an unversioned URL means users
execute whatever that package publishes next, unreviewed.
Switch to the /inline entrypoints, which embed the WASM in the Vite build:
no runtime fetch, and the version is fixed by package-lock.json (verified:
zero `new URL(...perspective-server...)` in perspective.inline.js).
- pin client/viewer/viewer-datagrid/server exact at 5.2.0. The explicit
`server` pin matters: client declares it as "" (an empty range), which npm
also resolves to latest -- the same break, at install time instead.
- drop the viewer-d3fc import. pf_app never selects a chart plugin, and d3fc
has no 5.x; loading 4.4.1 against a 5.x viewer only emits
`get_static_config is not a function` per plugin.
- themes move from a CDN <link> to @perspective-dev/viewer/themes.
Verified end-to-end with every non-localhost request aborted: no external
requests are attempted, both custom elements register, the Arrow stream
ingests, and the pivot renders (TOTAL 17,235.97 = -7,573.97 + 30,907.47
- 6,097.53). apache-arrow 21.1.0 ingests cleanly against the 5.2.0 WASM.
Bundle grows 263 KB -> 11.6 MB (5.4 MB gzipped); that is the embedded WASM.
PERSPECTIVE.md also records findings from the same investigation:
- §3a: expression columns are row-level, evaluated before aggregation, so a
ratio like "revenue"/"qty" summed per row is wrong under any pivot (not
just split_by). Fix is a weighted-mean aggregate, whose weight column must
be a NESTED array: ['weighted mean', ['qty']]. Works in 4.4.0 and survives
incremental table.update().
- §2: withdraws the recommendation of the 4.5.1-core + 4.4.1-d3fc pair. It
does not deliver charts, so the trilemma is really a dilemma: inline
bundling XOR charts. dataflow is on that pair and needs the same migration.
- §5: cleanLayout() does not sanitize `aggregates`; guard it before adopting
the weighted-mean pattern or a dropped column aborts the whole restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SBt3EtKaP9D2mmWJ6Q4bov
Replaced native <select> (macOS ignores CSS on option elements) with a
custom button+ul dropdown. Background/text/border colors are applied via
useTheme so they respond correctly to dark mode toggle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GET /api/dim-period/cols queries information_schema for pf.dim_period columns
(excluding sdat/edat/drange/ndays) so the UI always reflects actual columns.
Setup col_meta editor now shows a dropdown populated from that endpoint instead
of a free-text field, preventing invalid column names like the cash source had.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL generator no longer requires a units col; recode/clone/scale omit units
expressions when none is configured in col_meta
- Source registration validation drops units from required roles (value + date
are the only hard requirements)
- DELETE /api/sources/:id returns 409 when existing versions reference the source
- Setup.jsx surfaces the 409 error via flash instead of silently failing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta gets dim_period_col field: maps a dimension column to its pf.dim_period counterpart (e.g. year -> cal_year, month -> cal_month)
- When the date column is is_key of a dim_group and any sibling dimension has dim_period_col set, baseline and reference SQL JOIN pf.dim_period on the shifted date instead of copying raw source values
- No dim_period config = identical SQL to before (fully backwards compatible)
- Setup UI: period col input in col_meta editor, enabled for dimension columns with a dim_group set
- Schema migration applied: dim_period_col text null on pf.col_meta
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/sources/:id/lookup?col=X&value=Y — given a key column value, queries the source table for sibling column values in the same dim_group; returns null if no match or ambiguous
- Recode and Clone panels: key columns (is_key + dim_group) trigger lookup on blur and auto-fill sibling inputs that the user hasn't already typed into
- Row labels now use col_meta label field when set, falling back to cname
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta: add dim_group field to group related columns (dimension hierarchies, date-adjacent columns); is_key now enabled for date role to mark group parent
- sources.js: upsert includes dim_group
- Setup.jsx: group column in col_meta editor, key checkbox enabled for date role
- gen_dim_period.sql: create and populate pf.dim_period with calendar and fiscal period cuts (monthly grain, 2018-2035)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch server Arrow encoding from tableFromJSON (row objects) to
tableFromArrays (column arrays) — cuts peak Node heap 3-5x for large
datasets by avoiding one JS object per row
- Remove unused pf.log JOIN from data endpoint; forecast rows only
- Load Perspective viewer with direct table reference instead of worker
Server object — fixes "No Table attached" error on large datasets where
named-table registry lookup raced against WASM initialization
- Pre-emptively clean up stale named table in worker registry before
creating, eliminating the "already exists" retry path that silently
swallowed errors (finally ran but flash never fired)
- Strip cfg.table from restore configs since table is loaded by reference
- Throttle progress bar updates to 100ms intervals (was every chunk)
- Persist load errors until dismissed; add console.error for devtools
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Add PayloadPreview component showing the exact JSON that will be POSTed,
live-updating as form fields change (value_incr shown as computed delta)
- buildEffectiveSlice strips expression/system columns and converts
Perspective ms-timestamps to ISO date strings for date-role columns
- fetchCurrentTotals now includes date columns in Perspective view filter
(passing ms number as Perspective expects) so subtotals respect the
clicked date
- Server buildWhere now receives filterCols (dimensions + date cols) so
date values reach the SQL WHERE clause correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Bootstrap fill icons with Feather-style stroke SVGs (sun with
rays + crescent moon) in StatusBar toggle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forecast falls back to a saved per-source layout when no version-local
layout is cached, so new versions of a source open with a sensible pivot
without each user reconfiguring it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reference segments can now apply a date offset just like baselines.
SQL template gains the {{date_offset}} token; both POST /reference and
PUT baseline/:logid pass it through. Existing sources need to
regenerate SQL to pick up the new template — old stored reference SQL
ignores the token (preserving prior verbatim behavior). The Baseline
form drops the "dates land verbatim" hint and shows the offset
control for both segment types.
Editing a segment now color-codes the source row amber with a ring
and tints the form border + header amber so the active connection is
visually obvious. Header label reads "Edit segment #3 — baseline —
note" instead of just "#43" (the internal log id).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New forecasts opened the pivot with all dimensions stacked as
group_by and the date column as split_by — wide and slow to read.
Open with just the value column showing and pf_iter as rows so the
first thing you see is iteration totals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The segment form is now one component rendered in either 'view' or
'edit' mode — the expanded segment row in the list and the
add/edit form below share the same layout, view mode just disables
the inputs. Edit and View are visually identical so toggling between
them feels like enabling fields, not switching tools.
Filters become groups (conditions AND-ed inside, groups OR-ed
between) with + AND condition and + Add OR group affordances. The
compiled WHERE renders live below the groups so you can see what's
being built. A "Switch to manual SQL" toggle flips to a textarea
seeded with the compiled clause; backend baseline POST/PUT and
reference POST accept raw_where alongside filters and store whichever
arrived in pf.log.params for round-tripping.
The Add form is hidden until you click "+ Add segment" at the
bottom of the segments table; Edit also opens it. Cancel/Close
returns the table to its compact state.
/versions/:id/log now also returns value_total, units_total, and the
column names so the segments table can show row count and value sum
inline (header uses the source's actual value column name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds PUT /versions/:id/baseline/:logid that, in one transaction, drops
the segment's rows and log entry and replays the baseline or reference
SQL with new params. The endpoint refuses (409) if any scale, recode,
or clone has been applied — those operations were calibrated against
the old totals and would silently misreconcile.
Baseline view gets an Edit button on each segment (hidden once
forecast operations exist), populating the form with the original
filters, offset, and note. Submit issues PUT in edit mode, POST
otherwise. POST baseline and POST reference now also persist the
structured filters in pf.log.params so edit can reload them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Save/restore went through both viewer and plugin, where the explicit
plugin.restore could stomp the column formatting the viewer had
already applied. Capture via viewer.save() alone (it includes
plugin_config) and restore via a single viewer.restore call with
edit_mode merged in. Added a perspective-config-update listener so
formatting, sort, and other in-place changes persist to the last-used
cache without an explicit Save.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The slice panel was a single muted line; now it shows a breakdown
table — value, units, and derived price for baseline / scale / recode /
clone, with a bold total row when more than one iteration applies.
Numbers use full text contrast so the current state is legible at a
glance during adjustments. Scale gains a price input that holds units
constant and translates to a value-target call (target value =
new_price × current_units).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reuse a single Perspective worker across version switches and delete
the previous table instead of terminating the worker — terminate was
returning a rejecting promise the sync try/catch missed, and each new
worker leaked WASM memory. applyLayout no longer leaks a view per call;
it reads schema directly from the table. An init id guards against
concurrent runs (StrictMode, rapid version switches) clobbering each
other, and a catch on "already exists" recovers via open_table+delete
when a stale table from a previous run is still hosted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
App.jsx owns sources/sourceId/versions/versionId and persists them
across reloads. StatusBar renders the dropdowns plus a status badge —
Source-only on Setup, Source · Version · status on Baseline/Forecast.
The duplicate in-view selector bars in Forecast and Baseline are gone;
Baseline keeps its version actions (New/Close/Reopen/Delete) inline.
Setup reports source-list mutations up via refreshSources so the bar
stays in sync after register/deregister.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
App chrome now uses Pro Dark's neutral grays (#242526 background,
#2a2c2f panels, #4c505b borders) so the surrounding UI sits cleanly
against the viewer instead of clashing with its warmer tone. Status
accents are desaturated to match. Forecast view sets theme="Pro Dark"
or "Pro Light" on the perspective-viewer in sync with the toggle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Perspective table is now created with index: 'pf_id'. Delete endpoints
return the pf_ids they removed; the client calls table.remove(pf_ids)
in undoEntry. Avoids the full /data refetch that dominated undo time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
pg now returns bigint/numeric as JS numbers so Arrow infers Int/Float64
instead of Dictionary<Utf8>. /data accumulates rows and emits a single
record batch to avoid dictionary REPLACEMENT messages that crash
Perspective's WASM reader. Forecast view streams the response body and
shows received/total bytes while loading. Drops stale public/ static
middleware that was shadowing the React build at /.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Server streams rows from a pg cursor in 10k-row batches, building Arrow
record batches incrementally and piping them as chunked HTTP response —
Node.js heap stays bounded regardless of dataset size.
Client fetches as arrayBuffer() and loads directly into Perspective worker
(native Arrow path, no JSON deserialization). X-Row-Count header drives
a non-blocking banner for datasets >= 500k rows. validCols now derived
from col_meta rather than from row keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-applies fetchCurrentTotals dimension-only filter (prevents Perspective errors
from split_by/expression columns in the filter), toolbar three-group reorganization
(Layout | Expand | Data with dividers), always-visible Save as…, msg in toolbar,
resizable panel, change log modal with undo and inline note editing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Expression columns (bucket, computed) are defined in cfg.expressions and
are valid pivot axes, but weren't in validCols (raw table columns), so
they were filtered out of group_by/split_by on every layout restore.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/versions/:id/log — log entries with row counts via JOIN
- DELETE /api/log/:logid — undo in a transaction (delete fc rows + log entry)
- PATCH /api/log/:logid — update note text
- History button opens a modal: op badge, slice, editable note, row count, Undo per entry
- Undo triggers full Perspective table reload via initViewer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix perspective-click handler to use event filter triples instead of
__ROW_PATH__ — Perspective encodes row position as [col,'==',val] in
detail.config.filter
- buildWhere now skips unrecognised slice keys (e.g. pf_iter) instead of
throwing, so only dimension columns reach the WHERE clause
- Add draggable resize handle on the operation panel (160–480px)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Baseline.jsx: merge Reference section into Add Segment form with baseline/reference toggle; segment rows now clickable to expand stored WHERE clause + timeline; date filter inputs use type="date" for date-role columns
- Timeline.jsx: add type prop ('baseline'|'reference'); reference band uses purple; single-band height shrinks to 52px; canvas uses requestAnimationFrame to fix offsetWidth=0 on mount
- operations.js: reference route now accepts where_clause like baseline (drops date_from/date_to)
- sql_generator.js: reference SQL template uses {{filter_clause}} instead of hardcoded BETWEEN
Note: existing sources need Generate SQL re-run to pick up the new reference template.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ui/: React + Vite + Tailwind app (Setup, Baseline, Forecast views, collapsible sidebar, status bar, canvas timeline)
- server.js: serve built UI from public/app/
- package.json: add build script (cd ui && npm run build)
- routes/sources.js: default new col_meta role to 'dimension' instead of 'ignore'
- .gitignore: exclude public/app/ build output
- pf_spec.md: update tech stack, nav, frontend section, and project status to reflect current implementation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
