CLAUDE.md said is_key "marks dimensions used in slice WHERE clauses", which
is not true and never was: assertSelective and buildWhere validate against
filterCols, which is every dimension plus every date column regardless.
What it actually drives is the key of a dim_group, value completion, the
sibling autofill trigger, and the dim_period anchor -- and the first of
those wants exactly one column per group while the others want several. When
a group has more than one, find() silently takes the lowest opos, which is
how segment_new outranked part and a master-data refresh keyed on a column
that is null throughout: zero members, reported as success.
Written down because the failure gives no signal at all -- the refresh
succeeds, the list is simply empty, and every symptom appears somewhere
else entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things a reader would otherwise have to rediscover, and one of them is
load-bearing: observerShim.js has to be main.jsx's first import or it
silently intercepts nothing, which no amount of reading the shim itself
tells you. Also records the two false trails found while getting there --
getTable() resolving while getView() throws, and getView() returning a fresh
wrapper every call so identity cannot detect a rebuild.
Known issues rewritten against what is actually true now: the operation
panel wiring and the progress-bar throttle are done, and the load-time entry
now says what was measured -- that the cost is row count, not payload, and
dynamic grain is the remaining lever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The selection is restored from localStorage, but it was only validated once
at mount. Deregistering the selected source left App holding an id that no
longer exists, so every subsequent call 404'd "Source not found" with no way
out but a reload — Setup.deleteSource clears its own selectedSource and never
tells App. Deleting the selected version had the same shape.
Move both checks into effects keyed on the lists themselves. A sourcesLoaded
flag keeps the source effect from firing against the initial empty array and
wiping the restored id before the fetch resolves.
Also coerce both refresh callbacks to an array. A 401 returns an error object,
and data.some() then threw inside an unhandled promise, stranding the
selection instead of clearing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e6e2faeb37)
The server had no authentication: every /api route was open, CORS
allowed any origin, and the identity written to the audit log came from
the request body — the UI sent a hardcoded pf_user: 'admin', which any
client could have set to anything it liked.
Accounts live in pf.app_user with scrypt hashes from node's own crypto,
so there is no native build step and the parameters travel with each
hash. Sessions are express-session over connect-pg-simple in pf.session:
a restart no longer signs everyone out, and a session can be revoked by
deleting its row, which is how disable-user cuts off access immediately
rather than at cookie expiry.
Everything under /api except login/logout/me now requires a session, and
the React app is mounted only once there is one — its load effects call
the API on mount, so a logged-out mount would just fire a burst of 401s.
A session that expires while the app is open lands back on the login
screen: auth.jsx wraps fetch once rather than teaching every call site
to check.
Identity is now read from the session for pf_user, created_by and
closed_by, and the body values are ignored.
Hardened for an internet-facing deployment: trust proxy so req.ip and
secure-cookie detection are right behind TLS termination, httpOnly +
SameSite=Lax + Secure cookies, ten login failures per IP per fifteen
minutes, one error message for unknown, wrong and disabled alike, and a
fresh session id on success. CORS is off entirely unless CORS_ORIGIN
names an origin — a wildcard alongside a session cookie would be CSRF by
construction. The server refuses to boot without SESSION_SECRET rather
than falling back to a guessable default.
pf.sh grows add-user, passwd, list-users, disable-user and enable-user;
passwords are read on stdin and hashed before they reach psql, so no
plaintext in argv or shell history. install.sh generates the secret,
applies 02_auth.sql, and creates the first account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The row axis has had Expand 0/1/2/3 since the pivot landed; the column
axis had nothing. With a year over month split there was no way to step
back to whole years short of dragging split_by apart in the settings
panel and putting it back afterwards.
Perspective gives the two axes nothing in common here. Rows collapse
because the GROUP BY ROLLUP view holds every level at once and
view.set_depth() hides the deeper ones. For columns there is no
equivalent: expand()/collapse() take a row index, ViewConfig has
group_by_depth but no split_by_depth, and split_rollup_mode only chooses
whether subtotal column groups are emitted — a view shape, not an
interaction. So applySplitDepth() collapses by restoring a truncated
split_by, which rebuilds the view.
Three consequences of that rebuild, each handled:
- Once collapsed, viewer.save() only reports the short split_by, so the
full hierarchy is held separately (splitFullRef) and persisted into the
layout as split_full. Without it, collapsing would be a one-way door:
reload while collapsed and the deeper levels are gone. adoptSplit() is
the single place it is set.
- perspective-config-update fires for our own restore as well as for the
user rearranging the pivot, and the two mean opposite things — one must
adopt the new hierarchy, the other must not. collapsingRef separates
them.
- Row depth lives on the discarded view, so it is re-applied afterwards.
The selection is cleared on each change: slices name the split_by
dimensions they were cut from, and the highlight is keyed on grid
coordinates. Neither survives a column axis that just changed shape.
Buttons are named for the level they show — Total, then one per split_by
column — rather than numbered like Expand, since the levels are named and
a number would say nothing about what you are collapsing to.
Whole-axis, not per-branch: Excel can collapse 2025 while 2026 stays
expanded, and this cannot. `columns` selects which measures appear, not
individual split combinations, so there is no way to hide one branch's
leaves while keeping another's.
Verified in the browser against cash/test with split_by Year x Reason:
Reason -> Total -> Year -> Reason all render the expected column sets and
the right button highlights; and a reload while collapsed to Year comes
back collapsed with Reason still offered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Brings the Perspective work forward onto the display-grain decision in
c5b12aa. Four commits:
- Load Perspective from the bundled /inline entrypoints and pin every
package at 5.2.0, after the 4.x CDN bundle resolved its server WASM to
an unversioned path and took the app down on 2026-08-10.
- Multi-slice operations (slices array, apply_mode prorate/each),
per-measure resolution, target_basis, initiative tags, the tagged
bridge, and the reworked adjustment panel.
- Drag to select a region, and show the selection on the grid.
Merges clean against c5b12aa; the two touch disjoint files.
Two gaps in how the pivot reports a selection. Dragging across cells did
nothing, and a selection built from several ctrl-clicks was invisible on
the grid — the panel listed the slices but nothing on screen said which
cells they came from.
Drag-select
- The perspective-select handler was reading detail.selected and
detail.insertConfigs. In 5.2.0 that event carries a ViewWindow —
{ start_row, end_row, start_col, end_col }; insertConfigs only appears
on perspective-global-filter, and only in SELECT_ROW_TREE mode. So the
handler always returned early and the whole path was dead.
- It fires on every mouseover as the region grows, so the handler now
records the latest window and a window-level mouseup commits it. A
single-cell region is skipped there: perspective-click already owns
plain and modifier clicks, and handling it in both places would undo a
ctrl-click toggle. Modifier+drag adds to the selection.
- Turning a region back into slices re-derives, per cell, the same
filters Perspective attaches to a click — row dimensions from the
view's __ROW_PATH__, column dimensions from the split_by segments of
the column name. Reading the path rather than the rendered cell is
what keeps dates as epoch millis instead of whatever the grid
formatted them as. The grand-total row resolves to no dimension at
all — that would mean the whole version — and is skipped.
Highlight
- The datagrid already highlights whatever sits in its own
model._selection_state.selected_areas, and wipes that list on every
mousedown, so a multi-click selection only ever showed the last cell.
Keep a sliceKey -> rectangles map parallel to `slices` and push the
full set back after each change, which gets the native highlight for
every selected cell without any styling of our own.
- Deselecting anywhere — ctrl-click, the panel's x, Clear selection —
prunes the map by live slice key, so the grid and the panel cannot
disagree.
Also: edit_mode is forced to SELECT_REGION on restore rather than
defaulted, since a saved layout's plugin_config could previously
override it and turn selection off entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Documentation had drifted far enough to mislead: the spec described the
single-slice panel, the target/delta modes that no longer exist, and the
old `slice` request shape, while both files still claimed Perspective
4.4.0 from a CDN when it has been bundled inline at 5.2.0 since August.
pf_spec.md
- pf.log gains `tag`, with why it is written by a follow-up UPDATE
rather than through the per-source templates in pf.sql.
- Operations envelope documents `slices`, `apply_mode`, and the OR-of-
AND-groups WHERE clause (and why it cannot be flattened to IN lists).
- Scale documents per-measure resolution and `target_basis`, plus the
two guards: non-selective slices, and proration across a ~zero pool.
- New routes: PATCH /log/:logid, table-info, bridge, source tags.
- Forecast View rewritten for the dockable panel and the ledger; adds
the selection caveat around pf_iter and the expand-depth explanation.
- New Bridge View section; Log View gains inline tag editing.
- Status block refreshed, with a Fixed subsection recording the three
correctness bugs and their causes.
- Open Questions: adds bridge drill-down and targeting one iter band;
notes what the bridge partially answers.
CLAUDE.md
- Corrects the Perspective version and the CDN claim.
- Project layout now lists components/, including the two new files.
- Selection section covers multi-select and why duplicate effective
slices are collapsed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the outcome of the Option C spike (server-side DuckDB as a
Perspective virtual server) and the resulting architecture decision.
pf_perspective_options.md:
- Spike findings: latency is excellent (~12ms round trip, 5-29ms
aggregation, ~1.5-1.9s to materialize 534,902 rows) but Perspective's
GenericSQLVirtualServerModel ignores group_by_depth and has no
ViewConfig field for per-node expansion state, so interactive
drill-down is not achievable. This affects options B, C and D alike
since they share that SQL model.
- Decision: the real lever is grain, not transport. Pre-aggregating to
display grain collapses 534,902 -> 4,642 rows (~115x) on osm_stack
while keeping the native Perspective engine, so expand/collapse/
depth/sort/filter continue to work.
- Two candidate designs (Path A live virtual server vs Path B
pre-aggregated extract) with the deciding question: do real cuts ever
exceed the browser's leaf-row ceiling?
pf_spec.md:
- Concrete Path B design: pf.col_meta.in_grain, GET /api/versions/:id/agg,
synthetic pf_gkey index, and the append-deltas write/undo model that
mirrors the prior Excel pivot-cache workflow.
Drop pf_ux_mockup.md — an ASCII mockup of UI that is now built; the
views in ui/src/views are the current reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the @perspective-dev distribution (not FINOS @finos/perspective):
loader (npm /inline vs CDN), the version trilemma (inline needs 4.5.x,
viewer-d3fc caps at 4.4.1, charts need 4.4.1 — can't have all three),
Arrow vs JSON delivery constraints, deploy pattern, and an upgrade smoke
test. Correct CLAUDE.md's stale perspective.finos.org link to the actual
@perspective-dev repo.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- units role is now optional; spec and CLAUDE.md reflect conditionality in SQL patterns
- pf.col_meta gains dim_group and dim_period_col fields (documented in both files)
- pf.dim_period calendar table added to schema docs
- pf.source default_layout column added to spec DDL
- Forecast table metadata columns corrected to pf_iter/pf_logid/pf_created_at throughout spec
- SQL patterns updated with correct CTE structure and RETURNING * to match generated code
- Project status updated to 2026-06-12; stale Arrow IPC open question removed
- todo.md deleted; open items retained in CLAUDE.md known issues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Bootstrap fill icons with Feather-style stroke SVGs (sun with
rays + crescent moon) in StatusBar toggle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
