sales_usd and qty run to eight digits here, where two decimals are noise.
Price is the opposite case: it sits around 0.27, so two places barely show a
move and four still round away part of one.
derive() hardcoded two decimals, so the editable rows would have disagreed
with the lines above them. It now takes the measure's own precision. The
percentage row keeps one place, being its own scale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Price was blanked on the bridge lines and on the excluded row, so a ledger
that had both measures on every line showed a price on two of them. Baseline
sat there as 2,242,180.67 over 8,111,717.10 with the column empty.
A bridge line's price is the implied price of that initiative's own
contribution -- Scale #116 above works out to 0.2397 against a current
0.2735 -- which is the number that says whether the initiative was a price
move or a volume move. That is the same question the plug toggle asks about
the edit, answered for the adjustments already made.
A line with no units is left blank, since its price is undefined rather
than zero -- the "price" bridge line in that example moved dollars alone.
The price column's hint said "holds units", which stopped being true when
plug arrived: it holds units under plug = price and moves them under plug =
volume. It now just says what the column is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each ledger row derived only from its own input, so typing a dollar figure
left the units and price rows blank -- which are precisely the two numbers
that say whether you just asked for a volume change or a price change. The
plug toggle decided it and then showed you nothing.
Adds a Result line under the edit rows carrying value, units and price
together, resolved the same way the server resolves the increments: a price
input multiplies through volume, a dollar input with plug = volume scales
units in proportion, anything else holds units. Measures that actually move
are picked out in blue against the ones that hold, so plug = price reads as
price moving alone and plug = volume as price standing still.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sales figure on its own does not say which of price or volume moved, and
scale was answering silently: it wrote the value increment and left units
alone, so volume stayed flat and price absorbed everything. That is one of
the three behaviours the Excel form offered, picked by default rather than
by the user.
/opt/forecast_api/VBA/fpvt.frm has the semantics. Edit Sales, then plug one
side:
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
So 'volume' moves units in the same proportion as the dollars, holding
price; 'price' leaves units alone, as now. Checked against those formulas:
+100 on 1000/500 gives units 500 -> 550 with price held at 2.0000, and
-400 on 2000/800 gives 800 -> 640 at 2.5000.
Price still defaults, so nothing changes for an existing caller that does
not send `plug`. The control only appears once the edit is dollars-only --
naming units or price has already settled the question. A price target now
also honours a units target alongside it, which is the form's Edit Price
mode where both are inputs and dollars fall out.
Holding price when the selection has no value is refused rather than
divided by zero, the same case the form guarded with "Zero times any number
is zero".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the trace showed, one of them a wrong call on my part.
getTable() was the wrong thing to wait on. The re-apply threw "No table
set" two milliseconds after the event, meaning getTable() had already
resolved while getView() still had not -- it is the view that is missing
around a rebuild, not the table. There is no predicate for "the view is
ready", so attempt applyDepth and retry until it stops throwing, up to 5s.
And the shim matched nothing: not one callback reported a viewer. closest()
stops at a shadow boundary, so it cannot climb from an element inside
Perspective's shadow root out to the host, which is where the observed
elements live. Walk host to host via getRootNode().host instead.
The shim now also logs every callback it sees under [pf-obs], matched or
not, with a description of each target. If it still reports nothing at all,
the shim is not installed and the problem is import order rather than
matching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The version re-validation effect could not tell "the versions fetch has not
landed yet" from "this source has no versions", because both look like an
empty array. So a versionId restored from localStorage was cleared on the
first render and set straight back when the fetch returned: 29 -> '' -> 29.
Forecast's load effect is keyed on [versionId, sourceId], so that round
trip ran initViewer twice. Two /agg requests, two full aggregations in
Postgres, two Arrow payloads -- on version 29 that is the 285k-row
aggregate computed twice at ~17s each, which is most of the "it hangs
before it displays" and the Postgres process sitting at the top of htop.
sourcesLoaded already guarded the identical effect for sources; versions
just never got the same treatment. Cleared when the source changes, since
the list belongs to the previous source until the new fetch lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The WASM engine has no notion of focus or tab visibility. It re-renders
because IntersectionObserver or ResizeObserver told it the element is
visible again at some size, and that re-render discards the view -- taking
set_depth() and per-node expansion with it, since neither lives in
ViewConfig. Listening for window 'focus' was only a guess at when that
happened, which is why the re-apply kept landing while the viewer was still
detached from its table.
Perspective's viewer captures the constructors at module-evaluation time
(`var it=window.ResizeObserver; var st=window.IntersectionObserver`), so
replacing them on window before that import puts us in front of every
callback it gets. observerShim.js does exactly that and nothing else: the
original callback runs first and unchanged, and we re-emit as a CustomEvent
only when an entry's target is or contains a perspective-viewer. That has
to be the first import in main.jsx or there is nothing left to intercept.
focus/visibilitychange/pageshow stay as a backstop for the case where the
shim could not be installed. ResizeObserver fires on every frame of a panel
drag, so observer events trail by 150ms and act once the burst settles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The refocus timing is the sort of thing that will shift again -- a
Perspective upgrade, a different machine, a slower load -- and rebuilding
this instrumentation from scratch each time is wasted work. So it stays,
behind a flag, instead of being reverted.
probeView() returns before touching the viewer when the flag is off, since
it calls getView() purely to report identity.
Note for whoever reads the log next: the view ids it prints cannot be used
to detect a rebuild. getView() hands back a fresh wrapper object on every
call, so consecutive calls always look "rebuilt". The reliable signal is
whether the re-apply throws "No table set".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instrumentation showed the re-apply mostly throwing "No table set": the
viewer is detached from its table for a while after a refocus, the fixed
60ms delay fired inside that window, set_depth threw, and the tree rendered
fully expanded. The handful of times the delay happened to be long enough,
the log reads "re-apply done depth=0" and the collapse survived.
So poll for the table instead of guessing, up to 5s. queued now clears in a
finally after the work rather than at the top of the callback, so the focus
and visibilitychange that both fire for one window switch no longer each
run a re-apply -- that was the doubled applyDepth in the log.
Does not address per-node +/- collapse, which is still not recorded at all:
expandDepthRef stays null because only the toolbar buttons set it, and the
view exposes expand()/collapse() with no getter to read the state back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Temporary. Row grouping re-expands whenever the browser regains focus, for
both the toolbar EXPAND buttons and the per-row +/-. set_depth() and
per-node expansion both live on the view rather than in ViewConfig, so a
rebuilt view loses them -- but nothing so far distinguishes "the view was
rebuilt" from "the re-apply lost its race with the redraw".
Tags each view object with an id via a WeakMap and logs it at every point
that could rebuild one: focus/visibilitychange/pageshow, config-update,
applyDepth, the theme effect, and initViewer. A changed id across a refocus
means the view was discarded; an unchanged id means the depth re-apply
itself is at fault. Each bail-out in the refocus handler now says which
branch it took, which also shows whether expandDepthRef was ever set.
Prefix [pf-depth]. Revert this commit once the cause is known.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_gkey is the Perspective table index -- an opaque handle read only by
table.update() and table.remove(). It was the raw concat_ws of every grain
column, which on the 24-column grain of fc_osm_skinny_29 averaged 233
characters. Being unique per row by construction, it also defeated Arrow's
dictionary encoding, so it alone accounted for 65.6 MB of a 109 MB payload
-- more than the other 31 columns combined, all of which do dictionary
cleanly.
md5 of the same string keeps the determinism undo depends on (routes/log.js
recomputes the key through this same grainOf) and keeps collisions out of
reach at 128 bits, while fixing the width at 32 characters.
Measured on the live 285,685-row aggregate: payload 109.0 MB -> 54.2 MB,
all 285,685 keys still distinct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The load filter dropdown listed role 'date' and role 'filter' only, so a
segment could not be cut on sseas without recoding it -- and role is a
single value, so that trade would have taken it off the pivot to put it in
a dropdown.
Nothing downstream required the restriction: the server drops filter_clause
into the load's WHERE against the source table without consulting col_meta,
and this form takes values as free text rather than through the key-only
values endpoint. So the fix is to stop filtering them out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ordinary npm churn: dropped peer markers where those packages are now
direct dependents, plus the optional @emnapi platform packages.
.env is ignored but a .env.bak sitting next to it is not, and it holds the
same credentials. Widen the pattern to .env.* so a backup can't be staged
by a careless git add -A.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The label on a baseline or reference load ("Open Orders", "Prior Year")
lived only on the pf.log row; the data stream was a straight dump of the
forecast table, so Perspective saw pf_logid and never the name.
The stream now joins pf.log and emits two columns rather than one, because
commingling them makes neither useful: pf_segment names the load a row came
from, and is '(adjustment)' for everything else; pf_note carries the free
text on scale/recode/clone and is null on loads. The operation routes stamp
the same two fields on the rows they push back incrementally.
The tag column those labels belong in is declared in 01_schema.sql but
predates some installs, so the backfill seeding it from note joins the
ALTER already there. Adjustment notes are free text, not labels, and are
left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generator picked columns with find(), so a source with more than one
date or value column silently loaded only the first of each. The version's
DDL is built from col_meta separately, so the extra columns existed in the
forecast table and stayed null for its whole life -- gs.osm_skinny lost
sdate and stdcost_usd across 295k rows that way.
Loads copy the source row wholesale, so they now carry all of them, with
the offset shifting every date column together. The adjustment operations
are genuinely single-measure -- scale distributes one {{value_incr}} -- so
they keep the narrow column list until that has a defined meaning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The selection is restored from localStorage, but it was only validated once
at mount. Deregistering the selected source left App holding an id that no
longer exists, so every subsequent call 404'd "Source not found" with no way
out but a reload — Setup.deleteSource clears its own selectedSource and never
tells App. Deleting the selected version had the same shape.
Move both checks into effects keyed on the lists themselves. A sourcesLoaded
flag keeps the source effect from firing against the initial empty array and
wiping the restored id before the fetch resolves.
Also coerce both refresh callbacks to an array. A 401 returns an error object,
and data.some() then threw inside an unhandled promise, stranding the
selection instead of clearing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e6e2faeb37)
The server had no authentication: every /api route was open, CORS
allowed any origin, and the identity written to the audit log came from
the request body — the UI sent a hardcoded pf_user: 'admin', which any
client could have set to anything it liked.
Accounts live in pf.app_user with scrypt hashes from node's own crypto,
so there is no native build step and the parameters travel with each
hash. Sessions are express-session over connect-pg-simple in pf.session:
a restart no longer signs everyone out, and a session can be revoked by
deleting its row, which is how disable-user cuts off access immediately
rather than at cookie expiry.
Everything under /api except login/logout/me now requires a session, and
the React app is mounted only once there is one — its load effects call
the API on mount, so a logged-out mount would just fire a burst of 401s.
A session that expires while the app is open lands back on the login
screen: auth.jsx wraps fetch once rather than teaching every call site
to check.
Identity is now read from the session for pf_user, created_by and
closed_by, and the body values are ignored.
Hardened for an internet-facing deployment: trust proxy so req.ip and
secure-cookie detection are right behind TLS termination, httpOnly +
SameSite=Lax + Secure cookies, ten login failures per IP per fifteen
minutes, one error message for unknown, wrong and disabled alike, and a
fresh session id on success. CORS is off entirely unless CORS_ORIGIN
names an origin — a wildcard alongside a session cookie would be CSRF by
construction. The server refuses to boot without SESSION_SECRET rather
than falling back to a guessable default.
pf.sh grows add-user, passwd, list-users, disable-user and enable-user;
passwords are read on stdin and hashed before they reach psql, so no
plaintext in argv or shell history. install.sh generates the secret,
applies 02_auth.sql, and creates the first account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
server.js builds its pg pool from DB_HOST/DB_PORT/DB_NAME/DB_USER/
DB_PASSWORD, and that is what install.sh writes, but pf.sh had been
written against a DATABASE_URL/PF_USER scheme that nothing consumes.
The consequences were real: `status` always reported the database
unreachable, and `config` rewrote .env with cat >, replacing a working
connection with keys the server ignores.
Connect through a run_psql helper built from the DB_* vars, and have
config prompt for those six keys instead. Each prompt defaults to the
current value so entering through changes nothing, the password prompt
is hidden and keeps the stored one when left blank, and any other key
already in .env is carried over rather than dropped.
PF_USER goes away with it — pf_user reaches the server in the request
body, never from the environment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Perspective's column axis cannot be collapsed. The row axis has had it
forever — GROUP BY ROLLUP holds every level and view.set_depth() hides the
deeper ones — but nothing equivalent is exposed for split_by, so a Year over
Month pivot can only ever be shown fully expanded.
The engine already implements it. t_ctx2 is symmetric: set_depth(t_header,
depth), open(t_header, idx) and close(t_header, idx) each have a real
HEADER_COLUMN branch on m_ctraversal mirroring m_rtraversal, and
t_view_config carries m_column_pivot_depth which server.cpp already applies.
None of it is reachable: set_column_pivot_depth() is never called, so the
depth stays -1, and View<t_ctx2>::expand/collapse hardcode HEADER_ROW. The
patch is 193 lines of wiring across the protobuf, the Rust client and the
datagrid — no new engine logic.
Two capabilities result, mirroring the row axis:
- split_by_depth in ViewConfig, the split_by counterpart to group_by_depth
- expand_column()/collapse_column(), addressed by column traversal index
exactly as the row methods are addressed by row index
which together give the Excel behaviour — one year folded to its subtotal
while its siblings stay expanded — that no combination of existing config
could produce. Verified in this app against fc_cash_9: clicking a Year
header goes from 27 columns to 15, totals reconciling at every level.
Vendored rather than aliased
- The previous approach pointed vite at a local checkout, which built only on
one laptop and left package.json claiming npm 5.2.0 while the build used
something else. The four packages are now committed as npm tarballs and
package.json names them, so the declaration is true and `pf.sh deploy`
works unchanged — npm install expands them like any registry package.
- Packed with `pnpm pack`, not `npm pack`: Perspective is a pnpm workspace and
cross-package deps are `workspace:^`, which npm rejects outright. pnpm
rewrites those to real version ranges at pack time.
- All four move together. Perspective couples loader, package versions, data
format and apache-arrow; a partial vendor reintroduces exactly that drift.
Cost, stated plainly: 12MB of opaque binaries in git that do not delta, a
fork to maintain, and a second engine build for anyone changing it.
rebuild-perspective.sh makes that repeatable and PROVENANCE.txt records the
commit each tarball came from, because a committed .tgz otherwise has no
recoverable source. README.md says how to delete all of it once upstream
ships the feature.
This also moves the app from 5.2.0 to 5.4.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
The row axis has had Expand 0/1/2/3 since the pivot landed; the column
axis had nothing. With a year over month split there was no way to step
back to whole years short of dragging split_by apart in the settings
panel and putting it back afterwards.
Perspective gives the two axes nothing in common here. Rows collapse
because the GROUP BY ROLLUP view holds every level at once and
view.set_depth() hides the deeper ones. For columns there is no
equivalent: expand()/collapse() take a row index, ViewConfig has
group_by_depth but no split_by_depth, and split_rollup_mode only chooses
whether subtotal column groups are emitted — a view shape, not an
interaction. So applySplitDepth() collapses by restoring a truncated
split_by, which rebuilds the view.
Three consequences of that rebuild, each handled:
- Once collapsed, viewer.save() only reports the short split_by, so the
full hierarchy is held separately (splitFullRef) and persisted into the
layout as split_full. Without it, collapsing would be a one-way door:
reload while collapsed and the deeper levels are gone. adoptSplit() is
the single place it is set.
- perspective-config-update fires for our own restore as well as for the
user rearranging the pivot, and the two mean opposite things — one must
adopt the new hierarchy, the other must not. collapsingRef separates
them.
- Row depth lives on the discarded view, so it is re-applied afterwards.
The selection is cleared on each change: slices name the split_by
dimensions they were cut from, and the highlight is keyed on grid
coordinates. Neither survives a column axis that just changed shape.
Buttons are named for the level they show — Total, then one per split_by
column — rather than numbered like Expand, since the levels are named and
a number would say nothing about what you are collapsing to.
Whole-axis, not per-branch: Excel can collapse 2025 while 2026 stays
expanded, and this cannot. `columns` selects which measures appear, not
individual split combinations, so there is no way to hide one branch's
leaves while keeping another's.
Verified in the browser against cash/test with split_by Year x Reason:
Reason -> Total -> Year -> Reason all render the expected column sets and
the right button highlights; and a reload while collapsed to Year comes
back collapsed with Reason still offered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Brings the Perspective work forward onto the display-grain decision in
c5b12aa. Four commits:
- Load Perspective from the bundled /inline entrypoints and pin every
package at 5.2.0, after the 4.x CDN bundle resolved its server WASM to
an unversioned path and took the app down on 2026-08-10.
- Multi-slice operations (slices array, apply_mode prorate/each),
per-measure resolution, target_basis, initiative tags, the tagged
bridge, and the reworked adjustment panel.
- Drag to select a region, and show the selection on the grid.
Merges clean against c5b12aa; the two touch disjoint files.
Two gaps in how the pivot reports a selection. Dragging across cells did
nothing, and a selection built from several ctrl-clicks was invisible on
the grid — the panel listed the slices but nothing on screen said which
cells they came from.
Drag-select
- The perspective-select handler was reading detail.selected and
detail.insertConfigs. In 5.2.0 that event carries a ViewWindow —
{ start_row, end_row, start_col, end_col }; insertConfigs only appears
on perspective-global-filter, and only in SELECT_ROW_TREE mode. So the
handler always returned early and the whole path was dead.
- It fires on every mouseover as the region grows, so the handler now
records the latest window and a window-level mouseup commits it. A
single-cell region is skipped there: perspective-click already owns
plain and modifier clicks, and handling it in both places would undo a
ctrl-click toggle. Modifier+drag adds to the selection.
- Turning a region back into slices re-derives, per cell, the same
filters Perspective attaches to a click — row dimensions from the
view's __ROW_PATH__, column dimensions from the split_by segments of
the column name. Reading the path rather than the rendered cell is
what keeps dates as epoch millis instead of whatever the grid
formatted them as. The grand-total row resolves to no dimension at
all — that would mean the whole version — and is skipped.
Highlight
- The datagrid already highlights whatever sits in its own
model._selection_state.selected_areas, and wipes that list on every
mousedown, so a multi-click selection only ever showed the last cell.
Keep a sliceKey -> rectangles map parallel to `slices` and push the
full set back after each change, which gets the native highlight for
every selected cell without any styling of our own.
- Deselecting anywhere — ctrl-click, the panel's x, Clear selection —
prunes the map by live slice key, so the grid and the panel cannot
disagree.
Also: edit_mode is forced to SELECT_REGION on restore rather than
defaulted, since a saved layout's plugin_config could previously
override it and turn selection off entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Documentation had drifted far enough to mislead: the spec described the
single-slice panel, the target/delta modes that no longer exist, and the
old `slice` request shape, while both files still claimed Perspective
4.4.0 from a CDN when it has been bundled inline at 5.2.0 since August.
pf_spec.md
- pf.log gains `tag`, with why it is written by a follow-up UPDATE
rather than through the per-source templates in pf.sql.
- Operations envelope documents `slices`, `apply_mode`, and the OR-of-
AND-groups WHERE clause (and why it cannot be flattened to IN lists).
- Scale documents per-measure resolution and `target_basis`, plus the
two guards: non-selective slices, and proration across a ~zero pool.
- New routes: PATCH /log/:logid, table-info, bridge, source tags.
- Forecast View rewritten for the dockable panel and the ledger; adds
the selection caveat around pf_iter and the expand-depth explanation.
- New Bridge View section; Log View gains inline tag editing.
- Status block refreshed, with a Fixed subsection recording the three
correctness bugs and their causes.
- Open Questions: adds bridge drill-down and targeting one iter band;
notes what the bridge partially answers.
CLAUDE.md
- Corrects the Perspective version and the CDN claim.
- Project layout now lists components/, including the two new files.
- Selection section covers multi-select and why duplicate effective
slices are collapsed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the outcome of the Option C spike (server-side DuckDB as a
Perspective virtual server) and the resulting architecture decision.
pf_perspective_options.md:
- Spike findings: latency is excellent (~12ms round trip, 5-29ms
aggregation, ~1.5-1.9s to materialize 534,902 rows) but Perspective's
GenericSQLVirtualServerModel ignores group_by_depth and has no
ViewConfig field for per-node expansion state, so interactive
drill-down is not achievable. This affects options B, C and D alike
since they share that SQL model.
- Decision: the real lever is grain, not transport. Pre-aggregating to
display grain collapses 534,902 -> 4,642 rows (~115x) on osm_stack
while keeping the native Perspective engine, so expand/collapse/
depth/sort/filter continue to work.
- Two candidate designs (Path A live virtual server vs Path B
pre-aggregated extract) with the deciding question: do real cuts ever
exceed the browser's leaf-row ceiling?
pf_spec.md:
- Concrete Path B design: pf.col_meta.in_grain, GET /api/versions/:id/agg,
synthetic pf_gkey index, and the append-deltas write/undo model that
mirrors the prior Excel pivot-cache workflow.
Drop pf_ux_mockup.md — an ASCII mockup of UI that is now built; the
views in ui/src/views are the current reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pivot stopped rendering with:
LinkError: WebAssembly.instantiate(): Import #8 "env" "psp_opfs_load":
function import requires a callable
Nobody changed anything. The 4.x CDN bundle resolves its server WASM with
new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)
which from .../client@4.4.0/dist/cdn/ resolves to
.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm -- with no
version. jsdelivr serves @latest, so when @perspective-dev/server@5.2.0 was
published on 2026-08-10 every page load began linking a 5.2.0 WASM against a
4.4.0 client. 4.4.1 and 4.5.2 carry the identical unversioned pattern, so no
4.x pin is safe over CDN. Beyond the outage, an unversioned URL means users
execute whatever that package publishes next, unreviewed.
Switch to the /inline entrypoints, which embed the WASM in the Vite build:
no runtime fetch, and the version is fixed by package-lock.json (verified:
zero `new URL(...perspective-server...)` in perspective.inline.js).
- pin client/viewer/viewer-datagrid/server exact at 5.2.0. The explicit
`server` pin matters: client declares it as "" (an empty range), which npm
also resolves to latest -- the same break, at install time instead.
- drop the viewer-d3fc import. pf_app never selects a chart plugin, and d3fc
has no 5.x; loading 4.4.1 against a 5.x viewer only emits
`get_static_config is not a function` per plugin.
- themes move from a CDN <link> to @perspective-dev/viewer/themes.
Verified end-to-end with every non-localhost request aborted: no external
requests are attempted, both custom elements register, the Arrow stream
ingests, and the pivot renders (TOTAL 17,235.97 = -7,573.97 + 30,907.47
- 6,097.53). apache-arrow 21.1.0 ingests cleanly against the 5.2.0 WASM.
Bundle grows 263 KB -> 11.6 MB (5.4 MB gzipped); that is the embedded WASM.
PERSPECTIVE.md also records findings from the same investigation:
- §3a: expression columns are row-level, evaluated before aggregation, so a
ratio like "revenue"/"qty" summed per row is wrong under any pivot (not
just split_by). Fix is a weighted-mean aggregate, whose weight column must
be a NESTED array: ['weighted mean', ['qty']]. Works in 4.4.0 and survives
incremental table.update().
- §2: withdraws the recommendation of the 4.5.1-core + 4.4.1-d3fc pair. It
does not deliver charts, so the trilemma is really a dilemma: inline
bundling XOR charts. dataflow is on that pair and needs the same migration.
- §5: cleanLayout() does not sanitize `aggregates`; guard it before adopting
the weighted-mean pattern or a dropped column aborts the whole restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SBt3EtKaP9D2mmWJ6Q4bov
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the @perspective-dev distribution (not FINOS @finos/perspective):
loader (npm /inline vs CDN), the version trilemma (inline needs 4.5.x,
viewer-d3fc caps at 4.4.1, charts need 4.4.1 — can't have all three),
Arrow vs JSON delivery constraints, deploy pattern, and an upgrade smoke
test. Correct CLAUDE.md's stale perspective.finos.org link to the actual
@perspective-dev repo.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaced native <select> (macOS ignores CSS on option elements) with a
custom button+ul dropdown. Background/text/border colors are applied via
useTheme so they respond correctly to dark mode toggle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GET /api/dim-period/cols queries information_schema for pf.dim_period columns
(excluding sdat/edat/drange/ndays) so the UI always reflects actual columns.
Setup col_meta editor now shows a dropdown populated from that endpoint instead
of a free-text field, preventing invalid column names like the cash source had.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL generator no longer requires a units col; recode/clone/scale omit units
expressions when none is configured in col_meta
- Source registration validation drops units from required roles (value + date
are the only hard requirements)
- DELETE /api/sources/:id returns 409 when existing versions reference the source
- Setup.jsx surfaces the 409 error via flash instead of silently failing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- units role is now optional; spec and CLAUDE.md reflect conditionality in SQL patterns
- pf.col_meta gains dim_group and dim_period_col fields (documented in both files)
- pf.dim_period calendar table added to schema docs
- pf.source default_layout column added to spec DDL
- Forecast table metadata columns corrected to pf_iter/pf_logid/pf_created_at throughout spec
- SQL patterns updated with correct CTE structure and RETURNING * to match generated code
- Project status updated to 2026-06-12; stale Arrow IPC open question removed
- todo.md deleted; open items retained in CLAUDE.md known issues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta gets dim_period_col field: maps a dimension column to its pf.dim_period counterpart (e.g. year -> cal_year, month -> cal_month)
- When the date column is is_key of a dim_group and any sibling dimension has dim_period_col set, baseline and reference SQL JOIN pf.dim_period on the shifted date instead of copying raw source values
- No dim_period config = identical SQL to before (fully backwards compatible)
- Setup UI: period col input in col_meta editor, enabled for dimension columns with a dim_group set
- Schema migration applied: dim_period_col text null on pf.col_meta
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/sources/:id/lookup?col=X&value=Y — given a key column value, queries the source table for sibling column values in the same dim_group; returns null if no match or ambiguous
- Recode and Clone panels: key columns (is_key + dim_group) trigger lookup on blur and auto-fill sibling inputs that the user hasn't already typed into
- Row labels now use col_meta label field when set, falling back to cname
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PATCH /api/log/:logid — saves note updates to pf.log (was missing, frontend call was silently failing)
- GET /api/versions/:id/log — joins fc_table to return row_count per entry so the change log modal shows rows affected instead of '—'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta: add dim_group field to group related columns (dimension hierarchies, date-adjacent columns); is_key now enabled for date role to mark group parent
- sources.js: upsert includes dim_group
- Setup.jsx: group column in col_meta editor, key checkbox enabled for date role
- gen_dim_period.sql: create and populate pf.dim_period with calendar and fiscal period cuts (monthly grain, 2018-2035)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch server Arrow encoding from tableFromJSON (row objects) to
tableFromArrays (column arrays) — cuts peak Node heap 3-5x for large
datasets by avoiding one JS object per row
- Remove unused pf.log JOIN from data endpoint; forecast rows only
- Load Perspective viewer with direct table reference instead of worker
Server object — fixes "No Table attached" error on large datasets where
named-table registry lookup raced against WASM initialization
- Pre-emptively clean up stale named table in worker registry before
creating, eliminating the "already exists" retry path that silently
swallowed errors (finally ran but flash never fired)
- Strip cfg.table from restore configs since table is loaded by reference
- Throttle progress bar updates to 100ms intervals (was every chunk)
- Persist load errors until dismissed; add console.error for devtools
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Add PayloadPreview component showing the exact JSON that will be POSTed,
live-updating as form fields change (value_incr shown as computed delta)
- buildEffectiveSlice strips expression/system columns and converts
Perspective ms-timestamps to ISO date strings for date-role columns
- fetchCurrentTotals now includes date columns in Perspective view filter
(passing ms number as Perspective expects) so subtotals respect the
clicked date
- Server buildWhere now receives filterCols (dimensions + date cols) so
date values reach the SQL WHERE clause correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Bootstrap fill icons with Feather-style stroke SVGs (sun with
rays + crescent moon) in StatusBar toggle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forecast falls back to a saved per-source layout when no version-local
layout is cached, so new versions of a source open with a sensible pivot
without each user reconfiguring it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reference segments can now apply a date offset just like baselines.
SQL template gains the {{date_offset}} token; both POST /reference and
PUT baseline/:logid pass it through. Existing sources need to
regenerate SQL to pick up the new template — old stored reference SQL
ignores the token (preserving prior verbatim behavior). The Baseline
form drops the "dates land verbatim" hint and shows the offset
control for both segment types.
Editing a segment now color-codes the source row amber with a ring
and tints the form border + header amber so the active connection is
visually obvious. Header label reads "Edit segment #3 — baseline —
note" instead of just "#43" (the internal log id).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New forecasts opened the pivot with all dimensions stacked as
group_by and the date column as split_by — wide and slow to read.
Open with just the value column showing and pf_iter as rows so the
first thing you see is iteration totals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The segment form is now one component rendered in either 'view' or
'edit' mode — the expanded segment row in the list and the
add/edit form below share the same layout, view mode just disables
the inputs. Edit and View are visually identical so toggling between
them feels like enabling fields, not switching tools.
Filters become groups (conditions AND-ed inside, groups OR-ed
between) with + AND condition and + Add OR group affordances. The
compiled WHERE renders live below the groups so you can see what's
being built. A "Switch to manual SQL" toggle flips to a textarea
seeded with the compiled clause; backend baseline POST/PUT and
reference POST accept raw_where alongside filters and store whichever
arrived in pf.log.params for round-tripping.
The Add form is hidden until you click "+ Add segment" at the
bottom of the segments table; Edit also opens it. Cancel/Close
returns the table to its compact state.
/versions/:id/log now also returns value_total, units_total, and the
column names so the segments table can show row count and value sum
inline (header uses the source's actual value column name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Annotates each item with the design choice or open question. Notes
where existing spec coverage already addresses items 3 and 4
(structured filter groups and raw_where escape hatch) and where the
remaining work is wiring vs. greenfield.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
