Shifting backwards already worked -- the offset is added as a Postgres
interval, and '-90 days' is a perfectly good one -- but the suggestions only
listed forward shifts, so nothing said so. Pulling a plan back a quarter is
a normal thing to want.
A day-level shift is also the case that most needs the dim_period
derivation: 15 Mar 2027 less 90 days is 15 Dec 2026, which crosses from
2027/10 - Mar into 2027/07 - Dec. Copying the period columns across would
have labelled it March.
The offset is interpolated into the statement, so a typo surfaced as a
Postgres parse error from the middle of a CTE. It is parsed on its own
first, where the failure is cheap and can name the field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel required at least one override for both recode and clone. For
recode that is right -- rewriting rows as themselves does nothing. For clone
it is not: copying prior year forward changes the dates and, through
dim_period, the season and month dimensions, which is the entire point of
cloning from a reference segment. The server-side requirement was dropped
when from_logid went in; this one was missed, so the operation was blocked
in the UI with "Enter at least one override value".
Clone now only has to be doing something -- an override, a shift, or a
factor other than 1 -- and says so when it is doing none of the three.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge filtered on exclude_iters, so it dropped every reference row --
including Open Orders, which is loaded as reference precisely so nothing
adjusts it and is still part of the forecast. Membership is pf_bucket now,
which is the axis that answers this question; pf_iter answers a different
one.
What you compare against depends on what you are building: an AOP is built
off a prior period, a forecast off an update to the AOP. So the basis is
chosen rather than assumed, and the walk stays exact either way because
Forecast - Basis = (Forecast loads - Basis) + adjustments
The opening step is that first term, every tagged adjustment explains the
rest, and the bars sum to the endpoint by construction. No residual to
explain away -- which is what a bridge from prior year to forecast would
otherwise be, since the adjustments describe movement from the forecast's
own loads and not from last year.
With no basis it is the composition instead: the forecast's loads as the
opening anchor, then the adjustments. The picker only appears once something
carries a bucket, so a version that has not been labelled behaves as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The slice filters the rows being copied, so cloning prior year forward means
selecting last December and shifting it, not selecting the December you want
to fill. Selecting the destination matches nothing -- prior year's rows are
labelled with prior year's periods, so a 2027 slice and a 2026 segment share
no rows -- and it fails silently, as zero rows cloned.
Nothing in the form said which way round it was, and "copy from <segment>"
reads like the selection is the target. It now says the selection is what
gets copied, and the segment narrows it rather than replacing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes, one feature: a period with no baseline borrows a shape from
prior year or plan.
The generator handled one date group. sql_generator.js used find(), so a
source with order, requested and ship date groups derived the order one and
copied the other two raw -- shifted dates against unshifted period labels.
Every group now gets its own join, and they are LEFT rather than inner: an
order that has not shipped has no ship date, and an inner join would have
dropped the row from the load entirely rather than leaving its period
columns empty. dateGroupsOf() is now shared, like grainOf, so the routes and
the generator agree on membership and on join aliases.
Clone carries every date column rather than only the primary one, since it
is the operation that moves rows through time, and re-derives the period
dimensions from pf.dim_period against the shifted date instead of copying
them from the row being cloned.
from_logid names the segment to copy from, replacing the exclude clause for
that one entry rather than widening it. Rows written stay pf_iter = 'clone',
so scale applies to them afterwards as a second step.
set is no longer required: copying a segment forward unchanged is a
reasonable thing to ask for.
Needs dim_period_col set in Setup (fisc_year on oseas/rseas/sseas_e,
fisc_month_abbr on omon/rmon/smon_e) and Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_bucket mirrored pf_segment for scale/recode/clone rows, so adjustments
showed as '(adjustment)' -- a banner of their own, sitting outside the
forecast they are adjustments to. Grouping by pf_bucket therefore split the
forecast in two.
An adjustment is always part of the forecast: exclude_iters keeps operations
off the reference segments, so there is no adjustment that is not. It falls
back to 'Forecast' now, and still yields to an explicit bucket on its own
log entry.
pf_segment keeps '(adjustment)', which is right there -- it answers which
segment a row came from, and an adjustment came from none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The effect decided which dim_groups to load from colMetaRef.current, but
that ref is filled inside initViewer, which is async. The effect ran first,
saw an empty array, found no groups, and fetched nothing -- and a ref
changing does not re-run an effect, so it never recovered. dimMembers stayed
{} for the whole session.
Everything therefore fell back to the source lookup, which is precisely the
query pf.dim_member exists to replace: XCP06500G18B112 has two attribute
combinations across history, so DISTINCT ... LIMIT 2 returned two rows and
the route answered null. The member row had the answer all along, picked as
the most recent by order date.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md said is_key "marks dimensions used in slice WHERE clauses", which
is not true and never was: assertSelective and buildWhere validate against
filterCols, which is every dimension plus every date column regardless.
What it actually drives is the key of a dim_group, value completion, the
sibling autofill trigger, and the dim_period anchor -- and the first of
those wants exactly one column per group while the others want several. When
a group has more than one, find() silently takes the lowest opos, which is
how segment_new outranked part and a master-data refresh keyed on a column
that is null throughout: zero members, reported as success.
Written down because the failure gives no signal at all -- the refresh
succeeds, the list is simply empty, and every symptom appears somewhere
else entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Autofill filled only empty boxes, so it appeared to work exactly once. Type a
part, get its nine attributes; type a different part, and every box is
already full, so nothing updates and the form still describes the previous
part. That is worse than blank -- the recode would have been submitted with
one part's code and another's attributes.
These columns describe the key that was just entered, so they are replaced
outright now.
A lookup that finds nothing also said nothing, which is indistinguishable
from a lookup that did not run. It warns, and names the value it could not
find, so an unknown part reads differently from a broken autofill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Autofill stopped filling anything as soon as the member endpoint answered,
because a hit was treated as the only possible answer: if the key was not in
the list, it returned rather than trying the source. An empty list counted as
an answer too, which is the state while a refresh is still running -- so
wiring up master data broke the behaviour it was meant to improve, for
everyone who had not finished building the list yet.
A miss is not an answer. Empty list, or a key the list does not know, both
fall through to the source lookup as before. It costs one request, on a path
that only runs when someone types a value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every question about a part -- what values exist, what attributes go with
one -- was answered by querying the source, and the source is the wrong
place to ask. It is a view over a transaction table, so the query is slow
(76s for one ILIKE against 6.9M rows), it describes only what was
transacted, and it cannot express intent: there is no way to say a part is
discontinued, or to name one that has not sold yet.
pf.dim_member holds the app's own list: one row per key value per group,
siblings in jsonb, keyed on (source_id, dim_group, key_value). Refresh is a
merge rather than a replace, so curation survives it -- members absent from
the source are marked source_seen = false, not deleted. Triggered from
Setup, next to Generate SQL, because it reads the whole source and the
answer only changes when the catalogue does.
A key can carry several attribute sets across history -- 11,290 parts
against 13,662 combinations on osm_skinny -- so the refresh takes the most
recent by the source's date column. That also fixes the sibling autofill,
which used to run a DISTINCT ... LIMIT 2 against the source and silently
fill nothing whenever a part came back ambiguous. A member row is one
definition by construction.
The client fetches each group's list once per source and does both
completion and autofill against it in memory, so neither costs a request.
Columns outside a group, or a group never refreshed, still fall back to the
version's values endpoint.
Run 01_schema.sql to create the table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
55814ee introduced calls to sliceUnits() in the recode and clone routes
without ever defining it, so both have thrown "sliceUnits is not defined"
since that commit. Scale worked only because it inlined the same three lines
rather than calling the helper.
Defined from scale's copy, and scale now calls it too. Recode and clone
passed req.body.apply_mode straight through, where scale normalised it
first, so an absent or unrecognised value would have fallen to the 'each'
branch by accident; they normalise the same way now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pointing completion at the source made every keystroke a 76-second query:
gs.osm_skinny is a plain view over rlarp.osm_stack, so ILIKE '%1601%'
scanned 6.9M rows and read 1.3M buffers off disk, 64s of it I/O. Nothing had
called that endpoint before, so the cost only appeared once a debounced
input was wired to it.
The source is the wrong list anyway. It reaches back over all of history and
would offer parts discontinued years ago; the version holds what was
actually loaded, which is what the forecast is being written against.
So GET /versions/:id/values/:col reads the version's own forecast table --
2.0s for 11,290 parts on fc_osm_skinny_29 -- and holds the result in memory,
keyed on that version's latest pf.log id. Any load, adjustment or undo moves
the id and the next request rebuilds, so nothing has to remember to
invalidate. Filtering happens over the cached array, so typing costs one
small max(id) query.
The column name is interpolated into the DISTINCT, so it is checked against
col_meta first.
The source-side endpoint keeps its new q/limit but no longer has a caller;
its ILIKE path against a view is the expensive one and should stay unused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recoding to a part meant typing a code from memory into a plain text box,
with the only feedback being that the sibling autofill either fired or
silently did nothing.
The values endpoint already existed but returned every distinct value with
no filter and no limit, which for part on osm_skinny is 11,290 rows -- too
slow to open and no easier to read than a short list. It now takes ?q= and
?limit=, so the field fetches matches for what has been typed so far,
debounced 200ms, and offers them through a native datalist.
Only key columns get it, which is the same condition the endpoint already
enforced, and the same one that decides whether the dim_group sibling
lookup runs on blur. So on osm_skinny it is part and customer: type 1601
and the eight parts containing it are offered; pick one and the nine
part-group columns fill themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things a reader would otherwise have to rediscover, and one of them is
load-bearing: observerShim.js has to be main.jsx's first import or it
silently intercepts nothing, which no amount of reading the shim itself
tells you. Also records the two false trails found while getting there --
getTable() resolving while getView() throws, and getView() returning a fresh
wrapper every call so identity cannot detect a rebuild.
Known issues rewritten against what is actually true now: the operation
panel wiring and the progress-bar throttle are done, and the load-time entry
now says what was measured -- that the cost is row count, not payload, and
dynamic grain is the remaining lever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding the Value column pushed the table past max-w-4xl: roughly 624px of
declared column widths inside an 896px dialog, leaving Slice and Note to
fight over the rest. With auto table layout the cells won and the table grew
wider than its container, so the dialog got a horizontal scrollbar along the
bottom.
table-fixed makes the declared widths hold and the free-text columns
truncate instead of widening the table -- which is what the truncate classes
on them already assumed. Dialog widened to max-w-6xl, gutters down from
px-4 to px-3, and Note given an explicit share rather than whatever was
left.
max-w-xs on the slice and note cells was doing nothing useful under fixed
layout and is now overflow-hidden, which is what makes truncate work in a
table cell.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things about the change log, from using it.
The slice column grew to a block per entry once it listed lines, which is
too much for a table you scan. It is one truncated line now: a single slice
reads as its fields, and a dragged region as "5 slices · omon = 07 - Dec …
11 - Apr" -- enough to recognise the entry.
There was no indication of what an entry did to the numbers, which is the
first thing you want from a change log. The route already returned
value_total and it simply was not rendered. Added, signed and coloured, so
a list of adjustments reads as a list of impacts.
And clicking a row now opens slice and params as formatted JSON -- the exact
payload that went over the route, including the increments and the plug,
which no amount of summarising in the cell was going to convey.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A multi-slice operation logs an array of slices, and Object.entries over an
array yields its indices -- so the column read "0 = [object Object], 1 =
[object Object], ...", which says nothing about what was adjusted.
Each slice now gets a line. And since a dragged region is one dimension
walked across a fixed set of others, the shared part is pulled out: log 118
is five slices differing only in omon, which now reads as one line of the
four fixed fields and one listing Dec through Apr, rather than the same four
fields five times over.
Only collapsed when exactly one key varies. Two independent axes would lose
their pairing when flattened, so those stay listed slice by slice.
Dates arrive as epoch millis and are shown as dates; the raw payload is on
the cell's title for anything the rendering has flattened.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Filtering the loads out in the dialog left the server still joining the
whole forecast table for them and discarding the answer. ?kind=adjustments
moves the filter into the WHERE, so they never enter the join: on version 29
the join goes from 2,556,821 rows to 25, and the query from 1666ms to 719ms.
Still 719ms, because nothing indexes pf_logid and it stays a sequential scan
of 2.5M rows. New forecast tables now get an index on it. That column is how
every entry-level operation finds its rows -- undo deletes by it, this
aggregate groups by it, and it is part of the grain key -- so the scan was
being paid on all of them.
Existing tables predate the index and still scan; fc_osm_skinny_29 would
want it added by hand.
The route keeps returning everything by default: Baseline.jsx lists the
segments from the same endpoint, and the ledger's tag lookup needs every
entry to label its lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Baseline and reference loads are segment construction, not forecasting.
Listing them beside the adjustments buries what the log is actually for --
on version 29 that is four load entries against however many adjustments --
and offers an undo next to them that would silently gut the version.
They are managed in the Baseline view instead, which is where they can be
edited in place and their date ranges seen. Filtered in the dialog rather
than in the route, since Baseline.jsx and the ledger's tag lookup both read
the same endpoint and do want every entry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sales_usd and qty run to eight digits here, where two decimals are noise.
Price is the opposite case: it sits around 0.27, so two places barely show a
move and four still round away part of one.
derive() hardcoded two decimals, so the editable rows would have disagreed
with the lines above them. It now takes the measure's own precision. The
percentage row keeps one place, being its own scale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Price was blanked on the bridge lines and on the excluded row, so a ledger
that had both measures on every line showed a price on two of them. Baseline
sat there as 2,242,180.67 over 8,111,717.10 with the column empty.
A bridge line's price is the implied price of that initiative's own
contribution -- Scale #116 above works out to 0.2397 against a current
0.2735 -- which is the number that says whether the initiative was a price
move or a volume move. That is the same question the plug toggle asks about
the edit, answered for the adjustments already made.
A line with no units is left blank, since its price is undefined rather
than zero -- the "price" bridge line in that example moved dollars alone.
The price column's hint said "holds units", which stopped being true when
plug arrived: it holds units under plug = price and moves them under plug =
volume. It now just says what the column is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each ledger row derived only from its own input, so typing a dollar figure
left the units and price rows blank -- which are precisely the two numbers
that say whether you just asked for a volume change or a price change. The
plug toggle decided it and then showed you nothing.
Adds a Result line under the edit rows carrying value, units and price
together, resolved the same way the server resolves the increments: a price
input multiplies through volume, a dollar input with plug = volume scales
units in proportion, anything else holds units. Measures that actually move
are picked out in blue against the ones that hold, so plug = price reads as
price moving alone and plug = volume as price standing still.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sales figure on its own does not say which of price or volume moved, and
scale was answering silently: it wrote the value increment and left units
alone, so volume stayed flat and price absorbed everything. That is one of
the three behaviours the Excel form offered, picked by default rather than
by the user.
/opt/forecast_api/VBA/fpvt.frm has the semantics. Edit Sales, then plug one
side:
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
So 'volume' moves units in the same proportion as the dollars, holding
price; 'price' leaves units alone, as now. Checked against those formulas:
+100 on 1000/500 gives units 500 -> 550 with price held at 2.0000, and
-400 on 2000/800 gives 800 -> 640 at 2.5000.
Price still defaults, so nothing changes for an existing caller that does
not send `plug`. The control only appears once the edit is dollars-only --
naming units or price has already settled the question. A price target now
also honours a units target alongside it, which is the form's Edit Price
mode where both are inputs and dollars fall out.
Holding price when the selection has no value is refused rather than
divided by zero, the same case the form guarded with "Zero times any number
is zero".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the trace showed, one of them a wrong call on my part.
getTable() was the wrong thing to wait on. The re-apply threw "No table
set" two milliseconds after the event, meaning getTable() had already
resolved while getView() still had not -- it is the view that is missing
around a rebuild, not the table. There is no predicate for "the view is
ready", so attempt applyDepth and retry until it stops throwing, up to 5s.
And the shim matched nothing: not one callback reported a viewer. closest()
stops at a shadow boundary, so it cannot climb from an element inside
Perspective's shadow root out to the host, which is where the observed
elements live. Walk host to host via getRootNode().host instead.
The shim now also logs every callback it sees under [pf-obs], matched or
not, with a description of each target. If it still reports nothing at all,
the shim is not installed and the problem is import order rather than
matching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The version re-validation effect could not tell "the versions fetch has not
landed yet" from "this source has no versions", because both look like an
empty array. So a versionId restored from localStorage was cleared on the
first render and set straight back when the fetch returned: 29 -> '' -> 29.
Forecast's load effect is keyed on [versionId, sourceId], so that round
trip ran initViewer twice. Two /agg requests, two full aggregations in
Postgres, two Arrow payloads -- on version 29 that is the 285k-row
aggregate computed twice at ~17s each, which is most of the "it hangs
before it displays" and the Postgres process sitting at the top of htop.
sourcesLoaded already guarded the identical effect for sources; versions
just never got the same treatment. Cleared when the source changes, since
the list belongs to the previous source until the new fetch lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The WASM engine has no notion of focus or tab visibility. It re-renders
because IntersectionObserver or ResizeObserver told it the element is
visible again at some size, and that re-render discards the view -- taking
set_depth() and per-node expansion with it, since neither lives in
ViewConfig. Listening for window 'focus' was only a guess at when that
happened, which is why the re-apply kept landing while the viewer was still
detached from its table.
Perspective's viewer captures the constructors at module-evaluation time
(`var it=window.ResizeObserver; var st=window.IntersectionObserver`), so
replacing them on window before that import puts us in front of every
callback it gets. observerShim.js does exactly that and nothing else: the
original callback runs first and unchanged, and we re-emit as a CustomEvent
only when an entry's target is or contains a perspective-viewer. That has
to be the first import in main.jsx or there is nothing left to intercept.
focus/visibilitychange/pageshow stay as a backstop for the case where the
shim could not be installed. ResizeObserver fires on every frame of a panel
drag, so observer events trail by 150ms and act once the burst settles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The refocus timing is the sort of thing that will shift again -- a
Perspective upgrade, a different machine, a slower load -- and rebuilding
this instrumentation from scratch each time is wasted work. So it stays,
behind a flag, instead of being reverted.
probeView() returns before touching the viewer when the flag is off, since
it calls getView() purely to report identity.
Note for whoever reads the log next: the view ids it prints cannot be used
to detect a rebuild. getView() hands back a fresh wrapper object on every
call, so consecutive calls always look "rebuilt". The reliable signal is
whether the re-apply throws "No table set".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instrumentation showed the re-apply mostly throwing "No table set": the
viewer is detached from its table for a while after a refocus, the fixed
60ms delay fired inside that window, set_depth threw, and the tree rendered
fully expanded. The handful of times the delay happened to be long enough,
the log reads "re-apply done depth=0" and the collapse survived.
So poll for the table instead of guessing, up to 5s. queued now clears in a
finally after the work rather than at the top of the callback, so the focus
and visibilitychange that both fire for one window switch no longer each
run a re-apply -- that was the doubled applyDepth in the log.
Does not address per-node +/- collapse, which is still not recorded at all:
expandDepthRef stays null because only the toolbar buttons set it, and the
view exposes expand()/collapse() with no getter to read the state back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Temporary. Row grouping re-expands whenever the browser regains focus, for
both the toolbar EXPAND buttons and the per-row +/-. set_depth() and
per-node expansion both live on the view rather than in ViewConfig, so a
rebuilt view loses them -- but nothing so far distinguishes "the view was
rebuilt" from "the re-apply lost its race with the redraw".
Tags each view object with an id via a WeakMap and logs it at every point
that could rebuild one: focus/visibilitychange/pageshow, config-update,
applyDepth, the theme effect, and initViewer. A changed id across a refocus
means the view was discarded; an unchanged id means the depth re-apply
itself is at fault. Each bail-out in the refocus handler now says which
branch it took, which also shows whether expandDepthRef was ever set.
Prefix [pf-depth]. Revert this commit once the cause is known.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_gkey is the Perspective table index -- an opaque handle read only by
table.update() and table.remove(). It was the raw concat_ws of every grain
column, which on the 24-column grain of fc_osm_skinny_29 averaged 233
characters. Being unique per row by construction, it also defeated Arrow's
dictionary encoding, so it alone accounted for 65.6 MB of a 109 MB payload
-- more than the other 31 columns combined, all of which do dictionary
cleanly.
md5 of the same string keeps the determinism undo depends on (routes/log.js
recomputes the key through this same grainOf) and keeps collisions out of
reach at 128 bits, while fixing the width at 32 characters.
Measured on the live 285,685-row aggregate: payload 109.0 MB -> 54.2 MB,
all 285,685 keys still distinct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The load filter dropdown listed role 'date' and role 'filter' only, so a
segment could not be cut on sseas without recoding it -- and role is a
single value, so that trade would have taken it off the pivot to put it in
a dropdown.
Nothing downstream required the restriction: the server drops filter_clause
into the load's WHERE against the source table without consulting col_meta,
and this form takes values as free text rather than through the key-only
values endpoint. So the fix is to stop filtering them out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ordinary npm churn: dropped peer markers where those packages are now
direct dependents, plus the optional @emnapi platform packages.
.env is ignored but a .env.bak sitting next to it is not, and it holds the
same credentials. Widen the pattern to .env.* so a backup can't be staged
by a careless git add -A.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The label on a baseline or reference load ("Open Orders", "Prior Year")
lived only on the pf.log row; the data stream was a straight dump of the
forecast table, so Perspective saw pf_logid and never the name.
The stream now joins pf.log and emits two columns rather than one, because
commingling them makes neither useful: pf_segment names the load a row came
from, and is '(adjustment)' for everything else; pf_note carries the free
text on scale/recode/clone and is null on loads. The operation routes stamp
the same two fields on the rows they push back incrementally.
The tag column those labels belong in is declared in 01_schema.sql but
predates some installs, so the backfill seeding it from note joins the
ALTER already there. Adjustment notes are free text, not labels, and are
left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generator picked columns with find(), so a source with more than one
date or value column silently loaded only the first of each. The version's
DDL is built from col_meta separately, so the extra columns existed in the
forecast table and stayed null for its whole life -- gs.osm_skinny lost
sdate and stdcost_usd across 295k rows that way.
Loads copy the source row wholesale, so they now carry all of them, with
the offset shifting every date column together. The adjustment operations
are genuinely single-measure -- scale distributes one {{value_incr}} -- so
they keep the narrow column list until that has a defined meaning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The selection is restored from localStorage, but it was only validated once
at mount. Deregistering the selected source left App holding an id that no
longer exists, so every subsequent call 404'd "Source not found" with no way
out but a reload — Setup.deleteSource clears its own selectedSource and never
tells App. Deleting the selected version had the same shape.
Move both checks into effects keyed on the lists themselves. A sourcesLoaded
flag keeps the source effect from firing against the initial empty array and
wiping the restored id before the fetch resolves.
Also coerce both refresh callbacks to an array. A 401 returns an error object,
and data.some() then threw inside an unhandled promise, stranding the
selection instead of clearing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e6e2faeb37)
The server had no authentication: every /api route was open, CORS
allowed any origin, and the identity written to the audit log came from
the request body — the UI sent a hardcoded pf_user: 'admin', which any
client could have set to anything it liked.
Accounts live in pf.app_user with scrypt hashes from node's own crypto,
so there is no native build step and the parameters travel with each
hash. Sessions are express-session over connect-pg-simple in pf.session:
a restart no longer signs everyone out, and a session can be revoked by
deleting its row, which is how disable-user cuts off access immediately
rather than at cookie expiry.
Everything under /api except login/logout/me now requires a session, and
the React app is mounted only once there is one — its load effects call
the API on mount, so a logged-out mount would just fire a burst of 401s.
A session that expires while the app is open lands back on the login
screen: auth.jsx wraps fetch once rather than teaching every call site
to check.
Identity is now read from the session for pf_user, created_by and
closed_by, and the body values are ignored.
Hardened for an internet-facing deployment: trust proxy so req.ip and
secure-cookie detection are right behind TLS termination, httpOnly +
SameSite=Lax + Secure cookies, ten login failures per IP per fifteen
minutes, one error message for unknown, wrong and disabled alike, and a
fresh session id on success. CORS is off entirely unless CORS_ORIGIN
names an origin — a wildcard alongside a session cookie would be CSRF by
construction. The server refuses to boot without SESSION_SECRET rather
than falling back to a guessable default.
pf.sh grows add-user, passwd, list-users, disable-user and enable-user;
passwords are read on stdin and hashed before they reach psql, so no
plaintext in argv or shell history. install.sh generates the secret,
applies 02_auth.sql, and creates the first account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
server.js builds its pg pool from DB_HOST/DB_PORT/DB_NAME/DB_USER/
DB_PASSWORD, and that is what install.sh writes, but pf.sh had been
written against a DATABASE_URL/PF_USER scheme that nothing consumes.
The consequences were real: `status` always reported the database
unreachable, and `config` rewrote .env with cat >, replacing a working
connection with keys the server ignores.
Connect through a run_psql helper built from the DB_* vars, and have
config prompt for those six keys instead. Each prompt defaults to the
current value so entering through changes nothing, the password prompt
is hidden and keeps the stored one when left blank, and any other key
already in .env is carried over rather than dropped.
PF_USER goes away with it — pf_user reaches the server in the request
body, never from the environment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Perspective's column axis cannot be collapsed. The row axis has had it
forever — GROUP BY ROLLUP holds every level and view.set_depth() hides the
deeper ones — but nothing equivalent is exposed for split_by, so a Year over
Month pivot can only ever be shown fully expanded.
The engine already implements it. t_ctx2 is symmetric: set_depth(t_header,
depth), open(t_header, idx) and close(t_header, idx) each have a real
HEADER_COLUMN branch on m_ctraversal mirroring m_rtraversal, and
t_view_config carries m_column_pivot_depth which server.cpp already applies.
None of it is reachable: set_column_pivot_depth() is never called, so the
depth stays -1, and View<t_ctx2>::expand/collapse hardcode HEADER_ROW. The
patch is 193 lines of wiring across the protobuf, the Rust client and the
datagrid — no new engine logic.
Two capabilities result, mirroring the row axis:
- split_by_depth in ViewConfig, the split_by counterpart to group_by_depth
- expand_column()/collapse_column(), addressed by column traversal index
exactly as the row methods are addressed by row index
which together give the Excel behaviour — one year folded to its subtotal
while its siblings stay expanded — that no combination of existing config
could produce. Verified in this app against fc_cash_9: clicking a Year
header goes from 27 columns to 15, totals reconciling at every level.
Vendored rather than aliased
- The previous approach pointed vite at a local checkout, which built only on
one laptop and left package.json claiming npm 5.2.0 while the build used
something else. The four packages are now committed as npm tarballs and
package.json names them, so the declaration is true and `pf.sh deploy`
works unchanged — npm install expands them like any registry package.
- Packed with `pnpm pack`, not `npm pack`: Perspective is a pnpm workspace and
cross-package deps are `workspace:^`, which npm rejects outright. pnpm
rewrites those to real version ranges at pack time.
- All four move together. Perspective couples loader, package versions, data
format and apache-arrow; a partial vendor reintroduces exactly that drift.
Cost, stated plainly: 12MB of opaque binaries in git that do not delta, a
fork to maintain, and a second engine build for anyone changing it.
rebuild-perspective.sh makes that repeatable and PROVENANCE.txt records the
commit each tarball came from, because a committed .tgz otherwise has no
recoverable source. README.md says how to delete all of it once upstream
ships the feature.
This also moves the app from 5.2.0 to 5.4.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
The row axis has had Expand 0/1/2/3 since the pivot landed; the column
axis had nothing. With a year over month split there was no way to step
back to whole years short of dragging split_by apart in the settings
panel and putting it back afterwards.
Perspective gives the two axes nothing in common here. Rows collapse
because the GROUP BY ROLLUP view holds every level at once and
view.set_depth() hides the deeper ones. For columns there is no
equivalent: expand()/collapse() take a row index, ViewConfig has
group_by_depth but no split_by_depth, and split_rollup_mode only chooses
whether subtotal column groups are emitted — a view shape, not an
interaction. So applySplitDepth() collapses by restoring a truncated
split_by, which rebuilds the view.
Three consequences of that rebuild, each handled:
- Once collapsed, viewer.save() only reports the short split_by, so the
full hierarchy is held separately (splitFullRef) and persisted into the
layout as split_full. Without it, collapsing would be a one-way door:
reload while collapsed and the deeper levels are gone. adoptSplit() is
the single place it is set.
- perspective-config-update fires for our own restore as well as for the
user rearranging the pivot, and the two mean opposite things — one must
adopt the new hierarchy, the other must not. collapsingRef separates
them.
- Row depth lives on the discarded view, so it is re-applied afterwards.
The selection is cleared on each change: slices name the split_by
dimensions they were cut from, and the highlight is keyed on grid
coordinates. Neither survives a column axis that just changed shape.
Buttons are named for the level they show — Total, then one per split_by
column — rather than numbered like Expand, since the levels are named and
a number would say nothing about what you are collapsing to.
Whole-axis, not per-branch: Excel can collapse 2025 while 2026 stays
expanded, and this cannot. `columns` selects which measures appear, not
individual split combinations, so there is no way to hide one branch's
leaves while keeping another's.
Verified in the browser against cash/test with split_by Year x Reason:
Reason -> Total -> Year -> Reason all render the expected column sets and
the right button highlights; and a reload while collapsed to Year comes
back collapsed with Reason still offered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Brings the Perspective work forward onto the display-grain decision in
c5b12aa. Four commits:
- Load Perspective from the bundled /inline entrypoints and pin every
package at 5.2.0, after the 4.x CDN bundle resolved its server WASM to
an unversioned path and took the app down on 2026-08-10.
- Multi-slice operations (slices array, apply_mode prorate/each),
per-measure resolution, target_basis, initiative tags, the tagged
bridge, and the reworked adjustment panel.
- Drag to select a region, and show the selection on the grid.
Merges clean against c5b12aa; the two touch disjoint files.
Two gaps in how the pivot reports a selection. Dragging across cells did
nothing, and a selection built from several ctrl-clicks was invisible on
the grid — the panel listed the slices but nothing on screen said which
cells they came from.
Drag-select
- The perspective-select handler was reading detail.selected and
detail.insertConfigs. In 5.2.0 that event carries a ViewWindow —
{ start_row, end_row, start_col, end_col }; insertConfigs only appears
on perspective-global-filter, and only in SELECT_ROW_TREE mode. So the
handler always returned early and the whole path was dead.
- It fires on every mouseover as the region grows, so the handler now
records the latest window and a window-level mouseup commits it. A
single-cell region is skipped there: perspective-click already owns
plain and modifier clicks, and handling it in both places would undo a
ctrl-click toggle. Modifier+drag adds to the selection.
- Turning a region back into slices re-derives, per cell, the same
filters Perspective attaches to a click — row dimensions from the
view's __ROW_PATH__, column dimensions from the split_by segments of
the column name. Reading the path rather than the rendered cell is
what keeps dates as epoch millis instead of whatever the grid
formatted them as. The grand-total row resolves to no dimension at
all — that would mean the whole version — and is skipped.
Highlight
- The datagrid already highlights whatever sits in its own
model._selection_state.selected_areas, and wipes that list on every
mousedown, so a multi-click selection only ever showed the last cell.
Keep a sliceKey -> rectangles map parallel to `slices` and push the
full set back after each change, which gets the native highlight for
every selected cell without any styling of our own.
- Deselecting anywhere — ctrl-click, the panel's x, Clear selection —
prunes the map by live slice key, so the grid and the panel cannot
disagree.
Also: edit_mode is forced to SELECT_REGION on restore rather than
defaulted, since a saved layout's plugin_config could previously
override it and turn selection off entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH
Documentation had drifted far enough to mislead: the spec described the
single-slice panel, the target/delta modes that no longer exist, and the
old `slice` request shape, while both files still claimed Perspective
4.4.0 from a CDN when it has been bundled inline at 5.2.0 since August.
pf_spec.md
- pf.log gains `tag`, with why it is written by a follow-up UPDATE
rather than through the per-source templates in pf.sql.
- Operations envelope documents `slices`, `apply_mode`, and the OR-of-
AND-groups WHERE clause (and why it cannot be flattened to IN lists).
- Scale documents per-measure resolution and `target_basis`, plus the
two guards: non-selective slices, and proration across a ~zero pool.
- New routes: PATCH /log/:logid, table-info, bridge, source tags.
- Forecast View rewritten for the dockable panel and the ledger; adds
the selection caveat around pf_iter and the expand-depth explanation.
- New Bridge View section; Log View gains inline tag editing.
- Status block refreshed, with a Fixed subsection recording the three
correctness bugs and their causes.
- Open Questions: adds bridge drill-down and targeting one iter band;
notes what the bridge partially answers.
CLAUDE.md
- Corrects the Perspective version and the CDN claim.
- Project layout now lists components/, including the two new files.
- Selection section covers multi-select and why duplicate effective
slices are collapsed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the outcome of the Option C spike (server-side DuckDB as a
Perspective virtual server) and the resulting architecture decision.
pf_perspective_options.md:
- Spike findings: latency is excellent (~12ms round trip, 5-29ms
aggregation, ~1.5-1.9s to materialize 534,902 rows) but Perspective's
GenericSQLVirtualServerModel ignores group_by_depth and has no
ViewConfig field for per-node expansion state, so interactive
drill-down is not achievable. This affects options B, C and D alike
since they share that SQL model.
- Decision: the real lever is grain, not transport. Pre-aggregating to
display grain collapses 534,902 -> 4,642 rows (~115x) on osm_stack
while keeping the native Perspective engine, so expand/collapse/
depth/sort/filter continue to work.
- Two candidate designs (Path A live virtual server vs Path B
pre-aggregated extract) with the deciding question: do real cuts ever
exceed the browser's leaf-row ceiling?
pf_spec.md:
- Concrete Path B design: pf.col_meta.in_grain, GET /api/versions/:id/agg,
synthetic pf_gkey index, and the append-deltas write/undo model that
mirrors the prior Excel pivot-cache workflow.
Drop pf_ux_mockup.md — an ASCII mockup of UI that is now built; the
views in ui/src/views are the current reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pivot stopped rendering with:
LinkError: WebAssembly.instantiate(): Import #8 "env" "psp_opfs_load":
function import requires a callable
Nobody changed anything. The 4.x CDN bundle resolves its server WASM with
new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)
which from .../client@4.4.0/dist/cdn/ resolves to
.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm -- with no
version. jsdelivr serves @latest, so when @perspective-dev/server@5.2.0 was
published on 2026-08-10 every page load began linking a 5.2.0 WASM against a
4.4.0 client. 4.4.1 and 4.5.2 carry the identical unversioned pattern, so no
4.x pin is safe over CDN. Beyond the outage, an unversioned URL means users
execute whatever that package publishes next, unreviewed.
Switch to the /inline entrypoints, which embed the WASM in the Vite build:
no runtime fetch, and the version is fixed by package-lock.json (verified:
zero `new URL(...perspective-server...)` in perspective.inline.js).
- pin client/viewer/viewer-datagrid/server exact at 5.2.0. The explicit
`server` pin matters: client declares it as "" (an empty range), which npm
also resolves to latest -- the same break, at install time instead.
- drop the viewer-d3fc import. pf_app never selects a chart plugin, and d3fc
has no 5.x; loading 4.4.1 against a 5.x viewer only emits
`get_static_config is not a function` per plugin.
- themes move from a CDN <link> to @perspective-dev/viewer/themes.
Verified end-to-end with every non-localhost request aborted: no external
requests are attempted, both custom elements register, the Arrow stream
ingests, and the pivot renders (TOTAL 17,235.97 = -7,573.97 + 30,907.47
- 6,097.53). apache-arrow 21.1.0 ingests cleanly against the 5.2.0 WASM.
Bundle grows 263 KB -> 11.6 MB (5.4 MB gzipped); that is the embedded WASM.
PERSPECTIVE.md also records findings from the same investigation:
- §3a: expression columns are row-level, evaluated before aggregation, so a
ratio like "revenue"/"qty" summed per row is wrong under any pivot (not
just split_by). Fix is a weighted-mean aggregate, whose weight column must
be a NESTED array: ['weighted mean', ['qty']]. Works in 4.4.0 and survives
incremental table.update().
- §2: withdraws the recommendation of the 4.5.1-core + 4.4.1-d3fc pair. It
does not deliver charts, so the trilemma is really a dilemma: inline
bundling XOR charts. dataflow is on that pair and needs the same migration.
- §5: cleanLayout() does not sanitize `aggregates`; guard it before adopting
the weighted-mean pattern or a dropped column aborts the whole restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SBt3EtKaP9D2mmWJ6Q4bov
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
