Two gaps you hit. There was nowhere to set "counts toward" while defining a
segment -- only in the list afterwards -- and the Edit buttons disappear
entirely once any adjustment exists.
That guard is right in principle and too broad in practice. Editing a
segment's filters or date offset after a scale would silently recalibrate a
distribution that was sized against the old rows, so it stays gated. But the
label and the bucket are presentation: they change what the pivot shows and
what the segment counts toward, never which rows were loaded. Those are now
editable in the list at any time, and settable on the create form.
pf.log.label is new: the segment's display name, falling back to tag then
note. Separate from both because those have jobs already -- tag groups
adjustments into initiatives for the bridge, note is commentary -- and
because the label is where sort order lives. Perspective orders column
groups by the value string, so a leading "01 - " is how ordering gets
expressed, and putting that in the note would put it in every note.
The load routes do not yet carry bucket and label through: they return only
rows_affected, with no log id to attach them to. That comes with the switch
away from computed prefixes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Perspective orders column groups by the value string, and SortDir's col asc /
col desc only reverses that -- so Prior Year -> Plan -> Actual -> Forecast is
expressible as neither, being alphabetical in neither direction. The order
has to be part of the value, as a "01 · " prefix.
Done as Perspective expression columns, named Bucket and Segment, generated
from pf.version.bucket_order and pf.log.seq. The first attempt computed the
prefix in the served SQL (archived on feature/column-sequencing-sql), which
was the wrong layer: the prefix is a pivot-ordering concern, and putting it
in the query put it in every other reader too -- the change log and the
bridge's basis list would both have read "04 · Forecast". It also meant a
Generate SQL to introduce the placeholder, and a reload to see any change.
As expressions it stays in the pivot, travels with saved layouts because it
lives in ViewConfig, and reordering on the Baseline page takes effect
immediately -- the expressions are rebuilt and the pivot re-renders, no
reload.
Kept from the SQL attempt: pf.version.bucket_order and pf.log.seq, which are
needed either way, and the Baseline controls -- a seq column per segment and
a reorderable row of bucket chips. bucket_order is on the version because
the Baseline page is version-scoped; Setup is the only source-level context
and not where anyone would look for this.
Anything unordered falls through to the raw column, so it sorts after the
ordered entries (digits before letters) rather than silently landing first.
The expressions are merged into the live config rather than replacing it, so
a user's own expressions survive, and an expression that stops existing is
dropped from the axes first -- restore() rejects a config that pivots on an
expression it no longer defines.
Needs 01_schema.sql for the two columns. No Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Filtering the loads out in the dialog left the server still joining the
whole forecast table for them and discarding the answer. ?kind=adjustments
moves the filter into the WHERE, so they never enter the join: on version 29
the join goes from 2,556,821 rows to 25, and the query from 1666ms to 719ms.
Still 719ms, because nothing indexes pf_logid and it stays a sequential scan
of 2.5M rows. New forecast tables now get an index on it. That column is how
every entry-level operation finds its rows -- undo deletes by it, this
aggregate groups by it, and it is part of the grain key -- so the scan was
being paid on all of them.
Existing tables predate the index and still scan; fc_osm_skinny_29 would
want it added by hand.
The route keeps returning everything by default: Baseline.jsx lists the
segments from the same endpoint, and the ledger's tag lookup needs every
entry to label its lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PATCH /api/log/:logid — saves note updates to pf.log (was missing, frontend call was silently failing)
- GET /api/versions/:id/log — joins fc_table to return row_count per entry so the change log modal shows rows affected instead of '—'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Perspective table is now created with index: 'pf_id'. Delete endpoints
return the pf_ids they removed; the client calls table.remove(pf_ids)
in undoEntry. Avoids the full /data refetch that dominated undo time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Sources page: left column with stacked DB tables + registered sources panels,
right column as full-height column mapping workbench
- Add compact table search, column search, table preview button, delete source button
- Rename fc_table system columns to pf_ prefix (pf_id, pf_iter, pf_logid,
pf_created_at) to avoid collisions with source table columns like 'id'
- Remove 'filter' col_meta role — any non-ignore column usable in baseline filters
- Replace structured filter row builder with free-form SQL WHERE clause textarea
and clickable column chips for insertion; fully flexible AND/OR logic
- Baseline segment cards now display raw WHERE clause text + offset
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
