Three changes, one feature: a period with no baseline borrows a shape from
prior year or plan.
The generator handled one date group. sql_generator.js used find(), so a
source with order, requested and ship date groups derived the order one and
copied the other two raw -- shifted dates against unshifted period labels.
Every group now gets its own join, and they are LEFT rather than inner: an
order that has not shipped has no ship date, and an inner join would have
dropped the row from the load entirely rather than leaving its period
columns empty. dateGroupsOf() is now shared, like grainOf, so the routes and
the generator agree on membership and on join aliases.
Clone carries every date column rather than only the primary one, since it
is the operation that moves rows through time, and re-derives the period
dimensions from pf.dim_period against the shifted date instead of copying
them from the row being cloned.
from_logid names the segment to copy from, replacing the exclude clause for
that one entry rather than widening it. Rows written stay pf_iter = 'clone',
so scale applies to them afterwards as a second step.
set is no longer required: copying a segment forward unchanged is a
reasonable thing to ask for.
Needs dim_period_col set in Setup (fisc_year on oseas/rseas/sseas_e,
fisc_month_abbr on omon/rmon/smon_e) and Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_bucket mirrored pf_segment for scale/recode/clone rows, so adjustments
showed as '(adjustment)' -- a banner of their own, sitting outside the
forecast they are adjustments to. Grouping by pf_bucket therefore split the
forecast in two.
An adjustment is always part of the forecast: exclude_iters keeps operations
off the reference segments, so there is no adjustment that is not. It falls
back to 'Forecast' now, and still yields to an explicit bucket on its own
log entry.
pf_segment keeps '(adjustment)', which is right there -- it answers which
segment a row came from, and an adjustment came from none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_gkey is the Perspective table index -- an opaque handle read only by
table.update() and table.remove(). It was the raw concat_ws of every grain
column, which on the 24-column grain of fc_osm_skinny_29 averaged 233
characters. Being unique per row by construction, it also defeated Arrow's
dictionary encoding, so it alone accounted for 65.6 MB of a 109 MB payload
-- more than the other 31 columns combined, all of which do dictionary
cleanly.
md5 of the same string keeps the determinism undo depends on (routes/log.js
recomputes the key through this same grainOf) and keeps collisions out of
reach at 128 bits, while fixing the width at 32 characters.
Measured on the live 285,685-row aggregate: payload 109.0 MB -> 54.2 MB,
all 285,685 keys still distinct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generator picked columns with find(), so a source with more than one
date or value column silently loaded only the first of each. The version's
DDL is built from col_meta separately, so the extra columns existed in the
forecast table and stayed null for its whole life -- gs.osm_skinny lost
sdate and stdcost_usd across 295k rows that way.
Loads copy the source row wholesale, so they now carry all of them, with
the offset shifting every date column together. The adjustment operations
are genuinely single-measure -- scale distributes one {{value_incr}} -- so
they keep the narrow column list until that has a defined meaning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- SQL generator no longer requires a units col; recode/clone/scale omit units
expressions when none is configured in col_meta
- Source registration validation drops units from required roles (value + date
are the only hard requirements)
- DELETE /api/sources/:id returns 409 when existing versions reference the source
- Setup.jsx surfaces the 409 error via flash instead of silently failing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta gets dim_period_col field: maps a dimension column to its pf.dim_period counterpart (e.g. year -> cal_year, month -> cal_month)
- When the date column is is_key of a dim_group and any sibling dimension has dim_period_col set, baseline and reference SQL JOIN pf.dim_period on the shifted date instead of copying raw source values
- No dim_period config = identical SQL to before (fully backwards compatible)
- Setup UI: period col input in col_meta editor, enabled for dimension columns with a dim_group set
- Schema migration applied: dim_period_col text null on pf.col_meta
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reference segments can now apply a date offset just like baselines.
SQL template gains the {{date_offset}} token; both POST /reference and
PUT baseline/:logid pass it through. Existing sources need to
regenerate SQL to pick up the new template — old stored reference SQL
ignores the token (preserving prior verbatim behavior). The Baseline
form drops the "dates land verbatim" hint and shows the offset
control for both segment types.
Editing a segment now color-codes the source row amber with a ring
and tints the form border + header amber so the active connection is
visually obvious. Header label reads "Edit segment #3 — baseline —
note" instead of just "#43" (the internal log id).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix perspective-click handler to use event filter triples instead of
__ROW_PATH__ — Perspective encodes row position as [col,'==',val] in
detail.config.filter
- buildWhere now skips unrecognised slice keys (e.g. pf_iter) instead of
throwing, so only dimension columns reach the WHERE clause
- Add draggable resize handle on the operation panel (160–480px)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Baseline.jsx: merge Reference section into Add Segment form with baseline/reference toggle; segment rows now clickable to expand stored WHERE clause + timeline; date filter inputs use type="date" for date-role columns
- Timeline.jsx: add type prop ('baseline'|'reference'); reference band uses purple; single-band height shrinks to 52px; canvas uses requestAnimationFrame to fix offsetWidth=0 on mount
- operations.js: reference route now accepts where_clause like baseline (drops date_from/date_to)
- sql_generator.js: reference SQL template uses {{filter_clause}} instead of hardcoded BETWEEN
Note: existing sources need Generate SQL re-run to pick up the new reference template.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Sources page: left column with stacked DB tables + registered sources panels,
right column as full-height column mapping workbench
- Add compact table search, column search, table preview button, delete source button
- Rename fc_table system columns to pf_ prefix (pf_id, pf_iter, pf_logid,
pf_created_at) to avoid collisions with source table columns like 'id'
- Remove 'filter' col_meta role — any non-ignore column usable in baseline filters
- Replace structured filter row builder with free-form SQL WHERE clause textarea
and clickable column chips for insertion; fully flexible AND/OR logic
- Baseline segment cards now display raw WHERE clause text + offset
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New Baseline nav view replaces the simple Load Baseline modal
- Baseline loads are now additive; each segment is independently undoable
- Filter builder: any date/filter-role column, full operator set
- Timeline preview shows source → projected period bars for date BETWEEN filters
- Clear Baseline action deletes all baseline rows and log entries
- DELETE /api/versions/:id/baseline route
- buildFilterClause() added to sql_generator
- filter role added to col_meta editor
- Reminder: re-run generate-sql for each source after this change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The baseline operation now accepts a date_offset interval (e.g. "1 year",
"6 months") and applies it to every date when inserting rows, shifting
historical actuals into the target forecast period.
SQL: {date_col} + '{{date_offset}}'::interval)::date at insert time.
Route: defaults to '0 days' if omitted so existing calls are unaffected.
UI: year/month spinners with a live before→after month chip preview so
the projected landing period is visible before submitting.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of re-fetching all forecast data after scale/recode/clone/reference,
the routes now return the inserted rows directly. The frontend uses ag-Grid's
applyTransaction to add only the new rows, eliminating the full reload round-trip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
