params says what was asked for. That is not enough to explain a surprising
result, because the same intent produces different rows depending on state
the entry does not carry, and because the translation from intent to SQL is
itself a place bugs live.
So both, not one. env records the state that cannot be reconstructed later:
the territory in force, the version's exclude_iters, and when the template
was generated -- all mutable rows elsewhere with nothing remembering what
they were. sql_text records the statement as executed, territory and scope
already resolved into it.
The template generation is a fingerprint rather than a version. It cannot
bring the old template back; it can tell you the entry did not run under the
current one, which is what would otherwise make a comparison quietly wrong.
Generate SQL has overwritten those templates four times today.
The statement is fetched on demand through GET /log/:logid/debug and left out
of the list, which is opened to scan rather than to read SQL. Under an
entry's payload in the change log there is now an "executed SQL" toggle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PATCH /log/:logid had no check, so any account could edit the tag, note,
label and bucket of any entry -- including loads whose rows it cannot see.
That reads as harmless annotation and is not: label and bucket name the
pivot's columns for everyone in the version, so a rep could rename the
company's segments.
Same rule as undo now, author or admin, with the fields shown read-only
rather than editable-then-403 -- in the change log's tag and note cells and
on the Baseline page's label and bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Territory filtering was deferred from v1, so every account saw and could
write every row. With the sales team about to adjust their own territories
that is the thing standing in the way, and it is also why a rep would wait
fifteen seconds to load 2.7M rows to work on a few thousand.
The list lives on pf.app_user.territory with is_admin beside it, and
col_meta.is_territory marks which column of a source the values belong to --
flagged rather than named in code, so a second source can be divided by
something other than a sales rep.
Fail closed: buildTerritoryClause returns FALSE for an empty list or an
unflagged source. An account nobody configured sees nothing, rather than
everything because a column was left null.
Built from the session, never the request. That is what separates it from
`scope`, which the browser sends and should: a filter the user chose belongs
in the payload, a permission cannot come from the thing it restrains. It is
ANDed on last, where nothing in the request can undo it.
Enforced on /data (the cursor and the count behind X-Row-Count), on /agg
before the GROUP BY since the territory column need not be in the grain, on
every operation through sliceUnits, and on the value completion endpoint --
which reads the source table, so without it a dropdown enumerates every
customer and rep in the business to someone shown none of their rows.
Undo is gated by owner rather than territory: it removes an entry's rows
wholesale, so half-undoing one would leave a state nothing describes. Recode
refuses to set the territory column unless you are an admin, since moving a
row between territories is reassignment, not forecasting.
./pf.sh gains set-territory, set-admin and orphan-territory. The last lists
territory values no account owns -- work under one is invisible to everybody
but an admin, which a typo causes easily and nothing in the app reveals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change log joined the whole forecast table on every open to total rows
it had just written -- 2.5M rows to report a few thousand, and the Baseline
page's row and value columns paid the same cost again.
The totals go onto pf.log at write time instead. This is not a cache that
can drift: a log entry's forecast rows never change once written, because
only the operation owning the logid inserts them and the only thing that
removes them is undo, which deletes the log row too. measure_cols records
which columns the figures are denominated in, since the value and units
roles can be reassigned in col_meta and the numbers would otherwise quietly
come to mean something else.
Stamping is best-effort and runs after the commit: a failure to record what
happened must not roll back the thing that happened. The loads return their
new log id to make it possible, the adjustments take theirs from the rows
they return, and apply_mode 'each' writes one entry per slice, so it is a
set rather than a single id.
?recount=1 does it the old way and writes back what it finds. Stored totals
cannot drift on their own, but nothing stops someone deleting forecast rows
by hand, and a stored figure has no way to notice -- so there is a way back,
which doubles as the backfill for entries written before the columns
existed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bucket_order and log.seq were still settable -- PUT /versions/:id took the
first, PATCH /log/:logid the second -- with nothing left to read either. A
field that only ever gets written is worse than a missing one: the call
succeeds, so the caller has no way to find out it did nothing.
The columns themselves stay, marked vestigial where they are declared.
Dropping a column is not worth a migration to reclaim two that cost nothing.
pf.log.bucket is untouched and stays exactly as it was -- what a row counts
toward, read first by BUCKET_EXPR. It is the bucket *order* that no longer
needs storing, the text having become the order.
Also removes a comment head in Forecast.jsx that survived its function and
had glued itself onto fitColumns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps you hit. There was nowhere to set "counts toward" while defining a
segment -- only in the list afterwards -- and the Edit buttons disappear
entirely once any adjustment exists.
That guard is right in principle and too broad in practice. Editing a
segment's filters or date offset after a scale would silently recalibrate a
distribution that was sized against the old rows, so it stays gated. But the
label and the bucket are presentation: they change what the pivot shows and
what the segment counts toward, never which rows were loaded. Those are now
editable in the list at any time, and settable on the create form.
pf.log.label is new: the segment's display name, falling back to tag then
note. Separate from both because those have jobs already -- tag groups
adjustments into initiatives for the bridge, note is commentary -- and
because the label is where sort order lives. Perspective orders column
groups by the value string, so a leading "01 - " is how ordering gets
expressed, and putting that in the note would put it in every note.
The load routes do not yet carry bucket and label through: they return only
rows_affected, with no log id to attach them to. That comes with the switch
away from computed prefixes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Perspective orders column groups by the value string, and SortDir's col asc /
col desc only reverses that -- so Prior Year -> Plan -> Actual -> Forecast is
expressible as neither, being alphabetical in neither direction. The order
has to be part of the value, as a "01 · " prefix.
Done as Perspective expression columns, named Bucket and Segment, generated
from pf.version.bucket_order and pf.log.seq. The first attempt computed the
prefix in the served SQL (archived on feature/column-sequencing-sql), which
was the wrong layer: the prefix is a pivot-ordering concern, and putting it
in the query put it in every other reader too -- the change log and the
bridge's basis list would both have read "04 · Forecast". It also meant a
Generate SQL to introduce the placeholder, and a reload to see any change.
As expressions it stays in the pivot, travels with saved layouts because it
lives in ViewConfig, and reordering on the Baseline page takes effect
immediately -- the expressions are rebuilt and the pivot re-renders, no
reload.
Kept from the SQL attempt: pf.version.bucket_order and pf.log.seq, which are
needed either way, and the Baseline controls -- a seq column per segment and
a reorderable row of bucket chips. bucket_order is on the version because
the Baseline page is version-scoped; Setup is the only source-level context
and not where anyone would look for this.
Anything unordered falls through to the raw column, so it sorts after the
ordered entries (digits before letters) rather than silently landing first.
The expressions are merged into the live config rather than replacing it, so
a user's own expressions survive, and an expression that stops existing is
dropped from the axes first -- restore() rejects a config that pivots on an
expression it no longer defines.
Needs 01_schema.sql for the two columns. No Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Filtering the loads out in the dialog left the server still joining the
whole forecast table for them and discarding the answer. ?kind=adjustments
moves the filter into the WHERE, so they never enter the join: on version 29
the join goes from 2,556,821 rows to 25, and the query from 1666ms to 719ms.
Still 719ms, because nothing indexes pf_logid and it stays a sequential scan
of 2.5M rows. New forecast tables now get an index on it. That column is how
every entry-level operation finds its rows -- undo deletes by it, this
aggregate groups by it, and it is part of the grain key -- so the scan was
being paid on all of them.
Existing tables predate the index and still scan; fc_osm_skinny_29 would
want it added by hand.
The route keeps returning everything by default: Baseline.jsx lists the
segments from the same endpoint, and the ledger's tag lookup needs every
entry to label its lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PATCH /api/log/:logid — saves note updates to pf.log (was missing, frontend call was silently failing)
- GET /api/versions/:id/log — joins fc_table to return row_count per entry so the change log modal shows rows affected instead of '—'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Perspective table is now created with index: 'pf_id'. Delete endpoints
return the pf_ids they removed; the client calls table.remove(pf_ids)
in undoEntry. Avoids the full /data refetch that dominated undo time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Sources page: left column with stacked DB tables + registered sources panels,
right column as full-height column mapping workbench
- Add compact table search, column search, table preview button, delete source button
- Rename fc_table system columns to pf_ prefix (pf_id, pf_iter, pf_logid,
pf_created_at) to avoid collisions with source table columns like 'id'
- Remove 'filter' col_meta role — any non-ignore column usable in baseline filters
- Replace structured filter row builder with free-form SQL WHERE clause textarea
and clickable column chips for insertion; fully flexible AND/OR logic
- Baseline segment cards now display raw WHERE clause text + offset
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
