pf_app/docs/pf_spec.md

Pivot Forecast — Design

What the app does, how the pieces fit, and why — including what each design replaced, where that history still explains the current shape. CLAUDE.md holds the rules an agent needs while editing; this holds the reasoning. Where either disagrees with the code (routes/, lib/, ui/src/, setup_sql/), the code is the truth.

Related: pf_perspective_options.md (transport options, the DuckDB spike, why the pre-aggregated path won), PERSPECTIVE.md (bundling), ui/vendor/README.md (the patched Perspective build and how to rebuild it).

---

Overview

A web app for building named forecast scenarios against a PostgreSQL table:

1. Load actuals as a baseline — segments pulled from the source, optionally date-shifted into the forecast period, so the starting point is "no change from last year".
2. Load reference rows — prior year, plan, open orders — for comparison. Operations do not adjust them.
3. Adjust — select pivot cells and scale, recode or clone them.

Every write is append-only and produces exactly one pf.log entry. An adjustment inserts new rows carrying its entry's id; undo deletes those rows and the entry. So a version is always the sum of its log, every figure is attributable to the entry that wrote it, and any entry can be reversed alone.

Postgres holds the raw rows and is the single source of truth. The browser holds a copy in Perspective (WASM), pre-aggregated to the display grain, and updates it incrementally from each operation's response rather than reloading.

Lineage

A port of an Excel model: a data tab of pre-aggregated rows, a pivot cache over it, a double-click on a cell opening a VBA form (/opt/forecast_api/VBA/fpvt.frm) that appended incremental rows, and a cache refresh. The append-and-let-the-pivot-sum model follows that directly, and the price/volume rules in §Operations come from the form's calc_val / calc_price.

Stack

Node/Express (server.js, one pg pool, type parsers coercing bigint and numeric to numbers so Arrow does not infer them as dictionary text); PostgreSQL, everything in schema pf; React + Vite + Tailwind in ui/, built to public/app/; Perspective 5.4.0 from a vendored patched fork (§The pivot).

Pages and navigation

Three steps in a collapsible sidebar: Setup (register sources, assign column roles, Generate SQL), Baseline (versions, load segments, version defaults, tags), Forecast (the pivot and the operation panel). Setup and Baseline are admin-only to write (§Access).

The forecasts are listed in the sidebar, above the steps, each with its status and a star for the default. They used to be two dropdowns in the status bar, which is otherwise read-only, so the one pair of controls in it did not read as controls. The list is not nested under the Forecast step, where the idea started: Baseline works on the same version, so hanging it off one step would misrepresent its scope or duplicate it. The sidebar reads what am I working on, then what step am I on.

The source is not in the nav. A raw table name means nothing to someone editing a forecast. Picking a forecast sets the source, since pf.version.source_id comes with it; the source picker appears only on Setup. versions (the selected source's, which Baseline and Forecast need) and allVersions (the sidebar's, a client-side fan-out over /sources/:id/versions) are kept apart so widening one does not change the other.

The status bar keeps what follows from the selection: the forecast's name and row count, with the physical table in a click-to-copy hover panel — pasting it into psql is why it was ever on screen.

Known collision: "forecast" names both the scenario and the third step. The scenario should own the word, leaving the step wanting another (Adjust). Undecided.

Where a browser lands

App.jsx restores the last source and version from localStorage, re-validated against the live list so a deleted version re-points rather than 404ing every call. With nothing stored it falls back to pf.version.is_default (one per source, partial unique index). It used to fall back to versions[0] — newest first, so whatever somebody created last, a scratch copy as readily as the plan — and accounts that never pick a version, which is every sales account, followed the forecasting work around.

Setting the default is admin only, since it decides where everyone else lands; the Baseline checkbox is greyed for others. It is two statements (clear old, set new) in one transaction, because the index refuses the instant between. A stored selection still wins — the default is a fallback, so an account that landed in the wrong scenario keeps landing there until it switches once.

Renaming a version

Safe at any time, including closed. Nothing derives from the name: the table is fc_{tname}_{version_id}, pf.log and pf.layout key on the id, and client storage keys are id-based. The Baseline page's This forecast panel saves name and description on blur via PUT /versions/:id; a name collision answers 409, and a blank name is refused client-side because the route COALESCEs it away.

API surface

All under /api; all but the auth routes need a session.

Area	Routes
Auth	POST /login, POST /logout, GET /me
Browser	GET /tables, GET /tables/:schema/:tname/preview
Sources	GET/POST /sources, GET/PUT /sources/:id/cols, POST /sources/:id/generate-sql, GET /sources/:id/sql, GET /sources/:id/values/:col, GET /sources/:id/dim/:group, POST /sources/:id/dim/:group/refresh, GET /sources/:id/lookup, GET /dim-period/cols, DELETE /sources/:id
Versions	GET/POST /sources/:id/versions, PUT/DELETE /versions/:id, POST /versions/:id/{close,reopen}, GET /versions/:id/table-info, GET /versions/:id/values/:col, GET/PATCH /versions/:id/tags, GET /versions/:id/bridge
Data	GET /versions/:id/data, GET /versions/:id/agg
Loads	POST /versions/:id/{baseline,reference}, PUT /versions/:id/baseline/:logid, DELETE /versions/:id/baseline
Plans	GET/POST /versions/:id/plans, PATCH/DELETE /plans/:pid
Operations	POST /versions/:id/{scale,recode,clone}
Log	GET /versions/:id/log, GET /log/:logid/debug, PATCH/DELETE /log/:logid
Layouts	GET/POST /versions/:id/layouts, PATCH/DELETE /layouts/:lid

DELETE /sources/:id answers 409 while the source has versions (pf.version.source_id is ON DELETE RESTRICT) and never drops forecast tables.

---

Data model

Install scripts in setup_sql/: 01_schema.sql (metadata tables, with ADD COLUMN IF NOT EXISTS backfills and one-off data migrations inline), 02_auth.sql (pf.app_user, pf.session), gen_dim_period.sql (creates and fills pf.dim_period; fiscal start month set at the top; re-runnable).

01_schema.sql is idempotent but not a migration runner. Schema changes are applied to the live database directly and mirrored into setup_sql/ for fresh installs.

pf.source

schema, tname, label, status, created_by. default_layout is dead — migrated into pf.layout, read by nothing. Registration seeds one col_meta row per column as dimension.

Columns come from pg_catalog, not information_schema, which omits materialized views — they are not in the SQL standard — and the app's primary source is one. It looked columnless: registration seeded nothing and version creation failed "No usable columns in col_meta" while col_meta plainly held them. RELATION_COLUMNS_SQL in lib/utils.js returns the shape information_schema did (format_type minus modifier; precision and scale unpacked from atttypmod), so mapType and callers did not change. The table browser lists from pg_class by relkind for the same reason.

pf.col_meta

Column	Meaning
role	dimension | value | units | date | filter | ignore
label, opos	display name; ordinal, which orders everything generated
is_key	four separate uses, below
dim_group	groups functionally dependent columns — a part and its attributes, a date and its periods
dim_period_col	the pf.dim_period column this dimension is derived from at load
in_grain	part of the display grain (§Display-grain pre-aggregation)
is_territory	the column territory is expressed in; at most one per source, enforced on save

Roles: dimension is categorical and sliceable. value (required) and units (optional) are the measures — several may be marked, loads carry all, but the operations use only the first of each by opos (§Proposed: additional measures). date (at least one) — every date column is stored, shifted and carried by every operation, and sliceable. filter feeds the baseline filter builder and is not stored. ignore is not stored. Slices are validated against every dimension and date column regardless of is_key.

dim_group + dim_period_col. A date column that is is_key with a dim_group anchors the group: its sibling dimensions with a dim_period_col are looked up in pf.dim_period against the shifted date at load time instead of copied raw, so fiscal periods follow a baseline moved forward a year. Every such group is honoured; this used to be a .find(), so a source with order, requested and ship dates derived one set and copied the other two raw — shifted dates under unshifted labels.

is_key is read in four places, none of them what the name suggests:

1. The key of a dim_group — resolveGroup() in routes/sources.js takes members.find(c => c.is_key) as the column pf.dim_member is keyed on
2. Value completion — /sources/:id/values/:col serves only is_key columns (the operation panel now completes from /versions/:id/values/:col instead, which takes any dimension or date)
3. Sibling autofill — /sources/:id/lookup, for is_key && dim_group
4. The dim_period anchor — role = 'date' && is_key && dim_group

Uses 2 and 3 want several flagged; use 1 needs exactly one per group, and with more .find() silently takes the lowest opos. That has happened: a column null throughout outranked its group's real key, so a member refresh matched nothing and reported success with zero members. The meanings want separating — a per-group key choice, or the rule that the key is the column the group is named after — and until then a refresh finding two keys should refuse rather than guess.

pf.dim_period

One row per month keyed on sdat, a GiST-indexed drange, and calendar and fiscal year/quarter/month columns in several spellings. Joined as LEFT JOIN pf.dim_period dpN ON dpN.drange @> (date + offset)::date — left, because an unshipped order has no ship date and must keep its row.

pf.dim_member

Master data per dim_group: one row per key value, siblings in attrs, with is_active and source_seen. The source is transactional, often a view over all history, so deriving members from it is slow (a full scan) and wrong (it describes only what has been transacted — it cannot say a part is discontinued, or that a new one exists before it sells). This is the app's own list, refreshed from the source by an admin action but curatable apart from it. A refresh merges: members gone from the source become source_seen = false, not deleted, so curation survives; a key with several attribute sets across history takes the most recent. It stamps an explicit run time rather than now(), which inside a transaction is the transaction start.

pf.version

Column	Meaning
name, description	(source_id, name) unique
status, closed_at, closed_by	a closed version refuses every write but annotation
exclude_iters	jsonb, default ["reference"] — pf_iter values scale and recode will not write
is_default	where a browser with no saved selection lands
adjustment_segment, adjustment_bucket	names a new adjustment is stamped with (§Naming)
unlabeled_load, bucket_order	dead — read by nothing

Creating a version inserts the row and creates its forecast table and pf_logid index in one transaction. Deleting drops the table; pf.log, pf.layout and pf.load_plan cascade.

pf.fc_{tname}_{version_id}

One table per version holding operational rows (pf_iter = baseline, scale, recode, clone) and reference rows (reference):

CREATE TABLE pf.fc_{tname}_{id} (
    pf_id          bigserial   PRIMARY KEY,
    -- every non-ignored col_meta column, typed from the source by mapType
    pf_iter        text        NOT NULL,
    pf_logid       bigint      NOT NULL,
    pf_user        text,
    pf_created_at  timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX fc_{tname}_{id}_logid_idx ON pf.fc_{tname}_{id} (pf_logid);

No version_id column: the table is the version. pf_logid is not a foreign key — undo deletes rows then log row in one transaction, and those are the only two paths that remove either. The index is how undo, the change-log recount and the grain key find an entry's rows; without it each is a sequential scan. Tables created before it was added lack it.

pf.log

Column	Meaning
version_id	ON DELETE CASCADE
pf_user, stamp	the session account; when
operation	baseline | reference | scale | recode | clone
slice, params	the slice(s) written to (null on loads); intent plus what the server resolved
label, bucket	segment name and what it counts toward; required by the log_named CHECK
tag, note	initiative name, which the bridge groups on; free commentary
row_count, value_total, units_total, measure_cols	what the entry wrote, stamped at write time
env, sql_text	what it ran against; the statement as executed
seq	dead

Who writes what. Load templates insert label, bucket and tag in their ilog CTE. Adjustment templates insert label and bucket; their tag goes on in a follow-up UPDATE in the same transaction (tagLog()), a hold-over from when a new {{tag}} token would have stranded sources that had not re-run Generate SQL. Totals, env and sql_text are written after commit by stampLogTotals().

Stamped totals are not a cache. An entry's rows never change after it is written — only its own operation inserts them and only undo removes them, taking this row too — so the totals are fixed. The change log used to join the whole forecast table on every open to recompute them. measure_cols records which columns they are in, since roles can be reassigned. Stamping is best-effort: a failure does not roll back a write that succeeded, since totals can be recomputed and the adjustment cannot. GET /versions/:id/log?recount=1 recomputes by join and writes back — the repair for rows deleted by hand, and the backfill for entries older than the columns.

params and sql_text are both kept on purpose. One is the intent (slices, scope, resolved increments), the other what it became. Keeping only one assumes the translation is correct, which is exactly what is in doubt when rows look wrong. env holds what cannot be reconstructed afterwards — the territory, the version's exclude_iters, and the template's generated_at, a fingerprint (not a version) showing whether an entry ran under a different template. GET /log/:logid/debug returns the whole row; the list omits sql_text and reports has_sql.

pf.sql

One generated template per (source_id, operation): get_data, get_agg (only with a grain), baseline, reference, scale, recode, clone, undo. generateSQL() in lib/sql_generator.js bakes in column names and leaves {{token}}s — fc_table, version_id, pf_user, note, tag, label, bucket, params, slice, filter_clause, date_offset, where_clause, exclude_clause, value_incr, units_incr, set_clause, scale_factor, territory_clause — for applyTokens() at request time. Generation deletes any operation it no longer produces, since a stale get_agg after clearing the grain would still be picked up. The stored undo is documentation; the route runs its own statements.

Values are escaped (esc() doubles quotes), not parameterised — consistent with the codebase, and the executed SQL reads as-is in the server log and pf.log.sql_text.

Editing the generator changes nothing until Generate SQL runs, since the templates are stored; a template carrying a token the running code does not substitute fails at the database, reading as an unrelated client error. Restart first, then Generate SQL, or the old code writes the templates.

Other tables

pf.layout (§The pivot), pf.load_plan (§Loads and segments), pf.app_user and pf.session (§Access).

---

Loads and segments

A version's rows start as segments: independent queries against the source, each appended as one log entry, each undoable alone. Two kinds, differing only in pf_iter:

• baseline — the rows operations adjust
• reference — kept out of scale and recode by exclude_iters

pf_iter answers can operations write these rows. Whether rows belong in the forecast number is the segment's bucket (§Naming): open orders load as reference so nothing adjusts them, and are still part of the forecast. Neither can be derived from the other. All load routes are admin-only.

How a load's WHERE is built

The Baseline page's filter builder takes groups of conditions — AND within, OR between — over date and filter columns (=, !=, IN, NOT IN, BETWEEN, IS NULL, IS NOT NULL). A period is a filter like any other. A toggle swaps the builder for a raw WHERE body.

The client builds the clause; the route injects it. buildFilterClause() in Baseline.jsx renders (a AND b) OR (c) and sends it as where_clause beside the structured filters (or raw_where). The route substitutes raw_where || where_clause as {{filter_clause}} verbatim and keeps filters / raw_where in params only so a segment can be re-opened and a plan derived. Nothing server-side validates columns or escapes values, and the client does not escape them either; lib/sql_generator.js's own buildFilterClause(), which does, is called by no route. The protection is requireAdmin: whoever can reach the route can already send raw SQL, so validating the structured path would not narrow what they can do. An empty clause becomes TRUE — the whole source — and the client refuses to send one.

Date offsets

date_offset is a Postgres interval (1 year, -90 days) applied to every date column, so order and ship dates stay in step, with period dimensions looked up against the shifted dates. Filter columns are never shifted: the offset moves where rows land, not which are chosen.

It must be whole days (assertInterval()): shifted dates are cast back to dates, so anything finer is rounded away, and Postgres reads a bare number as seconds — "30" used to be accepted and move nothing. 30 days is not a month either; use 1 month. Checked on baseline loads, segment edits, recode and clone — not on reference loads, where a bad offset surfaces as a Postgres error.

The offset is typed as an interval string. It replaced year/month spinners, which could not express days and clamped at zero, so a segment could only move forward in whole months. The timeline preview parses it only far enough to draw the source and shifted periods; Postgres is the authority.

Naming, editing, clearing

A load's label is required (400 without) — the segment name the pivot shows and reports group on — and its bucket defaults to it (loadNames()).

PUT /versions/:id/baseline/:logid edits a segment by deleting its rows and log row and re-running it in one transaction, under a new logid. Refused (409) once any adjustment exists, since those were sized against the old totals. Annotations carry over with ??, not ||: the form has no tag input, so an absent tag must survive, while an explicit '' clears.

DELETE /versions/:id/baseline removes every baseline row and entry (not reference). Not undoable.

Load plans — pf.load_plan

A segment used to exist only once run, so cloning a definition meant re-running every load, and there was nowhere to hold "this cut, but with the dates changed first". A plan is the definition alone — kind, label, bucket, tag, note, filters, raw_where, date_offset, seq — with run_logid naming the entry it produced, or null.

• Deliberately not pf.log: every row there means a write happened, and unexecuted intent would cost it that meaning.
• run_logid is ON DELETE SET NULL, so undoing a load returns its plan to unrun instead of deleting the definition. The list reads the entry's row_count through the FK to tell run from unrun.
• Running is two round trips — the ordinary load POST with the plan's fields, then PATCH /plans/:pid with the log_id. The alternative was a second copy of the load route's substitution; a plan that reads unrun while its rows exist is visible and fixable, a divergent copy of the load SQL is neither.
• Run all stops at the first failure. A definition's segments are not independent — references date-shift off a baseline — so carrying on past a broken one builds something nobody described.

Known issue: runPlan() does not send the plan's tag, so a tagged plan loads untagged.

Cloning a version's definition

POST /sources/:id/versions with clone_from copies the template's settings (description unless given, exclude_iters, adjustment names), its version-scoped layouts, and its segments as plans — and runs nothing. The point of cloning a definition is to change the dates before anything loads; a clone that fetched the baseline first would have to be undone to be edited.

Not copied: adjustments, in any form — a scale's target was sized against the rows that were there, so replaying it against another load is arithmetic nobody asked for; and source-wide layouts, which already apply and would duplicate by name.

cloneDefinition() reads the template's plans, or derives them from its baseline/reference entries when it has none — every version built before plans existed. params already stores filters, raw_where and date_offset, so this is a read, not a reconstruction.

Replay

Re-running a version's adjustments against a restated baseline is not built. Nothing accepts a replay flag and nothing answers 501, whatever earlier specs said. See §Open questions.

---

Naming: segments, buckets, notes

Four pivot columns come from pf.log, joined on pf_logid, not from the forecast table:

Column	From
pf_segment	l.label — which load, or which kind of adjustment
pf_bucket	l.bucket — what it counts toward: Actual, Plan, Forecast, Prior Year
pf_note	adjustments: the tag, else the note; loads: null
pf_op	l.operation — /agg and pushed-back rows only

Separate on purpose: commingling a segment name with an adjustment's note makes neither pivotable. In grain mode pf_logid is in the grain, so the join adds no rows. SEGMENT_EXPR / BUCKET_EXPR / NOTE_EXPR in lib/sql_generator.js are the single definition, shared by the generated /agg and the hand-written /data cursor, which must agree. The operation routes stamp the same fields on the rows they return, since RETURNING * lacks them. The /data cursor does not emit pf_op, so a raw-mode pivot cannot group on it.

Every log entry is named

label and bucket are stored, not derived, and every entry has both — log_named refuses one without. A report gets the pivot's segment and bucket by joining pf.log and selecting two columns, with no rule of its own. Stamped at write time:

• A load — label required, bucket defaulting to it.
• An adjustment — no name inputs, so the version's adjustment_segment / adjustment_bucket, else ADJUSTMENT_SEGMENT = '99 - Adjustments' and ADJUSTMENT_BUCKET = '04 - Forecast'. adjustmentNames() resolves them once per request, for the INSERT and the returned rows alike.

The version defaults are set on the Baseline page (Defaults for new adjustments) and name the next adjustment; existing entries keep theirs. Renaming an entry is PATCH /log/:logid — from the Baseline page's segment list — which renames but refuses to clear, owner or admin only because the names are the pivot's columns for everyone. Relabelling needs a page reload to show, since the label is part of the aggregated row. A source registered before names were stored needs Generate SQL once so its templates write them.

What this replaced. The names were derived at read time: a load's label, else tag, else note, else the version's unlabeled_load, else Unlabeled; an adjustment's label, else the version default, else the constant — through a join on pf.version. That made the pivot the only reader that knew a row's bucket: the CASE was copied into /agg, the /data cursor and a reporting procedure in /opt/forecast, and every direct query had to copy it again. And editing a version's fallback silently renamed every past adjustment. The backfill in 01_schema.sql stamped existing entries with that rule, so nothing on screen changed name.

Column order is stored text

Perspective orders column groups by the value string, and asc/desc only reverses that, so Prior Year → Plan → Actual → Forecast is expressible in neither direction. A "01 - " prefix is the only lever, and it lives in the stored label and bucket, typed by whoever names the segment. Nothing derives it.

The one hardcoded ordinal is 99 - Adjustments, keeping unnamed adjustments after every numbered segment. That proviso is the scheme: ordering is string ordering, so 99 lands last only while loads carry 01–0n, and an unnumbered segment sorts after it (digits precede letters). The old '(adjustment)' sorted first for the same reason read the other way. Relabelling an adjustment's log row is how one kind of adjustment is split out from the rest.

What this replaced. The prefix was computed client-side as Perspective expression columns (pf_bucket_ord, pf_segment_ord) from pf.log.seq and pf.version.bucket_order. It ordered the pivot and nothing else, so every other reader disagreed; restore() replaces expressions wholesale, so it had to be re-applied after every layout load; and ExprTK's scanner tests each byte with isprint(), so a label with anything outside printable ASCII could not be ordered at all. DEAD_ORDER_EXPRS in Forecast.jsx strips those names from old layouts.

Tags

A tag names an initiative — a line in the ledger and the bridge. It never affects rows, so it stays editable, even on a closed version. It is not a row anywhere, only a value repeated across a version's entries; it exists because somebody typed it.

Completions are per version. GET /versions/:id/tags returns this version's tags with uses, last_used, owners, can_edit, refetched on version change and after every tagged write. It used to be GET /sources/:id/tags, across every version of the source — which put last year's initiatives in front of this year's and reported use counts earned elsewhere. The cost is that a fresh version completes from nothing until it has a tag, which is honest: cloning a definition copies no adjustments.

Managed on the Baseline page. PATCH /versions/:id/tags { from, to } rewrites every entry carrying from. A to that already exists merges — usually the point, since "Pull Forward" and "Pulled Forward" are one initiative reported as two. An empty to clears. Both are confirmed first, being invisible as a diff afterwards. Permission is ownership of every entry the tag is on — a half-applied rename leaves the initiative split, the state being fixed — so rows the reader cannot rename are greyed; the route locks the rows before checking, so check and rewrite see the same set. The Tags panel sits outside the page's read-only fieldset: a rep may rename what they named.

Completion matches on words. TagInput in OperationPanel.jsx lowercases, strips punctuation and stopwords and lightly stems both sides, so "Pulled Forward" finds "Pull Forward" and one inserted word does not hide an existing tag. Something that reduces to an existing tag without being it says so ("Already used as … — use that"). Nothing merges automatically. The change log's inline tag cell still uses a plain datalist.

The panel used to carry six quick-pick pills under the tag field; they wrapped to two rows once tags were longer than a word and pushed the docked form past its border. The list shows the tag alone — an option's text renders as a second column, so a use count widened every row for a number nobody picks on.

---

Selection

A selection is a list of slices ({column: value}) plus the pivot's own filter as scope. The server turns them into a WHERE clause; the ledger totals the loaded table through the same filters, so the panel and the write agree.

From a click to a slice

perspective-click's own filter list cannot be trusted: the datagrid's getCellConfig builds it with pivot_value ? [pivot, '==', pivot_value] : undefined, so the engine drops a blank group key — and, being a falsy test, a 0. A dropped key widens the write: clicking the blank node under a customer selected the whole customer, and the ledger agreed, totalling through the same list. So cellFilters() in Forecast.jsx re-derives both axes:

• Rows from detail.row.__ROW_PATH__ — raw values, so dates stay epoch millis. A subtotal's shorter path leaves deeper levels absent.
• Columns from the clicked column's name (taken from the mousedown's cell; the event names only the measure). The name is the visible split values joined with |, measure last, so the measure is popped before mapping split_by positionally. Mapping every segment used to read the measure as a value of the first collapsed dimension — a slice matching no row.

A plain click replaces the selection; ctrl/⌘/shift toggles a slice. The event carries no modifier flags, so they are read off the preceding mousedown. buildEffectiveSlice() keeps dimension, date and pf_ keys, turns dates into ISO strings, and drops the rest; duplicates are collapsed, since under each a repeat would apply twice. A Perspective computed column does not map back to stored rows, so only native columns slice.

Dragged regions

The datagrid runs in edit_mode: SELECT_REGION, forced on every restore so a layout cannot switch it off. A drag fires perspective-select with a ViewWindow — { start_row, end_row, start_col, end_col } — on every mouseover, so the handler keeps the latest and a window mouseup commits it, each cell going through cellFilters(). A single-cell region is ignored: perspective-click owns clicks, and handling both would undo a ctrl-click.

Blanks: null and '' are different values

Both occur in real data as different rows, so they stay distinct from click to WHERE; only undefined means absent.

• buildWhere() renders null as IS NULL. It used to fall through esc() to = '', selecting the other blank — the slice named one node and the operation wrote the other. In a list the null arm is ORed on, since IN (NULL) is never true.
• The ledger filters null with is null, not == null: the engine matches on type, and a null threshold on a string column is another dropped predicate.
• On the column axis, t_tscalar::to_string renders '' as '' and a null as the text "null", so "null" is read back as null. This was believed impossible — a null was thought to join as '' too — so '' cells were refused and "null" was taken literally, a slice matching nothing. The one ambiguity left is a value that really is the text null.
• The panel prints (null) / (blank), and — only for a column not in the slice: isolating the blank node and ignoring the column are opposite statements and used to look identical.

The pivot's pf_ columns are part of the slice

• Table columns — pf_iter, pf_user, pf_logid (TABLE_SLICE_COLS) — compared directly.
• Log columns — pf_segment, pf_bucket, pf_note, pf_op (COMPUTED_SLICE_COLS) — resolved to pf_logid IN (SELECT id FROM pf.log WHERE version_id = … AND <expr> …), exact because the name lives on the log row.
• Any other pf_ key is refused — skipping is how selections widened.

They narrow a slice but do not make it selective (pf_iter = scale alone is every adjustment): assertSelective() needs a dimension, date, pf_segment or pf_bucket key.

What this replaced. pf_iter was stripped on the theory that cells differing only by iter band are one slice; pf_note and pf_op were refused, and the client pre-empted that by stripping; pf_segment and pf_bucket were dropped before that. The ledger stripped none of them. So under the default layout, grouped by pf_iter, a click on a customer's scale row totalled the scale rows and wrote to all the customer's rows; a click on one bucket's cell scaled every bucket there.

Several slices, and total cells

Several slices become an OR of AND-groups, not per-column IN lists — {A:1,B:1} + {A:2,B:2} as A IN (1,2) AND B IN (1,2) would also match A:1,B:2 — parenthesised, since callers append AND pf_iter NOT IN (…).

A total cell is its displayed amount. The grand-total row names no row dimension, so its slice is whatever the column axis and the pivot filter leave — possibly {}, everything the filter shows. {} is accepted only with a non-empty scope (normalizeSlices(), assertSelective()); alone it would be the whole version and is refused on both sides. buildWhereAny() reads a union containing {} as TRUE, the scope then ANDed on — dropping it would narrow the write to the other cells. The panel names it "Total".

Scope: the pivot's filter travels with the slices

A slice can only mean equality, so a bounded season, an in-list or a substring has no slice form. The filter is read off viewer.save() — in the table's own types, with the user's operators — and sent as scope, [col, op, value] triples; the ledger totals against the same list. Without it the ledger counted rows the grid was hiding and the operation wrote them. It is ANDed onto each statement rather than folded into the slices, which under each would repeat it per statement.

SCOPE_OPS in lib/sql_generator.js is the whole allowed list, under Perspective's names: ==, !=, >, >=, <, <=, in, not in, is null, is not null, and the substring family. Anything else is refused — a scope silently dropped is a write wider than the panel that authorised it. pf_ log columns take == or in only.

• Substring operators map to ILIKE, because the engine's t_tscalar::contains / begins_with / ends_with lowercase both sides. LIKE would write a fraction of what the grid showed, and narrower is no safer than wider — neither is the set the ledger counted. The engine matches the term literally, so %, _ and \ are escaped. Residual difference: the engine folds ASCII only, ILIKE follows the collation.
• matches / not matches stay refused. RE2 in the engine, not in Postgres — \b is a word boundary in one and a backspace in the other — so a pattern valid in both could match different rows, which is worse than a 400.
• An empty term is refused. An unfinished filter row arrives as [col, '==', null], and ILIKE '%%' or = '' would read as every row or none rather than as the mistake it is.

The ledger used to drop a slice's column whenever the scope named it, assuming they could not disagree; under a substring filter, clicking one part then totalled every matching part. It now drops only an exact == repeat. Territory (§Access) is ANDed on after the scope, from the session.

Selection highlight

The datagrid highlights model._selection_state.selected_areas and wipes it on every mousedown, so a ctrl-click selection would show only its last cell. areasRef, a sliceKey → rectangles map beside slices, is pushed back after every change and pruned by live slice key on any deselect, so grid and panel cannot disagree.

Rectangles are re-derived when the grid moves. A sort reorders rows and a new segment column shifts every column; tracked coordinates would then highlight whoever took the position while slices, being value-based, still named the right row — the grid would be the only thing lying. relocateAreas() sweeps the displayed rows as a drag would and keeps cells whose slice is still selected. A slice can therefore light more cells than were clicked (a slice not naming the segment matches the customer under old and new segments alike), which is what it covers. Past MAX_RELOCATE_ROWS the highlight is cleared rather than left wrong.

---

Operations

POST /api/versions/:id/{scale|recode|clone}, common body:

{ "slices": [ { "channel": "WHS" } ], "scope": [ ["season", "<=", 2027] ],
  "apply_mode": "prorate", "tag": "volume_push", "note": "optional" }

slices — one or more (a legacy single slice is accepted). apply_mode — prorate (default) is one statement over the union and one entry; each is one per slice, so each reaches its target alone and undoes separately (for recode and clone it only decides how many entries). pf_user is the session's. A closed version answers 403. Operations are not admin-gated; territory and ownership govern them. The response is the new rows — whole in raw mode, aggregated in grain mode — for table.update(); no reload.

SQL shape

An ilog CTE inserts the log row and the insert CTEs reference its id; {{where_clause}} is the selection and {{exclude_clause}} keeps exclude_iters out.

Every date column travels — the written columns are every dimension, every date and the first value and units column. Scale and recode used to carry only the first date, so on a source with order, requested and ship dates they wrote the others null. The period dimensions were fine, being dimensions — which hid it: labels right, dates gone. Those dates are sliceable, so an adjustment scoped to a ship date matched no adjustment ever written. Clone and loads were always right. Rows written before the fix keep the nulls; deleting and replaying those entries is the honest repair, since re-deriving them would be inference.

One row per coordinate, not per row read. Reads GROUP BY every dimension and date. One-for-one, every operation inherited the row count of everything before it at that coordinate, and the table grew super-linearly with the work done. Only pf_logid and pf_iter collapse, so nothing goes null or unsliceable; the maths is unchanged, sum(sum(x)) OVER () being the aggregate of aggregates.

Scale

Distributes an increment in proportion to each coordinate's current amount:

round(("value" / NULLIF(total_value, 0)) * {{value_incr}}, 2)
round(("units" / NULLIF(total_units, 0)) * {{units_incr}}, 5)

resolveIncrs() in routes/operations.js computes the two increments per statement, which is what makes each land per slice. Value and units each take at most one of a target (target_value/target_units), a percentage (value_pct/units_pct) or a change (value_incr/units_incr), resolved independently; target_price is a third input; legacy pct: true is honoured.

target_basis — what a target or percentage measures against:

• adjustable — only the rows the operation can write. The server default.
• selected — everything the pivot shows, exclude_iters rows included. Those cannot move, so the adjustable rows absorb the difference and the pivot lands on the number asked for; otherwise a target set against a visible total overshoots by the fixed rows.

The panel defaults to selected and sends it only when the selection holds excluded rows (otherwise they are identical). The basis row is tinted and tagged "target", and clicking either row switches.

Price does not move unless it is named

A price move is invisible in the figure being edited — the dollars land either way, and only the implied price says whether volume followed. So:

plug answers the one ambiguous case, dollars alone: volume (default, price holds, units scale in proportion) or price (volume holds). From the Excel form:

plug volume:  pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price:   fVol = pVol + bVol

A qty input follows the same rule: with dollars unnamed they follow at the current price, value = (curValue / curUnits) * unitsIncr. Holding them made every qty edit a price edit — more cases at the same revenue is a price cut nobody asked for. A slice with no units has no price, so its dollars stay put.

The price held is the adjustable rows' own, never blended with the fixed rows. The SQL prorates over the adjustable rows only, so an increment sized at a blended price moved qty by a different factor than dollars wherever reference rows priced differently — and taking a selection to its fixed total drove qty past zero, grains with positive sales and negative qty. The ledger preview uses the same price.

Naming the price (target_price, the Excel form's Edit Price) holds it, and the other named measure decides the rest, on the chosen basis:

Named	Result
price alone	qty holds; dollars = price × qty
price + qty	qty lands as named; dollars = price × that qty
price + dollars	dollars land as named; qty = dollars / price
all three	refused — two degrees of freedom, three numbers

This branch used to read only a qty target (a qty change sized dollars on the old qty), took the fixed qty off a target but not the fixed dollars, and let a dollar figure outrank the price. Worse, the panel sends no plug with a price, so the server read volume and scaled qty by the dollar change too, putting price back where it started. A named price now bypasses the plug.

So the deliberate ways to move price are dollars and qty together and naming the price; both outrank the plug.

What this replaced. plug defaulted to price, being what the API did before the flag existed — so an unqualified dollar edit and every qty edit moved price, and only a hand-picked volume did not. The client's preference is keyed pf_scale_plug_v2 so a browser carrying the old price choice does not go on silently moving price.

The ledger and its three boxes

The scale form is one continuous statement: baseline, adjustments grouped by tag (untagged ones collapsed), fixed segments, total, then the edit, with a running total closing on the selected total. Value, qty and price each have New / Change / % change boxes; typing in one derives the others, and the box typed in is what is sent — the row you type in is the mode. A Result line previews all three under the server's rules and marks the column that cannot move · holds.

The boxes foot — price = value / qty — so at most two are inputs. effectiveScaleInputs() in OperationPanel.jsx decides which when all three are typed: value typed last lets the plug decide (volume drops qty, price drops price), since a dollar change says nothing about price or volume; otherwise the more recently touched of the other two stays. The dropped box keeps its text, so flipping the plug brings it back. buildPayload() in Forecast.jsx uses the same function, so what is posted is what is shown. Clicking into a derived figure pins it — "units are this" rather than "units may move" — and unpin releases it. Rough edge: all three typed, dollars last, the choice is really which earlier number to keep, and "Volume" there moves price too.

The Moves control is the plug. It lights whatever the edit moves — both if both do — and takes a click only while it is the one deciding. It used to render only for a dollars-only edit, so it was absent before anything was typed and vanished when a measure was pinned, even by an accidental click — two states that both looked like a missing control. It now stays, greyed when the inputs have answered it, showing their answer.

A price change or percentage is resolved client-side to an absolute target_price; the server has no price increment form.

What the log records

params carries the intent (pickIntent(): inputs, target_basis, scope, plug) plus:

• resolved — value_incr, units_incr and method: price (named), both (dollars and qty named), volume (qty named), else the plug. The request alone cannot say which moved — the plug is a per-browser preference — so a price edit used to be findable only by noticing qty had not moved.
• expected — the panel's Result line as a change on its basis (under each, combined and flagged so), beside resolved so a panel/server disagreement shows in the log. Nothing compares them yet.

Refusals

• A pool netting to ~zero (below 1% of gross): shares are row / total, so as the net nears zero rows fly to extreme opposite values. Offsetting slices are the usual cause; each handles them.
• Nothing to scale — every unit already on target or empty; individually skipped slices come back in slices_skipped.
• Holding price on a selection with no value — no price to hold.

Rounding each row to 2dp can leave a target a cent off; landing exactly would need a correction row.

Recode

Moves the selection to other dimension values in one entry: negative rows zero it out where it was, positive rows put it back with {{set_clause}} replacing the named dimensions.

A date shift moves the put-back leg only, so a shifted recode is a timing move — pull forward, push out — and set may be empty. Moved rows' period dimensions are re-derived against the shifted date, as clone's are; an explicit set wins; a row with a null date keeps its label, since a lookup on nothing would blank it. Without a shift nothing is derived.

Coordinates already netting to zero are skipped (HAVING sum(x) <> 0), rather than writing two empty rows each — which a second recode over cells a first one emptied would do for all of them. If nothing is left the request is refused ("Nothing to recode") rather than committing an empty entry. A scoped account cannot set the territory column (§Access).

Clone

Copies the selection as new business: set overrides dimensions, scale multiplies (default 1), date_offset shifts every date; the original is untouched. Period dimensions come from the calendar against the shifted date — else a mix moved forward a year keeps last year's labels — and set wins. Unlike recode, a null date blanks them.

exclude_iters does not apply to clone. It stops operations modifying reference rows — scale would attribute movement to prior-year rows, recode would write negatives against them. Clone does neither; copying a plan or prior year out of reference is the operation working as intended, and excluding them made a deliberate selection silently produce nothing. from_logid narrows instead, for a selection spanning segments where only one is wanted.

Known issue: a clone matching nothing commits an empty entry with rows_affected: 0, where scale and recode refuse.

Undo

DELETE /log/:logid deletes the entry's rows then its log row in one transaction and returns pf_gkeys (grain mode) or pf_ids (raw) for table.remove(); the view re-sums. Undo is by ownership, not territory — your own entries or an admin's override — because it removes rows wholesale, and half-undoing one would leave a state nothing describes (and two accounts can share a territory). Closed versions refuse. Undoing a load returns its plan to unrun.

Known issues: the pf_gkeys come from live col_meta, so after a grain change they match nothing in the loaded table and the pivot keeps the rows (the database is still right); recode and clone log apply_mode: unit.mode, never set, so their params omit it.

---

Display-grain pre-aggregation

Built, with a static grain. Path B of pf_perspective_options.md — a pre-aggregated extract in a native Perspective table — rather than Path A, live aggregation through a DuckDB virtual server (parked on spike/duckdb-virtual-server), which loses expand, collapse, depth, sort and filter.

The problem. Shipping raw rows makes the browser pivot all of them and load time grows with raw rows. Aggregating server-side to the grain the pivot displays cuts the row count by orders of magnitude while Perspective still does all the rollup locally.

Grain definition — pf.col_meta.in_grain

The flagged dimension/date columns, plus always pf_iter and pf_logid. grainOf() in lib/sql_generator.js is the single definition (Setup.jsx and routes/log.js mirror it), baked into the templates at Generate SQL so load and operations agree by construction.

• Additive measures only. Sums are exact; an average would ship as sum + count; count(DISTINCT …) cannot be pre-aggregated. Forecast measures are sums.
• An unflagged dimension cannot be pivoted or filtered on. Leaving high-cardinality columns (part numbers, raw dates) out is what keeps the grain from exploding back toward raw; the price is that they are unavailable. Dates usually enter through their derived period dimensions.
• Generation refuses a grain column the table never populates, or a grain with no measure.

in_grain means eligible; today the grain is exactly the eligible set.

The model: append deltas, let the view sum

The Excel pivot cache again: pre-aggregated rows go to the table, the pivot cuts them locally, and each adjustment is appended, never a recomputed total. Rows are aggregated per pf_logid — one per grain tuple × pf_iter × entry — and the view sums them. It was also the smallest change from raw mode, which already appended results with table.update().

Synthetic key. Perspective indexes on one column, so each row carries

md5(concat_ws(chr(31),
    COALESCE(col1::text, chr(30)), …, pf_iter, pf_logid::text)) AS pf_gkey

pf_logid keeps each entry's contribution distinct, so an append accumulates rather than replacing a bucket and undo removes exactly that entry. Plain concat_ws skips NULLs, so ('a', NULL) and (NULL, 'a') would collide and silently merge two groups; chr(30) also keeps NULL apart from ''. The md5 because the key is an opaque handle read only by update()/remove(): raw, it was long and, unique per row, defeated Arrow's dictionary encoding — the largest column in the payload. 128 bits keeps collisions unreachable.

Initial load — GET /api/versions/:id/agg

SELECT
     t.<grain cols>, t.pf_iter, t.pf_logid, <pf_gkey>
    ,SUM(t.<value>) AS <value>, SUM(t.<units>) AS <units>
    ,l.label AS pf_segment, l.bucket AS pf_bucket
    ,<NOTE_EXPR> AS pf_note, l.operation AS pf_op
FROM {{fc_table}} t
LEFT JOIN pf.log l
    ON l.id = t.pf_logid
WHERE {{territory_clause}}
GROUP BY <grain cols>, pf_iter, pf_logid, l.operation, l.label, l.tag, l.note, l.bucket

The pf.log columns depend on pf_logid, but Postgres will not infer it, so they are grouped (LABEL_GROUP_COLS). Territory applies before the GROUP BY, since its column need not be in the grain. The route sets LOCAL enable_sort and enable_incremental_sort off: seeing pf_logid pre-sorted off its index, the planner picks a sort-based GroupAggregate for the "free" sort, but comparing dozens of keys per row makes it several times slower than a HashAggregate, whatever work_mem or statistics say. The client loads the result with worker.table(buffer, { index: 'pf_gkey' }).

Write and delete paths

Operations insert raw rows under a new logid as always; in grain mode the final select aggregates that entry's rows to grain (FROM ins, or recode's neg UNION ALL ins) — safe because they all carry the new logid. The keys are therefore always new, update() appends, and nothing is recomputed.

Undo removes an entry's keys and lets the view re-sum — no re-aggregation, no emptied-bucket handling. RETURNING does not take DISTINCT, so:

WITH del AS (
    DELETE FROM {{fc_table}} WHERE pf_logid = $1
    RETURNING <grain cols>, pf_iter, pf_logid
)
SELECT count(*)::int AS rows_deleted, array_agg(DISTINCT <pf_gkey>) AS pf_gkeys
FROM del;

Raw mode

With nothing flagged, no get_agg exists and the client uses /data: every row plus pf_segment, pf_bucket and pf_note, indexed on pf_id; operations return whole rows and undo returns pf_ids.

Transport

Both endpoints return Arrow IPC in one record batch with X-Row-Count. /data reads a cursor in batches into column arrays (row objects multiplied peak heap) and builds one table with tableFromArrays; /agg does the same from its result. One batch because per-batch encoding creates independent dictionaries, and Perspective's WASM reader crashes on the replacement messages. An earlier plan to stream batches into update() behind a large-dataset banner was never built, for that reason.

Load time is dominated by row count, not payload size. Even a good grain on a wide source leaves enough rows that getting them out of Postgres and into JS dwarfs the aggregation and the Arrow build; halving the payload (the md5 key) barely moved it. The remaining lever is the dynamic grain.

The dynamic variant

The static grain means an unflagged dimension is never pivotable and every flagged one is paid for even when the pivot uses none of them. The additive next step: group by the fields the current pivot uses — the viewer's group_by, split_by and filter intersected with the eligible set — and re-fetch when they change. That needs the grain passed per request rather than baked into pf.sql, with operation responses and undo keys computed at the loaded table's grain.

Grain drift. Changing in_grain after a load needs Generate SQL and a page reload, since the loaded table's index and columns are fixed; until then undo's keys, from live col_meta, match nothing.

---

The pivot

Perspective, vendored

Perspective 5.4.0, @perspective-dev/* — not FINOS @finos/perspective — built from a patched fork and committed as tarballs in ui/vendor, whose README covers the patches (column-axis expand/collapse, split_by_depth, depth applied on config update), provenance and rebuild. Bundled through the /inline entrypoints, never a CDN: the 4.x CDN bundle resolved its server WASM to an unversioned path and silently pulled whatever was newest.

Layouts — pf.layout

A named ViewConfig owned by an account. Published — everyone on the forecast lists and can apply it, only owner or admin may change it — or private. Scope is the version by default, where people enter; version_id IS NULL applies to every version of the source. is_default (one per scope, published only) is restored on a first load, version-scoped beating source-wide — the narrower answer is the one somebody chose for this forecast. With no default: group_by: ['pf_iter'] over the first value column.

Load order: this version's last-used unnamed config in localStorage (pf_layout_v{vid}) if any — per-browser continuity, not something anyone names or shares — else the default. The old named list (pf_layouts_v{vid}) is lifted into pf.layout as private rows once and cleared.

Permissions are the pf.log rule and can_edit rides on every row so the Layout ▾ menu greys the rest. Applying is never restricted: the guarantee is that a published layout cannot be changed under people, not that it cannot be adapted — Save is withheld on others' layouts, Save as… forks. No territory clause: layouts are display config and territory restricts rows.

cleanLayout() gates every stored config before restore(), which is all-or-nothing: it drops unknown columns from the axes and aggregates (including a multi-arg weight column) entry by entry rather than dropping the layout, and strips theme, the dead ordering expressions and pf_totals.

The dirty dot is a comparison, not a flag. restore() itself fires perspective-config-update, so a flag set there would light the moment a layout was applied. sameConfig() compares against activeConfigRef, ignoring table (per-load) and theme.

Grand totals ride in the layout as pf_totals: { row, col } — the Total toggles. The engine has no "subtotals but no grand total" mode, so both are hidden in the grid's DOM; captureConfig() adds the key and cleanLayout() strips it. Without it: row shown, column hidden.

What this replaced. Named layouts lived only in localStorage, invisible to others and gone on another machine. The one server-side layout was pf.source.default_layout, an anonymous blob PUT /sources/:id/default-layout let any account overwrite for every account — a published layout with no owner. The route is gone; the column is migrated and unread.

Axis depth

Both axes collapse by a depth in ViewConfig set through restore() — group_by_depth (applyDepth()) and split_by_depth (applySplitDepth()). In the config it survives every rebuild, rides into layouts via viewer.save(), and needs nothing re-applied.

Each axis is a stepper from DepthButtons.jsx — ◂ level ▸, the middle opening the level list for a jump — three controls whatever the depth. It was one button per level, growing with the pivot, the current one a blue among greys so reading it meant scanning; before that a fixed 0 1 2 3 that named nothing and offered levels a two-field group_by lacks. It shows an em dash until a layout has been read back, and its middle is sized to the longest level so the toolbar holds still.

Depths are 1-based counts of levels to show, on the stepper and in the config, so the value crosses unchanged; server.cpp sets set_depth(n - 1) and t_traversal::set_depth adds 1 back. The toolbar used to send d + 1 and subtract it on read-back — self-consistent, and one level off the grid everywhere. The +1 dodged depth 0, which reaches the engine as set_depth(-1) on an unsigned t_depth, underflows and clamps to fully expanded. applyDepth now floors at 1, so the Total slot still shows the first level, not a bare total — a wart kept rather than traded for the underflow. A real total needs group_rollup_mode: 'total', unverified on this engine.

This required patching the engine. ViewConfig::apply_update applied ten fields and neither depth, so a depth through restore() was accepted and dropped — and Session::update_view_config returns early when nothing changed, so no view was rebuilt. ui/vendor/0001-apply-depth-fields-on-config-update.patch.

What this replaced:

• Row depth was imperative — view.set_depth() on an object the viewer discards whenever it re-renders — so restoring it meant guessing when that happened: an observerShim.js patching IntersectionObserver and ResizeObserver, focus and visibility listeners, a retry loop for getView() throwing No table set, a flag for whether the viewer had "gone away". That produced three visible faults: the tree fully expanding on refocus, snapping on any reflow, and snapping when the settings sidebar opened.
• Column collapse restored a truncated split_by, so discarded levels were remembered separately (splitFull, persisted as split_full), our collapse had to be told apart from the user rearranging, and the selection was cleared on every collapse. split_full and expand_depth are still read from old layouts.

Depth changes rebuild the view — restore() makes the session tear down and rebuild it, a full traversal, where a manual collapse mutates in place. Noticeable on a large grain. Detecting a depth-only change and calling view.set_depth() while still writing the config would fix it, at the risk of the two drifting.

Per-node expansion is not solved. It is view state with no config form, and the API is set-only (expand(row_index) / collapse(row_index), no getter), so it cannot be captured and replayed and is lost on every rebuild. The honest route is a ViewConfig field of expanded row paths (indices shift as the tree opens), applied in server.cpp beside the depths — needing node enumeration in the C++ traversal, a proto field and the apply step. expand_column / collapse_column share the limitation. And depth is whole-axis: Excel can collapse one year while the next stays open.

Auto-pause is off

<perspective-viewer> auto-pauses by default — an IntersectionObserver plus visibilitychange — and pausing deletes the view, so returning to the tab means restore_and_render(): a fresh traversal of the whole grain, a multi-second chug per tab switch on a large one, losing per-node expansion. initViewer() calls viewer.setAutoPause(false) after load(). Nothing updates the table while hidden — every operation is driven from this page — so the pause bought nothing.

Toolbar

Two zones. Left, view: Layout ▾ and the Rows and Columns steppers with their Total chips — all ViewConfig, nothing reaching the server. Right, forecast: Refresh data, Change log, Bridge, Operations — re-running the load, opening something over the pivot, or writing.

The distance between them is the separation. Each was once an outlined box too, the view one tinted too faintly to see and, in dark mode, reading as a sibling; before that everything was one row of identical hairlines, making a column-width control and the version's history look like peers. Three weights do the separating: .pf-section-label names a group and is heavier than its controls, .pf-group-rule divides groups within a zone, the gap divides zones. Each axis is one joined control, ◂ | level | ▸, then its Total chip.

Blue is for actions, pressed grey for state. Save and Post are blue; the Total chips and an open Operations use .pf-pressed. Operations used to relabel itself "Hide panel" in a blue outline, looking like a second Save.

Fit widths is gone, for three independent reasons: auto-size measures over the rows in the viewport, so anything below the fold never counted; draw({invalid_columns}) passes an option DrawOptions lacks, and draw() is throttled, so a click during a pending draw cleared widths without re-measuring; and the persistLayout after it saved nothing, since saveColumnSizes() returns only overrides, which resetAutoSize() had emptied — and this fork discards column_size_override whenever split_by is set, which here is always. Recoverable from git if ever worth fixing.

The operation panel docks bottom, right or floats, persisted locally; the toolbar toggle shows the selection count while it is shut.

Calculating indicator

Calculating… shows in the status bar while the pivot rebuilds. Perspective's beaker renders only while its settings panel is open, vanishing exactly when nothing else says to wait. A counter — rebuilds overlap — rises in applyDepth, applySplitDepth and applyLayout, falls after viewer.flush(), and travels as pf-pivot-busy on the window, like pf-data-changed.

Not driven by perspective-config-update, the obvious hook: on a depth change over a large grain it arrives after the rebuild, seconds late, and the flush after it resolves at once. The cost is that sorting or filtering from the viewer's own chrome shows nothing, since the app is never told they began.

Light and dark

ui/src/theme.jsx holds a context stored as pf_dark, defaulting to prefers-color-scheme; the status-bar toggle sets .dark on <html>. index.css defines properties under :root and .dark, with Tailwind overrides written as .dark .bg-white { … } — no Tailwind dark config. Dark uses Perspective's Pro Dark palette.

The pivot takes a partial viewer.restore({ theme: 'Pro Dark' | 'Pro Light' }) on load and on toggle. It used to be setAttribute('theme', …), which does nothing in 5.4.0 — the theme is a ViewerConfig field — so the pivot stayed light whatever the toggle said. The other half is cleanLayout() dropping theme: viewer.save() includes it, so every layout reasserted its author's theme. It is the reader's choice, so a published layout cannot impose one.

---

Access

Authentication and sessions

Everything under /api but /login, /logout, /me sits behind requireAuth; the React app mounts only with a session (Gate in main.jsx), since its effects call the API at once, and auth.jsx wraps fetch so a 401 returns to login.

• Accounts — pf.app_user, scrypt hashes whose stored form carries its parameters, so they can be raised without invalidating rows. ./pf.sh add-user | passwd | list-users | disable-user | enable-user; passwords go in on stdin through node lib/auth.js hash, never argv or psql.
• Sessions — connect-pg-simple in pf.session, so restarts keep people signed in and a session is revoked by deleting its row (disable-user does). Cookie pf.sid: httpOnly, SameSite=Lax, Secure unless COOKIE_SECURE=false, 12h rolling.
• Config — the server exits without SESSION_SECRET rather than using a guessable default. TRUST_PROXY (default 1) keeps req.ip and secure cookies right behind the TLS proxy. CORS is enabled only by CORS_ORIGIN, by name — a wildcard origin plus a session cookie is CSRF by construction.
• Login hardening — ten failures per IP per fifteen minutes (in memory), one message for unknown/wrong/disabled, session id regenerated on success.

Identity is server-side. pf_user, created_by, closed_by come from sessionUser(req), never the body. The UI used to send a hardcoded pf_user: 'admin', which any client could set to anything.

Setup and Baseline are admin-only to write

requireAdmin sat exported in lib/auth.js and applied to nothing, so every non-admin could call every Setup and Baseline endpoint. It now guards source writes (register, cols, Generate SQL, member refresh, deregister), version writes (create, update, close, reopen, delete), loads (baseline, reference, segment edit, clear) and every plan write. is_default is further admin-checked inside PUT /versions/:id.

Not gated, each for a reason. No read, including the table browser — the pages stay visible, and gating what a page needs to render turns it into a 403. Not the forecast operations, which carry territory and ownership rules. Not PATCH /versions/:id/tags, whose permission is ownership.

The pages go read-only rather than hidden: one <fieldset disabled> per page instead of a flag threaded through every control — Baseline alone has dozens, and the one missed would be the one that mattered — plus a ReadOnlyNotice banner. Setup wraps a real box because the fieldset is its flex row; Baseline uses display: contents so its layout is untouched. None of this is the permission; it only stops offering buttons that would answer 403.

A 403 used to blank the whole app. Setup put responses straight into state and .mapped them; an error body is {error}, and throwing in render unmounts React. Its list fetches now coerce to arrays, so a bad response costs an empty panel, not the session. The fragility predated the gating, which only made it reachable.

Territory scoping

An account sees and changes only its territory: pf.app_user.territory (jsonb list), is_admin for the unrestricted, and col_meta.is_territory naming which column of a source the values belong to — flagged, not coded, so another source can be divided by something other than a sales rep.

Fail closed. No territory and not admin means no rows: buildTerritoryClause() returns FALSE, not TRUE, for an empty list or an unflagged source, so a forgotten account sees nothing rather than the book.

Built from the session, never the request — the difference from scope, which the browser rightly sends as a filter the user chose. A permission cannot come from the thing it restrains, so the predicate is ANDed on last, in sliceUnits() for writes and per route for reads. Enforced at:

• /data — cursor and the X-Row-Count count
• /agg — {{territory_clause}}, before the GROUP BY
• /versions/:id/table-info — unscoped, it told an account its load was fetching the whole table
• every operation (pf.log.env records the territory in force)
• /sources/:id/values/:col — it reads the source, untouched by any scope
• recode's set — a scoped account cannot set the territory column. That is reassignment, not forecasting, and the row would vanish from the view that would have shown what happened.

Ownership, not territory, governs DELETE /log/:logid, PATCH /log/:logid and PATCH /versions/:id/tags. The log PATCH looks like a private annotation and is not: label and bucket name the pivot's columns for everyone, so unguarded it let any account rename the company's segments, including on loads whose rows it cannot see.

Known gaps, verified in code:

• Completion now uses unscoped routes. Recode and clone complete from GET /versions/:id/values/:col (the version's own table, cached in memory keyed on the latest logid), the member list from /sources/:id/dim/:group, sibling autofill from /sources/:id/lookup. None applies the territory clause.
• Territory is read onto the session at login, so a change reaches a signed-in account only when it signs in again. Re-reading it in requireAuth would also make disabling immediate.

./pf.sh set-territory | set-admin | orphan-territory; the last lists values in the data that no account owns — work under one is invisible to all but admins, which a typo causes easily and nothing in the app reveals.

---

Bridge and change log

Bridge

A waterfall of how this forecast got from where it started to where it stands, one step per initiative tag. BridgeView.jsx; the arithmetic is in exported pure buildSteps() / layoutSteps(), checkable against real data without a browser.

Computed from the table already in the browser, not GET /versions/:id/bridge, so it reconciles with the pivot. The endpoint remains for API consumers, grouping by tag over rows outside exclude_iters, with no notion of buckets.

Scope is the selection (the union of its slices, an operation's reach) or the pivot's filter, following the live selection until a scope button is picked. There is no whole-version scope: the filter is part of what the figures mean, and dropping it moves one end of the walk and not the other — rows outside a season filter sat wholly in the forecast loads with no counterpart in the plan, inflating the endpoint without explanation. View filters are AND-only, so selection scope builds a view per slice, each with the pivot's filter.

Membership comes from pf_bucket, not pf_iter: open orders load as reference and are still forecast, and an iter-keyed bridge dropped them. The forecast bucket is the version's adjustment_bucket — the value unbucketed adjustments carry, so the two agree by construction. It used to be the literal 'Forecast'; once buckets carried their sort prefix nothing matched, every row became comparison, and the basis cancelled itself to zero. With no match it falls back to whichever bucket holds the adjustments. Selection scope does not filter on pf_bucket/pf_segment even when a slice names one, since Compare to needs rows from every bucket.

Compare to sets a bucket as basis — Plan when building a forecast, Prior Year when building the plan — resting on

Forecast − Basis = (Forecast loads − Basis) + adjustments

The opening step is the forecast's loads against the basis and the tagged adjustments explain the rest: no residual, no plug, the bars summing to the endpoint by construction. With no basis it shows the forecast's composition.

Steps group by tag alone, case-insensitively, showing the commonest spelling. They grouped by label first once, to match pf_segment, but every adjustment carries the same default label, so no tag ever showed — the label is the segment, the tag the initiative. Untagged adjustments collapse to one "N untagged adjustments" step, as in the ledger, rather than a wall of identical bars.

Colour encodes polarity: increase and decrease as two poles of a validated blue/red diverging pair (green/red is the classic colour-vision failure), neutral grey anchors. Everything not a bar follows the theme via CSS variables set through style, where var() is reliable. Every bar is labelled, a table view gives full precision, and the panel can be dragged.

Known issues:

• Overlapping slices are deduplicated by pf_id, which grain rows lack, so in grain mode a row matching two selected slices counts twice; it should key on pf_gkey.
• Selection scope keeps only dimension and date keys, dropping the pf_iter / pf_note / pf_op / pf_user / pf_logid keys operations honour, so it bridges a wider set than the operation would write; a total-cell {} slice is skipped outright.
• No drill-down from a step to its entries.

Change log

A modal of the version's entries, newest first — time, operation, slice, tag, note, rows and totals, author — with undo per row and tag and note edited inline (Enter saves, Esc cancels). PATCH /log/:logid branches on whether a field was sent, so "" clears rather than meaning "leave alone" — except label and bucket, renamed from the Baseline page and never cleared. A saved tag regroups the ledger and bridge at once, so history can be reclassified. Multi-slice entries list a line per slice, pulling out shared fields when only one dimension varies (a dragged region); {} reads "Total within the pivot filter". An entry with stored SQL opens its debug record — intent, environment, executed statement.

It shows an entry's full impact, not the reader's share: stamped totals are company-wide, so an admin's version-wide scale reads the same everywhere — deliberately, and labelled, rather than re-aggregated per territory.

Everyone / Mine. Everyone's entries are listed, with the reader's own marked in place — a blue rule down the row and a blue By cell. With several active accounts, finding your own meant reading By down the list. Marked and filterable: the mark tells without hiding anybody, the filter is for wanting only yours. It opens on Mine every time, unpersisted — the log is almost always opened to check your own recent work, and a stale "Everyone" is the worse default. isMine() is strict authorship, not canUndo(): an admin can undo everything, and marking every row as theirs would say nothing. The fetch stays unfiltered, since the control's counts need the whole set. Rows the reader cannot change are greyed rather than offering a 403.

?kind=adjustments drops load entries, which own nearly every row, from the recount join.

Known issue: the log does not show an entry's id, so there is no way to name one when asking about it.

---

Proposed: additional measures

Status: proposed — not built. Nothing below exists in code yet.

The problem

The app forecasts exactly one dollar measure (role = value) and one quantity (role = units). col_meta lets a second column be marked value, but only the loads carry it (generateSQL in lib/sql_generator.js: "the adjustment operations are single-measure … and use only the first of each"). So a second measure is right on the baseline and wrong everywhere an adjustment has been made:

• scale, recode and clone write adjustment rows with that column null
• a recode moves the sales and leaves the cost behind
• in grain mode /agg sums only the first value and the units column, so it is not in the pivot at all

Extended cost is the first case wanted — margin is sales minus cost — and the same shape recurs for rebates, surcharges, freight and the like.

The model: every extra measure is a rate on a driver

An extra measure is almost never forecast on its own. It is a rate applied to something already being forecast:

Measure	Driver	Rate held by an ordinary edit
Extended cost	qty	unit cost
Surcharge per unit	qty	surcharge per unit
Rebate	sales	rebate as a % of sales
Freight (fixed)	none	— carried, not moved by scale

Declared once, in Setup: col_meta.role = 'measure', plus a new col_meta.driver of units | value | none. The existing value and units roles are unchanged and remain the two the ledger edits.

How each operation treats a measure

The rule is that an edit never changes a measure's rate unless the rate is what is being edited — the same rule the scale already applies to price.

Operation	Measure driven by qty	by sales	none
Load	copied	copied	copied
Scale	moves by the qty factor	moves by the dollar factor	untouched
Recode	both legs, like the other cols	both legs	both legs
Clone	× scale factor	× scale factor	× scale factor
Undo	removed with its rows	removed with its rows	removed

So a price-only edit moves no qty-driven cost (qty did not move) but does move a sales-driven rebate (dollars did). The scale SQL already prorates value and units by one factor per coordinate; each measure is prorated by its driver's factor the same way, per row, so the rate holds per row and not only in total.

Because scale moves a measure at the slice's current rate, a rate set earlier is carried forward by every later edit, which is what makes Set rate (below) stick.

Setting a rate: its own operation, not the ledger

The ledger stays three boxes — value, qty, price — and the measures stay out of it. The ledger is already the hardest part of the panel to read, and a rate is a different question from a price-or-volume edit.

The case that needs it: a part used as a proxy for a new part not yet set up, where the new part's cost is roughly known. The workflow is clone or recode the proxy's volume onto the new part, then set the new part's cost.

Set rate · measure ext_cost · per qty · new rate 0.31

• Writes adjustment rows that change only that measure: the slice's cost goes to rate × its qty; sales and qty do not move. The same shape as a price edit, on a different measure.
• A sales-driven measure takes its rate as a % of sales.
• A none measure takes a plain target or change amount.
• One pf.log entry, operation = 'rate', params recording the measure, the rate and the resolved change; undoable, listed in the change log, and a line in the bridge like any other adjustment.
• Measured against the same basis choice as scale (adjustable rows or the selected total), with the same territory and scope rules.

In the pivot

Each measure is summed like sales and qty — /agg and the /data cursor carry every measure column. Rates and margins are not stored: they are Perspective expression columns computed from the sums (unit cost = cost / qty, rebate % = rebate / sales, margin = sales − cost, margin % = margin / sales), so they are right at every level of the tree.

What has to change

• col_meta: role = 'measure', new driver column; Setup's editor offers both.
• lib/sql_generator.js: loads and /agg carry every measure; scale prorates each by its driver's factor; recode and clone carry them; a new rate template.
• routes/operations.js: a rate route; scale passes the factors through.
• OperationPanel.jsx: a Set rate form beside Scale / Recode / Clone.
• Existing versions: the forecast table needs the measure columns, so a version created before a measure was added needs them added (or recreating) — the same col_meta drift noted under §Open questions.

Open questions

• Should a measure be allowed more than one driver (a surcharge that is part per-unit, part percentage)? Proposed: no — split it into two measures.
• Does Set rate on a slice with no qty (a qty-driven measure) refuse, as the price hold does on a slice with no value? Proposed: yes.

---

Open questions

• Replay — re-running a version's adjustments against a restated baseline. Not built: no route takes replay and nothing returns 501. The hard part is the one cloning a definition declines: a scale's target was sized against the rows that were there, so a replay must choose between the figure and the intent (params holds both inputs and resolved). env's template fingerprint exists partly so a replay can tell an entry ran under another template.
• Approval workflow — a submission an admin approves before others see it. Deferred.
• Export — CSV, or a push to a reporting table. The bridge's table view is a partial stand-in; there is no download. Stored names (§Naming) mean a reporting query needs only a join to pf.log.
• Cross-version comparison — side by side, or a bridge between versions; the isolated tables make it a UNION. Compare to answers the within-version form (against a plan bucket loaded into the same version); across versions is open.
• Bridge drill-down — a step listing its entries, or selecting them back in the pivot.
• Multi-connection support — one database via .env today. The vision: a pf.connection table (host, port, dbname, user, password as an env-var reference), connection_id on pf.source, per-connection pools; pf stays on a home connection and sources live anywhere, with a Connections panel in Setup. It changes the source schema, so it wants a clean install.
• Col_meta / version schema drift — roles changed after a forecast table exists put the generated SQL and the table's columns out of sync, unnoticed. Setup should compare col_meta with each table's columns, warn, and offer to rebuild (recreate the table, keep the version and log). Today: delete and recreate the version. Additional measures would hit this on every version.
• Separating is_key's meanings — §Data model → pf.col_meta.
• A per-source default layout beyond the pf_iter fallback, built from col_meta when nothing is published.
• Dynamic grain — §Display-grain pre-aggregation.
• Per-node expansion that survives a rebuild — §The pivot → Axis depth.
• Tests — SQL generation is token substitution against append-only tables, exercised only by hand.