# Pivot Forecast — Design What the app does, how the pieces fit, and why — including what each design replaced, where that history still explains the current shape. `CLAUDE.md` holds the rules an agent needs while editing; this holds the reasoning. Where either disagrees with the code (`routes/`, `lib/`, `ui/src/`, `setup_sql/`), the code is the truth. Related: `pf_perspective_options.md` (transport options, the DuckDB spike, why the pre-aggregated path won), `PERSPECTIVE.md` (bundling), `ui/vendor/README.md` (the patched Perspective build and how to rebuild it). --- ## Overview A web app for building named forecast scenarios against a PostgreSQL table: 1. **Load actuals as a baseline** — segments pulled from the source, optionally date-shifted into the forecast period, so the starting point is "no change from last year". 2. **Load reference rows** — prior year, plan, open orders — for comparison. Operations do not adjust them. 3. **Adjust** — select pivot cells and scale, recode or clone them. Every write is **append-only** and produces exactly one `pf.log` entry. An adjustment inserts new rows carrying its entry's id; undo deletes those rows and the entry. So a version is always the sum of its log, every figure is attributable to the entry that wrote it, and any entry can be reversed alone. Postgres holds the raw rows and is the single source of truth. The browser holds a copy in [Perspective](https://github.com/perspective-dev/perspective) (WASM), pre-aggregated to the display grain, and updates it incrementally from each operation's response rather than reloading. ### Lineage A port of an Excel model: a data tab of pre-aggregated rows, a pivot cache over it, a double-click on a cell opening a VBA form (`/opt/forecast_api/VBA/fpvt.frm`) that appended incremental rows, and a cache refresh. The append-and-let-the-pivot-sum model follows that directly, and the price/volume rules in §Operations come from the form's `calc_val` / `calc_price`. ### Stack Node/Express (`server.js`, one `pg` pool, type parsers coercing `bigint` and `numeric` to numbers so Arrow does not infer them as dictionary text); PostgreSQL, everything in schema `pf`; React + Vite + Tailwind in `ui/`, built to `public/app/`; Perspective 5.4.0 from a vendored patched fork (§The pivot). ### Pages and navigation Three steps in a collapsible sidebar: **Setup** (register sources, assign column roles, Generate SQL), **Baseline** (versions, load segments, version defaults, tags), **Forecast** (the pivot and the operation panel). Setup and Baseline are admin-only to write (§Access). **The forecasts are listed in the sidebar**, above the steps, each with its status and a star for the default. They used to be two dropdowns in the status bar, which is otherwise read-only, so the one pair of controls in it did not read as controls. The list is *not* nested under the Forecast step, where the idea started: Baseline works on the same version, so hanging it off one step would misrepresent its scope or duplicate it. The sidebar reads *what am I working on*, then *what step am I on*. **The source is not in the nav.** A raw table name means nothing to someone editing a forecast. Picking a forecast sets the source, since `pf.version.source_id` comes with it; the source picker appears only on Setup. `versions` (the selected source's, which Baseline and Forecast need) and `allVersions` (the sidebar's, a client-side fan-out over `/sources/:id/versions`) are kept apart so widening one does not change the other. The status bar keeps what follows from the selection: the forecast's name and row count, with the physical table in a click-to-copy hover panel — pasting it into psql is why it was ever on screen. **Known collision:** "forecast" names both the scenario and the third step. The scenario should own the word, leaving the step wanting another (`Adjust`). Undecided. ### Where a browser lands `App.jsx` restores the last source and version from `localStorage`, re-validated against the live list so a deleted version re-points rather than 404ing every call. With nothing stored it falls back to **`pf.version.is_default`** (one per source, partial unique index). It used to fall back to `versions[0]` — newest first, so whatever somebody created last, a scratch copy as readily as the plan — and accounts that never pick a version, which is every sales account, followed the forecasting work around. Setting the default is **admin only**, since it decides where everyone else lands; the Baseline checkbox is greyed for others. It is two statements (clear old, set new) in one transaction, because the index refuses the instant between. **A stored selection still wins** — the default is a fallback, so an account that landed in the wrong scenario keeps landing there until it switches once. ### Renaming a version Safe at any time, including closed. Nothing derives from the name: the table is `fc_{tname}_{version_id}`, `pf.log` and `pf.layout` key on the id, and client storage keys are id-based. The Baseline page's **This forecast** panel saves name and description on blur via `PUT /versions/:id`; a name collision answers 409, and a blank name is refused client-side because the route COALESCEs it away. ### API surface All under `/api`; all but the auth routes need a session. | Area | Routes | |---|---| | Auth | `POST /login`, `POST /logout`, `GET /me` | | Browser | `GET /tables`, `GET /tables/:schema/:tname/preview` | | Sources | `GET/POST /sources`, `GET/PUT /sources/:id/cols`, `POST /sources/:id/generate-sql`, `GET /sources/:id/sql`, `GET /sources/:id/values/:col`, `GET /sources/:id/dim/:group`, `POST /sources/:id/dim/:group/refresh`, `GET /sources/:id/lookup`, `GET /dim-period/cols`, `DELETE /sources/:id` | | Versions | `GET/POST /sources/:id/versions`, `PUT/DELETE /versions/:id`, `POST /versions/:id/{close,reopen}`, `GET /versions/:id/table-info`, `GET /versions/:id/values/:col`, `GET/PATCH /versions/:id/tags`, `GET /versions/:id/bridge` | | Data | `GET /versions/:id/data`, `GET /versions/:id/agg` | | Loads | `POST /versions/:id/{baseline,reference}`, `PUT /versions/:id/baseline/:logid`, `DELETE /versions/:id/baseline` | | Plans | `GET/POST /versions/:id/plans`, `PATCH/DELETE /plans/:pid` | | Operations | `POST /versions/:id/{scale,recode,clone}` | | Log | `GET /versions/:id/log`, `GET /log/:logid/debug`, `PATCH/DELETE /log/:logid` | | Layouts | `GET/POST /versions/:id/layouts`, `PATCH/DELETE /layouts/:lid` | `DELETE /sources/:id` answers 409 while the source has versions (`pf.version.source_id` is `ON DELETE RESTRICT`) and never drops forecast tables. --- ## Data model Install scripts in `setup_sql/`: `01_schema.sql` (metadata tables, with `ADD COLUMN IF NOT EXISTS` backfills and one-off data migrations inline), `02_auth.sql` (`pf.app_user`, `pf.session`), `gen_dim_period.sql` (creates and fills `pf.dim_period`; fiscal start month set at the top; re-runnable). `01_schema.sql` is idempotent but **not a migration runner**. Schema changes are applied to the live database directly and mirrored into `setup_sql/` for fresh installs. ### `pf.source` `schema`, `tname`, `label`, `status`, `created_by`. `default_layout` is dead — migrated into `pf.layout`, read by nothing. Registration seeds one `col_meta` row per column as `dimension`. **Columns come from `pg_catalog`, not `information_schema`**, which omits materialized views — they are not in the SQL standard — and the app's primary source is one. It looked columnless: registration seeded nothing and version creation failed "No usable columns in col_meta" while col_meta plainly held them. `RELATION_COLUMNS_SQL` in `lib/utils.js` returns the shape `information_schema` did (`format_type` minus modifier; precision and scale unpacked from `atttypmod`), so `mapType` and callers did not change. The table browser lists from `pg_class` by `relkind` for the same reason. ### `pf.col_meta` | Column | Meaning | |---|---| | `role` | `dimension` \| `value` \| `units` \| `date` \| `filter` \| `ignore` | | `label`, `opos` | display name; ordinal, which orders everything generated | | `is_key` | four separate uses, below | | `dim_group` | groups functionally dependent columns — a part and its attributes, a date and its periods | | `dim_period_col` | the `pf.dim_period` column this dimension is derived from at load | | `in_grain` | part of the display grain (§Display-grain pre-aggregation) | | `is_territory` | the column territory is expressed in; at most one per source, enforced on save | Roles: `dimension` is categorical and sliceable. `value` (required) and `units` (optional) are the measures — several may be marked, loads carry all, but the operations use only the first of each by `opos` (§Proposed: additional measures). `date` (at least one) — every date column is stored, shifted and carried by every operation, and sliceable. `filter` feeds the baseline filter builder and is not stored. `ignore` is not stored. Slices are validated against every `dimension` and `date` column regardless of `is_key`. **`dim_group` + `dim_period_col`.** A `date` column that is `is_key` with a `dim_group` anchors the group: its sibling dimensions with a `dim_period_col` are looked up in `pf.dim_period` against the *shifted* date at load time instead of copied raw, so fiscal periods follow a baseline moved forward a year. Every such group is honoured; this used to be a `.find()`, so a source with order, requested and ship dates derived one set and copied the other two raw — shifted dates under unshifted labels. **`is_key`** is read in four places, none of them what the name suggests: 1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes `members.find(c => c.is_key)` as the column `pf.dim_member` is keyed on 2. **Value completion** — `/sources/:id/values/:col` serves only `is_key` columns (the operation panel now completes from `/versions/:id/values/:col` instead, which takes any dimension or date) 3. **Sibling autofill** — `/sources/:id/lookup`, for `is_key && dim_group` 4. **The `dim_period` anchor** — `role = 'date' && is_key && dim_group` Uses 2 and 3 want several flagged; use 1 needs exactly one per group, and with more `.find()` silently takes the lowest `opos`. That has happened: a column null throughout outranked its group's real key, so a member refresh matched nothing and reported success with zero members. The meanings want separating — a per-group key choice, or the rule that the key is the column the group is named after — and until then a refresh finding two keys should refuse rather than guess. ### `pf.dim_period` One row per month keyed on `sdat`, a GiST-indexed `drange`, and calendar and fiscal year/quarter/month columns in several spellings. Joined as `LEFT JOIN pf.dim_period dpN ON dpN.drange @> (date + offset)::date` — left, because an unshipped order has no ship date and must keep its row. ### `pf.dim_member` Master data per `dim_group`: one row per key value, siblings in `attrs`, with `is_active` and `source_seen`. The source is transactional, often a view over all history, so deriving members from it is slow (a full scan) and wrong (it describes only what has been transacted — it cannot say a part is discontinued, or that a new one exists before it sells). This is the app's own list, refreshed from the source by an admin action but curatable apart from it. A refresh **merges**: members gone from the source become `source_seen = false`, not deleted, so curation survives; a key with several attribute sets across history takes the most recent. It stamps an explicit run time rather than `now()`, which inside a transaction is the transaction start. ### `pf.version` | Column | Meaning | |---|---| | `name`, `description` | `(source_id, name)` unique | | `status`, `closed_at`, `closed_by` | a closed version refuses every write but annotation | | `exclude_iters` | jsonb, default `["reference"]` — `pf_iter` values scale and recode will not write | | `is_default` | where a browser with no saved selection lands | | `adjustment_segment`, `adjustment_bucket` | names a new adjustment is stamped with (§Naming) | | `unlabeled_load`, `bucket_order` | dead — read by nothing | Creating a version inserts the row and creates its forecast table and `pf_logid` index in one transaction. Deleting drops the table; `pf.log`, `pf.layout` and `pf.load_plan` cascade. ### `pf.fc_{tname}_{version_id}` One table per version holding operational rows (`pf_iter` = `baseline`, `scale`, `recode`, `clone`) and reference rows (`reference`): ```sql CREATE TABLE pf.fc_{tname}_{id} ( pf_id bigserial PRIMARY KEY, -- every non-ignored col_meta column, typed from the source by mapType pf_iter text NOT NULL, pf_logid bigint NOT NULL, pf_user text, pf_created_at timestamptz NOT NULL DEFAULT now() ); CREATE INDEX fc_{tname}_{id}_logid_idx ON pf.fc_{tname}_{id} (pf_logid); ``` No `version_id` column: the table is the version. **`pf_logid` is not a foreign key** — undo deletes rows then log row in one transaction, and those are the only two paths that remove either. **The index** is how undo, the change-log recount and the grain key find an entry's rows; without it each is a sequential scan. Tables created before it was added lack it. ### `pf.log` | Column | Meaning | |---|---| | `version_id` | `ON DELETE CASCADE` | | `pf_user`, `stamp` | the session account; when | | `operation` | `baseline` \| `reference` \| `scale` \| `recode` \| `clone` | | `slice`, `params` | the slice(s) written to (null on loads); intent plus what the server resolved | | `label`, `bucket` | segment name and what it counts toward; required by the `log_named` CHECK | | `tag`, `note` | initiative name, which the bridge groups on; free commentary | | `row_count`, `value_total`, `units_total`, `measure_cols` | what the entry wrote, stamped at write time | | `env`, `sql_text` | what it ran against; the statement as executed | | `seq` | dead | **Who writes what.** Load templates insert `label`, `bucket` and `tag` in their `ilog` CTE. Adjustment templates insert `label` and `bucket`; their `tag` goes on in a follow-up `UPDATE` in the same transaction (`tagLog()`), a hold-over from when a new `{{tag}}` token would have stranded sources that had not re-run Generate SQL. Totals, `env` and `sql_text` are written after commit by `stampLogTotals()`. **Stamped totals are not a cache.** An entry's rows never change after it is written — only its own operation inserts them and only undo removes them, taking this row too — so the totals are fixed. The change log used to join the whole forecast table on every open to recompute them. `measure_cols` records which columns they are in, since roles can be reassigned. Stamping is best-effort: a failure does not roll back a write that succeeded, since totals can be recomputed and the adjustment cannot. `GET /versions/:id/log?recount=1` recomputes by join and writes back — the repair for rows deleted by hand, and the backfill for entries older than the columns. **`params` and `sql_text` are both kept on purpose.** One is the intent (slices, scope, resolved increments), the other what it became. Keeping only one assumes the translation is correct, which is exactly what is in doubt when rows look wrong. `env` holds what cannot be reconstructed afterwards — the territory, the version's `exclude_iters`, and the template's `generated_at`, a fingerprint (not a version) showing whether an entry ran under a different template. `GET /log/:logid/debug` returns the whole row; the list omits `sql_text` and reports `has_sql`. ### `pf.sql` One generated template per `(source_id, operation)`: `get_data`, `get_agg` (only with a grain), `baseline`, `reference`, `scale`, `recode`, `clone`, `undo`. `generateSQL()` in `lib/sql_generator.js` bakes in column names and leaves `{{token}}`s — `fc_table`, `version_id`, `pf_user`, `note`, `tag`, `label`, `bucket`, `params`, `slice`, `filter_clause`, `date_offset`, `where_clause`, `exclude_clause`, `value_incr`, `units_incr`, `set_clause`, `scale_factor`, `territory_clause` — for `applyTokens()` at request time. Generation deletes any operation it no longer produces, since a stale `get_agg` after clearing the grain would still be picked up. The stored `undo` is documentation; the route runs its own statements. Values are escaped (`esc()` doubles quotes), not parameterised — consistent with the codebase, and the executed SQL reads as-is in the server log and `pf.log.sql_text`. **Editing the generator changes nothing until Generate SQL runs**, since the templates are stored; a template carrying a token the running code does not substitute fails at the database, reading as an unrelated client error. Restart first, then Generate SQL, or the old code writes the templates. ### Other tables `pf.layout` (§The pivot), `pf.load_plan` (§Loads and segments), `pf.app_user` and `pf.session` (§Access). --- ## Loads and segments A version's rows start as **segments**: independent queries against the source, each appended as one log entry, each undoable alone. Two kinds, differing only in `pf_iter`: - **baseline** — the rows operations adjust - **reference** — kept out of scale and recode by `exclude_iters` `pf_iter` answers *can operations write these rows*. Whether rows belong in the forecast number is the segment's **bucket** (§Naming): open orders load as reference so nothing adjusts them, and are still part of the forecast. Neither can be derived from the other. All load routes are admin-only. ### How a load's WHERE is built The Baseline page's filter builder takes **groups** of conditions — AND within, OR between — over `date` and `filter` columns (`=`, `!=`, `IN`, `NOT IN`, `BETWEEN`, `IS NULL`, `IS NOT NULL`). A period is a filter like any other. A toggle swaps the builder for a raw WHERE body. **The client builds the clause; the route injects it.** `buildFilterClause()` in `Baseline.jsx` renders `(a AND b) OR (c)` and sends it as `where_clause` beside the structured `filters` (or `raw_where`). The route substitutes `raw_where || where_clause` as `{{filter_clause}}` verbatim and keeps `filters` / `raw_where` in `params` only so a segment can be re-opened and a plan derived. Nothing server-side validates columns or escapes values, and the client does not escape them either; `lib/sql_generator.js`'s own `buildFilterClause()`, which does, is called by no route. The protection is `requireAdmin`: whoever can reach the route can already send raw SQL, so validating the structured path would not narrow what they can do. An empty clause becomes `TRUE` — the whole source — and the client refuses to send one. ### Date offsets `date_offset` is a Postgres interval (`1 year`, `-90 days`) applied to **every date column**, so order and ship dates stay in step, with period dimensions looked up against the shifted dates. Filter columns are never shifted: the offset moves where rows land, not which are chosen. It must be **whole days** (`assertInterval()`): shifted dates are cast back to dates, so anything finer is rounded away, and Postgres reads a bare number as seconds — `"30"` used to be accepted and move nothing. `30 days` is not a month either; use `1 month`. Checked on baseline loads, segment edits, recode and clone — **not** on reference loads, where a bad offset surfaces as a Postgres error. The offset is typed as an interval string. It replaced year/month spinners, which could not express days and clamped at zero, so a segment could only move forward in whole months. The timeline preview parses it only far enough to draw the source and shifted periods; Postgres is the authority. ### Naming, editing, clearing A load's **label is required** (400 without) — the segment name the pivot shows and reports group on — and its bucket defaults to it (`loadNames()`). `PUT /versions/:id/baseline/:logid` edits a segment by deleting its rows and log row and re-running it in one transaction, under a **new** logid. Refused (409) once any adjustment exists, since those were sized against the old totals. Annotations carry over with `??`, not `||`: the form has no tag input, so an absent tag must survive, while an explicit `''` clears. `DELETE /versions/:id/baseline` removes every `baseline` row and entry (not reference). Not undoable. ### Load plans — `pf.load_plan` A segment used to exist only once run, so cloning a definition meant re-running every load, and there was nowhere to hold "this cut, but with the dates changed first". A plan is the definition alone — `kind`, `label`, `bucket`, `tag`, `note`, `filters`, `raw_where`, `date_offset`, `seq` — with `run_logid` naming the entry it produced, or null. - **Deliberately not `pf.log`**: every row there means a write happened, and unexecuted intent would cost it that meaning. - **`run_logid` is `ON DELETE SET NULL`**, so undoing a load returns its plan to unrun instead of deleting the definition. The list reads the entry's `row_count` through the FK to tell run from unrun. - **Running is two round trips** — the ordinary load POST with the plan's fields, then `PATCH /plans/:pid` with the `log_id`. The alternative was a second copy of the load route's substitution; a plan that reads unrun while its rows exist is visible and fixable, a divergent copy of the load SQL is neither. - **Run all stops at the first failure.** A definition's segments are not independent — references date-shift off a baseline — so carrying on past a broken one builds something nobody described. Known issue: `runPlan()` does not send the plan's `tag`, so a tagged plan loads untagged. ### Cloning a version's definition `POST /sources/:id/versions` with `clone_from` copies the template's settings (description unless given, `exclude_iters`, adjustment names), its **version-scoped** layouts, and its segments as plans — and **runs nothing**. The point of cloning a definition is to change the dates before anything loads; a clone that fetched the baseline first would have to be undone to be edited. Not copied: **adjustments**, in any form — a scale's target was sized against the rows that were there, so replaying it against another load is arithmetic nobody asked for; and **source-wide layouts**, which already apply and would duplicate by name. `cloneDefinition()` reads the template's plans, or derives them from its `baseline`/`reference` entries when it has none — every version built before plans existed. `params` already stores `filters`, `raw_where` and `date_offset`, so this is a read, not a reconstruction. ### Replay Re-running a version's adjustments against a restated baseline is **not built**. Nothing accepts a `replay` flag and nothing answers 501, whatever earlier specs said. See §Open questions. --- ## Naming: segments, buckets, notes Four pivot columns come from `pf.log`, joined on `pf_logid`, not from the forecast table: | Column | From | |---|---| | `pf_segment` | `l.label` — which load, or which kind of adjustment | | `pf_bucket` | `l.bucket` — what it counts toward: Actual, Plan, Forecast, Prior Year | | `pf_note` | adjustments: the tag, else the note; loads: null | | `pf_op` | `l.operation` — `/agg` and pushed-back rows only | Separate on purpose: commingling a segment name with an adjustment's note makes neither pivotable. In grain mode `pf_logid` is in the grain, so the join adds no rows. `SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are the single definition, shared by the generated `/agg` and the hand-written `/data` cursor, which must agree. The operation routes stamp the same fields on the rows they return, since `RETURNING *` lacks them. The `/data` cursor does not emit `pf_op`, so a raw-mode pivot cannot group on it. ### Every log entry is named `label` and `bucket` are **stored, not derived**, and every entry has both — `log_named` refuses one without. A report gets the pivot's segment and bucket by joining `pf.log` and selecting two columns, with no rule of its own. Stamped at write time: - **A load** — `label` required, `bucket` defaulting to it. - **An adjustment** — no name inputs, so the version's `adjustment_segment` / `adjustment_bucket`, else `ADJUSTMENT_SEGMENT = '99 - Adjustments'` and `ADJUSTMENT_BUCKET = '04 - Forecast'`. `adjustmentNames()` resolves them once per request, for the INSERT and the returned rows alike. The version defaults are set on the Baseline page (**Defaults for new adjustments**) and name the *next* adjustment; existing entries keep theirs. Renaming an entry is `PATCH /log/:logid` — from the Baseline page's segment list — which renames but refuses to clear, owner or admin only because the names are the pivot's columns for everyone. Relabelling needs a page reload to show, since the label is part of the aggregated row. A source registered before names were stored needs Generate SQL once so its templates write them. **What this replaced.** The names were derived at read time: a load's `label`, else `tag`, else `note`, else the version's `unlabeled_load`, else `Unlabeled`; an adjustment's `label`, else the version default, else the constant — through a join on `pf.version`. That made the pivot the only reader that knew a row's bucket: the CASE was copied into `/agg`, the `/data` cursor and a reporting procedure in `/opt/forecast`, and every direct query had to copy it again. And editing a version's fallback silently renamed every past adjustment. The backfill in `01_schema.sql` stamped existing entries with that rule, so nothing on screen changed name. ### Column order is stored text Perspective orders column groups by the value string, and `asc`/`desc` only reverses that, so Prior Year → Plan → Actual → Forecast is expressible in neither direction. A `"01 - "` prefix is the only lever, and it lives in the stored label and bucket, typed by whoever names the segment. Nothing derives it. The one hardcoded ordinal is `99 - Adjustments`, keeping unnamed adjustments after every *numbered* segment. That proviso is the scheme: ordering is string ordering, so `99` lands last only while loads carry `01`–`0n`, and an unnumbered segment sorts after it (digits precede letters). The old `'(adjustment)'` sorted *first* for the same reason read the other way. Relabelling an adjustment's log row is how one kind of adjustment is split out from the rest. **What this replaced.** The prefix was computed client-side as Perspective expression columns (`pf_bucket_ord`, `pf_segment_ord`) from `pf.log.seq` and `pf.version.bucket_order`. It ordered the pivot and nothing else, so every other reader disagreed; `restore()` replaces `expressions` wholesale, so it had to be re-applied after every layout load; and ExprTK's scanner tests each *byte* with `isprint()`, so a label with anything outside printable ASCII could not be ordered at all. `DEAD_ORDER_EXPRS` in `Forecast.jsx` strips those names from old layouts. ### Tags A tag names an **initiative** — a line in the ledger and the bridge. It never affects rows, so it stays editable, even on a closed version. It is not a row anywhere, only a value repeated across a version's entries; it exists because somebody typed it. **Completions are per version.** `GET /versions/:id/tags` returns this version's tags with `uses`, `last_used`, `owners`, `can_edit`, refetched on version change and after every tagged write. It used to be `GET /sources/:id/tags`, across every version of the source — which put last year's initiatives in front of this year's and reported use counts earned elsewhere. The cost is that a fresh version completes from nothing until it has a tag, which is honest: cloning a definition copies no adjustments. **Managed on the Baseline page.** `PATCH /versions/:id/tags` `{ from, to }` rewrites every entry carrying `from`. A `to` that already exists **merges** — usually the point, since "Pull Forward" and "Pulled Forward" are one initiative reported as two. An empty `to` clears. Both are confirmed first, being invisible as a diff afterwards. Permission is ownership of *every* entry the tag is on — a half-applied rename leaves the initiative split, the state being fixed — so rows the reader cannot rename are greyed; the route locks the rows before checking, so check and rewrite see the same set. The Tags panel sits outside the page's read-only fieldset: a rep may rename what they named. **Completion matches on words.** `TagInput` in `OperationPanel.jsx` lowercases, strips punctuation and stopwords and lightly stems both sides, so "Pulled Forward" finds "Pull Forward" and one inserted word does not hide an existing tag. Something that reduces to an existing tag without being it says so (*"Already used as … — use that"*). Nothing merges automatically. The change log's inline tag cell still uses a plain datalist. The panel used to carry six quick-pick pills under the tag field; they wrapped to two rows once tags were longer than a word and pushed the docked form past its border. The list shows the tag alone — an option's text renders as a second column, so a use count widened every row for a number nobody picks on. --- ## Selection A selection is a list of **slices** (`{column: value}`) plus the pivot's own filter as **scope**. The server turns them into a WHERE clause; the ledger totals the loaded table through the same filters, so the panel and the write agree. ### From a click to a slice `perspective-click`'s own filter list cannot be trusted: the datagrid's `getCellConfig` builds it with `pivot_value ? [pivot, '==', pivot_value] : undefined`, so **the engine drops a blank group key** — and, being a falsy test, a `0`. A dropped key widens the write: clicking the blank node under a customer selected the whole customer, and the ledger agreed, totalling through the same list. So `cellFilters()` in `Forecast.jsx` re-derives both axes: - **Rows** from `detail.row.__ROW_PATH__` — raw values, so dates stay epoch millis. A subtotal's shorter path leaves deeper levels absent. - **Columns** from the clicked column's name (taken from the mousedown's cell; the event names only the measure). The name is the visible split values joined with `|`, measure last, so the measure is popped before mapping `split_by` positionally. Mapping every segment used to read the measure as a value of the first collapsed dimension — a slice matching no row. A plain click replaces the selection; ctrl/⌘/shift toggles a slice. The event carries no modifier flags, so they are read off the preceding mousedown. `buildEffectiveSlice()` keeps dimension, date and `pf_` keys, turns dates into ISO strings, and drops the rest; duplicates are collapsed, since under `each` a repeat would apply twice. A Perspective computed column does not map back to stored rows, so only native columns slice. ### Dragged regions The datagrid runs in `edit_mode: SELECT_REGION`, forced on every restore so a layout cannot switch it off. A drag fires `perspective-select` with a **ViewWindow** — `{ start_row, end_row, start_col, end_col }` — on every mouseover, so the handler keeps the latest and a window `mouseup` commits it, each cell going through `cellFilters()`. A single-cell region is ignored: `perspective-click` owns clicks, and handling both would undo a ctrl-click. ### Blanks: null and `''` are different values Both occur in real data as different rows, so they stay distinct from click to WHERE; only `undefined` means absent. - `buildWhere()` renders null as `IS NULL`. It used to fall through `esc()` to `= ''`, selecting *the other blank* — the slice named one node and the operation wrote the other. In a list the null arm is ORed on, since `IN (NULL)` is never true. - The ledger filters null with `is null`, not `== null`: the engine matches on type, and a null threshold on a string column is another dropped predicate. - On the column axis, `t_tscalar::to_string` renders `''` as `''` and a null as the text `"null"`, so `"null"` is read back as null. This was believed impossible — a null was thought to join as `''` too — so `''` cells were refused and `"null"` was taken literally, a slice matching nothing. The one ambiguity left is a value that really is the text `null`. - The panel prints `(null)` / `(blank)`, and `—` only for a column not in the slice: isolating the blank node and ignoring the column are opposite statements and used to look identical. ### The pivot's `pf_` columns are part of the slice - **Table columns** — `pf_iter`, `pf_user`, `pf_logid` (`TABLE_SLICE_COLS`) — compared directly. - **Log columns** — `pf_segment`, `pf_bucket`, `pf_note`, `pf_op` (`COMPUTED_SLICE_COLS`) — resolved to `pf_logid IN (SELECT id FROM pf.log WHERE version_id = … AND …)`, exact because the name lives on the log row. - **Any other `pf_` key is refused** — skipping is how selections widened. They **narrow a slice but do not make it selective** (`pf_iter = scale` alone is every adjustment): `assertSelective()` needs a dimension, date, `pf_segment` or `pf_bucket` key. **What this replaced.** `pf_iter` was stripped on the theory that cells differing only by iter band are one slice; `pf_note` and `pf_op` were refused, and the client pre-empted that by stripping; `pf_segment` and `pf_bucket` were dropped before that. The ledger stripped none of them. So under the default layout, grouped by `pf_iter`, a click on a customer's `scale` row totalled the scale rows and wrote to all the customer's rows; a click on one bucket's cell scaled every bucket there. ### Several slices, and total cells Several slices become an OR of AND-groups, not per-column `IN` lists — `{A:1,B:1}` + `{A:2,B:2}` as `A IN (1,2) AND B IN (1,2)` would also match `A:1,B:2` — parenthesised, since callers append `AND pf_iter NOT IN (…)`. **A total cell is its displayed amount.** The grand-total row names no row dimension, so its slice is whatever the column axis and the pivot filter leave — possibly `{}`, *everything the filter shows*. `{}` is accepted only with a non-empty scope (`normalizeSlices()`, `assertSelective()`); alone it would be the whole version and is refused on both sides. `buildWhereAny()` reads a union containing `{}` as `TRUE`, the scope then ANDed on — dropping it would narrow the write to the other cells. The panel names it "Total". ### Scope: the pivot's filter travels with the slices A slice can only mean equality, so a bounded season, an in-list or a substring has no slice form. The filter is read off `viewer.save()` — in the table's own types, with the user's operators — and sent as `scope`, `[col, op, value]` triples; the ledger totals against the same list. Without it the ledger counted rows the grid was hiding and the operation wrote them. It is ANDed onto each statement rather than folded into the slices, which under `each` would repeat it per statement. `SCOPE_OPS` in `lib/sql_generator.js` is the whole allowed list, under **Perspective's** names: `==`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `not in`, `is null`, `is not null`, and the substring family. Anything else is refused — a scope silently dropped is a write wider than the panel that authorised it. `pf_` log columns take `==` or `in` only. - **Substring operators map to `ILIKE`**, because the engine's `t_tscalar::contains` / `begins_with` / `ends_with` lowercase both sides. `LIKE` would write a fraction of what the grid showed, and narrower is no safer than wider — neither is the set the ledger counted. The engine matches the term literally, so `%`, `_` and `\` are escaped. Residual difference: the engine folds ASCII only, `ILIKE` follows the collation. - **`matches` / `not matches` stay refused.** RE2 in the engine, not in Postgres — `\b` is a word boundary in one and a backspace in the other — so a pattern valid in both could match different rows, which is worse than a 400. - **An empty term is refused.** An unfinished filter row arrives as `[col, '==', null]`, and `ILIKE '%%'` or `= ''` would read as every row or none rather than as the mistake it is. The ledger used to drop a slice's column whenever the scope named it, assuming they could not disagree; under a substring filter, clicking one part then totalled every matching part. It now drops only an exact `==` repeat. Territory (§Access) is ANDed on after the scope, from the session. ### Selection highlight The datagrid highlights `model._selection_state.selected_areas` and wipes it on every mousedown, so a ctrl-click selection would show only its last cell. `areasRef`, a `sliceKey → rectangles` map beside `slices`, is pushed back after every change and pruned by live slice key on any deselect, so grid and panel cannot disagree. Rectangles are **re-derived when the grid moves**. A sort reorders rows and a new segment column shifts every column; tracked coordinates would then highlight whoever took the position while `slices`, being value-based, still named the right row — the grid would be the only thing lying. `relocateAreas()` sweeps the displayed rows as a drag would and keeps cells whose slice is still selected. A slice can therefore light more cells than were clicked (a slice not naming the segment matches the customer under old and new segments alike), which is what it covers. Past `MAX_RELOCATE_ROWS` the highlight is cleared rather than left wrong. --- ## Operations `POST /api/versions/:id/{scale|recode|clone}`, common body: ```json { "slices": [ { "channel": "WHS" } ], "scope": [ ["season", "<=", 2027] ], "apply_mode": "prorate", "tag": "volume_push", "note": "optional" } ``` `slices` — one or more (a legacy single `slice` is accepted). `apply_mode` — `prorate` (default) is one statement over the union and one entry; `each` is one per slice, so each reaches its target alone and undoes separately (for recode and clone it only decides how many entries). `pf_user` is the session's. A closed version answers 403. Operations are not admin-gated; territory and ownership govern them. The response is the new rows — whole in raw mode, aggregated in grain mode — for `table.update()`; no reload. ### SQL shape An `ilog` CTE inserts the log row and the insert CTEs reference its id; `{{where_clause}}` is the selection and `{{exclude_clause}}` keeps `exclude_iters` out. **Every date column travels** — the written columns are every dimension, every date and the first value and units column. Scale and recode used to carry only the first date, so on a source with order, requested and ship dates they wrote the others **null**. The period dimensions were fine, being dimensions — which hid it: labels right, dates gone. Those dates are sliceable, so an adjustment scoped to a ship date matched no adjustment ever written. Clone and loads were always right. Rows written before the fix keep the nulls; deleting and replaying those entries is the honest repair, since re-deriving them would be inference. **One row per coordinate, not per row read.** Reads GROUP BY every dimension and date. One-for-one, every operation inherited the row count of everything before it at that coordinate, and the table grew super-linearly with the work done. Only `pf_logid` and `pf_iter` collapse, so nothing goes null or unsliceable; the maths is unchanged, `sum(sum(x)) OVER ()` being the aggregate of aggregates. ### Scale Distributes an increment in proportion to each coordinate's current amount: ```sql round(("value" / NULLIF(total_value, 0)) * {{value_incr}}, 2) round(("units" / NULLIF(total_units, 0)) * {{units_incr}}, 5) ``` `resolveIncrs()` in `routes/operations.js` computes the two increments per statement, which is what makes `each` land per slice. Value and units each take at most one of a target (`target_value`/`target_units`), a percentage (`value_pct`/`units_pct`) or a change (`value_incr`/`units_incr`), resolved independently; `target_price` is a third input; legacy `pct: true` is honoured. **`target_basis`** — what a target or percentage measures against: - `adjustable` — only the rows the operation can write. The **server default**. - `selected` — everything the pivot shows, `exclude_iters` rows included. Those cannot move, so the adjustable rows absorb the difference and the pivot lands on the number asked for; otherwise a target set against a visible total overshoots by the fixed rows. The panel defaults to `selected` and sends it only when the selection holds excluded rows (otherwise they are identical). The basis row is tinted and tagged "target", and clicking either row switches. #### Price does not move unless it is named A price move is invisible in the figure being edited — the dollars land either way, and only the implied price says whether volume followed. So: **`plug`** answers the one ambiguous case, dollars alone: `volume` (default, price holds, units scale in proportion) or `price` (volume holds). From the Excel form: ``` plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange plug price: fVol = pVol + bVol ``` **A qty input follows the same rule**: with dollars unnamed they follow at the current price, `value = (curValue / curUnits) * unitsIncr`. Holding them made every qty edit a price edit — more cases at the same revenue is a price cut nobody asked for. A slice with no units has no price, so its dollars stay put. **The price held is the adjustable rows' own**, never blended with the fixed rows. The SQL prorates over the adjustable rows only, so an increment sized at a blended price moved qty by a different factor than dollars wherever reference rows priced differently — and taking a selection to its fixed total drove qty past zero, grains with positive sales and negative qty. The ledger preview uses the same price. **Naming the price** (`target_price`, the Excel form's Edit Price) holds it, and the other named measure decides the rest, on the chosen basis: | Named | Result | |---|---| | price alone | qty holds; dollars = price × qty | | price + qty | qty lands as named; dollars = price × that qty | | price + dollars | dollars land as named; qty = dollars / price | | all three | refused — two degrees of freedom, three numbers | This branch used to read only a qty *target* (a qty change sized dollars on the old qty), took the fixed qty off a target but not the fixed dollars, and let a dollar figure outrank the price. Worse, the panel sends no `plug` with a price, so the server read `volume` and scaled qty by the dollar change too, putting price back where it started. A named price now bypasses the plug. So the deliberate ways to move price are **dollars and qty together** and **naming the price**; both outrank the plug. **What this replaced.** `plug` defaulted to `price`, being what the API did before the flag existed — so an unqualified dollar edit and every qty edit moved price, and only a hand-picked `volume` did not. The client's preference is keyed `pf_scale_plug_v2` so a browser carrying the old `price` choice does not go on silently moving price. #### The ledger and its three boxes The scale form is one continuous statement: baseline, adjustments grouped by tag (untagged ones collapsed), fixed segments, total, then the edit, with a running total closing on the selected total. Value, qty and price each have `New` / `Change` / `% change` boxes; typing in one derives the others, and the box typed in is what is sent — the row you type in *is* the mode. A **Result** line previews all three under the server's rules and marks the column that cannot move `· holds`. **The boxes foot** — price = value / qty — so at most two are inputs. `effectiveScaleInputs()` in `OperationPanel.jsx` decides which when all three are typed: value typed last lets the plug decide (`volume` drops qty, `price` drops price), since a dollar change says nothing about price or volume; otherwise the more recently touched of the other two stays. The dropped box keeps its text, so flipping the plug brings it back. `buildPayload()` in `Forecast.jsx` uses the same function, so what is posted is what is shown. Clicking into a derived figure pins it — "units are this" rather than "units may move" — and **unpin** releases it. Rough edge: all three typed, dollars last, the choice is really which earlier number to keep, and "Volume" there moves price too. **The Moves control** is the plug. It lights whatever the edit moves — both if both do — and takes a click only while it is the one deciding. It used to render only for a dollars-only edit, so it was absent before anything was typed and vanished when a measure was pinned, even by an accidental click — two states that both looked like a missing control. It now stays, greyed when the inputs have answered it, showing their answer. A price change or percentage is resolved client-side to an absolute `target_price`; the server has no price increment form. #### What the log records `params` carries the intent (`pickIntent()`: inputs, `target_basis`, `scope`, `plug`) plus: - **`resolved`** — `value_incr`, `units_incr` and **`method`**: `price` (named), `both` (dollars and qty named), `volume` (qty named), else the plug. The request alone cannot say which moved — the plug is a per-browser preference — so a price edit used to be findable only by noticing qty had not moved. - **`expected`** — the panel's Result line as a change on its basis (under `each`, combined and flagged so), beside `resolved` so a panel/server disagreement shows in the log. Nothing compares them yet. #### Refusals - **A pool netting to ~zero** (below 1% of gross): shares are `row / total`, so as the net nears zero rows fly to extreme opposite values. Offsetting slices are the usual cause; `each` handles them. - **Nothing to scale** — every unit already on target or empty; individually skipped slices come back in `slices_skipped`. - **Holding price on a selection with no value** — no price to hold. Rounding each row to 2dp can leave a target a cent off; landing exactly would need a correction row. ### Recode Moves the selection to other dimension values in one entry: **negative rows** zero it out where it was, **positive rows** put it back with `{{set_clause}}` replacing the named dimensions. **A date shift moves the put-back leg only**, so a shifted recode is a timing move — pull forward, push out — and `set` may be empty. Moved rows' period dimensions are re-derived against the shifted date, as clone's are; an explicit `set` wins; a row with a null date keeps its label, since a lookup on nothing would blank it. Without a shift nothing is derived. **Coordinates already netting to zero are skipped** (`HAVING sum(x) <> 0`), rather than writing two empty rows each — which a second recode over cells a first one emptied would do for all of them. If nothing is left the request is refused ("Nothing to recode") rather than committing an empty entry. A scoped account cannot set the territory column (§Access). ### Clone Copies the selection as new business: `set` overrides dimensions, `scale` multiplies (default 1), `date_offset` shifts every date; the original is untouched. Period dimensions come from the calendar against the shifted date — else a mix moved forward a year keeps last year's labels — and `set` wins. Unlike recode, a null date blanks them. **`exclude_iters` does not apply to clone.** It stops operations *modifying* reference rows — scale would attribute movement to prior-year rows, recode would write negatives against them. Clone does neither; copying a plan or prior year out of reference is the operation working as intended, and excluding them made a deliberate selection silently produce nothing. **`from_logid`** narrows instead, for a selection spanning segments where only one is wanted. Known issue: a clone matching nothing commits an empty entry with `rows_affected: 0`, where scale and recode refuse. ### Undo `DELETE /log/:logid` deletes the entry's rows then its log row in one transaction and returns `pf_gkeys` (grain mode) or `pf_ids` (raw) for `table.remove()`; the view re-sums. Undo is by **ownership**, not territory — your own entries or an admin's override — because it removes rows wholesale, and half-undoing one would leave a state nothing describes (and two accounts can share a territory). Closed versions refuse. Undoing a load returns its plan to unrun. Known issues: the `pf_gkeys` come from **live** col_meta, so after a grain change they match nothing in the loaded table and the pivot keeps the rows (the database is still right); recode and clone log `apply_mode: unit.mode`, never set, so their `params` omit it. --- ## Display-grain pre-aggregation **Built, with a static grain.** Path B of `pf_perspective_options.md` — a pre-aggregated extract in a **native** Perspective table — rather than Path A, live aggregation through a DuckDB virtual server (parked on `spike/duckdb-virtual-server`), which loses expand, collapse, depth, sort and filter. **The problem.** Shipping raw rows makes the browser pivot all of them and load time grows with raw rows. Aggregating server-side to the grain the pivot displays cuts the row count by orders of magnitude while Perspective still does all the rollup locally. ### Grain definition — `pf.col_meta.in_grain` The flagged `dimension`/`date` columns, plus always `pf_iter` and `pf_logid`. `grainOf()` in `lib/sql_generator.js` is the single definition (`Setup.jsx` and `routes/log.js` mirror it), baked into the templates at Generate SQL so load and operations agree by construction. - **Additive measures only.** Sums are exact; an average would ship as sum + count; `count(DISTINCT …)` cannot be pre-aggregated. Forecast measures are sums. - **An unflagged dimension cannot be pivoted or filtered on.** Leaving high-cardinality columns (part numbers, raw dates) out is what keeps the grain from exploding back toward raw; the price is that they are unavailable. Dates usually enter through their derived period dimensions. - Generation refuses a grain column the table never populates, or a grain with no measure. `in_grain` means *eligible*; today the grain is exactly the eligible set. ### The model: append deltas, let the view sum The Excel pivot cache again: pre-aggregated rows go to the table, the pivot cuts them locally, and each adjustment is **appended**, never a recomputed total. Rows are aggregated **per `pf_logid`** — one per grain tuple × `pf_iter` × entry — and the view sums them. It was also the smallest change from raw mode, which already appended results with `table.update()`. **Synthetic key.** Perspective indexes on one column, so each row carries ```sql md5(concat_ws(chr(31), COALESCE(col1::text, chr(30)), …, pf_iter, pf_logid::text)) AS pf_gkey ``` `pf_logid` keeps each entry's contribution distinct, so an append accumulates rather than replacing a bucket and undo removes exactly that entry. Plain `concat_ws` skips NULLs, so `('a', NULL)` and `(NULL, 'a')` would collide and silently merge two groups; `chr(30)` also keeps NULL apart from `''`. The md5 because the key is an opaque handle read only by `update()`/`remove()`: raw, it was long and, unique per row, defeated Arrow's dictionary encoding — the largest column in the payload. 128 bits keeps collisions unreachable. ### Initial load — `GET /api/versions/:id/agg` ```sql SELECT t., t.pf_iter, t.pf_logid, ,SUM(t.) AS , SUM(t.) AS ,l.label AS pf_segment, l.bucket AS pf_bucket , AS pf_note, l.operation AS pf_op FROM {{fc_table}} t LEFT JOIN pf.log l ON l.id = t.pf_logid WHERE {{territory_clause}} GROUP BY , pf_iter, pf_logid, l.operation, l.label, l.tag, l.note, l.bucket ``` The `pf.log` columns depend on `pf_logid`, but Postgres will not infer it, so they are grouped (`LABEL_GROUP_COLS`). **Territory applies before the GROUP BY**, since its column need not be in the grain. The route sets `LOCAL enable_sort` and `enable_incremental_sort` off: seeing `pf_logid` pre-sorted off its index, the planner picks a sort-based GroupAggregate for the "free" sort, but comparing dozens of keys per row makes it several times slower than a HashAggregate, whatever `work_mem` or statistics say. The client loads the result with `worker.table(buffer, { index: 'pf_gkey' })`. ### Write and delete paths Operations insert raw rows under a new logid as always; in grain mode the final select aggregates **that entry's rows** to grain (`FROM ins`, or recode's `neg UNION ALL ins`) — safe because they all carry the new logid. The keys are therefore always new, `update()` **appends**, and nothing is recomputed. Undo removes an entry's keys and lets the view re-sum — no re-aggregation, no emptied-bucket handling. `RETURNING` does not take `DISTINCT`, so: ```sql WITH del AS ( DELETE FROM {{fc_table}} WHERE pf_logid = $1 RETURNING , pf_iter, pf_logid ) SELECT count(*)::int AS rows_deleted, array_agg(DISTINCT ) AS pf_gkeys FROM del; ``` ### Raw mode With nothing flagged, no `get_agg` exists and the client uses `/data`: every row plus `pf_segment`, `pf_bucket` and `pf_note`, indexed on `pf_id`; operations return whole rows and undo returns `pf_ids`. ### Transport Both endpoints return Arrow IPC in **one record batch** with `X-Row-Count`. `/data` reads a cursor in batches into column arrays (row objects multiplied peak heap) and builds one table with `tableFromArrays`; `/agg` does the same from its result. One batch because per-batch encoding creates independent dictionaries, and Perspective's WASM reader crashes on the replacement messages. An earlier plan to stream batches into `update()` behind a large-dataset banner was never built, for that reason. **Load time is dominated by row count, not payload size.** Even a good grain on a wide source leaves enough rows that getting them out of Postgres and into JS dwarfs the aggregation and the Arrow build; halving the payload (the md5 key) barely moved it. The remaining lever is the dynamic grain. ### The dynamic variant The static grain means an unflagged dimension is never pivotable and every flagged one is paid for even when the pivot uses none of them. The additive next step: group by the fields the current pivot uses — the viewer's `group_by`, `split_by` and `filter` intersected with the eligible set — and re-fetch when they change. That needs the grain passed per request rather than baked into `pf.sql`, with operation responses and undo keys computed at the loaded table's grain. **Grain drift.** Changing `in_grain` after a load needs Generate SQL and a page reload, since the loaded table's index and columns are fixed; until then undo's keys, from live col_meta, match nothing. --- ## The pivot ### Perspective, vendored Perspective 5.4.0, `@perspective-dev/*` — **not** FINOS `@finos/perspective` — built from a patched fork and committed as tarballs in `ui/vendor`, whose README covers the patches (column-axis expand/collapse, `split_by_depth`, depth applied on config update), provenance and rebuild. Bundled through the `/inline` entrypoints, **never a CDN**: the 4.x CDN bundle resolved its server WASM to an unversioned path and silently pulled whatever was newest. ### Layouts — `pf.layout` A named `ViewConfig` owned by an account. **Published** — everyone on the forecast lists and can apply it, only owner or admin may change it — or **private**. Scope is the **version** by default, where people enter; `version_id IS NULL` applies to every version of the source. `is_default` (one per scope, published only) is restored on a first load, version-scoped beating source-wide — the narrower answer is the one somebody chose for this forecast. With no default: `group_by: ['pf_iter']` over the first value column. Load order: this version's last-used unnamed config in `localStorage` (`pf_layout_v{vid}`) if any — per-browser continuity, not something anyone names or shares — else the default. The old named list (`pf_layouts_v{vid}`) is lifted into `pf.layout` as private rows once and cleared. **Permissions are the `pf.log` rule** and `can_edit` rides on every row so the **Layout ▾** menu greys the rest. Applying is never restricted: the guarantee is that a published layout cannot be *changed* under people, not that it cannot be adapted — Save is withheld on others' layouts, Save as… forks. **No territory clause**: layouts are display config and territory restricts rows. `cleanLayout()` gates every stored config before `restore()`, which is all-or-nothing: it drops unknown columns from the axes and `aggregates` (including a multi-arg weight column) entry by entry rather than dropping the layout, and strips `theme`, the dead ordering expressions and `pf_totals`. **The dirty dot is a comparison, not a flag.** `restore()` itself fires `perspective-config-update`, so a flag set there would light the moment a layout was applied. `sameConfig()` compares against `activeConfigRef`, ignoring `table` (per-load) and `theme`. **Grand totals ride in the layout as `pf_totals: { row, col }`** — the Total toggles. The engine has no "subtotals but no grand total" mode, so both are hidden in the grid's DOM; `captureConfig()` adds the key and `cleanLayout()` strips it. Without it: row shown, column hidden. **What this replaced.** Named layouts lived only in `localStorage`, invisible to others and gone on another machine. The one server-side layout was `pf.source.default_layout`, an anonymous blob `PUT /sources/:id/default-layout` let *any* account overwrite for *every* account — a published layout with no owner. The route is gone; the column is migrated and unread. ### Axis depth Both axes collapse by a **depth in `ViewConfig`** set through `restore()` — `group_by_depth` (`applyDepth()`) and `split_by_depth` (`applySplitDepth()`). In the config it survives every rebuild, rides into layouts via `viewer.save()`, and needs nothing re-applied. Each axis is a **stepper** from `DepthButtons.jsx` — `◂ level ▸`, the middle opening the level list for a jump — three controls whatever the depth. It was one button per level, growing with the pivot, the current one a blue among greys so reading it meant scanning; before that a fixed `0 1 2 3` that named nothing and offered levels a two-field `group_by` lacks. It shows an em dash until a layout has been read back, and its middle is sized to the longest level so the toolbar holds still. **Depths are 1-based counts of levels to show**, on the stepper and in the config, so the value crosses unchanged; `server.cpp` sets `set_depth(n - 1)` and `t_traversal::set_depth` adds 1 back. The toolbar used to send `d + 1` and subtract it on read-back — self-consistent, and one level off the grid everywhere. The `+1` dodged depth 0, which reaches the engine as `set_depth(-1)` on an unsigned `t_depth`, underflows and clamps to fully expanded. `applyDepth` now floors at 1, so the **Total slot still shows the first level, not a bare total** — a wart kept rather than traded for the underflow. A real total needs `group_rollup_mode: 'total'`, unverified on this engine. **This required patching the engine.** `ViewConfig::apply_update` applied ten fields and neither depth, so a depth through `restore()` was accepted and dropped — and `Session::update_view_config` returns early when nothing changed, so no view was rebuilt. `ui/vendor/0001-apply-depth-fields-on-config-update.patch`. **What this replaced:** - Row depth was imperative — `view.set_depth()` on an object the viewer discards whenever it re-renders — so restoring it meant guessing when that happened: an `observerShim.js` patching `IntersectionObserver` and `ResizeObserver`, focus and visibility listeners, a retry loop for `getView()` throwing `No table set`, a flag for whether the viewer had "gone away". That produced three visible faults: the tree fully expanding on refocus, snapping on any reflow, and snapping when the settings sidebar opened. - Column collapse restored a **truncated `split_by`**, so discarded levels were remembered separately (`splitFull`, persisted as `split_full`), our collapse had to be told apart from the user rearranging, and the selection was cleared on every collapse. `split_full` and `expand_depth` are still read from old layouts. **Depth changes rebuild the view** — `restore()` makes the session tear down and rebuild it, a full traversal, where a manual collapse mutates in place. Noticeable on a large grain. Detecting a depth-only change and calling `view.set_depth()` while still writing the config would fix it, at the risk of the two drifting. **Per-node expansion is not solved.** It is view state with no config form, and the API is set-only (`expand(row_index)` / `collapse(row_index)`, no getter), so it cannot be captured and replayed and is lost on every rebuild. The honest route is a `ViewConfig` field of expanded row *paths* (indices shift as the tree opens), applied in `server.cpp` beside the depths — needing node enumeration in the C++ traversal, a proto field and the apply step. `expand_column` / `collapse_column` share the limitation. And depth is whole-axis: Excel can collapse one year while the next stays open. ### Auto-pause is off `` auto-pauses by default — an `IntersectionObserver` plus `visibilitychange` — and pausing **deletes the view**, so returning to the tab means `restore_and_render()`: a fresh traversal of the whole grain, a multi-second chug per tab switch on a large one, losing per-node expansion. `initViewer()` calls `viewer.setAutoPause(false)` after `load()`. Nothing updates the table while hidden — every operation is driven from this page — so the pause bought nothing. ### Toolbar Two zones. Left, **view**: `Layout ▾` and the Rows and Columns steppers with their Total chips — all `ViewConfig`, nothing reaching the server. Right, **forecast**: Refresh data, Change log, Bridge, Operations — re-running the load, opening something over the pivot, or writing. The distance between them is the separation. Each was once an outlined box too, the view one tinted too faintly to see and, in dark mode, reading as a sibling; before that everything was one row of identical hairlines, making a column-width control and the version's history look like peers. Three weights do the separating: `.pf-section-label` names a group and is heavier than its controls, `.pf-group-rule` divides groups within a zone, the gap divides zones. Each axis is one joined control, `◂ | level | ▸`, then its Total chip. **Blue is for actions, pressed grey for state.** Save and Post are blue; the Total chips and an open Operations use `.pf-pressed`. Operations used to relabel itself "Hide panel" in a blue outline, looking like a second Save. **Fit widths is gone**, for three independent reasons: auto-size measures over the rows *in the viewport*, so anything below the fold never counted; `draw({invalid_columns})` passes an option `DrawOptions` lacks, and `draw()` is throttled, so a click during a pending draw cleared widths without re-measuring; and the `persistLayout` after it saved nothing, since `saveColumnSizes()` returns only overrides, which `resetAutoSize()` had emptied — and this fork discards `column_size_override` whenever `split_by` is set, which here is always. Recoverable from git if ever worth fixing. The operation panel docks **bottom**, **right** or **floats**, persisted locally; the toolbar toggle shows the selection count while it is shut. ### Calculating indicator **Calculating…** shows in the status bar while the pivot rebuilds. Perspective's beaker renders only while its settings panel is open, vanishing exactly when nothing else says to wait. A counter — rebuilds overlap — rises in `applyDepth`, `applySplitDepth` and `applyLayout`, falls after `viewer.flush()`, and travels as `pf-pivot-busy` on the window, like `pf-data-changed`. **Not** driven by `perspective-config-update`, the obvious hook: on a depth change over a large grain it arrives *after* the rebuild, seconds late, and the flush after it resolves at once. The cost is that sorting or filtering from the viewer's own chrome shows nothing, since the app is never told they began. ### Light and dark `ui/src/theme.jsx` holds a context stored as `pf_dark`, defaulting to `prefers-color-scheme`; the status-bar toggle sets `.dark` on ``. `index.css` defines properties under `:root` and `.dark`, with Tailwind overrides written as `.dark .bg-white { … }` — no Tailwind dark config. Dark uses Perspective's Pro Dark palette. The pivot takes a **partial** `viewer.restore({ theme: 'Pro Dark' | 'Pro Light' })` on load and on toggle. It used to be `setAttribute('theme', …)`, which does nothing in 5.4.0 — the theme is a `ViewerConfig` field — so the pivot stayed light whatever the toggle said. The other half is `cleanLayout()` dropping `theme`: `viewer.save()` includes it, so every layout reasserted its author's theme. It is the reader's choice, so a published layout cannot impose one. --- ## Access ### Authentication and sessions Everything under `/api` but `/login`, `/logout`, `/me` sits behind `requireAuth`; the React app mounts only with a session (`Gate` in `main.jsx`), since its effects call the API at once, and `auth.jsx` wraps `fetch` so a 401 returns to login. - **Accounts** — `pf.app_user`, scrypt hashes whose stored form carries its parameters, so they can be raised without invalidating rows. `./pf.sh add-user | passwd | list-users | disable-user | enable-user`; passwords go in on stdin through `node lib/auth.js hash`, never argv or psql. - **Sessions** — `connect-pg-simple` in `pf.session`, so restarts keep people signed in and a session is revoked by deleting its row (`disable-user` does). Cookie `pf.sid`: httpOnly, SameSite=Lax, Secure unless `COOKIE_SECURE=false`, 12h rolling. - **Config** — the server exits without `SESSION_SECRET` rather than using a guessable default. `TRUST_PROXY` (default 1) keeps `req.ip` and secure cookies right behind the TLS proxy. CORS is enabled only by `CORS_ORIGIN`, by name — a wildcard origin plus a session cookie is CSRF by construction. - **Login hardening** — ten failures per IP per fifteen minutes (in memory), one message for unknown/wrong/disabled, session id regenerated on success. **Identity is server-side.** `pf_user`, `created_by`, `closed_by` come from `sessionUser(req)`, never the body. The UI used to send a hardcoded `pf_user: 'admin'`, which any client could set to anything. ### Setup and Baseline are admin-only to write `requireAdmin` sat exported in `lib/auth.js` and applied to nothing, so every non-admin could call every Setup and Baseline endpoint. It now guards source writes (register, cols, Generate SQL, member refresh, deregister), version writes (create, update, close, reopen, delete), loads (baseline, reference, segment edit, clear) and every plan write. `is_default` is further admin-checked inside `PUT /versions/:id`. **Not gated, each for a reason.** No read, including the table browser — the pages stay visible, and gating what a page needs to *render* turns it into a 403. Not the forecast operations, which carry territory and ownership rules. Not `PATCH /versions/:id/tags`, whose permission is ownership. **The pages go read-only rather than hidden**: one `
` per page instead of a flag threaded through every control — Baseline alone has dozens, and the one missed would be the one that mattered — plus a `ReadOnlyNotice` banner. Setup wraps a real box because the fieldset *is* its flex row; Baseline uses `display: contents` so its layout is untouched. **None of this is the permission**; it only stops offering buttons that would answer 403. **A 403 used to blank the whole app.** Setup put responses straight into state and `.map`ped them; an error body is `{error}`, and throwing in render unmounts React. Its list fetches now coerce to arrays, so a bad response costs an empty panel, not the session. The fragility predated the gating, which only made it reachable. ### Territory scoping An account sees and changes only its territory: `pf.app_user.territory` (jsonb list), `is_admin` for the unrestricted, and `col_meta.is_territory` naming which column of a source the values belong to — flagged, not coded, so another source can be divided by something other than a sales rep. **Fail closed.** No territory and not admin means no rows: `buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an unflagged source, so a forgotten account sees nothing rather than the book. **Built from the session, never the request** — the difference from `scope`, which the browser rightly sends as a filter the user chose. A permission cannot come from the thing it restrains, so the predicate is ANDed on last, in `sliceUnits()` for writes and per route for reads. Enforced at: - `/data` — cursor *and* the `X-Row-Count` count - `/agg` — `{{territory_clause}}`, before the GROUP BY - `/versions/:id/table-info` — unscoped, it told an account its load was fetching the whole table - every operation (`pf.log.env` records the territory in force) - `/sources/:id/values/:col` — it reads the *source*, untouched by any scope - recode's `set` — a scoped account cannot set the territory column. That is reassignment, not forecasting, and the row would vanish from the view that would have shown what happened. **Ownership, not territory**, governs `DELETE /log/:logid`, `PATCH /log/:logid` and `PATCH /versions/:id/tags`. The log PATCH looks like a private annotation and is not: `label` and `bucket` name the pivot's columns for everyone, so unguarded it let any account rename the company's segments, including on loads whose rows it cannot see. Known gaps, verified in code: - **Completion now uses unscoped routes.** Recode and clone complete from `GET /versions/:id/values/:col` (the version's own table, cached in memory keyed on the latest logid), the member list from `/sources/:id/dim/:group`, sibling autofill from `/sources/:id/lookup`. None applies the territory clause. - **Territory is read onto the session at login**, so a change reaches a signed-in account only when it signs in again. Re-reading it in `requireAuth` would also make disabling immediate. `./pf.sh set-territory | set-admin | orphan-territory`; the last lists values in the data that no account owns — work under one is invisible to all but admins, which a typo causes easily and nothing in the app reveals. --- ## Bridge and change log ### Bridge A waterfall of *how this forecast got from where it started to where it stands*, one step per initiative tag. `BridgeView.jsx`; the arithmetic is in exported pure `buildSteps()` / `layoutSteps()`, checkable against real data without a browser. **Computed from the table already in the browser**, not `GET /versions/:id/bridge`, so it reconciles with the pivot. The endpoint remains for API consumers, grouping by tag over rows outside `exclude_iters`, with no notion of buckets. **Scope** is the selection (the union of its slices, an operation's reach) or the pivot's filter, following the live selection until a scope button is picked. There is **no whole-version scope**: the filter is part of what the figures mean, and dropping it moves one end of the walk and not the other — rows outside a season filter sat wholly in the forecast loads with no counterpart in the plan, inflating the endpoint without explanation. View filters are AND-only, so selection scope builds a view per slice, each with the pivot's filter. **Membership comes from `pf_bucket`, not `pf_iter`**: open orders load as reference and are still forecast, and an iter-keyed bridge dropped them. The forecast bucket is the version's `adjustment_bucket` — the value unbucketed adjustments carry, so the two agree by construction. It used to be the literal `'Forecast'`; once buckets carried their sort prefix nothing matched, every row became comparison, and the basis cancelled itself to zero. With no match it falls back to whichever bucket holds the adjustments. Selection scope does not filter on `pf_bucket`/`pf_segment` even when a slice names one, since Compare to needs rows from every bucket. **Compare to** sets a bucket as basis — Plan when building a forecast, Prior Year when building the plan — resting on Forecast − Basis = (Forecast loads − Basis) + adjustments The opening step is the forecast's loads against the basis and the tagged adjustments explain the rest: no residual, no plug, the bars summing to the endpoint by construction. With no basis it shows the forecast's composition. **Steps group by tag alone**, case-insensitively, showing the commonest spelling. They grouped by label first once, to match `pf_segment`, but every adjustment carries the same default label, so no tag ever showed — the label is the segment, the tag the initiative. Untagged adjustments collapse to one "N untagged adjustments" step, as in the ledger, rather than a wall of identical bars. **Colour encodes polarity**: increase and decrease as two poles of a validated blue/red diverging pair (green/red is the classic colour-vision failure), neutral grey anchors. Everything not a bar follows the theme via CSS variables set through `style`, where `var()` is reliable. Every bar is labelled, a table view gives full precision, and the panel can be dragged. Known issues: - **Overlapping slices are deduplicated by `pf_id`**, which grain rows lack, so in grain mode a row matching two selected slices counts twice; it should key on `pf_gkey`. - **Selection scope keeps only dimension and date keys**, dropping the `pf_iter` / `pf_note` / `pf_op` / `pf_user` / `pf_logid` keys operations honour, so it bridges a wider set than the operation would write; a total-cell `{}` slice is skipped outright. - **No drill-down** from a step to its entries. ### Change log A modal of the version's entries, newest first — time, operation, slice, tag, note, rows and totals, author — with undo per row and tag and note edited inline (Enter saves, Esc cancels). `PATCH /log/:logid` branches on whether a field was sent, so `""` clears rather than meaning "leave alone" — except `label` and `bucket`, renamed from the Baseline page and never cleared. A saved tag regroups the ledger and bridge at once, so history can be reclassified. Multi-slice entries list a line per slice, pulling out shared fields when only one dimension varies (a dragged region); `{}` reads "Total within the pivot filter". An entry with stored SQL opens its debug record — intent, environment, executed statement. **It shows an entry's full impact**, not the reader's share: stamped totals are company-wide, so an admin's version-wide scale reads the same everywhere — deliberately, and labelled, rather than re-aggregated per territory. **Everyone / Mine.** Everyone's entries are listed, with the reader's own marked in place — a blue rule down the row and a blue `By` cell. With several active accounts, finding your own meant reading `By` down the list. Marked *and* filterable: the mark tells without hiding anybody, the filter is for wanting only yours. It opens on Mine every time, unpersisted — the log is almost always opened to check your own recent work, and a stale "Everyone" is the worse default. `isMine()` is strict authorship, not `canUndo()`: an admin can undo everything, and marking every row as theirs would say nothing. The fetch stays unfiltered, since the control's counts need the whole set. Rows the reader cannot change are greyed rather than offering a 403. `?kind=adjustments` drops load entries, which own nearly every row, from the recount join. Known issue: the log does not show an entry's id, so there is no way to name one when asking about it. --- ## Proposed: additional measures Status: **proposed — not built.** Nothing below exists in code yet. ### The problem The app forecasts exactly one dollar measure (`role = value`) and one quantity (`role = units`). `col_meta` lets a second column be marked `value`, but only the loads carry it (`generateSQL` in `lib/sql_generator.js`: "the adjustment operations are single-measure … and use only the first of each"). So a second measure is right on the baseline and wrong everywhere an adjustment has been made: - scale, recode and clone write adjustment rows with that column **null** - a recode moves the sales and leaves the cost behind - in grain mode `/agg` sums only the first value and the units column, so it is not in the pivot at all Extended cost is the first case wanted — margin is sales minus cost — and the same shape recurs for rebates, surcharges, freight and the like. ### The model: every extra measure is a rate on a driver An extra measure is almost never forecast on its own. It is a rate applied to something already being forecast: | Measure | Driver | Rate held by an ordinary edit | |--------------------|--------|-----------------------------------| | Extended cost | qty | unit cost | | Surcharge per unit | qty | surcharge per unit | | Rebate | sales | rebate as a % of sales | | Freight (fixed) | none | — carried, not moved by scale | Declared once, in Setup: `col_meta.role = 'measure'`, plus a new `col_meta.driver` of `units` | `value` | `none`. The existing `value` and `units` roles are unchanged and remain the two the ledger edits. ### How each operation treats a measure The rule is that **an edit never changes a measure's rate unless the rate is what is being edited** — the same rule the scale already applies to price. | Operation | Measure driven by qty | by sales | none | |--------------------|---------------------------------|--------------------------------|----------------------| | Load | copied | copied | copied | | Scale | moves by the qty factor | moves by the dollar factor | untouched | | Recode | both legs, like the other cols | both legs | both legs | | Clone | × scale factor | × scale factor | × scale factor | | Undo | removed with its rows | removed with its rows | removed | So a price-only edit moves no qty-driven cost (qty did not move) but does move a sales-driven rebate (dollars did). The scale SQL already prorates value and units by one factor per coordinate; each measure is prorated by its driver's factor the same way, per row, so the rate holds per row and not only in total. Because scale moves a measure at the slice's **current** rate, a rate set earlier is carried forward by every later edit, which is what makes Set rate (below) stick. ### Setting a rate: its own operation, not the ledger The ledger stays three boxes — value, qty, price — and the measures stay out of it. The ledger is already the hardest part of the panel to read, and a rate is a different question from a price-or-volume edit. The case that needs it: a part used as a proxy for a new part not yet set up, where the new part's cost is roughly known. The workflow is clone or recode the proxy's volume onto the new part, then set the new part's cost. Set rate · measure ext_cost · per qty · new rate 0.31 - Writes adjustment rows that change **only that measure**: the slice's cost goes to rate × its qty; sales and qty do not move. The same shape as a price edit, on a different measure. - A sales-driven measure takes its rate as a % of sales. - A `none` measure takes a plain target or change amount. - One `pf.log` entry, `operation = 'rate'`, `params` recording the measure, the rate and the resolved change; undoable, listed in the change log, and a line in the bridge like any other adjustment. - Measured against the same basis choice as scale (adjustable rows or the selected total), with the same territory and scope rules. ### In the pivot Each measure is summed like sales and qty — `/agg` and the `/data` cursor carry every `measure` column. Rates and margins are not stored: they are Perspective expression columns computed from the sums (unit cost = cost / qty, rebate % = rebate / sales, margin = sales − cost, margin % = margin / sales), so they are right at every level of the tree. ### What has to change - `col_meta`: `role = 'measure'`, new `driver` column; Setup's editor offers both. - `lib/sql_generator.js`: loads and `/agg` carry every measure; scale prorates each by its driver's factor; recode and clone carry them; a new `rate` template. - `routes/operations.js`: a `rate` route; scale passes the factors through. - `OperationPanel.jsx`: a Set rate form beside Scale / Recode / Clone. - Existing versions: the forecast table needs the measure columns, so a version created before a measure was added needs them added (or recreating) — the same col_meta drift noted under §Open questions. ### Open questions - Should a measure be allowed more than one driver (a surcharge that is part per-unit, part percentage)? Proposed: no — split it into two measures. - Does Set rate on a slice with no qty (a qty-driven measure) refuse, as the price hold does on a slice with no value? Proposed: yes. --- ## Open questions - **Replay** — re-running a version's adjustments against a restated baseline. Not built: no route takes `replay` and nothing returns 501. The hard part is the one cloning a definition declines: a scale's target was sized against the rows that were there, so a replay must choose between the *figure* and the *intent* (`params` holds both inputs and `resolved`). `env`'s template fingerprint exists partly so a replay can tell an entry ran under another template. - **Approval workflow** — a submission an admin approves before others see it. Deferred. - **Export** — CSV, or a push to a reporting table. The bridge's table view is a partial stand-in; there is no download. Stored names (§Naming) mean a reporting query needs only a join to `pf.log`. - **Cross-version comparison** — side by side, or a bridge between versions; the isolated tables make it a UNION. Compare to answers the within-version form (against a plan bucket loaded into the same version); across versions is open. - **Bridge drill-down** — a step listing its entries, or selecting them back in the pivot. - **Multi-connection support** — one database via `.env` today. The vision: a `pf.connection` table (host, port, dbname, user, password as an env-var reference), `connection_id` on `pf.source`, per-connection pools; `pf` stays on a home connection and sources live anywhere, with a Connections panel in Setup. It changes the source schema, so it wants a clean install. - **Col_meta / version schema drift** — roles changed after a forecast table exists put the generated SQL and the table's columns out of sync, unnoticed. Setup should compare col_meta with each table's columns, warn, and offer to rebuild (recreate the table, keep the version and log). Today: delete and recreate the version. Additional measures would hit this on every version. - **Separating `is_key`'s meanings** — §Data model → `pf.col_meta`. - **A per-source default layout** beyond the `pf_iter` fallback, built from col_meta when nothing is published. - **Dynamic grain** — §Display-grain pre-aggregation. - **Per-node expansion that survives a rebuild** — §The pivot → Axis depth. - **Tests** — SQL generation is token substitution against append-only tables, exercised only by hand.