pf_app/CLAUDE.md

# Pivot Forecast — CLAUDE.md

## What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec: `pf_spec.md`
Data transport architecture options: `pf_perspective_options.md`

---

## Tech stack

- **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`

---

## Project layout

```
server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation
  versions.js           Version CRUD, baseline/reference load, data stream, tag completions
  operations.js         scale, recode, clone, undo — the core forecast ops
  log.js                GET /api/versions/:id/log, DELETE /api/log/:logid
  layouts.js            Named pivot layouts — list per version, create, patch, delete
  plans.js              Planned load segments — list, create, patch, delete
lib/
  sql_generator.js      buildFilterClause, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
  utils.js
setup_sql/
  01_schema.sql         pf schema DDL — run once to install
  02_auth.sql           pf.app_user + pf.session
ui/src/
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    DepthButtons.jsx    One axis's collapse control; both axes are drawn from it
    LayoutMenu.jsx      The Layout ▾ control — Published / Mine, with the write actions
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         3-step collapsible nav
    StatusBar.jsx       Source · version · write target · row counts · theme
    Timeline.jsx        Date-range preview bar for baseline segments
```

---

## Database schema (`pf`)

- **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below); `is_key` is described under §`is_key` below
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it.
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.layout`** — named Perspective view configs; see §Pivot layouts
- **`pf.load_plan`** — load segment definitions, before and apart from having been run; see §Cloning a version's definition
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`
- **`pf.session`** — express-session store (connect-pg-simple layout)
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month

### `is_key`

Read in four places, and **not** the one it sounds like — slices are validated
against `filterCols`, which is every `dimension` plus every `date` column
regardless of `is_key`.

1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes
   `members.find(c => c.is_key)`, the column every other member is keyed on for
   `pf.dim_member`
2. **Value completion** — only `is_key` columns get a dropdown, and
   `/sources/:id/values/:col` refuses anything else
3. **Sibling autofill** — fires on blur only when `is_key && dim_group`
4. **The `dim_period` anchor** — `role === 'date' && is_key && dim_group` picks the
   date whose siblings are derived from the calendar

Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group.
**When a group has more than one, `.find()` silently takes the lowest `opos`.**
That is not hypothetical: `segment_new` (opos 12) outranked `part` (opos 15) in
the `part` group, so a refresh keyed on a column that is null throughout, matched
nothing, and reported success with zero members. `customer`'s group has four keys
and picks the right one only by `opos` luck.

The two meanings want separating — a per-group key choice, or a rule that the
group key is the column named by the group (which these groups nearly follow
already, except `sdate` → `sdate_e`). Until then, a refresh that finds two keys
should refuse rather than guess.

### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`

---

## Core data flow

### Initial load (Forecast view)
`Forecast.jsx` fetches col_meta first, then picks the endpoint:

- **grain mode** (any `in_grain` column) — `GET /api/versions/:id/agg`, rows pre-aggregated to the grain, table indexed on `pf_gkey`
- **raw mode** (no grain) — `GET /api/versions/:id/data`, raw forecast rows, table indexed on `pf_id`

Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM. `fetchArrow()` handles both.

**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.

### Display grain
Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings.

### Segment and note columns
`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table
does not itself carry:

- **`pf_segment`** — `pf.log.label`, else `tag`, else `note`, else `Unlabeled`;
  `'99 - Adjustments'` for an adjustment that has no label of its own
- **`pf_note`** — the free text on a scale/recode/clone; null on loads

They are deliberately separate: commingling a segment name with an adjustment note
makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join
adds no rows. The operation routes stamp the same two fields onto the rows they push
back incrementally, since those come from `RETURNING *` and would otherwise arrive
without them.

### Column order is stored text

Perspective orders column groups by the value string, and `SortDir`'s `col asc` /
`col desc` only reverses that — so Prior Year → Plan → Actual → Forecast is
alphabetical in neither direction and expressible as neither. A `"01 - "` prefix
is the only lever, and it lives in **`pf.log.label`** (and `pf.log.bucket`),
typed by whoever names the segment. Nothing derives it.

`SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are the
single definition, shared with the `/data` cursor in `routes/operations.js` —
`/agg` is generated, `/data` is not, and the two have to agree.

The one hardcoded ordinal is `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`, which
keeps unlabelled adjustments after every *numbered* segment. That proviso is the
whole scheme, not a caveat on it: ordering is string ordering, so `99` only lands
last once the loads carry `01`–`0n`, and an unnumbered segment sorts after it
(digits precede letters — `9` is `0x39`, `A` is `0x41`). The old `'(adjustment)'`
sorted *first* for the same reason read the other way, `(` being `0x28`.
Unlabelled loads read plain `Unlabeled` and so land at the very end, which is
where a segment nobody has named belongs. Labelling an adjustment's own log row
overrides the fallback, which is how one kind of adjustment is split out from the
rest.

### Hardcoded display names

Every name the pivot can show that does not come from `pf.log`. If a segment or
bucket appears under a name nobody typed, it is one of these. All three are in
the `DISPLAY DEFAULTS` block at the top of `lib/sql_generator.js`, exported so
the `/data` cursor and the operation routes' incremental row stamps use the same
values the generated `/agg` does.

| constant | `pf.version` column | built-in | applies to |
|---|---|---|---|
| `ADJUSTMENT_SEGMENT` | `adjustment_segment` | `99 - Adjustments` | `pf_segment` for a scale/recode/clone with no `label` |
| `ADJUSTMENT_BUCKET` | `adjustment_bucket` | `04 - Forecast` | `pf_bucket` for a scale/recode/clone with no `bucket` |
| `UNLABELED_LOAD` | `unlabeled_load` | `Unlabeled` | `pf_segment` and `pf_bucket` for a load with no `label`, `tag` or `note` |

Each is set per scenario on the Baseline page, under **Fallback names**; blank
falls back to the built-in. Anything typed on the log row overrides both, so
none of these appears once a segment is named.

**Why the join rather than a token.** `pf.sql` is keyed on
`(source_id, operation)` — one template shared by every version of a source — so
a value baked in at Generate SQL time could not vary by version, and
regenerating for one version would silently change the others. The names are
therefore read through `VERSION_JOIN` at query time, which also means changing
one takes effect on the next load with nothing regenerated.

The built-ins are still a convention guess: `ADJUSTMENT_BUCKET`'s `04 - ` only
suits one numbering. A version that numbers its buckets differently sets its
own rather than inheriting that.

**What this replaced.** The prefix used to be computed client-side, as
Perspective expression columns (`pf_bucket_ord`, `pf_segment_ord`) built from
`pf.log.seq` and `pf.version.bucket_order`. It ordered the pivot and nothing
else, so every other reader disagreed with it; `restore()` replaces
`expressions` wholesale, so it had to be re-applied after every layout load; and
ExprTK's string scanner tests each *byte* with `isprint()`, so a label
containing anything outside printable ASCII could not be ordered at all (`·` is
two bytes, of which `isprint(0xC2)` is false). `DEAD_ORDER_EXPRS` in
`Forecast.jsx` strips the expression names out of layouts saved under that
scheme. `pf.log.seq` and `pf.version.bucket_order` are no longer read; the
columns remain.

Relabelling now needs a page reload to show, because the label is part of the
aggregated row rather than something the pivot can re-derive. Editing a label
afterwards is a `PATCH /api/log/:logid` and needs nothing regenerated, but a
source registered before this change needs **Generate SQL** run once, so its
stored load templates write `label` and `bucket` onto the log row at all.

### Forecast operations
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums.

### Price or volume (`plug`)

**Price does not move unless it is named.** That is the rule the whole section
implements, and it is a rule because a price move is invisible in the figure
being edited: the dollars land on target whether volume followed or not, and
only the implied price says which happened.

A sales figure alone does not say which of price or volume moved, so scale takes
`plug` — `'volume'` (the default: price holds, units scale in proportion) or
`'price'` (volume holds, price moves). Resolved in `resolveIncrs()` in
`routes/operations.js`; the panel only *offers* it when the edit is dollars-only,
because naming units or price has already answered it, but it stays on screen
greyed out in the other cases rather than vanishing. The semantics come from the
predecessor Excel model, `/opt/forecast_api/VBA/fpvt.frm` → `calc_val` /
`calc_price`:

```
plug volume:  pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price:   fVol = pVol + bVol
```

**A qty input follows the same rule.** With dollars unnamed they follow at the
current price rather than holding — `value = (curValue / curUnits) * unitsIncr`.
Holding them made every qty edit a price edit: 10% more cases at the same revenue
is a 9% price cut nobody asked for. A slice with no units has no price to hold,
so there the dollars do stay put and the edit is a pure qty change.

The two deliberate ways to move price, both of which outrank the above:

- **name the dollars and the qty together** — price falls out of the two
- **name the price** (`target_price`), which is the Excel form's Edit Price mode:
  with a `target_units` alongside, both are inputs and dollars fall out

The ledger's **Result** line previews value, units and price together using the
same rules, so what you see is what gets written, and the column that cannot move
is labelled `· holds`.

**What this replaced.** `plug` defaulted to `'price'` — volume holds, price
absorbs — because that is what the API did before the flag existed. So an
unqualified dollar edit moved price, and a qty edit moved price, and the only
edit that did not was one where somebody had picked `'volume'` by hand. The
client's stored preference is keyed `pf_scale_plug_v2` for that reason: a browser
carrying the old `'price'` choice would have gone on silently moving price.

### Undo
`DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload.

---

## Axis depth (collapse / expand)

Both axes collapse the same way: a **depth in `ViewConfig`**, set through
`restore()`.

- **Rows** — `group_by_depth`, driven by the `Rows` buttons via `applyDepth()`
- **Columns** — `split_by_depth`, driven by the `Columns` buttons via
  `applySplitDepth()`

Both axes are drawn by `components/DepthButtons.jsx` from the axis's own field
list (`groupFull` / `splitFull`), as a **stepper**: `◂ partgroup ▸`, with the
middle button opening the full level list for a jump rather than a step.

Three controls per axis whatever the depth. It was one button per level, which
grows with the pivot — a 5-by-3 layout rendered ten of them — and the current
level was a blue button among the greys, so reading the state meant scanning.
Before that the row axis was a fixed `0 1 2 3`, which named nothing and offered
levels a two-field `group_by` does not have.

`depth` is null until the first layout has been read back; the stepper shows an
em dash rather than guessing a level.

Both are **1-based**: they count the levels to show, where the imperative
`view.set_depth()` counts the boundary below them. `server.cpp` does
`ctx1->set_depth(row_pivot_depth - 1)` and
`ctx2->set_depth(HEADER_COLUMN, column_pivot_depth - 1)`, so the toolbar sends
`d + 1` and subtracts one again when reading a layout back.

Because a depth lives in the config, it survives every view rebuild, rides into
the persisted and named layouts through `viewer.save()`, and needs nothing
re-applied afterwards.

**This required patching the engine.** `ViewConfig::apply_update` applied ten
fields and neither depth, so a depth sent through `restore()` was accepted,
deserialized and dropped — and since `Session::update_view_config` returns early
when `apply_update` reports no change, no view was rebuilt at all. See
`ui/vendor/0001-apply-depth-fields-on-config-update.patch`; the vendored build
carries it.

### What this replaced

Worth knowing, because a lot of machinery existed to work around it and is now
gone:

- Row depth used to be imperative — `getView()` then `view.set_depth()` — which
  put it on an object the viewer discards whenever it re-renders. Restoring it
  meant guessing when that had happened: an `observerShim.js` patching
  `window.IntersectionObserver` and `window.ResizeObserver`, focus and
  visibility listeners, a retry loop for `getView()` throwing `No table set`
  while `getTable()` resolved, and a flag tracking whether the viewer had "gone
  away". That guesswork produced three distinct visible faults — the tree fully
  expanding on refocus, snapping on any reflow, and snapping when Perspective's
  settings sidebar opened.
- Column collapse used to restore a **truncated `split_by`**. The discarded
  levels therefore had to be remembered separately (`splitFull`, persisted as
  `split_full`), our own collapse had to be told apart from the user rearranging
  the pivot (`collapsingRef` plus a prefix test), and the selection was cleared
  on every collapse because the axis was changing shape.

`split_full` is still *read* on load, for layouts saved under the old scheme.

### Auto-pause is off

`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on
itself plus the document's `visibilitychange` drive `AutoPauseState::apply()`,
and pausing **deletes the view** (`session.set_pause(true)` →
`view_sub.take().delete()`). Returning to the tab is therefore not a redraw but
`restore_and_render()` — a fresh view and a fresh traversal of the whole grain,
which on a large one is a multi-second chug on every tab switch, and takes any
per-node expansion with it.

`initViewer()` calls `viewer.setAutoPause(false)` right after `viewer.load()`.
Nothing updates the table while the tab is hidden — every operation is driven
from this page — so the pause bought nothing and cost a rebuild. The view is now
held while the tab is backgrounded.

### Still not solved: per-node expansion

Expanding one specific branch is view state with no config representation, and
the API has **no getter**:

```
expand(row_index: number): Promise<number>
collapse(row_index: number): Promise<number>
```

It can be set but not read, so it cannot be captured and replayed — this is why
it is lost on every rebuild, and why no client-side fix has worked. The honest
route is a `ViewConfig` field carrying expanded row *paths* (indices shift as
the tree opens), applied in `server.cpp` where the depths are. Bigger than the
depth patch: it needs a way to enumerate expanded nodes in the C++ traversal, a
proto field, and the apply step.

The fork's own `header_click.ts` / `expand_column` / `collapse_column` is the
column-axis equivalent and has the same limitation.

**Limitation that remains either way:** depth is whole-axis. Excel can collapse
2025 while 2026 stays expanded; a depth collapses every group at that level
together. `columns` selects which *measures* appear, not individual split
combinations.


## Tag completions are per version

`GET /versions/:id/tags` — the tags already used on **this** version, with their
use counts, for completing the next adjustment. `Forecast.jsx` refetches on
`versionId` and after every tagged write.

It used to be `GET /sources/:id/tags`, joined through `pf.version` and grouped
across every version of the source. A tag names an initiative *inside* a
scenario, so that put last year's work in front of this year's and reported a
`uses` count earned on a version the tag did not belong to. The cost of the
narrower scope is that a fresh version completes from nothing until it has a tag
of its own — which is the honest state of it, since cloning a definition copies
no adjustments.

The `pf_bucket` datalist on the Baseline page was already per version
(`bucketsInUse`), so the two now agree.

**The list is managed on the Baseline page, not the Forecast one.** A tag is
not a row anywhere — only a value repeated across this version's `pf.log`
entries — so there is no `/tags/:tag` to address and no list to curate ahead of
use: a tag exists because somebody typed it on an adjustment.

`PATCH /versions/:id/tags` takes `{ from, to }` and rewrites every entry
carrying `from`. `to` naming a tag that already exists **merges** the two, which
is usually the reason to be there: the bridge groups on this column, so
`Pull Forward` and `Pulled Forward` are one initiative reported as two. An empty
`to` clears the tag. The Baseline panel confirms both before sending, since
neither is visible as a diff afterwards.

Permission is the `pf.log` rule applied to *every* entry the tag is on, not
most of them — a half-applied rename leaves the initiative split, which is the
state the rename was fixing. `GET /versions/:id/tags` therefore returns `owners`
and `can_edit`, and the panel greys the row out rather than offering a rename
that answers 403. Editing one entry's tag is still the change log on the
Forecast page; this is the whole-version view of the same column.

**Completion matches on words, not substrings.** `TagInput` in
`OperationPanel.jsx` replaces the native datalist on the tag field: both sides
are lowercased, stripped of punctuation, relieved of stopwords and lightly
stemmed, so `Pulled Forward` finds `Pull Forward` and `moved to different sku`
finds `moved to a different sku` — neither of which a substring match reaches,
one inserted word being enough to hide the tag that already exists. Typing
something that reduces to an existing tag without being it says so outright
(*"Already used as … — use that"*), because the ledger will otherwise report the
two as separate initiatives. Nothing is merged automatically; the suggestion is
offered and the typing stays yours. The change log's inline tag cell still uses
the plain datalist.

**Completion is otherwise the list and nothing else.** The operation panel used to
carry six quick-pick pills under the tag field as well. They wrapped to two rows
as soon as the tags were longer than a word, and the panel docked at the bottom
has a fixed height, so they pushed the form past its own border. The datalist
costs no space, is not capped at six, and now shows the tag alone — an option's
text content renders as a second column beside the value, so the use count was
widening every row of the pick list to carry a number nobody picks on. `uses`
and `last_used` still come back from the route; nothing reads them.

---

## Where a browser lands

`App.jsx` restores `pf_sourceId` / `pf_versionId` from `localStorage` and
re-checks both against the live list. What it falls back to when there is
nothing stored — a new account, a new machine, a cleared browser — is
**`pf.version.is_default`**, one per source by partial unique index
(`version_one_default`), the way `pf.layout.is_default` works.

It used to fall back to `versions[0]`, and the list is ordered
`created_at DESC`, so that meant *the newest version* — whatever somebody
created last, a scratch copy as readily as the plan. Accounts that never pick a
version themselves, which is every sales account, therefore followed the
forecasting work around.

**Admin only**, enforced in `PUT /api/versions/:id`: it decides where everyone
else lands, so it follows the `pf.log` rule rather than being the version's own
setting like its name. The checkbox on the Baseline page's **This forecast**
panel is greyed out for other accounts rather than offering a click that
answers 403. The `StatusBar` picker marks the default `(default)`.

**A stored selection still wins.** The default is the fallback, not an
override — someone who has been working in another version stays there. The
corollary is that an account which already landed in the wrong scenario keeps
landing there until it switches once.

Setting the default is two statements — clear the old, set the new — so the
route runs them in one transaction; the index refuses the instant in between.

## Renaming a version

Safe at any time, including on a closed one. Nothing derives from the name: the
forecast table is `fc_{tname}_{version_id}`, `pf.log` and `pf.layout` key on the
id, and the UI's `localStorage` keys are id-based. The Baseline page's **This
forecast** panel saves `name` and `description` on blur through
`PUT /api/versions/:id`, which has accepted both all along — it was only ever
called for the fallback names.

Two edges the panel handles: `(source_id, name)` is unique, so the route
answers 409 on a collision rather than a raw Postgres error; and the `UPDATE`
COALESCEs a falsy name back to the existing one, so a blank box is refused
client-side rather than looking saved and changing nothing.

---

## Cloning a version's definition

`POST /sources/:id/versions` takes `clone_from`. It copies the template
version's settings (`description`, `exclude_iters`, the three fallback names,
`bucket_order`), its **version-scoped** layouts, and its load segments — and
**runs nothing**. The reason to clone a definition is to change the dates before
anything loads, and a clone that fetched two million rows first would have to be
undone before it could be edited.

Not copied: adjustments, in any form. A scale's target is a figure sized against
the rows that were there, so replaying one against a different load is
arithmetic nobody asked for. Source-wide layouts (`version_id IS NULL`) are not
copied either — they already apply to the new version, and a copy would
duplicate the name.

### `pf.load_plan`

A segment used to exist only once it had been run: a `pf.log` entry with
forecast rows behind it. A plan is the definition on its own — `kind`, `label`,
`bucket`, `tag`, `note`, `filters`, `raw_where`, `date_offset`, `seq` — with
`run_logid` naming the `pf.log` entry it produced, or null while it is still a
plan.

**Deliberately not `pf.log`.** Every row there means a write happened; holding
unexecuted intent in the audit log would cost it that meaning, and the change
log would list entries that never ran.

`run_logid` is `ON DELETE SET NULL`, so undoing a load returns its plan to unrun
rather than deleting the definition with it. The plans list reads the row count
back through that FK, which is how it tells run from unrun.

**Derivation from an old version.** `cloneDefinition()` in `routes/versions.js`
reads `pf.load_plan` when the template has plans and falls back to deriving them
from its `baseline`/`reference` log entries when it does not — which is every
version built before plans existed. `pf.log.params` already stores `filters`,
`raw_where` and `date_offset`, so this is a read, not a reconstruction.

**Run all stops at the first failure.** The segments of a definition are not
independent — a baseline followed by references that date-shift off it — so
carrying on past a broken one builds something nobody described.

---

## Pivot layouts

A layout is a named `ViewConfig`, stored in **`pf.layout`** and owned by an
account. Two kinds, one table:

- **published** — everyone on the forecast lists it and can apply it; only its
  owner or an admin may change it
- **private** — yours, nobody else lists it

Scope is the **version** by default, because that is where people enter the app.
`version_id IS NULL` means the layout applies to every version of the source;
that is where the old source default went, and what a brand-new version picks up
before anyone has published anything for it. `is_default` (at most one per scope,
by partial unique index) is what `initViewer()` restores on a first load — a
version-scoped default beating a source-wide one, the narrower answer winning.

**Permissions are the `pf.log` rule verbatim** — your own, or an admin's
override, and the UI greys out the rest rather than offering a click that answers
403. `can_edit` rides on every row so the menu knows which. Applying is never
restricted: the guarantee is that a published layout cannot be *changed* out from
under people, not that it cannot be adapted — Save is withheld on a layout that
isn't yours, Save as… forks it into your own.

**No territory clause.** A layout is display config, and territory restricts rows,
not columns; `cleanLayout()` already drops anything the live schema lacks.

**What is still local.** `LAYOUT_KEY` (`pf_layout_v{vid}`) — the unnamed
last-used config — stays in `localStorage`, because it is per-browser session
continuity rather than a thing anyone names or shares. `LAYOUTS_KEY`
(`pf_layouts_v{vid}`) is the old named list; `loadLayouts()` lifts it into
`pf.layout` as private rows once per version and then clears the key.

**The dirty dot is a comparison, not a flag.** `restore()` itself fires
`perspective-config-update`, so anything set unconditionally in that handler
would light up the moment a layout was applied. `activeConfigRef` holds what the
pivot last matched and `sameConfig()` compares against it, ignoring `table`
(the per-load table name, different on every refresh).

**What this replaced.** Named layouts lived only in `localStorage` — invisible to
anyone else, gone on another machine. The one server-side layout was
`pf.source.default_layout`, a single anonymous blob that `PUT
/sources/:id/default-layout` let *any* account overwrite for *every* account: a
published layout with no owner. That route is gone; the column remains, migrated
and read by nothing.

`cleanLayout()` guards `aggregates` along with the axes — including the weight
column of the multi-arg form — and drops the offending *entry* rather than the
layout, since `restore()` is all-or-nothing.

---

## The Forecast toolbar is two zones

Left, on its own tinted ground: **view** — `Layout ▾`, the `Rows` and `Columns`
depth buttons, `Fit widths`. Nothing there reaches the server or changes the
forecast; it is all `ViewConfig` and `plugin_config` against rows already loaded.

Right: **forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. These
re-run the load, open something over the pivot, or write to the version.

They were one undifferentiated row separated by identical hairlines, which made
`Fit` and `Bridge` read as peers when one is a column width and the other is the
version's history. The split is the point; the tint is just what makes it
visible.

---

## Slice mechanics

When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).

Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow** — `{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.

Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".

**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.

`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.

**A blank dimension value is a value.** Both blanks occur in this data and they
are different rows — on `fc_osm_skinny_34`, `partgroup` is null on 51,964 rows
and `''` on 111,696 — so `null` and `''` stay distinct from the clicked cell
through to the WHERE clause, where one becomes `IS NULL` and the other `= ''`.
Only `undefined` means absent, which is how a subtotal row (a shorter row path)
and a collapsed column level still read as "not part of the slice".

They used to be dropped, and a dropped key does not narrow the write — it widens
it. Clicking the blank `partgroup` node under `ALTMAN PLANTS` selected the
customer's whole $27.6M rather than the $6.2M under the node clicked, and the
ledger agreed with it, because the panel totals through the same filter list.
The source of the drop is the **engine**: `getCellConfig` builds the click's
filter list with `pivot_value ? [pivot, '==', pivot_value] : undefined`, so the
blank is gone before `perspective-click` is dispatched — and being a falsy test
it takes a dimension whose value is `0` with it. `cellFilters()` in
`Forecast.jsx` therefore re-derives both axes from `detail.row.__ROW_PATH__` and
the clicked column name rather than trusting that list; the click and a dragged
region now go through the same derivation. The ledger filters a null with
`is null` rather than `== null`, since the engine matches on type and a null
threshold against a string column is the same dropped predicate again.

`buildWhere()` renders a null as `IS NULL`; it used to fall through `esc()` to
`= ''`, which selected *the other blank* — the slice named one node and the
operation wrote the other. In a list, the null arm is ORed on rather than folded
in, because `IN (NULL)` is never true.

The panel prints `(null)` and `(blank)` rather than nothing, and `—` only when
the column is genuinely absent from the slice: a slice isolated to the blank node
and one the column plays no part in are opposite statements and looked identical.

**A blank on the column axis cannot be isolated.** A column name is its values
joined with `|`, and a null joins as the empty string, so by the time the name
reaches the client the two blanks are the same text. Those cells are refused —
with a message, and counted once for a drag — rather than widened, since a slice
missing a split_by level means every value of it. Filtering on the column
instead (`scope`, which carries `is null`) is the way to reach them.

**The pivot's own filter travels alongside, as `scope`.** A slice is
`{col: value}` and can only mean equality, so a view narrowed to
`sseas_e <= 2027` has no slice form at all — the filter is read off
`viewer.save()` and sent as `[col, op, value]` triples, which is also what the
ledger totals against, so the panel and the write agree. `SCOPE_OPS` in
`lib/sql_generator.js` is the whole allowed list, under **Perspective's**
operator names; anything outside it is refused rather than dropped, since a
scope silently discarded is a write wider than the panel that authorised it.

The substring operators — `contains`, `begins with`, `ends with` and their
negations — map to **`ILIKE`**, not `LIKE`, because that is what the engine
does: `t_tscalar::contains` lowercases both sides before `std::string::find`.
On `partd`, `contains 'Keysto'` matches 16,864 rows case-insensitively and 35
case-sensitively, so `LIKE` would have written a fortieth of what the grid
showed. Narrower is no safer than wider — either way it is not the set the
ledger counted. The term is literal in the engine, so `%`, `_` and `\` are
escaped out of it before it becomes a pattern.

`matches` / `not matches` stay refused. They are RE2 in the engine, and
Postgres regexes are not RE2 — `\b` is a word boundary in one and a backspace
in the other — so a pattern accepted by both could quietly match a different
set of rows, which is worse than a 400.

Every operator that needs a term refuses an empty one: an unfinished filter row
in the pivot arrives as `[col, '==', null]`, and `ILIKE '%%'` or `= ''` would
have read as "every row" or "no rows" rather than as the mistake it is.

**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.

---

## Operation SQL patterns

All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.

**Every date column travels, not just the first.** `dataCols` is
`[...dims, ...dateCols, value, units]`. It used to be `dateCols[0]`, so on a
source with order, requested and ship dates, scale and recode wrote rows whose
`rdate` and `sdate_e` were **null** — and because the derived period dimensions
(`rseas`, `smon_e`) are dimensions and were copied with the rest, the labels
read correctly while the dates under them were gone. Both columns are
`role = 'date'` and therefore sliceable, so an adjustment scoped to a ship date
matched no adjustment row that had ever been written. Clone and the loads were
always right. The wider GROUP BY costs almost nothing on `osm_skinny`
(971,185 → 971,352 groups) because `ordnum`/`ordline` already make a row
near-unique.

- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched

`build_where()` validates every slice key against col_meta (only `role = dimension` allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.

---

## Authentication

Everything under `/api` except the auth routes sits behind a session; the React
app is only mounted once there is one (`Gate` in `main.jsx`), because its load
effects call the API immediately.

- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
  Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
  the password is read on stdin and hashed before it reaches psql.
- **Sessions:** `express-session` + `connect-pg-simple` in `pf.session`, so a
  restart doesn't sign everyone out and a session can be revoked by deleting its
  row (`disable-user` does exactly that). Cookie `pf.sid`: httpOnly, SameSite=Lax,
  Secure unless `COOKIE_SECURE=false`, 12h rolling.
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
  `TRUST_PROXY` (default 1) makes `req.ip` and secure-cookie detection correct
  behind the TLS proxy. `CORS_ORIGIN` is the only way CORS is enabled at all; a
  wildcard origin plus a session cookie would be cross-site request forgery by
  construction.
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
  minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
  regenerates the session id on success.

**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
`sessionUser(req)`, never from the request body — the UI used to send a hardcoded
`pf_user: 'admin'`, which any client could have set to anything. The audit log
now names the account that made the change.

## Source columns come from pg_catalog

`information_schema.columns` omits **materialized views** — they are not in the
SQL standard — and `gs.osm_skinny` is one. So the source the whole app is built
on looked like it had no columns: registering it seeded nothing, and creating a
version failed with "No usable columns in col_meta" while col_meta plainly held
thirty-six.

`RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version
creation, source registration and the table preview. It returns the same shape
information_schema did, so `mapType` and the callers were unchanged:
`data_type` is `format_type` with the modifier stripped, which gives the same
spelling (`character varying`, `numeric`), and precision and scale are unpacked
from `atttypmod`. The table browser lists from `pg_class` by `relkind` for the
same reason.

## Territory scoping

An account sees and changes only its own territory. The list lives on
`pf.app_user.territory` (jsonb array) with `is_admin` for the accounts that see
everything, and `col_meta.is_territory` marks which column of a given source
the values belong to — one per source, flagged rather than named in code so a
second source can be divided by something other than a sales rep.

**Fail closed.** No territory and not an admin means no rows.
`buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an
unflagged source: an account somebody forgot to configure sees nothing instead
of the whole book.

**Built from the session, never the request.** This is the difference between
it and `scope`, which the browser sends and which is right to send, being a
filter the user chose. A permission cannot come from the thing it restrains, so
the territory predicate is ANDed on last, in `sliceUnits()` for writes and per
route for reads, where nothing in the payload can remove it.

Enforced at:

- `/data` — clause on the cursor *and* on the count behind `X-Row-Count`
- `/agg` — a `{{territory_clause}}` token applied **before** the GROUP BY, since
  the territory column need not be part of the grain and may not survive it
- every operation, through `sliceUnits()`
- `/sources/:id/values/:col` — completion reads the *source* table, which no
  scope has touched, so without it a dropdown enumerates the whole business
- `DELETE /log/:logid` and `PATCH /log/:logid` — by owner, not territory. Undo
  removes an entry's rows wholesale, and half-undoing one would leave a state
  nothing describes. The PATCH looks like a private annotation and is not:
  `label` and `bucket` name the pivot's columns for everyone in the version, so
  unguarded it let any account rename the company's segments. Your own entries,
  or an admin's override, and the UI greys out the rest rather than offering a
  click that answers 403.
- recode's `set` — a scoped account cannot set the territory column at all.
  Moving a row between territories is reassignment, not forecasting, and it
  would vanish from the view that would have shown what happened.

**The change log shows an entry's full impact**, not the reader's share. The
totals stamped on `pf.log` are company-wide, so an admin's version-wide scale
reads the same in every account — deliberate, and labelled, rather than
re-aggregating per territory.

Managed with `./pf.sh set-territory | set-admin | orphan-territory`.
`orphan-territory` lists values present in the data that no account owns; work
under one is invisible to everyone but an admin, which a typo causes easily and
nothing inside the app reveals.

## Light / dark mode

Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.

- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`

## After a change: restart, and Generate SQL

Two steps, and which one you need depends on what changed.

- **Restart the server** for anything in `routes/`, `lib/` or `server.js`.
- **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes.
  The templates are *stored* in `pf.sql`, so editing the generator changes
  nothing until they are rebuilt — and a template carrying a token the running
  code does not substitute fails at the database rather than in JS, which reads
  as an unrelated client-side error.

The order matters: restart first, then Generate SQL, or the old code writes the
templates.

Schema changes are applied to the live database directly and mirrored into
`setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a
migration runner, so running it is not how an existing database gets a new
column.

## Known issues / active work

- **Rows written before 2026-09-21 have null `rdate`/`sdate_e`.** The generator
  fix above changes what gets written from here on; it does not repair what is
  there. On `fc_osm_skinny_29` that is ~270k scale and recode rows. A backfill
  would have to re-derive each from the baseline rows at the same coordinate,
  which is possible but is inference, not recovery — deleting and replaying
  those log entries is the honest repair

- **Zero-row operations report success.** Scale refuses with "Nothing to
  scale…" when its slice matches nothing; recode and clone commit an empty log
  entry and return `rows_affected: 0`. A recode of a rep whose rows are all
  `reference` looked like it worked and did nothing
- **The change log does not show an entry's id**, so there is no way to name
  one when asking about it
- **Territory is read onto the session at login**, so granting or changing one
  does not reach a signed-in account until it signs in again. Re-reading it per
  request in `requireAuth` would also make disabling someone immediate
- **Depth buttons rebuild the view.** A depth lives in `ViewConfig`, so changing
  it goes through `restore()` and `Session::update_view_config` tears down and
  rebuilds the view — a full traversal — where a manual collapse mutates the
  existing one in place. Noticeable on a large grain. The fix is to detect a
  depth-only change and call `view.set_depth()` imperatively while still writing
  it to the config, at the cost of the two being able to drift

- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29`
  (2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in
  pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then
  ~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The
  remaining lever is **dynamic grain** — group by the fields the current pivot
  actually uses rather than every `in_grain` column; see `pf_spec.md` →
  §Display-grain pre-aggregation, "the dynamic variant"
- Per-node expand/collapse is lost whenever the view rebuilds — set-only API, no
  getter; see §Axis depth
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
- Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`,
  `App.jsx`). It is re-validated against the live list whenever that list changes, so a
  deregistered source or deleted version re-points at the first remaining one instead of
  leaving a dead id that 404s every call
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
- Grain drift: changing `in_grain` after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. `routes/log.js` derives the grain from live col_meta, so a grain changed mid-session yields `pf_gkeys` that don't match the loaded table and undo silently removes nothing
- Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see `pf_spec.md` → §Display-grain pre-aggregation

## Deferred (not in v1)
Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`.