# Pivot Forecast — CLAUDE.md

## What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec, including the rationale and history this file leaves out: `docs/pf_spec.md`
Data transport architecture options: `docs/pf_perspective_options.md`

---

## Tech stack

- **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `docs/PERSPECTIVE.md`.
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`

---

## Project layout

```
server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
pf.sh                   Account and territory admin (add-user, passwd, set-territory, …)
install.sh              Fresh install
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation;
                        GET /sources/:id/values/:col, /sources/:id/lookup,
                        /sources/:id/dim/:group (+ POST …/refresh), /dim-period/cols
  versions.js           Version CRUD and clone; GET /versions/:id/tags (+ PATCH),
                        /versions/:id/bridge, /versions/:id/values/:col, /versions/:id/table-info
  operations.js         Data streams (GET /versions/:id/data, /versions/:id/agg); loads
                        (baseline, reference); scale, recode, clone
  log.js                GET /api/versions/:id/log, /log/:logid/debug; DELETE and PATCH /log/:logid
  layouts.js            Named pivot layouts — list per version, create, patch, delete
  plans.js              Planned load segments — list, create, patch, delete
lib/
  sql_generator.js      grainOf, buildWhere/buildWhereAny, SCOPE_OPS, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, requireAdmin, sessionUser; `node lib/auth.js hash` CLI
  utils.js              RELATION_COLUMNS_SQL, mapType
setup_sql/
  01_schema.sql         pf schema DDL — idempotent, not a migration runner
  02_auth.sql           pf.app_user + pf.session
  gen_dim_period.sql    Populates pf.dim_period; fiscal year start month is configurable
ui/vendor/              Patched Perspective build and its patches (see its README)
ui/src/
  main.jsx              ThemeProvider, AuthProvider, Gate (mounts the app only once signed in)
  App.jsx               Page routing; source/version selection and its localStorage restore
  theme.jsx             ThemeContext / useTheme
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load, tags panel
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    DepthButtons.jsx    One axis's collapse stepper; both axes are drawn from it
    LayoutMenu.jsx      The Layout ▾ control — Published / Mine, with the write actions
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms; effectiveScaleInputs
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         Forecast list + the 3 steps; collapsible
    StatusBar.jsx       Forecast name · row count · calculating · theme; source on Setup only
    ReadOnlyNotice.jsx  The banner over a page a non-admin may read but not change
    Timeline.jsx        Date-range preview bar for baseline segments
```

---

## Database schema (`pf`)

- **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain**; `is_territory` marks the column territories are expressed in (one per source); `is_key` is described under §`is_key` below
- **`pf.dim_member`** — members of each `dim_group`, keyed on the group's key column; refreshed by `POST /sources/:id/dim/:group/refresh`
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations; `adjustment_segment` / `adjustment_bucket` name new adjustments; `is_default` (one per source)
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows. Tables created before that index was added do not have it.
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb; `label` / `bucket` required (`log_named` CHECK). Stamped at write time: `row_count`, `value_total`, `units_total`, `measure_cols` (which columns the totals are denominated in), `env` (the state the intent executed against), `sql_text` (the statement as executed)
- **`pf.layout`** — named Perspective view configs; see §Pivot layouts
- **`pf.load_plan`** — load segment definitions, before and apart from having been run; see §Cloning a version's definition
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`, `is_admin`, `territory` (jsonb array)
- **`pf.session`** — express-session store (connect-pg-simple layout)
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql`

Dead columns, kept but read by nothing: `pf.version.unlabeled_load`, `pf.version.bucket_order`, `pf.log.seq`, `pf.source.default_layout`.

### `is_key`

Read in four places, and **not** the one it sounds like — slices are validated
against `filterCols`, which is every `dimension` plus every `date` column
regardless of `is_key`.

1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes
   `members.find(c => c.is_key)`, the column every other member is keyed on for
   `pf.dim_member`
2. **Value completion** — only `is_key` columns get a dropdown, and
   `/sources/:id/values/:col` refuses anything else
3. **Sibling autofill** — fires on blur only when `is_key && dim_group`
4. **The `dim_period` anchor** — `role === 'date' && is_key && dim_group` picks the
   date whose siblings are derived from the calendar

Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group.
**When a group has more than one, `.find()` silently takes the lowest `opos`** —
a refresh keyed on the wrong column matches nothing and reports success with
zero members. Until the two meanings are separated, a refresh that finds two
keys should refuse rather than guess. Rationale: docs/pf_spec.md → §Data model.

### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{territory_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{label}}`, `{{bucket}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`

---

## Core data flow

### Initial load (Forecast view)
`Forecast.jsx` fetches col_meta first, then picks the endpoint (both in `routes/operations.js`):

- **grain mode** (any `in_grain` column) — `GET /api/versions/:id/agg`, rows pre-aggregated to the grain, table indexed on `pf_gkey`
- **raw mode** (no grain) — `GET /api/versions/:id/data`, raw forecast rows, table indexed on `pf_id`

Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM. `fetchArrow()` handles both.

**One batch, not streaming.** Per-batch Arrow encoding creates independent dictionaries that crash Perspective WASM on dictionary replacement messages, so the server accumulates all rows and emits one record batch. pg returns `bigint`/`numeric` as strings by default — the type parsers in `server.js` coerce them to numbers; do not remove them.

### Display grain
Aggregating to the grain the pivot displays is the load-time fix, and it keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. **`grainOf()` in `lib/sql_generator.js` is the single definition of the grain** — `routes/log.js` calls it directly and `Setup.jsx` mirrors it. Design: docs/pf_spec.md → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `docs/pf_perspective_options.md` → §Spike findings.

### Log-derived columns
`/data` and `/agg` both LEFT JOIN `pf.log` and emit four columns the forecast
table does not itself carry:

- **`pf_segment`** — `pf.log.label` (`SEGMENT_EXPR`)
- **`pf_bucket`** — `pf.log.bucket` (`BUCKET_EXPR`)
- **`pf_note`** — tag, else note, on a scale/recode/clone; null on loads (`NOTE_EXPR`)
- **`pf_op`** — `pf.log.operation`

Only segment and bucket are stored names (see §Every log entry is named); the
note is derived. `SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in
`lib/sql_generator.js` are the single definition, shared with the `/data`
cursor in `routes/operations.js` — `/agg` is generated, `/data` is not, and the
two have to agree. `LABEL_GROUP_COLS` lists the `pf.log` columns they read, for
`/agg`'s GROUP BY. In grain mode `pf_logid` is part of the grain, so the join
adds no rows. The operation routes stamp the same fields onto the rows they push
back incrementally, since those come from `RETURNING *` and would otherwise
arrive without them.

### Column order is stored text

Perspective orders column groups by the value string, so a `"01 - "` prefix in
**`pf.log.label`** / **`pf.log.bucket`**, typed by whoever names the segment, is
the only lever; nothing derives it. `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`
lands last only when the loads carry `01`–`0n` prefixes (digits precede
letters). Do not reintroduce ordering as Perspective expressions — `restore()`
replaces `expressions` wholesale and ExprTK cannot handle non-ASCII labels;
`DEAD_ORDER_EXPRS` in `Forecast.jsx` strips the old ones from saved layouts.
Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.

### Every log entry is named

`pf.log.label` is a row's segment and `pf.log.bucket` what it counts toward, and
**every entry carries both** — the `log_named` CHECK refuses one without. A
report gets the pivot's segment and bucket by joining the forecast table to
`pf.log` on `pf_logid`; there is no rule to copy. Never derive them at read time.

They are stamped when the entry is written:

- **A load** — `label` is required (the routes answer 400 without one); `bucket`
  defaults to the label. `loadNames()` in `routes/operations.js`.
- **An adjustment** — scale, recode and clone have no name inputs, so they are
  stamped with the version's `adjustment_segment` / `adjustment_bucket`, else the
  built-ins `ADJUSTMENT_SEGMENT` = `99 - Adjustments` and `ADJUSTMENT_BUCKET` =
  `04 - Forecast` (`ADJUSTMENT DEFAULTS` in `lib/sql_generator.js`).
  `adjustmentNames()` resolves them once per request; they go into the template's
  INSERT as `{{label}}` / `{{bucket}}` and onto the rows pushed back to the pivot.

The two version defaults are set on the Baseline page under **Defaults for new
adjustments**. Changing one names the *next* adjustment; existing entries keep
what they were written with. Renaming past entries is `PATCH /api/log/:logid`,
one at a time, which renames but refuses to clear. A relabel needs a page
reload to show, since the label is part of the aggregated row. A source
registered before names were stored needs **Generate SQL** run once so its load
templates write `label` and `bucket` at all. Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.

### Forecast operations
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums.

### Price or volume (`plug`)

**Price does not move unless it is named.** A price move is invisible in the
figure being edited — the dollars land on target either way — so every default
holds price. All resolution is in `resolveIncrs()` in `routes/operations.js`;
the semantics come from the predecessor Excel model
(`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`).

- **Dollars only** — `plug` decides: `'volume'` (the default: price holds, units
  scale in proportion) or `'price'` (volume holds, price moves).
- **Qty only** — dollars follow at the current price. A slice with no units has
  no price to hold, so the dollars stay put.
- **Dollars and qty** — both land as named; price falls out.
- **A named price (`target_price`) holds**, and the other named measure decides
  the rest: price alone → qty holds; price + qty → qty lands as named, dollars
  follow; price + dollars → dollars land, qty follows. **All three is refused**
  server-side (400).
- **The held price is the adjustable rows' own**, never blended with the fixed
  (excluded-iter) rows, even under `target_basis: 'selected'` — the scale SQL
  prorates over the adjustable rows only, so a blended price drives qty and
  dollars apart (and qty negative when a slice is taken to its fixed total).

**The three boxes foot.** At most two are inputs and the third is derived.
`effectiveScaleInputs()` in `OperationPanel.jsx` is the single resolver: the
panel renders from it and `buildPayload` in `Forecast.jsx` builds the request
from it, so what is posted is what is shown. Typing a third drops one — the plug
decides when dollars came last, else the older of the other two. The ledger's
**Result** line previews value, units and price by the same rules, and the
column that cannot move is labelled `· holds`.

**The plug control is labelled "Moves"**: it lights whatever the edit moves
(both if both do) and takes a click only while it is the one deciding; otherwise
it stays on screen, not hidden.

**Logged.** `resolved.method` (`price` | `both` | `volume` | the plug) records
which side moved, since the request alone cannot say. `params` also keeps
`plug` and `expected` (what the Result line promised), beside `resolved`, so a
panel/server disagreement can be found by reading the log — nothing compares
them yet. The client's stored preference is keyed `pf_scale_plug_v2`; do not
reuse the old key, which could carry a stale `'price'`. Rationale: docs/pf_spec.md → §Operations.

### Undo
`DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload.

---

## Axis depth (collapse / expand)

Both axes collapse the same way: a **depth in `ViewConfig`**, set through
`restore()` — `group_by_depth` via `applyDepth()`, `split_by_depth` via
`applySplitDepth()`. Both steppers are drawn by `components/DepthButtons.jsx` from the axis's own field list
(`groupFull` / `splitFull`), as a stepper `◂ level ▸`, the middle button opening
the full level list. `depth` is null until the first layout has been read back;
the stepper then shows an em dash.

**Depths are 1-based counts of levels to show**, on both axes and in
`DepthButtons` (`levels[d - 1]`); the value crosses unchanged in each direction.
The engine agrees (`set_depth(n - 1)`, and the traversal adds the 1 back). **Never
send 0**: it reaches the engine as `set_depth(-1)` on an unsigned `t_depth`,
underflows, and renders fully expanded. `applyDepth` floors at 1, so the
stepper's "Total" slot shows the first level rather than a bare total — kept
deliberately. A real total would need `group_rollup_mode: 'total'`, unverified
against the local engine.

A depth lives in the config, so it survives view rebuilds and rides into saved
layouts through `viewer.save()`. **This required patching the engine** —
`ViewConfig::apply_update` dropped both depth fields; see
`ui/vendor/0001-apply-depth-fields-on-config-update.patch`. Do not go back to
imperative `view.set_depth()` alone: the viewer discards that view on every
re-render. `split_full` is still *read* on load, for layouts saved under the
old truncated-`split_by` scheme.

**Auto-pause is off.** `initViewer()` calls `viewer.setAutoPause(false)` right
after `viewer.load()`. Pausing deletes the view, so returning to the tab would
rebuild the whole grain and lose per-node expansion.

**Per-node expansion is not solved.** `expand(row_index)` / `collapse(row_index)`
are set-only with no getter, so they cannot be captured and replayed and are
lost on every rebuild. The route is a `ViewConfig` field of expanded row paths
applied in `server.cpp`. Depth is also whole-axis — one level collapses every
group at that level. Rationale: docs/pf_spec.md → §The pivot.

## Tag completions are per version

`GET /versions/:id/tags` — the tags used on **this** version, with `uses`,
`last_used`, `owners` and `can_edit`; `Forecast.jsx` refetches on `versionId`
and after every tagged write. A tag is not a row anywhere — only a value
repeated across `pf.log` entries.

**Managed on the Baseline page's Tags panel**, which lists each with its use
count and last-used date. `PATCH /versions/:id/tags` takes `{ from, to }` and
rewrites every entry carrying `from`; a `to` that already exists **merges**, an
empty `to` clears; the panel confirms both. Permission is the `pf.log` rule
applied to *every* entry the tag is on (a half-applied rename leaves the
initiative split), so a row with `can_edit` false is greyed out. One entry's
tag is edited in the Forecast page's change log.

**Completion matches on words, not substrings.** `TagInput` in
`OperationPanel.jsx` normalises both sides (case, punctuation, stopwords, light
stemming) and says so when a typed tag reduces to an existing one; nothing is
merged automatically. Rationale: docs/pf_spec.md → §Bridge and change log.

---

## Choosing a forecast

The **sidebar** lists every source's versions (`allVersions`, a client-side
fan-out over `/sources/:id/versions`) and sets `sourceId` from the one picked.
`versions` stays scoped to the selected source, because Baseline and Forecast
are — do not widen it. The source picker is only on Setup. Known collision:
"forecast" is both the scenario and the third step; undecided.

**Where a browser lands.** `App.jsx` restores `pf_sourceId` / `pf_versionId`
from `localStorage`, re-checked against the live list. With nothing stored it
falls back to **`pf.version.is_default`** (one per source, partial unique index
`version_one_default`), not `versions[0]`, which is the newest. A stored
selection wins. Setting the default is **admin only**, in `PUT /api/versions/:id`,
as two statements in one transaction — the index refuses the instant between.

**Renaming a version** is safe at any time; nothing derives from the name.
`PUT /api/versions/:id` answers 409 on a `(source_id, name)` collision, and
COALESCEs a falsy name back to the existing one, so a blank box is refused
client-side.

---

## Cloning a version's definition

`POST /sources/:id/versions` takes `clone_from`. It copies the template
version's settings (`description`, `exclude_iters`, the fallback names,
`bucket_order`), its **version-scoped** layouts, and its load segments — and
**runs nothing**, so the dates can be changed before anything loads.

Not copied: adjustments, in any form (a scale's target was sized against the
rows that were there), and source-wide layouts (`version_id IS NULL`), which
already apply to the new version.

### `pf.load_plan`

A plan is a load definition on its own — `kind`, `label`, `bucket`, `tag`,
`note`, `filters`, `raw_where`, `date_offset`, `seq` — with `run_logid` naming
the `pf.log` entry it produced, or null while unrun. **Deliberately not
`pf.log`**: every row there means a write happened.

`run_logid` is `ON DELETE SET NULL`, so undoing a load returns its plan to unrun.
The plans list reads the row count back through that FK.

`cloneDefinition()` in `routes/versions.js` reads `pf.load_plan` when the
template has plans and otherwise derives them from its `baseline`/`reference`
log entries (`pf.log.params` stores `filters`, `raw_where`, `date_offset`).

**Run all stops at the first failure** — segments are not independent.
Rationale: docs/pf_spec.md → §Loads and segments.

---

## Pivot layouts

A layout is a named `ViewConfig` in **`pf.layout`**, owned by an account:
**published** (everyone on the forecast lists and can apply it; only its owner
or an admin may change it) or **private**. Scope is the **version** by default;
`version_id IS NULL` means every version of the source. `is_default` (at most
one per scope, by partial unique index) is what `initViewer()` restores on a
first load — a version-scoped default beating a source-wide one.

**Permissions are the `pf.log` rule verbatim** — your own, or an admin's
override; `can_edit` rides on every row and the UI greys out the rest. Applying
is never restricted; Save is withheld on a layout that isn't yours, Save as…
forks it. No territory clause — a layout is display config.

**Local state.** `LAYOUT_KEY` (`pf_layout_v{vid}`), the unnamed last-used
config, stays in `localStorage`; `loadLayouts()` lifts the old named list
(`LAYOUTS_KEY`) into `pf.layout` as private rows once, then clears it.

**The dirty dot is a comparison, not a flag.** `restore()` itself fires
`perspective-config-update`, so `activeConfigRef` holds what the pivot last
matched and `sameConfig()` compares against it, ignoring `table` and `theme`.

**Grand totals ride in the layout under `pf_totals`** — `{ row, col }`. The
engine has no "subtotals but no grand total" mode, so both are hidden in the
grid's DOM; `captureConfig()` adds the key and `cleanLayout()` strips it before
`restore()`. Without it: row shown, column hidden.

**`cleanLayout()`** drops anything the live schema lacks — axes, `aggregates`
(including the weight column of the multi-arg form), `theme`, `pf_totals`, dead
expressions — dropping the offending *entry* rather than the layout, since
`restore()` is all-or-nothing. Rationale: docs/pf_spec.md → §The pivot.

---

## The Forecast toolbar is two zones

Left, **view** — `Layout ▾` and the `Rows` / `Columns` steppers with their
`Total` toggles; all `ViewConfig`, nothing reaches the server. Right,
**forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. **Blue is
for actions, pressed grey (`.pf-pressed`) for state.** `Fit widths` was removed
deliberately; do not restore it without fixing auto-size (see git history).

**`Calculating…`** in the status bar: `Forecast.jsx` raises a counter (rebuilds
overlap) in `applyDepth`, `applySplitDepth` and `applyLayout`, lowers it after
`viewer.flush()`, and broadcasts `pf-pivot-busy` on the window for `StatusBar`.
**Not** driven from `perspective-config-update`, which arrives after the
rebuild. Rationale: docs/pf_spec.md → §The pivot.

---

## Slice mechanics

A click (`perspective-click`) or a dragged region becomes a **slice** — a
`{col: value}` object. A plain click replaces the selection; ctrl/⌘/shift-click
toggles a slice in or out, so the panel holds a **list** sent as `slices` in
operation POST bodies (a single `slice` is still accepted server-side).

**The slice is re-derived by `cellFilters()` in `Forecast.jsx`**, from
`detail.row.__ROW_PATH__` (row axis, raw values, so dates stay epoch millis)
and the clicked column name (column axis, split by `|`). The click and a dragged
region go through the same derivation. `detail.config.filter` is used only as
the fallback when there is no cell to index the column axis with — it cannot be
trusted otherwise, because the engine builds it with a falsy test that drops
blank and `0` pivot values, and a dropped key **widens** the write.

**Regions.** The datagrid runs in `edit_mode: SELECT_REGION` (forced on
restore). `perspective-select` carries a **ViewWindow** —
`{ start_row, end_row, start_col, end_col }` — and fires on every mouseover, so
the handler records the latest window and a window-level `mouseup` commits it.
A single-cell region is ignored there: `perspective-click` owns it, and handling
it twice would undo a ctrl-click toggle.

**A total cell is its displayed amount.** The grand-total row resolves to no
row dimension, so its slice is whatever the column axis and the pivot's `==`
filters leave — possibly `{}`. An empty slice is accepted only alongside a
non-empty `scope`; with no filter at all it would be the whole version, and is
refused on both sides. `buildWhereAny()` reads a union containing `{}` as
`TRUE`, the scope then ANDed on — dropping it would narrow the write.

**The pivot's `pf_` columns stay in the slice.** `pf_segment`, `pf_bucket`,
`pf_note` and `pf_op` are resolved through the log (`COMPUTED_SLICE_COLS`);
`pf_iter`, `pf_user` and `pf_logid` are compared directly (`TABLE_SLICE_COLS`).
Stripping any of them made the write wider than the ledger's total. They narrow
a slice but do not make it selective on their own. Duplicates are collapsed
before the request — otherwise `apply_mode: each` applies the change twice.

**Selection highlight.** The datagrid wipes `model._selection_state.selected_areas`
on every mousedown, so `Forecast.jsx` keeps `areasRef` (`sliceKey -> rectangles`,
parallel to `slices`) and an effect pushes the full set back and redraws after
every change. Every deselect path prunes it by live slice key.

**A blank dimension value is a value.** `null` and `''` are different rows and
stay distinct through to the WHERE clause: `IS NULL` vs `= ''`. Only `undefined`
means absent (a subtotal row's shorter path, a collapsed column level). On the
column axis, `t_tscalar::to_string` renders a null as the literal text `"null"`
and an empty string as `''`, so `cellFilters()` maps `"null"` → null. The ledger
filters a null with `is null`, never `== null` (the engine matches on type). The
panel prints `(null)` and `(blank)`, and `—` only when the column is absent from
the slice.

**The pivot's own filter travels alongside, as `scope`** — `[col, op, value]`
triples read off `viewer.save()`, which is also what the ledger totals against.
`SCOPE_OPS` in `lib/sql_generator.js` is the whole allowed list, under
Perspective's operator names; anything outside it is **refused, not dropped**.

- Substring operators (`contains`, `begins with`, `ends with` and negations) map
  to **`ILIKE`**, matching the engine's case-insensitive `t_tscalar::contains`;
  `%`, `_` and `\` are escaped out of the term.
- `matches` / `not matches` stay refused: the engine is RE2, Postgres is not.
- Every operator that needs a term refuses an empty one (an unfinished filter
  row arrives as `[col, '==', null]`).

**Limitation:** computed columns created by Perspective expressions don't map
back to raw rows — only native columns work for slice extraction. Rationale:
docs/pf_spec.md → §Selection.

---

## Operation SQL patterns

All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slices; `{{exclude_clause}}` blocks `exclude_iters` rows; the territory clause is ANDed on last.

**`buildWhere()` in `lib/sql_generator.js`** turns a slice into SQL: it allows
`dimension` + `date` columns plus `COMPUTED_SLICE_COLS` and `TABLE_SLICE_COLS`,
**skips** unknown non-`pf_` keys, and **refuses** an unknown `pf_` key (skipping
one would widen the write). A null renders as `IS NULL`; in a list the null arm
is ORed on, since `IN (NULL)` is never true. Values are escaped but not
parameterized — consistent with existing patterns, debuggable in pg logs. The
"must name a filterable column" refusal is **`assertSelective()`** in
`routes/operations.js`.

**Every date column travels.** `dataCols` is `[...dims, ...dateCols, value, units]`
— never just the first date column, or scale and recode write rows with null
secondary dates under correctly labelled period dimensions.

- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions; refuses a pool whose net is ~zero relative to its gross (`assertProratable`)
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched

**Recode takes a date shift too.** `date_offset` moves the *put-back* leg only,
so a shifted recode is a timing move, and with a shift `set` may be empty. The
moved rows' period dimensions are re-derived from `pf.dim_period` against the
shifted date; a row whose date is null keeps the label it had. Without a shift
nothing is derived. Clone still blanks those labels on a null date.

**A date offset must be whole days.** `assertInterval()` refuses anything finer,
because every shifted date is cast back to a date — and Postgres reads a bare
number as *seconds*. `30 days` is not a month; use `1 month`.

Scale refuses with "Nothing to scale…" and recode with "Nothing to recode…"
when there is nothing to write. Rationale: docs/pf_spec.md → §Operations.

---

## Authentication

Everything under `/api` except the auth routes sits behind a session; the React
app is only mounted once there is one (`Gate` in `main.jsx`).

- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
  Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
  the password is read on stdin and hashed before it reaches psql.
- **Sessions:** `pf.session` (connect-pg-simple); revoke by deleting the row.
  Cookie `pf.sid`: httpOnly, SameSite=Lax, Secure unless `COOKIE_SECURE=false`, 12h rolling.
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
  `TRUST_PROXY` (default 1) for `req.ip` and secure cookies behind the TLS proxy.
  `CORS_ORIGIN` is the only way CORS is enabled; never a wildcard with a session
  cookie.
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
  minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
  regenerates the session id on success.

**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
`sessionUser(req)`, **never from the request body**.

## Setup and Baseline are admin-only to write

`requireAdmin` (`lib/auth.js`) is on:

- **sources.js** — `POST /sources`, `PUT /sources/:id/cols`, `POST
  /sources/:id/generate-sql`, `POST /sources/:id/dim/:group/refresh`,
  `DELETE /sources/:id`
- **versions.js** — `POST /sources/:id/versions`, `PUT /versions/:id`,
  `POST /versions/:id/{close,reopen}`, `DELETE /versions/:id`
- **operations.js** — the loads: `POST /versions/:id/{baseline,reference}`,
  `PUT /versions/:id/baseline/:logid`, `DELETE /versions/:id/baseline`
- **plans.js** — every write

**Not gated, deliberately:** reads (gating what a page needs to render turns it
into a 403); scale/recode/clone/undo, which carry their own owner and territory
rules; `PATCH /versions/:id/tags`, whose permission is ownership.

**The pages stay visible and go read-only** — one `<fieldset disabled>` per page
plus a `ReadOnlyNotice` banner. Setup wraps a real box; Baseline uses
`display: contents`. Baseline's Tags panel sits outside the fieldset. **None of
this is the permission** — the server check is. Setup's list fetches coerce
responses to an array: an error body rendered with `.map` throws during render
and unmounts the whole app. Rationale: docs/pf_spec.md → §Access.

## Source columns come from pg_catalog

`information_schema.columns` omits **materialized views**, and `gs.osm_skinny` is
one. `RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version
creation, source registration and the table preview; it returns the
information_schema shape (`data_type` from `format_type` with the modifier
stripped, precision/scale from `atttypmod`). The table browser lists from
`pg_class` by `relkind`. Do not query `information_schema.columns` for source
columns.

## Territory scoping

An account sees and changes only its own territory: `pf.app_user.territory`
(jsonb array), `is_admin` for accounts that see everything, and
`col_meta.is_territory` marks the column (one per source).

**Fail closed.** `buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an
empty list or an unflagged source.

**Built from the session, never the request.** Unlike `scope`, the territory
predicate is ANDed on last, in `sliceUnits()` for writes and per route for
reads, where nothing in the payload can remove it.

Enforced at:

- `/data` — clause on the cursor *and* on the count behind `X-Row-Count`
- `/agg` — `{{territory_clause}}`, applied **before** the GROUP BY, since the
  territory column need not be in the grain
- every operation, through `sliceUnits()`
- `/sources/:id/values/:col` — completion reads the source table
- `DELETE /log/:logid` and `PATCH /log/:logid` — by **owner**, not territory:
  your own entries, or an admin's override. Undo removes an entry wholesale, and
  `label`/`bucket` name the pivot's columns for everyone in the version
- recode's `set` — a scoped account cannot set the territory column at all

**The change log** lists everyone's entries, with an **Everyone / Mine** control
(resets to Mine on every open, not persisted) and the reader's own entries
marked. `isMine()` is strict authorship, deliberately not `canUndo()`. The fetch
stays unfiltered, since the control's counts need the whole set. It shows an
entry's **full** impact — the totals on `pf.log` are company-wide, deliberately.

Managed with `./pf.sh set-territory | set-admin | orphan-territory`.
`orphan-territory` lists values present in the data that no account owns.
Rationale: docs/pf_spec.md → §Access.

## Light / dark mode

`ui/src/theme.jsx` — `ThemeProvider` (in `main.jsx`) and `useTheme()` →
`{ dark, setDark }`. Stored as `pf_dark` in `localStorage`, else
`prefers-color-scheme`; the effect toggles `.dark` on `<html>`. `ui/src/index.css`
defines custom properties under `:root` and `.dark`, with Tailwind overrides
written as `.dark .bg-white { ... }` — no Tailwind dark-mode config. Palette is
Perspective's "Pro Dark".

The viewer's theme is `viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' })`,
on load and in `useEffect([dark, versionId])` — a **partial** restore, no
rebuild. **Not** `setAttribute('theme', …)`, which does nothing in 5.4.0.
`cleanLayout` drops `theme` and `sameConfig()` ignores it: the theme is the
reader's choice, never a layout's.

## After a change: restart, and Generate SQL

- **Restart the server** for anything in `routes/`, `lib/` or `server.js`.
- **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes.
  The templates are *stored* in `pf.sql`, so editing the generator changes
  nothing until they are rebuilt — and a template carrying a token the running
  code does not substitute fails at the database, which reads as an unrelated
  client-side error.

**Order matters: restart first, then Generate SQL**, or the old code writes the
templates.

Schema changes are applied to the live database directly and mirrored into
`setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a
migration runner.

## Known issues / active work

- **Rows written before the every-date-column fix have null secondary dates**
  (`rdate`/`sdate_e`) on scale and recode rows. Deleting and replaying those log
  entries is the honest repair; a backfill would be inference
- **Zero-row clones report success** — clone commits an empty log entry and
  returns `rows_affected: 0`, where scale and recode refuse
- **The change log does not show an entry's id**, so there is no way to name
  one when asking about it
- **Territory is read onto the session at login**, so a change does not reach a
  signed-in account until it signs in again
- **The bridge dedupes overlapping slices by `pf_id`**, which grain-mode rows do
  not carry, so overlapping slices double-count there; and it keeps only
  dimension/date slice keys, dropping the `pf_` keys operations now honour
  (`BridgeView.jsx`, the per-slice view loop)
- **Depth buttons rebuild the view** — `restore()` tears down and rebuilds, a
  full traversal, where a manual collapse mutates in place. Fix: detect a
  depth-only change and call `view.set_depth()` too, at the cost of drift
- **Load time is dominated by row count, not payload size** — serialising rows
  out of Postgres into JS dominates. The lever is **dynamic grain**; see
  docs/pf_spec.md → §Display-grain pre-aggregation
- **Per-node expand/collapse** is lost whenever the view rebuilds; see §Axis depth
- **Col_meta / version schema drift:** if col_meta roles change after a version's
  forecast table is created, SQL and DDL go out of sync — delete and recreate
  the version
- **Grain drift:** changing `in_grain` after a load requires Generate SQL + a page
  reload. `routes/log.js` derives the grain from live col_meta, so a grain
  changed mid-session yields `pf_gkeys` that don't match the loaded table and
  undo silently removes nothing
- **Grain is static per source** — an unflagged dimension cannot be pivoted on

## Deferred (not in v1)
Baseline replay (not built), approval workflow, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`. See docs/pf_spec.md → §Open questions.