pf_app/CLAUDE.md

Pivot Forecast — CLAUDE.md

What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec: pf_spec.md Data transport architecture options: pf_perspective_options.md

---

Tech stack

• Backend: Node.js / Express (server.js)
• Database: PostgreSQL — isolated pf schema
• Frontend: React + Vite + Tailwind CSS in ui/; built output lands in public/app/
• Pivot: Perspective (@perspective-dev/* distribution, not FINOS @finos/perspective) 5.4.0 from a patched build vendored in ui/vendor — see its README for what is patched and which host-tool versions the rebuild needs; bundled inline via the /inline entrypoints — never from a CDN (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See PERSPECTIVE.md.
• Dev: npm run dev (nodemon) in root; npm run build in ui/

---

Project layout

server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation
  versions.js           Version CRUD, baseline/reference load, data stream, tag completions
  operations.js         scale, recode, clone, undo — the core forecast ops
  log.js                GET /api/versions/:id/log, DELETE /api/log/:logid
  layouts.js            Named pivot layouts — list per version, create, patch, delete
  plans.js              Planned load segments — list, create, patch, delete
lib/
  sql_generator.js      buildFilterClause, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
  utils.js
setup_sql/
  01_schema.sql         pf schema DDL — run once to install
  02_auth.sql           pf.app_user + pf.session
ui/src/
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    DepthButtons.jsx    One axis's collapse control; both axes are drawn from it
    LayoutMenu.jsx      The Layout ▾ control — Published / Mine, with the write actions
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         Forecast list + the 3 steps; collapsible
    StatusBar.jsx       Forecast name · row count · calculating · theme; source on Setup only
    ReadOnlyNotice.jsx  The banner over a page a non-admin may read but not change
    Timeline.jsx        Date-range preview bar for baseline segments

---

Database schema (pf)

• pf.source — registered source tables
• pf.col_meta — column roles: dimension | value | units | date | filter | ignore; dim_group groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); dim_period_col maps a dimension to a pf.dim_period column so date-adjacent values are derived at load time rather than copied raw; in_grain flags dimension/date columns that define the display grain (see below); is_key is described under §is_key below
• pf.version — named forecast scenarios; exclude_iters (default ["reference"]) blocks those iter values from all operations
• pf.fc_{tname}_{version_id} — one forecast table per version; contains both operational rows (pf_iter = baseline|scale|recode|clone) and reference rows (pf_iter = reference). Indexed on pf_logid, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it.
• pf.log — audit log; every write gets one entry; slice + params stored as jsonb
• pf.layout — named Perspective view configs; see §Pivot layouts
• pf.load_plan — load segment definitions, before and apart from having been run; see §Cloning a version's definition
• pf.sql — generated SQL templates per source/operation; tokens substituted at request time
• pf.app_user — login accounts; scrypt pass_hash, is_active, last_login_at
• pf.session — express-session store (connect-pg-simple layout)
• pf.dim_period — calendar lookup table (2018–2035); one row per month keyed on sdat (month start date); provides cal/fiscal year, quarter, and month columns; populated by setup_sql/gen_dim_period.sql with a configurable fiscal year start month

is_key

Read in four places, and not the one it sounds like — slices are validated against filterCols, which is every dimension plus every date column regardless of is_key.

1. The key of a dim_group — resolveGroup() in routes/sources.js takes members.find(c => c.is_key), the column every other member is keyed on for pf.dim_member
2. Value completion — only is_key columns get a dropdown, and /sources/:id/values/:col refuses anything else
3. Sibling autofill — fires on blur only when is_key && dim_group
4. The dim_period anchor — role === 'date' && is_key && dim_group picks the date whose siblings are derived from the calendar

Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group. When a group has more than one, .find() silently takes the lowest opos. That is not hypothetical: segment_new (opos 12) outranked part (opos 15) in the part group, so a refresh keyed on a column that is null throughout, matched nothing, and reported success with zero members. customer's group has four keys and picks the right one only by opos luck.

The two meanings want separating — a per-group key choice, or a rule that the group key is the column named by the group (which these groups nearly follow already, except sdate → sdate_e). Until then, a refresh that finds two keys should refuse rather than guess.

Key token substitution tokens

{{fc_table}}, {{where_clause}}, {{exclude_clause}}, {{logid}}, {{pf_user}}, {{value_incr}}, {{units_incr}}, {{pct}}, {{set_clause}}, {{scale_factor}}, {{date_offset}}, {{filter_clause}}

---

Core data flow

Initial load (Forecast view)

Forecast.jsx fetches col_meta first, then picks the endpoint:

• grain mode (any in_grain column) — GET /api/versions/:id/agg, rows pre-aggregated to the grain, table indexed on pf_gkey
• raw mode (no grain) — GET /api/versions/:id/data, raw forecast rows, table indexed on pf_id

Either way: Arrow IPC binary stream → worker.table(buffer) in Perspective WASM. fetchArrow() handles both.

Why one batch (not streaming): pg returns bigint/numeric as strings by default — type parsers in server.js coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.

Display grain

Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on osm_stack. It keeps the native Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (in_grain per column); it is baked into pf.sql at Generate SQL time so load and operations agree. grainOf() in lib/sql_generator.js is the single definition of what the grain is — Setup.jsx and routes/log.js mirror it. Full design: pf_spec.md → §Display-grain pre-aggregation. Why not a DuckDB virtual server: pf_perspective_options.md → §Spike findings.

Segment and note columns

/data and /agg both LEFT JOIN pf.log and emit two columns the forecast table does not itself carry:

• pf_segment — pf.log.label
• pf_bucket — pf.log.bucket
• pf_note — the free text on a scale/recode/clone; null on loads

Both names are stored, not derived — see §Every log entry is named.

They are deliberately separate: commingling a segment name with an adjustment note makes neither pivotable. In grain mode pf_logid is part of the grain, so the join adds no rows. The operation routes stamp the same two fields onto the rows they push back incrementally, since those come from RETURNING * and would otherwise arrive without them.

Column order is stored text

Perspective orders column groups by the value string, and SortDir's col asc / col desc only reverses that — so Prior Year → Plan → Actual → Forecast is alphabetical in neither direction and expressible as neither. A "01 - " prefix is the only lever, and it lives in pf.log.label (and pf.log.bucket), typed by whoever names the segment. Nothing derives it.

SEGMENT_EXPR / BUCKET_EXPR / NOTE_EXPR in lib/sql_generator.js are the single definition, shared with the /data cursor in routes/operations.js — /agg is generated, /data is not, and the two have to agree.

The one hardcoded ordinal is ADJUSTMENT_SEGMENT = '99 - Adjustments', the name an adjustment is stamped with when its version sets no default, which keeps those adjustments after every numbered segment. That proviso is the whole scheme, not a caveat on it: ordering is string ordering, so 99 only lands last once the loads carry 01–0n, and an unnumbered segment sorts after it (digits precede letters — 9 is 0x39, A is 0x41). The old '(adjustment)' sorted first for the same reason read the other way, ( being 0x28. Relabelling an adjustment's own log row afterwards is how one kind of adjustment is split out from the rest.

Every log entry is named

pf.log.label is a row's segment and pf.log.bucket what it counts toward, and every entry carries both — the log_named CHECK refuses one without. So a report gets the pivot's segment and bucket by joining the forecast table to pf.log on pf_logid and selecting two columns; there is no rule to copy.

They are stamped when the entry is written:

• A load — label is required (the routes answer 400 without one); bucket defaults to the label. loadNames() in routes/operations.js.
• An adjustment — scale, recode and clone have no name inputs, so they are stamped with the version's adjustment_segment / adjustment_bucket, else the built-ins ADJUSTMENT_SEGMENT = 99 - Adjustments and ADJUSTMENT_BUCKET = 04 - Forecast (ADJUSTMENT DEFAULTS in lib/sql_generator.js). adjustmentNames() resolves them once per request; they go into the template's INSERT as {{label}} / {{bucket}} and onto the rows pushed back to the pivot.

The two version defaults are set on the Baseline page under Defaults for new adjustments. Changing one names the next adjustment; existing entries keep what they were written with. Renaming past entries is PATCH /api/log/:logid, one at a time, which renames but refuses to clear.

What this replaced. The names used to be derived at read time: a load's label, else tag, else note, else the version's unlabeled_load, else Unlabeled; an adjustment's label, else the version default, else the constant — read through a join on pf.version. That made the pivot the only reader that knew what bucket a row was in. The same CASE was copied into /agg, the /data cursor and rlarp.load_osm_pf in /opt/forecast, and any direct query had to copy it again. It also meant editing a version's fallback name silently renamed every past adjustment. The 2026-09-24 backfill in setup_sql/01_schema.sql stamped the existing entries with that rule; before it ran, the pre-backfill label/bucket values were saved to pf.bak_log_names_20260924. pf.version.unlabeled_load is read by nothing now; the column remains.

What this replaced. The prefix used to be computed client-side, as Perspective expression columns (pf_bucket_ord, pf_segment_ord) built from pf.log.seq and pf.version.bucket_order. It ordered the pivot and nothing else, so every other reader disagreed with it; restore() replaces expressions wholesale, so it had to be re-applied after every layout load; and ExprTK's string scanner tests each byte with isprint(), so a label containing anything outside printable ASCII could not be ordered at all (· is two bytes, of which isprint(0xC2) is false). DEAD_ORDER_EXPRS in Forecast.jsx strips the expression names out of layouts saved under that scheme. pf.log.seq and pf.version.bucket_order are no longer read; the columns remain.

Relabelling now needs a page reload to show, because the label is part of the aggregated row rather than something the pivot can re-derive. Editing a label afterwards is a PATCH /api/log/:logid and needs nothing regenerated, but a source registered before this change needs Generate SQL run once, so its stored load templates write label and bucket onto the log row at all.

Forecast operations

POST to /api/versions/:id/{scale|recode|clone} → SQL executed with RETURNING * → new rows returned as JSON → pspTable.update(rows) — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since pf_logid is part of pf_gkey those keys are always new, so update() appends and the view re-sums.

Price or volume (plug)

Price does not move unless it is named. That is the rule the whole section implements, and it is a rule because a price move is invisible in the figure being edited: the dollars land on target whether volume followed or not, and only the implied price says which happened.

A sales figure alone does not say which of price or volume moved, so scale takes plug — 'volume' (the default: price holds, units scale in proportion) or 'price' (volume holds, price moves). Resolved in resolveIncrs() in routes/operations.js; the panel only offers it when the edit is dollars-only, because naming units or price has already answered it, but it stays on screen greyed out in the other cases rather than vanishing. The semantics come from the predecessor Excel model, /opt/forecast_api/VBA/fpvt.frm → calc_val / calc_price:

plug volume:  pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price:   fVol = pVol + bVol

A qty input follows the same rule. With dollars unnamed they follow at the current price rather than holding — value = (curValue / curUnits) * unitsIncr. Holding them made every qty edit a price edit: 10% more cases at the same revenue is a 9% price cut nobody asked for. A slice with no units has no price to hold, so there the dollars do stay put and the edit is a pure qty change.

The two deliberate ways to move price, both of which outrank the above:

• name the dollars and the qty together — price falls out of the two
• name the price (target_price), which is the Excel form's Edit Price mode: with a target_units alongside, both are inputs and dollars fall out

The ledger's Result line previews value, units and price together using the same rules, so what you see is what gets written, and the column that cannot move is labelled · holds.

What this replaced. plug defaulted to 'price' — volume holds, price absorbs — because that is what the API did before the flag existed. So an unqualified dollar edit moved price, and a qty edit moved price, and the only edit that did not was one where somebody had picked 'volume' by hand. The client's stored preference is keyed pf_scale_plug_v2 for that reason: a browser carrying the old 'price' choice would have gone on silently moving price.

Undo

DELETE /api/log/:logid → removes rows by logid → table.remove() of the affected index values (pf_gkeys in grain mode, pf_ids in raw mode); the view re-sums. No full reload.

---

Axis depth (collapse / expand)

Both axes collapse the same way: a depth in ViewConfig, set through restore().

• Rows — group_by_depth, driven by the Rows buttons via applyDepth()
• Columns — split_by_depth, driven by the Columns buttons via applySplitDepth()

Both axes are drawn by components/DepthButtons.jsx from the axis's own field list (groupFull / splitFull), as a stepper: ◂ partgroup ▸, with the middle button opening the full level list for a jump rather than a step.

Three controls per axis whatever the depth. It was one button per level, which grows with the pivot — a 5-by-3 layout rendered ten of them — and the current level was a blue button among the greys, so reading the state meant scanning. Before that the row axis was a fixed 0 1 2 3, which named nothing and offered levels a two-field group_by does not have.

depth is null until the first layout has been read back; the stepper shows an em dash rather than guessing a level.

Both are 1-based counts of the levels to show, and so is the stepper's own depth — DepthButtons labels d as levels[d - 1]. Same unit on both sides, so the value crosses unchanged in each direction. The engine agrees: server.cpp turns group_by_depth into set_depth(n - 1) and t_traversal::set_depth adds the 1 back, so n renders n levels, on both axes (ctx2->set_depth(HEADER_COLUMN, column_pivot_depth - 1) is the same arithmetic).

The toolbar used to send d + 1 and subtract the same 1 when reading a layout back. That was self-consistent — the stepper always agreed with itself — and off by one against the grid at every position: picking partgroup showed one level past it. The +1 was there to dodge depth 0, which reaches the engine as set_depth(-1) on an unsigned t_depth, underflows, and clamps to fully expanded. applyDepth now floors the value at 1 instead, so the stepper's "Total" slot still shows the first level rather than a bare total — the long-standing wart, kept deliberately rather than traded for the underflow. A real total needs group_rollup_mode: 'total', which is a field in ViewConfigUpdate but is unverified against the local WASM engine.

Because a depth lives in the config, it survives every view rebuild, rides into the persisted and named layouts through viewer.save(), and needs nothing re-applied afterwards.

This required patching the engine. ViewConfig::apply_update applied ten fields and neither depth, so a depth sent through restore() was accepted, deserialized and dropped — and since Session::update_view_config returns early when apply_update reports no change, no view was rebuilt at all. See ui/vendor/0001-apply-depth-fields-on-config-update.patch; the vendored build carries it.

What this replaced

Worth knowing, because a lot of machinery existed to work around it and is now gone:

• Row depth used to be imperative — getView() then view.set_depth() — which put it on an object the viewer discards whenever it re-renders. Restoring it meant guessing when that had happened: an observerShim.js patching window.IntersectionObserver and window.ResizeObserver, focus and visibility listeners, a retry loop for getView() throwing No table set while getTable() resolved, and a flag tracking whether the viewer had "gone away". That guesswork produced three distinct visible faults — the tree fully expanding on refocus, snapping on any reflow, and snapping when Perspective's settings sidebar opened.
• Column collapse used to restore a truncated split_by. The discarded levels therefore had to be remembered separately (splitFull, persisted as split_full), our own collapse had to be told apart from the user rearranging the pivot (collapsingRef plus a prefix test), and the selection was cleared on every collapse because the axis was changing shape.

split_full is still read on load, for layouts saved under the old scheme.

Auto-pause is off

<perspective-viewer> auto-pauses by default: an IntersectionObserver on itself plus the document's visibilitychange drive AutoPauseState::apply(), and pausing deletes the view (session.set_pause(true) → view_sub.take().delete()). Returning to the tab is therefore not a redraw but restore_and_render() — a fresh view and a fresh traversal of the whole grain, which on a large one is a multi-second chug on every tab switch, and takes any per-node expansion with it.

initViewer() calls viewer.setAutoPause(false) right after viewer.load(). Nothing updates the table while the tab is hidden — every operation is driven from this page — so the pause bought nothing and cost a rebuild. The view is now held while the tab is backgrounded.

Still not solved: per-node expansion

Expanding one specific branch is view state with no config representation, and the API has no getter:

expand(row_index: number): Promise<number>
collapse(row_index: number): Promise<number>

It can be set but not read, so it cannot be captured and replayed — this is why it is lost on every rebuild, and why no client-side fix has worked. The honest route is a ViewConfig field carrying expanded row paths (indices shift as the tree opens), applied in server.cpp where the depths are. Bigger than the depth patch: it needs a way to enumerate expanded nodes in the C++ traversal, a proto field, and the apply step.

The fork's own header_click.ts / expand_column / collapse_column is the column-axis equivalent and has the same limitation.

Limitation that remains either way: depth is whole-axis. Excel can collapse 2025 while 2026 stays expanded; a depth collapses every group at that level together. columns selects which measures appear, not individual split combinations.

Tag completions are per version

GET /versions/:id/tags — the tags already used on this version, with their use counts, for completing the next adjustment. Forecast.jsx refetches on versionId and after every tagged write.

It used to be GET /sources/:id/tags, joined through pf.version and grouped across every version of the source. A tag names an initiative inside a scenario, so that put last year's work in front of this year's and reported a uses count earned on a version the tag did not belong to. The cost of the narrower scope is that a fresh version completes from nothing until it has a tag of its own — which is the honest state of it, since cloning a definition copies no adjustments.

The pf_bucket datalist on the Baseline page was already per version (bucketsInUse), so the two now agree.

The list is managed on the Baseline page, not the Forecast one. A tag is not a row anywhere — only a value repeated across this version's pf.log entries — so there is no /tags/:tag to address and no list to curate ahead of use: a tag exists because somebody typed it on an adjustment.

PATCH /versions/:id/tags takes { from, to } and rewrites every entry carrying from. to naming a tag that already exists merges the two, which is usually the reason to be there: the bridge groups on this column, so Pull Forward and Pulled Forward are one initiative reported as two. An empty to clears the tag. The Baseline panel confirms both before sending, since neither is visible as a diff afterwards.

Permission is the pf.log rule applied to every entry the tag is on, not most of them — a half-applied rename leaves the initiative split, which is the state the rename was fixing. GET /versions/:id/tags therefore returns owners and can_edit, and the panel greys the row out rather than offering a rename that answers 403. Editing one entry's tag is still the change log on the Forecast page; this is the whole-version view of the same column.

Completion matches on words, not substrings. TagInput in OperationPanel.jsx replaces the native datalist on the tag field: both sides are lowercased, stripped of punctuation, relieved of stopwords and lightly stemmed, so Pulled Forward finds Pull Forward and moved to different sku finds moved to a different sku — neither of which a substring match reaches, one inserted word being enough to hide the tag that already exists. Typing something that reduces to an existing tag without being it says so outright ("Already used as … — use that"), because the ledger will otherwise report the two as separate initiatives. Nothing is merged automatically; the suggestion is offered and the typing stays yours. The change log's inline tag cell still uses the plain datalist.

Completion is otherwise the list and nothing else. The operation panel used to carry six quick-pick pills under the tag field as well. They wrapped to two rows as soon as the tags were longer than a word, and the panel docked at the bottom has a fixed height, so they pushed the form past its own border. The datalist costs no space, is not capped at six, and now shows the tag alone — an option's text content renders as a second column beside the value, so the use count was widening every row of the pick list to carry a number nobody picks on. uses and last_used still come back from the route; nothing reads them.

---

Choosing a forecast

The forecasts are listed in the sidebar, above the three steps, each row carrying its status and a star for the default. They were two dropdowns in the status bar, which is otherwise read-only — so the one pair of controls in it did not read as controls at all.

Not nested under the Forecast step, which is where the idea started: Baseline works on the same version, so hanging the list off one step would either misrepresent the scope or duplicate it under two. It reads what am I working on, then what step am I on.

The source is not in the nav. gs.osm_skinny means nothing to someone editing a forecast. The sidebar lists every source's versions and sets sourceId from the one picked, since pf.version.source_id comes with it — so choosing a forecast settles the source, and the picker is only offered on Setup, the one page where the raw table name is the right label.

versions stays scoped to the selected source, because Baseline and Forecast are; the sidebar gets its own allVersions instead, so widening it does not change what versions means everywhere else. It is a client-side fan-out over /sources/:id/versions rather than a GET /versions — one request, at one registered source.

The status bar keeps what follows from the selection: the forecast's name and its row count. The table written to is in the hover panel, still click-to-copy, because pasting it into psql is the reason it was ever on screen.

Known collision: "forecast" is both the scenario and the third step. The scenario should own the word — it is what people name and say aloud — which leaves the step wanting a different one (Adjust). Undecided.

Where a browser lands

App.jsx restores pf_sourceId / pf_versionId from localStorage and re-checks both against the live list. What it falls back to when there is nothing stored — a new account, a new machine, a cleared browser — is pf.version.is_default, one per source by partial unique index (version_one_default), the way pf.layout.is_default works.

It used to fall back to versions[0], and the list is ordered created_at DESC, so that meant the newest version — whatever somebody created last, a scratch copy as readily as the plan. Accounts that never pick a version themselves, which is every sales account, therefore followed the forecasting work around.

Admin only, enforced in PUT /api/versions/:id: it decides where everyone else lands, so it follows the pf.log rule rather than being the version's own setting like its name. The checkbox on the Baseline page's This forecast panel is greyed out for other accounts rather than offering a click that answers 403. The sidebar's forecast list marks the default with a star.

A stored selection still wins. The default is the fallback, not an override — someone who has been working in another version stays there. The corollary is that an account which already landed in the wrong scenario keeps landing there until it switches once.

Setting the default is two statements — clear the old, set the new — so the route runs them in one transaction; the index refuses the instant in between.

Renaming a version

Safe at any time, including on a closed one. Nothing derives from the name: the forecast table is fc_{tname}_{version_id}, pf.log and pf.layout key on the id, and the UI's localStorage keys are id-based. The Baseline page's This forecast panel saves name and description on blur through PUT /api/versions/:id, which has accepted both all along — it was only ever called for the fallback names.

Two edges the panel handles: (source_id, name) is unique, so the route answers 409 on a collision rather than a raw Postgres error; and the UPDATE COALESCEs a falsy name back to the existing one, so a blank box is refused client-side rather than looking saved and changing nothing.

---

Cloning a version's definition

POST /sources/:id/versions takes clone_from. It copies the template version's settings (description, exclude_iters, the three fallback names, bucket_order), its version-scoped layouts, and its load segments — and runs nothing. The reason to clone a definition is to change the dates before anything loads, and a clone that fetched two million rows first would have to be undone before it could be edited.

Not copied: adjustments, in any form. A scale's target is a figure sized against the rows that were there, so replaying one against a different load is arithmetic nobody asked for. Source-wide layouts (version_id IS NULL) are not copied either — they already apply to the new version, and a copy would duplicate the name.

pf.load_plan

A segment used to exist only once it had been run: a pf.log entry with forecast rows behind it. A plan is the definition on its own — kind, label, bucket, tag, note, filters, raw_where, date_offset, seq — with run_logid naming the pf.log entry it produced, or null while it is still a plan.

Deliberately not pf.log. Every row there means a write happened; holding unexecuted intent in the audit log would cost it that meaning, and the change log would list entries that never ran.

run_logid is ON DELETE SET NULL, so undoing a load returns its plan to unrun rather than deleting the definition with it. The plans list reads the row count back through that FK, which is how it tells run from unrun.

Derivation from an old version. cloneDefinition() in routes/versions.js reads pf.load_plan when the template has plans and falls back to deriving them from its baseline/reference log entries when it does not — which is every version built before plans existed. pf.log.params already stores filters, raw_where and date_offset, so this is a read, not a reconstruction.

Run all stops at the first failure. The segments of a definition are not independent — a baseline followed by references that date-shift off it — so carrying on past a broken one builds something nobody described.

---

Pivot layouts

A layout is a named ViewConfig, stored in pf.layout and owned by an account. Two kinds, one table:

• published — everyone on the forecast lists it and can apply it; only its owner or an admin may change it
• private — yours, nobody else lists it

Scope is the version by default, because that is where people enter the app. version_id IS NULL means the layout applies to every version of the source; that is where the old source default went, and what a brand-new version picks up before anyone has published anything for it. is_default (at most one per scope, by partial unique index) is what initViewer() restores on a first load — a version-scoped default beating a source-wide one, the narrower answer winning.

Permissions are the pf.log rule verbatim — your own, or an admin's override, and the UI greys out the rest rather than offering a click that answers 403. can_edit rides on every row so the menu knows which. Applying is never restricted: the guarantee is that a published layout cannot be changed out from under people, not that it cannot be adapted — Save is withheld on a layout that isn't yours, Save as… forks it into your own.

No territory clause. A layout is display config, and territory restricts rows, not columns; cleanLayout() already drops anything the live schema lacks.

What is still local. LAYOUT_KEY (pf_layout_v{vid}) — the unnamed last-used config — stays in localStorage, because it is per-browser session continuity rather than a thing anyone names or shares. LAYOUTS_KEY (pf_layouts_v{vid}) is the old named list; loadLayouts() lifts it into pf.layout as private rows once per version and then clears the key.

The dirty dot is a comparison, not a flag. restore() itself fires perspective-config-update, so anything set unconditionally in that handler would light up the moment a layout was applied. activeConfigRef holds what the pivot last matched and sameConfig() compares against it, ignoring table (the per-load table name, different on every refresh).

What this replaced. Named layouts lived only in localStorage — invisible to anyone else, gone on another machine. The one server-side layout was pf.source.default_layout, a single anonymous blob that PUT /sources/:id/default-layout let any account overwrite for every account: a published layout with no owner. That route is gone; the column remains, migrated and read by nothing.

cleanLayout() guards aggregates along with the axes — including the weight column of the multi-arg form — and drops the offending entry rather than the layout, since restore() is all-or-nothing.

---

The Forecast toolbar is two zones

Left, on its own tinted ground: view — Layout ▾ and the Rows and Columns depth buttons. Nothing there reaches the server or changes the forecast; it is all ViewConfig against rows already loaded.

Right, in a bordered panel of its own: forecast — Refresh data, Change log, Bridge, Operations. These re-run the load, open something over the pivot, or write to the version.

They were one undifferentiated row separated by identical hairlines, which made Fit and Bridge read as peers when one is a column width and the other is the version's history. Three weights do the separating now, and they were all once the same: .pf-section-label names a group (Layout, Rows, Columns) and is heavier and darker than the controls under it, .pf-group-rule separates groups within a zone, and the zone borders separate the zones. A label that matched its buttons, and a group rule that matched a zone border, is what made the row read as one undifferentiated thing.

Fit widths is gone, button and fitColumns both, for three independent reasons. Auto-size measures the leaf header over the rows currently in the viewport, so anything below the fold never counted; draw({invalid_columns}) passes an option DrawOptions does not have, and draw() is throttled, so clicking during a pending draw cleared the widths without re-measuring and the columns fell to their 52/60px floors; and the persistLayout after it saved nothing, since saveColumnSizes() returns only overrides, which resetAutoSize() had just emptied — and this fork discards column_size_override whenever split_by is non-empty, which here is always. Recoverable from git if it is ever worth fixing rather than dropping.

A Calculating… indicator sits in the app's status bar while the pivot rebuilds. Perspective has its own beaker, but it is rendered only while the settings panel is open, so it vanishes exactly when nothing else tells you to wait. Forecast.jsx raises a counter — a counter, because rebuilds overlap — in applyDepth, applySplitDepth and applyLayout, lowers it after viewer.flush(), and broadcasts pf-pivot-busy on the window for StatusBar to render, the same way pf-data-changed already travels between them.

Deliberately not driven from perspective-config-update, which is the obvious hook and is the wrong one: measured on a depth change over a large grain, the event arrives after the rebuild — seconds late — and the flush that follows resolves in 6ms. It reports the end of the work. The cost is that sorting or filtering from the viewer's own chrome shows nothing, because the app is never told those began.

---

Slice mechanics

When the user clicks a pivot cell, perspective-click fires. The handler in Forecast.jsx extracts [col, '==', value] filters from detail.config.filter — only role = dimension and role = date columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a list of slices sent as slices in operation POST bodies (the single slice object is still accepted server-side).

Dragging across a block of cells selects a region. The datagrid runs in edit_mode: SELECT_REGION (forced on restore, so a saved layout can't switch it off) and reports the region as a perspective-select event carrying a Perspective ViewWindow — { start_row, end_row, start_col, end_col }, not the per-row insertConfigs payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level mouseup commits it. A single-cell region is ignored there: perspective-click already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.

Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's __ROW_PATH__ (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".

Selection highlight. The datagrid highlights whatever sits in its own model._selection_state.selected_areas, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. Forecast.jsx keeps areasRef, a sliceKey -> rectangles map parallel to slices, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.

pf_iter is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, apply_mode: each would apply the same change twice.

A blank dimension value is a value. Both blanks occur in this data and they are different rows — on fc_osm_skinny_34, partgroup is null on 51,964 rows and '' on 111,696 — so null and '' stay distinct from the clicked cell through to the WHERE clause, where one becomes IS NULL and the other = ''. Only undefined means absent, which is how a subtotal row (a shorter row path) and a collapsed column level still read as "not part of the slice".

They used to be dropped, and a dropped key does not narrow the write — it widens it. Clicking the blank partgroup node under ALTMAN PLANTS selected the customer's whole $27.6M rather than the $6.2M under the node clicked, and the ledger agreed with it, because the panel totals through the same filter list. The source of the drop is the engine: getCellConfig builds the click's filter list with pivot_value ? [pivot, '==', pivot_value] : undefined, so the blank is gone before perspective-click is dispatched — and being a falsy test it takes a dimension whose value is 0 with it. cellFilters() in Forecast.jsx therefore re-derives both axes from detail.row.__ROW_PATH__ and the clicked column name rather than trusting that list; the click and a dragged region now go through the same derivation. The ledger filters a null with is null rather than == null, since the engine matches on type and a null threshold against a string column is the same dropped predicate again.

buildWhere() renders a null as IS NULL; it used to fall through esc() to = '', which selected the other blank — the slice named one node and the operation wrote the other. In a list, the null arm is ORed on rather than folded in, because IN (NULL) is never true.

The panel prints (null) and (blank) rather than nothing, and — only when the column is genuinely absent from the slice: a slice isolated to the blank node and one the column plays no part in are opposite statements and looked identical.

A blank on the column axis cannot be isolated. A column name is its values joined with |, and a null joins as the empty string, so by the time the name reaches the client the two blanks are the same text. Those cells are refused — with a message, and counted once for a drag — rather than widened, since a slice missing a split_by level means every value of it. Filtering on the column instead (scope, which carries is null) is the way to reach them.

The pivot's own filter travels alongside, as scope. A slice is {col: value} and can only mean equality, so a view narrowed to sseas_e <= 2027 has no slice form at all — the filter is read off viewer.save() and sent as [col, op, value] triples, which is also what the ledger totals against, so the panel and the write agree. SCOPE_OPS in lib/sql_generator.js is the whole allowed list, under Perspective's operator names; anything outside it is refused rather than dropped, since a scope silently discarded is a write wider than the panel that authorised it.

The substring operators — contains, begins with, ends with and their negations — map to ILIKE, not LIKE, because that is what the engine does: t_tscalar::contains lowercases both sides before std::string::find. On partd, contains 'Keysto' matches 16,864 rows case-insensitively and 35 case-sensitively, so LIKE would have written a fortieth of what the grid showed. Narrower is no safer than wider — either way it is not the set the ledger counted. The term is literal in the engine, so %, _ and \ are escaped out of it before it becomes a pattern.

matches / not matches stay refused. They are RE2 in the engine, and Postgres regexes are not RE2 — \b is a word boundary in one and a backspace in the other — so a pattern accepted by both could quietly match a different set of rows, which is worse than a 400.

Every operator that needs a term refuses an empty one: an unfinished filter row in the pivot arrives as [col, '==', null], and ILIKE '%%' or = '' would have read as "every row" or "no rows" rather than as the mistake it is.

Limitation: computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.

---

Operation SQL patterns

All three operations follow the same structure: insert a pf.log row in a CTE, then insert forecast rows referencing its id. {{where_clause}} is built from the slice; {{exclude_clause}} blocks exclude_iters rows.

Every date column travels, not just the first. dataCols is [...dims, ...dateCols, value, units]. It used to be dateCols[0], so on a source with order, requested and ship dates, scale and recode wrote rows whose rdate and sdate_e were null — and because the derived period dimensions (rseas, smon_e) are dimensions and were copied with the rest, the labels read correctly while the dates under them were gone. Both columns are role = 'date' and therefore sliceable, so an adjustment scoped to a ship date matched no adjustment row that had ever been written. Clone and the loads were always right. The wider GROUP BY costs almost nothing on osm_skinny (971,185 → 971,352 groups) because ordnum/ordline already make a row near-unique.

• Scale — distributes value_incr/units_incr proportionally across rows in the slice using window functions
• Recode — inserts negative rows (zero out original) + positive rows with {{set_clause}} dimension overrides; both share the same logid
• Clone — copies the slice with {{set_clause}} overrides and {{scale_factor}} multiplier; original untouched

build_where() validates every slice key against col_meta (only role = dimension allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.

---

Authentication

Everything under /api except the auth routes sits behind a session; the React app is only mounted once there is one (Gate in main.jsx), because its load effects call the API immediately.

• Accounts: pf.app_user — scrypt hashes from lib/auth.js, never plaintext. Managed with ./pf.sh add-user | passwd | list-users | disable-user | enable-user; the password is read on stdin and hashed before it reaches psql.
• Sessions: express-session + connect-pg-simple in pf.session, so a restart doesn't sign everyone out and a session can be revoked by deleting its row (disable-user does exactly that). Cookie pf.sid: httpOnly, SameSite=Lax, Secure unless COOKIE_SECURE=false, 12h rolling.
• Config: SESSION_SECRET is required — the server exits at boot without one. TRUST_PROXY (default 1) makes req.ip and secure-cookie detection correct behind the TLS proxy. CORS_ORIGIN is the only way CORS is enabled at all; a wildcard origin plus a session cookie would be cross-site request forgery by construction.
• Login hardening: routes/auth.js throttles to 10 failures per IP per 15 minutes (in-memory), returns one message for unknown/wrong/disabled alike, and regenerates the session id on success.

Identity is server-side. pf_user, created_by and closed_by come from sessionUser(req), never from the request body — the UI used to send a hardcoded pf_user: 'admin', which any client could have set to anything. The audit log now names the account that made the change.

Setup and Baseline are admin-only to write

requireAdmin sat in lib/auth.js, exported, and applied to nothing — so every non-admin account (26 of the 29) could call every Setup and Baseline endpoint directly. It is now on the writes those two pages lead to:

• sources.js — POST /sources, PUT /sources/:id/cols, POST /sources/:id/generate-sql, POST /sources/:id/dim/:group/refresh, DELETE /sources/:id
• versions.js — POST /sources/:id/versions, PUT /versions/:id, POST /versions/:id/{close,reopen}, DELETE /versions/:id
• operations.js — the loads: POST /versions/:id/{baseline,reference}, PUT /versions/:id/baseline/:logid, DELETE /versions/:id/baseline
• plans.js — every write

Not gated, and each for a reason. No read is, including the table browser — the pages stay visible, and gating what a page needs in order to render turns it into a 403 (see below). The forecast operations are not, because scale, recode, clone and undo carry their own owner and territory rules. Nor is PATCH /versions/:id/tags: its permission is ownership, and a rep may rename the initiatives they named themselves.

The pages stay visible and go read-only. One <fieldset disabled> per page rather than a flag threaded through every control — Baseline alone has forty-five, and the one that got missed would be the one that mattered — plus a ReadOnlyNotice banner saying why. Setup wraps a real box because that fieldset is its flex row; Baseline uses display: contents so the page's layout is untouched, and disabled applies either way, being a DOM property. Baseline's Tags panel sits outside the fieldset, for the ownership reason above. None of this is the permission — it only stops an account being offered a button that would answer 403.

A 403 used to blank the whole app. Setup fed responses straight into state and rendered them with .map; an error body is {error}, and throwing during render unmounts React. Its list fetches now coerce to an array, so an unexpected response costs an empty panel rather than the session. That fragility predated the gating — the gating only made it reachable.

Source columns come from pg_catalog

information_schema.columns omits materialized views — they are not in the SQL standard — and gs.osm_skinny is one. So the source the whole app is built on looked like it had no columns: registering it seeded nothing, and creating a version failed with "No usable columns in col_meta" while col_meta plainly held thirty-six.

RELATION_COLUMNS_SQL in lib/utils.js is the replacement, used by version creation, source registration and the table preview. It returns the same shape information_schema did, so mapType and the callers were unchanged: data_type is format_type with the modifier stripped, which gives the same spelling (character varying, numeric), and precision and scale are unpacked from atttypmod. The table browser lists from pg_class by relkind for the same reason.

Territory scoping

An account sees and changes only its own territory. The list lives on pf.app_user.territory (jsonb array) with is_admin for the accounts that see everything, and col_meta.is_territory marks which column of a given source the values belong to — one per source, flagged rather than named in code so a second source can be divided by something other than a sales rep.

Fail closed. No territory and not an admin means no rows. buildTerritoryClause() returns FALSE, not TRUE, for an empty list or an unflagged source: an account somebody forgot to configure sees nothing instead of the whole book.

Built from the session, never the request. This is the difference between it and scope, which the browser sends and which is right to send, being a filter the user chose. A permission cannot come from the thing it restrains, so the territory predicate is ANDed on last, in sliceUnits() for writes and per route for reads, where nothing in the payload can remove it.

Enforced at:

• /data — clause on the cursor and on the count behind X-Row-Count
• /agg — a {{territory_clause}} token applied before the GROUP BY, since the territory column need not be part of the grain and may not survive it
• every operation, through sliceUnits()
• /sources/:id/values/:col — completion reads the source table, which no scope has touched, so without it a dropdown enumerates the whole business
• DELETE /log/:logid and PATCH /log/:logid — by owner, not territory. Undo removes an entry's rows wholesale, and half-undoing one would leave a state nothing describes. The PATCH looks like a private annotation and is not: label and bucket name the pivot's columns for everyone in the version, so unguarded it let any account rename the company's segments. Your own entries, or an admin's override, and the UI greys out the rest rather than offering a click that answers 403.
• recode's set — a scoped account cannot set the territory column at all. Moving a row between territories is reassignment, not forecasting, and it would vanish from the view that would have shown what happened.

Whose entries the change log lists. Everyone's, as before, with an Everyone / Mine control in the modal header and each account's own entries marked in place — a blue rule down the left of the row and its By cell in blue. On the version this was built against that is 127 adjustments from five accounts, one of them the reader's, so finding your own meant reading the By column down the whole list.

Marked and filterable on purpose: the mark is how you tell without hiding anybody, the filter is for when you want only yours. It resets to Mine on every open and is not persisted — opening the log is almost always to check your own recent work, and a stale "Everyone" from last session is a worse default than the reverse would be. isMine() is strict authorship, deliberately not canUndo(): an admin can undo everything, and marking every row as theirs would say nothing.

The fetch stays unfiltered, since the counts on the control need the whole set.

The change log shows an entry's full impact, not the reader's share. The totals stamped on pf.log are company-wide, so an admin's version-wide scale reads the same in every account — deliberate, and labelled, rather than re-aggregating per territory.

Managed with ./pf.sh set-territory | set-admin | orphan-territory. orphan-territory lists values present in the data that no account owns; work under one is invisible to everyone but an admin, which a typo causes easily and nothing inside the app reveals.

Light / dark mode

Theme state lives in ui/src/theme.jsx — a React context (ThemeContext) with a ThemeProvider that wraps the app in main.jsx.

• Storage key: pf_dark in localStorage; falls back to window.matchMedia('(prefers-color-scheme: dark)') on first visit

• Toggle: setDark(d => !d) in StatusBar.jsx; effect writes localStorage and toggles the .dark class on <html>

• CSS: ui/src/index.css defines CSS custom properties under :root (light) and .dark. All Tailwind color overrides are written as .dark .bg-white { ... } etc. — no Tailwind dark-mode config needed

• Palette: dark mode uses Perspective's "Pro Dark" colours (--bg-primary: #242526, panels #2a2c2f, gridlines #3b3f46, text #c5c9d0)

• Perspective viewer: Forecast.jsx calls viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' }) both on initial load and in a useEffect([dark, versionId]). A partial restore, so it does not rebuild the view the way applyDepth's whole-config one does.

  It used to be setAttribute('theme', …), which in 5.4.0 does nothing — the theme is a ViewerConfig field (ViewerConfig.theme, a theme name), and the element observes no such attribute. The pivot simply stayed light whatever the toggle said.

  Half the fix is in cleanLayout, which drops theme. viewer.save() includes it, so every stored layout had one baked in, and restoring one reasserted the author's theme and undid the toggle. It is the reader's choice, not a property of the layout — which also means a published layout cannot impose its theme on whoever applies it. sameConfig() ignores theme for the same reason: toggling is not a layout edit and must not light the dirty dot.

• Consuming the theme: import useTheme from '../theme.jsx' then const { dark, setDark } = useTheme()

After a change: restart, and Generate SQL

Two steps, and which one you need depends on what changed.

• Restart the server for anything in routes/, lib/ or server.js.
• Generate SQL (Setup, per source) whenever lib/sql_generator.js changes. The templates are stored in pf.sql, so editing the generator changes nothing until they are rebuilt — and a template carrying a token the running code does not substitute fails at the database rather than in JS, which reads as an unrelated client-side error.

The order matters: restart first, then Generate SQL, or the old code writes the templates.

Schema changes are applied to the live database directly and mirrored into setup_sql/ for a fresh install; 01_schema.sql is idempotent but is not a migration runner, so running it is not how an existing database gets a new column.

Known issues / active work

• Rows written before 2026-09-21 have null rdate/sdate_e. The generator fix above changes what gets written from here on; it does not repair what is there. On fc_osm_skinny_29 that is ~270k scale and recode rows. A backfill would have to re-derive each from the baseline rows at the same coordinate, which is possible but is inference, not recovery — deleting and replaying those log entries is the honest repair

• Zero-row operations report success. Scale refuses with "Nothing to scale…" when its slice matches nothing; recode and clone commit an empty log entry and return rows_affected: 0. A recode of a rep whose rows are all reference looked like it worked and did nothing

• The change log does not show an entry's id, so there is no way to name one when asking about it

• Territory is read onto the session at login, so granting or changing one does not reach a signed-in account until it signs in again. Re-reading it per request in requireAuth would also make disabling someone immediate

• Depth buttons rebuild the view. A depth lives in ViewConfig, so changing it goes through restore() and Session::update_view_config tears down and rebuilds the view — a full traversal — where a manual collapse mutates the existing one in place. Noticeable on a large grain. The fix is to detect a depth-only change and call view.set_depth() imperatively while still writing it to the config, at the cost of the two being able to drift

• Load time is dominated by row count, not payload size. On fc_osm_skinny_29 (2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then ~3s to build Arrow. Halving the payload (the pf_gkey md5) barely moved it. The remaining lever is dynamic grain — group by the fields the current pivot actually uses rather than every in_grain column; see pf_spec.md → §Display-grain pre-aggregation, "the dynamic variant"

• Per-node expand/collapse is lost whenever the view rebuilds — set-only API, no getter; see §Axis depth

• Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)

• Source/version selection persists in localStorage (pf_sourceId / pf_versionId, App.jsx). It is re-validated against the live list whenever that list changes, so a deregistered source or deleted version re-points at the first remaining one instead of leaving a dead id that 404s every call

• Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version

• Grain drift: changing in_grain after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. routes/log.js derives the grain from live col_meta, so a grain changed mid-session yields pf_gkeys that don't match the loaded table and undo silently removes nothing

• Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see pf_spec.md → §Display-grain pre-aggregation

Deferred (not in v1)

Baseline replay (replay: true returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch spike/duckdb-virtual-server.