pf_app/CLAUDE.md

Pivot Forecast — CLAUDE.md

What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec, including the rationale and history this file leaves out: docs/pf_spec.md Data transport architecture options: docs/pf_perspective_options.md

---

Tech stack

• Backend: Node.js / Express (server.js)
• Database: PostgreSQL — isolated pf schema
• Frontend: React + Vite + Tailwind CSS in ui/; built output lands in public/app/
• Pivot: Perspective (@perspective-dev/* distribution, not FINOS @finos/perspective) 5.4.0 from a patched build vendored in ui/vendor — see its README for what is patched and which host-tool versions the rebuild needs; bundled inline via the /inline entrypoints — never from a CDN (the CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See docs/PERSPECTIVE.md.
• Dev: npm run dev (nodemon) in root; npm run build in ui/

---

Project layout

server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
pf.sh                   Account and territory admin (add-user, passwd, set-territory, …)
install.sh              Fresh install
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation;
                        GET /sources/:id/values/:col, /sources/:id/lookup,
                        /sources/:id/dim/:group (+ POST …/refresh), /dim-period/cols
  versions.js           Version CRUD and clone; GET /versions/:id/tags (+ PATCH),
                        /versions/:id/bridge, /versions/:id/values/:col, /versions/:id/table-info
  operations.js         Data streams (GET /versions/:id/data, /versions/:id/agg); loads
                        (baseline, reference); scale, recode, clone
  log.js                GET /api/versions/:id/log, /log/:logid/debug; DELETE and PATCH /log/:logid
  layouts.js            Named pivot layouts — list per version, create, patch, delete
  plans.js              Planned load segments — list, create, patch, delete
lib/
  sql_generator.js      grainOf, buildWhere/buildWhereAny, SCOPE_OPS, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, requireAdmin, sessionUser; `node lib/auth.js hash` CLI
  utils.js              RELATION_COLUMNS_SQL, mapType
setup_sql/
  01_schema.sql         pf schema DDL — idempotent, not a migration runner
  02_auth.sql           pf.app_user + pf.session
  gen_dim_period.sql    Populates pf.dim_period; fiscal year start month is configurable
ui/vendor/              Patched Perspective build and its patches (see its README)
ui/src/
  main.jsx              ThemeProvider, AuthProvider, Gate (mounts the app only once signed in)
  App.jsx               Page routing; source/version selection and its localStorage restore
  theme.jsx             ThemeContext / useTheme
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load, tags panel
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    DepthButtons.jsx    One axis's collapse stepper; both axes are drawn from it
    LayoutMenu.jsx      The Layout ▾ control — Published / Mine, with the write actions
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms; effectiveScaleInputs
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         Forecast list + the 3 steps; collapsible
    StatusBar.jsx       Forecast name · row count · calculating · theme; source on Setup only
    ReadOnlyNotice.jsx  The banner over a page a non-admin may read but not change
    Timeline.jsx        Date-range preview bar for baseline segments

---

Database schema (pf)

• pf.source — registered source tables
• pf.col_meta — column roles: dimension | value | units | date | filter | ignore; dim_group groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); dim_period_col maps a dimension to a pf.dim_period column so date-adjacent values are derived at load time rather than copied raw; in_grain flags dimension/date columns that define the display grain; is_territory marks the column territories are expressed in (one per source); is_key is described under §is_key below
• pf.dim_member — members of each dim_group, keyed on the group's key column; refreshed by POST /sources/:id/dim/:group/refresh
• pf.version — named forecast scenarios; exclude_iters (default ["reference"]) blocks those iter values from all operations; adjustment_segment / adjustment_bucket name new adjustments; is_default (one per source)
• pf.fc_{tname}_{version_id} — one forecast table per version; contains both operational rows (pf_iter = baseline|scale|recode|clone) and reference rows (pf_iter = reference). Indexed on pf_logid, which is how undo, the change-log aggregate and the grain key all find their rows. Tables created before that index was added do not have it.
• pf.log — audit log; every write gets one entry; slice + params stored as jsonb; label / bucket required (log_named CHECK). Stamped at write time: row_count, value_total, units_total, measure_cols (which columns the totals are denominated in), env (the state the intent executed against), sql_text (the statement as executed)
• pf.layout — named Perspective view configs; see §Pivot layouts
• pf.load_plan — load segment definitions, before and apart from having been run; see §Cloning a version's definition
• pf.sql — generated SQL templates per source/operation; tokens substituted at request time
• pf.app_user — login accounts; scrypt pass_hash, is_active, last_login_at, is_admin, territory (jsonb array)
• pf.session — express-session store (connect-pg-simple layout)
• pf.dim_period — calendar lookup table (2018–2035); one row per month keyed on sdat (month start date); provides cal/fiscal year, quarter, and month columns; populated by setup_sql/gen_dim_period.sql

Dead columns, kept but read by nothing: pf.version.unlabeled_load, pf.version.bucket_order, pf.log.seq, pf.source.default_layout.

is_key

Read in four places, and not the one it sounds like — slices are validated against filterCols, which is every dimension plus every date column regardless of is_key.

1. The key of a dim_group — resolveGroup() in routes/sources.js takes members.find(c => c.is_key), the column every other member is keyed on for pf.dim_member
2. Value completion — only is_key columns get a dropdown, and /sources/:id/values/:col refuses anything else
3. Sibling autofill — fires on blur only when is_key && dim_group
4. The dim_period anchor — role === 'date' && is_key && dim_group picks the date whose siblings are derived from the calendar

Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group. When a group has more than one, .find() silently takes the lowest opos — a refresh keyed on the wrong column matches nothing and reports success with zero members. Until the two meanings are separated, a refresh that finds two keys should refuse rather than guess. Rationale: docs/pf_spec.md → §Data model.

Key token substitution tokens

{{fc_table}}, {{where_clause}}, {{exclude_clause}}, {{territory_clause}}, {{logid}}, {{pf_user}}, {{label}}, {{bucket}}, {{value_incr}}, {{units_incr}}, {{pct}}, {{set_clause}}, {{scale_factor}}, {{date_offset}}, {{filter_clause}}

---

Core data flow

Initial load (Forecast view)

Forecast.jsx fetches col_meta first, then picks the endpoint (both in routes/operations.js):

• grain mode (any in_grain column) — GET /api/versions/:id/agg, rows pre-aggregated to the grain, table indexed on pf_gkey
• raw mode (no grain) — GET /api/versions/:id/data, raw forecast rows, table indexed on pf_id

Either way: Arrow IPC binary stream → worker.table(buffer) in Perspective WASM. fetchArrow() handles both.

One batch, not streaming. Per-batch Arrow encoding creates independent dictionaries that crash Perspective WASM on dictionary replacement messages, so the server accumulates all rows and emits one record batch. pg returns bigint/numeric as strings by default — the type parsers in server.js coerce them to numbers; do not remove them.

Display grain

Aggregating to the grain the pivot displays is the load-time fix, and it keeps the native Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (in_grain per column); it is baked into pf.sql at Generate SQL time so load and operations agree. grainOf() in lib/sql_generator.js is the single definition of the grain — routes/log.js calls it directly and Setup.jsx mirrors it. Design: docs/pf_spec.md → §Display-grain pre-aggregation. Why not a DuckDB virtual server: docs/pf_perspective_options.md → §Spike findings.

Log-derived columns

/data and /agg both LEFT JOIN pf.log and emit four columns the forecast table does not itself carry:

• pf_segment — pf.log.label (SEGMENT_EXPR)
• pf_bucket — pf.log.bucket (BUCKET_EXPR)
• pf_note — tag, else note, on a scale/recode/clone; null on loads (NOTE_EXPR)
• pf_op — pf.log.operation

Only segment and bucket are stored names (see §Every log entry is named); the note is derived. SEGMENT_EXPR / BUCKET_EXPR / NOTE_EXPR in lib/sql_generator.js are the single definition, shared with the /data cursor in routes/operations.js — /agg is generated, /data is not, and the two have to agree. LABEL_GROUP_COLS lists the pf.log columns they read, for /agg's GROUP BY. In grain mode pf_logid is part of the grain, so the join adds no rows. The operation routes stamp the same fields onto the rows they push back incrementally, since those come from RETURNING * and would otherwise arrive without them.

Column order is stored text

Perspective orders column groups by the value string, so a "01 - " prefix in pf.log.label / pf.log.bucket, typed by whoever names the segment, is the only lever; nothing derives it. ADJUSTMENT_SEGMENT = '99 - Adjustments' lands last only when the loads carry 01–0n prefixes (digits precede letters). Do not reintroduce ordering as Perspective expressions — restore() replaces expressions wholesale and ExprTK cannot handle non-ASCII labels; DEAD_ORDER_EXPRS in Forecast.jsx strips the old ones from saved layouts. Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.

Every log entry is named

pf.log.label is a row's segment and pf.log.bucket what it counts toward, and every entry carries both — the log_named CHECK refuses one without. A report gets the pivot's segment and bucket by joining the forecast table to pf.log on pf_logid; there is no rule to copy. Never derive them at read time.

They are stamped when the entry is written:

• A load — label is required (the routes answer 400 without one); bucket defaults to the label. loadNames() in routes/operations.js.
• An adjustment — scale, recode and clone have no name inputs, so they are stamped with the version's adjustment_segment / adjustment_bucket, else the built-ins ADJUSTMENT_SEGMENT = 99 - Adjustments and ADJUSTMENT_BUCKET = 04 - Forecast (ADJUSTMENT DEFAULTS in lib/sql_generator.js). adjustmentNames() resolves them once per request; they go into the template's INSERT as {{label}} / {{bucket}} and onto the rows pushed back to the pivot.

The two version defaults are set on the Baseline page under Defaults for new adjustments. Changing one names the next adjustment; existing entries keep what they were written with. Renaming past entries is PATCH /api/log/:logid, one at a time, which renames but refuses to clear. A relabel needs a page reload to show, since the label is part of the aggregated row. A source registered before names were stored needs Generate SQL run once so its load templates write label and bucket at all. Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.

Forecast operations

POST to /api/versions/:id/{scale|recode|clone} → SQL executed with RETURNING * → new rows returned as JSON → pspTable.update(rows) — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since pf_logid is part of pf_gkey those keys are always new, so update() appends and the view re-sums.

Price or volume (plug)

Price does not move unless it is named. A price move is invisible in the figure being edited — the dollars land on target either way — so every default holds price. All resolution is in resolveIncrs() in routes/operations.js; the semantics come from the predecessor Excel model (/opt/forecast_api/VBA/fpvt.frm → calc_val / calc_price).

• Dollars only — plug decides: 'volume' (the default: price holds, units scale in proportion) or 'price' (volume holds, price moves).
• Qty only — dollars follow at the current price. A slice with no units has no price to hold, so the dollars stay put.
• Dollars and qty — both land as named; price falls out.
• A named price (target_price) holds, and the other named measure decides the rest: price alone → qty holds; price + qty → qty lands as named, dollars follow; price + dollars → dollars land, qty follows. All three is refused server-side (400).
• The held price is the adjustable rows' own, never blended with the fixed (excluded-iter) rows, even under target_basis: 'selected' — the scale SQL prorates over the adjustable rows only, so a blended price drives qty and dollars apart (and qty negative when a slice is taken to its fixed total).

The three boxes foot. At most two are inputs and the third is derived. effectiveScaleInputs() in OperationPanel.jsx is the single resolver: the panel renders from it and buildPayload in Forecast.jsx builds the request from it, so what is posted is what is shown. Typing a third drops one — the plug decides when dollars came last, else the older of the other two. The ledger's Result line previews value, units and price by the same rules, and the column that cannot move is labelled · holds.

The plug control is labelled "Moves": it lights whatever the edit moves (both if both do) and takes a click only while it is the one deciding; otherwise it stays on screen, not hidden.

Logged. resolved.method (price | both | volume | the plug) records which side moved, since the request alone cannot say. params also keeps plug and expected (what the Result line promised), beside resolved, so a panel/server disagreement can be found by reading the log — nothing compares them yet. The client's stored preference is keyed pf_scale_plug_v2; do not reuse the old key, which could carry a stale 'price'. Rationale: docs/pf_spec.md → §Operations.

Undo

DELETE /api/log/:logid → removes rows by logid → table.remove() of the affected index values (pf_gkeys in grain mode, pf_ids in raw mode); the view re-sums. No full reload.

---

Axis depth (collapse / expand)

Both axes collapse the same way: a depth in ViewConfig, set through restore() — group_by_depth via applyDepth(), split_by_depth via applySplitDepth(). Both steppers are drawn by components/DepthButtons.jsx from the axis's own field list (groupFull / splitFull), as a stepper ◂ level ▸, the middle button opening the full level list. depth is null until the first layout has been read back; the stepper then shows an em dash.

Depths are 1-based counts of levels to show, on both axes and in DepthButtons (levels[d - 1]); the value crosses unchanged in each direction. The engine agrees (set_depth(n - 1), and the traversal adds the 1 back). Never send 0: it reaches the engine as set_depth(-1) on an unsigned t_depth, underflows, and renders fully expanded. applyDepth floors at 1, so the stepper's "Total" slot shows the first level rather than a bare total — kept deliberately. A real total would need group_rollup_mode: 'total', unverified against the local engine.

A depth lives in the config, so it survives view rebuilds and rides into saved layouts through viewer.save(). This required patching the engine — ViewConfig::apply_update dropped both depth fields; see ui/vendor/0001-apply-depth-fields-on-config-update.patch. Do not go back to imperative view.set_depth() alone: the viewer discards that view on every re-render. split_full is still read on load, for layouts saved under the old truncated-split_by scheme.

Auto-pause is off. initViewer() calls viewer.setAutoPause(false) right after viewer.load(). Pausing deletes the view, so returning to the tab would rebuild the whole grain and lose per-node expansion.

Per-node expansion is not solved. expand(row_index) / collapse(row_index) are set-only with no getter, so they cannot be captured and replayed and are lost on every rebuild. The route is a ViewConfig field of expanded row paths applied in server.cpp. Depth is also whole-axis — one level collapses every group at that level. Rationale: docs/pf_spec.md → §The pivot.

Tag completions are per version

GET /versions/:id/tags — the tags used on this version, with uses, last_used, owners and can_edit; Forecast.jsx refetches on versionId and after every tagged write. A tag is not a row anywhere — only a value repeated across pf.log entries.

Managed on the Baseline page's Tags panel, which lists each with its use count and last-used date. PATCH /versions/:id/tags takes { from, to } and rewrites every entry carrying from; a to that already exists merges, an empty to clears; the panel confirms both. Permission is the pf.log rule applied to every entry the tag is on (a half-applied rename leaves the initiative split), so a row with can_edit false is greyed out. One entry's tag is edited in the Forecast page's change log.

Completion matches on words, not substrings. TagInput in OperationPanel.jsx normalises both sides (case, punctuation, stopwords, light stemming) and says so when a typed tag reduces to an existing one; nothing is merged automatically. Rationale: docs/pf_spec.md → §Bridge and change log.

---

Choosing a forecast

The sidebar lists every source's versions (allVersions, a client-side fan-out over /sources/:id/versions) and sets sourceId from the one picked. versions stays scoped to the selected source, because Baseline and Forecast are — do not widen it. The source picker is only on Setup. Known collision: "forecast" is both the scenario and the third step; undecided.

Where a browser lands. App.jsx restores pf_sourceId / pf_versionId from localStorage, re-checked against the live list. With nothing stored it falls back to pf.version.is_default (one per source, partial unique index version_one_default), not versions[0], which is the newest. A stored selection wins. Setting the default is admin only, in PUT /api/versions/:id, as two statements in one transaction — the index refuses the instant between.

Renaming a version is safe at any time; nothing derives from the name. PUT /api/versions/:id answers 409 on a (source_id, name) collision, and COALESCEs a falsy name back to the existing one, so a blank box is refused client-side.

---

Cloning a version's definition

POST /sources/:id/versions takes clone_from. It copies the template version's settings (description, exclude_iters, the fallback names, bucket_order), its version-scoped layouts, and its load segments — and runs nothing, so the dates can be changed before anything loads.

Not copied: adjustments, in any form (a scale's target was sized against the rows that were there), and source-wide layouts (version_id IS NULL), which already apply to the new version.

pf.load_plan

A plan is a load definition on its own — kind, label, bucket, tag, note, filters, raw_where, date_offset, seq — with run_logid naming the pf.log entry it produced, or null while unrun. Deliberately not pf.log: every row there means a write happened.

run_logid is ON DELETE SET NULL, so undoing a load returns its plan to unrun. The plans list reads the row count back through that FK.

cloneDefinition() in routes/versions.js reads pf.load_plan when the template has plans and otherwise derives them from its baseline/reference log entries (pf.log.params stores filters, raw_where, date_offset).

Run all stops at the first failure — segments are not independent. Rationale: docs/pf_spec.md → §Loads and segments.

---

Pivot layouts

A layout is a named ViewConfig in pf.layout, owned by an account: published (everyone on the forecast lists and can apply it; only its owner or an admin may change it) or private. Scope is the version by default; version_id IS NULL means every version of the source. is_default (at most one per scope, by partial unique index) is what initViewer() restores on a first load — a version-scoped default beating a source-wide one.

Permissions are the pf.log rule verbatim — your own, or an admin's override; can_edit rides on every row and the UI greys out the rest. Applying is never restricted; Save is withheld on a layout that isn't yours, Save as… forks it. No territory clause — a layout is display config.

Local state. LAYOUT_KEY (pf_layout_v{vid}), the unnamed last-used config, stays in localStorage; loadLayouts() lifts the old named list (LAYOUTS_KEY) into pf.layout as private rows once, then clears it.

The dirty dot is a comparison, not a flag. restore() itself fires perspective-config-update, so activeConfigRef holds what the pivot last matched and sameConfig() compares against it, ignoring table and theme.

Grand totals ride in the layout under pf_totals — { row, col }. The engine has no "subtotals but no grand total" mode, so both are hidden in the grid's DOM; captureConfig() adds the key and cleanLayout() strips it before restore(). Without it: row shown, column hidden.

cleanLayout() drops anything the live schema lacks — axes, aggregates (including the weight column of the multi-arg form), theme, pf_totals, dead expressions — dropping the offending entry rather than the layout, since restore() is all-or-nothing. Rationale: docs/pf_spec.md → §The pivot.

---

The Forecast toolbar is two zones

Left, view — Layout ▾ and the Rows / Columns steppers with their Total toggles; all ViewConfig, nothing reaches the server. Right, forecast — Refresh data, Change log, Bridge, Operations. Blue is for actions, pressed grey (.pf-pressed) for state. Fit widths was removed deliberately; do not restore it without fixing auto-size (see git history).

Calculating… in the status bar: Forecast.jsx raises a counter (rebuilds overlap) in applyDepth, applySplitDepth and applyLayout, lowers it after viewer.flush(), and broadcasts pf-pivot-busy on the window for StatusBar. Not driven from perspective-config-update, which arrives after the rebuild. Rationale: docs/pf_spec.md → §The pivot.

---

Slice mechanics

A click (perspective-click) or a dragged region becomes a slice — a {col: value} object. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out, so the panel holds a list sent as slices in operation POST bodies (a single slice is still accepted server-side).

The slice is re-derived by cellFilters() in Forecast.jsx, from detail.row.__ROW_PATH__ (row axis, raw values, so dates stay epoch millis) and the clicked column name (column axis, split by |). The click and a dragged region go through the same derivation. detail.config.filter is used only as the fallback when there is no cell to index the column axis with — it cannot be trusted otherwise, because the engine builds it with a falsy test that drops blank and 0 pivot values, and a dropped key widens the write.

Regions. The datagrid runs in edit_mode: SELECT_REGION (forced on restore). perspective-select carries a ViewWindow — { start_row, end_row, start_col, end_col } — and fires on every mouseover, so the handler records the latest window and a window-level mouseup commits it. A single-cell region is ignored there: perspective-click owns it, and handling it twice would undo a ctrl-click toggle.

A total cell is its displayed amount. The grand-total row resolves to no row dimension, so its slice is whatever the column axis and the pivot's == filters leave — possibly {}. An empty slice is accepted only alongside a non-empty scope; with no filter at all it would be the whole version, and is refused on both sides. buildWhereAny() reads a union containing {} as TRUE, the scope then ANDed on — dropping it would narrow the write.

The pivot's pf_ columns stay in the slice. pf_segment, pf_bucket, pf_note and pf_op are resolved through the log (COMPUTED_SLICE_COLS); pf_iter, pf_user and pf_logid are compared directly (TABLE_SLICE_COLS). Stripping any of them made the write wider than the ledger's total. They narrow a slice but do not make it selective on their own. Duplicates are collapsed before the request — otherwise apply_mode: each applies the change twice.

Selection highlight. The datagrid wipes model._selection_state.selected_areas on every mousedown, so Forecast.jsx keeps areasRef (sliceKey -> rectangles, parallel to slices) and an effect pushes the full set back and redraws after every change. Every deselect path prunes it by live slice key.

A blank dimension value is a value. null and '' are different rows and stay distinct through to the WHERE clause: IS NULL vs = ''. Only undefined means absent (a subtotal row's shorter path, a collapsed column level). On the column axis, t_tscalar::to_string renders a null as the literal text "null" and an empty string as '', so cellFilters() maps "null" → null. The ledger filters a null with is null, never == null (the engine matches on type). The panel prints (null) and (blank), and — only when the column is absent from the slice.

The pivot's own filter travels alongside, as scope — [col, op, value] triples read off viewer.save(), which is also what the ledger totals against. SCOPE_OPS in lib/sql_generator.js is the whole allowed list, under Perspective's operator names; anything outside it is refused, not dropped.

• Substring operators (contains, begins with, ends with and negations) map to ILIKE, matching the engine's case-insensitive t_tscalar::contains; %, _ and \ are escaped out of the term.
• matches / not matches stay refused: the engine is RE2, Postgres is not.
• Every operator that needs a term refuses an empty one (an unfinished filter row arrives as [col, '==', null]).

Limitation: computed columns created by Perspective expressions don't map back to raw rows — only native columns work for slice extraction. Rationale: docs/pf_spec.md → §Selection.

---

Operation SQL patterns

All three operations follow the same structure: insert a pf.log row in a CTE, then insert forecast rows referencing its id. {{where_clause}} is built from the slices; {{exclude_clause}} blocks exclude_iters rows; the territory clause is ANDed on last.

buildWhere() in lib/sql_generator.js turns a slice into SQL: it allows dimension + date columns plus COMPUTED_SLICE_COLS and TABLE_SLICE_COLS, skips unknown non-pf_ keys, and refuses an unknown pf_ key (skipping one would widen the write). A null renders as IS NULL; in a list the null arm is ORed on, since IN (NULL) is never true. Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs. The "must name a filterable column" refusal is assertSelective() in routes/operations.js.

Every date column travels. dataCols is [...dims, ...dateCols, value, units] — never just the first date column, or scale and recode write rows with null secondary dates under correctly labelled period dimensions.

• Scale — distributes value_incr/units_incr proportionally across rows in the slice using window functions; refuses a pool whose net is ~zero relative to its gross (assertProratable)
• Recode — inserts negative rows (zero out original) + positive rows with {{set_clause}} dimension overrides; both share the same logid
• Clone — copies the slice with {{set_clause}} overrides and {{scale_factor}} multiplier; original untouched

Recode takes a date shift too. date_offset moves the put-back leg only, so a shifted recode is a timing move, and with a shift set may be empty. The moved rows' period dimensions are re-derived from pf.dim_period against the shifted date; a row whose date is null keeps the label it had. Without a shift nothing is derived. Clone still blanks those labels on a null date.

A date offset must be whole days. assertInterval() refuses anything finer, because every shifted date is cast back to a date — and Postgres reads a bare number as seconds. 30 days is not a month; use 1 month.

Scale refuses with "Nothing to scale…" and recode with "Nothing to recode…" when there is nothing to write. Rationale: docs/pf_spec.md → §Operations.

---

Authentication

Everything under /api except the auth routes sits behind a session; the React app is only mounted once there is one (Gate in main.jsx).

• Accounts: pf.app_user — scrypt hashes from lib/auth.js, never plaintext. Managed with ./pf.sh add-user | passwd | list-users | disable-user | enable-user; the password is read on stdin and hashed before it reaches psql.
• Sessions: pf.session (connect-pg-simple); revoke by deleting the row. Cookie pf.sid: httpOnly, SameSite=Lax, Secure unless COOKIE_SECURE=false, 12h rolling.
• Config: SESSION_SECRET is required — the server exits at boot without one. TRUST_PROXY (default 1) for req.ip and secure cookies behind the TLS proxy. CORS_ORIGIN is the only way CORS is enabled; never a wildcard with a session cookie.
• Login hardening: routes/auth.js throttles to 10 failures per IP per 15 minutes (in-memory), returns one message for unknown/wrong/disabled alike, and regenerates the session id on success.

Identity is server-side. pf_user, created_by and closed_by come from sessionUser(req), never from the request body.

Setup and Baseline are admin-only to write

requireAdmin (lib/auth.js) is on:

• sources.js — POST /sources, PUT /sources/:id/cols, POST /sources/:id/generate-sql, POST /sources/:id/dim/:group/refresh, DELETE /sources/:id
• versions.js — POST /sources/:id/versions, PUT /versions/:id, POST /versions/:id/{close,reopen}, DELETE /versions/:id
• operations.js — the loads: POST /versions/:id/{baseline,reference}, PUT /versions/:id/baseline/:logid, DELETE /versions/:id/baseline
• plans.js — every write

Not gated, deliberately: reads (gating what a page needs to render turns it into a 403); scale/recode/clone/undo, which carry their own owner and territory rules; PATCH /versions/:id/tags, whose permission is ownership.

The pages stay visible and go read-only — one <fieldset disabled> per page plus a ReadOnlyNotice banner. Setup wraps a real box; Baseline uses display: contents. Baseline's Tags panel sits outside the fieldset. None of this is the permission — the server check is. Setup's list fetches coerce responses to an array: an error body rendered with .map throws during render and unmounts the whole app. Rationale: docs/pf_spec.md → §Access.

Source columns come from pg_catalog

information_schema.columns omits materialized views, and gs.osm_skinny is one. RELATION_COLUMNS_SQL in lib/utils.js is the replacement, used by version creation, source registration and the table preview; it returns the information_schema shape (data_type from format_type with the modifier stripped, precision/scale from atttypmod). The table browser lists from pg_class by relkind. Do not query information_schema.columns for source columns.

Territory scoping

An account sees and changes only its own territory: pf.app_user.territory (jsonb array), is_admin for accounts that see everything, and col_meta.is_territory marks the column (one per source).

Fail closed. buildTerritoryClause() returns FALSE, not TRUE, for an empty list or an unflagged source.

Built from the session, never the request. Unlike scope, the territory predicate is ANDed on last, in sliceUnits() for writes and per route for reads, where nothing in the payload can remove it.

Enforced at:

• /data — clause on the cursor and on the count behind X-Row-Count
• /agg — {{territory_clause}}, applied before the GROUP BY, since the territory column need not be in the grain
• every operation, through sliceUnits()
• /sources/:id/values/:col — completion reads the source table
• DELETE /log/:logid and PATCH /log/:logid — by owner, not territory: your own entries, or an admin's override. Undo removes an entry wholesale, and label/bucket name the pivot's columns for everyone in the version
• recode's set — a scoped account cannot set the territory column at all

The change log lists everyone's entries, with an Everyone / Mine control (resets to Mine on every open, not persisted) and the reader's own entries marked. isMine() is strict authorship, deliberately not canUndo(). The fetch stays unfiltered, since the control's counts need the whole set. It shows an entry's full impact — the totals on pf.log are company-wide, deliberately.

Managed with ./pf.sh set-territory | set-admin | orphan-territory. orphan-territory lists values present in the data that no account owns. Rationale: docs/pf_spec.md → §Access.

Light / dark mode

ui/src/theme.jsx — ThemeProvider (in main.jsx) and useTheme() → { dark, setDark }. Stored as pf_dark in localStorage, else prefers-color-scheme; the effect toggles .dark on <html>. ui/src/index.css defines custom properties under :root and .dark, with Tailwind overrides written as .dark .bg-white { ... } — no Tailwind dark-mode config. Palette is Perspective's "Pro Dark".

The viewer's theme is viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' }), on load and in useEffect([dark, versionId]) — a partial restore, no rebuild. Not setAttribute('theme', …), which does nothing in 5.4.0. cleanLayout drops theme and sameConfig() ignores it: the theme is the reader's choice, never a layout's.

After a change: restart, and Generate SQL

• Restart the server for anything in routes/, lib/ or server.js.
• Generate SQL (Setup, per source) whenever lib/sql_generator.js changes. The templates are stored in pf.sql, so editing the generator changes nothing until they are rebuilt — and a template carrying a token the running code does not substitute fails at the database, which reads as an unrelated client-side error.

Order matters: restart first, then Generate SQL, or the old code writes the templates.

Schema changes are applied to the live database directly and mirrored into setup_sql/ for a fresh install; 01_schema.sql is idempotent but is not a migration runner.

Known issues / active work

• Rows written before the every-date-column fix have null secondary dates (rdate/sdate_e) on scale and recode rows. Deleting and replaying those log entries is the honest repair; a backfill would be inference
• Zero-row clones report success — clone commits an empty log entry and returns rows_affected: 0, where scale and recode refuse
• The change log does not show an entry's id, so there is no way to name one when asking about it
• Territory is read onto the session at login, so a change does not reach a signed-in account until it signs in again
• The bridge dedupes overlapping slices by pf_id, which grain-mode rows do not carry, so overlapping slices double-count there; and it keeps only dimension/date slice keys, dropping the pf_ keys operations now honour (BridgeView.jsx, the per-slice view loop)
• Depth buttons rebuild the view — restore() tears down and rebuilds, a full traversal, where a manual collapse mutates in place. Fix: detect a depth-only change and call view.set_depth() too, at the cost of drift
• Load time is dominated by row count, not payload size — serialising rows out of Postgres into JS dominates. The lever is dynamic grain; see docs/pf_spec.md → §Display-grain pre-aggregation
• Per-node expand/collapse is lost whenever the view rebuilds; see §Axis depth
• Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — delete and recreate the version
• Grain drift: changing in_grain after a load requires Generate SQL + a page reload. routes/log.js derives the grain from live col_meta, so a grain changed mid-session yields pf_gkeys that don't match the loaded table and undo silently removes nothing
• Grain is static per source — an unflagged dimension cannot be pivoted on

Deferred (not in v1)

Baseline replay (not built), approval workflow, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch spike/duckdb-virtual-server. See docs/pf_spec.md → §Open questions.