diff --git a/CLAUDE.md b/CLAUDE.md index effafa2..ead2d07 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,8 +4,8 @@ A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry. -Full spec: `pf_spec.md` -Data transport architecture options: `pf_perspective_options.md` +Full spec, including the rationale and history this file leaves out: `docs/pf_spec.md` +Data transport architecture options: `docs/pf_perspective_options.md` --- @@ -14,7 +14,7 @@ Data transport architecture options: `pf_perspective_options.md` - **Backend:** Node.js / Express (`server.js`) - **Database:** PostgreSQL — isolated `pf` schema - **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/` -- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`. +- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `docs/PERSPECTIVE.md`. - **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/` --- @@ -23,33 +23,44 @@ Data transport architecture options: `pf_perspective_options.md` ``` server.js Express entry point; pg pool; session; type parsers for bigint/numeric +pf.sh Account and territory admin (add-user, passwd, set-territory, …) +install.sh Fresh install routes/ auth.js POST /api/login, /api/logout, GET /api/me; login throttle tables.js GET /api/tables, /api/tables/:schema/:tname/preview - sources.js Source registration, col_meta, SQL generation - versions.js Version CRUD, baseline/reference load, data stream, tag completions - operations.js scale, recode, clone, undo — the core forecast ops - log.js GET /api/versions/:id/log, DELETE /api/log/:logid + sources.js Source registration, col_meta, SQL generation; + GET /sources/:id/values/:col, /sources/:id/lookup, + /sources/:id/dim/:group (+ POST …/refresh), /dim-period/cols + versions.js Version CRUD and clone; GET /versions/:id/tags (+ PATCH), + /versions/:id/bridge, /versions/:id/values/:col, /versions/:id/table-info + operations.js Data streams (GET /versions/:id/data, /versions/:id/agg); loads + (baseline, reference); scale, recode, clone + log.js GET /api/versions/:id/log, /log/:logid/debug; DELETE and PATCH /log/:logid layouts.js Named pivot layouts — list per version, create, patch, delete plans.js Planned load segments — list, create, patch, delete lib/ - sql_generator.js buildFilterClause, token substitution helpers - auth.js scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI - utils.js + sql_generator.js grainOf, buildWhere/buildWhereAny, SCOPE_OPS, token substitution helpers + auth.js scrypt hash/verify, requireAuth, requireAdmin, sessionUser; `node lib/auth.js hash` CLI + utils.js RELATION_COLUMNS_SQL, mapType setup_sql/ - 01_schema.sql pf schema DDL — run once to install + 01_schema.sql pf schema DDL — idempotent, not a migration runner 02_auth.sql pf.app_user + pf.session + gen_dim_period.sql Populates pf.dim_period; fiscal year start month is configurable +ui/vendor/ Patched Perspective build and its patches (see its README) ui/src/ + main.jsx ThemeProvider, AuthProvider, Gate (mounts the app only once signed in) + App.jsx Page routing; source/version selection and its localStorage restore + theme.jsx ThemeContext / useTheme auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login views/ Login.jsx Sign-in form Setup.jsx DB browser, source registration, col_meta editor - Baseline.jsx Version management, baseline workbench, reference load + Baseline.jsx Version management, baseline workbench, reference load, tags panel Forecast.jsx Perspective pivot, selection handling, operation dispatch components/ - DepthButtons.jsx One axis's collapse control; both axes are drawn from it + DepthButtons.jsx One axis's collapse stepper; both axes are drawn from it LayoutMenu.jsx The Layout ▾ control — Published / Mine, with the write actions - OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms + OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms; effectiveScaleInputs BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps) Sidebar.jsx Forecast list + the 3 steps; collapsible StatusBar.jsx Forecast name · row count · calculating · theme; source on Setup only @@ -62,16 +73,19 @@ ui/src/ ## Database schema (`pf`) - **`pf.source`** — registered source tables -- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below); `is_key` is described under §`is_key` below -- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations -- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it. -- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb +- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain**; `is_territory` marks the column territories are expressed in (one per source); `is_key` is described under §`is_key` below +- **`pf.dim_member`** — members of each `dim_group`, keyed on the group's key column; refreshed by `POST /sources/:id/dim/:group/refresh` +- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations; `adjustment_segment` / `adjustment_bucket` name new adjustments; `is_default` (one per source) +- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows. Tables created before that index was added do not have it. +- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb; `label` / `bucket` required (`log_named` CHECK). Stamped at write time: `row_count`, `value_total`, `units_total`, `measure_cols` (which columns the totals are denominated in), `env` (the state the intent executed against), `sql_text` (the statement as executed) - **`pf.layout`** — named Perspective view configs; see §Pivot layouts - **`pf.load_plan`** — load segment definitions, before and apart from having been run; see §Cloning a version's definition - **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time -- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at` +- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`, `is_admin`, `territory` (jsonb array) - **`pf.session`** — express-session store (connect-pg-simple layout) -- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month +- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` + +Dead columns, kept but read by nothing: `pf.version.unlabeled_load`, `pf.version.bucket_order`, `pf.log.seq`, `pf.source.default_layout`. ### `is_key` @@ -89,81 +103,67 @@ regardless of `is_key`. date whose siblings are derived from the calendar Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group. -**When a group has more than one, `.find()` silently takes the lowest `opos`.** -That is not hypothetical: `segment_new` (opos 12) outranked `part` (opos 15) in -the `part` group, so a refresh keyed on a column that is null throughout, matched -nothing, and reported success with zero members. `customer`'s group has four keys -and picks the right one only by `opos` luck. - -The two meanings want separating — a per-group key choice, or a rule that the -group key is the column named by the group (which these groups nearly follow -already, except `sdate` → `sdate_e`). Until then, a refresh that finds two keys -should refuse rather than guess. +**When a group has more than one, `.find()` silently takes the lowest `opos`** — +a refresh keyed on the wrong column matches nothing and reports success with +zero members. Until the two meanings are separated, a refresh that finds two +keys should refuse rather than guess. Rationale: docs/pf_spec.md → §Data model. ### Key token substitution tokens -`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}` +`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{territory_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{label}}`, `{{bucket}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}` --- ## Core data flow ### Initial load (Forecast view) -`Forecast.jsx` fetches col_meta first, then picks the endpoint: +`Forecast.jsx` fetches col_meta first, then picks the endpoint (both in `routes/operations.js`): - **grain mode** (any `in_grain` column) — `GET /api/versions/:id/agg`, rows pre-aggregated to the grain, table indexed on `pf_gkey` - **raw mode** (no grain) — `GET /api/versions/:id/data`, raw forecast rows, table indexed on `pf_id` Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM. `fetchArrow()` handles both. -**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch. +**One batch, not streaming.** Per-batch Arrow encoding creates independent dictionaries that crash Perspective WASM on dictionary replacement messages, so the server accumulates all rows and emits one record batch. pg returns `bigint`/`numeric` as strings by default — the type parsers in `server.js` coerce them to numbers; do not remove them. ### Display grain -Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings. +Aggregating to the grain the pivot displays is the load-time fix, and it keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. **`grainOf()` in `lib/sql_generator.js` is the single definition of the grain** — `routes/log.js` calls it directly and `Setup.jsx` mirrors it. Design: docs/pf_spec.md → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `docs/pf_perspective_options.md` → §Spike findings. -### Segment and note columns -`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table -does not itself carry: +### Log-derived columns +`/data` and `/agg` both LEFT JOIN `pf.log` and emit four columns the forecast +table does not itself carry: -- **`pf_segment`** — `pf.log.label` -- **`pf_bucket`** — `pf.log.bucket` -- **`pf_note`** — the free text on a scale/recode/clone; null on loads +- **`pf_segment`** — `pf.log.label` (`SEGMENT_EXPR`) +- **`pf_bucket`** — `pf.log.bucket` (`BUCKET_EXPR`) +- **`pf_note`** — tag, else note, on a scale/recode/clone; null on loads (`NOTE_EXPR`) +- **`pf_op`** — `pf.log.operation` -Both names are **stored**, not derived — see §Every log entry is named. - -They are deliberately separate: commingling a segment name with an adjustment note -makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join -adds no rows. The operation routes stamp the same two fields onto the rows they push -back incrementally, since those come from `RETURNING *` and would otherwise arrive -without them. +Only segment and bucket are stored names (see §Every log entry is named); the +note is derived. `SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in +`lib/sql_generator.js` are the single definition, shared with the `/data` +cursor in `routes/operations.js` — `/agg` is generated, `/data` is not, and the +two have to agree. `LABEL_GROUP_COLS` lists the `pf.log` columns they read, for +`/agg`'s GROUP BY. In grain mode `pf_logid` is part of the grain, so the join +adds no rows. The operation routes stamp the same fields onto the rows they push +back incrementally, since those come from `RETURNING *` and would otherwise +arrive without them. ### Column order is stored text -Perspective orders column groups by the value string, and `SortDir`'s `col asc` / -`col desc` only reverses that — so Prior Year → Plan → Actual → Forecast is -alphabetical in neither direction and expressible as neither. A `"01 - "` prefix -is the only lever, and it lives in **`pf.log.label`** (and `pf.log.bucket`), -typed by whoever names the segment. Nothing derives it. - -`SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are the -single definition, shared with the `/data` cursor in `routes/operations.js` — -`/agg` is generated, `/data` is not, and the two have to agree. - -The one hardcoded ordinal is `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`, the -name an adjustment is stamped with when its version sets no default, which keeps -those adjustments after every *numbered* segment. That proviso is the -whole scheme, not a caveat on it: ordering is string ordering, so `99` only lands -last once the loads carry `01`–`0n`, and an unnumbered segment sorts after it -(digits precede letters — `9` is `0x39`, `A` is `0x41`). The old `'(adjustment)'` -sorted *first* for the same reason read the other way, `(` being `0x28`. -Relabelling an adjustment's own log row afterwards is how one kind of adjustment -is split out from the rest. +Perspective orders column groups by the value string, so a `"01 - "` prefix in +**`pf.log.label`** / **`pf.log.bucket`**, typed by whoever names the segment, is +the only lever; nothing derives it. `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'` +lands last only when the loads carry `01`–`0n` prefixes (digits precede +letters). Do not reintroduce ordering as Perspective expressions — `restore()` +replaces `expressions` wholesale and ExprTK cannot handle non-ASCII labels; +`DEAD_ORDER_EXPRS` in `Forecast.jsx` strips the old ones from saved layouts. +Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes. ### Every log entry is named `pf.log.label` is a row's segment and `pf.log.bucket` what it counts toward, and -**every entry carries both** — the `log_named` CHECK refuses one without. So a +**every entry carries both** — the `log_named` CHECK refuses one without. A report gets the pivot's segment and bucket by joining the forecast table to -`pf.log` on `pf_logid` and selecting two columns; there is no rule to copy. +`pf.log` on `pf_logid`; there is no rule to copy. Never derive them at read time. They are stamped when the entry is written: @@ -179,83 +179,54 @@ They are stamped when the entry is written: The two version defaults are set on the Baseline page under **Defaults for new adjustments**. Changing one names the *next* adjustment; existing entries keep what they were written with. Renaming past entries is `PATCH /api/log/:logid`, -one at a time, which renames but refuses to clear. - -**What this replaced.** The names used to be derived at read time: a load's -`label`, else `tag`, else `note`, else the version's `unlabeled_load`, else -`Unlabeled`; an adjustment's `label`, else the version default, else the -constant — read through a join on `pf.version`. That made the pivot the only -reader that knew what bucket a row was in. The same CASE was copied into -`/agg`, the `/data` cursor and `rlarp.load_osm_pf` in `/opt/forecast`, and any -direct query had to copy it again. It also meant editing a version's fallback -name silently renamed every past adjustment. The backfill in -`setup_sql/01_schema.sql` stamped the existing entries with that rule. -`pf.version.unlabeled_load` is read by nothing now; the column remains. - -**What this replaced.** The prefix used to be computed client-side, as -Perspective expression columns (`pf_bucket_ord`, `pf_segment_ord`) built from -`pf.log.seq` and `pf.version.bucket_order`. It ordered the pivot and nothing -else, so every other reader disagreed with it; `restore()` replaces -`expressions` wholesale, so it had to be re-applied after every layout load; and -ExprTK's string scanner tests each *byte* with `isprint()`, so a label -containing anything outside printable ASCII could not be ordered at all (`·` is -two bytes, of which `isprint(0xC2)` is false). `DEAD_ORDER_EXPRS` in -`Forecast.jsx` strips the expression names out of layouts saved under that -scheme. `pf.log.seq` and `pf.version.bucket_order` are no longer read; the -columns remain. - -Relabelling now needs a page reload to show, because the label is part of the -aggregated row rather than something the pivot can re-derive. Editing a label -afterwards is a `PATCH /api/log/:logid` and needs nothing regenerated, but a -source registered before this change needs **Generate SQL** run once, so its -stored load templates write `label` and `bucket` onto the log row at all. +one at a time, which renames but refuses to clear. A relabel needs a page +reload to show, since the label is part of the aggregated row. A source +registered before names were stored needs **Generate SQL** run once so its load +templates write `label` and `bucket` at all. Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes. ### Forecast operations POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums. ### Price or volume (`plug`) -**Price does not move unless it is named.** That is the rule the whole section -implements, and it is a rule because a price move is invisible in the figure -being edited: the dollars land on target whether volume followed or not, and -only the implied price says which happened. +**Price does not move unless it is named.** A price move is invisible in the +figure being edited — the dollars land on target either way — so every default +holds price. All resolution is in `resolveIncrs()` in `routes/operations.js`; +the semantics come from the predecessor Excel model +(`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`). -A sales figure alone does not say which of price or volume moved, so scale takes -`plug` — `'volume'` (the default: price holds, units scale in proportion) or -`'price'` (volume holds, price moves). Resolved in `resolveIncrs()` in -`routes/operations.js`; the panel only *offers* it when the edit is dollars-only, -because naming units or price has already answered it, but it stays on screen -greyed out in the other cases rather than vanishing. The semantics come from the -predecessor Excel model, `/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / -`calc_price`: +- **Dollars only** — `plug` decides: `'volume'` (the default: price holds, units + scale in proportion) or `'price'` (volume holds, price moves). +- **Qty only** — dollars follow at the current price. A slice with no units has + no price to hold, so the dollars stay put. +- **Dollars and qty** — both land as named; price falls out. +- **A named price (`target_price`) holds**, and the other named measure decides + the rest: price alone → qty holds; price + qty → qty lands as named, dollars + follow; price + dollars → dollars land, qty follows. **All three is refused** + server-side (400). +- **The held price is the adjustable rows' own**, never blended with the fixed + (excluded-iter) rows, even under `target_basis: 'selected'` — the scale SQL + prorates over the adjustable rows only, so a blended price drives qty and + dollars apart (and qty negative when a slice is taken to its fixed total). -``` -plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange -plug price: fVol = pVol + bVol -``` +**The three boxes foot.** At most two are inputs and the third is derived. +`effectiveScaleInputs()` in `OperationPanel.jsx` is the single resolver: the +panel renders from it and `buildPayload` in `Forecast.jsx` builds the request +from it, so what is posted is what is shown. Typing a third drops one — the plug +decides when dollars came last, else the older of the other two. The ledger's +**Result** line previews value, units and price by the same rules, and the +column that cannot move is labelled `· holds`. -**A qty input follows the same rule.** With dollars unnamed they follow at the -current price rather than holding — `value = (curValue / curUnits) * unitsIncr`. -Holding them made every qty edit a price edit: 10% more cases at the same revenue -is a 9% price cut nobody asked for. A slice with no units has no price to hold, -so there the dollars do stay put and the edit is a pure qty change. +**The plug control is labelled "Moves"**: it lights whatever the edit moves +(both if both do) and takes a click only while it is the one deciding; otherwise +it stays on screen, not hidden. -The two deliberate ways to move price, both of which outrank the above: - -- **name the dollars and the qty together** — price falls out of the two -- **name the price** (`target_price`), which is the Excel form's Edit Price mode: - with a `target_units` alongside, both are inputs and dollars fall out - -The ledger's **Result** line previews value, units and price together using the -same rules, so what you see is what gets written, and the column that cannot move -is labelled `· holds`. - -**What this replaced.** `plug` defaulted to `'price'` — volume holds, price -absorbs — because that is what the API did before the flag existed. So an -unqualified dollar edit moved price, and a qty edit moved price, and the only -edit that did not was one where somebody had picked `'volume'` by hand. The -client's stored preference is keyed `pf_scale_plug_v2` for that reason: a browser -carrying the old `'price'` choice would have gone on silently moving price. +**Logged.** `resolved.method` (`price` | `both` | `volume` | the plug) records +which side moved, since the request alone cannot say. `params` also keeps +`plug` and `expected` (what the Result line promised), beside `resolved`, so a +panel/server disagreement can be found by reading the log — nothing compares +them yet. The client's stored preference is keyed `pf_scale_plug_v2`; do not +reuse the old key, which could carry a stale `'price'`. Rationale: docs/pf_spec.md → §Operations. ### Undo `DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload. @@ -265,561 +236,291 @@ carrying the old `'price'` choice would have gone on silently moving price. ## Axis depth (collapse / expand) Both axes collapse the same way: a **depth in `ViewConfig`**, set through -`restore()`. +`restore()` — `group_by_depth` via `applyDepth()`, `split_by_depth` via +`applySplitDepth()`. Both steppers are drawn by `components/DepthButtons.jsx` from the axis's own field list +(`groupFull` / `splitFull`), as a stepper `◂ level ▸`, the middle button opening +the full level list. `depth` is null until the first layout has been read back; +the stepper then shows an em dash. -- **Rows** — `group_by_depth`, driven by the `Rows` buttons via `applyDepth()` -- **Columns** — `split_by_depth`, driven by the `Columns` buttons via - `applySplitDepth()` +**Depths are 1-based counts of levels to show**, on both axes and in +`DepthButtons` (`levels[d - 1]`); the value crosses unchanged in each direction. +The engine agrees (`set_depth(n - 1)`, and the traversal adds the 1 back). **Never +send 0**: it reaches the engine as `set_depth(-1)` on an unsigned `t_depth`, +underflows, and renders fully expanded. `applyDepth` floors at 1, so the +stepper's "Total" slot shows the first level rather than a bare total — kept +deliberately. A real total would need `group_rollup_mode: 'total'`, unverified +against the local engine. -Both axes are drawn by `components/DepthButtons.jsx` from the axis's own field -list (`groupFull` / `splitFull`), as a **stepper**: `◂ partgroup ▸`, with the -middle button opening the full level list for a jump rather than a step. +A depth lives in the config, so it survives view rebuilds and rides into saved +layouts through `viewer.save()`. **This required patching the engine** — +`ViewConfig::apply_update` dropped both depth fields; see +`ui/vendor/0001-apply-depth-fields-on-config-update.patch`. Do not go back to +imperative `view.set_depth()` alone: the viewer discards that view on every +re-render. `split_full` is still *read* on load, for layouts saved under the +old truncated-`split_by` scheme. -Three controls per axis whatever the depth. It was one button per level, which -grows with the pivot — a 5-by-3 layout rendered ten of them — and the current -level was a blue button among the greys, so reading the state meant scanning. -Before that the row axis was a fixed `0 1 2 3`, which named nothing and offered -levels a two-field `group_by` does not have. - -`depth` is null until the first layout has been read back; the stepper shows an -em dash rather than guessing a level. - -Both are **1-based counts of the levels to show**, and so is the stepper's own -`depth` — `DepthButtons` labels `d` as `levels[d - 1]`. Same unit on both -sides, so the value crosses unchanged in each direction. The engine agrees: -`server.cpp` turns `group_by_depth` into `set_depth(n - 1)` and -`t_traversal::set_depth` adds the 1 back, so `n` renders `n` levels, on both -axes (`ctx2->set_depth(HEADER_COLUMN, column_pivot_depth - 1)` is the same -arithmetic). - -The toolbar used to send `d + 1` and subtract the same 1 when reading a layout -back. That was self-consistent — the stepper always agreed with itself — and -off by one against the grid at every position: picking `partgroup` showed one -level past it. The `+1` was there to dodge `depth 0`, which reaches the engine -as `set_depth(-1)` on an unsigned `t_depth`, underflows, and clamps to *fully -expanded*. `applyDepth` now floors the value at 1 instead, so the stepper's -**"Total" slot still shows the first level rather than a bare total** — the -long-standing wart, kept deliberately rather than traded for the underflow. A -real total needs `group_rollup_mode: 'total'`, which is a field in -`ViewConfigUpdate` but is unverified against the local WASM engine. - -Because a depth lives in the config, it survives every view rebuild, rides into -the persisted and named layouts through `viewer.save()`, and needs nothing -re-applied afterwards. - -**This required patching the engine.** `ViewConfig::apply_update` applied ten -fields and neither depth, so a depth sent through `restore()` was accepted, -deserialized and dropped — and since `Session::update_view_config` returns early -when `apply_update` reports no change, no view was rebuilt at all. See -`ui/vendor/0001-apply-depth-fields-on-config-update.patch`; the vendored build -carries it. - -### What this replaced - -Worth knowing, because a lot of machinery existed to work around it and is now -gone: - -- Row depth used to be imperative — `getView()` then `view.set_depth()` — which - put it on an object the viewer discards whenever it re-renders. Restoring it - meant guessing when that had happened: an `observerShim.js` patching - `window.IntersectionObserver` and `window.ResizeObserver`, focus and - visibility listeners, a retry loop for `getView()` throwing `No table set` - while `getTable()` resolved, and a flag tracking whether the viewer had "gone - away". That guesswork produced three distinct visible faults — the tree fully - expanding on refocus, snapping on any reflow, and snapping when Perspective's - settings sidebar opened. -- Column collapse used to restore a **truncated `split_by`**. The discarded - levels therefore had to be remembered separately (`splitFull`, persisted as - `split_full`), our own collapse had to be told apart from the user rearranging - the pivot (`collapsingRef` plus a prefix test), and the selection was cleared - on every collapse because the axis was changing shape. - -`split_full` is still *read* on load, for layouts saved under the old scheme. - -### Auto-pause is off - -`` auto-pauses by default: an `IntersectionObserver` on -itself plus the document's `visibilitychange` drive `AutoPauseState::apply()`, -and pausing **deletes the view** (`session.set_pause(true)` → -`view_sub.take().delete()`). Returning to the tab is therefore not a redraw but -`restore_and_render()` — a fresh view and a fresh traversal of the whole grain, -which on a large one is a multi-second chug on every tab switch, and takes any -per-node expansion with it. - -`initViewer()` calls `viewer.setAutoPause(false)` right after `viewer.load()`. -Nothing updates the table while the tab is hidden — every operation is driven -from this page — so the pause bought nothing and cost a rebuild. The view is now -held while the tab is backgrounded. - -### Still not solved: per-node expansion - -Expanding one specific branch is view state with no config representation, and -the API has **no getter**: - -``` -expand(row_index: number): Promise -collapse(row_index: number): Promise -``` - -It can be set but not read, so it cannot be captured and replayed — this is why -it is lost on every rebuild, and why no client-side fix has worked. The honest -route is a `ViewConfig` field carrying expanded row *paths* (indices shift as -the tree opens), applied in `server.cpp` where the depths are. Bigger than the -depth patch: it needs a way to enumerate expanded nodes in the C++ traversal, a -proto field, and the apply step. - -The fork's own `header_click.ts` / `expand_column` / `collapse_column` is the -column-axis equivalent and has the same limitation. - -**Limitation that remains either way:** depth is whole-axis. Excel can collapse -2025 while 2026 stays expanded; a depth collapses every group at that level -together. `columns` selects which *measures* appear, not individual split -combinations. +**Auto-pause is off.** `initViewer()` calls `viewer.setAutoPause(false)` right +after `viewer.load()`. Pausing deletes the view, so returning to the tab would +rebuild the whole grain and lose per-node expansion. +**Per-node expansion is not solved.** `expand(row_index)` / `collapse(row_index)` +are set-only with no getter, so they cannot be captured and replayed and are +lost on every rebuild. The route is a `ViewConfig` field of expanded row paths +applied in `server.cpp`. Depth is also whole-axis — one level collapses every +group at that level. Rationale: docs/pf_spec.md → §The pivot. ## Tag completions are per version -`GET /versions/:id/tags` — the tags already used on **this** version, with their -use counts, for completing the next adjustment. `Forecast.jsx` refetches on -`versionId` and after every tagged write. +`GET /versions/:id/tags` — the tags used on **this** version, with `uses`, +`last_used`, `owners` and `can_edit`; `Forecast.jsx` refetches on `versionId` +and after every tagged write. A tag is not a row anywhere — only a value +repeated across `pf.log` entries. -It used to be `GET /sources/:id/tags`, joined through `pf.version` and grouped -across every version of the source. A tag names an initiative *inside* a -scenario, so that put last year's work in front of this year's and reported a -`uses` count earned on a version the tag did not belong to. The cost of the -narrower scope is that a fresh version completes from nothing until it has a tag -of its own — which is the honest state of it, since cloning a definition copies -no adjustments. - -The `pf_bucket` datalist on the Baseline page was already per version -(`bucketsInUse`), so the two now agree. - -**The list is managed on the Baseline page, not the Forecast one.** A tag is -not a row anywhere — only a value repeated across this version's `pf.log` -entries — so there is no `/tags/:tag` to address and no list to curate ahead of -use: a tag exists because somebody typed it on an adjustment. - -`PATCH /versions/:id/tags` takes `{ from, to }` and rewrites every entry -carrying `from`. `to` naming a tag that already exists **merges** the two, which -is usually the reason to be there: the bridge groups on this column, so -`Pull Forward` and `Pulled Forward` are one initiative reported as two. An empty -`to` clears the tag. The Baseline panel confirms both before sending, since -neither is visible as a diff afterwards. - -Permission is the `pf.log` rule applied to *every* entry the tag is on, not -most of them — a half-applied rename leaves the initiative split, which is the -state the rename was fixing. `GET /versions/:id/tags` therefore returns `owners` -and `can_edit`, and the panel greys the row out rather than offering a rename -that answers 403. Editing one entry's tag is still the change log on the -Forecast page; this is the whole-version view of the same column. +**Managed on the Baseline page's Tags panel**, which lists each with its use +count and last-used date. `PATCH /versions/:id/tags` takes `{ from, to }` and +rewrites every entry carrying `from`; a `to` that already exists **merges**, an +empty `to` clears; the panel confirms both. Permission is the `pf.log` rule +applied to *every* entry the tag is on (a half-applied rename leaves the +initiative split), so a row with `can_edit` false is greyed out. One entry's +tag is edited in the Forecast page's change log. **Completion matches on words, not substrings.** `TagInput` in -`OperationPanel.jsx` replaces the native datalist on the tag field: both sides -are lowercased, stripped of punctuation, relieved of stopwords and lightly -stemmed, so `Pulled Forward` finds `Pull Forward` and `moved to different sku` -finds `moved to a different sku` — neither of which a substring match reaches, -one inserted word being enough to hide the tag that already exists. Typing -something that reduces to an existing tag without being it says so outright -(*"Already used as … — use that"*), because the ledger will otherwise report the -two as separate initiatives. Nothing is merged automatically; the suggestion is -offered and the typing stays yours. The change log's inline tag cell still uses -the plain datalist. - -**Completion is otherwise the list and nothing else.** The operation panel used to -carry six quick-pick pills under the tag field as well. They wrapped to two rows -as soon as the tags were longer than a word, and the panel docked at the bottom -has a fixed height, so they pushed the form past its own border. The datalist -costs no space, is not capped at six, and now shows the tag alone — an option's -text content renders as a second column beside the value, so the use count was -widening every row of the pick list to carry a number nobody picks on. `uses` -and `last_used` still come back from the route; nothing reads them. +`OperationPanel.jsx` normalises both sides (case, punctuation, stopwords, light +stemming) and says so when a typed tag reduces to an existing one; nothing is +merged automatically. Rationale: docs/pf_spec.md → §Bridge and change log. --- ## Choosing a forecast -The forecasts are listed in the **sidebar**, above the three steps, each row -carrying its status and a star for the default. They were two dropdowns in the -status bar, which is otherwise read-only — so the one pair of controls in it did -not read as controls at all. - -**Not nested under the `Forecast` step**, which is where the idea started: -Baseline works on the same version, so hanging the list off one step would -either misrepresent the scope or duplicate it under two. It reads *what am I -working on*, then *what step am I on*. - -**The source is not in the nav.** `gs.osm_skinny` means nothing to someone -editing a forecast. The sidebar lists every source's versions and sets -`sourceId` from the one picked, since `pf.version.source_id` comes with it — so -choosing a forecast settles the source, and the picker is only offered on Setup, -the one page where the raw table name is the right label. - +The **sidebar** lists every source's versions (`allVersions`, a client-side +fan-out over `/sources/:id/versions`) and sets `sourceId` from the one picked. `versions` stays scoped to the selected source, because Baseline and Forecast -are; the sidebar gets its own `allVersions` instead, so widening it does not -change what `versions` means everywhere else. It is a client-side fan-out over -`/sources/:id/versions` rather than a `GET /versions` — one request, at one -registered source. +are — do not widen it. The source picker is only on Setup. Known collision: +"forecast" is both the scenario and the third step; undecided. -The status bar keeps what follows *from* the selection: the forecast's name and -its row count. The table written to is in the hover panel, still click-to-copy, -because pasting it into psql is the reason it was ever on screen. +**Where a browser lands.** `App.jsx` restores `pf_sourceId` / `pf_versionId` +from `localStorage`, re-checked against the live list. With nothing stored it +falls back to **`pf.version.is_default`** (one per source, partial unique index +`version_one_default`), not `versions[0]`, which is the newest. A stored +selection wins. Setting the default is **admin only**, in `PUT /api/versions/:id`, +as two statements in one transaction — the index refuses the instant between. -**Known collision:** "forecast" is both the scenario and the third step. The -scenario should own the word — it is what people name and say aloud — which -leaves the step wanting a different one (`Adjust`). Undecided. - -## Where a browser lands - -`App.jsx` restores `pf_sourceId` / `pf_versionId` from `localStorage` and -re-checks both against the live list. What it falls back to when there is -nothing stored — a new account, a new machine, a cleared browser — is -**`pf.version.is_default`**, one per source by partial unique index -(`version_one_default`), the way `pf.layout.is_default` works. - -It used to fall back to `versions[0]`, and the list is ordered -`created_at DESC`, so that meant *the newest version* — whatever somebody -created last, a scratch copy as readily as the plan. Accounts that never pick a -version themselves, which is every sales account, therefore followed the -forecasting work around. - -**Admin only**, enforced in `PUT /api/versions/:id`: it decides where everyone -else lands, so it follows the `pf.log` rule rather than being the version's own -setting like its name. The checkbox on the Baseline page's **This forecast** -panel is greyed out for other accounts rather than offering a click that -answers 403. The sidebar's forecast list marks the default with a star. - -**A stored selection still wins.** The default is the fallback, not an -override — someone who has been working in another version stays there. The -corollary is that an account which already landed in the wrong scenario keeps -landing there until it switches once. - -Setting the default is two statements — clear the old, set the new — so the -route runs them in one transaction; the index refuses the instant in between. - -## Renaming a version - -Safe at any time, including on a closed one. Nothing derives from the name: the -forecast table is `fc_{tname}_{version_id}`, `pf.log` and `pf.layout` key on the -id, and the UI's `localStorage` keys are id-based. The Baseline page's **This -forecast** panel saves `name` and `description` on blur through -`PUT /api/versions/:id`, which has accepted both all along — it was only ever -called for the fallback names. - -Two edges the panel handles: `(source_id, name)` is unique, so the route -answers 409 on a collision rather than a raw Postgres error; and the `UPDATE` +**Renaming a version** is safe at any time; nothing derives from the name. +`PUT /api/versions/:id` answers 409 on a `(source_id, name)` collision, and COALESCEs a falsy name back to the existing one, so a blank box is refused -client-side rather than looking saved and changing nothing. +client-side. --- ## Cloning a version's definition `POST /sources/:id/versions` takes `clone_from`. It copies the template -version's settings (`description`, `exclude_iters`, the three fallback names, +version's settings (`description`, `exclude_iters`, the fallback names, `bucket_order`), its **version-scoped** layouts, and its load segments — and -**runs nothing**. The reason to clone a definition is to change the dates before -anything loads, and a clone that fetched two million rows first would have to be -undone before it could be edited. +**runs nothing**, so the dates can be changed before anything loads. -Not copied: adjustments, in any form. A scale's target is a figure sized against -the rows that were there, so replaying one against a different load is -arithmetic nobody asked for. Source-wide layouts (`version_id IS NULL`) are not -copied either — they already apply to the new version, and a copy would -duplicate the name. +Not copied: adjustments, in any form (a scale's target was sized against the +rows that were there), and source-wide layouts (`version_id IS NULL`), which +already apply to the new version. ### `pf.load_plan` -A segment used to exist only once it had been run: a `pf.log` entry with -forecast rows behind it. A plan is the definition on its own — `kind`, `label`, -`bucket`, `tag`, `note`, `filters`, `raw_where`, `date_offset`, `seq` — with -`run_logid` naming the `pf.log` entry it produced, or null while it is still a -plan. +A plan is a load definition on its own — `kind`, `label`, `bucket`, `tag`, +`note`, `filters`, `raw_where`, `date_offset`, `seq` — with `run_logid` naming +the `pf.log` entry it produced, or null while unrun. **Deliberately not +`pf.log`**: every row there means a write happened. -**Deliberately not `pf.log`.** Every row there means a write happened; holding -unexecuted intent in the audit log would cost it that meaning, and the change -log would list entries that never ran. +`run_logid` is `ON DELETE SET NULL`, so undoing a load returns its plan to unrun. +The plans list reads the row count back through that FK. -`run_logid` is `ON DELETE SET NULL`, so undoing a load returns its plan to unrun -rather than deleting the definition with it. The plans list reads the row count -back through that FK, which is how it tells run from unrun. +`cloneDefinition()` in `routes/versions.js` reads `pf.load_plan` when the +template has plans and otherwise derives them from its `baseline`/`reference` +log entries (`pf.log.params` stores `filters`, `raw_where`, `date_offset`). -**Derivation from an old version.** `cloneDefinition()` in `routes/versions.js` -reads `pf.load_plan` when the template has plans and falls back to deriving them -from its `baseline`/`reference` log entries when it does not — which is every -version built before plans existed. `pf.log.params` already stores `filters`, -`raw_where` and `date_offset`, so this is a read, not a reconstruction. - -**Run all stops at the first failure.** The segments of a definition are not -independent — a baseline followed by references that date-shift off it — so -carrying on past a broken one builds something nobody described. +**Run all stops at the first failure** — segments are not independent. +Rationale: docs/pf_spec.md → §Loads and segments. --- ## Pivot layouts -A layout is a named `ViewConfig`, stored in **`pf.layout`** and owned by an -account. Two kinds, one table: - -- **published** — everyone on the forecast lists it and can apply it; only its - owner or an admin may change it -- **private** — yours, nobody else lists it - -Scope is the **version** by default, because that is where people enter the app. -`version_id IS NULL` means the layout applies to every version of the source; -that is where the old source default went, and what a brand-new version picks up -before anyone has published anything for it. `is_default` (at most one per scope, -by partial unique index) is what `initViewer()` restores on a first load — a -version-scoped default beating a source-wide one, the narrower answer winning. +A layout is a named `ViewConfig` in **`pf.layout`**, owned by an account: +**published** (everyone on the forecast lists and can apply it; only its owner +or an admin may change it) or **private**. Scope is the **version** by default; +`version_id IS NULL` means every version of the source. `is_default` (at most +one per scope, by partial unique index) is what `initViewer()` restores on a +first load — a version-scoped default beating a source-wide one. **Permissions are the `pf.log` rule verbatim** — your own, or an admin's -override, and the UI greys out the rest rather than offering a click that answers -403. `can_edit` rides on every row so the menu knows which. Applying is never -restricted: the guarantee is that a published layout cannot be *changed* out from -under people, not that it cannot be adapted — Save is withheld on a layout that -isn't yours, Save as… forks it into your own. +override; `can_edit` rides on every row and the UI greys out the rest. Applying +is never restricted; Save is withheld on a layout that isn't yours, Save as… +forks it. No territory clause — a layout is display config. -**No territory clause.** A layout is display config, and territory restricts rows, -not columns; `cleanLayout()` already drops anything the live schema lacks. - -**What is still local.** `LAYOUT_KEY` (`pf_layout_v{vid}`) — the unnamed -last-used config — stays in `localStorage`, because it is per-browser session -continuity rather than a thing anyone names or shares. `LAYOUTS_KEY` -(`pf_layouts_v{vid}`) is the old named list; `loadLayouts()` lifts it into -`pf.layout` as private rows once per version and then clears the key. +**Local state.** `LAYOUT_KEY` (`pf_layout_v{vid}`), the unnamed last-used +config, stays in `localStorage`; `loadLayouts()` lifts the old named list +(`LAYOUTS_KEY`) into `pf.layout` as private rows once, then clears it. **The dirty dot is a comparison, not a flag.** `restore()` itself fires -`perspective-config-update`, so anything set unconditionally in that handler -would light up the moment a layout was applied. `activeConfigRef` holds what the -pivot last matched and `sameConfig()` compares against it, ignoring `table` -(the per-load table name, different on every refresh). +`perspective-config-update`, so `activeConfigRef` holds what the pivot last +matched and `sameConfig()` compares against it, ignoring `table` and `theme`. -**What this replaced.** Named layouts lived only in `localStorage` — invisible to -anyone else, gone on another machine. The one server-side layout was -`pf.source.default_layout`, a single anonymous blob that `PUT -/sources/:id/default-layout` let *any* account overwrite for *every* account: a -published layout with no owner. That route is gone; the column remains, migrated -and read by nothing. +**Grand totals ride in the layout under `pf_totals`** — `{ row, col }`. The +engine has no "subtotals but no grand total" mode, so both are hidden in the +grid's DOM; `captureConfig()` adds the key and `cleanLayout()` strips it before +`restore()`. Without it: row shown, column hidden. -**Grand totals ride in the layout under `pf_totals`** — `{ row, col }`, the -toolbar's *Total row* / *Total column* toggles. The engine has no rollup mode -for "subtotals but no grand total", so both are hidden in the grid's DOM rather -than in `ViewConfig`; `captureConfig()` adds the key and `cleanLayout()` strips -it before `restore()`. A layout without it gets the old behaviour: row shown, -column hidden. - -`cleanLayout()` guards `aggregates` along with the axes — including the weight -column of the multi-arg form — and drops the offending *entry* rather than the -layout, since `restore()` is all-or-nothing. +**`cleanLayout()`** drops anything the live schema lacks — axes, `aggregates` +(including the weight column of the multi-arg form), `theme`, `pf_totals`, dead +expressions — dropping the offending *entry* rather than the layout, since +`restore()` is all-or-nothing. Rationale: docs/pf_spec.md → §The pivot. --- ## The Forecast toolbar is two zones -Left: **view** — `Layout ▾` and the `Rows` and `Columns` depth steppers, each -with its `Total` toggle. Nothing there reaches the server or changes the -forecast; it is all `ViewConfig` against rows already loaded. +Left, **view** — `Layout ▾` and the `Rows` / `Columns` steppers with their +`Total` toggles; all `ViewConfig`, nothing reaches the server. Right, +**forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. **Blue is +for actions, pressed grey (`.pf-pressed`) for state.** `Fit widths` was removed +deliberately; do not restore it without fixing auto-size (see git history). -Right: **forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. -These re-run the load, open something over the pivot, or write to the version. - -The distance between them is the separation. Each zone used to be an outlined -box as well, the view one on a `gray-50` tint — too faint to see against white, -and in dark mode the two boxes read as siblings. Within the view zone each axis -is one joined control, `◂ | level | ▸`, in the style of the operation panel's -`Scale | Recode | Clone`, followed by its `Total` chip. - -**Blue is for actions, pressed grey for state.** `Save` and `Post` are blue; -the `Total` chips and `Operations` while its panel is open use `.pf-pressed`. -`Operations` used to relabel itself `Hide panel` in a blue outline, which made -it look like a second Save. - -They were one undifferentiated row separated by identical hairlines, which made -`Fit` and `Bridge` read as peers when one is a column width and the other is the -version's history. Three weights do the separating now, and they were all once -the same: `.pf-section-label` names a group (`Layout`, `Rows`, `Columns`) and is -heavier and darker than the controls under it, `.pf-group-rule` separates groups -*within* a zone, and the gap between the zones separates them. A label that -matched its buttons, and a group rule that matched a zone border, is what made -the row read as one undifferentiated thing. - -**`Fit widths` is gone**, button and `fitColumns` both, for three independent -reasons. Auto-size measures the leaf header over the rows *currently in the -viewport*, so anything below the fold never counted; `draw({invalid_columns})` -passes an option `DrawOptions` does not have, and `draw()` is throttled, so -clicking during a pending draw cleared the widths without re-measuring and the -columns fell to their 52/60px floors; and the `persistLayout` after it saved -nothing, since `saveColumnSizes()` returns only overrides, which `resetAutoSize()` -had just emptied — and this fork discards `column_size_override` whenever -`split_by` is non-empty, which here is always. Recoverable from git if it is ever -worth fixing rather than dropping. - -**A `Calculating…` indicator** sits in the app's status bar while the pivot -rebuilds. Perspective has its own beaker, but it is rendered only while the -settings panel is open, so it vanishes exactly when nothing else tells you to -wait. `Forecast.jsx` raises a counter — a counter, because rebuilds overlap — -in `applyDepth`, `applySplitDepth` and `applyLayout`, lowers it after -`viewer.flush()`, and broadcasts `pf-pivot-busy` on the window for `StatusBar` -to render, the same way `pf-data-changed` already travels between them. - -Deliberately **not** driven from `perspective-config-update`, which is the -obvious hook and is the wrong one: measured on a depth change over a large -grain, the event arrives *after* the rebuild — seconds late — and the flush that -follows resolves in 6ms. It reports the end of the work. The cost is that -sorting or filtering from the viewer's own chrome shows nothing, because the app -is never told those began. +**`Calculating…`** in the status bar: `Forecast.jsx` raises a counter (rebuilds +overlap) in `applyDepth`, `applySplitDepth` and `applyLayout`, lowers it after +`viewer.flush()`, and broadcasts `pf-pivot-busy` on the window for `StatusBar`. +**Not** driven from `perspective-config-update`, which arrives after the +rebuild. Rationale: docs/pf_spec.md → §The pivot. --- ## Slice mechanics -When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side). +A click (`perspective-click`) or a dragged region becomes a **slice** — a +`{col: value}` object. A plain click replaces the selection; ctrl/⌘/shift-click +toggles a slice in or out, so the panel holds a **list** sent as `slices` in +operation POST bodies (a single `slice` is still accepted server-side). -Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow** — `{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle. +**The slice is re-derived by `cellFilters()` in `Forecast.jsx`**, from +`detail.row.__ROW_PATH__` (row axis, raw values, so dates stay epoch millis) +and the clicked column name (column axis, split by `|`). The click and a dragged +region go through the same derivation. `detail.config.filter` is used only as +the fallback when there is no cell to index the column axis with — it cannot be +trusted otherwise, because the engine builds it with a falsy test that drops +blank and `0` pivot values, and a dropped key **widens** the write. -Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. **A total cell is its displayed amount.** The grand-total row resolves to no row dimension, so its slice is whatever the column axis and the pivot's `==` filters leave — possibly `{}`, meaning *everything the pivot's filter shows*. An empty slice is accepted only alongside a non-empty `scope` (`normalizeSlices()` / `assertSelective()` in `routes/operations.js`); with no filter at all it would be the whole version, and is refused on both sides. `buildWhereAny()` reads a union containing `{}` as `TRUE`, the scope then ANDed on — dropping it would narrow the write to the other cells. +**Regions.** The datagrid runs in `edit_mode: SELECT_REGION` (forced on +restore). `perspective-select` carries a **ViewWindow** — +`{ start_row, end_row, start_col, end_col }` — and fires on every mouseover, so +the handler records the latest window and a window-level `mouseup` commits it. +A single-cell region is ignored there: `perspective-click` owns it, and handling +it twice would undo a ctrl-click toggle. -**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree. +**A total cell is its displayed amount.** The grand-total row resolves to no +row dimension, so its slice is whatever the column axis and the pivot's `==` +filters leave — possibly `{}`. An empty slice is accepted only alongside a +non-empty `scope`; with no filter at all it would be the whole version, and is +refused on both sides. `buildWhereAny()` reads a union containing `{}` as +`TRUE`, the scope then ANDed on — dropping it would narrow the write. -**The `pf_` columns the pivot shows are slice columns too.** `pf_segment`, `pf_bucket`, `pf_note` and `pf_op` resolve to log ids (`COMPUTED_SLICE_COLS`); `pf_iter`, `pf_user` and `pf_logid` are forecast-table columns compared directly (`TABLE_SLICE_COLS`). They used to be stripped — `pf_iter` on the theory that two cells differing only by iter band are the same slice — while the ledger filtered on them, so under the default layout (grouped by `pf_iter`) a click on a customer's `scale` row totalled the scale rows and wrote to every row the customer has. They narrow a slice but do not make one selective on their own: `pf_iter = scale` alone is every adjustment in the version, and `assertSelective` still wants a dimension, date, `pf_segment` or `pf_bucket`. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice. +**The pivot's `pf_` columns stay in the slice.** `pf_segment`, `pf_bucket`, +`pf_note` and `pf_op` are resolved through the log (`COMPUTED_SLICE_COLS`); +`pf_iter`, `pf_user` and `pf_logid` are compared directly (`TABLE_SLICE_COLS`). +Stripping any of them made the write wider than the ledger's total. They narrow +a slice but do not make it selective on their own. Duplicates are collapsed +before the request — otherwise `apply_mode: each` applies the change twice. -A scale also carries `expected` — the panel's Result line as a change (`value_incr`, `units_incr`, `price`, `basis`) — which the log keeps beside `resolved` so a disagreement between panel and server can be found by reading it. Nothing compares them yet. +**Selection highlight.** The datagrid wipes `model._selection_state.selected_areas` +on every mousedown, so `Forecast.jsx` keeps `areasRef` (`sliceKey -> rectangles`, +parallel to `slices`) and an effect pushes the full set back and redraws after +every change. Every deselect path prunes it by live slice key. -**A blank dimension value is a value.** Both blanks occur in this data and they -are different rows — on `fc_osm_skinny_34`, `partgroup` is null on 51,964 rows -and `''` on 111,696 — so `null` and `''` stay distinct from the clicked cell -through to the WHERE clause, where one becomes `IS NULL` and the other `= ''`. -Only `undefined` means absent, which is how a subtotal row (a shorter row path) -and a collapsed column level still read as "not part of the slice". +**A blank dimension value is a value.** `null` and `''` are different rows and +stay distinct through to the WHERE clause: `IS NULL` vs `= ''`. Only `undefined` +means absent (a subtotal row's shorter path, a collapsed column level). On the +column axis, `t_tscalar::to_string` renders a null as the literal text `"null"` +and an empty string as `''`, so `cellFilters()` maps `"null"` → null. The ledger +filters a null with `is null`, never `== null` (the engine matches on type). The +panel prints `(null)` and `(blank)`, and `—` only when the column is absent from +the slice. -They used to be dropped, and a dropped key does not narrow the write — it widens -it. Clicking the blank `partgroup` node under `ALTMAN PLANTS` selected the -customer's whole $27.6M rather than the $6.2M under the node clicked, and the -ledger agreed with it, because the panel totals through the same filter list. -The source of the drop is the **engine**: `getCellConfig` builds the click's -filter list with `pivot_value ? [pivot, '==', pivot_value] : undefined`, so the -blank is gone before `perspective-click` is dispatched — and being a falsy test -it takes a dimension whose value is `0` with it. `cellFilters()` in -`Forecast.jsx` therefore re-derives both axes from `detail.row.__ROW_PATH__` and -the clicked column name rather than trusting that list; the click and a dragged -region now go through the same derivation. The ledger filters a null with -`is null` rather than `== null`, since the engine matches on type and a null -threshold against a string column is the same dropped predicate again. +**The pivot's own filter travels alongside, as `scope`** — `[col, op, value]` +triples read off `viewer.save()`, which is also what the ledger totals against. +`SCOPE_OPS` in `lib/sql_generator.js` is the whole allowed list, under +Perspective's operator names; anything outside it is **refused, not dropped**. -`buildWhere()` renders a null as `IS NULL`; it used to fall through `esc()` to -`= ''`, which selected *the other blank* — the slice named one node and the -operation wrote the other. In a list, the null arm is ORed on rather than folded -in, because `IN (NULL)` is never true. +- Substring operators (`contains`, `begins with`, `ends with` and negations) map + to **`ILIKE`**, matching the engine's case-insensitive `t_tscalar::contains`; + `%`, `_` and `\` are escaped out of the term. +- `matches` / `not matches` stay refused: the engine is RE2, Postgres is not. +- Every operator that needs a term refuses an empty one (an unfinished filter + row arrives as `[col, '==', null]`). -The panel prints `(null)` and `(blank)` rather than nothing, and `—` only when -the column is genuinely absent from the slice: a slice isolated to the blank node -and one the column plays no part in are opposite statements and looked identical. - -**A blank on the column axis is read back from the column name.** A column -name is its values joined with `|`, each rendered by `t_tscalar::to_string` in -`scalar.cpp`, which gives `''` for an empty string and the literal text `"null"` -for a null. So `cellFilters()` maps `"null"` to null and `''` to `''`, and both -blanks are isolatable on either axis. The one ambiguity left is a dimension -whose value is really the text `null`; none exists in this data. - -This used to be believed impossible — the claim was that a null joined as `''` -too — so `''` cells were refused, and `"null"` was taken at its word: the slice -became `segment_new = 'null'`, matched no row, and the adjustment reported zero. - -**The pivot's own filter travels alongside, as `scope`.** A slice is -`{col: value}` and can only mean equality, so a view narrowed to -`sseas_e <= 2027` has no slice form at all — the filter is read off -`viewer.save()` and sent as `[col, op, value]` triples, which is also what the -ledger totals against, so the panel and the write agree. `SCOPE_OPS` in -`lib/sql_generator.js` is the whole allowed list, under **Perspective's** -operator names; anything outside it is refused rather than dropped, since a -scope silently discarded is a write wider than the panel that authorised it. - -The substring operators — `contains`, `begins with`, `ends with` and their -negations — map to **`ILIKE`**, not `LIKE`, because that is what the engine -does: `t_tscalar::contains` lowercases both sides before `std::string::find`. -On `partd`, `contains 'Keysto'` matches 16,864 rows case-insensitively and 35 -case-sensitively, so `LIKE` would have written a fortieth of what the grid -showed. Narrower is no safer than wider — either way it is not the set the -ledger counted. The term is literal in the engine, so `%`, `_` and `\` are -escaped out of it before it becomes a pattern. - -`matches` / `not matches` stay refused. They are RE2 in the engine, and -Postgres regexes are not RE2 — `\b` is a word boundary in one and a backspace -in the other — so a pattern accepted by both could quietly match a different -set of rows, which is worse than a 400. - -Every operator that needs a term refuses an empty one: an unfinished filter row -in the pivot arrives as `[col, '==', null]`, and `ILIKE '%%'` or `= ''` would -have read as "every row" or "no rows" rather than as the mistake it is. - -**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction. +**Limitation:** computed columns created by Perspective expressions don't map +back to raw rows — only native columns work for slice extraction. Rationale: +docs/pf_spec.md → §Selection. --- ## Operation SQL patterns -All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows. +All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slices; `{{exclude_clause}}` blocks `exclude_iters` rows; the territory clause is ANDed on last. -**Every date column travels, not just the first.** `dataCols` is -`[...dims, ...dateCols, value, units]`. It used to be `dateCols[0]`, so on a -source with order, requested and ship dates, scale and recode wrote rows whose -`rdate` and `sdate_e` were **null** — and because the derived period dimensions -(`rseas`, `smon_e`) are dimensions and were copied with the rest, the labels -read correctly while the dates under them were gone. Both columns are -`role = 'date'` and therefore sliceable, so an adjustment scoped to a ship date -matched no adjustment row that had ever been written. Clone and the loads were -always right. The wider GROUP BY costs almost nothing on `osm_skinny` -(971,185 → 971,352 groups) because `ordnum`/`ordline` already make a row -near-unique. +**`buildWhere()` in `lib/sql_generator.js`** turns a slice into SQL: it allows +`dimension` + `date` columns plus `COMPUTED_SLICE_COLS` and `TABLE_SLICE_COLS`, +**skips** unknown non-`pf_` keys, and **refuses** an unknown `pf_` key (skipping +one would widen the write). A null renders as `IS NULL`; in a list the null arm +is ORed on, since `IN (NULL)` is never true. Values are escaped but not +parameterized — consistent with existing patterns, debuggable in pg logs. The +"must name a filterable column" refusal is **`assertSelective()`** in +`routes/operations.js`. -- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions +**Every date column travels.** `dataCols` is `[...dims, ...dateCols, value, units]` +— never just the first date column, or scale and recode write rows with null +secondary dates under correctly labelled period dimensions. + +- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions; refuses a pool whose net is ~zero relative to its gross (`assertProratable`) - **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid - **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched -**Recode takes a date shift too.** `date_offset` moves the *put-back* leg only — -the negative rows stay where the business was — so a shifted recode is a timing -move (pull forward, push out), and with a shift `set` may be empty. The moved -rows' period dimensions are re-derived from `pf.dim_period` against the shifted -date, as clone's are; a row whose date is null keeps the label it had, since a -calendar lookup on nothing would blank it. Without a shift nothing is derived and recode copies as it always -did. Clone still blanks those labels on a null date; it has not been changed. +**Recode takes a date shift too.** `date_offset` moves the *put-back* leg only, +so a shifted recode is a timing move, and with a shift `set` may be empty. The +moved rows' period dimensions are re-derived from `pf.dim_period` against the +shifted date; a row whose date is null keeps the label it had. Without a shift +nothing is derived. Clone still blanks those labels on a null date. -`build_where()` validates every slice key against col_meta (only `role = dimension` allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs. +**A date offset must be whole days.** `assertInterval()` refuses anything finer, +because every shifted date is cast back to a date — and Postgres reads a bare +number as *seconds*. `30 days` is not a month; use `1 month`. + +Scale refuses with "Nothing to scale…" and recode with "Nothing to recode…" +when there is nothing to write. Rationale: docs/pf_spec.md → §Operations. --- ## Authentication Everything under `/api` except the auth routes sits behind a session; the React -app is only mounted once there is one (`Gate` in `main.jsx`), because its load -effects call the API immediately. +app is only mounted once there is one (`Gate` in `main.jsx`). - **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext. Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`; the password is read on stdin and hashed before it reaches psql. -- **Sessions:** `express-session` + `connect-pg-simple` in `pf.session`, so a - restart doesn't sign everyone out and a session can be revoked by deleting its - row (`disable-user` does exactly that). Cookie `pf.sid`: httpOnly, SameSite=Lax, - Secure unless `COOKIE_SECURE=false`, 12h rolling. +- **Sessions:** `pf.session` (connect-pg-simple); revoke by deleting the row. + Cookie `pf.sid`: httpOnly, SameSite=Lax, Secure unless `COOKIE_SECURE=false`, 12h rolling. - **Config:** `SESSION_SECRET` is required — the server exits at boot without one. - `TRUST_PROXY` (default 1) makes `req.ip` and secure-cookie detection correct - behind the TLS proxy. `CORS_ORIGIN` is the only way CORS is enabled at all; a - wildcard origin plus a session cookie would be cross-site request forgery by - construction. + `TRUST_PROXY` (default 1) for `req.ip` and secure cookies behind the TLS proxy. + `CORS_ORIGIN` is the only way CORS is enabled; never a wildcard with a session + cookie. - **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15 minutes (in-memory), returns one message for unknown/wrong/disabled alike, and regenerates the session id on success. **Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from -`sessionUser(req)`, never from the request body — the UI used to send a hardcoded -`pf_user: 'admin'`, which any client could have set to anything. The audit log -now names the account that made the change. +`sessionUser(req)`, **never from the request body**. ## Setup and Baseline are admin-only to write -`requireAdmin` sat in `lib/auth.js`, exported, and applied to **nothing** — so -every non-admin account (26 of the 29) could call every Setup and Baseline -endpoint directly. It is now on the writes those two pages lead to: +`requireAdmin` (`lib/auth.js`) is on: - **sources.js** — `POST /sources`, `PUT /sources/:id/cols`, `POST /sources/:id/generate-sql`, `POST /sources/:id/dim/:group/refresh`, @@ -830,192 +531,123 @@ endpoint directly. It is now on the writes those two pages lead to: `PUT /versions/:id/baseline/:logid`, `DELETE /versions/:id/baseline` - **plans.js** — every write -**Not gated, and each for a reason.** No read is, including the table browser — -the pages stay visible, and gating what a page needs in order to *render* turns -it into a 403 (see below). The forecast operations are not, because scale, -recode, clone and undo carry their own owner and territory rules. Nor is -`PATCH /versions/:id/tags`: its permission is ownership, and a rep may rename -the initiatives they named themselves. +**Not gated, deliberately:** reads (gating what a page needs to render turns it +into a 403); scale/recode/clone/undo, which carry their own owner and territory +rules; `PATCH /versions/:id/tags`, whose permission is ownership. -**The pages stay visible and go read-only.** One `
` per page -rather than a flag threaded through every control — Baseline alone has -forty-five, and the one that got missed would be the one that mattered — plus a -`ReadOnlyNotice` banner saying why. Setup wraps a real box because that fieldset -*is* its flex row; Baseline uses `display: contents` so the page's layout is -untouched, and `disabled` applies either way, being a DOM property. Baseline's -Tags panel sits outside the fieldset, for the ownership reason above. **None of -this is the permission** — it only stops an account being offered a button that -would answer 403. - -**A 403 used to blank the whole app.** Setup fed responses straight into state -and rendered them with `.map`; an error body is `{error}`, and throwing during -render unmounts React. Its list fetches now coerce to an array, so an unexpected -response costs an empty panel rather than the session. That fragility predated -the gating — the gating only made it reachable. +**The pages stay visible and go read-only** — one `
` per page +plus a `ReadOnlyNotice` banner. Setup wraps a real box; Baseline uses +`display: contents`. Baseline's Tags panel sits outside the fieldset. **None of +this is the permission** — the server check is. Setup's list fetches coerce +responses to an array: an error body rendered with `.map` throws during render +and unmounts the whole app. Rationale: docs/pf_spec.md → §Access. ## Source columns come from pg_catalog -`information_schema.columns` omits **materialized views** — they are not in the -SQL standard — and `gs.osm_skinny` is one. So the source the whole app is built -on looked like it had no columns: registering it seeded nothing, and creating a -version failed with "No usable columns in col_meta" while col_meta plainly held -thirty-six. - -`RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version -creation, source registration and the table preview. It returns the same shape -information_schema did, so `mapType` and the callers were unchanged: -`data_type` is `format_type` with the modifier stripped, which gives the same -spelling (`character varying`, `numeric`), and precision and scale are unpacked -from `atttypmod`. The table browser lists from `pg_class` by `relkind` for the -same reason. +`information_schema.columns` omits **materialized views**, and `gs.osm_skinny` is +one. `RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version +creation, source registration and the table preview; it returns the +information_schema shape (`data_type` from `format_type` with the modifier +stripped, precision/scale from `atttypmod`). The table browser lists from +`pg_class` by `relkind`. Do not query `information_schema.columns` for source +columns. ## Territory scoping -An account sees and changes only its own territory. The list lives on -`pf.app_user.territory` (jsonb array) with `is_admin` for the accounts that see -everything, and `col_meta.is_territory` marks which column of a given source -the values belong to — one per source, flagged rather than named in code so a -second source can be divided by something other than a sales rep. +An account sees and changes only its own territory: `pf.app_user.territory` +(jsonb array), `is_admin` for accounts that see everything, and +`col_meta.is_territory` marks the column (one per source). -**Fail closed.** No territory and not an admin means no rows. -`buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an -unflagged source: an account somebody forgot to configure sees nothing instead -of the whole book. +**Fail closed.** `buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an +empty list or an unflagged source. -**Built from the session, never the request.** This is the difference between -it and `scope`, which the browser sends and which is right to send, being a -filter the user chose. A permission cannot come from the thing it restrains, so -the territory predicate is ANDed on last, in `sliceUnits()` for writes and per -route for reads, where nothing in the payload can remove it. +**Built from the session, never the request.** Unlike `scope`, the territory +predicate is ANDed on last, in `sliceUnits()` for writes and per route for +reads, where nothing in the payload can remove it. Enforced at: - `/data` — clause on the cursor *and* on the count behind `X-Row-Count` -- `/agg` — a `{{territory_clause}}` token applied **before** the GROUP BY, since - the territory column need not be part of the grain and may not survive it +- `/agg` — `{{territory_clause}}`, applied **before** the GROUP BY, since the + territory column need not be in the grain - every operation, through `sliceUnits()` -- `/sources/:id/values/:col` — completion reads the *source* table, which no - scope has touched, so without it a dropdown enumerates the whole business -- `DELETE /log/:logid` and `PATCH /log/:logid` — by owner, not territory. Undo - removes an entry's rows wholesale, and half-undoing one would leave a state - nothing describes. The PATCH looks like a private annotation and is not: - `label` and `bucket` name the pivot's columns for everyone in the version, so - unguarded it let any account rename the company's segments. Your own entries, - or an admin's override, and the UI greys out the rest rather than offering a - click that answers 403. -- recode's `set` — a scoped account cannot set the territory column at all. - Moving a row between territories is reassignment, not forecasting, and it - would vanish from the view that would have shown what happened. +- `/sources/:id/values/:col` — completion reads the source table +- `DELETE /log/:logid` and `PATCH /log/:logid` — by **owner**, not territory: + your own entries, or an admin's override. Undo removes an entry wholesale, and + `label`/`bucket` name the pivot's columns for everyone in the version +- recode's `set` — a scoped account cannot set the territory column at all -**Whose entries the change log lists.** Everyone's, as before, with an -**Everyone / Mine** control in the modal header and each account's own entries -marked in place — a blue rule down the left of the row and its `By` cell in -blue. On the version this was built against that is 127 adjustments from five -accounts, one of them the reader's, so finding your own meant reading the `By` -column down the whole list. - -Marked *and* filterable on purpose: the mark is how you tell without hiding -anybody, the filter is for when you want only yours. It resets to Mine on -every open and is not persisted — opening the log is almost always to check -your own recent work, and a stale "Everyone" from last session is a worse -default than the reverse would be. `isMine()` is strict authorship, -deliberately not `canUndo()`: an admin can undo everything, and marking every -row as theirs would say nothing. - -The fetch stays unfiltered, since the counts on the control need the whole set. - -**The change log shows an entry's full impact**, not the reader's share. The -totals stamped on `pf.log` are company-wide, so an admin's version-wide scale -reads the same in every account — deliberate, and labelled, rather than -re-aggregating per territory. +**The change log** lists everyone's entries, with an **Everyone / Mine** control +(resets to Mine on every open, not persisted) and the reader's own entries +marked. `isMine()` is strict authorship, deliberately not `canUndo()`. The fetch +stays unfiltered, since the control's counts need the whole set. It shows an +entry's **full** impact — the totals on `pf.log` are company-wide, deliberately. Managed with `./pf.sh set-territory | set-admin | orphan-territory`. -`orphan-territory` lists values present in the data that no account owns; work -under one is invisible to everyone but an admin, which a typo causes easily and -nothing inside the app reveals. +`orphan-territory` lists values present in the data that no account owns. +Rationale: docs/pf_spec.md → §Access. ## Light / dark mode -Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`. +`ui/src/theme.jsx` — `ThemeProvider` (in `main.jsx`) and `useTheme()` → +`{ dark, setDark }`. Stored as `pf_dark` in `localStorage`, else +`prefers-color-scheme`; the effect toggles `.dark` on ``. `ui/src/index.css` +defines custom properties under `:root` and `.dark`, with Tailwind overrides +written as `.dark .bg-white { ... }` — no Tailwind dark-mode config. Palette is +Perspective's "Pro Dark". -- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit -- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `` -- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed -- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`) -- **Perspective viewer:** `Forecast.jsx` calls `viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' })` both on initial load and in a `useEffect([dark, versionId])`. A **partial** restore, so it does not rebuild the view the way `applyDepth`'s whole-config one does. - - It used to be `setAttribute('theme', …)`, which in 5.4.0 does **nothing** — the theme is a `ViewerConfig` field (`ViewerConfig.theme`, a theme *name*), and the element observes no such attribute. The pivot simply stayed light whatever the toggle said. - - Half the fix is in `cleanLayout`, which **drops `theme`**. `viewer.save()` includes it, so every stored layout had one baked in, and restoring one reasserted the author's theme and undid the toggle. It is the reader's choice, not a property of the layout — which also means a published layout cannot impose its theme on whoever applies it. `sameConfig()` ignores `theme` for the same reason: toggling is not a layout edit and must not light the dirty dot. -- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()` +The viewer's theme is `viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' })`, +on load and in `useEffect([dark, versionId])` — a **partial** restore, no +rebuild. **Not** `setAttribute('theme', …)`, which does nothing in 5.4.0. +`cleanLayout` drops `theme` and `sameConfig()` ignores it: the theme is the +reader's choice, never a layout's. ## After a change: restart, and Generate SQL -Two steps, and which one you need depends on what changed. - - **Restart the server** for anything in `routes/`, `lib/` or `server.js`. - **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes. The templates are *stored* in `pf.sql`, so editing the generator changes nothing until they are rebuilt — and a template carrying a token the running - code does not substitute fails at the database rather than in JS, which reads - as an unrelated client-side error. + code does not substitute fails at the database, which reads as an unrelated + client-side error. -The order matters: restart first, then Generate SQL, or the old code writes the +**Order matters: restart first, then Generate SQL**, or the old code writes the templates. Schema changes are applied to the live database directly and mirrored into `setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a -migration runner, so running it is not how an existing database gets a new -column. +migration runner. ## Known issues / active work -- **Rows written before 2026-09-21 have null `rdate`/`sdate_e`.** The generator - fix above changes what gets written from here on; it does not repair what is - there. On `fc_osm_skinny_29` that is ~270k scale and recode rows. A backfill - would have to re-derive each from the baseline rows at the same coordinate, - which is possible but is inference, not recovery — deleting and replaying - those log entries is the honest repair - -- **Zero-row clones report success.** Scale refuses with "Nothing to scale…" - and recode with "Nothing to recode…" when there is nothing to write (recode - also skips coordinates that already net to zero, rather than writing an empty - pair for each); clone still commits an empty log entry and returns - `rows_affected: 0` -- **A date offset must be whole days.** `assertInterval()` refuses anything - finer, because every shifted date is cast back to a date — and Postgres reads - a bare number as *seconds*, so `"30"` used to be accepted and move nothing. - `30 days` is not a month either: a row on 1 October lands on the 31st; use - `1 month` to move a month into the next +- **Rows written before the every-date-column fix have null secondary dates** + (`rdate`/`sdate_e`) on scale and recode rows. Deleting and replaying those log + entries is the honest repair; a backfill would be inference +- **Zero-row clones report success** — clone commits an empty log entry and + returns `rows_affected: 0`, where scale and recode refuse - **The change log does not show an entry's id**, so there is no way to name one when asking about it -- **Territory is read onto the session at login**, so granting or changing one - does not reach a signed-in account until it signs in again. Re-reading it per - request in `requireAuth` would also make disabling someone immediate -- **Depth buttons rebuild the view.** A depth lives in `ViewConfig`, so changing - it goes through `restore()` and `Session::update_view_config` tears down and - rebuilds the view — a full traversal — where a manual collapse mutates the - existing one in place. Noticeable on a large grain. The fix is to detect a - depth-only change and call `view.set_depth()` imperatively while still writing - it to the config, at the cost of the two being able to drift - -- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29` - (2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in - pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then - ~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The - remaining lever is **dynamic grain** — group by the fields the current pivot - actually uses rather than every `in_grain` column; see `pf_spec.md` → - §Display-grain pre-aggregation, "the dynamic variant" -- Per-node expand/collapse is lost whenever the view rebuilds — set-only API, no - getter; see §Axis depth -- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions) -- Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`, - `App.jsx`). It is re-validated against the live list whenever that list changes, so a - deregistered source or deleted version re-points at the first remaining one instead of - leaving a dead id that 404s every call -- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version -- Grain drift: changing `in_grain` after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. `routes/log.js` derives the grain from live col_meta, so a grain changed mid-session yields `pf_gkeys` that don't match the loaded table and undo silently removes nothing -- Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see `pf_spec.md` → §Display-grain pre-aggregation +- **Territory is read onto the session at login**, so a change does not reach a + signed-in account until it signs in again +- **The bridge dedupes overlapping slices by `pf_id`**, which grain-mode rows do + not carry, so overlapping slices double-count there; and it keeps only + dimension/date slice keys, dropping the `pf_` keys operations now honour + (`BridgeView.jsx`, the per-slice view loop) +- **Depth buttons rebuild the view** — `restore()` tears down and rebuilds, a + full traversal, where a manual collapse mutates in place. Fix: detect a + depth-only change and call `view.set_depth()` too, at the cost of drift +- **Load time is dominated by row count, not payload size** — serialising rows + out of Postgres into JS dominates. The lever is **dynamic grain**; see + docs/pf_spec.md → §Display-grain pre-aggregation +- **Per-node expand/collapse** is lost whenever the view rebuilds; see §Axis depth +- **Col_meta / version schema drift:** if col_meta roles change after a version's + forecast table is created, SQL and DDL go out of sync — delete and recreate + the version +- **Grain drift:** changing `in_grain` after a load requires Generate SQL + a page + reload. `routes/log.js` derives the grain from live col_meta, so a grain + changed mid-session yields `pf_gkeys` that don't match the loaded table and + undo silently removes nothing +- **Grain is static per source** — an unflagged dimension cannot be pivoted on ## Deferred (not in v1) -Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`. +Baseline replay (not built), approval workflow, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`. See docs/pf_spec.md → §Open questions. diff --git a/PERSPECTIVE.md b/PERSPECTIVE.md deleted file mode 100644 index ca8782a..0000000 --- a/PERSPECTIVE.md +++ /dev/null @@ -1,362 +0,0 @@ -# Perspective — configuration & deployment reference - -Canonical guide for how Perspective should be configured, fed, and shipped across our -apps (**pf_app**, **dataflow**). Both embed the same `` web component -but made different early choices; this doc defines the target state and the rationale, so -the two converge instead of drifting. - -> **Distribution note:** we use the **`@perspective-dev/*`** packages -> (repo: , home: -> ) — **not** the FINOS/OpenJS `@finos/perspective` -> packages from . Same engine lineage, but a separate -> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why -> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the -> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two -> scopes. - ---- - -## Core principle: Perspective is one locked unit - -These four things must move together and be pinned together. Bumping one without the -others is the source of nearly every Perspective bug we've hit: - -1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN) -2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc` -3. **Data format** — Arrow IPC vs JSON rows -4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC - format the client WASM understands - -Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test -at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`. - ---- - -## 1. Loading — npm `/inline`, pinned exact (not CDN) - -```js -import perspective from '@perspective-dev/client/inline' -import '@perspective-dev/viewer/inline' -import '@perspective-dev/viewer-datagrid' -import '@perspective-dev/viewer-d3fc' -import '@perspective-dev/viewer/themes' -``` - -- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network - fetch, works offline / behind a firewall, reproducible from the lockfile.** -- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build, - one-line version bumps) but in production it means: app breaks if the CDN is - unreachable, version isn't captured in `package-lock.json`, slower cold start, and you - pull executable WASM from a third party on every load. - - > **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle - > resolves its server WASM with - > `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`, - > which from `.../client@4.4.0/dist/cdn/` resolves to - > `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**. - > jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every - > pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw - > `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`. - > Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern. - > - > The security framing matters as much as the outage: an unversioned URL means your - > users execute whatever that package publishes next, automatically, unreviewed. - > - > 5.2.0 fixes it by carrying the client's version across - > (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`: - > the WASM is embedded in the bundle and there is no runtime fetch to hijack - > (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`). - - pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17. -- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a - `` in `index.html` — so it's bundled and versioned too. - ---- - -## 2. Version policy — a real trilemma (read carefully) - -The version choice is constrained by two hard facts about the `@perspective-dev` packages -**(verified against installed metadata, 2026-06)**: - -- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line / - Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even - though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the - trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright. -- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`, - `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist - in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x. - -So you can have at most **two** of these three: - -| Want | Requires | -|---|---| -| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client | -| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) | -| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) | - -There is **no** version where all three hold. Pick by what the app needs: - -> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts -> -> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1` -> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.** -> -> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`, -> running exactly that pair), *every chart type other than Datagrid fails.* -> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1 -> `viewer` throws `get_static_config is not a function` — once per chart plugin. The -> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same -> load against a coherent 4.4.1 viewer produces no such error. -> -> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is -> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one. -> -> Consequence: dataflow currently has the worst of both worlds. It carries the -> mixed-version complexity *specifically* to keep charts, and does not have charts. -> Both directions are strictly better than standing still: down to a coherent **4.4.1** -> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer -> engine — §3a, `split_rollup_mode`, `edit_mode` persistence). -> -> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a -> real bundled build. It is the documented-and-untested assumption this whole policy -> rests on — establish it before betting a version choice on it. - -- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+ - `^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a - working Datagrid and no charts. If you are on it today, you are choosing inline - bundling, not charts; be explicit about which one you actually want. -- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to - **4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give - up `/inline` bundling. - -Whatever you pick, **commit the lockfile** so the resolved set can't drift on -`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or -later (then a fully-coherent inline-capable suite becomes possible — and only then does -"both" come back on the table). - ---- - -## 3. Data delivery — match the format to the workload - -| Workload | Format | Why | -|---|---|---| -| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. | -| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. | - -### Arrow constraints (read before choosing it) - -If you deliver Arrow, three pieces are coupled and must stay aligned: - -- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you - must coerce them to JS numbers before encoding, or `apache-arrow` infers - `Dictionary` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20, - 1700). -- **Single record batch.** Per-batch Arrow builds independent dictionaries; the - Perspective WASM crashes on dictionary-replacement messages. The server must - accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows), - 'stream')`). Consequence: the client "stream" is just a chunked *download* of one - batch — nothing renders progressively, and the server holds the full result set in - memory per request. -- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server - `package.json` and treat it as part of the locked unit (principle above). Note the - `@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while - pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning - the server toward the arrow major the client was built against, or at minimum make - Arrow ingestion (smoke test §7) the gate on any arrow bump. - -JSON avoids all three but pays in payload size and parse cost, and pushes type handling -to client-side heuristics — acceptable only at small scale. - ---- - -## 3a. Expression columns and aggregation order (ratio correctness) - -**Expression columns are row-level.** Perspective evaluates every expression against -each *raw row* first, then feeds the result into the column's aggregate. It has no -post-aggregate expression stage. So a ratio written the obvious way: - -```js -expressions: { price: '"revenue" / "qty"' } // default aggregate: sum -``` - -...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic -sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group -whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`. - -This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column -grouping just makes it visible by putting several wrong numbers side by side. No -scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and -`high`/`low`/`median`/`dominant` are all wrong for the same reason. - -### The fix: a weighted-mean aggregate - -Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is -algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes: - -```js -expressions: { price: '"revenue" / "qty"' }, -aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array -``` - -**The nested array matters.** The type is -`Aggregate = string | [string, Array]` (`ts-rs/Aggregate.d.ts`), so the weight -column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected -with the unhelpful `data did not match any variant of untagged enum Aggregate` — which -reads like "no such aggregate" and is easy to misread as the feature being absent. - -Available since **4.4.0** — no version bump needed, and because the aggregate is -evaluated inside the engine, incremental `table.update()` stays correct (verified: a -`table.update()` on a live view re-derives the weighted mean from the merged rows -without a reload). - -### Rules of thumb - -- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below) - → leave the default `sum`. Those are unaffected by any of this. -- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a - `['weighted mean', ['']]` aggregate, or it is wrong under any pivot. -- Mixing both in one view is fine and was verified. - -### Type inference can silently break expressions - -`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred** -as `integer` — which happens to numeric-looking strings when the table is created from -inferred JSON. With an explicit `string` schema the same expression is correct. Neither -literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no -error. Give period/year columns an explicit `string` type at table creation. - -### Not fixed by any of this: column-axis expand/collapse - -`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the -row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the -docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds -`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups -statically (no interactivity) — the nearest thing to Excel-style column subtotals, and -it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though -(§2: `viewer-d3fc` still caps at 4.4.1). - ---- - -## 4. Theming - -- One toggle drives both app CSS and the viewer: - `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`. -- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer - re-themes when the toggle fires (not just on mount). -- App-level dark mode is plain CSS custom properties + a `.dark` class on ``; the - viewer theme name is the only Perspective-specific piece. - ---- - -## 5. Layout persistence - -- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to - `localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`. -- **Guard restores against schema drift.** Before restoring, filter the saved config's - `columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually - exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is - the reference implementation; a stale layout referencing a dropped column otherwise - throws on restore. -- **`aggregates` needs the same guard.** pf_app's `cleanLayout()` has it as of 2026-09; - **dataflow's does not** and should adopt it. Filtering - `columns`/`group_by`/`split_by`/`sort`/`filter` and stopping there is harmless only - while `viewer.save()` emits `aggregates: {}`, which it does until someone sets an - aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a - dropped column aborts the entire restore — both `table.view()` and `viewer.restore()` - throw `Could not get dtype for column 'X' as it does not exist in the schema`. An - aggregate entry references a *target* column and, in the multi-arg form, a *weight* - column; both need validating, and the entry is what gets dropped, never the layout. - Add this guard as part of adopting §3a, not after. - -### Auto-pause deletes the view — turn it off for a static pivot - -`` auto-pauses by default: an `IntersectionObserver` on itself -(scrolled out of the viewport, `display: none`) combined with the document's -`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint -optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the -**view object is destroyed**. Becoming visible again calls -`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full -traversal, every time. - -For a viewer streaming live updates nobody is watching, that is the right trade. -For a pivot over a large static table it is the wrong one — on pf_app's -`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it -silently discards per-node expand/collapse (§"Not fixed by any of this"), which -has no config representation and so cannot be restored. - -```js -await viewer.load(table) -try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {} -``` - -Guard the call: it is a method on the custom element and absent on older builds. -Leave auto-pause **on** where the table is fed by a live stream the user does not -need to have kept up with while away. This is long-standing viewer behaviour, not -a 5.x regression — worth knowing before blaming a rebuild on your own code. - ---- - -## 6. Build & deploy (target) - -- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via - `outDir: '../public/app'`). The API server serves it statically (`express.static`). -- **Process:** run the Node API under **systemd** with `Restart=always` and an - `EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot). - dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy - automation yet** and should adopt the same pattern. -- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx, - systemd unit); later runs rebuild the UI and restart the service. -- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd - unit — never commit it. - ---- - -## 7. Upgrade checklist / smoke test - -Run this whenever bumping **any** Perspective package or `apache-arrow`: - -1. Confirm `viewer-d3fc` publishes the target version - (`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others. -2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`; - commit the lockfile. - - **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares - it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to - `latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0** - and the WASM fails to link: - `Import #8 "env" "psp_opfs_load": function import requires a callable`. - Five packages, not four. This is invisible while pf_app loads from the CDN, and - will bite on the "move off CDN" open item below. -3. `vite build` — no unresolved imports. -4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests - without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a - string/dictionary. -5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders. -6. Toggle dark/light; confirm the viewer re-themes. -7. Save a layout, reload, confirm it restores; then drop a column and confirm the - cleaned restore doesn't throw. - ---- - -## Per-project state (2026-06) - -| | pf_app | dataflow | Target | -|---|---|---|---| -| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** | -| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** | -| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app | -| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) | -| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test | -| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** | - -**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline -bundling and d3fc charts (§2). Leave it; just keep the lockfile committed. - -**Open items:** -- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages - pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host - blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders. -- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction): - its d3fc charts do not work, so it is paying mixed-version complexity for nothing. - Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually - render before choosing the latter, because nobody has confirmed they do. -- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`). diff --git a/docs/PERSPECTIVE.md b/docs/PERSPECTIVE.md new file mode 100644 index 0000000..4d6a0dd --- /dev/null +++ b/docs/PERSPECTIVE.md @@ -0,0 +1,187 @@ +# Perspective — lessons and rules + +What we have learned the hard way about configuring, feeding and upgrading +Perspective, written for any app that embeds `` (**pf_app**, +**dataflow**). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how +to rebuild it — live in `ui/vendor/README.md`; how pf_app uses the viewer day to +day is in `CLAUDE.md` and `pf_spec.md` → §The pivot. + +> **Distribution:** we use the **`@perspective-dev/*`** packages +> () — **not** FINOS/OpenJS +> `@finos/perspective`. Same engine lineage, but a separate npm scope, release +> cadence and "Pro" theme set (`Pro Dark` / `Pro Light`). Don't mix the scopes. + +--- + +## Perspective is one locked unit + +These move together and are pinned together. Bumping one without the others is +the source of nearly every Perspective bug we have hit: + +1. **Loader** — how the JS/WASM gets into the page (npm `/inline`, never a CDN) +2. **Package versions** — `client`, `server`, `viewer`, `viewer-datagrid` (and + `viewer-d3fc` where charts are used) +3. **Data format** — Arrow IPC vs JSON rows +4. **`apache-arrow` on the server**, when delivering Arrow — it must write an IPC + format the client WASM reads + +**Pin `@perspective-dev/server` explicitly.** `@perspective-dev/client` declares +it as `"@perspective-dev/server": ""` — an empty range, which npm resolves to +`latest`. Installing an older client pulls whatever server is newest and the WASM +fails to link (`Import #8 "env" "psp_opfs_load": function import requires a +callable`). Five packages, not four. + +--- + +## Load inline, never from a CDN + +```js +import perspective from '@perspective-dev/client/inline' +import '@perspective-dev/viewer/inline' +import '@perspective-dev/viewer-datagrid' +import '@perspective-dev/viewer/themes' +``` + +The `/inline` entrypoints bundle the WASM into the Vite build: no runtime fetch, +works behind a firewall, reproducible from the lockfile. The themes CSS is +imported in JS for the same reason, not ``ed from `index.html`. + +**This is not hypothetical — a CDN load took pf_app down on 2026-08-10.** The 4.x +CDN bundle resolves its server WASM with +`new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`, +which from `.../client@4.4.0/dist/cdn/` lands on +`.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no +version**, so jsdelivr serves `@latest`. The moment a 5.x server published, every +page load linked a 5.x WASM against a 4.4.0 client and threw the `LinkError` +above. Nobody changed anything. The security framing matters as much as the +outage: an unversioned URL means users execute whatever that package publishes +next, unreviewed. `/inline` has no runtime fetch to hijack. + +--- + +## Charts versus inline bundling (apps that need d3fc) + +pf_app uses the datagrid only and does not carry `viewer-d3fc`. For an app that +does, the constraint as of 2026-08: `viewer-d3fc` published nothing past 4.4.1, +while `/inline` and `/themes` exist only from 4.5.x. **A 4.5.x viewer with a 4.4.1 +d3fc does not work** — every chart plugin throws `get_static_config is not a +function` (reproduced in isolation, and seen in dataflow's install, where every +chart type but the datagrid fails). So it is inline bundling **or** d3fc charts, +not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1 +suite renders them — check before relying on it), or 5.x without them. Re-check +when d3fc publishes a newer version. + +--- + +## Arrow delivery: three coupled constraints + +Arrow IPC is the right format for large, numeric, incrementally-updated tables; +JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples +three things: + +- **Numeric type parsers, server-side.** pg returns `bigint` / `numeric` as + strings; coerce them to numbers before encoding, or `apache-arrow` infers + `Dictionary` instead of `Int` / `Float64`. pf_app: the type parsers in + `server.js` (oids 20 and 1700). +- **One record batch.** Per-batch encoding builds independent dictionaries, and + the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit + a single batch — pf_app does `tableToIPC(tableFromArrays(columns), 'stream')`. + The client's "stream" is therefore a chunked download of one batch; nothing + renders progressively. +- **`apache-arrow` versus the client WASM.** The client is built against an older + arrow major than pf_app's server pins; the pair was verified working, but any + arrow bump has to be gated on the ingest smoke test below. + +--- + +## Expression columns are row-level: ratios need a weighted mean + +Perspective evaluates every expression against each **raw row**, then feeds the +result into the column's aggregate; there is no post-aggregate expression stage. +So `price: '"revenue" / "qty"'` under the default `sum` **sums the per-row +ratios** — wrong at every grouped level, with or without `split_by`. `avg` gives +the average of ratios, which is also wrong. + +Weight the ratio by its own denominator — `sum(price_i × qty_i) / sum(qty_i)` is +exactly `sum(revenue) / sum(qty)`, at every level of both axes: + +```js +expressions: { price: '"revenue" / "qty"' }, +aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array +``` + +**The nested array matters.** `Aggregate = string | [string, Array]`, so +the weight column goes in its own array. The flat `['weighted mean', 'qty']` is +rejected with `data did not match any variant of untagged enum Aggregate`, which +reads like the feature is missing. Being engine-side, it stays correct across +incremental `table.update()`. + +- A **sum of a measure** (including `if(...)` subtotal expressions) → default + `sum`. +- A **ratio, rate, price or per-unit figure** → must carry + `['weighted mean', ['']]`. + +**Type inference can silently break expressions.** `if("Year" == '2026', +"Amount", 0)` returns 0 on every row if `Year` was *inferred* as integer from +numeric-looking strings, with no error, and neither literal form fixes it. Give +period and year columns an explicit `string` type at table creation. + +--- + +## Theming + +The viewer theme is a `ViewerConfig` field: `viewer.restore({ theme: 'Pro Dark' })`. +**`setAttribute('theme', …)` does nothing in 5.x** — the element observes no such +attribute. Apply it on load and whenever the app toggle changes. `viewer.save()` +includes `theme`, so strip it from stored layouts and ignore it when comparing +configs — otherwise restoring a layout reasserts its author's theme. + +--- + +## Layouts: guard every restore against schema drift + +Before `viewer.restore(cfg)`, filter the saved `columns` / `group_by` / +`split_by` / `sort` / `filter` against the columns the current table actually has +(plus its `expressions`); a stale layout naming a dropped column otherwise throws. +**`aggregates` needs the same guard** — once a layout carries a weighted mean, a +dropped column aborts the whole restore (`Could not get dtype for column 'X'`). +An aggregate names a target column and, in the multi-arg form, a weight column; +validate both and drop the *entry*, never the layout, since `restore()` is +all-or-nothing. pf_app's `cleanLayout()` does all of this; dataflow's does not yet +guard `aggregates`. + +--- + +## Auto-pause deletes the view — turn it off for a static pivot + +`` auto-pauses by default (an `IntersectionObserver` on itself +plus the document's `visibilitychange`). Pausing is not a paint optimisation: +`session.set_pause(true)` runs `view_sub.take().delete()`, so the view is +destroyed, and becoming visible again rebuilds it with a full traversal — a +multi-second stall per tab switch on a large table, and any per-node +expand/collapse is lost with it. + +```js +await viewer.load(table) +try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {} +``` + +Leave it on only where a live stream feeds a table nobody needs kept current +while away. Long-standing viewer behaviour, not a regression. + +--- + +## Upgrade smoke test + +Run on any change to a Perspective package or `apache-arrow` (for pf_app, after +rebuilding the vendored tarballs per `ui/vendor/README.md`): + +1. All five packages (and `apache-arrow`) pinned exact and matching; lockfile + committed. +2. `vite build` — no unresolved imports. +3. Arrow apps: load a real dataset; `worker.table(buffer)` ingests without a + dictionary error, and a numeric column comes through as `Float64` / `Int`. +4. Charts, where used: open a d3fc plugin and confirm it renders. +5. Toggle dark/light; the viewer re-themes. +6. Save a layout, reload, confirm it restores; drop a column and confirm the + cleaned restore does not throw. diff --git a/pf_perspective_options.md b/docs/pf_perspective_options.md similarity index 96% rename from pf_perspective_options.md rename to docs/pf_perspective_options.md index 00eb32d..cb94361 100644 --- a/pf_perspective_options.md +++ b/docs/pf_perspective_options.md @@ -1,5 +1,17 @@ # Perspective Architecture Options +> **Decision record, written 2026-08.** The outcome was the display-grain +> pre-aggregation (§Spike findings onward), and it shipped. **"Current +> architecture" below describes the app as it was then**, not now: it has +> since moved to a vendored Perspective 5.4.0 bundled inline, `tableFromArrays` +> on the server, `table.remove()` on undo, and `/data` / `/agg` emitting +> `pf_segment`, `pf_bucket`, `pf_note` and `pf_op`. For today's data flow see +> `CLAUDE.md` → §Core data flow and `pf_spec.md` → §Display-grain +> pre-aggregation. What remains current here is the spike findings, the +> display-grain reasoning, and the Path A / Path B analysis behind the +> "dynamic grain" next step. The DuckDB virtual-server spike is parked on +> branch `spike/duckdb-virtual-server`. + This document weighs how the Forecast view should source data for the Perspective pivot. The current implementation hits practical limits on initial load (~30s for 350k rows × ~55 cols), and growth is expected. Choosing an architecture diff --git a/docs/pf_spec.md b/docs/pf_spec.md new file mode 100644 index 0000000..ef8eb97 --- /dev/null +++ b/docs/pf_spec.md @@ -0,0 +1,1589 @@ +# Pivot Forecast — Design + +What the app does, how the pieces fit, and why — including what each design +replaced, where that history still explains the current shape. `CLAUDE.md` +holds the rules an agent needs while editing; this holds the reasoning. Where +either disagrees with the code (`routes/`, `lib/`, `ui/src/`, `setup_sql/`), +the code is the truth. + +Related: `pf_perspective_options.md` (transport options, the DuckDB spike, why +the pre-aggregated path won), `PERSPECTIVE.md` (bundling), `ui/vendor/README.md` +(the patched Perspective build and how to rebuild it). + +--- + +## Overview + +A web app for building named forecast scenarios against a PostgreSQL table: + +1. **Load actuals as a baseline** — segments pulled from the source, + optionally date-shifted into the forecast period, so the starting point is + "no change from last year". +2. **Load reference rows** — prior year, plan, open orders — for comparison. + Operations do not adjust them. +3. **Adjust** — select pivot cells and scale, recode or clone them. + +Every write is **append-only** and produces exactly one `pf.log` entry. An +adjustment inserts new rows carrying its entry's id; undo deletes those rows +and the entry. So a version is always the sum of its log, every figure is +attributable to the entry that wrote it, and any entry can be reversed alone. + +Postgres holds the raw rows and is the single source of truth. The browser +holds a copy in [Perspective](https://github.com/perspective-dev/perspective) +(WASM), pre-aggregated to the display grain, and updates it incrementally from +each operation's response rather than reloading. + +### Lineage + +A port of an Excel model: a data tab of pre-aggregated rows, a pivot cache over +it, a double-click on a cell opening a VBA form (`/opt/forecast_api/VBA/fpvt.frm`) +that appended incremental rows, and a cache refresh. The append-and-let-the-pivot-sum +model follows that directly, and the price/volume rules in §Operations come from +the form's `calc_val` / `calc_price`. + +### Stack + +Node/Express (`server.js`, one `pg` pool, type parsers coercing `bigint` and +`numeric` to numbers so Arrow does not infer them as dictionary text); +PostgreSQL, everything in schema `pf`; React + Vite + Tailwind in `ui/`, built to +`public/app/`; Perspective 5.4.0 from a vendored patched fork (§The pivot). + +### Pages and navigation + +Three steps in a collapsible sidebar: **Setup** (register sources, assign column +roles, Generate SQL), **Baseline** (versions, load segments, version defaults, +tags), **Forecast** (the pivot and the operation panel). Setup and Baseline are +admin-only to write (§Access). + +**The forecasts are listed in the sidebar**, above the steps, each with its +status and a star for the default. They used to be two dropdowns in the status +bar, which is otherwise read-only, so the one pair of controls in it did not read +as controls. The list is *not* nested under the Forecast step, where the idea +started: Baseline works on the same version, so hanging it off one step would +misrepresent its scope or duplicate it. The sidebar reads *what am I working on*, +then *what step am I on*. + +**The source is not in the nav.** A raw table name means nothing to someone +editing a forecast. Picking a forecast sets the source, since +`pf.version.source_id` comes with it; the source picker appears only on Setup. +`versions` (the selected source's, which Baseline and Forecast need) and +`allVersions` (the sidebar's, a client-side fan-out over `/sources/:id/versions`) +are kept apart so widening one does not change the other. + +The status bar keeps what follows from the selection: the forecast's name and row +count, with the physical table in a click-to-copy hover panel — pasting it into +psql is why it was ever on screen. + +**Known collision:** "forecast" names both the scenario and the third step. The +scenario should own the word, leaving the step wanting another (`Adjust`). +Undecided. + +### Where a browser lands + +`App.jsx` restores the last source and version from `localStorage`, re-validated +against the live list so a deleted version re-points rather than 404ing every +call. With nothing stored it falls back to **`pf.version.is_default`** (one per +source, partial unique index). It used to fall back to `versions[0]` — newest +first, so whatever somebody created last, a scratch copy as readily as the plan — +and accounts that never pick a version, which is every sales account, followed +the forecasting work around. + +Setting the default is **admin only**, since it decides where everyone else +lands; the Baseline checkbox is greyed for others. It is two statements (clear +old, set new) in one transaction, because the index refuses the instant between. +**A stored selection still wins** — the default is a fallback, so an account that +landed in the wrong scenario keeps landing there until it switches once. + +### Renaming a version + +Safe at any time, including closed. Nothing derives from the name: the table is +`fc_{tname}_{version_id}`, `pf.log` and `pf.layout` key on the id, and client +storage keys are id-based. The Baseline page's **This forecast** panel saves name +and description on blur via `PUT /versions/:id`; a name collision answers 409, +and a blank name is refused client-side because the route COALESCEs it away. + +### API surface + +All under `/api`; all but the auth routes need a session. + +| Area | Routes | +|---|---| +| Auth | `POST /login`, `POST /logout`, `GET /me` | +| Browser | `GET /tables`, `GET /tables/:schema/:tname/preview` | +| Sources | `GET/POST /sources`, `GET/PUT /sources/:id/cols`, `POST /sources/:id/generate-sql`, `GET /sources/:id/sql`, `GET /sources/:id/values/:col`, `GET /sources/:id/dim/:group`, `POST /sources/:id/dim/:group/refresh`, `GET /sources/:id/lookup`, `GET /dim-period/cols`, `DELETE /sources/:id` | +| Versions | `GET/POST /sources/:id/versions`, `PUT/DELETE /versions/:id`, `POST /versions/:id/{close,reopen}`, `GET /versions/:id/table-info`, `GET /versions/:id/values/:col`, `GET/PATCH /versions/:id/tags`, `GET /versions/:id/bridge` | +| Data | `GET /versions/:id/data`, `GET /versions/:id/agg` | +| Loads | `POST /versions/:id/{baseline,reference}`, `PUT /versions/:id/baseline/:logid`, `DELETE /versions/:id/baseline` | +| Plans | `GET/POST /versions/:id/plans`, `PATCH/DELETE /plans/:pid` | +| Operations | `POST /versions/:id/{scale,recode,clone}` | +| Log | `GET /versions/:id/log`, `GET /log/:logid/debug`, `PATCH/DELETE /log/:logid` | +| Layouts | `GET/POST /versions/:id/layouts`, `PATCH/DELETE /layouts/:lid` | + +`DELETE /sources/:id` answers 409 while the source has versions +(`pf.version.source_id` is `ON DELETE RESTRICT`) and never drops forecast tables. + +--- + +## Data model + +Install scripts in `setup_sql/`: `01_schema.sql` (metadata tables, with +`ADD COLUMN IF NOT EXISTS` backfills and one-off data migrations inline), +`02_auth.sql` (`pf.app_user`, `pf.session`), `gen_dim_period.sql` (creates and +fills `pf.dim_period`; fiscal start month set at the top; re-runnable). + +`01_schema.sql` is idempotent but **not a migration runner**. Schema changes are +applied to the live database directly and mirrored into `setup_sql/` for fresh +installs. + +### `pf.source` + +`schema`, `tname`, `label`, `status`, `created_by`. `default_layout` is dead — +migrated into `pf.layout`, read by nothing. Registration seeds one `col_meta` row +per column as `dimension`. + +**Columns come from `pg_catalog`, not `information_schema`**, which omits +materialized views — they are not in the SQL standard — and the app's primary +source is one. It looked columnless: registration seeded nothing and version +creation failed "No usable columns in col_meta" while col_meta plainly held them. +`RELATION_COLUMNS_SQL` in `lib/utils.js` returns the shape `information_schema` +did (`format_type` minus modifier; precision and scale unpacked from +`atttypmod`), so `mapType` and callers did not change. The table browser lists +from `pg_class` by `relkind` for the same reason. + +### `pf.col_meta` + +| Column | Meaning | +|---|---| +| `role` | `dimension` \| `value` \| `units` \| `date` \| `filter` \| `ignore` | +| `label`, `opos` | display name; ordinal, which orders everything generated | +| `is_key` | four separate uses, below | +| `dim_group` | groups functionally dependent columns — a part and its attributes, a date and its periods | +| `dim_period_col` | the `pf.dim_period` column this dimension is derived from at load | +| `in_grain` | part of the display grain (§Display-grain pre-aggregation) | +| `is_territory` | the column territory is expressed in; at most one per source, enforced on save | + +Roles: `dimension` is categorical and sliceable. `value` (required) and `units` +(optional) are the measures — several may be marked, loads carry all, but the +operations use only the first of each by `opos` (§Proposed: additional +measures). `date` (at least one) — every date column is stored, shifted and +carried by every operation, and sliceable. `filter` feeds the baseline filter +builder and is not stored. `ignore` is not stored. Slices are validated against +every `dimension` and `date` column regardless of `is_key`. + +**`dim_group` + `dim_period_col`.** A `date` column that is `is_key` with a +`dim_group` anchors the group: its sibling dimensions with a `dim_period_col` are +looked up in `pf.dim_period` against the *shifted* date at load time instead of +copied raw, so fiscal periods follow a baseline moved forward a year. Every such +group is honoured; this used to be a `.find()`, so a source with order, requested +and ship dates derived one set and copied the other two raw — shifted dates under +unshifted labels. + +**`is_key`** is read in four places, none of them what the name suggests: + +1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes + `members.find(c => c.is_key)` as the column `pf.dim_member` is keyed on +2. **Value completion** — `/sources/:id/values/:col` serves only `is_key` + columns (the operation panel now completes from `/versions/:id/values/:col` + instead, which takes any dimension or date) +3. **Sibling autofill** — `/sources/:id/lookup`, for `is_key && dim_group` +4. **The `dim_period` anchor** — `role = 'date' && is_key && dim_group` + +Uses 2 and 3 want several flagged; use 1 needs exactly one per group, and with +more `.find()` silently takes the lowest `opos`. That has happened: a column null +throughout outranked its group's real key, so a member refresh matched nothing +and reported success with zero members. The meanings want separating — a +per-group key choice, or the rule that the key is the column the group is named +after — and until then a refresh finding two keys should refuse rather than guess. + +### `pf.dim_period` + +One row per month keyed on `sdat`, a GiST-indexed `drange`, and calendar and +fiscal year/quarter/month columns in several spellings. Joined as +`LEFT JOIN pf.dim_period dpN ON dpN.drange @> (date + offset)::date` — left, +because an unshipped order has no ship date and must keep its row. + +### `pf.dim_member` + +Master data per `dim_group`: one row per key value, siblings in `attrs`, with +`is_active` and `source_seen`. The source is transactional, often a view over all +history, so deriving members from it is slow (a full scan) and wrong (it describes +only what has been transacted — it cannot say a part is discontinued, or that a +new one exists before it sells). This is the app's own list, refreshed from the +source by an admin action but curatable apart from it. A refresh **merges**: +members gone from the source become `source_seen = false`, not deleted, so +curation survives; a key with several attribute sets across history takes the +most recent. It stamps an explicit run time rather than `now()`, which inside a +transaction is the transaction start. + +### `pf.version` + +| Column | Meaning | +|---|---| +| `name`, `description` | `(source_id, name)` unique | +| `status`, `closed_at`, `closed_by` | a closed version refuses every write but annotation | +| `exclude_iters` | jsonb, default `["reference"]` — `pf_iter` values scale and recode will not write | +| `is_default` | where a browser with no saved selection lands | +| `adjustment_segment`, `adjustment_bucket` | names a new adjustment is stamped with (§Naming) | +| `unlabeled_load`, `bucket_order` | dead — read by nothing | + +Creating a version inserts the row and creates its forecast table and `pf_logid` +index in one transaction. Deleting drops the table; `pf.log`, `pf.layout` and +`pf.load_plan` cascade. + +### `pf.fc_{tname}_{version_id}` + +One table per version holding operational rows (`pf_iter` = `baseline`, `scale`, +`recode`, `clone`) and reference rows (`reference`): + +```sql +CREATE TABLE pf.fc_{tname}_{id} ( + pf_id bigserial PRIMARY KEY, + -- every non-ignored col_meta column, typed from the source by mapType + pf_iter text NOT NULL, + pf_logid bigint NOT NULL, + pf_user text, + pf_created_at timestamptz NOT NULL DEFAULT now() +); +CREATE INDEX fc_{tname}_{id}_logid_idx ON pf.fc_{tname}_{id} (pf_logid); +``` + +No `version_id` column: the table is the version. **`pf_logid` is not a foreign +key** — undo deletes rows then log row in one transaction, and those are the only +two paths that remove either. **The index** is how undo, the change-log recount +and the grain key find an entry's rows; without it each is a sequential scan. +Tables created before it was added lack it. + +### `pf.log` + +| Column | Meaning | +|---|---| +| `version_id` | `ON DELETE CASCADE` | +| `pf_user`, `stamp` | the session account; when | +| `operation` | `baseline` \| `reference` \| `scale` \| `recode` \| `clone` | +| `slice`, `params` | the slice(s) written to (null on loads); intent plus what the server resolved | +| `label`, `bucket` | segment name and what it counts toward; required by the `log_named` CHECK | +| `tag`, `note` | initiative name, which the bridge groups on; free commentary | +| `row_count`, `value_total`, `units_total`, `measure_cols` | what the entry wrote, stamped at write time | +| `env`, `sql_text` | what it ran against; the statement as executed | +| `seq` | dead | + +**Who writes what.** Load templates insert `label`, `bucket` and `tag` in their +`ilog` CTE. Adjustment templates insert `label` and `bucket`; their `tag` goes on +in a follow-up `UPDATE` in the same transaction (`tagLog()`), a hold-over from +when a new `{{tag}}` token would have stranded sources that had not re-run +Generate SQL. Totals, `env` and `sql_text` are written after commit by +`stampLogTotals()`. + +**Stamped totals are not a cache.** An entry's rows never change after it is +written — only its own operation inserts them and only undo removes them, taking +this row too — so the totals are fixed. The change log used to join the whole +forecast table on every open to recompute them. `measure_cols` records which +columns they are in, since roles can be reassigned. Stamping is best-effort: a +failure does not roll back a write that succeeded, since totals can be recomputed +and the adjustment cannot. `GET /versions/:id/log?recount=1` recomputes by join +and writes back — the repair for rows deleted by hand, and the backfill for +entries older than the columns. + +**`params` and `sql_text` are both kept on purpose.** One is the intent (slices, +scope, resolved increments), the other what it became. Keeping only one assumes +the translation is correct, which is exactly what is in doubt when rows look +wrong. `env` holds what cannot be reconstructed afterwards — the territory, the +version's `exclude_iters`, and the template's `generated_at`, a fingerprint (not +a version) showing whether an entry ran under a different template. +`GET /log/:logid/debug` returns the whole row; the list omits `sql_text` and +reports `has_sql`. + +### `pf.sql` + +One generated template per `(source_id, operation)`: `get_data`, `get_agg` (only +with a grain), `baseline`, `reference`, `scale`, `recode`, `clone`, `undo`. +`generateSQL()` in `lib/sql_generator.js` bakes in column names and leaves +`{{token}}`s — `fc_table`, `version_id`, `pf_user`, `note`, `tag`, `label`, +`bucket`, `params`, `slice`, `filter_clause`, `date_offset`, `where_clause`, +`exclude_clause`, `value_incr`, `units_incr`, `set_clause`, `scale_factor`, +`territory_clause` — for `applyTokens()` at request time. Generation deletes any +operation it no longer produces, since a stale `get_agg` after clearing the grain +would still be picked up. The stored `undo` is documentation; the route runs its +own statements. + +Values are escaped (`esc()` doubles quotes), not parameterised — consistent with +the codebase, and the executed SQL reads as-is in the server log and +`pf.log.sql_text`. + +**Editing the generator changes nothing until Generate SQL runs**, since the +templates are stored; a template carrying a token the running code does not +substitute fails at the database, reading as an unrelated client error. Restart +first, then Generate SQL, or the old code writes the templates. + +### Other tables + +`pf.layout` (§The pivot), `pf.load_plan` (§Loads and segments), `pf.app_user` and +`pf.session` (§Access). + +--- + +## Loads and segments + +A version's rows start as **segments**: independent queries against the source, +each appended as one log entry, each undoable alone. Two kinds, differing only in +`pf_iter`: + +- **baseline** — the rows operations adjust +- **reference** — kept out of scale and recode by `exclude_iters` + +`pf_iter` answers *can operations write these rows*. Whether rows belong in the +forecast number is the segment's **bucket** (§Naming): open orders load as +reference so nothing adjusts them, and are still part of the forecast. Neither +can be derived from the other. All load routes are admin-only. + +### How a load's WHERE is built + +The Baseline page's filter builder takes **groups** of conditions — AND within, +OR between — over `date` and `filter` columns (`=`, `!=`, `IN`, `NOT IN`, +`BETWEEN`, `IS NULL`, `IS NOT NULL`). A period is a filter like any other. A +toggle swaps the builder for a raw WHERE body. + +**The client builds the clause; the route injects it.** `buildFilterClause()` in +`Baseline.jsx` renders `(a AND b) OR (c)` and sends it as `where_clause` beside +the structured `filters` (or `raw_where`). The route substitutes +`raw_where || where_clause` as `{{filter_clause}}` verbatim and keeps `filters` / +`raw_where` in `params` only so a segment can be re-opened and a plan derived. +Nothing server-side validates columns or escapes values, and the client does not +escape them either; `lib/sql_generator.js`'s own `buildFilterClause()`, which +does, is called by no route. The protection is `requireAdmin`: whoever can reach +the route can already send raw SQL, so validating the structured path would not +narrow what they can do. An empty clause becomes `TRUE` — the whole source — and +the client refuses to send one. + +### Date offsets + +`date_offset` is a Postgres interval (`1 year`, `-90 days`) applied to **every +date column**, so order and ship dates stay in step, with period dimensions +looked up against the shifted dates. Filter columns are never shifted: the offset +moves where rows land, not which are chosen. + +It must be **whole days** (`assertInterval()`): shifted dates are cast back to +dates, so anything finer is rounded away, and Postgres reads a bare number as +seconds — `"30"` used to be accepted and move nothing. `30 days` is not a month +either; use `1 month`. Checked on baseline loads, segment edits, recode and clone +— **not** on reference loads, where a bad offset surfaces as a Postgres error. + +The offset is typed as an interval string. It replaced year/month spinners, +which could not express days and clamped at zero, so a segment could only move +forward in whole months. The timeline preview parses it only far enough to draw +the source and shifted periods; Postgres is the authority. + +### Naming, editing, clearing + +A load's **label is required** (400 without) — the segment name the pivot shows +and reports group on — and its bucket defaults to it (`loadNames()`). + +`PUT /versions/:id/baseline/:logid` edits a segment by deleting its rows and log +row and re-running it in one transaction, under a **new** logid. Refused (409) +once any adjustment exists, since those were sized against the old totals. +Annotations carry over with `??`, not `||`: the form has no tag input, so an +absent tag must survive, while an explicit `''` clears. + +`DELETE /versions/:id/baseline` removes every `baseline` row and entry (not +reference). Not undoable. + +### Load plans — `pf.load_plan` + +A segment used to exist only once run, so cloning a definition meant re-running +every load, and there was nowhere to hold "this cut, but with the dates changed +first". A plan is the definition alone — `kind`, `label`, `bucket`, `tag`, `note`, +`filters`, `raw_where`, `date_offset`, `seq` — with `run_logid` naming the entry +it produced, or null. + +- **Deliberately not `pf.log`**: every row there means a write happened, and + unexecuted intent would cost it that meaning. +- **`run_logid` is `ON DELETE SET NULL`**, so undoing a load returns its plan to + unrun instead of deleting the definition. The list reads the entry's + `row_count` through the FK to tell run from unrun. +- **Running is two round trips** — the ordinary load POST with the plan's fields, + then `PATCH /plans/:pid` with the `log_id`. The alternative was a second copy of + the load route's substitution; a plan that reads unrun while its rows exist is + visible and fixable, a divergent copy of the load SQL is neither. +- **Run all stops at the first failure.** A definition's segments are not + independent — references date-shift off a baseline — so carrying on past a + broken one builds something nobody described. + +Known issue: `runPlan()` does not send the plan's `tag`, so a tagged plan loads +untagged. + +### Cloning a version's definition + +`POST /sources/:id/versions` with `clone_from` copies the template's settings +(description unless given, `exclude_iters`, adjustment names), its +**version-scoped** layouts, and its segments as plans — and **runs nothing**. The +point of cloning a definition is to change the dates before anything loads; a +clone that fetched the baseline first would have to be undone to be edited. + +Not copied: **adjustments**, in any form — a scale's target was sized against the +rows that were there, so replaying it against another load is arithmetic nobody +asked for; and **source-wide layouts**, which already apply and would duplicate +by name. + +`cloneDefinition()` reads the template's plans, or derives them from its +`baseline`/`reference` entries when it has none — every version built before +plans existed. `params` already stores `filters`, `raw_where` and `date_offset`, +so this is a read, not a reconstruction. + +### Replay + +Re-running a version's adjustments against a restated baseline is **not built**. +Nothing accepts a `replay` flag and nothing answers 501, whatever earlier specs +said. See §Open questions. + +--- + +## Naming: segments, buckets, notes + +Four pivot columns come from `pf.log`, joined on `pf_logid`, not from the +forecast table: + +| Column | From | +|---|---| +| `pf_segment` | `l.label` — which load, or which kind of adjustment | +| `pf_bucket` | `l.bucket` — what it counts toward: Actual, Plan, Forecast, Prior Year | +| `pf_note` | adjustments: the tag, else the note; loads: null | +| `pf_op` | `l.operation` — `/agg` and pushed-back rows only | + +Separate on purpose: commingling a segment name with an adjustment's note makes +neither pivotable. In grain mode `pf_logid` is in the grain, so the join adds no +rows. `SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are +the single definition, shared by the generated `/agg` and the hand-written +`/data` cursor, which must agree. The operation routes stamp the same fields on +the rows they return, since `RETURNING *` lacks them. The `/data` cursor does not +emit `pf_op`, so a raw-mode pivot cannot group on it. + +### Every log entry is named + +`label` and `bucket` are **stored, not derived**, and every entry has both — +`log_named` refuses one without. A report gets the pivot's segment and bucket by +joining `pf.log` and selecting two columns, with no rule of its own. Stamped at +write time: + +- **A load** — `label` required, `bucket` defaulting to it. +- **An adjustment** — no name inputs, so the version's `adjustment_segment` / + `adjustment_bucket`, else `ADJUSTMENT_SEGMENT = '99 - Adjustments'` and + `ADJUSTMENT_BUCKET = '04 - Forecast'`. `adjustmentNames()` resolves them once + per request, for the INSERT and the returned rows alike. + +The version defaults are set on the Baseline page (**Defaults for new +adjustments**) and name the *next* adjustment; existing entries keep theirs. +Renaming an entry is `PATCH /log/:logid` — from the Baseline page's segment list — +which renames but refuses to clear, owner or admin only because the names are the +pivot's columns for everyone. Relabelling needs a page reload to show, since the +label is part of the aggregated row. A source registered before names were +stored needs Generate SQL once so its templates write them. + +**What this replaced.** The names were derived at read time: a load's `label`, +else `tag`, else `note`, else the version's `unlabeled_load`, else `Unlabeled`; +an adjustment's `label`, else the version default, else the constant — through a +join on `pf.version`. That made the pivot the only reader that knew a row's +bucket: the CASE was copied into `/agg`, the `/data` cursor and a reporting +procedure in `/opt/forecast`, and every direct query had to copy it again. And +editing a version's fallback silently renamed every past adjustment. The backfill +in `01_schema.sql` stamped existing entries with that rule, so nothing on screen +changed name. + +### Column order is stored text + +Perspective orders column groups by the value string, and `asc`/`desc` only +reverses that, so Prior Year → Plan → Actual → Forecast is expressible in neither +direction. A `"01 - "` prefix is the only lever, and it lives in the stored label +and bucket, typed by whoever names the segment. Nothing derives it. + +The one hardcoded ordinal is `99 - Adjustments`, keeping unnamed adjustments +after every *numbered* segment. That proviso is the scheme: ordering is string +ordering, so `99` lands last only while loads carry `01`–`0n`, and an unnumbered +segment sorts after it (digits precede letters). The old `'(adjustment)'` sorted +*first* for the same reason read the other way. Relabelling an adjustment's log +row is how one kind of adjustment is split out from the rest. + +**What this replaced.** The prefix was computed client-side as Perspective +expression columns (`pf_bucket_ord`, `pf_segment_ord`) from `pf.log.seq` and +`pf.version.bucket_order`. It ordered the pivot and nothing else, so every other +reader disagreed; `restore()` replaces `expressions` wholesale, so it had to be +re-applied after every layout load; and ExprTK's scanner tests each *byte* with +`isprint()`, so a label with anything outside printable ASCII could not be +ordered at all. `DEAD_ORDER_EXPRS` in `Forecast.jsx` strips those names from old +layouts. + +### Tags + +A tag names an **initiative** — a line in the ledger and the bridge. It never +affects rows, so it stays editable, even on a closed version. It is not a row +anywhere, only a value repeated across a version's entries; it exists because +somebody typed it. + +**Completions are per version.** `GET /versions/:id/tags` returns this version's +tags with `uses`, `last_used`, `owners`, `can_edit`, refetched on version change +and after every tagged write. It used to be `GET /sources/:id/tags`, across every +version of the source — which put last year's initiatives in front of this +year's and reported use counts earned elsewhere. The cost is that a fresh version +completes from nothing until it has a tag, which is honest: cloning a definition +copies no adjustments. + +**Managed on the Baseline page.** `PATCH /versions/:id/tags` `{ from, to }` +rewrites every entry carrying `from`. A `to` that already exists **merges** — +usually the point, since "Pull Forward" and "Pulled Forward" are one initiative +reported as two. An empty `to` clears. Both are confirmed first, being invisible +as a diff afterwards. Permission is ownership of *every* entry the tag is on — a +half-applied rename leaves the initiative split, the state being fixed — so rows +the reader cannot rename are greyed; the route locks the rows before checking, so +check and rewrite see the same set. The Tags panel sits outside the page's +read-only fieldset: a rep may rename what they named. + +**Completion matches on words.** `TagInput` in `OperationPanel.jsx` lowercases, +strips punctuation and stopwords and lightly stems both sides, so "Pulled +Forward" finds "Pull Forward" and one inserted word does not hide an existing +tag. Something that reduces to an existing tag without being it says so +(*"Already used as … — use that"*). Nothing merges automatically. The change +log's inline tag cell still uses a plain datalist. + +The panel used to carry six quick-pick pills under the tag field; they wrapped to +two rows once tags were longer than a word and pushed the docked form past its +border. The list shows the tag alone — an option's text renders as a second +column, so a use count widened every row for a number nobody picks on. + +--- + +## Selection + +A selection is a list of **slices** (`{column: value}`) plus the pivot's own +filter as **scope**. The server turns them into a WHERE clause; the ledger totals +the loaded table through the same filters, so the panel and the write agree. + +### From a click to a slice + +`perspective-click`'s own filter list cannot be trusted: the datagrid's +`getCellConfig` builds it with `pivot_value ? [pivot, '==', pivot_value] : +undefined`, so **the engine drops a blank group key** — and, being a falsy test, +a `0`. A dropped key widens the write: clicking the blank node under a customer +selected the whole customer, and the ledger agreed, totalling through the same +list. So `cellFilters()` in `Forecast.jsx` re-derives both axes: + +- **Rows** from `detail.row.__ROW_PATH__` — raw values, so dates stay epoch + millis. A subtotal's shorter path leaves deeper levels absent. +- **Columns** from the clicked column's name (taken from the mousedown's cell; + the event names only the measure). The name is the visible split values joined + with `|`, measure last, so the measure is popped before mapping `split_by` + positionally. Mapping every segment used to read the measure as a value of the + first collapsed dimension — a slice matching no row. + +A plain click replaces the selection; ctrl/⌘/shift toggles a slice. The event +carries no modifier flags, so they are read off the preceding mousedown. +`buildEffectiveSlice()` keeps dimension, date and `pf_` keys, turns dates into +ISO strings, and drops the rest; duplicates are collapsed, since under `each` a +repeat would apply twice. A Perspective computed column does not map back to +stored rows, so only native columns slice. + +### Dragged regions + +The datagrid runs in `edit_mode: SELECT_REGION`, forced on every restore so a +layout cannot switch it off. A drag fires `perspective-select` with a +**ViewWindow** — `{ start_row, end_row, start_col, end_col }` — on every +mouseover, so the handler keeps the latest and a window `mouseup` commits it, +each cell going through `cellFilters()`. A single-cell region is ignored: +`perspective-click` owns clicks, and handling both would undo a ctrl-click. + +### Blanks: null and `''` are different values + +Both occur in real data as different rows, so they stay distinct from click to +WHERE; only `undefined` means absent. + +- `buildWhere()` renders null as `IS NULL`. It used to fall through `esc()` to + `= ''`, selecting *the other blank* — the slice named one node and the + operation wrote the other. In a list the null arm is ORed on, since `IN (NULL)` + is never true. +- The ledger filters null with `is null`, not `== null`: the engine matches on + type, and a null threshold on a string column is another dropped predicate. +- On the column axis, `t_tscalar::to_string` renders `''` as `''` and a null as + the text `"null"`, so `"null"` is read back as null. This was believed + impossible — a null was thought to join as `''` too — so `''` cells were + refused and `"null"` was taken literally, a slice matching nothing. The one + ambiguity left is a value that really is the text `null`. +- The panel prints `(null)` / `(blank)`, and `—` only for a column not in the + slice: isolating the blank node and ignoring the column are opposite + statements and used to look identical. + +### The pivot's `pf_` columns are part of the slice + +- **Table columns** — `pf_iter`, `pf_user`, `pf_logid` (`TABLE_SLICE_COLS`) — + compared directly. +- **Log columns** — `pf_segment`, `pf_bucket`, `pf_note`, `pf_op` + (`COMPUTED_SLICE_COLS`) — resolved to `pf_logid IN (SELECT id FROM pf.log WHERE + version_id = … AND …)`, exact because the name lives on the log row. +- **Any other `pf_` key is refused** — skipping is how selections widened. + +They **narrow a slice but do not make it selective** (`pf_iter = scale` alone is +every adjustment): `assertSelective()` needs a dimension, date, `pf_segment` or +`pf_bucket` key. + +**What this replaced.** `pf_iter` was stripped on the theory that cells differing +only by iter band are one slice; `pf_note` and `pf_op` were refused, and the +client pre-empted that by stripping; `pf_segment` and `pf_bucket` were dropped +before that. The ledger stripped none of them. So under the default layout, +grouped by `pf_iter`, a click on a customer's `scale` row totalled the scale rows +and wrote to all the customer's rows; a click on one bucket's cell scaled every +bucket there. + +### Several slices, and total cells + +Several slices become an OR of AND-groups, not per-column `IN` lists — +`{A:1,B:1}` + `{A:2,B:2}` as `A IN (1,2) AND B IN (1,2)` would also match +`A:1,B:2` — parenthesised, since callers append `AND pf_iter NOT IN (…)`. + +**A total cell is its displayed amount.** The grand-total row names no row +dimension, so its slice is whatever the column axis and the pivot filter leave — +possibly `{}`, *everything the filter shows*. `{}` is accepted only with a +non-empty scope (`normalizeSlices()`, `assertSelective()`); alone it would be the +whole version and is refused on both sides. `buildWhereAny()` reads a union +containing `{}` as `TRUE`, the scope then ANDed on — dropping it would narrow the +write to the other cells. The panel names it "Total". + +### Scope: the pivot's filter travels with the slices + +A slice can only mean equality, so a bounded season, an in-list or a substring +has no slice form. The filter is read off `viewer.save()` — in the table's own +types, with the user's operators — and sent as `scope`, `[col, op, value]` +triples; the ledger totals against the same list. Without it the ledger counted +rows the grid was hiding and the operation wrote them. It is ANDed onto each +statement rather than folded into the slices, which under `each` would repeat it +per statement. + +`SCOPE_OPS` in `lib/sql_generator.js` is the whole allowed list, under +**Perspective's** names: `==`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `not in`, +`is null`, `is not null`, and the substring family. Anything else is refused — +a scope silently dropped is a write wider than the panel that authorised it. +`pf_` log columns take `==` or `in` only. + +- **Substring operators map to `ILIKE`**, because the engine's + `t_tscalar::contains` / `begins_with` / `ends_with` lowercase both sides. `LIKE` + would write a fraction of what the grid showed, and narrower is no safer than + wider — neither is the set the ledger counted. The engine matches the term + literally, so `%`, `_` and `\` are escaped. Residual difference: the engine + folds ASCII only, `ILIKE` follows the collation. +- **`matches` / `not matches` stay refused.** RE2 in the engine, not in Postgres — + `\b` is a word boundary in one and a backspace in the other — so a pattern + valid in both could match different rows, which is worse than a 400. +- **An empty term is refused.** An unfinished filter row arrives as + `[col, '==', null]`, and `ILIKE '%%'` or `= ''` would read as every row or none + rather than as the mistake it is. + +The ledger used to drop a slice's column whenever the scope named it, assuming +they could not disagree; under a substring filter, clicking one part then totalled +every matching part. It now drops only an exact `==` repeat. Territory (§Access) +is ANDed on after the scope, from the session. + +### Selection highlight + +The datagrid highlights `model._selection_state.selected_areas` and wipes it on +every mousedown, so a ctrl-click selection would show only its last cell. +`areasRef`, a `sliceKey → rectangles` map beside `slices`, is pushed back after +every change and pruned by live slice key on any deselect, so grid and panel +cannot disagree. + +Rectangles are **re-derived when the grid moves**. A sort reorders rows and a new +segment column shifts every column; tracked coordinates would then highlight +whoever took the position while `slices`, being value-based, still named the +right row — the grid would be the only thing lying. `relocateAreas()` sweeps the +displayed rows as a drag would and keeps cells whose slice is still selected. A +slice can therefore light more cells than were clicked (a slice not naming the +segment matches the customer under old and new segments alike), which is what it +covers. Past `MAX_RELOCATE_ROWS` the highlight is cleared rather than left wrong. + +--- + +## Operations + +`POST /api/versions/:id/{scale|recode|clone}`, common body: + +```json +{ "slices": [ { "channel": "WHS" } ], "scope": [ ["season", "<=", 2027] ], + "apply_mode": "prorate", "tag": "volume_push", "note": "optional" } +``` + +`slices` — one or more (a legacy single `slice` is accepted). `apply_mode` — +`prorate` (default) is one statement over the union and one entry; `each` is one +per slice, so each reaches its target alone and undoes separately (for recode and +clone it only decides how many entries). `pf_user` is the session's. A closed +version answers 403. Operations are not admin-gated; territory and ownership +govern them. The response is the new rows — whole in raw mode, aggregated in +grain mode — for `table.update()`; no reload. + +### SQL shape + +An `ilog` CTE inserts the log row and the insert CTEs reference its id; +`{{where_clause}}` is the selection and `{{exclude_clause}}` keeps +`exclude_iters` out. + +**Every date column travels** — the written columns are every dimension, every +date and the first value and units column. Scale and recode used to carry only +the first date, so on a source with order, requested and ship dates they wrote +the others **null**. The period dimensions were fine, being dimensions — which +hid it: labels right, dates gone. Those dates are sliceable, so an adjustment +scoped to a ship date matched no adjustment ever written. Clone and loads were +always right. Rows written before the fix keep the nulls; deleting and replaying +those entries is the honest repair, since re-deriving them would be inference. + +**One row per coordinate, not per row read.** Reads GROUP BY every dimension and +date. One-for-one, every operation inherited the row count of everything before +it at that coordinate, and the table grew super-linearly with the work done. +Only `pf_logid` and `pf_iter` collapse, so nothing goes null or unsliceable; the +maths is unchanged, `sum(sum(x)) OVER ()` being the aggregate of aggregates. + +### Scale + +Distributes an increment in proportion to each coordinate's current amount: + +```sql +round(("value" / NULLIF(total_value, 0)) * {{value_incr}}, 2) +round(("units" / NULLIF(total_units, 0)) * {{units_incr}}, 5) +``` + +`resolveIncrs()` in `routes/operations.js` computes the two increments per +statement, which is what makes `each` land per slice. Value and units each take +at most one of a target (`target_value`/`target_units`), a percentage +(`value_pct`/`units_pct`) or a change (`value_incr`/`units_incr`), resolved +independently; `target_price` is a third input; legacy `pct: true` is honoured. + +**`target_basis`** — what a target or percentage measures against: + +- `adjustable` — only the rows the operation can write. The **server default**. +- `selected` — everything the pivot shows, `exclude_iters` rows included. Those + cannot move, so the adjustable rows absorb the difference and the pivot lands + on the number asked for; otherwise a target set against a visible total + overshoots by the fixed rows. + +The panel defaults to `selected` and sends it only when the selection holds +excluded rows (otherwise they are identical). The basis row is tinted and tagged +"target", and clicking either row switches. + +#### Price does not move unless it is named + +A price move is invisible in the figure being edited — the dollars land either +way, and only the implied price says whether volume followed. So: + +**`plug`** answers the one ambiguous case, dollars alone: `volume` (default, +price holds, units scale in proportion) or `price` (volume holds). From the Excel +form: + +``` +plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange +plug price: fVol = pVol + bVol +``` + +**A qty input follows the same rule**: with dollars unnamed they follow at the +current price, `value = (curValue / curUnits) * unitsIncr`. Holding them made +every qty edit a price edit — more cases at the same revenue is a price cut +nobody asked for. A slice with no units has no price, so its dollars stay put. + +**The price held is the adjustable rows' own**, never blended with the fixed +rows. The SQL prorates over the adjustable rows only, so an increment sized at a +blended price moved qty by a different factor than dollars wherever reference +rows priced differently — and taking a selection to its fixed total drove qty past +zero, grains with positive sales and negative qty. The ledger preview uses the +same price. + +**Naming the price** (`target_price`, the Excel form's Edit Price) holds it, and +the other named measure decides the rest, on the chosen basis: + +| Named | Result | +|---|---| +| price alone | qty holds; dollars = price × qty | +| price + qty | qty lands as named; dollars = price × that qty | +| price + dollars | dollars land as named; qty = dollars / price | +| all three | refused — two degrees of freedom, three numbers | + +This branch used to read only a qty *target* (a qty change sized dollars on the +old qty), took the fixed qty off a target but not the fixed dollars, and let a +dollar figure outrank the price. Worse, the panel sends no `plug` with a price, +so the server read `volume` and scaled qty by the dollar change too, putting +price back where it started. A named price now bypasses the plug. + +So the deliberate ways to move price are **dollars and qty together** and +**naming the price**; both outrank the plug. + +**What this replaced.** `plug` defaulted to `price`, being what the API did +before the flag existed — so an unqualified dollar edit and every qty edit moved +price, and only a hand-picked `volume` did not. The client's preference is keyed +`pf_scale_plug_v2` so a browser carrying the old `price` choice does not go on +silently moving price. + +#### The ledger and its three boxes + +The scale form is one continuous statement: baseline, adjustments grouped by tag +(untagged ones collapsed), fixed segments, total, then the edit, with a running +total closing on the selected total. Value, qty and price each have `New` / +`Change` / `% change` boxes; typing in one derives the others, and the box typed +in is what is sent — the row you type in *is* the mode. A **Result** line +previews all three under the server's rules and marks the column that cannot +move `· holds`. + +**The boxes foot** — price = value / qty — so at most two are inputs. +`effectiveScaleInputs()` in `OperationPanel.jsx` decides which when all three are +typed: value typed last lets the plug decide (`volume` drops qty, `price` drops +price), since a dollar change says nothing about price or volume; otherwise the +more recently touched of the other two stays. The dropped box keeps its text, so +flipping the plug brings it back. `buildPayload()` in `Forecast.jsx` uses the same +function, so what is posted is what is shown. Clicking into a derived figure pins +it — "units are this" rather than "units may move" — and **unpin** releases it. +Rough edge: all three typed, dollars last, the choice is really which earlier +number to keep, and "Volume" there moves price too. + +**The Moves control** is the plug. It lights whatever the edit moves — both if +both do — and takes a click only while it is the one deciding. It used to render +only for a dollars-only edit, so it was absent before anything was typed and +vanished when a measure was pinned, even by an accidental click — two states that +both looked like a missing control. It now stays, greyed when the inputs have +answered it, showing their answer. + +A price change or percentage is resolved client-side to an absolute +`target_price`; the server has no price increment form. + +#### What the log records + +`params` carries the intent (`pickIntent()`: inputs, `target_basis`, `scope`, +`plug`) plus: + +- **`resolved`** — `value_incr`, `units_incr` and **`method`**: `price` (named), + `both` (dollars and qty named), `volume` (qty named), else the plug. The + request alone cannot say which moved — the plug is a per-browser preference — + so a price edit used to be findable only by noticing qty had not moved. +- **`expected`** — the panel's Result line as a change on its basis (under + `each`, combined and flagged so), beside `resolved` so a panel/server + disagreement shows in the log. Nothing compares them yet. + +#### Refusals + +- **A pool netting to ~zero** (below 1% of gross): shares are `row / total`, so + as the net nears zero rows fly to extreme opposite values. Offsetting slices + are the usual cause; `each` handles them. +- **Nothing to scale** — every unit already on target or empty; individually + skipped slices come back in `slices_skipped`. +- **Holding price on a selection with no value** — no price to hold. + +Rounding each row to 2dp can leave a target a cent off; landing exactly would +need a correction row. + +### Recode + +Moves the selection to other dimension values in one entry: **negative rows** +zero it out where it was, **positive rows** put it back with `{{set_clause}}` +replacing the named dimensions. + +**A date shift moves the put-back leg only**, so a shifted recode is a timing move +— pull forward, push out — and `set` may be empty. Moved rows' period dimensions +are re-derived against the shifted date, as clone's are; an explicit `set` wins; +a row with a null date keeps its label, since a lookup on nothing would blank it. +Without a shift nothing is derived. + +**Coordinates already netting to zero are skipped** (`HAVING sum(x) <> 0`), +rather than writing two empty rows each — which a second recode over cells a +first one emptied would do for all of them. If nothing is left the request is +refused ("Nothing to recode") rather than committing an empty entry. A scoped +account cannot set the territory column (§Access). + +### Clone + +Copies the selection as new business: `set` overrides dimensions, `scale` +multiplies (default 1), `date_offset` shifts every date; the original is +untouched. Period dimensions come from the calendar against the shifted date — +else a mix moved forward a year keeps last year's labels — and `set` wins. Unlike +recode, a null date blanks them. + +**`exclude_iters` does not apply to clone.** It stops operations *modifying* +reference rows — scale would attribute movement to prior-year rows, recode would +write negatives against them. Clone does neither; copying a plan or prior year +out of reference is the operation working as intended, and excluding them made a +deliberate selection silently produce nothing. **`from_logid`** narrows instead, +for a selection spanning segments where only one is wanted. + +Known issue: a clone matching nothing commits an empty entry with +`rows_affected: 0`, where scale and recode refuse. + +### Undo + +`DELETE /log/:logid` deletes the entry's rows then its log row in one transaction +and returns `pf_gkeys` (grain mode) or `pf_ids` (raw) for `table.remove()`; the +view re-sums. Undo is by **ownership**, not territory — your own entries or an +admin's override — because it removes rows wholesale, and half-undoing one would +leave a state nothing describes (and two accounts can share a territory). Closed +versions refuse. Undoing a load returns its plan to unrun. + +Known issues: the `pf_gkeys` come from **live** col_meta, so after a grain change +they match nothing in the loaded table and the pivot keeps the rows (the database +is still right); recode and clone log `apply_mode: unit.mode`, never set, so +their `params` omit it. + +--- + +## Display-grain pre-aggregation + +**Built, with a static grain.** Path B of `pf_perspective_options.md` — a +pre-aggregated extract in a **native** Perspective table — rather than Path A, +live aggregation through a DuckDB virtual server (parked on +`spike/duckdb-virtual-server`), which loses expand, collapse, depth, sort and +filter. + +**The problem.** Shipping raw rows makes the browser pivot all of them and load +time grows with raw rows. Aggregating server-side to the grain the pivot +displays cuts the row count by orders of magnitude while Perspective still does +all the rollup locally. + +### Grain definition — `pf.col_meta.in_grain` + +The flagged `dimension`/`date` columns, plus always `pf_iter` and `pf_logid`. +`grainOf()` in `lib/sql_generator.js` is the single definition (`Setup.jsx` and +`routes/log.js` mirror it), baked into the templates at Generate SQL so load and +operations agree by construction. + +- **Additive measures only.** Sums are exact; an average would ship as + sum + count; `count(DISTINCT …)` cannot be pre-aggregated. Forecast measures + are sums. +- **An unflagged dimension cannot be pivoted or filtered on.** Leaving + high-cardinality columns (part numbers, raw dates) out is what keeps the grain + from exploding back toward raw; the price is that they are unavailable. Dates + usually enter through their derived period dimensions. +- Generation refuses a grain column the table never populates, or a grain with + no measure. + +`in_grain` means *eligible*; today the grain is exactly the eligible set. + +### The model: append deltas, let the view sum + +The Excel pivot cache again: pre-aggregated rows go to the table, the pivot cuts +them locally, and each adjustment is **appended**, never a recomputed total. Rows +are aggregated **per `pf_logid`** — one per grain tuple × `pf_iter` × entry — and +the view sums them. It was also the smallest change from raw mode, which already +appended results with `table.update()`. + +**Synthetic key.** Perspective indexes on one column, so each row carries + +```sql +md5(concat_ws(chr(31), + COALESCE(col1::text, chr(30)), …, pf_iter, pf_logid::text)) AS pf_gkey +``` + +`pf_logid` keeps each entry's contribution distinct, so an append accumulates +rather than replacing a bucket and undo removes exactly that entry. Plain +`concat_ws` skips NULLs, so `('a', NULL)` and `(NULL, 'a')` would collide and +silently merge two groups; `chr(30)` also keeps NULL apart from `''`. The md5 +because the key is an opaque handle read only by `update()`/`remove()`: raw, it +was long and, unique per row, defeated Arrow's dictionary encoding — the largest +column in the payload. 128 bits keeps collisions unreachable. + +### Initial load — `GET /api/versions/:id/agg` + +```sql +SELECT + t., t.pf_iter, t.pf_logid, + ,SUM(t.) AS , SUM(t.) AS + ,l.label AS pf_segment, l.bucket AS pf_bucket + , AS pf_note, l.operation AS pf_op +FROM {{fc_table}} t +LEFT JOIN pf.log l + ON l.id = t.pf_logid +WHERE {{territory_clause}} +GROUP BY , pf_iter, pf_logid, l.operation, l.label, l.tag, l.note, l.bucket +``` + +The `pf.log` columns depend on `pf_logid`, but Postgres will not infer it, so +they are grouped (`LABEL_GROUP_COLS`). **Territory applies before the GROUP BY**, +since its column need not be in the grain. The route sets `LOCAL enable_sort` and +`enable_incremental_sort` off: seeing `pf_logid` pre-sorted off its index, the +planner picks a sort-based GroupAggregate for the "free" sort, but comparing +dozens of keys per row makes it several times slower than a HashAggregate, +whatever `work_mem` or statistics say. The client loads the result with +`worker.table(buffer, { index: 'pf_gkey' })`. + +### Write and delete paths + +Operations insert raw rows under a new logid as always; in grain mode the final +select aggregates **that entry's rows** to grain (`FROM ins`, or recode's +`neg UNION ALL ins`) — safe because they all carry the new logid. The keys are +therefore always new, `update()` **appends**, and nothing is recomputed. + +Undo removes an entry's keys and lets the view re-sum — no re-aggregation, no +emptied-bucket handling. `RETURNING` does not take `DISTINCT`, so: + +```sql +WITH del AS ( + DELETE FROM {{fc_table}} WHERE pf_logid = $1 + RETURNING , pf_iter, pf_logid +) +SELECT count(*)::int AS rows_deleted, array_agg(DISTINCT ) AS pf_gkeys +FROM del; +``` + +### Raw mode + +With nothing flagged, no `get_agg` exists and the client uses `/data`: every row +plus `pf_segment`, `pf_bucket` and `pf_note`, indexed on `pf_id`; operations +return whole rows and undo returns `pf_ids`. + +### Transport + +Both endpoints return Arrow IPC in **one record batch** with `X-Row-Count`. +`/data` reads a cursor in batches into column arrays (row objects multiplied peak +heap) and builds one table with `tableFromArrays`; `/agg` does the same from its +result. One batch because per-batch encoding creates independent dictionaries, +and Perspective's WASM reader crashes on the replacement messages. An earlier +plan to stream batches into `update()` behind a large-dataset banner was never +built, for that reason. + +**Load time is dominated by row count, not payload size.** Even a good grain on a +wide source leaves enough rows that getting them out of Postgres and into JS +dwarfs the aggregation and the Arrow build; halving the payload (the md5 key) +barely moved it. The remaining lever is the dynamic grain. + +### The dynamic variant + +The static grain means an unflagged dimension is never pivotable and every +flagged one is paid for even when the pivot uses none of them. The additive next +step: group by the fields the current pivot uses — the viewer's `group_by`, +`split_by` and `filter` intersected with the eligible set — and re-fetch when they +change. That needs the grain passed per request rather than baked into `pf.sql`, +with operation responses and undo keys computed at the loaded table's grain. + +**Grain drift.** Changing `in_grain` after a load needs Generate SQL and a page +reload, since the loaded table's index and columns are fixed; until then undo's +keys, from live col_meta, match nothing. + +--- + +## The pivot + +### Perspective, vendored + +Perspective 5.4.0, `@perspective-dev/*` — **not** FINOS `@finos/perspective` — +built from a patched fork and committed as tarballs in `ui/vendor`, whose README +covers the patches (column-axis expand/collapse, `split_by_depth`, depth applied +on config update), provenance and rebuild. Bundled through the `/inline` +entrypoints, **never a CDN**: the 4.x CDN bundle resolved its server WASM to an +unversioned path and silently pulled whatever was newest. + +### Layouts — `pf.layout` + +A named `ViewConfig` owned by an account. **Published** — everyone on the +forecast lists and can apply it, only owner or admin may change it — or +**private**. Scope is the **version** by default, where people enter; +`version_id IS NULL` applies to every version of the source. `is_default` (one +per scope, published only) is restored on a first load, version-scoped beating +source-wide — the narrower answer is the one somebody chose for this forecast. +With no default: `group_by: ['pf_iter']` over the first value column. + +Load order: this version's last-used unnamed config in `localStorage` +(`pf_layout_v{vid}`) if any — per-browser continuity, not something anyone names +or shares — else the default. The old named list (`pf_layouts_v{vid}`) is lifted +into `pf.layout` as private rows once and cleared. + +**Permissions are the `pf.log` rule** and `can_edit` rides on every row so the +**Layout ▾** menu greys the rest. Applying is never restricted: the guarantee is +that a published layout cannot be *changed* under people, not that it cannot be +adapted — Save is withheld on others' layouts, Save as… forks. **No territory +clause**: layouts are display config and territory restricts rows. + +`cleanLayout()` gates every stored config before `restore()`, which is +all-or-nothing: it drops unknown columns from the axes and `aggregates` +(including a multi-arg weight column) entry by entry rather than dropping the +layout, and strips `theme`, the dead ordering expressions and `pf_totals`. + +**The dirty dot is a comparison, not a flag.** `restore()` itself fires +`perspective-config-update`, so a flag set there would light the moment a layout +was applied. `sameConfig()` compares against `activeConfigRef`, ignoring `table` +(per-load) and `theme`. + +**Grand totals ride in the layout as `pf_totals: { row, col }`** — the Total +toggles. The engine has no "subtotals but no grand total" mode, so both are +hidden in the grid's DOM; `captureConfig()` adds the key and `cleanLayout()` +strips it. Without it: row shown, column hidden. + +**What this replaced.** Named layouts lived only in `localStorage`, invisible to +others and gone on another machine. The one server-side layout was +`pf.source.default_layout`, an anonymous blob `PUT /sources/:id/default-layout` +let *any* account overwrite for *every* account — a published layout with no +owner. The route is gone; the column is migrated and unread. + +### Axis depth + +Both axes collapse by a **depth in `ViewConfig`** set through `restore()` — +`group_by_depth` (`applyDepth()`) and `split_by_depth` (`applySplitDepth()`). In +the config it survives every rebuild, rides into layouts via `viewer.save()`, and +needs nothing re-applied. + +Each axis is a **stepper** from `DepthButtons.jsx` — `◂ level ▸`, the middle +opening the level list for a jump — three controls whatever the depth. It was one +button per level, growing with the pivot, the current one a blue among greys so +reading it meant scanning; before that a fixed `0 1 2 3` that named nothing and +offered levels a two-field `group_by` lacks. It shows an em dash until a layout +has been read back, and its middle is sized to the longest level so the toolbar +holds still. + +**Depths are 1-based counts of levels to show**, on the stepper and in the +config, so the value crosses unchanged; `server.cpp` sets `set_depth(n - 1)` and +`t_traversal::set_depth` adds 1 back. The toolbar used to send `d + 1` and +subtract it on read-back — self-consistent, and one level off the grid +everywhere. The `+1` dodged depth 0, which reaches the engine as +`set_depth(-1)` on an unsigned `t_depth`, underflows and clamps to fully +expanded. `applyDepth` now floors at 1, so the **Total slot still shows the first +level, not a bare total** — a wart kept rather than traded for the underflow. A +real total needs `group_rollup_mode: 'total'`, unverified on this engine. + +**This required patching the engine.** `ViewConfig::apply_update` applied ten +fields and neither depth, so a depth through `restore()` was accepted and +dropped — and `Session::update_view_config` returns early when nothing changed, +so no view was rebuilt. `ui/vendor/0001-apply-depth-fields-on-config-update.patch`. + +**What this replaced:** + +- Row depth was imperative — `view.set_depth()` on an object the viewer discards + whenever it re-renders — so restoring it meant guessing when that happened: an + `observerShim.js` patching `IntersectionObserver` and `ResizeObserver`, focus + and visibility listeners, a retry loop for `getView()` throwing `No table set`, + a flag for whether the viewer had "gone away". That produced three visible + faults: the tree fully expanding on refocus, snapping on any reflow, and + snapping when the settings sidebar opened. +- Column collapse restored a **truncated `split_by`**, so discarded levels were + remembered separately (`splitFull`, persisted as `split_full`), our collapse + had to be told apart from the user rearranging, and the selection was cleared + on every collapse. `split_full` and `expand_depth` are still read from old + layouts. + +**Depth changes rebuild the view** — `restore()` makes the session tear down and +rebuild it, a full traversal, where a manual collapse mutates in place. +Noticeable on a large grain. Detecting a depth-only change and calling +`view.set_depth()` while still writing the config would fix it, at the risk of the +two drifting. + +**Per-node expansion is not solved.** It is view state with no config form, and +the API is set-only (`expand(row_index)` / `collapse(row_index)`, no getter), so +it cannot be captured and replayed and is lost on every rebuild. The honest route +is a `ViewConfig` field of expanded row *paths* (indices shift as the tree +opens), applied in `server.cpp` beside the depths — needing node enumeration in +the C++ traversal, a proto field and the apply step. `expand_column` / +`collapse_column` share the limitation. And depth is whole-axis: Excel can +collapse one year while the next stays open. + +### Auto-pause is off + +`` auto-pauses by default — an `IntersectionObserver` plus +`visibilitychange` — and pausing **deletes the view**, so returning to the tab +means `restore_and_render()`: a fresh traversal of the whole grain, a +multi-second chug per tab switch on a large one, losing per-node expansion. +`initViewer()` calls `viewer.setAutoPause(false)` after `load()`. Nothing updates +the table while hidden — every operation is driven from this page — so the pause +bought nothing. + +### Toolbar + +Two zones. Left, **view**: `Layout ▾` and the Rows and Columns steppers with +their Total chips — all `ViewConfig`, nothing reaching the server. Right, +**forecast**: Refresh data, Change log, Bridge, Operations — re-running the load, +opening something over the pivot, or writing. + +The distance between them is the separation. Each was once an outlined box too, +the view one tinted too faintly to see and, in dark mode, reading as a sibling; +before that everything was one row of identical hairlines, making a +column-width control and the version's history look like peers. Three weights do +the separating: `.pf-section-label` names a group and is heavier than its +controls, `.pf-group-rule` divides groups within a zone, the gap divides zones. +Each axis is one joined control, `◂ | level | ▸`, then its Total chip. + +**Blue is for actions, pressed grey for state.** Save and Post are blue; the +Total chips and an open Operations use `.pf-pressed`. Operations used to relabel +itself "Hide panel" in a blue outline, looking like a second Save. + +**Fit widths is gone**, for three independent reasons: auto-size measures over +the rows *in the viewport*, so anything below the fold never counted; +`draw({invalid_columns})` passes an option `DrawOptions` lacks, and `draw()` is +throttled, so a click during a pending draw cleared widths without re-measuring; +and the `persistLayout` after it saved nothing, since `saveColumnSizes()` returns +only overrides, which `resetAutoSize()` had emptied — and this fork discards +`column_size_override` whenever `split_by` is set, which here is always. +Recoverable from git if ever worth fixing. + +The operation panel docks **bottom**, **right** or **floats**, persisted locally; +the toolbar toggle shows the selection count while it is shut. + +### Calculating indicator + +**Calculating…** shows in the status bar while the pivot rebuilds. Perspective's +beaker renders only while its settings panel is open, vanishing exactly when +nothing else says to wait. A counter — rebuilds overlap — rises in `applyDepth`, +`applySplitDepth` and `applyLayout`, falls after `viewer.flush()`, and travels as +`pf-pivot-busy` on the window, like `pf-data-changed`. + +**Not** driven by `perspective-config-update`, the obvious hook: on a depth +change over a large grain it arrives *after* the rebuild, seconds late, and the +flush after it resolves at once. The cost is that sorting or filtering from the +viewer's own chrome shows nothing, since the app is never told they began. + +### Light and dark + +`ui/src/theme.jsx` holds a context stored as `pf_dark`, defaulting to +`prefers-color-scheme`; the status-bar toggle sets `.dark` on ``. +`index.css` defines properties under `:root` and `.dark`, with Tailwind overrides +written as `.dark .bg-white { … }` — no Tailwind dark config. Dark uses +Perspective's Pro Dark palette. + +The pivot takes a **partial** `viewer.restore({ theme: 'Pro Dark' | 'Pro Light' })` +on load and on toggle. It used to be `setAttribute('theme', …)`, which does +nothing in 5.4.0 — the theme is a `ViewerConfig` field — so the pivot stayed light +whatever the toggle said. The other half is `cleanLayout()` dropping `theme`: +`viewer.save()` includes it, so every layout reasserted its author's theme. It is +the reader's choice, so a published layout cannot impose one. + +--- + +## Access + +### Authentication and sessions + +Everything under `/api` but `/login`, `/logout`, `/me` sits behind +`requireAuth`; the React app mounts only with a session (`Gate` in `main.jsx`), +since its effects call the API at once, and `auth.jsx` wraps `fetch` so a 401 +returns to login. + +- **Accounts** — `pf.app_user`, scrypt hashes whose stored form carries its + parameters, so they can be raised without invalidating rows. `./pf.sh add-user + | passwd | list-users | disable-user | enable-user`; passwords go in on stdin + through `node lib/auth.js hash`, never argv or psql. +- **Sessions** — `connect-pg-simple` in `pf.session`, so restarts keep people + signed in and a session is revoked by deleting its row (`disable-user` does). + Cookie `pf.sid`: httpOnly, SameSite=Lax, Secure unless `COOKIE_SECURE=false`, + 12h rolling. +- **Config** — the server exits without `SESSION_SECRET` rather than using a + guessable default. `TRUST_PROXY` (default 1) keeps `req.ip` and secure cookies + right behind the TLS proxy. CORS is enabled only by `CORS_ORIGIN`, by name — + a wildcard origin plus a session cookie is CSRF by construction. +- **Login hardening** — ten failures per IP per fifteen minutes (in memory), one + message for unknown/wrong/disabled, session id regenerated on success. + +**Identity is server-side.** `pf_user`, `created_by`, `closed_by` come from +`sessionUser(req)`, never the body. The UI used to send a hardcoded +`pf_user: 'admin'`, which any client could set to anything. + +### Setup and Baseline are admin-only to write + +`requireAdmin` sat exported in `lib/auth.js` and applied to nothing, so every +non-admin could call every Setup and Baseline endpoint. It now guards source +writes (register, cols, Generate SQL, member refresh, deregister), version writes +(create, update, close, reopen, delete), loads (baseline, reference, segment edit, +clear) and every plan write. `is_default` is further admin-checked inside +`PUT /versions/:id`. + +**Not gated, each for a reason.** No read, including the table browser — the +pages stay visible, and gating what a page needs to *render* turns it into a 403. +Not the forecast operations, which carry territory and ownership rules. Not +`PATCH /versions/:id/tags`, whose permission is ownership. + +**The pages go read-only rather than hidden**: one `
` per page +instead of a flag threaded through every control — Baseline alone has dozens, and +the one missed would be the one that mattered — plus a `ReadOnlyNotice` banner. +Setup wraps a real box because the fieldset *is* its flex row; Baseline uses +`display: contents` so its layout is untouched. **None of this is the +permission**; it only stops offering buttons that would answer 403. + +**A 403 used to blank the whole app.** Setup put responses straight into state +and `.map`ped them; an error body is `{error}`, and throwing in render unmounts +React. Its list fetches now coerce to arrays, so a bad response costs an empty +panel, not the session. The fragility predated the gating, which only made it +reachable. + +### Territory scoping + +An account sees and changes only its territory: `pf.app_user.territory` (jsonb +list), `is_admin` for the unrestricted, and `col_meta.is_territory` naming which +column of a source the values belong to — flagged, not coded, so another source +can be divided by something other than a sales rep. + +**Fail closed.** No territory and not admin means no rows: +`buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an +unflagged source, so a forgotten account sees nothing rather than the book. + +**Built from the session, never the request** — the difference from `scope`, +which the browser rightly sends as a filter the user chose. A permission cannot +come from the thing it restrains, so the predicate is ANDed on last, in +`sliceUnits()` for writes and per route for reads. Enforced at: + +- `/data` — cursor *and* the `X-Row-Count` count +- `/agg` — `{{territory_clause}}`, before the GROUP BY +- `/versions/:id/table-info` — unscoped, it told an account its load was + fetching the whole table +- every operation (`pf.log.env` records the territory in force) +- `/sources/:id/values/:col` — it reads the *source*, untouched by any scope +- recode's `set` — a scoped account cannot set the territory column. That is + reassignment, not forecasting, and the row would vanish from the view that + would have shown what happened. + +**Ownership, not territory**, governs `DELETE /log/:logid`, `PATCH /log/:logid` +and `PATCH /versions/:id/tags`. The log PATCH looks like a private annotation and +is not: `label` and `bucket` name the pivot's columns for everyone, so unguarded +it let any account rename the company's segments, including on loads whose rows +it cannot see. + +Known gaps, verified in code: + +- **Completion now uses unscoped routes.** Recode and clone complete from + `GET /versions/:id/values/:col` (the version's own table, cached in memory + keyed on the latest logid), the member list from `/sources/:id/dim/:group`, + sibling autofill from `/sources/:id/lookup`. None applies the territory clause. +- **Territory is read onto the session at login**, so a change reaches a + signed-in account only when it signs in again. Re-reading it in `requireAuth` + would also make disabling immediate. + +`./pf.sh set-territory | set-admin | orphan-territory`; the last lists values in +the data that no account owns — work under one is invisible to all but admins, +which a typo causes easily and nothing in the app reveals. + +--- + +## Bridge and change log + +### Bridge + +A waterfall of *how this forecast got from where it started to where it stands*, +one step per initiative tag. `BridgeView.jsx`; the arithmetic is in exported pure +`buildSteps()` / `layoutSteps()`, checkable against real data without a browser. + +**Computed from the table already in the browser**, not `GET /versions/:id/bridge`, +so it reconciles with the pivot. The endpoint remains for API consumers, grouping +by tag over rows outside `exclude_iters`, with no notion of buckets. + +**Scope** is the selection (the union of its slices, an operation's reach) or the +pivot's filter, following the live selection until a scope button is picked. +There is **no whole-version scope**: the filter is part of what the figures +mean, and dropping it moves one end of the walk and not the other — rows outside +a season filter sat wholly in the forecast loads with no counterpart in the plan, +inflating the endpoint without explanation. View filters are AND-only, so +selection scope builds a view per slice, each with the pivot's filter. + +**Membership comes from `pf_bucket`, not `pf_iter`**: open orders load as +reference and are still forecast, and an iter-keyed bridge dropped them. The +forecast bucket is the version's `adjustment_bucket` — the value unbucketed +adjustments carry, so the two agree by construction. It used to be the literal +`'Forecast'`; once buckets carried their sort prefix nothing matched, every row +became comparison, and the basis cancelled itself to zero. With no match it +falls back to whichever bucket holds the adjustments. Selection scope does not +filter on `pf_bucket`/`pf_segment` even when a slice names one, since Compare to +needs rows from every bucket. + +**Compare to** sets a bucket as basis — Plan when building a forecast, Prior Year +when building the plan — resting on + + Forecast − Basis = (Forecast loads − Basis) + adjustments + +The opening step is the forecast's loads against the basis and the tagged +adjustments explain the rest: no residual, no plug, the bars summing to the +endpoint by construction. With no basis it shows the forecast's composition. + +**Steps group by tag alone**, case-insensitively, showing the commonest spelling. +They grouped by label first once, to match `pf_segment`, but every adjustment +carries the same default label, so no tag ever showed — the label is the segment, +the tag the initiative. Untagged adjustments collapse to one "N untagged +adjustments" step, as in the ledger, rather than a wall of identical bars. + +**Colour encodes polarity**: increase and decrease as two poles of a validated +blue/red diverging pair (green/red is the classic colour-vision failure), neutral +grey anchors. Everything not a bar follows the theme via CSS variables set +through `style`, where `var()` is reliable. Every bar is labelled, a table view +gives full precision, and the panel can be dragged. + +Known issues: + +- **Overlapping slices are deduplicated by `pf_id`**, which grain rows lack, so + in grain mode a row matching two selected slices counts twice; it should key on + `pf_gkey`. +- **Selection scope keeps only dimension and date keys**, dropping the `pf_iter` / + `pf_note` / `pf_op` / `pf_user` / `pf_logid` keys operations honour, so it + bridges a wider set than the operation would write; a total-cell `{}` slice is + skipped outright. +- **No drill-down** from a step to its entries. + +### Change log + +A modal of the version's entries, newest first — time, operation, slice, tag, +note, rows and totals, author — with undo per row and tag and note edited inline +(Enter saves, Esc cancels). `PATCH /log/:logid` branches on whether a field was +sent, so `""` clears rather than meaning "leave alone" — except `label` and +`bucket`, renamed from the Baseline page and never cleared. A saved tag regroups +the ledger and bridge at once, so history can be reclassified. Multi-slice entries +list a line per slice, pulling out shared fields when only one dimension varies +(a dragged region); `{}` reads "Total within the pivot filter". An entry with +stored SQL opens its debug record — intent, environment, executed statement. + +**It shows an entry's full impact**, not the reader's share: stamped totals are +company-wide, so an admin's version-wide scale reads the same everywhere — +deliberately, and labelled, rather than re-aggregated per territory. + +**Everyone / Mine.** Everyone's entries are listed, with the reader's own marked +in place — a blue rule down the row and a blue `By` cell. With several active +accounts, finding your own meant reading `By` down the list. Marked *and* +filterable: the mark tells without hiding anybody, the filter is for wanting only +yours. It opens on Mine every time, unpersisted — the log is almost always opened +to check your own recent work, and a stale "Everyone" is the worse default. +`isMine()` is strict authorship, not `canUndo()`: an admin can undo everything, +and marking every row as theirs would say nothing. The fetch stays unfiltered, +since the control's counts need the whole set. Rows the reader cannot change are +greyed rather than offering a 403. + +`?kind=adjustments` drops load entries, which own nearly every row, from the +recount join. + +Known issue: the log does not show an entry's id, so there is no way to name one +when asking about it. + +--- + +## Proposed: additional measures + +Status: **proposed — not built.** Nothing below exists in code yet. + +### The problem + +The app forecasts exactly one dollar measure (`role = value`) and one quantity +(`role = units`). `col_meta` lets a second column be marked `value`, but only +the loads carry it (`generateSQL` in `lib/sql_generator.js`: "the adjustment +operations are single-measure … and use only the first of each"). So a second +measure is right on the baseline and wrong everywhere an adjustment has been +made: + +- scale, recode and clone write adjustment rows with that column **null** +- a recode moves the sales and leaves the cost behind +- in grain mode `/agg` sums only the first value and the units column, so it + is not in the pivot at all + +Extended cost is the first case wanted — margin is sales minus cost — and the +same shape recurs for rebates, surcharges, freight and the like. + +### The model: every extra measure is a rate on a driver + +An extra measure is almost never forecast on its own. It is a rate applied to +something already being forecast: + +| Measure | Driver | Rate held by an ordinary edit | +|--------------------|--------|-----------------------------------| +| Extended cost | qty | unit cost | +| Surcharge per unit | qty | surcharge per unit | +| Rebate | sales | rebate as a % of sales | +| Freight (fixed) | none | — carried, not moved by scale | + +Declared once, in Setup: `col_meta.role = 'measure'`, plus a new +`col_meta.driver` of `units` | `value` | `none`. The existing `value` and +`units` roles are unchanged and remain the two the ledger edits. + +### How each operation treats a measure + +The rule is that **an edit never changes a measure's rate unless the rate is +what is being edited** — the same rule the scale already applies to price. + +| Operation | Measure driven by qty | by sales | none | +|--------------------|---------------------------------|--------------------------------|----------------------| +| Load | copied | copied | copied | +| Scale | moves by the qty factor | moves by the dollar factor | untouched | +| Recode | both legs, like the other cols | both legs | both legs | +| Clone | × scale factor | × scale factor | × scale factor | +| Undo | removed with its rows | removed with its rows | removed | + +So a price-only edit moves no qty-driven cost (qty did not move) but does +move a sales-driven rebate (dollars did). The scale SQL already prorates +value and units by one factor per coordinate; each measure is prorated by its +driver's factor the same way, per row, so the rate holds per row and not only +in total. + +Because scale moves a measure at the slice's **current** rate, a rate set +earlier is carried forward by every later edit, which is what makes Set rate +(below) stick. + +### Setting a rate: its own operation, not the ledger + +The ledger stays three boxes — value, qty, price — and the measures stay out of +it. The ledger is already the hardest part of the panel to read, and a rate is +a different question from a price-or-volume edit. + +The case that needs it: a part used as a proxy for a new part not yet set up, +where the new part's cost is roughly known. The workflow is clone or recode the +proxy's volume onto the new part, then set the new part's cost. + + Set rate · measure ext_cost · per qty · new rate 0.31 + +- Writes adjustment rows that change **only that measure**: the slice's cost + goes to rate × its qty; sales and qty do not move. The same shape as a price + edit, on a different measure. +- A sales-driven measure takes its rate as a % of sales. +- A `none` measure takes a plain target or change amount. +- One `pf.log` entry, `operation = 'rate'`, `params` recording the measure, + the rate and the resolved change; undoable, listed in the change log, and a + line in the bridge like any other adjustment. +- Measured against the same basis choice as scale (adjustable rows or the + selected total), with the same territory and scope rules. + +### In the pivot + +Each measure is summed like sales and qty — `/agg` and the `/data` cursor carry +every `measure` column. Rates and margins are not stored: they are Perspective +expression columns computed from the sums (unit cost = cost / qty, rebate % = +rebate / sales, margin = sales − cost, margin % = margin / sales), so they are +right at every level of the tree. + +### What has to change + +- `col_meta`: `role = 'measure'`, new `driver` column; Setup's editor offers both. +- `lib/sql_generator.js`: loads and `/agg` carry every measure; scale prorates + each by its driver's factor; recode and clone carry them; a new `rate` + template. +- `routes/operations.js`: a `rate` route; scale passes the factors through. +- `OperationPanel.jsx`: a Set rate form beside Scale / Recode / Clone. +- Existing versions: the forecast table needs the measure columns, so a version + created before a measure was added needs them added (or recreating) — the + same col_meta drift noted under §Open questions. + +### Open questions + +- Should a measure be allowed more than one driver (a surcharge that is part + per-unit, part percentage)? Proposed: no — split it into two measures. +- Does Set rate on a slice with no qty (a qty-driven measure) refuse, as the + price hold does on a slice with no value? Proposed: yes. + +--- + +## Open questions + +- **Replay** — re-running a version's adjustments against a restated baseline. + Not built: no route takes `replay` and nothing returns 501. The hard part is + the one cloning a definition declines: a scale's target was sized against the + rows that were there, so a replay must choose between the *figure* and the + *intent* (`params` holds both inputs and `resolved`). `env`'s template + fingerprint exists partly so a replay can tell an entry ran under another + template. +- **Approval workflow** — a submission an admin approves before others see it. + Deferred. +- **Export** — CSV, or a push to a reporting table. The bridge's table view is a + partial stand-in; there is no download. Stored names (§Naming) mean a reporting + query needs only a join to `pf.log`. +- **Cross-version comparison** — side by side, or a bridge between versions; the + isolated tables make it a UNION. Compare to answers the within-version form + (against a plan bucket loaded into the same version); across versions is open. +- **Bridge drill-down** — a step listing its entries, or selecting them back in + the pivot. +- **Multi-connection support** — one database via `.env` today. The vision: a + `pf.connection` table (host, port, dbname, user, password as an env-var + reference), `connection_id` on `pf.source`, per-connection pools; `pf` stays on + a home connection and sources live anywhere, with a Connections panel in Setup. + It changes the source schema, so it wants a clean install. +- **Col_meta / version schema drift** — roles changed after a forecast table + exists put the generated SQL and the table's columns out of sync, unnoticed. + Setup should compare col_meta with each table's columns, warn, and offer to + rebuild (recreate the table, keep the version and log). Today: delete and + recreate the version. Additional measures would hit this on every version. +- **Separating `is_key`'s meanings** — §Data model → `pf.col_meta`. +- **A per-source default layout** beyond the `pf_iter` fallback, built from + col_meta when nothing is published. +- **Dynamic grain** — §Display-grain pre-aggregation. +- **Per-node expansion that survives a rebuild** — §The pivot → Axis depth. +- **Tests** — SQL generation is token substitution against append-only tables, + exercised only by hand. diff --git a/pf_spec.md b/pf_spec.md deleted file mode 100644 index ee59f0c..0000000 --- a/pf_spec.md +++ /dev/null @@ -1,1076 +0,0 @@ -# Pivot Forecast — Application Spec - -## Overview - -A web application for building named forecast scenarios against any PostgreSQL table. The core workflow is: load known historical actuals as a baseline, shift those dates forward by a specified interval into the forecast period to establish a no-change starting point, then apply incremental adjustments (scale, recode, clone) to build the plan. An admin configures a source table, generates a baseline, and opens it for users to make adjustments. Users interact with a pivot table to select slices of data and apply forecast operations. All changes are incremental (append-only), fully audited, and reversible. - ---- - -## Tech Stack - -- **Backend:** Node.js / Express -- **Database:** PostgreSQL — isolated `pf` schema, installs into any existing DB -- **Frontend:** React + Vite + Tailwind CSS; Perspective (forecast pivot) -- **Pattern:** Follows fc_webapp (shell) + pivot_forecast (operations) - ---- - -## Database Schema: `pf` - -Everything lives in the `pf` schema. Install via sequential SQL scripts. - -### `pf.source` -Registered source tables available for forecasting. - -```sql -CREATE TABLE pf.source ( - id serial PRIMARY KEY, - schema text NOT NULL, - tname text NOT NULL, - label text, -- friendly display name - status text DEFAULT 'active', -- active | archived - default_layout jsonb, -- Perspective view config used as per-source default - created_at timestamptz DEFAULT now(), - created_by text, - UNIQUE (schema, tname) -); -``` - -### `pf.col_meta` -Column configuration for each registered source table. Determines how the app treats each column. - -```sql -CREATE TABLE pf.col_meta ( - id serial PRIMARY KEY, - source_id integer REFERENCES pf.source(id), - cname text NOT NULL, -- column name in source table - label text, -- friendly display name - role text NOT NULL, -- 'dimension' | 'value' | 'units' | 'date' | 'filter' | 'ignore' - is_key boolean DEFAULT false, -- true = part of natural key (used in WHERE slice) - opos integer, -- ordinal position (for ordering) - dim_group text, -- groups functionally dependent columns (see below) - dim_period_col text, -- maps this dimension to a pf.dim_period column - UNIQUE (source_id, cname) -); -``` - -**Roles:** -- `dimension` — categorical field (customer, part, channel, rep, geography, etc.) — appears as pivot rows/cols, used in WHERE filters for operations -- `value` — the money/revenue field to scale (**required** — SQL generation fails without it) -- `units` — the quantity field to scale (**optional** — if absent, units columns are omitted from the forecast table and all SQL patterns) -- `date` — the primary date field; used for baseline/reference date range and stored in the forecast table (**required**) -- `filter` — columns available as filter conditions in the Baseline Workbench (e.g. order status, ship date, open flag); used in baseline WHERE clauses but **not stored** in the forecast table -- `ignore` — exclude from forecast table entirely - -**`dim_group`** — a free-text group name linking a `date` column to its derived dimension siblings. When the `date` column has `is_key = true` and a `dim_group` value, the SQL generator looks for `dimension` columns in the same group that also have a `dim_period_col` value. Those columns are sourced from `pf.dim_period` on baseline/reference load (via a JOIN on `drange @> date`) rather than copied raw from the source table. This allows fiscal year, quarter, and month columns to be stored in the forecast table with calendar-correct values even if those columns don't exist in the source. - -**`dim_period_col`** — names the column in `pf.dim_period` to use as the value for this dimension on load. Only meaningful when the column is in a `dim_group` whose `date` key has `is_key = true`. Example: `cal_year`, `fisc_quarter`, `fisc_label`. - -### `pf.dim_period` -Calendar lookup table. One row per month from 2018-01-01 through 2035-12-01. Keyed on `sdat` (month start date). Used to derive fiscal/calendar period columns at baseline load time when `dim_group` / `dim_period_col` are configured on col_meta. - -Populated by `setup_sql/gen_dim_period.sql` (safe to re-run; `ON CONFLICT DO NOTHING`). Fiscal year start month is configurable at the top of that script (default: June, i.e. fiscal month 1 = June). - -Key columns: `sdat`, `edat`, `drange` (GiST-indexed daterange), `cal_year`, `cal_quarter`, `cal_month`, `cal_month_abbr`, `cal_month_name`, `cal_label`, `fisc_year`, `fisc_quarter`, `fisc_quarter_label`, `fisc_month`, `fisc_month_abbr`, `fisc_month_name`, `fisc_label`, `period_key`. - -The baseline/reference SQL JOINs this table when `hasDimPeriod` is true: `JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`. - -### `pf.version` -Named forecast scenarios. One forecast table (`pf.fc_{tname}_{version_id}`) is created per version. - -```sql -CREATE TABLE pf.version ( - id serial PRIMARY KEY, - source_id integer REFERENCES pf.source(id), - name text NOT NULL, - description text, - status text DEFAULT 'open', -- open | closed - exclude_iters jsonb DEFAULT '["reference"]', -- iter values excluded from all operations - created_at timestamptz DEFAULT now(), - created_by text, - closed_at timestamptz, - closed_by text, - UNIQUE (source_id, name) -); -``` - -**`exclude_iters`:** jsonb array of `iter` values that are excluded from operation WHERE clauses. Defaults to `["reference"]`. Reference rows are still returned by `get_data` (visible in pivot) but are never touched by scale/recode/clone. Additional iters can be added to lock them from further adjustment. - -**Forecast table naming:** `pf.fc_{tname}_{version_id}` — e.g., `pf.fc_sales_3`. One table per version, physically isolated. Contains both operational rows and reference rows. - -Creating a version → `CREATE TABLE pf.fc_{tname}_{version_id} (...)` -Deleting a version → `DROP TABLE pf.fc_{tname}_{version_id}` + delete from `pf.version` + delete from `pf.log` - -### `pf.log` -Audit log. Every write operation gets one entry here. - -```sql -CREATE TABLE pf.log ( - id bigserial PRIMARY KEY, - version_id integer REFERENCES pf.version(id), - pf_user text NOT NULL, - stamp timestamptz DEFAULT now(), - operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone' - slice jsonb, -- the WHERE conditions that defined the selection - params jsonb, -- operation parameters (increments, new values, scale factor, etc.) - note text, -- user-provided comment - tag text -- initiative label, e.g. 'reduce_spend' -); -``` - -`tag` groups adjustments into initiatives. It is what the bridge walks: every entry -carrying the same tag becomes one step from baseline to current. Both `note` and `tag` -are annotations — they never affect forecast rows — so both stay editable after the -fact via `PATCH /api/log/:logid`. - -Tags are written by a follow-up `UPDATE` after the operation runs, not by the generated -SQL. The templates in `pf.sql` are stored per source, so adding a `{{tag}}` token would -silently stop recording tags for any source that had not re-run *Generate SQL*. - -### `pf.fc_{tname}_{version_id}` (dynamic, one per version) -Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`). - -```sql --- Example: source table "sales", version id 3 → pf.fc_sales_3 -CREATE TABLE pf.fc_sales_3 ( - id bigserial PRIMARY KEY, - - -- mirrored from source (role = dimension | value | units | date only): - customer text, - channel text, - part text, - geography text, - order_date date, - value numeric, - units numeric, -- omitted if no 'units' role in col_meta - - -- forecast metadata: - pf_iter text, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone' - pf_logid bigint REFERENCES pf.log(id), - pf_user text, - pf_created_at timestamptz DEFAULT now() -); -``` - -Note: no `version_id` column on the forecast table — it's implied by the table itself. The `units` column is only present when a column with `role = 'units'` exists in col_meta. - -### `pf.sql` -Generated SQL stored per source and operation. Built once when col_meta is finalized, fetched at request time. - -```sql -CREATE TABLE pf.sql ( - id serial PRIMARY KEY, - source_id integer REFERENCES pf.source(id), - operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone' | 'get_data' | 'undo' - sql text NOT NULL, - generated_at timestamptz DEFAULT now(), - UNIQUE (source_id, operation) -); -``` - -**Column names are baked in at generation time.** Runtime substitution tokens: - -| Token | Resolved from | -|-------|--------------| -| `{{fc_table}}` | `pf.fc_{tname}_{version_id}` — derived at request time | -| `{{where_clause}}` | built from `slice` JSON by `build_where()` in JS | -| `{{exclude_clause}}` | built from `version.exclude_iters` — e.g. `AND pf_iter NOT IN ('reference')` | -| `{{logid}}` | newly inserted `pf.log` id | -| `{{pf_user}}` | from request body | -| `{{date_from}}` / `{{date_to}}` | baseline/reference date range (source period) | -| `{{date_offset}}` | PostgreSQL interval string to shift dates into the forecast period — e.g. `1 year`, `6 months`, `2 years 3 months` (baseline only; empty string = no shift) | -| `{{value_incr}}` / `{{units_incr}}` | scale operation increments | -| `{{pct}}` | scale mode: absolute or percentage | -| `{{set_clause}}` | recode/clone dimension overrides | -| `{{scale_factor}}` | clone multiplier | - -**Request-time flow:** -1. Fetch SQL from `pf.sql` for `source_id` + `operation` -2. Fetch `version.exclude_iters`, build `{{exclude_clause}}` -3. Build `{{where_clause}}` from `slice` JSON via `build_where()` -4. Substitute all tokens -5. Execute — single round trip - -**WHERE clause safety:** `build_where()` validates every key in the slice against col_meta (only `role = 'dimension'` columns are permitted). Values are sanitized (escaped single quotes). No parameterization — consistent with existing projects, debuggable in Postgres logs. - ---- - -## Setup / Install Scripts - -``` -setup_sql/ - 01_schema.sql -- CREATE SCHEMA pf; create all metadata tables (source, col_meta, version, log, sql) -``` - -Source registration, col_meta configuration, SQL generation, version creation, and forecast table DDL all happen via API. - ---- - -## API Routes - -### DB Browser - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/api/tables` | List all tables in the DB with row counts | -| GET | `/api/tables/:schema/:tname/preview` | Preview columns + sample rows | - -### Source Management - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/api/sources` | List registered sources | -| POST | `/api/sources` | Register a source table | -| GET | `/api/sources/:id/cols` | Get col_meta for a source | -| PUT | `/api/sources/:id/cols` | Save col_meta configuration | -| POST | `/api/sources/:id/generate-sql` | Generate/regenerate all operation SQL into `pf.sql` | -| GET | `/api/sources/:id/sql` | View generated SQL for a source (inspection/debug) | -| DELETE | `/api/sources/:id` | Deregister a source (does not affect existing forecast tables) | - -### Forecast Versions - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/api/sources/:id/versions` | List versions for a source | -| POST | `/api/sources/:id/versions` | Create a new version (CREATE TABLE for forecast table) | -| PUT | `/api/versions/:id` | Update version (name, description, exclude_iters) | -| POST | `/api/versions/:id/close` | Close a version (blocks further edits) | -| POST | `/api/versions/:id/reopen` | Reopen a closed version | -| DELETE | `/api/versions/:id` | Delete a version (DROP TABLE + delete log entries) | - -### Baseline & Reference Data - -| Method | Route | Description | -|--------|-------|-------------| -| POST | `/api/versions/:id/baseline` | Load one baseline segment (additive — does not clear existing baseline rows) | -| DELETE | `/api/versions/:id/baseline` | Clear all baseline rows and baseline log entries for this version | -| POST | `/api/versions/:id/reference` | Load reference rows from source table for a date range (additive) | - -**Baseline load request body:** -```json -{ - "date_offset": "1 year", - "filters": [ - [ - { "col": "order_date", "op": "BETWEEN", "values": ["2024-01-01", "2024-12-31"] }, - { "col": "order_status", "op": "IN", "values": ["OPEN", "PENDING"] } - ], - [ - { "col": "order_status", "op": "IS NULL" } - ] - ], - "pf_user": "admin", - "note": "FY2024 actuals + open orders projected to FY2025", - "replay": false -} -``` - -The example above generates: `(order_date BETWEEN '2024-01-01' AND '2024-12-31' AND order_status IN ('OPEN','PENDING')) OR (order_status IS NULL)` - -- `date_offset` — PostgreSQL interval string applied to the primary `role = 'date'` column at insert time. Examples: `"1 year"`, `"6 months"`, `"2 years 3 months"`. Defaults to `"0 days"`. Applied to the stored date value only — filter columns are never shifted. -- `filters` — an array of **groups**. Conditions within a group are AND-ed; groups are OR-ed together. Each group is an array of one or more condition objects: - - `col` — must be `role = 'date'` or `role = 'filter'` in col_meta - - `op` — one of `=`, `!=`, `IN`, `NOT IN`, `BETWEEN`, `IS NULL`, `IS NOT NULL` - - `values` — array of strings; two elements for `BETWEEN`; multiple for `IN`/`NOT IN`; omitted for `IS NULL`/`IS NOT NULL` - - Backward compatibility: a flat array of condition objects (non-nested) is treated as a single group (all AND). -- At least one group with at least one condition is required. -- `raw_where` — optional string. When present, bypasses `filters` entirely and injects the value verbatim as the WHERE clause body. **Admin-only** — rejected with `403` if the requesting `pf_user` is not in the admin list. Not validated against col_meta. Caller is responsible for correctness and SQL safety. Stored as-is in `pf.log.params` for audit. Cannot be combined with `filters` — if both are present the request is rejected with `400`. -- Baseline loads are **additive** — existing `iter = 'baseline'` rows are not touched. Each load is its own log entry and is independently undoable. - -`replay` controls behavior when incremental rows exist (applies to Clear + reload, not individual segments): - -- `replay: false` (default) — after clearing, re-load baseline segments, leave incremental rows untouched -- `replay: true` — after clearing, re-load baseline, then re-execute each incremental log entry in chronological order - -**v1 note:** `replay: true` returns `501 Not Implemented` until the replay engine is built. - -**Clear baseline (`DELETE /api/versions/:id/baseline`)** — deletes all rows where `iter = 'baseline'` and all `operation = 'baseline'` log entries. Irreversible (no undo). Returns `{ rows_deleted, log_entries_deleted }`. - -**Reference request body:** same shape as baseline load without `replay`. Reference dates land verbatim (no offset). Additive — multiple reference loads stack independently, each undoable by logid. - -### Forecast Data - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/api/versions/:id/data` | Stream all rows for this version as an Arrow IPC binary | - -**Transport format — Apache Arrow IPC stream** - -The endpoint returns `Content-Type: application/vnd.apache.arrow.stream` (binary). JSON is not used for this route. The client fetches the response as `arrayBuffer()` and passes it directly to `worker.table(buffer)` — Perspective's native ingestion path with no JS deserialization overhead. - -Arrow's columnar layout with dictionary encoding on string dimension columns keeps payload size manageable at scale (typically 50–150 MB for 1M rows depending on string cardinality), compared to several times that for equivalent JSON. - -**Server-side streaming (cursor-based)** - -For datasets that may reach 1M+ rows, the server must not buffer the full query result in memory before writing the response. Instead: - -1. Open a PostgreSQL cursor over the `SELECT * FROM {{fc_table}}` query -2. Fetch rows in batches (target: 10 000 rows per batch) -3. For each batch, append a serialized Arrow record batch to the HTTP response using chunked transfer encoding -4. Close the cursor and end the response when all batches are written - -This means the first bytes of the Arrow stream reach the client while the server is still reading from the database, and Node.js heap stays bounded regardless of dataset size. - -**Client-side loading** - -- **Moderate datasets (< ~500k rows):** accumulate the full `arrayBuffer()` then call `worker.table(buffer)` once. Perspective becomes interactive after the stream completes. -- **Large datasets (≥ ~500k rows):** process Arrow record batches incrementally — call `worker.table(firstBatch)` to create the table, then `pspTable.update(batch)` for each subsequent batch. Perspective is interactive and browseable while remaining batches are still arriving. - -The client detects which path to use by checking the `X-Row-Count` response header (see below). - -**Row-count pre-check** - -Before opening the cursor, the server runs `SELECT COUNT(*) FROM {{fc_table}}`. The result is attached as the `X-Row-Count` response header so the client can choose its loading strategy. If the count exceeds 500 000, the UI displays a non-blocking notice ("Loading large dataset — pivot will become interactive as data arrives") rather than a blank screen. - -### Forecast Operations - -All operations share a common request envelope: - -```json -{ - "pf_user": "paul.trowbridge", - "note": "optional comment", - "tag": "reduce_spend", - "slices": [ { "channel": "WHS", "geography": "WEST" }, - { "channel": "DIR", "geography": "EAST" } ], - "apply_mode": "prorate" -} -``` - -- `slices` — one or more slices. The legacy single `slice` object is still accepted and - treated as a one-entry list. -- `apply_mode` — `prorate` (default) treats the selection as one pool; `each` runs the - operation once per slice, producing one log entry per slice so they can be undone - separately. With a single slice the two are identical. -- `tag` — optional initiative label, stored on the log entry. - -Slice keys must be `role = 'dimension'` or `role = 'date'` columns per col_meta. A slice -naming none of them is **rejected**: unknown keys are dropped when building the WHERE -clause, so such a slice would otherwise reduce to `TRUE` and apply the operation to the -entire version. - -Several slices become an `OR` of `AND`-groups, not per-column `IN` lists — flattening -`{A:1,B:1}` and `{A:2,B:2}` into `A IN (1,2) AND B IN (1,2)` would also match `A:1,B:2`. -The result is parenthesised because callers append `AND pf_iter NOT IN (...)`, and `AND` -binds tighter than `OR`. - -#### Scale -`POST /api/versions/:id/scale` - -```json -{ - "pf_user": "paul.trowbridge", - "note": "10% volume lift Q3 West", - "tag": "volume_push", - "slices": [ { "channel": "WHS", "geography": "WEST" } ], - "apply_mode": "prorate", - "target_value": 12000, - "units_pct": 10, - "target_basis": "selected" -} -``` - -Each measure is resolved **independently**, so a target on one and a percentage on the -other can be sent together. Per measure, exactly one of: - -| Field | Meaning | -|---|---| -| `target_value` / `target_units` | the total to end up with | -| `value_pct` / `units_pct` | a percentage of the current total | -| `value_incr` / `units_incr` | an absolute amount to add | -| `target_price` | target value/units ratio; holds units constant | - -The legacy global `pct: true` flag (meaning "the increments are percentages") is still -honoured. - -`target_basis` decides what a target or percentage measures against: - -- `adjustable` — only the rows the operation can write. -- `selected` (UI default) — everything the pivot shows for the slice, `exclude_iters` - rows included. Those rows cannot move, so the adjustable rows absorb the whole - difference and the pivot lands on the number you asked for. Without this, a target set - against a visible total overshoots by the excluded rows' contribution. - -Behaviour: - -- Excludes `exclude_iters` rows from the rows it writes, in every basis. -- Distributes the increment proportionally across rows in the slice. -- **Refuses to prorate a pool that nets to ~zero** — below 1% of gross. Each row's new - value is `(row / total) * increment`, so as the net approaches zero the multiplier - explodes and rows fly to extreme opposite values to reach the target. Offsetting - slices are the usual cause; `apply_mode: each` handles that correctly. -- Slices matching no rows, or already on target, are skipped and returned in - `slices_skipped` rather than silently counted as applied. -- Inserts rows tagged `iter = 'scale'` - -#### Recode -`POST /api/versions/:id/recode` - -```json -{ - "pf_user": "paul.trowbridge", - "note": "Part discontinued, replaced by new SKU", - "slices": [ { "part": "OLD-SKU-001" } ], - "set": { "part": "NEW-SKU-002" } -} -``` - -- `set` — one or more dimension fields to replace (can swap multiple at once) -- Inserts negative rows to zero out the original slice -- Inserts positive rows with replaced dimension values -- Both sets of rows share the same `logid` — undone together -- Inserts rows tagged `iter = 'recode'` - -#### Clone -`POST /api/versions/:id/clone` - -```json -{ - "pf_user": "paul.trowbridge", - "note": "New customer win, similar profile to existing", - "slices": [ { "customer": "EXISTING CO", "channel": "DIR" } ], - "set": { "customer": "NEW CO" }, - "scale": 0.75 -} -``` - -- `set` — dimension values to override on cloned rows -- `scale` — optional multiplier on value/units (default 1.0) -- Does not offset original slice -- Inserts rows tagged `iter = 'clone'` - -### Audit & Undo - -| Method | Route | Description | -|--------|-------|-------------| -| GET | `/api/versions/:id/log` | List all log entries for a version, newest first | -| DELETE | `/api/log/:logid` | Undo: delete all forecast rows with this logid, then delete log entry | -| PATCH | `/api/log/:logid` | Edit `note` and/or `tag`. Branches on whether a field was sent, so `""` clears rather than being read as "leave alone" | -| GET | `/api/versions/:id/table-info` | Physical forecast table, source table, and live row counts by `pf_iter` | -| GET | `/api/versions/:id/bridge` | Baseline → current rolled up by tag | -| GET | `/api/sources/:id/tags` | Tags used on this source with use counts, newest first — feeds tag autocomplete | - ---- - -## Frontend (Web UI) - -### Navigation (sidebar) - -Three-step collapsible sidebar (200 px expanded / 48 px collapsed, state persisted to `localStorage`): - -1. **① Setup** — browse DB tables, register sources, configure col_meta, generate SQL. One-time admin task. -2. **② Baseline** — create/manage versions, load baseline segments, timeline preview. One-time per version. -3. **③ Forecast** — main working view: Perspective pivot + operation panel. Primary ongoing use. - -### Setup View (① Setup) - -- Left panel: DB table browser — all tables with row counts; click a table to open a preview modal (column list + sample rows) -- Right panel: Registered sources list; click a source to open col_meta editor below -- Col_meta editor: inline table — role dropdown per column, is_key checkbox, label text input, ordinal position -- "Save" button — upserts col_meta; "Generate SQL" button — triggers generate-sql route, shows confirmation -- "Register source" button available in the table preview modal -- New columns default to role `dimension` on registration -- Must generate SQL before a version can be created against this source - -### Baseline View (② Baseline) - -Source and version selectors at top. Version management inline: create new version (explains that a forecast table will be created), Close / Reopen / Delete buttons. Delete drops the forecast table and removes all version records. - -### Baseline Workbench - -A dedicated view for constructing the baseline for the selected version. The baseline is built from one or more **segments** — each segment is an independent query against the source table that appends rows to `iter = 'baseline'`. Segments are additive; clearing is explicit. - -**Layout:** -``` -┌─────────────────────────────────────────────────────────────┐ -│ Baseline — [Version name] [Clear Baseline] │ -├─────────────────────────────────────────────────────────────┤ -│ Segments loaded (from log): │ -│ ┌──────┬────────────────┬──────────┬───────┬──────────┐ │ -│ │ ID │ Description │ Rows │ By │ [Undo] │ │ -│ └──────┴────────────────┴──────────┴───────┴──────────┘ │ -├─────────────────────────────────────────────────────────────┤ -│ Add Segment │ -│ │ -│ Description [_______________________________________] │ -│ │ -│ Date range [date_from] to [date_to] on [date col ▾] │ -│ Date offset [0] years [0] months │ -│ │ -│ Additional filters: │ -│ [ + Add filter ] │ -│ ┌──────────────────┬──────────┬──────────────┬───────┐ │ -│ │ Column │ Op │ Value(s) │ [ x ]│ │ -│ └──────────────────┴──────────┴──────────────┴───────┘ │ -│ │ -│ Preview: [projected month chips] │ -│ │ -│ Note [___________] [Load Segment] │ -└─────────────────────────────────────────────────────────────┘ -``` - -**Segments list** — shows all `operation = 'baseline'` log entries for this version, newest first. Each has an Undo button. Undo removes only that segment's rows (by logid), leaving other segments intact. - -**Clear Baseline** — deletes ALL `iter = 'baseline'` rows and all `operation = 'baseline'` log entries for this version. Prompts for confirmation. Used when starting over from scratch. - -**Add Segment form:** - -- **Description** — free text label stored as the log `note`, shown in the segments list -- **Date offset** — years + months spinners; shifts the primary `role = 'date'` column forward on insert -- **Filters** — one or more filter groups that define what rows to pull. Conditions within a group are AND-ed; groups are OR-ed. There is no separate "date range" section — period selection is just a filter like any other: - - Each group has a header row ("Group 1", "Group 2 — OR", …) and a `+ Add condition` link - - Within a group: Column (any `role = 'date'` or `role = 'filter'`), Operator (`=`, `!=`, `IN`, `NOT IN`, `BETWEEN`, `IS NULL`, `IS NOT NULL`), Value(s) - - Value inputs: `BETWEEN` → two date/text inputs; `IN`/`NOT IN` → comma-separated list; `=`/`!=` → single input; omitted for `IS NULL`/`IS NOT NULL` - - `+ Add OR group` button appends a new empty group below, joined by an "OR" separator label - - Groups with more than one condition render an "AND" badge between rows to make the logic explicit - - A group can be removed with `×` on its header (not available when only one group remains) - - At least one group with at least one condition is required to load a segment -- **Manual WHERE clause** (admin only) — a toggle link ("Switch to manual SQL") that replaces the filter builder with a plain textarea. The admin types a raw PostgreSQL WHERE clause body (no `WHERE` keyword). Switching back to the builder clears the textarea. When active, the filter builder is hidden and the structured `filters` field is not sent; `raw_where` is sent instead. A prominent warning banner reads: "Raw SQL is not validated. You are responsible for correctness and security." -- **Timeline preview** — rendered when any condition in any group is a `BETWEEN` or `=` on a `role = 'date'` column. Shows a horizontal bar (number-line style) for the source period and, if offset > 0, a second bar below for the projected period. Each bar shows start date on the left, end date on the right, duration in the centre. The two bars share the same visual width so the shift is immediately apparent. Not shown in manual WHERE mode or when no date condition is present. -- **Note** — optional free text -- **Load Segment** — submits; appends rows, does not clear existing baseline rows - -**Example — three-segment baseline:** - -| # | Description | Filter logic | Offset | -|---|-------------|--------------|--------| -| 1 | All orders taken 6/1/25–3/31/26 | `order_date BETWEEN 2025-06-01 AND 2026-03-31` | 0 | -| 2 | Open or unshipped orders (status missing or explicit) | `(status IN ('OPEN','PENDING')) OR (status IS NULL)` | 0 | -| 3 | Prior year book-and-ship 4/1/25–5/31/25 | `order_date BETWEEN 2025-04-01 AND 2025-05-31 AND ship_date BETWEEN 2025-04-01 AND 2025-05-31` | 0 | - -Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any combination is valid as long as at least one group with at least one condition is present. - -### Forecast View - -**Layout:** the operation panel docks **bottom** (default), **right**, or **floats** over -the pivot (drag its header to move, corner grip to resize). Position and size persist to -`localStorage`. It closes via its header ×, `Esc`, or the toolbar toggle, which shows the -selection count while shut. - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ [Layout…] [Expand 0 1 2 3] [Refresh] [Change log] [Bridge] │ -│ [Hide panel] │ -├─────────────────────────────────────────────────────────────────┤ -│ │ -│ Perspective Viewer (interactive pivot web component) │ -│ │ -├──────────────── drag to resize ─────────────────────────────────┤ -│ SLICE 2 selected │ scale recode clone │ Amount │ -│ channel=WHS │ Together | Each │ Baseline 1,000.00 │ -│ channel=DIR │ │ ▪ reduce_spend -20.00 │ -│ Clear selection │ │ ──────────────────── │ -│ │ │ Adjustable 1,070.00 │ -│ │ │ reference·fixed 921.72 │ -│ │ │ ──────────────────── │ -│ │ │ Selected total 1,991.72 │ -│ │ │ ──────────────────── │ -│ │ │ New value [ 2,000 ] │ -│ │ │ Change [ 8 ] │ -│ │ │ % change [ 0.4 ] │ -│ │ tag [reduce_spend] │ [Apply Scale] │ -└─────────────────────────────────────────────────────────────────┘ -``` - -**The ledger.** The scale form is one continuous statement rather than a totals display -plus a separate input form: baseline, each adjustment (grouped by tag), current, then the -edit. `New value`, `Change` and `% change` are three interchangeable editable rows — -typing in any one derives the other two, and whichever you typed in is what gets sent. -That replaces the old target/delta/percent mode toggle: the row you type in *is* the mode. - -Rows the pivot shows but operations cannot write (`exclude_iters`, typically `reference`) -appear as their own line with a `Selected total` beneath, and a control chooses which of -the two a target measures against — see `target_basis` above. - -**Pivot control:** [Perspective](https://github.com/perspective-dev/perspective) 5.2.0 (`@perspective-dev/*`), **bundled inline, not loaded from a CDN** — the `/inline` entrypoints embed the WASM so the version is pinned by `package-lock.json`. See `PERSPECTIVE.md`. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons. - -**Large-dataset loading sequence:** -1. Client issues `GET /api/versions/:id/data` -2. Server responds with `X-Row-Count` header and begins streaming Arrow record batches -3. If `X-Row-Count` ≥ 500 000, UI shows a non-blocking loading banner; otherwise no indicator -4. Client calls `worker.table(firstBatch)` on the first batch to make the pivot interactive immediately -5. Each subsequent batch is applied with `pspTable.update(batch)` as it arrives -6. Banner clears when the stream closes - -**Interaction flow:** -1. Click a cell or row in the pivot — the `perspective-click` event fires -2. `detail.config.filter` from the event is parsed: only `==` filters on `role = dimension` columns are extracted as the slice -3. A plain click replaces the selection; **ctrl/⌘/shift-click toggles** a slice in or out of - it. The `CustomEvent` carries no modifier flags, so they are read from the `mousedown` - that preceded it. `perspective-select` (region drag) is wired defensively alongside. -4. Slice populates the Operation Panel — pick operation tab, fill in parameters -5. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed -6. For recode, both the negative offset rows and positive replacement rows are returned and streamed - -**Selection caveat.** `pf_iter` is not a `col_meta` column, so it is stripped when a slice -is built. Two cells differing only by iter band (baseline vs reference) produce the same -effective slice; duplicates are collapsed before the request, and the panel warns when a -selection covers fewer distinct slices than cells clicked. There is currently no way to -target one band of a slice. - -**Expand depth.** Perspective's `GROUP BY ROLLUP` view contains every level of the -hierarchy, and `view.set_depth()` — which lives on the view, not in the saved config — is -the only thing hiding the deeper ones. The viewer rebuilds its view whenever it redraws, -which its Intersection/ResizeObserver triggers on tab refocus, leaving the tree fully -expanded. The last applied depth is therefore re-applied on `visibilitychange`, `focus` -and `pageshow`. - -**Pivot default layout:** built from col_meta — first two `dimension` columns as `group_by`, `date` column as `split_by`. User can rearrange in Perspective settings panel and save. - -**Reference rows** (`pf_iter = 'reference'`) are visible in the pivot for comparison context. Operations never affect them (enforced by `exclude_iters` in the version). - -### Log View - -Modal list of log entries — timestamp, operation, slice, **tag**, note, rows affected. -"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of -Perspective table). - -Tag and note are edited inline (click, Enter to save, Esc to cancel) via -`PATCH /api/log/:logid`; the tag field completes from tags already used on the source. -Saving a tag regroups the ledger and bridge immediately, so history can be reclassified -after the fact. - -### Bridge View - -A waterfall answering "how did this version get from its baseline to where it stands?", -one step per initiative tag, opened from the toolbar. - -``` - 6.0k ┤ ┌──────┐- - - -┐ - │ │+3,624│ │ - 4.0k ┤ │ │ −3,800│ - │ ┌─────┐- ┘ └ - - - ┘──┐ ┌─────┐ - 2.0k ┤ │2,734│ │+509│ │3,067│ - 0 ┴──┴─────┴────────────────┴────┴─┴─────┴── - Baseline clamp give food Current -``` - -**Scope:** the current slice selection (default when one exists), the pivot's current -filters, or the whole version. Selection scope uses the **union** of the selected slices — -the same reach an operation would have — with rows matching more than one slice deduped -by `pf_id` to match the `OR` semantics operations use. - -Computed from the Perspective table already loaded in the browser rather than from -`/api/versions/:id/bridge`, so the figures always reconcile with what the pivot is -showing. The endpoint remains for API consumers. - -**Colour** encodes polarity, not identity: increases and decreases are two poles of one -scale, so it uses a validated diverging pair (blue/red, CVD ΔE 21.6 — green/red is avoided -as the classic colourblind failure) with neutral grey anchors for baseline and current. -Every bar is directly labelled and a table view gives the same numbers at full precision. - ---- - -## Forecast SQL Patterns - -Column names baked in at generation time. Tokens substituted at request time. Metadata columns are `pf_iter`, `pf_logid`, `pf_user`, `pf_created_at`. - -**Units conditionality:** `{units_col}` appears in INSERT column lists and SELECT expressions only when a `units` role is configured in col_meta. The SQL generator omits it entirely otherwise — no placeholder column, no zero-fill. - -**dim_period JOIN:** when any `dimension` column has `dim_period_col` set (and its group's `date` key has `is_key = true`), the FROM clause becomes `{schema}.{tname} s JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`. Those dimension columns are selected as `dp.{dim_period_col} AS {col}` instead of `s.{col}`. - -### Baseline Load (one segment) - -```sql -WITH ilog AS ( - INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note) - VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}') - RETURNING id -) -,ins AS ( - INSERT INTO {{fc_table}} ( - {dimension_cols}, {date_col}, {value_col} [, {units_col}], - pf_iter, pf_logid, pf_user, pf_created_at - ) - SELECT - {dimension_cols}, - ({date_col} + '{{date_offset}}'::interval)::date, - {value_col} [, {units_col}], - 'baseline', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM - {schema}.{tname} -- or with dim_period JOIN (see above) - WHERE - {{filter_clause}} - RETURNING * -) -SELECT count(*) AS rows_affected FROM ins -``` - -Baseline loads are **additive** — no DELETE before INSERT. Each segment appends independently. - -Token details: -- `{{date_offset}}` — PostgreSQL interval string (e.g. `1 year`); defaults to `0 days`; applied only to the primary `role = 'date'` column on insert -- `{{filter_clause}}` — built from `filters` or `raw_where` at request time (not baked into stored SQL since conditions vary per segment). - - Structured path (`filters`): each group becomes a parenthesized AND block; groups are joined with `OR`. Every column is validated against col_meta (`role = 'date'` or `role = 'filter'`). Values are escaped (single quotes doubled). Supported operators: `=`, `!=`, `IN`, `NOT IN`, `BETWEEN`, `IS NULL`, `IS NOT NULL`. - - Raw path (`raw_where`): the string is injected verbatim. No col_meta validation. Admin-only. - -### Clear Baseline - -Two queries, run in a transaction: -```sql -DELETE FROM {{fc_table}} WHERE pf_iter = 'baseline'; -DELETE FROM pf.log WHERE version_id = {{version_id}} AND operation = 'baseline'; -``` - -### Reference Load - -```sql -WITH ilog AS ( - INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note) - VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}') - RETURNING id -) -,ins AS ( - INSERT INTO {{fc_table}} ( - {dimension_cols}, {date_col}, {value_col} [, {units_col}], - pf_iter, pf_logid, pf_user, pf_created_at - ) - SELECT - {dimension_cols}, {date_col}, {value_col} [, {units_col}], - 'reference', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM - {schema}.{tname} -- or with dim_period JOIN (see above) - WHERE - {{filter_clause}} - RETURNING * -) -SELECT count(*) AS rows_affected FROM ins -``` - -No date offset applied — reference rows land at their original dates for prior-period comparison. Same dim_period JOIN logic applies as baseline. - -### Scale - -```sql -WITH ilog AS ( - INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note) - VALUES ({{version_id}}, '{{pf_user}}', 'scale', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}') - RETURNING id -) -,base AS ( - SELECT - {dimension_cols}, {date_col}, - {value_col} [, {units_col}], - sum({value_col}) OVER () AS total_value - [, sum({units_col}) OVER () AS total_units] - FROM {{fc_table}} - WHERE {{where_clause}} - {{exclude_clause}} -) -,ins AS ( - INSERT INTO {{fc_table}} ( - {dimension_cols}, {date_col}, {value_col} [, {units_col}], - pf_iter, pf_logid, pf_user, pf_created_at - ) - SELECT - {dimension_cols}, {date_col}, - round(({value_col} / NULLIF(total_value, 0)) * {{value_incr}}, 2) - [, round(({units_col} / NULLIF(total_units, 0)) * {{units_incr}}, 5)], - 'scale', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM base - RETURNING * -) -SELECT * FROM ins -``` - -`{{value_incr}}` / `{{units_incr}}` are pre-computed in JS when `pct: true` (multiply slice total by pct). Units expressions are omitted when no units column is configured. - -### Recode - -```sql -WITH ilog AS ( - INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note) - VALUES ({{version_id}}, '{{pf_user}}', 'recode', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}') - RETURNING id -) -,src AS ( - SELECT {dimension_cols}, {date_col}, {value_col} [, {units_col}] - FROM {{fc_table}} - WHERE {{where_clause}} - {{exclude_clause}} -) -,neg AS ( - INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at) - SELECT {dimension_cols}, {date_col}, -{value_col} [, -{units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM src - RETURNING * -) -,ins AS ( - INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at) - SELECT {{set_clause}}, {date_col}, {value_col} [, {units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM src - RETURNING * -) -SELECT * FROM neg UNION ALL SELECT * FROM ins -``` - -`{{set_clause}}` replaces the listed dimension columns with new values, passes others through unchanged. Both the negative (zero-out) and positive (replacement) rows share the same `pf_logid` and are undone together. - -### Clone - -```sql -WITH ilog AS ( - INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note) - VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}') - RETURNING id -) -,ins AS ( - INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at) - SELECT - {{set_clause}}, {date_col}, - round({value_col} * {{scale_factor}}, 2) - [, round({units_col} * {{scale_factor}}, 5)], - 'clone', (SELECT id FROM ilog), '{{pf_user}}', now() - FROM {{fc_table}} - WHERE {{where_clause}} - {{exclude_clause}} - RETURNING * -) -SELECT * FROM ins -``` - -### Undo - -Two queries run sequentially (not in a CTE — FK ordering): - -```sql -DELETE FROM {{fc_table}} WHERE pf_logid = {{logid}}; -DELETE FROM pf.log WHERE id = {{logid}}; -``` - ---- - -## Display-grain pre-aggregation - -**Status:** built (static grain). This is **Path B** (pre-aggregated extract → -native Perspective table) of two candidate designs; rationale, the Path A -alternative (live virtual-server aggregation), the spike evidence, and the -A-vs-B trade-off live in `pf_perspective_options.md` (§Two candidate designs, -§Spike findings). - -The grain is **static** — set once per source in Setup and baked into the stored -`pf.sql` templates, so load and operations agree by construction. `in_grain` -means *eligible for the grain*, and in this version the grain is exactly the set -of eligible columns. Deriving a narrower grain per pivot at request time (the -dynamic variant) is then additive: intersect the viewer's field set with the -eligible set. Leaving high-cardinality columns (`part`, raw day dates, -currency-level detail) unflagged is what keeps the grain from exploding back -toward raw, regardless of what a user drags into the pivot. - -**Measured on `pf.fc_osm_stack_20`** at `pending_rep × customer × smon`: -534,902 → **6,154** rows (≈87×), `pf_gkey` unique across all 6,154, and both -measures reconcile exactly to the raw totals (283,296,087.67 / 962,142,261.46). - -**Problem it solves.** The current transport ships every raw forecast row to the -browser (≈535k rows / ~250 MB / ~2 min on `osm_stack`). Perspective then pivots -them in WASM. Pre-aggregating server-side to the grain the pivot actually -displays collapses the payload dramatically — measured 534,902 → **4,642 rows** -at `rep × customer × month` (≈115×) — while keeping Perspective's **native** -engine, so expand/collapse/depth/sort/filter all keep working (unlike the DuckDB -virtual-server path, which loses them). - -### Grain definition — `pf.col_meta.in_grain` - -Add a boolean `in_grain` to `pf.col_meta`. At **Generate SQL** time the flagged -columns (plus `pf_iter`, always) define the display grain; `value`/`units` -columns are the additive measures summed to that grain. The grain is baked into -the stored `pf.sql` templates, so initial load and operations agree on it. - -- A date column enters the grain at **period** resolution (via `dim_period_col`, - e.g. month `sdat`), not raw date. -- **Constraint — additive measures only.** Sums (sales, qty, units) are exact. - Averages must ship as `sum` + `count` and be derived; `count(DISTINCT …)` - cannot be pre-aggregated. Forecast measures are sums, so this holds. -- Dimensions **not** in the grain are unavailable for pivot/filter on that view. - Either include every dimension users pivot on (cardinality permitting — a - high-cardinality dim like `part` explodes the grain back toward raw), or - re-fetch at a new grain when the user changes `group_by`/`split_by`/`filter` - (each change = one ~25 ms aggregate returning a small native dataset). - -### Initial load — `GET /api/versions/:id/agg` - -Replaces the raw `/data` stream for grain-based versions. Aggregates the forecast -table to the stored grain and returns Arrow IPC. The template is stored in -`pf.sql` as operation `get_agg`, generated only when a grain is defined; clearing -the grain and regenerating removes it, and the client falls back to `/data`. -Both endpoints speak the same protocol (one record batch plus an `X-Row-Count` -header), so the client only chooses the URL: - -```sql -SELECT - {{grain_cols}} - ,pf_iter - ,pf_logid - ,{{grain_key}} AS pf_gkey -- synthetic index, see below - ,SUM({{value_col}}) AS {{value_col}} - ,SUM({{units_col}}) AS {{units_col}} -FROM {{fc_table}} -GROUP BY {{grain_cols}}, pf_iter, pf_logid -``` - -Client loads the result into a native `worker.table(buffer, { index: 'pf_gkey' })`. -The Perspective **view** sums measures across these rows for whatever the user -pivots on — exactly as an Excel pivot cache sums the data tab. - -**Model — append deltas, let the view sum (the Excel pivot-cache pattern).** This -is a direct port of the prior Excel workflow: pre-aggregated rows go to the table, -the pivot cuts them locally, and each adjustment is *appended* — never a -recomputed total. So rows are aggregated to grain **per `pf_logid`** (one row per -grain × `pf_iter` × originating log entry), and the view sums them. This is also -the smallest change from today's code, which already appends operation results via -`table.update()` and lets the view sum — we just feed pre-aggregated rows. - -**Synthetic key.** Perspective's index is a single column, so emit -`pf_gkey = {{grain_cols}} || pf_iter || pf_logid` (concatenated). Including -`pf_logid` keeps each operation's contribution a **distinct** row, so appends -accumulate (rather than replacing a bucket) and a delete can remove exactly that -operation's rows. - -As built, the concatenation is -`concat_ws(chr(31), COALESCE(col::text, chr(30)), …, pf_iter, pf_logid::text)`. -The separator and NULL sentinel matter: plain `concat_ws` skips NULLs, so -`('a', NULL)` and `(NULL, 'a')` would produce the same key and silently merge two -groups into one indexed row. `chr(30)` also keeps NULL distinct from `''`. - -### Write path (scale / recode / clone) — append the new log entry's rows - -Operations INSERT raw rows into `{{fc_table}}` under a new `pf_logid` as today; the -final CTE returns just **that log entry's rows aggregated to grain**: - -```sql -WITH -ins AS ( - INSERT INTO {{fc_table}} ( … ) SELECT … RETURNING {{grain_cols}}, pf_iter, pf_logid -) -SELECT - {{grain_cols}} - ,pf_iter - ,pf_logid - ,{{grain_key}} AS pf_gkey - ,SUM({{value_col}}) AS {{value_col}} - ,SUM({{units_col}}) AS {{units_col}} -FROM ins -GROUP BY {{grain_cols}}, pf_iter, pf_logid -``` - -Client applies `table.update(rows)` — these are new `pf_gkey`s, so they append and -the view re-sums. No bucket-total recomputation. (Aggregating `FROM ins` is safe -because each row carries the new, unique `pf_logid`.) - -### Delete path (undo) — remove that log entry's rows - -A logid's rows are uniquely keyed, so undo just removes them and lets the view -re-sum — no re-aggregation, no emptied-bucket handling, no snapshot caveat: - -`RETURNING` does not accept `DISTINCT`, so the delete feeds a CTE that reduces its -output to the distinct grain keys. `rows_deleted` still counts raw rows removed: - -```sql -WITH -del AS ( - DELETE FROM {{fc_table}} - WHERE pf_logid = {{logid}} - RETURNING {{grain_cols}}, pf_iter, pf_logid -) -SELECT - count(*)::int AS rows_deleted - ,array_agg(DISTINCT {{grain_key}}) AS pf_gkeys -FROM del; -DELETE FROM pf.log WHERE id = {{logid}}; -``` - -Client applies `table.remove(pf_gkeys)`. (A wholesale re-pull of `/agg` is an even -simpler fallback and is now cheap — ~25 ms.) - -### What this preserves / changes - -- **Preserves:** native Perspective interactivity, layout persistence, and the - slice-click → operation flow (a clicked grain cell still maps to a `WHERE` on - raw rows). pg raw rows remain the single source of truth. -- **Changes:** operation responses return grain-aggregated rows for the new - `pf_logid` (not raw `RETURNING *`); `/data` is superseded by `/agg` for grain - versions; the table is indexed by `pf_gkey` instead of `pf_id`; undo becomes a - targeted `table.remove()` and the "full reload" wart disappears. -- **Lineage:** this is a port of the prior Excel model — pre-agg rows → pivot - cache → double-click slice → VBA adjustment UI → append incremental rows → - refresh cache. The append-and-sum write/undo model mirrors that directly. - ---- - -## Admin Setup Flow (end-to-end) - -1. Open **Sources** view → browse DB tables → register source table -2. Open col_meta editor → assign roles to columns (`dimension`, `value`, `units`, `date`, `filter`, `ignore`), mark is_key dimensions, set labels -3. Click **Generate SQL** → app writes operation SQL to `pf.sql` -4. Open **Versions** view → create a named version (sets `exclude_iters`, creates forecast table) -5. Open **Baseline Workbench** → build the baseline from one or more segments: - - Each segment specifies a date range (on any date/filter column), date offset, and optional additional filter conditions - - Add segments until the baseline is complete; each is independently undoable - - Use "Clear Baseline" to start over if needed -6. Optionally load **Reference** → pick prior period date range → inserts `iter = 'reference'` rows at their original dates (for comparison in the pivot) -7. Open **Forecast** view → share with users - -## User Forecast Flow (end-to-end) - -1. Open **Forecast** view → select version -2. Pivot loads — explore data, identify slice to adjust -3. Select cells → Operation Panel populates with slice -4. Choose operation → fill in parameters → Submit -5. Grid refreshes — adjustment visible immediately -6. Repeat as needed -7. Admin closes version when forecasting is complete - ---- - -## Open Questions / Future Scope - -- **Display-grain pre-aggregation** — ship the pivot at its display grain instead of raw rows; see §Display-grain pre-aggregation above and `pf_perspective_options.md`. This is the chosen direction for the load-time + interactivity problem. -- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501 -- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred) -- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred) -- **Export** — download forecast as CSV or push results to a reporting table. The bridge's table view is a partial stand-in for reading the numbers out, but there is no download. -- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION). The bridge answers the within-version form of this question; across versions is still open. -- **Bridge drill-down** — click a step to list the adjustments behind it, or select that slice back in the pivot -- **Targeting one iter band** — make `pf_iter` part of a slice so an operation can act on, say, only the baseline rows of a selection (see Known issues) -- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually. -- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema. - ---- - -## Project Status — 2026-09-11 - -### What's working -- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo -- `units` column is optional — sources without a units column register and generate SQL correctly -- `dim_group` / `dim_period_col` on col_meta: baseline/reference load JOINs `pf.dim_period` to derive fiscal/calendar period columns rather than copying them raw from the source -- `pf.dim_period` calendar table (2018–2035): populated by `setup_sql/gen_dim_period.sql`, configurable fiscal year start -- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express -- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM -- 3-step collapsible sidebar (Setup / Baseline / Forecast) -- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation -- Baseline view: version management, multi-segment baseline workbench, canvas timeline, filter builder -- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage -- Incremental row streaming: operation results (`RETURNING *`) applied via `pspTable.update()` — no full reload -- **Multi-slice operations**: ctrl/⌘-click accumulates slices; `apply_mode` prorate/each -- **Per-measure resolution**: target, percent or change amount independently per measure -- **`target_basis`**: a target measures against the adjustable rows or everything the pivot shows -- **Ledger panel**: baseline → adjustments → current → three interchangeable editable rows, docked bottom/right/floating -- **Tags and bridge**: initiative tags on log entries, editable after the fact, with a waterfall view scoped to selection / filters / version -- **Status bar** names the physical table writes land in, with live row counts by iter - -### Known issues / next focus - -- **`pf_iter` not selectable** — it is not a col_meta column, so it is stripped from slices. Cells differing only by iter band collapse to one slice (duplicates are detected and collapsed, and the panel warns), and there is no way to operate on one band of a slice. -- **Per-row rounding drift** — the scale SQL rounds each row to 2dp, so a target of 1,000 across many rows can land on 999.99. Inherent to proportional distribution; a correction row would be needed to land exactly. -- **Manual caret expansion is not restored** — the depth re-apply on refocus only covers whole-tree depths set via the Expand buttons or a saved layout, since per-row expansion lives in the same discarded view. -- **Bridge has no drill-down** — clicking a step does not list its adjustments or select that slice back in the pivot. -- **Light surface only** — app chrome is light throughout; the dark toggle currently re-themes only the Perspective viewer. -- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version. -- **No migration sequence** — `01_schema.sql` carries `ADD COLUMN IF NOT EXISTS` inline for the `tag` column, which covers fresh installs and re-runs, but there is no ordered migration mechanism. -- **No tests** — SQL generation is token substitution against append-only tables and is entirely untested. - -### Fixed - -- **Non-selective slices applied to the whole version** — a slice naming no filterable column reduced to `TRUE`. Now rejected on all three operations. -- **Proration across a near-zero pool** — rows flew to extreme opposite values to reach a target. Refused when the net is below 1% of gross; `apply_mode: each` is the alternative. -- **Targets overshooting by excluded rows** — see `target_basis`. diff --git a/server.js b/server.js index aced83b..1e746e3 100644 --- a/server.js +++ b/server.js @@ -7,7 +7,7 @@ const { Pool, types } = require('pg'); const { requireAuth } = require('./lib/auth'); // Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings, -// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary. +// so apache-arrow encodes them as Int/Float64 rather than Dictionary. types.setTypeParser(20, v => v === null ? null : Number(v)); types.setTypeParser(1700, v => v === null ? null : Number(v)); diff --git a/ui/README.md b/ui/README.md deleted file mode 100644 index a36934d..0000000 --- a/ui/README.md +++ /dev/null @@ -1,16 +0,0 @@ -# React + Vite - -This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules. - -Currently, two official plugins are available: - -- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs) -- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/) - -## React Compiler - -The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation). - -## Expanding the ESLint configuration - -If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project. diff --git a/ui/src/views/Forecast.jsx b/ui/src/views/Forecast.jsx index f3735f3..be1e353 100644 --- a/ui/src/views/Forecast.jsx +++ b/ui/src/views/Forecast.jsx @@ -317,10 +317,9 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio // Mirrors ViewConfig.group_by_depth for the toolbar's active state. The // config is the source of truth; this only exists so the buttons can render. const [expandDepth, setExpandDepth] = useState(null) - // The column axis has no set_depth() — collapsing it means restoring a shorter - // split_by, so the full hierarchy has to be remembered separately to expand again. + // The column hierarchy as the live config has it, for the Columns stepper. + // Collapsing is split_by_depth now, so split_by itself is never truncated. const splitFullRef = useRef([]) - // a collapse from the user rearranging split_by themselves const initIdRef = useRef(0) const modifierRef = useRef(false) // the datagrid plugin element, for reading cell coordinates and driving its diff --git a/ui/vendor/README.md b/ui/vendor/README.md index 587b3fc..5acbedf 100644 --- a/ui/vendor/README.md +++ b/ui/vendor/README.md @@ -142,15 +142,23 @@ ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided The config field counts **levels to show**; `view.set_depth()` counts the boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and -`Forecast.jsx` sends `d + 1`. +`t_traversal::set_depth` adds the 1 back, so `n` renders `n` levels. +`Forecast.jsx` therefore sends the stepper's level count unchanged, floored at +1 — `0` would reach the engine as `set_depth(-1)` on an unsigned depth, +underflow, and clamp to fully expanded. (It used to send `d + 1` to dodge that, +which was off by one against the grid everywhere.) -To apply: +The patch is already committed on the fork's build branch (see +`PROVENANCE.txt`), so rebuilding needs only the script. The `.patch` file here +is the change on its own, for review or for carrying to a clean upstream +checkout: ```bash -cd $PSP_DIR # default ~/perspective -git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch -./ui/vendor/rebuild-perspective.sh -cd ui && npm install +./ui/vendor/rebuild-perspective.sh # from the fork branch +# then re-resolve by path, as above -- a plain `npm install` keeps the old code + +# only on a fresh upstream checkout: +cd $PSP_DIR && git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch ``` ### Host tools this tree needs