pt/pf_app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: pt:master
Branches Tags
pt:master
pt:feature/observer-shim
pt:feature/perspective-column-collapse
pt:fix/stale-source-selection
pt:spike/column-branch-collapse
pt:feature/column-collapse
pt:feature/migrations
pt:feature/static-grain
pt:spike/duckdb-virtual-server
..
compare: pt:feature/migrations
Branches Tags
pt:master
pt:feature/observer-shim
pt:feature/perspective-column-collapse
pt:fix/stale-source-selection
pt:spike/column-branch-collapse
pt:feature/column-collapse
pt:feature/migrations
pt:feature/static-grain
pt:spike/duckdb-virtual-server

1 Commits
master ... feature/mi

Author SHA1 Message Date
Paul Trowbridge
d1197df7d5 Replace idempotent schema script with tracked forward-only migrations 
setup_sql/01_schema.sql was an idempotent bootstrap script — CREATE TABLE
IF NOT EXISTS plus a tail of ALTER ... ADD COLUMN IF NOT EXISTS. It had no
record of what any given database had applied, which is how a branch could
declare col_meta.in_grain while the running database lacked it, with
nothing able to detect the mismatch. The symptom would have been a
confusing 'column "in_grain" does not exist' inside an unrelated request.

Migrations-only, no hand-maintained current-state file to drift:

- setup_sql/migrations/*.sql applied in filename order, recorded in
  pf.schema_version with a checksum. Split along the schema's actual
  evolution, so each column is declared exactly once — 01_schema.sql had
  grown to declare dim_group, dim_period_col and in_grain twice each.
- lib/migrations.js holds the bookkeeping, shared by the CLI and the boot
  check. scripts/migrate.js provides up | status | baseline.
- server.js refuses to start when the database is behind, listing what is
  pending. This converts silent drift into a clear boot message, which was
  the whole point. PF_SKIP_MIGRATION_CHECK=1 bypasses.
- Four integrity guards, each verified to fire: a migration modified after
  being applied, one recorded as applied but missing from disk, one that
  would apply out of order, and a re-run when already current.
- No IF NOT EXISTS on new migrations. The bookkeeping already guarantees
  one run each, and the guards hide ordering mistakes — that is exactly why
  01_schema.sql had ALTERs sitting above the CREATE TABLE they depended on,
  broken for anyone installing from scratch. 0004 keeps the guard only
  because it was applied by hand before migrations existed.

Verified: replaying all four migrations into a throwaway schema reproduces
the live pf schema exactly, 41 columns, column for column.

schema.generated.sql is a pg_dump snapshot for reading, refreshed by
npm run schema:dump. It excludes the runtime fc_* tables and strips
pg_dump's random \restrict token and version banner, so regenerating an
unchanged schema yields an identical file rather than a spurious diff.

pf.dim_period stays out of migrations — it is a parameterised data load
(fiscal year start month), not a schema change.

The dev database (ubm) has been baselined at all four migrations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-19 09:07:39 -04:00
56 changed files with 1805 additions and 9283 deletions
Show Stats Expand all files Collapse all files

15
.env.example
View File

@ -4,18 +4,3 @@ DB_NAME=your_database
DB_USER=your_user
DB_PASSWORD=your_password
PORT=3010
# Signs the session cookie. Generate with:
# node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))'
# or let ./pf.sh config do it. Changing it signs everyone out.
SESSION_SECRET=
# Send the session cookie over HTTPS only. Keep true behind a TLS proxy;
# set false only to reach the app over plain HTTP on a trusted network.
COOKIE_SECURE=true
# Reverse proxy hops express should trust for req.ip / protocol. Default 1.
#TRUST_PROXY=1
# Only needed if the UI is served from a different origin than the API.
#CORS_ORIGIN=https://forecast.example.com

1
.gitignore vendored
View File

@ -1,4 +1,3 @@
node_modules/
.env
.env.*
public/app/

498
CLAUDE.md
View File

@ -14,44 +14,40 @@ Data transport architecture options: `pf_perspective_options.md`
- **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
- **Schema:** forward-only SQL migrations in `setup_sql/migrations/`, applied by `npm run migrate` and tracked in `pf.schema_version`. `server.js` refuses to start when the database is behind. See `setup_sql/README.md`
---
## Project layout
```
server.js Express entry point; pg pool; session; type parsers for bigint/numeric
server.js Express entry point; pg pool; type parsers for bigint/numeric
routes/
auth.js POST /api/login, /api/logout, GET /api/me; login throttle
tables.js GET /api/tables, /api/tables/:schema/:tname/preview
sources.js Source registration, col_meta, SQL generation
versions.js Version CRUD, baseline/reference load, data stream
operations.js scale, recode, clone, undo — the core forecast ops
log.js GET /api/versions/:id/log, DELETE /api/log/:logid
layouts.js Named pivot layouts — list per version, create, patch, delete
lib/
sql_generator.js buildFilterClause, token substitution helpers
auth.js scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
utils.js
setup_sql/
01_schema.sql pf schema DDL — run once to install
02_auth.sql pf.app_user + pf.session
README.md migration workflow — read before changing the schema
migrations/ ordered .sql, applied once, tracked in pf.schema_version
gen_dim_period.sql parameterised calendar load (not a migration)
schema.generated.sql pg_dump reference snapshot; generated, never edited
scripts/
migrate.js migration CLI (up | status | baseline)
schema-dump.js regenerates schema.generated.sql
ui/src/
auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login
views/
Login.jsx Sign-in form
Setup.jsx DB browser, source registration, col_meta editor
Baseline.jsx Version management, baseline workbench, reference load
Forecast.jsx Perspective pivot, selection handling, operation dispatch
components/
DepthButtons.jsx One axis's collapse control; both axes are drawn from it
LayoutMenu.jsx The Layout ▾ control — Published / Mine, with the write actions
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms
BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
Forecast.jsx Perspective pivot + operation panel (Scale/Recode/Clone)
Sidebar.jsx 3-step collapsible nav
StatusBar.jsx Source · version · write target · row counts · theme
StatusBar.jsx Source · version · row count · status
Timeline.jsx Date-range preview bar for baseline segments
```
@ -60,42 +56,13 @@ ui/src/
## Database schema (`pf`)
- **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below); `is_key` is described under §`is_key` below
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below)
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it.
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.layout`** — named Perspective view configs; see §Pivot layouts
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`
- **`pf.session`** — express-session store (connect-pg-simple layout)
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
### `is_key`
Read in four places, and **not** the one it sounds like — slices are validated
against `filterCols`, which is every `dimension` plus every `date` column
regardless of `is_key`.
1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes
`members.find(c => c.is_key)`, the column every other member is keyed on for
`pf.dim_member`
2. **Value completion** — only `is_key` columns get a dropdown, and
`/sources/:id/values/:col` refuses anything else
3. **Sibling autofill** — fires on blur only when `is_key && dim_group`
4. **The `dim_period` anchor** — `role === 'date' && is_key && dim_group` picks the
date whose siblings are derived from the calendar
Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group.
**When a group has more than one, `.find()` silently takes the lowest `opos`.**
That is not hypothetical: `segment_new` (opos 12) outranked `part` (opos 15) in
the `part` group, so a refresh keyed on a column that is null throughout, matched
nothing, and reported success with zero members. `customer`'s group has four keys
and picks the right one only by `opos` luck.
The two meanings want separating — a per-group key choice, or a rule that the
group key is the column named by the group (which these groups nearly follow
already, except `sdate` → `sdate_e`). Until then, a refresh that finds two keys
should refuse rather than guess.
- **`pf.schema_version`** — applied migrations (filename + checksum); owned by the migration runner, never edited by hand
### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
@ -117,288 +84,17 @@ Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WA
### Display grain
Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings.
### Segment and note columns
`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table
does not itself carry:
- **`pf_segment`** — `pf.log.label`, else `tag`, else `note`, else `Unlabeled`;
`'99 - Adjustments'` for an adjustment that has no label of its own
- **`pf_note`** — the free text on a scale/recode/clone; null on loads
They are deliberately separate: commingling a segment name with an adjustment note
makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join
adds no rows. The operation routes stamp the same two fields onto the rows they push
back incrementally, since those come from `RETURNING *` and would otherwise arrive
without them.
### Column order is stored text
Perspective orders column groups by the value string, and `SortDir`'s `col asc` /
`col desc` only reverses that — so Prior Year → Plan → Actual → Forecast is
alphabetical in neither direction and expressible as neither. A `"01 - "` prefix
is the only lever, and it lives in **`pf.log.label`** (and `pf.log.bucket`),
typed by whoever names the segment. Nothing derives it.
`SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are the
single definition, shared with the `/data` cursor in `routes/operations.js` —
`/agg` is generated, `/data` is not, and the two have to agree.
The one hardcoded ordinal is `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`, which
keeps unlabelled adjustments after every *numbered* segment. That proviso is the
whole scheme, not a caveat on it: ordering is string ordering, so `99` only lands
last once the loads carry `01`–`0n`, and an unnumbered segment sorts after it
(digits precede letters — `9` is `0x39`, `A` is `0x41`). The old `'(adjustment)'`
sorted *first* for the same reason read the other way, `(` being `0x28`.
Unlabelled loads read plain `Unlabeled` and so land at the very end, which is
where a segment nobody has named belongs. Labelling an adjustment's own log row
overrides the fallback, which is how one kind of adjustment is split out from the
rest.
### Hardcoded display names
Every name the pivot can show that does not come from `pf.log`. If a segment or
bucket appears under a name nobody typed, it is one of these. All three are in
the `DISPLAY DEFAULTS` block at the top of `lib/sql_generator.js`, exported so
the `/data` cursor and the operation routes' incremental row stamps use the same
values the generated `/agg` does.
| constant | `pf.version` column | built-in | applies to |
|---|---|---|---|
| `ADJUSTMENT_SEGMENT` | `adjustment_segment` | `99 - Adjustments` | `pf_segment` for a scale/recode/clone with no `label` |
| `ADJUSTMENT_BUCKET` | `adjustment_bucket` | `04 - Forecast` | `pf_bucket` for a scale/recode/clone with no `bucket` |
| `UNLABELED_LOAD` | `unlabeled_load` | `Unlabeled` | `pf_segment` and `pf_bucket` for a load with no `label`, `tag` or `note` |
Each is set per scenario on the Baseline page, under **Fallback names**; blank
falls back to the built-in. Anything typed on the log row overrides both, so
none of these appears once a segment is named.
**Why the join rather than a token.** `pf.sql` is keyed on
`(source_id, operation)` — one template shared by every version of a source — so
a value baked in at Generate SQL time could not vary by version, and
regenerating for one version would silently change the others. The names are
therefore read through `VERSION_JOIN` at query time, which also means changing
one takes effect on the next load with nothing regenerated.
The built-ins are still a convention guess: `ADJUSTMENT_BUCKET`'s `04 - ` only
suits one numbering. A version that numbers its buckets differently sets its
own rather than inheriting that.
**What this replaced.** The prefix used to be computed client-side, as
Perspective expression columns (`pf_bucket_ord`, `pf_segment_ord`) built from
`pf.log.seq` and `pf.version.bucket_order`. It ordered the pivot and nothing
else, so every other reader disagreed with it; `restore()` replaces
`expressions` wholesale, so it had to be re-applied after every layout load; and
ExprTK's string scanner tests each *byte* with `isprint()`, so a label
containing anything outside printable ASCII could not be ordered at all (`·` is
two bytes, of which `isprint(0xC2)` is false). `DEAD_ORDER_EXPRS` in
`Forecast.jsx` strips the expression names out of layouts saved under that
scheme. `pf.log.seq` and `pf.version.bucket_order` are no longer read; the
columns remain.
Relabelling now needs a page reload to show, because the label is part of the
aggregated row rather than something the pivot can re-derive. Editing a label
afterwards is a `PATCH /api/log/:logid` and needs nothing regenerated, but a
source registered before this change needs **Generate SQL** run once, so its
stored load templates write `label` and `bucket` onto the log row at all.
### Forecast operations
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums.
### Price or volume (`plug`)
A sales figure alone does not say which of price or volume moved, so scale takes
`plug` — `'price'` (default, volume holds) or `'volume'` (price holds, units scale
in proportion). Resolved in `resolveIncrs()` in `routes/operations.js`; the panel
only offers it when the edit is dollars-only, because naming units or price has
already answered it. The semantics come from the predecessor Excel model,
`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`:
```
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
```
A `target_price` with a `target_units` alongside is that form's Edit Price mode,
where both are inputs and dollars fall out. The ledger's **Result** line previews
value, units and price together using the same rules, so what you see is what
gets written.
### Undo
`DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload.
---
## Axis depth (collapse / expand)
Both axes collapse the same way: a **depth in `ViewConfig`**, set through
`restore()`.
- **Rows** — `group_by_depth`, driven by the `Rows` buttons via `applyDepth()`
- **Columns** — `split_by_depth`, driven by the `Columns` buttons via
`applySplitDepth()`
Both sets are drawn by `components/DepthButtons.jsx` from the axis's own field
list (`groupFull` / `splitFull`), so each button is named after the level it
reveals. The row axis used to be a fixed `0 1 2 3`, which named nothing and
offered levels a two-field `group_by` does not have.
Both are **1-based**: they count the levels to show, where the imperative
`view.set_depth()` counts the boundary below them. `server.cpp` does
`ctx1->set_depth(row_pivot_depth - 1)` and
`ctx2->set_depth(HEADER_COLUMN, column_pivot_depth - 1)`, so the toolbar sends
`d + 1` and subtracts one again when reading a layout back.
Because a depth lives in the config, it survives every view rebuild, rides into
the persisted and named layouts through `viewer.save()`, and needs nothing
re-applied afterwards.
**This required patching the engine.** `ViewConfig::apply_update` applied ten
fields and neither depth, so a depth sent through `restore()` was accepted,
deserialized and dropped — and since `Session::update_view_config` returns early
when `apply_update` reports no change, no view was rebuilt at all. See
`ui/vendor/0001-apply-depth-fields-on-config-update.patch`; the vendored build
carries it.
### What this replaced
Worth knowing, because a lot of machinery existed to work around it and is now
gone:
- Row depth used to be imperative — `getView()` then `view.set_depth()` — which
put it on an object the viewer discards whenever it re-renders. Restoring it
meant guessing when that had happened: an `observerShim.js` patching
`window.IntersectionObserver` and `window.ResizeObserver`, focus and
visibility listeners, a retry loop for `getView()` throwing `No table set`
while `getTable()` resolved, and a flag tracking whether the viewer had "gone
away". That guesswork produced three distinct visible faults — the tree fully
expanding on refocus, snapping on any reflow, and snapping when Perspective's
settings sidebar opened.
- Column collapse used to restore a **truncated `split_by`**. The discarded
levels therefore had to be remembered separately (`splitFull`, persisted as
`split_full`), our own collapse had to be told apart from the user rearranging
the pivot (`collapsingRef` plus a prefix test), and the selection was cleared
on every collapse because the axis was changing shape.
`split_full` is still *read* on load, for layouts saved under the old scheme.
### Auto-pause is off
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on
itself plus the document's `visibilitychange` drive `AutoPauseState::apply()`,
and pausing **deletes the view** (`session.set_pause(true)` →
`view_sub.take().delete()`). Returning to the tab is therefore not a redraw but
`restore_and_render()` — a fresh view and a fresh traversal of the whole grain,
which on a large one is a multi-second chug on every tab switch, and takes any
per-node expansion with it.
`initViewer()` calls `viewer.setAutoPause(false)` right after `viewer.load()`.
Nothing updates the table while the tab is hidden — every operation is driven
from this page — so the pause bought nothing and cost a rebuild. The view is now
held while the tab is backgrounded.
### Still not solved: per-node expansion
Expanding one specific branch is view state with no config representation, and
the API has **no getter**:
```
expand(row_index: number): Promise<number>
collapse(row_index: number): Promise<number>
```
It can be set but not read, so it cannot be captured and replayed — this is why
it is lost on every rebuild, and why no client-side fix has worked. The honest
route is a `ViewConfig` field carrying expanded row *paths* (indices shift as
the tree opens), applied in `server.cpp` where the depths are. Bigger than the
depth patch: it needs a way to enumerate expanded nodes in the C++ traversal, a
proto field, and the apply step.
The fork's own `header_click.ts` / `expand_column` / `collapse_column` is the
column-axis equivalent and has the same limitation.
**Limitation that remains either way:** depth is whole-axis. Excel can collapse
2025 while 2026 stays expanded; a depth collapses every group at that level
together. `columns` selects which *measures* appear, not individual split
combinations.
## Pivot layouts
A layout is a named `ViewConfig`, stored in **`pf.layout`** and owned by an
account. Two kinds, one table:
- **published** — everyone on the forecast lists it and can apply it; only its
owner or an admin may change it
- **private** — yours, nobody else lists it
Scope is the **version** by default, because that is where people enter the app.
`version_id IS NULL` means the layout applies to every version of the source;
that is where the old source default went, and what a brand-new version picks up
before anyone has published anything for it. `is_default` (at most one per scope,
by partial unique index) is what `initViewer()` restores on a first load — a
version-scoped default beating a source-wide one, the narrower answer winning.
**Permissions are the `pf.log` rule verbatim** — your own, or an admin's
override, and the UI greys out the rest rather than offering a click that answers
403. `can_edit` rides on every row so the menu knows which. Applying is never
restricted: the guarantee is that a published layout cannot be *changed* out from
under people, not that it cannot be adapted — Save is withheld on a layout that
isn't yours, Save as… forks it into your own.
**No territory clause.** A layout is display config, and territory restricts rows,
not columns; `cleanLayout()` already drops anything the live schema lacks.
**What is still local.** `LAYOUT_KEY` (`pf_layout_v{vid}`) — the unnamed
last-used config — stays in `localStorage`, because it is per-browser session
continuity rather than a thing anyone names or shares. `LAYOUTS_KEY`
(`pf_layouts_v{vid}`) is the old named list; `loadLayouts()` lifts it into
`pf.layout` as private rows once per version and then clears the key.
**The dirty dot is a comparison, not a flag.** `restore()` itself fires
`perspective-config-update`, so anything set unconditionally in that handler
would light up the moment a layout was applied. `activeConfigRef` holds what the
pivot last matched and `sameConfig()` compares against it, ignoring `table`
(the per-load table name, different on every refresh).
**What this replaced.** Named layouts lived only in `localStorage` — invisible to
anyone else, gone on another machine. The one server-side layout was
`pf.source.default_layout`, a single anonymous blob that `PUT
/sources/:id/default-layout` let *any* account overwrite for *every* account: a
published layout with no owner. That route is gone; the column remains, migrated
and read by nothing.
`cleanLayout()` guards `aggregates` along with the axes — including the weight
column of the multi-arg form — and drops the offending *entry* rather than the
layout, since `restore()` is all-or-nothing.
---
## The Forecast toolbar is two zones
Left, on its own tinted ground: **view** — `Layout ▾`, the `Rows` and `Columns`
depth buttons, `Fit widths`. Nothing there reaches the server or changes the
forecast; it is all `ViewConfig` and `plugin_config` against rows already loaded.
Right: **forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. These
re-run the load, open something over the pivot, or write to the version.
They were one undifferentiated row separated by identical hairlines, which made
`Fit` and `Bridge` read as peers when one is a column width and the other is the
version's history. The split is the point; the tint is just what makes it
visible.
---
## Slice mechanics
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).
Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow** — `{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.
Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".
**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.
`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` columns are kept as the slice. This slice populates the operation panel and is sent as the `slice` object in all operation POST bodies.
**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.
@ -408,18 +104,6 @@ Turning a region back into slices re-derives, per cell, the same filters Perspec
All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.
**Every date column travels, not just the first.** `dataCols` is
`[...dims, ...dateCols, value, units]`. It used to be `dateCols[0]`, so on a
source with order, requested and ship dates, scale and recode wrote rows whose
`rdate` and `sdate_e` were **null** — and because the derived period dimensions
(`rseas`, `smon_e`) are dimensions and were copied with the rest, the labels
read correctly while the dates under them were gone. Both columns are
`role = 'date'` and therefore sliceable, so an adjustment scoped to a ship date
matched no adjustment row that had ever been written. Clone and the loads were
always right. The wider GROUP BY costs almost nothing on `osm_skinny`
(971,185 → 971,352 groups) because `ordnum`/`ordline` already make a row
near-unique.
- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched
@ -428,97 +112,6 @@ near-unique.
---
## Authentication
Everything under `/api` except the auth routes sits behind a session; the React
app is only mounted once there is one (`Gate` in `main.jsx`), because its load
effects call the API immediately.
- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
the password is read on stdin and hashed before it reaches psql.
- **Sessions:** `express-session` + `connect-pg-simple` in `pf.session`, so a
restart doesn't sign everyone out and a session can be revoked by deleting its
row (`disable-user` does exactly that). Cookie `pf.sid`: httpOnly, SameSite=Lax,
Secure unless `COOKIE_SECURE=false`, 12h rolling.
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
`TRUST_PROXY` (default 1) makes `req.ip` and secure-cookie detection correct
behind the TLS proxy. `CORS_ORIGIN` is the only way CORS is enabled at all; a
wildcard origin plus a session cookie would be cross-site request forgery by
construction.
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
regenerates the session id on success.
**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
`sessionUser(req)`, never from the request body — the UI used to send a hardcoded
`pf_user: 'admin'`, which any client could have set to anything. The audit log
now names the account that made the change.
## Source columns come from pg_catalog
`information_schema.columns` omits **materialized views** — they are not in the
SQL standard — and `gs.osm_skinny` is one. So the source the whole app is built
on looked like it had no columns: registering it seeded nothing, and creating a
version failed with "No usable columns in col_meta" while col_meta plainly held
thirty-six.
`RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version
creation, source registration and the table preview. It returns the same shape
information_schema did, so `mapType` and the callers were unchanged:
`data_type` is `format_type` with the modifier stripped, which gives the same
spelling (`character varying`, `numeric`), and precision and scale are unpacked
from `atttypmod`. The table browser lists from `pg_class` by `relkind` for the
same reason.
## Territory scoping
An account sees and changes only its own territory. The list lives on
`pf.app_user.territory` (jsonb array) with `is_admin` for the accounts that see
everything, and `col_meta.is_territory` marks which column of a given source
the values belong to — one per source, flagged rather than named in code so a
second source can be divided by something other than a sales rep.
**Fail closed.** No territory and not an admin means no rows.
`buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an
unflagged source: an account somebody forgot to configure sees nothing instead
of the whole book.
**Built from the session, never the request.** This is the difference between
it and `scope`, which the browser sends and which is right to send, being a
filter the user chose. A permission cannot come from the thing it restrains, so
the territory predicate is ANDed on last, in `sliceUnits()` for writes and per
route for reads, where nothing in the payload can remove it.
Enforced at:
- `/data` — clause on the cursor *and* on the count behind `X-Row-Count`
- `/agg` — a `{{territory_clause}}` token applied **before** the GROUP BY, since
the territory column need not be part of the grain and may not survive it
- every operation, through `sliceUnits()`
- `/sources/:id/values/:col` — completion reads the *source* table, which no
scope has touched, so without it a dropdown enumerates the whole business
- `DELETE /log/:logid` and `PATCH /log/:logid` — by owner, not territory. Undo
removes an entry's rows wholesale, and half-undoing one would leave a state
nothing describes. The PATCH looks like a private annotation and is not:
`label` and `bucket` name the pivot's columns for everyone in the version, so
unguarded it let any account rename the company's segments. Your own entries,
or an admin's override, and the UI greys out the rest rather than offering a
click that answers 403.
- recode's `set` — a scoped account cannot set the territory column at all.
Moving a row between territories is reassignment, not forecasting, and it
would vanish from the view that would have shown what happened.
**The change log shows an entry's full impact**, not the reader's share. The
totals stamped on `pf.log` are company-wide, so an admin's version-wide scale
reads the same in every account — deliberate, and labelled, rather than
re-aggregating per territory.
Managed with `./pf.sh set-territory | set-admin | orphan-territory`.
`orphan-territory` lists values present in the data that no account owns; work
under one is invisible to everyone but an admin, which a typo causes easily and
nothing inside the app reveals.
## Light / dark mode
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
@ -530,66 +123,15 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
## After a change: restart, and Generate SQL
Two steps, and which one you need depends on what changed.
- **Restart the server** for anything in `routes/`, `lib/` or `server.js`.
- **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes.
The templates are *stored* in `pf.sql`, so editing the generator changes
nothing until they are rebuilt — and a template carrying a token the running
code does not substitute fails at the database rather than in JS, which reads
as an unrelated client-side error.
The order matters: restart first, then Generate SQL, or the old code writes the
templates.
Schema changes are applied to the live database directly and mirrored into
`setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a
migration runner, so running it is not how an existing database gets a new
column.
## Known issues / active work
- **Rows written before 2026-09-21 have null `rdate`/`sdate_e`.** The generator
fix above changes what gets written from here on; it does not repair what is
there. On `fc_osm_skinny_29` that is ~270k scale and recode rows. A backfill
would have to re-derive each from the baseline rows at the same coordinate,
which is possible but is inference, not recovery — deleting and replaying
those log entries is the honest repair
- **Zero-row operations report success.** Scale refuses with "Nothing to
scale…" when its slice matches nothing; recode and clone commit an empty log
entry and return `rows_affected: 0`. A recode of a rep whose rows are all
`reference` looked like it worked and did nothing
- **The change log does not show an entry's id**, so there is no way to name
one when asking about it
- **Territory is read onto the session at login**, so granting or changing one
does not reach a signed-in account until it signs in again. Re-reading it per
request in `requireAuth` would also make disabling someone immediate
- **Depth buttons rebuild the view.** A depth lives in `ViewConfig`, so changing
it goes through `restore()` and `Session::update_view_config` tears down and
rebuilds the view — a full traversal — where a manual collapse mutates the
existing one in place. Noticeable on a large grain. The fix is to detect a
depth-only change and call `view.set_depth()` imperatively while still writing
it to the config, at the cost of the two being able to drift
- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29`
(2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in
pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then
~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The
remaining lever is **dynamic grain** — group by the fields the current pivot
actually uses rather than every `in_grain` column; see `pf_spec.md` →
§Display-grain pre-aggregation, "the dynamic variant"
- Per-node expand/collapse is lost whenever the view rebuilds — set-only API, no
getter; see §Axis depth
- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
- Load progress bar is jittery — needs throttle (~10 updates/sec)
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
- Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`,
`App.jsx`). It is re-validated against the live list whenever that list changes, so a
deregistered source or deleted version re-points at the first remaining one instead of
leaving a dead id that 404s every call
- Source/version selection doesn't persist across page reload
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
- Grain drift: changing `in_grain` after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. `routes/log.js` derives the grain from live col_meta, so a grain changed mid-session yields `pf_gkeys` that don't match the loaded table and undo silently removes nothing
- Migrations are forward-only — there are no down migrations. Rolling back a schema change means writing a new migration that reverses it
- Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see `pf_spec.md` → §Display-grain pre-aggregation
## Deferred (not in v1)

201
PERSPECTIVE.md
View File

@ -47,27 +47,8 @@ import '@perspective-dev/viewer/themes'
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
one-line version bumps) but in production it means: app breaks if the CDN is
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
pull executable WASM from a third party on every load.
> **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
> resolves its server WASM with
> `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
> which from `.../client@4.4.0/dist/cdn/` resolves to
> `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**.
> jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
> pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
> `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
> Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
>
> The security framing matters as much as the outage: an unversioned URL means your
> users execute whatever that package publishes next, automatically, unreviewed.
>
> 5.2.0 fixes it by carrying the client's version across
> (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
> the WASM is embedded in the bundle and there is no runtime fetch to hijack
> (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).
pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
pull executable WASM from a third party on every load. (pf_app currently does this in
`ui/src/views/Forecast.jsx` — migrating off it is the main open item.)
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
`<link>` in `index.html` — so it's bundled and versioned too.
@ -79,9 +60,7 @@ The version choice is constrained by two hard facts about the `@perspective-dev`
**(verified against installed metadata, 2026-06)**:
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even
though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
Treemap / Heatmap / etc.) live only in this package.
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
@ -92,47 +71,21 @@ So you can have at most **two** of these three:
|---|---|
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) |
| d3fc chart plugins | **4.4.1** viewer-d3fc |
There is **no** version where all three hold. Pick by what the app needs:
> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts
>
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1`
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.**
>
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
> running exactly that pair), *every chart type other than Datagrid fails.*
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
> `viewer` throws `get_static_config is not a function` — once per chart plugin. The
> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
> load against a coherent 4.4.1 viewer produces no such error.
>
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
>
> Consequence: dataflow currently has the worst of both worlds. It carries the
> mixed-version complexity *specifically* to keep charts, and does not have charts.
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
>
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
> real bundled build. It is the documented-and-untested assumption this whole policy
> rests on — establish it before betting a version choice on it.
- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
`^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
working Datagrid and no charts. If you are on it today, you are choosing inline
bundling, not charts; be explicit about which one you actually want.
- **Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
`^4.4.1` viewer-d3fc`. This is a deliberate, necessary mixed-version pair, *not* an
accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate
bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the
build breaks (`"./inline" is not exported`).
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
**4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give
up `/inline` bundling.
**4.4.1 exact**. Charts work; you give up `/inline` bundling.
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or
later (then a fully-coherent inline-capable suite becomes possible — and only then does
"both" come back on the table).
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x (then a
fully-coherent inline-capable 4.5.x suite becomes possible).
---
@ -169,74 +122,6 @@ to client-side heuristics — acceptable only at small scale.
---
## 3a. Expression columns and aggregation order (ratio correctness)
**Expression columns are row-level.** Perspective evaluates every expression against
each *raw row* first, then feeds the result into the column's aggregate. It has no
post-aggregate expression stage. So a ratio written the obvious way:
```js
expressions: { price: '"revenue" / "qty"' } // default aggregate: sum
```
...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.
This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
grouping just makes it visible by putting several wrong numbers side by side. No
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
`high`/`low`/`median`/`dominant` are all wrong for the same reason.
### The fix: a weighted-mean aggregate
Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:
```js
expressions: { price: '"revenue" / "qty"' },
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
```
**The nested array matters.** The type is
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
reads like "no such aggregate" and is easy to misread as the feature being absent.
Available since **4.4.0** — no version bump needed, and because the aggregate is
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
`table.update()` on a live view re-derives the weighted mean from the merged rows
without a reload).
### Rules of thumb
- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
→ leave the default `sum`. Those are unaffected by any of this.
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
`['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
- Mixing both in one view is fine and was verified.
### Type inference can silently break expressions
`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
as `integer` — which happens to numeric-looking strings when the table is created from
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
error. Give period/year columns an explicit `string` type at table creation.
### Not fixed by any of this: column-axis expand/collapse
`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
(§2: `viewer-d3fc` still caps at 4.4.1).
---
## 4. Theming
- One toggle drives both app CSS and the viewer:
@ -257,42 +142,6 @@ it would retire the `if(...)`-expression workaround. It costs the d3fc charts, t
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
the reference implementation; a stale layout referencing a dropped column otherwise
throws on restore.
- **`aggregates` needs the same guard.** pf_app's `cleanLayout()` has it as of 2026-09;
**dataflow's does not** and should adopt it. Filtering
`columns`/`group_by`/`split_by`/`sort`/`filter` and stopping there is harmless only
while `viewer.save()` emits `aggregates: {}`, which it does until someone sets an
aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
aggregate entry references a *target* column and, in the multi-arg form, a *weight*
column; both need validating, and the entry is what gets dropped, never the layout.
Add this guard as part of adopting §3a, not after.
### Auto-pause deletes the view — turn it off for a static pivot
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on itself
(scrolled out of the viewport, `display: none`) combined with the document's
`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint
optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the
**view object is destroyed**. Becoming visible again calls
`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full
traversal, every time.
For a viewer streaming live updates nobody is watching, that is the right trade.
For a pivot over a large static table it is the wrong one — on pf_app's
`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it
silently discards per-node expand/collapse (§"Not fixed by any of this"), which
has no config representation and so cannot be restored.
```js
await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
```
Guard the call: it is a method on the custom element and absent on older builds.
Leave auto-pause **on** where the table is fed by a live stream the user does not
need to have kept up with while away. This is long-standing viewer behaviour, not
a 5.x regression — worth knowing before blaming a rebuild on your own code.
---
@ -319,13 +168,6 @@ Run this whenever bumping **any** Perspective package or `apache-arrow`:
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
commit the lockfile.
- **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
`latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
and the WASM fails to link:
`Import #8 "env" "psp_opfs_load": function import requires a callable`.
Five packages, not four. This is invisible while pf_app loads from the CDN, and
will bite on the "move off CDN" open item below.
3. `vite build` — no unresolved imports.
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
@ -341,22 +183,17 @@ Run this whenever bumping **any** Perspective package or `apache-arrow`:
| | pf_app | dataflow | Target |
|---|---|---|---|
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** |
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** |
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
| Loader | CDN (runtime) | npm `/inline` | **npm `/inline`** |
| Version | 4.4.0 (CDN URLs) | 4.5.1 viewer/client + 4.4.1 d3fc | depends on loader (§2) |
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test |
| `apache-arrow` | `^21.1.0` (client built vs 17) | n/a | pin exact, match WASM |
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
**Open items:**
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages
pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host
blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction):
its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
render before choosing the latter, because nobody has confirmed they do.
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).
- pf_app → move off CDN. Note this forces the §2 choice: going npm-`/inline` means
4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and
load via the `.` entry instead of `/inline`. Either way, pin + commit the lockfile, and
add deploy automation (systemd + nginx + `deploy.sh`).

21
install.sh
View File

@ -26,12 +26,6 @@ echo ""
read -p "App port [3030]: " PORT
PORT=${PORT:-3030}
# Session cookies are signed with this; the server refuses to start without it.
SESSION_SECRET=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')
read -p "Send session cookie over HTTPS only? [Y/n]: " SECURE_ANS
case "${SECURE_ANS:-y}" in [Nn]*) COOKIE_SECURE=false ;; *) COOKIE_SECURE=true ;; esac
# ── Write .env ────────────────────────────────────────────────
cat > .env <<EOF
DB_HOST=${DB_HOST}
@ -40,10 +34,7 @@ DB_NAME=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
PORT=${PORT}
SESSION_SECRET=${SESSION_SECRET}
COOKIE_SECURE=${COOKIE_SECURE}
EOF
chmod 600 .env
echo "✓ .env written"
# ── npm install ───────────────────────────────────────────────
@ -60,23 +51,15 @@ PGPASSWORD=${DB_PASSWORD} psql \
-p "${DB_PORT}" \
-U "${DB_USER}" \
-d "${DB_NAME}" \
-v ON_ERROR_STOP=1 \
-f setup_sql/01_schema.sql \
-f setup_sql/02_auth.sql
-f setup_sql/01_schema.sql
echo "✓ schema installed"
# ── first account ─────────────────────────────────────────────
echo ""
echo "The app is behind a login. Create the first account now:"
./pf.sh add-user
# ── done ─────────────────────────────────────────────────────
echo ""
echo "========================================"
echo " Install complete"
echo " Start with: npm run dev
More accounts: ./pf.sh add-user"
echo " Start with: npm run dev"
echo " Open: http://$(hostname -I | awk '{print $1}'):${PORT}"
echo "========================================"
echo ""

92
lib/auth.js
View File

@ -1,92 +0,0 @@
// Password hashing and the route guard.
//
// Hashes are scrypt, from node's own crypto — no native build step, and the
// stored form carries its own parameters so they can be raised later without
// invalidating existing rows:
//
// scrypt$<N>$<r>$<p>$<salt base64>$<derived key base64>
const crypto = require('crypto');
const SCRYPT = { N: 16384, r: 8, p: 1, keylen: 64 };
function hashPassword(password, params = SCRYPT) {
const { N, r, p, keylen } = params;
const salt = crypto.randomBytes(16);
const dk = crypto.scryptSync(password, salt, keylen, { N, r, p, maxmem: 256 * 1024 * 1024 });
return `scrypt$${N}$${r}$${p}$${salt.toString('base64')}$${dk.toString('base64')}`;
}
// Constant-time compare. Returns false rather than throwing on a malformed or
// legacy hash, so one bad row can't 500 the login route.
function verifyPassword(password, stored) {
if (typeof stored !== 'string') return false;
const parts = stored.split('$');
if (parts.length !== 6 || parts[0] !== 'scrypt') return false;
const [, N, r, p, saltB64, dkB64] = parts;
try {
const salt = Buffer.from(saltB64, 'base64');
const expected = Buffer.from(dkB64, 'base64');
const actual = crypto.scryptSync(password, salt, expected.length, {
N: Number(N), r: Number(r), p: Number(p), maxmem: 256 * 1024 * 1024,
});
return crypto.timingSafeEqual(actual, expected);
} catch {
return false;
}
}
// Every /api route except the auth ones sits behind this.
function requireAuth(req, res, next) {
if (req.session?.user?.username) return next();
res.status(401).json({ error: 'Not authenticated' });
}
// The identity used for pf.log.pf_user and the created_by/closed_by columns.
// Read from the session only — never from the request body, which the browser
// controls and which used to carry a hardcoded 'admin'.
function sessionUser(req) {
return req.session?.user?.username || null;
}
// What this account may see and change, read from the session for the same
// reason the username is: the browser must not be able to widen it.
//
// Returns { admin: true } for an account that sees everything, or
// { admin: false, values: [...] } for a scoped one. An empty list is a real
// answer meaning "nothing", not a missing one meaning "everything" -- an
// account created without a territory sees no rows until it is granted some.
function sessionTerritory(req) {
const u = req.session?.user;
if (!u) return { admin: false, values: [] };
if (u.is_admin) return { admin: true, values: null };
return { admin: false, values: Array.isArray(u.territory) ? u.territory : [] };
}
function requireAdmin(req, res, next) {
if (req.session?.user?.is_admin) return next();
res.status(403).json({ error: 'Administrator access required' });
}
module.exports = {
hashPassword, verifyPassword, requireAuth, requireAdmin,
sessionUser, sessionTerritory, SCRYPT,
};
// CLI: `node lib/auth.js hash` reads a password on stdin and prints its hash,
// so ./pf.sh can create users without the plaintext touching argv or psql.
if (require.main === module) {
if (process.argv[2] !== 'hash') {
console.error('usage: node lib/auth.js hash (password on stdin)');
process.exit(2);
}
let input = '';
process.stdin.setEncoding('utf8');
process.stdin.on('data', chunk => { input += chunk; });
process.stdin.on('end', () => {
const password = input.replace(/\r?\n$/, '');
if (!password) { console.error('empty password'); process.exit(2); }
process.stdout.write(hashPassword(password) + '\n');
});
}

158
lib/migrations.js Normal file
View File

@ -0,0 +1,158 @@
// Migration bookkeeping shared by the runner (scripts/migrate.js) and the
// startup check in server.js.
//
// Migrations are plain .sql files in setup_sql/migrations, applied in filename
// order and recorded in pf.schema_version. They are immutable once applied: the
// runner stores a checksum and refuses to proceed if a file has changed, since
// editing an applied migration means databases silently disagree about what the
// schema is.
//
// pf.dim_period is deliberately not a migration — it is a parameterised data
// load (fiscal year start month), so it stays in setup_sql/gen_dim_period.sql.
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const MIGRATIONS_DIR = path.join(__dirname, '..', 'setup_sql', 'migrations');
// Bookkeeping table has to exist before the first migration can be recorded, so
// the runner creates it rather than a migration doing so.
const BOOKKEEPING_DDL = `
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE IF NOT EXISTS pf.schema_version (
filename text PRIMARY KEY,
checksum text NOT NULL,
applied_at timestamptz NOT NULL DEFAULT now(),
applied_by text
);
`;
function checksum(sql) {
return crypto.createHash('sha256').update(sql).digest('hex').slice(0, 16);
}
// every .sql file on disk, in apply order
function readMigrations() {
if (!fs.existsSync(MIGRATIONS_DIR)) return [];
return fs.readdirSync(MIGRATIONS_DIR)
.filter(f => f.endsWith('.sql'))
.sort()
.map(filename => {
const sql = fs.readFileSync(path.join(MIGRATIONS_DIR, filename), 'utf8');
return { filename, sql, checksum: checksum(sql) };
});
}
async function ensureBookkeeping(client) {
await client.query(BOOKKEEPING_DDL);
}
async function readApplied(client) {
const { rows } = await client.query(
`SELECT filename, checksum, applied_at FROM pf.schema_version ORDER BY filename`
);
return rows;
}
// Compare disk against the database. Returns pending migrations plus any
// integrity problems worth refusing to run on.
async function status(client) {
await ensureBookkeeping(client);
const onDisk = readMigrations();
const applied = await readApplied(client);
const appliedBy = new Map(applied.map(r => [r.filename, r]));
const diskBy = new Map(onDisk.map(m => [m.filename, m]));
const pending = onDisk.filter(m => !appliedBy.has(m.filename));
// an applied file whose contents changed — the schema is now undefined
const modified = onDisk
.filter(m => appliedBy.has(m.filename) && appliedBy.get(m.filename).checksum !== m.checksum)
.map(m => m.filename);
// recorded as applied but no longer on disk — someone deleted or renamed it
const missing = applied.filter(r => !diskBy.has(r.filename)).map(r => r.filename);
// a pending migration ordered before an applied one would apply out of sequence
const lastApplied = applied.length ? applied[applied.length - 1].filename : null;
const outOfOrder = lastApplied ? pending.filter(m => m.filename < lastApplied).map(m => m.filename) : [];
return { onDisk, applied, pending, modified, missing, outOfOrder };
}
// Apply pending migrations, each in its own transaction so a failure leaves
// earlier ones committed and the failing one fully rolled back.
async function migrate(client, { user, log = console.log } = {}) {
const st = await status(client);
if (st.modified.length) {
throw new Error(
`Applied migrations have been modified: ${st.modified.join(', ')}. ` +
`Migrations are immutable — revert the edit and add a new migration instead.`
);
}
if (st.missing.length) {
throw new Error(
`Migrations recorded as applied are missing from disk: ${st.missing.join(', ')}.`
);
}
if (st.outOfOrder.length) {
throw new Error(
`Migrations would apply out of order: ${st.outOfOrder.join(', ')} sort before ` +
`already-applied migrations. Renumber them after the latest applied migration.`
);
}
if (!st.pending.length) {
log('Schema is up to date — no migrations to apply.');
return [];
}
const done = [];
for (const m of st.pending) {
log(`applying ${m.filename} …`);
try {
await client.query('BEGIN');
await client.query(m.sql);
await client.query(
`INSERT INTO pf.schema_version (filename, checksum, applied_by) VALUES ($1, $2, $3)`,
[m.filename, m.checksum, user || null]
);
await client.query('COMMIT');
done.push(m.filename);
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
throw new Error(`${m.filename} failed: ${err.message}`);
}
}
log(`Applied ${done.length} migration(s).`);
return done;
}
// Record migrations as applied without running them — for a database whose schema
// already matches, from before migrations existed.
async function baseline(client, { user, upTo, log = console.log } = {}) {
const st = await status(client);
const target = upTo
? st.pending.filter(m => m.filename <= upTo)
: st.pending;
if (!target.length) {
log('Nothing to baseline — no pending migrations.');
return [];
}
await client.query('BEGIN');
try {
for (const m of target) {
await client.query(
`INSERT INTO pf.schema_version (filename, checksum, applied_by)
VALUES ($1, $2, $3)
ON CONFLICT (filename) DO NOTHING`,
[m.filename, m.checksum, user ? `${user} (baseline)` : 'baseline']
);
}
await client.query('COMMIT');
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
throw err;
}
log(`Marked ${target.length} migration(s) as applied without running them.`);
return target.map(m => m.filename);
}
module.exports = { MIGRATIONS_DIR, readMigrations, status, migrate, baseline, checksum };

492
lib/sql_generator.js
View File

@ -8,100 +8,9 @@
// Tokens baked in at generation time: column names, source schema.table
// Tokens substituted at request time: {{fc_table}}, {{where_clause}}, {{exclude_clause}},
// {{version_id}}, {{logid}}, {{pf_user}}, {{note}},
// {{label}}, {{bucket}}, {{tag}}, {{territory_clause}},
// {{params}}, {{slice}}, {{date_from}}, {{date_to}},
// {{value_incr}}, {{units_incr}}, {{set_clause}}, {{scale_factor}}
// What the pivot shows for a row's segment and its bucket.
//
// The ordering prefix is part of the stored text, not computed here. Perspective
// orders column groups by the value string, so "01 - Actual" is the only way an
// arbitrary order can be expressed -- and l.label is where a person types it.
// Nothing derives it, which is deliberate: an earlier design built the prefix from
// a separate seq column, as Perspective expressions on the client, and the prefix
// then existed only inside the pivot -- so every other reader disagreed with it,
// and a label that could not be expressed in ExprTK's printable-ASCII-per-byte
// string scanner could not be ordered at all. Stored text has neither problem.
//
// The single exception is the adjustment fallback, whose 99 keeps unlabelled
// adjustments last. Labelling an adjustment's log row overrides it, which is how
// one kind of adjustment is split out from the rest -- l.label rather than tag or
// note, so a segment name stays separable from adjustment commentary (pf_note).
//
// ---------------------------------------------------------------------------
// DISPLAY DEFAULTS -- every hardcoded name the pivot can show.
//
// These are the values a row falls back to when nobody has named it. They are
// the complete list: if a segment or bucket appears in the pivot under a name
// that is not in pf.log, it came from here. CLAUDE.md has the same list under
// "Hardcoded display names".
//
// They live on pf.version -- adjustment_segment, adjustment_bucket,
// unlabeled_load -- and the constants below are only the fallback for a version
// that has not set one. Read through a join at query time rather than
// substituted at generation: pf.sql templates are keyed on (source_id,
// operation) and shared by every version of a source, so a value baked in could
// not vary by version and regenerating for one would change the others.
//
// Exported because /data builds its own statement in routes/operations.js while
// /agg is generated here, and the two have to agree.
// ---------------------------------------------------------------------------
// An adjustment with no label of its own. The 99 keeps it after every numbered
// segment -- ordering is string ordering, so this only works while the loads
// carry 01-0n. Labelling an adjustment's log row overrides it, which is how one
// kind of adjustment is split out from the rest.
const ADJUSTMENT_SEGMENT = '99 - Adjustments';
// What an adjustment counts toward. Prefixed to match the segments it adjusts:
// unprefixed it read 'Forecast' while the loads read '04 - Forecast', and the
// column split in two -- the adjustments sitting apart from the rows they
// adjust. The number is a guess at the convention in use, which is the clearest
// argument for making this per-version.
const ADJUSTMENT_BUCKET = '04 - Forecast';
// A load nobody named. No prefix, so it sorts after everything numbered --
// letters follow digits in ASCII. The old '(unlabeled load)' sorted *first*,
// since '(' is 0x28 and digits begin at 0x30.
const UNLABELED_LOAD = 'Unlabeled';
const LOAD_SEGMENT = `COALESCE(NULLIF(l.label, ''), NULLIF(l.tag, ''), NULLIF(l.note, ''),
NULLIF(v.unlabeled_load, ''), '${UNLABELED_LOAD}')`;
const SEGMENT_EXPR = `CASE WHEN l.operation IN ('baseline','reference')
THEN ${LOAD_SEGMENT}
ELSE COALESCE(NULLIF(l.label, ''), NULLIF(v.adjustment_segment, ''), '${ADJUSTMENT_SEGMENT}')
END`;
// What the row counts towards. A load falls back to its own name until it is
// bucketed; an adjustment falls back to the forecast bucket, because that is
// what an adjustment is -- exclude_iters keeps operations off the reference
// segments, so there is no adjustment that is not part of the forecast.
const BUCKET_EXPR = `COALESCE(NULLIF(l.bucket, ''),
CASE WHEN l.operation IN ('baseline','reference')
THEN ${LOAD_SEGMENT}
ELSE COALESCE(NULLIF(v.adjustment_bucket, ''), '${ADJUSTMENT_BUCKET}')
END)`;
const NOTE_EXPR = `CASE WHEN l.operation IN ('baseline','reference')
THEN NULL
ELSE COALESCE(NULLIF(l.tag, ''), NULLIF(l.note, ''))
END`;
// Every pf.log column the two expressions above read, for /agg's GROUP BY: they
// are functionally dependent on pf_logid, which is in the grain, but Postgres
// will not infer that.
const LABEL_GROUP_COLS = ['l.operation', 'l.label', 'l.tag', 'l.note', 'l.bucket',
'v.adjustment_segment', 'v.adjustment_bucket', 'v.unlabeled_load'];
// The version carries the fallback names, so every statement that reads the
// expressions above needs it in scope as `v`. LEFT, not inner: a forecast row
// whose log entry somehow has no version should still come back, named by the
// constants.
const VERSION_JOIN = `
LEFT JOIN pf.version v
ON v.id = l.version_id`;
// wrap a column name in double quotes for safe use in SQL
function q(name) { return `"${name}"`; }
@ -125,149 +34,50 @@ function grainOf(colMeta) {
// parts and chr(30) stands in for NULL, so ('a', NULL) cannot collide with
// (NULL, 'a') and a NULL stays distinct from an empty string — a collision
// would silently merge two groups into one indexed row.
//
// md5 of that, rather than the concatenation itself, because the key is an
// opaque handle -- nothing reads it but table.update() and table.remove().
// The raw form averaged 233 chars on a 24-column grain and, being unique per
// row, defeated Arrow's dictionary encoding: 65.6 MB of a 109 MB payload,
// more than every other column combined. 128 bits keeps collisions unreachable.
const key = (pfx = '') => `md5(concat_ws(chr(31), ${[
const key = (pfx = '') => `concat_ws(chr(31), ${[
...cols.map(c => `COALESCE(${pfx}${q(c)}::text, chr(30))`),
`${pfx}pf_iter`,
`${pfx}pf_logid::text`
].join(', ')}))`;
].join(', ')})`;
const groupCols = (pfx = '') => [...cols.map(c => `${pfx}${q(c)}`), `${pfx}pf_iter`, `${pfx}pf_logid`];
return { cols, key, groupCols };
}
// Date columns that anchor a dim_group, each with the dimensions derived from
// pf.dim_period for it. Shared so the generator and the routes agree on both the
// membership and the join aliases -- the same reason grainOf is a single function.
function dateGroupsOf(colMeta) {
return colMeta
.filter(c => c.role === 'date' && c.is_key && c.dim_group)
.map((keyCol, i) => ({
alias: `dp${i + 1}`,
dateCol: keyCol.cname,
group: keyCol.dim_group,
derived: colMeta
.filter(c => c.role === 'dimension'
&& c.dim_group === keyCol.dim_group
&& c.dim_period_col)
.map(c => ({ cname: c.cname, periodCol: c.dim_period_col })),
}))
.filter(g => g.derived.length > 0);
}
// cname -> which join it comes from and which of its columns
function dimPeriodMapOf(dateGroups) {
return new Map(
dateGroups.flatMap(g => g.derived.map(d => [d.cname, { alias: g.alias, periodCol: d.periodCol }]))
);
}
// The joins themselves, against a date that has already had {{date_offset}}
// applied. LEFT because a null date -- an order not yet shipped has no ship date
// -- must leave the period columns empty rather than drop the row.
function dimPeriodJoins(dateGroups, alias = 's') {
return dateGroups.map(g =>
`\n LEFT JOIN pf.dim_period ${g.alias}`
+ ` ON ${g.alias}.drange @> (${alias}."${g.dateCol}" + '{{date_offset}}'::interval)::date`
).join('');
}
function generateSQL(source, colMeta) {
const dims = colMeta
.filter(c => c.role === 'dimension')
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
.map(c => c.cname);
// Every column of each measure/date role, in col_meta order. Loads carry all of
// them; the adjustment operations are single-measure (scale distributes one
// {{value_incr}}) and use only the first of each, below.
const byRole = role => colMeta
.filter(c => c.role === role)
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
.map(c => c.cname);
const valueCols = byRole('value');
const unitsCols = byRole('units');
const dateCols = byRole('date');
const valueCol = valueCols[0];
const unitsCol = unitsCols[0];
const valueCol = colMeta.find(c => c.role === 'value')?.cname;
const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
const dateCol = colMeta.find(c => c.role === 'date')?.cname;
if (!valueCol) throw new Error('No value column defined in col_meta');
if (!dateCols.length) throw new Error('No date column defined in col_meta');
if (!dateCol) throw new Error('No date column defined in col_meta');
if (dims.length === 0) throw new Error('No dimension columns defined in col_meta');
const srcTable = `"${source.schema}"."${source.tname}"`;
// Every date column, not just the primary one. Scale and recode used to
// carry dateCols[0] alone, so a source with order, requested and ship dates
// wrote adjustment rows whose rdate and sdate_e were *null* -- not stale,
// absent. Those columns are role 'date' and therefore sliceable, so a later
// adjustment scoped to a ship date matched no adjustment row ever written,
// and any reader that dated the forecast by ship date lost every
// adjustment silently. The derived period dimensions (rseas, smon_e) were
// fine throughout, being dimensions and copied with the rest -- which is
// what made it invisible: the labels were right and the dates underneath
// them were gone.
const dataCols = [...dims, ...dateCols, valueCol, unitsCol].filter(Boolean);
const dataCols = [...dims, dateCol, valueCol, unitsCol].filter(Boolean);
const effectiveValue = dataCols.includes(valueCol) ? valueCol : null;
const effectiveUnits = dataCols.includes(unitsCol) ? unitsCol : null;
const insertCols = [...dataCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
const selectData = dataCols.map(q).join(', ');
const dimsJoined = dims.map(q).join(', ');
// The date columns as a select list, in the same order dataCols lists them,
// so it lines up with insertCols positionally.
const datesJoined = dateCols.map(q).join(', ');
// Baseline and reference copy the source row wholesale, so they carry every
// measure and every date — not just the primary one the operations act on.
// Dropping the others would leave those columns null for the life of the version.
// Clone carries every date column, not just the primary one, because it is the
// operation that moves rows through time: {{date_offset}} shifts them all
// together, and the period dimensions are re-derived from pf.dim_period against
// the shifted dates rather than copied from the row being cloned. Cloning last
// year's mix forward a year otherwise produces rows dated 2027 still labelled
// with 2026's periods.
const cloneCols = [...dims, ...dateCols, effectiveValue, effectiveUnits].filter(Boolean);
const cloneInsertCols = [...cloneCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
// An adjustment writes one row per *coordinate* it touches, not one per row
// it reads.
//
// Reading rows one-for-one meant every operation inherited the row count of
// everything before it: the baseline's rows plus every prior adjustment's
// rows at the same coordinate, so the table grew super-linearly with how
// much work had been done on it. Eight Pull Forward entries and the next
// scale over the same slice writes nine times what it needs to.
//
// Collapsing is over pf_logid and pf_iter only -- every stored dimension and
// date stays in the GROUP BY -- so no column goes null and nothing becomes
// unsliceable later. The distribution maths is untouched either way, since
// the window sums see the same totals whether or not the rows underneath
// them have been added up first.
const groupCols = (cols) => cols.map(q).join(',\n ');
const loadCols = [...dims, ...dateCols, ...valueCols, ...unitsCols];
const loadInsertCols = [...loadCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
const dateColSet = new Set(dateCols);
// dim_period JOIN support: a date column that is the is_key of a dim_group
// anchors that group, and dimension siblings with dim_period_col set are
// derived from pf.dim_period instead of copied raw. Derivation is against the
// date *after* {{date_offset}}, which is the whole point -- shift a baseline
// forward a year and its period columns follow, rather than still naming the
// year it came from.
//
// Every such group, not just the first. This used to be a find(), so a source
// with order, requested and ship date groups derived the order one and copied
// the other two raw -- shifted dates against unshifted period labels.
const dateGroups = dateGroupsOf(colMeta);
const dimPeriodMap = dimPeriodMapOf(dateGroups);
// dim_period JOIN support: if the date column is the is_key of a dim_group,
// dimension siblings with dim_period_col set are derived from pf.dim_period
// instead of being copied raw from the source on baseline/reference load.
const dateKeyGroup = colMeta.find(c => c.role === 'date' && c.is_key && c.dim_group)?.dim_group;
const dimPeriodMap = new Map(
dateKeyGroup
? colMeta
.filter(c => c.role === 'dimension' && c.dim_group === dateKeyGroup && c.dim_period_col)
.map(c => [c.cname, c.dim_period_col])
: []
);
const hasDimPeriod = dimPeriodMap.size > 0;
// display grain — when set, initial load and operations both return rows
@ -307,23 +117,12 @@ function generateSQL(source, colMeta) {
// table indexed on pf_gkey and its view sums across these rows, exactly as an
// Excel pivot cache sums its data tab.
function buildGetAgg() {
// pf_logid is part of the grain, so joining pf.log adds no rows — each group
// already belongs to exactly one log entry. Without this the segment labels
// that /data surfaces would vanish the moment a source declares a grain.
return `
SELECT
${grainSelect('t.')}
,${SEGMENT_EXPR} AS pf_segment
,${BUCKET_EXPR} AS pf_bucket
,${NOTE_EXPR} AS pf_note
,l.operation AS pf_op
FROM {{fc_table}} t
LEFT JOIN pf.log l
ON l.id = t.pf_logid${VERSION_JOIN}
WHERE {{territory_clause}}
${grainSelect()}
FROM {{fc_table}}
GROUP BY
${grain.groupCols('t.').join('\n ,')}
,${LABEL_GROUP_COLS.join('\n ,')}`.trim();
${grain.groupCols().join('\n ,')}`.trim();
}
// grain columns + pf_gkey + summed measures, in the leading-comma style the
@ -352,33 +151,29 @@ GROUP BY
function buildLoadSelect(pfx) {
// pfx: table alias prefix ('s.' when joining dim_period, '' otherwise)
// The offset shifts every date column, so order date and ship date stay in step.
return loadCols.map(c => {
if (dateColSet.has(c)) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
if (dimPeriodMap.has(c)) {
const { alias, periodCol } = dimPeriodMap.get(c);
return `${alias}.${q(periodCol)} AS ${q(c)}`;
}
return dataCols.map(c => {
if (c === dateCol) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
if (dimPeriodMap.has(c)) return `dp.${q(dimPeriodMap.get(c))} AS ${q(c)}`;
return `${pfx}${q(c)}`;
}).join(',\n ');
}
function buildFromClause() {
if (!hasDimPeriod) return srcTable;
return srcTable + ' s' + dimPeriodJoins(dateGroups);
return `${srcTable} s\n JOIN pf.dim_period dp`
+ ` ON dp.drange @> (s.${q(dateCol)} + '{{date_offset}}'::interval)::date`;
}
function buildBaseline() {
return `
WITH
ilog AS (
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag)
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}',
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}')
RETURNING id
)
,ins AS (
INSERT INTO {{fc_table}} (${loadInsertCols})
INSERT INTO {{fc_table}} (${insertCols})
SELECT
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
@ -386,20 +181,19 @@ ilog AS (
WHERE {{filter_clause}}
RETURNING *
)
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim();
SELECT count(*) AS rows_affected FROM ins`.trim();
}
function buildReference() {
return `
WITH
ilog AS (
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag)
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}',
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}')
RETURNING id
)
,ins AS (
INSERT INTO {{fc_table}} (${loadInsertCols})
INSERT INTO {{fc_table}} (${insertCols})
SELECT
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
@ -407,7 +201,7 @@ ilog AS (
WHERE {{filter_clause}}
RETURNING *
)
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim();
SELECT count(*) AS rows_affected FROM ins`.trim();
}
function buildScale() {
@ -417,16 +211,13 @@ SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim
const uSel = effectiveUnits
? `round((${q(effectiveUnits)} / NULLIF(total_units, 0)) * {{units_incr}}, 5)`
: `0`;
// sum(sum(x)) OVER () is the aggregate of the aggregates: the window runs
// after the GROUP BY, so the total is over collapsed coordinates and
// comes to the same figure the ungrouped window produced.
const baseSelectParts = [
...dimsJoined ? [dimsJoined] : [],
datesJoined,
effectiveValue ? `sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : null,
effectiveUnits ? `sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : null,
effectiveValue ? `sum(sum(${q(effectiveValue)})) OVER () AS total_value` : null,
effectiveUnits ? `sum(sum(${q(effectiveUnits)})) OVER () AS total_units` : null
q(dateCol),
effectiveValue ? q(effectiveValue) : null,
effectiveUnits ? q(effectiveUnits) : null,
effectiveValue ? `sum(${q(effectiveValue)}) OVER () AS total_value` : null,
effectiveUnits ? `sum(${q(effectiveUnits)}) OVER () AS total_units` : null
].filter(Boolean).join(',\n ');
return `
WITH
@ -441,13 +232,11 @@ ilog AS (
FROM {{fc_table}}
WHERE {{where_clause}}
{{exclude_clause}}
GROUP BY
${groupCols([...dims, ...dateCols])}
)
,ins AS (
INSERT INTO {{fc_table}} (${insertCols})
SELECT
${[dimsJoined, datesJoined, ...(effectiveValue ? [vSel] : []), ...(effectiveUnits ? [uSel] : [])].join(',\n ')},
${[dimsJoined, q(dateCol), ...(effectiveValue ? [vSel] : []), ...(effectiveUnits ? [uSel] : [])].join(',\n ')},
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM base
RETURNING *
@ -464,25 +253,21 @@ ilog AS (
RETURNING id
)
,src AS (
SELECT
${dimsJoined},
${datesJoined}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''}
SELECT ${selectData}
FROM {{fc_table}}
WHERE {{where_clause}}
{{exclude_clause}}
GROUP BY
${groupCols([...dims, ...dateCols])}
)
,neg AS (
INSERT INTO {{fc_table}} (${insertCols})
SELECT ${dimsJoined}, ${datesJoined}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src
RETURNING *
)
,ins AS (
INSERT INTO {{fc_table}} (${insertCols})
SELECT {{set_clause}}, ${datesJoined}, ${effectiveValue ? q(effectiveValue) : '0'}${effectiveUnits ? `, ${q(effectiveUnits)}` : ''},
SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}${effectiveUnits ? `, ${q(effectiveUnits)}` : ''},
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src
RETURNING *
@ -496,16 +281,6 @@ ${opTail('allrows')}` : 'SELECT * FROM neg UNION ALL SELECT * FROM ins'}`.trim()
}
function buildClone() {
const select = [
// dims: whatever {{set_clause}} resolves them to. The route builds it,
// and substitutes the dim_period expression for any derived dimension
// the caller has not overridden outright.
'{{set_clause}}',
...dateCols.map(c => `(s.${q(c)} + '{{date_offset}}'::interval)::date`),
effectiveValue ? `round(s.${q(effectiveValue)} * {{scale_factor}}, 2)` : null,
effectiveUnits ? `round(s.${q(effectiveUnits)} * {{scale_factor}}, 5)` : null,
].filter(Boolean).join(',\n ');
return `
WITH
ilog AS (
@ -514,19 +289,15 @@ ilog AS (
RETURNING id
)
,ins AS (
INSERT INTO {{fc_table}} (${cloneInsertCols})
INSERT INTO {{fc_table}} (${insertCols})
SELECT
${select},
{{set_clause}},
${q(dateCol)},
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}${effectiveUnits ? `,\n round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : ''},
'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM (
SELECT
${groupCols([...dims, ...dateCols])}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''}
FROM {{fc_table}}
WHERE {{where_clause}}
{{exclude_clause}}
GROUP BY
${groupCols([...dims, ...dateCols])}
) s${hasDimPeriod ? dimPeriodJoins(dateGroups) : ''}
FROM {{fc_table}}
WHERE {{where_clause}}
{{exclude_clause}}
RETURNING *
)
${opTail('ins')}`.trim();
@ -556,53 +327,13 @@ function applyTokens(sql, tokens) {
// build a SQL WHERE clause string from a slice object
// only dimension columns are included; unrecognised keys are silently skipped
// pf_segment and pf_bucket are not columns on the forecast table -- they are
// computed at read time from the row's pf.log entry -- so a slice naming one
// cannot be compared directly. It resolves to a set of log ids instead, which is
// exact: the name lives on the log row, and every forecast row carries the
// pf_logid that points at it.
//
// Without this they were dropped from the slice, and clicking a single bucket's
// cell scaled every bucket at that dimension intersection while the panel showed
// only the one clicked.
const COMPUTED_SLICE_COLS = { pf_segment: SEGMENT_EXPR, pf_bucket: BUCKET_EXPR };
function computedSlicePredicate(col, val, versionId) {
if (versionId == null) {
const err = new Error(`Cannot filter on ${col} without a version`);
err.status = 500;
throw err;
}
const vals = (Array.isArray(val) ? val : [val]).map(v => `'${esc(v)}'`).join(', ');
return `pf_logid IN (
SELECT l.id
FROM pf.log l${VERSION_JOIN}
WHERE l.version_id = ${parseInt(versionId)}
AND ${COMPUTED_SLICE_COLS[col]} IN (${vals})
)`;
}
function buildWhere(slice, dimCols, versionId) {
function buildWhere(slice, dimCols) {
if (!slice || Object.keys(slice).length === 0) return 'TRUE';
const allowed = new Set(dimCols);
const parts = [];
for (const [col, val] of Object.entries(slice)) {
if (COMPUTED_SLICE_COLS[col]) {
parts.push(computedSlicePredicate(col, val, versionId));
continue;
}
// A pf_ key this does not understand is refused rather than skipped.
// Skipping is how a selection silently widened: the operation ran against
// everything the dropped key would have excluded. pf_iter is the one
// exception -- the client strips it deliberately, since two cells that
// differ only by iter band are the same slice.
if (col.startsWith('pf_') && col !== 'pf_iter') {
const err = new Error(`Slice names ${col}, which cannot be filtered on`);
err.status = 400;
throw err;
}
if (!allowed.has(col)) continue;
if (Array.isArray(val)) {
const escaped = val.map(v => esc(v));
@ -615,110 +346,6 @@ function buildWhere(slice, dimCols, versionId) {
return parts.length ? parts.join('\nAND ') : 'TRUE';
}
// The pivot's own filter, carried alongside the slices so an operation writes
// exactly the rows the ledger counted.
//
// A slice is {col: value} and can only ever mean equality, so a view filtered to
// sseas_e <= 2027 could not be expressed as one. Refusing it was safe but
// useless -- a bounded season is an ordinary way to scope a forecast -- so the
// operators travel as [col, op, value] triples instead.
//
// Perspective's operator names, not SQL's, since that is where these come from.
// Anything outside this list is refused rather than ignored: a scope silently
// dropped is a write that is wider than the panel that authorised it.
const SCOPE_OPS = {
'==': (c, v) => `${c} = ${v[0]}`,
'!=': (c, v) => `${c} != ${v[0]}`,
'>': (c, v) => `${c} > ${v[0]}`,
'>=': (c, v) => `${c} >= ${v[0]}`,
'<': (c, v) => `${c} < ${v[0]}`,
'<=': (c, v) => `${c} <= ${v[0]}`,
'in': (c, v) => `${c} IN (${v.join(', ')})`,
'not in': (c, v) => `${c} NOT IN (${v.join(', ')})`,
'is null': (c) => `${c} IS NULL`,
'is not null': (c) => `${c} IS NOT NULL`,
};
function buildScopeClause(scope, dimCols, versionId) {
if (!Array.isArray(scope) || scope.length === 0) return '';
const allowed = new Set(dimCols);
const parts = scope.map((entry) => {
if (!Array.isArray(entry) || entry.length < 2) {
const err = new Error(`Malformed scope entry ${JSON.stringify(entry)}`);
err.status = 400; throw err;
}
const [col, op, ...rest] = entry;
const vals = (Array.isArray(rest[0]) ? rest[0] : rest).filter(v => v !== undefined);
if (COMPUTED_SLICE_COLS[col]) {
if (op !== '==' && op !== 'in') {
const err = new Error(`${col} can only be scoped with == or in, not ${op}`);
err.status = 400; throw err;
}
return computedSlicePredicate(col, vals, versionId);
}
if (!allowed.has(col)) {
const err = new Error(`Column "${col}" is not available for filtering`);
err.status = 400; throw err;
}
const fn = SCOPE_OPS[op];
if (!fn) {
const err = new Error(`Unsupported filter operator "${op}" on ${col}`);
err.status = 400; throw err;
}
return fn(`"${col}"`, vals.map(v => `'${esc(String(v))}'`));
});
return parts.join('\nAND ');
}
// The territory predicate: what this account may see and change.
//
// Server-side by construction. Today's `scope` is sent by the browser, which is
// right for a filter the user chose and would be fatal for a permission -- so
// this one is built from the session and ANDed on last, where nothing in the
// request can remove it.
//
// FALSE, not TRUE, for an account with no territory. The whole point is that a
// missing grant means no rows: an empty list that fell through to TRUE would
// hand the entire book to the first account somebody forgot to configure.
function buildTerritoryClause(territory, territoryCol, alias = '') {
if (!territory || territory.admin) return '';
if (!territoryCol) return 'FALSE';
const vals = (territory.values || []).filter(v => v != null && v !== '');
if (vals.length === 0) return 'FALSE';
const pfx = alias ? `${alias}.` : '';
return `${pfx}"${territoryCol}" IN (${vals.map(v => `'${esc(String(v))}'`).join(', ')})`;
}
// build a WHERE clause spanning several slices — an OR of AND-groups.
// A union of slices cannot be flattened into one IN list per column: slices
// {Region:East, State:NY} and {Region:West, State:CA} would become
// Region IN (East,West) AND State IN (NY,CA), which also matches East/CA.
function buildWhereAny(slices, dimCols, versionId) {
const list = (slices || []).filter(s => s && Object.keys(s).length > 0);
if (list.length === 0) return 'TRUE';
if (list.length === 1) return buildWhere(list[0], dimCols, versionId);
const groups = list
.map(s => buildWhere(s, dimCols, versionId))
.filter(w => w !== 'TRUE');
// any slice that reduced to TRUE selects everything, so the union does too
if (groups.length !== list.length) return 'TRUE';
// outer parens matter: the caller appends `AND pf_iter NOT IN (...)`,
// and AND binds tighter than OR
return `(${groups.map(g => `(${g.replace(/\n/g, ' ')})`).join('\n OR ')})`;
}
// the bare predicate for "this row participates in operations", for use in a
// FILTER clause where the excluded rows still need to be counted separately
function buildExcludePredicate(excludeIters) {
if (!excludeIters || excludeIters.length === 0) return 'TRUE';
const list = excludeIters.map(i => `'${esc(i)}'`).join(', ');
return `pf_iter NOT IN (${list})`;
}
// build AND iter NOT IN (...) from a version's exclude_iters array
function buildExcludeClause(excludeIters) {
if (!excludeIters || excludeIters.length === 0) return '';
@ -728,21 +355,12 @@ function buildExcludeClause(excludeIters) {
// build the dimension columns portion of a SELECT for recode/clone
// replaces named dimensions with literal values, passes others through unchanged
// derivedExprs: cname -> a SQL expression to use when the caller has not set the
// column outright. Clone passes the dim_period expressions through here, so a
// cloned row's period dimensions come from the calendar against its shifted date
// rather than from the row it was copied from.
function buildSetClause(dimCols, setObj, opts = {}) {
const { derivedExprs, alias } = opts;
const pfx = alias ? `${alias}.` : '';
function buildSetClause(dimCols, setObj) {
return dimCols.map(col => {
if (setObj && setObj[col] !== undefined) {
return `'${esc(setObj[col])}' AS "${col}"`;
}
if (derivedExprs && derivedExprs[col]) {
return `${derivedExprs[col]} AS "${col}"`;
}
return `${pfx}"${col}"`;
return `"${col}"`;
}).join(', ');
}
@ -786,6 +404,4 @@ function esc(val) {
return String(val).replace(/'/g, "''");
}
module.exports = { generateSQL, grainOf, COMPUTED_SLICE_COLS, buildScopeClause, buildTerritoryClause,
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR, LABEL_GROUP_COLS, VERSION_JOIN,
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET, UNLABELED_LOAD, dateGroupsOf, dimPeriodMapOf, dimPeriodJoins, applyTokens, buildWhere, buildWhereAny, buildExcludeClause, buildExcludePredicate, buildSetClause, buildFilterClause, esc };
module.exports = { generateSQL, grainOf, applyTokens, buildWhere, buildExcludeClause, buildSetClause, buildFilterClause, esc };

38
lib/utils.js
View File

@ -3,41 +3,7 @@ function fcTable(tname, versionId) {
return `pf.fc_${tname}_${versionId}`;
}
// The columns of a relation, from pg_catalog rather than information_schema.
//
// information_schema.columns omits materialized views -- they are not in the
// SQL standard -- so a source built on one looked like it had no columns at
// all: registering it seeded nothing, and creating a version failed with "No
// usable columns in col_meta" while col_meta plainly had thirty-six.
//
// The shape matches what information_schema returned, so mapType and the
// callers did not have to change. data_type is format_type with the modifier
// stripped, which gives the same spelling information_schema uses ('character
// varying', 'numeric'), and the numeric precision and scale are unpacked from
// atttypmod the way information_schema does internally.
//
// Takes $1 = schema, $2 = relation name.
const RELATION_COLUMNS_SQL = `
SELECT a.attname AS column_name
,regexp_replace(format_type(a.atttypid, a.atttypmod), '\\(.*\\)$', '') AS data_type
,a.attnum AS ordinal_position
,CASE WHEN a.attnotnull THEN 'NO' ELSE 'YES' END AS is_nullable
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
THEN ((a.atttypmod - 4) >> 16) & 65535 END AS numeric_precision
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
THEN (a.atttypmod - 4) & 65535 END AS numeric_scale
FROM pg_attribute a
JOIN pg_class c ON c.oid = a.attrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE TRUE
AND n.nspname = $1
AND c.relname = $2
AND c.relkind IN ('r', 'v', 'm', 'f', 'p')
AND a.attnum > 0
AND NOT a.attisdropped
`;
// map a data_type name to a clean postgres column type
// map information_schema data_type to a clean postgres column type
function mapType(dataType, numericPrecision, numericScale) {
switch (dataType) {
case 'character varying':
@ -70,4 +36,4 @@ function mapType(dataType, numericPrecision, numericScale) {
}
}
module.exports = { fcTable, mapType, RELATION_COLUMNS_SQL };
module.exports = { fcTable, mapType };

68
package-lock.json generated
View File

@ -7,14 +7,11 @@
"": {
"name": "pf_app",
"version": "1.0.0",
"license": "MIT",
"dependencies": {
"apache-arrow": "^21.1.0",
"connect-pg-simple": "^10.0.0",
"cors": "^2.8.5",
"dotenv": "^16.0.0",
"express": "^4.18.2",
"express-session": "^1.19.0",
"pg": "^8.11.3"
},
"devDependencies": {
@ -372,18 +369,6 @@
"node": ">=12.20.0"
}
},
"node_modules/connect-pg-simple": {
"version": "10.0.0",
"resolved": "https://registry.npmjs.org/connect-pg-simple/-/connect-pg-simple-10.0.0.tgz",
"integrity": "sha512-pBGVazlqiMrackzCr0eKhn4LO5trJXsOX0nQoey9wCOayh80MYtThCbq8eoLsjpiWgiok/h+1/uti9/2/Una8A==",
"license": "MIT",
"dependencies": {
"pg": "^8.12.0"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=22.0.0"
}
},
"node_modules/content-disposition": {
"version": "0.5.4",
"resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
@ -597,29 +582,6 @@
"url": "https://opencollective.com/express"
}
},
"node_modules/express-session": {
"version": "1.19.0",
"resolved": "https://registry.npmjs.org/express-session/-/express-session-1.19.0.tgz",
"integrity": "sha512-0csaMkGq+vaiZTmSMMGkfdCOabYv192VbytFypcvI0MANrp+4i/7yEkJ0sbAEhycQjntaKGzYfjfXQyVb7BHMA==",
"license": "MIT",
"dependencies": {
"cookie": "~0.7.2",
"cookie-signature": "~1.0.7",
"debug": "~2.6.9",
"depd": "~2.0.0",
"on-headers": "~1.1.0",
"parseurl": "~1.3.3",
"safe-buffer": "~5.2.1",
"uid-safe": "~2.1.5"
},
"engines": {
"node": ">= 0.8.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/express"
}
},
"node_modules/fill-range": {
"version": "7.1.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz",
@ -1123,15 +1085,6 @@
"node": ">= 0.8"
}
},
"node_modules/on-headers": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/on-headers/-/on-headers-1.1.0.tgz",
"integrity": "sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==",
"license": "MIT",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/parseurl": {
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
@ -1323,15 +1276,6 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/random-bytes": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/random-bytes/-/random-bytes-1.0.0.tgz",
"integrity": "sha512-iv7LhNVO047HzYR3InF6pUcUsPQiHTM1Qal51DcGSuZFBil1aBBWG5eHPNek7bvILMaYJ/8RU1e8w1AMdHmLQQ==",
"license": "MIT",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/range-parser": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
@ -1648,18 +1592,6 @@
"node": ">=12.17"
}
},
"node_modules/uid-safe": {
"version": "2.1.5",
"resolved": "https://registry.npmjs.org/uid-safe/-/uid-safe-2.1.5.tgz",
"integrity": "sha512-KPHm4VL5dDXKz01UuEd88Df+KzynaohSL9fBh096KWAxSKZQDI2uBrVqtvRM4rwrIrRRKsdLNML/lnaaVSRioA==",
"license": "MIT",
"dependencies": {
"random-bytes": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/undefsafe": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz",

8
package.json
View File

@ -7,15 +7,17 @@
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js",
"build": "cd ui && npm run build"
"build": "cd ui && npm run build",
"migrate": "node scripts/migrate.js up",
"migrate:status": "node scripts/migrate.js status",
"migrate:baseline": "node scripts/migrate.js baseline",
"schema:dump": "node scripts/schema-dump.js"
},
"dependencies": {
"apache-arrow": "^21.1.0",
"connect-pg-simple": "^10.0.0",
"cors": "^2.8.5",
"dotenv": "^16.0.0",
"express": "^4.18.2",
"express-session": "^1.19.0",
"pg": "^8.11.3"
},
"devDependencies": {

362
pf.sh
View File

@ -4,8 +4,6 @@ set -euo pipefail
# ---------------------------------------------------------------------------
# pf.sh — Pivot Forecast management script
# Usage: ./pf.sh [deploy|start|stop|restart|status|logs|db-setup|config]
# ./pf.sh [add-user|passwd|list-users|disable-user|enable-user]
# ./pf.sh [set-territory|set-admin|orphan-territory]
# ./pf.sh (interactive menu)
# ---------------------------------------------------------------------------
@ -69,33 +67,13 @@ require_service() {
service_installed || die "systemd service not installed. Run: ./pf.sh install-service"
}
# The keys cmd_config manages; anything else in .env is left alone.
ENV_KEYS=(DB_HOST DB_PORT DB_NAME DB_USER DB_PASSWORD PORT SESSION_SECRET COOKIE_SECURE)
env_get() {
[[ -f "$ENV_FILE" ]] || return 0
grep -E "^$1=" "$ENV_FILE" | tail -1 | cut -d= -f2- | tr -d '"' || true
}
# psql against the DB_* connection in .env; extra args are passed through.
run_psql() {
PGPASSWORD="${DB_PASSWORD:-}" psql \
-h "${DB_HOST:-localhost}" \
-p "${DB_PORT:-5432}" \
-U "${DB_USER}" \
-d "${DB_NAME}" \
"$@"
}
db_ping() {
load_env
if [[ -z "${DB_NAME:-}" || -z "${DB_USER:-}" ]]; then
warn "DB_NAME / DB_USER not set in .env"
return 1
fi
local url="${DATABASE_URL:-}"
[[ -z "$url" ]] && { warn "DATABASE_URL not set in .env"; return 1; }
# Use psql if available for a real connectivity check
if command -v psql &>/dev/null; then
run_psql -tAc "SELECT 1" &>/dev/null && return 0 || return 1
psql "$url" -c "SELECT 1" &>/dev/null && return 0 || return 1
else
warn "psql not in PATH — skipping live DB check"
return 0
@ -190,25 +168,18 @@ cmd_logs() {
cmd_db_setup() {
require_env; load_env
[[ -n "${DB_NAME:-}" && -n "${DB_USER:-}" ]] || die "DB_NAME / DB_USER not set in .env — run: ./pf.sh config"
local url="${DATABASE_URL:-}"
[[ -z "$url" ]] && die "DATABASE_URL not set in .env"
command -v psql &>/dev/null || die "psql not found — install postgresql-client"
echo
bold "DB Setup — will run: setup_sql/01_schema.sql, setup_sql/02_auth.sql"
warn "This creates the pf schema, tables, and the account/session tables."
warn "Safe to re-run (CREATE IF NOT EXISTS)."
bold "DB Setup — will run: setup_sql/01_schema.sql"
warn "This creates the pf schema and tables. Safe to re-run (CREATE IF NOT EXISTS)."
read -rp " Continue? [y/N] " confirm
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; return; }
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/01_schema.sql"
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/02_auth.sql"
psql "$url" -f "${APP_DIR}/setup_sql/01_schema.sql"
success "Schema applied."
local n
n=$(run_psql -tAc "SELECT count(*) FROM pf.app_user WHERE is_active" 2>/dev/null || echo 0)
if [[ "${n:-0}" == "0" ]]; then
warn "No active accounts yet — create one with: ./pf.sh add-user"
fi
}
cmd_config() {
@ -217,284 +188,41 @@ cmd_config() {
echo " File: $ENV_FILE"
echo
local cur_host cur_port cur_name cur_user cur_pass cur_app_port
cur_host=$(env_get DB_HOST)
cur_port=$(env_get DB_PORT)
cur_name=$(env_get DB_NAME)
cur_user=$(env_get DB_USER)
cur_pass=$(env_get DB_PASSWORD)
cur_app_port=$(env_get PORT)
local current_url=""
local current_port=""
local current_user=""
local input
read -rp " DB_HOST [${cur_host:-localhost}]: " input
local host="${input:-${cur_host:-localhost}}"
read -rp " DB_PORT [${cur_port:-5432}]: " input
local port="${input:-${cur_port:-5432}}"
read -rp " DB_NAME [${cur_name:-not set}]: " input
local name="${input:-$cur_name}"
[[ -z "$name" ]] && die "DB_NAME is required."
read -rp " DB_USER [${cur_user:-$USER}]: " input
local user="${input:-${cur_user:-$USER}}"
if [[ -n "$cur_pass" ]]; then
read -rsp " DB_PASSWORD [keep existing]: " input; echo
else
read -rsp " DB_PASSWORD: " input; echo
fi
local pass="${input:-$cur_pass}"
read -rp " PORT (app) [${cur_app_port:-3010}]: " input
local app_port="${input:-${cur_app_port:-3010}}"
# Session cookies are signed with this; regenerating it signs everyone out,
# so an existing secret is kept rather than re-rolled on every config run.
local secret
secret=$(env_get SESSION_SECRET)
if [[ -z "$secret" ]]; then
secret=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')
success "SESSION_SECRET generated."
else
success "SESSION_SECRET kept (delete the line in .env to re-roll)."
fi
local cur_secure
cur_secure=$(env_get COOKIE_SECURE)
read -rp " COOKIE_SECURE — HTTPS-only cookie [${cur_secure:-true}]: " input
local cookie_secure="${input:-${cur_secure:-true}}"
# Rewrite the managed keys, carrying over any other lines already in .env.
local tmp
tmp=$(mktemp)
cat > "$tmp" <<EOF
DB_HOST=${host}
DB_PORT=${port}
DB_NAME=${name}
DB_USER=${user}
DB_PASSWORD=${pass}
PORT=${app_port}
SESSION_SECRET=${secret}
COOKIE_SECURE=${cookie_secure}
EOF
if [[ -f "$ENV_FILE" ]]; then
local managed
managed=$(IFS='|'; echo "${ENV_KEYS[*]}")
grep -vE "^(${managed})=" "$ENV_FILE" | grep -vE '^[[:space:]]*$' >> "$tmp" || true
current_url=$(grep -E '^DATABASE_URL=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
current_port=$(grep -E '^PORT=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
current_user=$(grep -E '^PF_USER=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
fi
mv "$tmp" "$ENV_FILE"
read -rp " DATABASE_URL [${current_url:-not set}]: " input_url
local url="${input_url:-$current_url}"
[[ -z "$url" ]] && die "DATABASE_URL is required."
read -rp " PORT [${current_port:-3010}]: " input_port
local port="${input_port:-${current_port:-3010}}"
read -rp " PF_USER [${current_user:-$USER}]: " input_user
local pf_user="${input_user:-${current_user:-$USER}}"
cat > "$ENV_FILE" <<EOF
DATABASE_URL=${url}
PORT=${port}
PF_USER=${pf_user}
EOF
chmod 600 "$ENV_FILE"
success ".env written."
if db_ping; then
success "Database connection verified."
else
warn "Could not reach the database — double-check the DB_* settings."
warn "Could not reach the database — double-check DATABASE_URL."
fi
}
# -- Accounts ----------------------------------------------------------------
# Reads a password twice without echo and hashes it with lib/auth.js, so the
# plaintext never reaches argv, psql, or the shell history.
read_new_password() {
local p1 p2
# Prompts and their newlines go to stderr: stdout is the hash, and a stray
# newline there ends up prefixed to it by the caller's $( ).
read -rsp " Password: " p1; echo >&2
[[ -z "$p1" ]] && { error "Password cannot be empty."; return 1; }
read -rsp " Confirm : " p2; echo >&2
[[ "$p1" != "$p2" ]] && { error "Passwords do not match."; return 1; }
printf '%s' "$p1" | node "${APP_DIR}/lib/auth.js" hash
}
# psql single-quoted literal: double any embedded quote.
sql_lit() { printf "%s" "${1//\'/\'\'}"; }
cmd_add_user() {
require_env; load_env
check_node >/dev/null
echo; bold "Add account"
local username display hash
read -rp " Username: " username
[[ -z "$username" ]] && die "Username is required."
read -rp " Display name [${username}]: " display
display="${display:-$username}"
hash=$(read_new_password) || return 1
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH ins AS (
INSERT INTO pf.app_user (username, pass_hash, display_name)
VALUES ('$(sql_lit "$username")', '$(sql_lit "$hash")', '$(sql_lit "$display")')
ON CONFLICT (username) DO NOTHING
RETURNING id
) SELECT id FROM ins" | grep -q . \
&& success "Account '${username}' created." \
|| die "Account '${username}' already exists — change its password with: ./pf.sh passwd"
}
cmd_passwd() {
require_env; load_env
check_node >/dev/null
echo; bold "Change password"
local username hash
read -rp " Username: " username
[[ -z "$username" ]] && die "Username is required."
hash=$(read_new_password) || return 1
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET pass_hash = '$(sql_lit "$hash")'
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
&& success "Password updated for '${username}'." \
|| die "No such account: ${username}"
}
cmd_list_users() {
require_env; load_env
echo; bold "Accounts"
run_psql -c "
SELECT username, display_name, is_active, is_admin,
coalesce(jsonb_array_length(territory), 0) AS territory_values,
to_char(last_login_at, 'YYYY-MM-DD HH24:MI') AS last_login
FROM pf.app_user ORDER BY username"
}
# Territory is what an account may see and change, as a list of values in the
# source's is_territory column. No territory and not an admin means no rows --
# so a new account is blind until this is run, which is the intended direction
# to fail in.
#
# Values are given comma-separated and have to match the column exactly, since
# that is what the SQL compares. set-territory with no values clears it.
cmd_set_territory() {
require_env; load_env
local username="${1:-}"; shift || true
[[ -z "$username" ]] && { read -rp " Username: " username; }
[[ -z "$username" ]] && die "Username is required."
local values="${*:-}"
[[ -z "$values" ]] && { read -rp " Territory values (comma separated, blank to clear): " values; }
local json="null"
if [[ -n "$values" ]]; then
json=$(python3 - "$values" <<'PYEOF'
import json, sys
vals = [v.strip() for v in sys.argv[1].split(',') if v.strip()]
print(json.dumps(vals))
PYEOF
)
fi
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET territory = $(if [[ "$json" == "null" ]]; then echo NULL; else echo "'$(sql_lit "$json")'::jsonb"; fi)
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING username
)
SELECT count(*) FROM upd" | grep -q '^1$' \
|| die "No such account: $username"
success "Territory updated for $username"
run_psql -c "SELECT username, is_admin, territory FROM pf.app_user WHERE lower(username) = lower('$(sql_lit "$username")')"
}
# An admin sees and changes everything, and is the only account that can recode
# the territory column or undo someone else's entry.
cmd_set_admin() {
require_env; load_env
local username="${1:-}" flag="${2:-true}"
[[ -z "$username" ]] && { read -rp " Username: " username; }
[[ -z "$username" ]] && die "Username is required."
[[ "$flag" != "true" && "$flag" != "false" ]] && die "Second argument must be true or false."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_admin = $flag
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING username
)
SELECT count(*) FROM upd" | grep -q '^1$' \
|| die "No such account: $username"
success "$username is_admin = $flag"
}
# Territory values present in the data that belong to no account. Work under one
# is invisible to everybody but an admin, which is easy to cause by a typo and
# impossible to notice from inside the app.
cmd_orphan_territory() {
require_env; load_env
local source_id="${1:-}"
[[ -z "$source_id" ]] && { read -rp " Source id: " source_id; }
[[ -z "$source_id" ]] && die "Source id is required."
# The column and table are data, so the query is built in two steps rather
# than one clever one: read the names, then run the listing.
local meta col schema tname
meta=$(run_psql -tAF'|' -c "
SELECT m.cname, x.schema, x.tname
FROM pf.col_meta m JOIN pf.source x ON x.id = m.source_id
WHERE m.source_id = $source_id AND m.is_territory")
[[ -z "$meta" ]] && die "Source $source_id has no column marked is_territory."
IFS='|' read -r col schema tname <<< "$meta"
echo; bold "Territory values in $schema.$tname with no account"
run_psql -c "
SELECT DISTINCT s.\"$col\" AS unassigned
FROM \"$schema\".\"$tname\" s
WHERE TRUE
AND s.\"$col\" IS NOT NULL
AND s.\"$col\"::text NOT IN (
SELECT jsonb_array_elements_text(territory)
FROM pf.app_user
WHERE territory IS NOT NULL
)
ORDER BY 1"
}
# Deactivating leaves the row (and its history) in place, and drops any live
# session so the account loses access immediately rather than at cookie expiry.
cmd_disable_user() {
require_env; load_env
local username="${1:-}"
[[ -z "$username" ]] && { read -rp " Username to disable: " username; }
[[ -z "$username" ]] && die "Username is required."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_active = false
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
|| die "No such account: ${username}"
run_psql -v ON_ERROR_STOP=1 -c "
DELETE FROM pf.session
WHERE sess::jsonb -> 'user' ->> 'username' ILIKE '$(sql_lit "$username")'" >/dev/null
success "Account '${username}' disabled and signed out."
}
cmd_enable_user() {
require_env; load_env
local username="${1:-}"
[[ -z "$username" ]] && { read -rp " Username to enable: " username; }
[[ -z "$username" ]] && die "Username is required."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_active = true
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
&& success "Account '${username}' enabled." \
|| die "No such account: ${username}"
}
cmd_install_service() {
require_systemd
require_env
@ -575,17 +303,9 @@ interactive_menu() {
echo " 5) status service + DB + git info"
echo " 6) logs tail journald logs"
echo " 7) db-setup apply setup_sql/01_schema.sql"
echo " 8) config set DB connection + app PORT"
echo " 8) config set DATABASE_URL / PORT / PF_USER"
echo " 9) install-service create systemd unit file"
echo " 10) uninstall-service remove systemd unit file"
echo " 11) add-user create a login account"
echo " 12) passwd change an account password"
echo " 13) list-users show accounts"
echo " 14) disable-user deactivate an account and sign it out"
echo " 15) enable-user reactivate an account"
echo " 16) set-territory grant an account the territory values it may see"
echo " 17) set-admin make an account an administrator"
echo " 18) orphan-territory territory values no account owns"
echo " q) quit"
echo
read -rp " Choice: " choice
@ -600,14 +320,6 @@ interactive_menu() {
8|config) cmd_config ;;
9|install-service) cmd_install_service ;;
10|uninstall-service) cmd_uninstall_service ;;
11|add-user) cmd_add_user ;;
12|passwd) cmd_passwd ;;
13|list-users) cmd_list_users ;;
14|disable-user) cmd_disable_user ;;
15|enable-user) cmd_enable_user ;;
16|set-territory) cmd_set_territory ;;
17|set-admin) cmd_set_admin ;;
18|orphan-territory) cmd_orphan_territory ;;
q|Q|quit|exit) echo "Bye."; exit 0 ;;
*) warn "Unknown option: $choice" ;;
esac
@ -627,14 +339,6 @@ case "${1:-}" in
config) cmd_config ;;
install-service) cmd_install_service ;;
uninstall-service) cmd_uninstall_service ;;
add-user) cmd_add_user ;;
passwd) cmd_passwd ;;
list-users) cmd_list_users ;;
set-territory) shift; cmd_set_territory "$@" ;;
set-admin) shift; cmd_set_admin "$@" ;;
orphan-territory) shift; cmd_orphan_territory "$@" ;;
disable-user) cmd_disable_user "${2:-}" ;;
enable-user) cmd_enable_user "${2:-}" ;;
"") interactive_menu ;;
*) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service add-user passwd list-users disable-user enable-user set-territory set-admin orphan-territory" ;;
*) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service" ;;
esac

270
pf_spec.md
View File

@ -113,20 +113,10 @@ CREATE TABLE pf.log (
operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
slice jsonb, -- the WHERE conditions that defined the selection
params jsonb, -- operation parameters (increments, new values, scale factor, etc.)
note text, -- user-provided comment
tag text -- initiative label, e.g. 'reduce_spend'
note text -- user-provided comment
);
```
`tag` groups adjustments into initiatives. It is what the bridge walks: every entry
carrying the same tag becomes one step from baseline to current. Both `note` and `tag`
are annotations — they never affect forecast rows — so both stay editable after the
fact via `PATCH /api/log/:logid`.
Tags are written by a follow-up `UPDATE` after the operation runs, not by the generated
SQL. The templates in `pf.sql` are stored per source, so adding a `{{tag}}` token would
silently stop recording tags for any source that had not re-run *Generate SQL*.
### `pf.fc_{tname}_{version_id}` (dynamic, one per version)
Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`).
@ -199,9 +189,22 @@ CREATE TABLE pf.sql (
```
setup_sql/
01_schema.sql -- CREATE SCHEMA pf; create all metadata tables (source, col_meta, version, log, sql)
migrations/ -- forward-only .sql, applied once each, tracked in pf.schema_version
0001_initial_schema.sql CREATE SCHEMA pf + metadata tables
0002_source_default_layout.sql
0003_col_meta_dim_group_period.sql
0004_col_meta_in_grain.sql
gen_dim_period.sql -- parameterised calendar load (fiscal year start month); not a migration
schema.generated.sql -- pg_dump reference snapshot; generated, never edited or applied
README.md -- migration workflow
```
Install with `npm run migrate`, then run `gen_dim_period.sql`. For a database that
already matches the schema, `npm run migrate:baseline` records the migrations as
applied without running them. `server.js` refuses to start when the database is
behind, so drift surfaces at boot rather than as a column-not-found error inside
an unrelated request.
Source registration, col_meta configuration, SQL generation, version creation, and forecast table DDL all happen via API.
---
@ -328,79 +331,35 @@ All operations share a common request envelope:
```json
{
"pf_user": "paul.trowbridge",
"note": "optional comment",
"tag": "reduce_spend",
"slices": [ { "channel": "WHS", "geography": "WEST" },
{ "channel": "DIR", "geography": "EAST" } ],
"apply_mode": "prorate"
"pf_user": "paul.trowbridge",
"note": "optional comment",
"slice": {
"channel": "WHS",
"geography": "WEST"
}
}
```
- `slices` — one or more slices. The legacy single `slice` object is still accepted and
treated as a one-entry list.
- `apply_mode` — `prorate` (default) treats the selection as one pool; `each` runs the
operation once per slice, producing one log entry per slice so they can be undone
separately. With a single slice the two are identical.
- `tag` — optional initiative label, stored on the log entry.
Slice keys must be `role = 'dimension'` or `role = 'date'` columns per col_meta. A slice
naming none of them is **rejected**: unknown keys are dropped when building the WHERE
clause, so such a slice would otherwise reduce to `TRUE` and apply the operation to the
entire version.
Several slices become an `OR` of `AND`-groups, not per-column `IN` lists — flattening
`{A:1,B:1}` and `{A:2,B:2}` into `A IN (1,2) AND B IN (1,2)` would also match `A:1,B:2`.
The result is parenthesised because callers append `AND pf_iter NOT IN (...)`, and `AND`
binds tighter than `OR`.
`slice` keys must be `role = 'dimension'` columns per col_meta. Stored in `pf.log` as the implicit link to affected rows.
#### Scale
`POST /api/versions/:id/scale`
```json
{
"pf_user": "paul.trowbridge",
"note": "10% volume lift Q3 West",
"tag": "volume_push",
"slices": [ { "channel": "WHS", "geography": "WEST" } ],
"apply_mode": "prorate",
"target_value": 12000,
"units_pct": 10,
"target_basis": "selected"
"pf_user": "paul.trowbridge",
"note": "10% volume lift Q3 West",
"slice": { "channel": "WHS", "geography": "WEST" },
"value_incr": null,
"units_incr": 5000,
"pct": false
}
```
Each measure is resolved **independently**, so a target on one and a percentage on the
other can be sent together. Per measure, exactly one of:
| Field | Meaning |
|---|---|
| `target_value` / `target_units` | the total to end up with |
| `value_pct` / `units_pct` | a percentage of the current total |
| `value_incr` / `units_incr` | an absolute amount to add |
| `target_price` | target value/units ratio; holds units constant |
The legacy global `pct: true` flag (meaning "the increments are percentages") is still
honoured.
`target_basis` decides what a target or percentage measures against:
- `adjustable` — only the rows the operation can write.
- `selected` (UI default) — everything the pivot shows for the slice, `exclude_iters`
rows included. Those rows cannot move, so the adjustable rows absorb the whole
difference and the pivot lands on the number you asked for. Without this, a target set
against a visible total overshoots by the excluded rows' contribution.
Behaviour:
- Excludes `exclude_iters` rows from the rows it writes, in every basis.
- Distributes the increment proportionally across rows in the slice.
- **Refuses to prorate a pool that nets to ~zero** — below 1% of gross. Each row's new
value is `(row / total) * increment`, so as the net approaches zero the multiplier
explodes and rows fly to extreme opposite values to reach the target. Offsetting
slices are the usual cause; `apply_mode: each` handles that correctly.
- Slices matching no rows, or already on target, are skipped and returned in
`slices_skipped` rather than silently counted as applied.
- `value_incr` / `units_incr` — absolute amounts to add (positive or negative). Either can be null.
- `pct: true` — treat as percentage of current slice total instead of absolute
- Excludes `exclude_iters` rows from the source selection
- Distributes increment proportionally across rows in the slice
- Inserts rows tagged `iter = 'scale'`
#### Recode
@ -410,7 +369,7 @@ Behaviour:
{
"pf_user": "paul.trowbridge",
"note": "Part discontinued, replaced by new SKU",
"slices": [ { "part": "OLD-SKU-001" } ],
"slice": { "part": "OLD-SKU-001" },
"set": { "part": "NEW-SKU-002" }
}
```
@ -428,7 +387,7 @@ Behaviour:
{
"pf_user": "paul.trowbridge",
"note": "New customer win, similar profile to existing",
"slices": [ { "customer": "EXISTING CO", "channel": "DIR" } ],
"slice": { "customer": "EXISTING CO", "channel": "DIR" },
"set": { "customer": "NEW CO" },
"scale": 0.75
}
@ -445,10 +404,6 @@ Behaviour:
|--------|-------|-------------|
| GET | `/api/versions/:id/log` | List all log entries for a version, newest first |
| DELETE | `/api/log/:logid` | Undo: delete all forecast rows with this logid, then delete log entry |
| PATCH | `/api/log/:logid` | Edit `note` and/or `tag`. Branches on whether a field was sent, so `""` clears rather than being read as "leave alone" |
| GET | `/api/versions/:id/table-info` | Physical forecast table, source table, and live row counts by `pf_iter` |
| GET | `/api/versions/:id/bridge` | Baseline → current rolled up by tag |
| GET | `/api/sources/:id/tags` | Tags used on this source with use counts, newest first — feeds tag autocomplete |
---
@ -542,47 +497,30 @@ Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any
### Forecast View
**Layout:** the operation panel docks **bottom** (default), **right**, or **floats** over
the pivot (drag its header to move, corner grip to resize). Position and size persist to
`localStorage`. It closes via its header ×, `Esc`, or the toolbar toggle, which shows the
selection count while shut.
**Layout:**
```
┌─────────────────────────────────────────────────────────────────┐
│ [Layout…] [Expand 0 1 2 3] [Refresh] [Change log] [Bridge] │
│ [Hide panel] │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Perspective Viewer (interactive pivot web component) │
│ │
├──────────────── drag to resize ─────────────────────────────────┤
│ SLICE 2 selected │ scale recode clone │ Amount │
│ channel=WHS │ Together | Each │ Baseline 1,000.00 │
│ channel=DIR │ │ ▪ reduce_spend -20.00 │
│ Clear selection │ │ ──────────────────── │
│ │ │ Adjustable 1,070.00 │
│ │ │ reference·fixed 921.72 │
│ │ │ ──────────────────── │
│ │ │ Selected total 1,991.72 │
│ │ │ ──────────────────── │
│ │ │ New value [ 2,000 ] │
│ │ │ Change [ 8 ] │
│ │ │ % change [ 0.4 ] │
│ │ tag [reduce_spend] │ [Apply Scale] │
└─────────────────────────────────────────────────────────────────┘
│ [Version label] [Refresh] [Save layout] [Reset layout] │
├──────────────────────────────────────┬──────────────────────────┤
│ │ │
│ Perspective Viewer │ Operation Panel │
│ (interactive pivot web component) │ (active when slice set) │
│ │ │
│ │ Slice: │
│ │ channel = WHS │
│ │ geography = WEST │
│ │ │
│ │ [ Scale ] [ Recode ] │
│ │ [ Clone ] │
│ │ │
│ │ ... operation form ... │
│ │ │
│ │ [ Submit ] │
│ │ │
└──────────────────────────────────────┴──────────────────────────┘
```
**The ledger.** The scale form is one continuous statement rather than a totals display
plus a separate input form: baseline, each adjustment (grouped by tag), current, then the
edit. `New value`, `Change` and `% change` are three interchangeable editable rows —
typing in any one derives the other two, and whichever you typed in is what gets sent.
That replaces the old target/delta/percent mode toggle: the row you type in *is* the mode.
Rows the pivot shows but operations cannot write (`exclude_iters`, typically `reference`)
appear as their own line with a `Selected total` beneath, and a control chooses which of
the two a target measures against — see `target_basis` above.
**Pivot control:** [Perspective](https://github.com/perspective-dev/perspective) 5.2.0 (`@perspective-dev/*`), **bundled inline, not loaded from a CDN** — the `/inline` entrypoints embed the WASM so the version is pinned by `package-lock.json`. See `PERSPECTIVE.md`. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
**Pivot control:** [Perspective](https://perspective.finos.org/) 4.4.0, loaded from CDN at runtime. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
**Large-dataset loading sequence:**
1. Client issues `GET /api/versions/:id/data`
@ -595,25 +533,9 @@ the two a target measures against — see `target_basis` above.
**Interaction flow:**
1. Click a cell or row in the pivot — the `perspective-click` event fires
2. `detail.config.filter` from the event is parsed: only `==` filters on `role = dimension` columns are extracted as the slice
3. A plain click replaces the selection; **ctrl/⌘/shift-click toggles** a slice in or out of
it. The `CustomEvent` carries no modifier flags, so they are read from the `mousedown`
that preceded it. `perspective-select` (region drag) is wired defensively alongside.
4. Slice populates the Operation Panel — pick operation tab, fill in parameters
5. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
6. For recode, both the negative offset rows and positive replacement rows are returned and streamed
**Selection caveat.** `pf_iter` is not a `col_meta` column, so it is stripped when a slice
is built. Two cells differing only by iter band (baseline vs reference) produce the same
effective slice; duplicates are collapsed before the request, and the panel warns when a
selection covers fewer distinct slices than cells clicked. There is currently no way to
target one band of a slice.
**Expand depth.** Perspective's `GROUP BY ROLLUP` view contains every level of the
hierarchy, and `view.set_depth()` — which lives on the view, not in the saved config — is
the only thing hiding the deeper ones. The viewer rebuilds its view whenever it redraws,
which its Intersection/ResizeObserver triggers on tab refocus, leaving the tree fully
expanded. The last applied depth is therefore re-applied on `visibilitychange`, `focus`
and `pageshow`.
3. Slice populates the Operation Panel — pick operation tab, fill in parameters
4. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
5. For recode, both the negative offset rows and positive replacement rows are returned and streamed
**Pivot default layout:** built from col_meta — first two `dimension` columns as `group_by`, `date` column as `split_by`. User can rearrange in Perspective settings panel and save.
@ -621,43 +543,8 @@ and `pageshow`.
### Log View
Modal list of log entries — timestamp, operation, slice, **tag**, note, rows affected.
"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of
Perspective table).
Tag and note are edited inline (click, Enter to save, Esc to cancel) via
`PATCH /api/log/:logid`; the tag field completes from tags already used on the source.
Saving a tag regroups the ledger and bridge immediately, so history can be reclassified
after the fact.
### Bridge View
A waterfall answering "how did this version get from its baseline to where it stands?",
one step per initiative tag, opened from the toolbar.
```
6.0k ┤ ┌──────┐- - - -┐
│ │+3,624│ │
4.0k ┤ │ │ −3,800│
│ ┌─────┐- ┘ └ - - - ┘──┐ ┌─────┐
2.0k ┤ │2,734│ │+509│ │3,067│
0 ┴──┴─────┴────────────────┴────┴─┴─────┴──
Baseline clamp give food Current
```
**Scope:** the current slice selection (default when one exists), the pivot's current
filters, or the whole version. Selection scope uses the **union** of the selected slices —
the same reach an operation would have — with rows matching more than one slice deduped
by `pf_id` to match the `OR` semantics operations use.
Computed from the Perspective table already loaded in the browser rather than from
`/api/versions/:id/bridge`, so the figures always reconcile with what the pivot is
showing. The endpoint remains for API consumers.
**Colour** encodes polarity, not identity: increases and decreases are two poles of one
scale, so it uses a validated diverging pair (blue/red, CVD ΔE 21.6 — green/red is avoided
as the classic colourblind failure) with neutral grey anchors for baseline and current.
Every bar is directly labelled and a table view gives the same numbers at full precision.
AG Grid list of log entries — user, timestamp, operation, slice, note, rows affected.
"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of Perspective table).
---
@ -1028,16 +915,14 @@ simpler fallback and is now cheap — ~25 ms.)
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
- **Export** — download forecast as CSV or push results to a reporting table. The bridge's table view is a partial stand-in for reading the numbers out, but there is no download.
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION). The bridge answers the within-version form of this question; across versions is still open.
- **Bridge drill-down** — click a step to list the adjustments behind it, or select that slice back in the pivot
- **Targeting one iter band** — make `pf_iter` part of a slice so an operation can act on, say, only the baseline rows of a selection (see Known issues)
- **Export** — download forecast as CSV or push results to a reporting table
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION)
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually.
- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema.
---
## Project Status — 2026-09-11
## Project Status — 2026-06-12
### What's working
- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo
@ -1047,30 +932,19 @@ simpler fallback and is now cheap — ~25 ms.)
- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express
- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM
- 3-step collapsible sidebar (Setup / Baseline / Forecast)
- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation
- Baseline view: version management, multi-segment baseline workbench, canvas timeline, filter builder
- Setup view: DB table browser with preview modal, source registration, col_meta editor (`dim_group`/`dim_period_col` fields included), SQL generation
- Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage
- Incremental row streaming: operation results (`RETURNING *`) applied via `pspTable.update()` — no full reload
- **Multi-slice operations**: ctrl/⌘-click accumulates slices; `apply_mode` prorate/each
- **Per-measure resolution**: target, percent or change amount independently per measure
- **`target_basis`**: a target measures against the adjustable rows or everything the pivot shows
- **Ledger panel**: baseline → adjustments → current → three interchangeable editable rows, docked bottom/right/floating
- **Tags and bridge**: initiative tags on log entries, editable after the fact, with a waterfall view scoped to selection / filters / version
- **Status bar** names the physical table writes land in, with live row counts by iter
- Slice extraction from `perspective-click` event feeds operation panel directly
- Incremental row streaming: operation results (`RETURNING *`) applied to Perspective table via `pspTable.update()` — no full reload
- Status bar: shows current source · version · baseline row count · status
### Known issues / next focus
- **`pf_iter` not selectable** — it is not a col_meta column, so it is stripped from slices. Cells differing only by iter band collapse to one slice (duplicates are detected and collapsed, and the panel warns), and there is no way to operate on one band of a slice.
- **Per-row rounding drift** — the scale SQL rounds each row to 2dp, so a target of 1,000 across many rows can land on 999.99. Inherent to proportional distribution; a correction row would be needed to land exactly.
- **Manual caret expansion is not restored** — the depth re-apply on refocus only covers whole-tree depths set via the Expand buttons or a saved layout, since per-row expansion lives in the same discarded view.
- **Bridge has no drill-down** — clicking a step does not list its adjustments or select that slice back in the pivot.
- **Light surface only** — app chrome is light throughout; the dark toggle currently re-themes only the Perspective viewer.
- **Forecast view** — operation panel SQL generation complete; UI wiring to API still needed
- **Load progress bar** — jittery at high throughput; throttle to ~10 updates/sec
- **Default pivot layout** — per-source configurable layout not yet implemented; currently hardcodes first 2 dimensions
- **No "current version" persistence** — source/version selection resets on page reload
- **Perspective slice limitation** — computed date columns (Month, YearDate) from split_by don't map back to raw rows; only native dimension columns work for slice extraction
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version.
- **No migration sequence** — `01_schema.sql` carries `ADD COLUMN IF NOT EXISTS` inline for the `tag` column, which covers fresh installs and re-runs, but there is no ordered migration mechanism.
- **No tests** — SQL generation is token substitution against append-only tables and is entirely untested.
### Fixed
- **Non-selective slices applied to the whole version** — a slice naming no filterable column reduced to `TRUE`. Now rejected on all three operations.
- **Proration across a near-zero pool** — rows flew to extreme opposite values to reach a target. Refused when the net is below 1% of gross; `apply_mode: each` is the alternative.
- **Targets overshooting by excluded rows** — see `target_basis`.

110
routes/auth.js
View File

@ -1,110 +0,0 @@
const express = require('express');
const { verifyPassword } = require('../lib/auth');
// Per-IP login throttle. In-memory on purpose: it only has to blunt online
// guessing, and a counter that resets on restart is the acceptable cost of not
// writing a failed-attempt row for every knock on an internet-facing port.
const WINDOW_MS = 15 * 60 * 1000;
const MAX_ATTEMPTS = 10;
const attempts = new Map(); // ip -> { count, resetAt }
function tooManyAttempts(ip) {
const rec = attempts.get(ip);
if (!rec || Date.now() > rec.resetAt) return false;
return rec.count >= MAX_ATTEMPTS;
}
function recordFailure(ip) {
const rec = attempts.get(ip);
if (!rec || Date.now() > rec.resetAt) {
attempts.set(ip, { count: 1, resetAt: Date.now() + WINDOW_MS });
} else {
rec.count += 1;
}
}
// Keep the map from growing without bound on a long-lived process.
setInterval(() => {
const now = Date.now();
for (const [ip, rec] of attempts) if (now > rec.resetAt) attempts.delete(ip);
}, WINDOW_MS).unref();
module.exports = function(pool) {
const router = express.Router();
router.post('/login', async (req, res) => {
const ip = req.ip;
if (tooManyAttempts(ip)) {
return res.status(429).json({ error: 'Too many failed attempts. Try again later.' });
}
const username = String(req.body?.username || '').trim();
const password = String(req.body?.password || '');
if (!username || !password) {
return res.status(400).json({ error: 'Username and password are required' });
}
try {
const result = await pool.query(
`SELECT id, username, display_name, pass_hash, is_active,
is_admin, territory
FROM pf.app_user WHERE lower(username) = lower($1)`,
[username]
);
const user = result.rows[0];
// Same message and roughly the same work either way: no unknown
// user / wrong password / disabled distinction to enumerate.
const ok = user && user.is_active && verifyPassword(password, user.pass_hash);
if (!ok) {
recordFailure(ip);
return res.status(401).json({ error: 'Invalid username or password' });
}
// New session id on login — an existing cookie can't be fixated.
req.session.regenerate(err => {
if (err) {
console.error(err);
return res.status(500).json({ error: 'Could not start session' });
}
req.session.user = {
id: user.id,
username: user.username,
display_name: user.display_name,
is_admin: !!user.is_admin,
territory: Array.isArray(user.territory) ? user.territory : [],
};
pool.query(`UPDATE pf.app_user SET last_login_at = now() WHERE id = $1`, [user.id])
.catch(e => console.error('last_login_at update failed', e));
req.session.save(err2 => {
if (err2) {
console.error(err2);
return res.status(500).json({ error: 'Could not start session' });
}
attempts.delete(ip);
res.json({ user: req.session.user });
});
});
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
router.post('/logout', (req, res) => {
const name = req.session?.cookie && req.app.get('session cookie name');
req.session.destroy(err => {
if (err) console.error(err);
res.clearCookie(name || 'pf.sid');
res.json({ ok: true });
});
});
// The UI calls this on load to decide between the login screen and the app.
router.get('/me', (req, res) => {
if (!req.session?.user) return res.status(401).json({ error: 'Not authenticated' });
res.json({ user: req.session.user });
});
return router;
};

193
routes/layouts.js
View File

@ -1,193 +0,0 @@
const express = require('express');
const { sessionUser } = require('../lib/auth');
// Named Perspective view configs. Two kinds, one table:
//
// private — yours, nobody else lists it
// published — everyone on the version sees it; only the owner or an admin
// may change it, the same rule pf.log uses for its entries
//
// Scope is the version by default, because that is where people enter the app.
// A layout with version_id NULL applies to every version of the source, which
// is where the old pf.source.default_layout went and what a brand-new version
// picks up before anyone has published anything for it.
module.exports = function(pool) {
const router = express.Router();
const SELECT_COLS = `id, source_id, version_id, name, config, owner,
visibility, is_default, created_at, updated_at`;
// What the client needs to know about a row it cannot write, so the UI can
// grey the control out rather than offer a click that answers 403.
const decorate = (row, req) => ({
...row,
scope: row.version_id == null ? 'source' : 'version',
can_edit: !!req.session?.user?.is_admin || row.owner === sessionUser(req)
});
// The owner/admin gate, shared by every write. Returns the row, or sends the
// response itself and returns null.
async function ownedLayout(req, res) {
const id = parseInt(req.params.lid);
const { rows } = await pool.query(
`SELECT ${SELECT_COLS} FROM pf.layout WHERE id = $1`, [id]
);
if (!rows.length) { res.status(404).json({ error: 'Layout not found' }); return null; }
const row = rows[0];
if (!req.session?.user?.is_admin && row.owner !== sessionUser(req)) {
res.status(403).json({
error: `“${row.name}” belongs to ${row.owner} — only they or an administrator can change it`
});
return null;
}
return row;
}
// Everything applicable to a version: its own published layouts, the
// source-wide published ones, and the caller's own private layouts at either
// scope. Deliberately not territory-filtered — a layout is display config,
// and territory restricts rows, not columns.
router.get('/versions/:id/layouts', async (req, res) => {
const versionId = parseInt(req.params.id);
try {
const ver = await pool.query(
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
);
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
const sourceId = ver.rows[0].source_id;
const { rows } = await pool.query(
`SELECT ${SELECT_COLS}
FROM pf.layout
WHERE TRUE
AND source_id = $1
AND (version_id = $2 OR version_id IS NULL)
AND (visibility = 'published' OR owner = $3)
ORDER BY visibility DESC, is_default DESC, lower(name)`,
[sourceId, versionId, sessionUser(req)]
);
res.json(rows.map(r => decorate(r, req)));
} catch (err) {
res.status(500).json({ error: err.message });
}
});
// Create. scope 'source' saves it against every version of the source;
// anything else is this version.
router.post('/versions/:id/layouts', async (req, res) => {
const versionId = parseInt(req.params.id);
const { name, config, visibility = 'private', scope = 'version' } = req.body || {};
if (!name || !String(name).trim()) return res.status(400).json({ error: 'Name is required' });
if (!config) return res.status(400).json({ error: 'Config is required' });
if (!['private', 'published'].includes(visibility)) {
return res.status(400).json({ error: 'visibility must be private or published' });
}
try {
const ver = await pool.query(
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
);
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
const { rows } = await pool.query(
`INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility)
VALUES ($1, $2, $3, $4, $5, $6)
RETURNING ${SELECT_COLS}`,
[ver.rows[0].source_id, scope === 'source' ? null : versionId,
String(name).trim(), config, sessionUser(req), visibility]
);
res.json(decorate(rows[0], req));
} catch (err) {
// the partial unique indexes, reported in the terms the user typed
if (err.code === '23505') {
return res.status(409).json({ error: `A layout named “${name}” already exists here` });
}
res.status(500).json({ error: err.message });
}
});
// Update any of name / config / visibility / is_default. Each is optional;
// a missing key leaves the column alone, which is why the flag-and-value
// pair is used rather than COALESCE on the value.
router.patch('/layouts/:lid', async (req, res) => {
const { name, config, visibility, is_default, scope } = req.body || {};
if (name === undefined && config === undefined && visibility === undefined
&& is_default === undefined && scope === undefined) {
return res.status(400).json({ error: 'Nothing to update' });
}
if (visibility !== undefined && !['private', 'published'].includes(visibility)) {
return res.status(400).json({ error: 'visibility must be private or published' });
}
const client = await pool.connect();
try {
const row = await ownedLayout(req, res);
if (!row) return;
// Unpublishing a default would leave a default nobody can see, which
// the table's CHECK refuses; clear the flag with it rather than
// failing on a constraint the user never mentioned.
const clearsDefault = visibility === 'private';
// One default per scope, enforced by a partial unique index. Stand
// the others down first rather than letting the update collide --
// and in one transaction, or a name clash on the second statement
// leaves the version with no default at all.
const target_version = scope === undefined ? row.version_id
: (scope === 'source' ? null : row.version_id);
await client.query('BEGIN');
if (is_default === true) {
await client.query(
`UPDATE pf.layout SET is_default = false
WHERE TRUE
AND source_id = $1
AND COALESCE(version_id, 0) = COALESCE($2::int, 0)
AND id <> $3
AND is_default`,
[row.source_id, target_version, row.id]
);
}
const { rows } = await client.query(
`UPDATE pf.layout SET
name = CASE WHEN $2::bool THEN $3::text ELSE name END,
config = CASE WHEN $4::bool THEN $5::jsonb ELSE config END,
visibility = CASE WHEN $6::bool THEN $7::text ELSE visibility END,
is_default = CASE WHEN $10::bool THEN false
WHEN $8::bool THEN $9::bool ELSE is_default END,
version_id = CASE WHEN $11::bool THEN $12::int ELSE version_id END,
updated_at = now()
WHERE id = $1
RETURNING ${SELECT_COLS}`,
[row.id,
name !== undefined, name !== undefined ? String(name).trim() : null,
config !== undefined, config !== undefined ? config : null,
visibility !== undefined, visibility !== undefined ? visibility : null,
is_default !== undefined, is_default === true,
clearsDefault,
scope !== undefined, target_version]
);
await client.query('COMMIT');
res.json(decorate(rows[0], req));
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
if (err.code === '23505') {
return res.status(409).json({ error: 'A layout with that name already exists here' });
}
res.status(500).json({ error: err.message });
} finally {
client.release();
}
});
router.delete('/layouts/:lid', async (req, res) => {
try {
const row = await ownedLayout(req, res);
if (!row) return;
await pool.query(`DELETE FROM pf.layout WHERE id = $1`, [row.id]);
res.json({ deleted: row.id });
} catch (err) {
res.status(500).json({ error: err.message });
}
});
return router;
};

156
routes/log.js
View File

@ -1,6 +1,5 @@
const express = require('express');
const { grainOf } = require('../lib/sql_generator');
const { sessionUser } = require('../lib/auth');
const { fcTable } = require('../lib/utils');
module.exports = function(pool) {
@ -31,96 +30,17 @@ module.exports = function(pool) {
unitsCol ? `sum(f."${unitsCol}")::float8 AS units_total` : `NULL::float8 AS units_total`
].join(', ');
// The totals are stamped onto the entry when it is written, so the
// normal read is a scan of a few dozen log rows rather than a join
// against millions of forecast rows.
//
// ?recount=1 does it the old way. Stored totals are fixed at write
// time and cannot drift on their own, but nothing stops someone
// deleting forecast rows by hand, and a stored figure has no way to
// notice. This is the way back -- and the backfill for entries
// written before the columns existed.
const recount = req.query.recount === '1' || req.query.recount === 'true';
const stamped = !recount && (await pool.query(
`SELECT count(*)::int AS n FROM pf.log
WHERE version_id = $1 AND row_count IS NULL`, [versionId]
)).rows[0].n === 0;
// ?kind=adjustments drops the baseline and reference entries. That is not
// only about what gets listed: the aggregate below joins the whole
// forecast table, and on a real version the load entries own almost
// every row of it -- 2.5M against a few thousand for the adjustments.
// Filtering in the WHERE keeps them out of the join rather than
// totalling them and discarding the answer.
const adjustmentsOnly = req.query.kind === 'adjustments';
const opFilter = adjustmentsOnly
? `AND l.operation NOT IN ('baseline', 'reference')`
: '';
const result = stamped
? await pool.query(`
SELECT l.*,
$2::text AS value_col,
$3::text AS units_col
FROM pf.log l
WHERE l.version_id = $1
${opFilter}
ORDER BY l.id DESC
`, [versionId, valueCol || null, unitsCol || null])
: await pool.query(`
SELECT l.*, ${aggCols},
$2::text AS value_col,
$3::text AS units_col
FROM pf.log l
LEFT JOIN ${table} f ON f.pf_logid = l.id
WHERE l.version_id = $1
${opFilter}
GROUP BY l.id
ORDER BY l.id DESC
`, [versionId, valueCol || null, unitsCol || null]);
// A recount is also a repair: write back what it found, so the next
// read is cheap again and the stored figure matches the rows.
if (recount) {
for (const r of result.rows) {
await pool.query(
`UPDATE pf.log SET row_count = $2, value_total = $3, units_total = $4,
measure_cols = $5::jsonb
WHERE id = $1`,
[r.id, r.row_count, r.value_total, r.units_total,
JSON.stringify({ value: valueCol || null, units: unitsCol || null })]
);
}
}
// The statement is kilobytes per entry and the list is opened to scan,
// not to read SQL. It stays in the row for the debug endpoint below.
res.json(result.rows.map(({ sql_text, ...r }) => ({
...r, has_sql: !!sql_text,
})));
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// Everything about one entry, for when the rows look wrong: what was asked
// for (params), what it ran against (env), and the statement that actually
// executed with territory and scope resolved into it (sql_text).
//
// Both the intent and the SQL, because the translation between them is
// exactly what is in doubt when a result is surprising.
router.get('/log/:logid/debug', async (req, res) => {
const logId = parseInt(req.params.logid);
try {
const { rows } = await pool.query(
`SELECT l.*, v.name AS version_name, s.schema, s.tname
FROM pf.log l
JOIN pf.version v ON v.id = l.version_id
JOIN pf.source s ON s.id = v.source_id
WHERE l.id = $1`, [logId]
);
if (!rows.length) return res.status(404).json({ error: 'Log entry not found' });
res.json(rows[0]);
const result = await pool.query(`
SELECT l.*, ${aggCols},
$2::text AS value_col,
$3::text AS units_col
FROM pf.log l
LEFT JOIN ${table} f ON f.pf_logid = l.id
WHERE l.version_id = $1
GROUP BY l.id
ORDER BY l.id DESC
`, [versionId, valueCol || null, unitsCol || null]);
res.json(result.rows);
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
@ -141,17 +61,6 @@ module.exports = function(pool) {
if (!logResult.rows.length) return res.status(404).json({ error: 'Log entry not found' });
const log = logResult.rows[0];
if (log.status === 'closed') return res.status(403).json({ error: 'Version is closed' });
// Undo deletes rows wholesale by logid, so it cannot be territory
// filtered the way a read or a write can -- half-undoing an entry
// would leave the version in a state nothing describes. Ownership
// instead: your own entries, or an admin's override. Territory alone
// would not do it anyway, since two accounts can share one.
if (!req.session?.user?.is_admin && log.pf_user !== sessionUser(req)) {
return res.status(403).json({
error: `That entry was made by ${log.pf_user || 'someone else'} — only they or an administrator can undo it`
});
}
const table = fcTable(log.tname, log.version_id);
// In grain mode the client's table is indexed on pf_gkey, so undo has to
@ -205,50 +114,13 @@ module.exports = function(pool) {
}
});
// update the note and/or tag on a log entry. Both are annotations — they never
// affect the forecast rows — so they stay editable after the fact, including on
// a closed version, where relabelling history is still legitimate.
// update the note on a log entry
router.patch('/log/:logid', async (req, res) => {
const logId = parseInt(req.params.logid);
const { note, tag, bucket, label } = req.body;
if (note === undefined && tag === undefined
&& bucket === undefined && label === undefined) {
return res.status(400).json({
error: 'Nothing to update — send note, tag, bucket and/or label'
});
}
const { note } = req.body;
try {
// Same rule as undo: your own entries, or an admin's. These are
// annotations, but label and bucket name the pivot's columns for
// everyone who opens the version, so an unguarded PATCH let any
// account rename the company's segments -- including on loads whose
// rows it cannot see.
const owner = await pool.query(
`SELECT pf_user FROM pf.log WHERE id = $1`, [logId]
);
if (!owner.rows.length) return res.status(404).json({ error: 'Log entry not found' });
if (!req.session?.user?.is_admin && owner.rows[0].pf_user !== sessionUser(req)) {
return res.status(403).json({
error: `That entry was made by ${owner.rows[0].pf_user || 'someone else'} — only they or an administrator can change it`
});
}
// COALESCE on the flag, not the value: an explicit null or '' must be
// able to clear a field, which COALESCE on the value alone would ignore
const result = await pool.query(
`UPDATE pf.log SET
note = CASE WHEN $2::bool THEN $3::text ELSE note END,
tag = CASE WHEN $4::bool THEN $5::text ELSE tag END,
bucket = CASE WHEN $6::bool THEN $7::text ELSE bucket END,
label = CASE WHEN $8::bool THEN $9::text ELSE label END
WHERE id = $1 RETURNING *`,
[
logId,
note !== undefined, note === undefined ? null : (String(note).trim() || null),
tag !== undefined, tag === undefined ? null : (String(tag).trim() || null),
bucket !== undefined, bucket === undefined ? null : (String(bucket).trim() || null),
label !== undefined, label === undefined ? null : (String(label).trim() || null),
]
`UPDATE pf.log SET note = $1 WHERE id = $2 RETURNING *`, [note ?? null, logId]
);
if (!result.rows.length) return res.status(404).json({ error: 'Log entry not found' });
res.json(result.rows[0]);

725
routes/operations.js
View File

@ -1,338 +1,14 @@
const express = require('express');
const { tableFromArrays, tableToIPC } = require('apache-arrow');
const { applyTokens, buildWhere, buildWhereAny, COMPUTED_SLICE_COLS, buildScopeClause, buildTerritoryClause, buildExcludeClause, buildExcludePredicate, buildSetClause, dateGroupsOf, dimPeriodMapOf, esc,
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR, VERSION_JOIN,
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET } = require('../lib/sql_generator');
const { sessionUser, sessionTerritory } = require('../lib/auth');
const { applyTokens, buildWhere, buildExcludeClause, buildSetClause, esc } = require('../lib/sql_generator');
const { fcTable } = require('../lib/utils');
module.exports = function(pool) {
const router = express.Router();
async function runSQL(sql, client) {
async function runSQL(sql) {
console.log('--- SQL ---\n', sql, '\n--- END SQL ---');
return (client || pool).query(sql);
}
// accept either the legacy single `slice` object or the newer `slices` array,
// and drop any empty entries so an empty selection can never widen to TRUE
function normalizeSlices(body) {
const raw = Array.isArray(body.slices) && body.slices.length ? body.slices : [body.slice];
return raw.filter(s => s && typeof s === 'object' && Object.keys(s).length > 0);
}
// Stamp the tag onto the log entry the operation just created.
// Done as a follow-up UPDATE rather than inside the generated SQL: those
// templates live in pf.sql per source, so adding a {{tag}} token would strand
// every source that has not re-run "Generate SQL".
async function tagLog(client, rows, tag) {
const clean = (tag || '').trim();
if (!clean) return null;
const ids = [...new Set(rows.map(r => r.pf_logid).filter(id => id != null))];
if (ids.length === 0) return null;
await client.query(`UPDATE pf.log SET tag = $1 WHERE id = ANY($2::bigint[])`, [clean, ids]);
return clean;
}
// How a multi-slice request is split into statements.
//
// 'each' — one statement per slice, so every slice reaches its target on
// its own and gets its own log entry
// 'prorate' — a single statement over all of them, letting the SQL's
// sum() OVER () distribute across the whole pool
//
// Only scale has a target to prorate towards; recode and clone rewrite rows
// rather than distribute an amount, so for them this only decides whether the
// work lands as one log entry or several.
// The scope is ANDed onto every unit rather than folded into the slices: it
// applies to all of them equally, and under apply_mode 'each' folding it in
// would repeat the same predicate in every statement for no gain.
function sliceUnits(slices, ctx, applyMode, scope, req) {
const vid = ctx.version.id;
const scl = buildScopeClause(scope, ctx.filterCols, vid);
const terr = req ? territoryOf(req, ctx) : '';
const and = (w) => {
let out = w;
for (const extra of [scl, terr]) {
if (!extra) continue;
out = (out === 'TRUE' || !out) ? extra : `${out}\nAND ${extra}`;
}
return out;
};
return applyMode === 'each'
? slices.map(sl => ({ slices: [sl], where: and(buildWhere(sl, ctx.filterCols, vid)) }))
: [{ slices, where: and(buildWhereAny(slices, ctx.filterCols, vid)) }];
}
// The offset is interpolated into the statement as an interval literal, so a
// typo would surface as a Postgres parse error from the middle of a CTE. Ask
// Postgres to parse it alone first, where the failure is cheap and can name the
// field it came from. Negative intervals are valid and useful -- '-90 days'
// pulls a plan back a quarter -- so this checks validity, not sign.
async function assertInterval(value, res) {
try {
await pool.query(`SELECT $1::interval`, [value]);
return true;
} catch {
res.status(400).json({
error: `"${value}" is not a valid interval. Try something like `
+ `"4 months", "1 year", "-90 days" or "0 days".`
});
return false;
}
}
// A slice is only meaningful if at least one of its keys is a filterable
// column. buildWhere silently drops unknown keys, so {"typo": "x"} would
// otherwise reduce to TRUE and apply the operation to the whole version.
// Refuse rather than let a malformed selection rewrite every row.
function assertSelective(slices, ctx) {
const allowed = new Set([...ctx.filterCols, ...Object.keys(COMPUTED_SLICE_COLS)]);
slices.forEach((sl, i) => {
const hits = Object.keys(sl).filter(k => allowed.has(k));
if (hits.length === 0) {
const err = new Error(
`Slice ${i + 1} does not name any filterable column ` +
`(${JSON.stringify(sl)}). Expected one of: ${ctx.filterCols.join(', ')}.`
);
err.status = 400;
throw err;
}
});
}
// Stamp what the entry did onto the entry itself.
//
// An indexed lookup on pf_logid, run once at write time, in place of the
// change log joining the whole forecast table on every open. Safe to store
// rather than derive because these rows never change: only this operation
// inserts them, and the only thing that removes them is undo, which deletes
// the log row too.
//
// Best-effort by design. A failure here must not roll back a write that
// succeeded -- the totals can always be recomputed, the adjustment cannot.
async function stampLogTotals(client, ctx, logId, extra = {}) {
if (!logId) return;
const v = ctx.valueCol, u = ctx.unitsCol;
// The state the write ran against, none of which can be reconstructed
// later: territory and exclude_iters are mutable rows elsewhere, and the
// template is overwritten in place every time Generate SQL runs. The
// generation timestamp is a fingerprint, not a version -- it cannot bring
// the old template back, only tell you the entry did not run under this
// one.
const env = {
territory: extra.territory ?? null,
territory_col: ctx.territoryCol || null,
exclude_iters: ctx.version.exclude_iters ?? null,
sql_generated_at: ctx.sqlGeneratedAt || null,
};
try {
await client.query(`
UPDATE pf.log SET
row_count = t.n,
value_total = t.v,
units_total = t.u,
measure_cols = $2::jsonb,
env = $3::jsonb,
sql_text = $4::text
FROM (
SELECT count(*)::int AS n
,${v ? `sum(f."${v}")::float8` : 'NULL::float8'} AS v
,${u ? `sum(f."${u}")::float8` : 'NULL::float8'} AS u
FROM ${ctx.table} f
WHERE f.pf_logid = $1
) t
WHERE pf.log.id = $1
`, [logId, JSON.stringify({ value: v || null, units: u || null }),
JSON.stringify(env), extra.sql || null]);
} catch (err) {
console.error('[stampLogTotals]', err);
}
}
// Moving a row between territories is reassignment, not forecasting, so a
// scoped account cannot set the territory column -- otherwise a rep could
// recode work into their own book, or quietly out of it, and the row would
// be gone from the view that would have shown what happened.
//
// Reads and writes are already scoped, so the *source* rows are safely the
// account's own; this is only about the destination.
function assertMayRecodeTerritory(req, ctx, set, res) {
if (!ctx.territoryCol) return true;
if (req.session?.user?.is_admin) return true;
if (!set || set[ctx.territoryCol] === undefined) return true;
res.status(403).json({
error: `Only an administrator can recode ${ctx.territoryCol} — that moves rows between territories`
});
return false;
}
// echo back what the caller asked for, for the audit log
function pickIntent(body) {
const keys = ['mode', 'target_basis', 'value_incr', 'units_incr', 'value_pct', 'units_pct', 'pct',
'target_value', 'target_units', 'target_price', 'scope'];
const out = {};
for (const k of keys) if (body[k] !== undefined && body[k] !== null && body[k] !== '') out[k] = body[k];
return out;
}
// Totals for a WHERE clause, split into the rows operations can change and the
// rows they cannot. Excluded iters (typically 'reference') are still visible in
// the pivot, so their contribution has to be reported rather than dropped —
// otherwise a target set against what the grid shows lands somewhere else.
async function sliceTotals(client, ctx, whereClause, excludeClause) {
const pred = buildExcludePredicate(ctx.version.exclude_iters);
const agg = (col, filter) => col ? `sum(${col}) FILTER (WHERE ${filter})` : 'NULL';
const v = ctx.valueCol ? `"${ctx.valueCol}"` : null;
const u = ctx.unitsCol ? `"${ctx.unitsCol}"` : null;
const r = await client.query(`
SELECT ${agg(v, pred)} AS total_value,
${agg(u, pred)} AS total_units,
${agg(v ? `abs(${v})` : null, pred)} AS abs_value,
${agg(u ? `abs(${u})` : null, pred)} AS abs_units,
${agg(v, `NOT (${pred})`)} AS excl_value,
${agg(u, `NOT (${pred})`)} AS excl_units
FROM ${ctx.table} WHERE ${whereClause}
`);
const n = (x) => parseFloat(r.rows[0][x]) || 0;
return {
value: n('total_value'), units: n('total_units'),
absValue: n('abs_value'), absUnits: n('abs_units'),
exclValue: n('excl_value'), exclUnits: n('excl_units'),
};
}
// Resolve each measure independently into the increment the scale SQL expects.
// Value and units each accept exactly one of: an absolute target, a change
// amount, or a percentage — whichever the caller sent. They are resolved
// separately so a target on one measure and a percentage on the other can be
// submitted together. Everything is measured against the totals of *this*
// WHERE clause, which is what makes apply_mode 'each' land per slice.
async function resolveIncrs(client, ctx, whereClause, excludeClause, body) {
const num = (v) => (v === undefined || v === null || v === '') ? null : parseFloat(v);
const tValue = num(body.target_value);
const tUnits = num(body.target_units);
const tPrice = num(body.target_price);
const vIncr = num(body.value_incr);
const uIncr = num(body.units_incr);
let vPct = num(body.value_pct);
let uPct = num(body.units_pct);
// legacy shape: a single `pct` flag meaning "the increments are percentages"
if (body.pct) {
if (vPct === null && vIncr !== null) vPct = vIncr;
if (uPct === null && uIncr !== null) uPct = uIncr;
}
const legacyPct = !!body.pct;
const anyInput = [tValue, tUnits, tPrice, vIncr, uIncr, vPct, uPct].some(v => v !== null);
if (!anyInput) return { value: 0, units: 0 };
const totals = await sliceTotals(client, ctx, whereClause, excludeClause);
// What the number is measured against:
// 'adjustable' — only the rows this operation can write (the default, and
// what every earlier version of this API did)
// 'selected' — everything the pivot shows for the slice, excluded rows
// included. Those rows cannot move, so reaching the target
// means the adjustable rows absorb the whole difference.
const basis = body.target_basis === 'selected' ? 'selected' : 'adjustable';
const fixedValue = basis === 'selected' ? totals.exclValue : 0;
const fixedUnits = basis === 'selected' ? totals.exclUnits : 0;
// one measure: target wins, then percentage, then a plain change amount
const resolve = (target, pct, incr, current, fixed) => {
// subtract the immovable part: current + incr + fixed === target
if (target !== null) return (target - fixed) - current;
// a percentage of the basis, which may include the immovable part
if (pct !== null) return (current + fixed) * pct / 100;
if (incr !== null && !legacyPct) return incr;
return 0;
};
let value = resolve(tValue, vPct, vIncr, totals.value, fixedValue);
let units = resolve(tUnits, uPct, uIncr, totals.units, fixedUnits);
// A price target is the "edit price" mode of the Excel form: price and
// volume are the inputs and dollars fall out of them. With a units target
// alongside it, both move; without one, volume holds and price alone carries
// the change. An explicit value target outranks it either way.
if (tPrice !== null && tValue === null) {
const targetUnits = tUnits !== null
? (tUnits - fixedUnits) + 0 // the units target is already absolute
: (totals.units + fixedUnits);
value = (tPrice * targetUnits) - (totals.value + fixedValue);
}
// Which side of price x volume absorbs a dollar change.
//
// 'price' — volume holds, so price moves. This is what the API has always
// done, and stays the default so existing callers are unaffected.
// 'volume' — price holds, so volume scales with the dollars.
//
// Only meaningful when dollars were the input and units were not given
// explicitly; naming both means the caller has already decided.
const plug = body.plug === 'volume' ? 'volume' : 'price';
const unitsGiven = [tUnits, uIncr, uPct].some(v => v !== null);
if (plug === 'volume' && value !== 0 && !unitsGiven) {
const curValue = totals.value + fixedValue;
const curUnits = totals.units + fixedUnits;
if (curValue === 0) {
const err = new Error(
'Cannot hold price constant here: the selection currently has no value, ' +
'so there is no price to hold. Scale units directly, or let price absorb ' +
'the change.'
);
err.status = 400; throw err;
}
// price constant means value and units move by the same proportion:
// fVol = curVol * (fVal / curVal), so the units delta is curVol * value/curVal
units = curUnits * (value / curValue);
}
// the scale SQL divides by the slice total; with no rows there is
// nothing to prorate across and the increment would vanish anyway
if (totals.value === 0 && totals.units === 0) return { value: 0, units: 0 };
// Refuse to prorate across a pool that nets to ~zero. Each row's new value is
// (row / total) * increment, so as the net approaches zero the multiplier
// explodes and rows fly apart in opposite directions to hit the target — a
// mathematically faithful, practically useless result. Selecting slices that
// offset each other is the usual cause, and 'each' handles that correctly.
assertProratable(totals, value, units);
return { value: round(value, 6), units: round(units, 6) };
}
// a pool is proratable only if its net is a meaningful fraction of its gross
const NET_TO_GROSS_FLOOR = 0.01;
function assertProratable(totals, value, units) {
const check = (net, gross, incr, label) => {
if (!incr) return;
if (gross === 0) return;
if (Math.abs(net) >= gross * NET_TO_GROSS_FLOOR) return;
const err = new Error(
`Cannot prorate ${label} across this selection: the rows net to ` +
`${net.toFixed(2)} against a gross of ${gross.toFixed(2)}, so they very ` +
`nearly cancel out. Scaling to a target would push them to extreme ` +
`opposite values. Use "Each" to scale every slice on its own, or narrow ` +
`the selection so it does not mix offsetting rows.`
);
err.status = 400;
throw err;
};
check(totals.value, totals.absValue, value, 'value');
check(totals.units, totals.absUnits, units, 'units');
}
function round(n, dp) {
if (!isFinite(n)) return 0;
const f = Math.pow(10, dp);
return Math.round(n * f) / f;
return pool.query(sql);
}
// fetch everything needed to execute an operation:
@ -360,7 +36,7 @@ module.exports = function(pool) {
const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
const sqlResult = await pool.query(
`SELECT sql, generated_at FROM pf.sql WHERE source_id = $1 AND operation = $2`,
`SELECT sql FROM pf.sql WHERE source_id = $1 AND operation = $2`,
[version.source_id, operation]
);
if (sqlResult.rows.length === 0) {
@ -377,26 +53,10 @@ module.exports = function(pool) {
filterCols: [...dimCols, ...dateCols],
valueCol,
unitsCol,
territoryCol: colMeta.find(c => c.is_territory)?.cname || null,
sql: sqlResult.rows[0].sql,
sqlGeneratedAt: sqlResult.rows[0].generated_at
sql: sqlResult.rows[0].sql
};
}
// Every read and every write goes through this, so a scoped account cannot
// reach a row outside its territory by any route. ANDed on last, after the
// slice and the client's own scope, where nothing in the request can undo
// it.
function territoryOf(req, ctx) {
return buildTerritoryClause(sessionTerritory(req), ctx.territoryCol);
}
function andTerritory(where, req, ctx) {
const t = territoryOf(req, ctx);
if (!t) return where;
return (!where || where === 'TRUE') ? t : `${where}\nAND ${t}`;
}
function guardOpen(version, res) {
if (version.status === 'closed') {
res.status(403).json({ error: 'Version is closed' });
@ -419,19 +79,7 @@ module.exports = function(pool) {
}
const tbl = fcTable(verResult.rows[0].tname, versionId);
// /data does not go through getContext, so it resolves the territory
// column itself. The count is scoped too, or the progress bar
// promises rows this account will never be sent.
const terrCol = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[verResult.rows[0].source_id]
)).rows[0]?.cname || null;
const territory = sessionTerritory(req);
const terrBare = buildTerritoryClause(territory, terrCol);
const terrAlias = buildTerritoryClause(territory, terrCol, 't');
const terrWhere = terrBare ? `WHERE ${terrBare}` : '';
const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl} ${terrWhere}`);
const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl}`);
const rowCount = parseInt(count);
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
@ -443,14 +91,7 @@ module.exports = function(pool) {
await client.query('BEGIN');
await client.query(`
DECLARE pf_cur CURSOR FOR
SELECT t.*
,${SEGMENT_EXPR} AS pf_segment
,${BUCKET_EXPR} AS pf_bucket
,${NOTE_EXPR} AS pf_note
FROM ${tbl} t
LEFT JOIN pf.log l
ON l.id = t.pf_logid${VERSION_JOIN}
${terrAlias ? `WHERE ${terrAlias}` : ''}
SELECT * FROM ${tbl}
`);
// Accumulate into column arrays (not row objects) to avoid allocating one JS
@ -494,13 +135,7 @@ module.exports = function(pool) {
router.get('/versions/:id/agg', async (req, res) => {
try {
const ctx = await getContext(parseInt(req.params.id), 'get_agg');
// Before the GROUP BY, not after: the territory column need not be
// part of the grain, so an aggregated row may not carry it at all.
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
territory_clause:
buildTerritoryClause(sessionTerritory(req), ctx.territoryCol, 't') || 'TRUE',
});
const sql = applyTokens(ctx.sql, { fc_table: ctx.table });
const { rows } = await runSQL(sql);
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
@ -525,10 +160,8 @@ module.exports = function(pool) {
// load baseline rows from source table — additive, no delete
router.post('/versions/:id/baseline', async (req, res) => {
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
const pf_user = sessionUser(req);
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const dateOffset = date_offset || '0 days';
if (!await assertInterval(dateOffset, res)) return;
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
try {
const ctx = await getContext(parseInt(req.params.id), 'baseline');
@ -543,16 +176,12 @@ module.exports = function(pool) {
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
label: esc(label || ''),
bucket: esc(bucket || ''),
tag: esc(tag || ''),
params: esc(paramsJson),
filter_clause: filterClause,
date_offset: esc(dateOffset)
});
const result = await runSQL(sql);
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json(result.rows[0]);
} catch (err) {
console.error(err);
@ -566,10 +195,8 @@ module.exports = function(pool) {
router.put('/versions/:id/baseline/:logid', async (req, res) => {
const versionId = parseInt(req.params.id);
const logid = parseInt(req.params.logid);
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
const pf_user = sessionUser(req);
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const dateOffset = date_offset || '0 days';
if (!await assertInterval(dateOffset, res)) return;
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
const client = await pool.connect();
@ -605,21 +232,11 @@ module.exports = function(pool) {
date_offset: dateOffset,
...(raw_where ? { raw_where } : (filters ? { filters } : {}))
});
// This route deletes the log row and inserts a fresh one, so every
// annotation on it has to be handed back or it is lost. `??`, not `||`:
// an empty string is the form clearing a field on purpose, undefined is
// the form not carrying it at all -- the segment form has no tag input,
// so tag is always the latter and must survive an edit made for any
// other reason.
const keep = (sent, prior) => esc(sent ?? prior ?? '');
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
label: keep(label, oldLog.label),
bucket: keep(bucket, oldLog.bucket),
tag: keep(tag, oldLog.tag),
params: esc(paramsJson),
filter_clause: filterClause,
date_offset: esc(dateOffset)
@ -633,7 +250,6 @@ module.exports = function(pool) {
await client.query(`DELETE FROM pf.log WHERE id = $1`, [logid]);
const insResult = await client.query(sql);
await client.query('COMMIT');
await stampLogTotals(pool, ctx, insResult.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json({
rows_deleted: delRows.rowCount,
@ -685,8 +301,7 @@ module.exports = function(pool) {
// load reference rows from source table (additive — does not clear prior reference rows)
router.post('/versions/:id/reference', async (req, res) => {
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
const pf_user = sessionUser(req);
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const dateOffset = date_offset || '0 days';
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
try {
@ -702,16 +317,12 @@ module.exports = function(pool) {
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
label: esc(label || ''),
bucket: esc(bucket || ''),
tag: esc(tag || ''),
params: esc(paramsJson),
filter_clause: filterClause,
date_offset: esc(dateOffset)
});
const result = await runSQL(sql);
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json(result.rows[0]);
} catch (err) {
console.error(err);
@ -719,275 +330,131 @@ module.exports = function(pool) {
}
});
// scale one or more slices — adjust value and/or units toward an absolute
// target or by an increment. With several slices selected, apply_mode decides
// whether they are treated as one pool ('prorate') or independently ('each').
// scale a slice — adjust value and/or units by absolute amount or percentage
router.post('/versions/:id/scale', async (req, res) => {
const { note, apply_mode } = req.body;
const pf_user = sessionUser(req);
const slices = normalizeSlices(req.body);
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
const applyMode = apply_mode === 'each' ? 'each' : 'prorate';
const { pf_user, note, slice, value_incr, units_incr, pct } = req.body;
if (!slice || Object.keys(slice).length === 0) {
return res.status(400).json({ error: 'slice is required' });
}
try {
const ctx = await getContext(parseInt(req.params.id), 'scale');
if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
// 'prorate' pools every slice into one WHERE and lets the SQL's
// sum() OVER () distribute the increment across the whole pool.
// 'each' runs the same statement once per slice, so every slice
// reaches the target on its own and gets its own log entry.
const units = sliceUnits(slices, ctx, applyMode, req.body.scope, req);
let absValueIncr = value_incr || 0;
let absUnitsIncr = units_incr || 0;
const client = await pool.connect();
let committed = false;
try {
await client.query('BEGIN');
const allRows = [];
// the statement that produced each entry, for pf.log.sql_text
const sqlByLogId = new Map();
let applied = 0;
const skipped = [];
for (const unit of units) {
const incr = await resolveIncrs(client, ctx, unit.where, excludeClause, req.body);
// no rows, or already at the target — nothing to write for this slice
if (incr.value === 0 && incr.units === 0) { skipped.push(...unit.slices); continue; }
applied++;
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({
slices: unit.slices,
apply_mode: applyMode,
...pickIntent(req.body),
resolved: { value_incr: incr.value, units_incr: incr.units }
})),
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
value_incr: incr.value,
units_incr: incr.units
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
if (allRows.length === 0) {
await client.query('ROLLBACK');
return res.status(400).json({
error: 'Nothing to scale — the target matches the current total, or the increment is zero'
});
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'scale' }));
res.json({
rows,
rows_affected: rows.length,
slices_applied: applied,
...(skipped.length ? { slices_skipped: skipped } : {})
});
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
// pct mode: run a quick totals query, convert percentages to absolutes
if (pct && (value_incr || units_incr)) {
const totals = await pool.query(`
SELECT
sum("${ctx.valueCol}") AS total_value,
sum("${ctx.unitsCol}") AS total_units
FROM ${ctx.table}
WHERE ${whereClause}
${excludeClause}
`);
const { total_value, total_units } = totals.rows[0];
if (value_incr) absValueIncr = (parseFloat(total_value) || 0) * value_incr / 100;
if (units_incr) absUnitsIncr = (parseFloat(total_units) || 0) * units_incr / 100;
}
if (absValueIncr === 0 && absUnitsIncr === 0) {
return res.status(400).json({ error: 'value_incr and/or units_incr must be non-zero' });
}
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slice, value_incr, units_incr, pct })),
slice: esc(JSON.stringify(slice)),
where_clause: whereClause,
exclude_clause: excludeClause,
value_incr: absValueIncr,
units_incr: absUnitsIncr
});
const result = await runSQL(sql);
const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'scale' }));
res.json({ rows, rows_affected: rows.length });
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// recode dimension values on one or more slices
// recode dimension values on a slice
// inserts negative rows to zero out the original, positive rows with new dimension values
router.post('/versions/:id/recode', async (req, res) => {
const { note, set, apply_mode } = req.body;
const pf_user = sessionUser(req);
const slices = normalizeSlices(req.body);
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
const { pf_user, note, slice, set } = req.body;
if (!slice || Object.keys(slice).length === 0) return res.status(400).json({ error: 'slice is required' });
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
try {
const ctx = await getContext(parseInt(req.params.id), 'recode');
if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
if (!assertMayRecodeTerritory(req, ctx, set, res)) return;
const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set);
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
const client = await pool.connect();
let committed = false;
try {
await client.query('BEGIN');
const allRows = [];
// the statement that produced each entry, for pf.log.sql_text
const sqlByLogId = new Map();
for (const unit of units) {
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slices: unit.slices, set, apply_mode: unit.mode })),
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
set_clause: setClause
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'recode' }));
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
}
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slice, set })),
slice: esc(JSON.stringify(slice)),
where_clause: whereClause,
exclude_clause: excludeClause,
set_clause: setClause
});
const result = await runSQL(sql);
const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'recode' }));
res.json({ rows, rows_affected: rows.length });
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// clone one or more slices as new business under new dimension values
// clone a slice as new business under new dimension values
// does not offset the original slice
router.post('/versions/:id/clone', async (req, res) => {
const { note, set, scale, apply_mode, from_logid, date_offset } = req.body;
const pf_user = sessionUser(req);
const slices = normalizeSlices(req.body);
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
const { pf_user, note, slice, set, scale } = req.body;
if (!slice || Object.keys(slice).length === 0) return res.status(400).json({ error: 'slice is required' });
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
try {
const ctx = await getContext(parseInt(req.params.id), 'clone');
if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
const dateOffset = (date_offset || '0 days').trim() || '0 days';
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set);
if (!await assertInterval(dateOffset, res)) return;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slice, set, scale: scaleFactor })),
slice: esc(JSON.stringify(slice)),
where_clause: whereClause,
exclude_clause: excludeClause,
set_clause: setClause,
scale_factor: scaleFactor
});
// exclude_iters deliberately does not apply here. It exists to stop
// operations *modifying* reference rows: scale would attribute forecast
// movement to prior-year rows by distributing across them, and recode
// writes negative rows that zero the original out. Clone does neither --
// it reads rows and writes new pf_iter = 'clone' rows, leaving the source
// untouched. Copying a plan or a prior year out of reference and into
// adjustments is the operation working as intended, and excluding them
// meant a visible, deliberate selection silently produced nothing.
//
// from_logid narrows instead: a selection spanning AOP and Prior Year
// where only one is wanted.
let excludeClause = '';
if (from_logid != null) {
const srcLog = await pool.query(
`SELECT id FROM pf.log WHERE id = $1 AND version_id = $2`,
[parseInt(from_logid), ctx.version.id]
);
if (!srcLog.rows.length) {
return res.status(400).json({ error: `No log entry ${from_logid} on this version` });
}
excludeClause = `AND pf_logid = ${parseInt(from_logid)}`;
}
// Period dimensions come from the calendar against the shifted date, not
// from the row being copied -- otherwise a mix moved forward a year
// keeps last year's period labels. An explicit set wins over both.
const dateGroups = dateGroupsOf(ctx.colMeta);
const derivedExprs = Object.fromEntries(
[...dimPeriodMapOf(dateGroups)].map(([cname, { alias, periodCol }]) =>
[cname, `${alias}."${periodCol}"`])
);
const setClause = buildSetClause(ctx.dimCols, set, { derivedExprs, alias: 's' });
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
const client = await pool.connect();
let committed = false;
try {
await client.query('BEGIN');
const allRows = [];
// the statement that produced each entry, for pf.log.sql_text
const sqlByLogId = new Map();
for (const unit of units) {
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({
slices: unit.slices, set, scale: scaleFactor, apply_mode: unit.mode,
date_offset: dateOffset,
...(from_logid != null ? { from_logid: parseInt(from_logid) } : {}),
})),
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
set_clause: setClause,
scale_factor: scaleFactor,
date_offset: esc(dateOffset)
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'clone' }));
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
}
const result = await runSQL(sql);
const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'clone' }));
res.json({ rows, rows_affected: rows.length });
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });

224
routes/sources.js
View File

@ -1,8 +1,5 @@
const express = require('express');
const { generateSQL, buildTerritoryClause } = require('../lib/sql_generator');
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
const { sessionTerritory } = require('../lib/auth');
const { sessionUser } = require('../lib/auth');
const { generateSQL } = require('../lib/sql_generator');
module.exports = function(pool) {
const router = express.Router();
@ -23,8 +20,7 @@ module.exports = function(pool) {
// register a source table
// auto-populates col_meta from information_schema with role='ignore'
router.post('/sources', async (req, res) => {
const { schema, tname, label } = req.body;
const created_by = sessionUser(req);
const { schema, tname, label, created_by } = req.body;
if (!schema || !tname) {
return res.status(400).json({ error: 'schema and tname are required' });
}
@ -43,14 +39,15 @@ module.exports = function(pool) {
);
const source = src.rows[0];
// seed col_meta from the source's real columns
// seed col_meta from information_schema
await client.query(`
INSERT INTO pf.col_meta (source_id, cname, role, opos)
SELECT $3, column_name, 'dimension', ordinal_position
FROM (${RELATION_COLUMNS_SQL}) c
SELECT $1, column_name, 'dimension', ordinal_position
FROM information_schema.columns
WHERE table_schema = $2 AND table_name = $3
ORDER BY ordinal_position
ON CONFLICT (source_id, cname) DO NOTHING
`, [schema, tname, source.id]);
`, [source.id, schema, tname]);
await client.query('COMMIT');
res.status(201).json(source);
@ -87,23 +84,13 @@ module.exports = function(pool) {
if (!Array.isArray(cols)) {
return res.status(400).json({ error: 'body must be an array' });
}
// Exactly one per source: the scope is a single IN list against a single
// column, and two flagged would silently mean whichever one a .find()
// reached first -- the trap is_key already fell into (see CLAUDE.md).
const territoryCols = cols.filter(c => c.is_territory).map(c => c.cname);
if (territoryCols.length > 1) {
return res.status(400).json({
error: `Only one column can be the territory. Flagged: ${territoryCols.join(', ')}`
});
}
const client = await pool.connect();
try {
await client.query('BEGIN');
for (const col of cols) {
await client.query(`
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, in_grain, is_territory, opos)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, in_grain, opos)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT (source_id, cname) DO UPDATE SET
label = EXCLUDED.label,
role = EXCLUDED.role,
@ -111,7 +98,6 @@ module.exports = function(pool) {
dim_group = EXCLUDED.dim_group,
dim_period_col = EXCLUDED.dim_period_col,
in_grain = EXCLUDED.in_grain,
is_territory = EXCLUDED.is_territory,
opos = EXCLUDED.opos
`, [
sourceId,
@ -122,7 +108,6 @@ module.exports = function(pool) {
col.dim_group || null,
col.dim_period_col || null,
col.in_grain || false,
col.is_territory || false,
col.opos || null
]);
}
@ -238,38 +223,10 @@ module.exports = function(pool) {
return res.status(400).json({ error: `"${col}" is not a key column` });
}
// ?q= narrows, ?limit= caps. A key column can be very wide -- part on
// osm_skinny has 11,290 distinct values -- so returning the lot to fill a
// completion list is both a slow query and a large response for a control
// that can only usefully show a handful.
const { schema, tname } = srcResult.rows[0];
const q = (req.query.q || '').trim();
const limit = Math.min(parseInt(req.query.limit) || 5000, 5000);
const params = [];
let filter = `WHERE "${col}" IS NOT NULL`;
// Completion reads the *source* table, which no territory scope has
// touched -- so without this a scoped account could enumerate every
// customer, part and rep in the business from a dropdown, having
// been shown none of their rows.
const terrRow = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[req.params.id]
)).rows[0];
const terrClause = buildTerritoryClause(sessionTerritory(req), terrRow?.cname || null);
if (terrClause) filter += ` AND ${terrClause}`;
if (q) {
params.push(`%${q}%`);
filter += ` AND "${col}"::text ILIKE $${params.length}`;
}
params.push(limit);
const result = await pool.query(
`SELECT DISTINCT "${col}"::text AS val FROM ${schema}.${tname}
${filter} ORDER BY 1 LIMIT $${params.length}`,
params
`SELECT DISTINCT "${col}" AS val FROM ${schema}.${tname}
WHERE "${col}" IS NOT NULL ORDER BY "${col}"`
);
res.json(result.rows.map(r => r.val));
} catch (err) {
@ -278,142 +235,6 @@ module.exports = function(pool) {
}
});
// Resolve a dim_group to its key column and siblings, or explain why it cannot be.
async function resolveGroup(sourceId, group) {
const { rows: meta } = await pool.query(
`SELECT * FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`, [sourceId]);
const members = meta.filter(c => c.dim_group === group);
if (!members.length) {
const err = new Error(`No columns are grouped as "${group}" on this source`);
err.status = 404; throw err;
}
const keyCol = members.find(c => c.is_key);
if (!keyCol) {
const err = new Error(
`Group "${group}" has no is_key column, so its members have nothing to be keyed on`);
err.status = 400; throw err;
}
return {
keyCol,
siblings: members.filter(c => c.cname !== keyCol.cname),
// recency column: the source's primary date, the same one the generator
// treats as the date for loads
dateCol: meta.find(c => c.role === 'date')?.cname || null,
};
}
// The member list for a group, as one array. Small enough to send whole --
// 11,290 parts on osm_skinny -- so the client holds it and filters locally
// instead of querying per keystroke.
router.get('/sources/:id/dim/:group', async (req, res) => {
try {
const sourceId = parseInt(req.params.id);
const { keyCol, siblings } = await resolveGroup(sourceId, req.params.group);
const includeInactive = req.query.all === '1';
const { rows } = await pool.query(`
SELECT key_value, attrs, is_active, source_seen
FROM pf.dim_member
WHERE source_id = $1 AND dim_group = $2
${includeInactive ? '' : 'AND is_active'}
ORDER BY key_value
`, [sourceId, req.params.group]);
res.json({
group: req.params.group,
key_col: keyCol.cname,
siblings: siblings.map(c => c.cname),
members: rows,
});
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// Rebuild a group's members from the source. A merge, not a replace: curation
// (a member deactivated by hand, or added before it ever sold) has to survive a
// refresh, so absent members are marked source_seen = false rather than deleted.
//
// Slow by nature -- it reads the whole source, which for a view over a
// transaction table is millions of rows -- so it is a deliberate action rather
// than something that happens on a page load.
router.post('/sources/:id/dim/:group/refresh', async (req, res) => {
const sourceId = parseInt(req.params.id);
const group = req.params.group;
try {
const srcResult = await pool.query(
`SELECT schema, tname FROM pf.source WHERE id = $1`, [sourceId]);
if (!srcResult.rows.length) return res.status(404).json({ error: 'Source not found' });
const { schema, tname } = srcResult.rows[0];
const { keyCol, siblings, dateCol } = await resolveGroup(sourceId, group);
if (!siblings.length) {
return res.status(400).json({ error: `Group "${group}" has no sibling columns to store` });
}
const q = (n) => `"${n}"`;
const attrs = siblings.map(c => `'${c.cname}', s.${q(c.cname)}::text`).join(', ');
// A key can carry more than one attribute set across history -- 11,290
// parts against 13,662 combinations on osm_skinny. Take the most recent
// by the source's date column, which is the live definition.
const recency = dateCol ? `s.${q(dateCol)} DESC NULLS LAST` : `1`;
const started = Date.now();
// An explicit stamp rather than now(): inside a transaction now() is the
// transaction's start time, so "refreshed in this run" and "refreshed in
// a run that began at the same instant" would be indistinguishable.
const runAt = new Date();
const client = await pool.connect();
try {
await client.query('BEGIN');
const { rows: [{ n }] } = await client.query(`
WITH ranked AS (
SELECT s.${q(keyCol.cname)}::text AS key_value,
jsonb_build_object(${attrs}) AS attrs,
row_number() OVER (
PARTITION BY s.${q(keyCol.cname)} ORDER BY ${recency}
) AS rn
FROM ${q(schema)}.${q(tname)} s
WHERE s.${q(keyCol.cname)} IS NOT NULL
)
,upserted AS (
INSERT INTO pf.dim_member
(source_id, dim_group, key_value, attrs, refreshed_at, source_seen)
SELECT $1, $2, key_value, attrs, $3, true
FROM ranked WHERE rn = 1
ON CONFLICT (source_id, dim_group, key_value) DO UPDATE SET
attrs = EXCLUDED.attrs,
source_seen = true,
refreshed_at = $3,
updated_at = now()
RETURNING 1
)
SELECT count(*)::int AS n FROM upserted
`, [sourceId, group, runAt]);
// anything this run did not touch is no longer in the source
const { rowCount: dropped } = await client.query(`
UPDATE pf.dim_member
SET source_seen = false, updated_at = now()
WHERE source_id = $1 AND dim_group = $2 AND source_seen
AND refreshed_at IS DISTINCT FROM $3
`, [sourceId, group, runAt]);
await client.query('COMMIT');
res.json({ group, members: n, no_longer_in_source: dropped, ms: Date.now() - started });
} catch (err) {
await client.query('ROLLBACK');
throw err;
} finally {
client.release();
}
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// given a key column value, look up sibling dim_group column values from source
// returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous
router.get('/sources/:id/lookup', async (req, res) => {
@ -449,12 +270,23 @@ module.exports = function(pool) {
}
});
// PUT /sources/:id/default-layout is gone. It wrote pf.source.default_layout,
// one anonymous blob per source that any account could overwrite for every
// other account -- a published layout with no owner. Its successor is
// pf.layout: named, owned, and writable only by its owner or an admin. The
// old column is left in place, already migrated into pf.layout by
// setup_sql/01_schema.sql, and read by nothing.
// set or clear the default Perspective layout for a source.
// Body: a Perspective view config (group_by, split_by, columns, plugin_config, …).
// Pass null or {} to clear.
router.put('/sources/:id/default-layout', async (req, res) => {
try {
const layout = req.body && Object.keys(req.body).length > 0 ? req.body : null;
const result = await pool.query(
`UPDATE pf.source SET default_layout = $1 WHERE id = $2 RETURNING *`,
[layout, req.params.id]
);
if (result.rows.length === 0) return res.status(404).json({ error: 'Source not found' });
res.json(result.rows[0]);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// deregister a source — does not drop existing forecast tables
router.get('/dim-period/cols', async (req, res) => {

26
routes/tables.js
View File

@ -1,5 +1,4 @@
const express = require('express');
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
module.exports = function(pool) {
const router = express.Router();
@ -8,18 +7,15 @@ module.exports = function(pool) {
router.get('/tables', async (req, res) => {
try {
const result = await pool.query(`
-- pg_class, not information_schema.tables, which omits
-- materialized views: gs.osm_skinny is one, and the browser
-- could not offer what the app is already built on.
SELECT
n.nspname AS schema,
c.relname AS tname,
t.table_schema AS schema,
t.table_name AS tname,
c.reltuples::bigint AS row_estimate
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r', 'v', 'm', 'f', 'p')
AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pf')
ORDER BY n.nspname, c.relname
FROM information_schema.tables t
LEFT JOIN pg_namespace n ON n.nspname = t.table_schema
LEFT JOIN pg_class c ON c.relname = t.table_name AND c.relnamespace = n.oid
WHERE t.table_schema NOT IN ('pg_catalog', 'information_schema', 'pf')
ORDER BY t.table_schema, t.table_name
`);
res.json(result.rows);
} catch (err) {
@ -35,8 +31,12 @@ module.exports = function(pool) {
return res.status(400).json({ error: 'Invalid schema or table name' });
}
try {
const cols = await pool.query(
`${RELATION_COLUMNS_SQL} ORDER BY ordinal_position`, [schema, tname]);
const cols = await pool.query(`
SELECT column_name, data_type, is_nullable, ordinal_position
FROM information_schema.columns
WHERE table_schema = $1 AND table_name = $2
ORDER BY ordinal_position
`, [schema, tname]);
const rows = await pool.query(
`SELECT * FROM ${schema}.${tname} LIMIT 5`

254
routes/versions.js
View File

@ -1,7 +1,5 @@
const express = require('express');
const { fcTable, mapType, RELATION_COLUMNS_SQL } = require('../lib/utils');
const { sessionUser, sessionTerritory } = require('../lib/auth');
const { buildTerritoryClause } = require('../lib/sql_generator');
const { fcTable, mapType } = require('../lib/utils');
module.exports = function(pool) {
const router = express.Router();
@ -24,8 +22,7 @@ module.exports = function(pool) {
// inserts version row, then CREATE TABLE pf.fc_{tname}_{version_id} in one transaction
router.post('/sources/:id/versions', async (req, res) => {
const sourceId = parseInt(req.params.id);
const { name, description, exclude_iters } = req.body;
const created_by = sessionUser(req);
const { name, description, created_by, exclude_iters } = req.body;
if (!name) return res.status(400).json({ error: 'name is required' });
const client = await pool.connect();
@ -39,9 +36,8 @@ module.exports = function(pool) {
}
const source = srcResult.rows[0];
// col_meta joined to the source's real columns, for the data types
// fetch col_meta joined to information_schema for data types
const colResult = await client.query(`
WITH cols AS (${RELATION_COLUMNS_SQL})
SELECT
m.cname,
m.role,
@ -50,11 +46,14 @@ module.exports = function(pool) {
i.numeric_precision,
i.numeric_scale
FROM pf.col_meta m
JOIN cols i ON i.column_name = m.cname
WHERE m.source_id = $3
JOIN information_schema.columns i
ON i.table_schema = $2
AND i.table_name = $3
AND i.column_name = m.cname
WHERE m.source_id = $1
AND m.role NOT IN ('ignore')
ORDER BY m.opos
`, [source.schema, source.tname, sourceId]);
`, [sourceId, source.schema, source.tname]);
if (colResult.rows.length === 0) {
return res.status(400).json({
@ -101,14 +100,6 @@ ${colDefs},
`;
await client.query(ddl);
// pf_logid is how every entry-level operation finds its rows: undo
// deletes by it, the change log aggregates by it, and it is part of the
// grain key. Without an index each of those is a sequential scan of the
// whole forecast table -- 2.5M rows to total two adjustments.
await client.query(
`CREATE INDEX ${table.split('.').pop()}_logid_idx ON ${table} (pf_logid)`
);
await client.query('COMMIT');
res.status(201).json({ ...version, fc_table: table });
} catch (err) {
@ -123,235 +114,22 @@ ${colDefs},
}
});
// where this version's writes actually land: the physical forecast table,
// its current row count, and the source table rows are read from.
// Surfaced in the status bar so the write target is never a mystery.
// Distinct values of a dimension as they appear in one version's forecast
// table, for completing recode and clone.
//
// Deliberately not the source: the source view reaches back over all of
// history, so completing from it offers parts discontinued years ago. The
// version holds what was actually loaded, which is what a forecast is being
// written against.
//
// Held in memory because the scan is not cheap -- 2.0s for 11,290 parts across
// 2.5M rows on fc_osm_skinny_29 -- and completion is typed into. Keyed on the
// version's latest log id, so any load, adjustment or undo rebuilds it on the
// next request without anything having to remember to invalidate.
const valueCache = new Map();
router.get('/versions/:id/values/:col', async (req, res) => {
const versionId = parseInt(req.params.id);
const col = req.params.col;
try {
const verResult = await pool.query(`
SELECT v.id, s.tname, s.id AS source_id
FROM pf.version v JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [versionId]);
if (!verResult.rows.length) return res.status(404).json({ error: 'Version not found' });
const { tname, source_id } = verResult.rows[0];
// the column name is interpolated, so it has to be one col_meta names
const okCol = await pool.query(`
SELECT 1 FROM pf.col_meta
WHERE source_id = $1 AND cname = $2 AND role IN ('dimension', 'date')
`, [source_id, col]);
if (!okCol.rows.length) {
return res.status(400).json({ error: `"${col}" is not a dimension on this source` });
}
const table = fcTable(tname, versionId);
const { rows: [{ rev }] } = await pool.query(
`SELECT coalesce(max(id), 0)::text AS rev FROM pf.log WHERE version_id = $1`,
[versionId]
);
const key = `${versionId}:${col}`;
let entry = valueCache.get(key);
if (!entry || entry.rev !== rev) {
const { rows } = await pool.query(
`SELECT DISTINCT "${col}"::text AS val FROM ${table}
WHERE "${col}" IS NOT NULL ORDER BY 1`
);
entry = { rev, values: rows.map(r => r.val) };
valueCache.set(key, entry);
}
const q = (req.query.q || '').trim().toLowerCase();
const limit = Math.min(parseInt(req.query.limit) || 50, 500);
const picked = q
? entry.values.filter(v => v.toLowerCase().includes(q))
: entry.values;
res.json(picked.slice(0, limit));
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
router.get('/versions/:id/table-info', async (req, res) => {
try {
const verResult = await pool.query(`
SELECT v.id, v.name, v.status, s.schema, s.tname, s.id AS source_id
FROM pf.version v
JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [req.params.id]);
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
const v = verResult.rows[0];
const fc = fcTable(v.tname, v.id);
const [schema, table] = fc.split('.');
const existsResult = await pool.query(
`SELECT to_regclass($1) IS NOT NULL AS exists`, [fc]
);
const exists = existsResult.rows[0].exists;
// Scoped like every other read. Unscoped it reported the whole
// table to an account that can see a fraction of it -- the status
// bar's row count, and the figure the load progress promises, both
// came from here: a rep with 330k rows was told the load was
// fetching 2.8M.
const terrCol = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[v.source_id]
)).rows[0]?.cname || null;
const terr = buildTerritoryClause(sessionTerritory(req), terrCol);
let rows = null, byIter = [];
if (exists) {
const countResult = await pool.query(
`SELECT pf_iter, count(*)::int AS n FROM ${fc}
${terr ? `WHERE ${terr}` : ''}
GROUP BY pf_iter ORDER BY pf_iter`
);
byIter = countResult.rows;
rows = byIter.reduce((a, r) => a + r.n, 0);
}
res.json({
version_id: v.id,
version_name: v.name,
status: v.status,
source: `${v.schema}.${v.tname}`,
fc_table: fc,
fc_schema: schema,
fc_tname: table,
exists,
rows,
by_iter: byIter
});
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// Tags already used on this source, newest first — feeds the tag autocomplete.
// Scoped to the source rather than the version so an initiative name carries
// across versions, which is the point of naming it.
router.get('/sources/:id/tags', async (req, res) => {
try {
const result = await pool.query(`
SELECT l.tag,
count(*)::int AS uses,
max(l.stamp) AS last_used
FROM pf.log l
JOIN pf.version v ON v.id = l.version_id
WHERE v.source_id = $1 AND l.tag IS NOT NULL AND l.tag <> ''
GROUP BY l.tag
ORDER BY max(l.stamp) DESC
`, [req.params.id]);
res.json(result.rows);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// Bridge: how this version got from its baseline to where it stands, grouped by
// initiative. Amounts come from the version's own forecast table, so the figures
// reconcile with the pivot rather than being recomputed from the log's params.
router.get('/versions/:id/bridge', async (req, res) => {
try {
const verResult = await pool.query(`
SELECT v.id, v.exclude_iters, s.schema, s.tname, s.id AS source_id
FROM pf.version v JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [req.params.id]);
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
const v = verResult.rows[0];
const fc = fcTable(v.tname, v.id);
const exists = await pool.query(`SELECT to_regclass($1) IS NOT NULL AS ok`, [fc]);
if (!exists.rows[0].ok) return res.json({ fc_table: fc, exists: false, rows: [] });
const colResult = await pool.query(
`SELECT cname, role FROM pf.col_meta WHERE source_id = $1`, [v.source_id]);
const valueCol = colResult.rows.find(c => c.role === 'value')?.cname;
const unitsCol = colResult.rows.find(c => c.role === 'units')?.cname;
if (!valueCol) return res.status(400).json({ error: 'No value column configured' });
const excl = (v.exclude_iters || []).length
? `t.pf_iter NOT IN (${v.exclude_iters.map(i => `'${String(i).replace(/'/g, "''")}'`).join(', ')})`
: 'TRUE';
const result = await pool.query(`
SELECT CASE WHEN t.pf_iter = 'baseline' THEN '(baseline)'
ELSE coalesce(nullif(l.tag, ''), '(untagged)') END AS tag,
bool_or(t.pf_iter = 'baseline') AS is_baseline,
count(DISTINCT l.id)::int AS entries,
count(*)::int AS row_count,
round(sum(t."${valueCol}")::numeric, 2) AS value
${unitsCol ? `, round(sum(t."${unitsCol}")::numeric, 2) AS units` : ''}
FROM ${fc} t
LEFT JOIN pf.log l ON l.id = t.pf_logid
WHERE ${excl}
GROUP BY 1
ORDER BY bool_or(t.pf_iter = 'baseline') DESC, min(l.id)
`);
res.json({ fc_table: fc, exists: true, value_col: valueCol, units_col: unitsCol, rows: result.rows });
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// update version name, description, exclude_iters, or the fallback display
// names.
//
// bucket_order is deliberately not settable: the bucket column order is the
// text in pf.log.bucket now, so a stored order would be a second answer to
// the same question, and a silent one -- nothing reads it.
//
// The three name columns are flag-and-value pairs rather than COALESCE:
// clearing one back to the built-in means writing null, which COALESCE on
// the value alone cannot tell from "not mentioned".
// update version name, description, or exclude_iters
router.put('/versions/:id', async (req, res) => {
const { name, description, exclude_iters,
adjustment_segment, adjustment_bucket, unlabeled_load } = req.body;
const set = (v) => (v === undefined ? null : (String(v).trim() || null));
const { name, description, exclude_iters } = req.body;
try {
const result = await pool.query(`
UPDATE pf.version SET
name = COALESCE($2, name),
description = COALESCE($3, description),
exclude_iters = COALESCE($4, exclude_iters),
adjustment_segment = CASE WHEN $5::bool THEN $6::text ELSE adjustment_segment END,
adjustment_bucket = CASE WHEN $7::bool THEN $8::text ELSE adjustment_bucket END,
unlabeled_load = CASE WHEN $9::bool THEN $10::text ELSE unlabeled_load END
name = COALESCE($2, name),
description = COALESCE($3, description),
exclude_iters = COALESCE($4, exclude_iters)
WHERE id = $1
RETURNING *
`, [
req.params.id,
name || null,
description || null,
exclude_iters ? JSON.stringify(exclude_iters) : null,
adjustment_segment !== undefined, set(adjustment_segment),
adjustment_bucket !== undefined, set(adjustment_bucket),
unlabeled_load !== undefined, set(unlabeled_load)
exclude_iters ? JSON.stringify(exclude_iters) : null
]);
if (result.rows.length === 0) {
return res.status(404).json({ error: 'Version not found' });
@ -365,7 +143,7 @@ ${colDefs},
// close a version — blocks further edits
router.post('/versions/:id/close', async (req, res) => {
const pf_user = sessionUser(req);
const { pf_user } = req.body;
try {
const result = await pool.query(`
UPDATE pf.version

72
scripts/migrate.js Normal file
View File

@ -0,0 +1,72 @@
#!/usr/bin/env node
// Migration CLI.
//
// npm run migrate apply pending migrations
// npm run migrate:status show what is applied and what is pending
// npm run migrate:baseline record pending migrations as applied WITHOUT running
// them — for a database that already matches
//
// Pass --up-to=<filename> to baseline only through a given migration.
require('dotenv').config();
const os = require('os');
const { Client } = require('pg');
const { status, migrate, baseline } = require('../lib/migrations');
const args = process.argv.slice(2);
const command = args.find(a => !a.startsWith('--')) || 'up';
const upTo = (args.find(a => a.startsWith('--up-to=')) || '').split('=')[1] || null;
function connect() {
return new Client({
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT) || 5432,
database: process.env.DB_NAME,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
ssl: false
});
}
async function showStatus(client) {
const st = await status(client);
console.log(`\n${process.env.DB_NAME} (${st.applied.length} applied, ${st.pending.length} pending)\n`);
for (const m of st.onDisk) {
const hit = st.applied.find(a => a.filename === m.filename);
const mark = !hit ? 'PENDING'
: st.modified.includes(m.filename) ? 'MODIFIED'
: 'applied';
const when = hit ? hit.applied_at.toISOString().slice(0, 19).replace('T', ' ') : '';
console.log(` ${mark.padEnd(9)} ${m.filename.padEnd(40)} ${when}`);
}
for (const f of st.missing) console.log(` MISSING ${f.padEnd(40)} recorded as applied but not on disk`);
if (st.modified.length) console.log(`\n ! modified after being applied: ${st.modified.join(', ')}`);
if (st.outOfOrder.length) console.log(` ! would apply out of order: ${st.outOfOrder.join(', ')}`);
console.log('');
return st;
}
(async () => {
const client = connect();
await client.connect();
try {
const user = `${os.userInfo().username}@${os.hostname()}`;
if (command === 'status') {
const st = await showStatus(client);
process.exitCode = (st.modified.length || st.missing.length) ? 1 : 0;
} else if (command === 'baseline') {
await baseline(client, { user, upTo });
await showStatus(client);
} else if (command === 'up') {
await migrate(client, { user });
} else {
console.error(`Unknown command "${command}" — expected up, status, or baseline.`);
process.exitCode = 2;
}
} catch (err) {
console.error(`\nmigration error: ${err.message}\n`);
process.exitCode = 1;
} finally {
await client.end();
}
})();

50
scripts/schema-dump.js Normal file
View File

@ -0,0 +1,50 @@
#!/usr/bin/env node
// Regenerate setup_sql/schema.generated.sql — a readable snapshot of the current
// pf schema, for reference only. Migrations are the source of truth; this file is
// generated so it cannot drift from the database the way a hand-maintained
// schema file does. Never edit it, and never apply it to create a database.
require('dotenv').config();
const { spawnSync } = require('child_process');
const fs = require('fs');
const path = require('path');
const out = path.join(__dirname, '..', 'setup_sql', 'schema.generated.sql');
const result = spawnSync('pg_dump', [
'--schema-only', '--no-owner', '--no-privileges',
'--schema=pf',
// per-version forecast tables are created at runtime — including them would
// make this file churn every time a version is created or dropped
'--exclude-table=pf.fc_*',
'-f', out
], {
stdio: 'inherit',
env: {
...process.env,
PGHOST: process.env.DB_HOST,
PGPORT: process.env.DB_PORT || '5432',
PGDATABASE: process.env.DB_NAME,
PGUSER: process.env.DB_USER,
PGPASSWORD: process.env.DB_PASSWORD
}
});
if (result.error) {
console.error(`pg_dump failed: ${result.error.message}`);
process.exit(1);
}
if (result.status !== 0) process.exit(result.status);
// Strip the lines pg_dump varies between runs — a random \restrict token and the
// server/client version banner — so regenerating an unchanged schema produces an
// identical file instead of a spurious diff.
const cleaned = fs.readFileSync(out, 'utf8')
.split('\n')
.filter(l => !/^\\(un)?restrict /.test(l))
.filter(l => !/^-- Dumped (from|by) /.test(l))
.join('\n')
.replace(/\n{3,}/g, '\n\n');
fs.writeFileSync(out, cleaned);
console.log(`wrote ${path.relative(process.cwd(), out)} (${cleaned.split('\n').length} lines)`);

87
server.js
View File

@ -1,10 +1,7 @@
require('dotenv').config();
const express = require('express');
const cors = require('cors');
const session = require('express-session');
const PgSession = require('connect-pg-simple')(session);
const { Pool, types } = require('pg');
const { requireAuth } = require('./lib/auth');
// Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings,
// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>.
@ -12,15 +9,7 @@ types.setTypeParser(20, v => v === null ? null : Number(v));
types.setTypeParser(1700, v => v === null ? null : Number(v));
const app = express();
// Sessions ride on a cookie, so a wildcard CORS origin would let any site make
// credentialed calls on behalf of a logged-in user. The UI is served from this
// same origin and needs no CORS at all; set CORS_ORIGIN only for a separate
// front-end host, and it is then allowed by name, never by wildcard.
if (process.env.CORS_ORIGIN) {
app.use(cors({ origin: process.env.CORS_ORIGIN.split(',').map(o => o.trim()), credentials: true }));
}
app.use(cors());
app.use(express.json());
app.use(express.static('public/app'));
@ -37,51 +26,45 @@ pool.on('error', (err) => {
console.error('pg pool error', err);
});
// ── Authentication ────────────────────────────────────────────
// Refuse to boot without a secret rather than fall back to a default one:
// a predictable secret means forgeable session cookies.
const sessionSecret = process.env.SESSION_SECRET;
if (!sessionSecret) {
console.error('SESSION_SECRET is not set. Run: ./pf.sh config');
process.exit(1);
}
// TLS terminates at the reverse proxy, so express has to trust its headers for
// req.ip (the login throttle) and for secure-cookie detection to be right.
app.set('trust proxy', process.env.TRUST_PROXY || 1);
const cookieSecure = process.env.COOKIE_SECURE !== 'false';
if (!cookieSecure) {
console.warn('COOKIE_SECURE=false — session cookie will be sent over plain HTTP.');
}
app.use(session({
name: 'pf.sid',
store: new PgSession({ pool, schemaName: 'pf', tableName: 'session', createTableIfMissing: false }),
secret: sessionSecret,
resave: false,
saveUninitialized: false,
rolling: true,
cookie: {
httpOnly: true,
sameSite: 'lax',
secure: cookieSecure,
maxAge: 1000 * 60 * 60 * 12,
},
}));
app.use('/api', require('./routes/auth')(pool));
// Everything below this line requires a session.
app.use('/api', requireAuth);
app.use('/api', require('./routes/tables')(pool));
app.use('/api', require('./routes/sources')(pool));
app.use('/api', require('./routes/versions')(pool));
app.use('/api', require('./routes/operations')(pool));
app.use('/api', require('./routes/log')(pool));
app.use('/api', require('./routes/layouts')(pool));
// Refuse to start against a database whose schema is behind the migrations on
// disk. Without this the mismatch surfaces later as a confusing query error
// ('column "x" does not exist') deep inside an unrelated request.
// Set PF_SKIP_MIGRATION_CHECK=1 to bypass.
async function checkSchema() {
if (process.env.PF_SKIP_MIGRATION_CHECK === '1') return;
const { status } = require('./lib/migrations');
const client = await pool.connect();
try {
const st = await status(client);
const problems = [
st.pending.length && `${st.pending.length} pending: ${st.pending.map(m => m.filename).join(', ')}`,
st.modified.length && `modified after being applied: ${st.modified.join(', ')}`,
st.missing.length && `applied but missing from disk: ${st.missing.join(', ')}`
].filter(Boolean);
if (problems.length) {
console.error('\n Database schema is out of date:');
for (const p of problems) console.error(` - ${p}`);
console.error('\n Run "npm run migrate" (or "npm run migrate:status" for detail).');
console.error(' For a database that already matches, "npm run migrate:baseline".\n');
throw new Error('schema out of date');
}
console.log(`schema up to date (${st.applied.length} migrations applied)`);
} finally {
client.release();
}
}
const port = process.env.PORT || 3010;
app.listen(port, '0.0.0.0', () => console.log(`pf_app started on port ${port}`));
checkSchema()
.then(() => app.listen(port, '0.0.0.0', () => console.log(`pf_app started on port ${port}`)))
.catch((err) => {
if (err.message !== 'schema out of date') console.error('startup failed:', err.message);
process.exit(1);
});

263
setup_sql/01_schema.sql
View File

@ -1,263 +0,0 @@
-- Pivot Forecast schema install
-- Run once against target database: psql -d <db> -f setup_sql/01_schema.sql
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE IF NOT EXISTS pf.source (
id serial PRIMARY KEY,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text NOT NULL DEFAULT 'active', -- active | archived
default_layout jsonb, -- Perspective view config used as the per-source default
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
UNIQUE (schema, tname)
);
-- backfill columns for existing installs
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
CREATE TABLE IF NOT EXISTS pf.col_meta (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
cname text NOT NULL,
label text,
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | ignore
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
dim_group text, -- groups functionally dependent columns
dim_period_col text, -- pf.dim_period column this dimension derives from
in_grain boolean NOT NULL DEFAULT false, -- true = column defines the display grain
opos integer,
UNIQUE (source_id, cname)
);
-- backfill columns for existing installs (must follow the CREATE above)
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;
CREATE TABLE IF NOT EXISTS pf.version (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
name text NOT NULL,
description text,
status text NOT NULL DEFAULT 'open', -- open | closed
exclude_iters jsonb NOT NULL DEFAULT '["reference"]'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
closed_at timestamptz,
closed_by text,
UNIQUE (source_id, name)
);
CREATE TABLE IF NOT EXISTS pf.log (
id bigserial PRIMARY KEY,
version_id integer NOT NULL REFERENCES pf.version(id) ON DELETE CASCADE,
pf_user text NOT NULL,
stamp timestamptz NOT NULL DEFAULT now(),
operation text NOT NULL, -- baseline | reference | scale | recode | clone
slice jsonb,
params jsonb,
note text,
tag text -- initiative label, e.g. 'reduce_spend'; groups
-- adjustments into a bridge from baseline to current
);
-- adding tags to an install that predates them
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS tag text;
CREATE INDEX IF NOT EXISTS log_tag_idx ON pf.log (tag) WHERE tag IS NOT NULL;
-- seed tags for loads that predate the column: a baseline/reference note is the
-- segment's name ('Open Orders', 'Prior Year'), which is exactly what tag holds.
-- Adjustment notes are free text, not labels, so they are left alone.
UPDATE pf.log
SET tag = note
WHERE TRUE
AND tag IS NULL
AND note IS NOT NULL
AND note <> ''
AND operation IN ('baseline', 'reference');
-- Display order for the pivot's segment and bucket columns.
--
-- The segment's display name in the pivot, falling back to tag then note.
--
-- Separate from both because those have jobs already -- tag groups adjustments
-- into initiatives for the bridge, note is free commentary -- and because the
-- label carries the sort order. Perspective orders column groups by the value
-- string, so a leading "01 - " is how ordering is expressed; putting that in the
-- note would put it in every note.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS label text;
-- Vestigial, both of them. They held the ordinal when the "01 - " prefix was
-- computed for the pivot rather than typed into label and bucket: bucket_order
-- sequenced the bucket columns, log.seq the segments within them. Nothing reads
-- either now, and nothing writes them -- kept only because dropping a column is
-- not worth a migration to reclaim two that cost nothing.
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS bucket_order jsonb;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS seq integer;
-- What a segment contributes to, independent of pf_iter.
--
-- pf_iter answers "can operations write to these rows"; bucket answers "does this
-- belong in the forecast number". Those are not the same question -- Open Orders is
-- loaded as reference so nothing adjusts it, yet it is part of the forecast -- so
-- neither can be derived from the other.
--
-- Free text with suggested values (Forecast / Prior Year / Prior Prior Year / Plan)
-- rather than an enum, so a new banner does not need a migration. Blank by default:
-- until a segment is labelled, the pivot falls back to showing its own name.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS bucket text;
CREATE INDEX IF NOT EXISTS log_bucket_idx ON pf.log (bucket) WHERE bucket IS NOT NULL;
-- The names a row falls back to when nobody has named it, per scenario. Null
-- means "use the built-in", which is the DISPLAY DEFAULTS block in
-- lib/sql_generator.js. Read through a join at query time, not baked into
-- pf.sql: those templates are keyed on (source_id, operation) and shared by
-- every version of a source.
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_segment text;
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_bucket text;
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS unlabeled_load text;
-- What the entry did, stamped when it did it.
--
-- Not a cache: a log entry's forecast rows never change after it is written.
-- Only the operation that owns the logid inserts them, and the only thing that
-- removes them is undo, which deletes this row too -- so these totals are fixed
-- at write time rather than derived from something that can move underneath
-- them. The change log was joining the whole forecast table to recompute them
-- on every open, 2.5M rows to total a few thousand.
--
-- measure_cols records which columns they are denominated in, since the value
-- and units roles can be reassigned in col_meta and the numbers would otherwise
-- quietly come to mean something else.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS row_count integer;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS value_total double precision;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS units_total double precision;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS measure_cols jsonb;
-- The column an account's territory is expressed in -- the rep, the region,
-- whatever this source divides ownership by. Exactly one per source.
--
-- Flagged here rather than named in code because it is source-specific, the
-- same way the grain is: a second source should not need a code change to be
-- scoped by something other than a sales rep.
--
-- It does two jobs. The server scopes every read and write to the values on the
-- account, and recode refuses to *set* this column unless the account is an
-- admin -- moving a row between territories is reassignment, not forecasting.
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS is_territory boolean NOT NULL DEFAULT false;
-- The debug path: what the entry was trying to do, and what that became.
--
-- Both, deliberately. params records the intent -- the slice, the scope, the
-- resolved increments -- and sql_text records the statement as executed, with
-- territory and scope already resolved into it. Keeping only one of them
-- assumes the translation between them is correct, which is exactly the
-- assumption in doubt when the rows look wrong.
--
-- env captures the state the intent was executed against that cannot be
-- reconstructed afterwards: the territory in force, the version's
-- exclude_iters, and when the template was generated. All three are mutable
-- rows elsewhere, and nothing remembers what they were.
--
-- The template generation is a fingerprint, not a version: it cannot reproduce
-- the old template, but it can tell you the entry ran under a different one,
-- which is what would otherwise make a replay quietly wrong.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS env jsonb;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS sql_text text;
-- Master data for a dim_group: one row per key value, with its sibling columns.
--
-- The source is transactional and often a view over all history, so deriving a
-- member list from it is both slow and wrong -- slow because it means scanning
-- millions of rows, wrong because it can only describe what was transacted and
-- has no way to say a part is discontinued or that a new one exists before it
-- has sold. This table is the app's own list, refreshed from the source but
-- curatable independently of it.
CREATE TABLE IF NOT EXISTS pf.dim_member (
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
dim_group text NOT NULL, -- matches pf.col_meta.dim_group
key_value text NOT NULL, -- the is_key column's value
attrs jsonb NOT NULL DEFAULT '{}'::jsonb, -- the sibling columns
is_active boolean NOT NULL DEFAULT true,
source_seen boolean NOT NULL DEFAULT true, -- present in the source at last refresh
added_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
refreshed_at timestamptz,
PRIMARY KEY (source_id, dim_group, key_value)
);
CREATE INDEX IF NOT EXISTS dim_member_active_idx
ON pf.dim_member (source_id, dim_group) WHERE is_active;
-- generated operation SQL per source, stored after col_meta is configured
CREATE TABLE IF NOT EXISTS pf.sql (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
operation text NOT NULL, -- get_data | baseline | reference | scale | recode | clone | undo
sql text NOT NULL,
generated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_id, operation)
);
-- pf.layout: named Perspective view configs.
--
-- Replaces the per-browser localStorage lists (pf_layouts_v*) and the single
-- anonymous pf.source.default_layout. A layout is either `private` -- visible
-- only to its owner -- or `published`, visible to everyone on the version and
-- writable only by its owner or an admin, the same rule pf.log already uses.
CREATE TABLE IF NOT EXISTS pf.layout (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
-- null = applies to every version of the source; that is where the old
-- source default lives, and what a new version picks up before anyone has
-- published a layout of its own.
version_id integer REFERENCES pf.version(id) ON DELETE CASCADE,
name text NOT NULL,
config jsonb NOT NULL,
owner text NOT NULL,
visibility text NOT NULL DEFAULT 'private', -- private | published
is_default boolean NOT NULL DEFAULT false, -- applied on first load
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CHECK (visibility IN ('private', 'published')),
-- only a published layout can be the default: a private one would be a
-- default nobody else could see.
CHECK (NOT is_default OR visibility = 'published')
);
-- COALESCE rather than the bare column: version_id is nullable, and in a unique
-- index NULLs are distinct, so source-wide rows would not be constrained at all.
CREATE UNIQUE INDEX IF NOT EXISTS layout_private_name
ON pf.layout (source_id, COALESCE(version_id, 0), owner, lower(name))
WHERE visibility = 'private';
CREATE UNIQUE INDEX IF NOT EXISTS layout_published_name
ON pf.layout (source_id, COALESCE(version_id, 0), lower(name))
WHERE visibility = 'published';
-- one default per scope
CREATE UNIQUE INDEX IF NOT EXISTS layout_one_default
ON pf.layout (source_id, COALESCE(version_id, 0))
WHERE is_default;
CREATE INDEX IF NOT EXISTS layout_lookup ON pf.layout (source_id, version_id);
-- Carry the old single source default across as a published, source-wide row.
-- Idempotent: skipped once a default exists for that source.
INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility, is_default)
SELECT s.id, NULL, 'Source default', s.default_layout, COALESCE(s.created_by, 'admin'), 'published', true
FROM pf.source s
WHERE TRUE
AND s.default_layout IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM pf.layout l
WHERE TRUE
AND l.source_id = s.id
AND l.version_id IS NULL
AND l.is_default
);

38
setup_sql/02_auth.sql
View File

@ -1,38 +0,0 @@
-- Pivot Forecast — authentication
-- Run after 01_schema.sql: psql -d <db> -f setup_sql/02_auth.sql
-- Safe to re-run.
-- Application accounts. Passwords are scrypt hashes written by lib/auth.js;
-- the plaintext never reaches the database. Manage with ./pf.sh add-user.
CREATE TABLE IF NOT EXISTS pf.app_user (
id serial PRIMARY KEY,
username text NOT NULL UNIQUE,
pass_hash text NOT NULL,
display_name text,
is_active boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT now(),
last_login_at timestamptz
);
-- Session store for express-session (connect-pg-simple layout). Sessions live
-- here rather than in memory so a restart doesn't sign everyone out, and so a
-- session can be revoked by deleting its row.
CREATE TABLE IF NOT EXISTS pf.session (
sid varchar PRIMARY KEY NOT NULL COLLATE "default",
sess json NOT NULL,
expire timestamp(6) NOT NULL
);
CREATE INDEX IF NOT EXISTS session_expire_idx ON pf.session (expire);
-- What an account may see and change.
--
-- territory is the list of values allowed in the source's territory column
-- (col_meta.is_territory). It is a filter the server applies to every read and
-- every write; it is never sent by the client, which could remove it.
--
-- Fail closed: no territory and not an admin means no rows. An account created
-- without one sees nothing until someone grants it, rather than seeing the whole
-- book because a column was left null.
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS territory jsonb;
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS is_admin boolean NOT NULL DEFAULT false;

64
setup_sql/README.md Normal file
View File

@ -0,0 +1,64 @@
# Database setup
## Migrations
`migrations/*.sql` are applied in filename order and recorded in
`pf.schema_version`. They are the **source of truth** for the `pf` schema — there
is no hand-maintained current-state file to drift out of sync.
```bash
npm run migrate # apply pending migrations
npm run migrate:status # what is applied, what is pending
npm run migrate:baseline # record pending as applied WITHOUT running them
```
`server.js` refuses to start when the database is behind, so drift surfaces at
boot rather than as `column "x" does not exist` inside an unrelated request. Set
`PF_SKIP_MIGRATION_CHECK=1` to bypass.
### Fresh database
```bash
npm run migrate
psql -d <db> -f setup_sql/gen_dim_period.sql
```
### Existing database that already matches
Use `baseline` so the runner does not try to re-create tables that exist:
```bash
npm run migrate:baseline
```
To baseline only part of the way — the schema matches through `0003` but not
`0004` — pass `--up-to` and then migrate the rest:
```bash
node scripts/migrate.js baseline --up-to=0003_col_meta_dim_group_period.sql
npm run migrate
```
## Writing a migration
- Name it `NNNN_short_description.sql`, numbered after the highest existing file.
- One concern per file. Keep it forward-only; there are no down migrations.
- **Applied migrations are immutable.** The runner stores a checksum and refuses
to proceed if a file changes after being applied, because editing one means
databases silently disagree about what the schema is. To fix a mistake, add a
new migration.
- No `IF NOT EXISTS` guards on new migrations. The bookkeeping already guarantees
each runs once, and the guards hide ordering mistakes — the reason the old
`01_schema.sql` had `ALTER`s sitting above the `CREATE TABLE` they depended on,
broken for anyone installing from scratch. `0004` is the one exception, since it
was applied by hand before migrations existed.
## Not migrations
- **`gen_dim_period.sql`** — creates and populates `pf.dim_period`. It is a
parameterised data load (configurable fiscal year start month), not a schema
change, so it stays a script you run deliberately.
- **`pf.fc_{tname}_{version_id}`** — per-version forecast tables, created and
dropped at runtime by `routes/versions.js` from `col_meta`. Never migrated.
- **`schema.generated.sql`** — a `pg_dump` snapshot for reading, refreshed with
`npm run schema:dump`. Generated, never edited, never applied.

63
setup_sql/migrations/0001_initial_schema.sql Normal file
View File

@ -0,0 +1,63 @@
-- Initial pf schema: sources, column metadata, versions, audit log, generated SQL.
--
-- This is the schema as it stood before the additive columns in later migrations.
-- Columns added afterwards are declared once, in their own migration — not here.
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE pf.source (
id serial PRIMARY KEY,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text NOT NULL DEFAULT 'active', -- active | archived
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
UNIQUE (schema, tname)
);
CREATE TABLE pf.col_meta (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
cname text NOT NULL,
label text,
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | filter | ignore
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
opos integer,
UNIQUE (source_id, cname)
);
CREATE TABLE pf.version (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
name text NOT NULL,
description text,
status text NOT NULL DEFAULT 'open', -- open | closed
exclude_iters jsonb NOT NULL DEFAULT '["reference"]'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
closed_at timestamptz,
closed_by text,
UNIQUE (source_id, name)
);
CREATE TABLE pf.log (
id bigserial PRIMARY KEY,
version_id integer NOT NULL REFERENCES pf.version(id) ON DELETE CASCADE,
pf_user text NOT NULL,
stamp timestamptz NOT NULL DEFAULT now(),
operation text NOT NULL, -- baseline | reference | scale | recode | clone
slice jsonb,
params jsonb,
note text
);
-- generated operation SQL per source, stored after col_meta is configured
CREATE TABLE pf.sql (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
operation text NOT NULL, -- get_data | get_agg | baseline | reference | scale | recode | clone | undo
sql text NOT NULL,
generated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_id, operation)
);

4
setup_sql/migrations/0002_source_default_layout.sql Normal file
View File

@ -0,0 +1,4 @@
-- Per-source default Perspective view config, applied when a version has no
-- saved layout of its own.
ALTER TABLE pf.source ADD COLUMN default_layout jsonb;

7
setup_sql/migrations/0003_col_meta_dim_group_period.sql Normal file
View File

@ -0,0 +1,7 @@
-- dim_group groups functionally dependent columns (e.g. a date and the year/month
-- dimensions derived from it). dim_period_col maps such a dimension to a
-- pf.dim_period column, so date-adjacent values are derived by JOIN at load time
-- rather than copied raw from the source.
ALTER TABLE pf.col_meta ADD COLUMN dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN dim_period_col text;

8
setup_sql/migrations/0004_col_meta_in_grain.sql Normal file
View File

@ -0,0 +1,8 @@
-- in_grain flags dimension/date columns that define the display grain: the
-- forecast load is pre-aggregated to the flagged columns instead of shipping raw
-- rows. See pf_spec.md -> Display-grain pre-aggregation.
--
-- Already applied by hand on the original dev database before migrations existed;
-- IF NOT EXISTS keeps replay safe there. New migrations should not need the guard.
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;

384
setup_sql/schema.generated.sql Normal file
View File

@ -0,0 +1,384 @@
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET transaction_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Name: pf; Type: SCHEMA; Schema: -; Owner: -
--
CREATE SCHEMA pf;
SET default_tablespace = '';
SET default_table_access_method = heap;
--
-- Name: col_meta; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.col_meta (
id integer NOT NULL,
source_id integer NOT NULL,
cname text NOT NULL,
label text,
role text DEFAULT 'ignore'::text NOT NULL,
is_key boolean DEFAULT false NOT NULL,
opos integer,
dim_group text,
dim_period_col text,
in_grain boolean DEFAULT false NOT NULL
);
--
-- Name: col_meta_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.col_meta_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: col_meta_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.col_meta_id_seq OWNED BY pf.col_meta.id;
--
-- Name: dim_period; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.dim_period (
sdat date NOT NULL,
edat date NOT NULL,
drange daterange NOT NULL,
ndays integer NOT NULL,
cal_year integer NOT NULL,
cal_quarter integer NOT NULL,
cal_month integer NOT NULL,
cal_month_abbr text NOT NULL,
cal_month_name text NOT NULL,
cal_label text NOT NULL,
fisc_year integer NOT NULL,
fisc_quarter integer NOT NULL,
fisc_quarter_label text NOT NULL,
fisc_month integer NOT NULL,
fisc_month_abbr text NOT NULL,
fisc_month_name text NOT NULL,
fisc_label text NOT NULL,
period_key text NOT NULL
);
--
-- Name: log; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.log (
id bigint NOT NULL,
version_id integer NOT NULL,
pf_user text NOT NULL,
stamp timestamp with time zone DEFAULT now() NOT NULL,
operation text NOT NULL,
slice jsonb,
params jsonb,
note text
);
--
-- Name: log_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.log_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: log_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.log_id_seq OWNED BY pf.log.id;
--
-- Name: schema_version; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.schema_version (
filename text NOT NULL,
checksum text NOT NULL,
applied_at timestamp with time zone DEFAULT now() NOT NULL,
applied_by text
);
--
-- Name: source; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.source (
id integer NOT NULL,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text DEFAULT 'active'::text NOT NULL,
created_at timestamp with time zone DEFAULT now() NOT NULL,
created_by text,
default_layout jsonb
);
--
-- Name: source_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.source_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: source_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.source_id_seq OWNED BY pf.source.id;
--
-- Name: sql; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.sql (
id integer NOT NULL,
source_id integer NOT NULL,
operation text NOT NULL,
sql text NOT NULL,
generated_at timestamp with time zone DEFAULT now() NOT NULL
);
--
-- Name: sql_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.sql_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: sql_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.sql_id_seq OWNED BY pf.sql.id;
--
-- Name: version; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.version (
id integer NOT NULL,
source_id integer NOT NULL,
name text NOT NULL,
description text,
status text DEFAULT 'open'::text NOT NULL,
exclude_iters jsonb DEFAULT '["reference"]'::jsonb NOT NULL,
created_at timestamp with time zone DEFAULT now() NOT NULL,
created_by text,
closed_at timestamp with time zone,
closed_by text
);
--
-- Name: version_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.version_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: version_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.version_id_seq OWNED BY pf.version.id;
--
-- Name: col_meta id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta ALTER COLUMN id SET DEFAULT nextval('pf.col_meta_id_seq'::regclass);
--
-- Name: log id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log ALTER COLUMN id SET DEFAULT nextval('pf.log_id_seq'::regclass);
--
-- Name: source id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source ALTER COLUMN id SET DEFAULT nextval('pf.source_id_seq'::regclass);
--
-- Name: sql id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql ALTER COLUMN id SET DEFAULT nextval('pf.sql_id_seq'::regclass);
--
-- Name: version id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version ALTER COLUMN id SET DEFAULT nextval('pf.version_id_seq'::regclass);
--
-- Name: col_meta col_meta_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_pkey PRIMARY KEY (id);
--
-- Name: col_meta col_meta_source_id_cname_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_source_id_cname_key UNIQUE (source_id, cname);
--
-- Name: dim_period dim_period_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.dim_period
ADD CONSTRAINT dim_period_pkey PRIMARY KEY (sdat);
--
-- Name: log log_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log
ADD CONSTRAINT log_pkey PRIMARY KEY (id);
--
-- Name: schema_version schema_version_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.schema_version
ADD CONSTRAINT schema_version_pkey PRIMARY KEY (filename);
--
-- Name: source source_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source
ADD CONSTRAINT source_pkey PRIMARY KEY (id);
--
-- Name: source source_schema_tname_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source
ADD CONSTRAINT source_schema_tname_key UNIQUE (schema, tname);
--
-- Name: sql sql_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_pkey PRIMARY KEY (id);
--
-- Name: sql sql_source_id_operation_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_source_id_operation_key UNIQUE (source_id, operation);
--
-- Name: version version_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_pkey PRIMARY KEY (id);
--
-- Name: version version_source_id_name_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_source_id_name_key UNIQUE (source_id, name);
--
-- Name: dim_period_cal_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_cal_idx ON pf.dim_period USING btree (cal_year, cal_month);
--
-- Name: dim_period_drange_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_drange_idx ON pf.dim_period USING gist (drange);
--
-- Name: dim_period_fisc_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_fisc_idx ON pf.dim_period USING btree (fisc_year, fisc_month);
--
-- Name: col_meta col_meta_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE CASCADE;
--
-- Name: log log_version_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log
ADD CONSTRAINT log_version_id_fkey FOREIGN KEY (version_id) REFERENCES pf.version(id) ON DELETE CASCADE;
--
-- Name: sql sql_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE CASCADE;
--
-- Name: version version_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE RESTRICT;
--
-- PostgreSQL database dump complete
--

1
ui/index.html
View File

@ -5,6 +5,7 @@
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pivot Forecast</title>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/css/themes.css" crossorigin="anonymous">
</head>
<body>
<div id="root"></div>

774
ui/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

4
ui/package.json
View File

@ -10,10 +10,6 @@
"preview": "vite preview"
},
"dependencies": {
"@perspective-dev/client": "file:vendor/perspective-dev-client-5.4.0.tgz",
"@perspective-dev/server": "file:vendor/perspective-dev-server-5.4.0.tgz",
"@perspective-dev/viewer": "file:vendor/perspective-dev-viewer-5.4.0.tgz",
"@perspective-dev/viewer-datagrid": "file:vendor/perspective-dev-viewer-datagrid-5.4.0.tgz",
"react": "^19.2.5",
"react-dom": "^19.2.5"
},

61
ui/src/App.jsx
View File

@ -10,10 +10,8 @@ export default function App() {
const [sidebarExpanded, setSidebarExpanded] = useState(() => localStorage.getItem('pf_sidebar') !== 'collapsed')
const [sources, setSources] = useState([])
const [sourcesLoaded, setSourcesLoaded] = useState(false)
const [sourceId, setSourceId] = useState(() => localStorage.getItem('pf_sourceId') || '')
const [versions, setVersions] = useState([])
const [versionsLoaded, setVersionsLoaded] = useState(false)
const [versionId, setVersionId] = useState(() => localStorage.getItem('pf_versionId') || '')
useEffect(() => { localStorage.setItem('pf_view', view) }, [view])
@ -23,58 +21,37 @@ export default function App() {
const refreshSources = useCallback(async () => {
const data = await fetch('/api/sources').then(r => r.json())
const list = Array.isArray(data) ? data : []
setSources(list)
setSourcesLoaded(true)
return list
setSources(data)
return data
}, [])
const refreshVersions = useCallback(async (sid) => {
const id = sid ?? sourceId
if (!id) { setVersions([]); setVersionsLoaded(true); return [] }
if (!id) { setVersions([]); return [] }
const data = await fetch(`/api/sources/${id}/versions`).then(r => r.json())
const list = Array.isArray(data) ? data : []
setVersions(list)
setVersionsLoaded(true)
return list
setVersions(data)
return data
}, [sourceId])
useEffect(() => { refreshSources() }, [])
// The selection is restored from localStorage and survives a deregister, so it
// has to be re-checked against the list itself rather than only at mount:
// deleting the selected source otherwise leaves a dead id behind and every
// call 404s "Source not found" until the page is reloaded.
useEffect(() => {
if (!sourcesLoaded) return
if (sources.length === 0) { setSourceId(''); return }
if (!sourceId || !sources.some(s => String(s.id) === String(sourceId))) {
setSourceId(String(sources[0].id))
}
}, [sources, sourcesLoaded, sourceId])
refreshSources().then(data => {
if (data.length === 0) { setSourceId(''); return }
if (!sourceId || !data.some(s => String(s.id) === String(sourceId))) {
setSourceId(String(data[0].id))
}
})
}, [])
useEffect(() => {
if (!sourceId) { setVersions([]); setVersionId(''); setVersionsLoaded(true); return }
// The list belongs to the previous source until the fetch lands.
setVersionsLoaded(false)
refreshVersions(sourceId)
if (!sourceId) { setVersions([]); setVersionId(''); return }
refreshVersions(sourceId).then(data => {
if (data.length === 0) { setVersionId(''); return }
if (!versionId || !data.some(v => String(v.id) === String(versionId))) {
setVersionId(String(data[0].id))
}
})
}, [sourceId])
// Same reasoning as sources: a deleted version must not stay selected.
//
// versionsLoaded matters more than it looks: without it, the empty initial
// state reads as "this source has no versions", so a versionId restored from
// localStorage is cleared and then set straight back when the fetch lands.
// Forecast's load effect is keyed on that id, so the round trip made every
// page load fetch and aggregate the whole version twice.
useEffect(() => {
if (!sourceId || !versionsLoaded) return
if (versions.length === 0) { setVersionId(''); return }
if (!versionId || !versions.some(v => String(v.id) === String(versionId))) {
setVersionId(String(versions[0].id))
}
}, [versions, versionsLoaded, sourceId, versionId])
const ctx = {
sources, sourceId, setSourceId,
versions, versionId, setVersionId,

62
ui/src/auth.jsx
View File

@ -1,62 +0,0 @@
import { createContext, useContext, useState, useEffect, useCallback } from 'react'
const AuthContext = createContext()
// A session can expire while the app is open. Rather than teach every one of
// the app's fetch calls to check for it, wrap fetch once: any 401 from /api
// drops the whole UI back to the login screen. Cookies ride along on their own
// — fetch defaults to same-origin credentials, and the UI is served from the
// same origin as the API.
function installUnauthorizedHandler(onUnauthorized) {
const original = window.fetch
window.fetch = async (...args) => {
const res = await original(...args)
const url = typeof args[0] === 'string' ? args[0] : args[0]?.url || ''
if (res.status === 401 && url.includes('/api/') && !url.includes('/api/login')) {
onUnauthorized()
}
return res
}
return () => { window.fetch = original }
}
export function AuthProvider({ children }) {
const [user, setUser] = useState(null)
const [checking, setCheck] = useState(true)
useEffect(() => {
fetch('/api/me')
.then(r => r.ok ? r.json() : null)
.then(d => setUser(d?.user || null))
.catch(() => setUser(null))
.finally(() => setCheck(false))
}, [])
useEffect(() => installUnauthorizedHandler(() => setUser(null)), [])
const login = useCallback(async (username, password) => {
const r = await fetch('/api/login', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ username, password }),
})
const data = await r.json().catch(() => ({}))
if (!r.ok) throw new Error(data.error || 'Login failed')
setUser(data.user)
return data.user
}, [])
const logout = useCallback(async () => {
try { await fetch('/api/logout', { method: 'POST' }) } catch {}
setUser(null)
}, [])
return (
<AuthContext.Provider value={{ user, checking, login, logout }}>
{children}
</AuthContext.Provider>
)
}
const useAuth = () => useContext(AuthContext)
export default useAuth

565
ui/src/components/BridgeView.jsx
View File

@ -1,565 +0,0 @@
// Bridge (waterfall): how a version got from its baseline to where it stands,
// one step per initiative tag.
//
// Data is computed from the Perspective table already in the browser rather than
// from the /bridge endpoint, so the figures always reconcile with what the pivot
// is showing — including when the view is scoped to the pivot's current filters.
//
// Colour is a POLARITY job, not a categorical one: increases and decreases are two
// poles of one scale, with baseline and current as neutral anchors. Blue/red is the
// validated diverging pair (CVD ΔE 21.6, normal-vision 32.3 against white);
// green/red is avoided precisely because it is the classic CVD failure.
import { useState, useEffect, useRef, useCallback } from 'react'
const UP = '#2a78d6' // increase
const DOWN = '#e34948' // decrease
const ANCHOR = '#6b7280' // baseline / current — neutral, 4.83:1 on white
const GRID = '#e5e7eb'
const INK = '#374151'
const INK_DIM = '#6b7280'
const fmt = (n, dp = 2) =>
n == null || !isFinite(n) ? '—'
: n.toLocaleString(undefined, { minimumFractionDigits: dp, maximumFractionDigits: dp })
const fmtSigned = (n, dp = 2) =>
n == null || !isFinite(n) ? '—' : `${n > 0 ? '+' : n < 0 ? '−' : ''}${fmt(Math.abs(n), dp)}`
// compact axis ticks — full precision belongs on the marks and in the table
function fmtAxis(n) {
const a = Math.abs(n)
if (a >= 1e9) return `${(n / 1e9).toFixed(1)}B`
if (a >= 1e6) return `${(n / 1e6).toFixed(1)}M`
if (a >= 1e3) return `${(n / 1e3).toFixed(1)}k`
return String(Math.round(n))
}
function niceTicks(min, max, count = 5) {
if (!isFinite(min) || !isFinite(max) || min === max) return [min || 0]
const span = max - min
const raw = span / count
const mag = Math.pow(10, Math.floor(Math.log10(raw)))
const step = [1, 2, 2.5, 5, 10].map(m => m * mag).find(s => s >= raw) || mag * 10
const out = []
for (let t = Math.ceil(min / step) * step; t <= max + 1e-9; t += step) out.push(t)
return out
}
// Turn raw forecast rows into the walk: baseline anchor, one floating step per
// initiative tag, current anchor. Pure and exported so the arithmetic can be
// checked against real data without a browser.
// The walk from a basis to the current forecast.
//
// Without a basis this is the forecast's own composition: its loads as the
// opening anchor, then one step per adjustment tag.
//
// With one -- Plan when building a forecast, Prior Year when building the AOP --
// it starts there instead, and the identity that makes it exact is
//
// Forecast - Basis = (Forecast loads - Basis) + adjustments
//
// so the opening step is the difference between the forecast's own loads and the
// basis, and every tagged adjustment explains the rest. No residual, no plug: the
// bars sum to the endpoint by construction rather than by hoping the tags cover
// everything.
//
// Membership comes from pf_bucket, not pf_iter. Those answer different questions
// -- Open Orders is loaded as reference so nothing adjusts it, and is still part
// of the forecast -- so a bridge keyed on iter silently dropped it.
export function buildSteps(rows, {
valueCol, unitsCol, logMeta = {},
excludeIters = ['reference'], // kept for callers with no bucket data
basis = null, // a pf_bucket name, or null for composition
forecastBucket,
}) {
const hasBuckets = rows.some(r => r.pf_bucket != null)
const excl = new Set(excludeIters)
// Which bucket is the forecast. It used to be the literal 'Forecast', and the
// moment the buckets were renamed to carry their sort prefix -- '04 - Forecast'
// -- no row matched it: every row counted as a comparison, the walk had no
// steps, and the bridge showed the basis cancelling itself to zero.
//
// The caller passes the version's adjustment_bucket, which is the same value
// an unbucketed adjustment is labelled with, so the two agree by construction.
// Falling back to whichever bucket actually holds the adjustments keeps a
// renamed or unconfigured version working rather than silently empty.
const fcBucket = (() => {
if (forecastBucket && rows.some(r => r.pf_bucket === forecastBucket)) return forecastBucket
const adjusted = rows.find(r => r.pf_bucket && !['baseline', 'reference'].includes(r.pf_iter))
if (adjusted) return adjusted.pf_bucket
const baseline = rows.find(r => r.pf_bucket && r.pf_iter === 'baseline')
return baseline ? baseline.pf_bucket : (forecastBucket || 'Forecast')
})()
const num = (r, col) => (col ? (parseFloat(r[col]) || 0) : 0)
const blank = () => ({ value: 0, units: 0, rows: 0 })
const loads = blank() // the forecast's own segments
const basisT = blank() // the comparison bucket
const byTag = new Map()
const buckets = new Map() // every bucket, for the picker and the markers
for (const r of rows) {
const v = num(r, valueCol)
const u = num(r, unitsCol)
const bucket = hasBuckets ? (r.pf_bucket || '') : null
if (bucket != null) {
const b = buckets.get(bucket) || blank()
b.value += v; b.units += u; b.rows += 1
buckets.set(bucket, b)
}
// Anything outside the forecast is a comparison, never a step.
if (hasBuckets && bucket !== fcBucket) {
if (basis && bucket === basis) { basisT.value += v; basisT.units += u; basisT.rows += 1 }
continue
}
if (!hasBuckets && excl.has(r.pf_iter)) continue
const isLoad = r.pf_iter === 'baseline' || r.pf_iter === 'reference'
if (isLoad) { loads.value += v; loads.units += u; loads.rows += 1; continue }
const meta = logMeta[r.pf_logid] || {}
// label first, the same precedence pf_segment uses, so the bridge and the
// pivot call a step by the same name
const tag = (meta.label || meta.tag || '').trim()
const label = tag || (meta.note || '').trim() ||
`${(meta.operation || r.pf_iter || 'adj')}${r.pf_logid != null ? ` #${r.pf_logid}` : ''}`
const key = tag ? `tag:${tag}` : `log:${r.pf_logid}`
const g = byTag.get(key) ||
{ key, label, tagged: !!tag, value: 0, units: 0, rows: 0, logIds: new Set(), first: r.pf_logid }
g.value += v; g.units += u; g.rows += 1
if (r.pf_logid != null) { g.logIds.add(r.pf_logid); g.first = Math.min(g.first ?? r.pf_logid, r.pf_logid) }
byTag.set(key, g)
}
const mid = [...byTag.values()].sort((a, b) => (a.first ?? 0) - (b.first ?? 0))
const useBasis = !!basis && basisT.rows > 0
const out = []
if (useBasis) {
out.push({
key: 'basis', label: basis, kind: 'anchor',
delta: basisT.value, start: 0, end: basisT.value,
units: basisT.units, rows: basisT.rows, entries: 1,
})
const gap = loads.value - basisT.value
out.push({
// "Loads" is our word for a segment import and means nothing to anyone
// reading a waterfall. This step is simply where the forecast began
// relative to the comparison.
//
// tagged, though it has no tag: the flag drives an "· untagged" suffix
// meant for adjustments nobody grouped into an initiative, and this is
// not an adjustment at all.
key: 'loads-vs-basis', label: `Starting point vs ${basis}`, kind: 'step',
tagged: true,
delta: gap, start: basisT.value, end: loads.value,
units: loads.units - basisT.units, rows: loads.rows, entries: 1,
})
} else {
out.push({
key: 'baseline', label: hasBuckets ? fcBucket + ' loads' : 'Baseline', kind: 'anchor',
delta: loads.value, start: 0, end: loads.value,
units: loads.units, rows: loads.rows, entries: 1,
})
}
let running = loads.value
for (const g of mid) {
const start = running
running += g.value
out.push({ ...g, kind: 'step', delta: g.value, start, end: running, entries: g.logIds.size })
}
out.push({
key: 'current', label: hasBuckets ? fcBucket : 'Current', kind: 'anchor',
delta: running, start: 0, end: running,
units: loads.units + mid.reduce((a, g) => a + (g.units || 0), 0),
rows: loads.rows + mid.reduce((a, g) => a + g.rows, 0),
entries: mid.reduce((a, g) => a + g.logIds.size, 0) + 1,
})
// Every bucket present, so the view can offer them as bases and show the ones
// that are not the basis as comparison markers.
out.buckets = [...buckets.entries()]
.map(([name, t]) => ({ name, ...t }))
.sort((a, b) => b.value - a.value)
return out
}
// Plot geometry, also pure: given the steps and a canvas size, where does each
// bar and label land? Exported so collisions and overflow can be checked.
// Break a label to the bar's width, on word boundaries.
//
// SVG text does not wrap, so the labels were cut at twelve characters and
// "04 - Forecast" and "04 - Forecasting" read the same. Character width is
// estimated rather than measured -- measuring means a DOM round trip per label
// on every render, and at this font a digit is about 0.55em, which is close
// enough for a centred label with a bar's width to play with.
//
// Three lines maximum, and a word longer than the line is left to overflow
// rather than broken: a truncated word helps nobody, and the bars are wide
// enough that it only happens to things like a pasted part number.
function wrapLabel(label, barW, fontSize = 10, maxLines = 3) {
const perLine = Math.max(6, Math.floor(barW / (fontSize * 0.55)))
const words = String(label || '').split(/\s+/).filter(Boolean)
const lines = []
let line = ''
for (const w of words) {
const next = line ? `${line} ${w}` : w
if (next.length <= perLine) { line = next; continue }
if (line) lines.push(line)
line = w
if (lines.length === maxLines) break
}
if (line && lines.length < maxLines) lines.push(line)
if (!lines.length) return ['']
// anything that did not fit is marked on the last line rather than dropped
const used = lines.join(' ').length
if (used < String(label).replace(/\s+/g, ' ').length) {
lines[lines.length - 1] = `${lines[lines.length - 1]}…`
}
return lines
}
export function layoutSteps(steps, width, H = 340, PAD = { t: 24, r: 16, b: 84, l: 68 }) {
const plotW = Math.max(120, width - PAD.l - PAD.r)
const plotH = H - PAD.t - PAD.b
const values = steps.flatMap(s => [s.start, s.end])
const rawMin = Math.min(0, ...values)
const rawMax = Math.max(0, ...values)
const span = (rawMax - rawMin) || 1
const yMin = rawMin - span * 0.08
const yMax = rawMax + span * 0.12
const y = (v) => PAD.t + plotH - ((v - yMin) / (yMax - yMin)) * plotH
const n = steps.length || 1
const band = plotW / n
const barW = Math.max(10, Math.min(64, band - 14))
const bars = steps.map((s, i) => {
const x = PAD.l + band * i + (band - barW) / 2
const top = y(Math.max(s.start, s.end))
const bot = y(Math.min(s.start, s.end))
return { key: s.key, x, w: barW, top, h: Math.max(2, bot - top), labelY: top - 6 }
})
return { PAD, plotW, plotH, yMin, yMax, y, band, barW, bars, H, width }
}
export default function BridgeView({
open, onClose, tableRef, viewerRef, logMeta = {},
valueCol, unitsCol, colMeta = [], slices = [],
excludeIters = ['reference'], versionName, forecastBucket,
}) {
const hasSelection = slices.length > 0
// 'selection' | 'filtered' | 'all'
const [scope, setScope] = useState(hasSelection ? 'selection' : 'filtered')
// Which bucket the walk starts from. Plan when building a forecast, Prior Year
// when building the AOP; empty means show the forecast's own composition.
const [basis, setBasis] = useState(() => localStorage.getItem('pf_bridge_basis') || '')
const [asTable, setAsTable] = useState(false)
const [steps, setSteps] = useState(null)
const [loading, setLoading] = useState(false)
const [error, setError] = useState(null)
const [hover, setHover] = useState(null)
const [width, setWidth] = useState(880)
const boxRef = useRef(null)
// Build the steps: baseline anchor, one floating step per tag, current anchor.
const compute = useCallback(async () => {
if (!tableRef?.current || !valueCol) return
setLoading(true); setError(null)
try {
let rows
if (scope === 'selection') {
// The union of the selected slices — the same reach an operation would
// have. Perspective view filters are AND-only, so each slice needs its own
// view; rows matching more than one slice are counted once.
const dimNames = new Set(colMeta.filter(c => c.role === 'dimension').map(c => c.cname))
const dateNames = new Set(colMeta.filter(c => c.role === 'date').map(c => c.cname))
const seen = new Set()
rows = []
for (const sl of slices) {
const f = [
...Object.entries(sl).filter(([c]) => dimNames.has(c)).map(([c, v]) => [c, '==', v]),
...Object.entries(sl).filter(([c]) => dateNames.has(c)).map(([c, v]) => [c, '==', Number(v)]),
]
if (!f.length) continue
// No expressions needed here: these filters are built from col_meta
// names, so they only ever reference real columns.
const view = await tableRef.current.view({ filter: f })
const part = await view.to_json()
await view.delete()
for (const r of part) {
if (r.pf_id != null && seen.has(r.pf_id)) continue
if (r.pf_id != null) seen.add(r.pf_id)
rows.push(r)
}
}
} else {
let filter = []
// Expression columns have to come with the filter that uses them: a
// filter can name a column that exists only as an expression, and a view
// built without it cannot resolve the column and fails outright.
let expressions = {}
if (scope === 'filtered' && viewerRef?.current) {
const cfg = await viewerRef.current.save()
filter = (cfg.filter || []).filter(f => Array.isArray(f) && f.length >= 2)
expressions = cfg.expressions || {}
}
const viewCfg = {}
if (filter.length) viewCfg.filter = filter
if (Object.keys(expressions).length) viewCfg.expressions = expressions
const view = await tableRef.current.view(viewCfg)
rows = await view.to_json()
await view.delete()
}
setSteps(buildSteps(rows, {
valueCol, unitsCol, logMeta, excludeIters, basis: basis || null, forecastBucket,
}))
} catch (err) {
setError(err.message || String(err))
setSteps(null)
} finally {
setLoading(false)
}
}, [tableRef, viewerRef, scope, logMeta, valueCol, unitsCol, excludeIters, slices, colMeta, basis, forecastBucket])
useEffect(() => {
if (!hasSelection && scope === 'selection') setScope('filtered')
}, [hasSelection, scope])
useEffect(() => { try { localStorage.setItem('pf_bridge_basis', basis) } catch {} }, [basis])
useEffect(() => { if (open) compute() }, [open, compute])
useEffect(() => {
if (!open || !boxRef.current) return
const ro = new ResizeObserver(([e]) => setWidth(Math.max(420, e.contentRect.width)))
ro.observe(boxRef.current)
return () => ro.disconnect()
}, [open])
if (!open) return null
const geom = layoutSteps(steps || [{ start: 0, end: 0 }], width)
const { PAD, plotW, plotH, yMin, yMax, y, barW, H, bars } = geom
const xOf = (i) => bars[i]?.x ?? PAD.l
const ticks = niceTicks(yMin, yMax, 5)
return (
<div className="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onClick={onClose}>
<div className="bg-white rounded-lg shadow-xl w-full max-w-5xl mx-4 flex flex-col max-h-[88vh]"
onClick={e => e.stopPropagation()}>
<div className="flex items-center justify-between px-5 py-3 border-b border-gray-200 shrink-0">
<div className="flex items-baseline gap-2">
<span className="font-medium text-gray-700 text-sm">Bridge</span>
{versionName && <span className="text-gray-600 text-xs">{versionName}</span>}
<span className="text-gray-600 text-xs">
· {scope === 'selection'
? `${slices.length} selected slice${slices.length === 1 ? '' : 's'}`
: scope === 'filtered' ? "pivot's filters" : 'whole version'}
</span>
</div>
<button onClick={onClose} className="text-gray-600 hover:text-gray-800 text-lg leading-none">×</button>
</div>
{/* controls — one row above the chart */}
<div className="flex items-center gap-3 px-5 py-2 border-b border-gray-100 shrink-0 text-xs flex-wrap">
<span className="text-gray-600">Scope</span>
<div className="inline-flex rounded border border-gray-200 overflow-hidden">
{[
['selection', hasSelection ? `Selection (${slices.length})` : 'Selection',
hasSelection ? 'The slices selected in the operation panel'
: 'Select one or more pivot rows first'],
['filtered', "Pivot's filters", 'Everything the pivot currently shows'],
['all', 'Whole version', 'Every row in the version, filters ignored'],
].map(([v, l, title]) => (
<button key={v} onClick={() => setScope(v)} title={title}
disabled={v === 'selection' && !hasSelection}
className={`px-3 py-1 disabled:opacity-40 disabled:cursor-not-allowed ${
scope === v ? 'bg-blue-600 text-white' : 'bg-white text-gray-700 hover:bg-gray-50'}`}>
{l}
</button>
))}
</div>
{/* Offered only once something carries a bucket, and only listing buckets
other than the forecast itself -- a walk from Forecast to Forecast is
the composition view, which is what the empty option gives. */}
{(steps?.buckets || []).some(b => b.name && b.name !== 'Forecast') && (
<>
<div className="w-px h-4 bg-gray-200" />
<span className="text-gray-600">Compare to</span>
<select value={basis} onChange={e => setBasis(e.target.value)}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 bg-white">
<option value="">nothing — show composition</option>
{(steps?.buckets || [])
.filter(b => b.name && b.name !== 'Forecast')
.map(b => <option key={b.name} value={b.name}>{b.name}</option>)}
</select>
</>
)}
<div className="w-px h-4 bg-gray-200" />
<button onClick={() => setAsTable(t => !t)}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 hover:bg-gray-50">
{asTable ? 'Show chart' : 'Show table'}
</button>
<button onClick={compute} disabled={loading}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 hover:bg-gray-50 disabled:opacity-40">
{loading ? 'Computing…' : 'Refresh'}
</button>
{/* legend — identity is never colour alone, but say it anyway */}
<div className="ml-auto flex items-center gap-3 text-gray-700">
{[['Increase', UP], ['Decrease', DOWN], ['Total', ANCHOR]].map(([l, c]) => (
<span key={l} className="inline-flex items-center gap-1.5">
<span className="inline-block w-2.5 h-2.5 rounded-sm" style={{ background: c }} />
{l}
</span>
))}
</div>
</div>
<div className="overflow-auto p-5" ref={boxRef}>
{error && <p className="text-red-600">{error}</p>}
{!error && !steps && <p className="text-gray-600">Computing…</p>}
{!error && steps && steps.length <= 2 && (
<p className="text-gray-600">
No adjustments in scope — the bridge shows the walk from baseline to current,
and this selection has only a baseline.
</p>
)}
{!error && steps && steps.length > 2 && !asTable && (
<div className="relative">
<svg width={width} height={H} role="img"
aria-label={`Bridge from baseline ${fmt(steps[0].end)} to current ${fmt(steps[steps.length - 1].end)}`}>
{/* recessive grid */}
{ticks.map(t => (
<g key={t}>
<line x1={PAD.l} x2={PAD.l + plotW} y1={y(t)} y2={y(t)}
stroke={t === 0 ? '#d1d5db' : GRID} strokeWidth={t === 0 ? 1.5 : 1} />
<text x={PAD.l - 8} y={y(t) + 3} textAnchor="end" fontSize="10" fill={INK_DIM}>
{fmtAxis(t)}
</text>
</g>
))}
{steps.map((s, i) => {
const isAnchor = s.kind === 'anchor'
const up = s.delta >= 0
const fill = isAnchor ? ANCHOR : (up ? UP : DOWN)
const top = y(Math.max(s.start, s.end))
const bot = y(Math.min(s.start, s.end))
const h = Math.max(2, bot - top)
const x = xOf(i)
const on = hover?.key === s.key
const labelLines = wrapLabel(s.label, barW)
const subY = PAD.t + plotH + 16 + labelLines.length * 11 + 2
return (
<g key={s.key}
onMouseEnter={() => setHover({ ...s, x: x + barW / 2, y: top })}
onMouseLeave={() => setHover(null)}>
{/* connector to the next bar, drawn behind */}
{i < steps.length - 1 && (
<line x1={x + barW} x2={xOf(i + 1)} y1={y(s.end)} y2={y(s.end)}
stroke="#cbd5e1" strokeWidth="1" strokeDasharray="2 2" />
)}
{/* hit target larger than the mark */}
<rect x={x - 6} y={PAD.t} width={barW + 12} height={plotH} fill="transparent" />
<rect x={x} y={top} width={barW} height={h} rx="4" fill={fill}
opacity={on ? 1 : 0.92}
stroke="#ffffff" strokeWidth="2" />
{/* direct label: few bars, so every one is labelled */}
<text x={x + barW / 2} y={top - 6} textAnchor="middle" fontSize="10"
fill={INK} fontWeight="500">
{isAnchor ? fmt(s.end, 0) : fmtSigned(s.delta, 0)}
</text>
<text x={x + barW / 2} y={PAD.t + plotH + 16} textAnchor="middle" fontSize="10" fill={INK}>
{labelLines.map((ln, li) => (
<tspan key={li} x={x + barW / 2} dy={li === 0 ? 0 : 11}>{ln}</tspan>
))}
</text>
{/* below the label, however many lines it took */}
{!isAnchor && s.entries > 1 && (
<text x={x + barW / 2} y={subY} textAnchor="middle" fontSize="9" fill={INK_DIM}>
×{s.entries}
</text>
)}
{!s.tagged && !isAnchor && (
<text x={x + barW / 2} y={subY} textAnchor="middle" fontSize="9" fill={INK_DIM}>
untagged
</text>
)}
</g>
)
})}
</svg>
{hover && (
<div className="absolute pointer-events-none bg-white border border-gray-300 rounded shadow-lg px-2.5 py-1.5 text-xs"
style={{ left: Math.min(hover.x + 10, width - 190), top: Math.max(0, hover.y - 10) }}>
<div className="font-medium text-gray-800">{hover.label}</div>
<div className="text-gray-700 font-mono tabular-nums">
{hover.kind === 'anchor' ? fmt(hover.end) : fmtSigned(hover.delta)}
</div>
{hover.kind === 'step' && (
<div className="text-gray-600">
running → <span className="font-mono tabular-nums">{fmt(hover.end)}</span>
</div>
)}
<div className="text-gray-600">
{hover.rows} row{hover.rows === 1 ? '' : 's'}
{hover.entries > 1 ? ` · ${hover.entries} adjustments` : ''}
</div>
</div>
)}
</div>
)}
{/* table view — the same numbers, at full precision */}
{!error && steps && steps.length > 2 && asTable && (
<table className="w-full text-xs">
<thead>
<tr className="text-gray-600 border-b border-gray-200">
<th className="text-left py-1.5 pr-3 font-medium">Step</th>
<th className="text-right py-1.5 px-2 font-medium">{valueCol}</th>
{unitsCol && <th className="text-right py-1.5 px-2 font-medium">{unitsCol}</th>}
<th className="text-right py-1.5 px-2 font-medium">Running</th>
<th className="text-right py-1.5 px-2 font-medium">Adjustments</th>
<th className="text-right py-1.5 pl-2 font-medium">Rows</th>
</tr>
</thead>
<tbody>
{steps.map(s => (
<tr key={s.key} className="border-b border-gray-100">
<td className="py-1.5 pr-3 text-gray-800">
{s.label}{!s.tagged && s.kind === 'step' && <span className="text-gray-600"> · untagged</span>}
</td>
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-800">
{s.kind === 'anchor' ? fmt(s.end) : fmtSigned(s.delta)}
</td>
{unitsCol && (
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-700">
{s.kind === 'anchor' ? fmt(s.units) : fmtSigned(s.units)}
</td>
)}
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-700">{fmt(s.end)}</td>
<td className="py-1.5 px-2 text-right text-gray-700">{s.kind === 'step' ? s.entries : '—'}</td>
<td className="py-1.5 pl-2 text-right text-gray-700">{s.rows}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</div>
</div>
)
}

26
ui/src/components/DepthButtons.jsx
View File

@ -1,26 +0,0 @@
// One axis's collapse control. Rows and columns are the same thing — a depth in
// ViewConfig, counting the levels to show — so they are drawn by one component
// rather than two that happen to look alike.
//
// The buttons are named after the level they reveal. The row axis used to be a
// fixed 0 1 2 3: a number means nothing without already knowing what group_by
// holds, and a fixed range offers levels that do not exist while hiding ones
// that do.
export default function DepthButtons({ label, levels, depth, onPick, totalLabel = 'Total' }) {
return (
<div className="flex items-center gap-1.5">
<span className="pf-section-label">{label}</span>
{Array.from({ length: levels.length + 1 }, (_, n) => (
<button key={n} onClick={() => onPick(n)}
title={n === 0
? `Collapse ${label.toLowerCase()} to a single total`
: `Show ${label.toLowerCase()} down to ${levels.slice(0, n).join(' › ')}`}
className={`border rounded px-1.5 py-0.5 transition-colors max-w-[9rem] truncate
${depth === n ? 'border-blue-300 text-blue-600 bg-blue-50'
: 'border-gray-200 text-gray-500 hover:border-gray-400'}`}>
{n === 0 ? totalLabel : levels[n - 1]}
</button>
))}
</div>
)
}

190
ui/src/components/LayoutMenu.jsx
View File

@ -1,190 +0,0 @@
import { useEffect, useRef, useState } from 'react'
// The pivot layout, as one control rather than a row of equal chips.
//
// Two lists, because a layout is one of two different things: Published, which
// everyone on this forecast sees and only its owner or an admin may change, and
// Mine, which nobody else lists. Everything you cannot write is still there to
// apply -- it is only the write controls that are withheld, so a click never
// answers 403.
export default function LayoutMenu({
layouts = [], activeLayoutId, dirty,
onApply, onSave, onSaveAs, onSetVisibility, onSetDefault, onRename, onDelete, onReset
}) {
const [open, setOpen] = useState(false)
const [saveName, setSaveName] = useState('')
const [savePublic, setSavePublic] = useState(false)
const [renaming, setRenaming] = useState(null)
const [renameTo, setRenameTo] = useState('')
const boxRef = useRef(null)
// Close on an outside click or Escape. Capture phase: the pivot lives in a
// shadow root and retargets its events, so a bubbled listener on document
// sees the host rather than the real target and cannot tell inside from out.
useEffect(() => {
if (!open) return
const onDown = (e) => { if (!boxRef.current?.contains(e.target)) setOpen(false) }
const onKey = (e) => { if (e.key === 'Escape') setOpen(false) }
document.addEventListener('mousedown', onDown, true)
window.addEventListener('keydown', onKey)
return () => {
document.removeEventListener('mousedown', onDown, true)
window.removeEventListener('keydown', onKey)
}
}, [open])
const active = layouts.find(l => l.id === activeLayoutId)
const published = layouts.filter(l => l.visibility === 'published')
const mine = layouts.filter(l => l.visibility === 'private')
const submitSaveAs = () => {
const name = saveName.trim()
if (!name) return
onSaveAs(name, savePublic ? 'published' : 'private')
setSaveName('')
setOpen(false)
}
const submitRename = (l) => {
const name = renameTo.trim()
if (name && name !== l.name) onRename(l, name)
setRenaming(null)
}
const iconBtn = 'px-1 rounded text-gray-400 hover:text-gray-700 hover:bg-gray-100 leading-none'
const row = (l) => (
<div key={l.id}
className={`group flex items-center gap-1 px-2 py-1 rounded cursor-pointer
${l.id === activeLayoutId ? 'bg-blue-50 text-blue-700' : 'hover:bg-gray-50 text-gray-700'}`}
onClick={() => { if (renaming !== l.id) { onApply(l); setOpen(false) } }}>
{renaming === l.id ? (
<input autoFocus value={renameTo} onClick={e => e.stopPropagation()}
onChange={e => setRenameTo(e.target.value)}
onBlur={() => submitRename(l)}
onKeyDown={e => {
if (e.key === 'Enter') submitRename(l)
if (e.key === 'Escape') setRenaming(null)
}}
className="flex-1 min-w-0 border border-blue-300 rounded px-1 py-0 outline-none bg-white" />
) : (
<>
<span className="flex-1 min-w-0 truncate">{l.name}</span>
{l.is_default && (
<span title="Applied when this forecast is opened for the first time"
className="text-amber-500 shrink-0">★</span>
)}
{l.scope === 'source' && (
<span title="Applies to every forecast of this source"
className="text-gray-300 shrink-0" style={{fontSize:'9px'}}>ALL</span>
)}
</>
)}
{/* Withheld rather than offered-and-refused: the server allows a write only
to the owner or an admin, and can_edit says which this is. */}
{l.can_edit && renaming !== l.id && (
<span className="hidden group-hover:flex items-center gap-0.5 shrink-0"
onClick={e => e.stopPropagation()}>
<button className={iconBtn} title="Rename"
onClick={() => { setRenaming(l.id); setRenameTo(l.name) }}>✎</button>
{l.visibility === 'private' ? (
<button className={iconBtn} title="Publish — everyone on this forecast sees it"
onClick={() => onSetVisibility(l, 'published')}>↑</button>
) : (
<>
{!l.is_default && (
<button className={iconBtn} title="Apply this when the forecast is first opened"
onClick={() => onSetDefault(l)}>★</button>
)}
<button className={iconBtn} title="Unpublish — keep it, but only for you"
onClick={() => onSetVisibility(l, 'private')}>↓</button>
</>
)}
<button className={`${iconBtn} hover:text-red-500`} title="Delete"
onClick={() => onDelete(l)}>×</button>
</span>
)}
{!l.can_edit && (
<span className="hidden group-hover:inline text-gray-300 shrink-0 truncate"
style={{fontSize:'9px'}} title={`Published by ${l.owner}`}>{l.owner}</span>
)}
</div>
)
return (
<div ref={boxRef} className="relative flex items-center gap-1.5">
<span className="pf-section-label">Layout</span>
<button onClick={() => setOpen(o => !o)}
className={`flex items-center gap-1 border rounded px-2 py-0.5 max-w-[14rem] transition-colors
${open ? 'border-blue-300 text-blue-700 bg-blue-50'
: 'border-gray-200 text-gray-600 hover:border-gray-400'}`}>
<span className="truncate">{active ? active.name : 'Unsaved'}</span>
{dirty && <span title="Changed since it was saved" className="text-amber-500 leading-none">•</span>}
<span className="text-gray-400 leading-none">▾</span>
</button>
{/* Save is the common action and stays outside the menu, but only when
there is something to save it to and it is yours to overwrite. */}
{active?.can_edit && dirty && (
<button onClick={() => onSave(active)}
className="border border-blue-200 text-blue-600 hover:text-blue-800 rounded px-2 py-0.5">
Save
</button>
)}
{open && (
<div className="absolute top-full left-0 mt-1 z-50 w-72 bg-white border border-gray-200
rounded shadow-lg py-1 text-xs">
{published.length > 0 && (
<>
<div className="px-2 pt-1 pb-0.5 text-gray-400 uppercase tracking-wide"
style={{fontSize:'9px'}}>Published</div>
<div className="px-1">{published.map(row)}</div>
</>
)}
{mine.length > 0 && (
<>
<div className="px-2 pt-2 pb-0.5 text-gray-400 uppercase tracking-wide"
style={{fontSize:'9px'}}>Mine</div>
<div className="px-1">{mine.map(row)}</div>
</>
)}
{!layouts.length && (
<div className="px-3 py-2 text-gray-400">No saved layouts yet</div>
)}
<div className="border-t border-gray-100 mt-1 pt-1 px-2">
<div className="flex items-center gap-1 py-1">
<input value={saveName} onChange={e => setSaveName(e.target.value)}
onKeyDown={e => { if (e.key === 'Enter') submitSaveAs() }}
placeholder="Save current view as…"
className="flex-1 min-w-0 border border-gray-300 rounded px-1.5 py-0.5
outline-none focus:border-blue-400" />
<button onClick={submitSaveAs} disabled={!saveName.trim()}
className="text-blue-600 hover:text-blue-800 px-1 disabled:opacity-30">Save</button>
</div>
<label className="flex items-center gap-1.5 py-0.5 text-gray-500 cursor-pointer">
<input type="checkbox" checked={savePublic}
onChange={e => setSavePublic(e.target.checked)} />
Publish to everyone on this forecast
</label>
</div>
<div className="border-t border-gray-100 mt-1 pt-1 px-2 pb-0.5">
<button onClick={() => { onReset(); setOpen(false) }}
className="text-gray-400 hover:text-gray-700 py-0.5">
Reset to a blank pivot
</button>
</div>
</div>
)}
</div>
)
}

1109
ui/src/components/OperationPanel.jsx
View File

File diff suppressed because it is too large Load Diff

104
ui/src/components/StatusBar.jsx
View File

@ -1,47 +1,12 @@
import { useState, useEffect, useCallback } from 'react'
import useTheme from '../theme.jsx'
import useAuth from '../auth.jsx'
export default function StatusBar({ view, sources = [], sourceId, setSourceId, versions = [], versionId, setVersionId }) {
const { dark, setDark } = useTheme()
const { user, logout } = useAuth()
const showVersion = view === 'baseline' || view === 'forecast'
const selectedVersion = versions.find(v => String(v.id) === String(versionId))
const [info, setInfo] = useState(null)
const [showInfo, setShow] = useState(false)
const [copied, setCopied] = useState(false)
const refreshInfo = useCallback(async () => {
if (!versionId || !showVersion) { setInfo(null); return }
try {
const r = await fetch(`/api/versions/${versionId}/table-info`)
setInfo(r.ok ? await r.json() : null)
} catch { setInfo(null) }
}, [versionId, showVersion])
useEffect(() => { refreshInfo() }, [refreshInfo])
// operations broadcast this after a write so the row count stays honest
useEffect(() => {
const onChange = () => refreshInfo()
window.addEventListener('pf-data-changed', onChange)
return () => window.removeEventListener('pf-data-changed', onChange)
}, [refreshInfo])
async function copyTable() {
if (!info?.fc_table) return
try {
await navigator.clipboard.writeText(info.fc_table)
setCopied(true)
setTimeout(() => setCopied(false), 1200)
} catch {}
}
const fmt = (n) => n == null ? '—' : n.toLocaleString()
return (
<div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs relative">
<div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs">
<span className="text-gray-400">Source</span>
<select
value={sourceId || ''}
@ -73,75 +38,10 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
{selectedVersion.status}
</span>
)}
{/* write target — the physical table every operation appends to */}
{info && (
<>
<span className="text-gray-200">|</span>
<span className="text-gray-400" title="Operations append to this table">writes to</span>
<button
onClick={copyTable}
onMouseEnter={() => setShow(true)}
onMouseLeave={() => setShow(false)}
className={`font-mono px-1.5 py-0.5 rounded border hover:bg-gray-50 ${
info.exists ? 'text-gray-700 border-gray-200' : 'text-amber-700 border-amber-200 bg-amber-50'
}`}
title={info.exists ? 'Click to copy table name' : 'Table does not exist yet'}
>
{copied ? 'copied!' : info.fc_table}
</button>
<span className="text-gray-400 font-mono">
{info.exists ? `${fmt(info.rows)} rows` : 'not created'}
</span>
{showInfo && (
<div className="absolute top-9 left-0 z-30 bg-white border border-gray-200 rounded shadow-lg p-3 text-xs min-w-[260px]">
<div className="text-gray-400 uppercase tracking-wide mb-2" style={{ fontSize: '10px' }}>Write target</div>
<div className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1">
<span className="text-gray-400">table</span>
<span className="font-mono text-gray-700">{info.fc_table}</span>
<span className="text-gray-400">reads from</span>
<span className="font-mono text-gray-700">{info.source}</span>
<span className="text-gray-400">total rows</span>
<span className="font-mono text-gray-700">{fmt(info.rows)}</span>
</div>
{info.by_iter?.length > 0 && (
<>
<div className="text-gray-400 uppercase tracking-wide mt-3 mb-1" style={{ fontSize: '10px' }}>Rows by iter</div>
<table className="w-full">
<tbody>
{info.by_iter.map(r => (
<tr key={r.pf_iter}>
<td className="text-gray-500 capitalize pr-3">{r.pf_iter}</td>
<td className="text-right font-mono text-gray-700">{fmt(r.n)}</td>
</tr>
))}
</tbody>
</table>
</>
)}
</div>
)}
</>
)}
</>
)}
<div className="ml-auto flex items-center gap-2">
{user && (
<>
<span className="text-gray-500" title={`Signed in as ${user.username}`}>
{user.display_name || user.username}
</span>
<button
onClick={logout}
className="text-xs text-gray-500 hover:text-gray-700 border border-gray-200 px-2 py-0.5 rounded"
title="Sign out"
>
Sign out
</button>
</>
)}
<div className="ml-auto">
<button
onClick={() => setDark(d => !d)}
className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100"

27
ui/src/components/Timeline.jsx
View File

@ -32,16 +32,11 @@ function roundRect(ctx, x, y, w, h, r, fill, stroke) {
if (stroke) ctx.stroke()
}
export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDays = 0, type = 'baseline' }) {
export default function Timeline({ dateFrom, dateTo, offsetYr, offsetMo, type = 'baseline' }) {
const canvasRef = useRef(null)
// Months and days are kept apart because they are not interchangeable: a month
// shift lands on the same day of a different month, a day shift can cross a
// month boundary. The preview adds months first, then days, as Postgres does.
const offsetMoTotal = offsetMonths || 0
const offsetDayTotal = offsetDays || 0
const shifted = offsetMoTotal !== 0 || offsetDayTotal !== 0
const twoBands = type === 'baseline' && shifted
const offsetMoTotal = (offsetYr || 0) * 12 + (offsetMo || 0)
const twoBands = type === 'baseline' && offsetMoTotal > 0
const canvasH = twoBands ? 90 : 52
useEffect(() => {
@ -66,9 +61,8 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
const srcEnd = parseDate(dateTo)
if (!srcStart || !srcEnd || isNaN(srcStart) || isNaN(srcEnd)) return
const addDays = (d, n) => { const x = new Date(d); x.setDate(x.getDate() + n); return x }
const projStart = addDays(addMonths(srcStart, offsetMoTotal), offsetDayTotal)
const projEnd = addDays(addMonths(srcEnd, offsetMoTotal), offsetDayTotal)
const projStart = addMonths(srcStart, offsetMoTotal)
const projEnd = addMonths(srcEnd, offsetMoTotal)
const winStart = addMonths(srcStart, -1)
const winEnd = addMonths(twoBands ? projEnd : srcEnd, 1)
@ -153,14 +147,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
ctx.lineTo(px1 - 4, arrowY + 4)
ctx.closePath()
ctx.fill()
const yrs = Math.trunc(offsetMoTotal / 12)
const mos = offsetMoTotal % 12
const sign = (offsetMoTotal + offsetDayTotal) < 0 ? '' : '+'
const offsetLabel = sign + [
yrs ? `${yrs}yr` : '',
mos ? `${mos}mo` : '',
offsetDayTotal ? `${offsetDayTotal}d` : '',
].filter(Boolean).join(' ')
const offsetLabel = '+' + (offsetYr ? offsetYr + 'yr ' : '') + (offsetMo ? offsetMo + 'mo' : '')
ctx.fillStyle = '#64748b'
ctx.font = '9px system-ui'
ctx.textAlign = 'center'
@ -169,7 +156,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
}
raf = requestAnimationFrame(draw)
return () => cancelAnimationFrame(raf)
}, [dateFrom, dateTo, offsetMoTotal, offsetDayTotal, type, twoBands, canvasH])
}, [dateFrom, dateTo, offsetYr, offsetMo, type, twoBands, canvasH])
return <canvas ref={canvasRef} height={canvasH} style={{ width: '100%', display: 'block' }} />
}

14
ui/src/index.css
View File

@ -37,20 +37,6 @@ body { margin: 0; background-color: var(--bg-primary); color: var(--text-primary
.dark .bg-gray-100 { background-color: var(--bg-tertiary); }
.dark .bg-gray-200 { background-color: var(--bg-tertiary); }
.dark .bg-gray-300 { background-color: var(--bg-tertiary); }
/* Toolbar section labels (LAYOUT / ROWS / COLUMNS). Their own class rather than
a grey utility: in dark mode text-gray-400 and text-gray-500 both collapse to
--text-muted, so changing the utility would have fixed light mode only. At
10px/gray-400 these sat near 2.5:1 on white and 2:1 on the dark panel, under
the 4.5:1 WCAG wants for text this size. */
.pf-section-label {
font-size: 11px;
line-height: 1;
letter-spacing: 0.06em;
text-transform: uppercase;
color: #6b7280;
}
.dark .pf-section-label { color: #9aa0aa; }
.dark .text-gray-300 { color: var(--text-muted); }
.dark .text-gray-400 { color: var(--text-muted); }
.dark .text-gray-500 { color: var(--text-muted); }

16
ui/src/main.jsx
View File

@ -1,27 +1,13 @@
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import { ThemeProvider } from './theme.jsx'
import { AuthProvider } from './auth.jsx'
import useAuth from './auth.jsx'
import Login from './views/Login.jsx'
import './index.css'
import App from './App.jsx'
// App is mounted only once there is a session — its load effects call /api
// straight away, and mounting it logged-out would just fire a burst of 401s.
function Gate() {
const { user, checking } = useAuth()
if (checking) return <div className="flex items-center justify-center h-screen text-sm text-gray-400">Loading…</div>
if (!user) return <Login />
return <App />
}
createRoot(document.getElementById('root')).render(
<StrictMode>
<ThemeProvider>
<AuthProvider>
<Gate />
</AuthProvider>
<App />
</ThemeProvider>
</StrictMode>,
)

298
ui/src/views/Baseline.jsx
View File

@ -1,6 +1,5 @@
import { useState, useEffect } from 'react'
import Timeline from '../components/Timeline.jsx'
import useAuth from '../auth.jsx'
const OPERATORS = ['BETWEEN', '=', '!=', 'IN', 'NOT IN', 'IS NULL', 'IS NOT NULL']
@ -49,28 +48,11 @@ function getDateRange(groups) {
return null
}
// The offset is stored and sent as a Postgres interval, so it is typed as one --
// "4 months", "1 year", "-90 days". This only parses it far enough to draw the
// timeline preview; Postgres remains the authority on what is valid, and the
// server rejects anything it will not accept.
//
// It replaced a pair of year/month number spinners, which could not express days
// at all and were clamped at zero, so a segment could only ever be shifted
// forward in whole months.
export function parseInterval(str) {
let months = 0, days = 0
if (!str) return { months, days }
const re = /([+-]?\d+(?:\.\d+)?)\s*(years?|yrs?|y|months?|mons?|mo|weeks?|wks?|w|days?|d)\b/gi
for (const [, n, unit] of str.matchAll(re)) {
const v = parseFloat(n)
const u = unit.toLowerCase()
if (u.startsWith('y')) months += v * 12
else if (u.startsWith('mo') || u === 'mons' || u === 'mon') months += v
else if (u.startsWith('w')) days += v * 7
else if (u.startsWith('d')) days += v
else months += v // 'm' alone reads as months here
}
return { months, days }
function parseOffset(offsetStr) {
if (!offsetStr || offsetStr === '0 days') return { yr: 0, mo: 0 }
const yr = parseInt(offsetStr.match(/(\d+)\s+year/)?.[1] || 0)
const mo = parseInt(offsetStr.match(/(\d+)\s+month/)?.[1] || 0)
return { yr, mo }
}
function emptyCondition(cols) {
@ -87,7 +69,6 @@ function normalizeFilters(stored) {
}
export default function Baseline({ sources = [], sourceId, versions = [], versionId, setVersionId, refreshVersions }) {
const { user: me } = useAuth()
const [filterCols, setFilterCols] = useState([])
const [log, setLog] = useState([])
@ -99,15 +80,13 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
// segment form
const [segType, setSegType] = useState('baseline')
const [description, setDescription] = useState('')
const [filters, setFilters] = useState([]) // [[cond,...], [cond,...]]
const [useRaw, setUseRaw] = useState(false)
const [rawSql, setRawSql] = useState('')
const [offset, setOffset] = useState('0 days')
const [offsetYr, setOffsetYr] = useState(0)
const [offsetMo, setOffsetMo] = useState(0)
const [segNote, setSegNote] = useState('')
// Presentation, not definition: what the segment counts toward and how it is
// labelled in the pivot. Safe to set at any time, unlike its filters.
const [segBucket, setSegBucket] = useState('')
const [segLabel, setSegLabel] = useState('')
const [submitting, setSubmitting] = useState(false)
const [editingLogId, setEditingLogId] = useState(null)
const [showAddForm, setShowAddForm] = useState(false)
@ -119,11 +98,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
useEffect(() => {
if (!sourceId) return
fetch(`/api/sources/${sourceId}/cols`).then(r => r.json()).then(cols => {
// What a load is filtered by and what the pivot groups by are separate
// concerns: a dimension is exactly the sort of thing a segment is cut on
// (sseas, channel_new), and forcing it to role 'filter' to get it here
// would cost it its place on the pivot. Only measures are excluded.
const fc = cols.filter(c => ['date', 'filter', 'dimension'].includes(c.role))
const fc = cols.filter(c => c.role === 'date' || c.role === 'filter')
setFilterCols(fc)
setFilters(fc.length > 0 ? [emptyGroup(fc)] : [])
})
@ -134,65 +109,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
loadLog()
}, [versionId])
// A segment's banner: what it counts toward, independent of pf_iter. Held
// locally while typing so the field does not fight the fetched value, and
// written on blur.
const [buckets, setBuckets] = useState({})
// Buckets already in use on this version, for the datalist. There is no stored
// bucket order any more: the order is whatever the typed text sorts as, so a
// bucket is named "02 - Forecast" and that is the whole mechanism.
const [bucketsInUse, setBucketsInUse] = useState([])
// Label and bucket are presentation, not definition: they change what the pivot
// shows and what the segment counts toward, never which rows were loaded. So
// they stay editable after adjustments exist, unlike the filters and offset,
// where an edit would silently recalibrate scales sized against the old rows.
//
// Both are only read at load time, hence the reload in the confirmation: the
// label is part of the aggregated row the pivot holds, not something it can
// re-derive in place.
// Same rule the server enforces: your own entries, or an admin's. A segment's
// label and bucket name the pivot's columns for everyone in the version, so
// they are not the private annotation they look like.
const canEdit = (entry) => !!me && (me.is_admin || entry.pf_user === me.username)
async function saveLogField(entry, field, value) {
const next = value.trim()
if (next === (entry[field] || '')) return
try {
const res = await fetch(`/api/log/${entry.id}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ [field]: next }),
})
if (!res.ok) { const d = await res.json(); flash(d.error, 'error'); return }
loadLog()
flash(next ? `Saved — reload the Forecast view to see it` : 'Cleared')
} catch (err) { flash(err.message, 'error') }
}
// Column widths read off the content rather than guessed. ch is the width of a
// '0', so for proportional text it runs slightly generous -- which is what is
// wanted for an input you are about to type a longer name into. The floors keep
// an empty table from collapsing its headers; the ceilings keep one long note
// from pushing the numbers off the side.
function widthCh(values, min, max) {
const longest = values.reduce((n, v) => Math.max(n, String(v || '').length), 0)
return `${Math.min(max, Math.max(min, longest + 2))}ch`
}
const labelW = widthCh(log.map(e => e.label || e.tag || e.note), 18, 34)
const bucketW = widthCh(log.map(e => e.bucket), 16, 24)
const noteW = widthCh(log.map(e => e.note), 22, 44)
function loadLog() {
fetch(`/api/versions/${versionId}/log`).then(r => r.json()).then(data => {
setLog(data.filter(e => e.operation === 'baseline' || e.operation === 'reference'))
setHasForecastOps(data.some(e => ['scale', 'recode', 'clone'].includes(e.operation)))
// Every bucket in use, adjustments included: typing one on a new segment
// should offer the ones already there rather than inviting a near-miss
// spelling, which would silently split the column in two.
setBucketsInUse([...new Set(data.map(e => (e.bucket || '').trim()).filter(Boolean))].sort())
})
}
@ -224,14 +144,13 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
const clause = useRaw ? rawSql.trim() : buildFilterClause(filters)
if (!clause) { flash(useRaw ? 'Enter a WHERE clause' : 'Add at least one filter', 'error'); return }
const isRef = segType === 'reference'
const offsetStr = offset.trim() || '0 days'
const offsetStr = [offsetYr > 0 ? `${offsetYr} year` : '', offsetMo > 0 ? `${offsetMo} month` : ''].filter(Boolean).join(' ') || '0 days'
const endpoint = isRef ? 'reference' : 'baseline'
const body = {
where_clause: clause,
note: segNote,
pf_user: 'admin',
note: description || segNote,
date_offset: offsetStr,
label: segLabel.trim(),
bucket: segBucket.trim(),
...(useRaw ? { raw_where: clause } : { filters }),
}
setSubmitting(true)
@ -267,9 +186,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
const params = entry.params || {}
setSegType(entry.operation)
setSegNote(entry.note || '')
setSegLabel(entry.label || '')
setSegBucket(entry.bucket || '')
setOffset(params.date_offset || '0 days')
setDescription('')
const off = parseOffset(params.date_offset)
setOffsetYr(off.yr)
setOffsetMo(off.mo)
const groups = normalizeFilters(params.filters)
if (groups) {
setUseRaw(false)
@ -295,9 +215,8 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
function cancelEdit() {
setEditingLogId(null)
setShowAddForm(false)
setDescription('')
setSegNote('')
setSegLabel('')
setSegBucket('')
setOffsetYr(0)
setOffsetMo(0)
setUseRaw(false)
@ -323,7 +242,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
async function closeVersion() {
const res = await fetch(`/api/versions/${versionId}/close`, {
method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({})
body: JSON.stringify({ pf_user: 'admin' })
})
const data = await res.json()
if (!res.ok) { flash(data.error, 'error'); return }
@ -354,33 +273,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
setTimeout(() => setMsg(null), 3000)
}
// The names a row falls back to when nobody has named it. Blank means "use the
// built-in", so these save on blur like the segment fields and an empty box is
// a meaningful value rather than a missing one.
async function saveVersionName(field, value) {
const next = value.trim()
if (next === (selectedVersion?.[field] || '')) return
try {
const res = await fetch(`/api/versions/${versionId}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ [field]: next }),
})
if (!res.ok) { const d = await res.json(); flash(d.error, 'error'); return }
await refreshVersions(sourceId)
flash('Saved — reload the Forecast view to see it')
} catch (err) { flash(err.message, 'error') }
}
const selectedVersion = versions.find(v => String(v.id) === versionId)
return (
<div className="h-full overflow-y-auto bg-gray-50">
{/* No page-wide cap and no stretching: at max-w-4xl (896px) the eleven-column
segment table always had something smashed, and uncapped it ran to the
window. items-start makes each block as wide as its own content needs,
which for the table is the measured column widths below. */}
<div className="p-4 flex flex-col gap-4 items-start max-w-full">
<div className="p-4 flex flex-col gap-4 max-w-4xl">
{msg && (
<div className={`px-3 py-2 text-xs rounded font-medium ${msg.type === 'error' ? 'bg-red-50 text-red-700' : 'bg-green-50 text-green-700'}`}>
@ -404,31 +301,6 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
)}
</div>
{/* Fallback names. Not part of any segment -- they are what the pivot shows
for rows nobody has named, so they belong to the version rather than to
a log entry. Blank falls back to the built-in in sql_generator's
DISPLAY DEFAULTS block. */}
{versionId && (
<div className="bg-white border border-gray-200 rounded p-3 flex items-end gap-3 flex-wrap">
<span className="text-xs text-gray-500 uppercase tracking-wide w-full">Fallback names</span>
{[
['adjustment_segment', 'Adjustment segment', '99 - Adjustments'],
['adjustment_bucket', 'Adjustment bucket', '04 - Forecast'],
['unlabeled_load', 'Unlabeled load', 'Unlabeled'],
].map(([field, label, builtin]) => (
<div key={field} className="flex flex-col gap-1">
<label className="text-xs text-gray-500">{label}</label>
<input
key={`${field}-${versionId}-${selectedVersion?.[field] || ''}`}
defaultValue={selectedVersion?.[field] || ''}
onBlur={e => saveVersionName(field, e.target.value)}
placeholder={builtin}
className="border border-gray-200 rounded px-2 py-1 text-sm w-48" />
</div>
))}
</div>
)}
{showNewVersion && (
<div className="bg-white border border-gray-200 rounded p-3 flex flex-col gap-3">
<div className="flex items-end gap-3">
@ -454,27 +326,17 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
{versionId && <>
{/* Segments loaded */}
<div className="bg-white border border-gray-200 rounded max-w-full">
<div className="bg-white border border-gray-200 rounded">
<div className="px-3 py-2 border-b border-gray-100 text-xs font-medium text-gray-500 uppercase tracking-wide flex items-center justify-between">
<span>Segments loaded</span>
<button onClick={clearBaseline} className="text-red-400 hover:text-red-600 text-xs normal-case font-normal">Clear all baseline</button>
</div>
<datalist id="pf-bucket-options">
{[...new Set([...bucketsInUse,
'01 - Prior Prior Year', '02 - Prior Year',
'03 - Plan', '04 - Forecast'])].map(b => (
<option key={b} value={b} />
))}
</datalist>
<table className="text-xs">
<table className="w-full text-xs">
<thead className="bg-gray-50">
<tr className="text-left text-gray-400 border-b border-gray-100">
<th className="px-3 py-1.5 font-medium w-6"></th>
<th className="px-3 py-1.5 font-medium">#</th>
<th className="px-3 py-1.5 font-medium w-20">kind</th>
<th className="px-3 py-1.5 font-medium" style={{ width: labelW }}>label</th>
<th className="px-3 py-1.5 font-medium" style={{ width: noteW }}>note</th>
<th className="px-3 py-1.5 font-medium" style={{ width: bucketW }}>counts toward</th>
<th className="px-3 py-1.5 font-medium">note</th>
<th className="px-3 py-1.5 font-medium text-right">rows</th>
<th className="px-3 py-1.5 font-medium text-right">{log[0]?.value_col || 'value'}</th>
<th className="px-3 py-1.5 font-medium">by</th>
@ -484,11 +346,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
</thead>
<tbody>
{log.length === 0 && (
<tr><td colSpan={11} className="px-3 py-3 text-gray-300 italic">No segments loaded yet</td></tr>
<tr><td colSpan={8} className="px-3 py-3 text-gray-300 italic">No segments loaded yet</td></tr>
)}
{!showAddForm && !editingLogId && (
<tr className="border-t border-gray-100">
<td colSpan={11} className="p-0">
<td colSpan={8} className="p-0">
<button
onClick={() => setShowAddForm(true)}
className="w-full px-3 py-2 text-xs text-blue-600 hover:bg-blue-50 text-left font-medium"
@ -510,49 +372,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
>
<td className="px-3 py-2 text-gray-400 w-6"><span className="text-gray-300 text-xs">{isOpen ? '▾' : '▸'}</span></td>
<td className="px-3 py-2 text-gray-400">{log.length - i}</td>
{/* The operation badge gets its own column. Sharing one with
the note put "reference" hard against "YTD Sales" as soon
as the note column lost width to label and bucket. */}
<td className="px-3 py-2">
<span className={`inline-block px-1.5 py-0.5 rounded text-xs font-medium ${entry.operation === 'reference' ? 'bg-purple-50 text-purple-600' : 'bg-blue-50 text-blue-600'}`}>
<span className={`inline-block mr-2 px-1.5 py-0.5 rounded text-xs font-medium ${entry.operation === 'reference' ? 'bg-purple-50 text-purple-600' : 'bg-blue-50 text-blue-600'}`}>
{entry.operation}
</span>
</td>
<td className="px-3 py-2" onClick={e => e.stopPropagation()}>
<input
defaultValue={entry.label || ''}
key={`label-${entry.id}-${entry.label || ''}`}
onBlur={e => saveLogField(entry, 'label', e.target.value)}
readOnly={!canEdit(entry)}
title={canEdit(entry) ? '' : `${entry.pf_user || 'Another account'} made this segment`}
placeholder={entry.tag || entry.note || '—'}
className="w-full border border-transparent hover:border-gray-200
focus:border-blue-400 rounded px-1 py-0.5 text-xs
focus:outline-none bg-transparent" />
</td>
{/* One line, clipped against the measured width above. The
note is provenance and can run long, so left to itself it
wrapped and pushed every row to two or three lines.
Expanding the row shows it in full. The cap is on the div,
not the cell: a max-width on a cell in an auto-layout table
is only a hint, and the column can still collapse to
min-content or grow past it. */}
<td className="px-3 py-2">
{entry.note
? <div className="truncate" style={{ maxWidth: noteW }} title={entry.note}>{entry.note}</div>
: <span className="text-gray-300">—</span>}
</td>
<td className="px-3 py-2" onClick={e => e.stopPropagation()}>
<input
value={buckets[entry.id] ?? entry.bucket ?? ''}
list="pf-bucket-options"
onChange={e => setBuckets(b => ({ ...b, [entry.id]: e.target.value }))}
onBlur={e => saveLogField(entry, 'bucket', e.target.value)}
readOnly={!canEdit(entry)}
title={canEdit(entry) ? '' : `${entry.pf_user || 'Another account'} made this segment`}
placeholder="—"
className="w-full border border-transparent hover:border-gray-200 focus:border-blue-400
rounded px-1 py-0.5 text-xs focus:outline-none bg-transparent" />
{entry.note || <span className="text-gray-300">—</span>}
</td>
<td className="px-3 py-2 text-right text-gray-700 font-mono">
{entry.row_count != null ? entry.row_count.toLocaleString() : '—'}
@ -571,7 +395,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
</tr>
{isOpen && (
<tr key={`${entry.id}-detail`} className="bg-blue-50 border-t border-blue-100">
<td colSpan={9} className="px-2 py-2">
<td colSpan={6} className="px-2 py-2">
<div className="bg-white border border-gray-200 rounded">
<SegmentForm mode="view" {...view} filterCols={filterCols} />
</div>
@ -610,10 +434,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
filters={filters} setFilters={setFilters}
useRaw={useRaw} setUseRaw={setUseRaw}
rawSql={rawSql} setRawSql={setRawSql}
description={description} setDescription={setDescription}
segNote={segNote} setSegNote={setSegNote}
segBucket={segBucket} setSegBucket={setSegBucket}
segLabel={segLabel} setSegLabel={setSegLabel}
offset={offset} setOffset={setOffset}
offsetYr={offsetYr} setOffsetYr={setOffsetYr}
offsetMo={offsetMo} setOffsetMo={setOffsetMo}
filterCols={filterCols}
onSubmit={loadSegment}
submitting={submitting}
@ -632,16 +456,17 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
// derive view-mode props for a saved segment
function segmentValuesFor(entry, filterCols) {
const params = entry.params || {}
const off = parseOffset(params.date_offset)
const groups = normalizeFilters(params.filters)
return {
segType: entry.operation === 'reference' ? 'reference' : 'baseline',
filters: groups || (filterCols.length > 0 ? [emptyGroup(filterCols)] : []),
useRaw: !groups && !!params.where_clause,
rawSql: params.where_clause || '',
description: '',
segNote: entry.note || '',
segBucket: entry.bucket || '',
segLabel: entry.label || '',
offset: params.date_offset || '0 days',
offsetYr: off.yr,
offsetMo: off.mo,
}
}
@ -651,10 +476,10 @@ function SegmentForm({
filters, setFilters,
useRaw, setUseRaw,
rawSql, setRawSql,
description, setDescription,
segNote, setSegNote,
segBucket, setSegBucket,
segLabel, setSegLabel,
offset, setOffset,
offsetYr, setOffsetYr,
offsetMo, setOffsetMo,
filterCols,
onSubmit,
submitting,
@ -724,6 +549,14 @@ function SegmentForm({
</div>
</div>
{/* Description (edit only) */}
{mode === 'edit' && (
<div className="flex items-center gap-3">
<label className="text-xs text-gray-500 w-28 shrink-0">Description</label>
<input value={description} onChange={e => setDescription(e.target.value)} placeholder="e.g. FY25 actuals +1yr" className="border border-gray-200 rounded px-2 py-1.5 text-sm flex-1 max-w-sm" />
</div>
)}
{/* Filters */}
<div>
<div className="flex items-center justify-between mb-2">
@ -823,19 +656,10 @@ function SegmentForm({
<div className="flex items-center gap-3">
<label className="text-xs text-gray-500 w-28 shrink-0">Date offset</label>
<div className="flex items-center gap-2">
<input disabled={disabled} value={offset} list="pf-offset-options"
onChange={e => setOffset(e.target.value)}
placeholder="0 days"
className={`${baseInp} text-sm w-32`} />
<datalist id="pf-offset-options">
<option value="12 months" />
<option value="1 year" />
<option value="4 months" />
<option value="-90 days" />
<option value="-12 months" />
<option value="0 days" />
</datalist>
<span className="text-xs text-gray-400">any Postgres interval</span>
<input disabled={disabled} type="number" value={offsetYr} min={0} onChange={e => setOffsetYr(parseInt(e.target.value) || 0)} className={`${baseInp} text-sm w-16 text-center`} />
<span className="text-xs text-gray-500">yr</span>
<input disabled={disabled} type="number" value={offsetMo} min={0} max={11} onChange={e => setOffsetMo(parseInt(e.target.value) || 0)} className={`${baseInp} text-sm w-16 text-center`} />
<span className="text-xs text-gray-500">mo</span>
</div>
</div>
@ -846,32 +670,16 @@ function SegmentForm({
<Timeline
dateFrom={dateRange.from}
dateTo={dateRange.to}
offsetMonths={parseInterval(offset).months}
offsetDays={parseInterval(offset).days}
offsetYr={offsetYr}
offsetMo={offsetMo}
type={segType}
/>
</div>
</div>
)}
{/* Label, bucket, note + submit.
Label and bucket are presentation: the label is what the pivot shows for
this segment, the bucket is what it counts toward. Both are free text and
both sort by what is typed, so a leading "01 - " is how ordering is set —
which is why they belong here, at the point the segment is defined, as
well as being editable in the list afterwards. */}
<div className="flex items-end gap-3 flex-wrap">
<div className="flex flex-col gap-1 max-w-xs">
<label className="text-xs text-gray-500">Label</label>
<input disabled={disabled} value={segLabel} onChange={e => setSegLabel(e.target.value)}
placeholder="defaults to the note" className={`${baseInp} text-sm py-1.5`} />
</div>
<div className="flex flex-col gap-1 max-w-xs">
<label className="text-xs text-gray-500">Counts toward</label>
<input disabled={disabled} value={segBucket} onChange={e => setSegBucket(e.target.value)}
list="pf-bucket-options" placeholder="e.g. 02 - Forecast"
className={`${baseInp} text-sm py-1.5`} />
</div>
{/* Note + submit */}
<div className="flex items-end gap-3">
<div className="flex flex-col gap-1 flex-1 max-w-xs">
<label className="text-xs text-gray-500">Note</label>
<input disabled={disabled} value={segNote} onChange={e => setSegNote(e.target.value)} placeholder="optional" className={`${baseInp} text-sm py-1.5`} />

2455
ui/src/views/Forecast.jsx
View File

File diff suppressed because it is too large Load Diff

68
ui/src/views/Login.jsx
View File

@ -1,68 +0,0 @@
import { useState } from 'react'
import useAuth from '../auth.jsx'
export default function Login() {
const { login } = useAuth()
const [username, setUsername] = useState('')
const [password, setPassword] = useState('')
const [error, setError] = useState('')
const [busy, setBusy] = useState(false)
async function submit(e) {
e.preventDefault()
setError(''); setBusy(true)
try {
await login(username.trim(), password)
} catch (err) {
setError(err.message)
setPassword('')
} finally {
setBusy(false)
}
}
return (
<div className="flex items-center justify-center h-screen w-full text-sm">
<form onSubmit={submit} className="bg-white border border-gray-200 rounded p-6 w-80 flex flex-col gap-4">
<div>
<div className="text-base font-medium">Pivot Forecast</div>
<div className="text-xs text-gray-500 mt-0.5">Sign in to continue</div>
</div>
<label className="flex flex-col gap-1">
<span className="text-xs text-gray-500">Username</span>
<input
value={username}
onChange={e => setUsername(e.target.value)}
autoFocus
autoComplete="username"
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
/>
</label>
<label className="flex flex-col gap-1">
<span className="text-xs text-gray-500">Password</span>
<input
type="password"
value={password}
onChange={e => setPassword(e.target.value)}
autoComplete="current-password"
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
/>
</label>
{error && (
<div className="px-3 py-2 text-xs rounded font-medium bg-red-50 text-red-700">{error}</div>
)}
<button
type="submit"
disabled={busy || !username.trim() || !password}
className="bg-blue-600 text-white text-xs px-3 py-2 rounded hover:bg-blue-700 disabled:opacity-50"
>
{busy ? 'Signing in…' : 'Sign in'}
</button>
</form>
</div>
)
}

61
ui/src/views/Setup.jsx
View File

@ -25,7 +25,6 @@ export default function Setup({ refreshSources }) {
const [sqlStatus, setSqlStatus] = useState({}) // sourceId -> bool
const [saving, setSaving] = useState(false)
const [generating, setGenerating] = useState(false)
const [refreshingDims, setRefreshingDims] = useState(false)
const [msg, setMsg] = useState(null)
const [dimPeriodCols, setDimPeriodCols] = useState([])
const [openPeriodIdx, setOpenPeriodIdx] = useState(null)
@ -154,35 +153,6 @@ export default function Setup({ refreshSources }) {
}
}
// Rebuild every keyed dim_group's member list from the source. Deliberate rather
// than automatic: it reads the whole source, which for a view over a transaction
// table is millions of rows, and the answer only changes when the catalogue does.
async function refreshDimMembers() {
const groups = [...new Set(cols.filter(c => c.dim_group && c.is_key).map(c => c.dim_group))]
if (groups.length === 0) {
flash('No dim_group has an is_key column, so there is nothing to build a list from', 'error')
return
}
setRefreshingDims(true)
try {
const done = []
for (const g of groups) {
const res = await fetch(`/api/sources/${selectedSource.id}/dim/${encodeURIComponent(g)}/refresh`,
{ method: 'POST' })
const data = await res.json()
if (!res.ok) { flash(`${g}: ${data.error}`, 'error'); return }
done.push(`${g}: ${data.members.toLocaleString()} members`
+ (data.no_longer_in_source ? `, ${data.no_longer_in_source} no longer in source` : '')
+ ` (${(data.ms / 1000).toFixed(1)}s)`)
}
flash(done.join(' · '))
} catch (err) {
flash(err.message, 'error')
} finally {
setRefreshingDims(false)
}
}
async function deleteSource(id, e) {
e.stopPropagation()
if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return
@ -325,17 +295,6 @@ export default function Setup({ refreshSources }) {
{saving ? 'Saving…' : 'Save'}
</button>
)}
{cols.some(c => c.dim_group && c.is_key) && (
<button
onClick={refreshDimMembers}
disabled={refreshingDims || colsDirty}
className="text-xs border border-gray-200 px-3 py-1 rounded hover:bg-gray-50 disabled:opacity-50"
title={colsDirty ? 'Save col meta first'
: 'Rebuild the member list for each keyed dim_group from the source. Reads the whole source, so it takes a while.'}
>
{refreshingDims ? 'Refreshing…' : 'Refresh master data'}
</button>
)}
<button
onClick={generateSQL}
disabled={generating || colsDirty}
@ -354,7 +313,6 @@ export default function Setup({ refreshSources }) {
<th className="px-3 py-1.5 font-medium">role</th>
<th className="px-3 py-1.5 font-medium text-center">key</th>
<th className="px-3 py-1.5 font-medium text-center" title="Include this column in the display grain — the load is pre-aggregated to the flagged columns">grain</th>
<th className="px-3 py-1.5 font-medium text-center" title="The column an account's territory is expressed in. Accounts see and change only rows whose value here is on their list. One per source.">territory</th>
<th className="px-3 py-1.5 font-medium">group</th>
<th className="px-3 py-1.5 font-medium">period col</th>
<th className="px-3 py-1.5 font-medium">label</th>
@ -391,25 +349,6 @@ export default function Setup({ refreshSources }) {
className="cursor-pointer disabled:opacity-20"
/>
</td>
{/* Radio, not a checkbox: exactly one column per source,
and the shape of the control should say so rather than
leaving it to a save-time error. */}
<td className="px-3 py-1.5 text-center">
<input
type="radio"
name="pf-territory-col"
checked={!!col.is_territory}
onChange={() => {}}
onClick={() => setEditedCols(prev => {
// clicking the chosen one again clears it, since a
// radio otherwise has no way back to "no territory"
const already = !!prev[i].is_territory
return prev.map((c, x) => ({ ...c, is_territory: !already && x === i }))
})}
disabled={col.role !== 'dimension'}
className="cursor-pointer disabled:opacity-20"
/>
</td>
<td className="px-3 py-1.5">
<input
type="text"

45
ui/vendor/0001-apply-depth-fields-on-config-update.patch vendored
View File

@ -1,45 +0,0 @@
diff --git a/rust/perspective-client/src/rust/config/view_config.rs b/rust/perspective-client/src/rust/config/view_config.rs
index fa4f36e..360b816 100644
--- a/rust/perspective-client/src/rust/config/view_config.rs
+++ b/rust/perspective-client/src/rust/config/view_config.rs
@@ -526,6 +526,23 @@ impl ViewConfig {
}
}
+ /// `_apply` for a field which is itself `Option`, where `None` in the update
+ /// means "not mentioned" rather than "clear it". `Option<Option<T>>` would be
+ /// needed to express both, and the wire format cannot carry the difference:
+ /// these fields are `skip_serializing_if = "Option::is_none"`, so an absent
+ /// field and an explicit null arrive identically. To lift a depth, send the
+ /// number of levels on that axis rather than clearing it.
+ fn _apply_optional<T: PartialEq>(field: &mut Option<T>, update: Option<T>) -> bool {
+ match update {
+ None => false,
+ Some(_) if *field == update => false,
+ Some(_) => {
+ *field = update;
+ true
+ },
+ }
+ }
+
pub fn reset(&mut self, reset_expressions: bool) {
let mut config = Self::default();
if !reset_expressions {
@@ -568,6 +585,16 @@ impl ViewConfig {
changed = Self::_apply(&mut self.windows, update.windows) || changed;
changed = Self::_apply(&mut self.group_rollup_mode, update.group_rollup_mode) || changed;
changed = Self::_apply(&mut self.split_rollup_mode, update.split_rollup_mode) || changed;
+
+ // Without these two, a depth can be set when a view is created --
+ // `table.view({ group_by_depth })` -- but never through `restore()`, which
+ // merges a `ViewConfigUpdate` onto the live config. The field arrives,
+ // deserializes, and is then dropped here, so the viewer's config and the
+ // engine never see it and nothing happens. That makes expand/collapse
+ // unreachable for anything driven by `restore()`, which is how a viewer
+ // changes its own configuration.
+ changed = Self::_apply_optional(&mut self.group_by_depth, update.group_by_depth) || changed;
+ changed = Self::_apply_optional(&mut self.split_by_depth, update.split_by_depth) || changed;
if self.group_rollup_mode == GroupRollupMode::Total && !self.group_by.is_empty() {
tracing::info!("`total` incompatible with `group_by`");
changed = true;

8
ui/vendor/PROVENANCE.txt vendored
View File

@ -1,8 +0,0 @@
Built from https://github.com/fleetside72/perspective
branch apply-depth-on-config-update
commit 71d1a7508aa86d7e7224ad91e247afe4c201313d
based on unknown
built 2026-09-17T14:34:21Z on usmidsap02
dirty 0 uncommitted file(s) in the source tree at build time
Regenerate with ui/vendor/rebuild-perspective.sh

121
ui/vendor/README.md vendored
View File

@ -1,121 +0,0 @@
# Vendored Perspective
pf_app runs a **patched build of Perspective**. Upstream's C++ engine has always
implemented column-axis expand/collapse — `t_ctx2::set_depth(HEADER_COLUMN, …)`
and `open`/`close(HEADER_COLUMN, idx)` are fully written — but nothing above C++
could reach it: `set_column_pivot_depth()` was never called, and
`View<t_ctx2>::expand/collapse` hardcoded `HEADER_ROW`. The patch is wiring, not
new engine logic.
It buys two things the released packages cannot do at all:
- `split_by_depth` in `ViewConfig`, the `split_by` counterpart to `group_by_depth`
- `expand_column()` / `collapse_column()`, so one column branch can fold to its
subtotal while its siblings stay expanded — the Excel behaviour
**Source:** https://github.com/fleetside72/perspective, branch
`column-axis-expand-collapse`. See `PROVENANCE.txt` for the exact commit these
tarballs were built from.
## Why tarballs and not npm
The feature is not released upstream. Until it is, the four packages are built
from the fork and committed here as npm tarballs. `npm install` expands them
exactly as it expands anything from the registry — no special tooling, and
`pf.sh deploy` works unchanged. A deploy machine needs node and nothing else:
no emscripten, no cmake, no protoc, no Rust.
All four move together, never a subset. Perspective couples loader, package
versions, data format and `apache-arrow`; vendoring a partial set reintroduces
exactly the drift that causes trouble.
## Changing the engine
./rebuild-perspective.sh # builds the fork, repacks, rewrites PROVENANCE.txt
cd .. && npm install
git add vendor && git commit
Push the fork first — the script warns if the source tree is dirty, because a
tarball built from uncommitted code has no recoverable source.
The build itself needs cmake >= 3.29.5, protoc >= 22 (its version silently
selects which protobuf source tree gets cloned), pnpm, and the Rust nightly the
repo pins. Roughly 40 minutes cold. Only ever on a machine changing the engine.
## Getting rid of this
This is a fork, with the maintenance that implies. The exit is upstream taking
the change — the patch is small and additive, and the engine work is already
theirs. When a release ships it, delete this directory and put normal version
ranges back in `ui/package.json`.
## Applied: depth fields on a config update
`0001-apply-depth-fields-on-config-update.patch` is **in** the vendored
tarballs as of the 2026-09-17 build, from branch
`apply-depth-on-config-update`. It fixes an upstream bug in
`rust/perspective-client/src/rust/config/view_config.rs`:
`ViewConfig::apply_update` applies ten fields and **neither `group_by_depth`
nor `split_by_depth` is among them**. So a depth can be set when a view is
created (`table.view({ group_by_depth: 1 })` — which is what the fork's own
`depth_test.mjs` exercises) but never through `restore()`, which is how a
viewer changes its own configuration. The field arrives, deserializes, and is
discarded before the engine sees it.
`group_by_depth` and the omission are both upstream; the fork mirrored
`split_by_depth` alongside it faithfully, including the omission.
Note the semantics, from `server.cpp`:
```cpp
ctx1->set_depth(row_pivot_depth - 1); // one-sided
ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided
```
The config field counts **levels to show**; `view.set_depth()` counts the
boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and
`Forecast.jsx` sends `d + 1`.
To apply:
```bash
cd $PSP_DIR # default ~/perspective
git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
./ui/vendor/rebuild-perspective.sh
cd ui && npm install
```
`cargo check` on this patch was clean — the 125 errors it reports without
`protoc` installed are unresolved generated protobuf modules, none of them in
`view_config.rs`.
`applyDepth()` in `Forecast.jsx` reads the config back after restoring it and
falls back to the imperative `view.set_depth()` when the value did not stick.
With the patched engine installed that fallback should no longer run; it is kept
for now so an unpatched engine still works, and can be removed once the
declarative path is confirmed.
### Host tools this tree needs
The repo pins its Rust toolchain (`nightly-2026-06-01`), Emscripten (4.0.9) and
Binaryen (132) exactly, and pins **nothing** for the host tools — no
`packageManager`, no `.nvmrc`. So `npm i -g pnpm` and `pip install cmake` both
give versions this tree was not written against. The script's preflight now
rejects them, but for the record:
- **pnpm 10.x.** 11+ rejects a repeated `--if-present`, which
`sh_perspective.mjs:189` emits once per package in scope.
- **cmake 3.x**, not 4. Both satisfy the `>= 3.29.5` floor; CMake 4 removed
support for `cmake_minimum_required < 3.5`, which the C++ dependencies still
declare, and fails partway through the Arrow build.
- **protoc >= 22** (33.2 known good).
- **`pnpm pack`, not `npm pack`.** These packages depend on each other as
`workspace:^`, and only pnpm rewrites that to a concrete version. An
`npm pack` tarball is rejected at install: `Unsupported URL Type
"workspace:"`.
Two commits on the build branch are local build concessions rather than
fixes, and should be dropped before the real patch goes anywhere upstream:
removing the `postinstall:playwright` step (it runs `playwright install
--with-deps`, which apt-installs system libraries and so needs root), and
filling in the `allowBuilds` block pnpm 12 demanded.

BIN
ui/vendor/perspective-dev-client-5.4.0.tgz vendored
View File

Binary file not shown.

BIN
ui/vendor/perspective-dev-server-5.4.0.tgz vendored
View File

Binary file not shown.

BIN
ui/vendor/perspective-dev-viewer-5.4.0.tgz vendored
View File

Binary file not shown.

BIN
ui/vendor/perspective-dev-viewer-datagrid-5.4.0.tgz vendored
View File

Binary file not shown.

127
ui/vendor/rebuild-perspective.sh vendored
View File

@ -1,127 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# ---------------------------------------------------------------------------
# rebuild-perspective.sh — rebuild the patched Perspective and re-vendor it
#
# pf_app runs a patched build of Perspective that exposes the column axis
# expand/collapse the engine already implements (split_by_depth, and
# expand_column/collapse_column). Upstream does not ship this yet, so the
# built packages are vendored into this directory as npm tarballs.
#
# Source of truth: https://github.com/fleetside72/perspective
# branch column-axis-expand-collapse
#
# This script exists because vendored binaries are opaque: once the .tgz files
# are committed, nothing in the repo records how to regenerate them. Run this
# after changing the fork, then commit the resulting tarballs.
#
# Only needed on a machine that is changing the engine. Deploys just run
# `npm install`, which expands the committed tarballs - see ../README in this
# directory.
# ---------------------------------------------------------------------------
PSP="${PSP_DIR:-$HOME/perspective}"
BRANCH="${PSP_BRANCH:-column-axis-expand-collapse}"
VENDOR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# The four packages that must move together. Perspective's own docs are
# emphatic that loader, packages, data format and apache-arrow are one unit;
# vendoring a subset would reintroduce exactly the drift that causes trouble.
PACKAGES=(
"rust/perspective-js"
"rust/perspective-server"
"rust/perspective-viewer"
"packages/viewer-datagrid"
)
info() { echo -e "\033[0;34m==>\033[0m $*"; }
ok() { echo -e "\033[0;32m ✓\033[0m $*"; }
die() { echo -e "\033[0;31m ✗\033[0m $*" >&2; exit 1; }
# -- preflight --------------------------------------------------------------
[[ -d "$PSP" ]] || die "No Perspective checkout at $PSP.
git clone https://github.com/fleetside72/perspective.git $PSP
cd $PSP && git checkout $BRANCH
Set PSP_DIR to use a different path."
command -v pnpm >/dev/null || die "pnpm not found. Perspective builds with pnpm, not npm."
# The repo pins its Rust toolchain, emsdk and Binaryen exactly, but pins nothing
# for the host tools -- no packageManager, no .nvmrc -- so `npm i -g pnpm` gets
# whatever is newest and that is not what this tree was written against. pnpm 11+
# rejects a repeated `--if-present`, which sh_perspective.mjs emits once per
# package in scope, and the build dies with:
# error: the argument '--if-present' cannot be used multiple times
pnpm_major=$(pnpm --version | cut -d. -f1)
if (( pnpm_major > 10 )); then
die "pnpm $(pnpm --version) is too new; this tree needs pnpm 10.x.
npm i -g pnpm@10"
fi
command -v protoc >/dev/null || die "protoc not found.
Its VERSION selects which protobuf source tree the build clones, and a
version below 22 pulls a layout the build cannot consume. Needs >= 22
(33.2 known good). Distro packages are usually far too old."
cmake_ver=$(cmake --version 2>/dev/null | head -1 | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?') || die "cmake not found"
cmake_major=${cmake_ver%%.*}; cmake_minor=$(echo "$cmake_ver" | cut -d. -f2)
if (( cmake_major < 3 || (cmake_major == 3 && cmake_minor < 29) )); then
die "cmake $cmake_ver is too old; Perspective needs >= 3.29.5.
A user-level install works: pip3 install --user 'cmake==3.31.*'"
fi
# And not too new: CMake 4 dropped compatibility with cmake_minimum_required
# below 3.5, which Perspective's C++ dependencies still declare. `pip install
# cmake` gives 4.x by default, which satisfies the floor above and then fails
# in the middle of the Arrow build.
if (( cmake_major >= 4 )); then
die "cmake $cmake_ver is too new; CMake 4 removed support for
cmake_minimum_required < 3.5, which the C++ dependencies still use.
pip3 install --user 'cmake==3.31.*'"
fi
info "Perspective checkout: $PSP"
git -C "$PSP" rev-parse --abbrev-ref HEAD | grep -qx "$BRANCH" \
|| echo " ! on branch $(git -C "$PSP" rev-parse --abbrev-ref HEAD), expected $BRANCH"
commit=$(git -C "$PSP" rev-parse --short HEAD)
dirty=$(git -C "$PSP" status --porcelain | wc -l)
echo " commit $commit$([[ $dirty -gt 0 ]] && echo " (+$dirty uncommitted files)")"
# -- build ------------------------------------------------------------------
# `metadata` first: it generates docs/expression_gen.md, which perspective-client
# includes at compile time. Building a scope without it fails on the missing file.
info "Building (this takes ~40 minutes cold, a few minutes warm)…"
( cd "$PSP" && PSP_ONCE=1 PACKAGE="metadata,server,client,viewer,viewer-datagrid" pnpm run build )
ok "build complete"
# -- pack -------------------------------------------------------------------
# pnpm pack, not npm pack. These packages depend on each other as
# `workspace:^`, and only pnpm rewrites that to the concrete version on pack.
# npm leaves it, and npm install then refuses the tarball outright:
# npm error Unsupported URL Type "workspace:": workspace:^
info "Packing tarballs into $VENDOR"
rm -f "$VENDOR"/*.tgz
for p in "${PACKAGES[@]}"; do
( cd "$PSP/$p" && pnpm pack --pack-destination "$VENDOR" >/dev/null )
ok "$(basename "$p")"
done
# -- record provenance ------------------------------------------------------
# A committed .tgz is an opaque binary; without this the tie back to source is
# only in someone's memory.
cat > "$VENDOR/PROVENANCE.txt" <<EOF
Built from https://github.com/fleetside72/perspective
branch $BRANCH
commit $(git -C "$PSP" rev-parse HEAD)
based on $(git -C "$PSP" describe --tags --abbrev=0 2>/dev/null || echo 'unknown')
built $(date -u +%Y-%m-%dT%H:%M:%SZ) on $(hostname)
dirty $dirty uncommitted file(s) in the source tree at build time
Regenerate with ui/vendor/rebuild-perspective.sh
EOF
echo
ls -la "$VENDOR"/*.tgz | awk '{printf " %-52s %5.1f MB\n", $NF, $5/1048576}'
echo
ok "Done. Now: cd ui && npm install && git add vendor && git commit"
[[ $dirty -gt 0 ]] && echo -e "\033[1;33m !\033[0m source tree had uncommitted changes — push them to the fork first"
exit 0