Compare commits

..

1 Commits

Author SHA1 Message Date
e6e2faeb37 Re-check the selected source and version against the live list
The selection is restored from localStorage, but it was only validated once
at mount. Deregistering the selected source left App holding an id that no
longer exists, so every subsequent call 404'd "Source not found" with no way
out but a reload — Setup.deleteSource clears its own selectedSource and never
tells App. Deleting the selected version had the same shape.

Move both checks into effects keyed on the lists themselves. A sourcesLoaded
flag keeps the source effect from firing against the initial empty array and
wiping the restored id before the fetch resolves.

Also coerce both refresh callbacks to an array. A 401 returns an error object,
and data.some() then threw inside an unhandled promise, stranding the
selection instead of clearing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 21:30:56 -04:00
45 changed files with 948 additions and 9368 deletions

View File

@ -4,18 +4,3 @@ DB_NAME=your_database
DB_USER=your_user DB_USER=your_user
DB_PASSWORD=your_password DB_PASSWORD=your_password
PORT=3010 PORT=3010
# Signs the session cookie. Generate with:
# node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))'
# or let ./pf.sh config do it. Changing it signs everyone out.
SESSION_SECRET=
# Send the session cookie over HTTPS only. Keep true behind a TLS proxy;
# set false only to reach the app over plain HTTP on a trusted network.
COOKIE_SECURE=true
# Reverse proxy hops express should trust for req.ip / protocol. Default 1.
#TRUST_PROXY=1
# Only needed if the UI is served from a different origin than the API.
#CORS_ORIGIN=https://forecast.example.com

1
.gitignore vendored
View File

@ -1,4 +1,3 @@
node_modules/ node_modules/
.env .env
.env.*
public/app/ public/app/

460
CLAUDE.md
View File

@ -14,7 +14,7 @@ Data transport architecture options: `pf_perspective_options.md`
- **Backend:** Node.js / Express (`server.js`) - **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema - **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/` - **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`. - **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/` - **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
--- ---
@ -22,35 +22,25 @@ Data transport architecture options: `pf_perspective_options.md`
## Project layout ## Project layout
``` ```
server.js Express entry point; pg pool; session; type parsers for bigint/numeric server.js Express entry point; pg pool; type parsers for bigint/numeric
routes/ routes/
auth.js POST /api/login, /api/logout, GET /api/me; login throttle
tables.js GET /api/tables, /api/tables/:schema/:tname/preview tables.js GET /api/tables, /api/tables/:schema/:tname/preview
sources.js Source registration, col_meta, SQL generation sources.js Source registration, col_meta, SQL generation
versions.js Version CRUD, baseline/reference load, data stream versions.js Version CRUD, baseline/reference load, data stream
operations.js scale, recode, clone, undo — the core forecast ops operations.js scale, recode, clone, undo — the core forecast ops
log.js GET /api/versions/:id/log, DELETE /api/log/:logid log.js GET /api/versions/:id/log, DELETE /api/log/:logid
layouts.js Named pivot layouts — list per version, create, patch, delete
lib/ lib/
sql_generator.js buildFilterClause, token substitution helpers sql_generator.js buildFilterClause, token substitution helpers
auth.js scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
utils.js utils.js
setup_sql/ setup_sql/
01_schema.sql pf schema DDL — run once to install 01_schema.sql pf schema DDL — run once to install
02_auth.sql pf.app_user + pf.session
ui/src/ ui/src/
auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login
views/ views/
Login.jsx Sign-in form
Setup.jsx DB browser, source registration, col_meta editor Setup.jsx DB browser, source registration, col_meta editor
Baseline.jsx Version management, baseline workbench, reference load Baseline.jsx Version management, baseline workbench, reference load
Forecast.jsx Perspective pivot, selection handling, operation dispatch Forecast.jsx Perspective pivot + operation panel (Scale/Recode/Clone)
components/
LayoutMenu.jsx The Layout ▾ control — Published / Mine, with the write actions
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms
BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
Sidebar.jsx 3-step collapsible nav Sidebar.jsx 3-step collapsible nav
StatusBar.jsx Source · version · write target · row counts · theme StatusBar.jsx Source · version · row count · status
Timeline.jsx Date-range preview bar for baseline segments Timeline.jsx Date-range preview bar for baseline segments
``` ```
@ -59,43 +49,13 @@ ui/src/
## Database schema (`pf`) ## Database schema (`pf`)
- **`pf.source`** — registered source tables - **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below); `is_key` is described under §`is_key` below - **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations - **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it. - **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb - **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.layout`** — named Perspective view configs; see §Pivot layouts
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time - **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`
- **`pf.session`** — express-session store (connect-pg-simple layout)
- **`pf.dim_period`** — calendar lookup table (20182035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month - **`pf.dim_period`** — calendar lookup table (20182035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
### `is_key`
Read in four places, and **not** the one it sounds like — slices are validated
against `filterCols`, which is every `dimension` plus every `date` column
regardless of `is_key`.
1. **The key of a `dim_group`**`resolveGroup()` in `routes/sources.js` takes
`members.find(c => c.is_key)`, the column every other member is keyed on for
`pf.dim_member`
2. **Value completion** — only `is_key` columns get a dropdown, and
`/sources/:id/values/:col` refuses anything else
3. **Sibling autofill** — fires on blur only when `is_key && dim_group`
4. **The `dim_period` anchor**`role === 'date' && is_key && dim_group` picks the
date whose siblings are derived from the calendar
Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group.
**When a group has more than one, `.find()` silently takes the lowest `opos`.**
That is not hypothetical: `segment_new` (opos 12) outranked `part` (opos 15) in
the `part` group, so a refresh keyed on a column that is null throughout, matched
nothing, and reported success with zero members. `customer`'s group has four keys
and picks the right one only by `opos` luck.
The two meanings want separating — a per-group key choice, or a rule that the
group key is the column named by the group (which these groups nearly follow
already, except `sdate``sdate_e`). Until then, a refresh that finds two keys
should refuse rather than guess.
### Key token substitution tokens ### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}` `{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
@ -104,280 +64,21 @@ should refuse rather than guess.
## Core data flow ## Core data flow
### Initial load (Forecast view) ### Initial load (Forecast view)
`Forecast.jsx` fetches col_meta first, then picks the endpoint: `GET /api/versions/:id/data` → Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM
- **grain mode** (any `in_grain` column) — `GET /api/versions/:id/agg`, rows pre-aggregated to the grain, table indexed on `pf_gkey`
- **raw mode** (no grain) — `GET /api/versions/:id/data`, raw forecast rows, table indexed on `pf_id`
Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM. `fetchArrow()` handles both.
**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch. **Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.
### Display grain
Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings.
### Segment and note columns
`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table
does not itself carry:
- **`pf_segment`** — `pf.log.label`, else `tag`, else `note`, else `Unlabeled`;
`'99 - Adjustments'` for an adjustment that has no label of its own
- **`pf_note`** — the free text on a scale/recode/clone; null on loads
They are deliberately separate: commingling a segment name with an adjustment note
makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join
adds no rows. The operation routes stamp the same two fields onto the rows they push
back incrementally, since those come from `RETURNING *` and would otherwise arrive
without them.
### Column order is stored text
Perspective orders column groups by the value string, and `SortDir`'s `col asc` /
`col desc` only reverses that — so Prior Year → Plan → Actual → Forecast is
alphabetical in neither direction and expressible as neither. A `"01 - "` prefix
is the only lever, and it lives in **`pf.log.label`** (and `pf.log.bucket`),
typed by whoever names the segment. Nothing derives it.
`SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in `lib/sql_generator.js` are the
single definition, shared with the `/data` cursor in `routes/operations.js`
`/agg` is generated, `/data` is not, and the two have to agree.
The one hardcoded ordinal is `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`, which
keeps unlabelled adjustments after every *numbered* segment. That proviso is the
whole scheme, not a caveat on it: ordering is string ordering, so `99` only lands
last once the loads carry `01``0n`, and an unnumbered segment sorts after it
(digits precede letters — `9` is `0x39`, `A` is `0x41`). The old `'(adjustment)'`
sorted *first* for the same reason read the other way, `(` being `0x28`.
Unlabelled loads read plain `Unlabeled` and so land at the very end, which is
where a segment nobody has named belongs. Labelling an adjustment's own log row
overrides the fallback, which is how one kind of adjustment is split out from the
rest.
### Hardcoded display names
Every name the pivot can show that does not come from `pf.log`. If a segment or
bucket appears under a name nobody typed, it is one of these. All three are in
the `DISPLAY DEFAULTS` block at the top of `lib/sql_generator.js`, exported so
the `/data` cursor and the operation routes' incremental row stamps use the same
values the generated `/agg` does.
| constant | `pf.version` column | built-in | applies to |
|---|---|---|---|
| `ADJUSTMENT_SEGMENT` | `adjustment_segment` | `99 - Adjustments` | `pf_segment` for a scale/recode/clone with no `label` |
| `ADJUSTMENT_BUCKET` | `adjustment_bucket` | `04 - Forecast` | `pf_bucket` for a scale/recode/clone with no `bucket` |
| `UNLABELED_LOAD` | `unlabeled_load` | `Unlabeled` | `pf_segment` and `pf_bucket` for a load with no `label`, `tag` or `note` |
Each is set per scenario on the Baseline page, under **Fallback names**; blank
falls back to the built-in. Anything typed on the log row overrides both, so
none of these appears once a segment is named.
**Why the join rather than a token.** `pf.sql` is keyed on
`(source_id, operation)` — one template shared by every version of a source — so
a value baked in at Generate SQL time could not vary by version, and
regenerating for one version would silently change the others. The names are
therefore read through `VERSION_JOIN` at query time, which also means changing
one takes effect on the next load with nothing regenerated.
The built-ins are still a convention guess: `ADJUSTMENT_BUCKET`'s `04 - ` only
suits one numbering. A version that numbers its buckets differently sets its
own rather than inheriting that.
**What this replaced.** The prefix used to be computed client-side, as
Perspective expression columns (`pf_bucket_ord`, `pf_segment_ord`) built from
`pf.log.seq` and `pf.version.bucket_order`. It ordered the pivot and nothing
else, so every other reader disagreed with it; `restore()` replaces
`expressions` wholesale, so it had to be re-applied after every layout load; and
ExprTK's string scanner tests each *byte* with `isprint()`, so a label
containing anything outside printable ASCII could not be ordered at all (`·` is
two bytes, of which `isprint(0xC2)` is false). `DEAD_ORDER_EXPRS` in
`Forecast.jsx` strips the expression names out of layouts saved under that
scheme. `pf.log.seq` and `pf.version.bucket_order` are no longer read; the
columns remain.
Relabelling now needs a page reload to show, because the label is part of the
aggregated row rather than something the pivot can re-derive. Editing a label
afterwards is a `PATCH /api/log/:logid` and needs nothing regenerated, but a
source registered before this change needs **Generate SQL** run once, so its
stored load templates write `label` and `bucket` onto the log row at all.
### Forecast operations ### Forecast operations
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums. POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload.
### Price or volume (`plug`)
A sales figure alone does not say which of price or volume moved, so scale takes
`plug``'price'` (default, volume holds) or `'volume'` (price holds, units scale
in proportion). Resolved in `resolveIncrs()` in `routes/operations.js`; the panel
only offers it when the edit is dollars-only, because naming units or price has
already answered it. The semantics come from the predecessor Excel model,
`/opt/forecast_api/VBA/fpvt.frm``calc_val` / `calc_price`:
```
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
```
A `target_price` with a `target_units` alongside is that form's Edit Price mode,
where both are inputs and dollars fall out. The ledger's **Result** line previews
value, units and price together using the same rules, so what you see is what
gets written.
### Undo ### Undo
`DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload. `DELETE /api/log/:logid` → removes rows by logid → **full Perspective reload** (known wart).
---
## Axis depth (collapse / expand)
Both axes collapse the same way: a **depth in `ViewConfig`**, set through
`restore()`.
- **Rows**`group_by_depth`, driven by the `EXPAND 0 1 2 3` buttons via
`applyDepth()`
- **Columns**`split_by_depth`, driven by the `COLUMNS` buttons via
`applySplitDepth()`
Both are **1-based**: they count the levels to show, where the imperative
`view.set_depth()` counts the boundary below them. `server.cpp` does
`ctx1->set_depth(row_pivot_depth - 1)` and
`ctx2->set_depth(HEADER_COLUMN, column_pivot_depth - 1)`, so the toolbar sends
`d + 1` and subtracts one again when reading a layout back.
Because a depth lives in the config, it survives every view rebuild, rides into
the persisted and named layouts through `viewer.save()`, and needs nothing
re-applied afterwards.
**This required patching the engine.** `ViewConfig::apply_update` applied ten
fields and neither depth, so a depth sent through `restore()` was accepted,
deserialized and dropped — and since `Session::update_view_config` returns early
when `apply_update` reports no change, no view was rebuilt at all. See
`ui/vendor/0001-apply-depth-fields-on-config-update.patch`; the vendored build
carries it.
### What this replaced
Worth knowing, because a lot of machinery existed to work around it and is now
gone:
- Row depth used to be imperative — `getView()` then `view.set_depth()` — which
put it on an object the viewer discards whenever it re-renders. Restoring it
meant guessing when that had happened: an `observerShim.js` patching
`window.IntersectionObserver` and `window.ResizeObserver`, focus and
visibility listeners, a retry loop for `getView()` throwing `No table set`
while `getTable()` resolved, and a flag tracking whether the viewer had "gone
away". That guesswork produced three distinct visible faults — the tree fully
expanding on refocus, snapping on any reflow, and snapping when Perspective's
settings sidebar opened.
- Column collapse used to restore a **truncated `split_by`**. The discarded
levels therefore had to be remembered separately (`splitFull`, persisted as
`split_full`), our own collapse had to be told apart from the user rearranging
the pivot (`collapsingRef` plus a prefix test), and the selection was cleared
on every collapse because the axis was changing shape.
`split_full` is still *read* on load, for layouts saved under the old scheme.
### Auto-pause is off
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on
itself plus the document's `visibilitychange` drive `AutoPauseState::apply()`,
and pausing **deletes the view** (`session.set_pause(true)` →
`view_sub.take().delete()`). Returning to the tab is therefore not a redraw but
`restore_and_render()` — a fresh view and a fresh traversal of the whole grain,
which on a large one is a multi-second chug on every tab switch, and takes any
per-node expansion with it.
`initViewer()` calls `viewer.setAutoPause(false)` right after `viewer.load()`.
Nothing updates the table while the tab is hidden — every operation is driven
from this page — so the pause bought nothing and cost a rebuild. The view is now
held while the tab is backgrounded.
### Still not solved: per-node expansion
Expanding one specific branch is view state with no config representation, and
the API has **no getter**:
```
expand(row_index: number): Promise<number>
collapse(row_index: number): Promise<number>
```
It can be set but not read, so it cannot be captured and replayed — this is why
it is lost on every rebuild, and why no client-side fix has worked. The honest
route is a `ViewConfig` field carrying expanded row *paths* (indices shift as
the tree opens), applied in `server.cpp` where the depths are. Bigger than the
depth patch: it needs a way to enumerate expanded nodes in the C++ traversal, a
proto field, and the apply step.
The fork's own `header_click.ts` / `expand_column` / `collapse_column` is the
column-axis equivalent and has the same limitation.
**Limitation that remains either way:** depth is whole-axis. Excel can collapse
2025 while 2026 stays expanded; a depth collapses every group at that level
together. `columns` selects which *measures* appear, not individual split
combinations.
## Pivot layouts
A layout is a named `ViewConfig`, stored in **`pf.layout`** and owned by an
account. Two kinds, one table:
- **published** — everyone on the forecast lists it and can apply it; only its
owner or an admin may change it
- **private** — yours, nobody else lists it
Scope is the **version** by default, because that is where people enter the app.
`version_id IS NULL` means the layout applies to every version of the source;
that is where the old source default went, and what a brand-new version picks up
before anyone has published anything for it. `is_default` (at most one per scope,
by partial unique index) is what `initViewer()` restores on a first load — a
version-scoped default beating a source-wide one, the narrower answer winning.
**Permissions are the `pf.log` rule verbatim** — your own, or an admin's
override, and the UI greys out the rest rather than offering a click that answers
403. `can_edit` rides on every row so the menu knows which. Applying is never
restricted: the guarantee is that a published layout cannot be *changed* out from
under people, not that it cannot be adapted — Save is withheld on a layout that
isn't yours, Save as… forks it into your own.
**No territory clause.** A layout is display config, and territory restricts rows,
not columns; `cleanLayout()` already drops anything the live schema lacks.
**What is still local.** `LAYOUT_KEY` (`pf_layout_v{vid}`) — the unnamed
last-used config — stays in `localStorage`, because it is per-browser session
continuity rather than a thing anyone names or shares. `LAYOUTS_KEY`
(`pf_layouts_v{vid}`) is the old named list; `loadLayouts()` lifts it into
`pf.layout` as private rows once per version and then clears the key.
**The dirty dot is a comparison, not a flag.** `restore()` itself fires
`perspective-config-update`, so anything set unconditionally in that handler
would light up the moment a layout was applied. `activeConfigRef` holds what the
pivot last matched and `sameConfig()` compares against it, ignoring `table`
(the per-load table name, different on every refresh).
**What this replaced.** Named layouts lived only in `localStorage` — invisible to
anyone else, gone on another machine. The one server-side layout was
`pf.source.default_layout`, a single anonymous blob that `PUT
/sources/:id/default-layout` let *any* account overwrite for *every* account: a
published layout with no owner. That route is gone; the column remains, migrated
and read by nothing.
`cleanLayout()` guards `aggregates` along with the axes — including the weight
column of the multi-arg form — and drops the offending *entry* rather than the
layout, since `restore()` is all-or-nothing.
--- ---
## Slice mechanics ## Slice mechanics
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side). When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` columns are kept as the slice. This slice populates the operation panel and is sent as the `slice` object in all operation POST bodies.
Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow**`{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.
Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".
**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.
`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.
**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction. **Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.
@ -395,97 +96,6 @@ All three operations follow the same structure: insert a `pf.log` row in a CTE,
--- ---
## Authentication
Everything under `/api` except the auth routes sits behind a session; the React
app is only mounted once there is one (`Gate` in `main.jsx`), because its load
effects call the API immediately.
- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
the password is read on stdin and hashed before it reaches psql.
- **Sessions:** `express-session` + `connect-pg-simple` in `pf.session`, so a
restart doesn't sign everyone out and a session can be revoked by deleting its
row (`disable-user` does exactly that). Cookie `pf.sid`: httpOnly, SameSite=Lax,
Secure unless `COOKIE_SECURE=false`, 12h rolling.
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
`TRUST_PROXY` (default 1) makes `req.ip` and secure-cookie detection correct
behind the TLS proxy. `CORS_ORIGIN` is the only way CORS is enabled at all; a
wildcard origin plus a session cookie would be cross-site request forgery by
construction.
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
regenerates the session id on success.
**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
`sessionUser(req)`, never from the request body — the UI used to send a hardcoded
`pf_user: 'admin'`, which any client could have set to anything. The audit log
now names the account that made the change.
## Source columns come from pg_catalog
`information_schema.columns` omits **materialized views** — they are not in the
SQL standard — and `gs.osm_skinny` is one. So the source the whole app is built
on looked like it had no columns: registering it seeded nothing, and creating a
version failed with "No usable columns in col_meta" while col_meta plainly held
thirty-six.
`RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version
creation, source registration and the table preview. It returns the same shape
information_schema did, so `mapType` and the callers were unchanged:
`data_type` is `format_type` with the modifier stripped, which gives the same
spelling (`character varying`, `numeric`), and precision and scale are unpacked
from `atttypmod`. The table browser lists from `pg_class` by `relkind` for the
same reason.
## Territory scoping
An account sees and changes only its own territory. The list lives on
`pf.app_user.territory` (jsonb array) with `is_admin` for the accounts that see
everything, and `col_meta.is_territory` marks which column of a given source
the values belong to — one per source, flagged rather than named in code so a
second source can be divided by something other than a sales rep.
**Fail closed.** No territory and not an admin means no rows.
`buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an empty list or an
unflagged source: an account somebody forgot to configure sees nothing instead
of the whole book.
**Built from the session, never the request.** This is the difference between
it and `scope`, which the browser sends and which is right to send, being a
filter the user chose. A permission cannot come from the thing it restrains, so
the territory predicate is ANDed on last, in `sliceUnits()` for writes and per
route for reads, where nothing in the payload can remove it.
Enforced at:
- `/data` — clause on the cursor *and* on the count behind `X-Row-Count`
- `/agg` — a `{{territory_clause}}` token applied **before** the GROUP BY, since
the territory column need not be part of the grain and may not survive it
- every operation, through `sliceUnits()`
- `/sources/:id/values/:col` — completion reads the *source* table, which no
scope has touched, so without it a dropdown enumerates the whole business
- `DELETE /log/:logid` and `PATCH /log/:logid` — by owner, not territory. Undo
removes an entry's rows wholesale, and half-undoing one would leave a state
nothing describes. The PATCH looks like a private annotation and is not:
`label` and `bucket` name the pivot's columns for everyone in the version, so
unguarded it let any account rename the company's segments. Your own entries,
or an admin's override, and the UI greys out the rest rather than offering a
click that answers 403.
- recode's `set` — a scoped account cannot set the territory column at all.
Moving a row between territories is reassignment, not forecasting, and it
would vanish from the view that would have shown what happened.
**The change log shows an entry's full impact**, not the reader's share. The
totals stamped on `pf.log` are company-wide, so an admin's version-wide scale
reads the same in every account — deliberate, and labelled, rather than
re-aggregating per territory.
Managed with `./pf.sh set-territory | set-admin | orphan-territory`.
`orphan-territory` lists values present in the data that no account owns; work
under one is invisible to everyone but an admin, which a typo causes easily and
nothing inside the app reveals.
## Light / dark mode ## Light / dark mode
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`. Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
@ -497,60 +107,16 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires - **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()` - **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
## After a change: restart, and Generate SQL
Two steps, and which one you need depends on what changed.
- **Restart the server** for anything in `routes/`, `lib/` or `server.js`.
- **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes.
The templates are *stored* in `pf.sql`, so editing the generator changes
nothing until they are rebuilt — and a template carrying a token the running
code does not substitute fails at the database rather than in JS, which reads
as an unrelated client-side error.
The order matters: restart first, then Generate SQL, or the old code writes the
templates.
Schema changes are applied to the live database directly and mirrored into
`setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a
migration runner, so running it is not how an existing database gets a new
column.
## Known issues / active work ## Known issues / active work
- **Zero-row operations report success.** Scale refuses with "Nothing to - Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
scale…" when its slice matches nothing; recode and clone commit an empty log - Load progress bar is jittery — needs throttle (~10 updates/sec)
entry and return `rows_affected: 0`. A recode of a rep whose rows are all
`reference` looked like it worked and did nothing
- **The change log does not show an entry's id**, so there is no way to name
one when asking about it
- **Territory is read onto the session at login**, so granting or changing one
does not reach a signed-in account until it signs in again. Re-reading it per
request in `requireAuth` would also make disabling someone immediate
- **Depth buttons rebuild the view.** A depth lives in `ViewConfig`, so changing
it goes through `restore()` and `Session::update_view_config` tears down and
rebuilds the view — a full traversal — where a manual collapse mutates the
existing one in place. Noticeable on a large grain. The fix is to detect a
depth-only change and call `view.set_depth()` imperatively while still writing
it to the config, at the cost of the two being able to drift
- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29`
(2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in
pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then
~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The
remaining lever is **dynamic grain** — group by the fields the current pivot
actually uses rather than every `in_grain` column; see `pf_spec.md`
§Display-grain pre-aggregation, "the dynamic variant"
- Per-node expand/collapse is lost whenever the view rebuilds — set-only API, no
getter; see §Axis depth
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions) - Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
- Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`, - Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`,
`App.jsx`). It is re-validated against the live list whenever that list changes, so a `App.jsx`). It is re-validated against the live list whenever that list changes, so a
deregistered source or deleted version re-points at the first remaining one instead of deregistered source or deleted version re-points at the first remaining one instead of
leaving a dead id that 404s every call leaving a dead id that 404s every call
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version - Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
- Grain drift: changing `in_grain` after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. `routes/log.js` derives the grain from live col_meta, so a grain changed mid-session yields `pf_gkeys` that don't match the loaded table and undo silently removes nothing
- Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see `pf_spec.md` → §Display-grain pre-aggregation
## Deferred (not in v1) ## Deferred (not in v1)
Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`. Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections.

View File

@ -47,27 +47,8 @@ import '@perspective-dev/viewer/themes'
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build, - **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
one-line version bumps) but in production it means: app breaks if the CDN is one-line version bumps) but in production it means: app breaks if the CDN is
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
pull executable WASM from a third party on every load. pull executable WASM from a third party on every load. (pf_app currently does this in
`ui/src/views/Forecast.jsx` — migrating off it is the main open item.)
> **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
> resolves its server WASM with
> `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
> which from `.../client@4.4.0/dist/cdn/` resolves to
> `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm`**no version**.
> jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
> pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
> `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
> Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
>
> The security framing matters as much as the outage: an unversioned URL means your
> users execute whatever that package publishes next, automatically, unreviewed.
>
> 5.2.0 fixes it by carrying the client's version across
> (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
> the WASM is embedded in the bundle and there is no runtime fetch to hijack
> (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).
pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a - The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
`<link>` in `index.html` — so it's bundled and versioned too. `<link>` in `index.html` — so it's bundled and versioned too.
@ -79,9 +60,7 @@ The version choice is constrained by two hard facts about the `@perspective-dev`
**(verified against installed metadata, 2026-06)**: **(verified against installed metadata, 2026-06)**:
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line / - **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even Treemap / Heatmap / etc.) live only in this package.
though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
- **The `/inline` and `/themes` entrypoints are 4.5.x-only**`@perspective-dev/client/inline`, - **The `/inline` and `/themes` entrypoints are 4.5.x-only**`@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x. in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
@ -92,47 +71,21 @@ So you can have at most **two** of these three:
|---|---| |---|---|
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client | | Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) | | One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
| d3fc chart plugins | **4.4.1** viewer-d3fc *and a 4.4.1 viewer to match* (see correction) | | d3fc chart plugins | **4.4.1** viewer-d3fc |
There is **no** version where all three hold. Pick by what the app needs: There is **no** version where all three hold. Pick by what the app needs:
> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts - **Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
> `^4.4.1` viewer-d3fc`. This is a deliberate, necessary mixed-version pair, *not* an
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1` accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.** bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the
> build breaks (`"./inline" is not exported`).
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
> running exactly that pair), *every chart type other than Datagrid fails.*
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
> `viewer` throws `get_static_config is not a function` — once per chart plugin. The
> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
> load against a coherent 4.4.1 viewer produces no such error.
>
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
>
> Consequence: dataflow currently has the worst of both worlds. It carries the
> mixed-version complexity *specifically* to keep charts, and does not have charts.
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
>
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
> real bundled build. It is the documented-and-untested assumption this whole policy
> rests on — establish it before betting a version choice on it.
- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
`^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
working Datagrid and no charts. If you are on it today, you are choosing inline
bundling, not charts; be explicit about which one you actually want.
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to - **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
**4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give **4.4.1 exact**. Charts work; you give up `/inline` bundling.
up `/inline` bundling.
Whatever you pick, **commit the lockfile** so the resolved set can't drift on Whatever you pick, **commit the lockfile** so the resolved set can't drift on
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or `npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x (then a
later (then a fully-coherent inline-capable suite becomes possible — and only then does fully-coherent inline-capable 4.5.x suite becomes possible).
"both" come back on the table).
--- ---
@ -169,74 +122,6 @@ to client-side heuristics — acceptable only at small scale.
--- ---
## 3a. Expression columns and aggregation order (ratio correctness)
**Expression columns are row-level.** Perspective evaluates every expression against
each *raw row* first, then feeds the result into the column's aggregate. It has no
post-aggregate expression stage. So a ratio written the obvious way:
```js
expressions: { price: '"revenue" / "qty"' } // default aggregate: sum
```
...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.
This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
grouping just makes it visible by putting several wrong numbers side by side. No
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
`high`/`low`/`median`/`dominant` are all wrong for the same reason.
### The fix: a weighted-mean aggregate
Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:
```js
expressions: { price: '"revenue" / "qty"' },
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
```
**The nested array matters.** The type is
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
reads like "no such aggregate" and is easy to misread as the feature being absent.
Available since **4.4.0** — no version bump needed, and because the aggregate is
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
`table.update()` on a live view re-derives the weighted mean from the merged rows
without a reload).
### Rules of thumb
- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
→ leave the default `sum`. Those are unaffected by any of this.
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
`['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
- Mixing both in one view is fine and was verified.
### Type inference can silently break expressions
`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
as `integer` — which happens to numeric-looking strings when the table is created from
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
error. Give period/year columns an explicit `string` type at table creation.
### Not fixed by any of this: column-axis expand/collapse
`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
(§2: `viewer-d3fc` still caps at 4.4.1).
---
## 4. Theming ## 4. Theming
- One toggle drives both app CSS and the viewer: - One toggle drives both app CSS and the viewer:
@ -257,42 +142,6 @@ it would retire the `if(...)`-expression workaround. It costs the d3fc charts, t
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
the reference implementation; a stale layout referencing a dropped column otherwise the reference implementation; a stale layout referencing a dropped column otherwise
throws on restore. throws on restore.
- **`aggregates` needs the same guard.** pf_app's `cleanLayout()` has it as of 2026-09;
**dataflow's does not** and should adopt it. Filtering
`columns`/`group_by`/`split_by`/`sort`/`filter` and stopping there is harmless only
while `viewer.save()` emits `aggregates: {}`, which it does until someone sets an
aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
aggregate entry references a *target* column and, in the multi-arg form, a *weight*
column; both need validating, and the entry is what gets dropped, never the layout.
Add this guard as part of adopting §3a, not after.
### Auto-pause deletes the view — turn it off for a static pivot
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on itself
(scrolled out of the viewport, `display: none`) combined with the document's
`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint
optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the
**view object is destroyed**. Becoming visible again calls
`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full
traversal, every time.
For a viewer streaming live updates nobody is watching, that is the right trade.
For a pivot over a large static table it is the wrong one — on pf_app's
`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it
silently discards per-node expand/collapse (§"Not fixed by any of this"), which
has no config representation and so cannot be restored.
```js
await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
```
Guard the call: it is a method on the custom element and absent on older builds.
Leave auto-pause **on** where the table is fed by a live stream the user does not
need to have kept up with while away. This is long-standing viewer behaviour, not
a 5.x regression — worth knowing before blaming a rebuild on your own code.
--- ---
@ -319,13 +168,6 @@ Run this whenever bumping **any** Perspective package or `apache-arrow`:
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others. (`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`; 2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
commit the lockfile. commit the lockfile.
- **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
`latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
and the WASM fails to link:
`Import #8 "env" "psp_opfs_load": function import requires a callable`.
Five packages, not four. This is invisible while pf_app loads from the CDN, and
will bite on the "move off CDN" open item below.
3. `vite build` — no unresolved imports. 3. `vite build` — no unresolved imports.
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests 4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
@ -341,22 +183,17 @@ Run this whenever bumping **any** Perspective package or `apache-arrow`:
| | pf_app | dataflow | Target | | | pf_app | dataflow | Target |
|---|---|---|---| |---|---|---|---|
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** | | Loader | CDN (runtime) | npm `/inline` | **npm `/inline`** |
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** | | Version | 4.4.0 (CDN URLs) | 4.5.1 viewer/client + 4.4.1 d3fc | depends on loader (§2) |
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) | | Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
| `apache-arrow` | `^21.1.0` **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test | | `apache-arrow` | `^21.1.0` (client built vs 17) | n/a | pin exact, match WASM |
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** | | Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline **dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed. bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
**Open items:** **Open items:**
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages - pf_app → move off CDN. Note this forces the §2 choice: going npm-`/inline` means
pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host 4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and
blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders. load via the `.` entry instead of `/inline`. Either way, pin + commit the lockfile, and
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction): add deploy automation (systemd + nginx + `deploy.sh`).
its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
render before choosing the latter, because nobody has confirmed they do.
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).

View File

@ -26,12 +26,6 @@ echo ""
read -p "App port [3030]: " PORT read -p "App port [3030]: " PORT
PORT=${PORT:-3030} PORT=${PORT:-3030}
# Session cookies are signed with this; the server refuses to start without it.
SESSION_SECRET=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')
read -p "Send session cookie over HTTPS only? [Y/n]: " SECURE_ANS
case "${SECURE_ANS:-y}" in [Nn]*) COOKIE_SECURE=false ;; *) COOKIE_SECURE=true ;; esac
# ── Write .env ──────────────────────────────────────────────── # ── Write .env ────────────────────────────────────────────────
cat > .env <<EOF cat > .env <<EOF
DB_HOST=${DB_HOST} DB_HOST=${DB_HOST}
@ -40,10 +34,7 @@ DB_NAME=${DB_NAME}
DB_USER=${DB_USER} DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD} DB_PASSWORD=${DB_PASSWORD}
PORT=${PORT} PORT=${PORT}
SESSION_SECRET=${SESSION_SECRET}
COOKIE_SECURE=${COOKIE_SECURE}
EOF EOF
chmod 600 .env
echo "✓ .env written" echo "✓ .env written"
# ── npm install ─────────────────────────────────────────────── # ── npm install ───────────────────────────────────────────────
@ -60,23 +51,15 @@ PGPASSWORD=${DB_PASSWORD} psql \
-p "${DB_PORT}" \ -p "${DB_PORT}" \
-U "${DB_USER}" \ -U "${DB_USER}" \
-d "${DB_NAME}" \ -d "${DB_NAME}" \
-v ON_ERROR_STOP=1 \ -f setup_sql/01_schema.sql
-f setup_sql/01_schema.sql \
-f setup_sql/02_auth.sql
echo "✓ schema installed" echo "✓ schema installed"
# ── first account ─────────────────────────────────────────────
echo ""
echo "The app is behind a login. Create the first account now:"
./pf.sh add-user
# ── done ───────────────────────────────────────────────────── # ── done ─────────────────────────────────────────────────────
echo "" echo ""
echo "========================================" echo "========================================"
echo " Install complete" echo " Install complete"
echo " Start with: npm run dev echo " Start with: npm run dev"
More accounts: ./pf.sh add-user"
echo " Open: http://$(hostname -I | awk '{print $1}'):${PORT}" echo " Open: http://$(hostname -I | awk '{print $1}'):${PORT}"
echo "========================================" echo "========================================"
echo "" echo ""

View File

@ -1,92 +0,0 @@
// Password hashing and the route guard.
//
// Hashes are scrypt, from node's own crypto — no native build step, and the
// stored form carries its own parameters so they can be raised later without
// invalidating existing rows:
//
// scrypt$<N>$<r>$<p>$<salt base64>$<derived key base64>
const crypto = require('crypto');
const SCRYPT = { N: 16384, r: 8, p: 1, keylen: 64 };
function hashPassword(password, params = SCRYPT) {
const { N, r, p, keylen } = params;
const salt = crypto.randomBytes(16);
const dk = crypto.scryptSync(password, salt, keylen, { N, r, p, maxmem: 256 * 1024 * 1024 });
return `scrypt$${N}$${r}$${p}$${salt.toString('base64')}$${dk.toString('base64')}`;
}
// Constant-time compare. Returns false rather than throwing on a malformed or
// legacy hash, so one bad row can't 500 the login route.
function verifyPassword(password, stored) {
if (typeof stored !== 'string') return false;
const parts = stored.split('$');
if (parts.length !== 6 || parts[0] !== 'scrypt') return false;
const [, N, r, p, saltB64, dkB64] = parts;
try {
const salt = Buffer.from(saltB64, 'base64');
const expected = Buffer.from(dkB64, 'base64');
const actual = crypto.scryptSync(password, salt, expected.length, {
N: Number(N), r: Number(r), p: Number(p), maxmem: 256 * 1024 * 1024,
});
return crypto.timingSafeEqual(actual, expected);
} catch {
return false;
}
}
// Every /api route except the auth ones sits behind this.
function requireAuth(req, res, next) {
if (req.session?.user?.username) return next();
res.status(401).json({ error: 'Not authenticated' });
}
// The identity used for pf.log.pf_user and the created_by/closed_by columns.
// Read from the session only — never from the request body, which the browser
// controls and which used to carry a hardcoded 'admin'.
function sessionUser(req) {
return req.session?.user?.username || null;
}
// What this account may see and change, read from the session for the same
// reason the username is: the browser must not be able to widen it.
//
// Returns { admin: true } for an account that sees everything, or
// { admin: false, values: [...] } for a scoped one. An empty list is a real
// answer meaning "nothing", not a missing one meaning "everything" -- an
// account created without a territory sees no rows until it is granted some.
function sessionTerritory(req) {
const u = req.session?.user;
if (!u) return { admin: false, values: [] };
if (u.is_admin) return { admin: true, values: null };
return { admin: false, values: Array.isArray(u.territory) ? u.territory : [] };
}
function requireAdmin(req, res, next) {
if (req.session?.user?.is_admin) return next();
res.status(403).json({ error: 'Administrator access required' });
}
module.exports = {
hashPassword, verifyPassword, requireAuth, requireAdmin,
sessionUser, sessionTerritory, SCRYPT,
};
// CLI: `node lib/auth.js hash` reads a password on stdin and prints its hash,
// so ./pf.sh can create users without the plaintext touching argv or psql.
if (require.main === module) {
if (process.argv[2] !== 'hash') {
console.error('usage: node lib/auth.js hash (password on stdin)');
process.exit(2);
}
let input = '';
process.stdin.setEncoding('utf8');
process.stdin.on('data', chunk => { input += chunk; });
process.stdin.on('end', () => {
const password = input.replace(/\r?\n$/, '');
if (!password) { console.error('empty password'); process.exit(2); }
process.stdout.write(hashPassword(password) + '\n');
});
}

View File

@ -1,204 +1,24 @@
// Generates operation SQL for a source table, baking in column names from col_meta. // Generates operation SQL for a source table, baking in column names from col_meta.
// Runtime values are left as {{token}} substitution points. // Runtime values are left as {{token}} substitution points.
// //
// Columns flagged col_meta.in_grain define a display grain. When one is set the
// initial load (get_agg) and every operation return rows pre-aggregated to that
// grain and keyed on pf_gkey, instead of raw forecast rows keyed on pf_id.
//
// Tokens baked in at generation time: column names, source schema.table // Tokens baked in at generation time: column names, source schema.table
// Tokens substituted at request time: {{fc_table}}, {{where_clause}}, {{exclude_clause}}, // Tokens substituted at request time: {{fc_table}}, {{where_clause}}, {{exclude_clause}},
// {{version_id}}, {{logid}}, {{pf_user}}, {{note}}, // {{version_id}}, {{logid}}, {{pf_user}}, {{note}},
// {{label}}, {{bucket}}, {{tag}}, {{territory_clause}},
// {{params}}, {{slice}}, {{date_from}}, {{date_to}}, // {{params}}, {{slice}}, {{date_from}}, {{date_to}},
// {{value_incr}}, {{units_incr}}, {{set_clause}}, {{scale_factor}} // {{value_incr}}, {{units_incr}}, {{set_clause}}, {{scale_factor}}
// What the pivot shows for a row's segment and its bucket.
//
// The ordering prefix is part of the stored text, not computed here. Perspective
// orders column groups by the value string, so "01 - Actual" is the only way an
// arbitrary order can be expressed -- and l.label is where a person types it.
// Nothing derives it, which is deliberate: an earlier design built the prefix from
// a separate seq column, as Perspective expressions on the client, and the prefix
// then existed only inside the pivot -- so every other reader disagreed with it,
// and a label that could not be expressed in ExprTK's printable-ASCII-per-byte
// string scanner could not be ordered at all. Stored text has neither problem.
//
// The single exception is the adjustment fallback, whose 99 keeps unlabelled
// adjustments last. Labelling an adjustment's log row overrides it, which is how
// one kind of adjustment is split out from the rest -- l.label rather than tag or
// note, so a segment name stays separable from adjustment commentary (pf_note).
//
// ---------------------------------------------------------------------------
// DISPLAY DEFAULTS -- every hardcoded name the pivot can show.
//
// These are the values a row falls back to when nobody has named it. They are
// the complete list: if a segment or bucket appears in the pivot under a name
// that is not in pf.log, it came from here. CLAUDE.md has the same list under
// "Hardcoded display names".
//
// They live on pf.version -- adjustment_segment, adjustment_bucket,
// unlabeled_load -- and the constants below are only the fallback for a version
// that has not set one. Read through a join at query time rather than
// substituted at generation: pf.sql templates are keyed on (source_id,
// operation) and shared by every version of a source, so a value baked in could
// not vary by version and regenerating for one would change the others.
//
// Exported because /data builds its own statement in routes/operations.js while
// /agg is generated here, and the two have to agree.
// ---------------------------------------------------------------------------
// An adjustment with no label of its own. The 99 keeps it after every numbered
// segment -- ordering is string ordering, so this only works while the loads
// carry 01-0n. Labelling an adjustment's log row overrides it, which is how one
// kind of adjustment is split out from the rest.
const ADJUSTMENT_SEGMENT = '99 - Adjustments';
// What an adjustment counts toward. Prefixed to match the segments it adjusts:
// unprefixed it read 'Forecast' while the loads read '04 - Forecast', and the
// column split in two -- the adjustments sitting apart from the rows they
// adjust. The number is a guess at the convention in use, which is the clearest
// argument for making this per-version.
const ADJUSTMENT_BUCKET = '04 - Forecast';
// A load nobody named. No prefix, so it sorts after everything numbered --
// letters follow digits in ASCII. The old '(unlabeled load)' sorted *first*,
// since '(' is 0x28 and digits begin at 0x30.
const UNLABELED_LOAD = 'Unlabeled';
const LOAD_SEGMENT = `COALESCE(NULLIF(l.label, ''), NULLIF(l.tag, ''), NULLIF(l.note, ''),
NULLIF(v.unlabeled_load, ''), '${UNLABELED_LOAD}')`;
const SEGMENT_EXPR = `CASE WHEN l.operation IN ('baseline','reference')
THEN ${LOAD_SEGMENT}
ELSE COALESCE(NULLIF(l.label, ''), NULLIF(v.adjustment_segment, ''), '${ADJUSTMENT_SEGMENT}')
END`;
// What the row counts towards. A load falls back to its own name until it is
// bucketed; an adjustment falls back to the forecast bucket, because that is
// what an adjustment is -- exclude_iters keeps operations off the reference
// segments, so there is no adjustment that is not part of the forecast.
const BUCKET_EXPR = `COALESCE(NULLIF(l.bucket, ''),
CASE WHEN l.operation IN ('baseline','reference')
THEN ${LOAD_SEGMENT}
ELSE COALESCE(NULLIF(v.adjustment_bucket, ''), '${ADJUSTMENT_BUCKET}')
END)`;
const NOTE_EXPR = `CASE WHEN l.operation IN ('baseline','reference')
THEN NULL
ELSE COALESCE(NULLIF(l.tag, ''), NULLIF(l.note, ''))
END`;
// Every pf.log column the two expressions above read, for /agg's GROUP BY: they
// are functionally dependent on pf_logid, which is in the grain, but Postgres
// will not infer that.
const LABEL_GROUP_COLS = ['l.operation', 'l.label', 'l.tag', 'l.note', 'l.bucket',
'v.adjustment_segment', 'v.adjustment_bucket', 'v.unlabeled_load'];
// The version carries the fallback names, so every statement that reads the
// expressions above needs it in scope as `v`. LEFT, not inner: a forecast row
// whose log entry somehow has no version should still come back, named by the
// constants.
const VERSION_JOIN = `
LEFT JOIN pf.version v
ON v.id = l.version_id`;
// wrap a column name in double quotes for safe use in SQL // wrap a column name in double quotes for safe use in SQL
function q(name) { return `"${name}"`; } function q(name) { return `"${name}"`; }
// The display grain: dimension/date columns flagged in_grain, plus pf_iter and
// pf_logid which are always part of it. Returns null when nothing is flagged —
// that is raw-row mode, where operations return whole rows and the client
// indexes on pf_id (the pre-grain behaviour).
//
// Keeping pf_logid in the grain is what makes the append model work: each
// operation's contribution stays a distinct row, so table.update() accumulates
// rather than replacing a bucket total, and undo can remove exactly that
// operation's rows.
function grainOf(colMeta) {
const cols = colMeta
.filter(c => c.in_grain && (c.role === 'dimension' || c.role === 'date'))
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
.map(c => c.cname);
if (cols.length === 0) return null;
// pf_gkey must be unique per grain tuple. chr(31) (unit separator) joins the
// parts and chr(30) stands in for NULL, so ('a', NULL) cannot collide with
// (NULL, 'a') and a NULL stays distinct from an empty string — a collision
// would silently merge two groups into one indexed row.
//
// md5 of that, rather than the concatenation itself, because the key is an
// opaque handle -- nothing reads it but table.update() and table.remove().
// The raw form averaged 233 chars on a 24-column grain and, being unique per
// row, defeated Arrow's dictionary encoding: 65.6 MB of a 109 MB payload,
// more than every other column combined. 128 bits keeps collisions unreachable.
const key = (pfx = '') => `md5(concat_ws(chr(31), ${[
...cols.map(c => `COALESCE(${pfx}${q(c)}::text, chr(30))`),
`${pfx}pf_iter`,
`${pfx}pf_logid::text`
].join(', ')}))`;
const groupCols = (pfx = '') => [...cols.map(c => `${pfx}${q(c)}`), `${pfx}pf_iter`, `${pfx}pf_logid`];
return { cols, key, groupCols };
}
// Date columns that anchor a dim_group, each with the dimensions derived from
// pf.dim_period for it. Shared so the generator and the routes agree on both the
// membership and the join aliases -- the same reason grainOf is a single function.
function dateGroupsOf(colMeta) {
return colMeta
.filter(c => c.role === 'date' && c.is_key && c.dim_group)
.map((keyCol, i) => ({
alias: `dp${i + 1}`,
dateCol: keyCol.cname,
group: keyCol.dim_group,
derived: colMeta
.filter(c => c.role === 'dimension'
&& c.dim_group === keyCol.dim_group
&& c.dim_period_col)
.map(c => ({ cname: c.cname, periodCol: c.dim_period_col })),
}))
.filter(g => g.derived.length > 0);
}
// cname -> which join it comes from and which of its columns
function dimPeriodMapOf(dateGroups) {
return new Map(
dateGroups.flatMap(g => g.derived.map(d => [d.cname, { alias: g.alias, periodCol: d.periodCol }]))
);
}
// The joins themselves, against a date that has already had {{date_offset}}
// applied. LEFT because a null date -- an order not yet shipped has no ship date
// -- must leave the period columns empty rather than drop the row.
function dimPeriodJoins(dateGroups, alias = 's') {
return dateGroups.map(g =>
`\n LEFT JOIN pf.dim_period ${g.alias}`
+ ` ON ${g.alias}.drange @> (${alias}."${g.dateCol}" + '{{date_offset}}'::interval)::date`
).join('');
}
function generateSQL(source, colMeta) { function generateSQL(source, colMeta) {
const dims = colMeta const dims = colMeta
.filter(c => c.role === 'dimension') .filter(c => c.role === 'dimension')
.sort((a, b) => (a.opos || 0) - (b.opos || 0)) .sort((a, b) => (a.opos || 0) - (b.opos || 0))
.map(c => c.cname); .map(c => c.cname);
// Every column of each measure/date role, in col_meta order. Loads carry all of const valueCol = colMeta.find(c => c.role === 'value')?.cname;
// them; the adjustment operations are single-measure (scale distributes one const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
// {{value_incr}}) and use only the first of each, below. const dateCol = colMeta.find(c => c.role === 'date')?.cname;
const byRole = role => colMeta
.filter(c => c.role === role)
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
.map(c => c.cname);
const valueCols = byRole('value');
const unitsCols = byRole('units');
const dateCols = byRole('date');
const valueCol = valueCols[0];
const unitsCol = unitsCols[0];
const dateCol = dateCols[0];
if (!valueCol) throw new Error('No value column defined in col_meta'); if (!valueCol) throw new Error('No value column defined in col_meta');
if (!dateCol) throw new Error('No date column defined in col_meta'); if (!dateCol) throw new Error('No date column defined in col_meta');
@ -212,72 +32,21 @@ function generateSQL(source, colMeta) {
const selectData = dataCols.map(q).join(', '); const selectData = dataCols.map(q).join(', ');
const dimsJoined = dims.map(q).join(', '); const dimsJoined = dims.map(q).join(', ');
// Baseline and reference copy the source row wholesale, so they carry every // dim_period JOIN support: if the date column is the is_key of a dim_group,
// measure and every date — not just the primary one the operations act on. // dimension siblings with dim_period_col set are derived from pf.dim_period
// Dropping the others would leave those columns null for the life of the version. // instead of being copied raw from the source on baseline/reference load.
// Clone carries every date column, not just the primary one, because it is the const dateKeyGroup = colMeta.find(c => c.role === 'date' && c.is_key && c.dim_group)?.dim_group;
// operation that moves rows through time: {{date_offset}} shifts them all const dimPeriodMap = new Map(
// together, and the period dimensions are re-derived from pf.dim_period against dateKeyGroup
// the shifted dates rather than copied from the row being cloned. Cloning last ? colMeta
// year's mix forward a year otherwise produces rows dated 2027 still labelled .filter(c => c.role === 'dimension' && c.dim_group === dateKeyGroup && c.dim_period_col)
// with 2026's periods. .map(c => [c.cname, c.dim_period_col])
const cloneCols = [...dims, ...dateCols, effectiveValue, effectiveUnits].filter(Boolean); : []
const cloneInsertCols = [...cloneCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', '); );
// An adjustment writes one row per *coordinate* it touches, not one per row
// it reads.
//
// Reading rows one-for-one meant every operation inherited the row count of
// everything before it: the baseline's rows plus every prior adjustment's
// rows at the same coordinate, so the table grew super-linearly with how
// much work had been done on it. Eight Pull Forward entries and the next
// scale over the same slice writes nine times what it needs to.
//
// Collapsing is over pf_logid and pf_iter only -- every stored dimension and
// date stays in the GROUP BY -- so no column goes null and nothing becomes
// unsliceable later. The distribution maths is untouched either way, since
// the window sums see the same totals whether or not the rows underneath
// them have been added up first.
const groupCols = (cols) => cols.map(q).join(',\n ');
const loadCols = [...dims, ...dateCols, ...valueCols, ...unitsCols];
const loadInsertCols = [...loadCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
const dateColSet = new Set(dateCols);
// dim_period JOIN support: a date column that is the is_key of a dim_group
// anchors that group, and dimension siblings with dim_period_col set are
// derived from pf.dim_period instead of copied raw. Derivation is against the
// date *after* {{date_offset}}, which is the whole point -- shift a baseline
// forward a year and its period columns follow, rather than still naming the
// year it came from.
//
// Every such group, not just the first. This used to be a find(), so a source
// with order, requested and ship date groups derived the order one and copied
// the other two raw -- shifted dates against unshifted period labels.
const dateGroups = dateGroupsOf(colMeta);
const dimPeriodMap = dimPeriodMapOf(dateGroups);
const hasDimPeriod = dimPeriodMap.size > 0; const hasDimPeriod = dimPeriodMap.size > 0;
// display grain — when set, initial load and operations both return rows
// pre-aggregated to it instead of raw forecast rows
const grain = grainOf(colMeta);
// A flag on anything other than a dimension/date column is ignored by grainOf,
// which is what we want — the role change is the source of truth, not a stale flag.
if (grain) {
const missing = grain.cols.filter(c => !dataCols.includes(c));
if (missing.length > 0) {
throw new Error(
`Grain columns are never populated in the forecast table: ${missing.join(', ')}`
);
}
if (!effectiveValue && !effectiveUnits) {
throw new Error('A grain requires at least one value or units column to aggregate');
}
}
return { return {
get_data: buildGetData(), get_data: buildGetData(),
...(grain ? { get_agg: buildGetAgg() } : {}),
baseline: buildBaseline(), baseline: buildBaseline(),
reference: buildReference(), reference: buildReference(),
scale: buildScale(), scale: buildScale(),
@ -290,83 +59,31 @@ function generateSQL(source, colMeta) {
return `SELECT * FROM {{fc_table}}`; return `SELECT * FROM {{fc_table}}`;
} }
// Aggregate the whole forecast table to the display grain. This is the initial
// load for grain sources — the client loads the result into a native Perspective
// table indexed on pf_gkey and its view sums across these rows, exactly as an
// Excel pivot cache sums its data tab.
function buildGetAgg() {
// pf_logid is part of the grain, so joining pf.log adds no rows — each group
// already belongs to exactly one log entry. Without this the segment labels
// that /data surfaces would vanish the moment a source declares a grain.
return `
SELECT
${grainSelect('t.')}
,${SEGMENT_EXPR} AS pf_segment
,${BUCKET_EXPR} AS pf_bucket
,${NOTE_EXPR} AS pf_note
,l.operation AS pf_op
FROM {{fc_table}} t
LEFT JOIN pf.log l
ON l.id = t.pf_logid${VERSION_JOIN}
WHERE {{territory_clause}}
GROUP BY
${grain.groupCols('t.').join('\n ,')}
,${LABEL_GROUP_COLS.join('\n ,')}`.trim();
}
// grain columns + pf_gkey + summed measures, in the leading-comma style the
// rest of the generated SQL uses
function grainSelect(pfx = '') {
return [
...grain.groupCols(pfx),
`${grain.key(pfx)} AS pf_gkey`,
effectiveValue ? `SUM(${pfx}${q(effectiveValue)}) AS ${q(effectiveValue)}` : null,
effectiveUnits ? `SUM(${pfx}${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : null
].filter(Boolean).join('\n ,');
}
// Tail of an operation statement: in grain mode the inserted rows come back
// aggregated to grain (the client appends them and lets the view re-sum);
// otherwise whole rows come back as before.
function opTail(cte) {
if (!grain) return `SELECT * FROM ${cte}`;
return `
SELECT
${grainSelect()}
FROM ${cte}
GROUP BY
${grain.groupCols().join('\n ,')}`.trim();
}
function buildLoadSelect(pfx) { function buildLoadSelect(pfx) {
// pfx: table alias prefix ('s.' when joining dim_period, '' otherwise) // pfx: table alias prefix ('s.' when joining dim_period, '' otherwise)
// The offset shifts every date column, so order date and ship date stay in step. return dataCols.map(c => {
return loadCols.map(c => { if (c === dateCol) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
if (dateColSet.has(c)) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`; if (dimPeriodMap.has(c)) return `dp.${q(dimPeriodMap.get(c))} AS ${q(c)}`;
if (dimPeriodMap.has(c)) {
const { alias, periodCol } = dimPeriodMap.get(c);
return `${alias}.${q(periodCol)} AS ${q(c)}`;
}
return `${pfx}${q(c)}`; return `${pfx}${q(c)}`;
}).join(',\n '); }).join(',\n ');
} }
function buildFromClause() { function buildFromClause() {
if (!hasDimPeriod) return srcTable; if (!hasDimPeriod) return srcTable;
return srcTable + ' s' + dimPeriodJoins(dateGroups); return `${srcTable} s\n JOIN pf.dim_period dp`
+ ` ON dp.drange @> (s.${q(dateCol)} + '{{date_offset}}'::interval)::date`;
} }
function buildBaseline() { function buildBaseline() {
return ` return `
WITH WITH
ilog AS ( ilog AS (
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag) INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}', VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}')
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
RETURNING id RETURNING id
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${loadInsertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT SELECT
${buildLoadSelect(hasDimPeriod ? 's.' : '')}, ${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now() 'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
@ -374,20 +91,19 @@ ilog AS (
WHERE {{filter_clause}} WHERE {{filter_clause}}
RETURNING * RETURNING *
) )
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim(); SELECT count(*) AS rows_affected FROM ins`.trim();
} }
function buildReference() { function buildReference() {
return ` return `
WITH WITH
ilog AS ( ilog AS (
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag) INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}', VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}')
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
RETURNING id RETURNING id
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${loadInsertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT SELECT
${buildLoadSelect(hasDimPeriod ? 's.' : '')}, ${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'reference', (SELECT id FROM ilog), '{{pf_user}}', now() 'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
@ -395,7 +111,7 @@ ilog AS (
WHERE {{filter_clause}} WHERE {{filter_clause}}
RETURNING * RETURNING *
) )
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim(); SELECT count(*) AS rows_affected FROM ins`.trim();
} }
function buildScale() { function buildScale() {
@ -405,16 +121,13 @@ SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim
const uSel = effectiveUnits const uSel = effectiveUnits
? `round((${q(effectiveUnits)} / NULLIF(total_units, 0)) * {{units_incr}}, 5)` ? `round((${q(effectiveUnits)} / NULLIF(total_units, 0)) * {{units_incr}}, 5)`
: `0`; : `0`;
// sum(sum(x)) OVER () is the aggregate of the aggregates: the window runs
// after the GROUP BY, so the total is over collapsed coordinates and
// comes to the same figure the ungrouped window produced.
const baseSelectParts = [ const baseSelectParts = [
...dimsJoined ? [dimsJoined] : [], ...dimsJoined ? [dimsJoined] : [],
q(dateCol), q(dateCol),
effectiveValue ? `sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : null, effectiveValue ? q(effectiveValue) : null,
effectiveUnits ? `sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : null, effectiveUnits ? q(effectiveUnits) : null,
effectiveValue ? `sum(sum(${q(effectiveValue)})) OVER () AS total_value` : null, effectiveValue ? `sum(${q(effectiveValue)}) OVER () AS total_value` : null,
effectiveUnits ? `sum(sum(${q(effectiveUnits)})) OVER () AS total_units` : null effectiveUnits ? `sum(${q(effectiveUnits)}) OVER () AS total_units` : null
].filter(Boolean).join(',\n '); ].filter(Boolean).join(',\n ');
return ` return `
WITH WITH
@ -429,8 +142,6 @@ ilog AS (
FROM {{fc_table}} FROM {{fc_table}}
WHERE {{where_clause}} WHERE {{where_clause}}
{{exclude_clause}} {{exclude_clause}}
GROUP BY
${groupCols([...dims, dateCol])}
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
@ -440,7 +151,7 @@ ilog AS (
FROM base FROM base
RETURNING * RETURNING *
) )
${opTail('ins')}`.trim(); SELECT * FROM ins`.trim();
} }
function buildRecode() { function buildRecode() {
@ -452,14 +163,10 @@ ilog AS (
RETURNING id RETURNING id
) )
,src AS ( ,src AS (
SELECT SELECT ${selectData}
${dimsJoined},
${q(dateCol)}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''}
FROM {{fc_table}} FROM {{fc_table}}
WHERE {{where_clause}} WHERE {{where_clause}}
{{exclude_clause}} {{exclude_clause}}
GROUP BY
${groupCols([...dims, dateCol])}
) )
,neg AS ( ,neg AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
@ -475,25 +182,10 @@ ilog AS (
FROM src FROM src
RETURNING * RETURNING *
) )
${grain ? `,allrows AS ( SELECT * FROM neg UNION ALL SELECT * FROM ins`.trim();
SELECT * FROM neg
UNION ALL
SELECT * FROM ins
)
${opTail('allrows')}` : 'SELECT * FROM neg UNION ALL SELECT * FROM ins'}`.trim();
} }
function buildClone() { function buildClone() {
const select = [
// dims: whatever {{set_clause}} resolves them to. The route builds it,
// and substitutes the dim_period expression for any derived dimension
// the caller has not overridden outright.
'{{set_clause}}',
...dateCols.map(c => `(s.${q(c)} + '{{date_offset}}'::interval)::date`),
effectiveValue ? `round(s.${q(effectiveValue)} * {{scale_factor}}, 2)` : null,
effectiveUnits ? `round(s.${q(effectiveUnits)} * {{scale_factor}}, 5)` : null,
].filter(Boolean).join(',\n ');
return ` return `
WITH WITH
ilog AS ( ilog AS (
@ -502,22 +194,18 @@ ilog AS (
RETURNING id RETURNING id
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${cloneInsertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT SELECT
${select}, {{set_clause}},
${q(dateCol)},
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}${effectiveUnits ? `,\n round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : ''},
'clone', (SELECT id FROM ilog), '{{pf_user}}', now() 'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM ( FROM {{fc_table}}
SELECT WHERE {{where_clause}}
${groupCols([...dims, ...dateCols])}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''} {{exclude_clause}}
FROM {{fc_table}}
WHERE {{where_clause}}
{{exclude_clause}}
GROUP BY
${groupCols([...dims, ...dateCols])}
) s${hasDimPeriod ? dimPeriodJoins(dateGroups) : ''}
RETURNING * RETURNING *
) )
${opTail('ins')}`.trim(); SELECT * FROM ins`.trim();
} }
function buildUndo() { function buildUndo() {
@ -544,53 +232,13 @@ function applyTokens(sql, tokens) {
// build a SQL WHERE clause string from a slice object // build a SQL WHERE clause string from a slice object
// only dimension columns are included; unrecognised keys are silently skipped // only dimension columns are included; unrecognised keys are silently skipped
// pf_segment and pf_bucket are not columns on the forecast table -- they are function buildWhere(slice, dimCols) {
// computed at read time from the row's pf.log entry -- so a slice naming one
// cannot be compared directly. It resolves to a set of log ids instead, which is
// exact: the name lives on the log row, and every forecast row carries the
// pf_logid that points at it.
//
// Without this they were dropped from the slice, and clicking a single bucket's
// cell scaled every bucket at that dimension intersection while the panel showed
// only the one clicked.
const COMPUTED_SLICE_COLS = { pf_segment: SEGMENT_EXPR, pf_bucket: BUCKET_EXPR };
function computedSlicePredicate(col, val, versionId) {
if (versionId == null) {
const err = new Error(`Cannot filter on ${col} without a version`);
err.status = 500;
throw err;
}
const vals = (Array.isArray(val) ? val : [val]).map(v => `'${esc(v)}'`).join(', ');
return `pf_logid IN (
SELECT l.id
FROM pf.log l${VERSION_JOIN}
WHERE l.version_id = ${parseInt(versionId)}
AND ${COMPUTED_SLICE_COLS[col]} IN (${vals})
)`;
}
function buildWhere(slice, dimCols, versionId) {
if (!slice || Object.keys(slice).length === 0) return 'TRUE'; if (!slice || Object.keys(slice).length === 0) return 'TRUE';
const allowed = new Set(dimCols); const allowed = new Set(dimCols);
const parts = []; const parts = [];
for (const [col, val] of Object.entries(slice)) { for (const [col, val] of Object.entries(slice)) {
if (COMPUTED_SLICE_COLS[col]) {
parts.push(computedSlicePredicate(col, val, versionId));
continue;
}
// A pf_ key this does not understand is refused rather than skipped.
// Skipping is how a selection silently widened: the operation ran against
// everything the dropped key would have excluded. pf_iter is the one
// exception -- the client strips it deliberately, since two cells that
// differ only by iter band are the same slice.
if (col.startsWith('pf_') && col !== 'pf_iter') {
const err = new Error(`Slice names ${col}, which cannot be filtered on`);
err.status = 400;
throw err;
}
if (!allowed.has(col)) continue; if (!allowed.has(col)) continue;
if (Array.isArray(val)) { if (Array.isArray(val)) {
const escaped = val.map(v => esc(v)); const escaped = val.map(v => esc(v));
@ -603,110 +251,6 @@ function buildWhere(slice, dimCols, versionId) {
return parts.length ? parts.join('\nAND ') : 'TRUE'; return parts.length ? parts.join('\nAND ') : 'TRUE';
} }
// The pivot's own filter, carried alongside the slices so an operation writes
// exactly the rows the ledger counted.
//
// A slice is {col: value} and can only ever mean equality, so a view filtered to
// sseas_e <= 2027 could not be expressed as one. Refusing it was safe but
// useless -- a bounded season is an ordinary way to scope a forecast -- so the
// operators travel as [col, op, value] triples instead.
//
// Perspective's operator names, not SQL's, since that is where these come from.
// Anything outside this list is refused rather than ignored: a scope silently
// dropped is a write that is wider than the panel that authorised it.
const SCOPE_OPS = {
'==': (c, v) => `${c} = ${v[0]}`,
'!=': (c, v) => `${c} != ${v[0]}`,
'>': (c, v) => `${c} > ${v[0]}`,
'>=': (c, v) => `${c} >= ${v[0]}`,
'<': (c, v) => `${c} < ${v[0]}`,
'<=': (c, v) => `${c} <= ${v[0]}`,
'in': (c, v) => `${c} IN (${v.join(', ')})`,
'not in': (c, v) => `${c} NOT IN (${v.join(', ')})`,
'is null': (c) => `${c} IS NULL`,
'is not null': (c) => `${c} IS NOT NULL`,
};
function buildScopeClause(scope, dimCols, versionId) {
if (!Array.isArray(scope) || scope.length === 0) return '';
const allowed = new Set(dimCols);
const parts = scope.map((entry) => {
if (!Array.isArray(entry) || entry.length < 2) {
const err = new Error(`Malformed scope entry ${JSON.stringify(entry)}`);
err.status = 400; throw err;
}
const [col, op, ...rest] = entry;
const vals = (Array.isArray(rest[0]) ? rest[0] : rest).filter(v => v !== undefined);
if (COMPUTED_SLICE_COLS[col]) {
if (op !== '==' && op !== 'in') {
const err = new Error(`${col} can only be scoped with == or in, not ${op}`);
err.status = 400; throw err;
}
return computedSlicePredicate(col, vals, versionId);
}
if (!allowed.has(col)) {
const err = new Error(`Column "${col}" is not available for filtering`);
err.status = 400; throw err;
}
const fn = SCOPE_OPS[op];
if (!fn) {
const err = new Error(`Unsupported filter operator "${op}" on ${col}`);
err.status = 400; throw err;
}
return fn(`"${col}"`, vals.map(v => `'${esc(String(v))}'`));
});
return parts.join('\nAND ');
}
// The territory predicate: what this account may see and change.
//
// Server-side by construction. Today's `scope` is sent by the browser, which is
// right for a filter the user chose and would be fatal for a permission -- so
// this one is built from the session and ANDed on last, where nothing in the
// request can remove it.
//
// FALSE, not TRUE, for an account with no territory. The whole point is that a
// missing grant means no rows: an empty list that fell through to TRUE would
// hand the entire book to the first account somebody forgot to configure.
function buildTerritoryClause(territory, territoryCol, alias = '') {
if (!territory || territory.admin) return '';
if (!territoryCol) return 'FALSE';
const vals = (territory.values || []).filter(v => v != null && v !== '');
if (vals.length === 0) return 'FALSE';
const pfx = alias ? `${alias}.` : '';
return `${pfx}"${territoryCol}" IN (${vals.map(v => `'${esc(String(v))}'`).join(', ')})`;
}
// build a WHERE clause spanning several slices — an OR of AND-groups.
// A union of slices cannot be flattened into one IN list per column: slices
// {Region:East, State:NY} and {Region:West, State:CA} would become
// Region IN (East,West) AND State IN (NY,CA), which also matches East/CA.
function buildWhereAny(slices, dimCols, versionId) {
const list = (slices || []).filter(s => s && Object.keys(s).length > 0);
if (list.length === 0) return 'TRUE';
if (list.length === 1) return buildWhere(list[0], dimCols, versionId);
const groups = list
.map(s => buildWhere(s, dimCols, versionId))
.filter(w => w !== 'TRUE');
// any slice that reduced to TRUE selects everything, so the union does too
if (groups.length !== list.length) return 'TRUE';
// outer parens matter: the caller appends `AND pf_iter NOT IN (...)`,
// and AND binds tighter than OR
return `(${groups.map(g => `(${g.replace(/\n/g, ' ')})`).join('\n OR ')})`;
}
// the bare predicate for "this row participates in operations", for use in a
// FILTER clause where the excluded rows still need to be counted separately
function buildExcludePredicate(excludeIters) {
if (!excludeIters || excludeIters.length === 0) return 'TRUE';
const list = excludeIters.map(i => `'${esc(i)}'`).join(', ');
return `pf_iter NOT IN (${list})`;
}
// build AND iter NOT IN (...) from a version's exclude_iters array // build AND iter NOT IN (...) from a version's exclude_iters array
function buildExcludeClause(excludeIters) { function buildExcludeClause(excludeIters) {
if (!excludeIters || excludeIters.length === 0) return ''; if (!excludeIters || excludeIters.length === 0) return '';
@ -716,21 +260,12 @@ function buildExcludeClause(excludeIters) {
// build the dimension columns portion of a SELECT for recode/clone // build the dimension columns portion of a SELECT for recode/clone
// replaces named dimensions with literal values, passes others through unchanged // replaces named dimensions with literal values, passes others through unchanged
// derivedExprs: cname -> a SQL expression to use when the caller has not set the function buildSetClause(dimCols, setObj) {
// column outright. Clone passes the dim_period expressions through here, so a
// cloned row's period dimensions come from the calendar against its shifted date
// rather than from the row it was copied from.
function buildSetClause(dimCols, setObj, opts = {}) {
const { derivedExprs, alias } = opts;
const pfx = alias ? `${alias}.` : '';
return dimCols.map(col => { return dimCols.map(col => {
if (setObj && setObj[col] !== undefined) { if (setObj && setObj[col] !== undefined) {
return `'${esc(setObj[col])}' AS "${col}"`; return `'${esc(setObj[col])}' AS "${col}"`;
} }
if (derivedExprs && derivedExprs[col]) { return `"${col}"`;
return `${derivedExprs[col]} AS "${col}"`;
}
return `${pfx}"${col}"`;
}).join(', '); }).join(', ');
} }
@ -774,6 +309,4 @@ function esc(val) {
return String(val).replace(/'/g, "''"); return String(val).replace(/'/g, "''");
} }
module.exports = { generateSQL, grainOf, COMPUTED_SLICE_COLS, buildScopeClause, buildTerritoryClause, module.exports = { generateSQL, applyTokens, buildWhere, buildExcludeClause, buildSetClause, buildFilterClause, esc };
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR, LABEL_GROUP_COLS, VERSION_JOIN,
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET, UNLABELED_LOAD, dateGroupsOf, dimPeriodMapOf, dimPeriodJoins, applyTokens, buildWhere, buildWhereAny, buildExcludeClause, buildExcludePredicate, buildSetClause, buildFilterClause, esc };

View File

@ -3,41 +3,7 @@ function fcTable(tname, versionId) {
return `pf.fc_${tname}_${versionId}`; return `pf.fc_${tname}_${versionId}`;
} }
// The columns of a relation, from pg_catalog rather than information_schema. // map information_schema data_type to a clean postgres column type
//
// information_schema.columns omits materialized views -- they are not in the
// SQL standard -- so a source built on one looked like it had no columns at
// all: registering it seeded nothing, and creating a version failed with "No
// usable columns in col_meta" while col_meta plainly had thirty-six.
//
// The shape matches what information_schema returned, so mapType and the
// callers did not have to change. data_type is format_type with the modifier
// stripped, which gives the same spelling information_schema uses ('character
// varying', 'numeric'), and the numeric precision and scale are unpacked from
// atttypmod the way information_schema does internally.
//
// Takes $1 = schema, $2 = relation name.
const RELATION_COLUMNS_SQL = `
SELECT a.attname AS column_name
,regexp_replace(format_type(a.atttypid, a.atttypmod), '\\(.*\\)$', '') AS data_type
,a.attnum AS ordinal_position
,CASE WHEN a.attnotnull THEN 'NO' ELSE 'YES' END AS is_nullable
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
THEN ((a.atttypmod - 4) >> 16) & 65535 END AS numeric_precision
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
THEN (a.atttypmod - 4) & 65535 END AS numeric_scale
FROM pg_attribute a
JOIN pg_class c ON c.oid = a.attrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE TRUE
AND n.nspname = $1
AND c.relname = $2
AND c.relkind IN ('r', 'v', 'm', 'f', 'p')
AND a.attnum > 0
AND NOT a.attisdropped
`;
// map a data_type name to a clean postgres column type
function mapType(dataType, numericPrecision, numericScale) { function mapType(dataType, numericPrecision, numericScale) {
switch (dataType) { switch (dataType) {
case 'character varying': case 'character varying':
@ -70,4 +36,4 @@ function mapType(dataType, numericPrecision, numericScale) {
} }
} }
module.exports = { fcTable, mapType, RELATION_COLUMNS_SQL }; module.exports = { fcTable, mapType };

68
package-lock.json generated
View File

@ -7,14 +7,11 @@
"": { "": {
"name": "pf_app", "name": "pf_app",
"version": "1.0.0", "version": "1.0.0",
"license": "MIT",
"dependencies": { "dependencies": {
"apache-arrow": "^21.1.0", "apache-arrow": "^21.1.0",
"connect-pg-simple": "^10.0.0",
"cors": "^2.8.5", "cors": "^2.8.5",
"dotenv": "^16.0.0", "dotenv": "^16.0.0",
"express": "^4.18.2", "express": "^4.18.2",
"express-session": "^1.19.0",
"pg": "^8.11.3" "pg": "^8.11.3"
}, },
"devDependencies": { "devDependencies": {
@ -372,18 +369,6 @@
"node": ">=12.20.0" "node": ">=12.20.0"
} }
}, },
"node_modules/connect-pg-simple": {
"version": "10.0.0",
"resolved": "https://registry.npmjs.org/connect-pg-simple/-/connect-pg-simple-10.0.0.tgz",
"integrity": "sha512-pBGVazlqiMrackzCr0eKhn4LO5trJXsOX0nQoey9wCOayh80MYtThCbq8eoLsjpiWgiok/h+1/uti9/2/Una8A==",
"license": "MIT",
"dependencies": {
"pg": "^8.12.0"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=22.0.0"
}
},
"node_modules/content-disposition": { "node_modules/content-disposition": {
"version": "0.5.4", "version": "0.5.4",
"resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz", "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
@ -597,29 +582,6 @@
"url": "https://opencollective.com/express" "url": "https://opencollective.com/express"
} }
}, },
"node_modules/express-session": {
"version": "1.19.0",
"resolved": "https://registry.npmjs.org/express-session/-/express-session-1.19.0.tgz",
"integrity": "sha512-0csaMkGq+vaiZTmSMMGkfdCOabYv192VbytFypcvI0MANrp+4i/7yEkJ0sbAEhycQjntaKGzYfjfXQyVb7BHMA==",
"license": "MIT",
"dependencies": {
"cookie": "~0.7.2",
"cookie-signature": "~1.0.7",
"debug": "~2.6.9",
"depd": "~2.0.0",
"on-headers": "~1.1.0",
"parseurl": "~1.3.3",
"safe-buffer": "~5.2.1",
"uid-safe": "~2.1.5"
},
"engines": {
"node": ">= 0.8.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/express"
}
},
"node_modules/fill-range": { "node_modules/fill-range": {
"version": "7.1.1", "version": "7.1.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz",
@ -1123,15 +1085,6 @@
"node": ">= 0.8" "node": ">= 0.8"
} }
}, },
"node_modules/on-headers": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/on-headers/-/on-headers-1.1.0.tgz",
"integrity": "sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==",
"license": "MIT",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/parseurl": { "node_modules/parseurl": {
"version": "1.3.3", "version": "1.3.3",
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
@ -1323,15 +1276,6 @@
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
}, },
"node_modules/random-bytes": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/random-bytes/-/random-bytes-1.0.0.tgz",
"integrity": "sha512-iv7LhNVO047HzYR3InF6pUcUsPQiHTM1Qal51DcGSuZFBil1aBBWG5eHPNek7bvILMaYJ/8RU1e8w1AMdHmLQQ==",
"license": "MIT",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/range-parser": { "node_modules/range-parser": {
"version": "1.2.1", "version": "1.2.1",
"resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
@ -1648,18 +1592,6 @@
"node": ">=12.17" "node": ">=12.17"
} }
}, },
"node_modules/uid-safe": {
"version": "2.1.5",
"resolved": "https://registry.npmjs.org/uid-safe/-/uid-safe-2.1.5.tgz",
"integrity": "sha512-KPHm4VL5dDXKz01UuEd88Df+KzynaohSL9fBh096KWAxSKZQDI2uBrVqtvRM4rwrIrRRKsdLNML/lnaaVSRioA==",
"license": "MIT",
"dependencies": {
"random-bytes": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/undefsafe": { "node_modules/undefsafe": {
"version": "2.0.5", "version": "2.0.5",
"resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz", "resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz",

View File

@ -11,11 +11,9 @@
}, },
"dependencies": { "dependencies": {
"apache-arrow": "^21.1.0", "apache-arrow": "^21.1.0",
"connect-pg-simple": "^10.0.0",
"cors": "^2.8.5", "cors": "^2.8.5",
"dotenv": "^16.0.0", "dotenv": "^16.0.0",
"express": "^4.18.2", "express": "^4.18.2",
"express-session": "^1.19.0",
"pg": "^8.11.3" "pg": "^8.11.3"
}, },
"devDependencies": { "devDependencies": {

362
pf.sh
View File

@ -4,8 +4,6 @@ set -euo pipefail
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# pf.sh — Pivot Forecast management script # pf.sh — Pivot Forecast management script
# Usage: ./pf.sh [deploy|start|stop|restart|status|logs|db-setup|config] # Usage: ./pf.sh [deploy|start|stop|restart|status|logs|db-setup|config]
# ./pf.sh [add-user|passwd|list-users|disable-user|enable-user]
# ./pf.sh [set-territory|set-admin|orphan-territory]
# ./pf.sh (interactive menu) # ./pf.sh (interactive menu)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -69,33 +67,13 @@ require_service() {
service_installed || die "systemd service not installed. Run: ./pf.sh install-service" service_installed || die "systemd service not installed. Run: ./pf.sh install-service"
} }
# The keys cmd_config manages; anything else in .env is left alone.
ENV_KEYS=(DB_HOST DB_PORT DB_NAME DB_USER DB_PASSWORD PORT SESSION_SECRET COOKIE_SECURE)
env_get() {
[[ -f "$ENV_FILE" ]] || return 0
grep -E "^$1=" "$ENV_FILE" | tail -1 | cut -d= -f2- | tr -d '"' || true
}
# psql against the DB_* connection in .env; extra args are passed through.
run_psql() {
PGPASSWORD="${DB_PASSWORD:-}" psql \
-h "${DB_HOST:-localhost}" \
-p "${DB_PORT:-5432}" \
-U "${DB_USER}" \
-d "${DB_NAME}" \
"$@"
}
db_ping() { db_ping() {
load_env load_env
if [[ -z "${DB_NAME:-}" || -z "${DB_USER:-}" ]]; then local url="${DATABASE_URL:-}"
warn "DB_NAME / DB_USER not set in .env" [[ -z "$url" ]] && { warn "DATABASE_URL not set in .env"; return 1; }
return 1
fi
# Use psql if available for a real connectivity check # Use psql if available for a real connectivity check
if command -v psql &>/dev/null; then if command -v psql &>/dev/null; then
run_psql -tAc "SELECT 1" &>/dev/null && return 0 || return 1 psql "$url" -c "SELECT 1" &>/dev/null && return 0 || return 1
else else
warn "psql not in PATH — skipping live DB check" warn "psql not in PATH — skipping live DB check"
return 0 return 0
@ -190,25 +168,18 @@ cmd_logs() {
cmd_db_setup() { cmd_db_setup() {
require_env; load_env require_env; load_env
[[ -n "${DB_NAME:-}" && -n "${DB_USER:-}" ]] || die "DB_NAME / DB_USER not set in .env — run: ./pf.sh config" local url="${DATABASE_URL:-}"
[[ -z "$url" ]] && die "DATABASE_URL not set in .env"
command -v psql &>/dev/null || die "psql not found — install postgresql-client" command -v psql &>/dev/null || die "psql not found — install postgresql-client"
echo echo
bold "DB Setup — will run: setup_sql/01_schema.sql, setup_sql/02_auth.sql" bold "DB Setup — will run: setup_sql/01_schema.sql"
warn "This creates the pf schema, tables, and the account/session tables." warn "This creates the pf schema and tables. Safe to re-run (CREATE IF NOT EXISTS)."
warn "Safe to re-run (CREATE IF NOT EXISTS)."
read -rp " Continue? [y/N] " confirm read -rp " Continue? [y/N] " confirm
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; return; } [[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; return; }
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/01_schema.sql" psql "$url" -f "${APP_DIR}/setup_sql/01_schema.sql"
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/02_auth.sql"
success "Schema applied." success "Schema applied."
local n
n=$(run_psql -tAc "SELECT count(*) FROM pf.app_user WHERE is_active" 2>/dev/null || echo 0)
if [[ "${n:-0}" == "0" ]]; then
warn "No active accounts yet — create one with: ./pf.sh add-user"
fi
} }
cmd_config() { cmd_config() {
@ -217,284 +188,41 @@ cmd_config() {
echo " File: $ENV_FILE" echo " File: $ENV_FILE"
echo echo
local cur_host cur_port cur_name cur_user cur_pass cur_app_port local current_url=""
cur_host=$(env_get DB_HOST) local current_port=""
cur_port=$(env_get DB_PORT) local current_user=""
cur_name=$(env_get DB_NAME)
cur_user=$(env_get DB_USER)
cur_pass=$(env_get DB_PASSWORD)
cur_app_port=$(env_get PORT)
local input
read -rp " DB_HOST [${cur_host:-localhost}]: " input
local host="${input:-${cur_host:-localhost}}"
read -rp " DB_PORT [${cur_port:-5432}]: " input
local port="${input:-${cur_port:-5432}}"
read -rp " DB_NAME [${cur_name:-not set}]: " input
local name="${input:-$cur_name}"
[[ -z "$name" ]] && die "DB_NAME is required."
read -rp " DB_USER [${cur_user:-$USER}]: " input
local user="${input:-${cur_user:-$USER}}"
if [[ -n "$cur_pass" ]]; then
read -rsp " DB_PASSWORD [keep existing]: " input; echo
else
read -rsp " DB_PASSWORD: " input; echo
fi
local pass="${input:-$cur_pass}"
read -rp " PORT (app) [${cur_app_port:-3010}]: " input
local app_port="${input:-${cur_app_port:-3010}}"
# Session cookies are signed with this; regenerating it signs everyone out,
# so an existing secret is kept rather than re-rolled on every config run.
local secret
secret=$(env_get SESSION_SECRET)
if [[ -z "$secret" ]]; then
secret=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')
success "SESSION_SECRET generated."
else
success "SESSION_SECRET kept (delete the line in .env to re-roll)."
fi
local cur_secure
cur_secure=$(env_get COOKIE_SECURE)
read -rp " COOKIE_SECURE — HTTPS-only cookie [${cur_secure:-true}]: " input
local cookie_secure="${input:-${cur_secure:-true}}"
# Rewrite the managed keys, carrying over any other lines already in .env.
local tmp
tmp=$(mktemp)
cat > "$tmp" <<EOF
DB_HOST=${host}
DB_PORT=${port}
DB_NAME=${name}
DB_USER=${user}
DB_PASSWORD=${pass}
PORT=${app_port}
SESSION_SECRET=${secret}
COOKIE_SECURE=${cookie_secure}
EOF
if [[ -f "$ENV_FILE" ]]; then if [[ -f "$ENV_FILE" ]]; then
local managed current_url=$(grep -E '^DATABASE_URL=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
managed=$(IFS='|'; echo "${ENV_KEYS[*]}") current_port=$(grep -E '^PORT=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
grep -vE "^(${managed})=" "$ENV_FILE" | grep -vE '^[[:space:]]*$' >> "$tmp" || true current_user=$(grep -E '^PF_USER=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
fi fi
mv "$tmp" "$ENV_FILE"
read -rp " DATABASE_URL [${current_url:-not set}]: " input_url
local url="${input_url:-$current_url}"
[[ -z "$url" ]] && die "DATABASE_URL is required."
read -rp " PORT [${current_port:-3010}]: " input_port
local port="${input_port:-${current_port:-3010}}"
read -rp " PF_USER [${current_user:-$USER}]: " input_user
local pf_user="${input_user:-${current_user:-$USER}}"
cat > "$ENV_FILE" <<EOF
DATABASE_URL=${url}
PORT=${port}
PF_USER=${pf_user}
EOF
chmod 600 "$ENV_FILE" chmod 600 "$ENV_FILE"
success ".env written." success ".env written."
if db_ping; then if db_ping; then
success "Database connection verified." success "Database connection verified."
else else
warn "Could not reach the database — double-check the DB_* settings." warn "Could not reach the database — double-check DATABASE_URL."
fi fi
} }
# -- Accounts ----------------------------------------------------------------
# Reads a password twice without echo and hashes it with lib/auth.js, so the
# plaintext never reaches argv, psql, or the shell history.
read_new_password() {
local p1 p2
# Prompts and their newlines go to stderr: stdout is the hash, and a stray
# newline there ends up prefixed to it by the caller's $( ).
read -rsp " Password: " p1; echo >&2
[[ -z "$p1" ]] && { error "Password cannot be empty."; return 1; }
read -rsp " Confirm : " p2; echo >&2
[[ "$p1" != "$p2" ]] && { error "Passwords do not match."; return 1; }
printf '%s' "$p1" | node "${APP_DIR}/lib/auth.js" hash
}
# psql single-quoted literal: double any embedded quote.
sql_lit() { printf "%s" "${1//\'/\'\'}"; }
cmd_add_user() {
require_env; load_env
check_node >/dev/null
echo; bold "Add account"
local username display hash
read -rp " Username: " username
[[ -z "$username" ]] && die "Username is required."
read -rp " Display name [${username}]: " display
display="${display:-$username}"
hash=$(read_new_password) || return 1
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH ins AS (
INSERT INTO pf.app_user (username, pass_hash, display_name)
VALUES ('$(sql_lit "$username")', '$(sql_lit "$hash")', '$(sql_lit "$display")')
ON CONFLICT (username) DO NOTHING
RETURNING id
) SELECT id FROM ins" | grep -q . \
&& success "Account '${username}' created." \
|| die "Account '${username}' already exists — change its password with: ./pf.sh passwd"
}
cmd_passwd() {
require_env; load_env
check_node >/dev/null
echo; bold "Change password"
local username hash
read -rp " Username: " username
[[ -z "$username" ]] && die "Username is required."
hash=$(read_new_password) || return 1
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET pass_hash = '$(sql_lit "$hash")'
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
&& success "Password updated for '${username}'." \
|| die "No such account: ${username}"
}
cmd_list_users() {
require_env; load_env
echo; bold "Accounts"
run_psql -c "
SELECT username, display_name, is_active, is_admin,
coalesce(jsonb_array_length(territory), 0) AS territory_values,
to_char(last_login_at, 'YYYY-MM-DD HH24:MI') AS last_login
FROM pf.app_user ORDER BY username"
}
# Territory is what an account may see and change, as a list of values in the
# source's is_territory column. No territory and not an admin means no rows --
# so a new account is blind until this is run, which is the intended direction
# to fail in.
#
# Values are given comma-separated and have to match the column exactly, since
# that is what the SQL compares. set-territory with no values clears it.
cmd_set_territory() {
require_env; load_env
local username="${1:-}"; shift || true
[[ -z "$username" ]] && { read -rp " Username: " username; }
[[ -z "$username" ]] && die "Username is required."
local values="${*:-}"
[[ -z "$values" ]] && { read -rp " Territory values (comma separated, blank to clear): " values; }
local json="null"
if [[ -n "$values" ]]; then
json=$(python3 - "$values" <<'PYEOF'
import json, sys
vals = [v.strip() for v in sys.argv[1].split(',') if v.strip()]
print(json.dumps(vals))
PYEOF
)
fi
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET territory = $(if [[ "$json" == "null" ]]; then echo NULL; else echo "'$(sql_lit "$json")'::jsonb"; fi)
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING username
)
SELECT count(*) FROM upd" | grep -q '^1$' \
|| die "No such account: $username"
success "Territory updated for $username"
run_psql -c "SELECT username, is_admin, territory FROM pf.app_user WHERE lower(username) = lower('$(sql_lit "$username")')"
}
# An admin sees and changes everything, and is the only account that can recode
# the territory column or undo someone else's entry.
cmd_set_admin() {
require_env; load_env
local username="${1:-}" flag="${2:-true}"
[[ -z "$username" ]] && { read -rp " Username: " username; }
[[ -z "$username" ]] && die "Username is required."
[[ "$flag" != "true" && "$flag" != "false" ]] && die "Second argument must be true or false."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_admin = $flag
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING username
)
SELECT count(*) FROM upd" | grep -q '^1$' \
|| die "No such account: $username"
success "$username is_admin = $flag"
}
# Territory values present in the data that belong to no account. Work under one
# is invisible to everybody but an admin, which is easy to cause by a typo and
# impossible to notice from inside the app.
cmd_orphan_territory() {
require_env; load_env
local source_id="${1:-}"
[[ -z "$source_id" ]] && { read -rp " Source id: " source_id; }
[[ -z "$source_id" ]] && die "Source id is required."
# The column and table are data, so the query is built in two steps rather
# than one clever one: read the names, then run the listing.
local meta col schema tname
meta=$(run_psql -tAF'|' -c "
SELECT m.cname, x.schema, x.tname
FROM pf.col_meta m JOIN pf.source x ON x.id = m.source_id
WHERE m.source_id = $source_id AND m.is_territory")
[[ -z "$meta" ]] && die "Source $source_id has no column marked is_territory."
IFS='|' read -r col schema tname <<< "$meta"
echo; bold "Territory values in $schema.$tname with no account"
run_psql -c "
SELECT DISTINCT s.\"$col\" AS unassigned
FROM \"$schema\".\"$tname\" s
WHERE TRUE
AND s.\"$col\" IS NOT NULL
AND s.\"$col\"::text NOT IN (
SELECT jsonb_array_elements_text(territory)
FROM pf.app_user
WHERE territory IS NOT NULL
)
ORDER BY 1"
}
# Deactivating leaves the row (and its history) in place, and drops any live
# session so the account loses access immediately rather than at cookie expiry.
cmd_disable_user() {
require_env; load_env
local username="${1:-}"
[[ -z "$username" ]] && { read -rp " Username to disable: " username; }
[[ -z "$username" ]] && die "Username is required."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_active = false
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
|| die "No such account: ${username}"
run_psql -v ON_ERROR_STOP=1 -c "
DELETE FROM pf.session
WHERE sess::jsonb -> 'user' ->> 'username' ILIKE '$(sql_lit "$username")'" >/dev/null
success "Account '${username}' disabled and signed out."
}
cmd_enable_user() {
require_env; load_env
local username="${1:-}"
[[ -z "$username" ]] && { read -rp " Username to enable: " username; }
[[ -z "$username" ]] && die "Username is required."
run_psql -v ON_ERROR_STOP=1 -tAc "
WITH upd AS (
UPDATE pf.app_user SET is_active = true
WHERE lower(username) = lower('$(sql_lit "$username")')
RETURNING id
) SELECT id FROM upd" | grep -q . \
&& success "Account '${username}' enabled." \
|| die "No such account: ${username}"
}
cmd_install_service() { cmd_install_service() {
require_systemd require_systemd
require_env require_env
@ -575,17 +303,9 @@ interactive_menu() {
echo " 5) status service + DB + git info" echo " 5) status service + DB + git info"
echo " 6) logs tail journald logs" echo " 6) logs tail journald logs"
echo " 7) db-setup apply setup_sql/01_schema.sql" echo " 7) db-setup apply setup_sql/01_schema.sql"
echo " 8) config set DB connection + app PORT" echo " 8) config set DATABASE_URL / PORT / PF_USER"
echo " 9) install-service create systemd unit file" echo " 9) install-service create systemd unit file"
echo " 10) uninstall-service remove systemd unit file" echo " 10) uninstall-service remove systemd unit file"
echo " 11) add-user create a login account"
echo " 12) passwd change an account password"
echo " 13) list-users show accounts"
echo " 14) disable-user deactivate an account and sign it out"
echo " 15) enable-user reactivate an account"
echo " 16) set-territory grant an account the territory values it may see"
echo " 17) set-admin make an account an administrator"
echo " 18) orphan-territory territory values no account owns"
echo " q) quit" echo " q) quit"
echo echo
read -rp " Choice: " choice read -rp " Choice: " choice
@ -600,14 +320,6 @@ interactive_menu() {
8|config) cmd_config ;; 8|config) cmd_config ;;
9|install-service) cmd_install_service ;; 9|install-service) cmd_install_service ;;
10|uninstall-service) cmd_uninstall_service ;; 10|uninstall-service) cmd_uninstall_service ;;
11|add-user) cmd_add_user ;;
12|passwd) cmd_passwd ;;
13|list-users) cmd_list_users ;;
14|disable-user) cmd_disable_user ;;
15|enable-user) cmd_enable_user ;;
16|set-territory) cmd_set_territory ;;
17|set-admin) cmd_set_admin ;;
18|orphan-territory) cmd_orphan_territory ;;
q|Q|quit|exit) echo "Bye."; exit 0 ;; q|Q|quit|exit) echo "Bye."; exit 0 ;;
*) warn "Unknown option: $choice" ;; *) warn "Unknown option: $choice" ;;
esac esac
@ -627,14 +339,6 @@ case "${1:-}" in
config) cmd_config ;; config) cmd_config ;;
install-service) cmd_install_service ;; install-service) cmd_install_service ;;
uninstall-service) cmd_uninstall_service ;; uninstall-service) cmd_uninstall_service ;;
add-user) cmd_add_user ;;
passwd) cmd_passwd ;;
list-users) cmd_list_users ;;
set-territory) shift; cmd_set_territory "$@" ;;
set-admin) shift; cmd_set_admin "$@" ;;
orphan-territory) shift; cmd_orphan_territory "$@" ;;
disable-user) cmd_disable_user "${2:-}" ;;
enable-user) cmd_enable_user "${2:-}" ;;
"") interactive_menu ;; "") interactive_menu ;;
*) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service add-user passwd list-users disable-user enable-user set-territory set-admin orphan-territory" ;; *) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service" ;;
esac esac

View File

@ -113,20 +113,10 @@ CREATE TABLE pf.log (
operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone' operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
slice jsonb, -- the WHERE conditions that defined the selection slice jsonb, -- the WHERE conditions that defined the selection
params jsonb, -- operation parameters (increments, new values, scale factor, etc.) params jsonb, -- operation parameters (increments, new values, scale factor, etc.)
note text, -- user-provided comment note text -- user-provided comment
tag text -- initiative label, e.g. 'reduce_spend'
); );
``` ```
`tag` groups adjustments into initiatives. It is what the bridge walks: every entry
carrying the same tag becomes one step from baseline to current. Both `note` and `tag`
are annotations — they never affect forecast rows — so both stay editable after the
fact via `PATCH /api/log/:logid`.
Tags are written by a follow-up `UPDATE` after the operation runs, not by the generated
SQL. The templates in `pf.sql` are stored per source, so adding a `{{tag}}` token would
silently stop recording tags for any source that had not re-run *Generate SQL*.
### `pf.fc_{tname}_{version_id}` (dynamic, one per version) ### `pf.fc_{tname}_{version_id}` (dynamic, one per version)
Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`). Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`).
@ -328,79 +318,35 @@ All operations share a common request envelope:
```json ```json
{ {
"pf_user": "paul.trowbridge", "pf_user": "paul.trowbridge",
"note": "optional comment", "note": "optional comment",
"tag": "reduce_spend", "slice": {
"slices": [ { "channel": "WHS", "geography": "WEST" }, "channel": "WHS",
{ "channel": "DIR", "geography": "EAST" } ], "geography": "WEST"
"apply_mode": "prorate" }
} }
``` ```
- `slices` — one or more slices. The legacy single `slice` object is still accepted and `slice` keys must be `role = 'dimension'` columns per col_meta. Stored in `pf.log` as the implicit link to affected rows.
treated as a one-entry list.
- `apply_mode``prorate` (default) treats the selection as one pool; `each` runs the
operation once per slice, producing one log entry per slice so they can be undone
separately. With a single slice the two are identical.
- `tag` — optional initiative label, stored on the log entry.
Slice keys must be `role = 'dimension'` or `role = 'date'` columns per col_meta. A slice
naming none of them is **rejected**: unknown keys are dropped when building the WHERE
clause, so such a slice would otherwise reduce to `TRUE` and apply the operation to the
entire version.
Several slices become an `OR` of `AND`-groups, not per-column `IN` lists — flattening
`{A:1,B:1}` and `{A:2,B:2}` into `A IN (1,2) AND B IN (1,2)` would also match `A:1,B:2`.
The result is parenthesised because callers append `AND pf_iter NOT IN (...)`, and `AND`
binds tighter than `OR`.
#### Scale #### Scale
`POST /api/versions/:id/scale` `POST /api/versions/:id/scale`
```json ```json
{ {
"pf_user": "paul.trowbridge", "pf_user": "paul.trowbridge",
"note": "10% volume lift Q3 West", "note": "10% volume lift Q3 West",
"tag": "volume_push", "slice": { "channel": "WHS", "geography": "WEST" },
"slices": [ { "channel": "WHS", "geography": "WEST" } ], "value_incr": null,
"apply_mode": "prorate", "units_incr": 5000,
"target_value": 12000, "pct": false
"units_pct": 10,
"target_basis": "selected"
} }
``` ```
Each measure is resolved **independently**, so a target on one and a percentage on the - `value_incr` / `units_incr` — absolute amounts to add (positive or negative). Either can be null.
other can be sent together. Per measure, exactly one of: - `pct: true` — treat as percentage of current slice total instead of absolute
- Excludes `exclude_iters` rows from the source selection
| Field | Meaning | - Distributes increment proportionally across rows in the slice
|---|---|
| `target_value` / `target_units` | the total to end up with |
| `value_pct` / `units_pct` | a percentage of the current total |
| `value_incr` / `units_incr` | an absolute amount to add |
| `target_price` | target value/units ratio; holds units constant |
The legacy global `pct: true` flag (meaning "the increments are percentages") is still
honoured.
`target_basis` decides what a target or percentage measures against:
- `adjustable` — only the rows the operation can write.
- `selected` (UI default) — everything the pivot shows for the slice, `exclude_iters`
rows included. Those rows cannot move, so the adjustable rows absorb the whole
difference and the pivot lands on the number you asked for. Without this, a target set
against a visible total overshoots by the excluded rows' contribution.
Behaviour:
- Excludes `exclude_iters` rows from the rows it writes, in every basis.
- Distributes the increment proportionally across rows in the slice.
- **Refuses to prorate a pool that nets to ~zero** — below 1% of gross. Each row's new
value is `(row / total) * increment`, so as the net approaches zero the multiplier
explodes and rows fly to extreme opposite values to reach the target. Offsetting
slices are the usual cause; `apply_mode: each` handles that correctly.
- Slices matching no rows, or already on target, are skipped and returned in
`slices_skipped` rather than silently counted as applied.
- Inserts rows tagged `iter = 'scale'` - Inserts rows tagged `iter = 'scale'`
#### Recode #### Recode
@ -410,7 +356,7 @@ Behaviour:
{ {
"pf_user": "paul.trowbridge", "pf_user": "paul.trowbridge",
"note": "Part discontinued, replaced by new SKU", "note": "Part discontinued, replaced by new SKU",
"slices": [ { "part": "OLD-SKU-001" } ], "slice": { "part": "OLD-SKU-001" },
"set": { "part": "NEW-SKU-002" } "set": { "part": "NEW-SKU-002" }
} }
``` ```
@ -428,7 +374,7 @@ Behaviour:
{ {
"pf_user": "paul.trowbridge", "pf_user": "paul.trowbridge",
"note": "New customer win, similar profile to existing", "note": "New customer win, similar profile to existing",
"slices": [ { "customer": "EXISTING CO", "channel": "DIR" } ], "slice": { "customer": "EXISTING CO", "channel": "DIR" },
"set": { "customer": "NEW CO" }, "set": { "customer": "NEW CO" },
"scale": 0.75 "scale": 0.75
} }
@ -445,10 +391,6 @@ Behaviour:
|--------|-------|-------------| |--------|-------|-------------|
| GET | `/api/versions/:id/log` | List all log entries for a version, newest first | | GET | `/api/versions/:id/log` | List all log entries for a version, newest first |
| DELETE | `/api/log/:logid` | Undo: delete all forecast rows with this logid, then delete log entry | | DELETE | `/api/log/:logid` | Undo: delete all forecast rows with this logid, then delete log entry |
| PATCH | `/api/log/:logid` | Edit `note` and/or `tag`. Branches on whether a field was sent, so `""` clears rather than being read as "leave alone" |
| GET | `/api/versions/:id/table-info` | Physical forecast table, source table, and live row counts by `pf_iter` |
| GET | `/api/versions/:id/bridge` | Baseline → current rolled up by tag |
| GET | `/api/sources/:id/tags` | Tags used on this source with use counts, newest first — feeds tag autocomplete |
--- ---
@ -542,47 +484,30 @@ Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any
### Forecast View ### Forecast View
**Layout:** the operation panel docks **bottom** (default), **right**, or **floats** over **Layout:**
the pivot (drag its header to move, corner grip to resize). Position and size persist to
`localStorage`. It closes via its header ×, `Esc`, or the toolbar toggle, which shows the
selection count while shut.
``` ```
┌─────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────┐
│ [Layout…] [Expand 0 1 2 3] [Refresh] [Change log] [Bridge] │ │ [Version label] [Refresh] [Save layout] [Reset layout] │
│ [Hide panel] │ ├──────────────────────────────────────┬──────────────────────────┤
├─────────────────────────────────────────────────────────────────┤ │ │ │
│ │ │ Perspective Viewer │ Operation Panel │
│ Perspective Viewer (interactive pivot web component) │ │ (interactive pivot web component) │ (active when slice set) │
│ │ │ │ │
├──────────────── drag to resize ─────────────────────────────────┤ │ │ Slice: │
│ SLICE 2 selected │ scale recode clone │ Amount │ │ │ channel = WHS │
│ channel=WHS │ Together | Each │ Baseline 1,000.00 │ │ │ geography = WEST │
│ channel=DIR │ │ ▪ reduce_spend -20.00 │ │ │ │
│ Clear selection │ │ ──────────────────── │ │ │ [ Scale ] [ Recode ] │
│ │ │ Adjustable 1,070.00 │ │ │ [ Clone ] │
│ │ │ reference·fixed 921.72 │ │ │ │
│ │ │ ──────────────────── │ │ │ ... operation form ... │
│ │ │ Selected total 1,991.72 │ │ │ │
│ │ │ ──────────────────── │ │ │ [ Submit ] │
│ │ │ New value [ 2,000 ] │ │ │ │
│ │ │ Change [ 8 ] │ └──────────────────────────────────────┴──────────────────────────┘
│ │ │ % change [ 0.4 ] │
│ │ tag [reduce_spend] │ [Apply Scale] │
└─────────────────────────────────────────────────────────────────┘
``` ```
**The ledger.** The scale form is one continuous statement rather than a totals display **Pivot control:** [Perspective](https://perspective.finos.org/) 4.4.0, loaded from CDN at runtime. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
plus a separate input form: baseline, each adjustment (grouped by tag), current, then the
edit. `New value`, `Change` and `% change` are three interchangeable editable rows —
typing in any one derives the other two, and whichever you typed in is what gets sent.
That replaces the old target/delta/percent mode toggle: the row you type in *is* the mode.
Rows the pivot shows but operations cannot write (`exclude_iters`, typically `reference`)
appear as their own line with a `Selected total` beneath, and a control chooses which of
the two a target measures against — see `target_basis` above.
**Pivot control:** [Perspective](https://github.com/perspective-dev/perspective) 5.2.0 (`@perspective-dev/*`), **bundled inline, not loaded from a CDN** — the `/inline` entrypoints embed the WASM so the version is pinned by `package-lock.json`. See `PERSPECTIVE.md`. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
**Large-dataset loading sequence:** **Large-dataset loading sequence:**
1. Client issues `GET /api/versions/:id/data` 1. Client issues `GET /api/versions/:id/data`
@ -595,25 +520,9 @@ the two a target measures against — see `target_basis` above.
**Interaction flow:** **Interaction flow:**
1. Click a cell or row in the pivot — the `perspective-click` event fires 1. Click a cell or row in the pivot — the `perspective-click` event fires
2. `detail.config.filter` from the event is parsed: only `==` filters on `role = dimension` columns are extracted as the slice 2. `detail.config.filter` from the event is parsed: only `==` filters on `role = dimension` columns are extracted as the slice
3. A plain click replaces the selection; **ctrl/⌘/shift-click toggles** a slice in or out of 3. Slice populates the Operation Panel — pick operation tab, fill in parameters
it. The `CustomEvent` carries no modifier flags, so they are read from the `mousedown` 4. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
that preceded it. `perspective-select` (region drag) is wired defensively alongside. 5. For recode, both the negative offset rows and positive replacement rows are returned and streamed
4. Slice populates the Operation Panel — pick operation tab, fill in parameters
5. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
6. For recode, both the negative offset rows and positive replacement rows are returned and streamed
**Selection caveat.** `pf_iter` is not a `col_meta` column, so it is stripped when a slice
is built. Two cells differing only by iter band (baseline vs reference) produce the same
effective slice; duplicates are collapsed before the request, and the panel warns when a
selection covers fewer distinct slices than cells clicked. There is currently no way to
target one band of a slice.
**Expand depth.** Perspective's `GROUP BY ROLLUP` view contains every level of the
hierarchy, and `view.set_depth()` — which lives on the view, not in the saved config — is
the only thing hiding the deeper ones. The viewer rebuilds its view whenever it redraws,
which its Intersection/ResizeObserver triggers on tab refocus, leaving the tree fully
expanded. The last applied depth is therefore re-applied on `visibilitychange`, `focus`
and `pageshow`.
**Pivot default layout:** built from col_meta — first two `dimension` columns as `group_by`, `date` column as `split_by`. User can rearrange in Perspective settings panel and save. **Pivot default layout:** built from col_meta — first two `dimension` columns as `group_by`, `date` column as `split_by`. User can rearrange in Perspective settings panel and save.
@ -621,43 +530,8 @@ and `pageshow`.
### Log View ### Log View
Modal list of log entries — timestamp, operation, slice, **tag**, note, rows affected. AG Grid list of log entries — user, timestamp, operation, slice, note, rows affected.
"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of "Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of Perspective table).
Perspective table).
Tag and note are edited inline (click, Enter to save, Esc to cancel) via
`PATCH /api/log/:logid`; the tag field completes from tags already used on the source.
Saving a tag regroups the ledger and bridge immediately, so history can be reclassified
after the fact.
### Bridge View
A waterfall answering "how did this version get from its baseline to where it stands?",
one step per initiative tag, opened from the toolbar.
```
6.0k ┤ ┌──────┐- - - -┐
│ │+3,624│ │
4.0k ┤ │ │ 3,800│
│ ┌─────┐- ┘ └ - - - ┘──┐ ┌─────┐
2.0k ┤ │2,734│ │+509│ │3,067│
0 ┴──┴─────┴────────────────┴────┴─┴─────┴──
Baseline clamp give food Current
```
**Scope:** the current slice selection (default when one exists), the pivot's current
filters, or the whole version. Selection scope uses the **union** of the selected slices —
the same reach an operation would have — with rows matching more than one slice deduped
by `pf_id` to match the `OR` semantics operations use.
Computed from the Perspective table already loaded in the browser rather than from
`/api/versions/:id/bridge`, so the figures always reconcile with what the pivot is
showing. The endpoint remains for API consumers.
**Colour** encodes polarity, not identity: increases and decreases are two poles of one
scale, so it uses a validated diverging pair (blue/red, CVD ΔE 21.6 — green/red is avoided
as the classic colourblind failure) with neutral grey anchors for baseline and current.
Every bar is directly labelled and a table view gives the same numbers at full precision.
--- ---
@ -840,26 +714,13 @@ DELETE FROM pf.log WHERE id = {{logid}};
--- ---
## Display-grain pre-aggregation ## Display-grain pre-aggregation (planned)
**Status:** built (static grain). This is **Path B** (pre-aggregated extract → **Status:** designed, not yet built. This is the concrete design for **Path B**
native Perspective table) of two candidate designs; rationale, the Path A (pre-aggregated extract → native Perspective table) of two candidate designs;
alternative (live virtual-server aggregation), the spike evidence, and the rationale, the Path A alternative (live virtual-server aggregation), the spike
A-vs-B trade-off live in `pf_perspective_options.md` (§Two candidate designs, evidence, and the A-vs-B trade-off live in `pf_perspective_options.md`
§Spike findings). (§Two candidate designs, §Spike findings).
The grain is **static** — set once per source in Setup and baked into the stored
`pf.sql` templates, so load and operations agree by construction. `in_grain`
means *eligible for the grain*, and in this version the grain is exactly the set
of eligible columns. Deriving a narrower grain per pivot at request time (the
dynamic variant) is then additive: intersect the viewer's field set with the
eligible set. Leaving high-cardinality columns (`part`, raw day dates,
currency-level detail) unflagged is what keeps the grain from exploding back
toward raw, regardless of what a user drags into the pivot.
**Measured on `pf.fc_osm_stack_20`** at `pending_rep × customer × smon`:
534,902 → **6,154** rows (≈87×), `pf_gkey` unique across all 6,154, and both
measures reconcile exactly to the raw totals (283,296,087.67 / 962,142,261.46).
**Problem it solves.** The current transport ships every raw forecast row to the **Problem it solves.** The current transport ships every raw forecast row to the
browser (≈535k rows / ~250 MB / ~2 min on `osm_stack`). Perspective then pivots browser (≈535k rows / ~250 MB / ~2 min on `osm_stack`). Perspective then pivots
@ -890,11 +751,7 @@ the stored `pf.sql` templates, so initial load and operations agree on it.
### Initial load — `GET /api/versions/:id/agg` ### Initial load — `GET /api/versions/:id/agg`
Replaces the raw `/data` stream for grain-based versions. Aggregates the forecast Replaces the raw `/data` stream for grain-based versions. Aggregates the forecast
table to the stored grain and returns Arrow IPC. The template is stored in table to the stored grain and returns Arrow IPC:
`pf.sql` as operation `get_agg`, generated only when a grain is defined; clearing
the grain and regenerating removes it, and the client falls back to `/data`.
Both endpoints speak the same protocol (one record batch plus an `X-Row-Count`
header), so the client only chooses the URL:
```sql ```sql
SELECT SELECT
@ -926,12 +783,6 @@ the smallest change from today's code, which already appends operation results v
accumulate (rather than replacing a bucket) and a delete can remove exactly that accumulate (rather than replacing a bucket) and a delete can remove exactly that
operation's rows. operation's rows.
As built, the concatenation is
`concat_ws(chr(31), COALESCE(col::text, chr(30)), …, pf_iter, pf_logid::text)`.
The separator and NULL sentinel matter: plain `concat_ws` skips NULLs, so
`('a', NULL)` and `(NULL, 'a')` would produce the same key and silently merge two
groups into one indexed row. `chr(30)` also keeps NULL distinct from `''`.
### Write path (scale / recode / clone) — append the new log entry's rows ### Write path (scale / recode / clone) — append the new log entry's rows
Operations INSERT raw rows into `{{fc_table}}` under a new `pf_logid` as today; the Operations INSERT raw rows into `{{fc_table}}` under a new `pf_logid` as today; the
@ -962,20 +813,9 @@ because each row carries the new, unique `pf_logid`.)
A logid's rows are uniquely keyed, so undo just removes them and lets the view A logid's rows are uniquely keyed, so undo just removes them and lets the view
re-sum — no re-aggregation, no emptied-bucket handling, no snapshot caveat: re-sum — no re-aggregation, no emptied-bucket handling, no snapshot caveat:
`RETURNING` does not accept `DISTINCT`, so the delete feeds a CTE that reduces its
output to the distinct grain keys. `rows_deleted` still counts raw rows removed:
```sql ```sql
WITH DELETE FROM {{fc_table}} WHERE pf_logid = {{logid}}
del AS ( RETURNING DISTINCT {{grain_cols}}, pf_iter, pf_logid; -- → pf_gkeys to remove
DELETE FROM {{fc_table}}
WHERE pf_logid = {{logid}}
RETURNING {{grain_cols}}, pf_iter, pf_logid
)
SELECT
count(*)::int AS rows_deleted
,array_agg(DISTINCT {{grain_key}}) AS pf_gkeys
FROM del;
DELETE FROM pf.log WHERE id = {{logid}}; DELETE FROM pf.log WHERE id = {{logid}};
``` ```
@ -1028,16 +868,14 @@ simpler fallback and is now cheap — ~25 ms.)
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501 - **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred) - **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred) - **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
- **Export** — download forecast as CSV or push results to a reporting table. The bridge's table view is a partial stand-in for reading the numbers out, but there is no download. - **Export** — download forecast as CSV or push results to a reporting table
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION). The bridge answers the within-version form of this question; across versions is still open. - **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION)
- **Bridge drill-down** — click a step to list the adjustments behind it, or select that slice back in the pivot
- **Targeting one iter band** — make `pf_iter` part of a slice so an operation can act on, say, only the baseline rows of a selection (see Known issues)
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually. - **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually.
- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema. - **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema.
--- ---
## Project Status — 2026-09-11 ## Project Status — 2026-06-12
### What's working ### What's working
- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo - Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo
@ -1047,30 +885,19 @@ simpler fallback and is now cheap — ~25 ms.)
- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express - React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express
- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM - Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM
- 3-step collapsible sidebar (Setup / Baseline / Forecast) - 3-step collapsible sidebar (Setup / Baseline / Forecast)
- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation - Setup view: DB table browser with preview modal, source registration, col_meta editor (`dim_group`/`dim_period_col` fields included), SQL generation
- Baseline view: version management, multi-segment baseline workbench, canvas timeline, filter builder - Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage - Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage
- Incremental row streaming: operation results (`RETURNING *`) applied via `pspTable.update()` — no full reload - Slice extraction from `perspective-click` event feeds operation panel directly
- **Multi-slice operations**: ctrl/⌘-click accumulates slices; `apply_mode` prorate/each - Incremental row streaming: operation results (`RETURNING *`) applied to Perspective table via `pspTable.update()` — no full reload
- **Per-measure resolution**: target, percent or change amount independently per measure - Status bar: shows current source · version · baseline row count · status
- **`target_basis`**: a target measures against the adjustable rows or everything the pivot shows
- **Ledger panel**: baseline → adjustments → current → three interchangeable editable rows, docked bottom/right/floating
- **Tags and bridge**: initiative tags on log entries, editable after the fact, with a waterfall view scoped to selection / filters / version
- **Status bar** names the physical table writes land in, with live row counts by iter
### Known issues / next focus ### Known issues / next focus
- **`pf_iter` not selectable** — it is not a col_meta column, so it is stripped from slices. Cells differing only by iter band collapse to one slice (duplicates are detected and collapsed, and the panel warns), and there is no way to operate on one band of a slice. - **Forecast view** — operation panel SQL generation complete; UI wiring to API still needed
- **Per-row rounding drift** — the scale SQL rounds each row to 2dp, so a target of 1,000 across many rows can land on 999.99. Inherent to proportional distribution; a correction row would be needed to land exactly. - **Load progress bar** — jittery at high throughput; throttle to ~10 updates/sec
- **Manual caret expansion is not restored** — the depth re-apply on refocus only covers whole-tree depths set via the Expand buttons or a saved layout, since per-row expansion lives in the same discarded view. - **Default pivot layout** — per-source configurable layout not yet implemented; currently hardcodes first 2 dimensions
- **Bridge has no drill-down** — clicking a step does not list its adjustments or select that slice back in the pivot. - **No "current version" persistence** — source/version selection resets on page reload
- **Light surface only** — app chrome is light throughout; the dark toggle currently re-themes only the Perspective viewer. - **Perspective slice limitation** — computed date columns (Month, YearDate) from split_by don't map back to raw rows; only native dimension columns work for slice extraction
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version. - **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version.
- **No migration sequence**`01_schema.sql` carries `ADD COLUMN IF NOT EXISTS` inline for the `tag` column, which covers fresh installs and re-runs, but there is no ordered migration mechanism.
- **No tests** — SQL generation is token substitution against append-only tables and is entirely untested.
### Fixed
- **Non-selective slices applied to the whole version** — a slice naming no filterable column reduced to `TRUE`. Now rejected on all three operations.
- **Proration across a near-zero pool** — rows flew to extreme opposite values to reach a target. Refused when the net is below 1% of gross; `apply_mode: each` is the alternative.
- **Targets overshooting by excluded rows** — see `target_basis`.

View File

@ -1,110 +0,0 @@
const express = require('express');
const { verifyPassword } = require('../lib/auth');
// Per-IP login throttle. In-memory on purpose: it only has to blunt online
// guessing, and a counter that resets on restart is the acceptable cost of not
// writing a failed-attempt row for every knock on an internet-facing port.
const WINDOW_MS = 15 * 60 * 1000;
const MAX_ATTEMPTS = 10;
const attempts = new Map(); // ip -> { count, resetAt }
function tooManyAttempts(ip) {
const rec = attempts.get(ip);
if (!rec || Date.now() > rec.resetAt) return false;
return rec.count >= MAX_ATTEMPTS;
}
function recordFailure(ip) {
const rec = attempts.get(ip);
if (!rec || Date.now() > rec.resetAt) {
attempts.set(ip, { count: 1, resetAt: Date.now() + WINDOW_MS });
} else {
rec.count += 1;
}
}
// Keep the map from growing without bound on a long-lived process.
setInterval(() => {
const now = Date.now();
for (const [ip, rec] of attempts) if (now > rec.resetAt) attempts.delete(ip);
}, WINDOW_MS).unref();
module.exports = function(pool) {
const router = express.Router();
router.post('/login', async (req, res) => {
const ip = req.ip;
if (tooManyAttempts(ip)) {
return res.status(429).json({ error: 'Too many failed attempts. Try again later.' });
}
const username = String(req.body?.username || '').trim();
const password = String(req.body?.password || '');
if (!username || !password) {
return res.status(400).json({ error: 'Username and password are required' });
}
try {
const result = await pool.query(
`SELECT id, username, display_name, pass_hash, is_active,
is_admin, territory
FROM pf.app_user WHERE lower(username) = lower($1)`,
[username]
);
const user = result.rows[0];
// Same message and roughly the same work either way: no unknown
// user / wrong password / disabled distinction to enumerate.
const ok = user && user.is_active && verifyPassword(password, user.pass_hash);
if (!ok) {
recordFailure(ip);
return res.status(401).json({ error: 'Invalid username or password' });
}
// New session id on login — an existing cookie can't be fixated.
req.session.regenerate(err => {
if (err) {
console.error(err);
return res.status(500).json({ error: 'Could not start session' });
}
req.session.user = {
id: user.id,
username: user.username,
display_name: user.display_name,
is_admin: !!user.is_admin,
territory: Array.isArray(user.territory) ? user.territory : [],
};
pool.query(`UPDATE pf.app_user SET last_login_at = now() WHERE id = $1`, [user.id])
.catch(e => console.error('last_login_at update failed', e));
req.session.save(err2 => {
if (err2) {
console.error(err2);
return res.status(500).json({ error: 'Could not start session' });
}
attempts.delete(ip);
res.json({ user: req.session.user });
});
});
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
router.post('/logout', (req, res) => {
const name = req.session?.cookie && req.app.get('session cookie name');
req.session.destroy(err => {
if (err) console.error(err);
res.clearCookie(name || 'pf.sid');
res.json({ ok: true });
});
});
// The UI calls this on load to decide between the login screen and the app.
router.get('/me', (req, res) => {
if (!req.session?.user) return res.status(401).json({ error: 'Not authenticated' });
res.json({ user: req.session.user });
});
return router;
};

View File

@ -1,193 +0,0 @@
const express = require('express');
const { sessionUser } = require('../lib/auth');
// Named Perspective view configs. Two kinds, one table:
//
// private — yours, nobody else lists it
// published — everyone on the version sees it; only the owner or an admin
// may change it, the same rule pf.log uses for its entries
//
// Scope is the version by default, because that is where people enter the app.
// A layout with version_id NULL applies to every version of the source, which
// is where the old pf.source.default_layout went and what a brand-new version
// picks up before anyone has published anything for it.
module.exports = function(pool) {
const router = express.Router();
const SELECT_COLS = `id, source_id, version_id, name, config, owner,
visibility, is_default, created_at, updated_at`;
// What the client needs to know about a row it cannot write, so the UI can
// grey the control out rather than offer a click that answers 403.
const decorate = (row, req) => ({
...row,
scope: row.version_id == null ? 'source' : 'version',
can_edit: !!req.session?.user?.is_admin || row.owner === sessionUser(req)
});
// The owner/admin gate, shared by every write. Returns the row, or sends the
// response itself and returns null.
async function ownedLayout(req, res) {
const id = parseInt(req.params.lid);
const { rows } = await pool.query(
`SELECT ${SELECT_COLS} FROM pf.layout WHERE id = $1`, [id]
);
if (!rows.length) { res.status(404).json({ error: 'Layout not found' }); return null; }
const row = rows[0];
if (!req.session?.user?.is_admin && row.owner !== sessionUser(req)) {
res.status(403).json({
error: `${row.name}” belongs to ${row.owner} — only they or an administrator can change it`
});
return null;
}
return row;
}
// Everything applicable to a version: its own published layouts, the
// source-wide published ones, and the caller's own private layouts at either
// scope. Deliberately not territory-filtered — a layout is display config,
// and territory restricts rows, not columns.
router.get('/versions/:id/layouts', async (req, res) => {
const versionId = parseInt(req.params.id);
try {
const ver = await pool.query(
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
);
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
const sourceId = ver.rows[0].source_id;
const { rows } = await pool.query(
`SELECT ${SELECT_COLS}
FROM pf.layout
WHERE TRUE
AND source_id = $1
AND (version_id = $2 OR version_id IS NULL)
AND (visibility = 'published' OR owner = $3)
ORDER BY visibility DESC, is_default DESC, lower(name)`,
[sourceId, versionId, sessionUser(req)]
);
res.json(rows.map(r => decorate(r, req)));
} catch (err) {
res.status(500).json({ error: err.message });
}
});
// Create. scope 'source' saves it against every version of the source;
// anything else is this version.
router.post('/versions/:id/layouts', async (req, res) => {
const versionId = parseInt(req.params.id);
const { name, config, visibility = 'private', scope = 'version' } = req.body || {};
if (!name || !String(name).trim()) return res.status(400).json({ error: 'Name is required' });
if (!config) return res.status(400).json({ error: 'Config is required' });
if (!['private', 'published'].includes(visibility)) {
return res.status(400).json({ error: 'visibility must be private or published' });
}
try {
const ver = await pool.query(
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
);
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
const { rows } = await pool.query(
`INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility)
VALUES ($1, $2, $3, $4, $5, $6)
RETURNING ${SELECT_COLS}`,
[ver.rows[0].source_id, scope === 'source' ? null : versionId,
String(name).trim(), config, sessionUser(req), visibility]
);
res.json(decorate(rows[0], req));
} catch (err) {
// the partial unique indexes, reported in the terms the user typed
if (err.code === '23505') {
return res.status(409).json({ error: `A layout named “${name}” already exists here` });
}
res.status(500).json({ error: err.message });
}
});
// Update any of name / config / visibility / is_default. Each is optional;
// a missing key leaves the column alone, which is why the flag-and-value
// pair is used rather than COALESCE on the value.
router.patch('/layouts/:lid', async (req, res) => {
const { name, config, visibility, is_default, scope } = req.body || {};
if (name === undefined && config === undefined && visibility === undefined
&& is_default === undefined && scope === undefined) {
return res.status(400).json({ error: 'Nothing to update' });
}
if (visibility !== undefined && !['private', 'published'].includes(visibility)) {
return res.status(400).json({ error: 'visibility must be private or published' });
}
const client = await pool.connect();
try {
const row = await ownedLayout(req, res);
if (!row) return;
// Unpublishing a default would leave a default nobody can see, which
// the table's CHECK refuses; clear the flag with it rather than
// failing on a constraint the user never mentioned.
const clearsDefault = visibility === 'private';
// One default per scope, enforced by a partial unique index. Stand
// the others down first rather than letting the update collide --
// and in one transaction, or a name clash on the second statement
// leaves the version with no default at all.
const target_version = scope === undefined ? row.version_id
: (scope === 'source' ? null : row.version_id);
await client.query('BEGIN');
if (is_default === true) {
await client.query(
`UPDATE pf.layout SET is_default = false
WHERE TRUE
AND source_id = $1
AND COALESCE(version_id, 0) = COALESCE($2::int, 0)
AND id <> $3
AND is_default`,
[row.source_id, target_version, row.id]
);
}
const { rows } = await client.query(
`UPDATE pf.layout SET
name = CASE WHEN $2::bool THEN $3::text ELSE name END,
config = CASE WHEN $4::bool THEN $5::jsonb ELSE config END,
visibility = CASE WHEN $6::bool THEN $7::text ELSE visibility END,
is_default = CASE WHEN $10::bool THEN false
WHEN $8::bool THEN $9::bool ELSE is_default END,
version_id = CASE WHEN $11::bool THEN $12::int ELSE version_id END,
updated_at = now()
WHERE id = $1
RETURNING ${SELECT_COLS}`,
[row.id,
name !== undefined, name !== undefined ? String(name).trim() : null,
config !== undefined, config !== undefined ? config : null,
visibility !== undefined, visibility !== undefined ? visibility : null,
is_default !== undefined, is_default === true,
clearsDefault,
scope !== undefined, target_version]
);
await client.query('COMMIT');
res.json(decorate(rows[0], req));
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
if (err.code === '23505') {
return res.status(409).json({ error: 'A layout with that name already exists here' });
}
res.status(500).json({ error: err.message });
} finally {
client.release();
}
});
router.delete('/layouts/:lid', async (req, res) => {
try {
const row = await ownedLayout(req, res);
if (!row) return;
await pool.query(`DELETE FROM pf.layout WHERE id = $1`, [row.id]);
res.json({ deleted: row.id });
} catch (err) {
res.status(500).json({ error: err.message });
}
});
return router;
};

View File

@ -1,6 +1,4 @@
const express = require('express'); const express = require('express');
const { grainOf } = require('../lib/sql_generator');
const { sessionUser } = require('../lib/auth');
const { fcTable } = require('../lib/utils'); const { fcTable } = require('../lib/utils');
module.exports = function(pool) { module.exports = function(pool) {
@ -31,96 +29,17 @@ module.exports = function(pool) {
unitsCol ? `sum(f."${unitsCol}")::float8 AS units_total` : `NULL::float8 AS units_total` unitsCol ? `sum(f."${unitsCol}")::float8 AS units_total` : `NULL::float8 AS units_total`
].join(', '); ].join(', ');
// The totals are stamped onto the entry when it is written, so the const result = await pool.query(`
// normal read is a scan of a few dozen log rows rather than a join SELECT l.*, ${aggCols},
// against millions of forecast rows. $2::text AS value_col,
// $3::text AS units_col
// ?recount=1 does it the old way. Stored totals are fixed at write FROM pf.log l
// time and cannot drift on their own, but nothing stops someone LEFT JOIN ${table} f ON f.pf_logid = l.id
// deleting forecast rows by hand, and a stored figure has no way to WHERE l.version_id = $1
// notice. This is the way back -- and the backfill for entries GROUP BY l.id
// written before the columns existed. ORDER BY l.id DESC
const recount = req.query.recount === '1' || req.query.recount === 'true'; `, [versionId, valueCol || null, unitsCol || null]);
const stamped = !recount && (await pool.query( res.json(result.rows);
`SELECT count(*)::int AS n FROM pf.log
WHERE version_id = $1 AND row_count IS NULL`, [versionId]
)).rows[0].n === 0;
// ?kind=adjustments drops the baseline and reference entries. That is not
// only about what gets listed: the aggregate below joins the whole
// forecast table, and on a real version the load entries own almost
// every row of it -- 2.5M against a few thousand for the adjustments.
// Filtering in the WHERE keeps them out of the join rather than
// totalling them and discarding the answer.
const adjustmentsOnly = req.query.kind === 'adjustments';
const opFilter = adjustmentsOnly
? `AND l.operation NOT IN ('baseline', 'reference')`
: '';
const result = stamped
? await pool.query(`
SELECT l.*,
$2::text AS value_col,
$3::text AS units_col
FROM pf.log l
WHERE l.version_id = $1
${opFilter}
ORDER BY l.id DESC
`, [versionId, valueCol || null, unitsCol || null])
: await pool.query(`
SELECT l.*, ${aggCols},
$2::text AS value_col,
$3::text AS units_col
FROM pf.log l
LEFT JOIN ${table} f ON f.pf_logid = l.id
WHERE l.version_id = $1
${opFilter}
GROUP BY l.id
ORDER BY l.id DESC
`, [versionId, valueCol || null, unitsCol || null]);
// A recount is also a repair: write back what it found, so the next
// read is cheap again and the stored figure matches the rows.
if (recount) {
for (const r of result.rows) {
await pool.query(
`UPDATE pf.log SET row_count = $2, value_total = $3, units_total = $4,
measure_cols = $5::jsonb
WHERE id = $1`,
[r.id, r.row_count, r.value_total, r.units_total,
JSON.stringify({ value: valueCol || null, units: unitsCol || null })]
);
}
}
// The statement is kilobytes per entry and the list is opened to scan,
// not to read SQL. It stays in the row for the debug endpoint below.
res.json(result.rows.map(({ sql_text, ...r }) => ({
...r, has_sql: !!sql_text,
})));
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// Everything about one entry, for when the rows look wrong: what was asked
// for (params), what it ran against (env), and the statement that actually
// executed with territory and scope resolved into it (sql_text).
//
// Both the intent and the SQL, because the translation between them is
// exactly what is in doubt when a result is surprising.
router.get('/log/:logid/debug', async (req, res) => {
const logId = parseInt(req.params.logid);
try {
const { rows } = await pool.query(
`SELECT l.*, v.name AS version_name, s.schema, s.tname
FROM pf.log l
JOIN pf.version v ON v.id = l.version_id
JOIN pf.source s ON s.id = v.source_id
WHERE l.id = $1`, [logId]
);
if (!rows.length) return res.status(404).json({ error: 'Log entry not found' });
res.json(rows[0]);
} catch (err) { } catch (err) {
console.error(err); console.error(err);
res.status(err.status || 500).json({ error: err.message }); res.status(err.status || 500).json({ error: err.message });
@ -132,7 +51,7 @@ module.exports = function(pool) {
const logId = parseInt(req.params.logid); const logId = parseInt(req.params.logid);
try { try {
const logResult = await pool.query(` const logResult = await pool.query(`
SELECT l.*, v.status, s.tname, v.id AS version_id, v.source_id SELECT l.*, v.status, s.tname, v.id AS version_id
FROM pf.log l FROM pf.log l
JOIN pf.version v ON v.id = l.version_id JOIN pf.version v ON v.id = l.version_id
JOIN pf.source s ON s.id = v.source_id JOIN pf.source s ON s.id = v.source_id
@ -141,58 +60,19 @@ module.exports = function(pool) {
if (!logResult.rows.length) return res.status(404).json({ error: 'Log entry not found' }); if (!logResult.rows.length) return res.status(404).json({ error: 'Log entry not found' });
const log = logResult.rows[0]; const log = logResult.rows[0];
if (log.status === 'closed') return res.status(403).json({ error: 'Version is closed' }); if (log.status === 'closed') return res.status(403).json({ error: 'Version is closed' });
// Undo deletes rows wholesale by logid, so it cannot be territory
// filtered the way a read or a write can -- half-undoing an entry
// would leave the version in a state nothing describes. Ownership
// instead: your own entries, or an admin's override. Territory alone
// would not do it anyway, since two accounts can share one.
if (!req.session?.user?.is_admin && log.pf_user !== sessionUser(req)) {
return res.status(403).json({
error: `That entry was made by ${log.pf_user || 'someone else'} — only they or an administrator can undo it`
});
}
const table = fcTable(log.tname, log.version_id); const table = fcTable(log.tname, log.version_id);
// In grain mode the client's table is indexed on pf_gkey, so undo has to
// report the grain keys to remove rather than raw pf_ids. The keys are
// distinct while rows_deleted still counts the raw rows removed.
const colMeta = await pool.query(
`SELECT cname, role, in_grain, opos FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`,
[log.source_id]
);
const grain = grainOf(colMeta.rows);
const client = await pool.connect(); const client = await pool.connect();
try { try {
await client.query('BEGIN'); await client.query('BEGIN');
const deleted = grain const deleted = await client.query(
? await client.query(` `DELETE FROM ${table} WHERE pf_logid = $1 RETURNING pf_id`, [logId]
WITH );
del AS (
DELETE FROM ${table}
WHERE pf_logid = $1
RETURNING ${grain.groupCols().join(', ')}
)
SELECT
count(*)::int AS rows_deleted
,array_agg(DISTINCT ${grain.key()}) AS pf_gkeys
FROM del
`, [logId])
: await client.query(
`DELETE FROM ${table} WHERE pf_logid = $1 RETURNING pf_id`, [logId]
);
await client.query('DELETE FROM pf.log WHERE id = $1', [logId]); await client.query('DELETE FROM pf.log WHERE id = $1', [logId]);
await client.query('COMMIT'); await client.query('COMMIT');
res.json(grain res.json({
? { rows_deleted: deleted.rowCount,
rows_deleted: deleted.rows[0].rows_deleted, pf_ids: deleted.rows.map(r => r.pf_id)
pf_gkeys: deleted.rows[0].pf_gkeys || [] });
}
: {
rows_deleted: deleted.rowCount,
pf_ids: deleted.rows.map(r => r.pf_id)
});
} catch (err) { } catch (err) {
await client.query('ROLLBACK'); await client.query('ROLLBACK');
throw err; throw err;
@ -205,50 +85,13 @@ module.exports = function(pool) {
} }
}); });
// update the note and/or tag on a log entry. Both are annotations — they never // update the note on a log entry
// affect the forecast rows — so they stay editable after the fact, including on
// a closed version, where relabelling history is still legitimate.
router.patch('/log/:logid', async (req, res) => { router.patch('/log/:logid', async (req, res) => {
const logId = parseInt(req.params.logid); const logId = parseInt(req.params.logid);
const { note, tag, bucket, label } = req.body; const { note } = req.body;
if (note === undefined && tag === undefined
&& bucket === undefined && label === undefined) {
return res.status(400).json({
error: 'Nothing to update — send note, tag, bucket and/or label'
});
}
try { try {
// Same rule as undo: your own entries, or an admin's. These are
// annotations, but label and bucket name the pivot's columns for
// everyone who opens the version, so an unguarded PATCH let any
// account rename the company's segments -- including on loads whose
// rows it cannot see.
const owner = await pool.query(
`SELECT pf_user FROM pf.log WHERE id = $1`, [logId]
);
if (!owner.rows.length) return res.status(404).json({ error: 'Log entry not found' });
if (!req.session?.user?.is_admin && owner.rows[0].pf_user !== sessionUser(req)) {
return res.status(403).json({
error: `That entry was made by ${owner.rows[0].pf_user || 'someone else'} — only they or an administrator can change it`
});
}
// COALESCE on the flag, not the value: an explicit null or '' must be
// able to clear a field, which COALESCE on the value alone would ignore
const result = await pool.query( const result = await pool.query(
`UPDATE pf.log SET `UPDATE pf.log SET note = $1 WHERE id = $2 RETURNING *`, [note ?? null, logId]
note = CASE WHEN $2::bool THEN $3::text ELSE note END,
tag = CASE WHEN $4::bool THEN $5::text ELSE tag END,
bucket = CASE WHEN $6::bool THEN $7::text ELSE bucket END,
label = CASE WHEN $8::bool THEN $9::text ELSE label END
WHERE id = $1 RETURNING *`,
[
logId,
note !== undefined, note === undefined ? null : (String(note).trim() || null),
tag !== undefined, tag === undefined ? null : (String(tag).trim() || null),
bucket !== undefined, bucket === undefined ? null : (String(bucket).trim() || null),
label !== undefined, label === undefined ? null : (String(label).trim() || null),
]
); );
if (!result.rows.length) return res.status(404).json({ error: 'Log entry not found' }); if (!result.rows.length) return res.status(404).json({ error: 'Log entry not found' });
res.json(result.rows[0]); res.json(result.rows[0]);

View File

@ -1,338 +1,14 @@
const express = require('express'); const express = require('express');
const { tableFromArrays, tableToIPC } = require('apache-arrow'); const { tableFromArrays, tableToIPC } = require('apache-arrow');
const { applyTokens, buildWhere, buildWhereAny, COMPUTED_SLICE_COLS, buildScopeClause, buildTerritoryClause, buildExcludeClause, buildExcludePredicate, buildSetClause, dateGroupsOf, dimPeriodMapOf, esc, const { applyTokens, buildWhere, buildExcludeClause, buildSetClause, esc } = require('../lib/sql_generator');
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR, VERSION_JOIN,
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET } = require('../lib/sql_generator');
const { sessionUser, sessionTerritory } = require('../lib/auth');
const { fcTable } = require('../lib/utils'); const { fcTable } = require('../lib/utils');
module.exports = function(pool) { module.exports = function(pool) {
const router = express.Router(); const router = express.Router();
async function runSQL(sql, client) { async function runSQL(sql) {
console.log('--- SQL ---\n', sql, '\n--- END SQL ---'); console.log('--- SQL ---\n', sql, '\n--- END SQL ---');
return (client || pool).query(sql); return pool.query(sql);
}
// accept either the legacy single `slice` object or the newer `slices` array,
// and drop any empty entries so an empty selection can never widen to TRUE
function normalizeSlices(body) {
const raw = Array.isArray(body.slices) && body.slices.length ? body.slices : [body.slice];
return raw.filter(s => s && typeof s === 'object' && Object.keys(s).length > 0);
}
// Stamp the tag onto the log entry the operation just created.
// Done as a follow-up UPDATE rather than inside the generated SQL: those
// templates live in pf.sql per source, so adding a {{tag}} token would strand
// every source that has not re-run "Generate SQL".
async function tagLog(client, rows, tag) {
const clean = (tag || '').trim();
if (!clean) return null;
const ids = [...new Set(rows.map(r => r.pf_logid).filter(id => id != null))];
if (ids.length === 0) return null;
await client.query(`UPDATE pf.log SET tag = $1 WHERE id = ANY($2::bigint[])`, [clean, ids]);
return clean;
}
// How a multi-slice request is split into statements.
//
// 'each' — one statement per slice, so every slice reaches its target on
// its own and gets its own log entry
// 'prorate' — a single statement over all of them, letting the SQL's
// sum() OVER () distribute across the whole pool
//
// Only scale has a target to prorate towards; recode and clone rewrite rows
// rather than distribute an amount, so for them this only decides whether the
// work lands as one log entry or several.
// The scope is ANDed onto every unit rather than folded into the slices: it
// applies to all of them equally, and under apply_mode 'each' folding it in
// would repeat the same predicate in every statement for no gain.
function sliceUnits(slices, ctx, applyMode, scope, req) {
const vid = ctx.version.id;
const scl = buildScopeClause(scope, ctx.filterCols, vid);
const terr = req ? territoryOf(req, ctx) : '';
const and = (w) => {
let out = w;
for (const extra of [scl, terr]) {
if (!extra) continue;
out = (out === 'TRUE' || !out) ? extra : `${out}\nAND ${extra}`;
}
return out;
};
return applyMode === 'each'
? slices.map(sl => ({ slices: [sl], where: and(buildWhere(sl, ctx.filterCols, vid)) }))
: [{ slices, where: and(buildWhereAny(slices, ctx.filterCols, vid)) }];
}
// The offset is interpolated into the statement as an interval literal, so a
// typo would surface as a Postgres parse error from the middle of a CTE. Ask
// Postgres to parse it alone first, where the failure is cheap and can name the
// field it came from. Negative intervals are valid and useful -- '-90 days'
// pulls a plan back a quarter -- so this checks validity, not sign.
async function assertInterval(value, res) {
try {
await pool.query(`SELECT $1::interval`, [value]);
return true;
} catch {
res.status(400).json({
error: `"${value}" is not a valid interval. Try something like `
+ `"4 months", "1 year", "-90 days" or "0 days".`
});
return false;
}
}
// A slice is only meaningful if at least one of its keys is a filterable
// column. buildWhere silently drops unknown keys, so {"typo": "x"} would
// otherwise reduce to TRUE and apply the operation to the whole version.
// Refuse rather than let a malformed selection rewrite every row.
function assertSelective(slices, ctx) {
const allowed = new Set([...ctx.filterCols, ...Object.keys(COMPUTED_SLICE_COLS)]);
slices.forEach((sl, i) => {
const hits = Object.keys(sl).filter(k => allowed.has(k));
if (hits.length === 0) {
const err = new Error(
`Slice ${i + 1} does not name any filterable column ` +
`(${JSON.stringify(sl)}). Expected one of: ${ctx.filterCols.join(', ')}.`
);
err.status = 400;
throw err;
}
});
}
// Stamp what the entry did onto the entry itself.
//
// An indexed lookup on pf_logid, run once at write time, in place of the
// change log joining the whole forecast table on every open. Safe to store
// rather than derive because these rows never change: only this operation
// inserts them, and the only thing that removes them is undo, which deletes
// the log row too.
//
// Best-effort by design. A failure here must not roll back a write that
// succeeded -- the totals can always be recomputed, the adjustment cannot.
async function stampLogTotals(client, ctx, logId, extra = {}) {
if (!logId) return;
const v = ctx.valueCol, u = ctx.unitsCol;
// The state the write ran against, none of which can be reconstructed
// later: territory and exclude_iters are mutable rows elsewhere, and the
// template is overwritten in place every time Generate SQL runs. The
// generation timestamp is a fingerprint, not a version -- it cannot bring
// the old template back, only tell you the entry did not run under this
// one.
const env = {
territory: extra.territory ?? null,
territory_col: ctx.territoryCol || null,
exclude_iters: ctx.version.exclude_iters ?? null,
sql_generated_at: ctx.sqlGeneratedAt || null,
};
try {
await client.query(`
UPDATE pf.log SET
row_count = t.n,
value_total = t.v,
units_total = t.u,
measure_cols = $2::jsonb,
env = $3::jsonb,
sql_text = $4::text
FROM (
SELECT count(*)::int AS n
,${v ? `sum(f."${v}")::float8` : 'NULL::float8'} AS v
,${u ? `sum(f."${u}")::float8` : 'NULL::float8'} AS u
FROM ${ctx.table} f
WHERE f.pf_logid = $1
) t
WHERE pf.log.id = $1
`, [logId, JSON.stringify({ value: v || null, units: u || null }),
JSON.stringify(env), extra.sql || null]);
} catch (err) {
console.error('[stampLogTotals]', err);
}
}
// Moving a row between territories is reassignment, not forecasting, so a
// scoped account cannot set the territory column -- otherwise a rep could
// recode work into their own book, or quietly out of it, and the row would
// be gone from the view that would have shown what happened.
//
// Reads and writes are already scoped, so the *source* rows are safely the
// account's own; this is only about the destination.
function assertMayRecodeTerritory(req, ctx, set, res) {
if (!ctx.territoryCol) return true;
if (req.session?.user?.is_admin) return true;
if (!set || set[ctx.territoryCol] === undefined) return true;
res.status(403).json({
error: `Only an administrator can recode ${ctx.territoryCol} — that moves rows between territories`
});
return false;
}
// echo back what the caller asked for, for the audit log
function pickIntent(body) {
const keys = ['mode', 'target_basis', 'value_incr', 'units_incr', 'value_pct', 'units_pct', 'pct',
'target_value', 'target_units', 'target_price', 'scope'];
const out = {};
for (const k of keys) if (body[k] !== undefined && body[k] !== null && body[k] !== '') out[k] = body[k];
return out;
}
// Totals for a WHERE clause, split into the rows operations can change and the
// rows they cannot. Excluded iters (typically 'reference') are still visible in
// the pivot, so their contribution has to be reported rather than dropped —
// otherwise a target set against what the grid shows lands somewhere else.
async function sliceTotals(client, ctx, whereClause, excludeClause) {
const pred = buildExcludePredicate(ctx.version.exclude_iters);
const agg = (col, filter) => col ? `sum(${col}) FILTER (WHERE ${filter})` : 'NULL';
const v = ctx.valueCol ? `"${ctx.valueCol}"` : null;
const u = ctx.unitsCol ? `"${ctx.unitsCol}"` : null;
const r = await client.query(`
SELECT ${agg(v, pred)} AS total_value,
${agg(u, pred)} AS total_units,
${agg(v ? `abs(${v})` : null, pred)} AS abs_value,
${agg(u ? `abs(${u})` : null, pred)} AS abs_units,
${agg(v, `NOT (${pred})`)} AS excl_value,
${agg(u, `NOT (${pred})`)} AS excl_units
FROM ${ctx.table} WHERE ${whereClause}
`);
const n = (x) => parseFloat(r.rows[0][x]) || 0;
return {
value: n('total_value'), units: n('total_units'),
absValue: n('abs_value'), absUnits: n('abs_units'),
exclValue: n('excl_value'), exclUnits: n('excl_units'),
};
}
// Resolve each measure independently into the increment the scale SQL expects.
// Value and units each accept exactly one of: an absolute target, a change
// amount, or a percentage — whichever the caller sent. They are resolved
// separately so a target on one measure and a percentage on the other can be
// submitted together. Everything is measured against the totals of *this*
// WHERE clause, which is what makes apply_mode 'each' land per slice.
async function resolveIncrs(client, ctx, whereClause, excludeClause, body) {
const num = (v) => (v === undefined || v === null || v === '') ? null : parseFloat(v);
const tValue = num(body.target_value);
const tUnits = num(body.target_units);
const tPrice = num(body.target_price);
const vIncr = num(body.value_incr);
const uIncr = num(body.units_incr);
let vPct = num(body.value_pct);
let uPct = num(body.units_pct);
// legacy shape: a single `pct` flag meaning "the increments are percentages"
if (body.pct) {
if (vPct === null && vIncr !== null) vPct = vIncr;
if (uPct === null && uIncr !== null) uPct = uIncr;
}
const legacyPct = !!body.pct;
const anyInput = [tValue, tUnits, tPrice, vIncr, uIncr, vPct, uPct].some(v => v !== null);
if (!anyInput) return { value: 0, units: 0 };
const totals = await sliceTotals(client, ctx, whereClause, excludeClause);
// What the number is measured against:
// 'adjustable' — only the rows this operation can write (the default, and
// what every earlier version of this API did)
// 'selected' — everything the pivot shows for the slice, excluded rows
// included. Those rows cannot move, so reaching the target
// means the adjustable rows absorb the whole difference.
const basis = body.target_basis === 'selected' ? 'selected' : 'adjustable';
const fixedValue = basis === 'selected' ? totals.exclValue : 0;
const fixedUnits = basis === 'selected' ? totals.exclUnits : 0;
// one measure: target wins, then percentage, then a plain change amount
const resolve = (target, pct, incr, current, fixed) => {
// subtract the immovable part: current + incr + fixed === target
if (target !== null) return (target - fixed) - current;
// a percentage of the basis, which may include the immovable part
if (pct !== null) return (current + fixed) * pct / 100;
if (incr !== null && !legacyPct) return incr;
return 0;
};
let value = resolve(tValue, vPct, vIncr, totals.value, fixedValue);
let units = resolve(tUnits, uPct, uIncr, totals.units, fixedUnits);
// A price target is the "edit price" mode of the Excel form: price and
// volume are the inputs and dollars fall out of them. With a units target
// alongside it, both move; without one, volume holds and price alone carries
// the change. An explicit value target outranks it either way.
if (tPrice !== null && tValue === null) {
const targetUnits = tUnits !== null
? (tUnits - fixedUnits) + 0 // the units target is already absolute
: (totals.units + fixedUnits);
value = (tPrice * targetUnits) - (totals.value + fixedValue);
}
// Which side of price x volume absorbs a dollar change.
//
// 'price' — volume holds, so price moves. This is what the API has always
// done, and stays the default so existing callers are unaffected.
// 'volume' — price holds, so volume scales with the dollars.
//
// Only meaningful when dollars were the input and units were not given
// explicitly; naming both means the caller has already decided.
const plug = body.plug === 'volume' ? 'volume' : 'price';
const unitsGiven = [tUnits, uIncr, uPct].some(v => v !== null);
if (plug === 'volume' && value !== 0 && !unitsGiven) {
const curValue = totals.value + fixedValue;
const curUnits = totals.units + fixedUnits;
if (curValue === 0) {
const err = new Error(
'Cannot hold price constant here: the selection currently has no value, ' +
'so there is no price to hold. Scale units directly, or let price absorb ' +
'the change.'
);
err.status = 400; throw err;
}
// price constant means value and units move by the same proportion:
// fVol = curVol * (fVal / curVal), so the units delta is curVol * value/curVal
units = curUnits * (value / curValue);
}
// the scale SQL divides by the slice total; with no rows there is
// nothing to prorate across and the increment would vanish anyway
if (totals.value === 0 && totals.units === 0) return { value: 0, units: 0 };
// Refuse to prorate across a pool that nets to ~zero. Each row's new value is
// (row / total) * increment, so as the net approaches zero the multiplier
// explodes and rows fly apart in opposite directions to hit the target — a
// mathematically faithful, practically useless result. Selecting slices that
// offset each other is the usual cause, and 'each' handles that correctly.
assertProratable(totals, value, units);
return { value: round(value, 6), units: round(units, 6) };
}
// a pool is proratable only if its net is a meaningful fraction of its gross
const NET_TO_GROSS_FLOOR = 0.01;
function assertProratable(totals, value, units) {
const check = (net, gross, incr, label) => {
if (!incr) return;
if (gross === 0) return;
if (Math.abs(net) >= gross * NET_TO_GROSS_FLOOR) return;
const err = new Error(
`Cannot prorate ${label} across this selection: the rows net to ` +
`${net.toFixed(2)} against a gross of ${gross.toFixed(2)}, so they very ` +
`nearly cancel out. Scaling to a target would push them to extreme ` +
`opposite values. Use "Each" to scale every slice on its own, or narrow ` +
`the selection so it does not mix offsetting rows.`
);
err.status = 400;
throw err;
};
check(totals.value, totals.absValue, value, 'value');
check(totals.units, totals.absUnits, units, 'units');
}
function round(n, dp) {
if (!isFinite(n)) return 0;
const f = Math.pow(10, dp);
return Math.round(n * f) / f;
} }
// fetch everything needed to execute an operation: // fetch everything needed to execute an operation:
@ -360,7 +36,7 @@ module.exports = function(pool) {
const unitsCol = colMeta.find(c => c.role === 'units')?.cname; const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
const sqlResult = await pool.query( const sqlResult = await pool.query(
`SELECT sql, generated_at FROM pf.sql WHERE source_id = $1 AND operation = $2`, `SELECT sql FROM pf.sql WHERE source_id = $1 AND operation = $2`,
[version.source_id, operation] [version.source_id, operation]
); );
if (sqlResult.rows.length === 0) { if (sqlResult.rows.length === 0) {
@ -377,26 +53,10 @@ module.exports = function(pool) {
filterCols: [...dimCols, ...dateCols], filterCols: [...dimCols, ...dateCols],
valueCol, valueCol,
unitsCol, unitsCol,
territoryCol: colMeta.find(c => c.is_territory)?.cname || null, sql: sqlResult.rows[0].sql
sql: sqlResult.rows[0].sql,
sqlGeneratedAt: sqlResult.rows[0].generated_at
}; };
} }
// Every read and every write goes through this, so a scoped account cannot
// reach a row outside its territory by any route. ANDed on last, after the
// slice and the client's own scope, where nothing in the request can undo
// it.
function territoryOf(req, ctx) {
return buildTerritoryClause(sessionTerritory(req), ctx.territoryCol);
}
function andTerritory(where, req, ctx) {
const t = territoryOf(req, ctx);
if (!t) return where;
return (!where || where === 'TRUE') ? t : `${where}\nAND ${t}`;
}
function guardOpen(version, res) { function guardOpen(version, res) {
if (version.status === 'closed') { if (version.status === 'closed') {
res.status(403).json({ error: 'Version is closed' }); res.status(403).json({ error: 'Version is closed' });
@ -419,19 +79,7 @@ module.exports = function(pool) {
} }
const tbl = fcTable(verResult.rows[0].tname, versionId); const tbl = fcTable(verResult.rows[0].tname, versionId);
// /data does not go through getContext, so it resolves the territory const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl}`);
// column itself. The count is scoped too, or the progress bar
// promises rows this account will never be sent.
const terrCol = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[verResult.rows[0].source_id]
)).rows[0]?.cname || null;
const territory = sessionTerritory(req);
const terrBare = buildTerritoryClause(territory, terrCol);
const terrAlias = buildTerritoryClause(territory, terrCol, 't');
const terrWhere = terrBare ? `WHERE ${terrBare}` : '';
const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl} ${terrWhere}`);
const rowCount = parseInt(count); const rowCount = parseInt(count);
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream'); res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
@ -443,14 +91,7 @@ module.exports = function(pool) {
await client.query('BEGIN'); await client.query('BEGIN');
await client.query(` await client.query(`
DECLARE pf_cur CURSOR FOR DECLARE pf_cur CURSOR FOR
SELECT t.* SELECT * FROM ${tbl}
,${SEGMENT_EXPR} AS pf_segment
,${BUCKET_EXPR} AS pf_bucket
,${NOTE_EXPR} AS pf_note
FROM ${tbl} t
LEFT JOIN pf.log l
ON l.id = t.pf_logid${VERSION_JOIN}
${terrAlias ? `WHERE ${terrAlias}` : ''}
`); `);
// Accumulate into column arrays (not row objects) to avoid allocating one JS // Accumulate into column arrays (not row objects) to avoid allocating one JS
@ -486,49 +127,10 @@ module.exports = function(pool) {
} }
}); });
// Aggregate a version to its display grain and return it as Arrow IPC.
// This replaces /data for sources that define a grain (col_meta.in_grain):
// the aggregation collapses the row count by orders of magnitude, so the
// result loads as one small native Perspective table indexed on pf_gkey and
// the WASM view still does all rollup/expand/collapse locally.
router.get('/versions/:id/agg', async (req, res) => {
try {
const ctx = await getContext(parseInt(req.params.id), 'get_agg');
// Before the GROUP BY, not after: the territory column need not be
// part of the grain, so an aggregated row may not carry it at all.
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
territory_clause:
buildTerritoryClause(sessionTerritory(req), ctx.territoryCol, 't') || 'TRUE',
});
const { rows } = await runSQL(sql);
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
res.setHeader('X-Row-Count', String(rows.length));
if (rows.length === 0) { res.end(); return; }
// column arrays, one Arrow record batch — same constraint as /data:
// per-batch dictionaries crash Perspective's Arrow reader
const colArrays = Object.fromEntries(Object.keys(rows[0]).map(k => [k, []]));
for (const row of rows) {
for (const k of Object.keys(colArrays)) colArrays[k].push(row[k]);
}
const buf = tableToIPC(tableFromArrays(colArrays), 'stream');
res.setHeader('Content-Length', String(buf.byteLength));
res.end(Buffer.from(buf.buffer, buf.byteOffset, buf.byteLength));
} catch (err) {
console.error(err);
if (!res.headersSent) res.status(err.status || 500).json({ error: err.message });
else res.destroy();
}
});
// load baseline rows from source table — additive, no delete // load baseline rows from source table — additive, no delete
router.post('/versions/:id/baseline', async (req, res) => { router.post('/versions/:id/baseline', async (req, res) => {
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body; const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const pf_user = sessionUser(req);
const dateOffset = date_offset || '0 days'; const dateOffset = date_offset || '0 days';
if (!await assertInterval(dateOffset, res)) return;
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE'; const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
try { try {
const ctx = await getContext(parseInt(req.params.id), 'baseline'); const ctx = await getContext(parseInt(req.params.id), 'baseline');
@ -543,16 +145,12 @@ module.exports = function(pool) {
version_id: ctx.version.id, version_id: ctx.version.id,
pf_user: esc(pf_user || ''), pf_user: esc(pf_user || ''),
note: esc(note || ''), note: esc(note || ''),
label: esc(label || ''),
bucket: esc(bucket || ''),
tag: esc(tag || ''),
params: esc(paramsJson), params: esc(paramsJson),
filter_clause: filterClause, filter_clause: filterClause,
date_offset: esc(dateOffset) date_offset: esc(dateOffset)
}); });
const result = await runSQL(sql); const result = await runSQL(sql);
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json(result.rows[0]); res.json(result.rows[0]);
} catch (err) { } catch (err) {
console.error(err); console.error(err);
@ -566,10 +164,8 @@ module.exports = function(pool) {
router.put('/versions/:id/baseline/:logid', async (req, res) => { router.put('/versions/:id/baseline/:logid', async (req, res) => {
const versionId = parseInt(req.params.id); const versionId = parseInt(req.params.id);
const logid = parseInt(req.params.logid); const logid = parseInt(req.params.logid);
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body; const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const pf_user = sessionUser(req);
const dateOffset = date_offset || '0 days'; const dateOffset = date_offset || '0 days';
if (!await assertInterval(dateOffset, res)) return;
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE'; const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
const client = await pool.connect(); const client = await pool.connect();
@ -605,21 +201,11 @@ module.exports = function(pool) {
date_offset: dateOffset, date_offset: dateOffset,
...(raw_where ? { raw_where } : (filters ? { filters } : {})) ...(raw_where ? { raw_where } : (filters ? { filters } : {}))
}); });
// This route deletes the log row and inserts a fresh one, so every
// annotation on it has to be handed back or it is lost. `??`, not `||`:
// an empty string is the form clearing a field on purpose, undefined is
// the form not carrying it at all -- the segment form has no tag input,
// so tag is always the latter and must survive an edit made for any
// other reason.
const keep = (sent, prior) => esc(sent ?? prior ?? '');
const sql = applyTokens(ctx.sql, { const sql = applyTokens(ctx.sql, {
fc_table: ctx.table, fc_table: ctx.table,
version_id: ctx.version.id, version_id: ctx.version.id,
pf_user: esc(pf_user || ''), pf_user: esc(pf_user || ''),
note: esc(note || ''), note: esc(note || ''),
label: keep(label, oldLog.label),
bucket: keep(bucket, oldLog.bucket),
tag: keep(tag, oldLog.tag),
params: esc(paramsJson), params: esc(paramsJson),
filter_clause: filterClause, filter_clause: filterClause,
date_offset: esc(dateOffset) date_offset: esc(dateOffset)
@ -633,7 +219,6 @@ module.exports = function(pool) {
await client.query(`DELETE FROM pf.log WHERE id = $1`, [logid]); await client.query(`DELETE FROM pf.log WHERE id = $1`, [logid]);
const insResult = await client.query(sql); const insResult = await client.query(sql);
await client.query('COMMIT'); await client.query('COMMIT');
await stampLogTotals(pool, ctx, insResult.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json({ res.json({
rows_deleted: delRows.rowCount, rows_deleted: delRows.rowCount,
@ -685,8 +270,7 @@ module.exports = function(pool) {
// load reference rows from source table (additive — does not clear prior reference rows) // load reference rows from source table (additive — does not clear prior reference rows)
router.post('/versions/:id/reference', async (req, res) => { router.post('/versions/:id/reference', async (req, res) => {
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body; const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
const pf_user = sessionUser(req);
const dateOffset = date_offset || '0 days'; const dateOffset = date_offset || '0 days';
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE'; const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
try { try {
@ -702,16 +286,12 @@ module.exports = function(pool) {
version_id: ctx.version.id, version_id: ctx.version.id,
pf_user: esc(pf_user || ''), pf_user: esc(pf_user || ''),
note: esc(note || ''), note: esc(note || ''),
label: esc(label || ''),
bucket: esc(bucket || ''),
tag: esc(tag || ''),
params: esc(paramsJson), params: esc(paramsJson),
filter_clause: filterClause, filter_clause: filterClause,
date_offset: esc(dateOffset) date_offset: esc(dateOffset)
}); });
const result = await runSQL(sql); const result = await runSQL(sql);
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
res.json(result.rows[0]); res.json(result.rows[0]);
} catch (err) { } catch (err) {
console.error(err); console.error(err);
@ -719,275 +299,131 @@ module.exports = function(pool) {
} }
}); });
// scale one or more slices — adjust value and/or units toward an absolute // scale a slice — adjust value and/or units by absolute amount or percentage
// target or by an increment. With several slices selected, apply_mode decides
// whether they are treated as one pool ('prorate') or independently ('each').
router.post('/versions/:id/scale', async (req, res) => { router.post('/versions/:id/scale', async (req, res) => {
const { note, apply_mode } = req.body; const { pf_user, note, slice, value_incr, units_incr, pct } = req.body;
const pf_user = sessionUser(req); if (!slice || Object.keys(slice).length === 0) {
const slices = normalizeSlices(req.body); return res.status(400).json({ error: 'slice is required' });
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' }); }
const applyMode = apply_mode === 'each' ? 'each' : 'prorate';
try { try {
const ctx = await getContext(parseInt(req.params.id), 'scale'); const ctx = await getContext(parseInt(req.params.id), 'scale');
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters); const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
// 'prorate' pools every slice into one WHERE and lets the SQL's let absValueIncr = value_incr || 0;
// sum() OVER () distribute the increment across the whole pool. let absUnitsIncr = units_incr || 0;
// 'each' runs the same statement once per slice, so every slice
// reaches the target on its own and gets its own log entry.
const units = sliceUnits(slices, ctx, applyMode, req.body.scope, req);
const client = await pool.connect(); // pct mode: run a quick totals query, convert percentages to absolutes
let committed = false; if (pct && (value_incr || units_incr)) {
try { const totals = await pool.query(`
await client.query('BEGIN'); SELECT
const allRows = []; sum("${ctx.valueCol}") AS total_value,
// the statement that produced each entry, for pf.log.sql_text sum("${ctx.unitsCol}") AS total_units
const sqlByLogId = new Map(); FROM ${ctx.table}
let applied = 0; WHERE ${whereClause}
const skipped = []; ${excludeClause}
`);
for (const unit of units) { const { total_value, total_units } = totals.rows[0];
const incr = await resolveIncrs(client, ctx, unit.where, excludeClause, req.body); if (value_incr) absValueIncr = (parseFloat(total_value) || 0) * value_incr / 100;
// no rows, or already at the target — nothing to write for this slice if (units_incr) absUnitsIncr = (parseFloat(total_units) || 0) * units_incr / 100;
if (incr.value === 0 && incr.units === 0) { skipped.push(...unit.slices); continue; }
applied++;
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({
slices: unit.slices,
apply_mode: applyMode,
...pickIntent(req.body),
resolved: { value_incr: incr.value, units_incr: incr.units }
})),
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
value_incr: incr.value,
units_incr: incr.units
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
if (allRows.length === 0) {
await client.query('ROLLBACK');
return res.status(400).json({
error: 'Nothing to scale — the target matches the current total, or the increment is zero'
});
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'scale' }));
res.json({
rows,
rows_affected: rows.length,
slices_applied: applied,
...(skipped.length ? { slices_skipped: skipped } : {})
});
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
} }
if (absValueIncr === 0 && absUnitsIncr === 0) {
return res.status(400).json({ error: 'value_incr and/or units_incr must be non-zero' });
}
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slice, value_incr, units_incr, pct })),
slice: esc(JSON.stringify(slice)),
where_clause: whereClause,
exclude_clause: excludeClause,
value_incr: absValueIncr,
units_incr: absUnitsIncr
});
const result = await runSQL(sql);
const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'scale' }));
res.json({ rows, rows_affected: rows.length });
} catch (err) { } catch (err) {
console.error(err); console.error(err);
res.status(err.status || 500).json({ error: err.message }); res.status(err.status || 500).json({ error: err.message });
} }
}); });
// recode dimension values on one or more slices // recode dimension values on a slice
// inserts negative rows to zero out the original, positive rows with new dimension values // inserts negative rows to zero out the original, positive rows with new dimension values
router.post('/versions/:id/recode', async (req, res) => { router.post('/versions/:id/recode', async (req, res) => {
const { note, set, apply_mode } = req.body; const { pf_user, note, slice, set } = req.body;
const pf_user = sessionUser(req); if (!slice || Object.keys(slice).length === 0) return res.status(400).json({ error: 'slice is required' });
const slices = normalizeSlices(req.body); if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
try { try {
const ctx = await getContext(parseInt(req.params.id), 'recode'); const ctx = await getContext(parseInt(req.params.id), 'recode');
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
if (!assertMayRecodeTerritory(req, ctx, set, res)) return;
const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters); const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set); const setClause = buildSetClause(ctx.dimCols, set);
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
const client = await pool.connect(); const sql = applyTokens(ctx.sql, {
let committed = false; fc_table: ctx.table,
try { version_id: ctx.version.id,
await client.query('BEGIN'); pf_user: esc(pf_user || ''),
const allRows = []; note: esc(note || ''),
// the statement that produced each entry, for pf.log.sql_text params: esc(JSON.stringify({ slice, set })),
const sqlByLogId = new Map(); slice: esc(JSON.stringify(slice)),
for (const unit of units) { where_clause: whereClause,
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices; exclude_clause: excludeClause,
const sql = applyTokens(ctx.sql, { set_clause: setClause
fc_table: ctx.table, });
version_id: ctx.version.id,
pf_user: esc(pf_user || ''), const result = await runSQL(sql);
note: esc(note || ''), const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'recode' }));
params: esc(JSON.stringify({ slices: unit.slices, set, apply_mode: unit.mode })), res.json({ rows, rows_affected: rows.length });
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
set_clause: setClause
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'recode' }));
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
}
} catch (err) { } catch (err) {
console.error(err); console.error(err);
res.status(err.status || 500).json({ error: err.message }); res.status(err.status || 500).json({ error: err.message });
} }
}); });
// clone one or more slices as new business under new dimension values // clone a slice as new business under new dimension values
// does not offset the original slice // does not offset the original slice
router.post('/versions/:id/clone', async (req, res) => { router.post('/versions/:id/clone', async (req, res) => {
const { note, set, scale, apply_mode, from_logid, date_offset } = req.body; const { pf_user, note, slice, set, scale } = req.body;
const pf_user = sessionUser(req); if (!slice || Object.keys(slice).length === 0) return res.status(400).json({ error: 'slice is required' });
const slices = normalizeSlices(req.body); if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
try { try {
const ctx = await getContext(parseInt(req.params.id), 'clone'); const ctx = await getContext(parseInt(req.params.id), 'clone');
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
assertSelective(slices, ctx);
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0; const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
const dateOffset = (date_offset || '0 days').trim() || '0 days'; const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set);
if (!await assertInterval(dateOffset, res)) return; const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({ slice, set, scale: scaleFactor })),
slice: esc(JSON.stringify(slice)),
where_clause: whereClause,
exclude_clause: excludeClause,
set_clause: setClause,
scale_factor: scaleFactor
});
// exclude_iters deliberately does not apply here. It exists to stop const result = await runSQL(sql);
// operations *modifying* reference rows: scale would attribute forecast const rows = result.rows.map(r => ({ ...r, pf_note: note || null, pf_op: 'clone' }));
// movement to prior-year rows by distributing across them, and recode res.json({ rows, rows_affected: rows.length });
// writes negative rows that zero the original out. Clone does neither --
// it reads rows and writes new pf_iter = 'clone' rows, leaving the source
// untouched. Copying a plan or a prior year out of reference and into
// adjustments is the operation working as intended, and excluding them
// meant a visible, deliberate selection silently produced nothing.
//
// from_logid narrows instead: a selection spanning AOP and Prior Year
// where only one is wanted.
let excludeClause = '';
if (from_logid != null) {
const srcLog = await pool.query(
`SELECT id FROM pf.log WHERE id = $1 AND version_id = $2`,
[parseInt(from_logid), ctx.version.id]
);
if (!srcLog.rows.length) {
return res.status(400).json({ error: `No log entry ${from_logid} on this version` });
}
excludeClause = `AND pf_logid = ${parseInt(from_logid)}`;
}
// Period dimensions come from the calendar against the shifted date, not
// from the row being copied -- otherwise a mix moved forward a year
// keeps last year's period labels. An explicit set wins over both.
const dateGroups = dateGroupsOf(ctx.colMeta);
const derivedExprs = Object.fromEntries(
[...dimPeriodMapOf(dateGroups)].map(([cname, { alias, periodCol }]) =>
[cname, `${alias}."${periodCol}"`])
);
const setClause = buildSetClause(ctx.dimCols, set, { derivedExprs, alias: 's' });
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
const client = await pool.connect();
let committed = false;
try {
await client.query('BEGIN');
const allRows = [];
// the statement that produced each entry, for pf.log.sql_text
const sqlByLogId = new Map();
for (const unit of units) {
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
const sql = applyTokens(ctx.sql, {
fc_table: ctx.table,
version_id: ctx.version.id,
pf_user: esc(pf_user || ''),
note: esc(note || ''),
params: esc(JSON.stringify({
slices: unit.slices, set, scale: scaleFactor, apply_mode: unit.mode,
date_offset: dateOffset,
...(from_logid != null ? { from_logid: parseInt(from_logid) } : {}),
})),
slice: esc(JSON.stringify(loggedSlice)),
where_clause: unit.where,
exclude_clause: excludeClause,
set_clause: setClause,
scale_factor: scaleFactor,
date_offset: esc(dateOffset)
});
const result = await runSQL(sql, client);
await tagLog(client, result.rows, req.body.tag);
for (const r of result.rows) {
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
}
allRows.push(...result.rows);
}
await client.query('COMMIT');
committed = true;
// one log id per unit: apply_mode 'each' writes an entry per slice
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
await stampLogTotals(pool, ctx, id, {
sql: sqlByLogId.get(id) || null,
territory: sessionTerritory(req),
});
}
const opLabel = (req.body.tag || '').trim() || note || null;
const rows = allRows.map(r => ({ ...r, pf_segment: ADJUSTMENT_SEGMENT, pf_bucket: ADJUSTMENT_BUCKET, pf_note: opLabel, pf_op: 'clone' }));
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
} finally {
if (!committed) try { await client.query('ROLLBACK'); } catch {}
client.release();
}
} catch (err) { } catch (err) {
console.error(err); console.error(err);
res.status(err.status || 500).json({ error: err.message }); res.status(err.status || 500).json({ error: err.message });

View File

@ -1,8 +1,5 @@
const express = require('express'); const express = require('express');
const { generateSQL, buildTerritoryClause } = require('../lib/sql_generator'); const { generateSQL } = require('../lib/sql_generator');
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
const { sessionTerritory } = require('../lib/auth');
const { sessionUser } = require('../lib/auth');
module.exports = function(pool) { module.exports = function(pool) {
const router = express.Router(); const router = express.Router();
@ -23,8 +20,7 @@ module.exports = function(pool) {
// register a source table // register a source table
// auto-populates col_meta from information_schema with role='ignore' // auto-populates col_meta from information_schema with role='ignore'
router.post('/sources', async (req, res) => { router.post('/sources', async (req, res) => {
const { schema, tname, label } = req.body; const { schema, tname, label, created_by } = req.body;
const created_by = sessionUser(req);
if (!schema || !tname) { if (!schema || !tname) {
return res.status(400).json({ error: 'schema and tname are required' }); return res.status(400).json({ error: 'schema and tname are required' });
} }
@ -43,14 +39,15 @@ module.exports = function(pool) {
); );
const source = src.rows[0]; const source = src.rows[0];
// seed col_meta from the source's real columns // seed col_meta from information_schema
await client.query(` await client.query(`
INSERT INTO pf.col_meta (source_id, cname, role, opos) INSERT INTO pf.col_meta (source_id, cname, role, opos)
SELECT $3, column_name, 'dimension', ordinal_position SELECT $1, column_name, 'dimension', ordinal_position
FROM (${RELATION_COLUMNS_SQL}) c FROM information_schema.columns
WHERE table_schema = $2 AND table_name = $3
ORDER BY ordinal_position ORDER BY ordinal_position
ON CONFLICT (source_id, cname) DO NOTHING ON CONFLICT (source_id, cname) DO NOTHING
`, [schema, tname, source.id]); `, [source.id, schema, tname]);
await client.query('COMMIT'); await client.query('COMMIT');
res.status(201).json(source); res.status(201).json(source);
@ -87,31 +84,19 @@ module.exports = function(pool) {
if (!Array.isArray(cols)) { if (!Array.isArray(cols)) {
return res.status(400).json({ error: 'body must be an array' }); return res.status(400).json({ error: 'body must be an array' });
} }
// Exactly one per source: the scope is a single IN list against a single
// column, and two flagged would silently mean whichever one a .find()
// reached first -- the trap is_key already fell into (see CLAUDE.md).
const territoryCols = cols.filter(c => c.is_territory).map(c => c.cname);
if (territoryCols.length > 1) {
return res.status(400).json({
error: `Only one column can be the territory. Flagged: ${territoryCols.join(', ')}`
});
}
const client = await pool.connect(); const client = await pool.connect();
try { try {
await client.query('BEGIN'); await client.query('BEGIN');
for (const col of cols) { for (const col of cols) {
await client.query(` await client.query(`
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, in_grain, is_territory, opos) INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, opos)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (source_id, cname) DO UPDATE SET ON CONFLICT (source_id, cname) DO UPDATE SET
label = EXCLUDED.label, label = EXCLUDED.label,
role = EXCLUDED.role, role = EXCLUDED.role,
is_key = EXCLUDED.is_key, is_key = EXCLUDED.is_key,
dim_group = EXCLUDED.dim_group, dim_group = EXCLUDED.dim_group,
dim_period_col = EXCLUDED.dim_period_col, dim_period_col = EXCLUDED.dim_period_col,
in_grain = EXCLUDED.in_grain,
is_territory = EXCLUDED.is_territory,
opos = EXCLUDED.opos opos = EXCLUDED.opos
`, [ `, [
sourceId, sourceId,
@ -121,8 +106,6 @@ module.exports = function(pool) {
col.is_key || false, col.is_key || false,
col.dim_group || null, col.dim_group || null,
col.dim_period_col || null, col.dim_period_col || null,
col.in_grain || false,
col.is_territory || false,
col.opos || null col.opos || null
]); ]);
} }
@ -183,13 +166,6 @@ module.exports = function(pool) {
generated_at = EXCLUDED.generated_at generated_at = EXCLUDED.generated_at
`, [sourceId, operation, sql]); `, [sourceId, operation, sql]);
} }
// drop operations this generation no longer produces — e.g. get_agg
// after the grain has been cleared, which would otherwise leave a
// stale template the load path would still pick up
await client.query(
`DELETE FROM pf.sql WHERE source_id = $1 AND operation <> ALL($2::text[])`,
[sourceId, Object.keys(sqls)]
);
await client.query('COMMIT'); await client.query('COMMIT');
} catch (err) { } catch (err) {
await client.query('ROLLBACK'); await client.query('ROLLBACK');
@ -238,38 +214,10 @@ module.exports = function(pool) {
return res.status(400).json({ error: `"${col}" is not a key column` }); return res.status(400).json({ error: `"${col}" is not a key column` });
} }
// ?q= narrows, ?limit= caps. A key column can be very wide -- part on
// osm_skinny has 11,290 distinct values -- so returning the lot to fill a
// completion list is both a slow query and a large response for a control
// that can only usefully show a handful.
const { schema, tname } = srcResult.rows[0]; const { schema, tname } = srcResult.rows[0];
const q = (req.query.q || '').trim();
const limit = Math.min(parseInt(req.query.limit) || 5000, 5000);
const params = [];
let filter = `WHERE "${col}" IS NOT NULL`;
// Completion reads the *source* table, which no territory scope has
// touched -- so without this a scoped account could enumerate every
// customer, part and rep in the business from a dropdown, having
// been shown none of their rows.
const terrRow = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[req.params.id]
)).rows[0];
const terrClause = buildTerritoryClause(sessionTerritory(req), terrRow?.cname || null);
if (terrClause) filter += ` AND ${terrClause}`;
if (q) {
params.push(`%${q}%`);
filter += ` AND "${col}"::text ILIKE $${params.length}`;
}
params.push(limit);
const result = await pool.query( const result = await pool.query(
`SELECT DISTINCT "${col}"::text AS val FROM ${schema}.${tname} `SELECT DISTINCT "${col}" AS val FROM ${schema}.${tname}
${filter} ORDER BY 1 LIMIT $${params.length}`, WHERE "${col}" IS NOT NULL ORDER BY "${col}"`
params
); );
res.json(result.rows.map(r => r.val)); res.json(result.rows.map(r => r.val));
} catch (err) { } catch (err) {
@ -278,142 +226,6 @@ module.exports = function(pool) {
} }
}); });
// Resolve a dim_group to its key column and siblings, or explain why it cannot be.
async function resolveGroup(sourceId, group) {
const { rows: meta } = await pool.query(
`SELECT * FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`, [sourceId]);
const members = meta.filter(c => c.dim_group === group);
if (!members.length) {
const err = new Error(`No columns are grouped as "${group}" on this source`);
err.status = 404; throw err;
}
const keyCol = members.find(c => c.is_key);
if (!keyCol) {
const err = new Error(
`Group "${group}" has no is_key column, so its members have nothing to be keyed on`);
err.status = 400; throw err;
}
return {
keyCol,
siblings: members.filter(c => c.cname !== keyCol.cname),
// recency column: the source's primary date, the same one the generator
// treats as the date for loads
dateCol: meta.find(c => c.role === 'date')?.cname || null,
};
}
// The member list for a group, as one array. Small enough to send whole --
// 11,290 parts on osm_skinny -- so the client holds it and filters locally
// instead of querying per keystroke.
router.get('/sources/:id/dim/:group', async (req, res) => {
try {
const sourceId = parseInt(req.params.id);
const { keyCol, siblings } = await resolveGroup(sourceId, req.params.group);
const includeInactive = req.query.all === '1';
const { rows } = await pool.query(`
SELECT key_value, attrs, is_active, source_seen
FROM pf.dim_member
WHERE source_id = $1 AND dim_group = $2
${includeInactive ? '' : 'AND is_active'}
ORDER BY key_value
`, [sourceId, req.params.group]);
res.json({
group: req.params.group,
key_col: keyCol.cname,
siblings: siblings.map(c => c.cname),
members: rows,
});
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// Rebuild a group's members from the source. A merge, not a replace: curation
// (a member deactivated by hand, or added before it ever sold) has to survive a
// refresh, so absent members are marked source_seen = false rather than deleted.
//
// Slow by nature -- it reads the whole source, which for a view over a
// transaction table is millions of rows -- so it is a deliberate action rather
// than something that happens on a page load.
router.post('/sources/:id/dim/:group/refresh', async (req, res) => {
const sourceId = parseInt(req.params.id);
const group = req.params.group;
try {
const srcResult = await pool.query(
`SELECT schema, tname FROM pf.source WHERE id = $1`, [sourceId]);
if (!srcResult.rows.length) return res.status(404).json({ error: 'Source not found' });
const { schema, tname } = srcResult.rows[0];
const { keyCol, siblings, dateCol } = await resolveGroup(sourceId, group);
if (!siblings.length) {
return res.status(400).json({ error: `Group "${group}" has no sibling columns to store` });
}
const q = (n) => `"${n}"`;
const attrs = siblings.map(c => `'${c.cname}', s.${q(c.cname)}::text`).join(', ');
// A key can carry more than one attribute set across history -- 11,290
// parts against 13,662 combinations on osm_skinny. Take the most recent
// by the source's date column, which is the live definition.
const recency = dateCol ? `s.${q(dateCol)} DESC NULLS LAST` : `1`;
const started = Date.now();
// An explicit stamp rather than now(): inside a transaction now() is the
// transaction's start time, so "refreshed in this run" and "refreshed in
// a run that began at the same instant" would be indistinguishable.
const runAt = new Date();
const client = await pool.connect();
try {
await client.query('BEGIN');
const { rows: [{ n }] } = await client.query(`
WITH ranked AS (
SELECT s.${q(keyCol.cname)}::text AS key_value,
jsonb_build_object(${attrs}) AS attrs,
row_number() OVER (
PARTITION BY s.${q(keyCol.cname)} ORDER BY ${recency}
) AS rn
FROM ${q(schema)}.${q(tname)} s
WHERE s.${q(keyCol.cname)} IS NOT NULL
)
,upserted AS (
INSERT INTO pf.dim_member
(source_id, dim_group, key_value, attrs, refreshed_at, source_seen)
SELECT $1, $2, key_value, attrs, $3, true
FROM ranked WHERE rn = 1
ON CONFLICT (source_id, dim_group, key_value) DO UPDATE SET
attrs = EXCLUDED.attrs,
source_seen = true,
refreshed_at = $3,
updated_at = now()
RETURNING 1
)
SELECT count(*)::int AS n FROM upserted
`, [sourceId, group, runAt]);
// anything this run did not touch is no longer in the source
const { rowCount: dropped } = await client.query(`
UPDATE pf.dim_member
SET source_seen = false, updated_at = now()
WHERE source_id = $1 AND dim_group = $2 AND source_seen
AND refreshed_at IS DISTINCT FROM $3
`, [sourceId, group, runAt]);
await client.query('COMMIT');
res.json({ group, members: n, no_longer_in_source: dropped, ms: Date.now() - started });
} catch (err) {
await client.query('ROLLBACK');
throw err;
} finally {
client.release();
}
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
// given a key column value, look up sibling dim_group column values from source // given a key column value, look up sibling dim_group column values from source
// returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous // returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous
router.get('/sources/:id/lookup', async (req, res) => { router.get('/sources/:id/lookup', async (req, res) => {
@ -449,12 +261,23 @@ module.exports = function(pool) {
} }
}); });
// PUT /sources/:id/default-layout is gone. It wrote pf.source.default_layout, // set or clear the default Perspective layout for a source.
// one anonymous blob per source that any account could overwrite for every // Body: a Perspective view config (group_by, split_by, columns, plugin_config, …).
// other account -- a published layout with no owner. Its successor is // Pass null or {} to clear.
// pf.layout: named, owned, and writable only by its owner or an admin. The router.put('/sources/:id/default-layout', async (req, res) => {
// old column is left in place, already migrated into pf.layout by try {
// setup_sql/01_schema.sql, and read by nothing. const layout = req.body && Object.keys(req.body).length > 0 ? req.body : null;
const result = await pool.query(
`UPDATE pf.source SET default_layout = $1 WHERE id = $2 RETURNING *`,
[layout, req.params.id]
);
if (result.rows.length === 0) return res.status(404).json({ error: 'Source not found' });
res.json(result.rows[0]);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// deregister a source — does not drop existing forecast tables // deregister a source — does not drop existing forecast tables
router.get('/dim-period/cols', async (req, res) => { router.get('/dim-period/cols', async (req, res) => {

View File

@ -1,5 +1,4 @@
const express = require('express'); const express = require('express');
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
module.exports = function(pool) { module.exports = function(pool) {
const router = express.Router(); const router = express.Router();
@ -8,18 +7,15 @@ module.exports = function(pool) {
router.get('/tables', async (req, res) => { router.get('/tables', async (req, res) => {
try { try {
const result = await pool.query(` const result = await pool.query(`
-- pg_class, not information_schema.tables, which omits
-- materialized views: gs.osm_skinny is one, and the browser
-- could not offer what the app is already built on.
SELECT SELECT
n.nspname AS schema, t.table_schema AS schema,
c.relname AS tname, t.table_name AS tname,
c.reltuples::bigint AS row_estimate c.reltuples::bigint AS row_estimate
FROM pg_class c FROM information_schema.tables t
JOIN pg_namespace n ON n.oid = c.relnamespace LEFT JOIN pg_namespace n ON n.nspname = t.table_schema
WHERE c.relkind IN ('r', 'v', 'm', 'f', 'p') LEFT JOIN pg_class c ON c.relname = t.table_name AND c.relnamespace = n.oid
AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pf') WHERE t.table_schema NOT IN ('pg_catalog', 'information_schema', 'pf')
ORDER BY n.nspname, c.relname ORDER BY t.table_schema, t.table_name
`); `);
res.json(result.rows); res.json(result.rows);
} catch (err) { } catch (err) {
@ -35,8 +31,12 @@ module.exports = function(pool) {
return res.status(400).json({ error: 'Invalid schema or table name' }); return res.status(400).json({ error: 'Invalid schema or table name' });
} }
try { try {
const cols = await pool.query( const cols = await pool.query(`
`${RELATION_COLUMNS_SQL} ORDER BY ordinal_position`, [schema, tname]); SELECT column_name, data_type, is_nullable, ordinal_position
FROM information_schema.columns
WHERE table_schema = $1 AND table_name = $2
ORDER BY ordinal_position
`, [schema, tname]);
const rows = await pool.query( const rows = await pool.query(
`SELECT * FROM ${schema}.${tname} LIMIT 5` `SELECT * FROM ${schema}.${tname} LIMIT 5`

View File

@ -1,7 +1,5 @@
const express = require('express'); const express = require('express');
const { fcTable, mapType, RELATION_COLUMNS_SQL } = require('../lib/utils'); const { fcTable, mapType } = require('../lib/utils');
const { sessionUser, sessionTerritory } = require('../lib/auth');
const { buildTerritoryClause } = require('../lib/sql_generator');
module.exports = function(pool) { module.exports = function(pool) {
const router = express.Router(); const router = express.Router();
@ -24,8 +22,7 @@ module.exports = function(pool) {
// inserts version row, then CREATE TABLE pf.fc_{tname}_{version_id} in one transaction // inserts version row, then CREATE TABLE pf.fc_{tname}_{version_id} in one transaction
router.post('/sources/:id/versions', async (req, res) => { router.post('/sources/:id/versions', async (req, res) => {
const sourceId = parseInt(req.params.id); const sourceId = parseInt(req.params.id);
const { name, description, exclude_iters } = req.body; const { name, description, created_by, exclude_iters } = req.body;
const created_by = sessionUser(req);
if (!name) return res.status(400).json({ error: 'name is required' }); if (!name) return res.status(400).json({ error: 'name is required' });
const client = await pool.connect(); const client = await pool.connect();
@ -39,9 +36,8 @@ module.exports = function(pool) {
} }
const source = srcResult.rows[0]; const source = srcResult.rows[0];
// col_meta joined to the source's real columns, for the data types // fetch col_meta joined to information_schema for data types
const colResult = await client.query(` const colResult = await client.query(`
WITH cols AS (${RELATION_COLUMNS_SQL})
SELECT SELECT
m.cname, m.cname,
m.role, m.role,
@ -50,11 +46,14 @@ module.exports = function(pool) {
i.numeric_precision, i.numeric_precision,
i.numeric_scale i.numeric_scale
FROM pf.col_meta m FROM pf.col_meta m
JOIN cols i ON i.column_name = m.cname JOIN information_schema.columns i
WHERE m.source_id = $3 ON i.table_schema = $2
AND i.table_name = $3
AND i.column_name = m.cname
WHERE m.source_id = $1
AND m.role NOT IN ('ignore') AND m.role NOT IN ('ignore')
ORDER BY m.opos ORDER BY m.opos
`, [source.schema, source.tname, sourceId]); `, [sourceId, source.schema, source.tname]);
if (colResult.rows.length === 0) { if (colResult.rows.length === 0) {
return res.status(400).json({ return res.status(400).json({
@ -101,14 +100,6 @@ ${colDefs},
`; `;
await client.query(ddl); await client.query(ddl);
// pf_logid is how every entry-level operation finds its rows: undo
// deletes by it, the change log aggregates by it, and it is part of the
// grain key. Without an index each of those is a sequential scan of the
// whole forecast table -- 2.5M rows to total two adjustments.
await client.query(
`CREATE INDEX ${table.split('.').pop()}_logid_idx ON ${table} (pf_logid)`
);
await client.query('COMMIT'); await client.query('COMMIT');
res.status(201).json({ ...version, fc_table: table }); res.status(201).json({ ...version, fc_table: table });
} catch (err) { } catch (err) {
@ -123,235 +114,22 @@ ${colDefs},
} }
}); });
// where this version's writes actually land: the physical forecast table, // update version name, description, or exclude_iters
// its current row count, and the source table rows are read from.
// Surfaced in the status bar so the write target is never a mystery.
// Distinct values of a dimension as they appear in one version's forecast
// table, for completing recode and clone.
//
// Deliberately not the source: the source view reaches back over all of
// history, so completing from it offers parts discontinued years ago. The
// version holds what was actually loaded, which is what a forecast is being
// written against.
//
// Held in memory because the scan is not cheap -- 2.0s for 11,290 parts across
// 2.5M rows on fc_osm_skinny_29 -- and completion is typed into. Keyed on the
// version's latest log id, so any load, adjustment or undo rebuilds it on the
// next request without anything having to remember to invalidate.
const valueCache = new Map();
router.get('/versions/:id/values/:col', async (req, res) => {
const versionId = parseInt(req.params.id);
const col = req.params.col;
try {
const verResult = await pool.query(`
SELECT v.id, s.tname, s.id AS source_id
FROM pf.version v JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [versionId]);
if (!verResult.rows.length) return res.status(404).json({ error: 'Version not found' });
const { tname, source_id } = verResult.rows[0];
// the column name is interpolated, so it has to be one col_meta names
const okCol = await pool.query(`
SELECT 1 FROM pf.col_meta
WHERE source_id = $1 AND cname = $2 AND role IN ('dimension', 'date')
`, [source_id, col]);
if (!okCol.rows.length) {
return res.status(400).json({ error: `"${col}" is not a dimension on this source` });
}
const table = fcTable(tname, versionId);
const { rows: [{ rev }] } = await pool.query(
`SELECT coalesce(max(id), 0)::text AS rev FROM pf.log WHERE version_id = $1`,
[versionId]
);
const key = `${versionId}:${col}`;
let entry = valueCache.get(key);
if (!entry || entry.rev !== rev) {
const { rows } = await pool.query(
`SELECT DISTINCT "${col}"::text AS val FROM ${table}
WHERE "${col}" IS NOT NULL ORDER BY 1`
);
entry = { rev, values: rows.map(r => r.val) };
valueCache.set(key, entry);
}
const q = (req.query.q || '').trim().toLowerCase();
const limit = Math.min(parseInt(req.query.limit) || 50, 500);
const picked = q
? entry.values.filter(v => v.toLowerCase().includes(q))
: entry.values;
res.json(picked.slice(0, limit));
} catch (err) {
console.error(err);
res.status(err.status || 500).json({ error: err.message });
}
});
router.get('/versions/:id/table-info', async (req, res) => {
try {
const verResult = await pool.query(`
SELECT v.id, v.name, v.status, s.schema, s.tname, s.id AS source_id
FROM pf.version v
JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [req.params.id]);
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
const v = verResult.rows[0];
const fc = fcTable(v.tname, v.id);
const [schema, table] = fc.split('.');
const existsResult = await pool.query(
`SELECT to_regclass($1) IS NOT NULL AS exists`, [fc]
);
const exists = existsResult.rows[0].exists;
// Scoped like every other read. Unscoped it reported the whole
// table to an account that can see a fraction of it -- the status
// bar's row count, and the figure the load progress promises, both
// came from here: a rep with 330k rows was told the load was
// fetching 2.8M.
const terrCol = (await pool.query(
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
[v.source_id]
)).rows[0]?.cname || null;
const terr = buildTerritoryClause(sessionTerritory(req), terrCol);
let rows = null, byIter = [];
if (exists) {
const countResult = await pool.query(
`SELECT pf_iter, count(*)::int AS n FROM ${fc}
${terr ? `WHERE ${terr}` : ''}
GROUP BY pf_iter ORDER BY pf_iter`
);
byIter = countResult.rows;
rows = byIter.reduce((a, r) => a + r.n, 0);
}
res.json({
version_id: v.id,
version_name: v.name,
status: v.status,
source: `${v.schema}.${v.tname}`,
fc_table: fc,
fc_schema: schema,
fc_tname: table,
exists,
rows,
by_iter: byIter
});
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// Tags already used on this source, newest first — feeds the tag autocomplete.
// Scoped to the source rather than the version so an initiative name carries
// across versions, which is the point of naming it.
router.get('/sources/:id/tags', async (req, res) => {
try {
const result = await pool.query(`
SELECT l.tag,
count(*)::int AS uses,
max(l.stamp) AS last_used
FROM pf.log l
JOIN pf.version v ON v.id = l.version_id
WHERE v.source_id = $1 AND l.tag IS NOT NULL AND l.tag <> ''
GROUP BY l.tag
ORDER BY max(l.stamp) DESC
`, [req.params.id]);
res.json(result.rows);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// Bridge: how this version got from its baseline to where it stands, grouped by
// initiative. Amounts come from the version's own forecast table, so the figures
// reconcile with the pivot rather than being recomputed from the log's params.
router.get('/versions/:id/bridge', async (req, res) => {
try {
const verResult = await pool.query(`
SELECT v.id, v.exclude_iters, s.schema, s.tname, s.id AS source_id
FROM pf.version v JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [req.params.id]);
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
const v = verResult.rows[0];
const fc = fcTable(v.tname, v.id);
const exists = await pool.query(`SELECT to_regclass($1) IS NOT NULL AS ok`, [fc]);
if (!exists.rows[0].ok) return res.json({ fc_table: fc, exists: false, rows: [] });
const colResult = await pool.query(
`SELECT cname, role FROM pf.col_meta WHERE source_id = $1`, [v.source_id]);
const valueCol = colResult.rows.find(c => c.role === 'value')?.cname;
const unitsCol = colResult.rows.find(c => c.role === 'units')?.cname;
if (!valueCol) return res.status(400).json({ error: 'No value column configured' });
const excl = (v.exclude_iters || []).length
? `t.pf_iter NOT IN (${v.exclude_iters.map(i => `'${String(i).replace(/'/g, "''")}'`).join(', ')})`
: 'TRUE';
const result = await pool.query(`
SELECT CASE WHEN t.pf_iter = 'baseline' THEN '(baseline)'
ELSE coalesce(nullif(l.tag, ''), '(untagged)') END AS tag,
bool_or(t.pf_iter = 'baseline') AS is_baseline,
count(DISTINCT l.id)::int AS entries,
count(*)::int AS row_count,
round(sum(t."${valueCol}")::numeric, 2) AS value
${unitsCol ? `, round(sum(t."${unitsCol}")::numeric, 2) AS units` : ''}
FROM ${fc} t
LEFT JOIN pf.log l ON l.id = t.pf_logid
WHERE ${excl}
GROUP BY 1
ORDER BY bool_or(t.pf_iter = 'baseline') DESC, min(l.id)
`);
res.json({ fc_table: fc, exists: true, value_col: valueCol, units_col: unitsCol, rows: result.rows });
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// update version name, description, exclude_iters, or the fallback display
// names.
//
// bucket_order is deliberately not settable: the bucket column order is the
// text in pf.log.bucket now, so a stored order would be a second answer to
// the same question, and a silent one -- nothing reads it.
//
// The three name columns are flag-and-value pairs rather than COALESCE:
// clearing one back to the built-in means writing null, which COALESCE on
// the value alone cannot tell from "not mentioned".
router.put('/versions/:id', async (req, res) => { router.put('/versions/:id', async (req, res) => {
const { name, description, exclude_iters, const { name, description, exclude_iters } = req.body;
adjustment_segment, adjustment_bucket, unlabeled_load } = req.body;
const set = (v) => (v === undefined ? null : (String(v).trim() || null));
try { try {
const result = await pool.query(` const result = await pool.query(`
UPDATE pf.version SET UPDATE pf.version SET
name = COALESCE($2, name), name = COALESCE($2, name),
description = COALESCE($3, description), description = COALESCE($3, description),
exclude_iters = COALESCE($4, exclude_iters), exclude_iters = COALESCE($4, exclude_iters)
adjustment_segment = CASE WHEN $5::bool THEN $6::text ELSE adjustment_segment END,
adjustment_bucket = CASE WHEN $7::bool THEN $8::text ELSE adjustment_bucket END,
unlabeled_load = CASE WHEN $9::bool THEN $10::text ELSE unlabeled_load END
WHERE id = $1 WHERE id = $1
RETURNING * RETURNING *
`, [ `, [
req.params.id, req.params.id,
name || null, name || null,
description || null, description || null,
exclude_iters ? JSON.stringify(exclude_iters) : null, exclude_iters ? JSON.stringify(exclude_iters) : null
adjustment_segment !== undefined, set(adjustment_segment),
adjustment_bucket !== undefined, set(adjustment_bucket),
unlabeled_load !== undefined, set(unlabeled_load)
]); ]);
if (result.rows.length === 0) { if (result.rows.length === 0) {
return res.status(404).json({ error: 'Version not found' }); return res.status(404).json({ error: 'Version not found' });
@ -365,7 +143,7 @@ ${colDefs},
// close a version — blocks further edits // close a version — blocks further edits
router.post('/versions/:id/close', async (req, res) => { router.post('/versions/:id/close', async (req, res) => {
const pf_user = sessionUser(req); const { pf_user } = req.body;
try { try {
const result = await pool.query(` const result = await pool.query(`
UPDATE pf.version UPDATE pf.version

View File

@ -1,10 +1,7 @@
require('dotenv').config(); require('dotenv').config();
const express = require('express'); const express = require('express');
const cors = require('cors'); const cors = require('cors');
const session = require('express-session');
const PgSession = require('connect-pg-simple')(session);
const { Pool, types } = require('pg'); const { Pool, types } = require('pg');
const { requireAuth } = require('./lib/auth');
// Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings, // Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings,
// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>. // so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>.
@ -12,15 +9,7 @@ types.setTypeParser(20, v => v === null ? null : Number(v));
types.setTypeParser(1700, v => v === null ? null : Number(v)); types.setTypeParser(1700, v => v === null ? null : Number(v));
const app = express(); const app = express();
app.use(cors());
// Sessions ride on a cookie, so a wildcard CORS origin would let any site make
// credentialed calls on behalf of a logged-in user. The UI is served from this
// same origin and needs no CORS at all; set CORS_ORIGIN only for a separate
// front-end host, and it is then allowed by name, never by wildcard.
if (process.env.CORS_ORIGIN) {
app.use(cors({ origin: process.env.CORS_ORIGIN.split(',').map(o => o.trim()), credentials: true }));
}
app.use(express.json()); app.use(express.json());
app.use(express.static('public/app')); app.use(express.static('public/app'));
@ -37,50 +26,11 @@ pool.on('error', (err) => {
console.error('pg pool error', err); console.error('pg pool error', err);
}); });
// ── Authentication ────────────────────────────────────────────
// Refuse to boot without a secret rather than fall back to a default one:
// a predictable secret means forgeable session cookies.
const sessionSecret = process.env.SESSION_SECRET;
if (!sessionSecret) {
console.error('SESSION_SECRET is not set. Run: ./pf.sh config');
process.exit(1);
}
// TLS terminates at the reverse proxy, so express has to trust its headers for
// req.ip (the login throttle) and for secure-cookie detection to be right.
app.set('trust proxy', process.env.TRUST_PROXY || 1);
const cookieSecure = process.env.COOKIE_SECURE !== 'false';
if (!cookieSecure) {
console.warn('COOKIE_SECURE=false — session cookie will be sent over plain HTTP.');
}
app.use(session({
name: 'pf.sid',
store: new PgSession({ pool, schemaName: 'pf', tableName: 'session', createTableIfMissing: false }),
secret: sessionSecret,
resave: false,
saveUninitialized: false,
rolling: true,
cookie: {
httpOnly: true,
sameSite: 'lax',
secure: cookieSecure,
maxAge: 1000 * 60 * 60 * 12,
},
}));
app.use('/api', require('./routes/auth')(pool));
// Everything below this line requires a session.
app.use('/api', requireAuth);
app.use('/api', require('./routes/tables')(pool)); app.use('/api', require('./routes/tables')(pool));
app.use('/api', require('./routes/sources')(pool)); app.use('/api', require('./routes/sources')(pool));
app.use('/api', require('./routes/versions')(pool)); app.use('/api', require('./routes/versions')(pool));
app.use('/api', require('./routes/operations')(pool)); app.use('/api', require('./routes/operations')(pool));
app.use('/api', require('./routes/log')(pool)); app.use('/api', require('./routes/log')(pool));
app.use('/api', require('./routes/layouts')(pool));
const port = process.env.PORT || 3010; const port = process.env.PORT || 3010;

View File

@ -17,6 +17,8 @@ CREATE TABLE IF NOT EXISTS pf.source (
-- backfill columns for existing installs -- backfill columns for existing installs
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb; ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate -- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
@ -27,18 +29,10 @@ CREATE TABLE IF NOT EXISTS pf.col_meta (
label text, label text,
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | ignore role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | ignore
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
dim_group text, -- groups functionally dependent columns
dim_period_col text, -- pf.dim_period column this dimension derives from
in_grain boolean NOT NULL DEFAULT false, -- true = column defines the display grain
opos integer, opos integer,
UNIQUE (source_id, cname) UNIQUE (source_id, cname)
); );
-- backfill columns for existing installs (must follow the CREATE above)
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;
CREATE TABLE IF NOT EXISTS pf.version ( CREATE TABLE IF NOT EXISTS pf.version (
id serial PRIMARY KEY, id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT, source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
@ -61,139 +55,9 @@ CREATE TABLE IF NOT EXISTS pf.log (
operation text NOT NULL, -- baseline | reference | scale | recode | clone operation text NOT NULL, -- baseline | reference | scale | recode | clone
slice jsonb, slice jsonb,
params jsonb, params jsonb,
note text, note text
tag text -- initiative label, e.g. 'reduce_spend'; groups
-- adjustments into a bridge from baseline to current
); );
-- adding tags to an install that predates them
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS tag text;
CREATE INDEX IF NOT EXISTS log_tag_idx ON pf.log (tag) WHERE tag IS NOT NULL;
-- seed tags for loads that predate the column: a baseline/reference note is the
-- segment's name ('Open Orders', 'Prior Year'), which is exactly what tag holds.
-- Adjustment notes are free text, not labels, so they are left alone.
UPDATE pf.log
SET tag = note
WHERE TRUE
AND tag IS NULL
AND note IS NOT NULL
AND note <> ''
AND operation IN ('baseline', 'reference');
-- Display order for the pivot's segment and bucket columns.
--
-- The segment's display name in the pivot, falling back to tag then note.
--
-- Separate from both because those have jobs already -- tag groups adjustments
-- into initiatives for the bridge, note is free commentary -- and because the
-- label carries the sort order. Perspective orders column groups by the value
-- string, so a leading "01 - " is how ordering is expressed; putting that in the
-- note would put it in every note.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS label text;
-- Vestigial, both of them. They held the ordinal when the "01 - " prefix was
-- computed for the pivot rather than typed into label and bucket: bucket_order
-- sequenced the bucket columns, log.seq the segments within them. Nothing reads
-- either now, and nothing writes them -- kept only because dropping a column is
-- not worth a migration to reclaim two that cost nothing.
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS bucket_order jsonb;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS seq integer;
-- What a segment contributes to, independent of pf_iter.
--
-- pf_iter answers "can operations write to these rows"; bucket answers "does this
-- belong in the forecast number". Those are not the same question -- Open Orders is
-- loaded as reference so nothing adjusts it, yet it is part of the forecast -- so
-- neither can be derived from the other.
--
-- Free text with suggested values (Forecast / Prior Year / Prior Prior Year / Plan)
-- rather than an enum, so a new banner does not need a migration. Blank by default:
-- until a segment is labelled, the pivot falls back to showing its own name.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS bucket text;
CREATE INDEX IF NOT EXISTS log_bucket_idx ON pf.log (bucket) WHERE bucket IS NOT NULL;
-- The names a row falls back to when nobody has named it, per scenario. Null
-- means "use the built-in", which is the DISPLAY DEFAULTS block in
-- lib/sql_generator.js. Read through a join at query time, not baked into
-- pf.sql: those templates are keyed on (source_id, operation) and shared by
-- every version of a source.
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_segment text;
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_bucket text;
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS unlabeled_load text;
-- What the entry did, stamped when it did it.
--
-- Not a cache: a log entry's forecast rows never change after it is written.
-- Only the operation that owns the logid inserts them, and the only thing that
-- removes them is undo, which deletes this row too -- so these totals are fixed
-- at write time rather than derived from something that can move underneath
-- them. The change log was joining the whole forecast table to recompute them
-- on every open, 2.5M rows to total a few thousand.
--
-- measure_cols records which columns they are denominated in, since the value
-- and units roles can be reassigned in col_meta and the numbers would otherwise
-- quietly come to mean something else.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS row_count integer;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS value_total double precision;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS units_total double precision;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS measure_cols jsonb;
-- The column an account's territory is expressed in -- the rep, the region,
-- whatever this source divides ownership by. Exactly one per source.
--
-- Flagged here rather than named in code because it is source-specific, the
-- same way the grain is: a second source should not need a code change to be
-- scoped by something other than a sales rep.
--
-- It does two jobs. The server scopes every read and write to the values on the
-- account, and recode refuses to *set* this column unless the account is an
-- admin -- moving a row between territories is reassignment, not forecasting.
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS is_territory boolean NOT NULL DEFAULT false;
-- The debug path: what the entry was trying to do, and what that became.
--
-- Both, deliberately. params records the intent -- the slice, the scope, the
-- resolved increments -- and sql_text records the statement as executed, with
-- territory and scope already resolved into it. Keeping only one of them
-- assumes the translation between them is correct, which is exactly the
-- assumption in doubt when the rows look wrong.
--
-- env captures the state the intent was executed against that cannot be
-- reconstructed afterwards: the territory in force, the version's
-- exclude_iters, and when the template was generated. All three are mutable
-- rows elsewhere, and nothing remembers what they were.
--
-- The template generation is a fingerprint, not a version: it cannot reproduce
-- the old template, but it can tell you the entry ran under a different one,
-- which is what would otherwise make a replay quietly wrong.
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS env jsonb;
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS sql_text text;
-- Master data for a dim_group: one row per key value, with its sibling columns.
--
-- The source is transactional and often a view over all history, so deriving a
-- member list from it is both slow and wrong -- slow because it means scanning
-- millions of rows, wrong because it can only describe what was transacted and
-- has no way to say a part is discontinued or that a new one exists before it
-- has sold. This table is the app's own list, refreshed from the source but
-- curatable independently of it.
CREATE TABLE IF NOT EXISTS pf.dim_member (
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
dim_group text NOT NULL, -- matches pf.col_meta.dim_group
key_value text NOT NULL, -- the is_key column's value
attrs jsonb NOT NULL DEFAULT '{}'::jsonb, -- the sibling columns
is_active boolean NOT NULL DEFAULT true,
source_seen boolean NOT NULL DEFAULT true, -- present in the source at last refresh
added_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
refreshed_at timestamptz,
PRIMARY KEY (source_id, dim_group, key_value)
);
CREATE INDEX IF NOT EXISTS dim_member_active_idx
ON pf.dim_member (source_id, dim_group) WHERE is_active;
-- generated operation SQL per source, stored after col_meta is configured -- generated operation SQL per source, stored after col_meta is configured
CREATE TABLE IF NOT EXISTS pf.sql ( CREATE TABLE IF NOT EXISTS pf.sql (
id serial PRIMARY KEY, id serial PRIMARY KEY,
@ -203,61 +67,3 @@ CREATE TABLE IF NOT EXISTS pf.sql (
generated_at timestamptz NOT NULL DEFAULT now(), generated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_id, operation) UNIQUE (source_id, operation)
); );
-- pf.layout: named Perspective view configs.
--
-- Replaces the per-browser localStorage lists (pf_layouts_v*) and the single
-- anonymous pf.source.default_layout. A layout is either `private` -- visible
-- only to its owner -- or `published`, visible to everyone on the version and
-- writable only by its owner or an admin, the same rule pf.log already uses.
CREATE TABLE IF NOT EXISTS pf.layout (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
-- null = applies to every version of the source; that is where the old
-- source default lives, and what a new version picks up before anyone has
-- published a layout of its own.
version_id integer REFERENCES pf.version(id) ON DELETE CASCADE,
name text NOT NULL,
config jsonb NOT NULL,
owner text NOT NULL,
visibility text NOT NULL DEFAULT 'private', -- private | published
is_default boolean NOT NULL DEFAULT false, -- applied on first load
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CHECK (visibility IN ('private', 'published')),
-- only a published layout can be the default: a private one would be a
-- default nobody else could see.
CHECK (NOT is_default OR visibility = 'published')
);
-- COALESCE rather than the bare column: version_id is nullable, and in a unique
-- index NULLs are distinct, so source-wide rows would not be constrained at all.
CREATE UNIQUE INDEX IF NOT EXISTS layout_private_name
ON pf.layout (source_id, COALESCE(version_id, 0), owner, lower(name))
WHERE visibility = 'private';
CREATE UNIQUE INDEX IF NOT EXISTS layout_published_name
ON pf.layout (source_id, COALESCE(version_id, 0), lower(name))
WHERE visibility = 'published';
-- one default per scope
CREATE UNIQUE INDEX IF NOT EXISTS layout_one_default
ON pf.layout (source_id, COALESCE(version_id, 0))
WHERE is_default;
CREATE INDEX IF NOT EXISTS layout_lookup ON pf.layout (source_id, version_id);
-- Carry the old single source default across as a published, source-wide row.
-- Idempotent: skipped once a default exists for that source.
INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility, is_default)
SELECT s.id, NULL, 'Source default', s.default_layout, COALESCE(s.created_by, 'admin'), 'published', true
FROM pf.source s
WHERE TRUE
AND s.default_layout IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM pf.layout l
WHERE TRUE
AND l.source_id = s.id
AND l.version_id IS NULL
AND l.is_default
);

View File

@ -1,38 +0,0 @@
-- Pivot Forecast — authentication
-- Run after 01_schema.sql: psql -d <db> -f setup_sql/02_auth.sql
-- Safe to re-run.
-- Application accounts. Passwords are scrypt hashes written by lib/auth.js;
-- the plaintext never reaches the database. Manage with ./pf.sh add-user.
CREATE TABLE IF NOT EXISTS pf.app_user (
id serial PRIMARY KEY,
username text NOT NULL UNIQUE,
pass_hash text NOT NULL,
display_name text,
is_active boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT now(),
last_login_at timestamptz
);
-- Session store for express-session (connect-pg-simple layout). Sessions live
-- here rather than in memory so a restart doesn't sign everyone out, and so a
-- session can be revoked by deleting its row.
CREATE TABLE IF NOT EXISTS pf.session (
sid varchar PRIMARY KEY NOT NULL COLLATE "default",
sess json NOT NULL,
expire timestamp(6) NOT NULL
);
CREATE INDEX IF NOT EXISTS session_expire_idx ON pf.session (expire);
-- What an account may see and change.
--
-- territory is the list of values allowed in the source's territory column
-- (col_meta.is_territory). It is a filter the server applies to every read and
-- every write; it is never sent by the client, which could remove it.
--
-- Fail closed: no territory and not an admin means no rows. An account created
-- without one sees nothing until someone grants it, rather than seeing the whole
-- book because a column was left null.
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS territory jsonb;
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS is_admin boolean NOT NULL DEFAULT false;

View File

@ -5,6 +5,7 @@
<link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pivot Forecast</title> <title>Pivot Forecast</title>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@perspective-dev/viewer@4.4.0/dist/css/themes.css" crossorigin="anonymous">
</head> </head>
<body> <body>
<div id="root"></div> <div id="root"></div>

774
ui/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -10,10 +10,6 @@
"preview": "vite preview" "preview": "vite preview"
}, },
"dependencies": { "dependencies": {
"@perspective-dev/client": "file:vendor/perspective-dev-client-5.4.0.tgz",
"@perspective-dev/server": "file:vendor/perspective-dev-server-5.4.0.tgz",
"@perspective-dev/viewer": "file:vendor/perspective-dev-viewer-5.4.0.tgz",
"@perspective-dev/viewer-datagrid": "file:vendor/perspective-dev-viewer-datagrid-5.4.0.tgz",
"react": "^19.2.5", "react": "^19.2.5",
"react-dom": "^19.2.5" "react-dom": "^19.2.5"
}, },

View File

@ -13,7 +13,6 @@ export default function App() {
const [sourcesLoaded, setSourcesLoaded] = useState(false) const [sourcesLoaded, setSourcesLoaded] = useState(false)
const [sourceId, setSourceId] = useState(() => localStorage.getItem('pf_sourceId') || '') const [sourceId, setSourceId] = useState(() => localStorage.getItem('pf_sourceId') || '')
const [versions, setVersions] = useState([]) const [versions, setVersions] = useState([])
const [versionsLoaded, setVersionsLoaded] = useState(false)
const [versionId, setVersionId] = useState(() => localStorage.getItem('pf_versionId') || '') const [versionId, setVersionId] = useState(() => localStorage.getItem('pf_versionId') || '')
useEffect(() => { localStorage.setItem('pf_view', view) }, [view]) useEffect(() => { localStorage.setItem('pf_view', view) }, [view])
@ -31,11 +30,10 @@ export default function App() {
const refreshVersions = useCallback(async (sid) => { const refreshVersions = useCallback(async (sid) => {
const id = sid ?? sourceId const id = sid ?? sourceId
if (!id) { setVersions([]); setVersionsLoaded(true); return [] } if (!id) { setVersions([]); return [] }
const data = await fetch(`/api/sources/${id}/versions`).then(r => r.json()) const data = await fetch(`/api/sources/${id}/versions`).then(r => r.json())
const list = Array.isArray(data) ? data : [] const list = Array.isArray(data) ? data : []
setVersions(list) setVersions(list)
setVersionsLoaded(true)
return list return list
}, [sourceId]) }, [sourceId])
@ -54,26 +52,18 @@ export default function App() {
}, [sources, sourcesLoaded, sourceId]) }, [sources, sourcesLoaded, sourceId])
useEffect(() => { useEffect(() => {
if (!sourceId) { setVersions([]); setVersionId(''); setVersionsLoaded(true); return } if (!sourceId) { setVersions([]); setVersionId(''); return }
// The list belongs to the previous source until the fetch lands.
setVersionsLoaded(false)
refreshVersions(sourceId) refreshVersions(sourceId)
}, [sourceId]) }, [sourceId])
// Same reasoning as sources: a deleted version must not stay selected. // Same reasoning as sources: a deleted version must not stay selected.
//
// versionsLoaded matters more than it looks: without it, the empty initial
// state reads as "this source has no versions", so a versionId restored from
// localStorage is cleared and then set straight back when the fetch lands.
// Forecast's load effect is keyed on that id, so the round trip made every
// page load fetch and aggregate the whole version twice.
useEffect(() => { useEffect(() => {
if (!sourceId || !versionsLoaded) return if (!sourceId) return
if (versions.length === 0) { setVersionId(''); return } if (versions.length === 0) { setVersionId(''); return }
if (!versionId || !versions.some(v => String(v.id) === String(versionId))) { if (!versionId || !versions.some(v => String(v.id) === String(versionId))) {
setVersionId(String(versions[0].id)) setVersionId(String(versions[0].id))
} }
}, [versions, versionsLoaded, sourceId, versionId]) }, [versions, sourceId, versionId])
const ctx = { const ctx = {
sources, sourceId, setSourceId, sources, sourceId, setSourceId,

View File

@ -1,62 +0,0 @@
import { createContext, useContext, useState, useEffect, useCallback } from 'react'
const AuthContext = createContext()
// A session can expire while the app is open. Rather than teach every one of
// the app's fetch calls to check for it, wrap fetch once: any 401 from /api
// drops the whole UI back to the login screen. Cookies ride along on their own
// fetch defaults to same-origin credentials, and the UI is served from the
// same origin as the API.
function installUnauthorizedHandler(onUnauthorized) {
const original = window.fetch
window.fetch = async (...args) => {
const res = await original(...args)
const url = typeof args[0] === 'string' ? args[0] : args[0]?.url || ''
if (res.status === 401 && url.includes('/api/') && !url.includes('/api/login')) {
onUnauthorized()
}
return res
}
return () => { window.fetch = original }
}
export function AuthProvider({ children }) {
const [user, setUser] = useState(null)
const [checking, setCheck] = useState(true)
useEffect(() => {
fetch('/api/me')
.then(r => r.ok ? r.json() : null)
.then(d => setUser(d?.user || null))
.catch(() => setUser(null))
.finally(() => setCheck(false))
}, [])
useEffect(() => installUnauthorizedHandler(() => setUser(null)), [])
const login = useCallback(async (username, password) => {
const r = await fetch('/api/login', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ username, password }),
})
const data = await r.json().catch(() => ({}))
if (!r.ok) throw new Error(data.error || 'Login failed')
setUser(data.user)
return data.user
}, [])
const logout = useCallback(async () => {
try { await fetch('/api/logout', { method: 'POST' }) } catch {}
setUser(null)
}, [])
return (
<AuthContext.Provider value={{ user, checking, login, logout }}>
{children}
</AuthContext.Provider>
)
}
const useAuth = () => useContext(AuthContext)
export default useAuth

View File

@ -1,565 +0,0 @@
// Bridge (waterfall): how a version got from its baseline to where it stands,
// one step per initiative tag.
//
// Data is computed from the Perspective table already in the browser rather than
// from the /bridge endpoint, so the figures always reconcile with what the pivot
// is showing including when the view is scoped to the pivot's current filters.
//
// Colour is a POLARITY job, not a categorical one: increases and decreases are two
// poles of one scale, with baseline and current as neutral anchors. Blue/red is the
// validated diverging pair (CVD ΔE 21.6, normal-vision 32.3 against white);
// green/red is avoided precisely because it is the classic CVD failure.
import { useState, useEffect, useRef, useCallback } from 'react'
const UP = '#2a78d6' // increase
const DOWN = '#e34948' // decrease
const ANCHOR = '#6b7280' // baseline / current neutral, 4.83:1 on white
const GRID = '#e5e7eb'
const INK = '#374151'
const INK_DIM = '#6b7280'
const fmt = (n, dp = 2) =>
n == null || !isFinite(n) ? '—'
: n.toLocaleString(undefined, { minimumFractionDigits: dp, maximumFractionDigits: dp })
const fmtSigned = (n, dp = 2) =>
n == null || !isFinite(n) ? '—' : `${n > 0 ? '+' : n < 0 ? '' : ''}${fmt(Math.abs(n), dp)}`
// compact axis ticks full precision belongs on the marks and in the table
function fmtAxis(n) {
const a = Math.abs(n)
if (a >= 1e9) return `${(n / 1e9).toFixed(1)}B`
if (a >= 1e6) return `${(n / 1e6).toFixed(1)}M`
if (a >= 1e3) return `${(n / 1e3).toFixed(1)}k`
return String(Math.round(n))
}
function niceTicks(min, max, count = 5) {
if (!isFinite(min) || !isFinite(max) || min === max) return [min || 0]
const span = max - min
const raw = span / count
const mag = Math.pow(10, Math.floor(Math.log10(raw)))
const step = [1, 2, 2.5, 5, 10].map(m => m * mag).find(s => s >= raw) || mag * 10
const out = []
for (let t = Math.ceil(min / step) * step; t <= max + 1e-9; t += step) out.push(t)
return out
}
// Turn raw forecast rows into the walk: baseline anchor, one floating step per
// initiative tag, current anchor. Pure and exported so the arithmetic can be
// checked against real data without a browser.
// The walk from a basis to the current forecast.
//
// Without a basis this is the forecast's own composition: its loads as the
// opening anchor, then one step per adjustment tag.
//
// With one -- Plan when building a forecast, Prior Year when building the AOP --
// it starts there instead, and the identity that makes it exact is
//
// Forecast - Basis = (Forecast loads - Basis) + adjustments
//
// so the opening step is the difference between the forecast's own loads and the
// basis, and every tagged adjustment explains the rest. No residual, no plug: the
// bars sum to the endpoint by construction rather than by hoping the tags cover
// everything.
//
// Membership comes from pf_bucket, not pf_iter. Those answer different questions
// -- Open Orders is loaded as reference so nothing adjusts it, and is still part
// of the forecast -- so a bridge keyed on iter silently dropped it.
export function buildSteps(rows, {
valueCol, unitsCol, logMeta = {},
excludeIters = ['reference'], // kept for callers with no bucket data
basis = null, // a pf_bucket name, or null for composition
forecastBucket,
}) {
const hasBuckets = rows.some(r => r.pf_bucket != null)
const excl = new Set(excludeIters)
// Which bucket is the forecast. It used to be the literal 'Forecast', and the
// moment the buckets were renamed to carry their sort prefix -- '04 - Forecast'
// -- no row matched it: every row counted as a comparison, the walk had no
// steps, and the bridge showed the basis cancelling itself to zero.
//
// The caller passes the version's adjustment_bucket, which is the same value
// an unbucketed adjustment is labelled with, so the two agree by construction.
// Falling back to whichever bucket actually holds the adjustments keeps a
// renamed or unconfigured version working rather than silently empty.
const fcBucket = (() => {
if (forecastBucket && rows.some(r => r.pf_bucket === forecastBucket)) return forecastBucket
const adjusted = rows.find(r => r.pf_bucket && !['baseline', 'reference'].includes(r.pf_iter))
if (adjusted) return adjusted.pf_bucket
const baseline = rows.find(r => r.pf_bucket && r.pf_iter === 'baseline')
return baseline ? baseline.pf_bucket : (forecastBucket || 'Forecast')
})()
const num = (r, col) => (col ? (parseFloat(r[col]) || 0) : 0)
const blank = () => ({ value: 0, units: 0, rows: 0 })
const loads = blank() // the forecast's own segments
const basisT = blank() // the comparison bucket
const byTag = new Map()
const buckets = new Map() // every bucket, for the picker and the markers
for (const r of rows) {
const v = num(r, valueCol)
const u = num(r, unitsCol)
const bucket = hasBuckets ? (r.pf_bucket || '') : null
if (bucket != null) {
const b = buckets.get(bucket) || blank()
b.value += v; b.units += u; b.rows += 1
buckets.set(bucket, b)
}
// Anything outside the forecast is a comparison, never a step.
if (hasBuckets && bucket !== fcBucket) {
if (basis && bucket === basis) { basisT.value += v; basisT.units += u; basisT.rows += 1 }
continue
}
if (!hasBuckets && excl.has(r.pf_iter)) continue
const isLoad = r.pf_iter === 'baseline' || r.pf_iter === 'reference'
if (isLoad) { loads.value += v; loads.units += u; loads.rows += 1; continue }
const meta = logMeta[r.pf_logid] || {}
// label first, the same precedence pf_segment uses, so the bridge and the
// pivot call a step by the same name
const tag = (meta.label || meta.tag || '').trim()
const label = tag || (meta.note || '').trim() ||
`${(meta.operation || r.pf_iter || 'adj')}${r.pf_logid != null ? ` #${r.pf_logid}` : ''}`
const key = tag ? `tag:${tag}` : `log:${r.pf_logid}`
const g = byTag.get(key) ||
{ key, label, tagged: !!tag, value: 0, units: 0, rows: 0, logIds: new Set(), first: r.pf_logid }
g.value += v; g.units += u; g.rows += 1
if (r.pf_logid != null) { g.logIds.add(r.pf_logid); g.first = Math.min(g.first ?? r.pf_logid, r.pf_logid) }
byTag.set(key, g)
}
const mid = [...byTag.values()].sort((a, b) => (a.first ?? 0) - (b.first ?? 0))
const useBasis = !!basis && basisT.rows > 0
const out = []
if (useBasis) {
out.push({
key: 'basis', label: basis, kind: 'anchor',
delta: basisT.value, start: 0, end: basisT.value,
units: basisT.units, rows: basisT.rows, entries: 1,
})
const gap = loads.value - basisT.value
out.push({
// "Loads" is our word for a segment import and means nothing to anyone
// reading a waterfall. This step is simply where the forecast began
// relative to the comparison.
//
// tagged, though it has no tag: the flag drives an "· untagged" suffix
// meant for adjustments nobody grouped into an initiative, and this is
// not an adjustment at all.
key: 'loads-vs-basis', label: `Starting point vs ${basis}`, kind: 'step',
tagged: true,
delta: gap, start: basisT.value, end: loads.value,
units: loads.units - basisT.units, rows: loads.rows, entries: 1,
})
} else {
out.push({
key: 'baseline', label: hasBuckets ? fcBucket + ' loads' : 'Baseline', kind: 'anchor',
delta: loads.value, start: 0, end: loads.value,
units: loads.units, rows: loads.rows, entries: 1,
})
}
let running = loads.value
for (const g of mid) {
const start = running
running += g.value
out.push({ ...g, kind: 'step', delta: g.value, start, end: running, entries: g.logIds.size })
}
out.push({
key: 'current', label: hasBuckets ? fcBucket : 'Current', kind: 'anchor',
delta: running, start: 0, end: running,
units: loads.units + mid.reduce((a, g) => a + (g.units || 0), 0),
rows: loads.rows + mid.reduce((a, g) => a + g.rows, 0),
entries: mid.reduce((a, g) => a + g.logIds.size, 0) + 1,
})
// Every bucket present, so the view can offer them as bases and show the ones
// that are not the basis as comparison markers.
out.buckets = [...buckets.entries()]
.map(([name, t]) => ({ name, ...t }))
.sort((a, b) => b.value - a.value)
return out
}
// Plot geometry, also pure: given the steps and a canvas size, where does each
// bar and label land? Exported so collisions and overflow can be checked.
// Break a label to the bar's width, on word boundaries.
//
// SVG text does not wrap, so the labels were cut at twelve characters and
// "04 - Forecast" and "04 - Forecasting" read the same. Character width is
// estimated rather than measured -- measuring means a DOM round trip per label
// on every render, and at this font a digit is about 0.55em, which is close
// enough for a centred label with a bar's width to play with.
//
// Three lines maximum, and a word longer than the line is left to overflow
// rather than broken: a truncated word helps nobody, and the bars are wide
// enough that it only happens to things like a pasted part number.
function wrapLabel(label, barW, fontSize = 10, maxLines = 3) {
const perLine = Math.max(6, Math.floor(barW / (fontSize * 0.55)))
const words = String(label || '').split(/\s+/).filter(Boolean)
const lines = []
let line = ''
for (const w of words) {
const next = line ? `${line} ${w}` : w
if (next.length <= perLine) { line = next; continue }
if (line) lines.push(line)
line = w
if (lines.length === maxLines) break
}
if (line && lines.length < maxLines) lines.push(line)
if (!lines.length) return ['']
// anything that did not fit is marked on the last line rather than dropped
const used = lines.join(' ').length
if (used < String(label).replace(/\s+/g, ' ').length) {
lines[lines.length - 1] = `${lines[lines.length - 1]}`
}
return lines
}
export function layoutSteps(steps, width, H = 340, PAD = { t: 24, r: 16, b: 84, l: 68 }) {
const plotW = Math.max(120, width - PAD.l - PAD.r)
const plotH = H - PAD.t - PAD.b
const values = steps.flatMap(s => [s.start, s.end])
const rawMin = Math.min(0, ...values)
const rawMax = Math.max(0, ...values)
const span = (rawMax - rawMin) || 1
const yMin = rawMin - span * 0.08
const yMax = rawMax + span * 0.12
const y = (v) => PAD.t + plotH - ((v - yMin) / (yMax - yMin)) * plotH
const n = steps.length || 1
const band = plotW / n
const barW = Math.max(10, Math.min(64, band - 14))
const bars = steps.map((s, i) => {
const x = PAD.l + band * i + (band - barW) / 2
const top = y(Math.max(s.start, s.end))
const bot = y(Math.min(s.start, s.end))
return { key: s.key, x, w: barW, top, h: Math.max(2, bot - top), labelY: top - 6 }
})
return { PAD, plotW, plotH, yMin, yMax, y, band, barW, bars, H, width }
}
export default function BridgeView({
open, onClose, tableRef, viewerRef, logMeta = {},
valueCol, unitsCol, colMeta = [], slices = [],
excludeIters = ['reference'], versionName, forecastBucket,
}) {
const hasSelection = slices.length > 0
// 'selection' | 'filtered' | 'all'
const [scope, setScope] = useState(hasSelection ? 'selection' : 'filtered')
// Which bucket the walk starts from. Plan when building a forecast, Prior Year
// when building the AOP; empty means show the forecast's own composition.
const [basis, setBasis] = useState(() => localStorage.getItem('pf_bridge_basis') || '')
const [asTable, setAsTable] = useState(false)
const [steps, setSteps] = useState(null)
const [loading, setLoading] = useState(false)
const [error, setError] = useState(null)
const [hover, setHover] = useState(null)
const [width, setWidth] = useState(880)
const boxRef = useRef(null)
// Build the steps: baseline anchor, one floating step per tag, current anchor.
const compute = useCallback(async () => {
if (!tableRef?.current || !valueCol) return
setLoading(true); setError(null)
try {
let rows
if (scope === 'selection') {
// The union of the selected slices the same reach an operation would
// have. Perspective view filters are AND-only, so each slice needs its own
// view; rows matching more than one slice are counted once.
const dimNames = new Set(colMeta.filter(c => c.role === 'dimension').map(c => c.cname))
const dateNames = new Set(colMeta.filter(c => c.role === 'date').map(c => c.cname))
const seen = new Set()
rows = []
for (const sl of slices) {
const f = [
...Object.entries(sl).filter(([c]) => dimNames.has(c)).map(([c, v]) => [c, '==', v]),
...Object.entries(sl).filter(([c]) => dateNames.has(c)).map(([c, v]) => [c, '==', Number(v)]),
]
if (!f.length) continue
// No expressions needed here: these filters are built from col_meta
// names, so they only ever reference real columns.
const view = await tableRef.current.view({ filter: f })
const part = await view.to_json()
await view.delete()
for (const r of part) {
if (r.pf_id != null && seen.has(r.pf_id)) continue
if (r.pf_id != null) seen.add(r.pf_id)
rows.push(r)
}
}
} else {
let filter = []
// Expression columns have to come with the filter that uses them: a
// filter can name a column that exists only as an expression, and a view
// built without it cannot resolve the column and fails outright.
let expressions = {}
if (scope === 'filtered' && viewerRef?.current) {
const cfg = await viewerRef.current.save()
filter = (cfg.filter || []).filter(f => Array.isArray(f) && f.length >= 2)
expressions = cfg.expressions || {}
}
const viewCfg = {}
if (filter.length) viewCfg.filter = filter
if (Object.keys(expressions).length) viewCfg.expressions = expressions
const view = await tableRef.current.view(viewCfg)
rows = await view.to_json()
await view.delete()
}
setSteps(buildSteps(rows, {
valueCol, unitsCol, logMeta, excludeIters, basis: basis || null, forecastBucket,
}))
} catch (err) {
setError(err.message || String(err))
setSteps(null)
} finally {
setLoading(false)
}
}, [tableRef, viewerRef, scope, logMeta, valueCol, unitsCol, excludeIters, slices, colMeta, basis, forecastBucket])
useEffect(() => {
if (!hasSelection && scope === 'selection') setScope('filtered')
}, [hasSelection, scope])
useEffect(() => { try { localStorage.setItem('pf_bridge_basis', basis) } catch {} }, [basis])
useEffect(() => { if (open) compute() }, [open, compute])
useEffect(() => {
if (!open || !boxRef.current) return
const ro = new ResizeObserver(([e]) => setWidth(Math.max(420, e.contentRect.width)))
ro.observe(boxRef.current)
return () => ro.disconnect()
}, [open])
if (!open) return null
const geom = layoutSteps(steps || [{ start: 0, end: 0 }], width)
const { PAD, plotW, plotH, yMin, yMax, y, barW, H, bars } = geom
const xOf = (i) => bars[i]?.x ?? PAD.l
const ticks = niceTicks(yMin, yMax, 5)
return (
<div className="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onClick={onClose}>
<div className="bg-white rounded-lg shadow-xl w-full max-w-5xl mx-4 flex flex-col max-h-[88vh]"
onClick={e => e.stopPropagation()}>
<div className="flex items-center justify-between px-5 py-3 border-b border-gray-200 shrink-0">
<div className="flex items-baseline gap-2">
<span className="font-medium text-gray-700 text-sm">Bridge</span>
{versionName && <span className="text-gray-600 text-xs">{versionName}</span>}
<span className="text-gray-600 text-xs">
· {scope === 'selection'
? `${slices.length} selected slice${slices.length === 1 ? '' : 's'}`
: scope === 'filtered' ? "pivot's filters" : 'whole version'}
</span>
</div>
<button onClick={onClose} className="text-gray-600 hover:text-gray-800 text-lg leading-none">×</button>
</div>
{/* controls — one row above the chart */}
<div className="flex items-center gap-3 px-5 py-2 border-b border-gray-100 shrink-0 text-xs flex-wrap">
<span className="text-gray-600">Scope</span>
<div className="inline-flex rounded border border-gray-200 overflow-hidden">
{[
['selection', hasSelection ? `Selection (${slices.length})` : 'Selection',
hasSelection ? 'The slices selected in the operation panel'
: 'Select one or more pivot rows first'],
['filtered', "Pivot's filters", 'Everything the pivot currently shows'],
['all', 'Whole version', 'Every row in the version, filters ignored'],
].map(([v, l, title]) => (
<button key={v} onClick={() => setScope(v)} title={title}
disabled={v === 'selection' && !hasSelection}
className={`px-3 py-1 disabled:opacity-40 disabled:cursor-not-allowed ${
scope === v ? 'bg-blue-600 text-white' : 'bg-white text-gray-700 hover:bg-gray-50'}`}>
{l}
</button>
))}
</div>
{/* Offered only once something carries a bucket, and only listing buckets
other than the forecast itself -- a walk from Forecast to Forecast is
the composition view, which is what the empty option gives. */}
{(steps?.buckets || []).some(b => b.name && b.name !== 'Forecast') && (
<>
<div className="w-px h-4 bg-gray-200" />
<span className="text-gray-600">Compare to</span>
<select value={basis} onChange={e => setBasis(e.target.value)}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 bg-white">
<option value="">nothing show composition</option>
{(steps?.buckets || [])
.filter(b => b.name && b.name !== 'Forecast')
.map(b => <option key={b.name} value={b.name}>{b.name}</option>)}
</select>
</>
)}
<div className="w-px h-4 bg-gray-200" />
<button onClick={() => setAsTable(t => !t)}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 hover:bg-gray-50">
{asTable ? 'Show chart' : 'Show table'}
</button>
<button onClick={compute} disabled={loading}
className="border border-gray-200 rounded px-2 py-1 text-gray-700 hover:bg-gray-50 disabled:opacity-40">
{loading ? 'Computing…' : 'Refresh'}
</button>
{/* legend — identity is never colour alone, but say it anyway */}
<div className="ml-auto flex items-center gap-3 text-gray-700">
{[['Increase', UP], ['Decrease', DOWN], ['Total', ANCHOR]].map(([l, c]) => (
<span key={l} className="inline-flex items-center gap-1.5">
<span className="inline-block w-2.5 h-2.5 rounded-sm" style={{ background: c }} />
{l}
</span>
))}
</div>
</div>
<div className="overflow-auto p-5" ref={boxRef}>
{error && <p className="text-red-600">{error}</p>}
{!error && !steps && <p className="text-gray-600">Computing</p>}
{!error && steps && steps.length <= 2 && (
<p className="text-gray-600">
No adjustments in scope the bridge shows the walk from baseline to current,
and this selection has only a baseline.
</p>
)}
{!error && steps && steps.length > 2 && !asTable && (
<div className="relative">
<svg width={width} height={H} role="img"
aria-label={`Bridge from baseline ${fmt(steps[0].end)} to current ${fmt(steps[steps.length - 1].end)}`}>
{/* recessive grid */}
{ticks.map(t => (
<g key={t}>
<line x1={PAD.l} x2={PAD.l + plotW} y1={y(t)} y2={y(t)}
stroke={t === 0 ? '#d1d5db' : GRID} strokeWidth={t === 0 ? 1.5 : 1} />
<text x={PAD.l - 8} y={y(t) + 3} textAnchor="end" fontSize="10" fill={INK_DIM}>
{fmtAxis(t)}
</text>
</g>
))}
{steps.map((s, i) => {
const isAnchor = s.kind === 'anchor'
const up = s.delta >= 0
const fill = isAnchor ? ANCHOR : (up ? UP : DOWN)
const top = y(Math.max(s.start, s.end))
const bot = y(Math.min(s.start, s.end))
const h = Math.max(2, bot - top)
const x = xOf(i)
const on = hover?.key === s.key
const labelLines = wrapLabel(s.label, barW)
const subY = PAD.t + plotH + 16 + labelLines.length * 11 + 2
return (
<g key={s.key}
onMouseEnter={() => setHover({ ...s, x: x + barW / 2, y: top })}
onMouseLeave={() => setHover(null)}>
{/* connector to the next bar, drawn behind */}
{i < steps.length - 1 && (
<line x1={x + barW} x2={xOf(i + 1)} y1={y(s.end)} y2={y(s.end)}
stroke="#cbd5e1" strokeWidth="1" strokeDasharray="2 2" />
)}
{/* hit target larger than the mark */}
<rect x={x - 6} y={PAD.t} width={barW + 12} height={plotH} fill="transparent" />
<rect x={x} y={top} width={barW} height={h} rx="4" fill={fill}
opacity={on ? 1 : 0.92}
stroke="#ffffff" strokeWidth="2" />
{/* direct label: few bars, so every one is labelled */}
<text x={x + barW / 2} y={top - 6} textAnchor="middle" fontSize="10"
fill={INK} fontWeight="500">
{isAnchor ? fmt(s.end, 0) : fmtSigned(s.delta, 0)}
</text>
<text x={x + barW / 2} y={PAD.t + plotH + 16} textAnchor="middle" fontSize="10" fill={INK}>
{labelLines.map((ln, li) => (
<tspan key={li} x={x + barW / 2} dy={li === 0 ? 0 : 11}>{ln}</tspan>
))}
</text>
{/* below the label, however many lines it took */}
{!isAnchor && s.entries > 1 && (
<text x={x + barW / 2} y={subY} textAnchor="middle" fontSize="9" fill={INK_DIM}>
×{s.entries}
</text>
)}
{!s.tagged && !isAnchor && (
<text x={x + barW / 2} y={subY} textAnchor="middle" fontSize="9" fill={INK_DIM}>
untagged
</text>
)}
</g>
)
})}
</svg>
{hover && (
<div className="absolute pointer-events-none bg-white border border-gray-300 rounded shadow-lg px-2.5 py-1.5 text-xs"
style={{ left: Math.min(hover.x + 10, width - 190), top: Math.max(0, hover.y - 10) }}>
<div className="font-medium text-gray-800">{hover.label}</div>
<div className="text-gray-700 font-mono tabular-nums">
{hover.kind === 'anchor' ? fmt(hover.end) : fmtSigned(hover.delta)}
</div>
{hover.kind === 'step' && (
<div className="text-gray-600">
running <span className="font-mono tabular-nums">{fmt(hover.end)}</span>
</div>
)}
<div className="text-gray-600">
{hover.rows} row{hover.rows === 1 ? '' : 's'}
{hover.entries > 1 ? ` · ${hover.entries} adjustments` : ''}
</div>
</div>
)}
</div>
)}
{/* table view — the same numbers, at full precision */}
{!error && steps && steps.length > 2 && asTable && (
<table className="w-full text-xs">
<thead>
<tr className="text-gray-600 border-b border-gray-200">
<th className="text-left py-1.5 pr-3 font-medium">Step</th>
<th className="text-right py-1.5 px-2 font-medium">{valueCol}</th>
{unitsCol && <th className="text-right py-1.5 px-2 font-medium">{unitsCol}</th>}
<th className="text-right py-1.5 px-2 font-medium">Running</th>
<th className="text-right py-1.5 px-2 font-medium">Adjustments</th>
<th className="text-right py-1.5 pl-2 font-medium">Rows</th>
</tr>
</thead>
<tbody>
{steps.map(s => (
<tr key={s.key} className="border-b border-gray-100">
<td className="py-1.5 pr-3 text-gray-800">
{s.label}{!s.tagged && s.kind === 'step' && <span className="text-gray-600"> · untagged</span>}
</td>
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-800">
{s.kind === 'anchor' ? fmt(s.end) : fmtSigned(s.delta)}
</td>
{unitsCol && (
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-700">
{s.kind === 'anchor' ? fmt(s.units) : fmtSigned(s.units)}
</td>
)}
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-700">{fmt(s.end)}</td>
<td className="py-1.5 px-2 text-right text-gray-700">{s.kind === 'step' ? s.entries : '—'}</td>
<td className="py-1.5 pl-2 text-right text-gray-700">{s.rows}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</div>
</div>
)
}

View File

@ -1,190 +0,0 @@
import { useEffect, useRef, useState } from 'react'
// The pivot layout, as one control rather than a row of equal chips.
//
// Two lists, because a layout is one of two different things: Published, which
// everyone on this forecast sees and only its owner or an admin may change, and
// Mine, which nobody else lists. Everything you cannot write is still there to
// apply -- it is only the write controls that are withheld, so a click never
// answers 403.
export default function LayoutMenu({
layouts = [], activeLayoutId, dirty,
onApply, onSave, onSaveAs, onSetVisibility, onSetDefault, onRename, onDelete, onReset
}) {
const [open, setOpen] = useState(false)
const [saveName, setSaveName] = useState('')
const [savePublic, setSavePublic] = useState(false)
const [renaming, setRenaming] = useState(null)
const [renameTo, setRenameTo] = useState('')
const boxRef = useRef(null)
// Close on an outside click or Escape. Capture phase: the pivot lives in a
// shadow root and retargets its events, so a bubbled listener on document
// sees the host rather than the real target and cannot tell inside from out.
useEffect(() => {
if (!open) return
const onDown = (e) => { if (!boxRef.current?.contains(e.target)) setOpen(false) }
const onKey = (e) => { if (e.key === 'Escape') setOpen(false) }
document.addEventListener('mousedown', onDown, true)
window.addEventListener('keydown', onKey)
return () => {
document.removeEventListener('mousedown', onDown, true)
window.removeEventListener('keydown', onKey)
}
}, [open])
const active = layouts.find(l => l.id === activeLayoutId)
const published = layouts.filter(l => l.visibility === 'published')
const mine = layouts.filter(l => l.visibility === 'private')
const submitSaveAs = () => {
const name = saveName.trim()
if (!name) return
onSaveAs(name, savePublic ? 'published' : 'private')
setSaveName('')
setOpen(false)
}
const submitRename = (l) => {
const name = renameTo.trim()
if (name && name !== l.name) onRename(l, name)
setRenaming(null)
}
const iconBtn = 'px-1 rounded text-gray-400 hover:text-gray-700 hover:bg-gray-100 leading-none'
const row = (l) => (
<div key={l.id}
className={`group flex items-center gap-1 px-2 py-1 rounded cursor-pointer
${l.id === activeLayoutId ? 'bg-blue-50 text-blue-700' : 'hover:bg-gray-50 text-gray-700'}`}
onClick={() => { if (renaming !== l.id) { onApply(l); setOpen(false) } }}>
{renaming === l.id ? (
<input autoFocus value={renameTo} onClick={e => e.stopPropagation()}
onChange={e => setRenameTo(e.target.value)}
onBlur={() => submitRename(l)}
onKeyDown={e => {
if (e.key === 'Enter') submitRename(l)
if (e.key === 'Escape') setRenaming(null)
}}
className="flex-1 min-w-0 border border-blue-300 rounded px-1 py-0 outline-none bg-white" />
) : (
<>
<span className="flex-1 min-w-0 truncate">{l.name}</span>
{l.is_default && (
<span title="Applied when this forecast is opened for the first time"
className="text-amber-500 shrink-0"></span>
)}
{l.scope === 'source' && (
<span title="Applies to every forecast of this source"
className="text-gray-300 shrink-0" style={{fontSize:'9px'}}>ALL</span>
)}
</>
)}
{/* Withheld rather than offered-and-refused: the server allows a write only
to the owner or an admin, and can_edit says which this is. */}
{l.can_edit && renaming !== l.id && (
<span className="hidden group-hover:flex items-center gap-0.5 shrink-0"
onClick={e => e.stopPropagation()}>
<button className={iconBtn} title="Rename"
onClick={() => { setRenaming(l.id); setRenameTo(l.name) }}></button>
{l.visibility === 'private' ? (
<button className={iconBtn} title="Publish — everyone on this forecast sees it"
onClick={() => onSetVisibility(l, 'published')}></button>
) : (
<>
{!l.is_default && (
<button className={iconBtn} title="Apply this when the forecast is first opened"
onClick={() => onSetDefault(l)}></button>
)}
<button className={iconBtn} title="Unpublish — keep it, but only for you"
onClick={() => onSetVisibility(l, 'private')}></button>
</>
)}
<button className={`${iconBtn} hover:text-red-500`} title="Delete"
onClick={() => onDelete(l)}>×</button>
</span>
)}
{!l.can_edit && (
<span className="hidden group-hover:inline text-gray-300 shrink-0 truncate"
style={{fontSize:'9px'}} title={`Published by ${l.owner}`}>{l.owner}</span>
)}
</div>
)
return (
<div ref={boxRef} className="relative flex items-center gap-1.5">
<span className="text-gray-400 uppercase tracking-wide" style={{fontSize:'10px'}}>Layout</span>
<button onClick={() => setOpen(o => !o)}
className={`flex items-center gap-1 border rounded px-2 py-0.5 max-w-[14rem] transition-colors
${open ? 'border-blue-300 text-blue-700 bg-blue-50'
: 'border-gray-200 text-gray-600 hover:border-gray-400'}`}>
<span className="truncate">{active ? active.name : 'Unsaved'}</span>
{dirty && <span title="Changed since it was saved" className="text-amber-500 leading-none"></span>}
<span className="text-gray-400 leading-none"></span>
</button>
{/* Save is the common action and stays outside the menu, but only when
there is something to save it to and it is yours to overwrite. */}
{active?.can_edit && dirty && (
<button onClick={() => onSave(active)}
className="border border-blue-200 text-blue-600 hover:text-blue-800 rounded px-2 py-0.5">
Save
</button>
)}
{open && (
<div className="absolute top-full left-0 mt-1 z-50 w-72 bg-white border border-gray-200
rounded shadow-lg py-1 text-xs">
{published.length > 0 && (
<>
<div className="px-2 pt-1 pb-0.5 text-gray-400 uppercase tracking-wide"
style={{fontSize:'9px'}}>Published</div>
<div className="px-1">{published.map(row)}</div>
</>
)}
{mine.length > 0 && (
<>
<div className="px-2 pt-2 pb-0.5 text-gray-400 uppercase tracking-wide"
style={{fontSize:'9px'}}>Mine</div>
<div className="px-1">{mine.map(row)}</div>
</>
)}
{!layouts.length && (
<div className="px-3 py-2 text-gray-400">No saved layouts yet</div>
)}
<div className="border-t border-gray-100 mt-1 pt-1 px-2">
<div className="flex items-center gap-1 py-1">
<input value={saveName} onChange={e => setSaveName(e.target.value)}
onKeyDown={e => { if (e.key === 'Enter') submitSaveAs() }}
placeholder="Save current view as…"
className="flex-1 min-w-0 border border-gray-300 rounded px-1.5 py-0.5
outline-none focus:border-blue-400" />
<button onClick={submitSaveAs} disabled={!saveName.trim()}
className="text-blue-600 hover:text-blue-800 px-1 disabled:opacity-30">Save</button>
</div>
<label className="flex items-center gap-1.5 py-0.5 text-gray-500 cursor-pointer">
<input type="checkbox" checked={savePublic}
onChange={e => setSavePublic(e.target.checked)} />
Publish to everyone on this forecast
</label>
</div>
<div className="border-t border-gray-100 mt-1 pt-1 px-2 pb-0.5">
<button onClick={() => { onReset(); setOpen(false) }}
className="text-gray-400 hover:text-gray-700 py-0.5">
Reset to a blank pivot
</button>
</div>
</div>
)}
</div>
)
}

File diff suppressed because it is too large Load Diff

View File

@ -1,47 +1,12 @@
import { useState, useEffect, useCallback } from 'react'
import useTheme from '../theme.jsx' import useTheme from '../theme.jsx'
import useAuth from '../auth.jsx'
export default function StatusBar({ view, sources = [], sourceId, setSourceId, versions = [], versionId, setVersionId }) { export default function StatusBar({ view, sources = [], sourceId, setSourceId, versions = [], versionId, setVersionId }) {
const { dark, setDark } = useTheme() const { dark, setDark } = useTheme()
const { user, logout } = useAuth()
const showVersion = view === 'baseline' || view === 'forecast' const showVersion = view === 'baseline' || view === 'forecast'
const selectedVersion = versions.find(v => String(v.id) === String(versionId)) const selectedVersion = versions.find(v => String(v.id) === String(versionId))
const [info, setInfo] = useState(null)
const [showInfo, setShow] = useState(false)
const [copied, setCopied] = useState(false)
const refreshInfo = useCallback(async () => {
if (!versionId || !showVersion) { setInfo(null); return }
try {
const r = await fetch(`/api/versions/${versionId}/table-info`)
setInfo(r.ok ? await r.json() : null)
} catch { setInfo(null) }
}, [versionId, showVersion])
useEffect(() => { refreshInfo() }, [refreshInfo])
// operations broadcast this after a write so the row count stays honest
useEffect(() => {
const onChange = () => refreshInfo()
window.addEventListener('pf-data-changed', onChange)
return () => window.removeEventListener('pf-data-changed', onChange)
}, [refreshInfo])
async function copyTable() {
if (!info?.fc_table) return
try {
await navigator.clipboard.writeText(info.fc_table)
setCopied(true)
setTimeout(() => setCopied(false), 1200)
} catch {}
}
const fmt = (n) => n == null ? '—' : n.toLocaleString()
return ( return (
<div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs relative"> <div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs">
<span className="text-gray-400">Source</span> <span className="text-gray-400">Source</span>
<select <select
value={sourceId || ''} value={sourceId || ''}
@ -73,75 +38,10 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
{selectedVersion.status} {selectedVersion.status}
</span> </span>
)} )}
{/* write target — the physical table every operation appends to */}
{info && (
<>
<span className="text-gray-200">|</span>
<span className="text-gray-400" title="Operations append to this table">writes to</span>
<button
onClick={copyTable}
onMouseEnter={() => setShow(true)}
onMouseLeave={() => setShow(false)}
className={`font-mono px-1.5 py-0.5 rounded border hover:bg-gray-50 ${
info.exists ? 'text-gray-700 border-gray-200' : 'text-amber-700 border-amber-200 bg-amber-50'
}`}
title={info.exists ? 'Click to copy table name' : 'Table does not exist yet'}
>
{copied ? 'copied!' : info.fc_table}
</button>
<span className="text-gray-400 font-mono">
{info.exists ? `${fmt(info.rows)} rows` : 'not created'}
</span>
{showInfo && (
<div className="absolute top-9 left-0 z-30 bg-white border border-gray-200 rounded shadow-lg p-3 text-xs min-w-[260px]">
<div className="text-gray-400 uppercase tracking-wide mb-2" style={{ fontSize: '10px' }}>Write target</div>
<div className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1">
<span className="text-gray-400">table</span>
<span className="font-mono text-gray-700">{info.fc_table}</span>
<span className="text-gray-400">reads from</span>
<span className="font-mono text-gray-700">{info.source}</span>
<span className="text-gray-400">total rows</span>
<span className="font-mono text-gray-700">{fmt(info.rows)}</span>
</div>
{info.by_iter?.length > 0 && (
<>
<div className="text-gray-400 uppercase tracking-wide mt-3 mb-1" style={{ fontSize: '10px' }}>Rows by iter</div>
<table className="w-full">
<tbody>
{info.by_iter.map(r => (
<tr key={r.pf_iter}>
<td className="text-gray-500 capitalize pr-3">{r.pf_iter}</td>
<td className="text-right font-mono text-gray-700">{fmt(r.n)}</td>
</tr>
))}
</tbody>
</table>
</>
)}
</div>
)}
</>
)}
</> </>
)} )}
<div className="ml-auto flex items-center gap-2"> <div className="ml-auto">
{user && (
<>
<span className="text-gray-500" title={`Signed in as ${user.username}`}>
{user.display_name || user.username}
</span>
<button
onClick={logout}
className="text-xs text-gray-500 hover:text-gray-700 border border-gray-200 px-2 py-0.5 rounded"
title="Sign out"
>
Sign out
</button>
</>
)}
<button <button
onClick={() => setDark(d => !d)} onClick={() => setDark(d => !d)}
className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100" className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100"

View File

@ -32,16 +32,11 @@ function roundRect(ctx, x, y, w, h, r, fill, stroke) {
if (stroke) ctx.stroke() if (stroke) ctx.stroke()
} }
export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDays = 0, type = 'baseline' }) { export default function Timeline({ dateFrom, dateTo, offsetYr, offsetMo, type = 'baseline' }) {
const canvasRef = useRef(null) const canvasRef = useRef(null)
// Months and days are kept apart because they are not interchangeable: a month const offsetMoTotal = (offsetYr || 0) * 12 + (offsetMo || 0)
// shift lands on the same day of a different month, a day shift can cross a const twoBands = type === 'baseline' && offsetMoTotal > 0
// month boundary. The preview adds months first, then days, as Postgres does.
const offsetMoTotal = offsetMonths || 0
const offsetDayTotal = offsetDays || 0
const shifted = offsetMoTotal !== 0 || offsetDayTotal !== 0
const twoBands = type === 'baseline' && shifted
const canvasH = twoBands ? 90 : 52 const canvasH = twoBands ? 90 : 52
useEffect(() => { useEffect(() => {
@ -66,9 +61,8 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
const srcEnd = parseDate(dateTo) const srcEnd = parseDate(dateTo)
if (!srcStart || !srcEnd || isNaN(srcStart) || isNaN(srcEnd)) return if (!srcStart || !srcEnd || isNaN(srcStart) || isNaN(srcEnd)) return
const addDays = (d, n) => { const x = new Date(d); x.setDate(x.getDate() + n); return x } const projStart = addMonths(srcStart, offsetMoTotal)
const projStart = addDays(addMonths(srcStart, offsetMoTotal), offsetDayTotal) const projEnd = addMonths(srcEnd, offsetMoTotal)
const projEnd = addDays(addMonths(srcEnd, offsetMoTotal), offsetDayTotal)
const winStart = addMonths(srcStart, -1) const winStart = addMonths(srcStart, -1)
const winEnd = addMonths(twoBands ? projEnd : srcEnd, 1) const winEnd = addMonths(twoBands ? projEnd : srcEnd, 1)
@ -153,14 +147,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
ctx.lineTo(px1 - 4, arrowY + 4) ctx.lineTo(px1 - 4, arrowY + 4)
ctx.closePath() ctx.closePath()
ctx.fill() ctx.fill()
const yrs = Math.trunc(offsetMoTotal / 12) const offsetLabel = '+' + (offsetYr ? offsetYr + 'yr ' : '') + (offsetMo ? offsetMo + 'mo' : '')
const mos = offsetMoTotal % 12
const sign = (offsetMoTotal + offsetDayTotal) < 0 ? '' : '+'
const offsetLabel = sign + [
yrs ? `${yrs}yr` : '',
mos ? `${mos}mo` : '',
offsetDayTotal ? `${offsetDayTotal}d` : '',
].filter(Boolean).join(' ')
ctx.fillStyle = '#64748b' ctx.fillStyle = '#64748b'
ctx.font = '9px system-ui' ctx.font = '9px system-ui'
ctx.textAlign = 'center' ctx.textAlign = 'center'
@ -169,7 +156,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
} }
raf = requestAnimationFrame(draw) raf = requestAnimationFrame(draw)
return () => cancelAnimationFrame(raf) return () => cancelAnimationFrame(raf)
}, [dateFrom, dateTo, offsetMoTotal, offsetDayTotal, type, twoBands, canvasH]) }, [dateFrom, dateTo, offsetYr, offsetMo, type, twoBands, canvasH])
return <canvas ref={canvasRef} height={canvasH} style={{ width: '100%', display: 'block' }} /> return <canvas ref={canvasRef} height={canvasH} style={{ width: '100%', display: 'block' }} />
} }

View File

@ -1,27 +1,13 @@
import { StrictMode } from 'react' import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client' import { createRoot } from 'react-dom/client'
import { ThemeProvider } from './theme.jsx' import { ThemeProvider } from './theme.jsx'
import { AuthProvider } from './auth.jsx'
import useAuth from './auth.jsx'
import Login from './views/Login.jsx'
import './index.css' import './index.css'
import App from './App.jsx' import App from './App.jsx'
// App is mounted only once there is a session its load effects call /api
// straight away, and mounting it logged-out would just fire a burst of 401s.
function Gate() {
const { user, checking } = useAuth()
if (checking) return <div className="flex items-center justify-center h-screen text-sm text-gray-400">Loading</div>
if (!user) return <Login />
return <App />
}
createRoot(document.getElementById('root')).render( createRoot(document.getElementById('root')).render(
<StrictMode> <StrictMode>
<ThemeProvider> <ThemeProvider>
<AuthProvider> <App />
<Gate />
</AuthProvider>
</ThemeProvider> </ThemeProvider>
</StrictMode>, </StrictMode>,
) )

View File

@ -1,6 +1,5 @@
import { useState, useEffect } from 'react' import { useState, useEffect } from 'react'
import Timeline from '../components/Timeline.jsx' import Timeline from '../components/Timeline.jsx'
import useAuth from '../auth.jsx'
const OPERATORS = ['BETWEEN', '=', '!=', 'IN', 'NOT IN', 'IS NULL', 'IS NOT NULL'] const OPERATORS = ['BETWEEN', '=', '!=', 'IN', 'NOT IN', 'IS NULL', 'IS NOT NULL']
@ -49,28 +48,11 @@ function getDateRange(groups) {
return null return null
} }
// The offset is stored and sent as a Postgres interval, so it is typed as one -- function parseOffset(offsetStr) {
// "4 months", "1 year", "-90 days". This only parses it far enough to draw the if (!offsetStr || offsetStr === '0 days') return { yr: 0, mo: 0 }
// timeline preview; Postgres remains the authority on what is valid, and the const yr = parseInt(offsetStr.match(/(\d+)\s+year/)?.[1] || 0)
// server rejects anything it will not accept. const mo = parseInt(offsetStr.match(/(\d+)\s+month/)?.[1] || 0)
// return { yr, mo }
// It replaced a pair of year/month number spinners, which could not express days
// at all and were clamped at zero, so a segment could only ever be shifted
// forward in whole months.
export function parseInterval(str) {
let months = 0, days = 0
if (!str) return { months, days }
const re = /([+-]?\d+(?:\.\d+)?)\s*(years?|yrs?|y|months?|mons?|mo|weeks?|wks?|w|days?|d)\b/gi
for (const [, n, unit] of str.matchAll(re)) {
const v = parseFloat(n)
const u = unit.toLowerCase()
if (u.startsWith('y')) months += v * 12
else if (u.startsWith('mo') || u === 'mons' || u === 'mon') months += v
else if (u.startsWith('w')) days += v * 7
else if (u.startsWith('d')) days += v
else months += v // 'm' alone reads as months here
}
return { months, days }
} }
function emptyCondition(cols) { function emptyCondition(cols) {
@ -87,7 +69,6 @@ function normalizeFilters(stored) {
} }
export default function Baseline({ sources = [], sourceId, versions = [], versionId, setVersionId, refreshVersions }) { export default function Baseline({ sources = [], sourceId, versions = [], versionId, setVersionId, refreshVersions }) {
const { user: me } = useAuth()
const [filterCols, setFilterCols] = useState([]) const [filterCols, setFilterCols] = useState([])
const [log, setLog] = useState([]) const [log, setLog] = useState([])
@ -99,15 +80,13 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
// segment form // segment form
const [segType, setSegType] = useState('baseline') const [segType, setSegType] = useState('baseline')
const [description, setDescription] = useState('')
const [filters, setFilters] = useState([]) // [[cond,...], [cond,...]] const [filters, setFilters] = useState([]) // [[cond,...], [cond,...]]
const [useRaw, setUseRaw] = useState(false) const [useRaw, setUseRaw] = useState(false)
const [rawSql, setRawSql] = useState('') const [rawSql, setRawSql] = useState('')
const [offset, setOffset] = useState('0 days') const [offsetYr, setOffsetYr] = useState(0)
const [offsetMo, setOffsetMo] = useState(0)
const [segNote, setSegNote] = useState('') const [segNote, setSegNote] = useState('')
// Presentation, not definition: what the segment counts toward and how it is
// labelled in the pivot. Safe to set at any time, unlike its filters.
const [segBucket, setSegBucket] = useState('')
const [segLabel, setSegLabel] = useState('')
const [submitting, setSubmitting] = useState(false) const [submitting, setSubmitting] = useState(false)
const [editingLogId, setEditingLogId] = useState(null) const [editingLogId, setEditingLogId] = useState(null)
const [showAddForm, setShowAddForm] = useState(false) const [showAddForm, setShowAddForm] = useState(false)
@ -119,11 +98,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
useEffect(() => { useEffect(() => {
if (!sourceId) return if (!sourceId) return
fetch(`/api/sources/${sourceId}/cols`).then(r => r.json()).then(cols => { fetch(`/api/sources/${sourceId}/cols`).then(r => r.json()).then(cols => {
// What a load is filtered by and what the pivot groups by are separate const fc = cols.filter(c => c.role === 'date' || c.role === 'filter')
// concerns: a dimension is exactly the sort of thing a segment is cut on
// (sseas, channel_new), and forcing it to role 'filter' to get it here
// would cost it its place on the pivot. Only measures are excluded.
const fc = cols.filter(c => ['date', 'filter', 'dimension'].includes(c.role))
setFilterCols(fc) setFilterCols(fc)
setFilters(fc.length > 0 ? [emptyGroup(fc)] : []) setFilters(fc.length > 0 ? [emptyGroup(fc)] : [])
}) })
@ -134,65 +109,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
loadLog() loadLog()
}, [versionId]) }, [versionId])
// A segment's banner: what it counts toward, independent of pf_iter. Held
// locally while typing so the field does not fight the fetched value, and
// written on blur.
const [buckets, setBuckets] = useState({})
// Buckets already in use on this version, for the datalist. There is no stored
// bucket order any more: the order is whatever the typed text sorts as, so a
// bucket is named "02 - Forecast" and that is the whole mechanism.
const [bucketsInUse, setBucketsInUse] = useState([])
// Label and bucket are presentation, not definition: they change what the pivot
// shows and what the segment counts toward, never which rows were loaded. So
// they stay editable after adjustments exist, unlike the filters and offset,
// where an edit would silently recalibrate scales sized against the old rows.
//
// Both are only read at load time, hence the reload in the confirmation: the
// label is part of the aggregated row the pivot holds, not something it can
// re-derive in place.
// Same rule the server enforces: your own entries, or an admin's. A segment's
// label and bucket name the pivot's columns for everyone in the version, so
// they are not the private annotation they look like.
const canEdit = (entry) => !!me && (me.is_admin || entry.pf_user === me.username)
async function saveLogField(entry, field, value) {
const next = value.trim()
if (next === (entry[field] || '')) return
try {
const res = await fetch(`/api/log/${entry.id}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ [field]: next }),
})
if (!res.ok) { const d = await res.json(); flash(d.error, 'error'); return }
loadLog()
flash(next ? `Saved — reload the Forecast view to see it` : 'Cleared')
} catch (err) { flash(err.message, 'error') }
}
// Column widths read off the content rather than guessed. ch is the width of a
// '0', so for proportional text it runs slightly generous -- which is what is
// wanted for an input you are about to type a longer name into. The floors keep
// an empty table from collapsing its headers; the ceilings keep one long note
// from pushing the numbers off the side.
function widthCh(values, min, max) {
const longest = values.reduce((n, v) => Math.max(n, String(v || '').length), 0)
return `${Math.min(max, Math.max(min, longest + 2))}ch`
}
const labelW = widthCh(log.map(e => e.label || e.tag || e.note), 18, 34)
const bucketW = widthCh(log.map(e => e.bucket), 16, 24)
const noteW = widthCh(log.map(e => e.note), 22, 44)
function loadLog() { function loadLog() {
fetch(`/api/versions/${versionId}/log`).then(r => r.json()).then(data => { fetch(`/api/versions/${versionId}/log`).then(r => r.json()).then(data => {
setLog(data.filter(e => e.operation === 'baseline' || e.operation === 'reference')) setLog(data.filter(e => e.operation === 'baseline' || e.operation === 'reference'))
setHasForecastOps(data.some(e => ['scale', 'recode', 'clone'].includes(e.operation))) setHasForecastOps(data.some(e => ['scale', 'recode', 'clone'].includes(e.operation)))
// Every bucket in use, adjustments included: typing one on a new segment
// should offer the ones already there rather than inviting a near-miss
// spelling, which would silently split the column in two.
setBucketsInUse([...new Set(data.map(e => (e.bucket || '').trim()).filter(Boolean))].sort())
}) })
} }
@ -224,14 +144,13 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
const clause = useRaw ? rawSql.trim() : buildFilterClause(filters) const clause = useRaw ? rawSql.trim() : buildFilterClause(filters)
if (!clause) { flash(useRaw ? 'Enter a WHERE clause' : 'Add at least one filter', 'error'); return } if (!clause) { flash(useRaw ? 'Enter a WHERE clause' : 'Add at least one filter', 'error'); return }
const isRef = segType === 'reference' const isRef = segType === 'reference'
const offsetStr = offset.trim() || '0 days' const offsetStr = [offsetYr > 0 ? `${offsetYr} year` : '', offsetMo > 0 ? `${offsetMo} month` : ''].filter(Boolean).join(' ') || '0 days'
const endpoint = isRef ? 'reference' : 'baseline' const endpoint = isRef ? 'reference' : 'baseline'
const body = { const body = {
where_clause: clause, where_clause: clause,
note: segNote, pf_user: 'admin',
note: description || segNote,
date_offset: offsetStr, date_offset: offsetStr,
label: segLabel.trim(),
bucket: segBucket.trim(),
...(useRaw ? { raw_where: clause } : { filters }), ...(useRaw ? { raw_where: clause } : { filters }),
} }
setSubmitting(true) setSubmitting(true)
@ -267,9 +186,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
const params = entry.params || {} const params = entry.params || {}
setSegType(entry.operation) setSegType(entry.operation)
setSegNote(entry.note || '') setSegNote(entry.note || '')
setSegLabel(entry.label || '') setDescription('')
setSegBucket(entry.bucket || '') const off = parseOffset(params.date_offset)
setOffset(params.date_offset || '0 days') setOffsetYr(off.yr)
setOffsetMo(off.mo)
const groups = normalizeFilters(params.filters) const groups = normalizeFilters(params.filters)
if (groups) { if (groups) {
setUseRaw(false) setUseRaw(false)
@ -295,9 +215,8 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
function cancelEdit() { function cancelEdit() {
setEditingLogId(null) setEditingLogId(null)
setShowAddForm(false) setShowAddForm(false)
setDescription('')
setSegNote('') setSegNote('')
setSegLabel('')
setSegBucket('')
setOffsetYr(0) setOffsetYr(0)
setOffsetMo(0) setOffsetMo(0)
setUseRaw(false) setUseRaw(false)
@ -323,7 +242,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
async function closeVersion() { async function closeVersion() {
const res = await fetch(`/api/versions/${versionId}/close`, { const res = await fetch(`/api/versions/${versionId}/close`, {
method: 'POST', headers: { 'Content-Type': 'application/json' }, method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({}) body: JSON.stringify({ pf_user: 'admin' })
}) })
const data = await res.json() const data = await res.json()
if (!res.ok) { flash(data.error, 'error'); return } if (!res.ok) { flash(data.error, 'error'); return }
@ -354,33 +273,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
setTimeout(() => setMsg(null), 3000) setTimeout(() => setMsg(null), 3000)
} }
// The names a row falls back to when nobody has named it. Blank means "use the
// built-in", so these save on blur like the segment fields and an empty box is
// a meaningful value rather than a missing one.
async function saveVersionName(field, value) {
const next = value.trim()
if (next === (selectedVersion?.[field] || '')) return
try {
const res = await fetch(`/api/versions/${versionId}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ [field]: next }),
})
if (!res.ok) { const d = await res.json(); flash(d.error, 'error'); return }
await refreshVersions(sourceId)
flash('Saved — reload the Forecast view to see it')
} catch (err) { flash(err.message, 'error') }
}
const selectedVersion = versions.find(v => String(v.id) === versionId) const selectedVersion = versions.find(v => String(v.id) === versionId)
return ( return (
<div className="h-full overflow-y-auto bg-gray-50"> <div className="h-full overflow-y-auto bg-gray-50">
{/* No page-wide cap and no stretching: at max-w-4xl (896px) the eleven-column <div className="p-4 flex flex-col gap-4 max-w-4xl">
segment table always had something smashed, and uncapped it ran to the
window. items-start makes each block as wide as its own content needs,
which for the table is the measured column widths below. */}
<div className="p-4 flex flex-col gap-4 items-start max-w-full">
{msg && ( {msg && (
<div className={`px-3 py-2 text-xs rounded font-medium ${msg.type === 'error' ? 'bg-red-50 text-red-700' : 'bg-green-50 text-green-700'}`}> <div className={`px-3 py-2 text-xs rounded font-medium ${msg.type === 'error' ? 'bg-red-50 text-red-700' : 'bg-green-50 text-green-700'}`}>
@ -404,31 +301,6 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
)} )}
</div> </div>
{/* Fallback names. Not part of any segment -- they are what the pivot shows
for rows nobody has named, so they belong to the version rather than to
a log entry. Blank falls back to the built-in in sql_generator's
DISPLAY DEFAULTS block. */}
{versionId && (
<div className="bg-white border border-gray-200 rounded p-3 flex items-end gap-3 flex-wrap">
<span className="text-xs text-gray-500 uppercase tracking-wide w-full">Fallback names</span>
{[
['adjustment_segment', 'Adjustment segment', '99 - Adjustments'],
['adjustment_bucket', 'Adjustment bucket', '04 - Forecast'],
['unlabeled_load', 'Unlabeled load', 'Unlabeled'],
].map(([field, label, builtin]) => (
<div key={field} className="flex flex-col gap-1">
<label className="text-xs text-gray-500">{label}</label>
<input
key={`${field}-${versionId}-${selectedVersion?.[field] || ''}`}
defaultValue={selectedVersion?.[field] || ''}
onBlur={e => saveVersionName(field, e.target.value)}
placeholder={builtin}
className="border border-gray-200 rounded px-2 py-1 text-sm w-48" />
</div>
))}
</div>
)}
{showNewVersion && ( {showNewVersion && (
<div className="bg-white border border-gray-200 rounded p-3 flex flex-col gap-3"> <div className="bg-white border border-gray-200 rounded p-3 flex flex-col gap-3">
<div className="flex items-end gap-3"> <div className="flex items-end gap-3">
@ -454,27 +326,17 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
{versionId && <> {versionId && <>
{/* Segments loaded */} {/* Segments loaded */}
<div className="bg-white border border-gray-200 rounded max-w-full"> <div className="bg-white border border-gray-200 rounded">
<div className="px-3 py-2 border-b border-gray-100 text-xs font-medium text-gray-500 uppercase tracking-wide flex items-center justify-between"> <div className="px-3 py-2 border-b border-gray-100 text-xs font-medium text-gray-500 uppercase tracking-wide flex items-center justify-between">
<span>Segments loaded</span> <span>Segments loaded</span>
<button onClick={clearBaseline} className="text-red-400 hover:text-red-600 text-xs normal-case font-normal">Clear all baseline</button> <button onClick={clearBaseline} className="text-red-400 hover:text-red-600 text-xs normal-case font-normal">Clear all baseline</button>
</div> </div>
<datalist id="pf-bucket-options"> <table className="w-full text-xs">
{[...new Set([...bucketsInUse,
'01 - Prior Prior Year', '02 - Prior Year',
'03 - Plan', '04 - Forecast'])].map(b => (
<option key={b} value={b} />
))}
</datalist>
<table className="text-xs">
<thead className="bg-gray-50"> <thead className="bg-gray-50">
<tr className="text-left text-gray-400 border-b border-gray-100"> <tr className="text-left text-gray-400 border-b border-gray-100">
<th className="px-3 py-1.5 font-medium w-6"></th> <th className="px-3 py-1.5 font-medium w-6"></th>
<th className="px-3 py-1.5 font-medium">#</th> <th className="px-3 py-1.5 font-medium">#</th>
<th className="px-3 py-1.5 font-medium w-20">kind</th> <th className="px-3 py-1.5 font-medium">note</th>
<th className="px-3 py-1.5 font-medium" style={{ width: labelW }}>label</th>
<th className="px-3 py-1.5 font-medium" style={{ width: noteW }}>note</th>
<th className="px-3 py-1.5 font-medium" style={{ width: bucketW }}>counts toward</th>
<th className="px-3 py-1.5 font-medium text-right">rows</th> <th className="px-3 py-1.5 font-medium text-right">rows</th>
<th className="px-3 py-1.5 font-medium text-right">{log[0]?.value_col || 'value'}</th> <th className="px-3 py-1.5 font-medium text-right">{log[0]?.value_col || 'value'}</th>
<th className="px-3 py-1.5 font-medium">by</th> <th className="px-3 py-1.5 font-medium">by</th>
@ -484,11 +346,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
</thead> </thead>
<tbody> <tbody>
{log.length === 0 && ( {log.length === 0 && (
<tr><td colSpan={11} className="px-3 py-3 text-gray-300 italic">No segments loaded yet</td></tr> <tr><td colSpan={8} className="px-3 py-3 text-gray-300 italic">No segments loaded yet</td></tr>
)} )}
{!showAddForm && !editingLogId && ( {!showAddForm && !editingLogId && (
<tr className="border-t border-gray-100"> <tr className="border-t border-gray-100">
<td colSpan={11} className="p-0"> <td colSpan={8} className="p-0">
<button <button
onClick={() => setShowAddForm(true)} onClick={() => setShowAddForm(true)}
className="w-full px-3 py-2 text-xs text-blue-600 hover:bg-blue-50 text-left font-medium" className="w-full px-3 py-2 text-xs text-blue-600 hover:bg-blue-50 text-left font-medium"
@ -510,49 +372,11 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
> >
<td className="px-3 py-2 text-gray-400 w-6"><span className="text-gray-300 text-xs">{isOpen ? '▾' : '▸'}</span></td> <td className="px-3 py-2 text-gray-400 w-6"><span className="text-gray-300 text-xs">{isOpen ? '▾' : '▸'}</span></td>
<td className="px-3 py-2 text-gray-400">{log.length - i}</td> <td className="px-3 py-2 text-gray-400">{log.length - i}</td>
{/* The operation badge gets its own column. Sharing one with
the note put "reference" hard against "YTD Sales" as soon
as the note column lost width to label and bucket. */}
<td className="px-3 py-2"> <td className="px-3 py-2">
<span className={`inline-block px-1.5 py-0.5 rounded text-xs font-medium ${entry.operation === 'reference' ? 'bg-purple-50 text-purple-600' : 'bg-blue-50 text-blue-600'}`}> <span className={`inline-block mr-2 px-1.5 py-0.5 rounded text-xs font-medium ${entry.operation === 'reference' ? 'bg-purple-50 text-purple-600' : 'bg-blue-50 text-blue-600'}`}>
{entry.operation} {entry.operation}
</span> </span>
</td> {entry.note || <span className="text-gray-300"></span>}
<td className="px-3 py-2" onClick={e => e.stopPropagation()}>
<input
defaultValue={entry.label || ''}
key={`label-${entry.id}-${entry.label || ''}`}
onBlur={e => saveLogField(entry, 'label', e.target.value)}
readOnly={!canEdit(entry)}
title={canEdit(entry) ? '' : `${entry.pf_user || 'Another account'} made this segment`}
placeholder={entry.tag || entry.note || '—'}
className="w-full border border-transparent hover:border-gray-200
focus:border-blue-400 rounded px-1 py-0.5 text-xs
focus:outline-none bg-transparent" />
</td>
{/* One line, clipped against the measured width above. The
note is provenance and can run long, so left to itself it
wrapped and pushed every row to two or three lines.
Expanding the row shows it in full. The cap is on the div,
not the cell: a max-width on a cell in an auto-layout table
is only a hint, and the column can still collapse to
min-content or grow past it. */}
<td className="px-3 py-2">
{entry.note
? <div className="truncate" style={{ maxWidth: noteW }} title={entry.note}>{entry.note}</div>
: <span className="text-gray-300"></span>}
</td>
<td className="px-3 py-2" onClick={e => e.stopPropagation()}>
<input
value={buckets[entry.id] ?? entry.bucket ?? ''}
list="pf-bucket-options"
onChange={e => setBuckets(b => ({ ...b, [entry.id]: e.target.value }))}
onBlur={e => saveLogField(entry, 'bucket', e.target.value)}
readOnly={!canEdit(entry)}
title={canEdit(entry) ? '' : `${entry.pf_user || 'Another account'} made this segment`}
placeholder="—"
className="w-full border border-transparent hover:border-gray-200 focus:border-blue-400
rounded px-1 py-0.5 text-xs focus:outline-none bg-transparent" />
</td> </td>
<td className="px-3 py-2 text-right text-gray-700 font-mono"> <td className="px-3 py-2 text-right text-gray-700 font-mono">
{entry.row_count != null ? entry.row_count.toLocaleString() : '—'} {entry.row_count != null ? entry.row_count.toLocaleString() : '—'}
@ -571,7 +395,7 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
</tr> </tr>
{isOpen && ( {isOpen && (
<tr key={`${entry.id}-detail`} className="bg-blue-50 border-t border-blue-100"> <tr key={`${entry.id}-detail`} className="bg-blue-50 border-t border-blue-100">
<td colSpan={9} className="px-2 py-2"> <td colSpan={6} className="px-2 py-2">
<div className="bg-white border border-gray-200 rounded"> <div className="bg-white border border-gray-200 rounded">
<SegmentForm mode="view" {...view} filterCols={filterCols} /> <SegmentForm mode="view" {...view} filterCols={filterCols} />
</div> </div>
@ -610,10 +434,10 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
filters={filters} setFilters={setFilters} filters={filters} setFilters={setFilters}
useRaw={useRaw} setUseRaw={setUseRaw} useRaw={useRaw} setUseRaw={setUseRaw}
rawSql={rawSql} setRawSql={setRawSql} rawSql={rawSql} setRawSql={setRawSql}
description={description} setDescription={setDescription}
segNote={segNote} setSegNote={setSegNote} segNote={segNote} setSegNote={setSegNote}
segBucket={segBucket} setSegBucket={setSegBucket} offsetYr={offsetYr} setOffsetYr={setOffsetYr}
segLabel={segLabel} setSegLabel={setSegLabel} offsetMo={offsetMo} setOffsetMo={setOffsetMo}
offset={offset} setOffset={setOffset}
filterCols={filterCols} filterCols={filterCols}
onSubmit={loadSegment} onSubmit={loadSegment}
submitting={submitting} submitting={submitting}
@ -632,16 +456,17 @@ export default function Baseline({ sources = [], sourceId, versions = [], versio
// derive view-mode props for a saved segment // derive view-mode props for a saved segment
function segmentValuesFor(entry, filterCols) { function segmentValuesFor(entry, filterCols) {
const params = entry.params || {} const params = entry.params || {}
const off = parseOffset(params.date_offset)
const groups = normalizeFilters(params.filters) const groups = normalizeFilters(params.filters)
return { return {
segType: entry.operation === 'reference' ? 'reference' : 'baseline', segType: entry.operation === 'reference' ? 'reference' : 'baseline',
filters: groups || (filterCols.length > 0 ? [emptyGroup(filterCols)] : []), filters: groups || (filterCols.length > 0 ? [emptyGroup(filterCols)] : []),
useRaw: !groups && !!params.where_clause, useRaw: !groups && !!params.where_clause,
rawSql: params.where_clause || '', rawSql: params.where_clause || '',
description: '',
segNote: entry.note || '', segNote: entry.note || '',
segBucket: entry.bucket || '', offsetYr: off.yr,
segLabel: entry.label || '', offsetMo: off.mo,
offset: params.date_offset || '0 days',
} }
} }
@ -651,10 +476,10 @@ function SegmentForm({
filters, setFilters, filters, setFilters,
useRaw, setUseRaw, useRaw, setUseRaw,
rawSql, setRawSql, rawSql, setRawSql,
description, setDescription,
segNote, setSegNote, segNote, setSegNote,
segBucket, setSegBucket, offsetYr, setOffsetYr,
segLabel, setSegLabel, offsetMo, setOffsetMo,
offset, setOffset,
filterCols, filterCols,
onSubmit, onSubmit,
submitting, submitting,
@ -724,6 +549,14 @@ function SegmentForm({
</div> </div>
</div> </div>
{/* Description (edit only) */}
{mode === 'edit' && (
<div className="flex items-center gap-3">
<label className="text-xs text-gray-500 w-28 shrink-0">Description</label>
<input value={description} onChange={e => setDescription(e.target.value)} placeholder="e.g. FY25 actuals +1yr" className="border border-gray-200 rounded px-2 py-1.5 text-sm flex-1 max-w-sm" />
</div>
)}
{/* Filters */} {/* Filters */}
<div> <div>
<div className="flex items-center justify-between mb-2"> <div className="flex items-center justify-between mb-2">
@ -823,19 +656,10 @@ function SegmentForm({
<div className="flex items-center gap-3"> <div className="flex items-center gap-3">
<label className="text-xs text-gray-500 w-28 shrink-0">Date offset</label> <label className="text-xs text-gray-500 w-28 shrink-0">Date offset</label>
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<input disabled={disabled} value={offset} list="pf-offset-options" <input disabled={disabled} type="number" value={offsetYr} min={0} onChange={e => setOffsetYr(parseInt(e.target.value) || 0)} className={`${baseInp} text-sm w-16 text-center`} />
onChange={e => setOffset(e.target.value)} <span className="text-xs text-gray-500">yr</span>
placeholder="0 days" <input disabled={disabled} type="number" value={offsetMo} min={0} max={11} onChange={e => setOffsetMo(parseInt(e.target.value) || 0)} className={`${baseInp} text-sm w-16 text-center`} />
className={`${baseInp} text-sm w-32`} /> <span className="text-xs text-gray-500">mo</span>
<datalist id="pf-offset-options">
<option value="12 months" />
<option value="1 year" />
<option value="4 months" />
<option value="-90 days" />
<option value="-12 months" />
<option value="0 days" />
</datalist>
<span className="text-xs text-gray-400">any Postgres interval</span>
</div> </div>
</div> </div>
@ -846,32 +670,16 @@ function SegmentForm({
<Timeline <Timeline
dateFrom={dateRange.from} dateFrom={dateRange.from}
dateTo={dateRange.to} dateTo={dateRange.to}
offsetMonths={parseInterval(offset).months} offsetYr={offsetYr}
offsetDays={parseInterval(offset).days} offsetMo={offsetMo}
type={segType} type={segType}
/> />
</div> </div>
</div> </div>
)} )}
{/* Label, bucket, note + submit. {/* Note + submit */}
Label and bucket are presentation: the label is what the pivot shows for <div className="flex items-end gap-3">
this segment, the bucket is what it counts toward. Both are free text and
both sort by what is typed, so a leading "01 - " is how ordering is set
which is why they belong here, at the point the segment is defined, as
well as being editable in the list afterwards. */}
<div className="flex items-end gap-3 flex-wrap">
<div className="flex flex-col gap-1 max-w-xs">
<label className="text-xs text-gray-500">Label</label>
<input disabled={disabled} value={segLabel} onChange={e => setSegLabel(e.target.value)}
placeholder="defaults to the note" className={`${baseInp} text-sm py-1.5`} />
</div>
<div className="flex flex-col gap-1 max-w-xs">
<label className="text-xs text-gray-500">Counts toward</label>
<input disabled={disabled} value={segBucket} onChange={e => setSegBucket(e.target.value)}
list="pf-bucket-options" placeholder="e.g. 02 - Forecast"
className={`${baseInp} text-sm py-1.5`} />
</div>
<div className="flex flex-col gap-1 flex-1 max-w-xs"> <div className="flex flex-col gap-1 flex-1 max-w-xs">
<label className="text-xs text-gray-500">Note</label> <label className="text-xs text-gray-500">Note</label>
<input disabled={disabled} value={segNote} onChange={e => setSegNote(e.target.value)} placeholder="optional" className={`${baseInp} text-sm py-1.5`} /> <input disabled={disabled} value={segNote} onChange={e => setSegNote(e.target.value)} placeholder="optional" className={`${baseInp} text-sm py-1.5`} />

File diff suppressed because it is too large Load Diff

View File

@ -1,68 +0,0 @@
import { useState } from 'react'
import useAuth from '../auth.jsx'
export default function Login() {
const { login } = useAuth()
const [username, setUsername] = useState('')
const [password, setPassword] = useState('')
const [error, setError] = useState('')
const [busy, setBusy] = useState(false)
async function submit(e) {
e.preventDefault()
setError(''); setBusy(true)
try {
await login(username.trim(), password)
} catch (err) {
setError(err.message)
setPassword('')
} finally {
setBusy(false)
}
}
return (
<div className="flex items-center justify-center h-screen w-full text-sm">
<form onSubmit={submit} className="bg-white border border-gray-200 rounded p-6 w-80 flex flex-col gap-4">
<div>
<div className="text-base font-medium">Pivot Forecast</div>
<div className="text-xs text-gray-500 mt-0.5">Sign in to continue</div>
</div>
<label className="flex flex-col gap-1">
<span className="text-xs text-gray-500">Username</span>
<input
value={username}
onChange={e => setUsername(e.target.value)}
autoFocus
autoComplete="username"
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
/>
</label>
<label className="flex flex-col gap-1">
<span className="text-xs text-gray-500">Password</span>
<input
type="password"
value={password}
onChange={e => setPassword(e.target.value)}
autoComplete="current-password"
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
/>
</label>
{error && (
<div className="px-3 py-2 text-xs rounded font-medium bg-red-50 text-red-700">{error}</div>
)}
<button
type="submit"
disabled={busy || !username.trim() || !password}
className="bg-blue-600 text-white text-xs px-3 py-2 rounded hover:bg-blue-700 disabled:opacity-50"
>
{busy ? 'Signing in…' : 'Sign in'}
</button>
</form>
</div>
)
}

View File

@ -25,7 +25,6 @@ export default function Setup({ refreshSources }) {
const [sqlStatus, setSqlStatus] = useState({}) // sourceId -> bool const [sqlStatus, setSqlStatus] = useState({}) // sourceId -> bool
const [saving, setSaving] = useState(false) const [saving, setSaving] = useState(false)
const [generating, setGenerating] = useState(false) const [generating, setGenerating] = useState(false)
const [refreshingDims, setRefreshingDims] = useState(false)
const [msg, setMsg] = useState(null) const [msg, setMsg] = useState(null)
const [dimPeriodCols, setDimPeriodCols] = useState([]) const [dimPeriodCols, setDimPeriodCols] = useState([])
const [openPeriodIdx, setOpenPeriodIdx] = useState(null) const [openPeriodIdx, setOpenPeriodIdx] = useState(null)
@ -154,35 +153,6 @@ export default function Setup({ refreshSources }) {
} }
} }
// Rebuild every keyed dim_group's member list from the source. Deliberate rather
// than automatic: it reads the whole source, which for a view over a transaction
// table is millions of rows, and the answer only changes when the catalogue does.
async function refreshDimMembers() {
const groups = [...new Set(cols.filter(c => c.dim_group && c.is_key).map(c => c.dim_group))]
if (groups.length === 0) {
flash('No dim_group has an is_key column, so there is nothing to build a list from', 'error')
return
}
setRefreshingDims(true)
try {
const done = []
for (const g of groups) {
const res = await fetch(`/api/sources/${selectedSource.id}/dim/${encodeURIComponent(g)}/refresh`,
{ method: 'POST' })
const data = await res.json()
if (!res.ok) { flash(`${g}: ${data.error}`, 'error'); return }
done.push(`${g}: ${data.members.toLocaleString()} members`
+ (data.no_longer_in_source ? `, ${data.no_longer_in_source} no longer in source` : '')
+ ` (${(data.ms / 1000).toFixed(1)}s)`)
}
flash(done.join(' · '))
} catch (err) {
flash(err.message, 'error')
} finally {
setRefreshingDims(false)
}
}
async function deleteSource(id, e) { async function deleteSource(id, e) {
e.stopPropagation() e.stopPropagation()
if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return
@ -203,11 +173,6 @@ export default function Setup({ refreshSources }) {
const registeredKeys = new Set(sources.map(s => `${s.schema}.${s.tname}`)) const registeredKeys = new Set(sources.map(s => `${s.schema}.${s.tname}`))
// display grain must match grainOf() in lib/sql_generator.js
const grainCols = editedCols
.filter(c => c.in_grain && (c.role === 'dimension' || c.role === 'date'))
.map(c => c.cname)
return ( return (
<div className="h-full flex overflow-hidden text-sm"> <div className="h-full flex overflow-hidden text-sm">
@ -313,11 +278,6 @@ export default function Setup({ refreshSources }) {
<div className="px-3 py-2 border-b border-gray-100 flex items-center justify-between shrink-0"> <div className="px-3 py-2 border-b border-gray-100 flex items-center justify-between shrink-0">
<span className="text-xs font-medium text-gray-500 uppercase tracking-wide"> <span className="text-xs font-medium text-gray-500 uppercase tracking-wide">
Col Meta <span className="text-gray-700 normal-case">{selectedSource.schema}.{selectedSource.tname}</span> Col Meta <span className="text-gray-700 normal-case">{selectedSource.schema}.{selectedSource.tname}</span>
<span className="ml-3 normal-case font-normal text-gray-400" title="Columns the forecast load is pre-aggregated to">
grain: {grainCols.length
? <span className="font-mono text-gray-600">{grainCols.join(' × ')}</span>
: <span className="italic">none raw rows</span>}
</span>
</span> </span>
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
{colsDirty && ( {colsDirty && (
@ -325,17 +285,6 @@ export default function Setup({ refreshSources }) {
{saving ? 'Saving…' : 'Save'} {saving ? 'Saving…' : 'Save'}
</button> </button>
)} )}
{cols.some(c => c.dim_group && c.is_key) && (
<button
onClick={refreshDimMembers}
disabled={refreshingDims || colsDirty}
className="text-xs border border-gray-200 px-3 py-1 rounded hover:bg-gray-50 disabled:opacity-50"
title={colsDirty ? 'Save col meta first'
: 'Rebuild the member list for each keyed dim_group from the source. Reads the whole source, so it takes a while.'}
>
{refreshingDims ? 'Refreshing…' : 'Refresh master data'}
</button>
)}
<button <button
onClick={generateSQL} onClick={generateSQL}
disabled={generating || colsDirty} disabled={generating || colsDirty}
@ -353,8 +302,6 @@ export default function Setup({ refreshSources }) {
<th className="px-3 py-1.5 font-medium">column</th> <th className="px-3 py-1.5 font-medium">column</th>
<th className="px-3 py-1.5 font-medium">role</th> <th className="px-3 py-1.5 font-medium">role</th>
<th className="px-3 py-1.5 font-medium text-center">key</th> <th className="px-3 py-1.5 font-medium text-center">key</th>
<th className="px-3 py-1.5 font-medium text-center" title="Include this column in the display grain — the load is pre-aggregated to the flagged columns">grain</th>
<th className="px-3 py-1.5 font-medium text-center" title="The column an account's territory is expressed in. Accounts see and change only rows whose value here is on their list. One per source.">territory</th>
<th className="px-3 py-1.5 font-medium">group</th> <th className="px-3 py-1.5 font-medium">group</th>
<th className="px-3 py-1.5 font-medium">period col</th> <th className="px-3 py-1.5 font-medium">period col</th>
<th className="px-3 py-1.5 font-medium">label</th> <th className="px-3 py-1.5 font-medium">label</th>
@ -382,34 +329,6 @@ export default function Setup({ refreshSources }) {
className="cursor-pointer disabled:opacity-20" className="cursor-pointer disabled:opacity-20"
/> />
</td> </td>
<td className="px-3 py-1.5 text-center">
<input
type="checkbox"
checked={!!col.in_grain}
onChange={e => updateCol(i, 'in_grain', e.target.checked)}
disabled={col.role !== 'dimension' && col.role !== 'date'}
className="cursor-pointer disabled:opacity-20"
/>
</td>
{/* Radio, not a checkbox: exactly one column per source,
and the shape of the control should say so rather than
leaving it to a save-time error. */}
<td className="px-3 py-1.5 text-center">
<input
type="radio"
name="pf-territory-col"
checked={!!col.is_territory}
onChange={() => {}}
onClick={() => setEditedCols(prev => {
// clicking the chosen one again clears it, since a
// radio otherwise has no way back to "no territory"
const already = !!prev[i].is_territory
return prev.map((c, x) => ({ ...c, is_territory: !already && x === i }))
})}
disabled={col.role !== 'dimension'}
className="cursor-pointer disabled:opacity-20"
/>
</td>
<td className="px-3 py-1.5"> <td className="px-3 py-1.5">
<input <input
type="text" type="text"

View File

@ -1,45 +0,0 @@
diff --git a/rust/perspective-client/src/rust/config/view_config.rs b/rust/perspective-client/src/rust/config/view_config.rs
index fa4f36e..360b816 100644
--- a/rust/perspective-client/src/rust/config/view_config.rs
+++ b/rust/perspective-client/src/rust/config/view_config.rs
@@ -526,6 +526,23 @@ impl ViewConfig {
}
}
+ /// `_apply` for a field which is itself `Option`, where `None` in the update
+ /// means "not mentioned" rather than "clear it". `Option<Option<T>>` would be
+ /// needed to express both, and the wire format cannot carry the difference:
+ /// these fields are `skip_serializing_if = "Option::is_none"`, so an absent
+ /// field and an explicit null arrive identically. To lift a depth, send the
+ /// number of levels on that axis rather than clearing it.
+ fn _apply_optional<T: PartialEq>(field: &mut Option<T>, update: Option<T>) -> bool {
+ match update {
+ None => false,
+ Some(_) if *field == update => false,
+ Some(_) => {
+ *field = update;
+ true
+ },
+ }
+ }
+
pub fn reset(&mut self, reset_expressions: bool) {
let mut config = Self::default();
if !reset_expressions {
@@ -568,6 +585,16 @@ impl ViewConfig {
changed = Self::_apply(&mut self.windows, update.windows) || changed;
changed = Self::_apply(&mut self.group_rollup_mode, update.group_rollup_mode) || changed;
changed = Self::_apply(&mut self.split_rollup_mode, update.split_rollup_mode) || changed;
+
+ // Without these two, a depth can be set when a view is created --
+ // `table.view({ group_by_depth })` -- but never through `restore()`, which
+ // merges a `ViewConfigUpdate` onto the live config. The field arrives,
+ // deserializes, and is then dropped here, so the viewer's config and the
+ // engine never see it and nothing happens. That makes expand/collapse
+ // unreachable for anything driven by `restore()`, which is how a viewer
+ // changes its own configuration.
+ changed = Self::_apply_optional(&mut self.group_by_depth, update.group_by_depth) || changed;
+ changed = Self::_apply_optional(&mut self.split_by_depth, update.split_by_depth) || changed;
if self.group_rollup_mode == GroupRollupMode::Total && !self.group_by.is_empty() {
tracing::info!("`total` incompatible with `group_by`");
changed = true;

View File

@ -1,8 +0,0 @@
Built from https://github.com/fleetside72/perspective
branch apply-depth-on-config-update
commit 71d1a7508aa86d7e7224ad91e247afe4c201313d
based on unknown
built 2026-09-17T14:34:21Z on usmidsap02
dirty 0 uncommitted file(s) in the source tree at build time
Regenerate with ui/vendor/rebuild-perspective.sh

121
ui/vendor/README.md vendored
View File

@ -1,121 +0,0 @@
# Vendored Perspective
pf_app runs a **patched build of Perspective**. Upstream's C++ engine has always
implemented column-axis expand/collapse — `t_ctx2::set_depth(HEADER_COLUMN, …)`
and `open`/`close(HEADER_COLUMN, idx)` are fully written — but nothing above C++
could reach it: `set_column_pivot_depth()` was never called, and
`View<t_ctx2>::expand/collapse` hardcoded `HEADER_ROW`. The patch is wiring, not
new engine logic.
It buys two things the released packages cannot do at all:
- `split_by_depth` in `ViewConfig`, the `split_by` counterpart to `group_by_depth`
- `expand_column()` / `collapse_column()`, so one column branch can fold to its
subtotal while its siblings stay expanded — the Excel behaviour
**Source:** https://github.com/fleetside72/perspective, branch
`column-axis-expand-collapse`. See `PROVENANCE.txt` for the exact commit these
tarballs were built from.
## Why tarballs and not npm
The feature is not released upstream. Until it is, the four packages are built
from the fork and committed here as npm tarballs. `npm install` expands them
exactly as it expands anything from the registry — no special tooling, and
`pf.sh deploy` works unchanged. A deploy machine needs node and nothing else:
no emscripten, no cmake, no protoc, no Rust.
All four move together, never a subset. Perspective couples loader, package
versions, data format and `apache-arrow`; vendoring a partial set reintroduces
exactly the drift that causes trouble.
## Changing the engine
./rebuild-perspective.sh # builds the fork, repacks, rewrites PROVENANCE.txt
cd .. && npm install
git add vendor && git commit
Push the fork first — the script warns if the source tree is dirty, because a
tarball built from uncommitted code has no recoverable source.
The build itself needs cmake >= 3.29.5, protoc >= 22 (its version silently
selects which protobuf source tree gets cloned), pnpm, and the Rust nightly the
repo pins. Roughly 40 minutes cold. Only ever on a machine changing the engine.
## Getting rid of this
This is a fork, with the maintenance that implies. The exit is upstream taking
the change — the patch is small and additive, and the engine work is already
theirs. When a release ships it, delete this directory and put normal version
ranges back in `ui/package.json`.
## Applied: depth fields on a config update
`0001-apply-depth-fields-on-config-update.patch` is **in** the vendored
tarballs as of the 2026-09-17 build, from branch
`apply-depth-on-config-update`. It fixes an upstream bug in
`rust/perspective-client/src/rust/config/view_config.rs`:
`ViewConfig::apply_update` applies ten fields and **neither `group_by_depth`
nor `split_by_depth` is among them**. So a depth can be set when a view is
created (`table.view({ group_by_depth: 1 })` — which is what the fork's own
`depth_test.mjs` exercises) but never through `restore()`, which is how a
viewer changes its own configuration. The field arrives, deserializes, and is
discarded before the engine sees it.
`group_by_depth` and the omission are both upstream; the fork mirrored
`split_by_depth` alongside it faithfully, including the omission.
Note the semantics, from `server.cpp`:
```cpp
ctx1->set_depth(row_pivot_depth - 1); // one-sided
ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided
```
The config field counts **levels to show**; `view.set_depth()` counts the
boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and
`Forecast.jsx` sends `d + 1`.
To apply:
```bash
cd $PSP_DIR # default ~/perspective
git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
./ui/vendor/rebuild-perspective.sh
cd ui && npm install
```
`cargo check` on this patch was clean — the 125 errors it reports without
`protoc` installed are unresolved generated protobuf modules, none of them in
`view_config.rs`.
`applyDepth()` in `Forecast.jsx` reads the config back after restoring it and
falls back to the imperative `view.set_depth()` when the value did not stick.
With the patched engine installed that fallback should no longer run; it is kept
for now so an unpatched engine still works, and can be removed once the
declarative path is confirmed.
### Host tools this tree needs
The repo pins its Rust toolchain (`nightly-2026-06-01`), Emscripten (4.0.9) and
Binaryen (132) exactly, and pins **nothing** for the host tools — no
`packageManager`, no `.nvmrc`. So `npm i -g pnpm` and `pip install cmake` both
give versions this tree was not written against. The script's preflight now
rejects them, but for the record:
- **pnpm 10.x.** 11+ rejects a repeated `--if-present`, which
`sh_perspective.mjs:189` emits once per package in scope.
- **cmake 3.x**, not 4. Both satisfy the `>= 3.29.5` floor; CMake 4 removed
support for `cmake_minimum_required < 3.5`, which the C++ dependencies still
declare, and fails partway through the Arrow build.
- **protoc >= 22** (33.2 known good).
- **`pnpm pack`, not `npm pack`.** These packages depend on each other as
`workspace:^`, and only pnpm rewrites that to a concrete version. An
`npm pack` tarball is rejected at install: `Unsupported URL Type
"workspace:"`.
Two commits on the build branch are local build concessions rather than
fixes, and should be dropped before the real patch goes anywhere upstream:
removing the `postinstall:playwright` step (it runs `playwright install
--with-deps`, which apt-installs system libraries and so needs root), and
filling in the `allowBuilds` block pnpm 12 demanded.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -1,127 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# ---------------------------------------------------------------------------
# rebuild-perspective.sh — rebuild the patched Perspective and re-vendor it
#
# pf_app runs a patched build of Perspective that exposes the column axis
# expand/collapse the engine already implements (split_by_depth, and
# expand_column/collapse_column). Upstream does not ship this yet, so the
# built packages are vendored into this directory as npm tarballs.
#
# Source of truth: https://github.com/fleetside72/perspective
# branch column-axis-expand-collapse
#
# This script exists because vendored binaries are opaque: once the .tgz files
# are committed, nothing in the repo records how to regenerate them. Run this
# after changing the fork, then commit the resulting tarballs.
#
# Only needed on a machine that is changing the engine. Deploys just run
# `npm install`, which expands the committed tarballs - see ../README in this
# directory.
# ---------------------------------------------------------------------------
PSP="${PSP_DIR:-$HOME/perspective}"
BRANCH="${PSP_BRANCH:-column-axis-expand-collapse}"
VENDOR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# The four packages that must move together. Perspective's own docs are
# emphatic that loader, packages, data format and apache-arrow are one unit;
# vendoring a subset would reintroduce exactly the drift that causes trouble.
PACKAGES=(
"rust/perspective-js"
"rust/perspective-server"
"rust/perspective-viewer"
"packages/viewer-datagrid"
)
info() { echo -e "\033[0;34m==>\033[0m $*"; }
ok() { echo -e "\033[0;32m ✓\033[0m $*"; }
die() { echo -e "\033[0;31m ✗\033[0m $*" >&2; exit 1; }
# -- preflight --------------------------------------------------------------
[[ -d "$PSP" ]] || die "No Perspective checkout at $PSP.
git clone https://github.com/fleetside72/perspective.git $PSP
cd $PSP && git checkout $BRANCH
Set PSP_DIR to use a different path."
command -v pnpm >/dev/null || die "pnpm not found. Perspective builds with pnpm, not npm."
# The repo pins its Rust toolchain, emsdk and Binaryen exactly, but pins nothing
# for the host tools -- no packageManager, no .nvmrc -- so `npm i -g pnpm` gets
# whatever is newest and that is not what this tree was written against. pnpm 11+
# rejects a repeated `--if-present`, which sh_perspective.mjs emits once per
# package in scope, and the build dies with:
# error: the argument '--if-present' cannot be used multiple times
pnpm_major=$(pnpm --version | cut -d. -f1)
if (( pnpm_major > 10 )); then
die "pnpm $(pnpm --version) is too new; this tree needs pnpm 10.x.
npm i -g pnpm@10"
fi
command -v protoc >/dev/null || die "protoc not found.
Its VERSION selects which protobuf source tree the build clones, and a
version below 22 pulls a layout the build cannot consume. Needs >= 22
(33.2 known good). Distro packages are usually far too old."
cmake_ver=$(cmake --version 2>/dev/null | head -1 | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?') || die "cmake not found"
cmake_major=${cmake_ver%%.*}; cmake_minor=$(echo "$cmake_ver" | cut -d. -f2)
if (( cmake_major < 3 || (cmake_major == 3 && cmake_minor < 29) )); then
die "cmake $cmake_ver is too old; Perspective needs >= 3.29.5.
A user-level install works: pip3 install --user 'cmake==3.31.*'"
fi
# And not too new: CMake 4 dropped compatibility with cmake_minimum_required
# below 3.5, which Perspective's C++ dependencies still declare. `pip install
# cmake` gives 4.x by default, which satisfies the floor above and then fails
# in the middle of the Arrow build.
if (( cmake_major >= 4 )); then
die "cmake $cmake_ver is too new; CMake 4 removed support for
cmake_minimum_required < 3.5, which the C++ dependencies still use.
pip3 install --user 'cmake==3.31.*'"
fi
info "Perspective checkout: $PSP"
git -C "$PSP" rev-parse --abbrev-ref HEAD | grep -qx "$BRANCH" \
|| echo " ! on branch $(git -C "$PSP" rev-parse --abbrev-ref HEAD), expected $BRANCH"
commit=$(git -C "$PSP" rev-parse --short HEAD)
dirty=$(git -C "$PSP" status --porcelain | wc -l)
echo " commit $commit$([[ $dirty -gt 0 ]] && echo " (+$dirty uncommitted files)")"
# -- build ------------------------------------------------------------------
# `metadata` first: it generates docs/expression_gen.md, which perspective-client
# includes at compile time. Building a scope without it fails on the missing file.
info "Building (this takes ~40 minutes cold, a few minutes warm)…"
( cd "$PSP" && PSP_ONCE=1 PACKAGE="metadata,server,client,viewer,viewer-datagrid" pnpm run build )
ok "build complete"
# -- pack -------------------------------------------------------------------
# pnpm pack, not npm pack. These packages depend on each other as
# `workspace:^`, and only pnpm rewrites that to the concrete version on pack.
# npm leaves it, and npm install then refuses the tarball outright:
# npm error Unsupported URL Type "workspace:": workspace:^
info "Packing tarballs into $VENDOR"
rm -f "$VENDOR"/*.tgz
for p in "${PACKAGES[@]}"; do
( cd "$PSP/$p" && pnpm pack --pack-destination "$VENDOR" >/dev/null )
ok "$(basename "$p")"
done
# -- record provenance ------------------------------------------------------
# A committed .tgz is an opaque binary; without this the tie back to source is
# only in someone's memory.
cat > "$VENDOR/PROVENANCE.txt" <<EOF
Built from https://github.com/fleetside72/perspective
branch $BRANCH
commit $(git -C "$PSP" rev-parse HEAD)
based on $(git -C "$PSP" describe --tags --abbrev=0 2>/dev/null || echo 'unknown')
built $(date -u +%Y-%m-%dT%H:%M:%SZ) on $(hostname)
dirty $dirty uncommitted file(s) in the source tree at build time
Regenerate with ui/vendor/rebuild-perspective.sh
EOF
echo
ls -la "$VENDOR"/*.tgz | awk '{printf " %-52s %5.1f MB\n", $NF, $5/1048576}'
echo
ok "Done. Now: cd ui && npm install && git add vendor && git commit"
[[ $dirty -gt 0 ]] && echo -e "\033[1;33m !\033[0m source tree had uncommitted changes — push them to the fork first"
exit 0