Compare commits
2 Commits
master
...
spike/colu
| Author | SHA1 | Date | |
|---|---|---|---|
| 42d9d51f63 | |||
| d2e706b483 |
15
.env.example
15
.env.example
@ -4,18 +4,3 @@ DB_NAME=your_database
|
||||
DB_USER=your_user
|
||||
DB_PASSWORD=your_password
|
||||
PORT=3010
|
||||
|
||||
# Signs the session cookie. Generate with:
|
||||
# node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))'
|
||||
# or let ./pf.sh config do it. Changing it signs everyone out.
|
||||
SESSION_SECRET=
|
||||
|
||||
# Send the session cookie over HTTPS only. Keep true behind a TLS proxy;
|
||||
# set false only to reach the app over plain HTTP on a trusted network.
|
||||
COOKIE_SECURE=true
|
||||
|
||||
# Reverse proxy hops express should trust for req.ip / protocol. Default 1.
|
||||
#TRUST_PROXY=1
|
||||
|
||||
# Only needed if the UI is served from a different origin than the API.
|
||||
#CORS_ORIGIN=https://forecast.example.com
|
||||
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@ -1,4 +1,3 @@
|
||||
node_modules/
|
||||
.env
|
||||
.env.*
|
||||
public/app/
|
||||
|
||||
647
CLAUDE.md
647
CLAUDE.md
@ -4,8 +4,8 @@
|
||||
|
||||
A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.
|
||||
|
||||
Full spec, including the rationale and history this file leaves out: `docs/pf_spec.md`
|
||||
Data transport architecture options: `docs/pf_perspective_options.md`
|
||||
Full spec: `pf_spec.md`
|
||||
Data transport architecture options: `pf_perspective_options.md`
|
||||
|
||||
---
|
||||
|
||||
@ -14,7 +14,7 @@ Data transport architecture options: `docs/pf_perspective_options.md`
|
||||
- **Backend:** Node.js / Express (`server.js`)
|
||||
- **Database:** PostgreSQL — isolated `pf` schema
|
||||
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
|
||||
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.4.0 from a **patched build vendored in `ui/vendor`** — see its README for what is patched and which host-tool versions the rebuild needs; **bundled inline via the `/inline` entrypoints — never from a CDN** (the CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `docs/PERSPECTIVE.md`.
|
||||
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.2.0, **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
|
||||
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
|
||||
|
||||
---
|
||||
@ -22,50 +22,28 @@ Data transport architecture options: `docs/pf_perspective_options.md`
|
||||
## Project layout
|
||||
|
||||
```
|
||||
server.js Express entry point; pg pool; session; type parsers for bigint/numeric
|
||||
pf.sh Install, deploy and admin: config, db-setup, deploy, install-service,
|
||||
add-user, passwd, set-territory, … (fresh install: config → db-setup
|
||||
→ add-user → deploy → install-service)
|
||||
server.js Express entry point; pg pool; type parsers for bigint/numeric
|
||||
routes/
|
||||
auth.js POST /api/login, /api/logout, GET /api/me; login throttle
|
||||
tables.js GET /api/tables, /api/tables/:schema/:tname/preview
|
||||
sources.js Source registration, col_meta, SQL generation;
|
||||
GET /sources/:id/values/:col, /sources/:id/lookup,
|
||||
/sources/:id/dim/:group (+ POST …/refresh), /dim-period/cols
|
||||
versions.js Version CRUD and clone; GET /versions/:id/tags (+ PATCH),
|
||||
/versions/:id/bridge, /versions/:id/values/:col, /versions/:id/table-info
|
||||
operations.js Data streams (GET /versions/:id/data, /versions/:id/agg); loads
|
||||
(baseline, reference); scale, recode, clone
|
||||
log.js GET /api/versions/:id/log, /log/:logid/debug; DELETE and PATCH /log/:logid
|
||||
layouts.js Named pivot layouts — list per version, create, patch, delete
|
||||
plans.js Planned load segments — list, create, patch, delete
|
||||
sources.js Source registration, col_meta, SQL generation
|
||||
versions.js Version CRUD, baseline/reference load, data stream
|
||||
operations.js scale, recode, clone, undo — the core forecast ops
|
||||
log.js GET /api/versions/:id/log, DELETE /api/log/:logid
|
||||
lib/
|
||||
sql_generator.js grainOf, buildWhere/buildWhereAny, SCOPE_OPS, token substitution helpers
|
||||
auth.js scrypt hash/verify, requireAuth, requireAdmin, sessionUser; `node lib/auth.js hash` CLI
|
||||
utils.js RELATION_COLUMNS_SQL, mapType
|
||||
sql_generator.js buildFilterClause, token substitution helpers
|
||||
utils.js
|
||||
setup_sql/
|
||||
01_schema.sql pf schema DDL — idempotent, not a migration runner
|
||||
02_auth.sql pf.app_user + pf.session
|
||||
gen_dim_period.sql Populates pf.dim_period; fiscal year start month is configurable
|
||||
ui/vendor/ Patched Perspective build and its patches (see its README)
|
||||
01_schema.sql pf schema DDL — run once to install
|
||||
ui/src/
|
||||
main.jsx ThemeProvider, AuthProvider, Gate (mounts the app only once signed in)
|
||||
App.jsx Page routing; source/version selection and its localStorage restore
|
||||
theme.jsx ThemeContext / useTheme
|
||||
auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login
|
||||
views/
|
||||
Login.jsx Sign-in form
|
||||
Setup.jsx DB browser, source registration, col_meta editor
|
||||
Baseline.jsx Version management, baseline workbench, reference load, tags panel
|
||||
Baseline.jsx Version management, baseline workbench, reference load
|
||||
Forecast.jsx Perspective pivot, selection handling, operation dispatch
|
||||
components/
|
||||
DepthButtons.jsx One axis's collapse stepper; both axes are drawn from it
|
||||
LayoutMenu.jsx The Layout ▾ control — Published / Mine, with the write actions
|
||||
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms; effectiveScaleInputs
|
||||
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms
|
||||
BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
|
||||
Sidebar.jsx Forecast list + the 3 steps; collapsible
|
||||
StatusBar.jsx Forecast name · row count · calculating · theme; source on Setup only
|
||||
ReadOnlyNotice.jsx The banner over a page a non-admin may read but not change
|
||||
Sidebar.jsx 3-step collapsible nav
|
||||
StatusBar.jsx Source · version · write target · row counts · theme
|
||||
Timeline.jsx Date-range preview bar for baseline segments
|
||||
```
|
||||
|
||||
@ -74,581 +52,116 @@ ui/src/
|
||||
## Database schema (`pf`)
|
||||
|
||||
- **`pf.source`** — registered source tables
|
||||
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `dim_group` groups functionally dependent columns (e.g. a part and its attributes, or a date and its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain**; `is_territory` marks the column territories are expressed in (one per source); `is_key` is described under §`is_key` below
|
||||
- **`pf.dim_member`** — members of each `dim_group`, keyed on the group's key column; refreshed by `POST /sources/:id/dim/:group/refresh`
|
||||
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations; `adjustment_segment` / `adjustment_bucket` name new adjustments; `is_default` (one per source)
|
||||
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows. Tables created before that index was added do not have it.
|
||||
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb; `label` / `bucket` required (`log_named` CHECK). Stamped at write time: `row_count`, `value_total`, `units_total`, `measure_cols` (which columns the totals are denominated in), `env` (the state the intent executed against), `sql_text` (the statement as executed)
|
||||
- **`pf.layout`** — named Perspective view configs; see §Pivot layouts
|
||||
- **`pf.load_plan`** — load segment definitions, before and apart from having been run; see §Cloning a version's definition
|
||||
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
|
||||
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
|
||||
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
|
||||
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
|
||||
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
|
||||
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`, `is_admin`, `territory` (jsonb array)
|
||||
- **`pf.session`** — express-session store (connect-pg-simple layout)
|
||||
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql`
|
||||
|
||||
Dead columns, kept but read by nothing: `pf.version.unlabeled_load`, `pf.version.bucket_order`, `pf.log.seq`, `pf.source.default_layout`.
|
||||
|
||||
### `is_key`
|
||||
|
||||
Read in four places, and **not** the one it sounds like — slices are validated
|
||||
against `filterCols`, which is every `dimension` plus every `date` column
|
||||
regardless of `is_key`.
|
||||
|
||||
1. **The key of a `dim_group`** — `resolveGroup()` in `routes/sources.js` takes
|
||||
`members.find(c => c.is_key)`, the column every other member is keyed on for
|
||||
`pf.dim_member`
|
||||
2. **Value completion** — only `is_key` columns get a dropdown, and
|
||||
`/sources/:id/values/:col` refuses anything else
|
||||
3. **Sibling autofill** — fires on blur only when `is_key && dim_group`
|
||||
4. **The `dim_period` anchor** — `role === 'date' && is_key && dim_group` picks the
|
||||
date whose siblings are derived from the calendar
|
||||
|
||||
Uses 2 and 3 want several columns flagged; use 1 needs exactly one per group.
|
||||
**When a group has more than one, `.find()` silently takes the lowest `opos`** —
|
||||
a refresh keyed on the wrong column matches nothing and reports success with
|
||||
zero members. Until the two meanings are separated, a refresh that finds two
|
||||
keys should refuse rather than guess. Rationale: docs/pf_spec.md → §Data model.
|
||||
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
|
||||
|
||||
### Key token substitution tokens
|
||||
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{territory_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{label}}`, `{{bucket}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
|
||||
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
|
||||
|
||||
---
|
||||
|
||||
## Core data flow
|
||||
|
||||
### Initial load (Forecast view)
|
||||
`Forecast.jsx` fetches col_meta first, then picks the endpoint (both in `routes/operations.js`):
|
||||
`GET /api/versions/:id/data` → Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM
|
||||
|
||||
- **grain mode** (any `in_grain` column) — `GET /api/versions/:id/agg`, rows pre-aggregated to the grain, table indexed on `pf_gkey`
|
||||
- **raw mode** (no grain) — `GET /api/versions/:id/data`, raw forecast rows, table indexed on `pf_id`
|
||||
|
||||
Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM. `fetchArrow()` handles both.
|
||||
|
||||
**One batch, not streaming.** Per-batch Arrow encoding creates independent dictionaries that crash Perspective WASM on dictionary replacement messages, so the server accumulates all rows and emits one record batch. pg returns `bigint`/`numeric` as strings by default — the type parsers in `server.js` coerce them to numbers; do not remove them.
|
||||
|
||||
### Display grain
|
||||
Aggregating to the grain the pivot displays is the load-time fix, and it keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. **`grainOf()` in `lib/sql_generator.js` is the single definition of the grain** — `routes/log.js` calls it directly and `Setup.jsx` mirrors it. Design: docs/pf_spec.md → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `docs/pf_perspective_options.md` → §Spike findings.
|
||||
|
||||
### Log-derived columns
|
||||
`/data` and `/agg` both LEFT JOIN `pf.log` and emit four columns the forecast
|
||||
table does not itself carry:
|
||||
|
||||
- **`pf_segment`** — `pf.log.label` (`SEGMENT_EXPR`)
|
||||
- **`pf_bucket`** — `pf.log.bucket` (`BUCKET_EXPR`)
|
||||
- **`pf_note`** — tag, else note, on a scale/recode/clone; null on loads (`NOTE_EXPR`)
|
||||
- **`pf_op`** — `pf.log.operation`
|
||||
|
||||
Only segment and bucket are stored names (see §Every log entry is named); the
|
||||
note is derived. `SEGMENT_EXPR` / `BUCKET_EXPR` / `NOTE_EXPR` in
|
||||
`lib/sql_generator.js` are the single definition, shared with the `/data`
|
||||
cursor in `routes/operations.js` — `/agg` is generated, `/data` is not, and the
|
||||
two have to agree. `LABEL_GROUP_COLS` lists the `pf.log` columns they read, for
|
||||
`/agg`'s GROUP BY. In grain mode `pf_logid` is part of the grain, so the join
|
||||
adds no rows. The operation routes stamp the same fields onto the rows they push
|
||||
back incrementally, since those come from `RETURNING *` and would otherwise
|
||||
arrive without them.
|
||||
|
||||
### Column order is stored text
|
||||
|
||||
Perspective orders column groups by the value string, so a `"01 - "` prefix in
|
||||
**`pf.log.label`** / **`pf.log.bucket`**, typed by whoever names the segment, is
|
||||
the only lever; nothing derives it. `ADJUSTMENT_SEGMENT` = `'99 - Adjustments'`
|
||||
lands last only when the loads carry `01`–`0n` prefixes (digits precede
|
||||
letters). Do not reintroduce ordering as Perspective expressions — `restore()`
|
||||
replaces `expressions` wholesale and ExprTK cannot handle non-ASCII labels;
|
||||
`DEAD_ORDER_EXPRS` in `Forecast.jsx` strips the old ones from saved layouts.
|
||||
Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.
|
||||
|
||||
### Every log entry is named
|
||||
|
||||
`pf.log.label` is a row's segment and `pf.log.bucket` what it counts toward, and
|
||||
**every entry carries both** — the `log_named` CHECK refuses one without. A
|
||||
report gets the pivot's segment and bucket by joining the forecast table to
|
||||
`pf.log` on `pf_logid`; there is no rule to copy. Never derive them at read time.
|
||||
|
||||
They are stamped when the entry is written:
|
||||
|
||||
- **A load** — `label` is required (the routes answer 400 without one); `bucket`
|
||||
defaults to the label. `loadNames()` in `routes/operations.js`.
|
||||
- **An adjustment** — scale, recode and clone have no name inputs, so they are
|
||||
stamped with the version's `adjustment_segment` / `adjustment_bucket`, else the
|
||||
built-ins `ADJUSTMENT_SEGMENT` = `99 - Adjustments` and `ADJUSTMENT_BUCKET` =
|
||||
`04 - Forecast` (`ADJUSTMENT DEFAULTS` in `lib/sql_generator.js`).
|
||||
`adjustmentNames()` resolves them once per request; they go into the template's
|
||||
INSERT as `{{label}}` / `{{bucket}}` and onto the rows pushed back to the pivot.
|
||||
|
||||
The two version defaults are set on the Baseline page under **Defaults for new
|
||||
adjustments**. Changing one names the *next* adjustment; existing entries keep
|
||||
what they were written with. Renaming past entries is `PATCH /api/log/:logid`,
|
||||
one at a time, which renames but refuses to clear. A relabel needs a page
|
||||
reload to show, since the label is part of the aggregated row. A source
|
||||
registered before names were stored needs **Generate SQL** run once so its load
|
||||
templates write `label` and `bucket` at all. Rationale: docs/pf_spec.md → §Naming: segments, buckets, notes.
|
||||
**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.
|
||||
|
||||
### Forecast operations
|
||||
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums.
|
||||
|
||||
### Price or volume (`plug`)
|
||||
|
||||
**Price does not move unless it is named.** A price move is invisible in the
|
||||
figure being edited — the dollars land on target either way — so every default
|
||||
holds price. All resolution is in `resolveIncrs()` in `routes/operations.js`;
|
||||
the semantics come from the predecessor Excel model
|
||||
(`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`).
|
||||
|
||||
- **Dollars only** — `plug` decides: `'volume'` (the default: price holds, units
|
||||
scale in proportion) or `'price'` (volume holds, price moves).
|
||||
- **Qty only** — dollars follow at the current price. A slice with no units has
|
||||
no price to hold, so the dollars stay put.
|
||||
- **Dollars and qty** — both land as named; price falls out.
|
||||
- **A named price (`target_price`) holds**, and the other named measure decides
|
||||
the rest: price alone → qty holds; price + qty → qty lands as named, dollars
|
||||
follow; price + dollars → dollars land, qty follows. **All three is refused**
|
||||
server-side (400).
|
||||
- **The held price is the adjustable rows' own**, never blended with the fixed
|
||||
(excluded-iter) rows, even under `target_basis: 'selected'` — the scale SQL
|
||||
prorates over the adjustable rows only, so a blended price drives qty and
|
||||
dollars apart (and qty negative when a slice is taken to its fixed total).
|
||||
|
||||
**The three boxes foot.** At most two are inputs and the third is derived.
|
||||
`effectiveScaleInputs()` in `OperationPanel.jsx` is the single resolver: the
|
||||
panel renders from it and `buildPayload` in `Forecast.jsx` builds the request
|
||||
from it, so what is posted is what is shown. Typing a third drops one — the plug
|
||||
decides when dollars came last, else the older of the other two. The ledger's
|
||||
**Result** line previews value, units and price by the same rules, and the
|
||||
column that cannot move is labelled `· holds`.
|
||||
|
||||
**The plug control is labelled "Moves"**: it lights whatever the edit moves
|
||||
(both if both do) and takes a click only while it is the one deciding; otherwise
|
||||
it stays on screen, not hidden.
|
||||
|
||||
**Logged.** `resolved.method` (`price` | `both` | `volume` | the plug) records
|
||||
which side moved, since the request alone cannot say. `params` also keeps
|
||||
`plug` and `expected` (what the Result line promised), beside `resolved`, so a
|
||||
panel/server disagreement can be found by reading the log — nothing compares
|
||||
them yet. The client's stored preference is keyed `pf_scale_plug_v2`; do not
|
||||
reuse the old key, which could carry a stale `'price'`. Rationale: docs/pf_spec.md → §Operations.
|
||||
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload.
|
||||
|
||||
### Undo
|
||||
`DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload.
|
||||
|
||||
---
|
||||
|
||||
## Axis depth (collapse / expand)
|
||||
|
||||
Both axes collapse the same way: a **depth in `ViewConfig`**, set through
|
||||
`restore()` — `group_by_depth` via `applyDepth()`, `split_by_depth` via
|
||||
`applySplitDepth()`. Both steppers are drawn by `components/DepthButtons.jsx` from the axis's own field list
|
||||
(`groupFull` / `splitFull`), as a stepper `◂ level ▸`, the middle button opening
|
||||
the full level list. `depth` is null until the first layout has been read back;
|
||||
the stepper then shows an em dash.
|
||||
|
||||
**Depths are 1-based counts of levels to show**, on both axes and in
|
||||
`DepthButtons` (`levels[d - 1]`); the value crosses unchanged in each direction.
|
||||
The engine agrees (`set_depth(n - 1)`, and the traversal adds the 1 back). **Never
|
||||
send 0**: it reaches the engine as `set_depth(-1)` on an unsigned `t_depth`,
|
||||
underflows, and renders fully expanded. `applyDepth` floors at 1, so the
|
||||
stepper's "Total" slot shows the first level rather than a bare total — kept
|
||||
deliberately. A real total would need `group_rollup_mode: 'total'`, unverified
|
||||
against the local engine.
|
||||
|
||||
A depth lives in the config, so it survives view rebuilds and rides into saved
|
||||
layouts through `viewer.save()`. **This required patching the engine** —
|
||||
`ViewConfig::apply_update` dropped both depth fields; see
|
||||
`ui/vendor/0001-apply-depth-fields-on-config-update.patch`. Do not go back to
|
||||
imperative `view.set_depth()` alone: the viewer discards that view on every
|
||||
re-render. `split_full` is still *read* on load, for layouts saved under the
|
||||
old truncated-`split_by` scheme.
|
||||
|
||||
**Auto-pause is off.** `initViewer()` calls `viewer.setAutoPause(false)` right
|
||||
after `viewer.load()`. Pausing deletes the view, so returning to the tab would
|
||||
rebuild the whole grain and lose per-node expansion.
|
||||
|
||||
**Per-node expansion is not solved.** `expand(row_index)` / `collapse(row_index)`
|
||||
are set-only with no getter, so they cannot be captured and replayed and are
|
||||
lost on every rebuild. The route is a `ViewConfig` field of expanded row paths
|
||||
applied in `server.cpp`. Depth is also whole-axis — one level collapses every
|
||||
group at that level. Rationale: docs/pf_spec.md → §The pivot.
|
||||
|
||||
## Tag completions are per version
|
||||
|
||||
`GET /versions/:id/tags` — the tags used on **this** version, with `uses`,
|
||||
`last_used`, `owners` and `can_edit`; `Forecast.jsx` refetches on `versionId`
|
||||
and after every tagged write. A tag is not a row anywhere — only a value
|
||||
repeated across `pf.log` entries.
|
||||
|
||||
**Managed on the Baseline page's Tags panel**, which lists each with its use
|
||||
count and last-used date. `PATCH /versions/:id/tags` takes `{ from, to }` and
|
||||
rewrites every entry carrying `from`; a `to` that already exists **merges**, an
|
||||
empty `to` clears; the panel confirms both. Permission is the `pf.log` rule
|
||||
applied to *every* entry the tag is on (a half-applied rename leaves the
|
||||
initiative split), so a row with `can_edit` false is greyed out. One entry's
|
||||
tag is edited in the Forecast page's change log.
|
||||
|
||||
**Completion matches on words, not substrings.** `TagInput` in
|
||||
`OperationPanel.jsx` normalises both sides (case, punctuation, stopwords, light
|
||||
stemming) and says so when a typed tag reduces to an existing one; nothing is
|
||||
merged automatically. Rationale: docs/pf_spec.md → §Bridge and change log.
|
||||
|
||||
---
|
||||
|
||||
## Choosing a forecast
|
||||
|
||||
The **sidebar** lists every source's versions (`allVersions`, a client-side
|
||||
fan-out over `/sources/:id/versions`) and sets `sourceId` from the one picked.
|
||||
`versions` stays scoped to the selected source, because Baseline and Forecast
|
||||
are — do not widen it. The source picker is only on Setup. Known collision:
|
||||
"forecast" is both the scenario and the third step; undecided.
|
||||
|
||||
**Where a browser lands.** `App.jsx` restores `pf_sourceId` / `pf_versionId`
|
||||
from `localStorage`, re-checked against the live list. With nothing stored it
|
||||
falls back to **`pf.version.is_default`** (one per source, partial unique index
|
||||
`version_one_default`), not `versions[0]`, which is the newest. A stored
|
||||
selection wins. Setting the default is **admin only**, in `PUT /api/versions/:id`,
|
||||
as two statements in one transaction — the index refuses the instant between.
|
||||
|
||||
**Renaming a version** is safe at any time; nothing derives from the name.
|
||||
`PUT /api/versions/:id` answers 409 on a `(source_id, name)` collision, and
|
||||
COALESCEs a falsy name back to the existing one, so a blank box is refused
|
||||
client-side.
|
||||
|
||||
---
|
||||
|
||||
## Cloning a version's definition
|
||||
|
||||
`POST /sources/:id/versions` takes `clone_from`. It copies the template
|
||||
version's settings (`description`, `exclude_iters`, the fallback names,
|
||||
`bucket_order`), its **version-scoped** layouts, and its load segments — and
|
||||
**runs nothing**, so the dates can be changed before anything loads.
|
||||
|
||||
Not copied: adjustments, in any form (a scale's target was sized against the
|
||||
rows that were there), and source-wide layouts (`version_id IS NULL`), which
|
||||
already apply to the new version.
|
||||
|
||||
### `pf.load_plan`
|
||||
|
||||
A plan is a load definition on its own — `kind`, `label`, `bucket`, `tag`,
|
||||
`note`, `filters`, `raw_where`, `date_offset`, `seq` — with `run_logid` naming
|
||||
the `pf.log` entry it produced, or null while unrun. **Deliberately not
|
||||
`pf.log`**: every row there means a write happened.
|
||||
|
||||
`run_logid` is `ON DELETE SET NULL`, so undoing a load returns its plan to unrun.
|
||||
The plans list reads the row count back through that FK.
|
||||
|
||||
`cloneDefinition()` in `routes/versions.js` reads `pf.load_plan` when the
|
||||
template has plans and otherwise derives them from its `baseline`/`reference`
|
||||
log entries (`pf.log.params` stores `filters`, `raw_where`, `date_offset`).
|
||||
|
||||
**Run all stops at the first failure** — segments are not independent.
|
||||
Rationale: docs/pf_spec.md → §Loads and segments.
|
||||
|
||||
---
|
||||
|
||||
## Pivot layouts
|
||||
|
||||
A layout is a named `ViewConfig` in **`pf.layout`**, owned by an account:
|
||||
**published** (everyone on the forecast lists and can apply it; only its owner
|
||||
or an admin may change it) or **private**. Scope is the **version** by default;
|
||||
`version_id IS NULL` means every version of the source. `is_default` (at most
|
||||
one per scope, by partial unique index) is what `initViewer()` restores on a
|
||||
first load — a version-scoped default beating a source-wide one.
|
||||
|
||||
**Permissions are the `pf.log` rule verbatim** — your own, or an admin's
|
||||
override; `can_edit` rides on every row and the UI greys out the rest. Applying
|
||||
is never restricted; Save is withheld on a layout that isn't yours, Save as…
|
||||
forks it. No territory clause — a layout is display config.
|
||||
|
||||
**Local state.** `LAYOUT_KEY` (`pf_layout_v{vid}`), the unnamed last-used
|
||||
config, stays in `localStorage`; `loadLayouts()` lifts the old named list
|
||||
(`LAYOUTS_KEY`) into `pf.layout` as private rows once, then clears it.
|
||||
|
||||
**The dirty dot is a comparison, not a flag.** `restore()` itself fires
|
||||
`perspective-config-update`, so `activeConfigRef` holds what the pivot last
|
||||
matched and `sameConfig()` compares against it, ignoring `table` and `theme`.
|
||||
|
||||
**Grand totals ride in the layout under `pf_totals`** — `{ row, col }`. The
|
||||
engine has no "subtotals but no grand total" mode, so both are hidden in the
|
||||
grid's DOM; `captureConfig()` adds the key and `cleanLayout()` strips it before
|
||||
`restore()`. Without it: row shown, column hidden.
|
||||
|
||||
**`cleanLayout()`** drops anything the live schema lacks — axes, `aggregates`
|
||||
(including the weight column of the multi-arg form), `theme`, `pf_totals`, dead
|
||||
expressions — dropping the offending *entry* rather than the layout, since
|
||||
`restore()` is all-or-nothing. Rationale: docs/pf_spec.md → §The pivot.
|
||||
|
||||
---
|
||||
|
||||
## The Forecast toolbar is two zones
|
||||
|
||||
Left, **view** — `Layout ▾` and the `Rows` / `Columns` steppers with their
|
||||
`Total` toggles; all `ViewConfig`, nothing reaches the server. Right,
|
||||
**forecast** — `Refresh data`, `Change log`, `Bridge`, `Operations`. **Blue is
|
||||
for actions, pressed grey (`.pf-pressed`) for state.** `Fit widths` was removed
|
||||
deliberately; do not restore it without fixing auto-size (see git history).
|
||||
|
||||
**`Calculating…`** in the status bar: `Forecast.jsx` raises a counter (rebuilds
|
||||
overlap) in `applyDepth`, `applySplitDepth` and `applyLayout`, lowers it after
|
||||
`viewer.flush()`, and broadcasts `pf-pivot-busy` on the window for `StatusBar`.
|
||||
**Not** driven from `perspective-config-update`, which arrives after the
|
||||
rebuild. Rationale: docs/pf_spec.md → §The pivot.
|
||||
`DELETE /api/log/:logid` → removes rows by logid → **full Perspective reload** (known wart).
|
||||
|
||||
---
|
||||
|
||||
## Slice mechanics
|
||||
|
||||
A click (`perspective-click`) or a dragged region becomes a **slice** — a
|
||||
`{col: value}` object. A plain click replaces the selection; ctrl/⌘/shift-click
|
||||
toggles a slice in or out, so the panel holds a **list** sent as `slices` in
|
||||
operation POST bodies (a single `slice` is still accepted server-side).
|
||||
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).
|
||||
|
||||
**The slice is re-derived by `cellFilters()` in `Forecast.jsx`**, from
|
||||
`detail.row.__ROW_PATH__` (row axis, raw values, so dates stay epoch millis)
|
||||
and the clicked column name (column axis, split by `|`). The click and a dragged
|
||||
region go through the same derivation. `detail.config.filter` is used only as
|
||||
the fallback when there is no cell to index the column axis with — it cannot be
|
||||
trusted otherwise, because the engine builds it with a falsy test that drops
|
||||
blank and `0` pivot values, and a dropped key **widens** the write.
|
||||
Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow** — `{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.
|
||||
|
||||
**Regions.** The datagrid runs in `edit_mode: SELECT_REGION` (forced on
|
||||
restore). `perspective-select` carries a **ViewWindow** —
|
||||
`{ start_row, end_row, start_col, end_col }` — and fires on every mouseover, so
|
||||
the handler records the latest window and a window-level `mouseup` commits it.
|
||||
A single-cell region is ignored there: `perspective-click` owns it, and handling
|
||||
it twice would undo a ctrl-click toggle.
|
||||
Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".
|
||||
|
||||
**A total cell is its displayed amount.** The grand-total row resolves to no
|
||||
row dimension, so its slice is whatever the column axis and the pivot's `==`
|
||||
filters leave — possibly `{}`. An empty slice is accepted only alongside a
|
||||
non-empty `scope`; with no filter at all it would be the whole version, and is
|
||||
refused on both sides. `buildWhereAny()` reads a union containing `{}` as
|
||||
`TRUE`, the scope then ANDed on — dropping it would narrow the write.
|
||||
**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.
|
||||
|
||||
**The pivot's `pf_` columns stay in the slice.** `pf_segment`, `pf_bucket`,
|
||||
`pf_note` and `pf_op` are resolved through the log (`COMPUTED_SLICE_COLS`);
|
||||
`pf_iter`, `pf_user` and `pf_logid` are compared directly (`TABLE_SLICE_COLS`).
|
||||
Stripping any of them made the write wider than the ledger's total. They narrow
|
||||
a slice but do not make it selective on their own. Duplicates are collapsed
|
||||
before the request — otherwise `apply_mode: each` applies the change twice.
|
||||
`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.
|
||||
|
||||
**Selection highlight.** The datagrid wipes `model._selection_state.selected_areas`
|
||||
on every mousedown, so `Forecast.jsx` keeps `areasRef` (`sliceKey -> rectangles`,
|
||||
parallel to `slices`) and an effect pushes the full set back and redraws after
|
||||
every change. Every deselect path prunes it by live slice key.
|
||||
**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.
|
||||
|
||||
**A blank dimension value is a value.** `null` and `''` are different rows and
|
||||
stay distinct through to the WHERE clause: `IS NULL` vs `= ''`. Only `undefined`
|
||||
means absent (a subtotal row's shorter path, a collapsed column level). On the
|
||||
column axis, `t_tscalar::to_string` renders a null as the literal text `"null"`
|
||||
and an empty string as `''`, so `cellFilters()` maps `"null"` → null. The ledger
|
||||
filters a null with `is null`, never `== null` (the engine matches on type). The
|
||||
panel prints `(null)` and `(blank)`, and `—` only when the column is absent from
|
||||
the slice.
|
||||
---
|
||||
|
||||
**The pivot's own filter travels alongside, as `scope`** — `[col, op, value]`
|
||||
triples read off `viewer.save()`, which is also what the ledger totals against.
|
||||
`SCOPE_OPS` in `lib/sql_generator.js` is the whole allowed list, under
|
||||
Perspective's operator names; anything outside it is **refused, not dropped**.
|
||||
## Column hierarchy (collapse / expand)
|
||||
|
||||
- Substring operators (`contains`, `begins with`, `ends with` and negations) map
|
||||
to **`ILIKE`**, matching the engine's case-insensitive `t_tscalar::contains`;
|
||||
`%`, `_` and `\` are escaped out of the term.
|
||||
- `matches` / `not matches` stay refused: the engine is RE2, Postgres is not.
|
||||
- Every operator that needs a term refuses an empty one (an unfinished filter
|
||||
row arrives as `[col, '==', null]`).
|
||||
The two pivot axes collapse by completely different mechanisms, and the asymmetry is a
|
||||
Perspective constraint, not a choice:
|
||||
|
||||
**Limitation:** computed columns created by Perspective expressions don't map
|
||||
back to raw rows — only native columns work for slice extraction. Rationale:
|
||||
docs/pf_spec.md → §Selection.
|
||||
- **Rows.** The `GROUP BY ROLLUP` view holds every level at once; `view.set_depth()` — which
|
||||
lives on the view, not the config — hides the deeper ones. That is what the `EXPAND 0 1 2 3`
|
||||
buttons drive, via `applyDepth()`.
|
||||
- **Columns.** There is no equivalent. `expand()` / `collapse()` take a **row index**,
|
||||
`ViewConfig` has `group_by_depth` but no `split_by_depth`, and `split_rollup_mode`
|
||||
(`'flat' | 'rollup'`) only chooses whether subtotal column groups are *emitted* — it is a
|
||||
view shape, not an interaction. So `applySplitDepth(n)` collapses by restoring a
|
||||
**truncated `split_by`**, which rebuilds the view.
|
||||
|
||||
Three things follow from the rebuild, and each is handled:
|
||||
|
||||
1. The full hierarchy has to be remembered separately — once collapsed, `viewer.save()`
|
||||
only reports the short `split_by`. `splitFullRef` / `splitFull` hold it, and it is
|
||||
persisted into the saved layout as `split_full` so a reload while collapsed can still
|
||||
expand back. `adoptSplit()` is the single place it is set.
|
||||
2. `perspective-config-update` fires for our own restore as well as the user rearranging
|
||||
the pivot. `collapsingRef` distinguishes them — without it, a collapse would overwrite
|
||||
the full hierarchy with the truncated one and the deeper levels would be unreachable.
|
||||
3. Row depth lives on the discarded view, so `applyDepth(expandDepthRef.current)` is
|
||||
re-applied afterwards — the same wart as the refocus re-apply.
|
||||
|
||||
The selection is cleared on every change: slices name the split_by dimensions they were
|
||||
cut from, and the highlight is keyed on grid coordinates. Neither survives a column axis
|
||||
that just changed shape.
|
||||
|
||||
**Limitation:** this is whole-axis, not per-branch. Excel can collapse 2025 while 2026
|
||||
stays expanded; truncating `split_by` collapses every column group at that level together.
|
||||
Per-branch is not reachable — `columns` selects which *measures* appear, not individual
|
||||
split combinations.
|
||||
|
||||
---
|
||||
|
||||
## Operation SQL patterns
|
||||
|
||||
All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slices; `{{exclude_clause}}` blocks `exclude_iters` rows; the territory clause is ANDed on last.
|
||||
All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.
|
||||
|
||||
**`buildWhere()` in `lib/sql_generator.js`** turns a slice into SQL: it allows
|
||||
`dimension` + `date` columns plus `COMPUTED_SLICE_COLS` and `TABLE_SLICE_COLS`,
|
||||
**skips** unknown non-`pf_` keys, and **refuses** an unknown `pf_` key (skipping
|
||||
one would widen the write). A null renders as `IS NULL`; in a list the null arm
|
||||
is ORed on, since `IN (NULL)` is never true. Values are escaped but not
|
||||
parameterized — consistent with existing patterns, debuggable in pg logs. The
|
||||
"must name a filterable column" refusal is **`assertSelective()`** in
|
||||
`routes/operations.js`.
|
||||
|
||||
**Every date column travels.** `dataCols` is `[...dims, ...dateCols, value, units]`
|
||||
— never just the first date column, or scale and recode write rows with null
|
||||
secondary dates under correctly labelled period dimensions.
|
||||
|
||||
- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions; refuses a pool whose net is ~zero relative to its gross (`assertProratable`)
|
||||
- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions
|
||||
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
|
||||
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched
|
||||
|
||||
**Recode takes a date shift too.** `date_offset` moves the *put-back* leg only,
|
||||
so a shifted recode is a timing move, and with a shift `set` may be empty. The
|
||||
moved rows' period dimensions are re-derived from `pf.dim_period` against the
|
||||
shifted date; a row whose date is null keeps the label it had. Without a shift
|
||||
nothing is derived. Clone still blanks those labels on a null date.
|
||||
|
||||
**A date offset must be whole days.** `assertInterval()` refuses anything finer,
|
||||
because every shifted date is cast back to a date — and Postgres reads a bare
|
||||
number as *seconds*. `30 days` is not a month; use `1 month`.
|
||||
|
||||
Scale refuses with "Nothing to scale…" and recode with "Nothing to recode…"
|
||||
when there is nothing to write. Rationale: docs/pf_spec.md → §Operations.
|
||||
`build_where()` validates every slice key against col_meta (only `role = dimension` allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
Everything under `/api` except the auth routes sits behind a session; the React
|
||||
app is only mounted once there is one (`Gate` in `main.jsx`).
|
||||
|
||||
- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
|
||||
Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
|
||||
the password is read on stdin and hashed before it reaches psql.
|
||||
- **Sessions:** `pf.session` (connect-pg-simple); revoke by deleting the row.
|
||||
Cookie `pf.sid`: httpOnly, SameSite=Lax, Secure unless `COOKIE_SECURE=false`, 12h rolling.
|
||||
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
|
||||
`TRUST_PROXY` (default 1) for `req.ip` and secure cookies behind the TLS proxy.
|
||||
`CORS_ORIGIN` is the only way CORS is enabled; never a wildcard with a session
|
||||
cookie.
|
||||
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
|
||||
minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
|
||||
regenerates the session id on success.
|
||||
|
||||
**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
|
||||
`sessionUser(req)`, **never from the request body**.
|
||||
|
||||
## Setup and Baseline are admin-only to write
|
||||
|
||||
`requireAdmin` (`lib/auth.js`) is on:
|
||||
|
||||
- **sources.js** — `POST /sources`, `PUT /sources/:id/cols`, `POST
|
||||
/sources/:id/generate-sql`, `POST /sources/:id/dim/:group/refresh`,
|
||||
`DELETE /sources/:id`
|
||||
- **versions.js** — `POST /sources/:id/versions`, `PUT /versions/:id`,
|
||||
`POST /versions/:id/{close,reopen}`, `DELETE /versions/:id`
|
||||
- **operations.js** — the loads: `POST /versions/:id/{baseline,reference}`,
|
||||
`PUT /versions/:id/baseline/:logid`, `DELETE /versions/:id/baseline`
|
||||
- **plans.js** — every write
|
||||
|
||||
**Not gated, deliberately:** reads (gating what a page needs to render turns it
|
||||
into a 403); scale/recode/clone/undo, which carry their own owner and territory
|
||||
rules; `PATCH /versions/:id/tags`, whose permission is ownership.
|
||||
|
||||
**The pages stay visible and go read-only** — one `<fieldset disabled>` per page
|
||||
plus a `ReadOnlyNotice` banner. Setup wraps a real box; Baseline uses
|
||||
`display: contents`. Baseline's Tags panel sits outside the fieldset. **None of
|
||||
this is the permission** — the server check is. Setup's list fetches coerce
|
||||
responses to an array: an error body rendered with `.map` throws during render
|
||||
and unmounts the whole app. Rationale: docs/pf_spec.md → §Access.
|
||||
|
||||
## Source columns come from pg_catalog
|
||||
|
||||
`information_schema.columns` omits **materialized views**, and `gs.osm_skinny` is
|
||||
one. `RELATION_COLUMNS_SQL` in `lib/utils.js` is the replacement, used by version
|
||||
creation, source registration and the table preview; it returns the
|
||||
information_schema shape (`data_type` from `format_type` with the modifier
|
||||
stripped, precision/scale from `atttypmod`). The table browser lists from
|
||||
`pg_class` by `relkind`. Do not query `information_schema.columns` for source
|
||||
columns.
|
||||
|
||||
## Territory scoping
|
||||
|
||||
An account sees and changes only its own territory: `pf.app_user.territory`
|
||||
(jsonb array), `is_admin` for accounts that see everything, and
|
||||
`col_meta.is_territory` marks the column (one per source).
|
||||
|
||||
**Fail closed.** `buildTerritoryClause()` returns `FALSE`, not `TRUE`, for an
|
||||
empty list or an unflagged source.
|
||||
|
||||
**Built from the session, never the request.** Unlike `scope`, the territory
|
||||
predicate is ANDed on last, in `sliceUnits()` for writes and per route for
|
||||
reads, where nothing in the payload can remove it.
|
||||
|
||||
Enforced at:
|
||||
|
||||
- `/data` — clause on the cursor *and* on the count behind `X-Row-Count`
|
||||
- `/agg` — `{{territory_clause}}`, applied **before** the GROUP BY, since the
|
||||
territory column need not be in the grain
|
||||
- every operation, through `sliceUnits()`
|
||||
- `/sources/:id/values/:col` — completion reads the source table
|
||||
- `DELETE /log/:logid` and `PATCH /log/:logid` — by **owner**, not territory:
|
||||
your own entries, or an admin's override. Undo removes an entry wholesale, and
|
||||
`label`/`bucket` name the pivot's columns for everyone in the version
|
||||
- recode's `set` — a scoped account cannot set the territory column at all
|
||||
|
||||
**The change log** lists everyone's entries, with an **Everyone / Mine** control
|
||||
(resets to Mine on every open, not persisted) and the reader's own entries
|
||||
marked. `isMine()` is strict authorship, deliberately not `canUndo()`. The fetch
|
||||
stays unfiltered, since the control's counts need the whole set. It shows an
|
||||
entry's **full** impact — the totals on `pf.log` are company-wide, deliberately.
|
||||
|
||||
Managed with `./pf.sh set-territory | set-admin | orphan-territory`.
|
||||
`orphan-territory` lists values present in the data that no account owns.
|
||||
Rationale: docs/pf_spec.md → §Access.
|
||||
|
||||
## Light / dark mode
|
||||
|
||||
`ui/src/theme.jsx` — `ThemeProvider` (in `main.jsx`) and `useTheme()` →
|
||||
`{ dark, setDark }`. Stored as `pf_dark` in `localStorage`, else
|
||||
`prefers-color-scheme`; the effect toggles `.dark` on `<html>`. `ui/src/index.css`
|
||||
defines custom properties under `:root` and `.dark`, with Tailwind overrides
|
||||
written as `.dark .bg-white { ... }` — no Tailwind dark-mode config. Palette is
|
||||
Perspective's "Pro Dark".
|
||||
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
|
||||
|
||||
The viewer's theme is `viewer.restore({ theme: dark ? 'Pro Dark' : 'Pro Light' })`,
|
||||
on load and in `useEffect([dark, versionId])` — a **partial** restore, no
|
||||
rebuild. **Not** `setAttribute('theme', …)`, which does nothing in 5.4.0.
|
||||
`cleanLayout` drops `theme` and `sameConfig()` ignores it: the theme is the
|
||||
reader's choice, never a layout's.
|
||||
|
||||
## After a change: restart, and Generate SQL
|
||||
|
||||
- **Restart the server** for anything in `routes/`, `lib/` or `server.js`.
|
||||
- **Generate SQL** (Setup, per source) whenever `lib/sql_generator.js` changes.
|
||||
The templates are *stored* in `pf.sql`, so editing the generator changes
|
||||
nothing until they are rebuilt — and a template carrying a token the running
|
||||
code does not substitute fails at the database, which reads as an unrelated
|
||||
client-side error.
|
||||
|
||||
**Order matters: restart first, then Generate SQL**, or the old code writes the
|
||||
templates.
|
||||
|
||||
Schema changes are applied to the live database directly and mirrored into
|
||||
`setup_sql/` for a fresh install; `01_schema.sql` is idempotent but is not a
|
||||
migration runner.
|
||||
- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
|
||||
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
|
||||
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
|
||||
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
|
||||
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
|
||||
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
|
||||
|
||||
## Known issues / active work
|
||||
|
||||
- **Rows written before the every-date-column fix have null secondary dates**
|
||||
(`rdate`/`sdate_e`) on scale and recode rows. Deleting and replaying those log
|
||||
entries is the honest repair; a backfill would be inference
|
||||
- **Zero-row clones report success** — clone commits an empty log entry and
|
||||
returns `rows_affected: 0`, where scale and recode refuse
|
||||
- **The change log does not show an entry's id**, so there is no way to name
|
||||
one when asking about it
|
||||
- **Territory is read onto the session at login**, so a change does not reach a
|
||||
signed-in account until it signs in again
|
||||
- **The bridge dedupes overlapping slices by `pf_id`**, which grain-mode rows do
|
||||
not carry, so overlapping slices double-count there; and it keeps only
|
||||
dimension/date slice keys, dropping the `pf_` keys operations now honour
|
||||
(`BridgeView.jsx`, the per-slice view loop)
|
||||
- **Depth buttons rebuild the view** — `restore()` tears down and rebuilds, a
|
||||
full traversal, where a manual collapse mutates in place. Fix: detect a
|
||||
depth-only change and call `view.set_depth()` too, at the cost of drift
|
||||
- **Load time is dominated by row count, not payload size** — serialising rows
|
||||
out of Postgres into JS dominates. The lever is **dynamic grain**; see
|
||||
docs/pf_spec.md → §Display-grain pre-aggregation
|
||||
- **Per-node expand/collapse** is lost whenever the view rebuilds; see §Axis depth
|
||||
- **Col_meta / version schema drift:** if col_meta roles change after a version's
|
||||
forecast table is created, SQL and DDL go out of sync — delete and recreate
|
||||
the version
|
||||
- **Grain drift:** changing `in_grain` after a load requires Generate SQL + a page
|
||||
reload. `routes/log.js` derives the grain from live col_meta, so a grain
|
||||
changed mid-session yields `pf_gkeys` that don't match the loaded table and
|
||||
undo silently removes nothing
|
||||
- **Grain is static per source** — an unflagged dimension cannot be pivoted on
|
||||
- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
|
||||
- Load progress bar is jittery — needs throttle (~10 updates/sec)
|
||||
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
|
||||
- Source/version selection doesn't persist across page reload
|
||||
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
|
||||
|
||||
## Deferred (not in v1)
|
||||
Baseline replay (not built), approval workflow, export, version comparison, multi-DB connections. Live server-side aggregation (Path A / DuckDB virtual server) is parked on branch `spike/duckdb-virtual-server`. See docs/pf_spec.md → §Open questions.
|
||||
Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections.
|
||||
|
||||
336
PERSPECTIVE.md
Normal file
336
PERSPECTIVE.md
Normal file
@ -0,0 +1,336 @@
|
||||
# Perspective — configuration & deployment reference
|
||||
|
||||
Canonical guide for how Perspective should be configured, fed, and shipped across our
|
||||
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
|
||||
but made different early choices; this doc defines the target state and the rationale, so
|
||||
the two converge instead of drifting.
|
||||
|
||||
> **Distribution note:** we use the **`@perspective-dev/*`** packages
|
||||
> (repo: <https://github.com/perspective-dev/perspective>, home:
|
||||
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
|
||||
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
|
||||
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
|
||||
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
|
||||
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
|
||||
> scopes.
|
||||
|
||||
---
|
||||
|
||||
## Core principle: Perspective is one locked unit
|
||||
|
||||
These four things must move together and be pinned together. Bumping one without the
|
||||
others is the source of nearly every Perspective bug we've hit:
|
||||
|
||||
1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
|
||||
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
|
||||
3. **Data format** — Arrow IPC vs JSON rows
|
||||
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
|
||||
format the client WASM understands
|
||||
|
||||
Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
|
||||
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Loading — npm `/inline`, pinned exact (not CDN)
|
||||
|
||||
```js
|
||||
import perspective from '@perspective-dev/client/inline'
|
||||
import '@perspective-dev/viewer/inline'
|
||||
import '@perspective-dev/viewer-datagrid'
|
||||
import '@perspective-dev/viewer-d3fc'
|
||||
import '@perspective-dev/viewer/themes'
|
||||
```
|
||||
|
||||
- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
|
||||
fetch, works offline / behind a firewall, reproducible from the lockfile.**
|
||||
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
|
||||
one-line version bumps) but in production it means: app breaks if the CDN is
|
||||
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
|
||||
pull executable WASM from a third party on every load.
|
||||
|
||||
> **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
|
||||
> resolves its server WASM with
|
||||
> `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
|
||||
> which from `.../client@4.4.0/dist/cdn/` resolves to
|
||||
> `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**.
|
||||
> jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
|
||||
> pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
|
||||
> `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
|
||||
> Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
|
||||
>
|
||||
> The security framing matters as much as the outage: an unversioned URL means your
|
||||
> users execute whatever that package publishes next, automatically, unreviewed.
|
||||
>
|
||||
> 5.2.0 fixes it by carrying the client's version across
|
||||
> (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
|
||||
> the WASM is embedded in the bundle and there is no runtime fetch to hijack
|
||||
> (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).
|
||||
|
||||
pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
|
||||
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
|
||||
`<link>` in `index.html` — so it's bundled and versioned too.
|
||||
|
||||
---
|
||||
|
||||
## 2. Version policy — a real trilemma (read carefully)
|
||||
|
||||
The version choice is constrained by two hard facts about the `@perspective-dev` packages
|
||||
**(verified against installed metadata, 2026-06)**:
|
||||
|
||||
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
|
||||
Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even
|
||||
though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
|
||||
trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
|
||||
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
|
||||
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
|
||||
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
|
||||
|
||||
So you can have at most **two** of these three:
|
||||
|
||||
| Want | Requires |
|
||||
|---|---|
|
||||
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
|
||||
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
|
||||
| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) |
|
||||
|
||||
There is **no** version where all three hold. Pick by what the app needs:
|
||||
|
||||
> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts
|
||||
>
|
||||
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1`
|
||||
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.**
|
||||
>
|
||||
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
|
||||
> running exactly that pair), *every chart type other than Datagrid fails.*
|
||||
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
|
||||
> `viewer` throws `get_static_config is not a function` — once per chart plugin. The
|
||||
> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
|
||||
> load against a coherent 4.4.1 viewer produces no such error.
|
||||
>
|
||||
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
|
||||
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
|
||||
>
|
||||
> Consequence: dataflow currently has the worst of both worlds. It carries the
|
||||
> mixed-version complexity *specifically* to keep charts, and does not have charts.
|
||||
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
|
||||
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
|
||||
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
|
||||
>
|
||||
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
|
||||
> real bundled build. It is the documented-and-untested assumption this whole policy
|
||||
> rests on — establish it before betting a version choice on it.
|
||||
|
||||
- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
|
||||
`^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
|
||||
working Datagrid and no charts. If you are on it today, you are choosing inline
|
||||
bundling, not charts; be explicit about which one you actually want.
|
||||
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
|
||||
**4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give
|
||||
up `/inline` bundling.
|
||||
|
||||
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
|
||||
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or
|
||||
later (then a fully-coherent inline-capable suite becomes possible — and only then does
|
||||
"both" come back on the table).
|
||||
|
||||
---
|
||||
|
||||
## 3. Data delivery — match the format to the workload
|
||||
|
||||
| Workload | Format | Why |
|
||||
|---|---|---|
|
||||
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
|
||||
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |
|
||||
|
||||
### Arrow constraints (read before choosing it)
|
||||
|
||||
If you deliver Arrow, three pieces are coupled and must stay aligned:
|
||||
|
||||
- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
|
||||
must coerce them to JS numbers before encoding, or `apache-arrow` infers
|
||||
`Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
|
||||
1700).
|
||||
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
|
||||
Perspective WASM crashes on dictionary-replacement messages. The server must
|
||||
accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
|
||||
'stream')`). Consequence: the client "stream" is just a chunked *download* of one
|
||||
batch — nothing renders progressively, and the server holds the full result set in
|
||||
memory per request.
|
||||
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
|
||||
`package.json` and treat it as part of the locked unit (principle above). Note the
|
||||
`@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
|
||||
pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
|
||||
the server toward the arrow major the client was built against, or at minimum make
|
||||
Arrow ingestion (smoke test §7) the gate on any arrow bump.
|
||||
|
||||
JSON avoids all three but pays in payload size and parse cost, and pushes type handling
|
||||
to client-side heuristics — acceptable only at small scale.
|
||||
|
||||
---
|
||||
|
||||
## 3a. Expression columns and aggregation order (ratio correctness)
|
||||
|
||||
**Expression columns are row-level.** Perspective evaluates every expression against
|
||||
each *raw row* first, then feeds the result into the column's aggregate. It has no
|
||||
post-aggregate expression stage. So a ratio written the obvious way:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' } // default aggregate: sum
|
||||
```
|
||||
|
||||
...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
|
||||
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
|
||||
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.
|
||||
|
||||
This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
|
||||
grouping just makes it visible by putting several wrong numbers side by side. No
|
||||
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
|
||||
`high`/`low`/`median`/`dominant` are all wrong for the same reason.
|
||||
|
||||
### The fix: a weighted-mean aggregate
|
||||
|
||||
Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
|
||||
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' },
|
||||
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
|
||||
```
|
||||
|
||||
**The nested array matters.** The type is
|
||||
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
|
||||
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
|
||||
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
|
||||
reads like "no such aggregate" and is easy to misread as the feature being absent.
|
||||
|
||||
Available since **4.4.0** — no version bump needed, and because the aggregate is
|
||||
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
|
||||
`table.update()` on a live view re-derives the weighted mean from the merged rows
|
||||
without a reload).
|
||||
|
||||
### Rules of thumb
|
||||
|
||||
- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
|
||||
→ leave the default `sum`. Those are unaffected by any of this.
|
||||
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
|
||||
`['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
|
||||
- Mixing both in one view is fine and was verified.
|
||||
|
||||
### Type inference can silently break expressions
|
||||
|
||||
`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
|
||||
as `integer` — which happens to numeric-looking strings when the table is created from
|
||||
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
|
||||
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
|
||||
error. Give period/year columns an explicit `string` type at table creation.
|
||||
|
||||
### Not fixed by any of this: column-axis expand/collapse
|
||||
|
||||
`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
|
||||
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
|
||||
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
|
||||
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
|
||||
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
|
||||
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
|
||||
(§2: `viewer-d3fc` still caps at 4.4.1).
|
||||
|
||||
---
|
||||
|
||||
## 4. Theming
|
||||
|
||||
- One toggle drives both app CSS and the viewer:
|
||||
`viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
|
||||
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
|
||||
re-themes when the toggle fires (not just on mount).
|
||||
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
|
||||
viewer theme name is the only Perspective-specific piece.
|
||||
|
||||
---
|
||||
|
||||
## 5. Layout persistence
|
||||
|
||||
- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
|
||||
`localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
|
||||
- **Guard restores against schema drift.** Before restoring, filter the saved config's
|
||||
`columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
|
||||
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
|
||||
the reference implementation; a stale layout referencing a dropped column otherwise
|
||||
throws on restore.
|
||||
- **`aggregates` needs the same guard, and doesn't currently have it** (verified 2026-08,
|
||||
pf_app). Both existing `cleanLayout()` implementations filter
|
||||
`columns`/`group_by`/`split_by`/`sort`/`filter` but leave `aggregates` untouched. That
|
||||
is harmless *today* only because `viewer.save()` emits `aggregates: {}` until someone
|
||||
sets one explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
|
||||
dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
|
||||
throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
|
||||
aggregate entry references a *target* column and, in the multi-arg form, a *weight*
|
||||
column; both need validating, and the entry should be dropped rather than the layout.
|
||||
Add this guard as part of adopting §3a, not after.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build & deploy (target)
|
||||
|
||||
- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
|
||||
`outDir: '../public/app'`). The API server serves it statically (`express.static`).
|
||||
- **Process:** run the Node API under **systemd** with `Restart=always` and an
|
||||
`EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
|
||||
dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
|
||||
automation yet** and should adopt the same pattern.
|
||||
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
|
||||
systemd unit); later runs rebuild the UI and restart the service.
|
||||
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
|
||||
unit — never commit it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Upgrade checklist / smoke test
|
||||
|
||||
Run this whenever bumping **any** Perspective package or `apache-arrow`:
|
||||
|
||||
1. Confirm `viewer-d3fc` publishes the target version
|
||||
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
|
||||
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
|
||||
commit the lockfile.
|
||||
- **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
|
||||
it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
|
||||
`latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
|
||||
and the WASM fails to link:
|
||||
`Import #8 "env" "psp_opfs_load": function import requires a callable`.
|
||||
Five packages, not four. This is invisible while pf_app loads from the CDN, and
|
||||
will bite on the "move off CDN" open item below.
|
||||
3. `vite build` — no unresolved imports.
|
||||
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
|
||||
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
|
||||
string/dictionary.
|
||||
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
|
||||
6. Toggle dark/light; confirm the viewer re-themes.
|
||||
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
|
||||
cleaned restore doesn't throw.
|
||||
|
||||
---
|
||||
|
||||
## Per-project state (2026-06)
|
||||
|
||||
| | pf_app | dataflow | Target |
|
||||
|---|---|---|---|
|
||||
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** |
|
||||
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** |
|
||||
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
|
||||
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
|
||||
| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test |
|
||||
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
|
||||
|
||||
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
|
||||
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
|
||||
|
||||
**Open items:**
|
||||
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages
|
||||
pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host
|
||||
blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
|
||||
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction):
|
||||
its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
|
||||
Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
|
||||
render before choosing the latter, because nobody has confirmed they do.
|
||||
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).
|
||||
@ -1,187 +0,0 @@
|
||||
# Perspective — lessons and rules
|
||||
|
||||
What we have learned the hard way about configuring, feeding and upgrading
|
||||
Perspective, written for any app that embeds `<perspective-viewer>` (**pf_app**,
|
||||
**dataflow**). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how
|
||||
to rebuild it — live in `ui/vendor/README.md`; how pf_app uses the viewer day to
|
||||
day is in `CLAUDE.md` and `pf_spec.md` → §The pivot.
|
||||
|
||||
> **Distribution:** we use the **`@perspective-dev/*`** packages
|
||||
> (<https://github.com/perspective-dev/perspective>) — **not** FINOS/OpenJS
|
||||
> `@finos/perspective`. Same engine lineage, but a separate npm scope, release
|
||||
> cadence and "Pro" theme set (`Pro Dark` / `Pro Light`). Don't mix the scopes.
|
||||
|
||||
---
|
||||
|
||||
## Perspective is one locked unit
|
||||
|
||||
These move together and are pinned together. Bumping one without the others is
|
||||
the source of nearly every Perspective bug we have hit:
|
||||
|
||||
1. **Loader** — how the JS/WASM gets into the page (npm `/inline`, never a CDN)
|
||||
2. **Package versions** — `client`, `server`, `viewer`, `viewer-datagrid` (and
|
||||
`viewer-d3fc` where charts are used)
|
||||
3. **Data format** — Arrow IPC vs JSON rows
|
||||
4. **`apache-arrow` on the server**, when delivering Arrow — it must write an IPC
|
||||
format the client WASM reads
|
||||
|
||||
**Pin `@perspective-dev/server` explicitly.** `@perspective-dev/client` declares
|
||||
it as `"@perspective-dev/server": ""` — an empty range, which npm resolves to
|
||||
`latest`. Installing an older client pulls whatever server is newest and the WASM
|
||||
fails to link (`Import #8 "env" "psp_opfs_load": function import requires a
|
||||
callable`). Five packages, not four.
|
||||
|
||||
---
|
||||
|
||||
## Load inline, never from a CDN
|
||||
|
||||
```js
|
||||
import perspective from '@perspective-dev/client/inline'
|
||||
import '@perspective-dev/viewer/inline'
|
||||
import '@perspective-dev/viewer-datagrid'
|
||||
import '@perspective-dev/viewer/themes'
|
||||
```
|
||||
|
||||
The `/inline` entrypoints bundle the WASM into the Vite build: no runtime fetch,
|
||||
works behind a firewall, reproducible from the lockfile. The themes CSS is
|
||||
imported in JS for the same reason, not `<link>`ed from `index.html`.
|
||||
|
||||
**This is not hypothetical — a CDN load took pf_app down on 2026-08-10.** The 4.x
|
||||
CDN bundle resolves its server WASM with
|
||||
`new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
|
||||
which from `.../client@4.4.0/dist/cdn/` lands on
|
||||
`.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no
|
||||
version**, so jsdelivr serves `@latest`. The moment a 5.x server published, every
|
||||
page load linked a 5.x WASM against a 4.4.0 client and threw the `LinkError`
|
||||
above. Nobody changed anything. The security framing matters as much as the
|
||||
outage: an unversioned URL means users execute whatever that package publishes
|
||||
next, unreviewed. `/inline` has no runtime fetch to hijack.
|
||||
|
||||
---
|
||||
|
||||
## Charts versus inline bundling (apps that need d3fc)
|
||||
|
||||
pf_app uses the datagrid only and does not carry `viewer-d3fc`. For an app that
|
||||
does, the constraint as of 2026-08: `viewer-d3fc` published nothing past 4.4.1,
|
||||
while `/inline` and `/themes` exist only from 4.5.x. **A 4.5.x viewer with a 4.4.1
|
||||
d3fc does not work** — every chart plugin throws `get_static_config is not a
|
||||
function` (reproduced in isolation, and seen in dataflow's install, where every
|
||||
chart type but the datagrid fails). So it is inline bundling **or** d3fc charts,
|
||||
not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1
|
||||
suite renders them — check before relying on it), or 5.x without them. Re-check
|
||||
when d3fc publishes a newer version.
|
||||
|
||||
---
|
||||
|
||||
## Arrow delivery: three coupled constraints
|
||||
|
||||
Arrow IPC is the right format for large, numeric, incrementally-updated tables;
|
||||
JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples
|
||||
three things:
|
||||
|
||||
- **Numeric type parsers, server-side.** pg returns `bigint` / `numeric` as
|
||||
strings; coerce them to numbers before encoding, or `apache-arrow` infers
|
||||
`Dictionary<Utf8>` instead of `Int` / `Float64`. pf_app: the type parsers in
|
||||
`server.js` (oids 20 and 1700).
|
||||
- **One record batch.** Per-batch encoding builds independent dictionaries, and
|
||||
the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit
|
||||
a single batch — pf_app does `tableToIPC(tableFromArrays(columns), 'stream')`.
|
||||
The client's "stream" is therefore a chunked download of one batch; nothing
|
||||
renders progressively.
|
||||
- **`apache-arrow` versus the client WASM.** The client is built against an older
|
||||
arrow major than pf_app's server pins; the pair was verified working, but any
|
||||
arrow bump has to be gated on the ingest smoke test below.
|
||||
|
||||
---
|
||||
|
||||
## Expression columns are row-level: ratios need a weighted mean
|
||||
|
||||
Perspective evaluates every expression against each **raw row**, then feeds the
|
||||
result into the column's aggregate; there is no post-aggregate expression stage.
|
||||
So `price: '"revenue" / "qty"'` under the default `sum` **sums the per-row
|
||||
ratios** — wrong at every grouped level, with or without `split_by`. `avg` gives
|
||||
the average of ratios, which is also wrong.
|
||||
|
||||
Weight the ratio by its own denominator — `sum(price_i × qty_i) / sum(qty_i)` is
|
||||
exactly `sum(revenue) / sum(qty)`, at every level of both axes:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' },
|
||||
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
|
||||
```
|
||||
|
||||
**The nested array matters.** `Aggregate = string | [string, Array<string>]`, so
|
||||
the weight column goes in its own array. The flat `['weighted mean', 'qty']` is
|
||||
rejected with `data did not match any variant of untagged enum Aggregate`, which
|
||||
reads like the feature is missing. Being engine-side, it stays correct across
|
||||
incremental `table.update()`.
|
||||
|
||||
- A **sum of a measure** (including `if(...)` subtotal expressions) → default
|
||||
`sum`.
|
||||
- A **ratio, rate, price or per-unit figure** → must carry
|
||||
`['weighted mean', ['<denominator>']]`.
|
||||
|
||||
**Type inference can silently break expressions.** `if("Year" == '2026',
|
||||
"Amount", 0)` returns 0 on every row if `Year` was *inferred* as integer from
|
||||
numeric-looking strings, with no error, and neither literal form fixes it. Give
|
||||
period and year columns an explicit `string` type at table creation.
|
||||
|
||||
---
|
||||
|
||||
## Theming
|
||||
|
||||
The viewer theme is a `ViewerConfig` field: `viewer.restore({ theme: 'Pro Dark' })`.
|
||||
**`setAttribute('theme', …)` does nothing in 5.x** — the element observes no such
|
||||
attribute. Apply it on load and whenever the app toggle changes. `viewer.save()`
|
||||
includes `theme`, so strip it from stored layouts and ignore it when comparing
|
||||
configs — otherwise restoring a layout reasserts its author's theme.
|
||||
|
||||
---
|
||||
|
||||
## Layouts: guard every restore against schema drift
|
||||
|
||||
Before `viewer.restore(cfg)`, filter the saved `columns` / `group_by` /
|
||||
`split_by` / `sort` / `filter` against the columns the current table actually has
|
||||
(plus its `expressions`); a stale layout naming a dropped column otherwise throws.
|
||||
**`aggregates` needs the same guard** — once a layout carries a weighted mean, a
|
||||
dropped column aborts the whole restore (`Could not get dtype for column 'X'`).
|
||||
An aggregate names a target column and, in the multi-arg form, a weight column;
|
||||
validate both and drop the *entry*, never the layout, since `restore()` is
|
||||
all-or-nothing. pf_app's `cleanLayout()` does all of this; dataflow's does not yet
|
||||
guard `aggregates`.
|
||||
|
||||
---
|
||||
|
||||
## Auto-pause deletes the view — turn it off for a static pivot
|
||||
|
||||
`<perspective-viewer>` auto-pauses by default (an `IntersectionObserver` on itself
|
||||
plus the document's `visibilitychange`). Pausing is not a paint optimisation:
|
||||
`session.set_pause(true)` runs `view_sub.take().delete()`, so the view is
|
||||
destroyed, and becoming visible again rebuilds it with a full traversal — a
|
||||
multi-second stall per tab switch on a large table, and any per-node
|
||||
expand/collapse is lost with it.
|
||||
|
||||
```js
|
||||
await viewer.load(table)
|
||||
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
|
||||
```
|
||||
|
||||
Leave it on only where a live stream feeds a table nobody needs kept current
|
||||
while away. Long-standing viewer behaviour, not a regression.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade smoke test
|
||||
|
||||
Run on any change to a Perspective package or `apache-arrow` (for pf_app, after
|
||||
rebuilding the vendored tarballs per `ui/vendor/README.md`):
|
||||
|
||||
1. All five packages (and `apache-arrow`) pinned exact and matching; lockfile
|
||||
committed.
|
||||
2. `vite build` — no unresolved imports.
|
||||
3. Arrow apps: load a real dataset; `worker.table(buffer)` ingests without a
|
||||
dictionary error, and a numeric column comes through as `Float64` / `Int`.
|
||||
4. Charts, where used: open a d3fc plugin and confirm it renders.
|
||||
5. Toggle dark/light; the viewer re-themes.
|
||||
6. Save a layout, reload, confirm it restores; drop a column and confirm the
|
||||
cleaned restore does not throw.
|
||||
1589
docs/pf_spec.md
1589
docs/pf_spec.md
File diff suppressed because it is too large
Load Diff
65
install.sh
Executable file
65
install.sh
Executable file
@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo " Pivot Forecast — Install"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
|
||||
# ── DB connection ─────────────────────────────────────────────
|
||||
read -p "DB host [192.168.1.110]: " DB_HOST
|
||||
DB_HOST=${DB_HOST:-192.168.1.110}
|
||||
|
||||
read -p "DB port [5432]: " DB_PORT
|
||||
DB_PORT=${DB_PORT:-5432}
|
||||
|
||||
read -p "DB name [ubm]: " DB_NAME
|
||||
DB_NAME=${DB_NAME:-ubm}
|
||||
|
||||
read -p "DB user [ptrowbridge]: " DB_USER
|
||||
DB_USER=${DB_USER:-ptrowbridge}
|
||||
|
||||
read -s -p "DB password: " DB_PASSWORD
|
||||
echo ""
|
||||
|
||||
read -p "App port [3030]: " PORT
|
||||
PORT=${PORT:-3030}
|
||||
|
||||
# ── Write .env ────────────────────────────────────────────────
|
||||
cat > .env <<EOF
|
||||
DB_HOST=${DB_HOST}
|
||||
DB_PORT=${DB_PORT}
|
||||
DB_NAME=${DB_NAME}
|
||||
DB_USER=${DB_USER}
|
||||
DB_PASSWORD=${DB_PASSWORD}
|
||||
PORT=${PORT}
|
||||
EOF
|
||||
echo "✓ .env written"
|
||||
|
||||
# ── npm install ───────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "Installing node dependencies..."
|
||||
npm install --silent
|
||||
echo "✓ dependencies installed"
|
||||
|
||||
# ── schema install ────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "Installing pf schema into ${DB_NAME}..."
|
||||
PGPASSWORD=${DB_PASSWORD} psql \
|
||||
-h "${DB_HOST}" \
|
||||
-p "${DB_PORT}" \
|
||||
-U "${DB_USER}" \
|
||||
-d "${DB_NAME}" \
|
||||
-f setup_sql/01_schema.sql
|
||||
|
||||
echo "✓ schema installed"
|
||||
|
||||
# ── done ─────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo " Install complete"
|
||||
echo " Start with: npm run dev"
|
||||
echo " Open: http://$(hostname -I | awk '{print $1}'):${PORT}"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
92
lib/auth.js
92
lib/auth.js
@ -1,92 +0,0 @@
|
||||
// Password hashing and the route guard.
|
||||
//
|
||||
// Hashes are scrypt, from node's own crypto — no native build step, and the
|
||||
// stored form carries its own parameters so they can be raised later without
|
||||
// invalidating existing rows:
|
||||
//
|
||||
// scrypt$<N>$<r>$<p>$<salt base64>$<derived key base64>
|
||||
|
||||
const crypto = require('crypto');
|
||||
|
||||
const SCRYPT = { N: 16384, r: 8, p: 1, keylen: 64 };
|
||||
|
||||
function hashPassword(password, params = SCRYPT) {
|
||||
const { N, r, p, keylen } = params;
|
||||
const salt = crypto.randomBytes(16);
|
||||
const dk = crypto.scryptSync(password, salt, keylen, { N, r, p, maxmem: 256 * 1024 * 1024 });
|
||||
return `scrypt$${N}$${r}$${p}$${salt.toString('base64')}$${dk.toString('base64')}`;
|
||||
}
|
||||
|
||||
// Constant-time compare. Returns false rather than throwing on a malformed or
|
||||
// legacy hash, so one bad row can't 500 the login route.
|
||||
function verifyPassword(password, stored) {
|
||||
if (typeof stored !== 'string') return false;
|
||||
const parts = stored.split('$');
|
||||
if (parts.length !== 6 || parts[0] !== 'scrypt') return false;
|
||||
|
||||
const [, N, r, p, saltB64, dkB64] = parts;
|
||||
try {
|
||||
const salt = Buffer.from(saltB64, 'base64');
|
||||
const expected = Buffer.from(dkB64, 'base64');
|
||||
const actual = crypto.scryptSync(password, salt, expected.length, {
|
||||
N: Number(N), r: Number(r), p: Number(p), maxmem: 256 * 1024 * 1024,
|
||||
});
|
||||
return crypto.timingSafeEqual(actual, expected);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Every /api route except the auth ones sits behind this.
|
||||
function requireAuth(req, res, next) {
|
||||
if (req.session?.user?.username) return next();
|
||||
res.status(401).json({ error: 'Not authenticated' });
|
||||
}
|
||||
|
||||
// The identity used for pf.log.pf_user and the created_by/closed_by columns.
|
||||
// Read from the session only — never from the request body, which the browser
|
||||
// controls and which used to carry a hardcoded 'admin'.
|
||||
function sessionUser(req) {
|
||||
return req.session?.user?.username || null;
|
||||
}
|
||||
|
||||
// What this account may see and change, read from the session for the same
|
||||
// reason the username is: the browser must not be able to widen it.
|
||||
//
|
||||
// Returns { admin: true } for an account that sees everything, or
|
||||
// { admin: false, values: [...] } for a scoped one. An empty list is a real
|
||||
// answer meaning "nothing", not a missing one meaning "everything" -- an
|
||||
// account created without a territory sees no rows until it is granted some.
|
||||
function sessionTerritory(req) {
|
||||
const u = req.session?.user;
|
||||
if (!u) return { admin: false, values: [] };
|
||||
if (u.is_admin) return { admin: true, values: null };
|
||||
return { admin: false, values: Array.isArray(u.territory) ? u.territory : [] };
|
||||
}
|
||||
|
||||
function requireAdmin(req, res, next) {
|
||||
if (req.session?.user?.is_admin) return next();
|
||||
res.status(403).json({ error: 'Administrator access required' });
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
hashPassword, verifyPassword, requireAuth, requireAdmin,
|
||||
sessionUser, sessionTerritory, SCRYPT,
|
||||
};
|
||||
|
||||
// CLI: `node lib/auth.js hash` reads a password on stdin and prints its hash,
|
||||
// so ./pf.sh can create users without the plaintext touching argv or psql.
|
||||
if (require.main === module) {
|
||||
if (process.argv[2] !== 'hash') {
|
||||
console.error('usage: node lib/auth.js hash (password on stdin)');
|
||||
process.exit(2);
|
||||
}
|
||||
let input = '';
|
||||
process.stdin.setEncoding('utf8');
|
||||
process.stdin.on('data', chunk => { input += chunk; });
|
||||
process.stdin.on('end', () => {
|
||||
const password = input.replace(/\r?\n$/, '');
|
||||
if (!password) { console.error('empty password'); process.exit(2); }
|
||||
process.stdout.write(hashPassword(password) + '\n');
|
||||
});
|
||||
}
|
||||
@ -1,260 +1,52 @@
|
||||
// Generates operation SQL for a source table, baking in column names from col_meta.
|
||||
// Runtime values are left as {{token}} substitution points.
|
||||
//
|
||||
// Columns flagged col_meta.in_grain define a display grain. When one is set the
|
||||
// initial load (get_agg) and every operation return rows pre-aggregated to that
|
||||
// grain and keyed on pf_gkey, instead of raw forecast rows keyed on pf_id.
|
||||
//
|
||||
// Tokens baked in at generation time: column names, source schema.table
|
||||
// Tokens substituted at request time: {{fc_table}}, {{where_clause}}, {{exclude_clause}},
|
||||
// {{version_id}}, {{logid}}, {{pf_user}}, {{note}},
|
||||
// {{label}}, {{bucket}}, {{tag}}, {{territory_clause}},
|
||||
// {{params}}, {{slice}}, {{date_from}}, {{date_to}},
|
||||
// {{value_incr}}, {{units_incr}}, {{set_clause}}, {{scale_factor}}
|
||||
|
||||
// What the pivot shows for a row's segment and its bucket.
|
||||
//
|
||||
// Both are STORED on the row's pf.log entry -- l.label and l.bucket -- and read
|
||||
// back as plain columns. Every entry carries both: pf.log's log_named CHECK
|
||||
// refuses one without, so a report can join pf.log and select them with no rule
|
||||
// of its own.
|
||||
//
|
||||
// They used to be derived at read time, through a chain of fallbacks (a load's
|
||||
// label, else its tag, else its note, else the version's unlabeled_load; an
|
||||
// adjustment's label, else the version's adjustment_segment, else a constant).
|
||||
// That made the pivot the only reader that knew what bucket a row was in: the
|
||||
// same CASE was repeated in /agg, the /data cursor and the decorate proc in
|
||||
// /opt/forecast, and anything querying pf directly had to copy it again. It also
|
||||
// meant editing a version's fallback name silently renamed every past adjustment.
|
||||
//
|
||||
// The ordering prefix is part of the stored text, not computed here. Perspective
|
||||
// orders column groups by the value string, so "01 - Actual" is the only way an
|
||||
// arbitrary order can be expressed -- and l.label is where a person types it.
|
||||
//
|
||||
// ---------------------------------------------------------------------------
|
||||
// ADJUSTMENT DEFAULTS -- what a scale/recode/clone is stamped with.
|
||||
//
|
||||
// An adjustment has no label or bucket input, so the route stamps these onto its
|
||||
// log entry when it is written: the version's own adjustment_segment /
|
||||
// adjustment_bucket when set, else the constants below. Resolved once, at write
|
||||
// time -- changing a version's default afterwards names new adjustments, not old
|
||||
// ones. Exported so the operation routes stamp the rows they push back to the
|
||||
// pivot with the same values they wrote.
|
||||
//
|
||||
// A load has no default: its label is required, and its bucket is its label
|
||||
// unless one is given (the routes enforce both).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// An adjustment with no label of its own. The 99 keeps it after every numbered
|
||||
// segment -- ordering is string ordering, so this only works while the loads
|
||||
// carry 01-0n.
|
||||
const ADJUSTMENT_SEGMENT = '99 - Adjustments';
|
||||
|
||||
// What an adjustment counts toward. Prefixed to match the segments it adjusts,
|
||||
// so the adjustments sit in the same column as the rows they adjust.
|
||||
const ADJUSTMENT_BUCKET = '04 - Forecast';
|
||||
|
||||
const SEGMENT_EXPR = `l.label`;
|
||||
const BUCKET_EXPR = `l.bucket`;
|
||||
|
||||
const NOTE_EXPR = `CASE WHEN l.operation IN ('baseline','reference')
|
||||
THEN NULL
|
||||
ELSE COALESCE(NULLIF(l.tag, ''), NULLIF(l.note, ''))
|
||||
END`;
|
||||
|
||||
// Every pf.log column the expressions above read, for /agg's GROUP BY: they
|
||||
// are functionally dependent on pf_logid, which is in the grain, but Postgres
|
||||
// will not infer that.
|
||||
const LABEL_GROUP_COLS = ['l.operation', 'l.label', 'l.tag', 'l.note', 'l.bucket'];
|
||||
|
||||
// wrap a column name in double quotes for safe use in SQL
|
||||
function q(name) { return `"${name}"`; }
|
||||
|
||||
// The display grain: dimension/date columns flagged in_grain, plus pf_iter and
|
||||
// pf_logid which are always part of it. Returns null when nothing is flagged —
|
||||
// that is raw-row mode, where operations return whole rows and the client
|
||||
// indexes on pf_id (the pre-grain behaviour).
|
||||
//
|
||||
// Keeping pf_logid in the grain is what makes the append model work: each
|
||||
// operation's contribution stays a distinct row, so table.update() accumulates
|
||||
// rather than replacing a bucket total, and undo can remove exactly that
|
||||
// operation's rows.
|
||||
function grainOf(colMeta) {
|
||||
const cols = colMeta
|
||||
.filter(c => c.in_grain && (c.role === 'dimension' || c.role === 'date'))
|
||||
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
|
||||
.map(c => c.cname);
|
||||
if (cols.length === 0) return null;
|
||||
|
||||
// pf_gkey must be unique per grain tuple. chr(31) (unit separator) joins the
|
||||
// parts and chr(30) stands in for NULL, so ('a', NULL) cannot collide with
|
||||
// (NULL, 'a') and a NULL stays distinct from an empty string — a collision
|
||||
// would silently merge two groups into one indexed row.
|
||||
//
|
||||
// md5 of that, rather than the concatenation itself, because the key is an
|
||||
// opaque handle -- nothing reads it but table.update() and table.remove().
|
||||
// The raw form averaged 233 chars on a 24-column grain and, being unique per
|
||||
// row, defeated Arrow's dictionary encoding: 65.6 MB of a 109 MB payload,
|
||||
// more than every other column combined. 128 bits keeps collisions unreachable.
|
||||
const key = (pfx = '') => `md5(concat_ws(chr(31), ${[
|
||||
...cols.map(c => `COALESCE(${pfx}${q(c)}::text, chr(30))`),
|
||||
`${pfx}pf_iter`,
|
||||
`${pfx}pf_logid::text`
|
||||
].join(', ')}))`;
|
||||
|
||||
const groupCols = (pfx = '') => [...cols.map(c => `${pfx}${q(c)}`), `${pfx}pf_iter`, `${pfx}pf_logid`];
|
||||
|
||||
return { cols, key, groupCols };
|
||||
}
|
||||
|
||||
// Date columns that anchor a dim_group, each with the dimensions derived from
|
||||
// pf.dim_period for it. Shared so the generator and the routes agree on both the
|
||||
// membership and the join aliases -- the same reason grainOf is a single function.
|
||||
function dateGroupsOf(colMeta) {
|
||||
return colMeta
|
||||
.filter(c => c.role === 'date' && c.is_key && c.dim_group)
|
||||
.map((keyCol, i) => ({
|
||||
alias: `dp${i + 1}`,
|
||||
dateCol: keyCol.cname,
|
||||
group: keyCol.dim_group,
|
||||
derived: colMeta
|
||||
.filter(c => c.role === 'dimension'
|
||||
&& c.dim_group === keyCol.dim_group
|
||||
&& c.dim_period_col)
|
||||
.map(c => ({ cname: c.cname, periodCol: c.dim_period_col })),
|
||||
}))
|
||||
.filter(g => g.derived.length > 0);
|
||||
}
|
||||
|
||||
// cname -> which join it comes from and which of its columns
|
||||
function dimPeriodMapOf(dateGroups) {
|
||||
return new Map(
|
||||
dateGroups.flatMap(g => g.derived.map(d => [d.cname, { alias: g.alias, periodCol: d.periodCol }]))
|
||||
);
|
||||
}
|
||||
|
||||
// The joins themselves, against a date that has already had {{date_offset}}
|
||||
// applied. LEFT because a null date -- an order not yet shipped has no ship date
|
||||
// -- must leave the period columns empty rather than drop the row.
|
||||
function dimPeriodJoins(dateGroups, alias = 's') {
|
||||
return dateGroups.map(g =>
|
||||
`\n LEFT JOIN pf.dim_period ${g.alias}`
|
||||
+ ` ON ${g.alias}.drange @> (${alias}."${g.dateCol}" + '{{date_offset}}'::interval)::date`
|
||||
).join('');
|
||||
}
|
||||
|
||||
function generateSQL(source, colMeta) {
|
||||
const dims = colMeta
|
||||
.filter(c => c.role === 'dimension')
|
||||
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
|
||||
.map(c => c.cname);
|
||||
|
||||
// Every column of each measure/date role, in col_meta order. Loads carry all of
|
||||
// them; the adjustment operations are single-measure (scale distributes one
|
||||
// {{value_incr}}) and use only the first of each, below.
|
||||
const byRole = role => colMeta
|
||||
.filter(c => c.role === role)
|
||||
.sort((a, b) => (a.opos || 0) - (b.opos || 0))
|
||||
.map(c => c.cname);
|
||||
|
||||
const valueCols = byRole('value');
|
||||
const unitsCols = byRole('units');
|
||||
const dateCols = byRole('date');
|
||||
|
||||
const valueCol = valueCols[0];
|
||||
const unitsCol = unitsCols[0];
|
||||
const valueCol = colMeta.find(c => c.role === 'value')?.cname;
|
||||
const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
|
||||
const dateCol = colMeta.find(c => c.role === 'date')?.cname;
|
||||
|
||||
if (!valueCol) throw new Error('No value column defined in col_meta');
|
||||
if (!dateCols.length) throw new Error('No date column defined in col_meta');
|
||||
if (!dateCol) throw new Error('No date column defined in col_meta');
|
||||
if (dims.length === 0) throw new Error('No dimension columns defined in col_meta');
|
||||
|
||||
const srcTable = `"${source.schema}"."${source.tname}"`;
|
||||
// Every date column, not just the primary one. Scale and recode used to
|
||||
// carry dateCols[0] alone, so a source with order, requested and ship dates
|
||||
// wrote adjustment rows whose rdate and sdate_e were *null* -- not stale,
|
||||
// absent. Those columns are role 'date' and therefore sliceable, so a later
|
||||
// adjustment scoped to a ship date matched no adjustment row ever written,
|
||||
// and any reader that dated the forecast by ship date lost every
|
||||
// adjustment silently. The derived period dimensions (rseas, smon_e) were
|
||||
// fine throughout, being dimensions and copied with the rest -- which is
|
||||
// what made it invisible: the labels were right and the dates underneath
|
||||
// them were gone.
|
||||
const dataCols = [...dims, ...dateCols, valueCol, unitsCol].filter(Boolean);
|
||||
const dataCols = [...dims, dateCol, valueCol, unitsCol].filter(Boolean);
|
||||
const effectiveValue = dataCols.includes(valueCol) ? valueCol : null;
|
||||
const effectiveUnits = dataCols.includes(unitsCol) ? unitsCol : null;
|
||||
const insertCols = [...dataCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
|
||||
const selectData = dataCols.map(q).join(', ');
|
||||
const dimsJoined = dims.map(q).join(', ');
|
||||
// The date columns as a select list, in the same order dataCols lists them,
|
||||
// so it lines up with insertCols positionally.
|
||||
const datesJoined = dateCols.map(q).join(', ');
|
||||
|
||||
// Baseline and reference copy the source row wholesale, so they carry every
|
||||
// measure and every date — not just the primary one the operations act on.
|
||||
// Dropping the others would leave those columns null for the life of the version.
|
||||
// Clone carries every date column, not just the primary one, because it is the
|
||||
// operation that moves rows through time: {{date_offset}} shifts them all
|
||||
// together, and the period dimensions are re-derived from pf.dim_period against
|
||||
// the shifted dates rather than copied from the row being cloned. Cloning last
|
||||
// year's mix forward a year otherwise produces rows dated 2027 still labelled
|
||||
// with 2026's periods.
|
||||
const cloneCols = [...dims, ...dateCols, effectiveValue, effectiveUnits].filter(Boolean);
|
||||
const cloneInsertCols = [...cloneCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
|
||||
|
||||
// An adjustment writes one row per *coordinate* it touches, not one per row
|
||||
// it reads.
|
||||
//
|
||||
// Reading rows one-for-one meant every operation inherited the row count of
|
||||
// everything before it: the baseline's rows plus every prior adjustment's
|
||||
// rows at the same coordinate, so the table grew super-linearly with how
|
||||
// much work had been done on it. Eight Pull Forward entries and the next
|
||||
// scale over the same slice writes nine times what it needs to.
|
||||
//
|
||||
// Collapsing is over pf_logid and pf_iter only -- every stored dimension and
|
||||
// date stays in the GROUP BY -- so no column goes null and nothing becomes
|
||||
// unsliceable later. The distribution maths is untouched either way, since
|
||||
// the window sums see the same totals whether or not the rows underneath
|
||||
// them have been added up first.
|
||||
const groupCols = (cols) => cols.map(q).join(',\n ');
|
||||
|
||||
const loadCols = [...dims, ...dateCols, ...valueCols, ...unitsCols];
|
||||
const loadInsertCols = [...loadCols.map(q), 'pf_iter', 'pf_logid', 'pf_user', 'pf_created_at'].join(', ');
|
||||
const dateColSet = new Set(dateCols);
|
||||
|
||||
// dim_period JOIN support: a date column that is the is_key of a dim_group
|
||||
// anchors that group, and dimension siblings with dim_period_col set are
|
||||
// derived from pf.dim_period instead of copied raw. Derivation is against the
|
||||
// date *after* {{date_offset}}, which is the whole point -- shift a baseline
|
||||
// forward a year and its period columns follow, rather than still naming the
|
||||
// year it came from.
|
||||
//
|
||||
// Every such group, not just the first. This used to be a find(), so a source
|
||||
// with order, requested and ship date groups derived the order one and copied
|
||||
// the other two raw -- shifted dates against unshifted period labels.
|
||||
const dateGroups = dateGroupsOf(colMeta);
|
||||
const dimPeriodMap = dimPeriodMapOf(dateGroups);
|
||||
// dim_period JOIN support: if the date column is the is_key of a dim_group,
|
||||
// dimension siblings with dim_period_col set are derived from pf.dim_period
|
||||
// instead of being copied raw from the source on baseline/reference load.
|
||||
const dateKeyGroup = colMeta.find(c => c.role === 'date' && c.is_key && c.dim_group)?.dim_group;
|
||||
const dimPeriodMap = new Map(
|
||||
dateKeyGroup
|
||||
? colMeta
|
||||
.filter(c => c.role === 'dimension' && c.dim_group === dateKeyGroup && c.dim_period_col)
|
||||
.map(c => [c.cname, c.dim_period_col])
|
||||
: []
|
||||
);
|
||||
const hasDimPeriod = dimPeriodMap.size > 0;
|
||||
|
||||
// display grain — when set, initial load and operations both return rows
|
||||
// pre-aggregated to it instead of raw forecast rows
|
||||
const grain = grainOf(colMeta);
|
||||
// A flag on anything other than a dimension/date column is ignored by grainOf,
|
||||
// which is what we want — the role change is the source of truth, not a stale flag.
|
||||
if (grain) {
|
||||
const missing = grain.cols.filter(c => !dataCols.includes(c));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(
|
||||
`Grain columns are never populated in the forecast table: ${missing.join(', ')}`
|
||||
);
|
||||
}
|
||||
if (!effectiveValue && !effectiveUnits) {
|
||||
throw new Error('A grain requires at least one value or units column to aggregate');
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
get_data: buildGetData(),
|
||||
...(grain ? { get_agg: buildGetAgg() } : {}),
|
||||
baseline: buildBaseline(),
|
||||
reference: buildReference(),
|
||||
scale: buildScale(),
|
||||
@ -267,83 +59,31 @@ function generateSQL(source, colMeta) {
|
||||
return `SELECT * FROM {{fc_table}}`;
|
||||
}
|
||||
|
||||
// Aggregate the whole forecast table to the display grain. This is the initial
|
||||
// load for grain sources — the client loads the result into a native Perspective
|
||||
// table indexed on pf_gkey and its view sums across these rows, exactly as an
|
||||
// Excel pivot cache sums its data tab.
|
||||
function buildGetAgg() {
|
||||
// pf_logid is part of the grain, so joining pf.log adds no rows — each group
|
||||
// already belongs to exactly one log entry. Without this the segment labels
|
||||
// that /data surfaces would vanish the moment a source declares a grain.
|
||||
return `
|
||||
SELECT
|
||||
${grainSelect('t.')}
|
||||
,${SEGMENT_EXPR} AS pf_segment
|
||||
,${BUCKET_EXPR} AS pf_bucket
|
||||
,${NOTE_EXPR} AS pf_note
|
||||
,l.operation AS pf_op
|
||||
FROM {{fc_table}} t
|
||||
LEFT JOIN pf.log l
|
||||
ON l.id = t.pf_logid
|
||||
WHERE {{territory_clause}}
|
||||
GROUP BY
|
||||
${grain.groupCols('t.').join('\n ,')}
|
||||
,${LABEL_GROUP_COLS.join('\n ,')}`.trim();
|
||||
}
|
||||
|
||||
// grain columns + pf_gkey + summed measures, in the leading-comma style the
|
||||
// rest of the generated SQL uses
|
||||
function grainSelect(pfx = '') {
|
||||
return [
|
||||
...grain.groupCols(pfx),
|
||||
`${grain.key(pfx)} AS pf_gkey`,
|
||||
effectiveValue ? `SUM(${pfx}${q(effectiveValue)}) AS ${q(effectiveValue)}` : null,
|
||||
effectiveUnits ? `SUM(${pfx}${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : null
|
||||
].filter(Boolean).join('\n ,');
|
||||
}
|
||||
|
||||
// Tail of an operation statement: in grain mode the inserted rows come back
|
||||
// aggregated to grain (the client appends them and lets the view re-sum);
|
||||
// otherwise whole rows come back as before.
|
||||
function opTail(cte) {
|
||||
if (!grain) return `SELECT * FROM ${cte}`;
|
||||
return `
|
||||
SELECT
|
||||
${grainSelect()}
|
||||
FROM ${cte}
|
||||
GROUP BY
|
||||
${grain.groupCols().join('\n ,')}`.trim();
|
||||
}
|
||||
|
||||
function buildLoadSelect(pfx) {
|
||||
// pfx: table alias prefix ('s.' when joining dim_period, '' otherwise)
|
||||
// The offset shifts every date column, so order date and ship date stay in step.
|
||||
return loadCols.map(c => {
|
||||
if (dateColSet.has(c)) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
|
||||
if (dimPeriodMap.has(c)) {
|
||||
const { alias, periodCol } = dimPeriodMap.get(c);
|
||||
return `${alias}.${q(periodCol)} AS ${q(c)}`;
|
||||
}
|
||||
return dataCols.map(c => {
|
||||
if (c === dateCol) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
|
||||
if (dimPeriodMap.has(c)) return `dp.${q(dimPeriodMap.get(c))} AS ${q(c)}`;
|
||||
return `${pfx}${q(c)}`;
|
||||
}).join(',\n ');
|
||||
}
|
||||
|
||||
function buildFromClause() {
|
||||
if (!hasDimPeriod) return srcTable;
|
||||
return srcTable + ' s' + dimPeriodJoins(dateGroups);
|
||||
return `${srcTable} s\n JOIN pf.dim_period dp`
|
||||
+ ` ON dp.drange @> (s.${q(dateCol)} + '{{date_offset}}'::interval)::date`;
|
||||
}
|
||||
|
||||
function buildBaseline() {
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}',
|
||||
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${loadInsertCols})
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT
|
||||
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
|
||||
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
@ -351,20 +91,19 @@ ilog AS (
|
||||
WHERE {{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim();
|
||||
SELECT count(*) AS rows_affected FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildReference() {
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket, tag)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}',
|
||||
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''), NULLIF('{{tag}}', ''))
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${loadInsertCols})
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT
|
||||
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
|
||||
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
@ -372,7 +111,7 @@ ilog AS (
|
||||
WHERE {{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim();
|
||||
SELECT count(*) AS rows_affected FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildScale() {
|
||||
@ -382,23 +121,19 @@ SELECT count(*) AS rows_affected, (SELECT id FROM ilog) AS log_id FROM ins`.trim
|
||||
const uSel = effectiveUnits
|
||||
? `round((${q(effectiveUnits)} / NULLIF(total_units, 0)) * {{units_incr}}, 5)`
|
||||
: `0`;
|
||||
// sum(sum(x)) OVER () is the aggregate of the aggregates: the window runs
|
||||
// after the GROUP BY, so the total is over collapsed coordinates and
|
||||
// comes to the same figure the ungrouped window produced.
|
||||
const baseSelectParts = [
|
||||
...dimsJoined ? [dimsJoined] : [],
|
||||
datesJoined,
|
||||
effectiveValue ? `sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : null,
|
||||
effectiveUnits ? `sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : null,
|
||||
effectiveValue ? `sum(sum(${q(effectiveValue)})) OVER () AS total_value` : null,
|
||||
effectiveUnits ? `sum(sum(${q(effectiveUnits)})) OVER () AS total_units` : null
|
||||
q(dateCol),
|
||||
effectiveValue ? q(effectiveValue) : null,
|
||||
effectiveUnits ? q(effectiveUnits) : null,
|
||||
effectiveValue ? `sum(${q(effectiveValue)}) OVER () AS total_value` : null,
|
||||
effectiveUnits ? `sum(${q(effectiveUnits)}) OVER () AS total_units` : null
|
||||
].filter(Boolean).join(',\n ');
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'scale', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}',
|
||||
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''))
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'scale', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
,base AS (
|
||||
@ -407,105 +142,70 @@ ilog AS (
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
GROUP BY
|
||||
${groupCols([...dims, ...dateCols])}
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT
|
||||
${[dimsJoined, datesJoined, ...(effectiveValue ? [vSel] : []), ...(effectiveUnits ? [uSel] : [])].join(',\n ')},
|
||||
${[dimsJoined, q(dateCol), ...(effectiveValue ? [vSel] : []), ...(effectiveUnits ? [uSel] : [])].join(',\n ')},
|
||||
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM base
|
||||
RETURNING *
|
||||
)
|
||||
${opTail('ins')}`.trim();
|
||||
SELECT * FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildRecode() {
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'recode', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}',
|
||||
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''))
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'recode', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
,src AS (
|
||||
SELECT
|
||||
${dimsJoined},
|
||||
${datesJoined}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''}
|
||||
SELECT ${selectData}
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
GROUP BY
|
||||
${groupCols([...dims, ...dateCols])}
|
||||
-- a coordinate whose earlier adjustments already net it to zero has nothing to
|
||||
-- move; writing it anyway is two empty rows per coordinate. Rare on a first
|
||||
-- recode, but a second recode over cells a first one emptied would rewrite
|
||||
-- every one of them
|
||||
HAVING ${[effectiveValue, effectiveUnits].filter(Boolean).map(c => `sum(${q(c)}) <> 0`).join(' OR ')}
|
||||
)
|
||||
,neg AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT ${dimsJoined}, ${datesJoined}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
|
||||
SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
|
||||
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
RETURNING *
|
||||
)
|
||||
-- the put-back leg, moved by {{date_offset}} ('0 days' for a plain recode). The
|
||||
-- route resolves {{set_clause}} against alias s, deriving the period dimensions
|
||||
-- from the dim_period joins below only when the offset is non-zero.
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT {{set_clause}}, ${dateCols.map(c => `(s.${q(c)} + '{{date_offset}}'::interval)::date`).join(', ')}, ${effectiveValue ? `s.${q(effectiveValue)}` : '0'}${effectiveUnits ? `, s.${q(effectiveUnits)}` : ''},
|
||||
SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}${effectiveUnits ? `, ${q(effectiveUnits)}` : ''},
|
||||
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src s${hasDimPeriod ? dimPeriodJoins(dateGroups) : ''}
|
||||
FROM src
|
||||
RETURNING *
|
||||
)
|
||||
${grain ? `,allrows AS (
|
||||
SELECT * FROM neg
|
||||
UNION ALL
|
||||
SELECT * FROM ins
|
||||
)
|
||||
${opTail('allrows')}` : 'SELECT * FROM neg UNION ALL SELECT * FROM ins'}`.trim();
|
||||
SELECT * FROM neg UNION ALL SELECT * FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildClone() {
|
||||
const select = [
|
||||
// dims: whatever {{set_clause}} resolves them to. The route builds it,
|
||||
// and substitutes the dim_period expression for any derived dimension
|
||||
// the caller has not overridden outright.
|
||||
'{{set_clause}}',
|
||||
...dateCols.map(c => `(s.${q(c)} + '{{date_offset}}'::interval)::date`),
|
||||
effectiveValue ? `round(s.${q(effectiveValue)} * {{scale_factor}}, 2)` : null,
|
||||
effectiveUnits ? `round(s.${q(effectiveUnits)} * {{scale_factor}}, 5)` : null,
|
||||
].filter(Boolean).join(',\n ');
|
||||
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note, label, bucket)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}',
|
||||
NULLIF('{{label}}', ''), NULLIF('{{bucket}}', ''))
|
||||
INSERT INTO pf.log (version_id, pf_user, operation, slice, params, note)
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${cloneInsertCols})
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT
|
||||
${select},
|
||||
{{set_clause}},
|
||||
${q(dateCol)},
|
||||
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}${effectiveUnits ? `,\n round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : ''},
|
||||
'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM (
|
||||
SELECT
|
||||
${groupCols([...dims, ...dateCols])}${effectiveValue ? `,\n sum(${q(effectiveValue)}) AS ${q(effectiveValue)}` : ''}${effectiveUnits ? `,\n sum(${q(effectiveUnits)}) AS ${q(effectiveUnits)}` : ''}
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
GROUP BY
|
||||
${groupCols([...dims, ...dateCols])}
|
||||
) s${hasDimPeriod ? dimPeriodJoins(dateGroups) : ''}
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
${opTail('ins')}`.trim();
|
||||
SELECT * FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildUndo() {
|
||||
@ -532,98 +232,17 @@ function applyTokens(sql, tokens) {
|
||||
|
||||
// build a SQL WHERE clause string from a slice object
|
||||
// only dimension columns are included; unrecognised keys are silently skipped
|
||||
// pf_segment and pf_bucket are not columns on the forecast table -- they are
|
||||
// stored on the row's pf.log entry -- so a slice naming one cannot be compared
|
||||
// directly. It resolves to a set of log ids instead, which is
|
||||
// exact: the name lives on the log row, and every forecast row carries the
|
||||
// pf_logid that points at it.
|
||||
//
|
||||
// Without this they were dropped from the slice, and clicking a single bucket's
|
||||
// cell scaled every bucket at that dimension intersection while the panel showed
|
||||
// only the one clicked.
|
||||
//
|
||||
// pf_note and pf_op are the same kind of column -- the pivot shows them, from
|
||||
// the log entry -- and used to be refused, which the client pre-empted by
|
||||
// stripping them. The ledger did not strip them, so a pivot grouped by one
|
||||
// totalled a narrower set of rows than the operation then wrote.
|
||||
const COMPUTED_SLICE_COLS = {
|
||||
pf_segment: SEGMENT_EXPR,
|
||||
pf_bucket: BUCKET_EXPR,
|
||||
pf_note: NOTE_EXPR,
|
||||
pf_op: 'l.operation',
|
||||
};
|
||||
|
||||
// pf_ columns that are real columns of the forecast table, compared directly.
|
||||
// pf_iter was stripped by the client and skipped here, on the reasoning that
|
||||
// two cells differing only by iter band are the same slice. They are not to the
|
||||
// ledger, which filtered on it: with the default layout grouped by pf_iter, a
|
||||
// click on a customer's `scale` row showed the scale rows' total and wrote to
|
||||
// every row the customer has.
|
||||
const TABLE_SLICE_COLS = new Set(['pf_iter', 'pf_user', 'pf_logid']);
|
||||
|
||||
function computedSlicePredicate(col, val, versionId) {
|
||||
if (versionId == null) {
|
||||
const err = new Error(`Cannot filter on ${col} without a version`);
|
||||
err.status = 500;
|
||||
throw err;
|
||||
}
|
||||
// a blank is a value here too -- a load has no note -- and `IN (NULL)` is
|
||||
// never true, so the null arm is ORed on as buildWhere does
|
||||
const list = Array.isArray(val) ? val : [val];
|
||||
const nulls = list.some(v => v === null);
|
||||
const vals = list.filter(v => v !== null).map(v => `'${esc(v)}'`);
|
||||
const expr = COMPUTED_SLICE_COLS[col];
|
||||
const inList = vals.length ? `(${expr}) IN (${vals.join(', ')})` : null;
|
||||
const test = nulls && inList ? `((${expr}) IS NULL OR ${inList})`
|
||||
: nulls ? `(${expr}) IS NULL`
|
||||
: inList;
|
||||
return `pf_logid IN (
|
||||
SELECT l.id
|
||||
FROM pf.log l
|
||||
WHERE l.version_id = ${parseInt(versionId)}
|
||||
AND ${test}
|
||||
)`;
|
||||
}
|
||||
|
||||
function buildWhere(slice, dimCols, versionId) {
|
||||
function buildWhere(slice, dimCols) {
|
||||
if (!slice || Object.keys(slice).length === 0) return 'TRUE';
|
||||
|
||||
const allowed = new Set(dimCols);
|
||||
const parts = [];
|
||||
|
||||
for (const [col, val] of Object.entries(slice)) {
|
||||
if (COMPUTED_SLICE_COLS[col]) {
|
||||
parts.push(computedSlicePredicate(col, val, versionId));
|
||||
continue;
|
||||
}
|
||||
// A pf_ key this does not understand is refused rather than skipped.
|
||||
// Skipping is how a selection silently widened: the operation ran against
|
||||
// everything the dropped key would have excluded.
|
||||
if (col.startsWith('pf_') && !TABLE_SLICE_COLS.has(col)) {
|
||||
const err = new Error(`Slice names ${col}, which cannot be filtered on`);
|
||||
err.status = 400;
|
||||
throw err;
|
||||
}
|
||||
if (!allowed.has(col) && !TABLE_SLICE_COLS.has(col)) continue;
|
||||
// A blank dimension value is a value. Both blanks exist in this data and
|
||||
// they are different rows -- on fc_osm_skinny_34, partgroup is null on
|
||||
// 51,964 and '' on 111,696 -- so a null has to become IS NULL. It used
|
||||
// to fall through esc() to `= ''`, which selected the *other* blank:
|
||||
// the slice named one node and the operation wrote the other.
|
||||
if (val === null) {
|
||||
parts.push(`"${col}" IS NULL`);
|
||||
} else if (Array.isArray(val)) {
|
||||
const nulls = val.some(v => v === null);
|
||||
const escaped = val.filter(v => v !== null).map(v => esc(v));
|
||||
const inList = escaped.length ? `"${col}" IN ('${escaped.join("', '")}')` : null;
|
||||
// an IN list cannot carry a null -- `IN (NULL)` is never true -- so
|
||||
// the null arm is ORed on rather than folded in
|
||||
parts.push(
|
||||
nulls && inList ? `("${col}" IS NULL OR ${inList})`
|
||||
: nulls ? `"${col}" IS NULL`
|
||||
: inList ? inList
|
||||
: 'FALSE'
|
||||
);
|
||||
if (!allowed.has(col)) continue;
|
||||
if (Array.isArray(val)) {
|
||||
const escaped = val.map(v => esc(v));
|
||||
parts.push(`"${col}" IN ('${escaped.join("', '")}')`);
|
||||
} else {
|
||||
parts.push(`"${col}" = '${esc(val)}'`);
|
||||
}
|
||||
@ -632,135 +251,17 @@ function buildWhere(slice, dimCols, versionId) {
|
||||
return parts.length ? parts.join('\nAND ') : 'TRUE';
|
||||
}
|
||||
|
||||
// The pivot's own filter, carried alongside the slices so an operation writes
|
||||
// exactly the rows the ledger counted.
|
||||
//
|
||||
// A slice is {col: value} and can only ever mean equality, so a view filtered to
|
||||
// sseas_e <= 2027 could not be expressed as one. Refusing it was safe but
|
||||
// useless -- a bounded season is an ordinary way to scope a forecast -- so the
|
||||
// operators travel as [col, op, value] triples instead.
|
||||
//
|
||||
// Perspective's operator names, not SQL's, since that is where these come from.
|
||||
// Anything outside this list is refused rather than ignored: a scope silently
|
||||
// dropped is a write that is wider than the panel that authorised it.
|
||||
//
|
||||
// The substring operators are the ones a text dimension is actually filtered
|
||||
// with -- `partd contains 'Keysto'` picks a product family out of a description
|
||||
// nobody would enumerate by hand -- and they are **case-insensitive** in the
|
||||
// engine: `t_tscalar::contains` lowercases both sides before
|
||||
// `std::string::find` (scalar.cpp), as do `begins_with` and `ends_with`. So the
|
||||
// SQL is ILIKE, not LIKE. LIKE would be *narrower* than the grid that
|
||||
// authorised the write, which is the failure this whole mechanism exists to
|
||||
// prevent, and narrower is no safer than wider: it writes a set the ledger did
|
||||
// not count either way.
|
||||
//
|
||||
// Their term is matched literally there, so LIKE's own metacharacters are
|
||||
// escaped out of it by `likeTerm` -- a filter on `50%` means those three
|
||||
// characters, not "50 followed by anything".
|
||||
//
|
||||
// Not identical in one corner: the engine folds case byte by byte, so ASCII
|
||||
// only, where ILIKE follows the database collation. They differ only on
|
||||
// non-ASCII letters.
|
||||
//
|
||||
// `matches` / `not matches` are deliberately still refused. They are RE2 in the
|
||||
// engine and Postgres regexes are not RE2 -- `\b` is a word boundary in one and
|
||||
// a backspace in the other -- so a pattern valid in both could quietly match a
|
||||
// different set of rows, which is worse than answering 400.
|
||||
const likeTerm = (val, pre, post) =>
|
||||
`'${esc(pre + String(val).replace(/[\\%_]/g, m => `\\${m}`) + post)}'`;
|
||||
|
||||
const SCOPE_OPS = {
|
||||
'==': (c, v) => `${c} = ${v[0]}`,
|
||||
'!=': (c, v) => `${c} != ${v[0]}`,
|
||||
'>': (c, v) => `${c} > ${v[0]}`,
|
||||
'>=': (c, v) => `${c} >= ${v[0]}`,
|
||||
'<': (c, v) => `${c} < ${v[0]}`,
|
||||
'<=': (c, v) => `${c} <= ${v[0]}`,
|
||||
'in': (c, v) => `${c} IN (${v.join(', ')})`,
|
||||
'not in': (c, v) => `${c} NOT IN (${v.join(', ')})`,
|
||||
'is null': (c) => `${c} IS NULL`,
|
||||
'is not null': (c) => `${c} IS NOT NULL`,
|
||||
'contains': (c, v, raw) => `${c} ILIKE ${likeTerm(raw[0], '%', '%')}`,
|
||||
'not contains': (c, v, raw) => `${c} NOT ILIKE ${likeTerm(raw[0], '%', '%')}`,
|
||||
'begins with': (c, v, raw) => `${c} ILIKE ${likeTerm(raw[0], '', '%')}`,
|
||||
'not begins with': (c, v, raw) => `${c} NOT ILIKE ${likeTerm(raw[0], '', '%')}`,
|
||||
'ends with': (c, v, raw) => `${c} ILIKE ${likeTerm(raw[0], '%', '')}`,
|
||||
'not ends with': (c, v, raw) => `${c} NOT ILIKE ${likeTerm(raw[0], '%', '')}`,
|
||||
};
|
||||
|
||||
// the operators that need a term, so an empty one is refused rather than
|
||||
// becoming ILIKE '%%' -- which is every non-null row, i.e. no scope at all
|
||||
const SCOPE_OPS_NEED_TERM = new Set(Object.keys(SCOPE_OPS).filter(op => !op.startsWith('is ')));
|
||||
|
||||
function buildScopeClause(scope, dimCols, versionId) {
|
||||
if (!Array.isArray(scope) || scope.length === 0) return '';
|
||||
const allowed = new Set(dimCols);
|
||||
const parts = scope.map((entry) => {
|
||||
if (!Array.isArray(entry) || entry.length < 2) {
|
||||
const err = new Error(`Malformed scope entry ${JSON.stringify(entry)}`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
const [col, op, ...rest] = entry;
|
||||
const vals = (Array.isArray(rest[0]) ? rest[0] : rest).filter(v => v !== undefined);
|
||||
|
||||
if (COMPUTED_SLICE_COLS[col]) {
|
||||
if (op !== '==' && op !== 'in') {
|
||||
const err = new Error(`${col} can only be scoped with == or in, not ${op}`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
return computedSlicePredicate(col, vals, versionId);
|
||||
}
|
||||
if (!allowed.has(col)) {
|
||||
const err = new Error(`Column "${col}" is not available for filtering`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
const fn = SCOPE_OPS[op];
|
||||
if (!fn) {
|
||||
const err = new Error(`Unsupported filter operator "${op}" on ${col}`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
if (SCOPE_OPS_NEED_TERM.has(op) && (vals.length === 0 || vals[0] === '' || vals[0] === null)) {
|
||||
const err = new Error(`Filter "${col} ${op}" has no value`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
return fn(`"${col}"`, vals.map(v => `'${esc(String(v))}'`), vals);
|
||||
});
|
||||
return parts.join('\nAND ');
|
||||
}
|
||||
|
||||
// The territory predicate: what this account may see and change.
|
||||
//
|
||||
// Server-side by construction. Today's `scope` is sent by the browser, which is
|
||||
// right for a filter the user chose and would be fatal for a permission -- so
|
||||
// this one is built from the session and ANDed on last, where nothing in the
|
||||
// request can remove it.
|
||||
//
|
||||
// FALSE, not TRUE, for an account with no territory. The whole point is that a
|
||||
// missing grant means no rows: an empty list that fell through to TRUE would
|
||||
// hand the entire book to the first account somebody forgot to configure.
|
||||
function buildTerritoryClause(territory, territoryCol, alias = '') {
|
||||
if (!territory || territory.admin) return '';
|
||||
if (!territoryCol) return 'FALSE';
|
||||
const vals = (territory.values || []).filter(v => v != null && v !== '');
|
||||
if (vals.length === 0) return 'FALSE';
|
||||
const pfx = alias ? `${alias}.` : '';
|
||||
return `${pfx}"${territoryCol}" IN (${vals.map(v => `'${esc(String(v))}'`).join(', ')})`;
|
||||
}
|
||||
|
||||
// build a WHERE clause spanning several slices — an OR of AND-groups.
|
||||
// A union of slices cannot be flattened into one IN list per column: slices
|
||||
// {Region:East, State:NY} and {Region:West, State:CA} would become
|
||||
// Region IN (East,West) AND State IN (NY,CA), which also matches East/CA.
|
||||
function buildWhereAny(slices, dimCols, versionId) {
|
||||
const list = (slices || []).filter(Boolean);
|
||||
// An empty slice is a total cell, and the union with a total is the total.
|
||||
// Dropping it instead would narrow the write to the other cells. The caller
|
||||
// only lets one through under a scope, which is ANDed on afterwards.
|
||||
if (list.length === 0 || list.some(s => Object.keys(s).length === 0)) return 'TRUE';
|
||||
if (list.length === 1) return buildWhere(list[0], dimCols, versionId);
|
||||
function buildWhereAny(slices, dimCols) {
|
||||
const list = (slices || []).filter(s => s && Object.keys(s).length > 0);
|
||||
if (list.length === 0) return 'TRUE';
|
||||
if (list.length === 1) return buildWhere(list[0], dimCols);
|
||||
|
||||
const groups = list
|
||||
.map(s => buildWhere(s, dimCols, versionId))
|
||||
.map(s => buildWhere(s, dimCols))
|
||||
.filter(w => w !== 'TRUE');
|
||||
|
||||
// any slice that reduced to TRUE selects everything, so the union does too
|
||||
@ -788,21 +289,12 @@ function buildExcludeClause(excludeIters) {
|
||||
|
||||
// build the dimension columns portion of a SELECT for recode/clone
|
||||
// replaces named dimensions with literal values, passes others through unchanged
|
||||
// derivedExprs: cname -> a SQL expression to use when the caller has not set the
|
||||
// column outright. Clone passes the dim_period expressions through here, so a
|
||||
// cloned row's period dimensions come from the calendar against its shifted date
|
||||
// rather than from the row it was copied from.
|
||||
function buildSetClause(dimCols, setObj, opts = {}) {
|
||||
const { derivedExprs, alias } = opts;
|
||||
const pfx = alias ? `${alias}.` : '';
|
||||
function buildSetClause(dimCols, setObj) {
|
||||
return dimCols.map(col => {
|
||||
if (setObj && setObj[col] !== undefined) {
|
||||
return `'${esc(setObj[col])}' AS "${col}"`;
|
||||
}
|
||||
if (derivedExprs && derivedExprs[col]) {
|
||||
return `${derivedExprs[col]} AS "${col}"`;
|
||||
}
|
||||
return `${pfx}"${col}"`;
|
||||
return `"${col}"`;
|
||||
}).join(', ');
|
||||
}
|
||||
|
||||
@ -846,6 +338,4 @@ function esc(val) {
|
||||
return String(val).replace(/'/g, "''");
|
||||
}
|
||||
|
||||
module.exports = { generateSQL, grainOf, COMPUTED_SLICE_COLS, TABLE_SLICE_COLS, buildScopeClause, buildTerritoryClause,
|
||||
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR, LABEL_GROUP_COLS,
|
||||
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET, dateGroupsOf, dimPeriodMapOf, dimPeriodJoins, applyTokens, buildWhere, buildWhereAny, buildExcludeClause, buildExcludePredicate, buildSetClause, buildFilterClause, esc };
|
||||
module.exports = { generateSQL, applyTokens, buildWhere, buildWhereAny, buildExcludeClause, buildExcludePredicate, buildSetClause, buildFilterClause, esc };
|
||||
|
||||
38
lib/utils.js
38
lib/utils.js
@ -3,41 +3,7 @@ function fcTable(tname, versionId) {
|
||||
return `pf.fc_${tname}_${versionId}`;
|
||||
}
|
||||
|
||||
// The columns of a relation, from pg_catalog rather than information_schema.
|
||||
//
|
||||
// information_schema.columns omits materialized views -- they are not in the
|
||||
// SQL standard -- so a source built on one looked like it had no columns at
|
||||
// all: registering it seeded nothing, and creating a version failed with "No
|
||||
// usable columns in col_meta" while col_meta plainly had thirty-six.
|
||||
//
|
||||
// The shape matches what information_schema returned, so mapType and the
|
||||
// callers did not have to change. data_type is format_type with the modifier
|
||||
// stripped, which gives the same spelling information_schema uses ('character
|
||||
// varying', 'numeric'), and the numeric precision and scale are unpacked from
|
||||
// atttypmod the way information_schema does internally.
|
||||
//
|
||||
// Takes $1 = schema, $2 = relation name.
|
||||
const RELATION_COLUMNS_SQL = `
|
||||
SELECT a.attname AS column_name
|
||||
,regexp_replace(format_type(a.atttypid, a.atttypmod), '\\(.*\\)$', '') AS data_type
|
||||
,a.attnum AS ordinal_position
|
||||
,CASE WHEN a.attnotnull THEN 'NO' ELSE 'YES' END AS is_nullable
|
||||
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
|
||||
THEN ((a.atttypmod - 4) >> 16) & 65535 END AS numeric_precision
|
||||
,CASE WHEN a.atttypid = 'numeric'::regtype AND a.atttypmod > 4
|
||||
THEN (a.atttypmod - 4) & 65535 END AS numeric_scale
|
||||
FROM pg_attribute a
|
||||
JOIN pg_class c ON c.oid = a.attrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE TRUE
|
||||
AND n.nspname = $1
|
||||
AND c.relname = $2
|
||||
AND c.relkind IN ('r', 'v', 'm', 'f', 'p')
|
||||
AND a.attnum > 0
|
||||
AND NOT a.attisdropped
|
||||
`;
|
||||
|
||||
// map a data_type name to a clean postgres column type
|
||||
// map information_schema data_type to a clean postgres column type
|
||||
function mapType(dataType, numericPrecision, numericScale) {
|
||||
switch (dataType) {
|
||||
case 'character varying':
|
||||
@ -70,4 +36,4 @@ function mapType(dataType, numericPrecision, numericScale) {
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { fcTable, mapType, RELATION_COLUMNS_SQL };
|
||||
module.exports = { fcTable, mapType };
|
||||
|
||||
68
package-lock.json
generated
68
package-lock.json
generated
@ -7,14 +7,11 @@
|
||||
"": {
|
||||
"name": "pf_app",
|
||||
"version": "1.0.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"apache-arrow": "^21.1.0",
|
||||
"connect-pg-simple": "^10.0.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.0.0",
|
||||
"express": "^4.18.2",
|
||||
"express-session": "^1.19.0",
|
||||
"pg": "^8.11.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
@ -372,18 +369,6 @@
|
||||
"node": ">=12.20.0"
|
||||
}
|
||||
},
|
||||
"node_modules/connect-pg-simple": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/connect-pg-simple/-/connect-pg-simple-10.0.0.tgz",
|
||||
"integrity": "sha512-pBGVazlqiMrackzCr0eKhn4LO5trJXsOX0nQoey9wCOayh80MYtThCbq8eoLsjpiWgiok/h+1/uti9/2/Una8A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"pg": "^8.12.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^18.18.0 || ^20.9.0 || >=22.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/content-disposition": {
|
||||
"version": "0.5.4",
|
||||
"resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
|
||||
@ -597,29 +582,6 @@
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/express-session": {
|
||||
"version": "1.19.0",
|
||||
"resolved": "https://registry.npmjs.org/express-session/-/express-session-1.19.0.tgz",
|
||||
"integrity": "sha512-0csaMkGq+vaiZTmSMMGkfdCOabYv192VbytFypcvI0MANrp+4i/7yEkJ0sbAEhycQjntaKGzYfjfXQyVb7BHMA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"cookie": "~0.7.2",
|
||||
"cookie-signature": "~1.0.7",
|
||||
"debug": "~2.6.9",
|
||||
"depd": "~2.0.0",
|
||||
"on-headers": "~1.1.0",
|
||||
"parseurl": "~1.3.3",
|
||||
"safe-buffer": "~5.2.1",
|
||||
"uid-safe": "~2.1.5"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.8.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/fill-range": {
|
||||
"version": "7.1.1",
|
||||
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz",
|
||||
@ -1123,15 +1085,6 @@
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/on-headers": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/on-headers/-/on-headers-1.1.0.tgz",
|
||||
"integrity": "sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/parseurl": {
|
||||
"version": "1.3.3",
|
||||
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
|
||||
@ -1323,15 +1276,6 @@
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/random-bytes": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/random-bytes/-/random-bytes-1.0.0.tgz",
|
||||
"integrity": "sha512-iv7LhNVO047HzYR3InF6pUcUsPQiHTM1Qal51DcGSuZFBil1aBBWG5eHPNek7bvILMaYJ/8RU1e8w1AMdHmLQQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/range-parser": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
|
||||
@ -1648,18 +1592,6 @@
|
||||
"node": ">=12.17"
|
||||
}
|
||||
},
|
||||
"node_modules/uid-safe": {
|
||||
"version": "2.1.5",
|
||||
"resolved": "https://registry.npmjs.org/uid-safe/-/uid-safe-2.1.5.tgz",
|
||||
"integrity": "sha512-KPHm4VL5dDXKz01UuEd88Df+KzynaohSL9fBh096KWAxSKZQDI2uBrVqtvRM4rwrIrRRKsdLNML/lnaaVSRioA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"random-bytes": "~1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/undefsafe": {
|
||||
"version": "2.0.5",
|
||||
"resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz",
|
||||
|
||||
@ -11,11 +11,9 @@
|
||||
},
|
||||
"dependencies": {
|
||||
"apache-arrow": "^21.1.0",
|
||||
"connect-pg-simple": "^10.0.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.0.0",
|
||||
"express": "^4.18.2",
|
||||
"express-session": "^1.19.0",
|
||||
"pg": "^8.11.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
366
pf.sh
366
pf.sh
@ -4,8 +4,6 @@ set -euo pipefail
|
||||
# ---------------------------------------------------------------------------
|
||||
# pf.sh — Pivot Forecast management script
|
||||
# Usage: ./pf.sh [deploy|start|stop|restart|status|logs|db-setup|config]
|
||||
# ./pf.sh [add-user|passwd|list-users|disable-user|enable-user]
|
||||
# ./pf.sh [set-territory|set-admin|orphan-territory]
|
||||
# ./pf.sh (interactive menu)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@ -69,33 +67,13 @@ require_service() {
|
||||
service_installed || die "systemd service not installed. Run: ./pf.sh install-service"
|
||||
}
|
||||
|
||||
# The keys cmd_config manages; anything else in .env is left alone.
|
||||
ENV_KEYS=(DB_HOST DB_PORT DB_NAME DB_USER DB_PASSWORD PORT SESSION_SECRET COOKIE_SECURE)
|
||||
|
||||
env_get() {
|
||||
[[ -f "$ENV_FILE" ]] || return 0
|
||||
grep -E "^$1=" "$ENV_FILE" | tail -1 | cut -d= -f2- | tr -d '"' || true
|
||||
}
|
||||
|
||||
# psql against the DB_* connection in .env; extra args are passed through.
|
||||
run_psql() {
|
||||
PGPASSWORD="${DB_PASSWORD:-}" psql \
|
||||
-h "${DB_HOST:-localhost}" \
|
||||
-p "${DB_PORT:-5432}" \
|
||||
-U "${DB_USER}" \
|
||||
-d "${DB_NAME}" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
db_ping() {
|
||||
load_env
|
||||
if [[ -z "${DB_NAME:-}" || -z "${DB_USER:-}" ]]; then
|
||||
warn "DB_NAME / DB_USER not set in .env"
|
||||
return 1
|
||||
fi
|
||||
local url="${DATABASE_URL:-}"
|
||||
[[ -z "$url" ]] && { warn "DATABASE_URL not set in .env"; return 1; }
|
||||
# Use psql if available for a real connectivity check
|
||||
if command -v psql &>/dev/null; then
|
||||
run_psql -tAc "SELECT 1" &>/dev/null && return 0 || return 1
|
||||
psql "$url" -c "SELECT 1" &>/dev/null && return 0 || return 1
|
||||
else
|
||||
warn "psql not in PATH — skipping live DB check"
|
||||
return 0
|
||||
@ -190,27 +168,18 @@ cmd_logs() {
|
||||
|
||||
cmd_db_setup() {
|
||||
require_env; load_env
|
||||
[[ -n "${DB_NAME:-}" && -n "${DB_USER:-}" ]] || die "DB_NAME / DB_USER not set in .env — run: ./pf.sh config"
|
||||
local url="${DATABASE_URL:-}"
|
||||
[[ -z "$url" ]] && die "DATABASE_URL not set in .env"
|
||||
command -v psql &>/dev/null || die "psql not found — install postgresql-client"
|
||||
|
||||
echo
|
||||
bold "DB Setup — will run: setup_sql/01_schema.sql, setup_sql/gen_dim_period.sql, setup_sql/02_auth.sql"
|
||||
warn "This creates the pf schema and tables, the pf.dim_period calendar, and the account/session tables."
|
||||
warn "The calendar's fiscal year starts in the month set in gen_dim_period.sql (fiscal_start_month) — check it before a first install."
|
||||
warn "Safe to re-run (CREATE IF NOT EXISTS; existing calendar rows are never overwritten)."
|
||||
bold "DB Setup — will run: setup_sql/01_schema.sql"
|
||||
warn "This creates the pf schema and tables. Safe to re-run (CREATE IF NOT EXISTS)."
|
||||
read -rp " Continue? [y/N] " confirm
|
||||
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; return; }
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/01_schema.sql"
|
||||
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/gen_dim_period.sql"
|
||||
run_psql -v ON_ERROR_STOP=1 -f "${APP_DIR}/setup_sql/02_auth.sql"
|
||||
psql "$url" -f "${APP_DIR}/setup_sql/01_schema.sql"
|
||||
success "Schema applied."
|
||||
|
||||
local n
|
||||
n=$(run_psql -tAc "SELECT count(*) FROM pf.app_user WHERE is_active" 2>/dev/null || echo 0)
|
||||
if [[ "${n:-0}" == "0" ]]; then
|
||||
warn "No active accounts yet — create one with: ./pf.sh add-user"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_config() {
|
||||
@ -219,284 +188,41 @@ cmd_config() {
|
||||
echo " File: $ENV_FILE"
|
||||
echo
|
||||
|
||||
local cur_host cur_port cur_name cur_user cur_pass cur_app_port
|
||||
cur_host=$(env_get DB_HOST)
|
||||
cur_port=$(env_get DB_PORT)
|
||||
cur_name=$(env_get DB_NAME)
|
||||
cur_user=$(env_get DB_USER)
|
||||
cur_pass=$(env_get DB_PASSWORD)
|
||||
cur_app_port=$(env_get PORT)
|
||||
local current_url=""
|
||||
local current_port=""
|
||||
local current_user=""
|
||||
|
||||
local input
|
||||
read -rp " DB_HOST [${cur_host:-localhost}]: " input
|
||||
local host="${input:-${cur_host:-localhost}}"
|
||||
|
||||
read -rp " DB_PORT [${cur_port:-5432}]: " input
|
||||
local port="${input:-${cur_port:-5432}}"
|
||||
|
||||
read -rp " DB_NAME [${cur_name:-not set}]: " input
|
||||
local name="${input:-$cur_name}"
|
||||
[[ -z "$name" ]] && die "DB_NAME is required."
|
||||
|
||||
read -rp " DB_USER [${cur_user:-$USER}]: " input
|
||||
local user="${input:-${cur_user:-$USER}}"
|
||||
|
||||
if [[ -n "$cur_pass" ]]; then
|
||||
read -rsp " DB_PASSWORD [keep existing]: " input; echo
|
||||
else
|
||||
read -rsp " DB_PASSWORD: " input; echo
|
||||
fi
|
||||
local pass="${input:-$cur_pass}"
|
||||
|
||||
read -rp " PORT (app) [${cur_app_port:-3010}]: " input
|
||||
local app_port="${input:-${cur_app_port:-3010}}"
|
||||
|
||||
# Session cookies are signed with this; regenerating it signs everyone out,
|
||||
# so an existing secret is kept rather than re-rolled on every config run.
|
||||
local secret
|
||||
secret=$(env_get SESSION_SECRET)
|
||||
if [[ -z "$secret" ]]; then
|
||||
secret=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')
|
||||
success "SESSION_SECRET generated."
|
||||
else
|
||||
success "SESSION_SECRET kept (delete the line in .env to re-roll)."
|
||||
fi
|
||||
|
||||
local cur_secure
|
||||
cur_secure=$(env_get COOKIE_SECURE)
|
||||
read -rp " COOKIE_SECURE — HTTPS-only cookie [${cur_secure:-true}]: " input
|
||||
local cookie_secure="${input:-${cur_secure:-true}}"
|
||||
|
||||
# Rewrite the managed keys, carrying over any other lines already in .env.
|
||||
local tmp
|
||||
tmp=$(mktemp)
|
||||
cat > "$tmp" <<EOF
|
||||
DB_HOST=${host}
|
||||
DB_PORT=${port}
|
||||
DB_NAME=${name}
|
||||
DB_USER=${user}
|
||||
DB_PASSWORD=${pass}
|
||||
PORT=${app_port}
|
||||
SESSION_SECRET=${secret}
|
||||
COOKIE_SECURE=${cookie_secure}
|
||||
EOF
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
local managed
|
||||
managed=$(IFS='|'; echo "${ENV_KEYS[*]}")
|
||||
grep -vE "^(${managed})=" "$ENV_FILE" | grep -vE '^[[:space:]]*$' >> "$tmp" || true
|
||||
current_url=$(grep -E '^DATABASE_URL=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
|
||||
current_port=$(grep -E '^PORT=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
|
||||
current_user=$(grep -E '^PF_USER=' "$ENV_FILE" | cut -d= -f2- | tr -d '"' || true)
|
||||
fi
|
||||
mv "$tmp" "$ENV_FILE"
|
||||
|
||||
read -rp " DATABASE_URL [${current_url:-not set}]: " input_url
|
||||
local url="${input_url:-$current_url}"
|
||||
[[ -z "$url" ]] && die "DATABASE_URL is required."
|
||||
|
||||
read -rp " PORT [${current_port:-3010}]: " input_port
|
||||
local port="${input_port:-${current_port:-3010}}"
|
||||
|
||||
read -rp " PF_USER [${current_user:-$USER}]: " input_user
|
||||
local pf_user="${input_user:-${current_user:-$USER}}"
|
||||
|
||||
cat > "$ENV_FILE" <<EOF
|
||||
DATABASE_URL=${url}
|
||||
PORT=${port}
|
||||
PF_USER=${pf_user}
|
||||
EOF
|
||||
chmod 600 "$ENV_FILE"
|
||||
success ".env written."
|
||||
|
||||
if db_ping; then
|
||||
success "Database connection verified."
|
||||
else
|
||||
warn "Could not reach the database — double-check the DB_* settings."
|
||||
warn "Could not reach the database — double-check DATABASE_URL."
|
||||
fi
|
||||
}
|
||||
|
||||
# -- Accounts ----------------------------------------------------------------
|
||||
|
||||
# Reads a password twice without echo and hashes it with lib/auth.js, so the
|
||||
# plaintext never reaches argv, psql, or the shell history.
|
||||
read_new_password() {
|
||||
local p1 p2
|
||||
# Prompts and their newlines go to stderr: stdout is the hash, and a stray
|
||||
# newline there ends up prefixed to it by the caller's $( ).
|
||||
read -rsp " Password: " p1; echo >&2
|
||||
[[ -z "$p1" ]] && { error "Password cannot be empty."; return 1; }
|
||||
read -rsp " Confirm : " p2; echo >&2
|
||||
[[ "$p1" != "$p2" ]] && { error "Passwords do not match."; return 1; }
|
||||
printf '%s' "$p1" | node "${APP_DIR}/lib/auth.js" hash
|
||||
}
|
||||
|
||||
# psql single-quoted literal: double any embedded quote.
|
||||
sql_lit() { printf "%s" "${1//\'/\'\'}"; }
|
||||
|
||||
cmd_add_user() {
|
||||
require_env; load_env
|
||||
check_node >/dev/null
|
||||
|
||||
echo; bold "Add account"
|
||||
local username display hash
|
||||
read -rp " Username: " username
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
read -rp " Display name [${username}]: " display
|
||||
display="${display:-$username}"
|
||||
|
||||
hash=$(read_new_password) || return 1
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH ins AS (
|
||||
INSERT INTO pf.app_user (username, pass_hash, display_name)
|
||||
VALUES ('$(sql_lit "$username")', '$(sql_lit "$hash")', '$(sql_lit "$display")')
|
||||
ON CONFLICT (username) DO NOTHING
|
||||
RETURNING id
|
||||
) SELECT id FROM ins" | grep -q . \
|
||||
&& success "Account '${username}' created." \
|
||||
|| die "Account '${username}' already exists — change its password with: ./pf.sh passwd"
|
||||
}
|
||||
|
||||
cmd_passwd() {
|
||||
require_env; load_env
|
||||
check_node >/dev/null
|
||||
|
||||
echo; bold "Change password"
|
||||
local username hash
|
||||
read -rp " Username: " username
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
|
||||
hash=$(read_new_password) || return 1
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH upd AS (
|
||||
UPDATE pf.app_user SET pass_hash = '$(sql_lit "$hash")'
|
||||
WHERE lower(username) = lower('$(sql_lit "$username")')
|
||||
RETURNING id
|
||||
) SELECT id FROM upd" | grep -q . \
|
||||
&& success "Password updated for '${username}'." \
|
||||
|| die "No such account: ${username}"
|
||||
}
|
||||
|
||||
cmd_list_users() {
|
||||
require_env; load_env
|
||||
echo; bold "Accounts"
|
||||
run_psql -c "
|
||||
SELECT username, display_name, is_active, is_admin,
|
||||
coalesce(jsonb_array_length(territory), 0) AS territory_values,
|
||||
to_char(last_login_at, 'YYYY-MM-DD HH24:MI') AS last_login
|
||||
FROM pf.app_user ORDER BY username"
|
||||
}
|
||||
|
||||
# Territory is what an account may see and change, as a list of values in the
|
||||
# source's is_territory column. No territory and not an admin means no rows --
|
||||
# so a new account is blind until this is run, which is the intended direction
|
||||
# to fail in.
|
||||
#
|
||||
# Values are given comma-separated and have to match the column exactly, since
|
||||
# that is what the SQL compares. set-territory with no values clears it.
|
||||
cmd_set_territory() {
|
||||
require_env; load_env
|
||||
local username="${1:-}"; shift || true
|
||||
[[ -z "$username" ]] && { read -rp " Username: " username; }
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
local values="${*:-}"
|
||||
[[ -z "$values" ]] && { read -rp " Territory values (comma separated, blank to clear): " values; }
|
||||
|
||||
local json="null"
|
||||
if [[ -n "$values" ]]; then
|
||||
json=$(python3 - "$values" <<'PYEOF'
|
||||
import json, sys
|
||||
vals = [v.strip() for v in sys.argv[1].split(',') if v.strip()]
|
||||
print(json.dumps(vals))
|
||||
PYEOF
|
||||
)
|
||||
fi
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH upd AS (
|
||||
UPDATE pf.app_user SET territory = $(if [[ "$json" == "null" ]]; then echo NULL; else echo "'$(sql_lit "$json")'::jsonb"; fi)
|
||||
WHERE lower(username) = lower('$(sql_lit "$username")')
|
||||
RETURNING username
|
||||
)
|
||||
SELECT count(*) FROM upd" | grep -q '^1$' \
|
||||
|| die "No such account: $username"
|
||||
success "Territory updated for $username"
|
||||
run_psql -c "SELECT username, is_admin, territory FROM pf.app_user WHERE lower(username) = lower('$(sql_lit "$username")')"
|
||||
}
|
||||
|
||||
# An admin sees and changes everything, and is the only account that can recode
|
||||
# the territory column or undo someone else's entry.
|
||||
cmd_set_admin() {
|
||||
require_env; load_env
|
||||
local username="${1:-}" flag="${2:-true}"
|
||||
[[ -z "$username" ]] && { read -rp " Username: " username; }
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
[[ "$flag" != "true" && "$flag" != "false" ]] && die "Second argument must be true or false."
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH upd AS (
|
||||
UPDATE pf.app_user SET is_admin = $flag
|
||||
WHERE lower(username) = lower('$(sql_lit "$username")')
|
||||
RETURNING username
|
||||
)
|
||||
SELECT count(*) FROM upd" | grep -q '^1$' \
|
||||
|| die "No such account: $username"
|
||||
success "$username is_admin = $flag"
|
||||
}
|
||||
|
||||
# Territory values present in the data that belong to no account. Work under one
|
||||
# is invisible to everybody but an admin, which is easy to cause by a typo and
|
||||
# impossible to notice from inside the app.
|
||||
cmd_orphan_territory() {
|
||||
require_env; load_env
|
||||
local source_id="${1:-}"
|
||||
[[ -z "$source_id" ]] && { read -rp " Source id: " source_id; }
|
||||
[[ -z "$source_id" ]] && die "Source id is required."
|
||||
|
||||
# The column and table are data, so the query is built in two steps rather
|
||||
# than one clever one: read the names, then run the listing.
|
||||
local meta col schema tname
|
||||
meta=$(run_psql -tAF'|' -c "
|
||||
SELECT m.cname, x.schema, x.tname
|
||||
FROM pf.col_meta m JOIN pf.source x ON x.id = m.source_id
|
||||
WHERE m.source_id = $source_id AND m.is_territory")
|
||||
[[ -z "$meta" ]] && die "Source $source_id has no column marked is_territory."
|
||||
IFS='|' read -r col schema tname <<< "$meta"
|
||||
|
||||
echo; bold "Territory values in $schema.$tname with no account"
|
||||
run_psql -c "
|
||||
SELECT DISTINCT s.\"$col\" AS unassigned
|
||||
FROM \"$schema\".\"$tname\" s
|
||||
WHERE TRUE
|
||||
AND s.\"$col\" IS NOT NULL
|
||||
AND s.\"$col\"::text NOT IN (
|
||||
SELECT jsonb_array_elements_text(territory)
|
||||
FROM pf.app_user
|
||||
WHERE territory IS NOT NULL
|
||||
)
|
||||
ORDER BY 1"
|
||||
}
|
||||
|
||||
# Deactivating leaves the row (and its history) in place, and drops any live
|
||||
# session so the account loses access immediately rather than at cookie expiry.
|
||||
cmd_disable_user() {
|
||||
require_env; load_env
|
||||
local username="${1:-}"
|
||||
[[ -z "$username" ]] && { read -rp " Username to disable: " username; }
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH upd AS (
|
||||
UPDATE pf.app_user SET is_active = false
|
||||
WHERE lower(username) = lower('$(sql_lit "$username")')
|
||||
RETURNING id
|
||||
) SELECT id FROM upd" | grep -q . \
|
||||
|| die "No such account: ${username}"
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -c "
|
||||
DELETE FROM pf.session
|
||||
WHERE sess::jsonb -> 'user' ->> 'username' ILIKE '$(sql_lit "$username")'" >/dev/null
|
||||
success "Account '${username}' disabled and signed out."
|
||||
}
|
||||
|
||||
cmd_enable_user() {
|
||||
require_env; load_env
|
||||
local username="${1:-}"
|
||||
[[ -z "$username" ]] && { read -rp " Username to enable: " username; }
|
||||
[[ -z "$username" ]] && die "Username is required."
|
||||
|
||||
run_psql -v ON_ERROR_STOP=1 -tAc "
|
||||
WITH upd AS (
|
||||
UPDATE pf.app_user SET is_active = true
|
||||
WHERE lower(username) = lower('$(sql_lit "$username")')
|
||||
RETURNING id
|
||||
) SELECT id FROM upd" | grep -q . \
|
||||
&& success "Account '${username}' enabled." \
|
||||
|| die "No such account: ${username}"
|
||||
}
|
||||
|
||||
cmd_install_service() {
|
||||
require_systemd
|
||||
require_env
|
||||
@ -576,18 +302,10 @@ interactive_menu() {
|
||||
echo " 4) restart restart server"
|
||||
echo " 5) status service + DB + git info"
|
||||
echo " 6) logs tail journald logs"
|
||||
echo " 7) db-setup apply the pf schema, calendar and auth tables"
|
||||
echo " 8) config set DB connection + app PORT"
|
||||
echo " 7) db-setup apply setup_sql/01_schema.sql"
|
||||
echo " 8) config set DATABASE_URL / PORT / PF_USER"
|
||||
echo " 9) install-service create systemd unit file"
|
||||
echo " 10) uninstall-service remove systemd unit file"
|
||||
echo " 11) add-user create a login account"
|
||||
echo " 12) passwd change an account password"
|
||||
echo " 13) list-users show accounts"
|
||||
echo " 14) disable-user deactivate an account and sign it out"
|
||||
echo " 15) enable-user reactivate an account"
|
||||
echo " 16) set-territory grant an account the territory values it may see"
|
||||
echo " 17) set-admin make an account an administrator"
|
||||
echo " 18) orphan-territory territory values no account owns"
|
||||
echo " q) quit"
|
||||
echo
|
||||
read -rp " Choice: " choice
|
||||
@ -602,14 +320,6 @@ interactive_menu() {
|
||||
8|config) cmd_config ;;
|
||||
9|install-service) cmd_install_service ;;
|
||||
10|uninstall-service) cmd_uninstall_service ;;
|
||||
11|add-user) cmd_add_user ;;
|
||||
12|passwd) cmd_passwd ;;
|
||||
13|list-users) cmd_list_users ;;
|
||||
14|disable-user) cmd_disable_user ;;
|
||||
15|enable-user) cmd_enable_user ;;
|
||||
16|set-territory) cmd_set_territory ;;
|
||||
17|set-admin) cmd_set_admin ;;
|
||||
18|orphan-territory) cmd_orphan_territory ;;
|
||||
q|Q|quit|exit) echo "Bye."; exit 0 ;;
|
||||
*) warn "Unknown option: $choice" ;;
|
||||
esac
|
||||
@ -629,14 +339,6 @@ case "${1:-}" in
|
||||
config) cmd_config ;;
|
||||
install-service) cmd_install_service ;;
|
||||
uninstall-service) cmd_uninstall_service ;;
|
||||
add-user) cmd_add_user ;;
|
||||
passwd) cmd_passwd ;;
|
||||
list-users) cmd_list_users ;;
|
||||
set-territory) shift; cmd_set_territory "$@" ;;
|
||||
set-admin) shift; cmd_set_admin "$@" ;;
|
||||
orphan-territory) shift; cmd_orphan_territory "$@" ;;
|
||||
disable-user) cmd_disable_user "${2:-}" ;;
|
||||
enable-user) cmd_enable_user "${2:-}" ;;
|
||||
"") interactive_menu ;;
|
||||
*) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service add-user passwd list-users disable-user enable-user set-territory set-admin orphan-territory" ;;
|
||||
*) die "Unknown command: $1. Valid: deploy start stop restart status logs db-setup config install-service uninstall-service" ;;
|
||||
esac
|
||||
|
||||
@ -1,17 +1,5 @@
|
||||
# Perspective Architecture Options
|
||||
|
||||
> **Decision record, written 2026-08.** The outcome was the display-grain
|
||||
> pre-aggregation (§Spike findings onward), and it shipped. **"Current
|
||||
> architecture" below describes the app as it was then**, not now: it has
|
||||
> since moved to a vendored Perspective 5.4.0 bundled inline, `tableFromArrays`
|
||||
> on the server, `table.remove()` on undo, and `/data` / `/agg` emitting
|
||||
> `pf_segment`, `pf_bucket`, `pf_note` and `pf_op`. For today's data flow see
|
||||
> `CLAUDE.md` → §Core data flow and `pf_spec.md` → §Display-grain
|
||||
> pre-aggregation. What remains current here is the spike findings, the
|
||||
> display-grain reasoning, and the Path A / Path B analysis behind the
|
||||
> "dynamic grain" next step. The DuckDB virtual-server spike is parked on
|
||||
> branch `spike/duckdb-virtual-server`.
|
||||
|
||||
This document weighs how the Forecast view should source data for the Perspective
|
||||
pivot. The current implementation hits practical limits on initial load
|
||||
(~30s for 350k rows × ~55 cols), and growth is expected. Choosing an architecture
|
||||
1042
pf_spec.md
Normal file
1042
pf_spec.md
Normal file
File diff suppressed because it is too large
Load Diff
110
routes/auth.js
110
routes/auth.js
@ -1,110 +0,0 @@
|
||||
const express = require('express');
|
||||
const { verifyPassword } = require('../lib/auth');
|
||||
|
||||
// Per-IP login throttle. In-memory on purpose: it only has to blunt online
|
||||
// guessing, and a counter that resets on restart is the acceptable cost of not
|
||||
// writing a failed-attempt row for every knock on an internet-facing port.
|
||||
const WINDOW_MS = 15 * 60 * 1000;
|
||||
const MAX_ATTEMPTS = 10;
|
||||
const attempts = new Map(); // ip -> { count, resetAt }
|
||||
|
||||
function tooManyAttempts(ip) {
|
||||
const rec = attempts.get(ip);
|
||||
if (!rec || Date.now() > rec.resetAt) return false;
|
||||
return rec.count >= MAX_ATTEMPTS;
|
||||
}
|
||||
|
||||
function recordFailure(ip) {
|
||||
const rec = attempts.get(ip);
|
||||
if (!rec || Date.now() > rec.resetAt) {
|
||||
attempts.set(ip, { count: 1, resetAt: Date.now() + WINDOW_MS });
|
||||
} else {
|
||||
rec.count += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Keep the map from growing without bound on a long-lived process.
|
||||
setInterval(() => {
|
||||
const now = Date.now();
|
||||
for (const [ip, rec] of attempts) if (now > rec.resetAt) attempts.delete(ip);
|
||||
}, WINDOW_MS).unref();
|
||||
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
router.post('/login', async (req, res) => {
|
||||
const ip = req.ip;
|
||||
if (tooManyAttempts(ip)) {
|
||||
return res.status(429).json({ error: 'Too many failed attempts. Try again later.' });
|
||||
}
|
||||
|
||||
const username = String(req.body?.username || '').trim();
|
||||
const password = String(req.body?.password || '');
|
||||
if (!username || !password) {
|
||||
return res.status(400).json({ error: 'Username and password are required' });
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await pool.query(
|
||||
`SELECT id, username, display_name, pass_hash, is_active,
|
||||
is_admin, territory
|
||||
FROM pf.app_user WHERE lower(username) = lower($1)`,
|
||||
[username]
|
||||
);
|
||||
const user = result.rows[0];
|
||||
|
||||
// Same message and roughly the same work either way: no unknown
|
||||
// user / wrong password / disabled distinction to enumerate.
|
||||
const ok = user && user.is_active && verifyPassword(password, user.pass_hash);
|
||||
if (!ok) {
|
||||
recordFailure(ip);
|
||||
return res.status(401).json({ error: 'Invalid username or password' });
|
||||
}
|
||||
|
||||
// New session id on login — an existing cookie can't be fixated.
|
||||
req.session.regenerate(err => {
|
||||
if (err) {
|
||||
console.error(err);
|
||||
return res.status(500).json({ error: 'Could not start session' });
|
||||
}
|
||||
req.session.user = {
|
||||
id: user.id,
|
||||
username: user.username,
|
||||
display_name: user.display_name,
|
||||
is_admin: !!user.is_admin,
|
||||
territory: Array.isArray(user.territory) ? user.territory : [],
|
||||
};
|
||||
pool.query(`UPDATE pf.app_user SET last_login_at = now() WHERE id = $1`, [user.id])
|
||||
.catch(e => console.error('last_login_at update failed', e));
|
||||
req.session.save(err2 => {
|
||||
if (err2) {
|
||||
console.error(err2);
|
||||
return res.status(500).json({ error: 'Could not start session' });
|
||||
}
|
||||
attempts.delete(ip);
|
||||
res.json({ user: req.session.user });
|
||||
});
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
router.post('/logout', (req, res) => {
|
||||
const name = req.session?.cookie && req.app.get('session cookie name');
|
||||
req.session.destroy(err => {
|
||||
if (err) console.error(err);
|
||||
res.clearCookie(name || 'pf.sid');
|
||||
res.json({ ok: true });
|
||||
});
|
||||
});
|
||||
|
||||
// The UI calls this on load to decide between the login screen and the app.
|
||||
router.get('/me', (req, res) => {
|
||||
if (!req.session?.user) return res.status(401).json({ error: 'Not authenticated' });
|
||||
res.json({ user: req.session.user });
|
||||
});
|
||||
|
||||
return router;
|
||||
};
|
||||
@ -1,193 +0,0 @@
|
||||
const express = require('express');
|
||||
const { sessionUser } = require('../lib/auth');
|
||||
|
||||
// Named Perspective view configs. Two kinds, one table:
|
||||
//
|
||||
// private — yours, nobody else lists it
|
||||
// published — everyone on the version sees it; only the owner or an admin
|
||||
// may change it, the same rule pf.log uses for its entries
|
||||
//
|
||||
// Scope is the version by default, because that is where people enter the app.
|
||||
// A layout with version_id NULL applies to every version of the source, which
|
||||
// is where the old pf.source.default_layout went and what a brand-new version
|
||||
// picks up before anyone has published anything for it.
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
const SELECT_COLS = `id, source_id, version_id, name, config, owner,
|
||||
visibility, is_default, created_at, updated_at`;
|
||||
|
||||
// What the client needs to know about a row it cannot write, so the UI can
|
||||
// grey the control out rather than offer a click that answers 403.
|
||||
const decorate = (row, req) => ({
|
||||
...row,
|
||||
scope: row.version_id == null ? 'source' : 'version',
|
||||
can_edit: !!req.session?.user?.is_admin || row.owner === sessionUser(req)
|
||||
});
|
||||
|
||||
// The owner/admin gate, shared by every write. Returns the row, or sends the
|
||||
// response itself and returns null.
|
||||
async function ownedLayout(req, res) {
|
||||
const id = parseInt(req.params.lid);
|
||||
const { rows } = await pool.query(
|
||||
`SELECT ${SELECT_COLS} FROM pf.layout WHERE id = $1`, [id]
|
||||
);
|
||||
if (!rows.length) { res.status(404).json({ error: 'Layout not found' }); return null; }
|
||||
const row = rows[0];
|
||||
if (!req.session?.user?.is_admin && row.owner !== sessionUser(req)) {
|
||||
res.status(403).json({
|
||||
error: `“${row.name}” belongs to ${row.owner} — only they or an administrator can change it`
|
||||
});
|
||||
return null;
|
||||
}
|
||||
return row;
|
||||
}
|
||||
|
||||
// Everything applicable to a version: its own published layouts, the
|
||||
// source-wide published ones, and the caller's own private layouts at either
|
||||
// scope. Deliberately not territory-filtered — a layout is display config,
|
||||
// and territory restricts rows, not columns.
|
||||
router.get('/versions/:id/layouts', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
try {
|
||||
const ver = await pool.query(
|
||||
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
|
||||
);
|
||||
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
|
||||
const sourceId = ver.rows[0].source_id;
|
||||
|
||||
const { rows } = await pool.query(
|
||||
`SELECT ${SELECT_COLS}
|
||||
FROM pf.layout
|
||||
WHERE TRUE
|
||||
AND source_id = $1
|
||||
AND (version_id = $2 OR version_id IS NULL)
|
||||
AND (visibility = 'published' OR owner = $3)
|
||||
ORDER BY visibility DESC, is_default DESC, lower(name)`,
|
||||
[sourceId, versionId, sessionUser(req)]
|
||||
);
|
||||
res.json(rows.map(r => decorate(r, req)));
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Create. scope 'source' saves it against every version of the source;
|
||||
// anything else is this version.
|
||||
router.post('/versions/:id/layouts', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const { name, config, visibility = 'private', scope = 'version' } = req.body || {};
|
||||
if (!name || !String(name).trim()) return res.status(400).json({ error: 'Name is required' });
|
||||
if (!config) return res.status(400).json({ error: 'Config is required' });
|
||||
if (!['private', 'published'].includes(visibility)) {
|
||||
return res.status(400).json({ error: 'visibility must be private or published' });
|
||||
}
|
||||
try {
|
||||
const ver = await pool.query(
|
||||
`SELECT source_id FROM pf.version WHERE id = $1`, [versionId]
|
||||
);
|
||||
if (!ver.rows.length) return res.status(404).json({ error: 'Version not found' });
|
||||
|
||||
const { rows } = await pool.query(
|
||||
`INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
RETURNING ${SELECT_COLS}`,
|
||||
[ver.rows[0].source_id, scope === 'source' ? null : versionId,
|
||||
String(name).trim(), config, sessionUser(req), visibility]
|
||||
);
|
||||
res.json(decorate(rows[0], req));
|
||||
} catch (err) {
|
||||
// the partial unique indexes, reported in the terms the user typed
|
||||
if (err.code === '23505') {
|
||||
return res.status(409).json({ error: `A layout named “${name}” already exists here` });
|
||||
}
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Update any of name / config / visibility / is_default. Each is optional;
|
||||
// a missing key leaves the column alone, which is why the flag-and-value
|
||||
// pair is used rather than COALESCE on the value.
|
||||
router.patch('/layouts/:lid', async (req, res) => {
|
||||
const { name, config, visibility, is_default, scope } = req.body || {};
|
||||
if (name === undefined && config === undefined && visibility === undefined
|
||||
&& is_default === undefined && scope === undefined) {
|
||||
return res.status(400).json({ error: 'Nothing to update' });
|
||||
}
|
||||
if (visibility !== undefined && !['private', 'published'].includes(visibility)) {
|
||||
return res.status(400).json({ error: 'visibility must be private or published' });
|
||||
}
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
const row = await ownedLayout(req, res);
|
||||
if (!row) return;
|
||||
|
||||
// Unpublishing a default would leave a default nobody can see, which
|
||||
// the table's CHECK refuses; clear the flag with it rather than
|
||||
// failing on a constraint the user never mentioned.
|
||||
const clearsDefault = visibility === 'private';
|
||||
|
||||
// One default per scope, enforced by a partial unique index. Stand
|
||||
// the others down first rather than letting the update collide --
|
||||
// and in one transaction, or a name clash on the second statement
|
||||
// leaves the version with no default at all.
|
||||
const target_version = scope === undefined ? row.version_id
|
||||
: (scope === 'source' ? null : row.version_id);
|
||||
await client.query('BEGIN');
|
||||
if (is_default === true) {
|
||||
await client.query(
|
||||
`UPDATE pf.layout SET is_default = false
|
||||
WHERE TRUE
|
||||
AND source_id = $1
|
||||
AND COALESCE(version_id, 0) = COALESCE($2::int, 0)
|
||||
AND id <> $3
|
||||
AND is_default`,
|
||||
[row.source_id, target_version, row.id]
|
||||
);
|
||||
}
|
||||
|
||||
const { rows } = await client.query(
|
||||
`UPDATE pf.layout SET
|
||||
name = CASE WHEN $2::bool THEN $3::text ELSE name END,
|
||||
config = CASE WHEN $4::bool THEN $5::jsonb ELSE config END,
|
||||
visibility = CASE WHEN $6::bool THEN $7::text ELSE visibility END,
|
||||
is_default = CASE WHEN $10::bool THEN false
|
||||
WHEN $8::bool THEN $9::bool ELSE is_default END,
|
||||
version_id = CASE WHEN $11::bool THEN $12::int ELSE version_id END,
|
||||
updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING ${SELECT_COLS}`,
|
||||
[row.id,
|
||||
name !== undefined, name !== undefined ? String(name).trim() : null,
|
||||
config !== undefined, config !== undefined ? config : null,
|
||||
visibility !== undefined, visibility !== undefined ? visibility : null,
|
||||
is_default !== undefined, is_default === true,
|
||||
clearsDefault,
|
||||
scope !== undefined, target_version]
|
||||
);
|
||||
await client.query('COMMIT');
|
||||
res.json(decorate(rows[0], req));
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK').catch(() => {});
|
||||
if (err.code === '23505') {
|
||||
return res.status(409).json({ error: 'A layout with that name already exists here' });
|
||||
}
|
||||
res.status(500).json({ error: err.message });
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
});
|
||||
|
||||
router.delete('/layouts/:lid', async (req, res) => {
|
||||
try {
|
||||
const row = await ownedLayout(req, res);
|
||||
if (!row) return;
|
||||
await pool.query(`DELETE FROM pf.layout WHERE id = $1`, [row.id]);
|
||||
res.json({ deleted: row.id });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
return router;
|
||||
};
|
||||
202
routes/log.js
202
routes/log.js
@ -1,6 +1,4 @@
|
||||
const express = require('express');
|
||||
const { grainOf } = require('../lib/sql_generator');
|
||||
const { sessionUser } = require('../lib/auth');
|
||||
const { fcTable } = require('../lib/utils');
|
||||
|
||||
module.exports = function(pool) {
|
||||
@ -31,96 +29,17 @@ module.exports = function(pool) {
|
||||
unitsCol ? `sum(f."${unitsCol}")::float8 AS units_total` : `NULL::float8 AS units_total`
|
||||
].join(', ');
|
||||
|
||||
// The totals are stamped onto the entry when it is written, so the
|
||||
// normal read is a scan of a few dozen log rows rather than a join
|
||||
// against millions of forecast rows.
|
||||
//
|
||||
// ?recount=1 does it the old way. Stored totals are fixed at write
|
||||
// time and cannot drift on their own, but nothing stops someone
|
||||
// deleting forecast rows by hand, and a stored figure has no way to
|
||||
// notice. This is the way back -- and the backfill for entries
|
||||
// written before the columns existed.
|
||||
const recount = req.query.recount === '1' || req.query.recount === 'true';
|
||||
const stamped = !recount && (await pool.query(
|
||||
`SELECT count(*)::int AS n FROM pf.log
|
||||
WHERE version_id = $1 AND row_count IS NULL`, [versionId]
|
||||
)).rows[0].n === 0;
|
||||
|
||||
// ?kind=adjustments drops the baseline and reference entries. That is not
|
||||
// only about what gets listed: the aggregate below joins the whole
|
||||
// forecast table, and on a real version the load entries own almost
|
||||
// every row of it -- 2.5M against a few thousand for the adjustments.
|
||||
// Filtering in the WHERE keeps them out of the join rather than
|
||||
// totalling them and discarding the answer.
|
||||
const adjustmentsOnly = req.query.kind === 'adjustments';
|
||||
const opFilter = adjustmentsOnly
|
||||
? `AND l.operation NOT IN ('baseline', 'reference')`
|
||||
: '';
|
||||
|
||||
const result = stamped
|
||||
? await pool.query(`
|
||||
SELECT l.*,
|
||||
$2::text AS value_col,
|
||||
$3::text AS units_col
|
||||
FROM pf.log l
|
||||
WHERE l.version_id = $1
|
||||
${opFilter}
|
||||
ORDER BY l.id DESC
|
||||
`, [versionId, valueCol || null, unitsCol || null])
|
||||
: await pool.query(`
|
||||
SELECT l.*, ${aggCols},
|
||||
$2::text AS value_col,
|
||||
$3::text AS units_col
|
||||
FROM pf.log l
|
||||
LEFT JOIN ${table} f ON f.pf_logid = l.id
|
||||
WHERE l.version_id = $1
|
||||
${opFilter}
|
||||
GROUP BY l.id
|
||||
ORDER BY l.id DESC
|
||||
`, [versionId, valueCol || null, unitsCol || null]);
|
||||
|
||||
// A recount is also a repair: write back what it found, so the next
|
||||
// read is cheap again and the stored figure matches the rows.
|
||||
if (recount) {
|
||||
for (const r of result.rows) {
|
||||
await pool.query(
|
||||
`UPDATE pf.log SET row_count = $2, value_total = $3, units_total = $4,
|
||||
measure_cols = $5::jsonb
|
||||
WHERE id = $1`,
|
||||
[r.id, r.row_count, r.value_total, r.units_total,
|
||||
JSON.stringify({ value: valueCol || null, units: unitsCol || null })]
|
||||
);
|
||||
}
|
||||
}
|
||||
// The statement is kilobytes per entry and the list is opened to scan,
|
||||
// not to read SQL. It stays in the row for the debug endpoint below.
|
||||
res.json(result.rows.map(({ sql_text, ...r }) => ({
|
||||
...r, has_sql: !!sql_text,
|
||||
})));
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(err.status || 500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Everything about one entry, for when the rows look wrong: what was asked
|
||||
// for (params), what it ran against (env), and the statement that actually
|
||||
// executed with territory and scope resolved into it (sql_text).
|
||||
//
|
||||
// Both the intent and the SQL, because the translation between them is
|
||||
// exactly what is in doubt when a result is surprising.
|
||||
router.get('/log/:logid/debug', async (req, res) => {
|
||||
const logId = parseInt(req.params.logid);
|
||||
try {
|
||||
const { rows } = await pool.query(
|
||||
`SELECT l.*, v.name AS version_name, s.schema, s.tname
|
||||
FROM pf.log l
|
||||
JOIN pf.version v ON v.id = l.version_id
|
||||
JOIN pf.source s ON s.id = v.source_id
|
||||
WHERE l.id = $1`, [logId]
|
||||
);
|
||||
if (!rows.length) return res.status(404).json({ error: 'Log entry not found' });
|
||||
res.json(rows[0]);
|
||||
const result = await pool.query(`
|
||||
SELECT l.*, ${aggCols},
|
||||
$2::text AS value_col,
|
||||
$3::text AS units_col
|
||||
FROM pf.log l
|
||||
LEFT JOIN ${table} f ON f.pf_logid = l.id
|
||||
WHERE l.version_id = $1
|
||||
GROUP BY l.id
|
||||
ORDER BY l.id DESC
|
||||
`, [versionId, valueCol || null, unitsCol || null]);
|
||||
res.json(result.rows);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(err.status || 500).json({ error: err.message });
|
||||
@ -132,7 +51,7 @@ module.exports = function(pool) {
|
||||
const logId = parseInt(req.params.logid);
|
||||
try {
|
||||
const logResult = await pool.query(`
|
||||
SELECT l.*, v.status, s.tname, v.id AS version_id, v.source_id
|
||||
SELECT l.*, v.status, s.tname, v.id AS version_id
|
||||
FROM pf.log l
|
||||
JOIN pf.version v ON v.id = l.version_id
|
||||
JOIN pf.source s ON s.id = v.source_id
|
||||
@ -141,58 +60,19 @@ module.exports = function(pool) {
|
||||
if (!logResult.rows.length) return res.status(404).json({ error: 'Log entry not found' });
|
||||
const log = logResult.rows[0];
|
||||
if (log.status === 'closed') return res.status(403).json({ error: 'Version is closed' });
|
||||
|
||||
// Undo deletes rows wholesale by logid, so it cannot be territory
|
||||
// filtered the way a read or a write can -- half-undoing an entry
|
||||
// would leave the version in a state nothing describes. Ownership
|
||||
// instead: your own entries, or an admin's override. Territory alone
|
||||
// would not do it anyway, since two accounts can share one.
|
||||
if (!req.session?.user?.is_admin && log.pf_user !== sessionUser(req)) {
|
||||
return res.status(403).json({
|
||||
error: `That entry was made by ${log.pf_user || 'someone else'} — only they or an administrator can undo it`
|
||||
});
|
||||
}
|
||||
const table = fcTable(log.tname, log.version_id);
|
||||
|
||||
// In grain mode the client's table is indexed on pf_gkey, so undo has to
|
||||
// report the grain keys to remove rather than raw pf_ids. The keys are
|
||||
// distinct while rows_deleted still counts the raw rows removed.
|
||||
const colMeta = await pool.query(
|
||||
`SELECT cname, role, in_grain, opos FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`,
|
||||
[log.source_id]
|
||||
);
|
||||
const grain = grainOf(colMeta.rows);
|
||||
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const deleted = grain
|
||||
? await client.query(`
|
||||
WITH
|
||||
del AS (
|
||||
DELETE FROM ${table}
|
||||
WHERE pf_logid = $1
|
||||
RETURNING ${grain.groupCols().join(', ')}
|
||||
)
|
||||
SELECT
|
||||
count(*)::int AS rows_deleted
|
||||
,array_agg(DISTINCT ${grain.key()}) AS pf_gkeys
|
||||
FROM del
|
||||
`, [logId])
|
||||
: await client.query(
|
||||
`DELETE FROM ${table} WHERE pf_logid = $1 RETURNING pf_id`, [logId]
|
||||
);
|
||||
const deleted = await client.query(
|
||||
`DELETE FROM ${table} WHERE pf_logid = $1 RETURNING pf_id`, [logId]
|
||||
);
|
||||
await client.query('DELETE FROM pf.log WHERE id = $1', [logId]);
|
||||
await client.query('COMMIT');
|
||||
res.json(grain
|
||||
? {
|
||||
rows_deleted: deleted.rows[0].rows_deleted,
|
||||
pf_gkeys: deleted.rows[0].pf_gkeys || []
|
||||
}
|
||||
: {
|
||||
rows_deleted: deleted.rowCount,
|
||||
pf_ids: deleted.rows.map(r => r.pf_id)
|
||||
});
|
||||
res.json({
|
||||
rows_deleted: deleted.rowCount,
|
||||
pf_ids: deleted.rows.map(r => r.pf_id)
|
||||
});
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK');
|
||||
throw err;
|
||||
@ -210,52 +90,22 @@ module.exports = function(pool) {
|
||||
// a closed version, where relabelling history is still legitimate.
|
||||
router.patch('/log/:logid', async (req, res) => {
|
||||
const logId = parseInt(req.params.logid);
|
||||
const { note, tag, bucket, label } = req.body;
|
||||
if (note === undefined && tag === undefined
|
||||
&& bucket === undefined && label === undefined) {
|
||||
return res.status(400).json({
|
||||
error: 'Nothing to update — send note, tag, bucket and/or label'
|
||||
});
|
||||
}
|
||||
// Every entry carries a label and a bucket (pf.log's log_named CHECK), so
|
||||
// they can be renamed but not cleared. Refused here for the message; the
|
||||
// constraint would refuse it anyway.
|
||||
for (const [field, value] of [['label', label], ['bucket', bucket]]) {
|
||||
if (value !== undefined && !String(value ?? '').trim()) {
|
||||
return res.status(400).json({ error: `The ${field} can be renamed but not cleared` });
|
||||
}
|
||||
const { note, tag } = req.body;
|
||||
if (note === undefined && tag === undefined) {
|
||||
return res.status(400).json({ error: 'Nothing to update — send note and/or tag' });
|
||||
}
|
||||
try {
|
||||
// Same rule as undo: your own entries, or an admin's. These are
|
||||
// annotations, but label and bucket name the pivot's columns for
|
||||
// everyone who opens the version, so an unguarded PATCH let any
|
||||
// account rename the company's segments -- including on loads whose
|
||||
// rows it cannot see.
|
||||
const owner = await pool.query(
|
||||
`SELECT pf_user FROM pf.log WHERE id = $1`, [logId]
|
||||
);
|
||||
if (!owner.rows.length) return res.status(404).json({ error: 'Log entry not found' });
|
||||
if (!req.session?.user?.is_admin && owner.rows[0].pf_user !== sessionUser(req)) {
|
||||
return res.status(403).json({
|
||||
error: `That entry was made by ${owner.rows[0].pf_user || 'someone else'} — only they or an administrator can change it`
|
||||
});
|
||||
}
|
||||
|
||||
// COALESCE on the flag, not the value: an explicit null or '' must be
|
||||
// able to clear a field, which COALESCE on the value alone would ignore
|
||||
const result = await pool.query(
|
||||
`UPDATE pf.log SET
|
||||
note = CASE WHEN $2::bool THEN $3::text ELSE note END,
|
||||
tag = CASE WHEN $4::bool THEN $5::text ELSE tag END,
|
||||
bucket = CASE WHEN $6::bool THEN $7::text ELSE bucket END,
|
||||
label = CASE WHEN $8::bool THEN $9::text ELSE label END
|
||||
note = CASE WHEN $2::bool THEN $3::text ELSE note END,
|
||||
tag = CASE WHEN $4::bool THEN $5::text ELSE tag END
|
||||
WHERE id = $1 RETURNING *`,
|
||||
[
|
||||
logId,
|
||||
note !== undefined, note === undefined ? null : (String(note).trim() || null),
|
||||
tag !== undefined, tag === undefined ? null : (String(tag).trim() || null),
|
||||
bucket !== undefined, bucket === undefined ? null : (String(bucket).trim() || null),
|
||||
label !== undefined, label === undefined ? null : (String(label).trim() || null),
|
||||
note !== undefined, note === undefined ? null : (String(note).trim() || null),
|
||||
tag !== undefined, tag === undefined ? null : (String(tag).trim() || null),
|
||||
]
|
||||
);
|
||||
if (!result.rows.length) return res.status(404).json({ error: 'Log entry not found' });
|
||||
|
||||
@ -1,55 +1,21 @@
|
||||
const express = require('express');
|
||||
const { tableFromArrays, tableToIPC } = require('apache-arrow');
|
||||
const { applyTokens, buildWhere, buildWhereAny, COMPUTED_SLICE_COLS, TABLE_SLICE_COLS, buildScopeClause, buildTerritoryClause, buildExcludeClause, buildExcludePredicate, buildSetClause, dateGroupsOf, dimPeriodMapOf, esc,
|
||||
SEGMENT_EXPR, BUCKET_EXPR, NOTE_EXPR,
|
||||
ADJUSTMENT_SEGMENT, ADJUSTMENT_BUCKET } = require('../lib/sql_generator');
|
||||
const { sessionUser, sessionTerritory, requireAdmin } = require('../lib/auth');
|
||||
const { applyTokens, buildWhere, buildWhereAny, buildExcludeClause, buildExcludePredicate, buildSetClause, esc } = require('../lib/sql_generator');
|
||||
const { fcTable } = require('../lib/utils');
|
||||
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
// The segment and bucket an adjustment is stamped with: written onto its
|
||||
// pf.log entry by the template, and onto the rows pushed back to the pivot,
|
||||
// so the two agree. The version's own defaults when set, else the
|
||||
// built-ins. Resolved at write time, so changing a version's default
|
||||
// afterwards names new adjustments and leaves existing ones as they are.
|
||||
function adjustmentNames(version) {
|
||||
const pick = (v, dflt) => (v || '').trim() || dflt;
|
||||
return {
|
||||
pf_segment: pick(version?.adjustment_segment, ADJUSTMENT_SEGMENT),
|
||||
pf_bucket: pick(version?.adjustment_bucket, ADJUSTMENT_BUCKET),
|
||||
};
|
||||
}
|
||||
|
||||
// The segment and bucket a load is stamped with. A load's label is its
|
||||
// segment name and is required -- it used to fall back through tag, note and
|
||||
// the version's unlabeled_load at read time, which left pf.log unable to say
|
||||
// what a row was called. The bucket is the label unless one is given, which
|
||||
// is how an unbucketed load has always read. Sends the 400 and returns null
|
||||
// when there is no label.
|
||||
function loadNames(label, bucket, res) {
|
||||
const segment = String(label ?? '').trim();
|
||||
if (!segment) {
|
||||
res.status(400).json({ error: 'Give the segment a label — it is what the pivot shows and what reports group on' });
|
||||
return null;
|
||||
}
|
||||
return { label: segment, bucket: String(bucket ?? '').trim() || segment };
|
||||
}
|
||||
|
||||
async function runSQL(sql, client) {
|
||||
console.log('--- SQL ---\n', sql, '\n--- END SQL ---');
|
||||
return (client || pool).query(sql);
|
||||
}
|
||||
|
||||
// accept either the legacy single `slice` object or the newer `slices` array.
|
||||
// An empty slice is a total cell -- everything the pivot's filter shows -- so
|
||||
// it is kept only when a scope came with it; without one it would be the
|
||||
// whole version, and is dropped so an empty selection can never widen to TRUE
|
||||
// accept either the legacy single `slice` object or the newer `slices` array,
|
||||
// and drop any empty entries so an empty selection can never widen to TRUE
|
||||
function normalizeSlices(body) {
|
||||
const raw = Array.isArray(body.slices) && body.slices.length ? body.slices : [body.slice];
|
||||
const scoped = Array.isArray(body.scope) && body.scope.length > 0;
|
||||
return raw.filter(s => s && typeof s === 'object' && (scoped || Object.keys(s).length > 0));
|
||||
return raw.filter(s => s && typeof s === 'object' && Object.keys(s).length > 0);
|
||||
}
|
||||
|
||||
// Stamp the tag onto the log entry the operation just created.
|
||||
@ -65,81 +31,13 @@ module.exports = function(pool) {
|
||||
return clean;
|
||||
}
|
||||
|
||||
// How a multi-slice request is split into statements.
|
||||
//
|
||||
// 'each' — one statement per slice, so every slice reaches its target on
|
||||
// its own and gets its own log entry
|
||||
// 'prorate' — a single statement over all of them, letting the SQL's
|
||||
// sum() OVER () distribute across the whole pool
|
||||
//
|
||||
// Only scale has a target to prorate towards; recode and clone rewrite rows
|
||||
// rather than distribute an amount, so for them this only decides whether the
|
||||
// work lands as one log entry or several.
|
||||
// The scope is ANDed onto every unit rather than folded into the slices: it
|
||||
// applies to all of them equally, and under apply_mode 'each' folding it in
|
||||
// would repeat the same predicate in every statement for no gain.
|
||||
function sliceUnits(slices, ctx, applyMode, scope, req) {
|
||||
const vid = ctx.version.id;
|
||||
const scl = buildScopeClause(scope, ctx.filterCols, vid);
|
||||
const terr = req ? territoryOf(req, ctx) : '';
|
||||
const and = (w) => {
|
||||
let out = w;
|
||||
for (const extra of [scl, terr]) {
|
||||
if (!extra) continue;
|
||||
out = (out === 'TRUE' || !out) ? extra : `${out}\nAND ${extra}`;
|
||||
}
|
||||
return out;
|
||||
};
|
||||
return applyMode === 'each'
|
||||
? slices.map(sl => ({ slices: [sl], where: and(buildWhere(sl, ctx.filterCols, vid)) }))
|
||||
: [{ slices, where: and(buildWhereAny(slices, ctx.filterCols, vid)) }];
|
||||
}
|
||||
|
||||
// The offset is interpolated into the statement as an interval literal, so a
|
||||
// typo would surface as a Postgres parse error from the middle of a CTE. Ask
|
||||
// Postgres to parse it alone first, where the failure is cheap and can name the
|
||||
// field it came from. Negative intervals are valid and useful -- '-90 days'
|
||||
// pulls a plan back a quarter -- so this checks validity, not sign.
|
||||
// A date offset has to be a whole number of days. Every date it shifts is
|
||||
// cast back to a date, so anything finer is rounded away -- and Postgres
|
||||
// reads a bare number as seconds, so "30" meant thirty seconds, moved
|
||||
// nothing, and a shifted recode took rows out of a cell and put them back
|
||||
// into the same one.
|
||||
async function assertInterval(value, res) {
|
||||
const example = `Try something like "30 days", "1 month", "-90 days" or "0 days".`;
|
||||
let whole;
|
||||
try {
|
||||
whole = (await pool.query(
|
||||
`SELECT date_trunc('day', $1::interval) = $1::interval AS whole`, [value]
|
||||
)).rows[0].whole;
|
||||
} catch {
|
||||
res.status(400).json({ error: `"${value}" is not a valid interval. ${example}` });
|
||||
return false;
|
||||
}
|
||||
if (!whole) {
|
||||
res.status(400).json({
|
||||
error: `"${value}" is not a whole number of days, so it would not move any date `
|
||||
+ `(a number with no unit is read as seconds). ${example}`
|
||||
});
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// A slice is only meaningful if at least one of its keys is a filterable
|
||||
// column. buildWhere silently drops unknown keys, so {"typo": "x"} would
|
||||
// otherwise reduce to TRUE and apply the operation to the whole version.
|
||||
// Refuse rather than let a malformed selection rewrite every row.
|
||||
// An empty slice passes only under a scope (see normalizeSlices), which
|
||||
// buildScopeClause refuses to let reduce to nothing.
|
||||
function assertSelective(slices, ctx, scope) {
|
||||
// pf_iter, pf_note, pf_op, pf_user and pf_logid narrow a slice once it
|
||||
// is selective, but do not make it so: they were only just accepted at
|
||||
// all, and `pf_iter = scale` alone is every adjustment in the version
|
||||
const allowed = new Set([...ctx.filterCols, 'pf_segment', 'pf_bucket']);
|
||||
const scoped = Array.isArray(scope) && scope.length > 0;
|
||||
function assertSelective(slices, ctx) {
|
||||
const allowed = new Set(ctx.filterCols);
|
||||
slices.forEach((sl, i) => {
|
||||
if (scoped && Object.keys(sl).length === 0) return;
|
||||
const hits = Object.keys(sl).filter(k => allowed.has(k));
|
||||
if (hits.length === 0) {
|
||||
const err = new Error(
|
||||
@ -152,80 +50,10 @@ module.exports = function(pool) {
|
||||
});
|
||||
}
|
||||
|
||||
// Stamp what the entry did onto the entry itself.
|
||||
//
|
||||
// An indexed lookup on pf_logid, run once at write time, in place of the
|
||||
// change log joining the whole forecast table on every open. Safe to store
|
||||
// rather than derive because these rows never change: only this operation
|
||||
// inserts them, and the only thing that removes them is undo, which deletes
|
||||
// the log row too.
|
||||
//
|
||||
// Best-effort by design. A failure here must not roll back a write that
|
||||
// succeeded -- the totals can always be recomputed, the adjustment cannot.
|
||||
async function stampLogTotals(client, ctx, logId, extra = {}) {
|
||||
if (!logId) return;
|
||||
const v = ctx.valueCol, u = ctx.unitsCol;
|
||||
// The state the write ran against, none of which can be reconstructed
|
||||
// later: territory and exclude_iters are mutable rows elsewhere, and the
|
||||
// template is overwritten in place every time Generate SQL runs. The
|
||||
// generation timestamp is a fingerprint, not a version -- it cannot bring
|
||||
// the old template back, only tell you the entry did not run under this
|
||||
// one.
|
||||
const env = {
|
||||
territory: extra.territory ?? null,
|
||||
territory_col: ctx.territoryCol || null,
|
||||
exclude_iters: ctx.version.exclude_iters ?? null,
|
||||
sql_generated_at: ctx.sqlGeneratedAt || null,
|
||||
};
|
||||
try {
|
||||
await client.query(`
|
||||
UPDATE pf.log SET
|
||||
row_count = t.n,
|
||||
value_total = t.v,
|
||||
units_total = t.u,
|
||||
measure_cols = $2::jsonb,
|
||||
env = $3::jsonb,
|
||||
sql_text = $4::text
|
||||
FROM (
|
||||
SELECT count(*)::int AS n
|
||||
,${v ? `sum(f."${v}")::float8` : 'NULL::float8'} AS v
|
||||
,${u ? `sum(f."${u}")::float8` : 'NULL::float8'} AS u
|
||||
FROM ${ctx.table} f
|
||||
WHERE f.pf_logid = $1
|
||||
) t
|
||||
WHERE pf.log.id = $1
|
||||
`, [logId, JSON.stringify({ value: v || null, units: u || null }),
|
||||
JSON.stringify(env), extra.sql || null]);
|
||||
} catch (err) {
|
||||
console.error('[stampLogTotals]', err);
|
||||
}
|
||||
}
|
||||
|
||||
// Moving a row between territories is reassignment, not forecasting, so a
|
||||
// scoped account cannot set the territory column -- otherwise a rep could
|
||||
// recode work into their own book, or quietly out of it, and the row would
|
||||
// be gone from the view that would have shown what happened.
|
||||
//
|
||||
// Reads and writes are already scoped, so the *source* rows are safely the
|
||||
// account's own; this is only about the destination.
|
||||
function assertMayRecodeTerritory(req, ctx, set, res) {
|
||||
if (!ctx.territoryCol) return true;
|
||||
if (req.session?.user?.is_admin) return true;
|
||||
if (!set || set[ctx.territoryCol] === undefined) return true;
|
||||
res.status(403).json({
|
||||
error: `Only an administrator can recode ${ctx.territoryCol} — that moves rows between territories`
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
// echo back what the caller asked for, for the audit log
|
||||
function pickIntent(body) {
|
||||
const keys = ['mode', 'target_basis', 'value_incr', 'units_incr', 'value_pct', 'units_pct', 'pct',
|
||||
'target_value', 'target_units', 'target_price', 'scope', 'plug',
|
||||
// What the panel's Result line promised, kept beside
|
||||
// `resolved` so a disagreement between the two can be
|
||||
// found by reading the log. Not checked -- yet.
|
||||
'expected'];
|
||||
'target_value', 'target_units', 'target_price'];
|
||||
const out = {};
|
||||
for (const k of keys) if (body[k] !== undefined && body[k] !== null && body[k] !== '') out[k] = body[k];
|
||||
return out;
|
||||
@ -308,96 +136,12 @@ module.exports = function(pool) {
|
||||
};
|
||||
|
||||
let value = resolve(tValue, vPct, vIncr, totals.value, fixedValue);
|
||||
let units = resolve(tUnits, uPct, uIncr, totals.units, fixedUnits);
|
||||
const units = resolve(tUnits, uPct, uIncr, totals.units, fixedUnits);
|
||||
|
||||
// A price target is the "edit price" mode of the Excel form. The price
|
||||
// named holds, and whichever other measure was named decides the rest:
|
||||
// price alone qty holds, dollars = price x qty
|
||||
// price + qty qty lands as named (target, change or %),
|
||||
// dollars = price x that qty
|
||||
// price + dollars dollars land as named, qty = dollars / price
|
||||
// all three refused -- two degrees of freedom, three numbers
|
||||
// All on the basis the panel measures against, so under 'selected' the
|
||||
// price is that of the whole selection, fixed rows included.
|
||||
//
|
||||
// This used to read only a qty *target*, so a qty change or % sized the
|
||||
// dollars on the old qty and missed the price; took the fixed qty off a
|
||||
// qty target but not the fixed dollars off the value, so under 'selected'
|
||||
// it landed short; and let a dollar figure outrank the price outright.
|
||||
const unitsGiven = [tUnits, uIncr, uPct].some(v => v !== null);
|
||||
const valueGiven = [tValue, vIncr, vPct].some(v => v !== null);
|
||||
if (tPrice !== null) {
|
||||
const bad = (msg) => { const err = new Error(msg); err.status = 400; return err; };
|
||||
if (valueGiven && unitsGiven) {
|
||||
throw bad('Price, dollars and qty were all given. Any two decide the third -- clear one of them.');
|
||||
}
|
||||
const curValue = totals.value + fixedValue;
|
||||
const curUnits = totals.units + fixedUnits;
|
||||
if (valueGiven) {
|
||||
if (!(tPrice > 0)) throw bad('A price of zero or less cannot set qty from dollars.');
|
||||
units = (curValue + value) / tPrice - curUnits;
|
||||
} else {
|
||||
value = tPrice * (curUnits + units) - curValue;
|
||||
}
|
||||
}
|
||||
|
||||
// Which side of price x volume absorbs a dollar change.
|
||||
//
|
||||
// 'volume' — price holds, so volume scales with the dollars. The default.
|
||||
// 'price' — volume holds, so price moves.
|
||||
//
|
||||
// Only meaningful when dollars were the input and units were not given
|
||||
// explicitly; naming both means the caller has already decided.
|
||||
//
|
||||
// 'price' was the default, being what the API did before the flag existed,
|
||||
// and that made every unqualified dollar edit a price edit. A forecast
|
||||
// that moves price when nobody asked is wrong in a way that is invisible
|
||||
// -- the dollars land on target either way -- so the default is now the
|
||||
// conservative one and moving price takes saying so.
|
||||
const plug = body.plug === 'price' ? 'price' : 'volume';
|
||||
|
||||
// The price held is the *adjustable* rows' own, never a blend with the
|
||||
// fixed ones: the scale SQL prorates both increments over the adjustable
|
||||
// rows only, so a units increment sized at the blended price cuts qty
|
||||
// faster than dollars whenever the fixed rows are priced differently.
|
||||
// Taking a slice to its fixed total -- dollars to zero -- then drove qty
|
||||
// past zero, leaving grains with positive sales and negative qty.
|
||||
//
|
||||
// Not when a price was named. That is an edit-price, not a dollar edit:
|
||||
// qty holds and the dollars were sized off the price above. The panel
|
||||
// greys the plug out then and does not send it, so `plug` reads as its
|
||||
// 'volume' default, and this block used to scale qty by the dollar
|
||||
// change as well -- which put price straight back where it started.
|
||||
if (plug === 'volume' && value !== 0 && !unitsGiven && tPrice === null) {
|
||||
if (totals.value === 0) {
|
||||
const err = new Error(
|
||||
'Cannot hold price constant here: the selection currently has no value, ' +
|
||||
'so there is no price to hold. Scale units directly, or let price absorb ' +
|
||||
'the change.'
|
||||
);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
// price constant means value and units move by the same proportion:
|
||||
// fVol = adjVol * (fVal / adjVal), so the units delta is adjVol * value/adjVal
|
||||
units = totals.units * (value / totals.value);
|
||||
}
|
||||
|
||||
// The same question asked of a qty input, and the answer has to match:
|
||||
// with dollars unnamed they follow at the current price, rather than
|
||||
// holding while price absorbs the difference.
|
||||
//
|
||||
// Holding them made every qty edit a price edit -- move 10% more cases
|
||||
// at the same revenue and the implied price drops 9% -- which is exactly
|
||||
// the silent price change the plug default above exists to prevent. The
|
||||
// deliberate forms are still available and both outrank this: name the
|
||||
// dollars alongside the qty (then price falls out of the two), or name
|
||||
// the price outright.
|
||||
// Again at the adjustable rows' own price, for the reason given above.
|
||||
if (units !== 0 && !valueGiven && tPrice === null) {
|
||||
// no units means no price to hold, so the dollars stay where they are
|
||||
// and the edit is a pure qty change -- adding units to a slice that
|
||||
// has none cannot infer what they are worth
|
||||
if (totals.units !== 0) value = (totals.value / totals.units) * units;
|
||||
// a price target holds units constant: new value = price x current units.
|
||||
// An explicit value target outranks it.
|
||||
if (tPrice !== null && tValue === null) {
|
||||
value = (tPrice * (totals.units + fixedUnits)) - (totals.value + fixedValue);
|
||||
}
|
||||
|
||||
// the scale SQL divides by the slice total; with no rows there is
|
||||
@ -411,19 +155,7 @@ module.exports = function(pool) {
|
||||
// offset each other is the usual cause, and 'each' handles that correctly.
|
||||
assertProratable(totals, value, units);
|
||||
|
||||
// Which of price or volume the edit moved, written onto the log entry.
|
||||
// The request's own fields do not say -- `plug` is a per-browser
|
||||
// preference, and an unqualified dollar figure is a price edit or a
|
||||
// volume edit depending on it -- so without this the log could not tell
|
||||
// the two apart, and the only way to find a price edit afterwards was to
|
||||
// notice qty had not moved.
|
||||
const method =
|
||||
tPrice !== null ? 'price' // price named
|
||||
: unitsGiven && valueGiven ? 'both' // dollars and qty named, price falls out
|
||||
: unitsGiven ? 'volume' // qty named, dollars follow at price
|
||||
: plug; // dollars only: the plug decides
|
||||
|
||||
return { value: round(value, 6), units: round(units, 6), method };
|
||||
return { value: round(value, 6), units: round(units, 6) };
|
||||
}
|
||||
|
||||
// a pool is proratable only if its net is a meaningful fraction of its gross
|
||||
@ -479,7 +211,7 @@ module.exports = function(pool) {
|
||||
const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
|
||||
|
||||
const sqlResult = await pool.query(
|
||||
`SELECT sql, generated_at FROM pf.sql WHERE source_id = $1 AND operation = $2`,
|
||||
`SELECT sql FROM pf.sql WHERE source_id = $1 AND operation = $2`,
|
||||
[version.source_id, operation]
|
||||
);
|
||||
if (sqlResult.rows.length === 0) {
|
||||
@ -496,26 +228,10 @@ module.exports = function(pool) {
|
||||
filterCols: [...dimCols, ...dateCols],
|
||||
valueCol,
|
||||
unitsCol,
|
||||
territoryCol: colMeta.find(c => c.is_territory)?.cname || null,
|
||||
sql: sqlResult.rows[0].sql,
|
||||
sqlGeneratedAt: sqlResult.rows[0].generated_at
|
||||
sql: sqlResult.rows[0].sql
|
||||
};
|
||||
}
|
||||
|
||||
// Every read and every write goes through this, so a scoped account cannot
|
||||
// reach a row outside its territory by any route. ANDed on last, after the
|
||||
// slice and the client's own scope, where nothing in the request can undo
|
||||
// it.
|
||||
function territoryOf(req, ctx) {
|
||||
return buildTerritoryClause(sessionTerritory(req), ctx.territoryCol);
|
||||
}
|
||||
|
||||
function andTerritory(where, req, ctx) {
|
||||
const t = territoryOf(req, ctx);
|
||||
if (!t) return where;
|
||||
return (!where || where === 'TRUE') ? t : `${where}\nAND ${t}`;
|
||||
}
|
||||
|
||||
function guardOpen(version, res) {
|
||||
if (version.status === 'closed') {
|
||||
res.status(403).json({ error: 'Version is closed' });
|
||||
@ -538,19 +254,7 @@ module.exports = function(pool) {
|
||||
}
|
||||
const tbl = fcTable(verResult.rows[0].tname, versionId);
|
||||
|
||||
// /data does not go through getContext, so it resolves the territory
|
||||
// column itself. The count is scoped too, or the progress bar
|
||||
// promises rows this account will never be sent.
|
||||
const terrCol = (await pool.query(
|
||||
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
|
||||
[verResult.rows[0].source_id]
|
||||
)).rows[0]?.cname || null;
|
||||
const territory = sessionTerritory(req);
|
||||
const terrBare = buildTerritoryClause(territory, terrCol);
|
||||
const terrAlias = buildTerritoryClause(territory, terrCol, 't');
|
||||
const terrWhere = terrBare ? `WHERE ${terrBare}` : '';
|
||||
|
||||
const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl} ${terrWhere}`);
|
||||
const { rows: [{ count }] } = await pool.query(`SELECT COUNT(*) FROM ${tbl}`);
|
||||
const rowCount = parseInt(count);
|
||||
|
||||
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
|
||||
@ -562,14 +266,7 @@ module.exports = function(pool) {
|
||||
await client.query('BEGIN');
|
||||
await client.query(`
|
||||
DECLARE pf_cur CURSOR FOR
|
||||
SELECT t.*
|
||||
,${SEGMENT_EXPR} AS pf_segment
|
||||
,${BUCKET_EXPR} AS pf_bucket
|
||||
,${NOTE_EXPR} AS pf_note
|
||||
FROM ${tbl} t
|
||||
LEFT JOIN pf.log l
|
||||
ON l.id = t.pf_logid
|
||||
${terrAlias ? `WHERE ${terrAlias}` : ''}
|
||||
SELECT * FROM ${tbl}
|
||||
`);
|
||||
|
||||
// Accumulate into column arrays (not row objects) to avoid allocating one JS
|
||||
@ -605,75 +302,11 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
// Aggregate a version to its display grain and return it as Arrow IPC.
|
||||
// This replaces /data for sources that define a grain (col_meta.in_grain):
|
||||
// the aggregation collapses the row count by orders of magnitude, so the
|
||||
// result loads as one small native Perspective table indexed on pf_gkey and
|
||||
// the WASM view still does all rollup/expand/collapse locally.
|
||||
router.get('/versions/:id/agg', async (req, res) => {
|
||||
let client;
|
||||
let committed = false;
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'get_agg');
|
||||
// Before the GROUP BY, not after: the territory column need not be
|
||||
// part of the grain, so an aggregated row may not carry it at all.
|
||||
const sql = applyTokens(ctx.sql, {
|
||||
fc_table: ctx.table,
|
||||
territory_clause:
|
||||
buildTerritoryClause(sessionTerritory(req), ctx.territoryCol, 't') || 'TRUE',
|
||||
});
|
||||
|
||||
// The GROUP BY spans every in_grain column plus pf_logid and the
|
||||
// pf.log/pf.version columns joined in for segment/bucket — 30+ keys
|
||||
// on a wide source. The planner sees pf_logid arrive pre-sorted off
|
||||
// its index and picks a sort-based GroupAggregate on the strength of
|
||||
// that "free" sort, but comparing that many columns per row makes it
|
||||
// 5-7x slower in practice than a HashAggregate — confirmed via
|
||||
// EXPLAIN ANALYZE on fc_osm_skinny_34 (34-50s sorted vs ~7s hashed,
|
||||
// independent of work_mem or stats). Scoped to this transaction only:
|
||||
// disabling incremental sort globally could hurt some other query's plan.
|
||||
client = await pool.connect();
|
||||
await client.query('BEGIN');
|
||||
await client.query('SET LOCAL enable_sort = off');
|
||||
await client.query('SET LOCAL enable_incremental_sort = off');
|
||||
const { rows } = await runSQL(sql, client);
|
||||
await client.query('COMMIT');
|
||||
committed = true;
|
||||
|
||||
res.setHeader('Content-Type', 'application/vnd.apache.arrow.stream');
|
||||
res.setHeader('X-Row-Count', String(rows.length));
|
||||
if (rows.length === 0) { res.end(); return; }
|
||||
|
||||
// column arrays, one Arrow record batch — same constraint as /data:
|
||||
// per-batch dictionaries crash Perspective's Arrow reader
|
||||
const colArrays = Object.fromEntries(Object.keys(rows[0]).map(k => [k, []]));
|
||||
for (const row of rows) {
|
||||
for (const k of Object.keys(colArrays)) colArrays[k].push(row[k]);
|
||||
}
|
||||
const buf = tableToIPC(tableFromArrays(colArrays), 'stream');
|
||||
res.setHeader('Content-Length', String(buf.byteLength));
|
||||
res.end(Buffer.from(buf.buffer, buf.byteOffset, buf.byteLength));
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
if (!res.headersSent) res.status(err.status || 500).json({ error: err.message });
|
||||
else res.destroy();
|
||||
} finally {
|
||||
if (client) {
|
||||
if (!committed) try { await client.query('ROLLBACK'); } catch {}
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// load baseline rows from source table — additive, no delete
|
||||
router.post('/versions/:id/baseline', requireAdmin, async (req, res) => {
|
||||
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
router.post('/versions/:id/baseline', async (req, res) => {
|
||||
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
|
||||
const dateOffset = date_offset || '0 days';
|
||||
if (!await assertInterval(dateOffset, res)) return;
|
||||
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
|
||||
const names = loadNames(label, bucket, res);
|
||||
if (!names) return;
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'baseline');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
@ -687,16 +320,12 @@ module.exports = function(pool) {
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.label),
|
||||
bucket: esc(names.bucket),
|
||||
tag: esc(tag || ''),
|
||||
params: esc(paramsJson),
|
||||
filter_clause: filterClause,
|
||||
date_offset: esc(dateOffset)
|
||||
});
|
||||
|
||||
const result = await runSQL(sql);
|
||||
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
@ -707,13 +336,11 @@ module.exports = function(pool) {
|
||||
// edit a baseline or reference segment in place — only allowed before any
|
||||
// scale/recode/clone has been applied on this version, since those would
|
||||
// have been calibrated against the old segment's totals.
|
||||
router.put('/versions/:id/baseline/:logid', requireAdmin, async (req, res) => {
|
||||
router.put('/versions/:id/baseline/:logid', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const logid = parseInt(req.params.logid);
|
||||
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
|
||||
const dateOffset = date_offset || '0 days';
|
||||
if (!await assertInterval(dateOffset, res)) return;
|
||||
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
|
||||
|
||||
const client = await pool.connect();
|
||||
@ -749,23 +376,11 @@ module.exports = function(pool) {
|
||||
date_offset: dateOffset,
|
||||
...(raw_where ? { raw_where } : (filters ? { filters } : {}))
|
||||
});
|
||||
// This route deletes the log row and inserts a fresh one, so every
|
||||
// annotation on it has to be handed back or it is lost. `??`, not `||`:
|
||||
// an empty string is the form clearing a field on purpose, undefined is
|
||||
// the form not carrying it at all -- the segment form has no tag input,
|
||||
// so tag is always the latter and must survive an edit made for any
|
||||
// other reason.
|
||||
const keep = (sent, prior) => esc(sent ?? prior ?? '');
|
||||
const names = loadNames(label ?? oldLog.label, bucket ?? oldLog.bucket, res);
|
||||
if (!names) return;
|
||||
const sql = applyTokens(ctx.sql, {
|
||||
fc_table: ctx.table,
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.label),
|
||||
bucket: esc(names.bucket),
|
||||
tag: keep(tag, oldLog.tag),
|
||||
params: esc(paramsJson),
|
||||
filter_clause: filterClause,
|
||||
date_offset: esc(dateOffset)
|
||||
@ -779,7 +394,6 @@ module.exports = function(pool) {
|
||||
await client.query(`DELETE FROM pf.log WHERE id = $1`, [logid]);
|
||||
const insResult = await client.query(sql);
|
||||
await client.query('COMMIT');
|
||||
await stampLogTotals(pool, ctx, insResult.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
|
||||
|
||||
res.json({
|
||||
rows_deleted: delRows.rowCount,
|
||||
@ -796,7 +410,7 @@ module.exports = function(pool) {
|
||||
});
|
||||
|
||||
// delete all baseline rows and log entries for a version
|
||||
router.delete('/versions/:id/baseline', requireAdmin, async (req, res) => {
|
||||
router.delete('/versions/:id/baseline', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
try {
|
||||
const ctx = await getContext(versionId, 'baseline');
|
||||
@ -830,13 +444,10 @@ module.exports = function(pool) {
|
||||
});
|
||||
|
||||
// load reference rows from source table (additive — does not clear prior reference rows)
|
||||
router.post('/versions/:id/reference', requireAdmin, async (req, res) => {
|
||||
const { where_clause, date_offset, note, filters, raw_where, label, bucket, tag } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
router.post('/versions/:id/reference', async (req, res) => {
|
||||
const { where_clause, date_offset, pf_user, note, filters, raw_where } = req.body;
|
||||
const dateOffset = date_offset || '0 days';
|
||||
const filterClause = (raw_where || where_clause || '').trim() || 'TRUE';
|
||||
const names = loadNames(label, bucket, res);
|
||||
if (!names) return;
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'reference');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
@ -850,16 +461,12 @@ module.exports = function(pool) {
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.label),
|
||||
bucket: esc(names.bucket),
|
||||
tag: esc(tag || ''),
|
||||
params: esc(paramsJson),
|
||||
filter_clause: filterClause,
|
||||
date_offset: esc(dateOffset)
|
||||
});
|
||||
|
||||
const result = await runSQL(sql);
|
||||
await stampLogTotals(pool, ctx, result.rows[0]?.log_id, { sql, territory: sessionTerritory(req) });
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
@ -871,8 +478,7 @@ module.exports = function(pool) {
|
||||
// target or by an increment. With several slices selected, apply_mode decides
|
||||
// whether they are treated as one pool ('prorate') or independently ('each').
|
||||
router.post('/versions/:id/scale', async (req, res) => {
|
||||
const { note, apply_mode } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
const { pf_user, note, apply_mode } = req.body;
|
||||
const slices = normalizeSlices(req.body);
|
||||
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
|
||||
|
||||
@ -881,7 +487,7 @@ module.exports = function(pool) {
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'scale');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
assertSelective(slices, ctx, req.body.scope);
|
||||
assertSelective(slices, ctx);
|
||||
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
|
||||
@ -889,23 +495,22 @@ module.exports = function(pool) {
|
||||
// sum() OVER () distribute the increment across the whole pool.
|
||||
// 'each' runs the same statement once per slice, so every slice
|
||||
// reaches the target on its own and gets its own log entry.
|
||||
const units = sliceUnits(slices, ctx, applyMode, req.body.scope, req);
|
||||
const units = applyMode === 'each'
|
||||
? slices.map(sl => ({ slices: [sl], where: buildWhere(sl, ctx.filterCols) }))
|
||||
: [{ slices, where: buildWhereAny(slices, ctx.filterCols) }];
|
||||
|
||||
const names = adjustmentNames(ctx.version);
|
||||
const client = await pool.connect();
|
||||
let committed = false;
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const allRows = [];
|
||||
// the statement that produced each entry, for pf.log.sql_text
|
||||
const sqlByLogId = new Map();
|
||||
let applied = 0;
|
||||
const skipped = [];
|
||||
|
||||
for (const unit of units) {
|
||||
const incr = await resolveIncrs(client, ctx, unit.where, excludeClause, req.body);
|
||||
// no rows, or already at the target — nothing to write for this slice
|
||||
if (incr.value === 0 && incr.units === 0) { for (const s of unit.slices) skipped.push(s); continue; }
|
||||
if (incr.value === 0 && incr.units === 0) { skipped.push(...unit.slices); continue; }
|
||||
applied++;
|
||||
|
||||
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
|
||||
@ -914,13 +519,11 @@ module.exports = function(pool) {
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.pf_segment),
|
||||
bucket: esc(names.pf_bucket),
|
||||
params: esc(JSON.stringify({
|
||||
slices: unit.slices,
|
||||
apply_mode: applyMode,
|
||||
...pickIntent(req.body),
|
||||
resolved: { value_incr: incr.value, units_incr: incr.units, method: incr.method }
|
||||
resolved: { value_incr: incr.value, units_incr: incr.units }
|
||||
})),
|
||||
slice: esc(JSON.stringify(loggedSlice)),
|
||||
where_clause: unit.where,
|
||||
@ -930,10 +533,7 @@ module.exports = function(pool) {
|
||||
});
|
||||
const result = await runSQL(sql, client);
|
||||
await tagLog(client, result.rows, req.body.tag);
|
||||
for (const r of result.rows) {
|
||||
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
|
||||
}
|
||||
for (const r of result.rows) allRows.push(r);
|
||||
allRows.push(...result.rows);
|
||||
}
|
||||
|
||||
if (allRows.length === 0) {
|
||||
@ -945,15 +545,7 @@ module.exports = function(pool) {
|
||||
|
||||
await client.query('COMMIT');
|
||||
committed = true;
|
||||
// one log id per unit: apply_mode 'each' writes an entry per slice
|
||||
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
|
||||
await stampLogTotals(pool, ctx, id, {
|
||||
sql: sqlByLogId.get(id) || null,
|
||||
territory: sessionTerritory(req),
|
||||
});
|
||||
}
|
||||
const opLabel = (req.body.tag || '').trim() || note || null;
|
||||
const rows = allRows.map(r => ({ ...r, ...names, pf_note: opLabel, pf_op: 'scale' }));
|
||||
const rows = allRows.map(r => ({ ...r, pf_note: note || null, pf_op: 'scale' }));
|
||||
res.json({
|
||||
rows,
|
||||
rows_affected: rows.length,
|
||||
@ -972,54 +564,26 @@ module.exports = function(pool) {
|
||||
|
||||
// recode dimension values on one or more slices
|
||||
// inserts negative rows to zero out the original, positive rows with new dimension values
|
||||
//
|
||||
// date_offset moves the positive rows in time as well -- the negative leg stays
|
||||
// where the rows were, so the pair is a timing move (pull forward, push out)
|
||||
// rather than a copy. With an offset, set may be empty: moving a slice without
|
||||
// renaming anything is the whole request.
|
||||
router.post('/versions/:id/recode', async (req, res) => {
|
||||
const { note, set, apply_mode, date_offset } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
const { pf_user, note, set, apply_mode } = req.body;
|
||||
const slices = normalizeSlices(req.body);
|
||||
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
|
||||
const dateOffset = String(date_offset ?? '').trim() || '0 days';
|
||||
if (!await assertInterval(dateOffset, res)) return;
|
||||
const shifted = (await pool.query(`SELECT $1::interval <> '0'::interval AS s`, [dateOffset])).rows[0].s;
|
||||
if ((!set || Object.keys(set).length === 0) && !shifted) {
|
||||
return res.status(400).json({ error: 'Recode needs a new value or a date offset' });
|
||||
}
|
||||
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
|
||||
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'recode');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
assertSelective(slices, ctx, req.body.scope);
|
||||
if (!assertMayRecodeTerritory(req, ctx, set, res)) return;
|
||||
assertSelective(slices, ctx);
|
||||
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
const setClause = buildSetClause(ctx.dimCols, set);
|
||||
const units = sliceUnits(slices, ctx, apply_mode);
|
||||
|
||||
// Moved rows take their period dimensions from the calendar against the
|
||||
// shifted date, as clone does; an explicit set still wins. A row whose
|
||||
// date is null has nothing to shift and keeps the label it had, rather
|
||||
// than having it blanked by a calendar lookup that found nothing. Not
|
||||
// derived at all without an offset, so a plain recode copies them as
|
||||
// it always has.
|
||||
const dateGroups = dateGroupsOf(ctx.colMeta);
|
||||
const dateColOf = Object.fromEntries(dateGroups.map(g => [g.alias, g.dateCol]));
|
||||
const derivedExprs = shifted ? Object.fromEntries(
|
||||
[...dimPeriodMapOf(dateGroups)].map(([cname, { alias, periodCol }]) =>
|
||||
[cname, `CASE WHEN s."${dateColOf[alias]}" IS NULL THEN s."${cname}" ELSE ${alias}."${periodCol}" END`])
|
||||
) : undefined;
|
||||
const setClause = buildSetClause(ctx.dimCols, set, { derivedExprs, alias: 's' });
|
||||
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
|
||||
|
||||
const names = adjustmentNames(ctx.version);
|
||||
const client = await pool.connect();
|
||||
let committed = false;
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const allRows = [];
|
||||
// the statement that produced each entry, for pf.log.sql_text
|
||||
const sqlByLogId = new Map();
|
||||
for (const unit of units) {
|
||||
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
|
||||
const sql = applyTokens(ctx.sql, {
|
||||
@ -1027,43 +591,19 @@ module.exports = function(pool) {
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.pf_segment),
|
||||
bucket: esc(names.pf_bucket),
|
||||
params: esc(JSON.stringify({ slices: unit.slices, set, apply_mode: unit.mode, date_offset: dateOffset })),
|
||||
params: esc(JSON.stringify({ slices: unit.slices, set, apply_mode: unit.mode })),
|
||||
slice: esc(JSON.stringify(loggedSlice)),
|
||||
where_clause: unit.where,
|
||||
exclude_clause: excludeClause,
|
||||
set_clause: setClause,
|
||||
date_offset: esc(dateOffset)
|
||||
set_clause: setClause
|
||||
});
|
||||
const result = await runSQL(sql, client);
|
||||
await tagLog(client, result.rows, req.body.tag);
|
||||
for (const r of result.rows) {
|
||||
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
|
||||
}
|
||||
for (const r of result.rows) allRows.push(r);
|
||||
allRows.push(...result.rows);
|
||||
}
|
||||
// Every coordinate in the selection nets to zero -- nothing to move.
|
||||
// Refused like scale's no-op rather than committing an empty log
|
||||
// entry that reads as a recode that happened.
|
||||
if (allRows.length === 0) {
|
||||
await client.query('ROLLBACK');
|
||||
return res.status(400).json({
|
||||
error: 'Nothing to recode — the selection nets to zero, or holds only rows recode cannot touch'
|
||||
});
|
||||
}
|
||||
|
||||
await client.query('COMMIT');
|
||||
committed = true;
|
||||
// one log id per unit: apply_mode 'each' writes an entry per slice
|
||||
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
|
||||
await stampLogTotals(pool, ctx, id, {
|
||||
sql: sqlByLogId.get(id) || null,
|
||||
territory: sessionTerritory(req),
|
||||
});
|
||||
}
|
||||
const opLabel = (req.body.tag || '').trim() || note || null;
|
||||
const rows = allRows.map(r => ({ ...r, ...names, pf_note: opLabel, pf_op: 'recode' }));
|
||||
const rows = allRows.map(r => ({ ...r, pf_note: note || null, pf_op: 'recode' }));
|
||||
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
|
||||
} finally {
|
||||
if (!committed) try { await client.query('ROLLBACK'); } catch {}
|
||||
@ -1078,63 +618,26 @@ module.exports = function(pool) {
|
||||
// clone one or more slices as new business under new dimension values
|
||||
// does not offset the original slice
|
||||
router.post('/versions/:id/clone', async (req, res) => {
|
||||
const { note, set, scale, apply_mode, from_logid, date_offset } = req.body;
|
||||
const pf_user = sessionUser(req);
|
||||
const { pf_user, note, set, scale, apply_mode } = req.body;
|
||||
const slices = normalizeSlices(req.body);
|
||||
if (slices.length === 0) return res.status(400).json({ error: 'slice is required' });
|
||||
if (!set || Object.keys(set).length === 0) return res.status(400).json({ error: 'set is required' });
|
||||
|
||||
try {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'clone');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
assertSelective(slices, ctx, req.body.scope);
|
||||
assertSelective(slices, ctx);
|
||||
|
||||
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
|
||||
const dateOffset = (date_offset || '0 days').trim() || '0 days';
|
||||
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
const setClause = buildSetClause(ctx.dimCols, set);
|
||||
const units = sliceUnits(slices, ctx, apply_mode);
|
||||
|
||||
if (!await assertInterval(dateOffset, res)) return;
|
||||
|
||||
// exclude_iters deliberately does not apply here. It exists to stop
|
||||
// operations *modifying* reference rows: scale would attribute forecast
|
||||
// movement to prior-year rows by distributing across them, and recode
|
||||
// writes negative rows that zero the original out. Clone does neither --
|
||||
// it reads rows and writes new pf_iter = 'clone' rows, leaving the source
|
||||
// untouched. Copying a plan or a prior year out of reference and into
|
||||
// adjustments is the operation working as intended, and excluding them
|
||||
// meant a visible, deliberate selection silently produced nothing.
|
||||
//
|
||||
// from_logid narrows instead: a selection spanning AOP and Prior Year
|
||||
// where only one is wanted.
|
||||
let excludeClause = '';
|
||||
if (from_logid != null) {
|
||||
const srcLog = await pool.query(
|
||||
`SELECT id FROM pf.log WHERE id = $1 AND version_id = $2`,
|
||||
[parseInt(from_logid), ctx.version.id]
|
||||
);
|
||||
if (!srcLog.rows.length) {
|
||||
return res.status(400).json({ error: `No log entry ${from_logid} on this version` });
|
||||
}
|
||||
excludeClause = `AND pf_logid = ${parseInt(from_logid)}`;
|
||||
}
|
||||
|
||||
// Period dimensions come from the calendar against the shifted date, not
|
||||
// from the row being copied -- otherwise a mix moved forward a year
|
||||
// keeps last year's period labels. An explicit set wins over both.
|
||||
const dateGroups = dateGroupsOf(ctx.colMeta);
|
||||
const derivedExprs = Object.fromEntries(
|
||||
[...dimPeriodMapOf(dateGroups)].map(([cname, { alias, periodCol }]) =>
|
||||
[cname, `${alias}."${periodCol}"`])
|
||||
);
|
||||
const setClause = buildSetClause(ctx.dimCols, set, { derivedExprs, alias: 's' });
|
||||
const units = sliceUnits(slices, ctx, apply_mode === 'each' ? 'each' : 'prorate', req.body.scope, req);
|
||||
|
||||
const names = adjustmentNames(ctx.version);
|
||||
const client = await pool.connect();
|
||||
let committed = false;
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const allRows = [];
|
||||
// the statement that produced each entry, for pf.log.sql_text
|
||||
const sqlByLogId = new Map();
|
||||
for (const unit of units) {
|
||||
const loggedSlice = unit.slices.length === 1 ? unit.slices[0] : unit.slices;
|
||||
const sql = applyTokens(ctx.sql, {
|
||||
@ -1142,38 +645,20 @@ module.exports = function(pool) {
|
||||
version_id: ctx.version.id,
|
||||
pf_user: esc(pf_user || ''),
|
||||
note: esc(note || ''),
|
||||
label: esc(names.pf_segment),
|
||||
bucket: esc(names.pf_bucket),
|
||||
params: esc(JSON.stringify({
|
||||
slices: unit.slices, set, scale: scaleFactor, apply_mode: unit.mode,
|
||||
date_offset: dateOffset,
|
||||
...(from_logid != null ? { from_logid: parseInt(from_logid) } : {}),
|
||||
})),
|
||||
params: esc(JSON.stringify({ slices: unit.slices, set, scale: scaleFactor, apply_mode: unit.mode })),
|
||||
slice: esc(JSON.stringify(loggedSlice)),
|
||||
where_clause: unit.where,
|
||||
exclude_clause: excludeClause,
|
||||
set_clause: setClause,
|
||||
scale_factor: scaleFactor,
|
||||
date_offset: esc(dateOffset)
|
||||
scale_factor: scaleFactor
|
||||
});
|
||||
const result = await runSQL(sql, client);
|
||||
await tagLog(client, result.rows, req.body.tag);
|
||||
for (const r of result.rows) {
|
||||
if (r.pf_logid != null) sqlByLogId.set(r.pf_logid, sql);
|
||||
}
|
||||
for (const r of result.rows) allRows.push(r);
|
||||
allRows.push(...result.rows);
|
||||
}
|
||||
await client.query('COMMIT');
|
||||
committed = true;
|
||||
// one log id per unit: apply_mode 'each' writes an entry per slice
|
||||
for (const id of new Set(allRows.map(r => r.pf_logid).filter(Boolean))) {
|
||||
await stampLogTotals(pool, ctx, id, {
|
||||
sql: sqlByLogId.get(id) || null,
|
||||
territory: sessionTerritory(req),
|
||||
});
|
||||
}
|
||||
const opLabel = (req.body.tag || '').trim() || note || null;
|
||||
const rows = allRows.map(r => ({ ...r, ...names, pf_note: opLabel, pf_op: 'clone' }));
|
||||
const rows = allRows.map(r => ({ ...r, pf_note: note || null, pf_op: 'clone' }));
|
||||
res.json({ rows, rows_affected: rows.length, slices_applied: units.length });
|
||||
} finally {
|
||||
if (!committed) try { await client.query('ROLLBACK'); } catch {}
|
||||
|
||||
125
routes/plans.js
125
routes/plans.js
@ -1,125 +0,0 @@
|
||||
const express = require('express');
|
||||
const { sessionUser, requireAdmin } = require('../lib/auth');
|
||||
|
||||
// Planned load segments — a version's definition, before it has been run.
|
||||
//
|
||||
// A plan carries exactly what POST /versions/:id/{baseline|reference} takes, so
|
||||
// running one is that same call with the plan's fields; the client then stamps
|
||||
// run_logid. It is two round trips rather than one because the alternative is a
|
||||
// second copy of the load route's token substitution, and a plan that reads
|
||||
// unrun while its rows exist is visible and fixable, where a divergent copy of
|
||||
// the load SQL is neither.
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
const FIELDS = ['seq', 'kind', 'label', 'bucket', 'tag', 'note',
|
||||
'filters', 'raw_where', 'date_offset'];
|
||||
|
||||
// A plan whose log entry has been undone is a plan again: the FK is
|
||||
// ON DELETE SET NULL, so run_logid clears itself. The join is for the
|
||||
// row count, which is what makes "already run" worth seeing.
|
||||
const SELECT_PLANS = `
|
||||
SELECT
|
||||
p.*
|
||||
,l.row_count AS run_rows
|
||||
,l.stamp AS run_at
|
||||
FROM pf.load_plan p
|
||||
LEFT JOIN pf.log l
|
||||
ON l.id = p.run_logid
|
||||
WHERE TRUE
|
||||
AND p.version_id = $1
|
||||
ORDER BY
|
||||
p.seq
|
||||
,p.id`;
|
||||
|
||||
router.get('/versions/:id/plans', async (req, res) => {
|
||||
try {
|
||||
const { rows } = await pool.query(SELECT_PLANS, [parseInt(req.params.id)]);
|
||||
res.json(rows);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
router.post('/versions/:id/plans', requireAdmin, async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const b = req.body || {};
|
||||
if (!['baseline', 'reference'].includes(b.kind)) {
|
||||
return res.status(400).json({ error: 'kind must be baseline or reference' });
|
||||
}
|
||||
try {
|
||||
// Append by default: a new plan runs after the ones already there.
|
||||
const next = await pool.query(
|
||||
`SELECT COALESCE(max(seq), -1) + 1 AS seq FROM pf.load_plan WHERE version_id = $1`,
|
||||
[versionId]
|
||||
);
|
||||
const { rows } = await pool.query(
|
||||
`INSERT INTO pf.load_plan
|
||||
(version_id, seq, kind, label, bucket, tag, note, filters, raw_where,
|
||||
date_offset, created_by)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, COALESCE($10, '0 days'), $11)
|
||||
RETURNING *`,
|
||||
[versionId, b.seq ?? next.rows[0].seq, b.kind,
|
||||
b.label || null, b.bucket || null, b.tag || null, b.note || null,
|
||||
b.filters ? JSON.stringify(b.filters) : null, b.raw_where || null,
|
||||
b.date_offset || null, sessionUser(req)]
|
||||
);
|
||||
res.status(201).json(rows[0]);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Any subset of the fields, plus run_logid when a run has just happened.
|
||||
// The flag-and-value pair rather than COALESCE on the value: clearing a
|
||||
// label to blank has to be distinguishable from not mentioning it.
|
||||
router.patch('/plans/:pid', requireAdmin, async (req, res) => {
|
||||
const pid = parseInt(req.params.pid);
|
||||
const b = req.body || {};
|
||||
const keys = [...FIELDS, 'run_logid'].filter(k => b[k] !== undefined);
|
||||
if (!keys.length) return res.status(400).json({ error: 'Nothing to update' });
|
||||
if (b.kind !== undefined && !['baseline', 'reference'].includes(b.kind)) {
|
||||
return res.status(400).json({ error: 'kind must be baseline or reference' });
|
||||
}
|
||||
try {
|
||||
const sets = [];
|
||||
const params = [pid];
|
||||
for (const k of keys) {
|
||||
let v = k === 'filters'
|
||||
? (b[k] === null ? null : JSON.stringify(b[k]))
|
||||
: (typeof b[k] === 'string' ? (b[k].trim() || null) : b[k]);
|
||||
// date_offset is NOT NULL, so a blank one means the default.
|
||||
// Coerced here rather than in the statement: a second assignment
|
||||
// to the same column is an error, not a fallback.
|
||||
if (k === 'date_offset' && !v) v = '0 days';
|
||||
params.push(v);
|
||||
sets.push(`${k} = $${params.length}${k === 'filters' ? '::jsonb' : ''}`);
|
||||
}
|
||||
const { rows } = await pool.query(
|
||||
`UPDATE pf.load_plan
|
||||
SET ${sets.join(', ')}, updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING *`,
|
||||
params
|
||||
);
|
||||
if (!rows.length) return res.status(404).json({ error: 'Plan not found' });
|
||||
res.json(rows[0]);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
router.delete('/plans/:pid', requireAdmin, async (req, res) => {
|
||||
try {
|
||||
const { rows } = await pool.query(
|
||||
`DELETE FROM pf.load_plan WHERE id = $1 RETURNING id`, [parseInt(req.params.pid)]
|
||||
);
|
||||
if (!rows.length) return res.status(404).json({ error: 'Plan not found' });
|
||||
res.json({ deleted: rows[0].id });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
return router;
|
||||
};
|
||||
@ -1,8 +1,5 @@
|
||||
const express = require('express');
|
||||
const { generateSQL, buildTerritoryClause } = require('../lib/sql_generator');
|
||||
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
|
||||
const { sessionTerritory, requireAdmin } = require('../lib/auth');
|
||||
const { sessionUser } = require('../lib/auth');
|
||||
const { generateSQL } = require('../lib/sql_generator');
|
||||
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
@ -22,9 +19,8 @@ module.exports = function(pool) {
|
||||
|
||||
// register a source table
|
||||
// auto-populates col_meta from information_schema with role='ignore'
|
||||
router.post('/sources', requireAdmin, async (req, res) => {
|
||||
const { schema, tname, label } = req.body;
|
||||
const created_by = sessionUser(req);
|
||||
router.post('/sources', async (req, res) => {
|
||||
const { schema, tname, label, created_by } = req.body;
|
||||
if (!schema || !tname) {
|
||||
return res.status(400).json({ error: 'schema and tname are required' });
|
||||
}
|
||||
@ -43,14 +39,15 @@ module.exports = function(pool) {
|
||||
);
|
||||
const source = src.rows[0];
|
||||
|
||||
// seed col_meta from the source's real columns
|
||||
// seed col_meta from information_schema
|
||||
await client.query(`
|
||||
INSERT INTO pf.col_meta (source_id, cname, role, opos)
|
||||
SELECT $3, column_name, 'dimension', ordinal_position
|
||||
FROM (${RELATION_COLUMNS_SQL}) c
|
||||
SELECT $1, column_name, 'dimension', ordinal_position
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = $2 AND table_name = $3
|
||||
ORDER BY ordinal_position
|
||||
ON CONFLICT (source_id, cname) DO NOTHING
|
||||
`, [schema, tname, source.id]);
|
||||
`, [source.id, schema, tname]);
|
||||
|
||||
await client.query('COMMIT');
|
||||
res.status(201).json(source);
|
||||
@ -81,37 +78,25 @@ module.exports = function(pool) {
|
||||
});
|
||||
|
||||
// save col_meta — accepts full array, upserts each row
|
||||
router.put('/sources/:id/cols', requireAdmin, async (req, res) => {
|
||||
router.put('/sources/:id/cols', async (req, res) => {
|
||||
const sourceId = parseInt(req.params.id);
|
||||
const cols = req.body;
|
||||
if (!Array.isArray(cols)) {
|
||||
return res.status(400).json({ error: 'body must be an array' });
|
||||
}
|
||||
// Exactly one per source: the scope is a single IN list against a single
|
||||
// column, and two flagged would silently mean whichever one a .find()
|
||||
// reached first -- the trap is_key already fell into (see CLAUDE.md).
|
||||
const territoryCols = cols.filter(c => c.is_territory).map(c => c.cname);
|
||||
if (territoryCols.length > 1) {
|
||||
return res.status(400).json({
|
||||
error: `Only one column can be the territory. Flagged: ${territoryCols.join(', ')}`
|
||||
});
|
||||
}
|
||||
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
for (const col of cols) {
|
||||
await client.query(`
|
||||
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, in_grain, is_territory, opos)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
|
||||
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, opos)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
ON CONFLICT (source_id, cname) DO UPDATE SET
|
||||
label = EXCLUDED.label,
|
||||
role = EXCLUDED.role,
|
||||
is_key = EXCLUDED.is_key,
|
||||
dim_group = EXCLUDED.dim_group,
|
||||
dim_period_col = EXCLUDED.dim_period_col,
|
||||
in_grain = EXCLUDED.in_grain,
|
||||
is_territory = EXCLUDED.is_territory,
|
||||
opos = EXCLUDED.opos
|
||||
`, [
|
||||
sourceId,
|
||||
@ -121,8 +106,6 @@ module.exports = function(pool) {
|
||||
col.is_key || false,
|
||||
col.dim_group || null,
|
||||
col.dim_period_col || null,
|
||||
col.in_grain || false,
|
||||
col.is_territory || false,
|
||||
col.opos || null
|
||||
]);
|
||||
}
|
||||
@ -142,7 +125,7 @@ module.exports = function(pool) {
|
||||
});
|
||||
|
||||
// generate SQL for all operations from current col_meta and store in pf.sql
|
||||
router.post('/sources/:id/generate-sql', requireAdmin, async (req, res) => {
|
||||
router.post('/sources/:id/generate-sql', async (req, res) => {
|
||||
const sourceId = parseInt(req.params.id);
|
||||
try {
|
||||
const srcResult = await pool.query(
|
||||
@ -183,13 +166,6 @@ module.exports = function(pool) {
|
||||
generated_at = EXCLUDED.generated_at
|
||||
`, [sourceId, operation, sql]);
|
||||
}
|
||||
// drop operations this generation no longer produces — e.g. get_agg
|
||||
// after the grain has been cleared, which would otherwise leave a
|
||||
// stale template the load path would still pick up
|
||||
await client.query(
|
||||
`DELETE FROM pf.sql WHERE source_id = $1 AND operation <> ALL($2::text[])`,
|
||||
[sourceId, Object.keys(sqls)]
|
||||
);
|
||||
await client.query('COMMIT');
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK');
|
||||
@ -238,38 +214,10 @@ module.exports = function(pool) {
|
||||
return res.status(400).json({ error: `"${col}" is not a key column` });
|
||||
}
|
||||
|
||||
// ?q= narrows, ?limit= caps. A key column can be very wide -- part on
|
||||
// osm_skinny has 11,290 distinct values -- so returning the lot to fill a
|
||||
// completion list is both a slow query and a large response for a control
|
||||
// that can only usefully show a handful.
|
||||
const { schema, tname } = srcResult.rows[0];
|
||||
const q = (req.query.q || '').trim();
|
||||
const limit = Math.min(parseInt(req.query.limit) || 5000, 5000);
|
||||
|
||||
const params = [];
|
||||
let filter = `WHERE "${col}" IS NOT NULL`;
|
||||
|
||||
// Completion reads the *source* table, which no territory scope has
|
||||
// touched -- so without this a scoped account could enumerate every
|
||||
// customer, part and rep in the business from a dropdown, having
|
||||
// been shown none of their rows.
|
||||
const terrRow = (await pool.query(
|
||||
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
|
||||
[req.params.id]
|
||||
)).rows[0];
|
||||
const terrClause = buildTerritoryClause(sessionTerritory(req), terrRow?.cname || null);
|
||||
if (terrClause) filter += ` AND ${terrClause}`;
|
||||
|
||||
if (q) {
|
||||
params.push(`%${q}%`);
|
||||
filter += ` AND "${col}"::text ILIKE $${params.length}`;
|
||||
}
|
||||
params.push(limit);
|
||||
|
||||
const result = await pool.query(
|
||||
`SELECT DISTINCT "${col}"::text AS val FROM ${schema}.${tname}
|
||||
${filter} ORDER BY 1 LIMIT $${params.length}`,
|
||||
params
|
||||
`SELECT DISTINCT "${col}" AS val FROM ${schema}.${tname}
|
||||
WHERE "${col}" IS NOT NULL ORDER BY "${col}"`
|
||||
);
|
||||
res.json(result.rows.map(r => r.val));
|
||||
} catch (err) {
|
||||
@ -278,142 +226,6 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
// Resolve a dim_group to its key column and siblings, or explain why it cannot be.
|
||||
async function resolveGroup(sourceId, group) {
|
||||
const { rows: meta } = await pool.query(
|
||||
`SELECT * FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`, [sourceId]);
|
||||
const members = meta.filter(c => c.dim_group === group);
|
||||
if (!members.length) {
|
||||
const err = new Error(`No columns are grouped as "${group}" on this source`);
|
||||
err.status = 404; throw err;
|
||||
}
|
||||
const keyCol = members.find(c => c.is_key);
|
||||
if (!keyCol) {
|
||||
const err = new Error(
|
||||
`Group "${group}" has no is_key column, so its members have nothing to be keyed on`);
|
||||
err.status = 400; throw err;
|
||||
}
|
||||
return {
|
||||
keyCol,
|
||||
siblings: members.filter(c => c.cname !== keyCol.cname),
|
||||
// recency column: the source's primary date, the same one the generator
|
||||
// treats as the date for loads
|
||||
dateCol: meta.find(c => c.role === 'date')?.cname || null,
|
||||
};
|
||||
}
|
||||
|
||||
// The member list for a group, as one array. Small enough to send whole --
|
||||
// 11,290 parts on osm_skinny -- so the client holds it and filters locally
|
||||
// instead of querying per keystroke.
|
||||
router.get('/sources/:id/dim/:group', async (req, res) => {
|
||||
try {
|
||||
const sourceId = parseInt(req.params.id);
|
||||
const { keyCol, siblings } = await resolveGroup(sourceId, req.params.group);
|
||||
const includeInactive = req.query.all === '1';
|
||||
|
||||
const { rows } = await pool.query(`
|
||||
SELECT key_value, attrs, is_active, source_seen
|
||||
FROM pf.dim_member
|
||||
WHERE source_id = $1 AND dim_group = $2
|
||||
${includeInactive ? '' : 'AND is_active'}
|
||||
ORDER BY key_value
|
||||
`, [sourceId, req.params.group]);
|
||||
|
||||
res.json({
|
||||
group: req.params.group,
|
||||
key_col: keyCol.cname,
|
||||
siblings: siblings.map(c => c.cname),
|
||||
members: rows,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(err.status || 500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Rebuild a group's members from the source. A merge, not a replace: curation
|
||||
// (a member deactivated by hand, or added before it ever sold) has to survive a
|
||||
// refresh, so absent members are marked source_seen = false rather than deleted.
|
||||
//
|
||||
// Slow by nature -- it reads the whole source, which for a view over a
|
||||
// transaction table is millions of rows -- so it is a deliberate action rather
|
||||
// than something that happens on a page load.
|
||||
router.post('/sources/:id/dim/:group/refresh', requireAdmin, async (req, res) => {
|
||||
const sourceId = parseInt(req.params.id);
|
||||
const group = req.params.group;
|
||||
try {
|
||||
const srcResult = await pool.query(
|
||||
`SELECT schema, tname FROM pf.source WHERE id = $1`, [sourceId]);
|
||||
if (!srcResult.rows.length) return res.status(404).json({ error: 'Source not found' });
|
||||
const { schema, tname } = srcResult.rows[0];
|
||||
|
||||
const { keyCol, siblings, dateCol } = await resolveGroup(sourceId, group);
|
||||
if (!siblings.length) {
|
||||
return res.status(400).json({ error: `Group "${group}" has no sibling columns to store` });
|
||||
}
|
||||
|
||||
const q = (n) => `"${n}"`;
|
||||
const attrs = siblings.map(c => `'${c.cname}', s.${q(c.cname)}::text`).join(', ');
|
||||
// A key can carry more than one attribute set across history -- 11,290
|
||||
// parts against 13,662 combinations on osm_skinny. Take the most recent
|
||||
// by the source's date column, which is the live definition.
|
||||
const recency = dateCol ? `s.${q(dateCol)} DESC NULLS LAST` : `1`;
|
||||
|
||||
const started = Date.now();
|
||||
// An explicit stamp rather than now(): inside a transaction now() is the
|
||||
// transaction's start time, so "refreshed in this run" and "refreshed in
|
||||
// a run that began at the same instant" would be indistinguishable.
|
||||
const runAt = new Date();
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const { rows: [{ n }] } = await client.query(`
|
||||
WITH ranked AS (
|
||||
SELECT s.${q(keyCol.cname)}::text AS key_value,
|
||||
jsonb_build_object(${attrs}) AS attrs,
|
||||
row_number() OVER (
|
||||
PARTITION BY s.${q(keyCol.cname)} ORDER BY ${recency}
|
||||
) AS rn
|
||||
FROM ${q(schema)}.${q(tname)} s
|
||||
WHERE s.${q(keyCol.cname)} IS NOT NULL
|
||||
)
|
||||
,upserted AS (
|
||||
INSERT INTO pf.dim_member
|
||||
(source_id, dim_group, key_value, attrs, refreshed_at, source_seen)
|
||||
SELECT $1, $2, key_value, attrs, $3, true
|
||||
FROM ranked WHERE rn = 1
|
||||
ON CONFLICT (source_id, dim_group, key_value) DO UPDATE SET
|
||||
attrs = EXCLUDED.attrs,
|
||||
source_seen = true,
|
||||
refreshed_at = $3,
|
||||
updated_at = now()
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT count(*)::int AS n FROM upserted
|
||||
`, [sourceId, group, runAt]);
|
||||
|
||||
// anything this run did not touch is no longer in the source
|
||||
const { rowCount: dropped } = await client.query(`
|
||||
UPDATE pf.dim_member
|
||||
SET source_seen = false, updated_at = now()
|
||||
WHERE source_id = $1 AND dim_group = $2 AND source_seen
|
||||
AND refreshed_at IS DISTINCT FROM $3
|
||||
`, [sourceId, group, runAt]);
|
||||
|
||||
await client.query('COMMIT');
|
||||
res.json({ group, members: n, no_longer_in_source: dropped, ms: Date.now() - started });
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK');
|
||||
throw err;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(err.status || 500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// given a key column value, look up sibling dim_group column values from source
|
||||
// returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous
|
||||
router.get('/sources/:id/lookup', async (req, res) => {
|
||||
@ -449,12 +261,23 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
// PUT /sources/:id/default-layout is gone. It wrote pf.source.default_layout,
|
||||
// one anonymous blob per source that any account could overwrite for every
|
||||
// other account -- a published layout with no owner. Its successor is
|
||||
// pf.layout: named, owned, and writable only by its owner or an admin. The
|
||||
// old column is left in place, already migrated into pf.layout by
|
||||
// setup_sql/01_schema.sql, and read by nothing.
|
||||
// set or clear the default Perspective layout for a source.
|
||||
// Body: a Perspective view config (group_by, split_by, columns, plugin_config, …).
|
||||
// Pass null or {} to clear.
|
||||
router.put('/sources/:id/default-layout', async (req, res) => {
|
||||
try {
|
||||
const layout = req.body && Object.keys(req.body).length > 0 ? req.body : null;
|
||||
const result = await pool.query(
|
||||
`UPDATE pf.source SET default_layout = $1 WHERE id = $2 RETURNING *`,
|
||||
[layout, req.params.id]
|
||||
);
|
||||
if (result.rows.length === 0) return res.status(404).json({ error: 'Source not found' });
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// deregister a source — does not drop existing forecast tables
|
||||
router.get('/dim-period/cols', async (req, res) => {
|
||||
@ -473,7 +296,7 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
router.delete('/sources/:id', requireAdmin, async (req, res) => {
|
||||
router.delete('/sources/:id', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(
|
||||
`DELETE FROM pf.source WHERE id = $1 RETURNING *`,
|
||||
|
||||
@ -1,9 +1,5 @@
|
||||
const express = require('express');
|
||||
const { RELATION_COLUMNS_SQL } = require('../lib/utils');
|
||||
|
||||
// Reads, deliberately open to everyone: Setup is visible read-only to
|
||||
// non-admins, so gating what the page needs in order to render turned it into
|
||||
// a 403 the client rendered as a crash. The writes it leads to are gated.
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
@ -11,18 +7,15 @@ module.exports = function(pool) {
|
||||
router.get('/tables', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
-- pg_class, not information_schema.tables, which omits
|
||||
-- materialized views: gs.osm_skinny is one, and the browser
|
||||
-- could not offer what the app is already built on.
|
||||
SELECT
|
||||
n.nspname AS schema,
|
||||
c.relname AS tname,
|
||||
t.table_schema AS schema,
|
||||
t.table_name AS tname,
|
||||
c.reltuples::bigint AS row_estimate
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE c.relkind IN ('r', 'v', 'm', 'f', 'p')
|
||||
AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pf')
|
||||
ORDER BY n.nspname, c.relname
|
||||
FROM information_schema.tables t
|
||||
LEFT JOIN pg_namespace n ON n.nspname = t.table_schema
|
||||
LEFT JOIN pg_class c ON c.relname = t.table_name AND c.relnamespace = n.oid
|
||||
WHERE t.table_schema NOT IN ('pg_catalog', 'information_schema', 'pf')
|
||||
ORDER BY t.table_schema, t.table_name
|
||||
`);
|
||||
res.json(result.rows);
|
||||
} catch (err) {
|
||||
@ -38,8 +31,12 @@ module.exports = function(pool) {
|
||||
return res.status(400).json({ error: 'Invalid schema or table name' });
|
||||
}
|
||||
try {
|
||||
const cols = await pool.query(
|
||||
`${RELATION_COLUMNS_SQL} ORDER BY ordinal_position`, [schema, tname]);
|
||||
const cols = await pool.query(`
|
||||
SELECT column_name, data_type, is_nullable, ordinal_position
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = $1 AND table_name = $2
|
||||
ORDER BY ordinal_position
|
||||
`, [schema, tname]);
|
||||
|
||||
const rows = await pool.query(
|
||||
`SELECT * FROM ${schema}.${tname} LIMIT 5`
|
||||
|
||||
@ -1,75 +1,5 @@
|
||||
const express = require('express');
|
||||
const { fcTable, mapType, RELATION_COLUMNS_SQL } = require('../lib/utils');
|
||||
const { sessionUser, sessionTerritory, requireAdmin } = require('../lib/auth');
|
||||
const { buildTerritoryClause } = require('../lib/sql_generator');
|
||||
|
||||
// Copy a version's definition onto a freshly created one: its load segments as
|
||||
// plans, and its saved layouts.
|
||||
//
|
||||
// The plans come from pf.load_plan when the template has them, and are derived
|
||||
// from its baseline/reference log entries when it does not — which is every
|
||||
// version created before plans existed, so a definition can be lifted out of
|
||||
// one that was built by hand. `params` on a load entry already stores the
|
||||
// filters, the raw WHERE when one was used, and the date offset, so the
|
||||
// derivation is a read rather than a reconstruction.
|
||||
async function cloneDefinition(client, template, version, created_by) {
|
||||
const existing = await client.query(
|
||||
`SELECT * FROM pf.load_plan WHERE version_id = $1 ORDER BY seq, id`,
|
||||
[template.id]
|
||||
);
|
||||
|
||||
let source = existing.rows.map(p => ({
|
||||
kind: p.kind, label: p.label, bucket: p.bucket, tag: p.tag, note: p.note,
|
||||
filters: p.filters, raw_where: p.raw_where, date_offset: p.date_offset
|
||||
}));
|
||||
|
||||
if (!source.length) {
|
||||
const log = await client.query(
|
||||
`SELECT operation, label, bucket, tag, note, params
|
||||
FROM pf.log
|
||||
WHERE TRUE
|
||||
AND version_id = $1
|
||||
AND operation IN ('baseline', 'reference')
|
||||
ORDER BY id`,
|
||||
[template.id]
|
||||
);
|
||||
source = log.rows.map(e => ({
|
||||
kind: e.operation,
|
||||
label: e.label, bucket: e.bucket, tag: e.tag, note: e.note,
|
||||
filters: e.params?.filters || null,
|
||||
raw_where: e.params?.raw_where || null,
|
||||
date_offset: e.params?.date_offset || '0 days'
|
||||
}));
|
||||
}
|
||||
|
||||
for (const [i, p] of source.entries()) {
|
||||
await client.query(
|
||||
`INSERT INTO pf.load_plan
|
||||
(version_id, seq, kind, label, bucket, tag, note, filters, raw_where,
|
||||
date_offset, created_by)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, COALESCE($10, '0 days'), $11)`,
|
||||
[version.id, i, p.kind, p.label, p.bucket, p.tag, p.note,
|
||||
p.filters ? JSON.stringify(p.filters) : null, p.raw_where,
|
||||
p.date_offset, created_by]
|
||||
);
|
||||
}
|
||||
|
||||
// Version-scoped layouts only. The source-wide ones (version_id IS NULL)
|
||||
// already apply to the new version, so copying them would duplicate them
|
||||
// under a name that now exists twice.
|
||||
const layouts = await client.query(
|
||||
`INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility, is_default)
|
||||
SELECT source_id, $2, name, config, owner, visibility, is_default
|
||||
FROM pf.layout
|
||||
WHERE TRUE
|
||||
AND version_id = $1
|
||||
RETURNING id`,
|
||||
[template.id, version.id]
|
||||
);
|
||||
|
||||
return { from: template.id, from_name: template.name,
|
||||
plans: source.length, layouts: layouts.rowCount };
|
||||
}
|
||||
const { fcTable, mapType } = require('../lib/utils');
|
||||
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
@ -90,10 +20,9 @@ module.exports = function(pool) {
|
||||
|
||||
// create a new version
|
||||
// inserts version row, then CREATE TABLE pf.fc_{tname}_{version_id} in one transaction
|
||||
router.post('/sources/:id/versions', requireAdmin, async (req, res) => {
|
||||
router.post('/sources/:id/versions', async (req, res) => {
|
||||
const sourceId = parseInt(req.params.id);
|
||||
const { name, description, exclude_iters, clone_from } = req.body;
|
||||
const created_by = sessionUser(req);
|
||||
const { name, description, created_by, exclude_iters } = req.body;
|
||||
if (!name) return res.status(400).json({ error: 'name is required' });
|
||||
|
||||
const client = await pool.connect();
|
||||
@ -107,9 +36,8 @@ module.exports = function(pool) {
|
||||
}
|
||||
const source = srcResult.rows[0];
|
||||
|
||||
// col_meta joined to the source's real columns, for the data types
|
||||
// fetch col_meta joined to information_schema for data types
|
||||
const colResult = await client.query(`
|
||||
WITH cols AS (${RELATION_COLUMNS_SQL})
|
||||
SELECT
|
||||
m.cname,
|
||||
m.role,
|
||||
@ -118,11 +46,14 @@ module.exports = function(pool) {
|
||||
i.numeric_precision,
|
||||
i.numeric_scale
|
||||
FROM pf.col_meta m
|
||||
JOIN cols i ON i.column_name = m.cname
|
||||
WHERE m.source_id = $3
|
||||
JOIN information_schema.columns i
|
||||
ON i.table_schema = $2
|
||||
AND i.table_name = $3
|
||||
AND i.column_name = m.cname
|
||||
WHERE m.source_id = $1
|
||||
AND m.role NOT IN ('ignore')
|
||||
ORDER BY m.opos
|
||||
`, [source.schema, source.tname, sourceId]);
|
||||
`, [sourceId, source.schema, source.tname]);
|
||||
|
||||
if (colResult.rows.length === 0) {
|
||||
return res.status(400).json({
|
||||
@ -130,52 +61,19 @@ module.exports = function(pool) {
|
||||
});
|
||||
}
|
||||
|
||||
// clone_from copies a version's *definition* -- its settings, its
|
||||
// saved layouts, and its load segments as plans -- and stops there.
|
||||
// The loads are deliberately not run: the reason to clone a
|
||||
// definition is to change the dates before it loads anything, and a
|
||||
// clone that fetched two million rows first would have to be undone
|
||||
// before it could be edited.
|
||||
//
|
||||
// Adjustments are not copied in any form. A scale's target is a
|
||||
// figure sized against the rows that were there, so replaying one
|
||||
// against a different load is arithmetic nobody asked for.
|
||||
let template = null;
|
||||
if (clone_from) {
|
||||
const t = await client.query(
|
||||
`SELECT * FROM pf.version WHERE id = $1 AND source_id = $2`,
|
||||
[parseInt(clone_from), sourceId]
|
||||
);
|
||||
if (!t.rows.length) {
|
||||
return res.status(404).json({
|
||||
error: 'Version to clone not found, or it belongs to a different source'
|
||||
});
|
||||
}
|
||||
template = t.rows[0];
|
||||
}
|
||||
|
||||
await client.query('BEGIN');
|
||||
|
||||
// insert version to get id
|
||||
const verResult = await client.query(`
|
||||
INSERT INTO pf.version (source_id, name, description, created_by, exclude_iters,
|
||||
adjustment_segment, adjustment_bucket, unlabeled_load,
|
||||
bucket_order)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
INSERT INTO pf.version (source_id, name, description, created_by, exclude_iters)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
RETURNING *
|
||||
`, [
|
||||
sourceId,
|
||||
name,
|
||||
// An explicit description wins; otherwise the template's is
|
||||
// carried, since it describes the definition being copied.
|
||||
description || template?.description || null,
|
||||
description || null,
|
||||
created_by || null,
|
||||
exclude_iters ? JSON.stringify(exclude_iters)
|
||||
: (template ? JSON.stringify(template.exclude_iters) : '["reference"]'),
|
||||
template?.adjustment_segment || null,
|
||||
template?.adjustment_bucket || null,
|
||||
template?.unlabeled_load || null,
|
||||
template?.bucket_order ? JSON.stringify(template.bucket_order) : null
|
||||
exclude_iters ? JSON.stringify(exclude_iters) : '["reference"]'
|
||||
]);
|
||||
const version = verResult.rows[0];
|
||||
|
||||
@ -202,19 +100,8 @@ ${colDefs},
|
||||
`;
|
||||
await client.query(ddl);
|
||||
|
||||
// pf_logid is how every entry-level operation finds its rows: undo
|
||||
// deletes by it, the change log aggregates by it, and it is part of the
|
||||
// grain key. Without an index each of those is a sequential scan of the
|
||||
// whole forecast table -- 2.5M rows to total two adjustments.
|
||||
await client.query(
|
||||
`CREATE INDEX ${table.split('.').pop()}_logid_idx ON ${table} (pf_logid)`
|
||||
);
|
||||
|
||||
let cloned = null;
|
||||
if (template) cloned = await cloneDefinition(client, template, version, created_by);
|
||||
|
||||
await client.query('COMMIT');
|
||||
res.status(201).json({ ...version, fc_table: table, cloned });
|
||||
res.status(201).json({ ...version, fc_table: table });
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK');
|
||||
console.error(err);
|
||||
@ -230,74 +117,10 @@ ${colDefs},
|
||||
// where this version's writes actually land: the physical forecast table,
|
||||
// its current row count, and the source table rows are read from.
|
||||
// Surfaced in the status bar so the write target is never a mystery.
|
||||
// Distinct values of a dimension as they appear in one version's forecast
|
||||
// table, for completing recode and clone.
|
||||
//
|
||||
// Deliberately not the source: the source view reaches back over all of
|
||||
// history, so completing from it offers parts discontinued years ago. The
|
||||
// version holds what was actually loaded, which is what a forecast is being
|
||||
// written against.
|
||||
//
|
||||
// Held in memory because the scan is not cheap -- 2.0s for 11,290 parts across
|
||||
// 2.5M rows on fc_osm_skinny_29 -- and completion is typed into. Keyed on the
|
||||
// version's latest log id, so any load, adjustment or undo rebuilds it on the
|
||||
// next request without anything having to remember to invalidate.
|
||||
const valueCache = new Map();
|
||||
|
||||
router.get('/versions/:id/values/:col', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const col = req.params.col;
|
||||
try {
|
||||
const verResult = await pool.query(`
|
||||
SELECT v.id, s.tname, s.id AS source_id
|
||||
FROM pf.version v JOIN pf.source s ON s.id = v.source_id
|
||||
WHERE v.id = $1
|
||||
`, [versionId]);
|
||||
if (!verResult.rows.length) return res.status(404).json({ error: 'Version not found' });
|
||||
const { tname, source_id } = verResult.rows[0];
|
||||
|
||||
// the column name is interpolated, so it has to be one col_meta names
|
||||
const okCol = await pool.query(`
|
||||
SELECT 1 FROM pf.col_meta
|
||||
WHERE source_id = $1 AND cname = $2 AND role IN ('dimension', 'date')
|
||||
`, [source_id, col]);
|
||||
if (!okCol.rows.length) {
|
||||
return res.status(400).json({ error: `"${col}" is not a dimension on this source` });
|
||||
}
|
||||
|
||||
const table = fcTable(tname, versionId);
|
||||
const { rows: [{ rev }] } = await pool.query(
|
||||
`SELECT coalesce(max(id), 0)::text AS rev FROM pf.log WHERE version_id = $1`,
|
||||
[versionId]
|
||||
);
|
||||
|
||||
const key = `${versionId}:${col}`;
|
||||
let entry = valueCache.get(key);
|
||||
if (!entry || entry.rev !== rev) {
|
||||
const { rows } = await pool.query(
|
||||
`SELECT DISTINCT "${col}"::text AS val FROM ${table}
|
||||
WHERE "${col}" IS NOT NULL ORDER BY 1`
|
||||
);
|
||||
entry = { rev, values: rows.map(r => r.val) };
|
||||
valueCache.set(key, entry);
|
||||
}
|
||||
|
||||
const q = (req.query.q || '').trim().toLowerCase();
|
||||
const limit = Math.min(parseInt(req.query.limit) || 50, 500);
|
||||
const picked = q
|
||||
? entry.values.filter(v => v.toLowerCase().includes(q))
|
||||
: entry.values;
|
||||
res.json(picked.slice(0, limit));
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(err.status || 500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
router.get('/versions/:id/table-info', async (req, res) => {
|
||||
try {
|
||||
const verResult = await pool.query(`
|
||||
SELECT v.id, v.name, v.status, s.schema, s.tname, s.id AS source_id
|
||||
SELECT v.id, v.name, v.status, s.schema, s.tname
|
||||
FROM pf.version v
|
||||
JOIN pf.source s ON s.id = v.source_id
|
||||
WHERE v.id = $1
|
||||
@ -313,23 +136,10 @@ ${colDefs},
|
||||
);
|
||||
const exists = existsResult.rows[0].exists;
|
||||
|
||||
// Scoped like every other read. Unscoped it reported the whole
|
||||
// table to an account that can see a fraction of it -- the status
|
||||
// bar's row count, and the figure the load progress promises, both
|
||||
// came from here: a rep with 330k rows was told the load was
|
||||
// fetching 2.8M.
|
||||
const terrCol = (await pool.query(
|
||||
`SELECT cname FROM pf.col_meta WHERE source_id = $1 AND is_territory LIMIT 1`,
|
||||
[v.source_id]
|
||||
)).rows[0]?.cname || null;
|
||||
const terr = buildTerritoryClause(sessionTerritory(req), terrCol);
|
||||
|
||||
let rows = null, byIter = [];
|
||||
if (exists) {
|
||||
const countResult = await pool.query(
|
||||
`SELECT pf_iter, count(*)::int AS n FROM ${fc}
|
||||
${terr ? `WHERE ${terr}` : ''}
|
||||
GROUP BY pf_iter ORDER BY pf_iter`
|
||||
`SELECT pf_iter, count(*)::int AS n FROM ${fc} GROUP BY pf_iter ORDER BY pf_iter`
|
||||
);
|
||||
byIter = countResult.rows;
|
||||
rows = byIter.reduce((a, r) => a + r.n, 0);
|
||||
@ -356,106 +166,25 @@ ${colDefs},
|
||||
// Tags already used on this source, newest first — feeds the tag autocomplete.
|
||||
// Scoped to the source rather than the version so an initiative name carries
|
||||
// across versions, which is the point of naming it.
|
||||
// Tags already used on one version, for completing the next adjustment.
|
||||
//
|
||||
// Scoped to the version, not the source. A tag names an initiative inside a
|
||||
// scenario -- what this plan is doing -- so offering every tag anyone has
|
||||
// ever used on the source put last year's work in the way of this year's,
|
||||
// and the `uses` count read as activity on a version it did not belong to.
|
||||
// The cost is that a fresh version completes from nothing until it has a
|
||||
// tag of its own, which is the honest state of it: cloning a definition
|
||||
// copies no adjustments, so it has no initiatives yet.
|
||||
router.get('/versions/:id/tags', async (req, res) => {
|
||||
router.get('/sources/:id/tags', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
SELECT l.tag,
|
||||
count(*)::int AS uses,
|
||||
max(l.stamp) AS last_used,
|
||||
array_agg(DISTINCT l.pf_user) AS owners
|
||||
count(*)::int AS uses,
|
||||
max(l.stamp) AS last_used
|
||||
FROM pf.log l
|
||||
WHERE TRUE
|
||||
AND l.version_id = $1
|
||||
AND l.tag IS NOT NULL
|
||||
AND l.tag <> ''
|
||||
JOIN pf.version v ON v.id = l.version_id
|
||||
WHERE v.source_id = $1 AND l.tag IS NOT NULL AND l.tag <> ''
|
||||
GROUP BY l.tag
|
||||
ORDER BY max(l.stamp) DESC
|
||||
`, [parseInt(req.params.id)]);
|
||||
|
||||
// Renaming a tag rewrites every entry that carries it, so the right
|
||||
// to do it is the right to edit all of them -- not most of them.
|
||||
// The tag manager greys the row out rather than offering a rename
|
||||
// that would half-apply.
|
||||
const me = sessionUser(req);
|
||||
const admin = !!req.session?.user?.is_admin;
|
||||
res.json(result.rows.map(r => ({
|
||||
...r,
|
||||
can_edit: admin || (r.owners || []).every(o => o === me)
|
||||
})));
|
||||
`, [req.params.id]);
|
||||
res.json(result.rows);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// Rename a tag across the whole version, or clear it. `to` empty means
|
||||
// clear; `to` naming a tag that already exists merges the two, which is the
|
||||
// point -- "Pull Forward" and "Pulled Forward" are one initiative reported
|
||||
// as two, and the bridge groups on this column.
|
||||
//
|
||||
// A PATCH on the version rather than on a tag: a tag is not a row anywhere,
|
||||
// only a value repeated across pf.log, so there is no /tags/:tag to address.
|
||||
router.patch('/versions/:id/tags', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const from = req.body.from === undefined ? null : String(req.body.from).trim();
|
||||
const to = req.body.to === undefined ? null : String(req.body.to).trim();
|
||||
if (!from) return res.status(400).json({ error: 'Which tag? Send `from`' });
|
||||
if (to === from) return res.status(400).json({ error: 'That is the same tag' });
|
||||
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
// Lock the rows before counting them, so the owner check and the
|
||||
// rewrite see the same set.
|
||||
const rows = await client.query(
|
||||
`SELECT id, pf_user FROM pf.log
|
||||
WHERE version_id = $1 AND tag = $2
|
||||
FOR UPDATE`,
|
||||
[versionId, from]
|
||||
);
|
||||
if (!rows.rows.length) {
|
||||
await client.query('ROLLBACK');
|
||||
return res.status(404).json({ error: `No entry on this version is tagged "${from}"` });
|
||||
}
|
||||
|
||||
// Same rule as undo and PATCH /log: your own entries, or an admin's
|
||||
// override. Applied to every entry carrying the tag, because a
|
||||
// partial rename leaves the initiative split in two.
|
||||
const me = sessionUser(req);
|
||||
if (!req.session?.user?.is_admin) {
|
||||
const others = [...new Set(rows.rows.map(r => r.pf_user).filter(u => u !== me))];
|
||||
if (others.length) {
|
||||
await client.query('ROLLBACK');
|
||||
return res.status(403).json({
|
||||
error: `"${from}" is also on entries made by ${others.join(', ')} — only they or an administrator can change those`
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const result = await client.query(
|
||||
`UPDATE pf.log SET tag = $3 WHERE version_id = $1 AND tag = $2`,
|
||||
[versionId, from, to || null]
|
||||
);
|
||||
await client.query('COMMIT');
|
||||
res.json({ from, to: to || null, rows_affected: result.rowCount });
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK').catch(() => {});
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
});
|
||||
|
||||
// Bridge: how this version got from its baseline to where it stands, grouped by
|
||||
// initiative. Amounts come from the version's own forecast table, so the figures
|
||||
// reconcile with the pivot rather than being recomputed from the log's params.
|
||||
@ -504,99 +233,36 @@ ${colDefs},
|
||||
}
|
||||
});
|
||||
|
||||
// update version name, description, exclude_iters, or the fallback display
|
||||
// names.
|
||||
//
|
||||
// bucket_order is deliberately not settable: the bucket column order is the
|
||||
// text in pf.log.bucket now, so a stored order would be a second answer to
|
||||
// the same question, and a silent one -- nothing reads it.
|
||||
//
|
||||
// The three name columns are flag-and-value pairs rather than COALESCE:
|
||||
// clearing one back to the built-in means writing null, which COALESCE on
|
||||
// the value alone cannot tell from "not mentioned".
|
||||
router.put('/versions/:id', requireAdmin, async (req, res) => {
|
||||
const { name, description, exclude_iters,
|
||||
adjustment_segment, adjustment_bucket, unlabeled_load,
|
||||
is_default } = req.body;
|
||||
const set = (v) => (v === undefined ? null : (String(v).trim() || null));
|
||||
|
||||
// is_default decides where every account without a saved selection
|
||||
// lands, so it is not the version's own setting the way its name is:
|
||||
// it follows the pf.log rule, an admin override and nothing else. The
|
||||
// UI greys the box out rather than offering a click that answers 403.
|
||||
if (is_default !== undefined && !req.session?.user?.is_admin) {
|
||||
return res.status(403).json({
|
||||
error: 'Only an admin can set the default forecast'
|
||||
});
|
||||
}
|
||||
|
||||
// The write is two statements when the default moves -- clear the old
|
||||
// one, set the new -- and the partial unique index refuses the moment
|
||||
// in between, so they share a transaction.
|
||||
const client = await pool.connect();
|
||||
// update version name, description, or exclude_iters
|
||||
router.put('/versions/:id', async (req, res) => {
|
||||
const { name, description, exclude_iters } = req.body;
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
if (is_default === true) {
|
||||
await client.query(`
|
||||
UPDATE pf.version SET is_default = false
|
||||
WHERE source_id = (SELECT source_id FROM pf.version WHERE id = $1)
|
||||
AND id <> $1
|
||||
AND is_default
|
||||
`, [req.params.id]);
|
||||
}
|
||||
const result = await client.query(`
|
||||
const result = await pool.query(`
|
||||
UPDATE pf.version SET
|
||||
name = COALESCE($2, name),
|
||||
description = COALESCE($3, description),
|
||||
exclude_iters = COALESCE($4, exclude_iters),
|
||||
adjustment_segment = CASE WHEN $5::bool THEN $6::text ELSE adjustment_segment END,
|
||||
adjustment_bucket = CASE WHEN $7::bool THEN $8::text ELSE adjustment_bucket END,
|
||||
unlabeled_load = CASE WHEN $9::bool THEN $10::text ELSE unlabeled_load END,
|
||||
is_default = CASE WHEN $11::bool THEN $12::bool ELSE is_default END
|
||||
name = COALESCE($2, name),
|
||||
description = COALESCE($3, description),
|
||||
exclude_iters = COALESCE($4, exclude_iters)
|
||||
WHERE id = $1
|
||||
RETURNING *
|
||||
`, [
|
||||
req.params.id,
|
||||
name || null,
|
||||
description || null,
|
||||
exclude_iters ? JSON.stringify(exclude_iters) : null,
|
||||
adjustment_segment !== undefined, set(adjustment_segment),
|
||||
adjustment_bucket !== undefined, set(adjustment_bucket),
|
||||
unlabeled_load !== undefined, set(unlabeled_load),
|
||||
is_default !== undefined, is_default === true
|
||||
exclude_iters ? JSON.stringify(exclude_iters) : null
|
||||
]);
|
||||
if (result.rows.length === 0) {
|
||||
await client.query('ROLLBACK');
|
||||
return res.status(404).json({ error: 'Version not found' });
|
||||
}
|
||||
await client.query('COMMIT');
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK').catch(() => {});
|
||||
console.error(err);
|
||||
// (source_id, name) is unique, so a rename can collide. The POST has
|
||||
// said so for as long as it has existed; this said it with a raw
|
||||
// Postgres message, which reads as a server fault rather than a name
|
||||
// that is already taken.
|
||||
if (err.code === '23505') {
|
||||
return res.status(409).json({
|
||||
// Two unique constraints raise 23505 here, and the name is
|
||||
// only one of them -- version_one_default is the other,
|
||||
// reachable only if two requests set the default at once.
|
||||
error: err.constraint === 'version_one_default'
|
||||
? 'Another forecast was just made the default — reload and try again'
|
||||
: 'A version with that name already exists for this source'
|
||||
});
|
||||
}
|
||||
res.status(500).json({ error: err.message });
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
});
|
||||
|
||||
// close a version — blocks further edits
|
||||
router.post('/versions/:id/close', requireAdmin, async (req, res) => {
|
||||
const pf_user = sessionUser(req);
|
||||
router.post('/versions/:id/close', async (req, res) => {
|
||||
const { pf_user } = req.body;
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
UPDATE pf.version
|
||||
@ -615,7 +281,7 @@ ${colDefs},
|
||||
});
|
||||
|
||||
// reopen a closed version
|
||||
router.post('/versions/:id/reopen', requireAdmin, async (req, res) => {
|
||||
router.post('/versions/:id/reopen', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
UPDATE pf.version
|
||||
@ -635,7 +301,7 @@ ${colDefs},
|
||||
|
||||
// delete a version — drops forecast table then deletes version record
|
||||
// log entries are removed by ON DELETE CASCADE on pf.log.version_id
|
||||
router.delete('/versions/:id', requireAdmin, async (req, res) => {
|
||||
router.delete('/versions/:id', async (req, res) => {
|
||||
const versionId = parseInt(req.params.id);
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
|
||||
55
server.js
55
server.js
@ -1,26 +1,15 @@
|
||||
require('dotenv').config();
|
||||
const express = require('express');
|
||||
const cors = require('cors');
|
||||
const session = require('express-session');
|
||||
const PgSession = require('connect-pg-simple')(session);
|
||||
const { Pool, types } = require('pg');
|
||||
const { requireAuth } = require('./lib/auth');
|
||||
|
||||
// Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings,
|
||||
// so apache-arrow encodes them as Int/Float64 rather than Dictionary<Utf8>.
|
||||
// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>.
|
||||
types.setTypeParser(20, v => v === null ? null : Number(v));
|
||||
types.setTypeParser(1700, v => v === null ? null : Number(v));
|
||||
|
||||
const app = express();
|
||||
|
||||
// Sessions ride on a cookie, so a wildcard CORS origin would let any site make
|
||||
// credentialed calls on behalf of a logged-in user. The UI is served from this
|
||||
// same origin and needs no CORS at all; set CORS_ORIGIN only for a separate
|
||||
// front-end host, and it is then allowed by name, never by wildcard.
|
||||
if (process.env.CORS_ORIGIN) {
|
||||
app.use(cors({ origin: process.env.CORS_ORIGIN.split(',').map(o => o.trim()), credentials: true }));
|
||||
}
|
||||
|
||||
app.use(cors());
|
||||
app.use(express.json());
|
||||
app.use(express.static('public/app'));
|
||||
|
||||
@ -37,51 +26,11 @@ pool.on('error', (err) => {
|
||||
console.error('pg pool error', err);
|
||||
});
|
||||
|
||||
// ── Authentication ────────────────────────────────────────────
|
||||
// Refuse to boot without a secret rather than fall back to a default one:
|
||||
// a predictable secret means forgeable session cookies.
|
||||
const sessionSecret = process.env.SESSION_SECRET;
|
||||
if (!sessionSecret) {
|
||||
console.error('SESSION_SECRET is not set. Run: ./pf.sh config');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// TLS terminates at the reverse proxy, so express has to trust its headers for
|
||||
// req.ip (the login throttle) and for secure-cookie detection to be right.
|
||||
app.set('trust proxy', process.env.TRUST_PROXY || 1);
|
||||
|
||||
const cookieSecure = process.env.COOKIE_SECURE !== 'false';
|
||||
if (!cookieSecure) {
|
||||
console.warn('COOKIE_SECURE=false — session cookie will be sent over plain HTTP.');
|
||||
}
|
||||
|
||||
app.use(session({
|
||||
name: 'pf.sid',
|
||||
store: new PgSession({ pool, schemaName: 'pf', tableName: 'session', createTableIfMissing: false }),
|
||||
secret: sessionSecret,
|
||||
resave: false,
|
||||
saveUninitialized: false,
|
||||
rolling: true,
|
||||
cookie: {
|
||||
httpOnly: true,
|
||||
sameSite: 'lax',
|
||||
secure: cookieSecure,
|
||||
maxAge: 1000 * 60 * 60 * 12,
|
||||
},
|
||||
}));
|
||||
|
||||
app.use('/api', require('./routes/auth')(pool));
|
||||
|
||||
// Everything below this line requires a session.
|
||||
app.use('/api', requireAuth);
|
||||
|
||||
app.use('/api', require('./routes/tables')(pool));
|
||||
app.use('/api', require('./routes/sources')(pool));
|
||||
app.use('/api', require('./routes/versions')(pool));
|
||||
app.use('/api', require('./routes/operations')(pool));
|
||||
app.use('/api', require('./routes/log')(pool));
|
||||
app.use('/api', require('./routes/layouts')(pool));
|
||||
app.use('/api', require('./routes/plans')(pool));
|
||||
|
||||
|
||||
const port = process.env.PORT || 3010;
|
||||
|
||||
@ -17,6 +17,8 @@ CREATE TABLE IF NOT EXISTS pf.source (
|
||||
|
||||
-- backfill columns for existing installs
|
||||
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
|
||||
|
||||
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
|
||||
|
||||
@ -27,18 +29,10 @@ CREATE TABLE IF NOT EXISTS pf.col_meta (
|
||||
label text,
|
||||
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | ignore
|
||||
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
|
||||
dim_group text, -- groups functionally dependent columns
|
||||
dim_period_col text, -- pf.dim_period column this dimension derives from
|
||||
in_grain boolean NOT NULL DEFAULT false, -- true = column defines the display grain
|
||||
opos integer,
|
||||
UNIQUE (source_id, cname)
|
||||
);
|
||||
|
||||
-- backfill columns for existing installs (must follow the CREATE above)
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS pf.version (
|
||||
id serial PRIMARY KEY,
|
||||
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
|
||||
@ -70,194 +64,6 @@ CREATE TABLE IF NOT EXISTS pf.log (
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS tag text;
|
||||
CREATE INDEX IF NOT EXISTS log_tag_idx ON pf.log (tag) WHERE tag IS NOT NULL;
|
||||
|
||||
-- seed tags for loads that predate the column: a baseline/reference note is the
|
||||
-- segment's name ('Open Orders', 'Prior Year'), which is exactly what tag holds.
|
||||
-- Adjustment notes are free text, not labels, so they are left alone.
|
||||
UPDATE pf.log
|
||||
SET tag = note
|
||||
WHERE TRUE
|
||||
AND tag IS NULL
|
||||
AND note IS NOT NULL
|
||||
AND note <> ''
|
||||
AND operation IN ('baseline', 'reference');
|
||||
|
||||
-- Display order for the pivot's segment and bucket columns.
|
||||
--
|
||||
-- The segment's display name in the pivot, falling back to tag then note.
|
||||
--
|
||||
-- Separate from both because those have jobs already -- tag groups adjustments
|
||||
-- into initiatives for the bridge, note is free commentary -- and because the
|
||||
-- label carries the sort order. Perspective orders column groups by the value
|
||||
-- string, so a leading "01 - " is how ordering is expressed; putting that in the
|
||||
-- note would put it in every note.
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS label text;
|
||||
|
||||
-- Vestigial, both of them. They held the ordinal when the "01 - " prefix was
|
||||
-- computed for the pivot rather than typed into label and bucket: bucket_order
|
||||
-- sequenced the bucket columns, log.seq the segments within them. Nothing reads
|
||||
-- either now, and nothing writes them -- kept only because dropping a column is
|
||||
-- not worth a migration to reclaim two that cost nothing.
|
||||
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS bucket_order jsonb;
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS seq integer;
|
||||
|
||||
-- What a segment contributes to, independent of pf_iter.
|
||||
--
|
||||
-- pf_iter answers "can operations write to these rows"; bucket answers "does this
|
||||
-- belong in the forecast number". Those are not the same question -- Open Orders is
|
||||
-- loaded as reference so nothing adjusts it, yet it is part of the forecast -- so
|
||||
-- neither can be derived from the other.
|
||||
--
|
||||
-- Free text with suggested values (Forecast / Prior Year / Prior Prior Year / Plan)
|
||||
-- rather than an enum, so a new banner does not need a migration. Blank by default:
|
||||
-- until a segment is labelled, the pivot falls back to showing its own name.
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS bucket text;
|
||||
CREATE INDEX IF NOT EXISTS log_bucket_idx ON pf.log (bucket) WHERE bucket IS NOT NULL;
|
||||
|
||||
-- The names a row falls back to when nobody has named it, per scenario. Null
|
||||
-- means "use the built-in", which is the DISPLAY DEFAULTS block in
|
||||
-- lib/sql_generator.js. Read through a join at query time, not baked into
|
||||
-- pf.sql: those templates are keyed on (source_id, operation) and shared by
|
||||
-- every version of a source.
|
||||
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_segment text;
|
||||
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS adjustment_bucket text;
|
||||
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS unlabeled_load text;
|
||||
|
||||
-- Every log entry is named. label is the row's segment and bucket what it counts
|
||||
-- toward, both stored rather than derived, so a report reads them off pf.log with
|
||||
-- no rule of its own. The routes stamp them on write: a load's label is required
|
||||
-- and its bucket defaults to the label; an adjustment gets the version's
|
||||
-- adjustment_segment / adjustment_bucket, else the built-ins in sql_generator.
|
||||
--
|
||||
-- The backfill names entries written before that, with the rule the pivot used to
|
||||
-- apply at read time, so nothing already on screen changes name. It is the last
|
||||
-- reader of pf.version.unlabeled_load; the column remains.
|
||||
WITH
|
||||
-- the read-time rule, verbatim
|
||||
named AS (
|
||||
SELECT
|
||||
l.id
|
||||
,CASE
|
||||
WHEN l.operation IN ('baseline', 'reference')
|
||||
THEN COALESCE(NULLIF(l.label, ''), NULLIF(l.tag, ''), NULLIF(l.note, ''), NULLIF(v.unlabeled_load, ''), 'Unlabeled')
|
||||
ELSE COALESCE(NULLIF(l.label, ''), NULLIF(v.adjustment_segment, ''), '99 - Adjustments')
|
||||
END AS label
|
||||
,l.operation
|
||||
,l.bucket
|
||||
,v.adjustment_bucket
|
||||
FROM
|
||||
pf.log l
|
||||
LEFT JOIN pf.version v
|
||||
ON v.id = l.version_id
|
||||
WHERE TRUE
|
||||
AND (NULLIF(l.label, '') IS NULL OR NULLIF(l.bucket, '') IS NULL)
|
||||
)
|
||||
UPDATE pf.log l
|
||||
SET
|
||||
label = n.label
|
||||
,bucket = COALESCE(
|
||||
NULLIF(n.bucket, '')
|
||||
,CASE
|
||||
WHEN n.operation IN ('baseline', 'reference') THEN n.label
|
||||
ELSE COALESCE(NULLIF(n.adjustment_bucket, ''), '04 - Forecast')
|
||||
END
|
||||
)
|
||||
FROM
|
||||
named n
|
||||
WHERE TRUE
|
||||
AND l.id = n.id;
|
||||
|
||||
ALTER TABLE pf.log DROP CONSTRAINT IF EXISTS log_named;
|
||||
ALTER TABLE pf.log ADD CONSTRAINT log_named
|
||||
CHECK (NULLIF(label, '') IS NOT NULL AND NULLIF(bucket, '') IS NOT NULL);
|
||||
|
||||
-- Where an account lands when it has no saved selection of its own.
|
||||
--
|
||||
-- Without it the app fell back to the newest version, which is whatever
|
||||
-- somebody created last -- a scratch scenario as readily as the plan. Sales
|
||||
-- accounts, which never pick a version, therefore followed whatever the
|
||||
-- forecasting work happened to be doing that morning.
|
||||
--
|
||||
-- At most one per source, by partial unique index, the way pf.layout does it:
|
||||
-- two defaults is not a state anything could resolve, so the database refuses
|
||||
-- it rather than the route picking one.
|
||||
ALTER TABLE pf.version ADD COLUMN IF NOT EXISTS is_default boolean NOT NULL DEFAULT false;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS version_one_default
|
||||
ON pf.version (source_id)
|
||||
WHERE is_default;
|
||||
|
||||
-- What the entry did, stamped when it did it.
|
||||
--
|
||||
-- Not a cache: a log entry's forecast rows never change after it is written.
|
||||
-- Only the operation that owns the logid inserts them, and the only thing that
|
||||
-- removes them is undo, which deletes this row too -- so these totals are fixed
|
||||
-- at write time rather than derived from something that can move underneath
|
||||
-- them. The change log was joining the whole forecast table to recompute them
|
||||
-- on every open, 2.5M rows to total a few thousand.
|
||||
--
|
||||
-- measure_cols records which columns they are denominated in, since the value
|
||||
-- and units roles can be reassigned in col_meta and the numbers would otherwise
|
||||
-- quietly come to mean something else.
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS row_count integer;
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS value_total double precision;
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS units_total double precision;
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS measure_cols jsonb;
|
||||
|
||||
-- The column an account's territory is expressed in -- the rep, the region,
|
||||
-- whatever this source divides ownership by. Exactly one per source.
|
||||
--
|
||||
-- Flagged here rather than named in code because it is source-specific, the
|
||||
-- same way the grain is: a second source should not need a code change to be
|
||||
-- scoped by something other than a sales rep.
|
||||
--
|
||||
-- It does two jobs. The server scopes every read and write to the values on the
|
||||
-- account, and recode refuses to *set* this column unless the account is an
|
||||
-- admin -- moving a row between territories is reassignment, not forecasting.
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS is_territory boolean NOT NULL DEFAULT false;
|
||||
|
||||
-- The debug path: what the entry was trying to do, and what that became.
|
||||
--
|
||||
-- Both, deliberately. params records the intent -- the slice, the scope, the
|
||||
-- resolved increments -- and sql_text records the statement as executed, with
|
||||
-- territory and scope already resolved into it. Keeping only one of them
|
||||
-- assumes the translation between them is correct, which is exactly the
|
||||
-- assumption in doubt when the rows look wrong.
|
||||
--
|
||||
-- env captures the state the intent was executed against that cannot be
|
||||
-- reconstructed afterwards: the territory in force, the version's
|
||||
-- exclude_iters, and when the template was generated. All three are mutable
|
||||
-- rows elsewhere, and nothing remembers what they were.
|
||||
--
|
||||
-- The template generation is a fingerprint, not a version: it cannot reproduce
|
||||
-- the old template, but it can tell you the entry ran under a different one,
|
||||
-- which is what would otherwise make a replay quietly wrong.
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS env jsonb;
|
||||
ALTER TABLE pf.log ADD COLUMN IF NOT EXISTS sql_text text;
|
||||
|
||||
-- Master data for a dim_group: one row per key value, with its sibling columns.
|
||||
--
|
||||
-- The source is transactional and often a view over all history, so deriving a
|
||||
-- member list from it is both slow and wrong -- slow because it means scanning
|
||||
-- millions of rows, wrong because it can only describe what was transacted and
|
||||
-- has no way to say a part is discontinued or that a new one exists before it
|
||||
-- has sold. This table is the app's own list, refreshed from the source but
|
||||
-- curatable independently of it.
|
||||
CREATE TABLE IF NOT EXISTS pf.dim_member (
|
||||
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
|
||||
dim_group text NOT NULL, -- matches pf.col_meta.dim_group
|
||||
key_value text NOT NULL, -- the is_key column's value
|
||||
attrs jsonb NOT NULL DEFAULT '{}'::jsonb, -- the sibling columns
|
||||
is_active boolean NOT NULL DEFAULT true,
|
||||
source_seen boolean NOT NULL DEFAULT true, -- present in the source at last refresh
|
||||
added_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
refreshed_at timestamptz,
|
||||
PRIMARY KEY (source_id, dim_group, key_value)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS dim_member_active_idx
|
||||
ON pf.dim_member (source_id, dim_group) WHERE is_active;
|
||||
|
||||
-- generated operation SQL per source, stored after col_meta is configured
|
||||
CREATE TABLE IF NOT EXISTS pf.sql (
|
||||
id serial PRIMARY KEY,
|
||||
@ -267,95 +73,3 @@ CREATE TABLE IF NOT EXISTS pf.sql (
|
||||
generated_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (source_id, operation)
|
||||
);
|
||||
|
||||
-- pf.layout: named Perspective view configs.
|
||||
--
|
||||
-- Replaces the per-browser localStorage lists (pf_layouts_v*) and the single
|
||||
-- anonymous pf.source.default_layout. A layout is either `private` -- visible
|
||||
-- only to its owner -- or `published`, visible to everyone on the version and
|
||||
-- writable only by its owner or an admin, the same rule pf.log already uses.
|
||||
CREATE TABLE IF NOT EXISTS pf.layout (
|
||||
id serial PRIMARY KEY,
|
||||
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
|
||||
-- null = applies to every version of the source; that is where the old
|
||||
-- source default lives, and what a new version picks up before anyone has
|
||||
-- published a layout of its own.
|
||||
version_id integer REFERENCES pf.version(id) ON DELETE CASCADE,
|
||||
name text NOT NULL,
|
||||
config jsonb NOT NULL,
|
||||
owner text NOT NULL,
|
||||
visibility text NOT NULL DEFAULT 'private', -- private | published
|
||||
is_default boolean NOT NULL DEFAULT false, -- applied on first load
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
CHECK (visibility IN ('private', 'published')),
|
||||
-- only a published layout can be the default: a private one would be a
|
||||
-- default nobody else could see.
|
||||
CHECK (NOT is_default OR visibility = 'published')
|
||||
);
|
||||
|
||||
-- COALESCE rather than the bare column: version_id is nullable, and in a unique
|
||||
-- index NULLs are distinct, so source-wide rows would not be constrained at all.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS layout_private_name
|
||||
ON pf.layout (source_id, COALESCE(version_id, 0), owner, lower(name))
|
||||
WHERE visibility = 'private';
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS layout_published_name
|
||||
ON pf.layout (source_id, COALESCE(version_id, 0), lower(name))
|
||||
WHERE visibility = 'published';
|
||||
|
||||
-- one default per scope
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS layout_one_default
|
||||
ON pf.layout (source_id, COALESCE(version_id, 0))
|
||||
WHERE is_default;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS layout_lookup ON pf.layout (source_id, version_id);
|
||||
|
||||
-- Carry the old single source default across as a published, source-wide row.
|
||||
-- Idempotent: skipped once a default exists for that source.
|
||||
INSERT INTO pf.layout (source_id, version_id, name, config, owner, visibility, is_default)
|
||||
SELECT s.id, NULL, 'Source default', s.default_layout, COALESCE(s.created_by, 'admin'), 'published', true
|
||||
FROM pf.source s
|
||||
WHERE TRUE
|
||||
AND s.default_layout IS NOT NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM pf.layout l
|
||||
WHERE TRUE
|
||||
AND l.source_id = s.id
|
||||
AND l.version_id IS NULL
|
||||
AND l.is_default
|
||||
);
|
||||
|
||||
-- pf.load_plan: a segment's definition, before and apart from having been run.
|
||||
--
|
||||
-- Until now a segment existed only once it had been loaded -- as a pf.log entry
|
||||
-- with forecast rows behind it -- so cloning a version's definition meant
|
||||
-- re-running every load, and there was nowhere to hold "this is the cut I want,
|
||||
-- but the dates need changing first".
|
||||
--
|
||||
-- Deliberately not pf.log. That table is the audit log: every row in it means a
|
||||
-- write happened, and putting unexecuted intent there would cost it that
|
||||
-- meaning. A plan is intent; run_logid is the pf.log entry it produced.
|
||||
CREATE TABLE IF NOT EXISTS pf.load_plan (
|
||||
id serial PRIMARY KEY,
|
||||
version_id integer NOT NULL REFERENCES pf.version(id) ON DELETE CASCADE,
|
||||
seq integer NOT NULL DEFAULT 0, -- run order
|
||||
kind text NOT NULL, -- baseline | reference
|
||||
label text,
|
||||
bucket text,
|
||||
tag text,
|
||||
note text,
|
||||
filters jsonb, -- the structured filter builder's own shape
|
||||
raw_where text, -- the raw-SQL escape hatch, when used instead
|
||||
date_offset text NOT NULL DEFAULT '0 days',
|
||||
-- the pf.log entry this plan produced, or null while it is still a plan.
|
||||
-- ON DELETE SET NULL so undoing a load returns its plan to unrun rather than
|
||||
-- deleting the definition with it.
|
||||
run_logid bigint REFERENCES pf.log(id) ON DELETE SET NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
created_by text,
|
||||
CHECK (kind IN ('baseline', 'reference'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS load_plan_version ON pf.load_plan (version_id, seq, id);
|
||||
|
||||
@ -1,38 +0,0 @@
|
||||
-- Pivot Forecast — authentication
|
||||
-- Run after 01_schema.sql: psql -d <db> -f setup_sql/02_auth.sql
|
||||
-- Safe to re-run.
|
||||
|
||||
-- Application accounts. Passwords are scrypt hashes written by lib/auth.js;
|
||||
-- the plaintext never reaches the database. Manage with ./pf.sh add-user.
|
||||
CREATE TABLE IF NOT EXISTS pf.app_user (
|
||||
id serial PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE,
|
||||
pass_hash text NOT NULL,
|
||||
display_name text,
|
||||
is_active boolean NOT NULL DEFAULT true,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
last_login_at timestamptz
|
||||
);
|
||||
|
||||
-- Session store for express-session (connect-pg-simple layout). Sessions live
|
||||
-- here rather than in memory so a restart doesn't sign everyone out, and so a
|
||||
-- session can be revoked by deleting its row.
|
||||
CREATE TABLE IF NOT EXISTS pf.session (
|
||||
sid varchar PRIMARY KEY NOT NULL COLLATE "default",
|
||||
sess json NOT NULL,
|
||||
expire timestamp(6) NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS session_expire_idx ON pf.session (expire);
|
||||
|
||||
-- What an account may see and change.
|
||||
--
|
||||
-- territory is the list of values allowed in the source's territory column
|
||||
-- (col_meta.is_territory). It is a filter the server applies to every read and
|
||||
-- every write; it is never sent by the client, which could remove it.
|
||||
--
|
||||
-- Fail closed: no territory and not an admin means no rows. An account created
|
||||
-- without one sees nothing until someone grants it, rather than seeing the whole
|
||||
-- book because a column was left null.
|
||||
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS territory jsonb;
|
||||
ALTER TABLE pf.app_user ADD COLUMN IF NOT EXISTS is_admin boolean NOT NULL DEFAULT false;
|
||||
16
ui/README.md
Normal file
16
ui/README.md
Normal file
@ -0,0 +1,16 @@
|
||||
# React + Vite
|
||||
|
||||
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
|
||||
|
||||
Currently, two official plugins are available:
|
||||
|
||||
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
|
||||
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
|
||||
|
||||
## React Compiler
|
||||
|
||||
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
|
||||
|
||||
## Expanding the ESLint configuration
|
||||
|
||||
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.
|
||||
80
ui/package-lock.json
generated
80
ui/package-lock.json
generated
@ -8,10 +8,10 @@
|
||||
"name": "ui",
|
||||
"version": "0.0.0",
|
||||
"dependencies": {
|
||||
"@perspective-dev/client": "file:vendor/perspective-dev-client-5.4.0.tgz",
|
||||
"@perspective-dev/server": "file:vendor/perspective-dev-server-5.4.0.tgz",
|
||||
"@perspective-dev/viewer": "file:vendor/perspective-dev-viewer-5.4.0.tgz",
|
||||
"@perspective-dev/viewer-datagrid": "file:vendor/perspective-dev-viewer-datagrid-5.4.0.tgz",
|
||||
"@perspective-dev/client": "5.2.0",
|
||||
"@perspective-dev/server": "5.2.0",
|
||||
"@perspective-dev/viewer": "5.2.0",
|
||||
"@perspective-dev/viewer-datagrid": "5.2.0",
|
||||
"react": "^19.2.5",
|
||||
"react-dom": "^19.2.5"
|
||||
},
|
||||
@ -59,6 +59,7 @@
|
||||
"integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/code-frame": "^7.29.0",
|
||||
"@babel/generator": "^7.29.0",
|
||||
@ -268,29 +269,6 @@
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/core": {
|
||||
"version": "1.11.3",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.3.tgz",
|
||||
"integrity": "sha512-zLpS5asjEb7lq8jYLq37N6XKaE41DIexlY1rF/z4/tIl3wo13Sqm28fRyfIsKZD+NZ8mM5RoKkpW/rBcuoSZSg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/wasi-threads": "1.2.3",
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/runtime": {
|
||||
"version": "1.11.3",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz",
|
||||
"integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/wasi-threads": {
|
||||
"version": "1.2.3",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.3.tgz",
|
||||
@ -576,43 +554,43 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@perspective-dev/client": {
|
||||
"version": "5.4.0",
|
||||
"resolved": "file:vendor/perspective-dev-client-5.4.0.tgz",
|
||||
"integrity": "sha512-d6TKVh6DFKuGpiLYN2PmA8U9GbkMtgiCy9iUrLx7STMABT2kYwUE4tVQLqm/sRlwcRiVJ4+K2d1l6J8cgX+6Aw==",
|
||||
"version": "5.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@perspective-dev/client/-/client-5.2.0.tgz",
|
||||
"integrity": "sha512-zkJmJFwdw0wMREoJt8gUJyCsflzi7s7Yc8ZtUCOLE9XB6fdyxJmDB99cxO3Q+hno/57kLsz8VNoz8XSm6PZNrg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@perspective-dev/server": "^5.4.0",
|
||||
"@perspective-dev/server": "",
|
||||
"pro_self_extracting_wasm": "0.0.9",
|
||||
"stoppable": "=1.1.0",
|
||||
"ws": "^8.17.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@perspective-dev/server": {
|
||||
"version": "5.4.0",
|
||||
"resolved": "file:vendor/perspective-dev-server-5.4.0.tgz",
|
||||
"integrity": "sha512-Z5oGEOYqTHMGoMElYbrfJL3YCtoGrAX6+22+3KwrxWDDLB5OlOOdsctfeEAdGqVeJ1/WFhqZp0fWBPEn4vcXrQ==",
|
||||
"version": "5.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@perspective-dev/server/-/server-5.2.0.tgz",
|
||||
"integrity": "sha512-WRBiokT2/BYM8Ipe7Dg1/2UYNb01Vsox79KBfFVI800VmwdId0GdqI95zufN5ZoFffeBfri1sr3S3AShBRFXKA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@perspective-dev/viewer": {
|
||||
"version": "5.4.0",
|
||||
"resolved": "file:vendor/perspective-dev-viewer-5.4.0.tgz",
|
||||
"integrity": "sha512-AkQwfacQFSJ0fZ7Ja142pa0HGLssxFK9/FM4jhrMvrXp31Rp74qRNIyueG8wiqGTPXmORaRDA+ePtW+Wfg5UpA==",
|
||||
"version": "5.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@perspective-dev/viewer/-/viewer-5.2.0.tgz",
|
||||
"integrity": "sha512-IIyqdPduofzzZ/QWrteq29IadJQR8IapTRz7U6XbrtBBm7eyiFsIBKQV0wFfcqWg4TRnhuvCAUvBRieuE0R8Eg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@perspective-dev/client": "^5.4.0",
|
||||
"@perspective-dev/client": "",
|
||||
"pro_self_extracting_wasm": "0.0.9",
|
||||
"regular-layout": "=0.6.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@perspective-dev/viewer-datagrid": {
|
||||
"version": "5.4.0",
|
||||
"resolved": "file:vendor/perspective-dev-viewer-datagrid-5.4.0.tgz",
|
||||
"integrity": "sha512-7wqr7T8cp6SI1ZZy69qy0Y6sZ5UJCpQjHPnDL6Fjs3ATJtmhmVEz/dpcMcjCgqwGrs5Sqd7xyKNQ+RuwcBsMEQ==",
|
||||
"version": "5.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@perspective-dev/viewer-datagrid/-/viewer-datagrid-5.2.0.tgz",
|
||||
"integrity": "sha512-v/SR/35YfyKfivO21nglJFiyG1iE5G8vMwIwWI6fQuwjW764pmNrm/zcrNWNY70x5XHkPRM94aY6LjkLsLkO2g==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@perspective-dev/client": "^5.4.0",
|
||||
"@perspective-dev/viewer": "^5.4.0",
|
||||
"regular-table": "=0.9.1"
|
||||
"@perspective-dev/client": "",
|
||||
"@perspective-dev/viewer": "",
|
||||
"regular-table": "=0.8.6"
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-android-arm64": {
|
||||
@ -1223,6 +1201,7 @@
|
||||
"integrity": "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"csstype": "^3.2.2"
|
||||
}
|
||||
@ -1269,6 +1248,7 @@
|
||||
"integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"acorn": "bin/acorn"
|
||||
},
|
||||
@ -1398,6 +1378,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"baseline-browser-mapping": "^2.10.12",
|
||||
"caniuse-lite": "^1.0.30001782",
|
||||
@ -1671,6 +1652,7 @@
|
||||
"integrity": "sha512-wiyGaKsDgqXvF40P8mDwiUp/KQjE1FdrIEJsM8PZ3XCiniTMXS3OHWWUe5FI5agoCnr8x4xPrTDZuxsBlNHl+Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@eslint-community/eslint-utils": "^4.8.0",
|
||||
"@eslint-community/regexpp": "^4.12.2",
|
||||
@ -2837,6 +2819,7 @@
|
||||
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@ -2941,6 +2924,7 @@
|
||||
"resolved": "https://registry.npmjs.org/react/-/react-19.2.5.tgz",
|
||||
"integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@ -2964,9 +2948,9 @@
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/regular-table": {
|
||||
"version": "0.9.1",
|
||||
"resolved": "https://registry.npmjs.org/regular-table/-/regular-table-0.9.1.tgz",
|
||||
"integrity": "sha512-3I/2I3NEhmyScCevW/r1AE9hyUkqrPRMDsgo/nDnGXhpXVZOxUddJQGBmEMLgyc7OSaQSzl9BtXllnMqw1MwCA==",
|
||||
"version": "0.8.6",
|
||||
"resolved": "https://registry.npmjs.org/regular-table/-/regular-table-0.8.6.tgz",
|
||||
"integrity": "sha512-jzJzu9WLtqwMgbf5ak/VdNm/YLHUDnDSCHD5SOksnHrHLuDN6l5i2stYfe7BjsHbQV1Gx9369Ma40nTBCgOFvA==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=16"
|
||||
@ -3303,6 +3287,7 @@
|
||||
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.4",
|
||||
@ -3461,6 +3446,7 @@
|
||||
"integrity": "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/colinhacks"
|
||||
}
|
||||
|
||||
@ -10,10 +10,10 @@
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@perspective-dev/client": "file:vendor/perspective-dev-client-5.4.0.tgz",
|
||||
"@perspective-dev/server": "file:vendor/perspective-dev-server-5.4.0.tgz",
|
||||
"@perspective-dev/viewer": "file:vendor/perspective-dev-viewer-5.4.0.tgz",
|
||||
"@perspective-dev/viewer-datagrid": "file:vendor/perspective-dev-viewer-datagrid-5.4.0.tgz",
|
||||
"@perspective-dev/client": "5.2.0",
|
||||
"@perspective-dev/server": "5.2.0",
|
||||
"@perspective-dev/viewer": "5.2.0",
|
||||
"@perspective-dev/viewer-datagrid": "5.2.0",
|
||||
"react": "^19.2.5",
|
||||
"react-dom": "^19.2.5"
|
||||
},
|
||||
|
||||
102
ui/src/App.jsx
102
ui/src/App.jsx
@ -10,10 +10,8 @@ export default function App() {
|
||||
const [sidebarExpanded, setSidebarExpanded] = useState(() => localStorage.getItem('pf_sidebar') !== 'collapsed')
|
||||
|
||||
const [sources, setSources] = useState([])
|
||||
const [sourcesLoaded, setSourcesLoaded] = useState(false)
|
||||
const [sourceId, setSourceId] = useState(() => localStorage.getItem('pf_sourceId') || '')
|
||||
const [versions, setVersions] = useState([])
|
||||
const [versionsLoaded, setVersionsLoaded] = useState(false)
|
||||
const [versionId, setVersionId] = useState(() => localStorage.getItem('pf_versionId') || '')
|
||||
|
||||
useEffect(() => { localStorage.setItem('pf_view', view) }, [view])
|
||||
@ -23,92 +21,37 @@ export default function App() {
|
||||
|
||||
const refreshSources = useCallback(async () => {
|
||||
const data = await fetch('/api/sources').then(r => r.json())
|
||||
const list = Array.isArray(data) ? data : []
|
||||
setSources(list)
|
||||
setSourcesLoaded(true)
|
||||
return list
|
||||
setSources(data)
|
||||
return data
|
||||
}, [])
|
||||
|
||||
const refreshVersions = useCallback(async (sid) => {
|
||||
const id = sid ?? sourceId
|
||||
if (!id) { setVersions([]); setVersionsLoaded(true); return [] }
|
||||
if (!id) { setVersions([]); return [] }
|
||||
const data = await fetch(`/api/sources/${id}/versions`).then(r => r.json())
|
||||
const list = Array.isArray(data) ? data : []
|
||||
setVersions(list)
|
||||
setVersionsLoaded(true)
|
||||
return list
|
||||
setVersions(data)
|
||||
return data
|
||||
}, [sourceId])
|
||||
|
||||
// Every source's forecasts, for the sidebar only.
|
||||
//
|
||||
// `versions` stays scoped to the selected source, because Baseline and
|
||||
// Forecast are, and widening it would change what it means everywhere. The
|
||||
// sidebar is the one place that has to be cross-source: it is how a forecast
|
||||
// gets chosen, and a forecast carries its own source_id, so choosing one
|
||||
// settles the source. Without this, a second registered source would be
|
||||
// unreachable from anywhere but Setup -- and invisible from everywhere else,
|
||||
// which is worse.
|
||||
//
|
||||
// A fan-out rather than an endpoint: there is one source today, so this is one
|
||||
// request, and a `GET /versions` is not worth adding until there are enough to
|
||||
// make the difference measurable.
|
||||
const [allVersions, setAllVersions] = useState([])
|
||||
const refreshAllVersions = useCallback(async (list) => {
|
||||
const srcs = list ?? sources
|
||||
if (!srcs.length) { setAllVersions([]); return }
|
||||
const lists = await Promise.all(srcs.map(s =>
|
||||
fetch(`/api/sources/${s.id}/versions`).then(r => r.json()).catch(() => [])))
|
||||
setAllVersions(lists.flat().filter(v => v && v.id != null))
|
||||
}, [sources])
|
||||
|
||||
useEffect(() => { refreshSources() }, [])
|
||||
|
||||
// Recomputed when the selected source's list changes, which is what creating
|
||||
// or deleting a version does -- so the sidebar follows without every caller
|
||||
// having to know it exists.
|
||||
useEffect(() => { if (sourcesLoaded) refreshAllVersions() }, [sourcesLoaded, sources, versions])
|
||||
|
||||
// The selection is restored from localStorage and survives a deregister, so it
|
||||
// has to be re-checked against the list itself rather than only at mount:
|
||||
// deleting the selected source otherwise leaves a dead id behind and every
|
||||
// call 404s "Source not found" until the page is reloaded.
|
||||
useEffect(() => {
|
||||
if (!sourcesLoaded) return
|
||||
if (sources.length === 0) { setSourceId(''); return }
|
||||
if (!sourceId || !sources.some(s => String(s.id) === String(sourceId))) {
|
||||
setSourceId(String(sources[0].id))
|
||||
}
|
||||
}, [sources, sourcesLoaded, sourceId])
|
||||
refreshSources().then(data => {
|
||||
if (data.length === 0) { setSourceId(''); return }
|
||||
if (!sourceId || !data.some(s => String(s.id) === String(sourceId))) {
|
||||
setSourceId(String(data[0].id))
|
||||
}
|
||||
})
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
if (!sourceId) { setVersions([]); setVersionId(''); setVersionsLoaded(true); return }
|
||||
// The list belongs to the previous source until the fetch lands.
|
||||
setVersionsLoaded(false)
|
||||
refreshVersions(sourceId)
|
||||
if (!sourceId) { setVersions([]); setVersionId(''); return }
|
||||
refreshVersions(sourceId).then(data => {
|
||||
if (data.length === 0) { setVersionId(''); return }
|
||||
if (!versionId || !data.some(v => String(v.id) === String(versionId))) {
|
||||
setVersionId(String(data[0].id))
|
||||
}
|
||||
})
|
||||
}, [sourceId])
|
||||
|
||||
// Same reasoning as sources: a deleted version must not stay selected.
|
||||
//
|
||||
// versionsLoaded matters more than it looks: without it, the empty initial
|
||||
// state reads as "this source has no versions", so a versionId restored from
|
||||
// localStorage is cleared and then set straight back when the fetch lands.
|
||||
// Forecast's load effect is keyed on that id, so the round trip made every
|
||||
// page load fetch and aggregate the whole version twice.
|
||||
//
|
||||
// What it falls back TO is the source's default version, set on the Baseline
|
||||
// page, and only the newest version when nobody has named one. The list is
|
||||
// ordered created_at DESC, so the old fallback landed an account on whatever
|
||||
// scenario was created last -- a scratch copy as readily as the plan. Sales
|
||||
// accounts never pick a version themselves, so that is where they stayed.
|
||||
useEffect(() => {
|
||||
if (!sourceId || !versionsLoaded) return
|
||||
if (versions.length === 0) { setVersionId(''); return }
|
||||
if (!versionId || !versions.some(v => String(v.id) === String(versionId))) {
|
||||
const landing = versions.find(v => v.is_default) || versions[0]
|
||||
setVersionId(String(landing.id))
|
||||
}
|
||||
}, [versions, versionsLoaded, sourceId, versionId])
|
||||
|
||||
const ctx = {
|
||||
sources, sourceId, setSourceId,
|
||||
versions, versionId, setVersionId,
|
||||
@ -117,12 +60,7 @@ export default function App() {
|
||||
|
||||
return (
|
||||
<div className="flex h-screen w-full text-sm overflow-hidden">
|
||||
<Sidebar
|
||||
view={view} setView={setView}
|
||||
expanded={sidebarExpanded} setExpanded={setSidebarExpanded}
|
||||
versions={allVersions} versionId={versionId} setVersionId={setVersionId}
|
||||
sourceId={sourceId} setSourceId={setSourceId}
|
||||
/>
|
||||
<Sidebar view={view} setView={setView} expanded={sidebarExpanded} setExpanded={setSidebarExpanded} />
|
||||
<div className="flex flex-col flex-1 overflow-hidden min-w-0">
|
||||
<StatusBar view={view} {...ctx} />
|
||||
<div className="flex-1 overflow-hidden">
|
||||
|
||||
@ -1,62 +0,0 @@
|
||||
import { createContext, useContext, useState, useEffect, useCallback } from 'react'
|
||||
|
||||
const AuthContext = createContext()
|
||||
|
||||
// A session can expire while the app is open. Rather than teach every one of
|
||||
// the app's fetch calls to check for it, wrap fetch once: any 401 from /api
|
||||
// drops the whole UI back to the login screen. Cookies ride along on their own
|
||||
// — fetch defaults to same-origin credentials, and the UI is served from the
|
||||
// same origin as the API.
|
||||
function installUnauthorizedHandler(onUnauthorized) {
|
||||
const original = window.fetch
|
||||
window.fetch = async (...args) => {
|
||||
const res = await original(...args)
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0]?.url || ''
|
||||
if (res.status === 401 && url.includes('/api/') && !url.includes('/api/login')) {
|
||||
onUnauthorized()
|
||||
}
|
||||
return res
|
||||
}
|
||||
return () => { window.fetch = original }
|
||||
}
|
||||
|
||||
export function AuthProvider({ children }) {
|
||||
const [user, setUser] = useState(null)
|
||||
const [checking, setCheck] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
fetch('/api/me')
|
||||
.then(r => r.ok ? r.json() : null)
|
||||
.then(d => setUser(d?.user || null))
|
||||
.catch(() => setUser(null))
|
||||
.finally(() => setCheck(false))
|
||||
}, [])
|
||||
|
||||
useEffect(() => installUnauthorizedHandler(() => setUser(null)), [])
|
||||
|
||||
const login = useCallback(async (username, password) => {
|
||||
const r = await fetch('/api/login', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ username, password }),
|
||||
})
|
||||
const data = await r.json().catch(() => ({}))
|
||||
if (!r.ok) throw new Error(data.error || 'Login failed')
|
||||
setUser(data.user)
|
||||
return data.user
|
||||
}, [])
|
||||
|
||||
const logout = useCallback(async () => {
|
||||
try { await fetch('/api/logout', { method: 'POST' }) } catch {}
|
||||
setUser(null)
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<AuthContext.Provider value={{ user, checking, login, logout }}>
|
||||
{children}
|
||||
</AuthContext.Provider>
|
||||
)
|
||||
}
|
||||
|
||||
const useAuth = () => useContext(AuthContext)
|
||||
export default useAuth
|
||||
178
ui/src/branchCollapse.js
Normal file
178
ui/src/branchCollapse.js
Normal file
@ -0,0 +1,178 @@
|
||||
// SPIKE — per-branch column collapse without forking the datagrid plugin.
|
||||
//
|
||||
// The question this answers: Excel can collapse 2025 while 2026 stays expanded.
|
||||
// Perspective cannot, because the column axis has no expand/collapse of any kind
|
||||
// (see the "Column hierarchy" section of CLAUDE.md). But under
|
||||
// `split_rollup_mode: 'rollup'` the engine already emits the whole column tree in
|
||||
// pre-order — grand total, then each branch's subtotal immediately followed by its
|
||||
// own leaves — so every number Excel would show is already on the client. What is
|
||||
// missing is only the ability to *not show* some of them.
|
||||
//
|
||||
// So this hides the leaves of a collapsed branch in the DOM, from a style listener,
|
||||
// and shrinks the spanning header cells to match. No fork, no engine change.
|
||||
//
|
||||
// What it cannot fix, and what the spike is really measuring: the plugin still
|
||||
// fetches and lays out the hidden columns. Its horizontal virtualisation asks the
|
||||
// view for a contiguous [start_col, end_col) window and indexes everything by that
|
||||
// x, so a hidden column still costs a fetch and still occupies an index. At small
|
||||
// column counts that is invisible; the open question is where it stops being so.
|
||||
|
||||
const ZERO_WIDTH = /^\u200b*$/
|
||||
|
||||
// The split_by levels a column actually belongs to, dropping the measure name.
|
||||
// 'Amount' -> [], '2025|Amount' -> ['2025'], '2025|AI|Amount' -> ['2025','AI']
|
||||
export function levelsOf(columnPath) {
|
||||
if (!columnPath) return []
|
||||
return columnPath.split('|').slice(0, -1)
|
||||
}
|
||||
|
||||
// Same, from a cell's column_header, where rolled-up levels arrive as runs of
|
||||
// zero-width spaces rather than being absent.
|
||||
export function levelsOfHeader(columnHeader, splitLen) {
|
||||
const out = []
|
||||
for (const level of (columnHeader || []).slice(0, splitLen)) {
|
||||
if (level == null || ZERO_WIDTH.test(level)) break
|
||||
out.push(level)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// How many consecutive columns from x share this cell's first chy+1 levels —
|
||||
// the span the group header would have if nothing were hidden.
|
||||
//
|
||||
// This has to be derived from the column paths on every draw. The obvious
|
||||
// shortcut, stashing the original span on the cell the first time we see it, is
|
||||
// wrong: regular-table recycles header cells between draws, so the stashed value
|
||||
// reappears on an unrelated group and silently truncates it.
|
||||
function fullSpan(columnPaths, x, chy) {
|
||||
const levels = levelsOf(columnPaths[x])
|
||||
if (levels.length < chy + 1) return 1 // a rollup placeholder covers only itself
|
||||
const prefix = levels.slice(0, chy + 1)
|
||||
let n = 0
|
||||
for (let i = x; i < columnPaths.length; i++) {
|
||||
const l = levelsOf(columnPaths[i])
|
||||
if (l.length < prefix.length || !prefix.every((v, k) => l[k] === v)) break
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
function isDescendant(levels, prefix) {
|
||||
return levels.length > prefix.length && prefix.every((v, i) => levels[i] === v)
|
||||
}
|
||||
|
||||
// Engine column indices to hide: the strict descendants of any collapsed branch.
|
||||
// The branch's own subtotal column survives — that is what you are collapsing to.
|
||||
export function hiddenColumns(columnPaths, collapsed) {
|
||||
const hidden = new Set()
|
||||
if (!collapsed.size) return hidden
|
||||
const prefixes = [...collapsed].map(c => c.split('|'))
|
||||
columnPaths.forEach((path, x) => {
|
||||
if (!path) return
|
||||
const levels = levelsOf(path)
|
||||
if (prefixes.some(p => isDescendant(levels, p))) hidden.add(x)
|
||||
})
|
||||
return hidden
|
||||
}
|
||||
|
||||
// The branch a header cell at column x, header row `chy`, would collapse: the
|
||||
// first chy+1 levels of its column path. Null when that is not a collapsible
|
||||
// group — the grand total, or the deepest level, which has no leaves to hide.
|
||||
export function branchAt(columnPaths, x, chy, splitLen) {
|
||||
const levels = levelsOf(columnPaths[x])
|
||||
if (levels.length < chy + 1) return null
|
||||
if (chy + 1 >= splitLen) return null
|
||||
return levels.slice(0, chy + 1).join('|')
|
||||
}
|
||||
|
||||
/**
|
||||
* Attach per-branch collapse to a datagrid plugin element.
|
||||
* Returns { toggle, collapsed, clear, detach } — `onChange` fires after a toggle
|
||||
* so the host can persist or re-render.
|
||||
*/
|
||||
export function attachBranchCollapse(grid, { onChange } = {}) {
|
||||
const collapsed = new Set()
|
||||
const rt = grid.regular_table
|
||||
|
||||
const splitLen = () => grid.model?._config?.split_by?.length || 0
|
||||
const paths = () => grid.model?._column_paths || []
|
||||
|
||||
function styleListener() {
|
||||
const len = splitLen()
|
||||
if (!len) return
|
||||
const hidden = hiddenColumns(paths(), collapsed)
|
||||
|
||||
const columnPaths = paths()
|
||||
|
||||
for (const cell of rt.querySelectorAll('tbody td, tbody th, thead th')) {
|
||||
const meta = rt.getMeta(cell)
|
||||
if (!meta) continue
|
||||
|
||||
// A group header stays visible but must shrink to the columns still
|
||||
// showing under it, or the header row and the body stop lining up.
|
||||
//
|
||||
// Only while something is actually collapsed. With nothing hidden this
|
||||
// must not touch colSpan at all — regular-table computes it each draw by
|
||||
// merging adjacent equal header values, and that answer is the right one.
|
||||
if (hidden.size && meta.type === 'column_header' && meta.x != null) {
|
||||
const chy = meta.column_header_y ?? 0
|
||||
if (chy < len) {
|
||||
const full = fullSpan(columnPaths, meta.x, chy)
|
||||
let visible = 0
|
||||
for (let i = meta.x; i < meta.x + full; i++) if (!hidden.has(i)) visible++
|
||||
cell.colSpan = Math.max(1, visible)
|
||||
}
|
||||
}
|
||||
|
||||
const hide = meta.x != null && hidden.has(meta.x)
|
||||
cell.style.display = hide ? 'none' : ''
|
||||
|
||||
// mark what can be clicked to collapse, and which way it would go
|
||||
if (meta.type === 'column_header' && meta.x != null && !hide) {
|
||||
const branch = branchAt(columnPaths, meta.x, meta.column_header_y ?? 0, len)
|
||||
if (branch) {
|
||||
cell.dataset.pfBranch = branch
|
||||
cell.style.cursor = 'pointer'
|
||||
cell.title = collapsed.has(branch)
|
||||
? `Expand ${branch}`
|
||||
: `Collapse ${branch} to its subtotal`
|
||||
} else {
|
||||
delete cell.dataset.pfBranch
|
||||
cell.style.cursor = ''
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function onClick(event) {
|
||||
const target = (event.composedPath?.()[0]) || event.target
|
||||
const th = target?.closest?.('th[data-pf-branch]')
|
||||
if (!th) return
|
||||
event.stopPropagation() // the plugin would otherwise sort or open the menu
|
||||
event.preventDefault()
|
||||
toggle(th.dataset.pfBranch)
|
||||
}
|
||||
|
||||
function toggle(branch) {
|
||||
if (collapsed.has(branch)) collapsed.delete(branch)
|
||||
else collapsed.add(branch)
|
||||
rt.draw({ preserve_width: true })?.catch?.(() => {})
|
||||
onChange?.(new Set(collapsed))
|
||||
}
|
||||
|
||||
function clear() {
|
||||
collapsed.clear()
|
||||
rt.draw({ preserve_width: true })?.catch?.(() => {})
|
||||
onChange?.(new Set(collapsed))
|
||||
}
|
||||
|
||||
rt.addStyleListener(styleListener)
|
||||
rt.addEventListener('click', onClick, true)
|
||||
|
||||
return {
|
||||
toggle,
|
||||
clear,
|
||||
collapsed: () => new Set(collapsed),
|
||||
detach() { rt.removeEventListener('click', onClick, true) },
|
||||
}
|
||||
}
|
||||
@ -15,15 +15,9 @@ import { useState, useEffect, useRef, useCallback } from 'react'
|
||||
const UP = '#2a78d6' // increase
|
||||
const DOWN = '#e34948' // decrease
|
||||
const ANCHOR = '#6b7280' // baseline / current — neutral, 4.83:1 on white
|
||||
// Everything that is not a bar follows the theme. These were hex literals, so in
|
||||
// dark mode the value labels stayed #374151 on a #2a2c2f panel and could not be
|
||||
// read. Applied through `style` rather than the presentation attribute, where
|
||||
// var() is not reliable across browsers.
|
||||
const GRID = 'var(--border-color)'
|
||||
const AXIS = 'var(--chart-axis)'
|
||||
const INK = 'var(--text-secondary)'
|
||||
const INK_DIM = 'var(--chart-ink-dim)'
|
||||
const PANEL = 'var(--bg-secondary)' // the bars' separating stroke
|
||||
const GRID = '#e5e7eb'
|
||||
const INK = '#374151'
|
||||
const INK_DIM = '#6b7280'
|
||||
|
||||
const fmt = (n, dp = 2) =>
|
||||
n == null || !isFinite(n) ? '—'
|
||||
@ -55,201 +49,59 @@ function niceTicks(min, max, count = 5) {
|
||||
// Turn raw forecast rows into the walk: baseline anchor, one floating step per
|
||||
// initiative tag, current anchor. Pure and exported so the arithmetic can be
|
||||
// checked against real data without a browser.
|
||||
// The walk from a basis to the current forecast.
|
||||
//
|
||||
// Without a basis this is the forecast's own composition: its loads as the
|
||||
// opening anchor, then one step per adjustment tag.
|
||||
//
|
||||
// With one -- Plan when building a forecast, Prior Year when building the AOP --
|
||||
// it starts there instead, and the identity that makes it exact is
|
||||
//
|
||||
// Forecast - Basis = (Forecast loads - Basis) + adjustments
|
||||
//
|
||||
// so the opening step is the difference between the forecast's own loads and the
|
||||
// basis, and every tagged adjustment explains the rest. No residual, no plug: the
|
||||
// bars sum to the endpoint by construction rather than by hoping the tags cover
|
||||
// everything.
|
||||
//
|
||||
// Membership comes from pf_bucket, not pf_iter. Those answer different questions
|
||||
// -- Open Orders is loaded as reference so nothing adjusts it, and is still part
|
||||
// of the forecast -- so a bridge keyed on iter silently dropped it.
|
||||
export function buildSteps(rows, {
|
||||
valueCol, unitsCol, logMeta = {},
|
||||
excludeIters = ['reference'], // kept for callers with no bucket data
|
||||
basis = null, // a pf_bucket name, or null for composition
|
||||
forecastBucket,
|
||||
}) {
|
||||
const hasBuckets = rows.some(r => r.pf_bucket != null)
|
||||
export function buildSteps(rows, { valueCol, unitsCol, logMeta = {}, excludeIters = ['reference'] }) {
|
||||
const excl = new Set(excludeIters)
|
||||
|
||||
// Which bucket is the forecast. It used to be the literal 'Forecast', and the
|
||||
// moment the buckets were renamed to carry their sort prefix -- '04 - Forecast'
|
||||
// -- no row matched it: every row counted as a comparison, the walk had no
|
||||
// steps, and the bridge showed the basis cancelling itself to zero.
|
||||
//
|
||||
// The caller passes the version's adjustment_bucket, which is the same value
|
||||
// an unbucketed adjustment is labelled with, so the two agree by construction.
|
||||
// Falling back to whichever bucket actually holds the adjustments keeps a
|
||||
// renamed or unconfigured version working rather than silently empty.
|
||||
const fcBucket = (() => {
|
||||
if (forecastBucket && rows.some(r => r.pf_bucket === forecastBucket)) return forecastBucket
|
||||
const adjusted = rows.find(r => r.pf_bucket && !['baseline', 'reference'].includes(r.pf_iter))
|
||||
if (adjusted) return adjusted.pf_bucket
|
||||
const baseline = rows.find(r => r.pf_bucket && r.pf_iter === 'baseline')
|
||||
return baseline ? baseline.pf_bucket : (forecastBucket || 'Forecast')
|
||||
})()
|
||||
|
||||
const num = (r, col) => (col ? (parseFloat(r[col]) || 0) : 0)
|
||||
const blank = () => ({ value: 0, units: 0, rows: 0 })
|
||||
|
||||
const loads = blank() // the forecast's own segments
|
||||
const basisT = blank() // the comparison bucket
|
||||
const byTag = new Map()
|
||||
const buckets = new Map() // every bucket, for the picker and the markers
|
||||
const baseline = { value: 0, units: 0, rows: 0 }
|
||||
const byTag = new Map()
|
||||
|
||||
for (const r of rows) {
|
||||
const v = num(r, valueCol)
|
||||
const u = num(r, unitsCol)
|
||||
const bucket = hasBuckets ? (r.pf_bucket || '') : null
|
||||
const iter = r.pf_iter
|
||||
if (excl.has(iter)) continue
|
||||
const v = parseFloat(r[valueCol]) || 0
|
||||
const u = unitsCol ? (parseFloat(r[unitsCol]) || 0) : 0
|
||||
|
||||
if (bucket != null) {
|
||||
const b = buckets.get(bucket) || blank()
|
||||
b.value += v; b.units += u; b.rows += 1
|
||||
buckets.set(bucket, b)
|
||||
}
|
||||
|
||||
// Anything outside the forecast is a comparison, never a step.
|
||||
if (hasBuckets && bucket !== fcBucket) {
|
||||
if (basis && bucket === basis) { basisT.value += v; basisT.units += u; basisT.rows += 1 }
|
||||
if (iter === 'baseline') {
|
||||
baseline.value += v; baseline.units += u; baseline.rows += 1
|
||||
continue
|
||||
}
|
||||
if (!hasBuckets && excl.has(r.pf_iter)) continue
|
||||
|
||||
const isLoad = r.pf_iter === 'baseline' || r.pf_iter === 'reference'
|
||||
if (isLoad) { loads.value += v; loads.units += u; loads.rows += 1; continue }
|
||||
|
||||
const meta = logMeta[r.pf_logid] || {}
|
||||
// The tag alone. This used to be label first, to match pf_segment, but every
|
||||
// adjustment now carries a label -- the version's adjustment segment, the
|
||||
// same one for all of them -- so the walk collapsed into a single bar and
|
||||
// no tag was ever shown. The label is the segment; the tag is the initiative.
|
||||
const tag = (meta.tag || '').trim()
|
||||
// Untagged entries collapse into one step, the same as the ledger's
|
||||
// "N untagged adjustments" line in OperationPanel.jsx -- one bar per
|
||||
// untagged scale/recode would otherwise turn a routine cleanup pass into
|
||||
// a wall of identical-looking bars. Tagged entries group case-
|
||||
// insensitively for the same reason the ledger does: "Pull Forward" and
|
||||
// "pull forward" are one initiative, not two, and the most-used spelling
|
||||
// is what gets shown.
|
||||
const key = tag ? `tag:${tag.toLowerCase()}` : 'untagged'
|
||||
const g = byTag.get(key) || {
|
||||
key, tagged: !!tag, value: 0, units: 0, rows: 0, logIds: new Set(), first: r.pf_logid,
|
||||
spellings: tag ? new Map() : null,
|
||||
}
|
||||
const meta = logMeta[r.pf_logid] || {}
|
||||
const tag = (meta.tag || '').trim()
|
||||
const label = tag || (meta.note || '').trim() ||
|
||||
`${(meta.operation || iter || 'adj')}${r.pf_logid != null ? ` #${r.pf_logid}` : ''}`
|
||||
const key = tag ? `tag:${tag}` : `log:${r.pf_logid}`
|
||||
const g = byTag.get(key) ||
|
||||
{ key, label, tagged: !!tag, value: 0, units: 0, rows: 0, logIds: new Set(), first: r.pf_logid }
|
||||
g.value += v; g.units += u; g.rows += 1
|
||||
if (r.pf_logid != null) { g.logIds.add(r.pf_logid); g.first = Math.min(g.first ?? r.pf_logid, r.pf_logid) }
|
||||
if (tag) g.spellings.set(tag, (g.spellings.get(tag) || 0) + 1)
|
||||
byTag.set(key, g)
|
||||
}
|
||||
|
||||
const mid = [...byTag.values()]
|
||||
.sort((a, b) => (a.first ?? 0) - (b.first ?? 0))
|
||||
.map(g => ({
|
||||
...g,
|
||||
label: g.tagged
|
||||
? [...g.spellings.entries()].sort((a, b) => b[1] - a[1])[0]?.[0] || ''
|
||||
: `${g.logIds.size} untagged adjustment${g.logIds.size === 1 ? '' : 's'}`,
|
||||
}))
|
||||
const useBasis = !!basis && basisT.rows > 0
|
||||
const mid = [...byTag.values()].sort((a, b) => (a.first ?? 0) - (b.first ?? 0))
|
||||
|
||||
const out = []
|
||||
if (useBasis) {
|
||||
out.push({
|
||||
key: 'basis', label: basis, kind: 'anchor',
|
||||
delta: basisT.value, start: 0, end: basisT.value,
|
||||
units: basisT.units, rows: basisT.rows, entries: 1,
|
||||
})
|
||||
const gap = loads.value - basisT.value
|
||||
out.push({
|
||||
// "Loads" is our word for a segment import and means nothing to anyone
|
||||
// reading a waterfall. This step is simply where the forecast began
|
||||
// relative to the comparison.
|
||||
//
|
||||
// tagged, though it has no tag: the flag drives an "· untagged" suffix
|
||||
// meant for adjustments nobody grouped into an initiative, and this is
|
||||
// not an adjustment at all.
|
||||
key: 'loads-vs-basis', label: `Starting point vs ${basis}`, kind: 'step',
|
||||
tagged: true,
|
||||
delta: gap, start: basisT.value, end: loads.value,
|
||||
units: loads.units - basisT.units, rows: loads.rows, entries: 1,
|
||||
})
|
||||
} else {
|
||||
out.push({
|
||||
key: 'baseline', label: hasBuckets ? fcBucket + ' loads' : 'Baseline', kind: 'anchor',
|
||||
delta: loads.value, start: 0, end: loads.value,
|
||||
units: loads.units, rows: loads.rows, entries: 1,
|
||||
})
|
||||
}
|
||||
|
||||
let running = loads.value
|
||||
let running = baseline.value
|
||||
const out = [{
|
||||
key: 'baseline', label: 'Baseline', kind: 'anchor',
|
||||
delta: baseline.value, start: 0, end: baseline.value,
|
||||
units: baseline.units, rows: baseline.rows, entries: 1,
|
||||
}]
|
||||
for (const g of mid) {
|
||||
const start = running
|
||||
running += g.value
|
||||
out.push({ ...g, kind: 'step', delta: g.value, start, end: running, entries: g.logIds.size })
|
||||
}
|
||||
|
||||
out.push({
|
||||
key: 'current', label: hasBuckets ? fcBucket : 'Current', kind: 'anchor',
|
||||
key: 'current', label: 'Current', kind: 'anchor',
|
||||
delta: running, start: 0, end: running,
|
||||
units: loads.units + mid.reduce((a, g) => a + (g.units || 0), 0),
|
||||
rows: loads.rows + mid.reduce((a, g) => a + g.rows, 0),
|
||||
units: out.reduce((a, s) => a + (s.kind === 'anchor' ? 0 : s.units || 0), baseline.units),
|
||||
rows: rows.filter(r => !excl.has(r.pf_iter)).length,
|
||||
entries: mid.reduce((a, g) => a + g.logIds.size, 0) + 1,
|
||||
})
|
||||
|
||||
// Every bucket present, so the view can offer them as bases and show the ones
|
||||
// that are not the basis as comparison markers.
|
||||
out.buckets = [...buckets.entries()]
|
||||
.map(([name, t]) => ({ name, ...t }))
|
||||
.sort((a, b) => b.value - a.value)
|
||||
return out
|
||||
}
|
||||
|
||||
// Plot geometry, also pure: given the steps and a canvas size, where does each
|
||||
// bar and label land? Exported so collisions and overflow can be checked.
|
||||
// Break a label to the bar's width, on word boundaries.
|
||||
//
|
||||
// SVG text does not wrap, so the labels were cut at twelve characters and
|
||||
// "04 - Forecast" and "04 - Forecasting" read the same. Character width is
|
||||
// estimated rather than measured -- measuring means a DOM round trip per label
|
||||
// on every render, and at this font a digit is about 0.55em, which is close
|
||||
// enough for a centred label with a bar's width to play with.
|
||||
//
|
||||
// Three lines maximum, and a word longer than the line is left to overflow
|
||||
// rather than broken: a truncated word helps nobody, and the bars are wide
|
||||
// enough that it only happens to things like a pasted part number.
|
||||
function wrapLabel(label, barW, fontSize = 10, maxLines = 3) {
|
||||
const perLine = Math.max(6, Math.floor(barW / (fontSize * 0.55)))
|
||||
const words = String(label || '').split(/\s+/).filter(Boolean)
|
||||
const lines = []
|
||||
let line = ''
|
||||
for (const w of words) {
|
||||
const next = line ? `${line} ${w}` : w
|
||||
if (next.length <= perLine) { line = next; continue }
|
||||
if (line) lines.push(line)
|
||||
line = w
|
||||
if (lines.length === maxLines) break
|
||||
}
|
||||
if (line && lines.length < maxLines) lines.push(line)
|
||||
if (!lines.length) return ['']
|
||||
// anything that did not fit is marked on the last line rather than dropped
|
||||
const used = lines.join(' ').length
|
||||
if (used < String(label).replace(/\s+/g, ' ').length) {
|
||||
lines[lines.length - 1] = `${lines[lines.length - 1]}…`
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
export function layoutSteps(steps, width, H = 340, PAD = { t: 24, r: 16, b: 84, l: 68 }) {
|
||||
export function layoutSteps(steps, width, H = 340, PAD = { t: 24, r: 16, b: 64, l: 68 }) {
|
||||
const plotW = Math.max(120, width - PAD.l - PAD.r)
|
||||
const plotH = H - PAD.t - PAD.b
|
||||
const values = steps.flatMap(s => [s.start, s.end])
|
||||
@ -275,26 +127,11 @@ export function layoutSteps(steps, width, H = 340, PAD = { t: 24, r: 16, b: 84,
|
||||
export default function BridgeView({
|
||||
open, onClose, tableRef, viewerRef, logMeta = {},
|
||||
valueCol, unitsCol, colMeta = [], slices = [],
|
||||
excludeIters = ['reference'], versionName, forecastBucket,
|
||||
excludeIters = ['reference'], versionName,
|
||||
}) {
|
||||
const hasSelection = slices.length > 0
|
||||
// 'selection' | 'filtered'. There is no whole-version scope: the pivot's
|
||||
// filter is part of what the figures mean, and dropping it moves one end of
|
||||
// the walk and not the other. On the version this was built against the grid
|
||||
// is scoped `sseas_e <= 2027`, and the rows outside it -- orders with a null
|
||||
// ship date, plus a tail beyond 2027 -- sit entirely in the forecast loads,
|
||||
// with no counterpart in Plan. Ignoring the filter therefore added $9.4M to
|
||||
// the endpoint and nothing to the basis, and the bridge said nothing about it.
|
||||
// 'selection' | 'filtered' | 'all'
|
||||
const [scope, setScope] = useState(hasSelection ? 'selection' : 'filtered')
|
||||
// Whether the user has picked a scope button themselves. Bridge mounts once,
|
||||
// up front, before anything has ever been selected on the pivot — so the
|
||||
// useState initializer above always seeds 'filtered' and nothing follows a
|
||||
// selection made afterward. Until the user overrides it, scope tracks the
|
||||
// live selection instead.
|
||||
const [scopeTouched, setScopeTouched] = useState(false)
|
||||
// Which bucket the walk starts from. Plan when building a forecast, Prior Year
|
||||
// when building the AOP; empty means show the forecast's own composition.
|
||||
const [basis, setBasis] = useState(() => localStorage.getItem('pf_bridge_basis') || '')
|
||||
const [asTable, setAsTable] = useState(false)
|
||||
const [steps, setSteps] = useState(null)
|
||||
const [loading, setLoading] = useState(false)
|
||||
@ -302,10 +139,6 @@ export default function BridgeView({
|
||||
const [hover, setHover] = useState(null)
|
||||
const [width, setWidth] = useState(880)
|
||||
const boxRef = useRef(null)
|
||||
// Where the panel has been dragged to, as an offset from centre. It opens
|
||||
// centred each time rather than wherever it was last left.
|
||||
const [pos, setPos] = useState({ x: 0, y: 0 })
|
||||
const dragRef = useRef(null)
|
||||
|
||||
// Build the steps: baseline anchor, one floating step per tag, current anchor.
|
||||
const compute = useCallback(async () => {
|
||||
@ -317,52 +150,17 @@ export default function BridgeView({
|
||||
// The union of the selected slices — the same reach an operation would
|
||||
// have. Perspective view filters are AND-only, so each slice needs its own
|
||||
// view; rows matching more than one slice are counted once.
|
||||
//
|
||||
// pf_bucket/pf_segment are deliberately left out here, even though a
|
||||
// click can carry one (e.g. the column axis split by bucket): the walk
|
||||
// needs rows from every bucket, not just the one the slice landed on,
|
||||
// or there is nothing left for "Compare to" to compare against and
|
||||
// every basis collapses to the one bucket fetched. Which bucket the
|
||||
// walk starts and ends on is decided inside buildSteps, from
|
||||
// forecastBucket, not by filtering it out of the fetch.
|
||||
const dimNames = new Set(colMeta.filter(c => c.role === 'dimension').map(c => c.cname))
|
||||
const dateNames = new Set(colMeta.filter(c => c.role === 'date').map(c => c.cname))
|
||||
const schema = await tableRef.current.schema()
|
||||
const typed = (col, val) => {
|
||||
switch (schema[col]) {
|
||||
case 'integer': case 'float': return Number(val)
|
||||
case 'boolean': return val === true || val === 'true'
|
||||
case 'date': case 'datetime': return Number(val)
|
||||
default: return String(val)
|
||||
}
|
||||
}
|
||||
// The pivot's own filter (e.g. sseas_e <= 2027) is not part of a
|
||||
// clicked slice — perspective-click reports only the cell's own
|
||||
// dimensions — so it has to be read off the viewer and applied
|
||||
// alongside every per-slice view, the same reason Forecast.jsx's
|
||||
// ledger totals do this. Without it, a grid scoped narrower than the
|
||||
// whole version bridges rows the grid is hiding.
|
||||
let viewFilter = []
|
||||
let expressions = {}
|
||||
if (viewerRef?.current) {
|
||||
const cfg = await viewerRef.current.save()
|
||||
viewFilter = (cfg.filter || []).filter(f => Array.isArray(f) && f.length >= 2)
|
||||
expressions = cfg.expressions || {}
|
||||
}
|
||||
const seen = new Set()
|
||||
rows = []
|
||||
for (const sl of slices) {
|
||||
const sliceFilters = Object.entries(sl)
|
||||
.filter(([c]) => (dimNames.has(c) || dateNames.has(c)) && !viewFilter.some(vf => vf[0] === c))
|
||||
// A null is filtered with `is null`, not `== null`: the engine
|
||||
// matches on type, and a null threshold against a string column is
|
||||
// the same dropped predicate that widens a slice back toward its
|
||||
// parent group — see totalsFor in Forecast.jsx for the same fix.
|
||||
.map(([c, v]) => (v === null ? [c, 'is null'] : [c, '==', typed(c, v)]))
|
||||
if (!sliceFilters.length) continue
|
||||
const viewCfg = { filter: [...viewFilter, ...sliceFilters] }
|
||||
if (Object.keys(expressions).length) viewCfg.expressions = expressions
|
||||
const view = await tableRef.current.view(viewCfg)
|
||||
const f = [
|
||||
...Object.entries(sl).filter(([c]) => dimNames.has(c)).map(([c, v]) => [c, '==', v]),
|
||||
...Object.entries(sl).filter(([c]) => dateNames.has(c)).map(([c, v]) => [c, '==', Number(v)]),
|
||||
]
|
||||
if (!f.length) continue
|
||||
const view = await tableRef.current.view({ filter: f })
|
||||
const part = await view.to_json()
|
||||
await view.delete()
|
||||
for (const r of part) {
|
||||
@ -373,80 +171,30 @@ export default function BridgeView({
|
||||
}
|
||||
} else {
|
||||
let filter = []
|
||||
// Expression columns have to come with the filter that uses them: a
|
||||
// filter can name a column that exists only as an expression, and a view
|
||||
// built without it cannot resolve the column and fails outright.
|
||||
let expressions = {}
|
||||
if (viewerRef?.current) {
|
||||
if (scope === 'filtered' && viewerRef?.current) {
|
||||
const cfg = await viewerRef.current.save()
|
||||
filter = (cfg.filter || []).filter(f => Array.isArray(f) && f.length >= 2)
|
||||
expressions = cfg.expressions || {}
|
||||
}
|
||||
const viewCfg = {}
|
||||
if (filter.length) viewCfg.filter = filter
|
||||
if (Object.keys(expressions).length) viewCfg.expressions = expressions
|
||||
const view = await tableRef.current.view(viewCfg)
|
||||
const view = await tableRef.current.view(filter.length ? { filter } : {})
|
||||
rows = await view.to_json()
|
||||
await view.delete()
|
||||
}
|
||||
|
||||
setSteps(buildSteps(rows, {
|
||||
valueCol, unitsCol, logMeta, excludeIters, basis: basis || null, forecastBucket,
|
||||
}))
|
||||
setSteps(buildSteps(rows, { valueCol, unitsCol, logMeta, excludeIters }))
|
||||
} catch (err) {
|
||||
setError(err.message || String(err))
|
||||
setSteps(null)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [tableRef, viewerRef, scope, logMeta, valueCol, unitsCol, excludeIters, slices, colMeta, basis, forecastBucket])
|
||||
}, [tableRef, viewerRef, scope, logMeta, valueCol, unitsCol, excludeIters, slices, colMeta])
|
||||
|
||||
// Follow the pivot's selection until the user chooses a scope explicitly.
|
||||
useEffect(() => {
|
||||
if (scopeTouched) return
|
||||
setScope(hasSelection ? 'selection' : 'filtered')
|
||||
}, [hasSelection, scopeTouched])
|
||||
|
||||
// Once the user has picked "Selection" -- themselves or via the effect above
|
||||
// -- and the selection then clears, there is nothing left to show under it;
|
||||
// fall back regardless of whether the choice was manual.
|
||||
useEffect(() => {
|
||||
if (!hasSelection && scope === 'selection') { setScope('filtered'); setScopeTouched(false) }
|
||||
if (!hasSelection && scope === 'selection') setScope('filtered')
|
||||
}, [hasSelection, scope])
|
||||
|
||||
useEffect(() => { try { localStorage.setItem('pf_bridge_basis', basis) } catch {} }, [basis])
|
||||
useEffect(() => { if (open) compute() }, [open, compute])
|
||||
|
||||
useEffect(() => { if (open) setPos({ x: 0, y: 0 }) }, [open])
|
||||
|
||||
// Escape closes it. There is no backdrop to click any more -- see below.
|
||||
useEffect(() => {
|
||||
if (!open) return
|
||||
const onKey = (e) => { if (e.key === 'Escape') onClose() }
|
||||
window.addEventListener('keydown', onKey)
|
||||
return () => window.removeEventListener('keydown', onKey)
|
||||
}, [open, onClose])
|
||||
|
||||
// Drag by the header. Pointer capture keeps the drag alive when the cursor
|
||||
// outruns the panel; the close button is excluded so clicking it still closes.
|
||||
const startDrag = (e) => {
|
||||
if (e.button !== 0 || e.target.closest('button')) return
|
||||
e.currentTarget.setPointerCapture(e.pointerId)
|
||||
dragRef.current = { px: e.clientX, py: e.clientY, x: pos.x, y: pos.y }
|
||||
}
|
||||
const moveDrag = (e) => {
|
||||
const d = dragRef.current
|
||||
if (!d) return
|
||||
// keep the header on screen, or there is nothing left to drag it back by
|
||||
const maxX = window.innerWidth / 2 - 40
|
||||
const maxY = window.innerHeight / 2 - 20
|
||||
setPos({
|
||||
x: Math.max(-maxX, Math.min(maxX, d.x + e.clientX - d.px)),
|
||||
y: Math.max(-maxY, Math.min(maxY, d.y + e.clientY - d.py)),
|
||||
})
|
||||
}
|
||||
const endDrag = () => { dragRef.current = null }
|
||||
|
||||
useEffect(() => {
|
||||
if (!open || !boxRef.current) return
|
||||
const ro = new ResizeObserver(([e]) => setWidth(Math.max(420, e.contentRect.width)))
|
||||
@ -463,25 +211,18 @@ export default function BridgeView({
|
||||
const ticks = niceTicks(yMin, yMax, 5)
|
||||
|
||||
return (
|
||||
// Not modal. The overlay passes clicks through, so the pivot stays usable
|
||||
// with the bridge open -- moving it aside to read the grid under it, or
|
||||
// changing the selection it is scoped to, which recomputes it. It used to
|
||||
// be a dimmed backdrop that closed on click, which made the panel immovable
|
||||
// and the pivot unreachable until it was dismissed.
|
||||
<div className="fixed inset-0 z-50 flex items-center justify-center pointer-events-none">
|
||||
<div className="pointer-events-auto bg-white rounded-lg shadow-2xl border border-gray-200 w-full max-w-5xl mx-4 flex flex-col max-h-[88vh]"
|
||||
style={{ transform: `translate(${pos.x}px, ${pos.y}px)` }}>
|
||||
<div className="fixed inset-0 z-50 flex items-center justify-center bg-black/40" onClick={onClose}>
|
||||
<div className="bg-white rounded-lg shadow-xl w-full max-w-5xl mx-4 flex flex-col max-h-[88vh]"
|
||||
onClick={e => e.stopPropagation()}>
|
||||
|
||||
<div className="flex items-center justify-between px-5 py-3 border-b border-gray-200 shrink-0 cursor-move select-none touch-none"
|
||||
onPointerDown={startDrag} onPointerMove={moveDrag}
|
||||
onPointerUp={endDrag} onPointerCancel={endDrag}>
|
||||
<div className="flex items-center justify-between px-5 py-3 border-b border-gray-200 shrink-0">
|
||||
<div className="flex items-baseline gap-2">
|
||||
<span className="font-medium text-gray-700 text-sm">Bridge</span>
|
||||
{versionName && <span className="text-gray-600 text-xs">{versionName}</span>}
|
||||
<span className="text-gray-600 text-xs">
|
||||
· {scope === 'selection'
|
||||
? `${slices.length} selected slice${slices.length === 1 ? '' : 's'}`
|
||||
: "pivot's filters"}
|
||||
: scope === 'filtered' ? "pivot's filters" : 'whole version'}
|
||||
</span>
|
||||
</div>
|
||||
<button onClick={onClose} className="text-gray-600 hover:text-gray-800 text-lg leading-none">×</button>
|
||||
@ -496,8 +237,9 @@ export default function BridgeView({
|
||||
hasSelection ? 'The slices selected in the operation panel'
|
||||
: 'Select one or more pivot rows first'],
|
||||
['filtered', "Pivot's filters", 'Everything the pivot currently shows'],
|
||||
['all', 'Whole version', 'Every row in the version, filters ignored'],
|
||||
].map(([v, l, title]) => (
|
||||
<button key={v} onClick={() => { setScope(v); setScopeTouched(true) }} title={title}
|
||||
<button key={v} onClick={() => setScope(v)} title={title}
|
||||
disabled={v === 'selection' && !hasSelection}
|
||||
className={`px-3 py-1 disabled:opacity-40 disabled:cursor-not-allowed ${
|
||||
scope === v ? 'bg-blue-600 text-white' : 'bg-white text-gray-700 hover:bg-gray-50'}`}>
|
||||
@ -505,23 +247,6 @@ export default function BridgeView({
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
{/* Offered only once something carries a bucket, and only listing buckets
|
||||
other than the forecast itself -- a walk from Forecast to Forecast is
|
||||
the composition view, which is what the empty option gives. */}
|
||||
{(steps?.buckets || []).some(b => b.name && b.name !== 'Forecast') && (
|
||||
<>
|
||||
<div className="w-px h-4 bg-gray-200" />
|
||||
<span className="text-gray-600">Compare to</span>
|
||||
<select value={basis} onChange={e => setBasis(e.target.value)}
|
||||
className="border border-gray-200 rounded px-2 py-1 text-gray-700 bg-white">
|
||||
<option value="">nothing — show composition</option>
|
||||
{(steps?.buckets || [])
|
||||
.filter(b => b.name && b.name !== 'Forecast')
|
||||
.map(b => <option key={b.name} value={b.name}>{b.name}</option>)}
|
||||
</select>
|
||||
</>
|
||||
)}
|
||||
|
||||
<div className="w-px h-4 bg-gray-200" />
|
||||
<button onClick={() => setAsTable(t => !t)}
|
||||
className="border border-gray-200 rounded px-2 py-1 text-gray-700 hover:bg-gray-50">
|
||||
@ -561,8 +286,8 @@ export default function BridgeView({
|
||||
{ticks.map(t => (
|
||||
<g key={t}>
|
||||
<line x1={PAD.l} x2={PAD.l + plotW} y1={y(t)} y2={y(t)}
|
||||
style={{ stroke: t === 0 ? AXIS : GRID }} strokeWidth={t === 0 ? 1.5 : 1} />
|
||||
<text x={PAD.l - 8} y={y(t) + 3} textAnchor="end" fontSize="10" style={{ fill: INK_DIM }}>
|
||||
stroke={t === 0 ? '#d1d5db' : GRID} strokeWidth={t === 0 ? 1.5 : 1} />
|
||||
<text x={PAD.l - 8} y={y(t) + 3} textAnchor="end" fontSize="10" fill={INK_DIM}>
|
||||
{fmtAxis(t)}
|
||||
</text>
|
||||
</g>
|
||||
@ -577,8 +302,6 @@ export default function BridgeView({
|
||||
const h = Math.max(2, bot - top)
|
||||
const x = xOf(i)
|
||||
const on = hover?.key === s.key
|
||||
const labelLines = wrapLabel(s.label, barW)
|
||||
const subY = PAD.t + plotH + 16 + labelLines.length * 11 + 2
|
||||
return (
|
||||
<g key={s.key}
|
||||
onMouseEnter={() => setHover({ ...s, x: x + barW / 2, y: top })}
|
||||
@ -586,31 +309,31 @@ export default function BridgeView({
|
||||
{/* connector to the next bar, drawn behind */}
|
||||
{i < steps.length - 1 && (
|
||||
<line x1={x + barW} x2={xOf(i + 1)} y1={y(s.end)} y2={y(s.end)}
|
||||
style={{ stroke: AXIS }} strokeWidth="1" strokeDasharray="2 2" />
|
||||
stroke="#cbd5e1" strokeWidth="1" strokeDasharray="2 2" />
|
||||
)}
|
||||
{/* hit target larger than the mark */}
|
||||
<rect x={x - 6} y={PAD.t} width={barW + 12} height={plotH} fill="transparent" />
|
||||
<rect x={x} y={top} width={barW} height={h} rx="4" fill={fill}
|
||||
opacity={on ? 1 : 0.92}
|
||||
style={{ stroke: PANEL }} strokeWidth="2" />
|
||||
stroke="#ffffff" strokeWidth="2" />
|
||||
{/* direct label: few bars, so every one is labelled */}
|
||||
<text x={x + barW / 2} y={top - 6} textAnchor="middle" fontSize="10"
|
||||
style={{ fill: INK }} fontWeight="500">
|
||||
fill={INK} fontWeight="500">
|
||||
{isAnchor ? fmt(s.end, 0) : fmtSigned(s.delta, 0)}
|
||||
</text>
|
||||
<text x={x + barW / 2} y={PAD.t + plotH + 16} textAnchor="middle" fontSize="10" style={{ fill: INK }}>
|
||||
{labelLines.map((ln, li) => (
|
||||
<tspan key={li} x={x + barW / 2} dy={li === 0 ? 0 : 11}>{ln}</tspan>
|
||||
))}
|
||||
<text x={x + barW / 2} y={PAD.t + plotH + 16} textAnchor="middle" fontSize="10" fill={INK}>
|
||||
{s.label.length > 12 ? `${s.label.slice(0, 11)}…` : s.label}
|
||||
</text>
|
||||
{/* below the label, however many lines it took. Skipped for the
|
||||
untagged bucket -- its label already reads "N untagged
|
||||
adjustments", so a ×N here would just repeat the count. */}
|
||||
{!isAnchor && s.tagged && s.entries > 1 && (
|
||||
<text x={x + barW / 2} y={subY} textAnchor="middle" fontSize="9" style={{ fill: INK_DIM }}>
|
||||
{!isAnchor && s.entries > 1 && (
|
||||
<text x={x + barW / 2} y={PAD.t + plotH + 29} textAnchor="middle" fontSize="9" fill={INK_DIM}>
|
||||
×{s.entries}
|
||||
</text>
|
||||
)}
|
||||
{!s.tagged && !isAnchor && (
|
||||
<text x={x + barW / 2} y={PAD.t + plotH + 29} textAnchor="middle" fontSize="9" fill={INK_DIM}>
|
||||
untagged
|
||||
</text>
|
||||
)}
|
||||
</g>
|
||||
)
|
||||
})}
|
||||
@ -653,7 +376,9 @@ export default function BridgeView({
|
||||
<tbody>
|
||||
{steps.map(s => (
|
||||
<tr key={s.key} className="border-b border-gray-100">
|
||||
<td className="py-1.5 pr-3 text-gray-800">{s.label}</td>
|
||||
<td className="py-1.5 pr-3 text-gray-800">
|
||||
{s.label}{!s.tagged && s.kind === 'step' && <span className="text-gray-600"> · untagged</span>}
|
||||
</td>
|
||||
<td className="py-1.5 px-2 text-right font-mono tabular-nums text-gray-800">
|
||||
{s.kind === 'anchor' ? fmt(s.end) : fmtSigned(s.delta)}
|
||||
</td>
|
||||
|
||||
@ -1,108 +0,0 @@
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
|
||||
// One axis's collapse control, as a stepper.
|
||||
//
|
||||
// Rows and columns are the same thing — a depth in ViewConfig, counting the
|
||||
// levels to show — so one component draws both.
|
||||
//
|
||||
// It used to be one button per level, which is N+1 controls that grow with the
|
||||
// pivot: a 5-by-3 layout rendered ten of them, and the active level was a blue
|
||||
// button somewhere among the greys, so reading the current state meant scanning.
|
||||
// A stepper is three controls whatever the depth, and says where you are rather
|
||||
// than making you find it. The full list is still one click away, for jumping
|
||||
// rather than stepping.
|
||||
export default function DepthButtons({ label, levels, depth, onPick, totalLabel = 'Total', children }) {
|
||||
const [open, setOpen] = useState(false)
|
||||
const boxRef = useRef(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!open) return
|
||||
const onDown = (e) => { if (!boxRef.current?.contains(e.target)) setOpen(false) }
|
||||
const onKey = (e) => { if (e.key === 'Escape') setOpen(false) }
|
||||
// Capture phase: the pivot lives in a shadow root and retargets its events,
|
||||
// so a bubbled listener sees the host rather than the real target.
|
||||
document.addEventListener('mousedown', onDown, true)
|
||||
window.addEventListener('keydown', onKey)
|
||||
return () => {
|
||||
document.removeEventListener('mousedown', onDown, true)
|
||||
window.removeEventListener('keydown', onKey)
|
||||
}
|
||||
}, [open])
|
||||
|
||||
const max = levels.length
|
||||
// depth is null until the first layout has been read back. Showing the last
|
||||
// level would be a guess; an em dash is honest and the arrows still work.
|
||||
const known = depth != null
|
||||
const current = !known ? '—' : (depth === 0 ? totalLabel : levels[depth - 1])
|
||||
const step = (d) => onPick(Math.min(max, Math.max(0, d)))
|
||||
// Capped so one long field name cannot push the rest of the toolbar off;
|
||||
// past the cap the label truncates, with the full name in the title.
|
||||
const widest = Math.min(18, Math.max(...[totalLabel, ...levels].map(l => String(l).length)))
|
||||
|
||||
// One joined control, in the style of the operation panel's
|
||||
// Scale | Recode | Clone. The arrows used to be separate outlined boxes a
|
||||
// few pixels wide, which in dark mode were barely visible and looked the
|
||||
// same enabled or disabled; joined, the axis reads as one object and the
|
||||
// arrows are halves of it rather than specks beside it.
|
||||
const seg = 'px-1.5 py-0.5 leading-none bg-white transition-colors ' +
|
||||
'text-gray-600 hover:bg-gray-100 disabled:text-gray-300 disabled:hover:bg-white disabled:cursor-default'
|
||||
|
||||
return (
|
||||
<div ref={boxRef} className="relative flex items-center gap-1.5">
|
||||
<span className="pf-section-label">{label}</span>
|
||||
|
||||
<div className="inline-flex rounded border border-gray-300 divide-x divide-gray-300 overflow-hidden">
|
||||
<button className={seg} disabled={!known || depth <= 0}
|
||||
title={`Collapse ${label.toLowerCase()} by one level`}
|
||||
onClick={() => step((depth ?? 0) - 1)}>◂</button>
|
||||
|
||||
{/* Sized to the axis's longest level, not the current one. Sized to
|
||||
its content it resized on every step, so the ▸ moved out from
|
||||
under the cursor between clicks. */}
|
||||
<button onClick={() => setOpen(o => !o)}
|
||||
title={`${label}: showing ${current}`}
|
||||
style={{ width: `calc(${widest}ch + 1rem)` }}
|
||||
className={`px-2 py-0.5 truncate text-left transition-colors
|
||||
${open ? 'bg-blue-50 text-blue-700' : 'bg-white text-gray-700 hover:bg-gray-100'}`}>
|
||||
{current}
|
||||
</button>
|
||||
|
||||
<button className={seg} disabled={!known || depth >= max}
|
||||
title={`Expand ${label.toLowerCase()} by one level`}
|
||||
onClick={() => step((depth ?? 0) + 1)}>▸</button>
|
||||
</div>
|
||||
|
||||
{children}
|
||||
|
||||
{open && (
|
||||
<div className="absolute top-full left-0 mt-1 z-50 min-w-[10rem] bg-white border
|
||||
border-gray-200 rounded shadow-lg py-1 text-xs">
|
||||
{Array.from({ length: max + 1 }, (_, n) => (
|
||||
<button key={n} onClick={() => { onPick(n); setOpen(false) }}
|
||||
title={n === 0
|
||||
? `Collapse ${label.toLowerCase()} to a single total`
|
||||
: `Show ${label.toLowerCase()} down to ${levels.slice(0, n).join(' › ')}`}
|
||||
className={`block w-full text-left px-3 py-1 truncate
|
||||
${depth === n ? 'bg-blue-50 text-blue-700' : 'text-gray-700 hover:bg-gray-50'}`}>
|
||||
{n === 0 ? totalLabel : levels[n - 1]}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
// An on/off control that looks pressed when on — the same state the toolbar's
|
||||
// Operations button uses for "open". Replaces the native checkboxes, which
|
||||
// drew in the light style on the dark theme and were the brightest thing on
|
||||
// the toolbar.
|
||||
export function ToggleChip({ on, onChange, title, children }) {
|
||||
return (
|
||||
<button type="button" aria-pressed={on} title={title} onClick={() => onChange(!on)}
|
||||
className={`border rounded px-2 py-0.5 transition-colors ${
|
||||
on ? 'pf-pressed' : 'border-gray-300 bg-white text-gray-500 hover:bg-gray-100'}`}>
|
||||
{children}
|
||||
</button>
|
||||
)
|
||||
}
|
||||
@ -1,197 +0,0 @@
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
|
||||
// The pivot layout, as one control rather than a row of equal chips.
|
||||
//
|
||||
// Two lists, because a layout is one of two different things: Published, which
|
||||
// everyone on this forecast sees and only its owner or an admin may change, and
|
||||
// Mine, which nobody else lists. Everything you cannot write is still there to
|
||||
// apply -- it is only the write controls that are withheld, so a click never
|
||||
// answers 403.
|
||||
export default function LayoutMenu({
|
||||
layouts = [], activeLayoutId, dirty,
|
||||
onApply, onSave, onSaveAs, onSetVisibility, onSetDefault, onRename, onDelete, onReset
|
||||
}) {
|
||||
const [open, setOpen] = useState(false)
|
||||
const [saveName, setSaveName] = useState('')
|
||||
const [savePublic, setSavePublic] = useState(false)
|
||||
const [renaming, setRenaming] = useState(null)
|
||||
const [renameTo, setRenameTo] = useState('')
|
||||
const boxRef = useRef(null)
|
||||
|
||||
// Close on an outside click or Escape. Capture phase: the pivot lives in a
|
||||
// shadow root and retargets its events, so a bubbled listener on document
|
||||
// sees the host rather than the real target and cannot tell inside from out.
|
||||
useEffect(() => {
|
||||
if (!open) return
|
||||
const onDown = (e) => { if (!boxRef.current?.contains(e.target)) setOpen(false) }
|
||||
const onKey = (e) => { if (e.key === 'Escape') setOpen(false) }
|
||||
document.addEventListener('mousedown', onDown, true)
|
||||
window.addEventListener('keydown', onKey)
|
||||
return () => {
|
||||
document.removeEventListener('mousedown', onDown, true)
|
||||
window.removeEventListener('keydown', onKey)
|
||||
}
|
||||
}, [open])
|
||||
|
||||
const active = layouts.find(l => l.id === activeLayoutId)
|
||||
const published = layouts.filter(l => l.visibility === 'published')
|
||||
const mine = layouts.filter(l => l.visibility === 'private')
|
||||
|
||||
const submitSaveAs = () => {
|
||||
const name = saveName.trim()
|
||||
if (!name) return
|
||||
onSaveAs(name, savePublic ? 'published' : 'private')
|
||||
setSaveName('')
|
||||
setOpen(false)
|
||||
}
|
||||
|
||||
const submitRename = (l) => {
|
||||
const name = renameTo.trim()
|
||||
if (name && name !== l.name) onRename(l, name)
|
||||
setRenaming(null)
|
||||
}
|
||||
|
||||
const iconBtn = 'px-1 rounded text-gray-400 hover:text-gray-700 hover:bg-gray-100 leading-none'
|
||||
|
||||
const row = (l) => (
|
||||
<div key={l.id}
|
||||
className={`group flex items-center gap-1 px-2 py-1 rounded cursor-pointer
|
||||
${l.id === activeLayoutId ? 'bg-blue-50 text-blue-700' : 'hover:bg-gray-50 text-gray-700'}`}
|
||||
onClick={() => { if (renaming !== l.id) { onApply(l); setOpen(false) } }}>
|
||||
|
||||
{renaming === l.id ? (
|
||||
<input autoFocus value={renameTo} onClick={e => e.stopPropagation()}
|
||||
onChange={e => setRenameTo(e.target.value)}
|
||||
onBlur={() => submitRename(l)}
|
||||
onKeyDown={e => {
|
||||
if (e.key === 'Enter') submitRename(l)
|
||||
if (e.key === 'Escape') setRenaming(null)
|
||||
}}
|
||||
className="flex-1 min-w-0 border border-blue-300 rounded px-1 py-0 outline-none bg-white" />
|
||||
) : (
|
||||
<>
|
||||
<span className="flex-1 min-w-0 truncate">{l.name}</span>
|
||||
{l.is_default && (
|
||||
<span title="Applied when this forecast is opened for the first time"
|
||||
className="text-amber-500 shrink-0">★</span>
|
||||
)}
|
||||
{l.scope === 'source' && (
|
||||
<span title="Applies to every forecast of this source"
|
||||
className="text-gray-300 shrink-0" style={{fontSize:'9px'}}>ALL</span>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* Withheld rather than offered-and-refused: the server allows a write only
|
||||
to the owner or an admin, and can_edit says which this is. */}
|
||||
{l.can_edit && renaming !== l.id && (
|
||||
<span className="hidden group-hover:flex items-center gap-0.5 shrink-0"
|
||||
onClick={e => e.stopPropagation()}>
|
||||
<button className={iconBtn} title="Rename"
|
||||
onClick={() => { setRenaming(l.id); setRenameTo(l.name) }}>✎</button>
|
||||
{l.visibility === 'private' ? (
|
||||
<button className={iconBtn} title="Publish — everyone on this forecast sees it"
|
||||
onClick={() => onSetVisibility(l, 'published')}>↑</button>
|
||||
) : (
|
||||
<>
|
||||
{!l.is_default && (
|
||||
<button className={iconBtn} title="Apply this when the forecast is first opened"
|
||||
onClick={() => onSetDefault(l)}>★</button>
|
||||
)}
|
||||
<button className={iconBtn} title="Unpublish — keep it, but only for you"
|
||||
onClick={() => onSetVisibility(l, 'private')}>↓</button>
|
||||
</>
|
||||
)}
|
||||
<button className={`${iconBtn} hover:text-red-500`} title="Delete"
|
||||
onClick={() => onDelete(l)}>×</button>
|
||||
</span>
|
||||
)}
|
||||
|
||||
{!l.can_edit && (
|
||||
<span className="hidden group-hover:inline text-gray-300 shrink-0 truncate"
|
||||
style={{fontSize:'9px'}} title={`Published by ${l.owner}`}>{l.owner}</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
|
||||
return (
|
||||
<div ref={boxRef} className="relative flex items-center gap-1.5">
|
||||
<span className="pf-section-label">Layout</span>
|
||||
|
||||
<button onClick={() => setOpen(o => !o)}
|
||||
className={`flex items-center gap-1 border rounded px-2 py-0.5 max-w-[22rem] transition-colors
|
||||
${open ? 'border-blue-300 text-blue-700 bg-blue-50'
|
||||
: 'border-gray-200 text-gray-600 hover:border-gray-400'}`}>
|
||||
<span className="truncate">{active ? active.name : 'Unsaved'}</span>
|
||||
{/* Always rendered, hidden when clean, so the button does not widen
|
||||
by a dot the moment the pivot changes. */}
|
||||
<span title={dirty ? 'Changed since it was saved' : undefined}
|
||||
className={`text-amber-500 leading-none ${dirty ? '' : 'invisible'}`}>•</span>
|
||||
<span className="text-gray-400 leading-none">▾</span>
|
||||
</button>
|
||||
|
||||
{/* Save is the common action and stays outside the menu, but only when
|
||||
there is something to save it to and it is yours to overwrite. Its
|
||||
slot is held while the layout is clean: appearing and vanishing, it
|
||||
shoved the Rows and Columns controls sideways under the cursor. */}
|
||||
{active?.can_edit && (
|
||||
<button onClick={() => onSave(active)} disabled={!dirty}
|
||||
aria-hidden={!dirty} tabIndex={dirty ? 0 : -1}
|
||||
className={`border border-blue-200 text-blue-600 hover:text-blue-800 rounded px-2 py-0.5
|
||||
${dirty ? '' : 'invisible'}`}>
|
||||
Save
|
||||
</button>
|
||||
)}
|
||||
|
||||
{open && (
|
||||
<div className="absolute top-full left-0 mt-1 z-50 w-72 bg-white border border-gray-200
|
||||
rounded shadow-lg py-1 text-xs">
|
||||
|
||||
{published.length > 0 && (
|
||||
<>
|
||||
<div className="px-2 pt-1 pb-0.5 text-gray-400 uppercase tracking-wide"
|
||||
style={{fontSize:'9px'}}>Published</div>
|
||||
<div className="px-1">{published.map(row)}</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
{mine.length > 0 && (
|
||||
<>
|
||||
<div className="px-2 pt-2 pb-0.5 text-gray-400 uppercase tracking-wide"
|
||||
style={{fontSize:'9px'}}>Mine</div>
|
||||
<div className="px-1">{mine.map(row)}</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
{!layouts.length && (
|
||||
<div className="px-3 py-2 text-gray-400">No saved layouts yet</div>
|
||||
)}
|
||||
|
||||
<div className="border-t border-gray-100 mt-1 pt-1 px-2">
|
||||
<div className="flex items-center gap-1 py-1">
|
||||
<input value={saveName} onChange={e => setSaveName(e.target.value)}
|
||||
onKeyDown={e => { if (e.key === 'Enter') submitSaveAs() }}
|
||||
placeholder="Save current view as…"
|
||||
className="flex-1 min-w-0 border border-gray-300 rounded px-1.5 py-0.5
|
||||
outline-none focus:border-blue-400" />
|
||||
<button onClick={submitSaveAs} disabled={!saveName.trim()}
|
||||
className="text-blue-600 hover:text-blue-800 px-1 disabled:opacity-30">Save</button>
|
||||
</div>
|
||||
<label className="flex items-center gap-1.5 py-0.5 text-gray-500 cursor-pointer">
|
||||
<input type="checkbox" checked={savePublic}
|
||||
onChange={e => setSavePublic(e.target.checked)} />
|
||||
Publish to everyone on this forecast
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="border-t border-gray-100 mt-1 pt-1 px-2 pb-0.5">
|
||||
<button onClick={() => { onReset(); setOpen(false) }}
|
||||
className="text-gray-400 hover:text-gray-700 py-0.5">
|
||||
Reset to a blank pivot
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,22 +0,0 @@
|
||||
// Says why everything on the page is greyed out.
|
||||
//
|
||||
// The disabling itself is one `<fieldset disabled>` around the page rather than
|
||||
// a flag threaded through every control -- Baseline alone has forty-five of
|
||||
// them, and the one that got missed would be the one that mattered. A disabled
|
||||
// fieldset takes its descendant form controls out with it natively, so nothing
|
||||
// has to remember to opt in.
|
||||
//
|
||||
// None of this is the permission. The routes carry requireAdmin; this only
|
||||
// stops an account being offered a button that would answer 403.
|
||||
export default function ReadOnlyNotice({ what }) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 px-3 py-1.5 bg-amber-50 border-b border-amber-200 text-amber-800 text-xs shrink-0">
|
||||
<svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor"
|
||||
strokeWidth="2" strokeLinecap="round" className="shrink-0">
|
||||
<rect x="4" y="11" width="16" height="10" rx="2" />
|
||||
<path d="M8 11V7a4 4 0 0 1 8 0v4" />
|
||||
</svg>
|
||||
Read-only — {what} is managed by an administrator.
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@ -32,41 +32,7 @@ const NAV = [
|
||||
},
|
||||
]
|
||||
|
||||
// The forecasts are listed here, above the steps, rather than offered as a
|
||||
// dropdown in the status bar. Two reasons: a scenario is a thing you open, not
|
||||
// a setting you change, and the status bar is otherwise read-only, so a control
|
||||
// sitting in it did not read as one.
|
||||
//
|
||||
// Not nested under the Forecast button, which is where this started: Baseline
|
||||
// works on the same version, so hanging it off one step would either
|
||||
// misrepresent the scope or duplicate the list under two. It reads
|
||||
// "what am I working on" and then "what step am I on".
|
||||
//
|
||||
// The source is deliberately absent. `gs.osm_skinny` means nothing to someone
|
||||
// editing a forecast, and a version already carries its source, so choosing a
|
||||
// forecast settles it. Picking the source is a Setup concern and the status bar
|
||||
// only offers it there.
|
||||
export default function Sidebar({
|
||||
view, setView, expanded, setExpanded,
|
||||
versions = [], versionId, setVersionId,
|
||||
sourceId, setSourceId,
|
||||
}) {
|
||||
const selected = versions.find(v => String(v.id) === String(versionId))
|
||||
|
||||
function pick(v) {
|
||||
// The source comes with the forecast. This is the whole reason the source
|
||||
// needs no control of its own outside Setup: every version carries its
|
||||
// source_id, so choosing one settles both. Set first, so the version is not
|
||||
// briefly validated against the outgoing source's list.
|
||||
if (v.source_id != null && String(v.source_id) !== String(sourceId)) {
|
||||
setSourceId(String(v.source_id))
|
||||
}
|
||||
setVersionId(String(v.id))
|
||||
// Setup is the one step a version means nothing to, so choosing one there
|
||||
// is a statement of intent to go and work on it.
|
||||
if (view === 'setup') setView('forecast')
|
||||
}
|
||||
|
||||
export default function Sidebar({ view, setView, expanded, setExpanded }) {
|
||||
return (
|
||||
<div
|
||||
className="bg-white border-r border-gray-200 flex flex-col shrink-0 overflow-hidden transition-all duration-150"
|
||||
@ -92,61 +58,6 @@ export default function Sidebar({
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{expanded ? (
|
||||
<div className="px-2 pt-2 pb-1.5 border-b border-gray-100 shrink-0">
|
||||
<div className="px-2 pb-1 text-gray-400 uppercase tracking-wide" style={{ fontSize: '10px' }}>
|
||||
Forecasts
|
||||
</div>
|
||||
{/* Capped and scrollable: the list grows with the work, and the steps
|
||||
below it must never be pushed off the bottom. */}
|
||||
<div className="max-h-52 overflow-y-auto">
|
||||
{versions.length === 0 && (
|
||||
<div className="px-2 py-1 text-xs text-gray-400">none yet</div>
|
||||
)}
|
||||
{versions.map(v => {
|
||||
const active = String(v.id) === String(versionId)
|
||||
return (
|
||||
<button
|
||||
key={v.id}
|
||||
onClick={() => pick(v)}
|
||||
title={`${v.name}${v.is_default ? ' · default' : ''} · ${v.status}`}
|
||||
className={`w-full flex items-center gap-1.5 rounded px-2 py-1 text-left transition-colors ${
|
||||
active ? 'bg-blue-50 text-blue-700' : 'text-gray-600 hover:bg-gray-100'
|
||||
}`}
|
||||
>
|
||||
<span className={`w-1.5 h-1.5 rounded-full shrink-0 ${active ? 'bg-blue-600' : 'bg-transparent'}`} />
|
||||
<span className="text-xs truncate flex-1">{v.name}</span>
|
||||
{v.is_default && (
|
||||
<span className="text-amber-500 shrink-0" style={{ fontSize: '10px' }} title="Default">★</span>
|
||||
)}
|
||||
<span
|
||||
className={`shrink-0 ${v.status === 'open' ? 'text-green-600' : 'text-gray-400'}`}
|
||||
style={{ fontSize: '10px' }}
|
||||
>
|
||||
{v.status}
|
||||
</span>
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
// Collapsed, the list has nowhere to go, so all that survives is which
|
||||
// one is open — as a tooltip, on a button that opens the rail again.
|
||||
<div className="p-2 border-b border-gray-100 shrink-0">
|
||||
<button
|
||||
onClick={() => setExpanded(true)}
|
||||
title={selected ? `Forecast: ${selected.name}` : 'Forecasts'}
|
||||
className="w-8 h-8 flex items-center justify-center rounded hover:bg-gray-100 text-gray-400"
|
||||
>
|
||||
<svg width="18" height="18" viewBox="0 0 20 20" fill="none" stroke="currentColor" strokeWidth="1.6" strokeLinecap="round" strokeLinejoin="round">
|
||||
<path d="M4 3h8l4 4v10a1 1 0 0 1-1 1H4a1 1 0 0 1-1-1V4a1 1 0 0 1 1-1z"/>
|
||||
<polyline points="12,3 12,7 16,7"/>
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<nav className="flex flex-col gap-0.5 p-2 flex-1">
|
||||
{NAV.map(item => {
|
||||
const active = view === item.id
|
||||
|
||||
@ -1,20 +1,14 @@
|
||||
import { useState, useEffect, useCallback } from 'react'
|
||||
import useTheme from '../theme.jsx'
|
||||
import useAuth from '../auth.jsx'
|
||||
|
||||
export default function StatusBar({ view, sources = [], sourceId, setSourceId, versions = [], versionId }) {
|
||||
export default function StatusBar({ view, sources = [], sourceId, setSourceId, versions = [], versionId, setVersionId }) {
|
||||
const { dark, setDark } = useTheme()
|
||||
const { user, logout } = useAuth()
|
||||
const showVersion = view === 'baseline' || view === 'forecast'
|
||||
const selectedVersion = versions.find(v => String(v.id) === String(versionId))
|
||||
|
||||
const [info, setInfo] = useState(null)
|
||||
const [showInfo, setShow] = useState(false)
|
||||
const [copied, setCopied] = useState(false)
|
||||
// Forecast.jsx mirrors the pivot's own updating state onto this event. The
|
||||
// viewer shows the same thing as a beaker in its settings panel, which is
|
||||
// exactly where you cannot see it once the panel is closed.
|
||||
const [busy, setBusy] = useState(false)
|
||||
|
||||
const refreshInfo = useCallback(async () => {
|
||||
if (!versionId || !showVersion) { setInfo(null); return }
|
||||
@ -33,12 +27,6 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
|
||||
return () => window.removeEventListener('pf-data-changed', onChange)
|
||||
}, [refreshInfo])
|
||||
|
||||
useEffect(() => {
|
||||
const onBusy = (e) => setBusy(!!e.detail?.busy)
|
||||
window.addEventListener('pf-pivot-busy', onBusy)
|
||||
return () => window.removeEventListener('pf-pivot-busy', onBusy)
|
||||
}, [])
|
||||
|
||||
async function copyTable() {
|
||||
if (!info?.fc_table) return
|
||||
try {
|
||||
@ -52,44 +40,55 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
|
||||
|
||||
return (
|
||||
<div className="bg-white border-b border-gray-200 px-3 h-9 flex items-center gap-3 shrink-0 text-xs relative">
|
||||
{/* Source is a Setup concern: the raw table name means nothing to someone
|
||||
editing a forecast, and a version carries its own source, so picking a
|
||||
forecast in the sidebar settles it everywhere else. */}
|
||||
{view === 'setup' && (
|
||||
<>
|
||||
<span className="text-gray-400">Source</span>
|
||||
<select
|
||||
value={sourceId || ''}
|
||||
onChange={e => setSourceId(e.target.value)}
|
||||
disabled={sources.length === 0}
|
||||
className="border border-gray-200 rounded px-2 py-0.5 bg-white"
|
||||
>
|
||||
{sources.length === 0
|
||||
? <option value="">— no sources —</option>
|
||||
: sources.map(s => <option key={s.id} value={s.id}>{s.tname}</option>)}
|
||||
</select>
|
||||
</>
|
||||
)}
|
||||
<span className="text-gray-400">Source</span>
|
||||
<select
|
||||
value={sourceId || ''}
|
||||
onChange={e => setSourceId(e.target.value)}
|
||||
disabled={sources.length === 0}
|
||||
className="border border-gray-200 rounded px-2 py-0.5 bg-white"
|
||||
>
|
||||
{sources.length === 0
|
||||
? <option value="">— no sources —</option>
|
||||
: sources.map(s => <option key={s.id} value={s.id}>{s.tname}</option>)}
|
||||
</select>
|
||||
|
||||
{showVersion && (
|
||||
<>
|
||||
{/* What is open, and how big it is -- the two things a forecaster
|
||||
wants from a status bar. The physical table it writes to is an
|
||||
implementation detail and means nothing to a sales account, so it
|
||||
has moved into the hover panel and onto the Baseline page, which
|
||||
is where a version's plumbing is managed. */}
|
||||
<span className="text-gray-200">|</span>
|
||||
<span className="text-gray-400">Version</span>
|
||||
<select
|
||||
value={versionId || ''}
|
||||
onChange={e => setVersionId(e.target.value)}
|
||||
disabled={versions.length === 0}
|
||||
className="border border-gray-200 rounded px-2 py-0.5 bg-white"
|
||||
>
|
||||
{versions.length === 0
|
||||
? <option value="">— no versions —</option>
|
||||
: versions.map(v => <option key={v.id} value={v.id}>{v.name}</option>)}
|
||||
</select>
|
||||
{selectedVersion && (
|
||||
<span className={`font-medium ${selectedVersion.status === 'open' ? 'text-green-600' : 'text-gray-400'}`}>
|
||||
{selectedVersion.status}
|
||||
</span>
|
||||
)}
|
||||
|
||||
{/* write target — the physical table every operation appends to */}
|
||||
{info && (
|
||||
<>
|
||||
<span className="text-gray-200">|</span>
|
||||
<span
|
||||
<span className="text-gray-400" title="Operations append to this table">writes to</span>
|
||||
<button
|
||||
onClick={copyTable}
|
||||
onMouseEnter={() => setShow(true)}
|
||||
onMouseLeave={() => setShow(false)}
|
||||
className="font-medium text-gray-700 cursor-default"
|
||||
title="Hover for the write target and row breakdown"
|
||||
className={`font-mono px-1.5 py-0.5 rounded border hover:bg-gray-50 ${
|
||||
info.exists ? 'text-gray-700 border-gray-200' : 'text-amber-700 border-amber-200 bg-amber-50'
|
||||
}`}
|
||||
title={info.exists ? 'Click to copy table name' : 'Table does not exist yet'}
|
||||
>
|
||||
{selectedVersion?.name || '—'}
|
||||
</span>
|
||||
<span className={`font-mono ${info.exists ? 'text-gray-400' : 'text-amber-700'}`}>
|
||||
{copied ? 'copied!' : info.fc_table}
|
||||
</button>
|
||||
<span className="text-gray-400 font-mono">
|
||||
{info.exists ? `${fmt(info.rows)} rows` : 'not created'}
|
||||
</span>
|
||||
|
||||
@ -98,15 +97,7 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
|
||||
<div className="text-gray-400 uppercase tracking-wide mb-2" style={{ fontSize: '10px' }}>Write target</div>
|
||||
<div className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1">
|
||||
<span className="text-gray-400">table</span>
|
||||
{/* Still copyable, just no longer the headline: pasting it
|
||||
into psql is the reason it was ever on screen. */}
|
||||
<button
|
||||
onClick={copyTable}
|
||||
title="Click to copy table name"
|
||||
className="font-mono text-gray-700 text-left hover:text-blue-700"
|
||||
>
|
||||
{copied ? 'copied!' : info.fc_table}
|
||||
</button>
|
||||
<span className="font-mono text-gray-700">{info.fc_table}</span>
|
||||
<span className="text-gray-400">reads from</span>
|
||||
<span className="font-mono text-gray-700">{info.source}</span>
|
||||
<span className="text-gray-400">total rows</span>
|
||||
@ -134,38 +125,7 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* Left of the ml-auto block, so it sits beside the pivot's own figures
|
||||
rather than out among the account controls. Occupies no space at all
|
||||
when idle: a permanently reserved slot reads as a thing that is broken
|
||||
when empty. */}
|
||||
{busy && view === 'forecast' && (
|
||||
<span
|
||||
className="flex items-center gap-1.5 text-blue-600"
|
||||
title="The pivot is rebuilding its view — sorting, filtering, or changing depth on a large grain takes a moment"
|
||||
>
|
||||
<svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor"
|
||||
strokeWidth="3" strokeLinecap="round" className="animate-spin">
|
||||
<path d="M12 3a9 9 0 1 0 9 9" />
|
||||
</svg>
|
||||
Calculating…
|
||||
</span>
|
||||
)}
|
||||
|
||||
<div className="ml-auto flex items-center gap-2">
|
||||
{user && (
|
||||
<>
|
||||
<span className="text-gray-500" title={`Signed in as ${user.username}`}>
|
||||
{user.display_name || user.username}
|
||||
</span>
|
||||
<button
|
||||
onClick={logout}
|
||||
className="text-xs text-gray-500 hover:text-gray-700 border border-gray-200 px-2 py-0.5 rounded"
|
||||
title="Sign out"
|
||||
>
|
||||
Sign out
|
||||
</button>
|
||||
</>
|
||||
)}
|
||||
<div className="ml-auto">
|
||||
<button
|
||||
onClick={() => setDark(d => !d)}
|
||||
className="w-6 h-6 flex items-center justify-center rounded hover:bg-gray-100"
|
||||
|
||||
@ -32,16 +32,11 @@ function roundRect(ctx, x, y, w, h, r, fill, stroke) {
|
||||
if (stroke) ctx.stroke()
|
||||
}
|
||||
|
||||
export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDays = 0, type = 'baseline' }) {
|
||||
export default function Timeline({ dateFrom, dateTo, offsetYr, offsetMo, type = 'baseline' }) {
|
||||
const canvasRef = useRef(null)
|
||||
|
||||
// Months and days are kept apart because they are not interchangeable: a month
|
||||
// shift lands on the same day of a different month, a day shift can cross a
|
||||
// month boundary. The preview adds months first, then days, as Postgres does.
|
||||
const offsetMoTotal = offsetMonths || 0
|
||||
const offsetDayTotal = offsetDays || 0
|
||||
const shifted = offsetMoTotal !== 0 || offsetDayTotal !== 0
|
||||
const twoBands = type === 'baseline' && shifted
|
||||
const offsetMoTotal = (offsetYr || 0) * 12 + (offsetMo || 0)
|
||||
const twoBands = type === 'baseline' && offsetMoTotal > 0
|
||||
const canvasH = twoBands ? 90 : 52
|
||||
|
||||
useEffect(() => {
|
||||
@ -66,9 +61,8 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
|
||||
const srcEnd = parseDate(dateTo)
|
||||
if (!srcStart || !srcEnd || isNaN(srcStart) || isNaN(srcEnd)) return
|
||||
|
||||
const addDays = (d, n) => { const x = new Date(d); x.setDate(x.getDate() + n); return x }
|
||||
const projStart = addDays(addMonths(srcStart, offsetMoTotal), offsetDayTotal)
|
||||
const projEnd = addDays(addMonths(srcEnd, offsetMoTotal), offsetDayTotal)
|
||||
const projStart = addMonths(srcStart, offsetMoTotal)
|
||||
const projEnd = addMonths(srcEnd, offsetMoTotal)
|
||||
|
||||
const winStart = addMonths(srcStart, -1)
|
||||
const winEnd = addMonths(twoBands ? projEnd : srcEnd, 1)
|
||||
@ -153,14 +147,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
|
||||
ctx.lineTo(px1 - 4, arrowY + 4)
|
||||
ctx.closePath()
|
||||
ctx.fill()
|
||||
const yrs = Math.trunc(offsetMoTotal / 12)
|
||||
const mos = offsetMoTotal % 12
|
||||
const sign = (offsetMoTotal + offsetDayTotal) < 0 ? '' : '+'
|
||||
const offsetLabel = sign + [
|
||||
yrs ? `${yrs}yr` : '',
|
||||
mos ? `${mos}mo` : '',
|
||||
offsetDayTotal ? `${offsetDayTotal}d` : '',
|
||||
].filter(Boolean).join(' ')
|
||||
const offsetLabel = '+' + (offsetYr ? offsetYr + 'yr ' : '') + (offsetMo ? offsetMo + 'mo' : '')
|
||||
ctx.fillStyle = '#64748b'
|
||||
ctx.font = '9px system-ui'
|
||||
ctx.textAlign = 'center'
|
||||
@ -169,7 +156,7 @@ export default function Timeline({ dateFrom, dateTo, offsetMonths = 0, offsetDay
|
||||
}
|
||||
raf = requestAnimationFrame(draw)
|
||||
return () => cancelAnimationFrame(raf)
|
||||
}, [dateFrom, dateTo, offsetMoTotal, offsetDayTotal, type, twoBands, canvasH])
|
||||
}, [dateFrom, dateTo, offsetYr, offsetMo, type, twoBands, canvasH])
|
||||
|
||||
return <canvas ref={canvasRef} height={canvasH} style={{ width: '100%', display: 'block' }} />
|
||||
}
|
||||
|
||||
@ -11,8 +11,6 @@
|
||||
--border-light: #f3f4f6;
|
||||
--accent-bg: #eff6ff;
|
||||
--accent-text: #1d4ed8;
|
||||
--chart-ink-dim: #6b7280;
|
||||
--chart-axis: #d1d5db;
|
||||
}
|
||||
|
||||
/* Dark palette tuned to Perspective's "Pro Dark" theme:
|
||||
@ -29,11 +27,6 @@
|
||||
--border-light: #3b3f46;
|
||||
--accent-bg: rgba(39, 113, 170, 0.32);
|
||||
--accent-text: #4778c2;
|
||||
--chart-ink-dim: #9ca3af;
|
||||
--chart-axis: #61656e;
|
||||
/* Native controls — checkboxes, scrollbars, date pickers — draw in the dark
|
||||
style. Without it every checkbox in the app was a white square. */
|
||||
color-scheme: dark;
|
||||
}
|
||||
|
||||
body { margin: 0; background-color: var(--bg-primary); color: var(--text-primary); }
|
||||
@ -41,68 +34,9 @@ body { margin: 0; background-color: var(--bg-primary); color: var(--text-primary
|
||||
|
||||
.dark .bg-white { background-color: var(--bg-secondary); }
|
||||
.dark .bg-gray-50 { background-color: var(--bg-tertiary); }
|
||||
/* The opacity variant is its own class and the rule above does not reach it, so
|
||||
without this the operation panel's header band stayed light grey in dark mode.
|
||||
--bg-tertiary at the same 60%. */
|
||||
.dark .bg-gray-50\/60 { background-color: rgb(59 63 70 / 0.6); }
|
||||
.dark .bg-gray-100 { background-color: var(--bg-tertiary); }
|
||||
.dark .bg-gray-200 { background-color: var(--bg-tertiary); }
|
||||
.dark .bg-gray-300 { background-color: var(--bg-tertiary); }
|
||||
/* Toolbar section labels (LAYOUT / ROWS / COLUMNS). Their own class rather than
|
||||
a grey utility: in dark mode text-gray-400 and text-gray-500 both collapse to
|
||||
--text-muted, so changing the utility would have fixed light mode only. At
|
||||
10px/gray-400 these sat near 2.5:1 on white and 2:1 on the dark panel, under
|
||||
the 4.5:1 WCAG wants for text this size. */
|
||||
/* Names the group of controls beside it — Layout, Rows, Columns. Weighted and
|
||||
tracked so it reads as a label rather than as another small grey control:
|
||||
the toolbar is dense, and at the old weight the labels and the buttons they
|
||||
name were the same visual object. */
|
||||
.pf-section-label {
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
line-height: 1;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
color: #4b5563;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.dark .pf-section-label { color: #c5c9d0; }
|
||||
|
||||
/* The rule between groups inside a zone. Taller and lighter than the old
|
||||
`w-px h-4 bg-gray-200`, which was the same weight as the zone borders, so
|
||||
the groups within a zone and the zones themselves separated identically. */
|
||||
.pf-group-rule {
|
||||
width: 1px;
|
||||
height: 20px;
|
||||
flex-shrink: 0;
|
||||
background: #b6bcc5;
|
||||
}
|
||||
.dark .pf-group-rule { background: #5a616b; }
|
||||
|
||||
/* A toggle that is on — the Total chips, and Operations while its panel is
|
||||
open. Pressed grey rather than blue: blue is Save and Post, actions, and
|
||||
these are states. Its own class so the dark values are defined once rather
|
||||
than hoping each utility has a dark override. */
|
||||
.pf-pressed {
|
||||
background-color: #e5e7eb;
|
||||
border-color: #9ca3af;
|
||||
color: #1f2937;
|
||||
box-shadow: inset 0 1px 2px rgb(0 0 0 / 0.08);
|
||||
}
|
||||
.dark .pf-pressed {
|
||||
background-color: var(--bg-tertiary);
|
||||
border-color: #61656e;
|
||||
color: var(--text-primary);
|
||||
box-shadow: inset 0 1px 2px rgb(0 0 0 / 0.3);
|
||||
}
|
||||
|
||||
/* The joined controls — the depth steppers and the operation panel's
|
||||
Scale | Recode | Clone — draw their inner rules with divide-*, which had no
|
||||
dark override and stayed light grey. */
|
||||
.dark .divide-gray-300 > :not(:last-child) { border-color: var(--border-color); }
|
||||
.dark .disabled\:text-gray-300:disabled { color: var(--text-muted); }
|
||||
.dark .disabled\:hover\:bg-white:disabled:hover { background-color: var(--bg-secondary); }
|
||||
|
||||
.dark .text-gray-300 { color: var(--text-muted); }
|
||||
.dark .text-gray-400 { color: var(--text-muted); }
|
||||
.dark .text-gray-500 { color: var(--text-muted); }
|
||||
|
||||
@ -1,27 +1,13 @@
|
||||
import { StrictMode } from 'react'
|
||||
import { createRoot } from 'react-dom/client'
|
||||
import { ThemeProvider } from './theme.jsx'
|
||||
import { AuthProvider } from './auth.jsx'
|
||||
import useAuth from './auth.jsx'
|
||||
import Login from './views/Login.jsx'
|
||||
import './index.css'
|
||||
import App from './App.jsx'
|
||||
|
||||
// App is mounted only once there is a session — its load effects call /api
|
||||
// straight away, and mounting it logged-out would just fire a burst of 401s.
|
||||
function Gate() {
|
||||
const { user, checking } = useAuth()
|
||||
if (checking) return <div className="flex items-center justify-center h-screen text-sm text-gray-400">Loading…</div>
|
||||
if (!user) return <Login />
|
||||
return <App />
|
||||
}
|
||||
|
||||
createRoot(document.getElementById('root')).render(
|
||||
<StrictMode>
|
||||
<ThemeProvider>
|
||||
<AuthProvider>
|
||||
<Gate />
|
||||
</AuthProvider>
|
||||
<App />
|
||||
</ThemeProvider>
|
||||
</StrictMode>,
|
||||
)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@ -1,68 +0,0 @@
|
||||
import { useState } from 'react'
|
||||
import useAuth from '../auth.jsx'
|
||||
|
||||
export default function Login() {
|
||||
const { login } = useAuth()
|
||||
const [username, setUsername] = useState('')
|
||||
const [password, setPassword] = useState('')
|
||||
const [error, setError] = useState('')
|
||||
const [busy, setBusy] = useState(false)
|
||||
|
||||
async function submit(e) {
|
||||
e.preventDefault()
|
||||
setError(''); setBusy(true)
|
||||
try {
|
||||
await login(username.trim(), password)
|
||||
} catch (err) {
|
||||
setError(err.message)
|
||||
setPassword('')
|
||||
} finally {
|
||||
setBusy(false)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex items-center justify-center h-screen w-full text-sm">
|
||||
<form onSubmit={submit} className="bg-white border border-gray-200 rounded p-6 w-80 flex flex-col gap-4">
|
||||
<div>
|
||||
<div className="text-base font-medium">Pivot Forecast</div>
|
||||
<div className="text-xs text-gray-500 mt-0.5">Sign in to continue</div>
|
||||
</div>
|
||||
|
||||
<label className="flex flex-col gap-1">
|
||||
<span className="text-xs text-gray-500">Username</span>
|
||||
<input
|
||||
value={username}
|
||||
onChange={e => setUsername(e.target.value)}
|
||||
autoFocus
|
||||
autoComplete="username"
|
||||
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
|
||||
/>
|
||||
</label>
|
||||
|
||||
<label className="flex flex-col gap-1">
|
||||
<span className="text-xs text-gray-500">Password</span>
|
||||
<input
|
||||
type="password"
|
||||
value={password}
|
||||
onChange={e => setPassword(e.target.value)}
|
||||
autoComplete="current-password"
|
||||
className="border border-gray-200 rounded px-2 py-1.5 text-sm"
|
||||
/>
|
||||
</label>
|
||||
|
||||
{error && (
|
||||
<div className="px-3 py-2 text-xs rounded font-medium bg-red-50 text-red-700">{error}</div>
|
||||
)}
|
||||
|
||||
<button
|
||||
type="submit"
|
||||
disabled={busy || !username.trim() || !password}
|
||||
className="bg-blue-600 text-white text-xs px-3 py-2 rounded hover:bg-blue-700 disabled:opacity-50"
|
||||
>
|
||||
{busy ? 'Signing in…' : 'Sign in'}
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@ -1,7 +1,5 @@
|
||||
import { useState, useEffect, useRef } from 'react'
|
||||
import useTheme from '../theme.jsx'
|
||||
import useAuth from '../auth.jsx'
|
||||
import ReadOnlyNotice from '../components/ReadOnlyNotice.jsx'
|
||||
|
||||
const ROLES = ['ignore', 'dimension', 'value', 'units', 'date', 'filter']
|
||||
|
||||
@ -15,8 +13,6 @@ const ROLE_STYLE = {
|
||||
}
|
||||
|
||||
export default function Setup({ refreshSources }) {
|
||||
const { user: me } = useAuth()
|
||||
const isAdmin = !!me?.is_admin
|
||||
const { dark } = useTheme()
|
||||
const [tables, setTables] = useState([])
|
||||
const [sources, setSources] = useState([])
|
||||
@ -29,22 +25,14 @@ export default function Setup({ refreshSources }) {
|
||||
const [sqlStatus, setSqlStatus] = useState({}) // sourceId -> bool
|
||||
const [saving, setSaving] = useState(false)
|
||||
const [generating, setGenerating] = useState(false)
|
||||
const [refreshingDims, setRefreshingDims] = useState(false)
|
||||
const [msg, setMsg] = useState(null)
|
||||
const [dimPeriodCols, setDimPeriodCols] = useState([])
|
||||
const [openPeriodIdx, setOpenPeriodIdx] = useState(null)
|
||||
const periodDropRef = useRef(null)
|
||||
|
||||
// Every list here is rendered with .map, so a response that is not an array
|
||||
// takes the whole app down rather than this one panel -- a 403 body is
|
||||
// {error}, and `{error}.map` threw during render, which unmounts React and
|
||||
// leaves a blank page. An endpoint answering something unexpected should cost
|
||||
// an empty list, not the session.
|
||||
const asList = (set) => (d) => set(Array.isArray(d) ? d : [])
|
||||
|
||||
useEffect(() => {
|
||||
fetch('/api/tables').then(r => r.json()).then(asList(setTables)).catch(console.error)
|
||||
fetch('/api/dim-period/cols').then(r => r.json()).then(asList(setDimPeriodCols)).catch(console.error)
|
||||
fetch('/api/tables').then(r => r.json()).then(setTables).catch(console.error)
|
||||
fetch('/api/dim-period/cols').then(r => r.json()).then(setDimPeriodCols).catch(console.error)
|
||||
loadSources()
|
||||
}, [])
|
||||
|
||||
@ -59,8 +47,7 @@ export default function Setup({ refreshSources }) {
|
||||
}, [openPeriodIdx])
|
||||
|
||||
function loadSources() {
|
||||
fetch('/api/sources').then(r => r.json()).then(raw => {
|
||||
const data = Array.isArray(raw) ? raw : []
|
||||
fetch('/api/sources').then(r => r.json()).then(data => {
|
||||
setSources(data)
|
||||
// check sql status for each source
|
||||
data.forEach(s => {
|
||||
@ -75,8 +62,7 @@ export default function Setup({ refreshSources }) {
|
||||
function selectSource(source) {
|
||||
setSelectedSource(source)
|
||||
setColsDirty(false)
|
||||
fetch(`/api/sources/${source.id}/cols`).then(r => r.json()).then(raw => {
|
||||
const data = Array.isArray(raw) ? raw : []
|
||||
fetch(`/api/sources/${source.id}/cols`).then(r => r.json()).then(data => {
|
||||
setCols(data)
|
||||
setEditedCols(data.map(c => ({ ...c })))
|
||||
})
|
||||
@ -167,35 +153,6 @@ export default function Setup({ refreshSources }) {
|
||||
}
|
||||
}
|
||||
|
||||
// Rebuild every keyed dim_group's member list from the source. Deliberate rather
|
||||
// than automatic: it reads the whole source, which for a view over a transaction
|
||||
// table is millions of rows, and the answer only changes when the catalogue does.
|
||||
async function refreshDimMembers() {
|
||||
const groups = [...new Set(cols.filter(c => c.dim_group && c.is_key).map(c => c.dim_group))]
|
||||
if (groups.length === 0) {
|
||||
flash('No dim_group has an is_key column, so there is nothing to build a list from', 'error')
|
||||
return
|
||||
}
|
||||
setRefreshingDims(true)
|
||||
try {
|
||||
const done = []
|
||||
for (const g of groups) {
|
||||
const res = await fetch(`/api/sources/${selectedSource.id}/dim/${encodeURIComponent(g)}/refresh`,
|
||||
{ method: 'POST' })
|
||||
const data = await res.json()
|
||||
if (!res.ok) { flash(`${g}: ${data.error}`, 'error'); return }
|
||||
done.push(`${g}: ${data.members.toLocaleString()} members`
|
||||
+ (data.no_longer_in_source ? `, ${data.no_longer_in_source} no longer in source` : '')
|
||||
+ ` (${(data.ms / 1000).toFixed(1)}s)`)
|
||||
}
|
||||
flash(done.join(' · '))
|
||||
} catch (err) {
|
||||
flash(err.message, 'error')
|
||||
} finally {
|
||||
setRefreshingDims(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function deleteSource(id, e) {
|
||||
e.stopPropagation()
|
||||
if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return
|
||||
@ -216,19 +173,8 @@ export default function Setup({ refreshSources }) {
|
||||
|
||||
const registeredKeys = new Set(sources.map(s => `${s.schema}.${s.tname}`))
|
||||
|
||||
// display grain — must match grainOf() in lib/sql_generator.js
|
||||
const grainCols = editedCols
|
||||
.filter(c => c.in_grain && (c.role === 'dimension' || c.role === 'date'))
|
||||
.map(c => c.cname)
|
||||
|
||||
return (
|
||||
<div className="h-full flex flex-col overflow-hidden text-sm">
|
||||
{!isAdmin && <ReadOnlyNotice what="source registration and column roles" />}
|
||||
{/* A real box here rather than `display: contents`, because this one has to
|
||||
be the flex row the two panes sit in. Baseline uses `contents` instead,
|
||||
where the parent's layout must be left alone. `disabled` is a DOM
|
||||
property and applies either way. */}
|
||||
<fieldset disabled={!isAdmin} className="flex-1 min-h-0 flex overflow-hidden border-0 p-0 m-0">
|
||||
<div className="h-full flex overflow-hidden text-sm">
|
||||
|
||||
{/* All Tables */}
|
||||
<div className="w-64 bg-white border-r border-gray-200 flex flex-col shrink-0">
|
||||
@ -332,11 +278,6 @@ export default function Setup({ refreshSources }) {
|
||||
<div className="px-3 py-2 border-b border-gray-100 flex items-center justify-between shrink-0">
|
||||
<span className="text-xs font-medium text-gray-500 uppercase tracking-wide">
|
||||
Col Meta — <span className="text-gray-700 normal-case">{selectedSource.schema}.{selectedSource.tname}</span>
|
||||
<span className="ml-3 normal-case font-normal text-gray-400" title="Columns the forecast load is pre-aggregated to">
|
||||
grain: {grainCols.length
|
||||
? <span className="font-mono text-gray-600">{grainCols.join(' × ')}</span>
|
||||
: <span className="italic">none — raw rows</span>}
|
||||
</span>
|
||||
</span>
|
||||
<div className="flex items-center gap-2">
|
||||
{colsDirty && (
|
||||
@ -344,17 +285,6 @@ export default function Setup({ refreshSources }) {
|
||||
{saving ? 'Saving…' : 'Save'}
|
||||
</button>
|
||||
)}
|
||||
{cols.some(c => c.dim_group && c.is_key) && (
|
||||
<button
|
||||
onClick={refreshDimMembers}
|
||||
disabled={refreshingDims || colsDirty}
|
||||
className="text-xs border border-gray-200 px-3 py-1 rounded hover:bg-gray-50 disabled:opacity-50"
|
||||
title={colsDirty ? 'Save col meta first'
|
||||
: 'Rebuild the member list for each keyed dim_group from the source. Reads the whole source, so it takes a while.'}
|
||||
>
|
||||
{refreshingDims ? 'Refreshing…' : 'Refresh master data'}
|
||||
</button>
|
||||
)}
|
||||
<button
|
||||
onClick={generateSQL}
|
||||
disabled={generating || colsDirty}
|
||||
@ -372,8 +302,6 @@ export default function Setup({ refreshSources }) {
|
||||
<th className="px-3 py-1.5 font-medium">column</th>
|
||||
<th className="px-3 py-1.5 font-medium">role</th>
|
||||
<th className="px-3 py-1.5 font-medium text-center">key</th>
|
||||
<th className="px-3 py-1.5 font-medium text-center" title="Include this column in the display grain — the load is pre-aggregated to the flagged columns">grain</th>
|
||||
<th className="px-3 py-1.5 font-medium text-center" title="The column an account's territory is expressed in. Accounts see and change only rows whose value here is on their list. One per source.">territory</th>
|
||||
<th className="px-3 py-1.5 font-medium">group</th>
|
||||
<th className="px-3 py-1.5 font-medium">period col</th>
|
||||
<th className="px-3 py-1.5 font-medium">label</th>
|
||||
@ -401,34 +329,6 @@ export default function Setup({ refreshSources }) {
|
||||
className="cursor-pointer disabled:opacity-20"
|
||||
/>
|
||||
</td>
|
||||
<td className="px-3 py-1.5 text-center">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={!!col.in_grain}
|
||||
onChange={e => updateCol(i, 'in_grain', e.target.checked)}
|
||||
disabled={col.role !== 'dimension' && col.role !== 'date'}
|
||||
className="cursor-pointer disabled:opacity-20"
|
||||
/>
|
||||
</td>
|
||||
{/* Radio, not a checkbox: exactly one column per source,
|
||||
and the shape of the control should say so rather than
|
||||
leaving it to a save-time error. */}
|
||||
<td className="px-3 py-1.5 text-center">
|
||||
<input
|
||||
type="radio"
|
||||
name="pf-territory-col"
|
||||
checked={!!col.is_territory}
|
||||
onChange={() => {}}
|
||||
onClick={() => setEditedCols(prev => {
|
||||
// clicking the chosen one again clears it, since a
|
||||
// radio otherwise has no way back to "no territory"
|
||||
const already = !!prev[i].is_territory
|
||||
return prev.map((c, x) => ({ ...c, is_territory: !already && x === i }))
|
||||
})}
|
||||
disabled={col.role !== 'dimension'}
|
||||
className="cursor-pointer disabled:opacity-20"
|
||||
/>
|
||||
</td>
|
||||
<td className="px-3 py-1.5">
|
||||
<input
|
||||
type="text"
|
||||
@ -572,7 +472,6 @@ export default function Setup({ refreshSources }) {
|
||||
</div>
|
||||
)}
|
||||
|
||||
</fieldset>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@ -1,45 +0,0 @@
|
||||
diff --git a/rust/perspective-client/src/rust/config/view_config.rs b/rust/perspective-client/src/rust/config/view_config.rs
|
||||
index fa4f36e..360b816 100644
|
||||
--- a/rust/perspective-client/src/rust/config/view_config.rs
|
||||
+++ b/rust/perspective-client/src/rust/config/view_config.rs
|
||||
@@ -526,6 +526,23 @@ impl ViewConfig {
|
||||
}
|
||||
}
|
||||
|
||||
+ /// `_apply` for a field which is itself `Option`, where `None` in the update
|
||||
+ /// means "not mentioned" rather than "clear it". `Option<Option<T>>` would be
|
||||
+ /// needed to express both, and the wire format cannot carry the difference:
|
||||
+ /// these fields are `skip_serializing_if = "Option::is_none"`, so an absent
|
||||
+ /// field and an explicit null arrive identically. To lift a depth, send the
|
||||
+ /// number of levels on that axis rather than clearing it.
|
||||
+ fn _apply_optional<T: PartialEq>(field: &mut Option<T>, update: Option<T>) -> bool {
|
||||
+ match update {
|
||||
+ None => false,
|
||||
+ Some(_) if *field == update => false,
|
||||
+ Some(_) => {
|
||||
+ *field = update;
|
||||
+ true
|
||||
+ },
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
pub fn reset(&mut self, reset_expressions: bool) {
|
||||
let mut config = Self::default();
|
||||
if !reset_expressions {
|
||||
@@ -568,6 +585,16 @@ impl ViewConfig {
|
||||
changed = Self::_apply(&mut self.windows, update.windows) || changed;
|
||||
changed = Self::_apply(&mut self.group_rollup_mode, update.group_rollup_mode) || changed;
|
||||
changed = Self::_apply(&mut self.split_rollup_mode, update.split_rollup_mode) || changed;
|
||||
+
|
||||
+ // Without these two, a depth can be set when a view is created --
|
||||
+ // `table.view({ group_by_depth })` -- but never through `restore()`, which
|
||||
+ // merges a `ViewConfigUpdate` onto the live config. The field arrives,
|
||||
+ // deserializes, and is then dropped here, so the viewer's config and the
|
||||
+ // engine never see it and nothing happens. That makes expand/collapse
|
||||
+ // unreachable for anything driven by `restore()`, which is how a viewer
|
||||
+ // changes its own configuration.
|
||||
+ changed = Self::_apply_optional(&mut self.group_by_depth, update.group_by_depth) || changed;
|
||||
+ changed = Self::_apply_optional(&mut self.split_by_depth, update.split_by_depth) || changed;
|
||||
if self.group_rollup_mode == GroupRollupMode::Total && !self.group_by.is_empty() {
|
||||
tracing::info!("`total` incompatible with `group_by`");
|
||||
changed = true;
|
||||
8
ui/vendor/PROVENANCE.txt
vendored
8
ui/vendor/PROVENANCE.txt
vendored
@ -1,8 +0,0 @@
|
||||
Built from https://github.com/fleetside72/perspective
|
||||
branch expand-blank-tree-nodes
|
||||
commit 751dcfe136c6455e3cd02a7b75a6df79cbe8e064
|
||||
based on unknown
|
||||
built 2026-09-22T04:18:51Z on usmidsap02
|
||||
dirty 0 uncommitted file(s) in the source tree at build time
|
||||
|
||||
Regenerate with ui/vendor/rebuild-perspective.sh
|
||||
187
ui/vendor/README.md
vendored
187
ui/vendor/README.md
vendored
@ -1,187 +0,0 @@
|
||||
# Vendored Perspective
|
||||
|
||||
pf_app runs a **patched build of Perspective**. Upstream's C++ engine has always
|
||||
implemented column-axis expand/collapse — `t_ctx2::set_depth(HEADER_COLUMN, …)`
|
||||
and `open`/`close(HEADER_COLUMN, idx)` are fully written — but nothing above C++
|
||||
could reach it: `set_column_pivot_depth()` was never called, and
|
||||
`View<t_ctx2>::expand/collapse` hardcoded `HEADER_ROW`. The patch is wiring, not
|
||||
new engine logic.
|
||||
|
||||
It buys two things the released packages cannot do at all:
|
||||
|
||||
- `split_by_depth` in `ViewConfig`, the `split_by` counterpart to `group_by_depth`
|
||||
- `expand_column()` / `collapse_column()`, so one column branch can fold to its
|
||||
subtotal while its siblings stay expanded — the Excel behaviour
|
||||
|
||||
**Source:** https://github.com/fleetside72/perspective, branch
|
||||
`expand-blank-tree-nodes` — the tip of the stack, carrying
|
||||
`column-axis-expand-collapse` and `apply-depth-on-config-update` beneath it.
|
||||
See `PROVENANCE.txt` for the exact commit these tarballs were built from.
|
||||
|
||||
## Why tarballs and not npm
|
||||
|
||||
The feature is not released upstream. Until it is, the four packages are built
|
||||
from the fork and committed here as npm tarballs. `npm install` expands them
|
||||
exactly as it expands anything from the registry — no special tooling, and
|
||||
`pf.sh deploy` works unchanged. A deploy machine needs node and nothing else:
|
||||
no emscripten, no cmake, no protoc, no Rust.
|
||||
|
||||
All four move together, never a subset. Perspective couples loader, package
|
||||
versions, data format and `apache-arrow`; vendoring a partial set reintroduces
|
||||
exactly the drift that causes trouble.
|
||||
|
||||
## Changing the engine
|
||||
|
||||
./rebuild-perspective.sh # builds the fork, repacks, rewrites PROVENANCE.txt
|
||||
cd .. && npm install ./vendor/perspective-dev-*-5.4.0.tgz # by path — see below
|
||||
git add vendor package-lock.json && git commit
|
||||
|
||||
Push the fork first — the script warns if the source tree is dirty, because a
|
||||
tarball built from uncommitted code has no recoverable source.
|
||||
|
||||
The build itself needs cmake >= 3.29.5, protoc >= 22 (its version silently
|
||||
selects which protobuf source tree gets cloned), pnpm, and the Rust nightly the
|
||||
repo pins. Roughly 40 minutes cold. Only ever on a machine changing the engine.
|
||||
|
||||
### `npm install` alone does not pick up a rebuilt tarball
|
||||
|
||||
The version in the filename never changes, and `package-lock.json` pins an
|
||||
`integrity` hash per package — so npm satisfies the dependency from its own
|
||||
cache and the freshly built tarball on disk is never read. Nothing fails; the
|
||||
old code is simply still there. Re-resolve by path instead, which recomputes
|
||||
the hashes and rewrites the lock:
|
||||
|
||||
```bash
|
||||
cd ui
|
||||
rm -rf node_modules/@perspective-dev
|
||||
npm install ./vendor/perspective-dev-{client,server,viewer,viewer-datagrid}-5.4.0.tgz
|
||||
```
|
||||
|
||||
Check the install rather than trusting it — the packages ship their TypeScript
|
||||
sources, so the change is readable without unminifying anything:
|
||||
|
||||
```bash
|
||||
grep -n is_label node_modules/@perspective-dev/viewer-datagrid/src/ts/style_handlers/table_cell/row_header.ts
|
||||
```
|
||||
|
||||
`cargo check` on this patch was clean — the 125 errors it reports without
|
||||
`protoc` installed are unresolved generated protobuf modules, none of them in
|
||||
`view_config.rs`.
|
||||
|
||||
`applyDepth()` in `Forecast.jsx` reads the config back after restoring it and
|
||||
falls back to the imperative `view.set_depth()` when the value did not stick.
|
||||
With the patched engine installed that fallback should no longer run; it is kept
|
||||
for now so an unpatched engine still works, and can be removed once the
|
||||
declarative path is confirmed.
|
||||
|
||||
## Getting rid of this
|
||||
|
||||
This is a fork, with the maintenance that implies. The exit is upstream taking
|
||||
the change — the patch is small and additive, and the engine work is already
|
||||
theirs. When a release ships it, delete this directory and put normal version
|
||||
ranges back in `ui/package.json`.
|
||||
|
||||
## Applied: expand a tree node whose group key is blank
|
||||
|
||||
In the vendored tarballs as of the 2026-09-22 build, from branch
|
||||
`expand-blank-tree-nodes` (which is `apply-depth-on-config-update` plus this).
|
||||
Upstream bug, in `packages/viewer-datagrid/src/ts/style_handlers/table_cell/row_header.ts`.
|
||||
|
||||
A tree row's header is the label cell padded on its left by one blank cell per
|
||||
level above it, and `cell_style_row_header` told padding from label by testing
|
||||
the cell's own **text**. A node whose group key is null or `''` therefore read
|
||||
as padding: no `psp-tree-label` class, so no `+` caret — and because
|
||||
`createClickListener` in `event_handlers/header_click.ts` gates expand/collapse
|
||||
on that same class, no way to open the node at all. The node was in the
|
||||
traversal and its children existed; only the affordance was missing. On
|
||||
`gs.osm_skinny` that is `partgroup`, which has 93k null and 111k empty rows.
|
||||
|
||||
The fix tells them apart structurally — the label is the last defined entry in
|
||||
`row_header` — which is how `is_collapse` two lines below already reads the
|
||||
following row. Flat rollup (`group_rollup_mode: 'flat'`) writes each level into
|
||||
its own cell with no padding, so there is nothing structural to test there and
|
||||
it keeps the text test.
|
||||
|
||||
The class was only half of it. `tbody th:empty { pointer-events: none }` in
|
||||
`regular_table.css` makes the padding cells inert, and a blank label cell is
|
||||
`:empty` as well — a `::before` caret is not content — so the fixed node drew
|
||||
its `+`, then ignored the click and would not even take the hover. The rule now
|
||||
excludes the tree classes, which is the distinction it was reaching for: a
|
||||
padding cell never carries one.
|
||||
|
||||
`row-hover.css` uses `:empty` the same way in six border selectors. Those are a
|
||||
hover tint and are left alone; a row whose first level is blank loses a border
|
||||
highlight, and nothing else.
|
||||
|
||||
Separate from the data question: null and `''` group as two distinct rows that
|
||||
look identical, so a `COALESCE(NULLIF(col,''), '(none)')` at load time is still
|
||||
worth doing on its own merits.
|
||||
|
||||
## Applied: depth fields on a config update
|
||||
|
||||
`0001-apply-depth-fields-on-config-update.patch` is **in** the vendored
|
||||
tarballs as of the 2026-09-17 build, from branch
|
||||
`apply-depth-on-config-update`. It fixes an upstream bug in
|
||||
`rust/perspective-client/src/rust/config/view_config.rs`:
|
||||
`ViewConfig::apply_update` applies ten fields and **neither `group_by_depth`
|
||||
nor `split_by_depth` is among them**. So a depth can be set when a view is
|
||||
created (`table.view({ group_by_depth: 1 })` — which is what the fork's own
|
||||
`depth_test.mjs` exercises) but never through `restore()`, which is how a
|
||||
viewer changes its own configuration. The field arrives, deserializes, and is
|
||||
discarded before the engine sees it.
|
||||
|
||||
`group_by_depth` and the omission are both upstream; the fork mirrored
|
||||
`split_by_depth` alongside it faithfully, including the omission.
|
||||
|
||||
Note the semantics, from `server.cpp`:
|
||||
|
||||
```cpp
|
||||
ctx1->set_depth(row_pivot_depth - 1); // one-sided
|
||||
ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided
|
||||
```
|
||||
|
||||
The config field counts **levels to show**; `view.set_depth()` counts the
|
||||
boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and
|
||||
`t_traversal::set_depth` adds the 1 back, so `n` renders `n` levels.
|
||||
`Forecast.jsx` therefore sends the stepper's level count unchanged, floored at
|
||||
1 — `0` would reach the engine as `set_depth(-1)` on an unsigned depth,
|
||||
underflow, and clamp to fully expanded. (It used to send `d + 1` to dodge that,
|
||||
which was off by one against the grid everywhere.)
|
||||
|
||||
The patch is already committed on the fork's build branch (see
|
||||
`PROVENANCE.txt`), so rebuilding needs only the script. The `.patch` file here
|
||||
is the change on its own, for review or for carrying to a clean upstream
|
||||
checkout:
|
||||
|
||||
```bash
|
||||
./ui/vendor/rebuild-perspective.sh # from the fork branch
|
||||
# then re-resolve by path, as above -- a plain `npm install` keeps the old code
|
||||
|
||||
# only on a fresh upstream checkout:
|
||||
cd $PSP_DIR && git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
|
||||
```
|
||||
|
||||
### Host tools this tree needs
|
||||
|
||||
The repo pins its Rust toolchain (`nightly-2026-06-01`), Emscripten (4.0.9) and
|
||||
Binaryen (132) exactly, and pins **nothing** for the host tools — no
|
||||
`packageManager`, no `.nvmrc`. So `npm i -g pnpm` and `pip install cmake` both
|
||||
give versions this tree was not written against. The script's preflight now
|
||||
rejects them, but for the record:
|
||||
|
||||
- **pnpm 10.x.** 11+ rejects a repeated `--if-present`, which
|
||||
`sh_perspective.mjs:189` emits once per package in scope.
|
||||
- **cmake 3.x**, not 4. Both satisfy the `>= 3.29.5` floor; CMake 4 removed
|
||||
support for `cmake_minimum_required < 3.5`, which the C++ dependencies still
|
||||
declare, and fails partway through the Arrow build.
|
||||
- **protoc >= 22** (33.2 known good).
|
||||
- **`pnpm pack`, not `npm pack`.** These packages depend on each other as
|
||||
`workspace:^`, and only pnpm rewrites that to a concrete version. An
|
||||
`npm pack` tarball is rejected at install: `Unsupported URL Type
|
||||
"workspace:"`.
|
||||
|
||||
Two commits on the build branch are local build concessions rather than
|
||||
fixes, and should be dropped before the real patch goes anywhere upstream:
|
||||
removing the `postinstall:playwright` step (it runs `playwright install
|
||||
--with-deps`, which apt-installs system libraries and so needs root), and
|
||||
filling in the `allowBuilds` block pnpm 12 demanded.
|
||||
BIN
ui/vendor/perspective-dev-client-5.4.0.tgz
vendored
BIN
ui/vendor/perspective-dev-client-5.4.0.tgz
vendored
Binary file not shown.
BIN
ui/vendor/perspective-dev-server-5.4.0.tgz
vendored
BIN
ui/vendor/perspective-dev-server-5.4.0.tgz
vendored
Binary file not shown.
BIN
ui/vendor/perspective-dev-viewer-5.4.0.tgz
vendored
BIN
ui/vendor/perspective-dev-viewer-5.4.0.tgz
vendored
Binary file not shown.
BIN
ui/vendor/perspective-dev-viewer-datagrid-5.4.0.tgz
vendored
BIN
ui/vendor/perspective-dev-viewer-datagrid-5.4.0.tgz
vendored
Binary file not shown.
132
ui/vendor/rebuild-perspective.sh
vendored
132
ui/vendor/rebuild-perspective.sh
vendored
@ -1,132 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# rebuild-perspective.sh — rebuild the patched Perspective and re-vendor it
|
||||
#
|
||||
# pf_app runs a patched build of Perspective that exposes the column axis
|
||||
# expand/collapse the engine already implements (split_by_depth, and
|
||||
# expand_column/collapse_column). Upstream does not ship this yet, so the
|
||||
# built packages are vendored into this directory as npm tarballs.
|
||||
#
|
||||
# Source of truth: https://github.com/fleetside72/perspective
|
||||
# branch expand-blank-tree-nodes (tip of the patch stack)
|
||||
#
|
||||
# This script exists because vendored binaries are opaque: once the .tgz files
|
||||
# are committed, nothing in the repo records how to regenerate them. Run this
|
||||
# after changing the fork, then commit the resulting tarballs.
|
||||
#
|
||||
# Only needed on a machine that is changing the engine. Deploys just run
|
||||
# `npm install`, which expands the committed tarballs - see ../README in this
|
||||
# directory.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
PSP="${PSP_DIR:-$HOME/perspective}"
|
||||
BRANCH="${PSP_BRANCH:-expand-blank-tree-nodes}"
|
||||
VENDOR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# The four packages that must move together. Perspective's own docs are
|
||||
# emphatic that loader, packages, data format and apache-arrow are one unit;
|
||||
# vendoring a subset would reintroduce exactly the drift that causes trouble.
|
||||
PACKAGES=(
|
||||
"rust/perspective-js"
|
||||
"rust/perspective-server"
|
||||
"rust/perspective-viewer"
|
||||
"packages/viewer-datagrid"
|
||||
)
|
||||
|
||||
info() { echo -e "\033[0;34m==>\033[0m $*"; }
|
||||
ok() { echo -e "\033[0;32m ✓\033[0m $*"; }
|
||||
die() { echo -e "\033[0;31m ✗\033[0m $*" >&2; exit 1; }
|
||||
|
||||
# -- preflight --------------------------------------------------------------
|
||||
[[ -d "$PSP" ]] || die "No Perspective checkout at $PSP.
|
||||
git clone https://github.com/fleetside72/perspective.git $PSP
|
||||
cd $PSP && git checkout $BRANCH
|
||||
Set PSP_DIR to use a different path."
|
||||
|
||||
command -v pnpm >/dev/null || die "pnpm not found. Perspective builds with pnpm, not npm."
|
||||
|
||||
# The repo pins its Rust toolchain, emsdk and Binaryen exactly, but pins nothing
|
||||
# for the host tools -- no packageManager, no .nvmrc -- so `npm i -g pnpm` gets
|
||||
# whatever is newest and that is not what this tree was written against. pnpm 11+
|
||||
# rejects a repeated `--if-present`, which sh_perspective.mjs emits once per
|
||||
# package in scope, and the build dies with:
|
||||
# error: the argument '--if-present' cannot be used multiple times
|
||||
pnpm_major=$(pnpm --version | cut -d. -f1)
|
||||
if (( pnpm_major > 10 )); then
|
||||
die "pnpm $(pnpm --version) is too new; this tree needs pnpm 10.x.
|
||||
npm i -g pnpm@10"
|
||||
fi
|
||||
command -v protoc >/dev/null || die "protoc not found.
|
||||
Its VERSION selects which protobuf source tree the build clones, and a
|
||||
version below 22 pulls a layout the build cannot consume. Needs >= 22
|
||||
(33.2 known good). Distro packages are usually far too old."
|
||||
|
||||
cmake_ver=$(cmake --version 2>/dev/null | head -1 | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?') || die "cmake not found"
|
||||
cmake_major=${cmake_ver%%.*}; cmake_minor=$(echo "$cmake_ver" | cut -d. -f2)
|
||||
if (( cmake_major < 3 || (cmake_major == 3 && cmake_minor < 29) )); then
|
||||
die "cmake $cmake_ver is too old; Perspective needs >= 3.29.5.
|
||||
A user-level install works: pip3 install --user 'cmake==3.31.*'"
|
||||
fi
|
||||
# And not too new: CMake 4 dropped compatibility with cmake_minimum_required
|
||||
# below 3.5, which Perspective's C++ dependencies still declare. `pip install
|
||||
# cmake` gives 4.x by default, which satisfies the floor above and then fails
|
||||
# in the middle of the Arrow build.
|
||||
if (( cmake_major >= 4 )); then
|
||||
die "cmake $cmake_ver is too new; CMake 4 removed support for
|
||||
cmake_minimum_required < 3.5, which the C++ dependencies still use.
|
||||
pip3 install --user 'cmake==3.31.*'"
|
||||
fi
|
||||
|
||||
info "Perspective checkout: $PSP"
|
||||
git -C "$PSP" rev-parse --abbrev-ref HEAD | grep -qx "$BRANCH" \
|
||||
|| echo " ! on branch $(git -C "$PSP" rev-parse --abbrev-ref HEAD), expected $BRANCH"
|
||||
commit=$(git -C "$PSP" rev-parse --short HEAD)
|
||||
dirty=$(git -C "$PSP" status --porcelain | wc -l)
|
||||
echo " commit $commit$([[ $dirty -gt 0 ]] && echo " (+$dirty uncommitted files)")"
|
||||
|
||||
# -- build ------------------------------------------------------------------
|
||||
# `metadata` first: it generates docs/expression_gen.md, which perspective-client
|
||||
# includes at compile time. Building a scope without it fails on the missing file.
|
||||
info "Building (this takes ~40 minutes cold, a few minutes warm)…"
|
||||
( cd "$PSP" && PSP_ONCE=1 PACKAGE="metadata,server,client,viewer,viewer-datagrid" pnpm run build )
|
||||
ok "build complete"
|
||||
|
||||
# -- pack -------------------------------------------------------------------
|
||||
# pnpm pack, not npm pack. These packages depend on each other as
|
||||
# `workspace:^`, and only pnpm rewrites that to the concrete version on pack.
|
||||
# npm leaves it, and npm install then refuses the tarball outright:
|
||||
# npm error Unsupported URL Type "workspace:": workspace:^
|
||||
info "Packing tarballs into $VENDOR"
|
||||
rm -f "$VENDOR"/*.tgz
|
||||
for p in "${PACKAGES[@]}"; do
|
||||
( cd "$PSP/$p" && pnpm pack --pack-destination "$VENDOR" >/dev/null )
|
||||
ok "$(basename "$p")"
|
||||
done
|
||||
|
||||
# -- record provenance ------------------------------------------------------
|
||||
# A committed .tgz is an opaque binary; without this the tie back to source is
|
||||
# only in someone's memory.
|
||||
cat > "$VENDOR/PROVENANCE.txt" <<EOF
|
||||
Built from https://github.com/fleetside72/perspective
|
||||
branch $BRANCH
|
||||
commit $(git -C "$PSP" rev-parse HEAD)
|
||||
based on $(git -C "$PSP" describe --tags --abbrev=0 2>/dev/null || echo 'unknown')
|
||||
built $(date -u +%Y-%m-%dT%H:%M:%SZ) on $(hostname)
|
||||
dirty $dirty uncommitted file(s) in the source tree at build time
|
||||
|
||||
Regenerate with ui/vendor/rebuild-perspective.sh
|
||||
EOF
|
||||
|
||||
echo
|
||||
ls -la "$VENDOR"/*.tgz | awk '{printf " %-52s %5.1f MB\n", $NF, $5/1048576}'
|
||||
echo
|
||||
ok "Done. Now, from ui/ — note the paths: npm resolves these from its cache
|
||||
otherwise, since the version in the filename never changes.
|
||||
|
||||
rm -rf node_modules/@perspective-dev
|
||||
npm install ./vendor/perspective-dev-{client,server,viewer,viewer-datagrid}-5.4.0.tgz
|
||||
git add vendor package-lock.json && git commit"
|
||||
[[ $dirty -gt 0 ]] && echo -e "\033[1;33m !\033[0m source tree had uncommitted changes — push them to the fork first"
|
||||
exit 0
|
||||
Loading…
Reference in New Issue
Block a user