Paul Trowbridge
119065ef59
Documentation had drifted far enough to mislead: the spec described the single-slice panel, the target/delta modes that no longer exist, and the old `slice` request shape, while both files still claimed Perspective 4.4.0 from a CDN when it has been bundled inline at 5.2.0 since August. pf_spec.md - pf.log gains `tag`, with why it is written by a follow-up UPDATE rather than through the per-source templates in pf.sql. - Operations envelope documents `slices`, `apply_mode`, and the OR-of- AND-groups WHERE clause (and why it cannot be flattened to IN lists). - Scale documents per-measure resolution and `target_basis`, plus the two guards: non-selective slices, and proration across a ~zero pool. - New routes: PATCH /log/:logid, table-info, bridge, source tags. - Forecast View rewritten for the dockable panel and the ledger; adds the selection caveat around pf_iter and the expand-depth explanation. - New Bridge View section; Log View gains inline tag editing. - Status block refreshed, with a Fixed subsection recording the three correctness bugs and their causes. - Open Questions: adds bridge drill-down and targeting one iter band; notes what the bridge partially answers. CLAUDE.md - Corrects the Perspective version and the CDN claim. - Project layout now lists components/, including the two new files. - Selection section covers multi-select and why duplicate effective slices are collapsed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
126 lines
8.1 KiB
Markdown
126 lines
8.1 KiB
Markdown
# Pivot Forecast — CLAUDE.md
|
||
|
||
## What this app is
|
||
|
||
A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.
|
||
|
||
Full spec: `pf_spec.md`
|
||
Data transport architecture options: `pf_perspective_options.md`
|
||
UX mockup: `pf_ux_mockup.md`
|
||
|
||
---
|
||
|
||
## Tech stack
|
||
|
||
- **Backend:** Node.js / Express (`server.js`)
|
||
- **Database:** PostgreSQL — isolated `pf` schema
|
||
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
|
||
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.2.0, **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
|
||
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
|
||
|
||
---
|
||
|
||
## Project layout
|
||
|
||
```
|
||
server.js Express entry point; pg pool; type parsers for bigint/numeric
|
||
routes/
|
||
tables.js GET /api/tables, /api/tables/:schema/:tname/preview
|
||
sources.js Source registration, col_meta, SQL generation
|
||
versions.js Version CRUD, baseline/reference load, data stream
|
||
operations.js scale, recode, clone, undo — the core forecast ops
|
||
log.js GET /api/versions/:id/log, DELETE /api/log/:logid
|
||
lib/
|
||
sql_generator.js buildFilterClause, token substitution helpers
|
||
utils.js
|
||
setup_sql/
|
||
01_schema.sql pf schema DDL — run once to install
|
||
ui/src/
|
||
views/
|
||
Setup.jsx DB browser, source registration, col_meta editor
|
||
Baseline.jsx Version management, baseline workbench, reference load
|
||
Forecast.jsx Perspective pivot, selection handling, operation dispatch
|
||
components/
|
||
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms
|
||
BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
|
||
Sidebar.jsx 3-step collapsible nav
|
||
StatusBar.jsx Source · version · write target · row counts · theme
|
||
Timeline.jsx Date-range preview bar for baseline segments
|
||
```
|
||
|
||
---
|
||
|
||
## Database schema (`pf`)
|
||
|
||
- **`pf.source`** — registered source tables
|
||
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
|
||
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
|
||
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
|
||
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
|
||
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
|
||
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
|
||
|
||
### Key token substitution tokens
|
||
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
|
||
|
||
---
|
||
|
||
## Core data flow
|
||
|
||
### Initial load (Forecast view)
|
||
`GET /api/versions/:id/data` → Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM
|
||
|
||
**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.
|
||
|
||
### Forecast operations
|
||
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload.
|
||
|
||
### Undo
|
||
`DELETE /api/log/:logid` → removes rows by logid → **full Perspective reload** (known wart).
|
||
|
||
---
|
||
|
||
## Slice mechanics
|
||
|
||
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).
|
||
|
||
`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.
|
||
|
||
**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.
|
||
|
||
---
|
||
|
||
## Operation SQL patterns
|
||
|
||
All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.
|
||
|
||
- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions
|
||
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
|
||
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched
|
||
|
||
`build_where()` validates every slice key against col_meta (only `role = dimension` allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.
|
||
|
||
---
|
||
|
||
## Light / dark mode
|
||
|
||
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
|
||
|
||
- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
|
||
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
|
||
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
|
||
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
|
||
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
|
||
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
|
||
|
||
## Known issues / active work
|
||
|
||
- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
|
||
- Load progress bar is jittery — needs throttle (~10 updates/sec)
|
||
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
|
||
- Source/version selection doesn't persist across page reload
|
||
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
|
||
|
||
## Deferred (not in v1)
|
||
Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections.
|