pf_app/CLAUDE.md

Pivot Forecast — CLAUDE.md

What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec: pf_spec.md Data transport architecture options: pf_perspective_options.md

---

Tech stack

• Backend: Node.js / Express (server.js)
• Database: PostgreSQL — isolated pf schema
• Frontend: React + Vite + Tailwind CSS in ui/; built output lands in public/app/
• Pivot: Perspective (@perspective-dev/* distribution, not FINOS @finos/perspective) 5.2.0, bundled inline via the /inline entrypoints — never from a CDN (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See PERSPECTIVE.md.
• Dev: npm run dev (nodemon) in root; npm run build in ui/

---

Project layout

server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation
  versions.js           Version CRUD, baseline/reference load, data stream
  operations.js         scale, recode, clone, undo — the core forecast ops
  log.js                GET /api/versions/:id/log, DELETE /api/log/:logid
lib/
  sql_generator.js      buildFilterClause, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
  utils.js
setup_sql/
  01_schema.sql         pf schema DDL — run once to install
  02_auth.sql           pf.app_user + pf.session
ui/src/
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         3-step collapsible nav
    StatusBar.jsx       Source · version · write target · row counts · theme
    Timeline.jsx        Date-range preview bar for baseline segments

---

Database schema (pf)

• pf.source — registered source tables
• pf.col_meta — column roles: dimension | value | units | date | filter | ignore; is_key marks dimensions used in slice WHERE clauses; dim_group groups functionally dependent columns (e.g. date + its derived year/month dimensions); dim_period_col maps a dimension to a pf.dim_period column so date-adjacent values are derived at load time rather than copied raw
• pf.version — named forecast scenarios; exclude_iters (default ["reference"]) blocks those iter values from all operations
• pf.fc_{tname}_{version_id} — one forecast table per version; contains both operational rows (pf_iter = baseline|scale|recode|clone) and reference rows (pf_iter = reference)
• pf.log — audit log; every write gets one entry; slice + params stored as jsonb
• pf.sql — generated SQL templates per source/operation; tokens substituted at request time
• pf.app_user — login accounts; scrypt pass_hash, is_active, last_login_at
• pf.session — express-session store (connect-pg-simple layout)
• pf.dim_period — calendar lookup table (2018–2035); one row per month keyed on sdat (month start date); provides cal/fiscal year, quarter, and month columns; populated by setup_sql/gen_dim_period.sql with a configurable fiscal year start month

Key token substitution tokens

{{fc_table}}, {{where_clause}}, {{exclude_clause}}, {{logid}}, {{pf_user}}, {{value_incr}}, {{units_incr}}, {{pct}}, {{set_clause}}, {{scale_factor}}, {{date_offset}}, {{filter_clause}}

---

Core data flow

Initial load (Forecast view)

GET /api/versions/:id/data → Arrow IPC binary stream → worker.table(buffer) in Perspective WASM

Why one batch (not streaming): pg returns bigint/numeric as strings by default — type parsers in server.js coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.

Forecast operations

POST to /api/versions/:id/{scale|recode|clone} → SQL executed with RETURNING * → new rows returned as JSON → pspTable.update(rows) — no full reload.

Undo

DELETE /api/log/:logid → removes rows by logid → full Perspective reload (known wart).

---

Slice mechanics

When the user clicks a pivot cell, perspective-click fires. The handler in Forecast.jsx extracts [col, '==', value] filters from detail.config.filter — only role = dimension and role = date columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a list of slices sent as slices in operation POST bodies (the single slice object is still accepted server-side).

Dragging across a block of cells selects a region. The datagrid runs in edit_mode: SELECT_REGION (forced on restore, so a saved layout can't switch it off) and reports the region as a perspective-select event carrying a Perspective ViewWindow — { start_row, end_row, start_col, end_col }, not the per-row insertConfigs payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level mouseup commits it. A single-cell region is ignored there: perspective-click already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.

Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's __ROW_PATH__ (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".

Selection highlight. The datagrid highlights whatever sits in its own model._selection_state.selected_areas, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. Forecast.jsx keeps areasRef, a sliceKey -> rectangles map parallel to slices, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.

pf_iter is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, apply_mode: each would apply the same change twice.

Limitation: computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.

---

Column hierarchy (collapse / expand)

The two pivot axes collapse by completely different mechanisms, and the asymmetry is a Perspective constraint, not a choice:

• Rows. The GROUP BY ROLLUP view holds every level at once; view.set_depth() — which lives on the view, not the config — hides the deeper ones. That is what the EXPAND 0 1 2 3 buttons drive, via applyDepth().
• Columns. There is no equivalent. expand() / collapse() take a row index, ViewConfig has group_by_depth but no split_by_depth, and split_rollup_mode ('flat' | 'rollup') only chooses whether subtotal column groups are emitted — it is a view shape, not an interaction. So applySplitDepth(n) collapses by restoring a truncated split_by, which rebuilds the view.

Three things follow from the rebuild, and each is handled:

1. The full hierarchy has to be remembered separately — once collapsed, viewer.save() only reports the short split_by. splitFullRef / splitFull hold it, and it is persisted into the saved layout as split_full so a reload while collapsed can still expand back. adoptSplit() is the single place it is set.
2. perspective-config-update fires for our own restore as well as the user rearranging the pivot. collapsingRef distinguishes them — without it, a collapse would overwrite the full hierarchy with the truncated one and the deeper levels would be unreachable.
3. Row depth lives on the discarded view, so applyDepth(expandDepthRef.current) is re-applied afterwards — the same wart as the refocus re-apply.

The selection is cleared on every change: slices name the split_by dimensions they were cut from, and the highlight is keyed on grid coordinates. Neither survives a column axis that just changed shape.

Limitation: this is whole-axis, not per-branch. Excel can collapse 2025 while 2026 stays expanded; truncating split_by collapses every column group at that level together. Per-branch is not reachable — columns selects which measures appear, not individual split combinations.

---

Operation SQL patterns

All three operations follow the same structure: insert a pf.log row in a CTE, then insert forecast rows referencing its id. {{where_clause}} is built from the slice; {{exclude_clause}} blocks exclude_iters rows.

• Scale — distributes value_incr/units_incr proportionally across rows in the slice using window functions
• Recode — inserts negative rows (zero out original) + positive rows with {{set_clause}} dimension overrides; both share the same logid
• Clone — copies the slice with {{set_clause}} overrides and {{scale_factor}} multiplier; original untouched

build_where() validates every slice key against col_meta (only role = dimension allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.

---

Authentication

Everything under /api except the auth routes sits behind a session; the React app is only mounted once there is one (Gate in main.jsx), because its load effects call the API immediately.

• Accounts: pf.app_user — scrypt hashes from lib/auth.js, never plaintext. Managed with ./pf.sh add-user | passwd | list-users | disable-user | enable-user; the password is read on stdin and hashed before it reaches psql.
• Sessions: express-session + connect-pg-simple in pf.session, so a restart doesn't sign everyone out and a session can be revoked by deleting its row (disable-user does exactly that). Cookie pf.sid: httpOnly, SameSite=Lax, Secure unless COOKIE_SECURE=false, 12h rolling.
• Config: SESSION_SECRET is required — the server exits at boot without one. TRUST_PROXY (default 1) makes req.ip and secure-cookie detection correct behind the TLS proxy. CORS_ORIGIN is the only way CORS is enabled at all; a wildcard origin plus a session cookie would be cross-site request forgery by construction.
• Login hardening: routes/auth.js throttles to 10 failures per IP per 15 minutes (in-memory), returns one message for unknown/wrong/disabled alike, and regenerates the session id on success.

Identity is server-side. pf_user, created_by and closed_by come from sessionUser(req), never from the request body — the UI used to send a hardcoded pf_user: 'admin', which any client could have set to anything. The audit log now names the account that made the change.

Light / dark mode

Theme state lives in ui/src/theme.jsx — a React context (ThemeContext) with a ThemeProvider that wraps the app in main.jsx.

• Storage key: pf_dark in localStorage; falls back to window.matchMedia('(prefers-color-scheme: dark)') on first visit
• Toggle: setDark(d => !d) in StatusBar.jsx; effect writes localStorage and toggles the .dark class on <html>
• CSS: ui/src/index.css defines CSS custom properties under :root (light) and .dark. All Tailwind color overrides are written as .dark .bg-white { ... } etc. — no Tailwind dark-mode config needed
• Palette: dark mode uses Perspective's "Pro Dark" colours (--bg-primary: #242526, panels #2a2c2f, gridlines #3b3f46, text #c5c9d0)
• Perspective viewer: Forecast.jsx calls viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light') both on initial load and in a useEffect([dark, versionId]) so the viewer stays in sync when the toggle fires
• Consuming the theme: import useTheme from '../theme.jsx' then const { dark, setDark } = useTheme()

Known issues / active work

• Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
• Load progress bar is jittery — needs throttle (~10 updates/sec)
• Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
• Source/version selection doesn't persist across page reload
• Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version

Deferred (not in v1)

Baseline replay (replay: true returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections.