pf_app/CLAUDE.md

# Pivot Forecast — CLAUDE.md

## What this app is

A web app for building named forecast scenarios against any PostgreSQL table. The workflow: load historical actuals as a baseline (optionally date-shifted into the forecast period), then apply incremental adjustments (scale, recode, clone) to build a plan. All changes are append-only, fully audited, and reversible by log entry.

Full spec: `pf_spec.md`
Data transport architecture options: `pf_perspective_options.md`

---

## Tech stack

- **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.2.0, **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`

---

## Project layout

```
server.js               Express entry point; pg pool; session; type parsers for bigint/numeric
routes/
  auth.js               POST /api/login, /api/logout, GET /api/me; login throttle
  tables.js             GET /api/tables, /api/tables/:schema/:tname/preview
  sources.js            Source registration, col_meta, SQL generation
  versions.js           Version CRUD, baseline/reference load, data stream
  operations.js         scale, recode, clone, undo — the core forecast ops
  log.js                GET /api/versions/:id/log, DELETE /api/log/:logid
lib/
  sql_generator.js      buildFilterClause, token substitution helpers
  auth.js               scrypt hash/verify, requireAuth, sessionUser; `node lib/auth.js hash` CLI
  utils.js
setup_sql/
  01_schema.sql         pf schema DDL — run once to install
  02_auth.sql           pf.app_user + pf.session
ui/src/
  auth.jsx              AuthProvider/useAuth; wraps fetch so any 401 returns to login
  views/
    Login.jsx           Sign-in form
    Setup.jsx           DB browser, source registration, col_meta editor
    Baseline.jsx        Version management, baseline workbench, reference load
    Forecast.jsx        Perspective pivot, selection handling, operation dispatch
  components/
    OperationPanel.jsx  The adjustment workbench — ledger + scale/recode/clone forms
    BridgeView.jsx      Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
    Sidebar.jsx         3-step collapsible nav
    StatusBar.jsx       Source · version · write target · row counts · theme
    Timeline.jsx        Date-range preview bar for baseline segments
```

---

## Database schema (`pf`)

- **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at`
- **`pf.session`** — express-session store (connect-pg-simple layout)
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month

### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`

---

## Core data flow

### Initial load (Forecast view)
`GET /api/versions/:id/data` → Arrow IPC binary stream → `worker.table(buffer)` in Perspective WASM

**Why one batch (not streaming):** pg returns `bigint`/`numeric` as strings by default — type parsers in `server.js` coerce them to numbers. Per-batch Arrow encoding creates independent dictionaries that cause Perspective WASM to crash on dictionary replacement messages. Server accumulates all rows, emits one record batch.

### Forecast operations
POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload.

### Undo
`DELETE /api/log/:logid` → removes rows by logid → **full Perspective reload** (known wart).

---

## Slice mechanics

When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).

Dragging across a block of cells selects a region. The datagrid runs in `edit_mode: SELECT_REGION` (forced on restore, so a saved layout can't switch it off) and reports the region as a `perspective-select` event carrying a Perspective **ViewWindow** — `{ start_row, end_row, start_col, end_col }`, *not* the per-row `insertConfigs` payload an older API used. It fires on every mouseover as the region grows, so the handler only records the latest window and a window-level `mouseup` commits it. A single-cell region is ignored there: `perspective-click` already owns plain and modifier clicks, and handling it in both places would undo a ctrl-click toggle.

Turning a region back into slices re-derives, per cell, the same filters Perspective attaches to a click — row dimensions from the view's `__ROW_PATH__` (raw values, so dates stay epoch millis rather than whatever the grid formatted them as), column dimensions from the split_by segments of the column name. The grand-total row resolves to no dimension at all and is skipped; that would mean "the whole version".

**Selection highlight.** The datagrid highlights whatever sits in its own `model._selection_state.selected_areas`, and wipes that list on every mousedown — so a multi-slice selection built up over several ctrl-clicks would only ever show the last cell. `Forecast.jsx` keeps `areasRef`, a `sliceKey -> rectangles` map parallel to `slices`, and an effect pushes the full set back and redraws after every change. Deselecting anywhere (ctrl-click, the panel's ×, Clear selection) prunes the map by live slice key, so the grid and the panel can't disagree.

`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.

**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.

---

## Column hierarchy (collapse / expand)

The two pivot axes collapse by completely different mechanisms, and the asymmetry is a
Perspective constraint, not a choice:

- **Rows.** The `GROUP BY ROLLUP` view holds every level at once; `view.set_depth()` — which
  lives on the view, not the config — hides the deeper ones. That is what the `EXPAND 0 1 2 3`
  buttons drive, via `applyDepth()`.
- **Columns.** There is no equivalent. `expand()` / `collapse()` take a **row index**,
  `ViewConfig` has `group_by_depth` but no `split_by_depth`, and `split_rollup_mode`
  (`'flat' | 'rollup'`) only chooses whether subtotal column groups are *emitted* — it is a
  view shape, not an interaction. So `applySplitDepth(n)` collapses by restoring a
  **truncated `split_by`**, which rebuilds the view.

Three things follow from the rebuild, and each is handled:

1. The full hierarchy has to be remembered separately — once collapsed, `viewer.save()`
   only reports the short `split_by`. `splitFullRef` / `splitFull` hold it, and it is
   persisted into the saved layout as `split_full` so a reload while collapsed can still
   expand back. `adoptSplit()` is the single place it is set.
2. `perspective-config-update` fires for our own restore as well as the user rearranging
   the pivot. `collapsingRef` distinguishes them — without it, a collapse would overwrite
   the full hierarchy with the truncated one and the deeper levels would be unreachable.
3. Row depth lives on the discarded view, so `applyDepth(expandDepthRef.current)` is
   re-applied afterwards — the same wart as the refocus re-apply.

The selection is cleared on every change: slices name the split_by dimensions they were
cut from, and the highlight is keyed on grid coordinates. Neither survives a column axis
that just changed shape.

**Limitation:** this is whole-axis, not per-branch. Excel can collapse 2025 while 2026
stays expanded; truncating `split_by` collapses every column group at that level together.
Per-branch is not reachable — `columns` selects which *measures* appear, not individual
split combinations.

---

## Operation SQL patterns

All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.

- **Scale** — distributes `value_incr`/`units_incr` proportionally across rows in the slice using window functions
- **Recode** — inserts negative rows (zero out original) + positive rows with `{{set_clause}}` dimension overrides; both share the same logid
- **Clone** — copies the slice with `{{set_clause}}` overrides and `{{scale_factor}}` multiplier; original untouched

`build_where()` validates every slice key against col_meta (only `role = dimension` allowed). Values are escaped but not parameterized — consistent with existing patterns, debuggable in pg logs.

---

## Authentication

Everything under `/api` except the auth routes sits behind a session; the React
app is only mounted once there is one (`Gate` in `main.jsx`), because its load
effects call the API immediately.

- **Accounts:** `pf.app_user` — scrypt hashes from `lib/auth.js`, never plaintext.
  Managed with `./pf.sh add-user | passwd | list-users | disable-user | enable-user`;
  the password is read on stdin and hashed before it reaches psql.
- **Sessions:** `express-session` + `connect-pg-simple` in `pf.session`, so a
  restart doesn't sign everyone out and a session can be revoked by deleting its
  row (`disable-user` does exactly that). Cookie `pf.sid`: httpOnly, SameSite=Lax,
  Secure unless `COOKIE_SECURE=false`, 12h rolling.
- **Config:** `SESSION_SECRET` is required — the server exits at boot without one.
  `TRUST_PROXY` (default 1) makes `req.ip` and secure-cookie detection correct
  behind the TLS proxy. `CORS_ORIGIN` is the only way CORS is enabled at all; a
  wildcard origin plus a session cookie would be cross-site request forgery by
  construction.
- **Login hardening:** `routes/auth.js` throttles to 10 failures per IP per 15
  minutes (in-memory), returns one message for unknown/wrong/disabled alike, and
  regenerates the session id on success.

**Identity is server-side.** `pf_user`, `created_by` and `closed_by` come from
`sessionUser(req)`, never from the request body — the UI used to send a hardcoded
`pf_user: 'admin'`, which any client could have set to anything. The audit log
now names the account that made the change.

## Light / dark mode

Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.

- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`

## Known issues / active work

- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
- Load progress bar is jittery — needs throttle (~10 updates/sec)
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
- Source/version selection doesn't persist across page reload
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version

## Deferred (not in v1)
Baseline replay (`replay: true` returns 501), approval workflow, territory filtering, export, version comparison, multi-DB connections.