diff --git a/CLAUDE.md b/CLAUDE.md index fd280a0..68dff00 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,6 +38,7 @@ setup_sql/ 01_schema.sql pf schema DDL — run once to install 02_auth.sql pf.app_user + pf.session ui/src/ + observerShim.js Wraps window.Intersection/ResizeObserver; MUST be main.jsx's first import auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login views/ Login.jsx Sign-in form @@ -59,7 +60,7 @@ ui/src/ - **`pf.source`** — registered source tables - **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below) - **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations -- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`) +- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it. - **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb - **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time - **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at` @@ -86,14 +87,78 @@ Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WA ### Display grain Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings. +### Segment and note columns +`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table +does not itself carry: + +- **`pf_segment`** — for a baseline or reference row, that load's label (`tag`, else + `note`); `'(adjustment)'` for everything else +- **`pf_note`** — the free text on a scale/recode/clone; null on loads + +They are deliberately separate: commingling a segment name with an adjustment note +makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join +adds no rows. The operation routes stamp the same two fields onto the rows they push +back incrementally, since those come from `RETURNING *` and would otherwise arrive +without them. + ### Forecast operations POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums. +### Price or volume (`plug`) +A sales figure alone does not say which of price or volume moved, so scale takes +`plug` — `'price'` (default, volume holds) or `'volume'` (price holds, units scale +in proportion). Resolved in `resolveIncrs()` in `routes/operations.js`; the panel +only offers it when the edit is dollars-only, because naming units or price has +already answered it. The semantics come from the predecessor Excel model, +`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`: + +``` +plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange +plug price: fVol = pVol + bVol +``` + +A `target_price` with a `target_units` alongside is that form's Edit Price mode, +where both are inputs and dollars fall out. The ledger's **Result** line previews +value, units and price together using the same rules, so what you see is what +gets written. + ### Undo `DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload. --- +## Row depth and the observer shim + +`set_depth()` and per-node expand/collapse live on the **view**, not in `ViewConfig`. +So every view rebuild starts fully expanded, and nothing in Perspective restores it — +the app has to re-apply depth itself. + +The engine has no notion of focus or tab visibility. It re-renders because +`IntersectionObserver` or `ResizeObserver` told it the element is visible again, and +*that* is what discards the view. `ui/src/observerShim.js` wraps both constructors so +the re-apply fires on the actual callback rather than on a window `focus` guess. + +**It must stay the first import in `main.jsx`.** Perspective's viewer captures the +constructors at module-evaluation time (`var it=window.ResizeObserver; var +st=window.IntersectionObserver`), so any import that reaches `perspective-viewer` +first leaves the shim with nothing to intercept. `focus`/`visibilitychange`/`pageshow` +remain as a backstop for when that happens. + +Two things learned the hard way, both easy to repeat: + +- Around a rebuild the viewer throws `No table set` from `getView()` while + `getTable()` resolves happily. Gating on the table does not help; + `applyDepthWhenReady()` retries `applyDepth` until it stops throwing. +- `getView()` returns a **fresh wrapper object every call**, so object identity + cannot be used to detect a rebuild. Two consecutive calls always look different. + +Tracing lives behind `localStorage.pf_debug = '1'`, prefixed `[pf-depth]` and +`[pf-obs]`. + +**Not solved:** per-node `+/−` is still lost on rebuild. The view exposes +`expand(row)` / `collapse(row)` with no getter, so the state cannot be read back — +it would have to be shadowed by intercepting the grid's own calls. + ## Slice mechanics When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side). @@ -199,8 +264,15 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit ## Known issues / active work -- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion -- Load progress bar is jittery — needs throttle (~10 updates/sec) +- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29` + (2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in + pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then + ~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The + remaining lever is **dynamic grain** — group by the fields the current pivot + actually uses rather than every `in_grain` column; see `pf_spec.md` → + §Display-grain pre-aggregation, "the dynamic variant" +- Per-node row expand/collapse is lost whenever the view rebuilds; see §Row depth and + the observer shim - Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions) - Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`, `App.jsx`). It is re-validated against the live list whenever that list changes, so a