From 088b6a30c5d328df7e282d748ee8f1cac07e7bce Mon Sep 17 00:00:00 2001 From: Paul Trowbridge Date: Wed, 16 Sep 2026 23:48:25 -0400 Subject: [PATCH] Document the plug, the segment columns, and the observer shim Three things a reader would otherwise have to rediscover, and one of them is load-bearing: observerShim.js has to be main.jsx's first import or it silently intercepts nothing, which no amount of reading the shim itself tells you. Also records the two false trails found while getting there -- getTable() resolving while getView() throws, and getView() returning a fresh wrapper every call so identity cannot detect a rebuild. Known issues rewritten against what is actually true now: the operation panel wiring and the progress-bar throttle are done, and the load-time entry now says what was measured -- that the cost is row count, not payload, and dynamic grain is the remaining lever. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 75 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fd280a0..68dff00 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,6 +38,7 @@ setup_sql/ 01_schema.sql pf schema DDL — run once to install 02_auth.sql pf.app_user + pf.session ui/src/ + observerShim.js Wraps window.Intersection/ResizeObserver; MUST be main.jsx's first import auth.jsx AuthProvider/useAuth; wraps fetch so any 401 returns to login views/ Login.jsx Sign-in form @@ -59,7 +60,7 @@ ui/src/ - **`pf.source`** — registered source tables - **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw; `in_grain` flags dimension/date columns that define the **display grain** (see below) - **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations -- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`) +- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`). Indexed on `pf_logid`, which is how undo, the change-log aggregate and the grain key all find their rows — without it each is a sequential scan of the whole table. Tables created before that index was added do not have it. - **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb - **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time - **`pf.app_user`** — login accounts; scrypt `pass_hash`, `is_active`, `last_login_at` @@ -86,14 +87,78 @@ Either way: Arrow IPC binary stream → `worker.table(buffer)` in Perspective WA ### Display grain Aggregating to the grain the pivot actually displays is the load-time fix — measured 534,902 → 6,154 rows on `osm_stack`. It keeps the **native** Perspective engine, so expand/collapse/depth/sort/filter all still work. Set the grain in Setup (`in_grain` per column); it is baked into `pf.sql` at Generate SQL time so load and operations agree. `grainOf()` in `lib/sql_generator.js` is the single definition of what the grain is — `Setup.jsx` and `routes/log.js` mirror it. Full design: `pf_spec.md` → §Display-grain pre-aggregation. Why not a DuckDB virtual server: `pf_perspective_options.md` → §Spike findings. +### Segment and note columns +`/data` and `/agg` both LEFT JOIN `pf.log` and emit two columns the forecast table +does not itself carry: + +- **`pf_segment`** — for a baseline or reference row, that load's label (`tag`, else + `note`); `'(adjustment)'` for everything else +- **`pf_note`** — the free text on a scale/recode/clone; null on loads + +They are deliberately separate: commingling a segment name with an adjustment note +makes neither pivotable. In grain mode `pf_logid` is part of the grain, so the join +adds no rows. The operation routes stamp the same two fields onto the rows they push +back incrementally, since those come from `RETURNING *` and would otherwise arrive +without them. + ### Forecast operations POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNING *` → new rows returned as JSON → `pspTable.update(rows)` — no full reload. In grain mode the operation's final CTE aggregates its own new rows to grain first; since `pf_logid` is part of `pf_gkey` those keys are always new, so `update()` **appends** and the view re-sums. +### Price or volume (`plug`) +A sales figure alone does not say which of price or volume moved, so scale takes +`plug` — `'price'` (default, volume holds) or `'volume'` (price holds, units scale +in proportion). Resolved in `resolveIncrs()` in `routes/operations.js`; the panel +only offers it when the edit is dollars-only, because naming units or price has +already answered it. The semantics come from the predecessor Excel model, +`/opt/forecast_api/VBA/fpvt.frm` → `calc_val` / `calc_price`: + +``` +plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange +plug price: fVol = pVol + bVol +``` + +A `target_price` with a `target_units` alongside is that form's Edit Price mode, +where both are inputs and dollars fall out. The ledger's **Result** line previews +value, units and price together using the same rules, so what you see is what +gets written. + ### Undo `DELETE /api/log/:logid` → removes rows by logid → `table.remove()` of the affected index values (`pf_gkeys` in grain mode, `pf_ids` in raw mode); the view re-sums. No full reload. --- +## Row depth and the observer shim + +`set_depth()` and per-node expand/collapse live on the **view**, not in `ViewConfig`. +So every view rebuild starts fully expanded, and nothing in Perspective restores it — +the app has to re-apply depth itself. + +The engine has no notion of focus or tab visibility. It re-renders because +`IntersectionObserver` or `ResizeObserver` told it the element is visible again, and +*that* is what discards the view. `ui/src/observerShim.js` wraps both constructors so +the re-apply fires on the actual callback rather than on a window `focus` guess. + +**It must stay the first import in `main.jsx`.** Perspective's viewer captures the +constructors at module-evaluation time (`var it=window.ResizeObserver; var +st=window.IntersectionObserver`), so any import that reaches `perspective-viewer` +first leaves the shim with nothing to intercept. `focus`/`visibilitychange`/`pageshow` +remain as a backstop for when that happens. + +Two things learned the hard way, both easy to repeat: + +- Around a rebuild the viewer throws `No table set` from `getView()` while + `getTable()` resolves happily. Gating on the table does not help; + `applyDepthWhenReady()` retries `applyDepth` until it stops throwing. +- `getView()` returns a **fresh wrapper object every call**, so object identity + cannot be used to detect a rebuild. Two consecutive calls always look different. + +Tracing lives behind `localStorage.pf_debug = '1'`, prefixed `[pf-depth]` and +`[pf-obs]`. + +**Not solved:** per-node `+/−` is still lost on rebuild. The view exposes +`expand(row)` / `collapse(row)` with no getter, so the state cannot be read back — +it would have to be shadowed by intercepting the grid's own calls. + ## Slice mechanics When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side). @@ -199,8 +264,15 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit ## Known issues / active work -- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion -- Load progress bar is jittery — needs throttle (~10 updates/sec) +- **Load time is dominated by row count, not payload size.** On `fc_osm_skinny_29` + (2.56M raw rows) a 24-column grain still yields 285,685 rows: ~2s to aggregate in + pg, but ~15s to serialise those rows out of Postgres and parse them into JS, then + ~3s to build Arrow. Halving the payload (the `pf_gkey` md5) barely moved it. The + remaining lever is **dynamic grain** — group by the fields the current pivot + actually uses rather than every `in_grain` column; see `pf_spec.md` → + §Display-grain pre-aggregation, "the dynamic variant" +- Per-node row expand/collapse is lost whenever the view rebuilds; see §Row depth and + the observer shim - Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions) - Source/version selection persists in `localStorage` (`pf_sourceId` / `pf_versionId`, `App.jsx`). It is re-validated against the live list whenever that list changes, so a