pt/pf_app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update spec and CLAUDE.md for multi-slice, tags, and the bridge

Browse Source 
Documentation had drifted far enough to mislead: the spec described the
single-slice panel, the target/delta modes that no longer exist, and the
old `slice` request shape, while both files still claimed Perspective
4.4.0 from a CDN when it has been bundled inline at 5.2.0 since August.

pf_spec.md
- pf.log gains `tag`, with why it is written by a follow-up UPDATE
  rather than through the per-source templates in pf.sql.
- Operations envelope documents `slices`, `apply_mode`, and the OR-of-
  AND-groups WHERE clause (and why it cannot be flattened to IN lists).
- Scale documents per-measure resolution and `target_basis`, plus the
  two guards: non-selective slices, and proration across a ~zero pool.
- New routes: PATCH /log/:logid, table-info, bridge, source tags.
- Forecast View rewritten for the dockable panel and the ledger; adds
  the selection caveat around pf_iter and the expand-depth explanation.
- New Bridge View section; Log View gains inline tag editing.
- Status block refreshed, with a Fixed subsection recording the three
  correctness bugs and their causes.
- Open Questions: adds bridge drill-down and targeting one iter band;
  notes what the bridge partially answers.

CLAUDE.md
- Corrects the Perspective version and the CDN claim.
- Project layout now lists components/, including the two new files.
- Selection section covers multi-select and why duplicate effective
  slices are collapsed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
This commit is contained in:
Paul Trowbridge 2026-09-11 23:34:08 -04:00
parent 55814ee0d5
commit 119065ef59
2 changed files with 206 additions and 62 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
CLAUDE.md
View File

@ -15,7 +15,7 @@ UX mockup: `pf_ux_mockup.md`
- **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 5.2.0, **bundled inline via the `/inline` entrypoints — never from a CDN** (the 4.x CDN bundle resolves its server WASM to an unversioned path and silently pulls whatever is newest). See `PERSPECTIVE.md`.
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
---
@ -39,9 +39,12 @@ ui/src/
views/
Setup.jsx DB browser, source registration, col_meta editor
Baseline.jsx Version management, baseline workbench, reference load
Forecast.jsx Perspective pivot + operation panel (Scale/Recode/Clone)
Forecast.jsx Perspective pivot, selection handling, operation dispatch
components/
OperationPanel.jsx The adjustment workbench — ledger + scale/recode/clone forms
BridgeView.jsx Baseline → current waterfall by tag (exports buildSteps/layoutSteps)
Sidebar.jsx 3-step collapsible nav
StatusBar.jsx Source · version · row count · status
StatusBar.jsx Source · version · write target · row counts · theme
Timeline.jsx Date-range preview bar for baseline segments
```
@ -79,7 +82,9 @@ POST to `/api/versions/:id/{scale|recode|clone}` → SQL executed with `RETURNIN
## Slice mechanics
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` columns are kept as the slice. This slice populates the operation panel and is sent as the `slice` object in all operation POST bodies.
When the user clicks a pivot cell, `perspective-click` fires. The handler in `Forecast.jsx` extracts `[col, '==', value]` filters from `detail.config.filter` — only `role = dimension` and `role = date` columns are kept as the slice. A plain click replaces the selection; ctrl/⌘/shift-click toggles a slice in or out of it, so the panel holds a **list** of slices sent as `slices` in operation POST bodies (the single `slice` object is still accepted server-side).
`pf_iter` is not a col_meta column, so it is stripped when a slice is built: two cells differing only by iter band produce the same effective slice. Duplicates are collapsed before the request — without that, `apply_mode: each` would apply the same change twice.
**Limitation:** computed columns created by Perspective's split_by (e.g. Month, YearDate) don't map back to raw rows — only native dimension columns work for slice extraction.

255
pf_spec.md
View File

@ -113,10 +113,20 @@ CREATE TABLE pf.log (
operation text NOT NULL, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
slice jsonb, -- the WHERE conditions that defined the selection
params jsonb, -- operation parameters (increments, new values, scale factor, etc.)
note text -- user-provided comment
note text, -- user-provided comment
tag text -- initiative label, e.g. 'reduce_spend'
);
```
`tag` groups adjustments into initiatives. It is what the bridge walks: every entry
carrying the same tag becomes one step from baseline to current. Both `note` and `tag`
are annotations — they never affect forecast rows — so both stay editable after the
fact via `PATCH /api/log/:logid`.
Tags are written by a follow-up `UPDATE` after the operation runs, not by the generated
SQL. The templates in `pf.sql` are stored per source, so adding a `{{tag}}` token would
silently stop recording tags for any source that had not re-run *Generate SQL*.
### `pf.fc_{tname}_{version_id}` (dynamic, one per version)
Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`).
@ -318,35 +328,79 @@ All operations share a common request envelope:
```json
{
"pf_user": "paul.trowbridge",
"note": "optional comment",
"slice": {
"channel": "WHS",
"geography": "WEST"
}
"pf_user": "paul.trowbridge",
"note": "optional comment",
"tag": "reduce_spend",
"slices": [ { "channel": "WHS", "geography": "WEST" },
{ "channel": "DIR", "geography": "EAST" } ],
"apply_mode": "prorate"
}
```
`slice` keys must be `role = 'dimension'` columns per col_meta. Stored in `pf.log` as the implicit link to affected rows.
- `slices` — one or more slices. The legacy single `slice` object is still accepted and
treated as a one-entry list.
- `apply_mode` — `prorate` (default) treats the selection as one pool; `each` runs the
operation once per slice, producing one log entry per slice so they can be undone
separately. With a single slice the two are identical.
- `tag` — optional initiative label, stored on the log entry.
Slice keys must be `role = 'dimension'` or `role = 'date'` columns per col_meta. A slice
naming none of them is **rejected**: unknown keys are dropped when building the WHERE
clause, so such a slice would otherwise reduce to `TRUE` and apply the operation to the
entire version.
Several slices become an `OR` of `AND`-groups, not per-column `IN` lists — flattening
`{A:1,B:1}` and `{A:2,B:2}` into `A IN (1,2) AND B IN (1,2)` would also match `A:1,B:2`.
The result is parenthesised because callers append `AND pf_iter NOT IN (...)`, and `AND`
binds tighter than `OR`.
#### Scale
`POST /api/versions/:id/scale`
```json
{
"pf_user": "paul.trowbridge",
"note": "10% volume lift Q3 West",
"slice": { "channel": "WHS", "geography": "WEST" },
"value_incr": null,
"units_incr": 5000,
"pct": false
"pf_user": "paul.trowbridge",
"note": "10% volume lift Q3 West",
"tag": "volume_push",
"slices": [ { "channel": "WHS", "geography": "WEST" } ],
"apply_mode": "prorate",
"target_value": 12000,
"units_pct": 10,
"target_basis": "selected"
}
```
- `value_incr` / `units_incr` — absolute amounts to add (positive or negative). Either can be null.
- `pct: true` — treat as percentage of current slice total instead of absolute
- Excludes `exclude_iters` rows from the source selection
- Distributes increment proportionally across rows in the slice
Each measure is resolved **independently**, so a target on one and a percentage on the
other can be sent together. Per measure, exactly one of:
| Field | Meaning |
|---|---|
| `target_value` / `target_units` | the total to end up with |
| `value_pct` / `units_pct` | a percentage of the current total |
| `value_incr` / `units_incr` | an absolute amount to add |
| `target_price` | target value/units ratio; holds units constant |
The legacy global `pct: true` flag (meaning "the increments are percentages") is still
honoured.
`target_basis` decides what a target or percentage measures against:
- `adjustable` — only the rows the operation can write.
- `selected` (UI default) — everything the pivot shows for the slice, `exclude_iters`
rows included. Those rows cannot move, so the adjustable rows absorb the whole
difference and the pivot lands on the number you asked for. Without this, a target set
against a visible total overshoots by the excluded rows' contribution.
Behaviour:
- Excludes `exclude_iters` rows from the rows it writes, in every basis.
- Distributes the increment proportionally across rows in the slice.
- **Refuses to prorate a pool that nets to ~zero** — below 1% of gross. Each row's new
value is `(row / total) * increment`, so as the net approaches zero the multiplier
explodes and rows fly to extreme opposite values to reach the target. Offsetting
slices are the usual cause; `apply_mode: each` handles that correctly.
- Slices matching no rows, or already on target, are skipped and returned in
`slices_skipped` rather than silently counted as applied.
- Inserts rows tagged `iter = 'scale'`
#### Recode
@ -356,7 +410,7 @@ All operations share a common request envelope:
{
"pf_user": "paul.trowbridge",
"note": "Part discontinued, replaced by new SKU",
"slice": { "part": "OLD-SKU-001" },
"slices": [ { "part": "OLD-SKU-001" } ],
"set": { "part": "NEW-SKU-002" }
}
```
@ -374,7 +428,7 @@ All operations share a common request envelope:
{
"pf_user": "paul.trowbridge",
"note": "New customer win, similar profile to existing",
"slice": { "customer": "EXISTING CO", "channel": "DIR" },
"slices": [ { "customer": "EXISTING CO", "channel": "DIR" } ],
"set": { "customer": "NEW CO" },
"scale": 0.75
}
@ -391,6 +445,10 @@ All operations share a common request envelope:
|--------|-------|-------------|
| GET | `/api/versions/:id/log` | List all log entries for a version, newest first |
| DELETE | `/api/log/:logid` | Undo: delete all forecast rows with this logid, then delete log entry |
| PATCH | `/api/log/:logid` | Edit `note` and/or `tag`. Branches on whether a field was sent, so `""` clears rather than being read as "leave alone" |
| GET | `/api/versions/:id/table-info` | Physical forecast table, source table, and live row counts by `pf_iter` |
| GET | `/api/versions/:id/bridge` | Baseline → current rolled up by tag |
| GET | `/api/sources/:id/tags` | Tags used on this source with use counts, newest first — feeds tag autocomplete |
---
@ -484,30 +542,47 @@ Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any
### Forecast View
**Layout:**
**Layout:** the operation panel docks **bottom** (default), **right**, or **floats** over
the pivot (drag its header to move, corner grip to resize). Position and size persist to
`localStorage`. It closes via its header ×, `Esc`, or the toolbar toggle, which shows the
selection count while shut.
```
┌─────────────────────────────────────────────────────────────────┐
│ [Version label] [Refresh] [Save layout] [Reset layout] │
├──────────────────────────────────────┬──────────────────────────┤
│ │ │
│ Perspective Viewer │ Operation Panel │
│ (interactive pivot web component) │ (active when slice set) │
│ │ │
│ │ Slice: │
│ │ channel = WHS │
│ │ geography = WEST │
│ │ │
│ │ [ Scale ] [ Recode ] │
│ │ [ Clone ] │
│ │ │
│ │ ... operation form ... │
│ │ │
│ │ [ Submit ] │
│ │ │
└──────────────────────────────────────┴──────────────────────────┘
│ [Layout…] [Expand 0 1 2 3] [Refresh] [Change log] [Bridge] │
│ [Hide panel] │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Perspective Viewer (interactive pivot web component) │
│ │
├──────────────── drag to resize ─────────────────────────────────┤
│ SLICE 2 selected │ scale recode clone │ Amount │
│ channel=WHS │ Together | Each │ Baseline 1,000.00 │
│ channel=DIR │ │ ▪ reduce_spend -20.00 │
│ Clear selection │ │ ──────────────────── │
│ │ │ Adjustable 1,070.00 │
│ │ │ reference·fixed 921.72 │
│ │ │ ──────────────────── │
│ │ │ Selected total 1,991.72 │
│ │ │ ──────────────────── │
│ │ │ New value [ 2,000 ] │
│ │ │ Change [ 8 ] │
│ │ │ % change [ 0.4 ] │
│ │ tag [reduce_spend] │ [Apply Scale] │
└─────────────────────────────────────────────────────────────────┘
```
**Pivot control:** [Perspective](https://perspective.finos.org/) 4.4.0, loaded from CDN at runtime. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
**The ledger.** The scale form is one continuous statement rather than a totals display
plus a separate input form: baseline, each adjustment (grouped by tag), current, then the
edit. `New value`, `Change` and `% change` are three interchangeable editable rows —
typing in any one derives the other two, and whichever you typed in is what gets sent.
That replaces the old target/delta/percent mode toggle: the row you type in *is* the mode.
Rows the pivot shows but operations cannot write (`exclude_iters`, typically `reference`)
appear as their own line with a `Selected total` beneath, and a control chooses which of
the two a target measures against — see `target_basis` above.
**Pivot control:** [Perspective](https://github.com/perspective-dev/perspective) 5.2.0 (`@perspective-dev/*`), **bundled inline, not loaded from a CDN** — the `/inline` entrypoints embed the WASM so the version is pinned by `package-lock.json`. See `PERSPECTIVE.md`. Data is fetched from `GET /api/versions/:id/data` as an Arrow IPC binary stream and loaded into an in-browser Perspective worker — Perspective's native ingestion path. Supports grouping, splitting, filtering, sorting, and charting interactively. Layout (group_by, split_by, filters, plugin) is saved per version to `localStorage` via Save layout / Reset layout buttons.
**Large-dataset loading sequence:**
1. Client issues `GET /api/versions/:id/data`
@ -520,9 +595,25 @@ Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any
**Interaction flow:**
1. Click a cell or row in the pivot — the `perspective-click` event fires
2. `detail.config.filter` from the event is parsed: only `==` filters on `role = dimension` columns are extracted as the slice
3. Slice populates the Operation Panel — pick operation tab, fill in parameters
4. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
5. For recode, both the negative offset rows and positive replacement rows are returned and streamed
3. A plain click replaces the selection; **ctrl/⌘/shift-click toggles** a slice in or out of
it. The `CustomEvent` carries no modifier flags, so they are read from the `mousedown`
that preceded it. `perspective-select` (region drag) is wired defensively alongside.
4. Slice populates the Operation Panel — pick operation tab, fill in parameters
5. Submit → POST to API → new rows returned via `RETURNING *` are streamed directly into the Perspective table (`pspTable.update(rows)`) — no full reload needed
6. For recode, both the negative offset rows and positive replacement rows are returned and streamed
**Selection caveat.** `pf_iter` is not a `col_meta` column, so it is stripped when a slice
is built. Two cells differing only by iter band (baseline vs reference) produce the same
effective slice; duplicates are collapsed before the request, and the panel warns when a
selection covers fewer distinct slices than cells clicked. There is currently no way to
target one band of a slice.
**Expand depth.** Perspective's `GROUP BY ROLLUP` view contains every level of the
hierarchy, and `view.set_depth()` — which lives on the view, not in the saved config — is
the only thing hiding the deeper ones. The viewer rebuilds its view whenever it redraws,
which its Intersection/ResizeObserver triggers on tab refocus, leaving the tree fully
expanded. The last applied depth is therefore re-applied on `visibilitychange`, `focus`
and `pageshow`.
**Pivot default layout:** built from col_meta — first two `dimension` columns as `group_by`, `date` column as `split_by`. User can rearrange in Perspective settings panel and save.
@ -530,8 +621,43 @@ Segment 2 uses two OR groups; segment 3 has two AND conditions in one group. Any
### Log View
AG Grid list of log entries — user, timestamp, operation, slice, note, rows affected.
"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of Perspective table).
Modal list of log entries — timestamp, operation, slice, **tag**, note, rows affected.
"Undo" button per row → `DELETE /api/log/:logid` → grid and pivot refresh (full reload of
Perspective table).
Tag and note are edited inline (click, Enter to save, Esc to cancel) via
`PATCH /api/log/:logid`; the tag field completes from tags already used on the source.
Saving a tag regroups the ledger and bridge immediately, so history can be reclassified
after the fact.
### Bridge View
A waterfall answering "how did this version get from its baseline to where it stands?",
one step per initiative tag, opened from the toolbar.
```
6.0k ┤ ┌──────┐- - - -┐
│ │+3,624│ │
4.0k ┤ │ │ −3,800│
│ ┌─────┐- ┘ └ - - - ┘──┐ ┌─────┐
2.0k ┤ │2,734│ │+509│ │3,067│
0 ┴──┴─────┴────────────────┴────┴─┴─────┴──
Baseline clamp give food Current
```
**Scope:** the current slice selection (default when one exists), the pivot's current
filters, or the whole version. Selection scope uses the **union** of the selected slices —
the same reach an operation would have — with rows matching more than one slice deduped
by `pf_id` to match the `OR` semantics operations use.
Computed from the Perspective table already loaded in the browser rather than from
`/api/versions/:id/bridge`, so the figures always reconcile with what the pivot is
showing. The endpoint remains for API consumers.
**Colour** encodes polarity, not identity: increases and decreases are two poles of one
scale, so it uses a validated diverging pair (blue/red, CVD ΔE 21.6 — green/red is avoided
as the classic colourblind failure) with neutral grey anchors for baseline and current.
Every bar is directly labelled and a table view gives the same numbers at full precision.
---
@ -744,14 +870,16 @@ DELETE FROM pf.log WHERE id = {{logid}};
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
- **Export** — download forecast as CSV or push results to a reporting table
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION)
- **Export** — download forecast as CSV or push results to a reporting table. The bridge's table view is a partial stand-in for reading the numbers out, but there is no download.
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION). The bridge answers the within-version form of this question; across versions is still open.
- **Bridge drill-down** — click a step to list the adjustments behind it, or select that slice back in the pivot
- **Targeting one iter band** — make `pf_iter` part of a slice so an operation can act on, say, only the baseline rows of a selection (see Known issues)
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually.
- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema.
---
## Project Status — 2026-06-12
## Project Status — 2026-09-11
### What's working
- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo
@ -761,19 +889,30 @@ DELETE FROM pf.log WHERE id = {{logid}};
- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express
- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM
- 3-step collapsible sidebar (Setup / Baseline / Forecast)
- Setup view: DB table browser with preview modal, source registration, col_meta editor (`dim_group`/`dim_period_col` fields included), SQL generation
- Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder
- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation
- Baseline view: version management, multi-segment baseline workbench, canvas timeline, filter builder
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage
- Slice extraction from `perspective-click` event feeds operation panel directly
- Incremental row streaming: operation results (`RETURNING *`) applied to Perspective table via `pspTable.update()` — no full reload
- Status bar: shows current source · version · baseline row count · status
- Incremental row streaming: operation results (`RETURNING *`) applied via `pspTable.update()` — no full reload
- **Multi-slice operations**: ctrl/⌘-click accumulates slices; `apply_mode` prorate/each
- **Per-measure resolution**: target, percent or change amount independently per measure
- **`target_basis`**: a target measures against the adjustable rows or everything the pivot shows
- **Ledger panel**: baseline → adjustments → current → three interchangeable editable rows, docked bottom/right/floating
- **Tags and bridge**: initiative tags on log entries, editable after the fact, with a waterfall view scoped to selection / filters / version
- **Status bar** names the physical table writes land in, with live row counts by iter
### Known issues / next focus
- **Forecast view** — operation panel SQL generation complete; UI wiring to API still needed
- **Load progress bar** — jittery at high throughput; throttle to ~10 updates/sec
- **Default pivot layout** — per-source configurable layout not yet implemented; currently hardcodes first 2 dimensions
- **No "current version" persistence** — source/version selection resets on page reload
- **Perspective slice limitation** — computed date columns (Month, YearDate) from split_by don't map back to raw rows; only native dimension columns work for slice extraction
- **`pf_iter` not selectable** — it is not a col_meta column, so it is stripped from slices. Cells differing only by iter band collapse to one slice (duplicates are detected and collapsed, and the panel warns), and there is no way to operate on one band of a slice.
- **Per-row rounding drift** — the scale SQL rounds each row to 2dp, so a target of 1,000 across many rows can land on 999.99. Inherent to proportional distribution; a correction row would be needed to land exactly.
- **Manual caret expansion is not restored** — the depth re-apply on refocus only covers whole-tree depths set via the Expand buttons or a saved layout, since per-row expansion lives in the same discarded view.
- **Bridge has no drill-down** — clicking a step does not list its adjustments or select that slice back in the pivot.
- **Light surface only** — app chrome is light throughout; the dark toggle currently re-themes only the Perspective viewer.
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version.
- **No migration sequence** — `01_schema.sql` carries `ADD COLUMN IF NOT EXISTS` inline for the `tag` column, which covers fresh installs and re-runs, but there is no ordered migration mechanism.
- **No tests** — SQL generation is token substitution against append-only tables and is entirely untested.
### Fixed
- **Non-selective slices applied to the whole version** — a slice naming no filterable column reduced to `TRUE`. Now rejected on all three operations.
- **Proration across a near-zero pool** — rows flew to extreme opposite values to reach a target. Refused when the net is below 1% of gross; `apply_mode: each` is the alternative.
- **Targets overshooting by excluded rows** — see `target_basis`.