pt/pf_app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the docs against the code, and move the design docs to docs/

Browse Source 
An audit against the code found every doc had drifted. pf_spec.md mostly
described behaviour the app no longer has and missed auth, territory,
layouts, load plans, segment naming and scope entirely; CLAUDE.md had
grown to carry the history as well as the rules, and loads into every
session.

- CLAUDE.md: rules, invariants and single-definition pointers only, with
  stale claims corrected (project layout, routes, schema, the pf_ slice
  columns, the price/volume and footing rules, known issues, deferred).
  Each section that lost its reasoning links to the spec.
- docs/pf_spec.md: rewritten as the current design doc under twelve fixed
  headings, holding the history and rationale moved out of CLAUDE.md, the
  code issues found on the way, and a proposed (not built) design for
  additional measures -- cost, rebates, surcharges -- as rates on a driver.
- docs/PERSPECTIVE.md: cut to the cross-app lessons that still hold; the
  fork and its rebuild are in ui/vendor/README.md.
- docs/pf_perspective_options.md: marked as a dated decision record.
- ui/vendor/README.md: depth is sent floored at 1, not d + 1; rebuild steps.
- ui/README.md: removed (Vite template boilerplate).
- Two stale code comments (server.js, Forecast.jsx).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-09-28 23:19:06 -04:00
parent 5feb28eb76
commit 1b1452b625
10 changed files with 2181 additions and 2208 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1120
CLAUDE.md
View File

File diff suppressed because it is too large Load Diff

362
PERSPECTIVE.md
View File

@ -1,362 +0,0 @@
# Perspective — configuration & deployment reference
Canonical guide for how Perspective should be configured, fed, and shipped across our
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
but made different early choices; this doc defines the target state and the rationale, so
the two converge instead of drifting.
> **Distribution note:** we use the **`@perspective-dev/*`** packages
> (repo: <https://github.com/perspective-dev/perspective>, home:
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
> scopes.
---
## Core principle: Perspective is one locked unit
These four things must move together and be pinned together. Bumping one without the
others is the source of nearly every Perspective bug we've hit:
1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
3. **Data format** — Arrow IPC vs JSON rows
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
format the client WASM understands
Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.
---
## 1. Loading — npm `/inline`, pinned exact (not CDN)
```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
```
- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
fetch, works offline / behind a firewall, reproducible from the lockfile.**
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
one-line version bumps) but in production it means: app breaks if the CDN is
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
pull executable WASM from a third party on every load.
> **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
> resolves its server WASM with
> `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
> which from `.../client@4.4.0/dist/cdn/` resolves to
> `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**.
> jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
> pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
> `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
> Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
>
> The security framing matters as much as the outage: an unversioned URL means your
> users execute whatever that package publishes next, automatically, unreviewed.
>
> 5.2.0 fixes it by carrying the client's version across
> (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
> the WASM is embedded in the bundle and there is no runtime fetch to hijack
> (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).
pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
`<link>` in `index.html` — so it's bundled and versioned too.
---
## 2. Version policy — a real trilemma (read carefully)
The version choice is constrained by two hard facts about the `@perspective-dev` packages
**(verified against installed metadata, 2026-06)**:
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even
though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
So you can have at most **two** of these three:
| Want | Requires |
|---|---|
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) |
There is **no** version where all three hold. Pick by what the app needs:
> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts
>
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1`
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.**
>
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
> running exactly that pair), *every chart type other than Datagrid fails.*
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
> `viewer` throws `get_static_config is not a function` — once per chart plugin. The
> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
> load against a coherent 4.4.1 viewer produces no such error.
>
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
>
> Consequence: dataflow currently has the worst of both worlds. It carries the
> mixed-version complexity *specifically* to keep charts, and does not have charts.
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
>
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
> real bundled build. It is the documented-and-untested assumption this whole policy
> rests on — establish it before betting a version choice on it.
- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
`^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
working Datagrid and no charts. If you are on it today, you are choosing inline
bundling, not charts; be explicit about which one you actually want.
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
**4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give
up `/inline` bundling.
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or
later (then a fully-coherent inline-capable suite becomes possible — and only then does
"both" come back on the table).
---
## 3. Data delivery — match the format to the workload
| Workload | Format | Why |
|---|---|---|
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |
### Arrow constraints (read before choosing it)
If you deliver Arrow, three pieces are coupled and must stay aligned:
- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
must coerce them to JS numbers before encoding, or `apache-arrow` infers
`Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
1700).
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
Perspective WASM crashes on dictionary-replacement messages. The server must
accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
'stream')`). Consequence: the client "stream" is just a chunked *download* of one
batch — nothing renders progressively, and the server holds the full result set in
memory per request.
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
`package.json` and treat it as part of the locked unit (principle above). Note the
`@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
the server toward the arrow major the client was built against, or at minimum make
Arrow ingestion (smoke test §7) the gate on any arrow bump.
JSON avoids all three but pays in payload size and parse cost, and pushes type handling
to client-side heuristics — acceptable only at small scale.
---
## 3a. Expression columns and aggregation order (ratio correctness)
**Expression columns are row-level.** Perspective evaluates every expression against
each *raw row* first, then feeds the result into the column's aggregate. It has no
post-aggregate expression stage. So a ratio written the obvious way:
```js
expressions: { price: '"revenue" / "qty"' } // default aggregate: sum
```
...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.
This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
grouping just makes it visible by putting several wrong numbers side by side. No
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
`high`/`low`/`median`/`dominant` are all wrong for the same reason.
### The fix: a weighted-mean aggregate
Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:
```js
expressions: { price: '"revenue" / "qty"' },
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
```
**The nested array matters.** The type is
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
reads like "no such aggregate" and is easy to misread as the feature being absent.
Available since **4.4.0** — no version bump needed, and because the aggregate is
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
`table.update()` on a live view re-derives the weighted mean from the merged rows
without a reload).
### Rules of thumb
- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
→ leave the default `sum`. Those are unaffected by any of this.
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
`['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
- Mixing both in one view is fine and was verified.
### Type inference can silently break expressions
`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
as `integer` — which happens to numeric-looking strings when the table is created from
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
error. Give period/year columns an explicit `string` type at table creation.
### Not fixed by any of this: column-axis expand/collapse
`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
(§2: `viewer-d3fc` still caps at 4.4.1).
---
## 4. Theming
- One toggle drives both app CSS and the viewer:
`viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
re-themes when the toggle fires (not just on mount).
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
viewer theme name is the only Perspective-specific piece.
---
## 5. Layout persistence
- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
`localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
- **Guard restores against schema drift.** Before restoring, filter the saved config's
`columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
the reference implementation; a stale layout referencing a dropped column otherwise
throws on restore.
- **`aggregates` needs the same guard.** pf_app's `cleanLayout()` has it as of 2026-09;
**dataflow's does not** and should adopt it. Filtering
`columns`/`group_by`/`split_by`/`sort`/`filter` and stopping there is harmless only
while `viewer.save()` emits `aggregates: {}`, which it does until someone sets an
aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
aggregate entry references a *target* column and, in the multi-arg form, a *weight*
column; both need validating, and the entry is what gets dropped, never the layout.
Add this guard as part of adopting §3a, not after.
### Auto-pause deletes the view — turn it off for a static pivot
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on itself
(scrolled out of the viewport, `display: none`) combined with the document's
`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint
optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the
**view object is destroyed**. Becoming visible again calls
`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full
traversal, every time.
For a viewer streaming live updates nobody is watching, that is the right trade.
For a pivot over a large static table it is the wrong one — on pf_app's
`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it
silently discards per-node expand/collapse (§"Not fixed by any of this"), which
has no config representation and so cannot be restored.
```js
await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
```
Guard the call: it is a method on the custom element and absent on older builds.
Leave auto-pause **on** where the table is fed by a live stream the user does not
need to have kept up with while away. This is long-standing viewer behaviour, not
a 5.x regression — worth knowing before blaming a rebuild on your own code.
---
## 6. Build & deploy (target)
- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
`outDir: '../public/app'`). The API server serves it statically (`express.static`).
- **Process:** run the Node API under **systemd** with `Restart=always` and an
`EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
automation yet** and should adopt the same pattern.
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
systemd unit); later runs rebuild the UI and restart the service.
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
unit — never commit it.
---
## 7. Upgrade checklist / smoke test
Run this whenever bumping **any** Perspective package or `apache-arrow`:
1. Confirm `viewer-d3fc` publishes the target version
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
commit the lockfile.
- **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
`latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
and the WASM fails to link:
`Import #8 "env" "psp_opfs_load": function import requires a callable`.
Five packages, not four. This is invisible while pf_app loads from the CDN, and
will bite on the "move off CDN" open item below.
3. `vite build` — no unresolved imports.
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
string/dictionary.
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
6. Toggle dark/light; confirm the viewer re-themes.
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
cleaned restore doesn't throw.
---
## Per-project state (2026-06)
| | pf_app | dataflow | Target |
|---|---|---|---|
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** |
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** |
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test |
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
**Open items:**
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages
pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host
blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction):
its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
render before choosing the latter, because nobody has confirmed they do.
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).

187
docs/PERSPECTIVE.md Normal file
View File

@ -0,0 +1,187 @@
# Perspective — lessons and rules
What we have learned the hard way about configuring, feeding and upgrading
Perspective, written for any app that embeds `<perspective-viewer>` (**pf_app**,
**dataflow**). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how
to rebuild it — live in `ui/vendor/README.md`; how pf_app uses the viewer day to
day is in `CLAUDE.md` and `pf_spec.md` → §The pivot.
> **Distribution:** we use the **`@perspective-dev/*`** packages
> (<https://github.com/perspective-dev/perspective>) — **not** FINOS/OpenJS
> `@finos/perspective`. Same engine lineage, but a separate npm scope, release
> cadence and "Pro" theme set (`Pro Dark` / `Pro Light`). Don't mix the scopes.
---
## Perspective is one locked unit
These move together and are pinned together. Bumping one without the others is
the source of nearly every Perspective bug we have hit:
1. **Loader** — how the JS/WASM gets into the page (npm `/inline`, never a CDN)
2. **Package versions** — `client`, `server`, `viewer`, `viewer-datagrid` (and
`viewer-d3fc` where charts are used)
3. **Data format** — Arrow IPC vs JSON rows
4. **`apache-arrow` on the server**, when delivering Arrow — it must write an IPC
format the client WASM reads
**Pin `@perspective-dev/server` explicitly.** `@perspective-dev/client` declares
it as `"@perspective-dev/server": ""` — an empty range, which npm resolves to
`latest`. Installing an older client pulls whatever server is newest and the WASM
fails to link (`Import #8 "env" "psp_opfs_load": function import requires a
callable`). Five packages, not four.
---
## Load inline, never from a CDN
```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer/themes'
```
The `/inline` entrypoints bundle the WASM into the Vite build: no runtime fetch,
works behind a firewall, reproducible from the lockfile. The themes CSS is
imported in JS for the same reason, not `<link>`ed from `index.html`.
**This is not hypothetical — a CDN load took pf_app down on 2026-08-10.** The 4.x
CDN bundle resolves its server WASM with
`new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
which from `.../client@4.4.0/dist/cdn/` lands on
`.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no
version**, so jsdelivr serves `@latest`. The moment a 5.x server published, every
page load linked a 5.x WASM against a 4.4.0 client and threw the `LinkError`
above. Nobody changed anything. The security framing matters as much as the
outage: an unversioned URL means users execute whatever that package publishes
next, unreviewed. `/inline` has no runtime fetch to hijack.
---
## Charts versus inline bundling (apps that need d3fc)
pf_app uses the datagrid only and does not carry `viewer-d3fc`. For an app that
does, the constraint as of 2026-08: `viewer-d3fc` published nothing past 4.4.1,
while `/inline` and `/themes` exist only from 4.5.x. **A 4.5.x viewer with a 4.4.1
d3fc does not work** — every chart plugin throws `get_static_config is not a
function` (reproduced in isolation, and seen in dataflow's install, where every
chart type but the datagrid fails). So it is inline bundling **or** d3fc charts,
not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1
suite renders them — check before relying on it), or 5.x without them. Re-check
when d3fc publishes a newer version.
---
## Arrow delivery: three coupled constraints
Arrow IPC is the right format for large, numeric, incrementally-updated tables;
JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples
three things:
- **Numeric type parsers, server-side.** pg returns `bigint` / `numeric` as
strings; coerce them to numbers before encoding, or `apache-arrow` infers
`Dictionary<Utf8>` instead of `Int` / `Float64`. pf_app: the type parsers in
`server.js` (oids 20 and 1700).
- **One record batch.** Per-batch encoding builds independent dictionaries, and
the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit
a single batch — pf_app does `tableToIPC(tableFromArrays(columns), 'stream')`.
The client's "stream" is therefore a chunked download of one batch; nothing
renders progressively.
- **`apache-arrow` versus the client WASM.** The client is built against an older
arrow major than pf_app's server pins; the pair was verified working, but any
arrow bump has to be gated on the ingest smoke test below.
---
## Expression columns are row-level: ratios need a weighted mean
Perspective evaluates every expression against each **raw row**, then feeds the
result into the column's aggregate; there is no post-aggregate expression stage.
So `price: '"revenue" / "qty"'` under the default `sum` **sums the per-row
ratios** — wrong at every grouped level, with or without `split_by`. `avg` gives
the average of ratios, which is also wrong.
Weight the ratio by its own denominator — `sum(price_i × qty_i) / sum(qty_i)` is
exactly `sum(revenue) / sum(qty)`, at every level of both axes:
```js
expressions: { price: '"revenue" / "qty"' },
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
```
**The nested array matters.** `Aggregate = string | [string, Array<string>]`, so
the weight column goes in its own array. The flat `['weighted mean', 'qty']` is
rejected with `data did not match any variant of untagged enum Aggregate`, which
reads like the feature is missing. Being engine-side, it stays correct across
incremental `table.update()`.
- A **sum of a measure** (including `if(...)` subtotal expressions) → default
`sum`.
- A **ratio, rate, price or per-unit figure** → must carry
`['weighted mean', ['<denominator>']]`.
**Type inference can silently break expressions.** `if("Year" == '2026',
"Amount", 0)` returns 0 on every row if `Year` was *inferred* as integer from
numeric-looking strings, with no error, and neither literal form fixes it. Give
period and year columns an explicit `string` type at table creation.
---
## Theming
The viewer theme is a `ViewerConfig` field: `viewer.restore({ theme: 'Pro Dark' })`.
**`setAttribute('theme', …)` does nothing in 5.x** — the element observes no such
attribute. Apply it on load and whenever the app toggle changes. `viewer.save()`
includes `theme`, so strip it from stored layouts and ignore it when comparing
configs — otherwise restoring a layout reasserts its author's theme.
---
## Layouts: guard every restore against schema drift
Before `viewer.restore(cfg)`, filter the saved `columns` / `group_by` /
`split_by` / `sort` / `filter` against the columns the current table actually has
(plus its `expressions`); a stale layout naming a dropped column otherwise throws.
**`aggregates` needs the same guard** — once a layout carries a weighted mean, a
dropped column aborts the whole restore (`Could not get dtype for column 'X'`).
An aggregate names a target column and, in the multi-arg form, a weight column;
validate both and drop the *entry*, never the layout, since `restore()` is
all-or-nothing. pf_app's `cleanLayout()` does all of this; dataflow's does not yet
guard `aggregates`.
---
## Auto-pause deletes the view — turn it off for a static pivot
`<perspective-viewer>` auto-pauses by default (an `IntersectionObserver` on itself
plus the document's `visibilitychange`). Pausing is not a paint optimisation:
`session.set_pause(true)` runs `view_sub.take().delete()`, so the view is
destroyed, and becoming visible again rebuilds it with a full traversal — a
multi-second stall per tab switch on a large table, and any per-node
expand/collapse is lost with it.
```js
await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
```
Leave it on only where a live stream feeds a table nobody needs kept current
while away. Long-standing viewer behaviour, not a regression.
---
## Upgrade smoke test
Run on any change to a Perspective package or `apache-arrow` (for pf_app, after
rebuilding the vendored tarballs per `ui/vendor/README.md`):
1. All five packages (and `apache-arrow`) pinned exact and matching; lockfile
committed.
2. `vite build` — no unresolved imports.
3. Arrow apps: load a real dataset; `worker.table(buffer)` ingests without a
dictionary error, and a numeric column comes through as `Float64` / `Int`.
4. Charts, where used: open a d3fc plugin and confirm it renders.
5. Toggle dark/light; the viewer re-themes.
6. Save a layout, reload, confirm it restores; drop a column and confirm the
cleaned restore does not throw.

12
pf_perspective_options.md → docs/pf_perspective_options.md
View File

@ -1,5 +1,17 @@
# Perspective Architecture Options # Perspective Architecture Options
> **Decision record, written 2026-08.** The outcome was the display-grain
> pre-aggregation (§Spike findings onward), and it shipped. **"Current
> architecture" below describes the app as it was then**, not now: it has
> since moved to a vendored Perspective 5.4.0 bundled inline, `tableFromArrays`
> on the server, `table.remove()` on undo, and `/data` / `/agg` emitting
> `pf_segment`, `pf_bucket`, `pf_note` and `pf_op`. For today's data flow see
> `CLAUDE.md` → §Core data flow and `pf_spec.md` → §Display-grain
> pre-aggregation. What remains current here is the spike findings, the
> display-grain reasoning, and the Path A / Path B analysis behind the
> "dynamic grain" next step. The DuckDB virtual-server spike is parked on
> branch `spike/duckdb-virtual-server`.
This document weighs how the Forecast view should source data for the Perspective This document weighs how the Forecast view should source data for the Perspective
pivot. The current implementation hits practical limits on initial load pivot. The current implementation hits practical limits on initial load
(~30s for 350k rows × ~55 cols), and growth is expected. Choosing an architecture (~30s for 350k rows × ~55 cols), and growth is expected. Choosing an architecture

1589
docs/pf_spec.md Normal file
View File

File diff suppressed because it is too large Load Diff

1076
pf_spec.md
View File

File diff suppressed because it is too large Load Diff

2
server.js
View File

@ -7,7 +7,7 @@ const { Pool, types } = require('pg');
const { requireAuth } = require('./lib/auth'); const { requireAuth } = require('./lib/auth');
// Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings, // Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings,
// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>. // so apache-arrow encodes them as Int/Float64 rather than Dictionary<Utf8>.
types.setTypeParser(20, v => v === null ? null : Number(v)); types.setTypeParser(20, v => v === null ? null : Number(v));
types.setTypeParser(1700, v => v === null ? null : Number(v)); types.setTypeParser(1700, v => v === null ? null : Number(v));

16
ui/README.md
View File

@ -1,16 +0,0 @@
# React + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
## React Compiler
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
## Expanding the ESLint configuration
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.

5
ui/src/views/Forecast.jsx
View File

@ -317,10 +317,9 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
// Mirrors ViewConfig.group_by_depth for the toolbar's active state. The // Mirrors ViewConfig.group_by_depth for the toolbar's active state. The
// config is the source of truth; this only exists so the buttons can render. // config is the source of truth; this only exists so the buttons can render.
const [expandDepth, setExpandDepth] = useState(null) const [expandDepth, setExpandDepth] = useState(null)
// The column axis has no set_depth() — collapsing it means restoring a shorter // The column hierarchy as the live config has it, for the Columns stepper.
// split_by, so the full hierarchy has to be remembered separately to expand again. // Collapsing is split_by_depth now, so split_by itself is never truncated.
const splitFullRef = useRef([]) const splitFullRef = useRef([])
// a collapse from the user rearranging split_by themselves
const initIdRef = useRef(0) const initIdRef = useRef(0)
const modifierRef = useRef(false) const modifierRef = useRef(false)
// the datagrid plugin element, for reading cell coordinates and driving its // the datagrid plugin element, for reading cell coordinates and driving its

20
ui/vendor/README.md vendored
View File

@ -142,15 +142,23 @@ ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided
The config field counts **levels to show**; `view.set_depth()` counts the The config field counts **levels to show**; `view.set_depth()` counts the
boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and
`Forecast.jsx` sends `d + 1`. `t_traversal::set_depth` adds the 1 back, so `n` renders `n` levels.
`Forecast.jsx` therefore sends the stepper's level count unchanged, floored at
1 — `0` would reach the engine as `set_depth(-1)` on an unsigned depth,
underflow, and clamp to fully expanded. (It used to send `d + 1` to dodge that,
which was off by one against the grid everywhere.)
To apply: The patch is already committed on the fork's build branch (see
`PROVENANCE.txt`), so rebuilding needs only the script. The `.patch` file here
is the change on its own, for review or for carrying to a clean upstream
checkout:
```bash ```bash
cd $PSP_DIR # default ~/perspective ./ui/vendor/rebuild-perspective.sh # from the fork branch
git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch # then re-resolve by path, as above -- a plain `npm install` keeps the old code
./ui/vendor/rebuild-perspective.sh
cd ui && npm install # only on a fresh upstream checkout:
cd $PSP_DIR && git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
``` ```
### Host tools this tree needs ### Host tools this tree needs