Paul Trowbridge
1b1452b625
An audit against the code found every doc had drifted. pf_spec.md mostly described behaviour the app no longer has and missed auth, territory, layouts, load plans, segment naming and scope entirely; CLAUDE.md had grown to carry the history as well as the rules, and loads into every session. - CLAUDE.md: rules, invariants and single-definition pointers only, with stale claims corrected (project layout, routes, schema, the pf_ slice columns, the price/volume and footing rules, known issues, deferred). Each section that lost its reasoning links to the spec. - docs/pf_spec.md: rewritten as the current design doc under twelve fixed headings, holding the history and rationale moved out of CLAUDE.md, the code issues found on the way, and a proposed (not built) design for additional measures -- cost, rebates, surcharges -- as rates on a driver. - docs/PERSPECTIVE.md: cut to the cross-app lessons that still hold; the fork and its rebuild are in ui/vendor/README.md. - docs/pf_perspective_options.md: marked as a dated decision record. - ui/vendor/README.md: depth is sent floored at 1, not d + 1; rebuild steps. - ui/README.md: removed (Vite template boilerplate). - Two stale code comments (server.js, Forecast.jsx). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
188 lines
8.5 KiB
Markdown
188 lines
8.5 KiB
Markdown
# Perspective — lessons and rules
|
||
|
||
What we have learned the hard way about configuring, feeding and upgrading
|
||
Perspective, written for any app that embeds `<perspective-viewer>` (**pf_app**,
|
||
**dataflow**). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how
|
||
to rebuild it — live in `ui/vendor/README.md`; how pf_app uses the viewer day to
|
||
day is in `CLAUDE.md` and `pf_spec.md` → §The pivot.
|
||
|
||
> **Distribution:** we use the **`@perspective-dev/*`** packages
|
||
> (<https://github.com/perspective-dev/perspective>) — **not** FINOS/OpenJS
|
||
> `@finos/perspective`. Same engine lineage, but a separate npm scope, release
|
||
> cadence and "Pro" theme set (`Pro Dark` / `Pro Light`). Don't mix the scopes.
|
||
|
||
---
|
||
|
||
## Perspective is one locked unit
|
||
|
||
These move together and are pinned together. Bumping one without the others is
|
||
the source of nearly every Perspective bug we have hit:
|
||
|
||
1. **Loader** — how the JS/WASM gets into the page (npm `/inline`, never a CDN)
|
||
2. **Package versions** — `client`, `server`, `viewer`, `viewer-datagrid` (and
|
||
`viewer-d3fc` where charts are used)
|
||
3. **Data format** — Arrow IPC vs JSON rows
|
||
4. **`apache-arrow` on the server**, when delivering Arrow — it must write an IPC
|
||
format the client WASM reads
|
||
|
||
**Pin `@perspective-dev/server` explicitly.** `@perspective-dev/client` declares
|
||
it as `"@perspective-dev/server": ""` — an empty range, which npm resolves to
|
||
`latest`. Installing an older client pulls whatever server is newest and the WASM
|
||
fails to link (`Import #8 "env" "psp_opfs_load": function import requires a
|
||
callable`). Five packages, not four.
|
||
|
||
---
|
||
|
||
## Load inline, never from a CDN
|
||
|
||
```js
|
||
import perspective from '@perspective-dev/client/inline'
|
||
import '@perspective-dev/viewer/inline'
|
||
import '@perspective-dev/viewer-datagrid'
|
||
import '@perspective-dev/viewer/themes'
|
||
```
|
||
|
||
The `/inline` entrypoints bundle the WASM into the Vite build: no runtime fetch,
|
||
works behind a firewall, reproducible from the lockfile. The themes CSS is
|
||
imported in JS for the same reason, not `<link>`ed from `index.html`.
|
||
|
||
**This is not hypothetical — a CDN load took pf_app down on 2026-08-10.** The 4.x
|
||
CDN bundle resolves its server WASM with
|
||
`new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
|
||
which from `.../client@4.4.0/dist/cdn/` lands on
|
||
`.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no
|
||
version**, so jsdelivr serves `@latest`. The moment a 5.x server published, every
|
||
page load linked a 5.x WASM against a 4.4.0 client and threw the `LinkError`
|
||
above. Nobody changed anything. The security framing matters as much as the
|
||
outage: an unversioned URL means users execute whatever that package publishes
|
||
next, unreviewed. `/inline` has no runtime fetch to hijack.
|
||
|
||
---
|
||
|
||
## Charts versus inline bundling (apps that need d3fc)
|
||
|
||
pf_app uses the datagrid only and does not carry `viewer-d3fc`. For an app that
|
||
does, the constraint as of 2026-08: `viewer-d3fc` published nothing past 4.4.1,
|
||
while `/inline` and `/themes` exist only from 4.5.x. **A 4.5.x viewer with a 4.4.1
|
||
d3fc does not work** — every chart plugin throws `get_static_config is not a
|
||
function` (reproduced in isolation, and seen in dataflow's install, where every
|
||
chart type but the datagrid fails). So it is inline bundling **or** d3fc charts,
|
||
not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1
|
||
suite renders them — check before relying on it), or 5.x without them. Re-check
|
||
when d3fc publishes a newer version.
|
||
|
||
---
|
||
|
||
## Arrow delivery: three coupled constraints
|
||
|
||
Arrow IPC is the right format for large, numeric, incrementally-updated tables;
|
||
JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples
|
||
three things:
|
||
|
||
- **Numeric type parsers, server-side.** pg returns `bigint` / `numeric` as
|
||
strings; coerce them to numbers before encoding, or `apache-arrow` infers
|
||
`Dictionary<Utf8>` instead of `Int` / `Float64`. pf_app: the type parsers in
|
||
`server.js` (oids 20 and 1700).
|
||
- **One record batch.** Per-batch encoding builds independent dictionaries, and
|
||
the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit
|
||
a single batch — pf_app does `tableToIPC(tableFromArrays(columns), 'stream')`.
|
||
The client's "stream" is therefore a chunked download of one batch; nothing
|
||
renders progressively.
|
||
- **`apache-arrow` versus the client WASM.** The client is built against an older
|
||
arrow major than pf_app's server pins; the pair was verified working, but any
|
||
arrow bump has to be gated on the ingest smoke test below.
|
||
|
||
---
|
||
|
||
## Expression columns are row-level: ratios need a weighted mean
|
||
|
||
Perspective evaluates every expression against each **raw row**, then feeds the
|
||
result into the column's aggregate; there is no post-aggregate expression stage.
|
||
So `price: '"revenue" / "qty"'` under the default `sum` **sums the per-row
|
||
ratios** — wrong at every grouped level, with or without `split_by`. `avg` gives
|
||
the average of ratios, which is also wrong.
|
||
|
||
Weight the ratio by its own denominator — `sum(price_i × qty_i) / sum(qty_i)` is
|
||
exactly `sum(revenue) / sum(qty)`, at every level of both axes:
|
||
|
||
```js
|
||
expressions: { price: '"revenue" / "qty"' },
|
||
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
|
||
```
|
||
|
||
**The nested array matters.** `Aggregate = string | [string, Array<string>]`, so
|
||
the weight column goes in its own array. The flat `['weighted mean', 'qty']` is
|
||
rejected with `data did not match any variant of untagged enum Aggregate`, which
|
||
reads like the feature is missing. Being engine-side, it stays correct across
|
||
incremental `table.update()`.
|
||
|
||
- A **sum of a measure** (including `if(...)` subtotal expressions) → default
|
||
`sum`.
|
||
- A **ratio, rate, price or per-unit figure** → must carry
|
||
`['weighted mean', ['<denominator>']]`.
|
||
|
||
**Type inference can silently break expressions.** `if("Year" == '2026',
|
||
"Amount", 0)` returns 0 on every row if `Year` was *inferred* as integer from
|
||
numeric-looking strings, with no error, and neither literal form fixes it. Give
|
||
period and year columns an explicit `string` type at table creation.
|
||
|
||
---
|
||
|
||
## Theming
|
||
|
||
The viewer theme is a `ViewerConfig` field: `viewer.restore({ theme: 'Pro Dark' })`.
|
||
**`setAttribute('theme', …)` does nothing in 5.x** — the element observes no such
|
||
attribute. Apply it on load and whenever the app toggle changes. `viewer.save()`
|
||
includes `theme`, so strip it from stored layouts and ignore it when comparing
|
||
configs — otherwise restoring a layout reasserts its author's theme.
|
||
|
||
---
|
||
|
||
## Layouts: guard every restore against schema drift
|
||
|
||
Before `viewer.restore(cfg)`, filter the saved `columns` / `group_by` /
|
||
`split_by` / `sort` / `filter` against the columns the current table actually has
|
||
(plus its `expressions`); a stale layout naming a dropped column otherwise throws.
|
||
**`aggregates` needs the same guard** — once a layout carries a weighted mean, a
|
||
dropped column aborts the whole restore (`Could not get dtype for column 'X'`).
|
||
An aggregate names a target column and, in the multi-arg form, a weight column;
|
||
validate both and drop the *entry*, never the layout, since `restore()` is
|
||
all-or-nothing. pf_app's `cleanLayout()` does all of this; dataflow's does not yet
|
||
guard `aggregates`.
|
||
|
||
---
|
||
|
||
## Auto-pause deletes the view — turn it off for a static pivot
|
||
|
||
`<perspective-viewer>` auto-pauses by default (an `IntersectionObserver` on itself
|
||
plus the document's `visibilitychange`). Pausing is not a paint optimisation:
|
||
`session.set_pause(true)` runs `view_sub.take().delete()`, so the view is
|
||
destroyed, and becoming visible again rebuilds it with a full traversal — a
|
||
multi-second stall per tab switch on a large table, and any per-node
|
||
expand/collapse is lost with it.
|
||
|
||
```js
|
||
await viewer.load(table)
|
||
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
|
||
```
|
||
|
||
Leave it on only where a live stream feeds a table nobody needs kept current
|
||
while away. Long-standing viewer behaviour, not a regression.
|
||
|
||
---
|
||
|
||
## Upgrade smoke test
|
||
|
||
Run on any change to a Perspective package or `apache-arrow` (for pf_app, after
|
||
rebuilding the vendored tarballs per `ui/vendor/README.md`):
|
||
|
||
1. All five packages (and `apache-arrow`) pinned exact and matching; lockfile
|
||
committed.
|
||
2. `vite build` — no unresolved imports.
|
||
3. Arrow apps: load a real dataset; `worker.table(buffer)` ingests without a
|
||
dictionary error, and a numeric column comes through as `Float64` / `Int`.
|
||
4. Charts, where used: open a d3fc plugin and confirm it renders.
|
||
5. Toggle dark/light; the viewer re-themes.
|
||
6. Save a layout, reload, confirm it restores; drop a column and confirm the
|
||
cleaned restore does not throw.
|