pf_app/docs/PERSPECTIVE.md

Perspective — lessons and rules

What we have learned the hard way about configuring, feeding and upgrading Perspective, written for any app that embeds <perspective-viewer> (pf_app, dataflow). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how to rebuild it — live in ui/vendor/README.md; how pf_app uses the viewer day to day is in CLAUDE.md and pf_spec.md → §The pivot.

Distribution: we use the @perspective-dev/* packages (https://github.com/perspective-dev/perspective) — not FINOS/OpenJS @finos/perspective. Same engine lineage, but a separate npm scope, release cadence and "Pro" theme set (Pro Dark / Pro Light). Don't mix the scopes.

---

Perspective is one locked unit

These move together and are pinned together. Bumping one without the others is the source of nearly every Perspective bug we have hit:

1. Loader — how the JS/WASM gets into the page (npm /inline, never a CDN)
2. Package versions — client, server, viewer, viewer-datagrid (and viewer-d3fc where charts are used)
3. Data format — Arrow IPC vs JSON rows
4. apache-arrow on the server, when delivering Arrow — it must write an IPC format the client WASM reads

Pin @perspective-dev/server explicitly. @perspective-dev/client declares it as "@perspective-dev/server": "" — an empty range, which npm resolves to latest. Installing an older client pulls whatever server is newest and the WASM fails to link (Import #8 "env" "psp_opfs_load": function import requires a callable). Five packages, not four.

---

Load inline, never from a CDN

import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer/themes'

The /inline entrypoints bundle the WASM into the Vite build: no runtime fetch, works behind a firewall, reproducible from the lockfile. The themes CSS is imported in JS for the same reason, not <link>ed from index.html.

This is not hypothetical — a CDN load took pf_app down on 2026-08-10. The 4.x CDN bundle resolves its server WASM with new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url), which from .../client@4.4.0/dist/cdn/ lands on .../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm — no version, so jsdelivr serves @latest. The moment a 5.x server published, every page load linked a 5.x WASM against a 4.4.0 client and threw the LinkError above. Nobody changed anything. The security framing matters as much as the outage: an unversioned URL means users execute whatever that package publishes next, unreviewed. /inline has no runtime fetch to hijack.

---

Charts versus inline bundling (apps that need d3fc)

pf_app uses the datagrid only and does not carry viewer-d3fc. For an app that does, the constraint as of 2026-08: viewer-d3fc published nothing past 4.4.1, while /inline and /themes exist only from 4.5.x. A 4.5.x viewer with a 4.4.1 d3fc does not work — every chart plugin throws get_static_config is not a function (reproduced in isolation, and seen in dataflow's install, where every chart type but the datagrid fails). So it is inline bundling or d3fc charts, not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1 suite renders them — check before relying on it), or 5.x without them. Re-check when d3fc publishes a newer version.

---

Arrow delivery: three coupled constraints

Arrow IPC is the right format for large, numeric, incrementally-updated tables; JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples three things:

• Numeric type parsers, server-side. pg returns bigint / numeric as strings; coerce them to numbers before encoding, or apache-arrow infers Dictionary<Utf8> instead of Int / Float64. pf_app: the type parsers in server.js (oids 20 and 1700).
• One record batch. Per-batch encoding builds independent dictionaries, and the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit a single batch — pf_app does tableToIPC(tableFromArrays(columns), 'stream'). The client's "stream" is therefore a chunked download of one batch; nothing renders progressively.
• apache-arrow versus the client WASM. The client is built against an older arrow major than pf_app's server pins; the pair was verified working, but any arrow bump has to be gated on the ingest smoke test below.

---

Expression columns are row-level: ratios need a weighted mean

Perspective evaluates every expression against each raw row, then feeds the result into the column's aggregate; there is no post-aggregate expression stage. So price: '"revenue" / "qty"' under the default sum sums the per-row ratios — wrong at every grouped level, with or without split_by. avg gives the average of ratios, which is also wrong.

Weight the ratio by its own denominator — sum(price_i × qty_i) / sum(qty_i) is exactly sum(revenue) / sum(qty), at every level of both axes:

expressions: { price: '"revenue" / "qty"' },
aggregates:  { price: ['weighted mean', ['qty']] }   // note the NESTED array

The nested array matters. Aggregate = string | [string, Array<string>], so the weight column goes in its own array. The flat ['weighted mean', 'qty'] is rejected with data did not match any variant of untagged enum Aggregate, which reads like the feature is missing. Being engine-side, it stays correct across incremental table.update().

• A sum of a measure (including if(...) subtotal expressions) → default sum.
• A ratio, rate, price or per-unit figure → must carry ['weighted mean', ['<denominator>']].

Type inference can silently break expressions. if("Year" == '2026', "Amount", 0) returns 0 on every row if Year was inferred as integer from numeric-looking strings, with no error, and neither literal form fixes it. Give period and year columns an explicit string type at table creation.

---

Theming

The viewer theme is a ViewerConfig field: viewer.restore({ theme: 'Pro Dark' }). setAttribute('theme', …) does nothing in 5.x — the element observes no such attribute. Apply it on load and whenever the app toggle changes. viewer.save() includes theme, so strip it from stored layouts and ignore it when comparing configs — otherwise restoring a layout reasserts its author's theme.

---

Layouts: guard every restore against schema drift

Before viewer.restore(cfg), filter the saved columns / group_by / split_by / sort / filter against the columns the current table actually has (plus its expressions); a stale layout naming a dropped column otherwise throws. aggregates needs the same guard — once a layout carries a weighted mean, a dropped column aborts the whole restore (Could not get dtype for column 'X'). An aggregate names a target column and, in the multi-arg form, a weight column; validate both and drop the entry, never the layout, since restore() is all-or-nothing. pf_app's cleanLayout() does all of this; dataflow's does not yet guard aggregates.

---

Auto-pause deletes the view — turn it off for a static pivot

<perspective-viewer> auto-pauses by default (an IntersectionObserver on itself plus the document's visibilitychange). Pausing is not a paint optimisation: session.set_pause(true) runs view_sub.take().delete(), so the view is destroyed, and becoming visible again rebuilds it with a full traversal — a multi-second stall per tab switch on a large table, and any per-node expand/collapse is lost with it.

await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}

Leave it on only where a live stream feeds a table nobody needs kept current while away. Long-standing viewer behaviour, not a regression.

---

Upgrade smoke test

Run on any change to a Perspective package or apache-arrow (for pf_app, after rebuilding the vendored tarballs per ui/vendor/README.md):

1. All five packages (and apache-arrow) pinned exact and matching; lockfile committed.
2. vite build — no unresolved imports.
3. Arrow apps: load a real dataset; worker.table(buffer) ingests without a dictionary error, and a numeric column comes through as Float64 / Int.
4. Charts, where used: open a d3fc plugin and confirm it renders.
5. Toggle dark/light; the viewer re-themes.
6. Save a layout, reload, confirm it restores; drop a column and confirm the cleaned restore does not throw.