pf_app/PERSPECTIVE.md

Perspective — configuration & deployment reference

Canonical guide for how Perspective should be configured, fed, and shipped across our apps (pf_app, dataflow). Both embed the same <perspective-viewer> web component but made different early choices; this doc defines the target state and the rationale, so the two converge instead of drifting.

Distribution note: we use the @perspective-dev/* packages (repo: https://github.com/perspective-dev/perspective, home: https://perspective-dev.github.io) — not the FINOS/OpenJS @finos/perspective packages from https://perspective.finos.org/. Same engine lineage, but a separate npm scope, release cadence, and "Pro" theme set (Pro Dark/Pro Light). This is why viewer-d3fc versions on a different schedule than viewer/client (§2), and why the client's bundled apache-arrow (17.x) can lag the server's (§3). Don't mix the two scopes.

---

Core principle: Perspective is one locked unit

These four things must move together and be pinned together. Bumping one without the others is the source of nearly every Perspective bug we've hit:

1. Loader — how the JS/WASM gets into the page (npm-inline vs CDN)
2. Package version — client / viewer / viewer-datagrid / viewer-d3fc
3. Data format — Arrow IPC vs JSON rows
4. apache-arrow version (server-side, only if using Arrow) — must speak an IPC format the client WASM understands

Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test at the bottom of this doc. Never let viewer/client drift ahead of viewer-d3fc.

---

1. Loading — npm /inline, pinned exact (not CDN)

import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'

• The /inline entrypoints bundle the WASM into the Vite build — no runtime network fetch, works offline / behind a firewall, reproducible from the lockfile.

• Do not load from a CDN at runtime. It's convenient for a prototype (smaller build, one-line version bumps) but in production it means: app breaks if the CDN is unreachable, version isn't captured in package-lock.json, slower cold start, and you pull executable WASM from a third party on every load.

  This is not hypothetical — it took pf_app down on 2026-08-10. The 4.x CDN bundle resolves its server WASM with new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url), which from .../client@4.4.0/dist/cdn/ resolves to .../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm — no version. jsdelivr serves @latest. The moment @perspective-dev/server@5.2.0 published, every pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable. Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.

  The security framing matters as much as the outage: an unversioned URL means your users execute whatever that package publishes next, automatically, unreviewed.

  5.2.0 fixes it by carrying the client's version across (/client@X/dist/cdn/… → /server@X/dist/wasm/…), but the durable fix is /inline: the WASM is embedded in the bundle and there is no runtime fetch to hijack (verified: zero new URL(...perspective-server...) in perspective.inline.js).

  pf_app migrated off CDN to npm /inline at 5.2.0 on 2026-08-17.

• The themes CSS is imported in JS (@perspective-dev/viewer/themes), not via a <link> in index.html — so it's bundled and versioned too.

---

2. Version policy — a real trilemma (read carefully)

The version choice is constrained by two hard facts about the @perspective-dev packages (verified against installed metadata, 2026-06):

• viewer-d3fc caps at 4.4.1 — npm publishes no 4.5.x. The d3fc charts (Bar / Line / Treemap / Heatmap / etc.) live only in this package. Still true as of 2026-08, even though client/viewer/viewer-datagrid now publish through 5.2.0 — so the trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
• The /inline and /themes entrypoints are 4.5.x-only — @perspective-dev/client/inline, @perspective-dev/viewer/inline, and @perspective-dev/viewer/themes do not exist in 4.4.1's exports map. Bundling inline WASM requires 4.5.x.

So you can have at most two of these three:

Want	Requires
Inline WASM bundling (/inline, /themes)	4.5.x viewer/client
One coherent single-version suite	4.4.1 everything (d3fc ceiling)
d3fc chart plugins	4.4.1 viewer-d3fc — and a 4.4.1 viewer to match (see correction)

There is no version where all three hold. Pick by what the app needs:

⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts

This section previously recommended ^4.5.1 viewer/client/datagrid + ^4.4.1 viewer-d3fc as "the only combo that keeps both." That recommendation was wrong.

• Confirmed by the app owner: in the deployed dataflow install (/opt/dataflow, running exactly that pair), every chart type other than Datagrid fails.
• Mechanism, reproduced in isolation: loading 4.4.1 viewer-d3fc against a 4.5.1 viewer throws get_static_config is not a function — once per chart plugin. The 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same load against a coherent 4.4.1 viewer produces no such error.

So the "Inline-bundled + charts" row below is not achievable. The trilemma is really a dilemma: inline WASM bundling XOR d3fc charts — pick one.

Consequence: dataflow currently has the worst of both worlds. It carries the mixed-version complexity specifically to keep charts, and does not have charts. Both directions are strictly better than standing still: down to a coherent 4.4.1 suite (if charts matter) or up to 5.x (if they don't, and you want the newer engine — §3a, split_rollup_mode, edit_mode persistence).

Still unverified: whether a coherent 4.4.1 suite actually renders charts in a real bundled build. It is the documented-and-untested assumption this whole policy rests on — establish it before betting a version choice on it.

• Inline-bundled + charts (dataflow's case) → ^4.5.1 viewer/client/datagrid **+ ^4.4.1 viewer-d3fc`. Withdrawn — see correction above. This pair yields a working Datagrid and no charts. If you are on it today, you are choosing inline bundling, not charts; be explicit about which one you actually want.
• Coherent single suite, no inline (e.g. CDN or .-entry loading) → pin all four to 4.4.1 exact. Charts are believed to work here (unverified — see above); you give up /inline bundling.

Whatever you pick, commit the lockfile so the resolved set can't drift on npm install. Re-evaluate the whole policy only when viewer-d3fc ships a 4.5.x or later (then a fully-coherent inline-capable suite becomes possible — and only then does "both" come back on the table).

---

3. Data delivery — match the format to the workload

Workload	Format	Why
Large (100k+ rows), numeric-heavy, writes/incremental updates	Arrow IPC	Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path.
Small (≤100k), read-only, click-to-inspect	JSON rows	Simpler, no encoding step, no dictionary pitfalls. dataflow's model.

Arrow constraints (read before choosing it)

If you deliver Arrow, three pieces are coupled and must stay aligned:

• Numeric type parsers, server-side. pg returns bigint/numeric as strings; you must coerce them to JS numbers before encoding, or apache-arrow infers Dictionary<Utf8> instead of Int/Float64. See server.js type parsers (oid 20, 1700).
• Single record batch. Per-batch Arrow builds independent dictionaries; the Perspective WASM crashes on dictionary-replacement messages. The server must accumulate all rows and emit one batch (tableToIPC(tableFromJSON(allRows), 'stream')). Consequence: the client "stream" is just a chunked download of one batch — nothing renders progressively, and the server holds the full result set in memory per request.
• apache-arrow pinned to match the client WASM. Pin it exact in the server package.json and treat it as part of the locked unit (principle above). Note the @perspective-dev/client build is tested against apache-arrow@17.0.0, while pf_app's server currently pins ^21.1.0 — a real IPC version gap. Prefer aligning the server toward the arrow major the client was built against, or at minimum make Arrow ingestion (smoke test §7) the gate on any arrow bump.

JSON avoids all three but pays in payload size and parse cost, and pushes type handling to client-side heuristics — acceptable only at small scale.

---

3a. Expression columns and aggregation order (ratio correctness)

Expression columns are row-level. Perspective evaluates every expression against each raw row first, then feeds the result into the column's aggregate. It has no post-aggregate expression stage. So a ratio written the obvious way:

expressions: { price: '"revenue" / "qty"' }   // default aggregate: sum

...computes revenue/qty per row and then sums the per-row ratios — the classic sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group whose true sum(revenue)/sum(qty) is 16.15, the pivot shows 42.

This is not a split_by bug. It is equally wrong with only group_by — column grouping just makes it visible by putting several wrong numbers side by side. No scalar aggregate fixes it: avg/mean give the average of ratios (14), and high/low/median/dominant are all wrong for the same reason.

The fix: a weighted-mean aggregate

Weight the ratio by its own denominator. sum(price_i × qty_i) / sum(qty_i) is algebraically sum(revenue)/sum(qty) — the correct answer at every level of both axes:

expressions: { price: '"revenue" / "qty"' },
aggregates:  { price: ['weighted mean', ['qty']] }   // note the NESTED array

The nested array matters. The type is Aggregate = string | [string, Array<string>] (ts-rs/Aggregate.d.ts), so the weight column goes in its own array. The flat form ['weighted mean', 'qty'] is rejected with the unhelpful data did not match any variant of untagged enum Aggregate — which reads like "no such aggregate" and is easy to misread as the feature being absent.

Available since 4.4.0 — no version bump needed, and because the aggregate is evaluated inside the engine, incremental table.update() stays correct (verified: a table.update() on a live view re-derives the weighted mean from the merged rows without a reload).

Rules of thumb

• Column is a sum of a measure (incl. if(...) column-subtotal expressions, §below) → leave the default sum. Those are unaffected by any of this.
• Column is a ratio, rate, price, or per-unit figure → it must carry a ['weighted mean', ['<denominator>']] aggregate, or it is wrong under any pivot.
• Mixing both in one view is fine and was verified.

Type inference can silently break expressions

if("Year" == '2026', "Amount", 0) returns 0 for every row if Year was inferred as integer — which happens to numeric-looking strings when the table is created from inferred JSON. With an explicit string schema the same expression is correct. Neither literal form ('2026' or 2026) works against a mis-inferred column, and there is no error. Give period/year columns an explicit string type at table creation.

Not fixed by any of this: column-axis expand/collapse

view.expand() / view.collapse() / set_depth() take a row index and act on the row axis only; there is no column-axis equivalent in 4.4.0 or 5.2.0, contrary to the docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds split_rollup_mode: 'rollup', which emits subtotal and grand-total column groups statically (no interactivity) — the nearest thing to Excel-style column subtotals, and it would retire the if(...)-expression workaround. It costs the d3fc charts, though (§2: viewer-d3fc still caps at 4.4.1).

---

4. Theming

• One toggle drives both app CSS and the viewer: viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light').
• Apply it on initial load and in an effect keyed on the dark flag, so the viewer re-themes when the toggle fires (not just on mount).
• App-level dark mode is plain CSS custom properties + a .dark class on <html>; the viewer theme name is the only Perspective-specific piece.

---

5. Layout persistence

• Persist the full viewer config (await viewer.save(), incl. plugin_config) to localStorage, keyed per source/version. Restore with viewer.restore(cfg).
• Guard restores against schema drift. Before restoring, filter the saved config's columns/group_by/split_by/sort/filter against the columns that actually exist in the current dataset (plus any expressions). dataflow's cleanLayout() is the reference implementation; a stale layout referencing a dropped column otherwise throws on restore.
• aggregates needs the same guard. pf_app's cleanLayout() has it as of 2026-09; dataflow's does not and should adopt it. Filtering columns/group_by/split_by/sort/filter and stopping there is harmless only while viewer.save() emits aggregates: {}, which it does until someone sets an aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a dropped column aborts the entire restore — both table.view() and viewer.restore() throw Could not get dtype for column 'X' as it does not exist in the schema. An aggregate entry references a target column and, in the multi-arg form, a weight column; both need validating, and the entry is what gets dropped, never the layout. Add this guard as part of adopting §3a, not after.

Auto-pause deletes the view — turn it off for a static pivot

<perspective-viewer> auto-pauses by default: an IntersectionObserver on itself (scrolled out of the viewport, display: none) combined with the document's visibilitychange (backgrounded tab, minimized window). "Pause" is not a paint optimisation — session.set_pause(true) runs view_sub.take().delete(), so the view object is destroyed. Becoming visible again calls restore_and_render(…, ViewerConfigUpdate::default()): a new view and a full traversal, every time.

For a viewer streaming live updates nobody is watching, that is the right trade. For a pivot over a large static table it is the wrong one — on pf_app's fc_osm_skinny_29 grain it is a multi-second stall on every tab switch, and it silently discards per-node expand/collapse (§"Not fixed by any of this"), which has no config representation and so cannot be restored.

await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}

Guard the call: it is a method on the custom element and absent on older builds. Leave auto-pause on where the table is fed by a live stream the user does not need to have kept up with while away. This is long-standing viewer behaviour, not a 5.x regression — worth knowing before blaming a rebuild on your own code.

---

6. Build & deploy (target)

• Build: vite build emits a static bundle (pf_app → public/app, via outDir: '../public/app'). The API server serves it statically (express.static).
• Process: run the Node API under systemd with Restart=always and an EnvironmentFile=.env; front it with nginx (reverse proxy + TLS via certbot). dataflow's dataflow.service + deploy.sh are the reference; pf_app has no deploy automation yet and should adopt the same pattern.
• deploy.sh should be idempotent: first run installs (db/schema, UI build, nginx, systemd unit); later runs rebuild the UI and restart the service.
• Keep secrets in .env (db creds), loaded by both the app (dotenv) and the systemd unit — never commit it.

---

7. Upgrade checklist / smoke test

Run this whenever bumping any Perspective package or apache-arrow:

1. Confirm viewer-d3fc publishes the target version (npm view @perspective-dev/viewer-d3fc versions). If not, don't bump the others.
2. Pin all four packages + apache-arrow to exact, matching versions; npm install; commit the lockfile.
   • Pin @perspective-dev/server explicitly too. @perspective-dev/client declares it as "@perspective-dev/server": "" — an empty range, which npm resolves to latest. A fresh npm i @perspective-dev/client@4.4.0 today pulls server 5.2.0 and the WASM fails to link: Import #8 "env" "psp_opfs_load": function import requires a callable. Five packages, not four. This is invisible while pf_app loads from the CDN, and will bite on the "move off CDN" open item below.
3. vite build — no unresolved imports.
4. Arrow apps: load a real dataset and confirm worker.table(buffer) ingests without a WASM dictionary error; verify a numeric column is Float64/Int, not a string/dictionary.
5. Open a d3fc chart plugin (not just datagrid) and confirm it renders.
6. Toggle dark/light; confirm the viewer re-themes.
7. Save a layout, reload, confirm it restores; then drop a column and confirm the cleaned restore doesn't throw.

---

Per-project state (2026-06)

	pf_app	dataflow	Target
Loader	npm /inline (was CDN until 2026-08-17)	npm /inline	npm /inline
Version	5.2.0 exact, all four (incl. server)	4.5.1 viewer/client + 4.4.1 d3fc	5.2.0 exact
Charts	none — d3fc import dropped (unused)	d3fc imported but broken (§2 correction)	decide per app
Data	Arrow IPC (single batch)	JSON (≤100k)	per workload (§3)
apache-arrow	^21.1.0 — verified OK against 5.2.0 WASM	n/a	pin exact; verify by test
Deploy	none	systemd + nginx + deploy.sh	systemd + nginx + deploy.sh

dataflow's 4.5.1/4.4.1 pair is correct — it's the only combo giving both inline bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.

Open items:

• pf_app → move off CDN. Done 2026-08-17 — npm /inline, all four packages pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
• dataflow → same migration. It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction): its d3fc charts do not work, so it is paying mixed-version complexity for nothing. Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually render before choosing the latter, because nobody has confirmed they do.
• pf_app still has no deploy automation (systemd + nginx + deploy.sh).