pf_app/PERSPECTIVE.md

Perspective — configuration & deployment reference

Canonical guide for how Perspective should be configured, fed, and shipped across our apps (pf_app, dataflow). Both embed the same <perspective-viewer> web component but made different early choices; this doc defines the target state and the rationale, so the two converge instead of drifting.

Distribution note: we use the @perspective-dev/* packages (repo: https://github.com/perspective-dev/perspective, home: https://perspective-dev.github.io) — not the FINOS/OpenJS @finos/perspective packages from https://perspective.finos.org/. Same engine lineage, but a separate npm scope, release cadence, and "Pro" theme set (Pro Dark/Pro Light). This is why viewer-d3fc versions on a different schedule than viewer/client (§2), and why the client's bundled apache-arrow (17.x) can lag the server's (§3). Don't mix the two scopes.

---

Core principle: Perspective is one locked unit

These four things must move together and be pinned together. Bumping one without the others is the source of nearly every Perspective bug we've hit:

1. Loader — how the JS/WASM gets into the page (npm-inline vs CDN)
2. Package version — client / viewer / viewer-datagrid / viewer-d3fc
3. Data format — Arrow IPC vs JSON rows
4. apache-arrow version (server-side, only if using Arrow) — must speak an IPC format the client WASM understands

Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test at the bottom of this doc. Never let viewer/client drift ahead of viewer-d3fc.

---

1. Loading — npm /inline, pinned exact (not CDN)

import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'

• The /inline entrypoints bundle the WASM into the Vite build — no runtime network fetch, works offline / behind a firewall, reproducible from the lockfile.
• Do not load from a CDN at runtime. It's convenient for a prototype (smaller build, one-line version bumps) but in production it means: app breaks if the CDN is unreachable, version isn't captured in package-lock.json, slower cold start, and you pull executable WASM from a third party on every load. (pf_app currently does this in ui/src/views/Forecast.jsx — migrating off it is the main open item.)
• The themes CSS is imported in JS (@perspective-dev/viewer/themes), not via a <link> in index.html — so it's bundled and versioned too.

---

2. Version policy — a real trilemma (read carefully)

The version choice is constrained by two hard facts about the @perspective-dev packages (verified against installed metadata, 2026-06):

• viewer-d3fc caps at 4.4.1 — npm publishes no 4.5.x. The d3fc charts (Bar / Line / Treemap / Heatmap / etc.) live only in this package.
• The /inline and /themes entrypoints are 4.5.x-only — @perspective-dev/client/inline, @perspective-dev/viewer/inline, and @perspective-dev/viewer/themes do not exist in 4.4.1's exports map. Bundling inline WASM requires 4.5.x.

So you can have at most two of these three:

Want	Requires
Inline WASM bundling (/inline, /themes)	4.5.x viewer/client
One coherent single-version suite	4.4.1 everything (d3fc ceiling)
d3fc chart plugins	4.4.1 viewer-d3fc

There is no version where all three hold. Pick by what the app needs:

• Inline-bundled + charts (dataflow's case) → ^4.5.1 viewer/client/datagrid **+ ^4.4.1 viewer-d3fc. This is a deliberate, necessary mixed-version pair, *not* an accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the build breaks ("./inline" is not exported`).
• Coherent single suite, no inline (e.g. CDN or .-entry loading) → pin all four to 4.4.1 exact. Charts work; you give up /inline bundling.

Whatever you pick, commit the lockfile so the resolved set can't drift on npm install. Re-evaluate the whole policy only when viewer-d3fc ships a 4.5.x (then a fully-coherent inline-capable 4.5.x suite becomes possible).

---

3. Data delivery — match the format to the workload

Workload	Format	Why
Large (100k+ rows), numeric-heavy, writes/incremental updates	Arrow IPC	Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path.
Small (≤100k), read-only, click-to-inspect	JSON rows	Simpler, no encoding step, no dictionary pitfalls. dataflow's model.

Arrow constraints (read before choosing it)

If you deliver Arrow, three pieces are coupled and must stay aligned:

• Numeric type parsers, server-side. pg returns bigint/numeric as strings; you must coerce them to JS numbers before encoding, or apache-arrow infers Dictionary<Utf8> instead of Int/Float64. See server.js type parsers (oid 20, 1700).
• Single record batch. Per-batch Arrow builds independent dictionaries; the Perspective WASM crashes on dictionary-replacement messages. The server must accumulate all rows and emit one batch (tableToIPC(tableFromJSON(allRows), 'stream')). Consequence: the client "stream" is just a chunked download of one batch — nothing renders progressively, and the server holds the full result set in memory per request.
• apache-arrow pinned to match the client WASM. Pin it exact in the server package.json and treat it as part of the locked unit (principle above). Note the @perspective-dev/client build is tested against apache-arrow@17.0.0, while pf_app's server currently pins ^21.1.0 — a real IPC version gap. Prefer aligning the server toward the arrow major the client was built against, or at minimum make Arrow ingestion (smoke test §7) the gate on any arrow bump.

JSON avoids all three but pays in payload size and parse cost, and pushes type handling to client-side heuristics — acceptable only at small scale.

---

4. Theming

• One toggle drives both app CSS and the viewer: viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light').
• Apply it on initial load and in an effect keyed on the dark flag, so the viewer re-themes when the toggle fires (not just on mount).
• App-level dark mode is plain CSS custom properties + a .dark class on <html>; the viewer theme name is the only Perspective-specific piece.

---

5. Layout persistence

• Persist the full viewer config (await viewer.save(), incl. plugin_config) to localStorage, keyed per source/version. Restore with viewer.restore(cfg).
• Guard restores against schema drift. Before restoring, filter the saved config's columns/group_by/split_by/sort/filter against the columns that actually exist in the current dataset (plus any expressions). dataflow's cleanLayout() is the reference implementation; a stale layout referencing a dropped column otherwise throws on restore.

---

6. Build & deploy (target)

• Build: vite build emits a static bundle (pf_app → public/app, via outDir: '../public/app'). The API server serves it statically (express.static).
• Process: run the Node API under systemd with Restart=always and an EnvironmentFile=.env; front it with nginx (reverse proxy + TLS via certbot). dataflow's dataflow.service + deploy.sh are the reference; pf_app has no deploy automation yet and should adopt the same pattern.
• deploy.sh should be idempotent: first run installs (db/schema, UI build, nginx, systemd unit); later runs rebuild the UI and restart the service.
• Keep secrets in .env (db creds), loaded by both the app (dotenv) and the systemd unit — never commit it.

---

7. Upgrade checklist / smoke test

Run this whenever bumping any Perspective package or apache-arrow:

1. Confirm viewer-d3fc publishes the target version (npm view @perspective-dev/viewer-d3fc versions). If not, don't bump the others.
2. Pin all four packages + apache-arrow to exact, matching versions; npm install; commit the lockfile.
3. vite build — no unresolved imports.
4. Arrow apps: load a real dataset and confirm worker.table(buffer) ingests without a WASM dictionary error; verify a numeric column is Float64/Int, not a string/dictionary.
5. Open a d3fc chart plugin (not just datagrid) and confirm it renders.
6. Toggle dark/light; confirm the viewer re-themes.
7. Save a layout, reload, confirm it restores; then drop a column and confirm the cleaned restore doesn't throw.

---

Per-project state (2026-06)

	pf_app	dataflow	Target
Loader	CDN (runtime)	npm /inline	npm /inline
Version	4.4.0 (CDN URLs)	4.5.1 viewer/client + 4.4.1 d3fc	depends on loader (§2)
Data	Arrow IPC (single batch)	JSON (≤100k)	per workload (§3)
apache-arrow	^21.1.0 (client built vs 17)	n/a	pin exact, match WASM
Deploy	none	systemd + nginx + deploy.sh	systemd + nginx + deploy.sh

dataflow's 4.5.1/4.4.1 pair is correct — it's the only combo giving both inline bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.

Open items:

• pf_app → move off CDN. Note this forces the §2 choice: going npm-/inline means 4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and load via the . entry instead of /inline. Either way, pin + commit the lockfile, and add deploy automation (systemd + nginx + deploy.sh).