pf_app/PERSPECTIVE.md

# Perspective — configuration & deployment reference

Canonical guide for how Perspective should be configured, fed, and shipped across our
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
but made different early choices; this doc defines the target state and the rationale, so
the two converge instead of drifting.

> **Distribution note:** we use the **`@perspective-dev/*`** packages
> (repo: <https://github.com/perspective-dev/perspective>, home:
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
> scopes.

---

## Core principle: Perspective is one locked unit

These four things must move together and be pinned together. Bumping one without the
others is the source of nearly every Perspective bug we've hit:

1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
3. **Data format** — Arrow IPC vs JSON rows
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
   format the client WASM understands

Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.

---

## 1. Loading — npm `/inline`, pinned exact (not CDN)

```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
```

- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
  fetch, works offline / behind a firewall, reproducible from the lockfile.**
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
  one-line version bumps) but in production it means: app breaks if the CDN is
  unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
  pull executable WASM from a third party on every load.

  > **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
  > resolves its server WASM with
  > `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
  > which from `.../client@4.4.0/dist/cdn/` resolves to
  > `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**.
  > jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
  > pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
  > `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
  > Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
  >
  > The security framing matters as much as the outage: an unversioned URL means your
  > users execute whatever that package publishes next, automatically, unreviewed.
  >
  > 5.2.0 fixes it by carrying the client's version across
  > (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
  > the WASM is embedded in the bundle and there is no runtime fetch to hijack
  > (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).

  pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
  `<link>` in `index.html` — so it's bundled and versioned too.

---

## 2. Version policy — a real trilemma (read carefully)

The version choice is constrained by two hard facts about the `@perspective-dev` packages
**(verified against installed metadata, 2026-06)**:

- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
  Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even
  though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
  trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
  `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
  in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.

So you can have at most **two** of these three:

| Want | Requires |
|---|---|
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) |

There is **no** version where all three hold. Pick by what the app needs:

> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts
>
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1`
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.**
>
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
>   running exactly that pair), *every chart type other than Datagrid fails.*
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
>   `viewer` throws `get_static_config is not a function` — once per chart plugin. The
>   4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
>   load against a coherent 4.4.1 viewer produces no such error.
>
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
>
> Consequence: dataflow currently has the worst of both worlds. It carries the
> mixed-version complexity *specifically* to keep charts, and does not have charts.
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
>
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
> real bundled build. It is the documented-and-untested assumption this whole policy
> rests on — establish it before betting a version choice on it.

- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
  `^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
  working Datagrid and no charts. If you are on it today, you are choosing inline
  bundling, not charts; be explicit about which one you actually want.
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
  **4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give
  up `/inline` bundling.

Whatever you pick, **commit the lockfile** so the resolved set can't drift on
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or
later (then a fully-coherent inline-capable suite becomes possible — and only then does
"both" come back on the table).

---

## 3. Data delivery — match the format to the workload

| Workload | Format | Why |
|---|---|---|
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |

### Arrow constraints (read before choosing it)

If you deliver Arrow, three pieces are coupled and must stay aligned:

- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
  must coerce them to JS numbers before encoding, or `apache-arrow` infers
  `Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
  1700).
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
  Perspective WASM crashes on dictionary-replacement messages. The server must
  accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
  'stream')`). Consequence: the client "stream" is just a chunked *download* of one
  batch — nothing renders progressively, and the server holds the full result set in
  memory per request.
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
  `package.json` and treat it as part of the locked unit (principle above). Note the
  `@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
  pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
  the server toward the arrow major the client was built against, or at minimum make
  Arrow ingestion (smoke test §7) the gate on any arrow bump.

JSON avoids all three but pays in payload size and parse cost, and pushes type handling
to client-side heuristics — acceptable only at small scale.

---

## 3a. Expression columns and aggregation order (ratio correctness)

**Expression columns are row-level.** Perspective evaluates every expression against
each *raw row* first, then feeds the result into the column's aggregate. It has no
post-aggregate expression stage. So a ratio written the obvious way:

```js
expressions: { price: '"revenue" / "qty"' }   // default aggregate: sum
```

...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.

This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
grouping just makes it visible by putting several wrong numbers side by side. No
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
`high`/`low`/`median`/`dominant` are all wrong for the same reason.

### The fix: a weighted-mean aggregate

Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:

```js
expressions: { price: '"revenue" / "qty"' },
aggregates:  { price: ['weighted mean', ['qty']] }   // note the NESTED array
```

**The nested array matters.** The type is
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
reads like "no such aggregate" and is easy to misread as the feature being absent.

Available since **4.4.0** — no version bump needed, and because the aggregate is
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
`table.update()` on a live view re-derives the weighted mean from the merged rows
without a reload).

### Rules of thumb

- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
  → leave the default `sum`. Those are unaffected by any of this.
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
  `['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
- Mixing both in one view is fine and was verified.

### Type inference can silently break expressions

`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
as `integer` — which happens to numeric-looking strings when the table is created from
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
error. Give period/year columns an explicit `string` type at table creation.

### Not fixed by any of this: column-axis expand/collapse

`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
(§2: `viewer-d3fc` still caps at 4.4.1).

---

## 4. Theming

- One toggle drives both app CSS and the viewer:
  `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
  re-themes when the toggle fires (not just on mount).
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
  viewer theme name is the only Perspective-specific piece.

---

## 5. Layout persistence

- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
  `localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
- **Guard restores against schema drift.** Before restoring, filter the saved config's
  `columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
  exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
  the reference implementation; a stale layout referencing a dropped column otherwise
  throws on restore.
- **`aggregates` needs the same guard, and doesn't currently have it** (verified 2026-08,
  pf_app). Both existing `cleanLayout()` implementations filter
  `columns`/`group_by`/`split_by`/`sort`/`filter` but leave `aggregates` untouched. That
  is harmless *today* only because `viewer.save()` emits `aggregates: {}` until someone
  sets one explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
  dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
  throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
  aggregate entry references a *target* column and, in the multi-arg form, a *weight*
  column; both need validating, and the entry should be dropped rather than the layout.
  Add this guard as part of adopting §3a, not after.

### Auto-pause deletes the view — turn it off for a static pivot

`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on itself
(scrolled out of the viewport, `display: none`) combined with the document's
`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint
optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the
**view object is destroyed**. Becoming visible again calls
`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full
traversal, every time.

For a viewer streaming live updates nobody is watching, that is the right trade.
For a pivot over a large static table it is the wrong one — on pf_app's
`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it
silently discards per-node expand/collapse (§"Not fixed by any of this"), which
has no config representation and so cannot be restored.

```js
await viewer.load(table)
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
```

Guard the call: it is a method on the custom element and absent on older builds.
Leave auto-pause **on** where the table is fed by a live stream the user does not
need to have kept up with while away. This is long-standing viewer behaviour, not
a 5.x regression — worth knowing before blaming a rebuild on your own code.

---

## 6. Build & deploy (target)

- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
  `outDir: '../public/app'`). The API server serves it statically (`express.static`).
- **Process:** run the Node API under **systemd** with `Restart=always` and an
  `EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
  dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
  automation yet** and should adopt the same pattern.
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
  systemd unit); later runs rebuild the UI and restart the service.
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
  unit — never commit it.

---

## 7. Upgrade checklist / smoke test

Run this whenever bumping **any** Perspective package or `apache-arrow`:

1. Confirm `viewer-d3fc` publishes the target version
   (`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
   commit the lockfile.
   - **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
     it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
     `latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
     and the WASM fails to link:
     `Import #8 "env" "psp_opfs_load": function import requires a callable`.
     Five packages, not four. This is invisible while pf_app loads from the CDN, and
     will bite on the "move off CDN" open item below.
3. `vite build` — no unresolved imports.
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
   without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
   string/dictionary.
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
6. Toggle dark/light; confirm the viewer re-themes.
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
   cleaned restore doesn't throw.

---

## Per-project state (2026-06)

| | pf_app | dataflow | Target |
|---|---|---|---|
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** |
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** |
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test |
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |

**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.

**Open items:**
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages
  pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host
  blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction):
  its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
  Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
  render before choosing the latter, because nobody has confirmed they do.
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).