From 64f3cc58e8779ad8e7fc060a22ab0dad25cd7935 Mon Sep 17 00:00:00 2001 From: Paul Trowbridge Date: Mon, 15 Jun 2026 23:49:53 -0400 Subject: [PATCH] Add PERSPECTIVE.md config/deploy reference; fix CLAUDE.md distribution link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the @perspective-dev distribution (not FINOS @finos/perspective): loader (npm /inline vs CDN), the version trilemma (inline needs 4.5.x, viewer-d3fc caps at 4.4.1, charts need 4.4.1 — can't have all three), Arrow vs JSON delivery constraints, deploy pattern, and an upgrade smoke test. Correct CLAUDE.md's stale perspective.finos.org link to the actual @perspective-dev repo. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 2 +- PERSPECTIVE.md | 199 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 200 insertions(+), 1 deletion(-) create mode 100644 PERSPECTIVE.md diff --git a/CLAUDE.md b/CLAUDE.md index 65ae2e6..1554c39 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,7 @@ UX mockup: `pf_ux_mockup.md` - **Backend:** Node.js / Express (`server.js`) - **Database:** PostgreSQL — isolated `pf` schema - **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/` -- **Pivot:** [Perspective](https://perspective.finos.org/) 4.4.0 loaded from CDN at runtime +- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance - **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/` --- diff --git a/PERSPECTIVE.md b/PERSPECTIVE.md new file mode 100644 index 0000000..7d19abe --- /dev/null +++ b/PERSPECTIVE.md @@ -0,0 +1,199 @@ +# Perspective — configuration & deployment reference + +Canonical guide for how Perspective should be configured, fed, and shipped across our +apps (**pf_app**, **dataflow**). Both embed the same `` web component +but made different early choices; this doc defines the target state and the rationale, so +the two converge instead of drifting. + +> **Distribution note:** we use the **`@perspective-dev/*`** packages +> (repo: , home: +> ) — **not** the FINOS/OpenJS `@finos/perspective` +> packages from . Same engine lineage, but a separate +> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why +> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the +> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two +> scopes. + +--- + +## Core principle: Perspective is one locked unit + +These four things must move together and be pinned together. Bumping one without the +others is the source of nearly every Perspective bug we've hit: + +1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN) +2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc` +3. **Data format** — Arrow IPC vs JSON rows +4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC + format the client WASM understands + +Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test +at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`. + +--- + +## 1. Loading — npm `/inline`, pinned exact (not CDN) + +```js +import perspective from '@perspective-dev/client/inline' +import '@perspective-dev/viewer/inline' +import '@perspective-dev/viewer-datagrid' +import '@perspective-dev/viewer-d3fc' +import '@perspective-dev/viewer/themes' +``` + +- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network + fetch, works offline / behind a firewall, reproducible from the lockfile.** +- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build, + one-line version bumps) but in production it means: app breaks if the CDN is + unreachable, version isn't captured in `package-lock.json`, slower cold start, and you + pull executable WASM from a third party on every load. (pf_app currently does this in + `ui/src/views/Forecast.jsx` — migrating off it is the main open item.) +- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a + `` in `index.html` — so it's bundled and versioned too. + +--- + +## 2. Version policy — a real trilemma (read carefully) + +The version choice is constrained by two hard facts about the `@perspective-dev` packages +**(verified against installed metadata, 2026-06)**: + +- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line / + Treemap / Heatmap / etc.) live only in this package. +- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`, + `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist + in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x. + +So you can have at most **two** of these three: + +| Want | Requires | +|---|---| +| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client | +| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) | +| d3fc chart plugins | **4.4.1** viewer-d3fc | + +There is **no** version where all three hold. Pick by what the app needs: + +- **Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+ + `^4.4.1` viewer-d3fc`. This is a deliberate, necessary mixed-version pair, *not* an + accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate + bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the + build breaks (`"./inline" is not exported`). +- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to + **4.4.1 exact**. Charts work; you give up `/inline` bundling. + +Whatever you pick, **commit the lockfile** so the resolved set can't drift on +`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x (then a +fully-coherent inline-capable 4.5.x suite becomes possible). + +--- + +## 3. Data delivery — match the format to the workload + +| Workload | Format | Why | +|---|---|---| +| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. | +| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. | + +### Arrow constraints (read before choosing it) + +If you deliver Arrow, three pieces are coupled and must stay aligned: + +- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you + must coerce them to JS numbers before encoding, or `apache-arrow` infers + `Dictionary` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20, + 1700). +- **Single record batch.** Per-batch Arrow builds independent dictionaries; the + Perspective WASM crashes on dictionary-replacement messages. The server must + accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows), + 'stream')`). Consequence: the client "stream" is just a chunked *download* of one + batch — nothing renders progressively, and the server holds the full result set in + memory per request. +- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server + `package.json` and treat it as part of the locked unit (principle above). Note the + `@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while + pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning + the server toward the arrow major the client was built against, or at minimum make + Arrow ingestion (smoke test §7) the gate on any arrow bump. + +JSON avoids all three but pays in payload size and parse cost, and pushes type handling +to client-side heuristics — acceptable only at small scale. + +--- + +## 4. Theming + +- One toggle drives both app CSS and the viewer: + `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`. +- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer + re-themes when the toggle fires (not just on mount). +- App-level dark mode is plain CSS custom properties + a `.dark` class on ``; the + viewer theme name is the only Perspective-specific piece. + +--- + +## 5. Layout persistence + +- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to + `localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`. +- **Guard restores against schema drift.** Before restoring, filter the saved config's + `columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually + exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is + the reference implementation; a stale layout referencing a dropped column otherwise + throws on restore. + +--- + +## 6. Build & deploy (target) + +- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via + `outDir: '../public/app'`). The API server serves it statically (`express.static`). +- **Process:** run the Node API under **systemd** with `Restart=always` and an + `EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot). + dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy + automation yet** and should adopt the same pattern. +- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx, + systemd unit); later runs rebuild the UI and restart the service. +- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd + unit — never commit it. + +--- + +## 7. Upgrade checklist / smoke test + +Run this whenever bumping **any** Perspective package or `apache-arrow`: + +1. Confirm `viewer-d3fc` publishes the target version + (`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others. +2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`; + commit the lockfile. +3. `vite build` — no unresolved imports. +4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests + without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a + string/dictionary. +5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders. +6. Toggle dark/light; confirm the viewer re-themes. +7. Save a layout, reload, confirm it restores; then drop a column and confirm the + cleaned restore doesn't throw. + +--- + +## Per-project state (2026-06) + +| | pf_app | dataflow | Target | +|---|---|---|---| +| Loader | CDN (runtime) | npm `/inline` | **npm `/inline`** | +| Version | 4.4.0 (CDN URLs) | 4.5.1 viewer/client + 4.4.1 d3fc | depends on loader (§2) | +| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) | +| `apache-arrow` | `^21.1.0` (client built vs 17) | n/a | pin exact, match WASM | +| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** | + +**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline +bundling and d3fc charts (§2). Leave it; just keep the lockfile committed. + +**Open items:** +- pf_app → move off CDN. Note this forces the §2 choice: going npm-`/inline` means + 4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and + load via the `.` entry instead of `/inline`. Either way, pin + commit the lockfile, and + add deploy automation (systemd + nginx + `deploy.sh`).