Rewrite the docs against the code, and move the design docs to docs/
An audit against the code found every doc had drifted. pf_spec.md mostly described behaviour the app no longer has and missed auth, territory, layouts, load plans, segment naming and scope entirely; CLAUDE.md had grown to carry the history as well as the rules, and loads into every session. - CLAUDE.md: rules, invariants and single-definition pointers only, with stale claims corrected (project layout, routes, schema, the pf_ slice columns, the price/volume and footing rules, known issues, deferred). Each section that lost its reasoning links to the spec. - docs/pf_spec.md: rewritten as the current design doc under twelve fixed headings, holding the history and rationale moved out of CLAUDE.md, the code issues found on the way, and a proposed (not built) design for additional measures -- cost, rebates, surcharges -- as rates on a driver. - docs/PERSPECTIVE.md: cut to the cross-app lessons that still hold; the fork and its rebuild are in ui/vendor/README.md. - docs/pf_perspective_options.md: marked as a dated decision record. - ui/vendor/README.md: depth is sent floored at 1, not d + 1; rebuild steps. - ui/README.md: removed (Vite template boilerplate). - Two stale code comments (server.js, Forecast.jsx). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
5feb28eb76
commit
1b1452b625
362
PERSPECTIVE.md
362
PERSPECTIVE.md
@ -1,362 +0,0 @@
|
||||
# Perspective — configuration & deployment reference
|
||||
|
||||
Canonical guide for how Perspective should be configured, fed, and shipped across our
|
||||
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
|
||||
but made different early choices; this doc defines the target state and the rationale, so
|
||||
the two converge instead of drifting.
|
||||
|
||||
> **Distribution note:** we use the **`@perspective-dev/*`** packages
|
||||
> (repo: <https://github.com/perspective-dev/perspective>, home:
|
||||
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
|
||||
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
|
||||
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
|
||||
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
|
||||
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
|
||||
> scopes.
|
||||
|
||||
---
|
||||
|
||||
## Core principle: Perspective is one locked unit
|
||||
|
||||
These four things must move together and be pinned together. Bumping one without the
|
||||
others is the source of nearly every Perspective bug we've hit:
|
||||
|
||||
1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
|
||||
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
|
||||
3. **Data format** — Arrow IPC vs JSON rows
|
||||
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
|
||||
format the client WASM understands
|
||||
|
||||
Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
|
||||
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Loading — npm `/inline`, pinned exact (not CDN)
|
||||
|
||||
```js
|
||||
import perspective from '@perspective-dev/client/inline'
|
||||
import '@perspective-dev/viewer/inline'
|
||||
import '@perspective-dev/viewer-datagrid'
|
||||
import '@perspective-dev/viewer-d3fc'
|
||||
import '@perspective-dev/viewer/themes'
|
||||
```
|
||||
|
||||
- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
|
||||
fetch, works offline / behind a firewall, reproducible from the lockfile.**
|
||||
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
|
||||
one-line version bumps) but in production it means: app breaks if the CDN is
|
||||
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
|
||||
pull executable WASM from a third party on every load.
|
||||
|
||||
> **This is not hypothetical — it took pf_app down on 2026-08-10.** The 4.x CDN bundle
|
||||
> resolves its server WASM with
|
||||
> `new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
|
||||
> which from `.../client@4.4.0/dist/cdn/` resolves to
|
||||
> `.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no version**.
|
||||
> jsdelivr serves `@latest`. The moment `@perspective-dev/server@5.2.0` published, every
|
||||
> pf_app page load linked a 5.2.0 WASM against a 4.4.0 client and threw
|
||||
> `LinkError: Import #8 "env" "psp_opfs_load": function import requires a callable`.
|
||||
> Nobody changed anything. 4.4.1 and 4.5.2 have the identical unversioned pattern.
|
||||
>
|
||||
> The security framing matters as much as the outage: an unversioned URL means your
|
||||
> users execute whatever that package publishes next, automatically, unreviewed.
|
||||
>
|
||||
> 5.2.0 fixes it by carrying the client's version across
|
||||
> (`/client@X/dist/cdn/… → /server@X/dist/wasm/…`), but the durable fix is `/inline`:
|
||||
> the WASM is embedded in the bundle and there is no runtime fetch to hijack
|
||||
> (verified: zero `new URL(...perspective-server...)` in `perspective.inline.js`).
|
||||
|
||||
pf_app migrated off CDN to npm `/inline` at 5.2.0 on 2026-08-17.
|
||||
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
|
||||
`<link>` in `index.html` — so it's bundled and versioned too.
|
||||
|
||||
---
|
||||
|
||||
## 2. Version policy — a real trilemma (read carefully)
|
||||
|
||||
The version choice is constrained by two hard facts about the `@perspective-dev` packages
|
||||
**(verified against installed metadata, 2026-06)**:
|
||||
|
||||
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
|
||||
Treemap / Heatmap / etc.) live only in this package. **Still true as of 2026-08**, even
|
||||
though `client`/`viewer`/`viewer-datagrid` now publish through **5.2.0** — so the
|
||||
trilemma below has widened, not closed: taking 5.x costs the d3fc charts outright.
|
||||
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
|
||||
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
|
||||
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
|
||||
|
||||
So you can have at most **two** of these three:
|
||||
|
||||
| Want | Requires |
|
||||
|---|---|
|
||||
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
|
||||
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
|
||||
| d3fc chart plugins | **4.4.1** viewer-d3fc — *and a 4.4.1 viewer to match* (see correction) |
|
||||
|
||||
There is **no** version where all three hold. Pick by what the app needs:
|
||||
|
||||
> ### ⚠️ CORRECTION (2026-08): the mixed pair does NOT deliver charts
|
||||
>
|
||||
> This section previously recommended `^4.5.1` viewer/client/datagrid + `^4.4.1`
|
||||
> viewer-d3fc as "the only combo that keeps both." **That recommendation was wrong.**
|
||||
>
|
||||
> - **Confirmed by the app owner:** in the deployed dataflow install (`/opt/dataflow`,
|
||||
> running exactly that pair), *every chart type other than Datagrid fails.*
|
||||
> - **Mechanism, reproduced in isolation:** loading 4.4.1 `viewer-d3fc` against a 4.5.1
|
||||
> `viewer` throws `get_static_config is not a function` — once per chart plugin. The
|
||||
> 4.5.x viewer calls a registration method the 4.4.1 plugins don't implement. The same
|
||||
> load against a coherent 4.4.1 viewer produces no such error.
|
||||
>
|
||||
> So the "Inline-bundled + charts" row below is **not achievable**. The trilemma is
|
||||
> really a **dilemma**: inline WASM bundling **XOR** d3fc charts — pick one.
|
||||
>
|
||||
> Consequence: dataflow currently has the worst of both worlds. It carries the
|
||||
> mixed-version complexity *specifically* to keep charts, and does not have charts.
|
||||
> Both directions are strictly better than standing still: down to a coherent **4.4.1**
|
||||
> suite (if charts matter) or up to **5.x** (if they don't, and you want the newer
|
||||
> engine — §3a, `split_rollup_mode`, `edit_mode` persistence).
|
||||
>
|
||||
> **Still unverified:** whether a coherent 4.4.1 suite actually *renders* charts in a
|
||||
> real bundled build. It is the documented-and-untested assumption this whole policy
|
||||
> rests on — establish it before betting a version choice on it.
|
||||
|
||||
- ~~**Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
|
||||
`^4.4.1` viewer-d3fc`.~~ **Withdrawn — see correction above.** This pair yields a
|
||||
working Datagrid and no charts. If you are on it today, you are choosing inline
|
||||
bundling, not charts; be explicit about which one you actually want.
|
||||
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
|
||||
**4.4.1 exact**. Charts are *believed* to work here (unverified — see above); you give
|
||||
up `/inline` bundling.
|
||||
|
||||
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
|
||||
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x or
|
||||
later (then a fully-coherent inline-capable suite becomes possible — and only then does
|
||||
"both" come back on the table).
|
||||
|
||||
---
|
||||
|
||||
## 3. Data delivery — match the format to the workload
|
||||
|
||||
| Workload | Format | Why |
|
||||
|---|---|---|
|
||||
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
|
||||
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |
|
||||
|
||||
### Arrow constraints (read before choosing it)
|
||||
|
||||
If you deliver Arrow, three pieces are coupled and must stay aligned:
|
||||
|
||||
- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
|
||||
must coerce them to JS numbers before encoding, or `apache-arrow` infers
|
||||
`Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
|
||||
1700).
|
||||
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
|
||||
Perspective WASM crashes on dictionary-replacement messages. The server must
|
||||
accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
|
||||
'stream')`). Consequence: the client "stream" is just a chunked *download* of one
|
||||
batch — nothing renders progressively, and the server holds the full result set in
|
||||
memory per request.
|
||||
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
|
||||
`package.json` and treat it as part of the locked unit (principle above). Note the
|
||||
`@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
|
||||
pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
|
||||
the server toward the arrow major the client was built against, or at minimum make
|
||||
Arrow ingestion (smoke test §7) the gate on any arrow bump.
|
||||
|
||||
JSON avoids all three but pays in payload size and parse cost, and pushes type handling
|
||||
to client-side heuristics — acceptable only at small scale.
|
||||
|
||||
---
|
||||
|
||||
## 3a. Expression columns and aggregation order (ratio correctness)
|
||||
|
||||
**Expression columns are row-level.** Perspective evaluates every expression against
|
||||
each *raw row* first, then feeds the result into the column's aggregate. It has no
|
||||
post-aggregate expression stage. So a ratio written the obvious way:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' } // default aggregate: sum
|
||||
```
|
||||
|
||||
...computes `revenue/qty` per row and then **sums the per-row ratios** — the classic
|
||||
sum-then-divide error. Verified on 4.4.0/4.4.1/4.5.2/5.2.0 (all identical): for a group
|
||||
whose true `sum(revenue)/sum(qty)` is `16.15`, the pivot shows `42`.
|
||||
|
||||
This is **not** a `split_by` bug. It is equally wrong with only `group_by` — column
|
||||
grouping just makes it visible by putting several wrong numbers side by side. No
|
||||
scalar aggregate fixes it: `avg`/`mean` give the average *of ratios* (`14`), and
|
||||
`high`/`low`/`median`/`dominant` are all wrong for the same reason.
|
||||
|
||||
### The fix: a weighted-mean aggregate
|
||||
|
||||
Weight the ratio by its own denominator. `sum(price_i × qty_i) / sum(qty_i)` is
|
||||
algebraically `sum(revenue)/sum(qty)` — the correct answer at *every* level of both axes:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' },
|
||||
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
|
||||
```
|
||||
|
||||
**The nested array matters.** The type is
|
||||
`Aggregate = string | [string, Array<string>]` (`ts-rs/Aggregate.d.ts`), so the weight
|
||||
column goes in *its own array*. The flat form `['weighted mean', 'qty']` is rejected
|
||||
with the unhelpful `data did not match any variant of untagged enum Aggregate` — which
|
||||
reads like "no such aggregate" and is easy to misread as the feature being absent.
|
||||
|
||||
Available since **4.4.0** — no version bump needed, and because the aggregate is
|
||||
evaluated inside the engine, incremental `table.update()` stays correct (verified: a
|
||||
`table.update()` on a live view re-derives the weighted mean from the merged rows
|
||||
without a reload).
|
||||
|
||||
### Rules of thumb
|
||||
|
||||
- Column is a **sum of a measure** (incl. `if(...)` column-subtotal expressions, §below)
|
||||
→ leave the default `sum`. Those are unaffected by any of this.
|
||||
- Column is a **ratio, rate, price, or per-unit figure** → it *must* carry a
|
||||
`['weighted mean', ['<denominator>']]` aggregate, or it is wrong under any pivot.
|
||||
- Mixing both in one view is fine and was verified.
|
||||
|
||||
### Type inference can silently break expressions
|
||||
|
||||
`if("Year" == '2026', "Amount", 0)` returns 0 for every row if `Year` was **inferred**
|
||||
as `integer` — which happens to numeric-looking strings when the table is created from
|
||||
inferred JSON. With an explicit `string` schema the same expression is correct. Neither
|
||||
literal form (`'2026'` or `2026`) works against a mis-inferred column, and there is no
|
||||
error. Give period/year columns an explicit `string` type at table creation.
|
||||
|
||||
### Not fixed by any of this: column-axis expand/collapse
|
||||
|
||||
`view.expand()` / `view.collapse()` / `set_depth()` take a **row index** and act on the
|
||||
row axis only; there is no column-axis equivalent in 4.4.0 **or** 5.2.0, contrary to the
|
||||
docs' claim that both axes support it. Confirmed directly against the API. 5.2.0 adds
|
||||
`split_rollup_mode: 'rollup'`, which *emits* subtotal and grand-total column groups
|
||||
statically (no interactivity) — the nearest thing to Excel-style column subtotals, and
|
||||
it would retire the `if(...)`-expression workaround. It costs the d3fc charts, though
|
||||
(§2: `viewer-d3fc` still caps at 4.4.1).
|
||||
|
||||
---
|
||||
|
||||
## 4. Theming
|
||||
|
||||
- One toggle drives both app CSS and the viewer:
|
||||
`viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
|
||||
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
|
||||
re-themes when the toggle fires (not just on mount).
|
||||
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
|
||||
viewer theme name is the only Perspective-specific piece.
|
||||
|
||||
---
|
||||
|
||||
## 5. Layout persistence
|
||||
|
||||
- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
|
||||
`localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
|
||||
- **Guard restores against schema drift.** Before restoring, filter the saved config's
|
||||
`columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
|
||||
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
|
||||
the reference implementation; a stale layout referencing a dropped column otherwise
|
||||
throws on restore.
|
||||
- **`aggregates` needs the same guard.** pf_app's `cleanLayout()` has it as of 2026-09;
|
||||
**dataflow's does not** and should adopt it. Filtering
|
||||
`columns`/`group_by`/`split_by`/`sort`/`filter` and stopping there is harmless only
|
||||
while `viewer.save()` emits `aggregates: {}`, which it does until someone sets an
|
||||
aggregate explicitly. The moment a layout adopts the weighted-mean pattern (§3a), a
|
||||
dropped column aborts the entire restore — both `table.view()` and `viewer.restore()`
|
||||
throw `Could not get dtype for column 'X' as it does not exist in the schema`. An
|
||||
aggregate entry references a *target* column and, in the multi-arg form, a *weight*
|
||||
column; both need validating, and the entry is what gets dropped, never the layout.
|
||||
Add this guard as part of adopting §3a, not after.
|
||||
|
||||
### Auto-pause deletes the view — turn it off for a static pivot
|
||||
|
||||
`<perspective-viewer>` auto-pauses by default: an `IntersectionObserver` on itself
|
||||
(scrolled out of the viewport, `display: none`) combined with the document's
|
||||
`visibilitychange` (backgrounded tab, minimized window). "Pause" is not a paint
|
||||
optimisation — `session.set_pause(true)` runs `view_sub.take().delete()`, so the
|
||||
**view object is destroyed**. Becoming visible again calls
|
||||
`restore_and_render(…, ViewerConfigUpdate::default())`: a new view and a full
|
||||
traversal, every time.
|
||||
|
||||
For a viewer streaming live updates nobody is watching, that is the right trade.
|
||||
For a pivot over a large static table it is the wrong one — on pf_app's
|
||||
`fc_osm_skinny_29` grain it is a multi-second stall on every tab switch, and it
|
||||
silently discards per-node expand/collapse (§"Not fixed by any of this"), which
|
||||
has no config representation and so cannot be restored.
|
||||
|
||||
```js
|
||||
await viewer.load(table)
|
||||
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
|
||||
```
|
||||
|
||||
Guard the call: it is a method on the custom element and absent on older builds.
|
||||
Leave auto-pause **on** where the table is fed by a live stream the user does not
|
||||
need to have kept up with while away. This is long-standing viewer behaviour, not
|
||||
a 5.x regression — worth knowing before blaming a rebuild on your own code.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build & deploy (target)
|
||||
|
||||
- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
|
||||
`outDir: '../public/app'`). The API server serves it statically (`express.static`).
|
||||
- **Process:** run the Node API under **systemd** with `Restart=always` and an
|
||||
`EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
|
||||
dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
|
||||
automation yet** and should adopt the same pattern.
|
||||
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
|
||||
systemd unit); later runs rebuild the UI and restart the service.
|
||||
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
|
||||
unit — never commit it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Upgrade checklist / smoke test
|
||||
|
||||
Run this whenever bumping **any** Perspective package or `apache-arrow`:
|
||||
|
||||
1. Confirm `viewer-d3fc` publishes the target version
|
||||
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
|
||||
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
|
||||
commit the lockfile.
|
||||
- **Pin `@perspective-dev/server` explicitly too.** `@perspective-dev/client` declares
|
||||
it as `"@perspective-dev/server": ""` — an *empty* range, which npm resolves to
|
||||
`latest`. A fresh `npm i @perspective-dev/client@4.4.0` today pulls **server 5.2.0**
|
||||
and the WASM fails to link:
|
||||
`Import #8 "env" "psp_opfs_load": function import requires a callable`.
|
||||
Five packages, not four. This is invisible while pf_app loads from the CDN, and
|
||||
will bite on the "move off CDN" open item below.
|
||||
3. `vite build` — no unresolved imports.
|
||||
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
|
||||
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
|
||||
string/dictionary.
|
||||
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
|
||||
6. Toggle dark/light; confirm the viewer re-themes.
|
||||
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
|
||||
cleaned restore doesn't throw.
|
||||
|
||||
---
|
||||
|
||||
## Per-project state (2026-06)
|
||||
|
||||
| | pf_app | dataflow | Target |
|
||||
|---|---|---|---|
|
||||
| Loader | **npm `/inline`** (was CDN until 2026-08-17) | npm `/inline` | **npm `/inline`** |
|
||||
| Version | **5.2.0 exact, all four** (incl. `server`) | 4.5.1 viewer/client + 4.4.1 d3fc | **5.2.0 exact** |
|
||||
| Charts | none — d3fc import dropped (unused) | d3fc imported but **broken** (§2 correction) | decide per app |
|
||||
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
|
||||
| `apache-arrow` | `^21.1.0` — **verified OK against 5.2.0 WASM** | n/a | pin exact; verify by test |
|
||||
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
|
||||
|
||||
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
|
||||
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
|
||||
|
||||
**Open items:**
|
||||
- ~~pf_app → move off CDN.~~ **Done 2026-08-17** — npm `/inline`, all four packages
|
||||
pinned exact at 5.2.0, lockfile committed. Verified end-to-end with every external host
|
||||
blocked: viewer + datagrid register, the real Arrow stream ingests, the pivot renders.
|
||||
- **dataflow → same migration.** It is on the withdrawn 4.5.1/4.4.1 pair (§2 correction):
|
||||
its d3fc charts do not work, so it is paying mixed-version complexity for nothing.
|
||||
Either drop d3fc and go to 5.2.0, or go coherent 4.4.1 — but verify charts actually
|
||||
render before choosing the latter, because nobody has confirmed they do.
|
||||
- pf_app still has no deploy automation (systemd + nginx + `deploy.sh`).
|
||||
187
docs/PERSPECTIVE.md
Normal file
187
docs/PERSPECTIVE.md
Normal file
@ -0,0 +1,187 @@
|
||||
# Perspective — lessons and rules
|
||||
|
||||
What we have learned the hard way about configuring, feeding and upgrading
|
||||
Perspective, written for any app that embeds `<perspective-viewer>` (**pf_app**,
|
||||
**dataflow**). The pf_app specifics — the vendored 5.4.0 fork, its patch, and how
|
||||
to rebuild it — live in `ui/vendor/README.md`; how pf_app uses the viewer day to
|
||||
day is in `CLAUDE.md` and `pf_spec.md` → §The pivot.
|
||||
|
||||
> **Distribution:** we use the **`@perspective-dev/*`** packages
|
||||
> (<https://github.com/perspective-dev/perspective>) — **not** FINOS/OpenJS
|
||||
> `@finos/perspective`. Same engine lineage, but a separate npm scope, release
|
||||
> cadence and "Pro" theme set (`Pro Dark` / `Pro Light`). Don't mix the scopes.
|
||||
|
||||
---
|
||||
|
||||
## Perspective is one locked unit
|
||||
|
||||
These move together and are pinned together. Bumping one without the others is
|
||||
the source of nearly every Perspective bug we have hit:
|
||||
|
||||
1. **Loader** — how the JS/WASM gets into the page (npm `/inline`, never a CDN)
|
||||
2. **Package versions** — `client`, `server`, `viewer`, `viewer-datagrid` (and
|
||||
`viewer-d3fc` where charts are used)
|
||||
3. **Data format** — Arrow IPC vs JSON rows
|
||||
4. **`apache-arrow` on the server**, when delivering Arrow — it must write an IPC
|
||||
format the client WASM reads
|
||||
|
||||
**Pin `@perspective-dev/server` explicitly.** `@perspective-dev/client` declares
|
||||
it as `"@perspective-dev/server": ""` — an empty range, which npm resolves to
|
||||
`latest`. Installing an older client pulls whatever server is newest and the WASM
|
||||
fails to link (`Import #8 "env" "psp_opfs_load": function import requires a
|
||||
callable`). Five packages, not four.
|
||||
|
||||
---
|
||||
|
||||
## Load inline, never from a CDN
|
||||
|
||||
```js
|
||||
import perspective from '@perspective-dev/client/inline'
|
||||
import '@perspective-dev/viewer/inline'
|
||||
import '@perspective-dev/viewer-datagrid'
|
||||
import '@perspective-dev/viewer/themes'
|
||||
```
|
||||
|
||||
The `/inline` entrypoints bundle the WASM into the Vite build: no runtime fetch,
|
||||
works behind a firewall, reproducible from the lockfile. The themes CSS is
|
||||
imported in JS for the same reason, not `<link>`ed from `index.html`.
|
||||
|
||||
**This is not hypothetical — a CDN load took pf_app down on 2026-08-10.** The 4.x
|
||||
CDN bundle resolves its server WASM with
|
||||
`new URL("../../../server/dist/wasm/perspective-server.wasm", import.meta.url)`,
|
||||
which from `.../client@4.4.0/dist/cdn/` lands on
|
||||
`.../npm/@perspective-dev/server/dist/wasm/perspective-server.wasm` — **no
|
||||
version**, so jsdelivr serves `@latest`. The moment a 5.x server published, every
|
||||
page load linked a 5.x WASM against a 4.4.0 client and threw the `LinkError`
|
||||
above. Nobody changed anything. The security framing matters as much as the
|
||||
outage: an unversioned URL means users execute whatever that package publishes
|
||||
next, unreviewed. `/inline` has no runtime fetch to hijack.
|
||||
|
||||
---
|
||||
|
||||
## Charts versus inline bundling (apps that need d3fc)
|
||||
|
||||
pf_app uses the datagrid only and does not carry `viewer-d3fc`. For an app that
|
||||
does, the constraint as of 2026-08: `viewer-d3fc` published nothing past 4.4.1,
|
||||
while `/inline` and `/themes` exist only from 4.5.x. **A 4.5.x viewer with a 4.4.1
|
||||
d3fc does not work** — every chart plugin throws `get_static_config is not a
|
||||
function` (reproduced in isolation, and seen in dataflow's install, where every
|
||||
chart type but the datagrid fails). So it is inline bundling **or** d3fc charts,
|
||||
not both: go coherent 4.4.1 for charts (still unverified that a bundled 4.4.1
|
||||
suite renders them — check before relying on it), or 5.x without them. Re-check
|
||||
when d3fc publishes a newer version.
|
||||
|
||||
---
|
||||
|
||||
## Arrow delivery: three coupled constraints
|
||||
|
||||
Arrow IPC is the right format for large, numeric, incrementally-updated tables;
|
||||
JSON rows are simpler and fine for small read-only ones. Delivering Arrow couples
|
||||
three things:
|
||||
|
||||
- **Numeric type parsers, server-side.** pg returns `bigint` / `numeric` as
|
||||
strings; coerce them to numbers before encoding, or `apache-arrow` infers
|
||||
`Dictionary<Utf8>` instead of `Int` / `Float64`. pf_app: the type parsers in
|
||||
`server.js` (oids 20 and 1700).
|
||||
- **One record batch.** Per-batch encoding builds independent dictionaries, and
|
||||
the WASM crashes on dictionary-replacement messages. Accumulate all rows and emit
|
||||
a single batch — pf_app does `tableToIPC(tableFromArrays(columns), 'stream')`.
|
||||
The client's "stream" is therefore a chunked download of one batch; nothing
|
||||
renders progressively.
|
||||
- **`apache-arrow` versus the client WASM.** The client is built against an older
|
||||
arrow major than pf_app's server pins; the pair was verified working, but any
|
||||
arrow bump has to be gated on the ingest smoke test below.
|
||||
|
||||
---
|
||||
|
||||
## Expression columns are row-level: ratios need a weighted mean
|
||||
|
||||
Perspective evaluates every expression against each **raw row**, then feeds the
|
||||
result into the column's aggregate; there is no post-aggregate expression stage.
|
||||
So `price: '"revenue" / "qty"'` under the default `sum` **sums the per-row
|
||||
ratios** — wrong at every grouped level, with or without `split_by`. `avg` gives
|
||||
the average of ratios, which is also wrong.
|
||||
|
||||
Weight the ratio by its own denominator — `sum(price_i × qty_i) / sum(qty_i)` is
|
||||
exactly `sum(revenue) / sum(qty)`, at every level of both axes:
|
||||
|
||||
```js
|
||||
expressions: { price: '"revenue" / "qty"' },
|
||||
aggregates: { price: ['weighted mean', ['qty']] } // note the NESTED array
|
||||
```
|
||||
|
||||
**The nested array matters.** `Aggregate = string | [string, Array<string>]`, so
|
||||
the weight column goes in its own array. The flat `['weighted mean', 'qty']` is
|
||||
rejected with `data did not match any variant of untagged enum Aggregate`, which
|
||||
reads like the feature is missing. Being engine-side, it stays correct across
|
||||
incremental `table.update()`.
|
||||
|
||||
- A **sum of a measure** (including `if(...)` subtotal expressions) → default
|
||||
`sum`.
|
||||
- A **ratio, rate, price or per-unit figure** → must carry
|
||||
`['weighted mean', ['<denominator>']]`.
|
||||
|
||||
**Type inference can silently break expressions.** `if("Year" == '2026',
|
||||
"Amount", 0)` returns 0 on every row if `Year` was *inferred* as integer from
|
||||
numeric-looking strings, with no error, and neither literal form fixes it. Give
|
||||
period and year columns an explicit `string` type at table creation.
|
||||
|
||||
---
|
||||
|
||||
## Theming
|
||||
|
||||
The viewer theme is a `ViewerConfig` field: `viewer.restore({ theme: 'Pro Dark' })`.
|
||||
**`setAttribute('theme', …)` does nothing in 5.x** — the element observes no such
|
||||
attribute. Apply it on load and whenever the app toggle changes. `viewer.save()`
|
||||
includes `theme`, so strip it from stored layouts and ignore it when comparing
|
||||
configs — otherwise restoring a layout reasserts its author's theme.
|
||||
|
||||
---
|
||||
|
||||
## Layouts: guard every restore against schema drift
|
||||
|
||||
Before `viewer.restore(cfg)`, filter the saved `columns` / `group_by` /
|
||||
`split_by` / `sort` / `filter` against the columns the current table actually has
|
||||
(plus its `expressions`); a stale layout naming a dropped column otherwise throws.
|
||||
**`aggregates` needs the same guard** — once a layout carries a weighted mean, a
|
||||
dropped column aborts the whole restore (`Could not get dtype for column 'X'`).
|
||||
An aggregate names a target column and, in the multi-arg form, a weight column;
|
||||
validate both and drop the *entry*, never the layout, since `restore()` is
|
||||
all-or-nothing. pf_app's `cleanLayout()` does all of this; dataflow's does not yet
|
||||
guard `aggregates`.
|
||||
|
||||
---
|
||||
|
||||
## Auto-pause deletes the view — turn it off for a static pivot
|
||||
|
||||
`<perspective-viewer>` auto-pauses by default (an `IntersectionObserver` on itself
|
||||
plus the document's `visibilitychange`). Pausing is not a paint optimisation:
|
||||
`session.set_pause(true)` runs `view_sub.take().delete()`, so the view is
|
||||
destroyed, and becoming visible again rebuilds it with a full traversal — a
|
||||
multi-second stall per tab switch on a large table, and any per-node
|
||||
expand/collapse is lost with it.
|
||||
|
||||
```js
|
||||
await viewer.load(table)
|
||||
try { if (viewer.setAutoPause) await viewer.setAutoPause(false) } catch {}
|
||||
```
|
||||
|
||||
Leave it on only where a live stream feeds a table nobody needs kept current
|
||||
while away. Long-standing viewer behaviour, not a regression.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade smoke test
|
||||
|
||||
Run on any change to a Perspective package or `apache-arrow` (for pf_app, after
|
||||
rebuilding the vendored tarballs per `ui/vendor/README.md`):
|
||||
|
||||
1. All five packages (and `apache-arrow`) pinned exact and matching; lockfile
|
||||
committed.
|
||||
2. `vite build` — no unresolved imports.
|
||||
3. Arrow apps: load a real dataset; `worker.table(buffer)` ingests without a
|
||||
dictionary error, and a numeric column comes through as `Float64` / `Int`.
|
||||
4. Charts, where used: open a d3fc plugin and confirm it renders.
|
||||
5. Toggle dark/light; the viewer re-themes.
|
||||
6. Save a layout, reload, confirm it restores; drop a column and confirm the
|
||||
cleaned restore does not throw.
|
||||
@ -1,5 +1,17 @@
|
||||
# Perspective Architecture Options
|
||||
|
||||
> **Decision record, written 2026-08.** The outcome was the display-grain
|
||||
> pre-aggregation (§Spike findings onward), and it shipped. **"Current
|
||||
> architecture" below describes the app as it was then**, not now: it has
|
||||
> since moved to a vendored Perspective 5.4.0 bundled inline, `tableFromArrays`
|
||||
> on the server, `table.remove()` on undo, and `/data` / `/agg` emitting
|
||||
> `pf_segment`, `pf_bucket`, `pf_note` and `pf_op`. For today's data flow see
|
||||
> `CLAUDE.md` → §Core data flow and `pf_spec.md` → §Display-grain
|
||||
> pre-aggregation. What remains current here is the spike findings, the
|
||||
> display-grain reasoning, and the Path A / Path B analysis behind the
|
||||
> "dynamic grain" next step. The DuckDB virtual-server spike is parked on
|
||||
> branch `spike/duckdb-virtual-server`.
|
||||
|
||||
This document weighs how the Forecast view should source data for the Perspective
|
||||
pivot. The current implementation hits practical limits on initial load
|
||||
(~30s for 350k rows × ~55 cols), and growth is expected. Choosing an architecture
|
||||
1589
docs/pf_spec.md
Normal file
1589
docs/pf_spec.md
Normal file
File diff suppressed because it is too large
Load Diff
1076
pf_spec.md
1076
pf_spec.md
File diff suppressed because it is too large
Load Diff
@ -7,7 +7,7 @@ const { Pool, types } = require('pg');
|
||||
const { requireAuth } = require('./lib/auth');
|
||||
|
||||
// Return bigint (oid 20) and numeric (oid 1700) as JS numbers instead of strings,
|
||||
// so apache-arrow's tableFromJSON infers Int/Float64 rather than Dictionary<Utf8>.
|
||||
// so apache-arrow encodes them as Int/Float64 rather than Dictionary<Utf8>.
|
||||
types.setTypeParser(20, v => v === null ? null : Number(v));
|
||||
types.setTypeParser(1700, v => v === null ? null : Number(v));
|
||||
|
||||
|
||||
16
ui/README.md
16
ui/README.md
@ -1,16 +0,0 @@
|
||||
# React + Vite
|
||||
|
||||
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
|
||||
|
||||
Currently, two official plugins are available:
|
||||
|
||||
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
|
||||
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
|
||||
|
||||
## React Compiler
|
||||
|
||||
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
|
||||
|
||||
## Expanding the ESLint configuration
|
||||
|
||||
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.
|
||||
@ -317,10 +317,9 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
|
||||
// Mirrors ViewConfig.group_by_depth for the toolbar's active state. The
|
||||
// config is the source of truth; this only exists so the buttons can render.
|
||||
const [expandDepth, setExpandDepth] = useState(null)
|
||||
// The column axis has no set_depth() — collapsing it means restoring a shorter
|
||||
// split_by, so the full hierarchy has to be remembered separately to expand again.
|
||||
// The column hierarchy as the live config has it, for the Columns stepper.
|
||||
// Collapsing is split_by_depth now, so split_by itself is never truncated.
|
||||
const splitFullRef = useRef([])
|
||||
// a collapse from the user rearranging split_by themselves
|
||||
const initIdRef = useRef(0)
|
||||
const modifierRef = useRef(false)
|
||||
// the datagrid plugin element, for reading cell coordinates and driving its
|
||||
|
||||
20
ui/vendor/README.md
vendored
20
ui/vendor/README.md
vendored
@ -142,15 +142,23 @@ ctx2->set_depth(t_header::HEADER_ROW, row_pivot_depth - 1); // two-sided
|
||||
|
||||
The config field counts **levels to show**; `view.set_depth()` counts the
|
||||
boundary below them. So `group_by_depth: n` equals `set_depth(n - 1)`, and
|
||||
`Forecast.jsx` sends `d + 1`.
|
||||
`t_traversal::set_depth` adds the 1 back, so `n` renders `n` levels.
|
||||
`Forecast.jsx` therefore sends the stepper's level count unchanged, floored at
|
||||
1 — `0` would reach the engine as `set_depth(-1)` on an unsigned depth,
|
||||
underflow, and clamp to fully expanded. (It used to send `d + 1` to dodge that,
|
||||
which was off by one against the grid everywhere.)
|
||||
|
||||
To apply:
|
||||
The patch is already committed on the fork's build branch (see
|
||||
`PROVENANCE.txt`), so rebuilding needs only the script. The `.patch` file here
|
||||
is the change on its own, for review or for carrying to a clean upstream
|
||||
checkout:
|
||||
|
||||
```bash
|
||||
cd $PSP_DIR # default ~/perspective
|
||||
git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
|
||||
./ui/vendor/rebuild-perspective.sh
|
||||
cd ui && npm install
|
||||
./ui/vendor/rebuild-perspective.sh # from the fork branch
|
||||
# then re-resolve by path, as above -- a plain `npm install` keeps the old code
|
||||
|
||||
# only on a fresh upstream checkout:
|
||||
cd $PSP_DIR && git apply /path/to/pf_app/ui/vendor/0001-apply-depth-fields-on-config-update.patch
|
||||
```
|
||||
|
||||
### Host tools this tree needs
|
||||
|
||||
Loading…
Reference in New Issue
Block a user