dataflow/docs/perspective.md

# Perspective

Everything about the Perspective pivot in dataflow: which packages and versions are
pinned and why, and a ground-truth reference for the parts of the API the official docs
don't cover.

Shared rationale across projects lives in the canonical guide at
`/home/pt/pf_app/PERSPECTIVE.md` (loading, version policy, Arrow constraints, deploy
pattern, upgrade smoke test). This file records what's specific to dataflow.

---

> **Distribution:** these are the **`@perspective-dev/*`** packages (repo
> github.com/perspective-dev/perspective), **not** FINOS `@finos/perspective`. Same
> engine, separate npm scope and release schedule — don't mix the two.

---

## Current state

- **Loader:** npm `/inline` (`ui/src/pages/Pivot.jsx`) — bundled WASM, offline-capable. ✅
  This is the target loader; pf_app should adopt it.
- **Data:** JSON rows via `api.getViewData(source, 100000, 0)`, capped at 100k. ✅
  Correct for dataflow's read-only, click-to-inspect model. No need to move to Arrow
  unless view sizes grow well past 100k.
- **Deploy:** `manage.py` + `dataflow.service` (systemd) + nginx. ✅ Reference pattern
  for the org; pf_app should copy it.
- **Charts:** `viewer-d3fc` is imported, so the chart plugins are available in the UI.
  Default plugin config is datagrid-only (`{ edit_mode: 'SELECT_REGION' }`).
- **Layout safety:** `cleanLayout()` filters saved configs against valid columns before
  restore — the reference implementation; keep it.

## The version pair is correct — do NOT "fix" it to 4.4.1

`ui/package.json` pins **viewer/client/datagrid at `^4.5.1`** and **`viewer-d3fc` at
`^4.4.1`**. This looks like a skew but is **deliberate and necessary** — it's the only
combination that keeps both of dataflow's hard requirements:

- **Inline WASM bundling.** `Pivot.jsx` imports `@perspective-dev/client/inline`,
  `@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes`. Those export
  paths **exist only in 4.5.x** — they are absent from 4.4.1's `exports` map.
- **d3fc chart plugins.** `viewer-d3fc` is published only up to **4.4.1**.

Verified the hard way: pinning all four to 4.4.1 and rebuilding fails with
`"./inline" is not exported … from @perspective-dev/client`. So the 4.5.1/4.4.1 pair
stays. Don't touch it.

**What to actually do:**
- Keep the versions as-is; **commit `package-lock.json`** so the resolved set can't drift
  on `npm install`. (Optionally tighten the carets to exact `4.5.1`/`4.4.1` to make that
  explicit.)
- Treat any Perspective bump as gated by the canonical smoke test (§7): a d3fc **chart**
  renders, dark/light re-themes, and save→reload→drop-column layout restore.
- Revisit only when `viewer-d3fc` ships a 4.5.x — then a fully-coherent inline-capable
  4.5.x suite becomes possible and the pair can collapse to one version.

---

# API Reference

Packages: `@perspective-dev` client/viewer/viewer-datagrid at **v4.5.1**, viewer-d3fc at
**v4.4.1** — installed via npm. API notes that reference v4.4.0 behaviour have not been
re-verified at 4.5.1 but are believed to still apply. The official docs are incomplete
for some of these APIs — treat this as a ground-truth supplement.

## Loading via npm

```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
```

The `inline` builds embed WebAssembly directly into the JS bundle — no separate `.wasm` file to serve. viewer-datagrid and viewer-d3fc have no inline variant; they import normally. viewer-d3fc is currently at v4.4.1 (no v4.5.x release yet); its chart plugins register but may not appear in the viewer due to an API change in v4.5.x's `registerPlugin`.

---

## Core Objects

```
perspective            — the module default export
  .worker()            — creates a Web Worker instance

worker
  .table(rows, opts)   — creates a named Table; returns the Table object
  .open_table(name)    — re-opens a previously created named table

table
  .view(config)        — creates a View (filtered/grouped projection)
  .update(rows)        — incremental row upsert/insert

view
  .to_json()           — returns rows as array of objects
  .set_depth(n)        — sets expansion depth for all grouped rows (see below)
  .delete()            — frees the view; always call when done

viewer (the <perspective-viewer> DOM element)
  .load(worker)        — attaches the worker to the viewer
  .save()              — returns full viewer config as plain object
  .restore(config)     — applies a config object to the viewer
  .flush()             — forces viewer to synchronize (limited effect on plugin state)
  .getPlugin()         — returns the active plugin element (e.g. datagrid)
  .getView()           — returns the current View object
  .toggleConfig()      — shows/hides the settings panel

plugin (datagrid element, from viewer.getPlugin())
  .save()              — returns plugin-specific state: { columns, scroll_lock, edit_mode }
  .restore(config)     — applies plugin-specific state
  .draw(view)          — redraws the plugin against the given View
```

---

## viewer.save() — Config Shape

```js
{
  table: "source_name",
  plugin: "datagrid",           // or "d3_y_bar", etc.
  plugin_config: { ... },       // NOT reliably populated — use plugin.save() instead
  group_by: ["field1"],
  split_by: ["field2"],
  columns: ["Amount"],
  filter: [["field", "op", "value"]],
  sort: [["field", "asc"]],
  expressions: { "ExprName": "// formula\n..." },
  settings: false,              // whether the config panel is open
}
```

**Important:** `plugin_config` in `viewer.save()` is NOT reliably populated in v4.4.0. Use `plugin.save()` separately to capture plugin state.

---

## plugin.save() — Plugin State Shape (datagrid)

```js
{
  columns: {},          // per-column formatting overrides
  scroll_lock: false,
  edit_mode: "SELECT_REGION"   // see valid values below
}
```

---

## Selection Modes (edit_mode)

Valid values for the datagrid plugin's `edit_mode` field:

| Value | Button label | Behavior |
|---|---|---|
| `READ_ONLY` | Read-Only | No selection highlight |
| `SELECT_ROW` | Select Row | Highlights full rows |
| `SELECT_COLUMN` | Select Column | Highlights full columns |
| `SELECT_REGION` | Select Region | Highlights clicked cell region |
| `EDIT` | Edit | Enables cell editing |

The built-in button in the viewer toolbar cycles through these in order.

**Setting the default:**
```js
// After viewer.restore(...), set it directly on the plugin:
const plugin = await viewer.getPlugin()
await plugin.restore({ edit_mode: 'SELECT_REGION' })
```

Setting via `viewer.restore({ plugin_config: { edit_mode: ... } })` does NOT reliably work in v4.4.0.

---

## Expand/Collapse Row Depth

Controls how many levels of the `group_by` hierarchy are expanded. This is the only working mechanism found in v4.4.0:

```js
const view = await viewer.getView()
await view.set_depth(depth)       // 0 = collapse all, 1 = expand one level, etc.
const plugin = await viewer.getPlugin()
await plugin.draw(view)           // required — viewer does not redraw automatically
```

**What does NOT work:**
- `viewer.restore({ plugin_config: { expand_depth: d } })` — silently ignored
- `view.set_depth(d)` alone — view state changes but display doesn't update
- `view.set_depth(d)` + `viewer.flush()` — still no visual update
- `plugin.restore({ expand_depth: d })` — "Unknown" field, ignored

**The `plugin.draw(view)` call is required** to make the datagrid re-render after `set_depth`.

---

## Saving and Restoring Full State

To capture complete state (viewer + plugin + expand depth):

```js
async function captureConfig(viewer, expandDepth) {
  const plugin = await viewer.getPlugin()
  const [viewerConfig, pluginConfig] = await Promise.all([viewer.save(), plugin.save()])
  return { ...viewerConfig, plugin_config: pluginConfig, expand_depth: expandDepth }
}
```

To restore:
```js
async function restoreConfig(viewer, config, applyDepth) {
  await viewer.restore(config)
  if (config.plugin_config) {
    const plugin = await viewer.getPlugin()
    await plugin.restore(config.plugin_config)
  }
  if (config.expand_depth != null) {
    await applyDepth(viewer, config.expand_depth)
  }
  await viewer.flush()
}

async function applyDepth(viewer, depth) {
  const view = await viewer.getView()
  await view.set_depth(depth)
  const plugin = await viewer.getPlugin()
  await plugin.draw(view)
}
```

---

## The perspective-click Event

Fires when the user clicks a cell. The event detail:

```js
viewer.addEventListener('perspective-click', async (e) => {
  const { row, column_names, config } = e.detail
  // row          — aggregated values for the clicked cell (keyed by "split|metric" format)
  // column_names — array of metric column names clicked
  // config       — { filter: [[field, op, value], ...] }
  //   filter includes:
  //     - group_by coordinate filters  (field == value, one per group_by level)
  //     - split_by coordinate filters  (field == value, one per split_by field)
  //     - user-set filters             (any op)
})
```

`__ROW_PATH__` in `row` contains the group_by path as an array.

**The `config.filter` array is the reliable way to get cell coordinates.** Do not try to zip `__ROW_PATH__` with `group_by` — the filter approach handles all cases including partial paths.

---

## Filtering Rows for a Clicked Cell

The click event's `filter` array can be applied to the underlying table via a new View, which correctly evaluates expression/computed columns (unlike filtering raw JS rows):

```js
const config = await viewer.save()
const view = await table.view({
  filter: eventFilters,
  expressions: config.expressions || [],
})
const rows = await view.to_json()
await view.delete()

// Strip expression columns from results (they're computed, not source fields)
const exprNames = new Set(Object.keys(config.expressions || {}))
const clean = rows.map(r =>
  Object.fromEntries(Object.entries(r).filter(([k]) => !exprNames.has(k)))
)
```

**Why not filter raw JS rows?** Expression columns (computed in Perspective) don't exist in the source data. `filterRowsByConfig` on raw rows will skip those filters, returning all rows for the group rather than the specific cell.

**Guard against no group_by:** Without `group_by`, the filter array has no coordinate filters and the view query returns the entire table (slow). Check first:

```js
const config = await viewer.save()
if ((config.group_by || []).length === 0) return  // no hierarchy — skip inspector
```

---

## Viewer Methods (full list, v4.4.0)

From `Object.getOwnPropertyNames(Object.getPrototypeOf(viewer))`:

`constructor`, `__destroy_into_raw`, `free`, `__get_model`, `connectedCallback`, `copy`, `delete`, `download`, `eject`, `export`, `flush`, `getAllPlugins`, `getClient`, `getEditPort`, `getPlugin`, `getRenderStats`, `getSelection`, `getTable`, `getView`, `getViewConfig`, `load`, `openColumnSettings`, `reset`, `resetError`, `resetThemes`, `resize`, `restore`, `restyleElement`, `save`, `setAutoPause`, `setAutoSize`, `setSelection`, `setThrottle`, `toggleColumnSettings`, `toggleConfig`

## Plugin Methods (datagrid, full list)

From `Object.getOwnPropertyNames(Object.getPrototypeOf(plugin))`:

`constructor`, `connectedCallback`, `disconnectedCallback`, `activate`, `name`, `category`, `select_mode`, `min_config_columns`, `config_column_names`, `group_rollups`, `priority`, `can_render_column_styles`, `column_style_controls`, `draw`, `update`, `render`, `resize`, `clear`, `save`, `restore`, `restyle`, `delete`

## View Methods (full list)

From `Object.getOwnPropertyNames(Object.getPrototypeOf(view))` — includes `set_depth`, `expand`, `collapse`, `to_json`, `to_csv`, `to_arrow`, `schema`, `num_rows`, `num_columns`, `delete`, and others.

---

## settings Panel

The `settings` key in `viewer.restore()` controls whether the config panel (gear icon) is open:

```js
// Hide on load:
await viewer.restore({ table: "name", settings: false, plugin_config: DEFAULT_PLUGIN_CONFIG })

// Toggle programmatically:
viewer.toggleConfig()
```

The settings state is saved by `viewer.save()` and restored on `viewer.restore()`, so it persists across layout saves automatically.

---

## Incremental Updates

To update the table data without a full reload:

```js
table.update(newRows)  // upserts by index (or by index_col if specified at table creation)
```

The viewer re-renders automatically after `table.update()`.

---

## Common Pitfalls

- **`plugin_config` in `viewer.restore()` is unreliable.** Always set plugin state via `plugin.restore()` separately after `viewer.restore()`.
- **`view.set_depth()` requires `plugin.draw(view)`.** The viewer won't redraw automatically.
- **Expression columns don't exist in raw data.** Filter via a Perspective View (`table.view({ filter, expressions })`), not against raw JS rows.
- **Always `await view.delete()`** after using a temporary view, or you'll leak worker memory.
- **Named tables:** `worker.table(rows, { name: 'foo' })` — the name is used by the viewer's `table` config key. Re-open with `worker.open_table('foo')`.