Working measurement spike for Option C: aggregation runs in a server-side
DuckDB instead of shipping every row to the browser. Opt-in via
?engine=duck; the default client-side WASM path is untouched.
- routes/perspective.js: POST /perspective/init materializes a version's
forecast table into a persistent in-process DuckDB (ATTACH postgres,
so 535k rows land in ~1.5s without streaming through Node); POST
/perspective/sql executes one Perspective-generated statement, with
Arrow IPC output for viewport reads.
- ui/src/duckServerHandler.js: VirtualServerHandler that delegates SQL
generation to Perspective's own GenericSQLVirtualServerModel and POSTs
the result to the server. Adapted from @perspective-dev/client's
virtual_servers/duckdb.ts — same SQL model, different transport.
- Forecast.jsx: DUCK_MODE branch replaces the heavy /data fetch with
/perspective/init, opens the hosted table over a message port, and
shows a latency overlay. applyDepth routes through viewer.restore()
since the SQL server cannot handle view.set_depth().
Preserved for a future faster-load effort. Findings and the reason this
is not the shipped path (no interactive tree expand/collapse) are in
pf_perspective_options.md on master.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pg now returns bigint/numeric as JS numbers so Arrow infers Int/Float64
instead of Dictionary<Utf8>. /data accumulates rows and emits a single
record batch to avoid dictionary REPLACEMENT messages that crash
Perspective's WASM reader. Forecast view streams the response body and
shows received/total bytes while loading. Drops stale public/ static
middleware that was shadowing the React build at /.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Server streams rows from a pg cursor in 10k-row batches, building Arrow
record batches incrementally and piping them as chunked HTTP response —
Node.js heap stays bounded regardless of dataset size.
Client fetches as arrayBuffer() and loads directly into Perspective worker
(native Arrow path, no JSON deserialization). X-Row-Count header drives
a non-blocking banner for datasets >= 500k rows. validCols now derived
from col_meta rather than from row keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
