Record DuckDB virtual-server spike findings; choose display-grain pre-aggregation
Document the outcome of the Option C spike (server-side DuckDB as a Perspective virtual server) and the resulting architecture decision. pf_perspective_options.md: - Spike findings: latency is excellent (~12ms round trip, 5-29ms aggregation, ~1.5-1.9s to materialize 534,902 rows) but Perspective's GenericSQLVirtualServerModel ignores group_by_depth and has no ViewConfig field for per-node expansion state, so interactive drill-down is not achievable. This affects options B, C and D alike since they share that SQL model. - Decision: the real lever is grain, not transport. Pre-aggregating to display grain collapses 534,902 -> 4,642 rows (~115x) on osm_stack while keeping the native Perspective engine, so expand/collapse/ depth/sort/filter continue to work. - Two candidate designs (Path A live virtual server vs Path B pre-aggregated extract) with the deciding question: do real cuts ever exceed the browser's leaf-row ceiling? pf_spec.md: - Concrete Path B design: pf.col_meta.in_grain, GET /api/versions/:id/agg, synthetic pf_gkey index, and the append-deltas write/undo model that mirrors the prior Excel pivot-cache workflow. Drop pf_ux_mockup.md — an ASCII mockup of UI that is now built; the views in ui/src/views are the current reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8d26629f32
commit
c5b12aac76
@ -6,7 +6,6 @@ A web app for building named forecast scenarios against any PostgreSQL table. Th
|
|||||||
|
|
||||||
Full spec: `pf_spec.md`
|
Full spec: `pf_spec.md`
|
||||||
Data transport architecture options: `pf_perspective_options.md`
|
Data transport architecture options: `pf_perspective_options.md`
|
||||||
UX mockup: `pf_ux_mockup.md`
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@ -255,3 +255,213 @@ A reasonable phased path: do A first (fast, low risk, ships value this
|
|||||||
week), live with it while planning, then move to D when row counts demand
|
week), live with it while planning, then move to D when row counts demand
|
||||||
it. C is a different shape and probably not warranted unless multi-user
|
it. C is a different shape and probably not warranted unless multi-user
|
||||||
emerges as a requirement.
|
emerges as a requirement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Spike findings — Option C (server-side DuckDB virtual server), 2026-06-18
|
||||||
|
|
||||||
|
A working spike was built to measure Option C on the real ~535k-row forecast
|
||||||
|
(`pf.fc_osm_stack_20`, version 20). It is opt-in behind `?engine=duck` and does
|
||||||
|
not touch the default client-side WASM path.
|
||||||
|
|
||||||
|
**Architecture built.** Perspective's `VirtualServer` + `GenericSQLVirtualServerModel`
|
||||||
|
run in the browser (reusing the already-loaded viewer WASM) and turn each
|
||||||
|
interaction into SQL. That SQL is POSTed to a new server endpoint
|
||||||
|
(`routes/perspective.js`, `POST /api/perspective/sql`) and executed against a
|
||||||
|
persistent in-process DuckDB that has the forecast table materialized from
|
||||||
|
Postgres. Only generated SQL + the aggregated viewport (Arrow IPC) cross the
|
||||||
|
wire. Files: `routes/perspective.js`, `ui/src/duckServerHandler.js`, a branch in
|
||||||
|
`Forecast.jsx`, and the `duckdb` node dependency.
|
||||||
|
|
||||||
|
**Latency — excellent, not the bottleneck.**
|
||||||
|
|
||||||
|
| Measure | Result |
|
||||||
|
|---|---|
|
||||||
|
| Materialize 534,902 rows into DuckDB (one-time, startup) | ~1.5–1.9 s |
|
||||||
|
| Pivot aggregation in DuckDB (1-col / 3-col rollup / filtered) | 5–29 ms |
|
||||||
|
| Per-query round trip incl. HTTP + Arrow (browser ↔ server) | **~12 ms** |
|
||||||
|
| Initial load (no rows over the wire — just schema + first viewport) | sub-second |
|
||||||
|
|
||||||
|
This is far below the doc's earlier 50–200 ms estimate and dwarfed by today's
|
||||||
|
~2-minute cold load on this dataset.
|
||||||
|
|
||||||
|
**Blocking limitation — no interactive group-tree drill-down.** Perspective's
|
||||||
|
generic SQL virtual server cannot drive the expandable rollup tree this app is
|
||||||
|
built around:
|
||||||
|
|
||||||
|
- `group_by_depth` is **ignored** by the SQL generator — verified by running
|
||||||
|
`GenericSQLVirtualServerModel` headlessly: the generated `tableMakeView` /
|
||||||
|
`viewGetData` SQL is byte-identical for depth 0/1/2/undefined. It always emits
|
||||||
|
the full `GROUP BY ROLLUP(...)`.
|
||||||
|
- The imperative `view.set_depth()` emits a `ViewSetDepthReq` the virtual server
|
||||||
|
does not handle → `Abort(): Unhandled request`.
|
||||||
|
- `ViewConfig` has **no field to express per-node expansion state**, so
|
||||||
|
expand/collapse cannot be communicated to the SQL model at all. The server
|
||||||
|
returns a static full rollup; the grid cannot reflow it. (Per-node +/- fires
|
||||||
|
the SQL round trips but produces no visual change.)
|
||||||
|
|
||||||
|
What makes expand/collapse/depth work *today* is Perspective's own WASM view
|
||||||
|
engine computing the tree. The virtual server **replaces** that engine with the
|
||||||
|
SQL model, which does not implement dynamic expansion.
|
||||||
|
|
||||||
|
**This limitation is not Option-C-specific.** Options **B, C, and D** all use the
|
||||||
|
same `DuckDBHandler` / `GenericSQLVirtualServerModel`. Any DuckDB-backed virtual
|
||||||
|
server therefore inherits the same loss of interactive tree expand/collapse/depth.
|
||||||
|
**Only Option A** keeps rows in a native Perspective `Table`, preserving full
|
||||||
|
interactivity.
|
||||||
|
|
||||||
|
**Implementation gotchas worth recording:**
|
||||||
|
|
||||||
|
- The SQL generator must be constructed from the *initialized* viewer WASM module
|
||||||
|
(`customElements.get('perspective-viewer').__wasm_module__`), not the bare
|
||||||
|
`@perspective-dev/client` default export — the latter's wasm glue is undefined
|
||||||
|
and `new perspective.GenericSQLVirtualServerModel()` throws.
|
||||||
|
- The handler must implement **both** `viewColumnSize` (→ `COUNT(*) FROM (DESCRIBE v)`,
|
||||||
|
column count) **and** `viewSize` (→ `COUNT(*) FROM v`, row count). The reference
|
||||||
|
`duckdb.ts` only shows `viewColumnSize`; omitting `viewSize` breaks row-count
|
||||||
|
rendering.
|
||||||
|
- DuckDB Arrow IPC output needs `INSTALL arrow FROM community; LOAD arrow;`
|
||||||
|
(the `to_arrow_ipc` table function is not built in to 1.x).
|
||||||
|
- DuckDB reads the pg forecast tables directly via `postgres_scanner`
|
||||||
|
(`ATTACH ... TYPE postgres, READ_ONLY`); the spike materializes a native copy so
|
||||||
|
aggregations are pure-columnar (writes would require a re-materialize/refresh).
|
||||||
|
|
||||||
|
**Conclusion.** The latency case for server-side DuckDB is strong, but the generic
|
||||||
|
SQL virtual server cannot support this app's expandable pivot UX. Choosing any
|
||||||
|
B/C/D path means either (a) writing a custom virtual server that implements
|
||||||
|
depth/expansion in SQL (rebuilding what the generic model omits), or (b) accepting
|
||||||
|
flat/fully-rolled-up views with no interactive drill-down. To keep the current UX,
|
||||||
|
**Option A** (optimize the encode/transport, keep the native Perspective table) is
|
||||||
|
the path.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision — display-grain pre-aggregation (2026-06-18)
|
||||||
|
|
||||||
|
The spike showed the real lever isn't the *transport*, it's the *grain*: the
|
||||||
|
browser pivots at a coarse display grain (e.g. `rep × customer × month`) but we
|
||||||
|
ship raw transaction rows. Pre-aggregating to that grain server-side (DuckDB or
|
||||||
|
plain pg `GROUP BY`) collapses **534,902 → 4,642 rows** (≈115×) on `osm_stack`,
|
||||||
|
and adding more dimensions barely moves it (4,642 → 4,743 with director +
|
||||||
|
channel) because the raw detail (part/plant/day/currency…) all rolls up.
|
||||||
|
|
||||||
|
This is the chosen direction. It is essentially "Option A done right" — reduce
|
||||||
|
**rows** by aggregating to grain, not just trim columns — and it keeps the
|
||||||
|
**native** Perspective engine, so expand/collapse/depth/sort/filter keep working
|
||||||
|
(the capability B/C/D sacrifice). The DB does the heavy aggregation (~25 ms);
|
||||||
|
the browser receives a few thousand native rows.
|
||||||
|
|
||||||
|
**Why it beats the alternatives here:**
|
||||||
|
|
||||||
|
- vs. current: ~250 MB / ~2 min → a few hundred KB / sub-second; render instant.
|
||||||
|
- vs. B/C/D (DuckDB virtual server): keeps interactive drill-down; no per-drag
|
||||||
|
round trips; no custom SQL generator to own.
|
||||||
|
- The undo "full reload" wart disappears — a re-pull is now ~25 ms.
|
||||||
|
|
||||||
|
**Trade-offs accepted:** additive measures only (sums exact; averages = sum+count;
|
||||||
|
distinct-counts not pre-aggregatable — fine for forecast); a view only supports
|
||||||
|
the dimensions in its grain (handled by re-fetching at a new grain when the user
|
||||||
|
changes group/split/filter, or by choosing a grain that covers the pivot set
|
||||||
|
while avoiding high-cardinality dims like `part`).
|
||||||
|
|
||||||
|
**This is a port of the prior Excel model.** The forecasting process previously
|
||||||
|
ran in Excel: pre-aggregated rows were sent to a data tab, pivot tables cut them
|
||||||
|
locally any way the user wanted, double-clicking a pivot cell let VBA capture the
|
||||||
|
slice and present an adjustment UI, and incremental rows were built in the DB,
|
||||||
|
**appended** to the data tab, and the pivot cache refreshed to include them. That
|
||||||
|
maps 1:1 onto native Perspective (pivot cache = the WASM view) — which is *why*
|
||||||
|
the native engine matters and the virtual server was the wrong fit: Excel never
|
||||||
|
re-queried the DB on every pivot move.
|
||||||
|
|
||||||
|
**Write/read model** (append deltas, let the view sum — the pivot-cache pattern;
|
||||||
|
single source of truth = pg raw rows):
|
||||||
|
|
||||||
|
- **Initial load** — `GET /api/versions/:id/agg`, `GROUP BY` grain × `pf_iter` ×
|
||||||
|
`pf_logid` → Arrow → native table indexed by a synthetic `pf_gkey`.
|
||||||
|
- **Write (scale/recode/clone)** — operation's final CTE returns just the new
|
||||||
|
`pf_logid`'s rows aggregated to grain → `table.update()` **appends** them; the
|
||||||
|
view re-sums. No bucket-total recomputation (keying on `pf_logid` keeps each
|
||||||
|
op's contribution a distinct row, so appends accumulate).
|
||||||
|
- **Delete (undo)** — `table.remove()` that logid's `pf_gkey`s; the view re-sums.
|
||||||
|
No re-aggregation, no emptied-bucket handling. (Or re-pull `/agg`, now cheap.)
|
||||||
|
|
||||||
|
**Concrete design** (schema flag `col_meta.in_grain`, the `/agg` endpoint, the
|
||||||
|
synthetic `pf_gkey` index, and the operation/undo SQL templates) is specified in
|
||||||
|
`pf_spec.md` → §Display-grain pre-aggregation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Two candidate designs for server-side aggregation (open — 2026-06-18)
|
||||||
|
|
||||||
|
Both keep **Perspective as the client** and **DuckDB/pg as the aggregation
|
||||||
|
engine** over raw pg rows (the source of truth), and both rely on additive
|
||||||
|
measures (sums; avg = sum+count; no distinct-count). The target experience for
|
||||||
|
both is "Excel PivotTable against a Power BI / SSAS tabular model": a good-looking
|
||||||
|
pivot driven by a fast columnar engine. They differ in **where the rollup happens**
|
||||||
|
and therefore **what crosses the wire** — which trades off against each other.
|
||||||
|
|
||||||
|
### Path A — Live server aggregation (virtual server)
|
||||||
|
|
||||||
|
- **Reads:** Perspective runs as a virtual server; each interaction →
|
||||||
|
SQL → DuckDB → just the viewport cells. Requires a **custom depth-aware SQL
|
||||||
|
generator** replacing `GenericSQLVirtualServerModel` (honor `group_by_depth`,
|
||||||
|
reproduce the `__ROW_PATH__`/`__GROUPING_ID__` contract, and eventually
|
||||||
|
`split_by`'s dynamic pivot + expression columns).
|
||||||
|
- **Writes/undo:** write raw rows to pg; the next query reflects them (re-query /
|
||||||
|
invalidate) — no client-side row bookkeeping.
|
||||||
|
- **Pros:** unbounded dataset — only the visible viewport ever ships; always live;
|
||||||
|
the truest "any cut, any size."
|
||||||
|
- **Cons:** the generator is a real component to build and own; **per-node**
|
||||||
|
expand/collapse is a hard frontier (the protocol delivers only a global depth to
|
||||||
|
the JS handler, not per-node expansion state — that needs a Rust/protocol fork);
|
||||||
|
per-interaction latency (~12 ms + RTT, measured in the spike).
|
||||||
|
- **Excel analog:** live SSAS / Power BI **MDX** connection.
|
||||||
|
|
||||||
|
### Path B — Pre-aggregated extract per cut (native table)
|
||||||
|
|
||||||
|
- **Reads:** the server ships a flat `GROUP BY` to the **current pivot's field
|
||||||
|
grain**; the result loads into a **native** Perspective table, and Perspective's
|
||||||
|
own engine does all rollup/expand/collapse/depth/sort **locally, with zero
|
||||||
|
calls**. A server round trip happens **only when the field set changes**
|
||||||
|
(add/remove a dimension, or filter on a non-grain dimension) — not on
|
||||||
|
expand/collapse/sort or moving a field between rows and columns.
|
||||||
|
- **Writes/undo:** append grain-aggregated delta rows keyed by `pf_logid`; the view
|
||||||
|
sums them. Undo removes that logid's rows. (The append model — concrete design in
|
||||||
|
`pf_spec.md` → §Display-grain pre-aggregation.)
|
||||||
|
- **Pros:** the server SQL is trivial and fully owned (no tree contract to
|
||||||
|
reproduce); zero-latency native interaction; native look/feel; payload is small
|
||||||
|
(only the pivot's columns, only distinct grain rows).
|
||||||
|
- **Cons:** ships the **entire current-grain leaf set**, not just the viewport — so
|
||||||
|
a high-cardinality cut (anything × `part`-like) can be large and eventually hit
|
||||||
|
Perspective's ~1–2M-row WASM ceiling; pays a re-fetch when the field set changes.
|
||||||
|
- **Excel analog:** Power Query **extract → pivot cache**.
|
||||||
|
|
||||||
|
### Comparison
|
||||||
|
|
||||||
|
| | Path A — live virtual server | Path B — pre-agg extract |
|
||||||
|
|---|---|---|
|
||||||
|
| Where rollup happens | server, per interaction | client (native), per cut |
|
||||||
|
| Crosses the wire | visible viewport only | whole current-grain leaf set (pivot cols only) |
|
||||||
|
| Expand/collapse/sort | server round trip each | local, zero calls |
|
||||||
|
| Server-side code | custom SQL generator (large) | flat `GROUP BY` (trivial) |
|
||||||
|
| Dataset ceiling | none (viewport only) | ~1–2M rows per cut (WASM) |
|
||||||
|
| Per-node expand | needs protocol/Rust fork | native (free) |
|
||||||
|
| Interaction latency | ~12 ms + RTT | 0 (local); re-fetch only on field-set change |
|
||||||
|
|
||||||
|
### The deciding question
|
||||||
|
|
||||||
|
It comes down to one thing: **do real cuts ever produce a leaf grain too large for
|
||||||
|
the browser?**
|
||||||
|
|
||||||
|
- If practical cuts stay under ~1M leaf rows → **Path B** — dramatically less code,
|
||||||
|
all of it yours, native feel, no generator to maintain.
|
||||||
|
- If you need truly unbounded any-cut-any-size → **Path A** — at the cost of
|
||||||
|
building/owning the generator and accepting the per-node-expand frontier.
|
||||||
|
|
||||||
|
**Hybrid (future option):** default to B; when a requested grain's distinct count
|
||||||
|
exceeds a threshold, fall back to A (virtual server) for that cut. Best of both,
|
||||||
|
more moving parts. Worth noting but not for a first build.
|
||||||
|
|
||||||
|
Next step toward deciding: map which dimensions are safe in a Path-B grain vs.
|
||||||
|
which are high-cardinality landmines (`part`, day-level dates, etc.) on the real
|
||||||
|
data — that tells us where B's ceiling actually bites.
|
||||||
|
|||||||
124
pf_spec.md
124
pf_spec.md
@ -714,6 +714,129 @@ DELETE FROM pf.log WHERE id = {{logid}};
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Display-grain pre-aggregation (planned)
|
||||||
|
|
||||||
|
**Status:** designed, not yet built. This is the concrete design for **Path B**
|
||||||
|
(pre-aggregated extract → native Perspective table) of two candidate designs;
|
||||||
|
rationale, the Path A alternative (live virtual-server aggregation), the spike
|
||||||
|
evidence, and the A-vs-B trade-off live in `pf_perspective_options.md`
|
||||||
|
(§Two candidate designs, §Spike findings).
|
||||||
|
|
||||||
|
**Problem it solves.** The current transport ships every raw forecast row to the
|
||||||
|
browser (≈535k rows / ~250 MB / ~2 min on `osm_stack`). Perspective then pivots
|
||||||
|
them in WASM. Pre-aggregating server-side to the grain the pivot actually
|
||||||
|
displays collapses the payload dramatically — measured 534,902 → **4,642 rows**
|
||||||
|
at `rep × customer × month` (≈115×) — while keeping Perspective's **native**
|
||||||
|
engine, so expand/collapse/depth/sort/filter all keep working (unlike the DuckDB
|
||||||
|
virtual-server path, which loses them).
|
||||||
|
|
||||||
|
### Grain definition — `pf.col_meta.in_grain`
|
||||||
|
|
||||||
|
Add a boolean `in_grain` to `pf.col_meta`. At **Generate SQL** time the flagged
|
||||||
|
columns (plus `pf_iter`, always) define the display grain; `value`/`units`
|
||||||
|
columns are the additive measures summed to that grain. The grain is baked into
|
||||||
|
the stored `pf.sql` templates, so initial load and operations agree on it.
|
||||||
|
|
||||||
|
- A date column enters the grain at **period** resolution (via `dim_period_col`,
|
||||||
|
e.g. month `sdat`), not raw date.
|
||||||
|
- **Constraint — additive measures only.** Sums (sales, qty, units) are exact.
|
||||||
|
Averages must ship as `sum` + `count` and be derived; `count(DISTINCT …)`
|
||||||
|
cannot be pre-aggregated. Forecast measures are sums, so this holds.
|
||||||
|
- Dimensions **not** in the grain are unavailable for pivot/filter on that view.
|
||||||
|
Either include every dimension users pivot on (cardinality permitting — a
|
||||||
|
high-cardinality dim like `part` explodes the grain back toward raw), or
|
||||||
|
re-fetch at a new grain when the user changes `group_by`/`split_by`/`filter`
|
||||||
|
(each change = one ~25 ms aggregate returning a small native dataset).
|
||||||
|
|
||||||
|
### Initial load — `GET /api/versions/:id/agg`
|
||||||
|
|
||||||
|
Replaces the raw `/data` stream for grain-based versions. Aggregates the forecast
|
||||||
|
table to the stored grain and returns Arrow IPC:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
{{grain_cols}}
|
||||||
|
,pf_iter
|
||||||
|
,pf_logid
|
||||||
|
,{{grain_key}} AS pf_gkey -- synthetic index, see below
|
||||||
|
,SUM({{value_col}}) AS {{value_col}}
|
||||||
|
,SUM({{units_col}}) AS {{units_col}}
|
||||||
|
FROM {{fc_table}}
|
||||||
|
GROUP BY {{grain_cols}}, pf_iter, pf_logid
|
||||||
|
```
|
||||||
|
|
||||||
|
Client loads the result into a native `worker.table(buffer, { index: 'pf_gkey' })`.
|
||||||
|
The Perspective **view** sums measures across these rows for whatever the user
|
||||||
|
pivots on — exactly as an Excel pivot cache sums the data tab.
|
||||||
|
|
||||||
|
**Model — append deltas, let the view sum (the Excel pivot-cache pattern).** This
|
||||||
|
is a direct port of the prior Excel workflow: pre-aggregated rows go to the table,
|
||||||
|
the pivot cuts them locally, and each adjustment is *appended* — never a
|
||||||
|
recomputed total. So rows are aggregated to grain **per `pf_logid`** (one row per
|
||||||
|
grain × `pf_iter` × originating log entry), and the view sums them. This is also
|
||||||
|
the smallest change from today's code, which already appends operation results via
|
||||||
|
`table.update()` and lets the view sum — we just feed pre-aggregated rows.
|
||||||
|
|
||||||
|
**Synthetic key.** Perspective's index is a single column, so emit
|
||||||
|
`pf_gkey = {{grain_cols}} || pf_iter || pf_logid` (concatenated). Including
|
||||||
|
`pf_logid` keeps each operation's contribution a **distinct** row, so appends
|
||||||
|
accumulate (rather than replacing a bucket) and a delete can remove exactly that
|
||||||
|
operation's rows.
|
||||||
|
|
||||||
|
### Write path (scale / recode / clone) — append the new log entry's rows
|
||||||
|
|
||||||
|
Operations INSERT raw rows into `{{fc_table}}` under a new `pf_logid` as today; the
|
||||||
|
final CTE returns just **that log entry's rows aggregated to grain**:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
WITH
|
||||||
|
ins AS (
|
||||||
|
INSERT INTO {{fc_table}} ( … ) SELECT … RETURNING {{grain_cols}}, pf_iter, pf_logid
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
{{grain_cols}}
|
||||||
|
,pf_iter
|
||||||
|
,pf_logid
|
||||||
|
,{{grain_key}} AS pf_gkey
|
||||||
|
,SUM({{value_col}}) AS {{value_col}}
|
||||||
|
,SUM({{units_col}}) AS {{units_col}}
|
||||||
|
FROM ins
|
||||||
|
GROUP BY {{grain_cols}}, pf_iter, pf_logid
|
||||||
|
```
|
||||||
|
|
||||||
|
Client applies `table.update(rows)` — these are new `pf_gkey`s, so they append and
|
||||||
|
the view re-sums. No bucket-total recomputation. (Aggregating `FROM ins` is safe
|
||||||
|
because each row carries the new, unique `pf_logid`.)
|
||||||
|
|
||||||
|
### Delete path (undo) — remove that log entry's rows
|
||||||
|
|
||||||
|
A logid's rows are uniquely keyed, so undo just removes them and lets the view
|
||||||
|
re-sum — no re-aggregation, no emptied-bucket handling, no snapshot caveat:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
DELETE FROM {{fc_table}} WHERE pf_logid = {{logid}}
|
||||||
|
RETURNING DISTINCT {{grain_cols}}, pf_iter, pf_logid; -- → pf_gkeys to remove
|
||||||
|
DELETE FROM pf.log WHERE id = {{logid}};
|
||||||
|
```
|
||||||
|
|
||||||
|
Client applies `table.remove(pf_gkeys)`. (A wholesale re-pull of `/agg` is an even
|
||||||
|
simpler fallback and is now cheap — ~25 ms.)
|
||||||
|
|
||||||
|
### What this preserves / changes
|
||||||
|
|
||||||
|
- **Preserves:** native Perspective interactivity, layout persistence, and the
|
||||||
|
slice-click → operation flow (a clicked grain cell still maps to a `WHERE` on
|
||||||
|
raw rows). pg raw rows remain the single source of truth.
|
||||||
|
- **Changes:** operation responses return grain-aggregated rows for the new
|
||||||
|
`pf_logid` (not raw `RETURNING *`); `/data` is superseded by `/agg` for grain
|
||||||
|
versions; the table is indexed by `pf_gkey` instead of `pf_id`; undo becomes a
|
||||||
|
targeted `table.remove()` and the "full reload" wart disappears.
|
||||||
|
- **Lineage:** this is a port of the prior Excel model — pre-agg rows → pivot
|
||||||
|
cache → double-click slice → VBA adjustment UI → append incremental rows →
|
||||||
|
refresh cache. The append-and-sum write/undo model mirrors that directly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Admin Setup Flow (end-to-end)
|
## Admin Setup Flow (end-to-end)
|
||||||
|
|
||||||
1. Open **Sources** view → browse DB tables → register source table
|
1. Open **Sources** view → browse DB tables → register source table
|
||||||
@ -741,6 +864,7 @@ DELETE FROM pf.log WHERE id = {{logid}};
|
|||||||
|
|
||||||
## Open Questions / Future Scope
|
## Open Questions / Future Scope
|
||||||
|
|
||||||
|
- **Display-grain pre-aggregation** — ship the pivot at its display grain instead of raw rows; see §Display-grain pre-aggregation above and `pf_perspective_options.md`. This is the chosen direction for the load-time + interactivity problem.
|
||||||
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
|
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
|
||||||
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
|
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
|
||||||
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
|
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
|
||||||
|
|||||||
112
pf_ux_mockup.md
112
pf_ux_mockup.md
@ -1,112 +0,0 @@
|
|||||||
# Pivot Forecast — UX Mockup
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Pivot Forecast │
|
|
||||||
│ ① Setup ② Baseline ③ Forecast ◀ (default landing) │
|
|
||||||
└─────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
① SETUP
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
|
|
||||||
┌──── All Tables ──────────────┐ ┌──── Registered Sources ─────────┐
|
|
||||||
│ schema table rows │ │ │
|
|
||||||
│ ────── ────────── ────── │ │ sales_orders ✓ SQL ready │
|
|
||||||
│ public sales_orders 48,291 │◀─│ invoices ✓ SQL ready │
|
|
||||||
│ public invoices 12,004 │ │ + Register table │
|
|
||||||
│ public products 891 │ └──────────────────────────────────┘
|
|
||||||
│ rpt summary_mv 3,442 │
|
|
||||||
└──────────────────────────────┘ ┌──── Col Meta: sales_orders ─────┐
|
|
||||||
│ column role key label│
|
|
||||||
│ ────────── ──────── ─── ─── │
|
|
||||||
│ customer dimension ✓ │
|
|
||||||
│ channel dimension ✓ │
|
|
||||||
│ part dimension │
|
|
||||||
│ geography dimension │
|
|
||||||
│ order_date date │
|
|
||||||
│ ship_date filter │
|
|
||||||
│ status filter │
|
|
||||||
│ units units │
|
|
||||||
│ revenue value │
|
|
||||||
│ internal_id ignore │
|
|
||||||
│ │
|
|
||||||
│ [Generate SQL ▶] │
|
|
||||||
└──────────────────────────────────┘
|
|
||||||
|
|
||||||
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
② BASELINE
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
|
|
||||||
Source [sales_orders ▾] Version [FY2026 Plan ▾] [+ New version]
|
|
||||||
|
|
||||||
┌──── Segments ──────────────────────────────────────────────────────┐
|
|
||||||
│ # description rows by date │
|
|
||||||
│ ─ ──────────────────────────── ────── ────── ────────────── │
|
|
||||||
│ 1 FY25 actuals +1yr 41,204 paul Apr 24 │
|
|
||||||
│ 2 Open orders 3,109 paul Apr 24 [Undo] │
|
|
||||||
│ │
|
|
||||||
│ Total baseline rows: 44,313 [Clear all baseline] │
|
|
||||||
└────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
┌──── Add Segment ────────────────────────────────────────────────────┐
|
|
||||||
│ │
|
|
||||||
│ Description [ ] │
|
|
||||||
│ │
|
|
||||||
│ Filters [+ Add filter] │
|
|
||||||
│ ┌─────────────────┬──────────┬─────────────────────┬───┐ │
|
|
||||||
│ │ order_date │ BETWEEN │ 2025-01-01 2025-12-31│ x │ │
|
|
||||||
│ └─────────────────┴──────────┴─────────────────────┴───┘ │
|
|
||||||
│ │
|
|
||||||
│ Date offset [1] yr [0] mo │
|
|
||||||
│ │
|
|
||||||
│ ·───────────────────────────· source │
|
|
||||||
│ Jan 2025 Dec 2025 │
|
|
||||||
│ ·───────────────────────────· projected (+1 yr) │
|
|
||||||
│ Jan 2026 Dec 2026 │
|
|
||||||
│ │
|
|
||||||
│ Note [ ] [Load Segment] │
|
|
||||||
└────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
┌──── Reference (optional) ──────────────────────────────────────────┐
|
|
||||||
│ Load prior-period rows for comparison in the pivot │
|
|
||||||
│ Date range [2024-01-01] to [2024-12-31] │
|
|
||||||
│ Note [ ] [Load Ref] │
|
|
||||||
└────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
③ FORECAST source: sales_orders
|
|
||||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
||||||
|
|
||||||
Version [FY2026 Plan ▾] [Refresh] [Save layout] [Reset layout]
|
|
||||||
|
|
||||||
┌──── Pivot ───────────────────────────────┐ ┌──── Operations ───────┐
|
|
||||||
│ │ │ │
|
|
||||||
│ (Perspective viewer) │ │ Slice │
|
|
||||||
│ │ │ channel = WHS │
|
|
||||||
│ channel │ Jan 2026 │ Feb 2026 │ ... │ │ geo = WEST │
|
|
||||||
│ ──────────┼──────────┼──────────┼─── │ │ │
|
|
||||||
│ DIR │ 412,000 │ 388,000 │ │ │ [Scale][Recode] │
|
|
||||||
│ WHS ◀ │ 290,000 │ 310,000 │ │ │ [Clone] │
|
|
||||||
│ ──────── │ │ │ │ │ ─────────────────── │
|
|
||||||
│ Total │ 702,000 │ 698,000 │ │ │ Value incr [ ] │
|
|
||||||
│ │ │ Units incr [ ] │
|
|
||||||
│ │ │ Pct? [ ] │
|
|
||||||
│ │ │ │
|
|
||||||
│ │ │ Note [ ] │
|
|
||||||
│ │ │ │
|
|
||||||
│ │ │ [Submit] │
|
|
||||||
└──────────────────────────────────────────┘ └───────────────────────┘
|
|
||||||
|
|
||||||
▼ Change log (12 entries)
|
|
||||||
┌────┬───────────┬──────────┬─────────────────────────┬────────────┐
|
|
||||||
│ id │ operation │ by │ slice │ │
|
|
||||||
│ ── │ ───────── │ ──────── │ ───────────────────── ─ │ │
|
|
||||||
│ 12 │ scale │ paul │ channel=WHS geo=WEST │ [Undo] │
|
|
||||||
│ 11 │ recode │ paul │ part=OLD-SKU │ [Undo] │
|
|
||||||
│ 10 │ scale │ paul │ channel=DIR │ [Undo] │
|
|
||||||
└────┴───────────┴──────────┴─────────────────────────┴────────────┘
|
|
||||||
```
|
|
||||||
Loading…
Reference in New Issue
Block a user