Merge remote-tracking branch 'origin/operations-panel'
This commit is contained in:
commit
41349f2dee
26
CLAUDE.md
26
CLAUDE.md
@ -7,16 +7,15 @@ A web app for building named forecast scenarios against any PostgreSQL table. Th
|
||||
Full spec: `pf_spec.md`
|
||||
Data transport architecture options: `pf_perspective_options.md`
|
||||
UX mockup: `pf_ux_mockup.md`
|
||||
Open work: `todo.md`
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
- **Backend:** Node.js / Express (`server.js`), runs on port 3010
|
||||
- **Backend:** Node.js / Express (`server.js`)
|
||||
- **Database:** PostgreSQL — isolated `pf` schema
|
||||
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
|
||||
- **Pivot:** [Perspective](https://perspective.finos.org/) 4.4.0 loaded from CDN at runtime
|
||||
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
|
||||
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
|
||||
|
||||
---
|
||||
@ -51,11 +50,12 @@ ui/src/
|
||||
## Database schema (`pf`)
|
||||
|
||||
- **`pf.source`** — registered source tables
|
||||
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses
|
||||
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
|
||||
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
|
||||
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`iter = baseline|scale|recode|clone`) and reference rows (`iter = reference`)
|
||||
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
|
||||
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
|
||||
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
|
||||
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
|
||||
|
||||
### Key token substitution tokens
|
||||
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
|
||||
@ -97,10 +97,20 @@ All three operations follow the same structure: insert a `pf.log` row in a CTE,
|
||||
|
||||
---
|
||||
|
||||
## Known issues / active work (see todo.md for detail)
|
||||
## Light / dark mode
|
||||
|
||||
- Operation panel (Scale/Recode/Clone) wiring to API is a stub — needs completion
|
||||
- Status bar is hardcoded — needs to reflect actual selected source/version
|
||||
Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
|
||||
|
||||
- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
|
||||
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
|
||||
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
|
||||
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
|
||||
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
|
||||
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
|
||||
|
||||
## Known issues / active work
|
||||
|
||||
- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
|
||||
- Load progress bar is jittery — needs throttle (~10 updates/sec)
|
||||
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
|
||||
- Source/version selection doesn't persist across page reload
|
||||
|
||||
199
PERSPECTIVE.md
Normal file
199
PERSPECTIVE.md
Normal file
@ -0,0 +1,199 @@
|
||||
# Perspective — configuration & deployment reference
|
||||
|
||||
Canonical guide for how Perspective should be configured, fed, and shipped across our
|
||||
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
|
||||
but made different early choices; this doc defines the target state and the rationale, so
|
||||
the two converge instead of drifting.
|
||||
|
||||
> **Distribution note:** we use the **`@perspective-dev/*`** packages
|
||||
> (repo: <https://github.com/perspective-dev/perspective>, home:
|
||||
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
|
||||
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
|
||||
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
|
||||
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
|
||||
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
|
||||
> scopes.
|
||||
|
||||
---
|
||||
|
||||
## Core principle: Perspective is one locked unit
|
||||
|
||||
These four things must move together and be pinned together. Bumping one without the
|
||||
others is the source of nearly every Perspective bug we've hit:
|
||||
|
||||
1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
|
||||
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
|
||||
3. **Data format** — Arrow IPC vs JSON rows
|
||||
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
|
||||
format the client WASM understands
|
||||
|
||||
Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
|
||||
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Loading — npm `/inline`, pinned exact (not CDN)
|
||||
|
||||
```js
|
||||
import perspective from '@perspective-dev/client/inline'
|
||||
import '@perspective-dev/viewer/inline'
|
||||
import '@perspective-dev/viewer-datagrid'
|
||||
import '@perspective-dev/viewer-d3fc'
|
||||
import '@perspective-dev/viewer/themes'
|
||||
```
|
||||
|
||||
- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
|
||||
fetch, works offline / behind a firewall, reproducible from the lockfile.**
|
||||
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
|
||||
one-line version bumps) but in production it means: app breaks if the CDN is
|
||||
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
|
||||
pull executable WASM from a third party on every load. (pf_app currently does this in
|
||||
`ui/src/views/Forecast.jsx` — migrating off it is the main open item.)
|
||||
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
|
||||
`<link>` in `index.html` — so it's bundled and versioned too.
|
||||
|
||||
---
|
||||
|
||||
## 2. Version policy — a real trilemma (read carefully)
|
||||
|
||||
The version choice is constrained by two hard facts about the `@perspective-dev` packages
|
||||
**(verified against installed metadata, 2026-06)**:
|
||||
|
||||
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
|
||||
Treemap / Heatmap / etc.) live only in this package.
|
||||
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
|
||||
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
|
||||
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
|
||||
|
||||
So you can have at most **two** of these three:
|
||||
|
||||
| Want | Requires |
|
||||
|---|---|
|
||||
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
|
||||
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
|
||||
| d3fc chart plugins | **4.4.1** viewer-d3fc |
|
||||
|
||||
There is **no** version where all three hold. Pick by what the app needs:
|
||||
|
||||
- **Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
|
||||
`^4.4.1` viewer-d3fc`. This is a deliberate, necessary mixed-version pair, *not* an
|
||||
accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate
|
||||
bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the
|
||||
build breaks (`"./inline" is not exported`).
|
||||
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
|
||||
**4.4.1 exact**. Charts work; you give up `/inline` bundling.
|
||||
|
||||
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
|
||||
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x (then a
|
||||
fully-coherent inline-capable 4.5.x suite becomes possible).
|
||||
|
||||
---
|
||||
|
||||
## 3. Data delivery — match the format to the workload
|
||||
|
||||
| Workload | Format | Why |
|
||||
|---|---|---|
|
||||
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
|
||||
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |
|
||||
|
||||
### Arrow constraints (read before choosing it)
|
||||
|
||||
If you deliver Arrow, three pieces are coupled and must stay aligned:
|
||||
|
||||
- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
|
||||
must coerce them to JS numbers before encoding, or `apache-arrow` infers
|
||||
`Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
|
||||
1700).
|
||||
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
|
||||
Perspective WASM crashes on dictionary-replacement messages. The server must
|
||||
accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
|
||||
'stream')`). Consequence: the client "stream" is just a chunked *download* of one
|
||||
batch — nothing renders progressively, and the server holds the full result set in
|
||||
memory per request.
|
||||
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
|
||||
`package.json` and treat it as part of the locked unit (principle above). Note the
|
||||
`@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
|
||||
pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
|
||||
the server toward the arrow major the client was built against, or at minimum make
|
||||
Arrow ingestion (smoke test §7) the gate on any arrow bump.
|
||||
|
||||
JSON avoids all three but pays in payload size and parse cost, and pushes type handling
|
||||
to client-side heuristics — acceptable only at small scale.
|
||||
|
||||
---
|
||||
|
||||
## 4. Theming
|
||||
|
||||
- One toggle drives both app CSS and the viewer:
|
||||
`viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
|
||||
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
|
||||
re-themes when the toggle fires (not just on mount).
|
||||
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
|
||||
viewer theme name is the only Perspective-specific piece.
|
||||
|
||||
---
|
||||
|
||||
## 5. Layout persistence
|
||||
|
||||
- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
|
||||
`localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
|
||||
- **Guard restores against schema drift.** Before restoring, filter the saved config's
|
||||
`columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
|
||||
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
|
||||
the reference implementation; a stale layout referencing a dropped column otherwise
|
||||
throws on restore.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build & deploy (target)
|
||||
|
||||
- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
|
||||
`outDir: '../public/app'`). The API server serves it statically (`express.static`).
|
||||
- **Process:** run the Node API under **systemd** with `Restart=always` and an
|
||||
`EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
|
||||
dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
|
||||
automation yet** and should adopt the same pattern.
|
||||
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
|
||||
systemd unit); later runs rebuild the UI and restart the service.
|
||||
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
|
||||
unit — never commit it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Upgrade checklist / smoke test
|
||||
|
||||
Run this whenever bumping **any** Perspective package or `apache-arrow`:
|
||||
|
||||
1. Confirm `viewer-d3fc` publishes the target version
|
||||
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
|
||||
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
|
||||
commit the lockfile.
|
||||
3. `vite build` — no unresolved imports.
|
||||
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
|
||||
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
|
||||
string/dictionary.
|
||||
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
|
||||
6. Toggle dark/light; confirm the viewer re-themes.
|
||||
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
|
||||
cleaned restore doesn't throw.
|
||||
|
||||
---
|
||||
|
||||
## Per-project state (2026-06)
|
||||
|
||||
| | pf_app | dataflow | Target |
|
||||
|---|---|---|---|
|
||||
| Loader | CDN (runtime) | npm `/inline` | **npm `/inline`** |
|
||||
| Version | 4.4.0 (CDN URLs) | 4.5.1 viewer/client + 4.4.1 d3fc | depends on loader (§2) |
|
||||
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
|
||||
| `apache-arrow` | `^21.1.0` (client built vs 17) | n/a | pin exact, match WASM |
|
||||
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
|
||||
|
||||
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
|
||||
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
|
||||
|
||||
**Open items:**
|
||||
- pf_app → move off CDN. Note this forces the §2 choice: going npm-`/inline` means
|
||||
4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and
|
||||
load via the `.` entry instead of `/inline`. Either way, pin + commit the lockfile, and
|
||||
add deploy automation (systemd + nginx + `deploy.sh`).
|
||||
@ -21,11 +21,10 @@ function generateSQL(source, colMeta) {
|
||||
const dateCol = colMeta.find(c => c.role === 'date')?.cname;
|
||||
|
||||
if (!valueCol) throw new Error('No value column defined in col_meta');
|
||||
if (!unitsCol) throw new Error('No units column defined in col_meta');
|
||||
if (!dateCol) throw new Error('No date column defined in col_meta');
|
||||
if (dims.length === 0) throw new Error('No dimension columns defined in col_meta');
|
||||
|
||||
const srcTable = `${source.schema}.${source.tname}`;
|
||||
const srcTable = `"${source.schema}"."${source.tname}"`;
|
||||
const dataCols = [...dims, dateCol, valueCol, unitsCol].filter(Boolean);
|
||||
const effectiveValue = dataCols.includes(valueCol) ? valueCol : null;
|
||||
const effectiveUnits = dataCols.includes(unitsCol) ? unitsCol : null;
|
||||
@ -33,6 +32,19 @@ function generateSQL(source, colMeta) {
|
||||
const selectData = dataCols.map(q).join(', ');
|
||||
const dimsJoined = dims.map(q).join(', ');
|
||||
|
||||
// dim_period JOIN support: if the date column is the is_key of a dim_group,
|
||||
// dimension siblings with dim_period_col set are derived from pf.dim_period
|
||||
// instead of being copied raw from the source on baseline/reference load.
|
||||
const dateKeyGroup = colMeta.find(c => c.role === 'date' && c.is_key && c.dim_group)?.dim_group;
|
||||
const dimPeriodMap = new Map(
|
||||
dateKeyGroup
|
||||
? colMeta
|
||||
.filter(c => c.role === 'dimension' && c.dim_group === dateKeyGroup && c.dim_period_col)
|
||||
.map(c => [c.cname, c.dim_period_col])
|
||||
: []
|
||||
);
|
||||
const hasDimPeriod = dimPeriodMap.size > 0;
|
||||
|
||||
return {
|
||||
get_data: buildGetData(),
|
||||
baseline: buildBaseline(),
|
||||
@ -47,10 +59,22 @@ function generateSQL(source, colMeta) {
|
||||
return `SELECT * FROM {{fc_table}}`;
|
||||
}
|
||||
|
||||
function buildLoadSelect(pfx) {
|
||||
// pfx: table alias prefix ('s.' when joining dim_period, '' otherwise)
|
||||
return dataCols.map(c => {
|
||||
if (c === dateCol) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
|
||||
if (dimPeriodMap.has(c)) return `dp.${q(dimPeriodMap.get(c))} AS ${q(c)}`;
|
||||
return `${pfx}${q(c)}`;
|
||||
}).join(',\n ');
|
||||
}
|
||||
|
||||
function buildFromClause() {
|
||||
if (!hasDimPeriod) return srcTable;
|
||||
return `${srcTable} s\n JOIN pf.dim_period dp`
|
||||
+ ` ON dp.drange @> (s.${q(dateCol)} + '{{date_offset}}'::interval)::date`;
|
||||
}
|
||||
|
||||
function buildBaseline() {
|
||||
const baselineSelect = dataCols.map(c =>
|
||||
c === dateCol ? `(${q(c)} + '{{date_offset}}'::interval)::date` : q(c)
|
||||
).join(', ');
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
@ -60,8 +84,10 @@ ilog AS (
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT ${baselineSelect}, 'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM ${srcTable}
|
||||
SELECT
|
||||
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
|
||||
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM ${buildFromClause()}
|
||||
WHERE {{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
@ -69,9 +95,6 @@ SELECT count(*) AS rows_affected FROM ins`.trim();
|
||||
}
|
||||
|
||||
function buildReference() {
|
||||
const referenceSelect = dataCols.map(c =>
|
||||
c === dateCol ? `(${q(c)} + '{{date_offset}}'::interval)::date` : q(c)
|
||||
).join(', ');
|
||||
return `
|
||||
WITH
|
||||
ilog AS (
|
||||
@ -81,8 +104,10 @@ ilog AS (
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT ${referenceSelect}, 'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM ${srcTable}
|
||||
SELECT
|
||||
${buildLoadSelect(hasDimPeriod ? 's.' : '')},
|
||||
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM ${buildFromClause()}
|
||||
WHERE {{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
@ -145,14 +170,14 @@ ilog AS (
|
||||
)
|
||||
,neg AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}, ${effectiveUnits ? `-${q(effectiveUnits)}` : '0'},
|
||||
SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
|
||||
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
RETURNING *
|
||||
)
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (${insertCols})
|
||||
SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}, ${effectiveUnits ? q(effectiveUnits) : '0'},
|
||||
SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}${effectiveUnits ? `, ${q(effectiveUnits)}` : ''},
|
||||
'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
RETURNING *
|
||||
@ -173,8 +198,7 @@ ilog AS (
|
||||
SELECT
|
||||
{{set_clause}},
|
||||
${q(dateCol)},
|
||||
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'},
|
||||
${effectiveUnits ? `round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : '0'},
|
||||
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}${effectiveUnits ? `,\n round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : ''},
|
||||
'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
|
||||
252
pf_spec.md
252
pf_spec.md
@ -24,13 +24,14 @@ Registered source tables available for forecasting.
|
||||
|
||||
```sql
|
||||
CREATE TABLE pf.source (
|
||||
id serial PRIMARY KEY,
|
||||
schema text NOT NULL,
|
||||
tname text NOT NULL,
|
||||
label text, -- friendly display name
|
||||
status text DEFAULT 'active', -- active | archived
|
||||
created_at timestamptz DEFAULT now(),
|
||||
created_by text,
|
||||
id serial PRIMARY KEY,
|
||||
schema text NOT NULL,
|
||||
tname text NOT NULL,
|
||||
label text, -- friendly display name
|
||||
status text DEFAULT 'active', -- active | archived
|
||||
default_layout jsonb, -- Perspective view config used as per-source default
|
||||
created_at timestamptz DEFAULT now(),
|
||||
created_by text,
|
||||
UNIQUE (schema, tname)
|
||||
);
|
||||
```
|
||||
@ -40,25 +41,40 @@ Column configuration for each registered source table. Determines how the app tr
|
||||
|
||||
```sql
|
||||
CREATE TABLE pf.col_meta (
|
||||
id serial PRIMARY KEY,
|
||||
source_id integer REFERENCES pf.source(id),
|
||||
cname text NOT NULL, -- column name in source table
|
||||
label text, -- friendly display name
|
||||
role text NOT NULL, -- 'dimension' | 'value' | 'units' | 'date' | 'ignore'
|
||||
is_key boolean DEFAULT false, -- true = part of natural key (used in WHERE slice)
|
||||
opos integer, -- ordinal position (for ordering)
|
||||
id serial PRIMARY KEY,
|
||||
source_id integer REFERENCES pf.source(id),
|
||||
cname text NOT NULL, -- column name in source table
|
||||
label text, -- friendly display name
|
||||
role text NOT NULL, -- 'dimension' | 'value' | 'units' | 'date' | 'filter' | 'ignore'
|
||||
is_key boolean DEFAULT false, -- true = part of natural key (used in WHERE slice)
|
||||
opos integer, -- ordinal position (for ordering)
|
||||
dim_group text, -- groups functionally dependent columns (see below)
|
||||
dim_period_col text, -- maps this dimension to a pf.dim_period column
|
||||
UNIQUE (source_id, cname)
|
||||
);
|
||||
```
|
||||
|
||||
**Roles:**
|
||||
- `dimension` — categorical field (customer, part, channel, rep, geography, etc.) — appears as pivot rows/cols, used in WHERE filters for operations
|
||||
- `value` — the money/revenue field to scale
|
||||
- `units` — the quantity field to scale
|
||||
- `date` — the primary date field; used for baseline/reference date range and stored in the forecast table
|
||||
- `value` — the money/revenue field to scale (**required** — SQL generation fails without it)
|
||||
- `units` — the quantity field to scale (**optional** — if absent, units columns are omitted from the forecast table and all SQL patterns)
|
||||
- `date` — the primary date field; used for baseline/reference date range and stored in the forecast table (**required**)
|
||||
- `filter` — columns available as filter conditions in the Baseline Workbench (e.g. order status, ship date, open flag); used in baseline WHERE clauses but **not stored** in the forecast table
|
||||
- `ignore` — exclude from forecast table entirely
|
||||
|
||||
**`dim_group`** — a free-text group name linking a `date` column to its derived dimension siblings. When the `date` column has `is_key = true` and a `dim_group` value, the SQL generator looks for `dimension` columns in the same group that also have a `dim_period_col` value. Those columns are sourced from `pf.dim_period` on baseline/reference load (via a JOIN on `drange @> date`) rather than copied raw from the source table. This allows fiscal year, quarter, and month columns to be stored in the forecast table with calendar-correct values even if those columns don't exist in the source.
|
||||
|
||||
**`dim_period_col`** — names the column in `pf.dim_period` to use as the value for this dimension on load. Only meaningful when the column is in a `dim_group` whose `date` key has `is_key = true`. Example: `cal_year`, `fisc_quarter`, `fisc_label`.
|
||||
|
||||
### `pf.dim_period`
|
||||
Calendar lookup table. One row per month from 2018-01-01 through 2035-12-01. Keyed on `sdat` (month start date). Used to derive fiscal/calendar period columns at baseline load time when `dim_group` / `dim_period_col` are configured on col_meta.
|
||||
|
||||
Populated by `setup_sql/gen_dim_period.sql` (safe to re-run; `ON CONFLICT DO NOTHING`). Fiscal year start month is configurable at the top of that script (default: June, i.e. fiscal month 1 = June).
|
||||
|
||||
Key columns: `sdat`, `edat`, `drange` (GiST-indexed daterange), `cal_year`, `cal_quarter`, `cal_month`, `cal_month_abbr`, `cal_month_name`, `cal_label`, `fisc_year`, `fisc_quarter`, `fisc_quarter_label`, `fisc_month`, `fisc_month_abbr`, `fisc_month_name`, `fisc_label`, `period_key`.
|
||||
|
||||
The baseline/reference SQL JOINs this table when `hasDimPeriod` is true: `JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`.
|
||||
|
||||
### `pf.version`
|
||||
Named forecast scenarios. One forecast table (`pf.fc_{tname}_{version_id}`) is created per version.
|
||||
|
||||
@ -102,31 +118,31 @@ CREATE TABLE pf.log (
|
||||
```
|
||||
|
||||
### `pf.fc_{tname}_{version_id}` (dynamic, one per version)
|
||||
Created when a version is created. Mirrors source table dimension/value/units/date columns plus forecast metadata. Contains both operational rows (`iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`iter = 'reference'`).
|
||||
Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`).
|
||||
|
||||
```sql
|
||||
-- Example: source table "sales", version id 3 → pf.fc_sales_3
|
||||
CREATE TABLE pf.fc_sales_3 (
|
||||
id bigserial PRIMARY KEY,
|
||||
id bigserial PRIMARY KEY,
|
||||
|
||||
-- mirrored from source (role = dimension | value | units | date only):
|
||||
customer text,
|
||||
channel text,
|
||||
part text,
|
||||
geography text,
|
||||
order_date date,
|
||||
units numeric,
|
||||
value numeric,
|
||||
customer text,
|
||||
channel text,
|
||||
part text,
|
||||
geography text,
|
||||
order_date date,
|
||||
value numeric,
|
||||
units numeric, -- omitted if no 'units' role in col_meta
|
||||
|
||||
-- forecast metadata:
|
||||
iter text, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
|
||||
logid bigint REFERENCES pf.log(id),
|
||||
pf_user text,
|
||||
created_at timestamptz DEFAULT now()
|
||||
pf_iter text, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
|
||||
pf_logid bigint REFERENCES pf.log(id),
|
||||
pf_user text,
|
||||
pf_created_at timestamptz DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
Note: no `version_id` column on the forecast table — it's implied by the table itself.
|
||||
Note: no `version_id` column on the forecast table — it's implied by the table itself. The `units` column is only present when a column with `role = 'units'` exists in col_meta.
|
||||
|
||||
### `pf.sql`
|
||||
Generated SQL stored per source and operation. Built once when col_meta is finalized, fetched at request time.
|
||||
@ -148,7 +164,7 @@ CREATE TABLE pf.sql (
|
||||
|-------|--------------|
|
||||
| `{{fc_table}}` | `pf.fc_{tname}_{version_id}` — derived at request time |
|
||||
| `{{where_clause}}` | built from `slice` JSON by `build_where()` in JS |
|
||||
| `{{exclude_clause}}` | built from `version.exclude_iters` — e.g. `AND iter NOT IN ('reference')` |
|
||||
| `{{exclude_clause}}` | built from `version.exclude_iters` — e.g. `AND pf_iter NOT IN ('reference')` |
|
||||
| `{{logid}}` | newly inserted `pf.log` id |
|
||||
| `{{pf_user}}` | from request body |
|
||||
| `{{date_from}}` / `{{date_to}}` | baseline/reference date range (source period) |
|
||||
@ -521,7 +537,11 @@ AG Grid list of log entries — user, timestamp, operation, slice, note, rows af
|
||||
|
||||
## Forecast SQL Patterns
|
||||
|
||||
Column names baked in at generation time. Tokens substituted at request time.
|
||||
Column names baked in at generation time. Tokens substituted at request time. Metadata columns are `pf_iter`, `pf_logid`, `pf_user`, `pf_created_at`.
|
||||
|
||||
**Units conditionality:** `{units_col}` appears in INSERT column lists and SELECT expressions only when a `units` role is configured in col_meta. The SQL generator omits it entirely otherwise — no placeholder column, no zero-fill.
|
||||
|
||||
**dim_period JOIN:** when any `dimension` column has `dim_period_col` set (and its group's `date` key has `is_key = true`), the FROM clause becomes `{schema}.{tname} s JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`. Those dimension columns are selected as `dp.{dim_period_col} AS {col}` instead of `s.{col}`.
|
||||
|
||||
### Baseline Load (one segment)
|
||||
|
||||
@ -531,18 +551,23 @@ WITH ilog AS (
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {value_col}, {units_col}, {date_col},
|
||||
iter, logid, pf_user, created_at
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {date_col}, {value_col} [, {units_col}],
|
||||
pf_iter, pf_logid, pf_user, pf_created_at
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols},
|
||||
({date_col} + '{{date_offset}}'::interval)::date,
|
||||
{value_col} [, {units_col}],
|
||||
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM
|
||||
{schema}.{tname} -- or with dim_period JOIN (see above)
|
||||
WHERE
|
||||
{{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols}, {value_col}, {units_col},
|
||||
({date_col} + '{{date_offset}}'::interval)::date,
|
||||
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM
|
||||
{schema}.{tname}
|
||||
WHERE
|
||||
{{filter_clause}}
|
||||
SELECT count(*) AS rows_affected FROM ins
|
||||
```
|
||||
|
||||
Baseline loads are **additive** — no DELETE before INSERT. Each segment appends independently.
|
||||
@ -557,7 +582,7 @@ Token details:
|
||||
|
||||
Two queries, run in a transaction:
|
||||
```sql
|
||||
DELETE FROM {{fc_table}} WHERE iter = 'baseline';
|
||||
DELETE FROM {{fc_table}} WHERE pf_iter = 'baseline';
|
||||
DELETE FROM pf.log WHERE version_id = {{version_id}} AND operation = 'baseline';
|
||||
```
|
||||
|
||||
@ -569,20 +594,24 @@ WITH ilog AS (
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {value_col}, {units_col}, {date_col},
|
||||
iter, logid, pf_user, created_at
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {date_col}, {value_col} [, {units_col}],
|
||||
pf_iter, pf_logid, pf_user, pf_created_at
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols}, {date_col}, {value_col} [, {units_col}],
|
||||
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM
|
||||
{schema}.{tname} -- or with dim_period JOIN (see above)
|
||||
WHERE
|
||||
{{filter_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols}, {value_col}, {units_col}, {date_col},
|
||||
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM
|
||||
{schema}.{tname}
|
||||
WHERE
|
||||
{date_col} BETWEEN '{{date_from}}' AND '{{date_to}}'
|
||||
SELECT count(*) AS rows_affected FROM ins
|
||||
```
|
||||
|
||||
No date offset — reference rows land at their original dates for prior-period comparison.
|
||||
No date offset applied — reference rows land at their original dates for prior-period comparison. Same dim_period JOIN logic applies as baseline.
|
||||
|
||||
### Scale
|
||||
|
||||
@ -595,26 +624,30 @@ WITH ilog AS (
|
||||
,base AS (
|
||||
SELECT
|
||||
{dimension_cols}, {date_col},
|
||||
{value_col}, {units_col},
|
||||
sum({value_col}) OVER () AS total_value,
|
||||
sum({units_col}) OVER () AS total_units
|
||||
{value_col} [, {units_col}],
|
||||
sum({value_col}) OVER () AS total_value
|
||||
[, sum({units_col}) OVER () AS total_units]
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
)
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {date_col}, {value_col}, {units_col},
|
||||
iter, logid, pf_user, created_at
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} (
|
||||
{dimension_cols}, {date_col}, {value_col} [, {units_col}],
|
||||
pf_iter, pf_logid, pf_user, pf_created_at
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols}, {date_col},
|
||||
round(({value_col} / NULLIF(total_value, 0)) * {{value_incr}}, 2)
|
||||
[, round(({units_col} / NULLIF(total_units, 0)) * {{units_incr}}, 5)],
|
||||
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM base
|
||||
RETURNING *
|
||||
)
|
||||
SELECT
|
||||
{dimension_cols}, {date_col},
|
||||
round(({value_col} / NULLIF(total_value, 0)) * {{value_incr}}, 2),
|
||||
round(({units_col} / NULLIF(total_units, 0)) * {{units_incr}}, 5),
|
||||
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM base
|
||||
SELECT * FROM ins
|
||||
```
|
||||
|
||||
`{{value_incr}}` / `{{units_incr}}` are pre-computed in JS when `pct: true` (multiply slice total by pct).
|
||||
`{{value_incr}}` / `{{units_incr}}` are pre-computed in JS when `pct: true` (multiply slice total by pct). Units expressions are omitted when no units column is configured.
|
||||
|
||||
### Recode
|
||||
|
||||
@ -625,22 +658,27 @@ WITH ilog AS (
|
||||
RETURNING id
|
||||
)
|
||||
,src AS (
|
||||
SELECT {dimension_cols}, {date_col}, {value_col}, {units_col}
|
||||
SELECT {dimension_cols}, {date_col}, {value_col} [, {units_col}]
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
)
|
||||
,negatives AS (
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at)
|
||||
SELECT {dimension_cols}, {date_col}, -{value_col}, -{units_col}, 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
,neg AS (
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
|
||||
SELECT {dimension_cols}, {date_col}, -{value_col} [, -{units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
RETURNING *
|
||||
)
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at)
|
||||
SELECT {{set_clause}}, {date_col}, {value_col}, {units_col}, 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
|
||||
SELECT {{set_clause}}, {date_col}, {value_col} [, {units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM src
|
||||
RETURNING *
|
||||
)
|
||||
SELECT * FROM neg UNION ALL SELECT * FROM ins
|
||||
```
|
||||
|
||||
`{{set_clause}}` replaces the listed dimension columns with new values, passes others through unchanged.
|
||||
`{{set_clause}}` replaces the listed dimension columns with new values, passes others through unchanged. Both the negative (zero-out) and positive (replacement) rows share the same `pf_logid` and are undone together.
|
||||
|
||||
### Clone
|
||||
|
||||
@ -650,21 +688,27 @@ WITH ilog AS (
|
||||
VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}')
|
||||
RETURNING id
|
||||
)
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at)
|
||||
SELECT
|
||||
{{set_clause}}, {date_col},
|
||||
round({value_col} * {{scale_factor}}, 2),
|
||||
round({units_col} * {{scale_factor}}, 5),
|
||||
'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
,ins AS (
|
||||
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
|
||||
SELECT
|
||||
{{set_clause}}, {date_col},
|
||||
round({value_col} * {{scale_factor}}, 2)
|
||||
[, round({units_col} * {{scale_factor}}, 5)],
|
||||
'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
|
||||
FROM {{fc_table}}
|
||||
WHERE {{where_clause}}
|
||||
{{exclude_clause}}
|
||||
RETURNING *
|
||||
)
|
||||
SELECT * FROM ins
|
||||
```
|
||||
|
||||
### Undo
|
||||
|
||||
Two queries run sequentially (not in a CTE — FK ordering):
|
||||
|
||||
```sql
|
||||
DELETE FROM {{fc_table}} WHERE logid = {{logid}};
|
||||
DELETE FROM {{fc_table}} WHERE pf_logid = {{logid}};
|
||||
DELETE FROM pf.log WHERE id = {{logid}};
|
||||
```
|
||||
|
||||
@ -698,38 +742,38 @@ DELETE FROM pf.log WHERE id = {{logid}};
|
||||
## Open Questions / Future Scope
|
||||
|
||||
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
|
||||
- **Arrow IPC for initial data load** — at large row counts (1M+) the `/versions/:id/data` JSON response becomes a bottleneck. Option: serve Arrow IPC binary instead of JSON; Perspective's `worker.table()` accepts Arrow buffers natively. Incremental operation rows (scale/recode/clone) can stay as JSON fed to `table.update()` since they're always small. Could be implemented with `pg` + `apache-arrow` in Node, or by adding a server-side DuckDB instance (Postgres scanner → Arrow IPC) if a caching layer is also needed.
|
||||
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
|
||||
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
|
||||
- **Export** — download forecast as CSV or push results to a reporting table
|
||||
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION)
|
||||
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync (e.g. a column added to SQL that doesn't exist in the table). UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). For now the workaround is to delete and recreate the version manually.
|
||||
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually.
|
||||
- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema.
|
||||
|
||||
---
|
||||
|
||||
## Project Status — 2026-04-25
|
||||
## Project Status — 2026-06-12
|
||||
|
||||
### What's working
|
||||
- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo
|
||||
- React + Vite + Tailwind CSS frontend scaffolded in `ui/`, built output to `public/app/`, served by Express
|
||||
- 3-step collapsible sidebar (Setup / Baseline / Forecast) — addresses prior UX concern about opaque 5-tab nav
|
||||
- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation
|
||||
- `units` column is optional — sources without a units column register and generate SQL correctly
|
||||
- `dim_group` / `dim_period_col` on col_meta: baseline/reference load JOINs `pf.dim_period` to derive fiscal/calendar period columns rather than copying them raw from the source
|
||||
- `pf.dim_period` calendar table (2018–2035): populated by `setup_sql/gen_dim_period.sql`, configurable fiscal year start
|
||||
- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express
|
||||
- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM
|
||||
- 3-step collapsible sidebar (Setup / Baseline / Forecast)
|
||||
- Setup view: DB table browser with preview modal, source registration, col_meta editor (`dim_group`/`dim_period_col` fields included), SQL generation
|
||||
- Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder
|
||||
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version
|
||||
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage
|
||||
- Slice extraction from `perspective-click` event feeds operation panel directly
|
||||
- Incremental row streaming: operation results (`RETURNING *`) stream into Perspective table without full reload
|
||||
- Incremental row streaming: operation results (`RETURNING *`) applied to Perspective table via `pspTable.update()` — no full reload
|
||||
- Status bar: shows current source · version · baseline row count · status
|
||||
|
||||
### Known issues / next focus
|
||||
|
||||
- **Forecast view** — operation panel (Scale / Recode / Clone) is a stub; needs wiring to API
|
||||
- **Status bar** — currently hardcoded; needs to reflect actual selected source/version from state
|
||||
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, the SQL and table DDL go out of sync. UI should detect this (compare col_meta against `information_schema`), warn, and offer rebuild. Workaround: delete and recreate the version.
|
||||
- **No "current version" persistence** — source/version selection resets on page reload; session context not persisted
|
||||
- **Perspective slice limitation** — computed date columns (Month, YearDate) extracted via split_by don't filter back to raw rows; only native dimension columns work for slice extraction
|
||||
|
||||
### Branch status
|
||||
- `baseline-workbench` — merged to origin, stable
|
||||
- `perspective-forecast` — active development branch; React UI scaffolded, Forecast operation panel pending
|
||||
- **Forecast view** — operation panel SQL generation complete; UI wiring to API still needed
|
||||
- **Load progress bar** — jittery at high throughput; throttle to ~10 updates/sec
|
||||
- **Default pivot layout** — per-source configurable layout not yet implemented; currently hardcodes first 2 dimensions
|
||||
- **No "current version" persistence** — source/version selection resets on page reload
|
||||
- **Perspective slice limitation** — computed date columns (Month, YearDate) from split_by don't map back to raw rows; only native dimension columns work for slice extraction
|
||||
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version.
|
||||
|
||||
|
||||
@ -4,11 +4,31 @@ const { fcTable } = require('../lib/utils');
|
||||
module.exports = function(pool) {
|
||||
const router = express.Router();
|
||||
|
||||
// list all log entries for a version, newest first
|
||||
// list all log entries for a version, newest first, with row counts from fc_table
|
||||
router.get('/versions/:id/log', async (req, res) => {
|
||||
try {
|
||||
const verResult = await pool.query(`
|
||||
SELECT v.*, s.tname
|
||||
FROM pf.version v
|
||||
JOIN pf.source s ON s.id = v.source_id
|
||||
WHERE v.id = $1
|
||||
`, [req.params.id]);
|
||||
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
|
||||
|
||||
const { tname, id: version_id } = verResult.rows[0];
|
||||
const table = fcTable(tname, version_id);
|
||||
|
||||
const result = await pool.query(
|
||||
`SELECT * FROM pf.log WHERE version_id = $1 ORDER BY stamp DESC`,
|
||||
`SELECT l.*,
|
||||
counts.row_count
|
||||
FROM pf.log l
|
||||
LEFT JOIN (
|
||||
SELECT pf_logid, count(*)::int AS row_count
|
||||
FROM ${table}
|
||||
GROUP BY pf_logid
|
||||
) counts ON counts.pf_logid = l.id
|
||||
WHERE l.version_id = $1
|
||||
ORDER BY l.stamp DESC`,
|
||||
[req.params.id]
|
||||
);
|
||||
res.json(result.rows);
|
||||
@ -18,6 +38,22 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
// update note on a log entry
|
||||
router.patch('/log/:logid', async (req, res) => {
|
||||
const { note } = req.body;
|
||||
try {
|
||||
const result = await pool.query(
|
||||
`UPDATE pf.log SET note = $1 WHERE id = $2 RETURNING *`,
|
||||
[note ?? null, parseInt(req.params.logid)]
|
||||
);
|
||||
if (result.rows.length === 0) return res.status(404).json({ error: 'Log entry not found' });
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// undo an operation — deletes all forecast rows with this logid, then the log entry
|
||||
// two separate queries in a transaction to avoid FK ordering issues
|
||||
router.delete('/log/:logid', async (req, res) => {
|
||||
|
||||
@ -31,6 +31,7 @@ module.exports = function(pool) {
|
||||
);
|
||||
const colMeta = colResult.rows;
|
||||
const dimCols = colMeta.filter(c => c.role === 'dimension').map(c => c.cname);
|
||||
const dateCols = colMeta.filter(c => c.role === 'date').map(c => c.cname);
|
||||
const valueCol = colMeta.find(c => c.role === 'value')?.cname;
|
||||
const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
|
||||
|
||||
@ -48,6 +49,8 @@ module.exports = function(pool) {
|
||||
table: fcTable(version.tname, version.id),
|
||||
colMeta,
|
||||
dimCols,
|
||||
dateCols,
|
||||
filterCols: [...dimCols, ...dateCols],
|
||||
valueCol,
|
||||
unitsCol,
|
||||
sql: sqlResult.rows[0].sql
|
||||
@ -306,7 +309,7 @@ module.exports = function(pool) {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'scale');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
|
||||
const whereClause = buildWhere(slice, ctx.dimCols);
|
||||
const whereClause = buildWhere(slice, ctx.filterCols);
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
|
||||
let absValueIncr = value_incr || 0;
|
||||
@ -364,7 +367,7 @@ module.exports = function(pool) {
|
||||
const ctx = await getContext(parseInt(req.params.id), 'recode');
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
|
||||
const whereClause = buildWhere(slice, ctx.dimCols);
|
||||
const whereClause = buildWhere(slice, ctx.filterCols);
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
const setClause = buildSetClause(ctx.dimCols, set);
|
||||
|
||||
@ -401,7 +404,7 @@ module.exports = function(pool) {
|
||||
if (!guardOpen(ctx.version, res)) return;
|
||||
|
||||
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
|
||||
const whereClause = buildWhere(slice, ctx.dimCols);
|
||||
const whereClause = buildWhere(slice, ctx.filterCols);
|
||||
const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
|
||||
const setClause = buildSetClause(ctx.dimCols, set);
|
||||
|
||||
|
||||
@ -89,20 +89,24 @@ module.exports = function(pool) {
|
||||
await client.query('BEGIN');
|
||||
for (const col of cols) {
|
||||
await client.query(`
|
||||
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, opos)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, opos)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
ON CONFLICT (source_id, cname) DO UPDATE SET
|
||||
label = EXCLUDED.label,
|
||||
role = EXCLUDED.role,
|
||||
is_key = EXCLUDED.is_key,
|
||||
opos = EXCLUDED.opos
|
||||
label = EXCLUDED.label,
|
||||
role = EXCLUDED.role,
|
||||
is_key = EXCLUDED.is_key,
|
||||
dim_group = EXCLUDED.dim_group,
|
||||
dim_period_col = EXCLUDED.dim_period_col,
|
||||
opos = EXCLUDED.opos
|
||||
`, [
|
||||
sourceId,
|
||||
col.cname,
|
||||
col.label || null,
|
||||
col.role || 'ignore',
|
||||
col.is_key || false,
|
||||
col.opos || null
|
||||
col.label || null,
|
||||
col.role || 'ignore',
|
||||
col.is_key || false,
|
||||
col.dim_group || null,
|
||||
col.dim_period_col || null,
|
||||
col.opos || null
|
||||
]);
|
||||
}
|
||||
await client.query('COMMIT');
|
||||
@ -139,7 +143,7 @@ module.exports = function(pool) {
|
||||
// validate required roles
|
||||
const colMeta = colResult.rows;
|
||||
const roles = new Set(colMeta.map(c => c.role));
|
||||
const missing = ['value', 'units', 'date'].filter(r => !roles.has(r));
|
||||
const missing = ['value', 'date'].filter(r => !roles.has(r));
|
||||
if (missing.length > 0) {
|
||||
return res.status(400).json({
|
||||
error: `col_meta is missing required roles: ${missing.join(', ')}`
|
||||
@ -222,6 +226,41 @@ module.exports = function(pool) {
|
||||
}
|
||||
});
|
||||
|
||||
// given a key column value, look up sibling dim_group column values from source
|
||||
// returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous
|
||||
router.get('/sources/:id/lookup', async (req, res) => {
|
||||
const { col, value } = req.query;
|
||||
if (!col || value == null || value === '') return res.json(null);
|
||||
try {
|
||||
const [srcResult, metaResult] = await Promise.all([
|
||||
pool.query(`SELECT schema, tname FROM pf.source WHERE id = $1`, [req.params.id]),
|
||||
pool.query(`SELECT * FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`, [req.params.id])
|
||||
]);
|
||||
if (srcResult.rows.length === 0) return res.status(404).json({ error: 'Source not found' });
|
||||
|
||||
const keyCol = metaResult.rows.find(c => c.cname === col && c.is_key && c.dim_group);
|
||||
if (!keyCol) return res.json(null);
|
||||
|
||||
const siblings = metaResult.rows.filter(c =>
|
||||
c.dim_group === keyCol.dim_group && c.cname !== col
|
||||
);
|
||||
if (!siblings.length) return res.json(null);
|
||||
|
||||
const { schema, tname } = srcResult.rows[0];
|
||||
const sibCols = siblings.map(c => `"${c.cname}"`).join(', ');
|
||||
const result = await pool.query(
|
||||
`SELECT DISTINCT ${sibCols} FROM "${schema}"."${tname}" WHERE "${col}" = $1 LIMIT 2`,
|
||||
[value]
|
||||
);
|
||||
|
||||
if (result.rows.length !== 1) return res.json(null);
|
||||
res.json(result.rows[0]);
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
// set or clear the default Perspective layout for a source.
|
||||
// Body: a Perspective view config (group_by, split_by, columns, plugin_config, …).
|
||||
// Pass null or {} to clear.
|
||||
@ -241,6 +280,22 @@ module.exports = function(pool) {
|
||||
});
|
||||
|
||||
// deregister a source — does not drop existing forecast tables
|
||||
router.get('/dim-period/cols', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
SELECT column_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = 'pf' AND table_name = 'dim_period'
|
||||
AND column_name NOT IN ('sdat', 'edat', 'drange', 'ndays')
|
||||
ORDER BY ordinal_position
|
||||
`);
|
||||
res.json(result.rows.map(r => r.column_name));
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
router.delete('/sources/:id', async (req, res) => {
|
||||
try {
|
||||
const result = await pool.query(
|
||||
@ -253,6 +308,9 @@ module.exports = function(pool) {
|
||||
res.json({ message: 'Source deregistered', source: result.rows[0] });
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
if (err.code === '23001' || err.code === '23503') {
|
||||
return res.status(409).json({ error: 'Source has existing versions — delete them first.' });
|
||||
}
|
||||
res.status(500).json({ error: err.message });
|
||||
}
|
||||
});
|
||||
|
||||
@ -15,8 +15,12 @@ CREATE TABLE IF NOT EXISTS pf.source (
|
||||
UNIQUE (schema, tname)
|
||||
);
|
||||
|
||||
-- backfill column for existing installs
|
||||
-- backfill columns for existing installs
|
||||
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
|
||||
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
|
||||
|
||||
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
|
||||
|
||||
CREATE TABLE IF NOT EXISTS pf.col_meta (
|
||||
id serial PRIMARY KEY,
|
||||
|
||||
94
setup_sql/gen_dim_period.sql
Normal file
94
setup_sql/gen_dim_period.sql
Normal file
@ -0,0 +1,94 @@
|
||||
-- pf.dim_period — create and populate
|
||||
-- Adjust fiscal_start_month: 1=Jan (calendar year), 4=Apr, 6=Jun, 7=Jul, 10=Oct, etc.
|
||||
-- Safe to re-run: ON CONFLICT DO NOTHING, so existing rows are never overwritten.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS pf.dim_period (
|
||||
sdat date NOT NULL PRIMARY KEY,
|
||||
edat date NOT NULL,
|
||||
drange daterange NOT NULL,
|
||||
ndays integer NOT NULL,
|
||||
-- calendar
|
||||
cal_year integer NOT NULL,
|
||||
cal_quarter integer NOT NULL,
|
||||
cal_month integer NOT NULL,
|
||||
cal_month_abbr text NOT NULL, -- 01 - Jan
|
||||
cal_month_name text NOT NULL, -- 01 - January
|
||||
cal_label text NOT NULL, -- 2025-01 Jan
|
||||
-- fiscal
|
||||
fisc_year integer NOT NULL,
|
||||
fisc_quarter integer NOT NULL,
|
||||
fisc_quarter_label text NOT NULL, -- FY2025 Q3
|
||||
fisc_month integer NOT NULL,
|
||||
fisc_month_abbr text NOT NULL, -- 07 - Jan
|
||||
fisc_month_name text NOT NULL, -- 07 - January
|
||||
fisc_label text NOT NULL, -- FY2025 P07
|
||||
-- sort key
|
||||
period_key text NOT NULL -- 2025.07 (ltree-compatible)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS dim_period_drange_idx ON pf.dim_period USING gist (drange);
|
||||
CREATE INDEX IF NOT EXISTS dim_period_fisc_idx ON pf.dim_period (fisc_year, fisc_month);
|
||||
CREATE INDEX IF NOT EXISTS dim_period_cal_idx ON pf.dim_period (cal_year, cal_month);
|
||||
|
||||
WITH
|
||||
cfg AS (
|
||||
SELECT 6 AS fiscal_start_month -- change to match your fiscal year start month
|
||||
)
|
||||
,periods AS (
|
||||
SELECT
|
||||
gs.d::date AS sdat,
|
||||
(gs.d + '1 month'::interval)::date AS edat,
|
||||
extract(year FROM gs.d)::int AS cal_year,
|
||||
extract(month FROM gs.d)::int AS cal_month,
|
||||
extract(quarter FROM gs.d)::int AS cal_quarter,
|
||||
((extract(month FROM gs.d)::int - cfg.fiscal_start_month + 12) % 12) + 1
|
||||
AS fisc_month,
|
||||
extract(year FROM gs.d)::int
|
||||
+ CASE
|
||||
WHEN cfg.fiscal_start_month > 1
|
||||
AND extract(month FROM gs.d)::int >= cfg.fiscal_start_month
|
||||
THEN 1 ELSE 0
|
||||
END AS fisc_year
|
||||
FROM
|
||||
generate_series('2018-01-01'::date, '2035-12-01'::date, '1 month') gs(d)
|
||||
CROSS JOIN cfg
|
||||
)
|
||||
INSERT INTO pf.dim_period (
|
||||
sdat, edat, drange, ndays,
|
||||
cal_year, cal_quarter, cal_month,
|
||||
cal_month_abbr, cal_month_name, cal_label,
|
||||
fisc_year, fisc_quarter, fisc_quarter_label,
|
||||
fisc_month, fisc_month_abbr, fisc_month_name,
|
||||
fisc_label, period_key
|
||||
)
|
||||
SELECT
|
||||
sdat,
|
||||
edat,
|
||||
daterange(sdat, edat) AS drange,
|
||||
edat - sdat AS ndays,
|
||||
cal_year,
|
||||
cal_quarter,
|
||||
cal_month,
|
||||
to_char(cal_month, 'FM00') || ' - ' || to_char(sdat, 'Mon') AS cal_month_abbr,
|
||||
to_char(cal_month, 'FM00') || ' - ' || to_char(sdat, 'Month') AS cal_month_name,
|
||||
to_char(sdat, 'YYYY-MM') || ' ' || to_char(sdat, 'Mon') AS cal_label,
|
||||
fisc_year,
|
||||
ceil(fisc_month / 3.0)::int AS fisc_quarter,
|
||||
'FY' || fisc_year || ' Q' || ceil(fisc_month / 3.0)::int AS fisc_quarter_label,
|
||||
fisc_month,
|
||||
to_char(fisc_month, 'FM00') || ' - ' || to_char(sdat, 'Mon') AS fisc_month_abbr,
|
||||
to_char(fisc_month, 'FM00') || ' - ' || to_char(sdat, 'Month') AS fisc_month_name,
|
||||
'FY' || fisc_year || ' P' || to_char(fisc_month, 'FM00') AS fisc_label,
|
||||
to_char(fisc_year, 'FM0000') || '.' || to_char(fisc_month, 'FM00') AS period_key
|
||||
FROM periods
|
||||
ON CONFLICT (sdat) DO NOTHING;
|
||||
|
||||
-- preview first 24 months
|
||||
SELECT
|
||||
period_key, sdat, edat, ndays,
|
||||
cal_year, cal_quarter, cal_month, cal_month_abbr,
|
||||
fisc_year, fisc_quarter, fisc_month, fisc_month_abbr,
|
||||
fisc_quarter_label, fisc_label, cal_label
|
||||
FROM pf.dim_period
|
||||
ORDER BY sdat
|
||||
LIMIT 24;
|
||||
113
todo.md
113
todo.md
@ -1,113 +0,0 @@
|
||||
- [ ] when you enter the forecast, be able to enter in a context so you dont have to open the whole thing (should show in status bar and be a filter for SQL and spi calls)
|
||||
|
||||
|
||||
- [x] should be able to edit and revise forecase segments that constitute baseline or reference. if you edit, maybe a warning that your forecast values wont mean a lot, and have an option to delete them.
|
||||
|
||||
Notes: A baseline/reference segment is a `pf.log` row plus the
|
||||
forecast rows it produced (joined by pf_logid). Editing has the
|
||||
shape of a delete-then-replay: drop the rows by pf_logid, drop the
|
||||
log entry, re-run the segment with the new params (offset, filter,
|
||||
iter type), insert the new log entry. New endpoint:
|
||||
`PUT /versions/:id/baseline/:logid` (and the same for reference).
|
||||
UI: an Edit button on each segment in Baseline view, populating the
|
||||
form with the original `params`.
|
||||
|
||||
Cascade warning: if any scale/recode/clone log entries exist *after*
|
||||
this segment was added, those operations were calibrated against
|
||||
the old totals and will no longer reconcile cleanly. Show a banner
|
||||
like "3 forecast operations applied after this segment may be
|
||||
invalidated. View / Delete / Continue." Probably want a CASCADE
|
||||
option that deletes downstream forecast entries too, plus a plain
|
||||
"edit only" option for the user who knows what they're doing.
|
||||
|
||||
Implementation order: API + cascade detection first (compare
|
||||
pf.log.stamp ordering); UI second.
|
||||
|
||||
- [~] be able to copy an existing forecast and it's segments to adjust some parameters without having to start from scrath.
|
||||
|
||||
Notes: A version is the unit of copy. Need a `POST /versions/:id/copy`
|
||||
endpoint that creates a new pf.version row with the same source/
|
||||
col_meta, creates the new fc_<tname>_<id> table via the same DDL
|
||||
path, and replays each pf.log entry's INSERT against the new table
|
||||
(preserving stamp ordering). Each log entry gets re-inserted
|
||||
pointing at the new version_id; the new pf_logid feeds the row
|
||||
inserts. Notes/users come along.
|
||||
|
||||
UI: "Copy" button next to each version in Baseline. Copy modal
|
||||
asks for a new name and optional description, then runs the API
|
||||
call (likely 5–30s for a 350k-row version since every segment is
|
||||
re-evaluated). Show progress.
|
||||
|
||||
Two design questions worth deciding up front:
|
||||
- Copy as-of-now (re-fetch source data, so freshly-arrived rows
|
||||
show up in baseline)? Or freeze (replay from existing forecast
|
||||
rows, i.e. clone the forecast table directly)? Different
|
||||
semantics, different SQL — pick one before building.
|
||||
- Should the copy track its origin? A `parent_version_id` column
|
||||
on pf.version makes "show me variants of FY2026 Plan" easy.
|
||||
|
||||
- [x] need the list of filters to have an and/or specification
|
||||
|
||||
Notes: Spec already covers this in `pf_spec.md:245` — `filters` is
|
||||
an array of groups; conditions within a group are AND-ed, groups
|
||||
OR-ed. Backend has `buildFilterClause` in
|
||||
`lib/sql_generator.js:247` but it's not wired into the routes
|
||||
(baseline currently takes raw `where_clause`). Wiring + UI is the
|
||||
remaining work.
|
||||
|
||||
UI: each group is a card with a header ("Group 1", "Group 2 — OR"),
|
||||
rows of `column / operator / values`, a `+ Add condition` link,
|
||||
and a `+ Add OR group` button at the bottom. The Baseline view
|
||||
already has a single-group filter builder; extend it to wrap the
|
||||
current rows in a group container and allow adding more groups.
|
||||
|
||||
- [x] the filters should have the option to just write the WHERE clause SQL
|
||||
|
||||
Notes: Spec covers this too (`pf_spec.md:251`, `:454`) as the
|
||||
`raw_where` admin-only escape hatch. The current baseline endpoint
|
||||
*already* takes `where_clause` as a raw string — so the API is
|
||||
effectively in "raw only" mode today; it's the structured side
|
||||
that's missing. Two things to add:
|
||||
|
||||
- Once structured `filters` is wired in, gate `raw_where` behind
|
||||
an admin check (`pf_user` in admin list — needs admin list
|
||||
config) and reject 400 if both are sent.
|
||||
- UI toggle: a "Switch to manual SQL" link in the Baseline filter
|
||||
builder swaps the structured rows for a `<textarea>`; warning
|
||||
banner: "Raw SQL is not validated. You are responsible for
|
||||
correctness and security."
|
||||
|
||||
- [ ] load status bar is super jittery and the numbers wildly change
|
||||
|
||||
Notes: `setLoadProgress` fires per chunk in the body-stream reader
|
||||
(Forecast.jsx:155–161). On localhost or fast connections the
|
||||
reader yields chunks in tight bursts and React re-renders the
|
||||
overlay text on each — the visible bytes value flickers because
|
||||
paints land out of order with respect to setState batching.
|
||||
|
||||
Fix: throttle to ~10 updates/sec. Either `if (now - lastUpdate >
|
||||
100)` before `setLoadProgress`, or accumulate received bytes into
|
||||
a ref and flush on `requestAnimationFrame`. Five-line change.
|
||||
|
||||
- [ ] default layout for the pivot should be sales_usd group by pending_rep, split by pf_iter
|
||||
|
||||
Notes: Default-layout logic lives in `initViewer` (Forecast.jsx
|
||||
~line 240) and currently picks `group_by = first 2 dimensions`,
|
||||
`split_by = date column`. `sales_usd` / `pending_rep` are
|
||||
source-specific, so hardcoding them in the view would break for
|
||||
any other source.
|
||||
|
||||
Two paths:
|
||||
- **Quick**: hardcode for the current source. Cheap, but rots
|
||||
the moment a second source comes along.
|
||||
- **Right**: store a default-layout config on `pf.source` (e.g., a
|
||||
JSON `default_view` column with `{ group_by, split_by, columns }`)
|
||||
and let initViewer read it. Setup view gets a small "Default
|
||||
pivot" editor — pick a value column, group_by columns, split_by
|
||||
column.
|
||||
|
||||
Suggest the right version since the spec already implies
|
||||
per-source customization in col_meta, and you're going to want
|
||||
this for any future source you register. The col_meta path is
|
||||
even tighter: extend col_meta with role flags like `default_value`,
|
||||
`default_group`, `default_split` that initViewer reads.
|
||||
@ -48,12 +48,20 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
|
||||
title={dark ? 'Switch to light mode' : 'Switch to dark mode'}
|
||||
>
|
||||
{dark ? (
|
||||
<svg width="14" height="14" viewBox="0 0 16 16" fill="currentColor">
|
||||
<path d="M8 12a4 4 0 1 0 0-8 4 4 0 0 0 0 8zM8 0a.5.5 0 0 1 .5.5v2a.5.5 0 0 1-1 0v-2A.5.5 0 0 1 8 0zm0 13a.5.5 0 0 1 .5.5v2a.5.5 0 0 1-1 0v-2A.5.5 0 0 1 8 13zm8-5a.5.5 0 0 1-.5.5h-2a.5.5 0 0 1 0-1h2a.5.5 0 0 1 .5.5zM3 8a.5.5 0 0 1-.5.5h-2a.5.5 0 0 1 0-1h2A.5.5 0 0 1 3 8z"/>
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
|
||||
<circle cx="12" cy="12" r="4"/>
|
||||
<line x1="12" y1="2" x2="12" y2="5"/>
|
||||
<line x1="12" y1="19" x2="12" y2="22"/>
|
||||
<line x1="4.93" y1="4.93" x2="7.05" y2="7.05"/>
|
||||
<line x1="16.95" y1="16.95" x2="19.07" y2="19.07"/>
|
||||
<line x1="2" y1="12" x2="5" y2="12"/>
|
||||
<line x1="19" y1="12" x2="22" y2="12"/>
|
||||
<line x1="4.93" y1="19.07" x2="7.05" y2="16.95"/>
|
||||
<line x1="16.95" y1="7.05" x2="19.07" y2="4.93"/>
|
||||
</svg>
|
||||
) : (
|
||||
<svg width="14" height="14" viewBox="0 0 16 16" fill="currentColor">
|
||||
<path d="M6 .278a.768.768 0 0 1 1.065.02A.75.75 0 0 1 5.792 15.5a.75.75 0 0 1-1.498-.075.768.768 0 0 1-.02-1.05A8 8 0 1 0 6.278 14.72a.768.768 0 0 1-1.055-.02A.75.75 0 0 1 2.5 13.75a.75.75 0 0 1 1.498.075A8 8 0 1 0 6 .278z"/>
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
|
||||
<path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
|
||||
</svg>
|
||||
)}
|
||||
</button>
|
||||
|
||||
@ -89,5 +89,6 @@ body { margin: 0; background-color: var(--bg-primary); color: var(--text-primary
|
||||
.dark ::selection { background-color: var(--accent-bg); color: var(--text-primary); }
|
||||
.dark input { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
|
||||
.dark select { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
|
||||
.dark select option { background-color: var(--bg-secondary); color: var(--text-primary); }
|
||||
.dark textarea { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
|
||||
.dark .bg-transparent { background-color: transparent; }
|
||||
|
||||
@ -113,11 +113,15 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
|
||||
const unitsCol = colMetaRef.current.find(c => c.role === 'units')?.cname
|
||||
if (!valueCol && !unitsCol) return
|
||||
try {
|
||||
const dimNames = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname))
|
||||
const dimNames = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname))
|
||||
const dateNames = new Set(colMetaRef.current.filter(c => c.role === 'date').map(c => c.cname))
|
||||
const filters = [
|
||||
...Object.entries(sliceObj)
|
||||
.filter(([col]) => dimNames.has(col))
|
||||
.map(([col, val]) => [col, '==', val]),
|
||||
...Object.entries(sliceObj)
|
||||
.filter(([col]) => dateNames.has(col))
|
||||
.map(([col, val]) => [col, '==', Number(val)]),
|
||||
['pf_iter', '!=', 'reference'],
|
||||
]
|
||||
const view = await tableRef.current.view({ filter: filters })
|
||||
@ -388,7 +392,9 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
|
||||
|
||||
async function submitOp(op) {
|
||||
if (!Object.keys(slice).length) { flash('Select a slice first', 'error'); return }
|
||||
let body = { pf_user: 'admin', slice }
|
||||
const effectiveSlice = buildEffectiveSlice(slice)
|
||||
if (!Object.keys(effectiveSlice).length) { flash('No dimension or date columns in slice — check col_meta', 'error'); return }
|
||||
let body = { pf_user: 'admin', slice: effectiveSlice }
|
||||
|
||||
if (op === 'scale') {
|
||||
let vi = null, ui = null
|
||||
@ -433,6 +439,65 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
|
||||
} catch (err) { flash(err.message, 'error') }
|
||||
}
|
||||
|
||||
async function lookupDerivedCols(col, value, setter) {
|
||||
if (!sourceId || !value.trim()) return
|
||||
const res = await fetch(`/api/sources/${sourceId}/lookup?col=${encodeURIComponent(col)}&value=${encodeURIComponent(value)}`)
|
||||
if (!res.ok) return
|
||||
const derived = await res.json()
|
||||
if (!derived) return
|
||||
setter(prev => {
|
||||
const next = { ...prev }
|
||||
for (const [k, v] of Object.entries(derived)) {
|
||||
if (!prev[k] || prev[k] === '') next[k] = String(v ?? '')
|
||||
}
|
||||
return next
|
||||
})
|
||||
}
|
||||
|
||||
function buildEffectiveSlice(raw) {
|
||||
const dimCols = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname))
|
||||
const dateCols = new Set(colMetaRef.current.filter(c => c.role === 'date').map(c => c.cname))
|
||||
const out = {}
|
||||
for (const [k, v] of Object.entries(raw)) {
|
||||
if (dimCols.has(k)) { out[k] = v; continue }
|
||||
if (dateCols.has(k)) {
|
||||
const ms = Number(v)
|
||||
out[k] = isFinite(ms) ? new Date(ms).toISOString().slice(0, 10) : v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
function buildPayload(op) {
|
||||
if (!Object.keys(slice).length) return null
|
||||
const effectiveSlice = buildEffectiveSlice(slice)
|
||||
let body = { pf_user: 'admin', slice: effectiveSlice }
|
||||
if (op === 'scale') {
|
||||
let vi = null, ui = null
|
||||
if (scaleMode === 'target') {
|
||||
const curValue = currentTotals?.total?.value
|
||||
const curUnits = currentTotals?.total?.units
|
||||
if (scalePrice !== '' && curUnits != null && curValue != null)
|
||||
vi = (parseFloat(scalePrice) * curUnits) - curValue
|
||||
if (scaleValue !== '' && curValue != null)
|
||||
vi = parseFloat(scaleValue) - curValue
|
||||
if (scaleUnits !== '' && curUnits != null)
|
||||
ui = parseFloat(scaleUnits) - curUnits
|
||||
} else {
|
||||
if (scaleValue !== '') vi = parseFloat(scaleValue)
|
||||
if (scaleUnits !== '') ui = parseFloat(scaleUnits)
|
||||
}
|
||||
body = { ...body, note: scaleNote || undefined, value_incr: vi, units_incr: ui, pct: scaleMode === 'delta' && scalePct }
|
||||
} else if (op === 'recode') {
|
||||
const set = Object.fromEntries(Object.entries(recodeSet).filter(([, v]) => v.trim()))
|
||||
body = { ...body, note: recodeNote || undefined, set }
|
||||
} else if (op === 'clone') {
|
||||
const set = Object.fromEntries(Object.entries(cloneSet).filter(([, v]) => v.trim()))
|
||||
body = { ...body, note: cloneNote || undefined, set, scale: parseFloat(cloneScale) || 1 }
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
function flash(text, type = 'ok') {
|
||||
setMsg({ text, type })
|
||||
if (type !== 'error') setTimeout(() => setMsg(null), 3000)
|
||||
@ -791,31 +856,46 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
|
||||
)}
|
||||
|
||||
<Row label="Note"><input value={scaleNote} onChange={e => setScaleNote(e.target.value)} placeholder="optional" className={inp} /></Row>
|
||||
<PayloadPreview payload={buildPayload('scale')} />
|
||||
<Submit onClick={() => submitOp('scale')}>Apply Scale</Submit>
|
||||
</>}
|
||||
|
||||
{activeOp === 'recode' && <>
|
||||
<p className="text-gray-400">New values for dimensions to replace. Leave blank to keep.</p>
|
||||
{dimCols.map(c => (
|
||||
<Row key={c.cname} label={c.cname}>
|
||||
<input value={recodeSet[c.cname] || ''} onChange={e => setRecodeSet(s => ({ ...s, [c.cname]: e.target.value }))}
|
||||
placeholder={slice[c.cname] || '—'} className={`${inp} font-mono`} />
|
||||
<Row key={c.cname} label={c.label || c.cname}>
|
||||
<input
|
||||
value={recodeSet[c.cname] || ''}
|
||||
onChange={e => setRecodeSet(s => ({ ...s, [c.cname]: e.target.value }))}
|
||||
onBlur={c.is_key && c.dim_group
|
||||
? e => lookupDerivedCols(c.cname, e.target.value, setRecodeSet)
|
||||
: undefined}
|
||||
placeholder={slice[c.cname] || '—'}
|
||||
className={`${inp} font-mono`} />
|
||||
</Row>
|
||||
))}
|
||||
<Row label="Note"><input value={recodeNote} onChange={e => setRecodeNote(e.target.value)} placeholder="optional" className={inp} /></Row>
|
||||
<PayloadPreview payload={buildPayload('recode')} />
|
||||
<Submit onClick={() => submitOp('recode')}>Apply Recode</Submit>
|
||||
</>}
|
||||
|
||||
{activeOp === 'clone' && <>
|
||||
<p className="text-gray-400">Override dimensions on cloned rows. Leave blank to keep.</p>
|
||||
{dimCols.map(c => (
|
||||
<Row key={c.cname} label={c.cname}>
|
||||
<input value={cloneSet[c.cname] || ''} onChange={e => setCloneSet(s => ({ ...s, [c.cname]: e.target.value }))}
|
||||
placeholder={slice[c.cname] || '—'} className={`${inp} font-mono`} />
|
||||
<Row key={c.cname} label={c.label || c.cname}>
|
||||
<input
|
||||
value={cloneSet[c.cname] || ''}
|
||||
onChange={e => setCloneSet(s => ({ ...s, [c.cname]: e.target.value }))}
|
||||
onBlur={c.is_key && c.dim_group
|
||||
? e => lookupDerivedCols(c.cname, e.target.value, setCloneSet)
|
||||
: undefined}
|
||||
placeholder={slice[c.cname] || '—'}
|
||||
className={`${inp} font-mono`} />
|
||||
</Row>
|
||||
))}
|
||||
<Row label="Scale"><input type="number" step="any" value={cloneScale} onChange={e => setCloneScale(e.target.value)} className={inp} /></Row>
|
||||
<Row label="Note"><input value={cloneNote} onChange={e => setCloneNote(e.target.value)} placeholder="optional" className={inp} /></Row>
|
||||
<PayloadPreview payload={buildPayload('clone')} />
|
||||
<Submit onClick={() => submitOp('clone')}>Apply Clone</Submit>
|
||||
</>}
|
||||
</div>
|
||||
@ -874,3 +954,12 @@ function Submit({ onClick, children }) {
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
function PayloadPreview({ payload }) {
|
||||
if (!payload) return null
|
||||
return (
|
||||
<pre className="text-xs font-mono text-gray-400 bg-gray-50 border border-gray-100 rounded p-2 overflow-auto max-h-36 leading-relaxed">
|
||||
{JSON.stringify(payload, null, 2)}
|
||||
</pre>
|
||||
)
|
||||
}
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
import { useState, useEffect } from 'react'
|
||||
import { useState, useEffect, useRef } from 'react'
|
||||
import useTheme from '../theme.jsx'
|
||||
|
||||
const ROLES = ['ignore', 'dimension', 'value', 'units', 'date', 'filter']
|
||||
|
||||
@ -12,6 +13,7 @@ const ROLE_STYLE = {
|
||||
}
|
||||
|
||||
export default function Setup({ refreshSources }) {
|
||||
const { dark } = useTheme()
|
||||
const [tables, setTables] = useState([])
|
||||
const [sources, setSources] = useState([])
|
||||
const [selectedSource, setSelectedSource] = useState(null)
|
||||
@ -24,12 +26,26 @@ export default function Setup({ refreshSources }) {
|
||||
const [saving, setSaving] = useState(false)
|
||||
const [generating, setGenerating] = useState(false)
|
||||
const [msg, setMsg] = useState(null)
|
||||
const [dimPeriodCols, setDimPeriodCols] = useState([])
|
||||
const [openPeriodIdx, setOpenPeriodIdx] = useState(null)
|
||||
const periodDropRef = useRef(null)
|
||||
|
||||
useEffect(() => {
|
||||
fetch('/api/tables').then(r => r.json()).then(setTables).catch(console.error)
|
||||
fetch('/api/dim-period/cols').then(r => r.json()).then(setDimPeriodCols).catch(console.error)
|
||||
loadSources()
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
if (openPeriodIdx === null) return
|
||||
function handleClick(e) {
|
||||
if (periodDropRef.current && !periodDropRef.current.contains(e.target))
|
||||
setOpenPeriodIdx(null)
|
||||
}
|
||||
document.addEventListener('mousedown', handleClick)
|
||||
return () => document.removeEventListener('mousedown', handleClick)
|
||||
}, [openPeriodIdx])
|
||||
|
||||
function loadSources() {
|
||||
fetch('/api/sources').then(r => r.json()).then(data => {
|
||||
setSources(data)
|
||||
@ -140,7 +156,12 @@ export default function Setup({ refreshSources }) {
|
||||
async function deleteSource(id, e) {
|
||||
e.stopPropagation()
|
||||
if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return
|
||||
await fetch(`/api/sources/${id}`, { method: 'DELETE' })
|
||||
const res = await fetch(`/api/sources/${id}`, { method: 'DELETE' })
|
||||
if (!res.ok) {
|
||||
const data = await res.json().catch(() => ({}))
|
||||
flash(data.error || 'Delete failed', 'err')
|
||||
return
|
||||
}
|
||||
if (selectedSource?.id === id) { setSelectedSource(null); setCols([]); setEditedCols([]) }
|
||||
loadSources()
|
||||
}
|
||||
@ -281,6 +302,8 @@ export default function Setup({ refreshSources }) {
|
||||
<th className="px-3 py-1.5 font-medium">column</th>
|
||||
<th className="px-3 py-1.5 font-medium">role</th>
|
||||
<th className="px-3 py-1.5 font-medium text-center">key</th>
|
||||
<th className="px-3 py-1.5 font-medium">group</th>
|
||||
<th className="px-3 py-1.5 font-medium">period col</th>
|
||||
<th className="px-3 py-1.5 font-medium">label</th>
|
||||
</tr>
|
||||
</thead>
|
||||
@ -302,10 +325,53 @@ export default function Setup({ refreshSources }) {
|
||||
type="checkbox"
|
||||
checked={!!col.is_key}
|
||||
onChange={e => updateCol(i, 'is_key', e.target.checked)}
|
||||
disabled={col.role !== 'dimension'}
|
||||
disabled={col.role !== 'dimension' && col.role !== 'date'}
|
||||
className="cursor-pointer disabled:opacity-20"
|
||||
/>
|
||||
</td>
|
||||
<td className="px-3 py-1.5">
|
||||
<input
|
||||
type="text"
|
||||
value={col.dim_group || ''}
|
||||
onChange={e => updateCol(i, 'dim_group', e.target.value || null)}
|
||||
placeholder="—"
|
||||
disabled={col.role !== 'dimension' && col.role !== 'date'}
|
||||
className="border border-transparent hover:border-gray-200 focus:border-gray-300 rounded px-1.5 py-0.5 w-full outline-none bg-transparent disabled:opacity-20 disabled:cursor-default"
|
||||
/>
|
||||
</td>
|
||||
<td className="px-3 py-1.5 relative">
|
||||
{(() => {
|
||||
const disabled = col.role !== 'dimension' || !col.dim_group
|
||||
const isOpen = openPeriodIdx === i
|
||||
return (
|
||||
<div ref={isOpen ? periodDropRef : null} className="relative">
|
||||
<button
|
||||
type="button"
|
||||
disabled={disabled}
|
||||
onClick={() => setOpenPeriodIdx(isOpen ? null : i)}
|
||||
className="text-xs px-1.5 py-0.5 rounded border border-transparent hover:border-gray-200 outline-none bg-white text-gray-700 disabled:opacity-20 disabled:cursor-default font-mono w-full text-left"
|
||||
>
|
||||
{col.dim_period_col || '—'}
|
||||
</button>
|
||||
{isOpen && (
|
||||
<ul style={{ backgroundColor: dark ? '#2a2c2f' : '#ffffff', color: dark ? '#c5c9d0' : '#374151', borderColor: dark ? '#3b3f46' : '#e5e7eb' }} className="absolute z-50 left-0 top-full mt-0.5 min-w-full border rounded shadow-lg text-xs font-mono max-h-48 overflow-y-auto">
|
||||
<li
|
||||
className="px-2 py-1 cursor-pointer hover:bg-gray-100"
|
||||
onMouseDown={() => { updateCol(i, 'dim_period_col', null); setOpenPeriodIdx(null) }}
|
||||
>—</li>
|
||||
{dimPeriodCols.map(c => (
|
||||
<li
|
||||
key={c}
|
||||
className={`px-2 py-1 cursor-pointer hover:bg-gray-100 ${col.dim_period_col === c ? 'bg-blue-50 text-blue-700' : ''}`}
|
||||
onMouseDown={() => { updateCol(i, 'dim_period_col', c); setOpenPeriodIdx(null) }}
|
||||
>{c}</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
})()}
|
||||
</td>
|
||||
<td className="px-3 py-1.5">
|
||||
<input
|
||||
type="text"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user