pt/pf_app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/operations-panel'

Browse Source
This commit is contained in:
Paul Trowbridge 2026-06-17 21:24:15 -04:00
commit 41349f2dee
14 changed files with 796 additions and 273 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
CLAUDE.md
View File

@ -7,16 +7,15 @@ A web app for building named forecast scenarios against any PostgreSQL table. Th
Full spec: `pf_spec.md` Full spec: `pf_spec.md`
Data transport architecture options: `pf_perspective_options.md` Data transport architecture options: `pf_perspective_options.md`
UX mockup: `pf_ux_mockup.md` UX mockup: `pf_ux_mockup.md`
Open work: `todo.md`
--- ---
## Tech stack ## Tech stack
- **Backend:** Node.js / Express (`server.js`), runs on port 3010 - **Backend:** Node.js / Express (`server.js`)
- **Database:** PostgreSQL — isolated `pf` schema - **Database:** PostgreSQL — isolated `pf` schema
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/` - **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://perspective.finos.org/) 4.4.0 loaded from CDN at runtime - **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/` - **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
--- ---
@ -51,11 +50,12 @@ ui/src/
## Database schema (`pf`) ## Database schema (`pf`)
- **`pf.source`** — registered source tables - **`pf.source`** — registered source tables
- **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses - **`pf.col_meta`** — column roles: `dimension` | `value` | `units` | `date` | `filter` | `ignore`; `is_key` marks dimensions used in slice WHERE clauses; `dim_group` groups functionally dependent columns (e.g. date + its derived year/month dimensions); `dim_period_col` maps a dimension to a `pf.dim_period` column so date-adjacent values are derived at load time rather than copied raw
- **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations - **`pf.version`** — named forecast scenarios; `exclude_iters` (default `["reference"]`) blocks those iter values from all operations
- **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`iter = baseline|scale|recode|clone`) and reference rows (`iter = reference`) - **`pf.fc_{tname}_{version_id}`** — one forecast table per version; contains both operational rows (`pf_iter = baseline|scale|recode|clone`) and reference rows (`pf_iter = reference`)
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb - **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time - **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
### Key token substitution tokens ### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}` `{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
@ -97,10 +97,20 @@ All three operations follow the same structure: insert a `pf.log` row in a CTE,
--- ---
## Known issues / active work (see todo.md for detail) ## Light / dark mode
- Operation panel (Scale/Recode/Clone) wiring to API is a stub — needs completion Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) with a `ThemeProvider` that wraps the app in `main.jsx`.
- Status bar is hardcoded — needs to reflect actual selected source/version
- **Storage key:** `pf_dark` in `localStorage`; falls back to `window.matchMedia('(prefers-color-scheme: dark)')` on first visit
- **Toggle:** `setDark(d => !d)` in `StatusBar.jsx`; effect writes `localStorage` and toggles the `.dark` class on `<html>`
- **CSS:** `ui/src/index.css` defines CSS custom properties under `:root` (light) and `.dark`. All Tailwind color overrides are written as `.dark .bg-white { ... }` etc. — no Tailwind dark-mode config needed
- **Palette:** dark mode uses Perspective's "Pro Dark" colours (`--bg-primary: #242526`, panels `#2a2c2f`, gridlines `#3b3f46`, text `#c5c9d0`)
- **Perspective viewer:** `Forecast.jsx` calls `viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')` both on initial load and in a `useEffect([dark, versionId])` so the viewer stays in sync when the toggle fires
- **Consuming the theme:** `import useTheme from '../theme.jsx'` then `const { dark, setDark } = useTheme()`
## Known issues / active work
- Operation panel (Scale/Recode/Clone) SQL generation and dim_period JOIN are complete; UI wiring to API still needs completion
- Load progress bar is jittery — needs throttle (~10 updates/sec) - Load progress bar is jittery — needs throttle (~10 updates/sec)
- Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions) - Default pivot layout should be configurable per source (currently hardcodes first 2 dimensions)
- Source/version selection doesn't persist across page reload - Source/version selection doesn't persist across page reload

199
PERSPECTIVE.md Normal file
View File

@ -0,0 +1,199 @@
# Perspective — configuration & deployment reference
Canonical guide for how Perspective should be configured, fed, and shipped across our
apps (**pf_app**, **dataflow**). Both embed the same `<perspective-viewer>` web component
but made different early choices; this doc defines the target state and the rationale, so
the two converge instead of drifting.
> **Distribution note:** we use the **`@perspective-dev/*`** packages
> (repo: <https://github.com/perspective-dev/perspective>, home:
> <https://perspective-dev.github.io>) — **not** the FINOS/OpenJS `@finos/perspective`
> packages from <https://perspective.finos.org/>. Same engine lineage, but a separate
> npm scope, release cadence, and "Pro" theme set (`Pro Dark`/`Pro Light`). This is why
> `viewer-d3fc` versions on a different schedule than `viewer`/`client` (§2), and why the
> client's bundled `apache-arrow` (17.x) can lag the server's (§3). Don't mix the two
> scopes.
---
## Core principle: Perspective is one locked unit
These four things must move together and be pinned together. Bumping one without the
others is the source of nearly every Perspective bug we've hit:
1. **Loader** — how the JS/WASM gets into the page (npm-inline vs CDN)
2. **Package version** — `client` / `viewer` / `viewer-datagrid` / `viewer-d3fc`
3. **Data format** — Arrow IPC vs JSON rows
4. **`apache-arrow` version** (server-side, only if using Arrow) — must speak an IPC
format the client WASM understands
Treat a Perspective upgrade as a coordinated change to all four, gated by the smoke test
at the bottom of this doc. Never let viewer/client drift ahead of `viewer-d3fc`.
---
## 1. Loading — npm `/inline`, pinned exact (not CDN)
```js
import perspective from '@perspective-dev/client/inline'
import '@perspective-dev/viewer/inline'
import '@perspective-dev/viewer-datagrid'
import '@perspective-dev/viewer-d3fc'
import '@perspective-dev/viewer/themes'
```
- The `/inline` entrypoints bundle the WASM into the Vite build — **no runtime network
fetch, works offline / behind a firewall, reproducible from the lockfile.**
- **Do not load from a CDN at runtime.** It's convenient for a prototype (smaller build,
one-line version bumps) but in production it means: app breaks if the CDN is
unreachable, version isn't captured in `package-lock.json`, slower cold start, and you
pull executable WASM from a third party on every load. (pf_app currently does this in
`ui/src/views/Forecast.jsx` — migrating off it is the main open item.)
- The themes CSS is imported in JS (`@perspective-dev/viewer/themes`), **not** via a
`<link>` in `index.html` — so it's bundled and versioned too.
---
## 2. Version policy — a real trilemma (read carefully)
The version choice is constrained by two hard facts about the `@perspective-dev` packages
**(verified against installed metadata, 2026-06)**:
- **`viewer-d3fc` caps at 4.4.1** — npm publishes no 4.5.x. The d3fc charts (Bar / Line /
Treemap / Heatmap / etc.) live only in this package.
- **The `/inline` and `/themes` entrypoints are 4.5.x-only** — `@perspective-dev/client/inline`,
`@perspective-dev/viewer/inline`, and `@perspective-dev/viewer/themes` do **not** exist
in 4.4.1's `exports` map. Bundling inline WASM requires 4.5.x.
So you can have at most **two** of these three:
| Want | Requires |
|---|---|
| Inline WASM bundling (`/inline`, `/themes`) | **4.5.x** viewer/client |
| One coherent single-version suite | **4.4.1** everything (d3fc ceiling) |
| d3fc chart plugins | **4.4.1** viewer-d3fc |
There is **no** version where all three hold. Pick by what the app needs:
- **Inline-bundled + charts** (dataflow's case) → `^4.5.1` viewer/client/datagrid **+
`^4.4.1` viewer-d3fc`. This is a deliberate, necessary mixed-version pair, *not* an
accident — it's the only combo that keeps both. Accept it; pin the lockfile and gate
bumps on the smoke test (§7). Do **not** "fix" it by pinning everything to 4.4.1 — the
build breaks (`"./inline" is not exported`).
- **Coherent single suite, no inline** (e.g. CDN or `.`-entry loading) → pin all four to
**4.4.1 exact**. Charts work; you give up `/inline` bundling.
Whatever you pick, **commit the lockfile** so the resolved set can't drift on
`npm install`. Re-evaluate the whole policy only when `viewer-d3fc` ships a 4.5.x (then a
fully-coherent inline-capable 4.5.x suite becomes possible).
---
## 3. Data delivery — match the format to the workload
| Workload | Format | Why |
|---|---|---|
| Large (100k+ rows), numeric-heavy, writes/incremental updates | **Arrow IPC** | Compact columnar binary, near-zero-copy ingest, carries types (no string coercion). Powers pf_app's 500k-row path. |
| Small (≤100k), read-only, click-to-inspect | **JSON rows** | Simpler, no encoding step, no dictionary pitfalls. dataflow's model. |
### Arrow constraints (read before choosing it)
If you deliver Arrow, three pieces are coupled and must stay aligned:
- **Numeric type parsers, server-side.** pg returns `bigint`/`numeric` as *strings*; you
must coerce them to JS numbers before encoding, or `apache-arrow` infers
`Dictionary<Utf8>` instead of `Int`/`Float64`. See `server.js` type parsers (oid 20,
1700).
- **Single record batch.** Per-batch Arrow builds independent dictionaries; the
Perspective WASM crashes on dictionary-replacement messages. The server must
accumulate all rows and emit **one** batch (`tableToIPC(tableFromJSON(allRows),
'stream')`). Consequence: the client "stream" is just a chunked *download* of one
batch — nothing renders progressively, and the server holds the full result set in
memory per request.
- **`apache-arrow` pinned to match the client WASM.** Pin it exact in the server
`package.json` and treat it as part of the locked unit (principle above). Note the
`@perspective-dev/client` build is tested against **`apache-arrow@17.0.0`**, while
pf_app's server currently pins **`^21.1.0`** — a real IPC version gap. Prefer aligning
the server toward the arrow major the client was built against, or at minimum make
Arrow ingestion (smoke test §7) the gate on any arrow bump.
JSON avoids all three but pays in payload size and parse cost, and pushes type handling
to client-side heuristics — acceptable only at small scale.
---
## 4. Theming
- One toggle drives both app CSS and the viewer:
`viewer.setAttribute('theme', dark ? 'Pro Dark' : 'Pro Light')`.
- Apply it on initial load **and** in an effect keyed on the dark flag, so the viewer
re-themes when the toggle fires (not just on mount).
- App-level dark mode is plain CSS custom properties + a `.dark` class on `<html>`; the
viewer theme name is the only Perspective-specific piece.
---
## 5. Layout persistence
- Persist the full viewer config (`await viewer.save()`, incl. `plugin_config`) to
`localStorage`, keyed per source/version. Restore with `viewer.restore(cfg)`.
- **Guard restores against schema drift.** Before restoring, filter the saved config's
`columns`/`group_by`/`split_by`/`sort`/`filter` against the columns that actually
exist in the current dataset (plus any `expressions`). dataflow's `cleanLayout()` is
the reference implementation; a stale layout referencing a dropped column otherwise
throws on restore.
---
## 6. Build & deploy (target)
- **Build:** `vite build` emits a static bundle (pf_app → `public/app`, via
`outDir: '../public/app'`). The API server serves it statically (`express.static`).
- **Process:** run the Node API under **systemd** with `Restart=always` and an
`EnvironmentFile=.env`; front it with **nginx** (reverse proxy + TLS via certbot).
dataflow's `dataflow.service` + `deploy.sh` are the reference; **pf_app has no deploy
automation yet** and should adopt the same pattern.
- **`deploy.sh`** should be idempotent: first run installs (db/schema, UI build, nginx,
systemd unit); later runs rebuild the UI and restart the service.
- Keep secrets in `.env` (db creds), loaded by both the app (`dotenv`) and the systemd
unit — never commit it.
---
## 7. Upgrade checklist / smoke test
Run this whenever bumping **any** Perspective package or `apache-arrow`:
1. Confirm `viewer-d3fc` publishes the target version
(`npm view @perspective-dev/viewer-d3fc versions`). If not, **don't bump** the others.
2. Pin all four packages + `apache-arrow` to exact, matching versions; `npm install`;
commit the lockfile.
3. `vite build` — no unresolved imports.
4. **Arrow apps:** load a real dataset and confirm `worker.table(buffer)` ingests
without a WASM dictionary error; verify a numeric column is `Float64`/`Int`, not a
string/dictionary.
5. Open a d3fc **chart** plugin (not just datagrid) and confirm it renders.
6. Toggle dark/light; confirm the viewer re-themes.
7. Save a layout, reload, confirm it restores; then drop a column and confirm the
cleaned restore doesn't throw.
---
## Per-project state (2026-06)
| | pf_app | dataflow | Target |
|---|---|---|---|
| Loader | CDN (runtime) | npm `/inline` | **npm `/inline`** |
| Version | 4.4.0 (CDN URLs) | 4.5.1 viewer/client + 4.4.1 d3fc | depends on loader (§2) |
| Data | Arrow IPC (single batch) | JSON (≤100k) | per workload (§3) |
| `apache-arrow` | `^21.1.0` (client built vs 17) | n/a | pin exact, match WASM |
| Deploy | none | systemd + nginx + `deploy.sh` | **systemd + nginx + `deploy.sh`** |
**dataflow's 4.5.1/4.4.1 pair is correct** — it's the only combo giving both inline
bundling and d3fc charts (§2). Leave it; just keep the lockfile committed.
**Open items:**
- pf_app → move off CDN. Note this forces the §2 choice: going npm-`/inline` means
4.5.x viewer/client + 4.4.1 d3fc (same pair as dataflow); or stay coherent at 4.4.x and
load via the `.` entry instead of `/inline`. Either way, pin + commit the lockfile, and
add deploy automation (systemd + nginx + `deploy.sh`).

56
lib/sql_generator.js
View File

@ -21,11 +21,10 @@ function generateSQL(source, colMeta) {
const dateCol = colMeta.find(c => c.role === 'date')?.cname; const dateCol = colMeta.find(c => c.role === 'date')?.cname;
if (!valueCol) throw new Error('No value column defined in col_meta'); if (!valueCol) throw new Error('No value column defined in col_meta');
if (!unitsCol) throw new Error('No units column defined in col_meta');
if (!dateCol) throw new Error('No date column defined in col_meta'); if (!dateCol) throw new Error('No date column defined in col_meta');
if (dims.length === 0) throw new Error('No dimension columns defined in col_meta'); if (dims.length === 0) throw new Error('No dimension columns defined in col_meta');
const srcTable = `${source.schema}.${source.tname}`; const srcTable = `"${source.schema}"."${source.tname}"`;
const dataCols = [...dims, dateCol, valueCol, unitsCol].filter(Boolean); const dataCols = [...dims, dateCol, valueCol, unitsCol].filter(Boolean);
const effectiveValue = dataCols.includes(valueCol) ? valueCol : null; const effectiveValue = dataCols.includes(valueCol) ? valueCol : null;
const effectiveUnits = dataCols.includes(unitsCol) ? unitsCol : null; const effectiveUnits = dataCols.includes(unitsCol) ? unitsCol : null;
@ -33,6 +32,19 @@ function generateSQL(source, colMeta) {
const selectData = dataCols.map(q).join(', '); const selectData = dataCols.map(q).join(', ');
const dimsJoined = dims.map(q).join(', '); const dimsJoined = dims.map(q).join(', ');
// dim_period JOIN support: if the date column is the is_key of a dim_group,
// dimension siblings with dim_period_col set are derived from pf.dim_period
// instead of being copied raw from the source on baseline/reference load.
const dateKeyGroup = colMeta.find(c => c.role === 'date' && c.is_key && c.dim_group)?.dim_group;
const dimPeriodMap = new Map(
dateKeyGroup
? colMeta
.filter(c => c.role === 'dimension' && c.dim_group === dateKeyGroup && c.dim_period_col)
.map(c => [c.cname, c.dim_period_col])
: []
);
const hasDimPeriod = dimPeriodMap.size > 0;
return { return {
get_data: buildGetData(), get_data: buildGetData(),
baseline: buildBaseline(), baseline: buildBaseline(),
@ -47,10 +59,22 @@ function generateSQL(source, colMeta) {
return `SELECT * FROM {{fc_table}}`; return `SELECT * FROM {{fc_table}}`;
} }
function buildLoadSelect(pfx) {
// pfx: table alias prefix ('s.' when joining dim_period, '' otherwise)
return dataCols.map(c => {
if (c === dateCol) return `(${pfx}${q(c)} + '{{date_offset}}'::interval)::date`;
if (dimPeriodMap.has(c)) return `dp.${q(dimPeriodMap.get(c))} AS ${q(c)}`;
return `${pfx}${q(c)}`;
}).join(',\n ');
}
function buildFromClause() {
if (!hasDimPeriod) return srcTable;
return `${srcTable} s\n JOIN pf.dim_period dp`
+ ` ON dp.drange @> (s.${q(dateCol)} + '{{date_offset}}'::interval)::date`;
}
function buildBaseline() { function buildBaseline() {
const baselineSelect = dataCols.map(c =>
c === dateCol ? `(${q(c)} + '{{date_offset}}'::interval)::date` : q(c)
).join(', ');
return ` return `
WITH WITH
ilog AS ( ilog AS (
@ -60,8 +84,10 @@ ilog AS (
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT ${baselineSelect}, 'baseline', (SELECT id FROM ilog), '{{pf_user}}', now() SELECT
FROM ${srcTable} ${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM ${buildFromClause()}
WHERE {{filter_clause}} WHERE {{filter_clause}}
RETURNING * RETURNING *
) )
@ -69,9 +95,6 @@ SELECT count(*) AS rows_affected FROM ins`.trim();
} }
function buildReference() { function buildReference() {
const referenceSelect = dataCols.map(c =>
c === dateCol ? `(${q(c)} + '{{date_offset}}'::interval)::date` : q(c)
).join(', ');
return ` return `
WITH WITH
ilog AS ( ilog AS (
@ -81,8 +104,10 @@ ilog AS (
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT ${referenceSelect}, 'reference', (SELECT id FROM ilog), '{{pf_user}}', now() SELECT
FROM ${srcTable} ${buildLoadSelect(hasDimPeriod ? 's.' : '')},
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM ${buildFromClause()}
WHERE {{filter_clause}} WHERE {{filter_clause}}
RETURNING * RETURNING *
) )
@ -145,14 +170,14 @@ ilog AS (
) )
,neg AS ( ,neg AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}, ${effectiveUnits ? `-${q(effectiveUnits)}` : '0'}, SELECT ${dimsJoined}, ${q(dateCol)}, ${effectiveValue ? `-${q(effectiveValue)}` : '0'}${effectiveUnits ? `, -${q(effectiveUnits)}` : ''},
'recode', (SELECT id FROM ilog), '{{pf_user}}', now() 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src FROM src
RETURNING * RETURNING *
) )
,ins AS ( ,ins AS (
INSERT INTO {{fc_table}} (${insertCols}) INSERT INTO {{fc_table}} (${insertCols})
SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}, ${effectiveUnits ? q(effectiveUnits) : '0'}, SELECT {{set_clause}}, ${q(dateCol)}, ${effectiveValue ? q(effectiveValue) : '0'}${effectiveUnits ? `, ${q(effectiveUnits)}` : ''},
'recode', (SELECT id FROM ilog), '{{pf_user}}', now() 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src FROM src
RETURNING * RETURNING *
@ -173,8 +198,7 @@ ilog AS (
SELECT SELECT
{{set_clause}}, {{set_clause}},
${q(dateCol)}, ${q(dateCol)},
${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}, ${effectiveValue ? `round(${q(effectiveValue)} * {{scale_factor}}, 2)` : '0'}${effectiveUnits ? `,\n round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : ''},
${effectiveUnits ? `round(${q(effectiveUnits)} * {{scale_factor}}, 5)` : '0'},
'clone', (SELECT id FROM ilog), '{{pf_user}}', now() 'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM {{fc_table}} FROM {{fc_table}}
WHERE {{where_clause}} WHERE {{where_clause}}

252
pf_spec.md
View File

@ -24,13 +24,14 @@ Registered source tables available for forecasting.
```sql ```sql
CREATE TABLE pf.source ( CREATE TABLE pf.source (
id serial PRIMARY KEY, id serial PRIMARY KEY,
schema text NOT NULL, schema text NOT NULL,
tname text NOT NULL, tname text NOT NULL,
label text, -- friendly display name label text, -- friendly display name
status text DEFAULT 'active', -- active | archived status text DEFAULT 'active', -- active | archived
created_at timestamptz DEFAULT now(), default_layout jsonb, -- Perspective view config used as per-source default
created_by text, created_at timestamptz DEFAULT now(),
created_by text,
UNIQUE (schema, tname) UNIQUE (schema, tname)
); );
``` ```
@ -40,25 +41,40 @@ Column configuration for each registered source table. Determines how the app tr
```sql ```sql
CREATE TABLE pf.col_meta ( CREATE TABLE pf.col_meta (
id serial PRIMARY KEY, id serial PRIMARY KEY,
source_id integer REFERENCES pf.source(id), source_id integer REFERENCES pf.source(id),
cname text NOT NULL, -- column name in source table cname text NOT NULL, -- column name in source table
label text, -- friendly display name label text, -- friendly display name
role text NOT NULL, -- 'dimension' | 'value' | 'units' | 'date' | 'ignore' role text NOT NULL, -- 'dimension' | 'value' | 'units' | 'date' | 'filter' | 'ignore'
is_key boolean DEFAULT false, -- true = part of natural key (used in WHERE slice) is_key boolean DEFAULT false, -- true = part of natural key (used in WHERE slice)
opos integer, -- ordinal position (for ordering) opos integer, -- ordinal position (for ordering)
dim_group text, -- groups functionally dependent columns (see below)
dim_period_col text, -- maps this dimension to a pf.dim_period column
UNIQUE (source_id, cname) UNIQUE (source_id, cname)
); );
``` ```
**Roles:** **Roles:**
- `dimension` — categorical field (customer, part, channel, rep, geography, etc.) — appears as pivot rows/cols, used in WHERE filters for operations - `dimension` — categorical field (customer, part, channel, rep, geography, etc.) — appears as pivot rows/cols, used in WHERE filters for operations
- `value` — the money/revenue field to scale - `value` — the money/revenue field to scale (**required** — SQL generation fails without it)
- `units` — the quantity field to scale - `units` — the quantity field to scale (**optional** — if absent, units columns are omitted from the forecast table and all SQL patterns)
- `date` — the primary date field; used for baseline/reference date range and stored in the forecast table - `date` — the primary date field; used for baseline/reference date range and stored in the forecast table (**required**)
- `filter` — columns available as filter conditions in the Baseline Workbench (e.g. order status, ship date, open flag); used in baseline WHERE clauses but **not stored** in the forecast table - `filter` — columns available as filter conditions in the Baseline Workbench (e.g. order status, ship date, open flag); used in baseline WHERE clauses but **not stored** in the forecast table
- `ignore` — exclude from forecast table entirely - `ignore` — exclude from forecast table entirely
**`dim_group`** — a free-text group name linking a `date` column to its derived dimension siblings. When the `date` column has `is_key = true` and a `dim_group` value, the SQL generator looks for `dimension` columns in the same group that also have a `dim_period_col` value. Those columns are sourced from `pf.dim_period` on baseline/reference load (via a JOIN on `drange @> date`) rather than copied raw from the source table. This allows fiscal year, quarter, and month columns to be stored in the forecast table with calendar-correct values even if those columns don't exist in the source.
**`dim_period_col`** — names the column in `pf.dim_period` to use as the value for this dimension on load. Only meaningful when the column is in a `dim_group` whose `date` key has `is_key = true`. Example: `cal_year`, `fisc_quarter`, `fisc_label`.
### `pf.dim_period`
Calendar lookup table. One row per month from 2018-01-01 through 2035-12-01. Keyed on `sdat` (month start date). Used to derive fiscal/calendar period columns at baseline load time when `dim_group` / `dim_period_col` are configured on col_meta.
Populated by `setup_sql/gen_dim_period.sql` (safe to re-run; `ON CONFLICT DO NOTHING`). Fiscal year start month is configurable at the top of that script (default: June, i.e. fiscal month 1 = June).
Key columns: `sdat`, `edat`, `drange` (GiST-indexed daterange), `cal_year`, `cal_quarter`, `cal_month`, `cal_month_abbr`, `cal_month_name`, `cal_label`, `fisc_year`, `fisc_quarter`, `fisc_quarter_label`, `fisc_month`, `fisc_month_abbr`, `fisc_month_name`, `fisc_label`, `period_key`.
The baseline/reference SQL JOINs this table when `hasDimPeriod` is true: `JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`.
### `pf.version` ### `pf.version`
Named forecast scenarios. One forecast table (`pf.fc_{tname}_{version_id}`) is created per version. Named forecast scenarios. One forecast table (`pf.fc_{tname}_{version_id}`) is created per version.
@ -102,31 +118,31 @@ CREATE TABLE pf.log (
``` ```
### `pf.fc_{tname}_{version_id}` (dynamic, one per version) ### `pf.fc_{tname}_{version_id}` (dynamic, one per version)
Created when a version is created. Mirrors source table dimension/value/units/date columns plus forecast metadata. Contains both operational rows (`iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`iter = 'reference'`). Created when a version is created. Mirrors source table dimension/value/date columns (and units if configured) plus any `dim_period_col`-derived dimension columns, plus forecast metadata. Contains both operational rows (`pf_iter = 'baseline' | 'scale' | 'recode' | 'clone'`) and reference rows (`pf_iter = 'reference'`).
```sql ```sql
-- Example: source table "sales", version id 3 → pf.fc_sales_3 -- Example: source table "sales", version id 3 → pf.fc_sales_3
CREATE TABLE pf.fc_sales_3 ( CREATE TABLE pf.fc_sales_3 (
id bigserial PRIMARY KEY, id bigserial PRIMARY KEY,
-- mirrored from source (role = dimension | value | units | date only): -- mirrored from source (role = dimension | value | units | date only):
customer text, customer text,
channel text, channel text,
part text, part text,
geography text, geography text,
order_date date, order_date date,
units numeric, value numeric,
value numeric, units numeric, -- omitted if no 'units' role in col_meta
-- forecast metadata: -- forecast metadata:
iter text, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone' pf_iter text, -- 'baseline' | 'reference' | 'scale' | 'recode' | 'clone'
logid bigint REFERENCES pf.log(id), pf_logid bigint REFERENCES pf.log(id),
pf_user text, pf_user text,
created_at timestamptz DEFAULT now() pf_created_at timestamptz DEFAULT now()
); );
``` ```
Note: no `version_id` column on the forecast table — it's implied by the table itself. Note: no `version_id` column on the forecast table — it's implied by the table itself. The `units` column is only present when a column with `role = 'units'` exists in col_meta.
### `pf.sql` ### `pf.sql`
Generated SQL stored per source and operation. Built once when col_meta is finalized, fetched at request time. Generated SQL stored per source and operation. Built once when col_meta is finalized, fetched at request time.
@ -148,7 +164,7 @@ CREATE TABLE pf.sql (
|-------|--------------| |-------|--------------|
| `{{fc_table}}` | `pf.fc_{tname}_{version_id}` — derived at request time | | `{{fc_table}}` | `pf.fc_{tname}_{version_id}` — derived at request time |
| `{{where_clause}}` | built from `slice` JSON by `build_where()` in JS | | `{{where_clause}}` | built from `slice` JSON by `build_where()` in JS |
| `{{exclude_clause}}` | built from `version.exclude_iters` — e.g. `AND iter NOT IN ('reference')` | | `{{exclude_clause}}` | built from `version.exclude_iters` — e.g. `AND pf_iter NOT IN ('reference')` |
| `{{logid}}` | newly inserted `pf.log` id | | `{{logid}}` | newly inserted `pf.log` id |
| `{{pf_user}}` | from request body | | `{{pf_user}}` | from request body |
| `{{date_from}}` / `{{date_to}}` | baseline/reference date range (source period) | | `{{date_from}}` / `{{date_to}}` | baseline/reference date range (source period) |
@ -521,7 +537,11 @@ AG Grid list of log entries — user, timestamp, operation, slice, note, rows af
## Forecast SQL Patterns ## Forecast SQL Patterns
Column names baked in at generation time. Tokens substituted at request time. Column names baked in at generation time. Tokens substituted at request time. Metadata columns are `pf_iter`, `pf_logid`, `pf_user`, `pf_created_at`.
**Units conditionality:** `{units_col}` appears in INSERT column lists and SELECT expressions only when a `units` role is configured in col_meta. The SQL generator omits it entirely otherwise — no placeholder column, no zero-fill.
**dim_period JOIN:** when any `dimension` column has `dim_period_col` set (and its group's `date` key has `is_key = true`), the FROM clause becomes `{schema}.{tname} s JOIN pf.dim_period dp ON dp.drange @> (s.{date_col} + '{{date_offset}}'::interval)::date`. Those dimension columns are selected as `dp.{dim_period_col} AS {col}` instead of `s.{col}`.
### Baseline Load (one segment) ### Baseline Load (one segment)
@ -531,18 +551,23 @@ WITH ilog AS (
VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}') VALUES ({{version_id}}, '{{pf_user}}', 'baseline', NULL, '{{params}}'::jsonb, '{{note}}')
RETURNING id RETURNING id
) )
INSERT INTO {{fc_table}} ( ,ins AS (
{dimension_cols}, {value_col}, {units_col}, {date_col}, INSERT INTO {{fc_table}} (
iter, logid, pf_user, created_at {dimension_cols}, {date_col}, {value_col} [, {units_col}],
pf_iter, pf_logid, pf_user, pf_created_at
)
SELECT
{dimension_cols},
({date_col} + '{{date_offset}}'::interval)::date,
{value_col} [, {units_col}],
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM
{schema}.{tname} -- or with dim_period JOIN (see above)
WHERE
{{filter_clause}}
RETURNING *
) )
SELECT SELECT count(*) AS rows_affected FROM ins
{dimension_cols}, {value_col}, {units_col},
({date_col} + '{{date_offset}}'::interval)::date,
'baseline', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM
{schema}.{tname}
WHERE
{{filter_clause}}
``` ```
Baseline loads are **additive** — no DELETE before INSERT. Each segment appends independently. Baseline loads are **additive** — no DELETE before INSERT. Each segment appends independently.
@ -557,7 +582,7 @@ Token details:
Two queries, run in a transaction: Two queries, run in a transaction:
```sql ```sql
DELETE FROM {{fc_table}} WHERE iter = 'baseline'; DELETE FROM {{fc_table}} WHERE pf_iter = 'baseline';
DELETE FROM pf.log WHERE version_id = {{version_id}} AND operation = 'baseline'; DELETE FROM pf.log WHERE version_id = {{version_id}} AND operation = 'baseline';
``` ```
@ -569,20 +594,24 @@ WITH ilog AS (
VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}') VALUES ({{version_id}}, '{{pf_user}}', 'reference', NULL, '{{params}}'::jsonb, '{{note}}')
RETURNING id RETURNING id
) )
INSERT INTO {{fc_table}} ( ,ins AS (
{dimension_cols}, {value_col}, {units_col}, {date_col}, INSERT INTO {{fc_table}} (
iter, logid, pf_user, created_at {dimension_cols}, {date_col}, {value_col} [, {units_col}],
pf_iter, pf_logid, pf_user, pf_created_at
)
SELECT
{dimension_cols}, {date_col}, {value_col} [, {units_col}],
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM
{schema}.{tname} -- or with dim_period JOIN (see above)
WHERE
{{filter_clause}}
RETURNING *
) )
SELECT SELECT count(*) AS rows_affected FROM ins
{dimension_cols}, {value_col}, {units_col}, {date_col},
'reference', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM
{schema}.{tname}
WHERE
{date_col} BETWEEN '{{date_from}}' AND '{{date_to}}'
``` ```
No date offset — reference rows land at their original dates for prior-period comparison. No date offset applied — reference rows land at their original dates for prior-period comparison. Same dim_period JOIN logic applies as baseline.
### Scale ### Scale
@ -595,26 +624,30 @@ WITH ilog AS (
,base AS ( ,base AS (
SELECT SELECT
{dimension_cols}, {date_col}, {dimension_cols}, {date_col},
{value_col}, {units_col}, {value_col} [, {units_col}],
sum({value_col}) OVER () AS total_value, sum({value_col}) OVER () AS total_value
sum({units_col}) OVER () AS total_units [, sum({units_col}) OVER () AS total_units]
FROM {{fc_table}} FROM {{fc_table}}
WHERE {{where_clause}} WHERE {{where_clause}}
{{exclude_clause}} {{exclude_clause}}
) )
INSERT INTO {{fc_table}} ( ,ins AS (
{dimension_cols}, {date_col}, {value_col}, {units_col}, INSERT INTO {{fc_table}} (
iter, logid, pf_user, created_at {dimension_cols}, {date_col}, {value_col} [, {units_col}],
pf_iter, pf_logid, pf_user, pf_created_at
)
SELECT
{dimension_cols}, {date_col},
round(({value_col} / NULLIF(total_value, 0)) * {{value_incr}}, 2)
[, round(({units_col} / NULLIF(total_units, 0)) * {{units_incr}}, 5)],
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM base
RETURNING *
) )
SELECT SELECT * FROM ins
{dimension_cols}, {date_col},
round(({value_col} / NULLIF(total_value, 0)) * {{value_incr}}, 2),
round(({units_col} / NULLIF(total_units, 0)) * {{units_incr}}, 5),
'scale', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM base
``` ```
`{{value_incr}}` / `{{units_incr}}` are pre-computed in JS when `pct: true` (multiply slice total by pct). `{{value_incr}}` / `{{units_incr}}` are pre-computed in JS when `pct: true` (multiply slice total by pct). Units expressions are omitted when no units column is configured.
### Recode ### Recode
@ -625,22 +658,27 @@ WITH ilog AS (
RETURNING id RETURNING id
) )
,src AS ( ,src AS (
SELECT {dimension_cols}, {date_col}, {value_col}, {units_col} SELECT {dimension_cols}, {date_col}, {value_col} [, {units_col}]
FROM {{fc_table}} FROM {{fc_table}}
WHERE {{where_clause}} WHERE {{where_clause}}
{{exclude_clause}} {{exclude_clause}}
) )
,negatives AS ( ,neg AS (
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at) INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
SELECT {dimension_cols}, {date_col}, -{value_col}, -{units_col}, 'recode', (SELECT id FROM ilog), '{{pf_user}}', now() SELECT {dimension_cols}, {date_col}, -{value_col} [, -{units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src FROM src
RETURNING *
) )
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at) ,ins AS (
SELECT {{set_clause}}, {date_col}, {value_col}, {units_col}, 'recode', (SELECT id FROM ilog), '{{pf_user}}', now() INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
FROM src SELECT {{set_clause}}, {date_col}, {value_col} [, {units_col}], 'recode', (SELECT id FROM ilog), '{{pf_user}}', now()
FROM src
RETURNING *
)
SELECT * FROM neg UNION ALL SELECT * FROM ins
``` ```
`{{set_clause}}` replaces the listed dimension columns with new values, passes others through unchanged. `{{set_clause}}` replaces the listed dimension columns with new values, passes others through unchanged. Both the negative (zero-out) and positive (replacement) rows share the same `pf_logid` and are undone together.
### Clone ### Clone
@ -650,21 +688,27 @@ WITH ilog AS (
VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}') VALUES ({{version_id}}, '{{pf_user}}', 'clone', '{{slice}}'::jsonb, '{{params}}'::jsonb, '{{note}}')
RETURNING id RETURNING id
) )
INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col}, {units_col}, iter, logid, pf_user, created_at) ,ins AS (
SELECT INSERT INTO {{fc_table}} ({dimension_cols}, {date_col}, {value_col} [, {units_col}], pf_iter, pf_logid, pf_user, pf_created_at)
{{set_clause}}, {date_col}, SELECT
round({value_col} * {{scale_factor}}, 2), {{set_clause}}, {date_col},
round({units_col} * {{scale_factor}}, 5), round({value_col} * {{scale_factor}}, 2)
'clone', (SELECT id FROM ilog), '{{pf_user}}', now() [, round({units_col} * {{scale_factor}}, 5)],
FROM {{fc_table}} 'clone', (SELECT id FROM ilog), '{{pf_user}}', now()
WHERE {{where_clause}} FROM {{fc_table}}
{{exclude_clause}} WHERE {{where_clause}}
{{exclude_clause}}
RETURNING *
)
SELECT * FROM ins
``` ```
### Undo ### Undo
Two queries run sequentially (not in a CTE — FK ordering):
```sql ```sql
DELETE FROM {{fc_table}} WHERE logid = {{logid}}; DELETE FROM {{fc_table}} WHERE pf_logid = {{logid}};
DELETE FROM pf.log WHERE id = {{logid}}; DELETE FROM pf.log WHERE id = {{logid}};
``` ```
@ -698,38 +742,38 @@ DELETE FROM pf.log WHERE id = {{logid}};
## Open Questions / Future Scope ## Open Questions / Future Scope
- **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501 - **Baseline replay** — re-execute change log against a restated baseline (`replay: true`); v1 returns 501
- **Arrow IPC for initial data load** — at large row counts (1M+) the `/versions/:id/data` JSON response becomes a bottleneck. Option: serve Arrow IPC binary instead of JSON; Perspective's `worker.table()` accepts Arrow buffers natively. Incremental operation rows (scale/recode/clone) can stay as JSON fed to `table.update()` since they're always small. Could be implemented with `pg` + `apache-arrow` in Node, or by adding a server-side DuckDB instance (Postgres scanner → Arrow IPC) if a caching layer is also needed.
- **Approval workflow** — user submits, admin approves before changes are visible to others (deferred) - **Approval workflow** — user submits, admin approves before changes are visible to others (deferred)
- **Territory filtering** — restrict what a user can see/edit by dimension value (deferred) - **Territory filtering** — restrict what a user can see/edit by dimension value (deferred)
- **Export** — download forecast as CSV or push results to a reporting table - **Export** — download forecast as CSV or push results to a reporting table
- **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION) - **Version comparison** — side-by-side view of two versions (facilitated by isolated tables via UNION)
- **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync (e.g. a column added to SQL that doesn't exist in the table). UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). For now the workaround is to delete and recreate the version manually. - **Col meta / version schema drift** — if col_meta roles are changed after a version's forecast table is already created, the generated SQL and the table DDL go out of sync. UI should detect this: compare col_meta against the forecast table's actual columns via `information_schema`, warn the user, and offer to rebuild the version (drop + recreate table, preserving the version record and log). Workaround: delete and recreate the version manually.
- **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema. - **Multi-connection support** — currently one DB via `.env`. Full vision: `pf.connection` table (host, port, dbname, user, password as env-var ref), `connection_id` on `pf.source`, per-connection pg pools at runtime. `pf` schema stays on a "home" connection; source data can live anywhere. Connections UI in Setup. Safe to defer while in dev — requires clean reinstall when added since it changes the source schema.
--- ---
## Project Status — 2026-04-25 ## Project Status — 2026-06-12
### What's working ### What's working
- Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo - Full backend: source registration, col_meta, SQL generation, versions, baseline segments, reference load, scale, recode, clone, undo
- React + Vite + Tailwind CSS frontend scaffolded in `ui/`, built output to `public/app/`, served by Express - `units` column is optional — sources without a units column register and generate SQL correctly
- 3-step collapsible sidebar (Setup / Baseline / Forecast) — addresses prior UX concern about opaque 5-tab nav - `dim_group` / `dim_period_col` on col_meta: baseline/reference load JOINs `pf.dim_period` to derive fiscal/calendar period columns rather than copying them raw from the source
- Setup view: DB table browser with preview modal, source registration, col_meta editor, SQL generation - `pf.dim_period` calendar table (2018–2035): populated by `setup_sql/gen_dim_period.sql`, configurable fiscal year start
- React + Vite + Tailwind CSS frontend in `ui/`, built output to `public/app/`, served by Express
- Data transport: Arrow IPC binary stream (`GET /api/versions/:id/data`); server accumulates all rows into one record batch; client hands buffer directly to Perspective WASM
- 3-step collapsible sidebar (Setup / Baseline / Forecast)
- Setup view: DB table browser with preview modal, source registration, col_meta editor (`dim_group`/`dim_period_col` fields included), SQL generation
- Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder - Baseline view: version management (create/close/reopen/delete), multi-segment baseline workbench, canvas timeline, filter builder
- Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version - Perspective pivot in Forecast view: loads all version rows, interactive group/split/filter/chart, layout saved per version to localStorage
- Slice extraction from `perspective-click` event feeds operation panel directly - Slice extraction from `perspective-click` event feeds operation panel directly
- Incremental row streaming: operation results (`RETURNING *`) stream into Perspective table without full reload - Incremental row streaming: operation results (`RETURNING *`) applied to Perspective table via `pspTable.update()` — no full reload
- Status bar: shows current source · version · baseline row count · status - Status bar: shows current source · version · baseline row count · status
### Known issues / next focus ### Known issues / next focus
- **Forecast view** — operation panel (Scale / Recode / Clone) is a stub; needs wiring to API - **Forecast view** — operation panel SQL generation complete; UI wiring to API still needed
- **Status bar** — currently hardcoded; needs to reflect actual selected source/version from state - **Load progress bar** — jittery at high throughput; throttle to ~10 updates/sec
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, the SQL and table DDL go out of sync. UI should detect this (compare col_meta against `information_schema`), warn, and offer rebuild. Workaround: delete and recreate the version. - **Default pivot layout** — per-source configurable layout not yet implemented; currently hardcodes first 2 dimensions
- **No "current version" persistence** — source/version selection resets on page reload; session context not persisted - **No "current version" persistence** — source/version selection resets on page reload
- **Perspective slice limitation** — computed date columns (Month, YearDate) extracted via split_by don't filter back to raw rows; only native dimension columns work for slice extraction - **Perspective slice limitation** — computed date columns (Month, YearDate) from split_by don't map back to raw rows; only native dimension columns work for slice extraction
- **Col_meta / version schema drift** — if col_meta changes after a version's forecast table is created, SQL and DDL go out of sync. Workaround: delete and recreate the version.
### Branch status
- `baseline-workbench` — merged to origin, stable
- `perspective-forecast` — active development branch; React UI scaffolded, Forecast operation panel pending

40
routes/log.js
View File

@ -4,11 +4,31 @@ const { fcTable } = require('../lib/utils');
module.exports = function(pool) { module.exports = function(pool) {
const router = express.Router(); const router = express.Router();
// list all log entries for a version, newest first // list all log entries for a version, newest first, with row counts from fc_table
router.get('/versions/:id/log', async (req, res) => { router.get('/versions/:id/log', async (req, res) => {
try { try {
const verResult = await pool.query(`
SELECT v.*, s.tname
FROM pf.version v
JOIN pf.source s ON s.id = v.source_id
WHERE v.id = $1
`, [req.params.id]);
if (verResult.rows.length === 0) return res.status(404).json({ error: 'Version not found' });
const { tname, id: version_id } = verResult.rows[0];
const table = fcTable(tname, version_id);
const result = await pool.query( const result = await pool.query(
`SELECT * FROM pf.log WHERE version_id = $1 ORDER BY stamp DESC`, `SELECT l.*,
counts.row_count
FROM pf.log l
LEFT JOIN (
SELECT pf_logid, count(*)::int AS row_count
FROM ${table}
GROUP BY pf_logid
) counts ON counts.pf_logid = l.id
WHERE l.version_id = $1
ORDER BY l.stamp DESC`,
[req.params.id] [req.params.id]
); );
res.json(result.rows); res.json(result.rows);
@ -18,6 +38,22 @@ module.exports = function(pool) {
} }
}); });
// update note on a log entry
router.patch('/log/:logid', async (req, res) => {
const { note } = req.body;
try {
const result = await pool.query(
`UPDATE pf.log SET note = $1 WHERE id = $2 RETURNING *`,
[note ?? null, parseInt(req.params.logid)]
);
if (result.rows.length === 0) return res.status(404).json({ error: 'Log entry not found' });
res.json(result.rows[0]);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// undo an operation — deletes all forecast rows with this logid, then the log entry // undo an operation — deletes all forecast rows with this logid, then the log entry
// two separate queries in a transaction to avoid FK ordering issues // two separate queries in a transaction to avoid FK ordering issues
router.delete('/log/:logid', async (req, res) => { router.delete('/log/:logid', async (req, res) => {

9
routes/operations.js
View File

@ -31,6 +31,7 @@ module.exports = function(pool) {
); );
const colMeta = colResult.rows; const colMeta = colResult.rows;
const dimCols = colMeta.filter(c => c.role === 'dimension').map(c => c.cname); const dimCols = colMeta.filter(c => c.role === 'dimension').map(c => c.cname);
const dateCols = colMeta.filter(c => c.role === 'date').map(c => c.cname);
const valueCol = colMeta.find(c => c.role === 'value')?.cname; const valueCol = colMeta.find(c => c.role === 'value')?.cname;
const unitsCol = colMeta.find(c => c.role === 'units')?.cname; const unitsCol = colMeta.find(c => c.role === 'units')?.cname;
@ -48,6 +49,8 @@ module.exports = function(pool) {
table: fcTable(version.tname, version.id), table: fcTable(version.tname, version.id),
colMeta, colMeta,
dimCols, dimCols,
dateCols,
filterCols: [...dimCols, ...dateCols],
valueCol, valueCol,
unitsCol, unitsCol,
sql: sqlResult.rows[0].sql sql: sqlResult.rows[0].sql
@ -306,7 +309,7 @@ module.exports = function(pool) {
const ctx = await getContext(parseInt(req.params.id), 'scale'); const ctx = await getContext(parseInt(req.params.id), 'scale');
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
const whereClause = buildWhere(slice, ctx.dimCols); const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters); const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
let absValueIncr = value_incr || 0; let absValueIncr = value_incr || 0;
@ -364,7 +367,7 @@ module.exports = function(pool) {
const ctx = await getContext(parseInt(req.params.id), 'recode'); const ctx = await getContext(parseInt(req.params.id), 'recode');
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
const whereClause = buildWhere(slice, ctx.dimCols); const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters); const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set); const setClause = buildSetClause(ctx.dimCols, set);
@ -401,7 +404,7 @@ module.exports = function(pool) {
if (!guardOpen(ctx.version, res)) return; if (!guardOpen(ctx.version, res)) return;
const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0; const scaleFactor = (scale != null) ? parseFloat(scale) : 1.0;
const whereClause = buildWhere(slice, ctx.dimCols); const whereClause = buildWhere(slice, ctx.filterCols);
const excludeClause = buildExcludeClause(ctx.version.exclude_iters); const excludeClause = buildExcludeClause(ctx.version.exclude_iters);
const setClause = buildSetClause(ctx.dimCols, set); const setClause = buildSetClause(ctx.dimCols, set);

80
routes/sources.js
View File

@ -89,20 +89,24 @@ module.exports = function(pool) {
await client.query('BEGIN'); await client.query('BEGIN');
for (const col of cols) { for (const col of cols) {
await client.query(` await client.query(`
INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, opos) INSERT INTO pf.col_meta (source_id, cname, label, role, is_key, dim_group, dim_period_col, opos)
VALUES ($1, $2, $3, $4, $5, $6) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (source_id, cname) DO UPDATE SET ON CONFLICT (source_id, cname) DO UPDATE SET
label = EXCLUDED.label, label = EXCLUDED.label,
role = EXCLUDED.role, role = EXCLUDED.role,
is_key = EXCLUDED.is_key, is_key = EXCLUDED.is_key,
opos = EXCLUDED.opos dim_group = EXCLUDED.dim_group,
dim_period_col = EXCLUDED.dim_period_col,
opos = EXCLUDED.opos
`, [ `, [
sourceId, sourceId,
col.cname, col.cname,
col.label || null, col.label || null,
col.role || 'ignore', col.role || 'ignore',
col.is_key || false, col.is_key || false,
col.opos || null col.dim_group || null,
col.dim_period_col || null,
col.opos || null
]); ]);
} }
await client.query('COMMIT'); await client.query('COMMIT');
@ -139,7 +143,7 @@ module.exports = function(pool) {
// validate required roles // validate required roles
const colMeta = colResult.rows; const colMeta = colResult.rows;
const roles = new Set(colMeta.map(c => c.role)); const roles = new Set(colMeta.map(c => c.role));
const missing = ['value', 'units', 'date'].filter(r => !roles.has(r)); const missing = ['value', 'date'].filter(r => !roles.has(r));
if (missing.length > 0) { if (missing.length > 0) {
return res.status(400).json({ return res.status(400).json({
error: `col_meta is missing required roles: ${missing.join(', ')}` error: `col_meta is missing required roles: ${missing.join(', ')}`
@ -222,6 +226,41 @@ module.exports = function(pool) {
} }
}); });
// given a key column value, look up sibling dim_group column values from source
// returns { sibling_col: value, ... } if exactly one match, null if none or ambiguous
router.get('/sources/:id/lookup', async (req, res) => {
const { col, value } = req.query;
if (!col || value == null || value === '') return res.json(null);
try {
const [srcResult, metaResult] = await Promise.all([
pool.query(`SELECT schema, tname FROM pf.source WHERE id = $1`, [req.params.id]),
pool.query(`SELECT * FROM pf.col_meta WHERE source_id = $1 ORDER BY opos`, [req.params.id])
]);
if (srcResult.rows.length === 0) return res.status(404).json({ error: 'Source not found' });
const keyCol = metaResult.rows.find(c => c.cname === col && c.is_key && c.dim_group);
if (!keyCol) return res.json(null);
const siblings = metaResult.rows.filter(c =>
c.dim_group === keyCol.dim_group && c.cname !== col
);
if (!siblings.length) return res.json(null);
const { schema, tname } = srcResult.rows[0];
const sibCols = siblings.map(c => `"${c.cname}"`).join(', ');
const result = await pool.query(
`SELECT DISTINCT ${sibCols} FROM "${schema}"."${tname}" WHERE "${col}" = $1 LIMIT 2`,
[value]
);
if (result.rows.length !== 1) return res.json(null);
res.json(result.rows[0]);
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
// set or clear the default Perspective layout for a source. // set or clear the default Perspective layout for a source.
// Body: a Perspective view config (group_by, split_by, columns, plugin_config, …). // Body: a Perspective view config (group_by, split_by, columns, plugin_config, …).
// Pass null or {} to clear. // Pass null or {} to clear.
@ -241,6 +280,22 @@ module.exports = function(pool) {
}); });
// deregister a source — does not drop existing forecast tables // deregister a source — does not drop existing forecast tables
router.get('/dim-period/cols', async (req, res) => {
try {
const result = await pool.query(`
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'pf' AND table_name = 'dim_period'
AND column_name NOT IN ('sdat', 'edat', 'drange', 'ndays')
ORDER BY ordinal_position
`);
res.json(result.rows.map(r => r.column_name));
} catch (err) {
console.error(err);
res.status(500).json({ error: err.message });
}
});
router.delete('/sources/:id', async (req, res) => { router.delete('/sources/:id', async (req, res) => {
try { try {
const result = await pool.query( const result = await pool.query(
@ -253,6 +308,9 @@ module.exports = function(pool) {
res.json({ message: 'Source deregistered', source: result.rows[0] }); res.json({ message: 'Source deregistered', source: result.rows[0] });
} catch (err) { } catch (err) {
console.error(err); console.error(err);
if (err.code === '23001' || err.code === '23503') {
return res.status(409).json({ error: 'Source has existing versions — delete them first.' });
}
res.status(500).json({ error: err.message }); res.status(500).json({ error: err.message });
} }
}); });

6
setup_sql/01_schema.sql
View File

@ -15,8 +15,12 @@ CREATE TABLE IF NOT EXISTS pf.source (
UNIQUE (schema, tname) UNIQUE (schema, tname)
); );
-- backfill column for existing installs -- backfill columns for existing installs
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb; ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
CREATE TABLE IF NOT EXISTS pf.col_meta ( CREATE TABLE IF NOT EXISTS pf.col_meta (
id serial PRIMARY KEY, id serial PRIMARY KEY,

94
setup_sql/gen_dim_period.sql Normal file
View File

@ -0,0 +1,94 @@
-- pf.dim_period — create and populate
-- Adjust fiscal_start_month: 1=Jan (calendar year), 4=Apr, 6=Jun, 7=Jul, 10=Oct, etc.
-- Safe to re-run: ON CONFLICT DO NOTHING, so existing rows are never overwritten.
CREATE TABLE IF NOT EXISTS pf.dim_period (
sdat date NOT NULL PRIMARY KEY,
edat date NOT NULL,
drange daterange NOT NULL,
ndays integer NOT NULL,
-- calendar
cal_year integer NOT NULL,
cal_quarter integer NOT NULL,
cal_month integer NOT NULL,
cal_month_abbr text NOT NULL, -- 01 - Jan
cal_month_name text NOT NULL, -- 01 - January
cal_label text NOT NULL, -- 2025-01 Jan
-- fiscal
fisc_year integer NOT NULL,
fisc_quarter integer NOT NULL,
fisc_quarter_label text NOT NULL, -- FY2025 Q3
fisc_month integer NOT NULL,
fisc_month_abbr text NOT NULL, -- 07 - Jan
fisc_month_name text NOT NULL, -- 07 - January
fisc_label text NOT NULL, -- FY2025 P07
-- sort key
period_key text NOT NULL -- 2025.07 (ltree-compatible)
);
CREATE INDEX IF NOT EXISTS dim_period_drange_idx ON pf.dim_period USING gist (drange);
CREATE INDEX IF NOT EXISTS dim_period_fisc_idx ON pf.dim_period (fisc_year, fisc_month);
CREATE INDEX IF NOT EXISTS dim_period_cal_idx ON pf.dim_period (cal_year, cal_month);
WITH
cfg AS (
SELECT 6 AS fiscal_start_month -- change to match your fiscal year start month
)
,periods AS (
SELECT
gs.d::date AS sdat,
(gs.d + '1 month'::interval)::date AS edat,
extract(year FROM gs.d)::int AS cal_year,
extract(month FROM gs.d)::int AS cal_month,
extract(quarter FROM gs.d)::int AS cal_quarter,
((extract(month FROM gs.d)::int - cfg.fiscal_start_month + 12) % 12) + 1
AS fisc_month,
extract(year FROM gs.d)::int
+ CASE
WHEN cfg.fiscal_start_month > 1
AND extract(month FROM gs.d)::int >= cfg.fiscal_start_month
THEN 1 ELSE 0
END AS fisc_year
FROM
generate_series('2018-01-01'::date, '2035-12-01'::date, '1 month') gs(d)
CROSS JOIN cfg
)
INSERT INTO pf.dim_period (
sdat, edat, drange, ndays,
cal_year, cal_quarter, cal_month,
cal_month_abbr, cal_month_name, cal_label,
fisc_year, fisc_quarter, fisc_quarter_label,
fisc_month, fisc_month_abbr, fisc_month_name,
fisc_label, period_key
)
SELECT
sdat,
edat,
daterange(sdat, edat) AS drange,
edat - sdat AS ndays,
cal_year,
cal_quarter,
cal_month,
to_char(cal_month, 'FM00') || ' - ' || to_char(sdat, 'Mon') AS cal_month_abbr,
to_char(cal_month, 'FM00') || ' - ' || to_char(sdat, 'Month') AS cal_month_name,
to_char(sdat, 'YYYY-MM') || ' ' || to_char(sdat, 'Mon') AS cal_label,
fisc_year,
ceil(fisc_month / 3.0)::int AS fisc_quarter,
'FY' || fisc_year || ' Q' || ceil(fisc_month / 3.0)::int AS fisc_quarter_label,
fisc_month,
to_char(fisc_month, 'FM00') || ' - ' || to_char(sdat, 'Mon') AS fisc_month_abbr,
to_char(fisc_month, 'FM00') || ' - ' || to_char(sdat, 'Month') AS fisc_month_name,
'FY' || fisc_year || ' P' || to_char(fisc_month, 'FM00') AS fisc_label,
to_char(fisc_year, 'FM0000') || '.' || to_char(fisc_month, 'FM00') AS period_key
FROM periods
ON CONFLICT (sdat) DO NOTHING;
-- preview first 24 months
SELECT
period_key, sdat, edat, ndays,
cal_year, cal_quarter, cal_month, cal_month_abbr,
fisc_year, fisc_quarter, fisc_month, fisc_month_abbr,
fisc_quarter_label, fisc_label, cal_label
FROM pf.dim_period
ORDER BY sdat
LIMIT 24;

113
todo.md
View File

@ -1,113 +0,0 @@
- [ ] when you enter the forecast, be able to enter in a context so you dont have to open the whole thing (should show in status bar and be a filter for SQL and spi calls)
- [x] should be able to edit and revise forecase segments that constitute baseline or reference. if you edit, maybe a warning that your forecast values wont mean a lot, and have an option to delete them.
Notes: A baseline/reference segment is a `pf.log` row plus the
forecast rows it produced (joined by pf_logid). Editing has the
shape of a delete-then-replay: drop the rows by pf_logid, drop the
log entry, re-run the segment with the new params (offset, filter,
iter type), insert the new log entry. New endpoint:
`PUT /versions/:id/baseline/:logid` (and the same for reference).
UI: an Edit button on each segment in Baseline view, populating the
form with the original `params`.
Cascade warning: if any scale/recode/clone log entries exist *after*
this segment was added, those operations were calibrated against
the old totals and will no longer reconcile cleanly. Show a banner
like "3 forecast operations applied after this segment may be
invalidated. View / Delete / Continue." Probably want a CASCADE
option that deletes downstream forecast entries too, plus a plain
"edit only" option for the user who knows what they're doing.
Implementation order: API + cascade detection first (compare
pf.log.stamp ordering); UI second.
- [~] be able to copy an existing forecast and it's segments to adjust some parameters without having to start from scrath.
Notes: A version is the unit of copy. Need a `POST /versions/:id/copy`
endpoint that creates a new pf.version row with the same source/
col_meta, creates the new fc_<tname>_<id> table via the same DDL
path, and replays each pf.log entry's INSERT against the new table
(preserving stamp ordering). Each log entry gets re-inserted
pointing at the new version_id; the new pf_logid feeds the row
inserts. Notes/users come along.
UI: "Copy" button next to each version in Baseline. Copy modal
asks for a new name and optional description, then runs the API
call (likely 5–30s for a 350k-row version since every segment is
re-evaluated). Show progress.
Two design questions worth deciding up front:
- Copy as-of-now (re-fetch source data, so freshly-arrived rows
show up in baseline)? Or freeze (replay from existing forecast
rows, i.e. clone the forecast table directly)? Different
semantics, different SQL — pick one before building.
- Should the copy track its origin? A `parent_version_id` column
on pf.version makes "show me variants of FY2026 Plan" easy.
- [x] need the list of filters to have an and/or specification
Notes: Spec already covers this in `pf_spec.md:245` — `filters` is
an array of groups; conditions within a group are AND-ed, groups
OR-ed. Backend has `buildFilterClause` in
`lib/sql_generator.js:247` but it's not wired into the routes
(baseline currently takes raw `where_clause`). Wiring + UI is the
remaining work.
UI: each group is a card with a header ("Group 1", "Group 2 — OR"),
rows of `column / operator / values`, a `+ Add condition` link,
and a `+ Add OR group` button at the bottom. The Baseline view
already has a single-group filter builder; extend it to wrap the
current rows in a group container and allow adding more groups.
- [x] the filters should have the option to just write the WHERE clause SQL
Notes: Spec covers this too (`pf_spec.md:251`, `:454`) as the
`raw_where` admin-only escape hatch. The current baseline endpoint
*already* takes `where_clause` as a raw string — so the API is
effectively in "raw only" mode today; it's the structured side
that's missing. Two things to add:
- Once structured `filters` is wired in, gate `raw_where` behind
an admin check (`pf_user` in admin list — needs admin list
config) and reject 400 if both are sent.
- UI toggle: a "Switch to manual SQL" link in the Baseline filter
builder swaps the structured rows for a `<textarea>`; warning
banner: "Raw SQL is not validated. You are responsible for
correctness and security."
- [ ] load status bar is super jittery and the numbers wildly change
Notes: `setLoadProgress` fires per chunk in the body-stream reader
(Forecast.jsx:155–161). On localhost or fast connections the
reader yields chunks in tight bursts and React re-renders the
overlay text on each — the visible bytes value flickers because
paints land out of order with respect to setState batching.
Fix: throttle to ~10 updates/sec. Either `if (now - lastUpdate >
100)` before `setLoadProgress`, or accumulate received bytes into
a ref and flush on `requestAnimationFrame`. Five-line change.
- [ ] default layout for the pivot should be sales_usd group by pending_rep, split by pf_iter
Notes: Default-layout logic lives in `initViewer` (Forecast.jsx
~line 240) and currently picks `group_by = first 2 dimensions`,
`split_by = date column`. `sales_usd` / `pending_rep` are
source-specific, so hardcoding them in the view would break for
any other source.
Two paths:
- **Quick**: hardcode for the current source. Cheap, but rots
the moment a second source comes along.
- **Right**: store a default-layout config on `pf.source` (e.g., a
JSON `default_view` column with `{ group_by, split_by, columns }`)
and let initViewer read it. Setup view gets a small "Default
pivot" editor — pick a value column, group_by columns, split_by
column.
Suggest the right version since the spec already implies
per-source customization in col_meta, and you're going to want
this for any future source you register. The col_meta path is
even tighter: extend col_meta with role flags like `default_value`,
`default_group`, `default_split` that initViewer reads.

16
ui/src/components/StatusBar.jsx
View File

@ -48,12 +48,20 @@ export default function StatusBar({ view, sources = [], sourceId, setSourceId, v
title={dark ? 'Switch to light mode' : 'Switch to dark mode'} title={dark ? 'Switch to light mode' : 'Switch to dark mode'}
> >
{dark ? ( {dark ? (
<svg width="14" height="14" viewBox="0 0 16 16" fill="currentColor"> <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M8 12a4 4 0 1 0 0-8 4 4 0 0 0 0 8zM8 0a.5.5 0 0 1 .5.5v2a.5.5 0 0 1-1 0v-2A.5.5 0 0 1 8 0zm0 13a.5.5 0 0 1 .5.5v2a.5.5 0 0 1-1 0v-2A.5.5 0 0 1 8 13zm8-5a.5.5 0 0 1-.5.5h-2a.5.5 0 0 1 0-1h2a.5.5 0 0 1 .5.5zM3 8a.5.5 0 0 1-.5.5h-2a.5.5 0 0 1 0-1h2A.5.5 0 0 1 3 8z"/> <circle cx="12" cy="12" r="4"/>
<line x1="12" y1="2" x2="12" y2="5"/>
<line x1="12" y1="19" x2="12" y2="22"/>
<line x1="4.93" y1="4.93" x2="7.05" y2="7.05"/>
<line x1="16.95" y1="16.95" x2="19.07" y2="19.07"/>
<line x1="2" y1="12" x2="5" y2="12"/>
<line x1="19" y1="12" x2="22" y2="12"/>
<line x1="4.93" y1="19.07" x2="7.05" y2="16.95"/>
<line x1="16.95" y1="7.05" x2="19.07" y2="4.93"/>
</svg> </svg>
) : ( ) : (
<svg width="14" height="14" viewBox="0 0 16 16" fill="currentColor"> <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M6 .278a.768.768 0 0 1 1.065.02A.75.75 0 0 1 5.792 15.5a.75.75 0 0 1-1.498-.075.768.768 0 0 1-.02-1.05A8 8 0 1 0 6.278 14.72a.768.768 0 0 1-1.055-.02A.75.75 0 0 1 2.5 13.75a.75.75 0 0 1 1.498.075A8 8 0 1 0 6 .278z"/> <path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
</svg> </svg>
)} )}
</button> </button>

1
ui/src/index.css
View File

@ -89,5 +89,6 @@ body { margin: 0; background-color: var(--bg-primary); color: var(--text-primary
.dark ::selection { background-color: var(--accent-bg); color: var(--text-primary); } .dark ::selection { background-color: var(--accent-bg); color: var(--text-primary); }
.dark input { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); } .dark input { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
.dark select { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); } .dark select { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
.dark select option { background-color: var(--bg-secondary); color: var(--text-primary); }
.dark textarea { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); } .dark textarea { background-color: var(--bg-secondary); color: var(--text-primary); border-color: var(--border-color); }
.dark .bg-transparent { background-color: transparent; } .dark .bg-transparent { background-color: transparent; }

105
ui/src/views/Forecast.jsx
View File

@ -113,11 +113,15 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
const unitsCol = colMetaRef.current.find(c => c.role === 'units')?.cname const unitsCol = colMetaRef.current.find(c => c.role === 'units')?.cname
if (!valueCol && !unitsCol) return if (!valueCol && !unitsCol) return
try { try {
const dimNames = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname)) const dimNames = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname))
const dateNames = new Set(colMetaRef.current.filter(c => c.role === 'date').map(c => c.cname))
const filters = [ const filters = [
...Object.entries(sliceObj) ...Object.entries(sliceObj)
.filter(([col]) => dimNames.has(col)) .filter(([col]) => dimNames.has(col))
.map(([col, val]) => [col, '==', val]), .map(([col, val]) => [col, '==', val]),
...Object.entries(sliceObj)
.filter(([col]) => dateNames.has(col))
.map(([col, val]) => [col, '==', Number(val)]),
['pf_iter', '!=', 'reference'], ['pf_iter', '!=', 'reference'],
] ]
const view = await tableRef.current.view({ filter: filters }) const view = await tableRef.current.view({ filter: filters })
@ -388,7 +392,9 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
async function submitOp(op) { async function submitOp(op) {
if (!Object.keys(slice).length) { flash('Select a slice first', 'error'); return } if (!Object.keys(slice).length) { flash('Select a slice first', 'error'); return }
let body = { pf_user: 'admin', slice } const effectiveSlice = buildEffectiveSlice(slice)
if (!Object.keys(effectiveSlice).length) { flash('No dimension or date columns in slice — check col_meta', 'error'); return }
let body = { pf_user: 'admin', slice: effectiveSlice }
if (op === 'scale') { if (op === 'scale') {
let vi = null, ui = null let vi = null, ui = null
@ -433,6 +439,65 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
} catch (err) { flash(err.message, 'error') } } catch (err) { flash(err.message, 'error') }
} }
async function lookupDerivedCols(col, value, setter) {
if (!sourceId || !value.trim()) return
const res = await fetch(`/api/sources/${sourceId}/lookup?col=${encodeURIComponent(col)}&value=${encodeURIComponent(value)}`)
if (!res.ok) return
const derived = await res.json()
if (!derived) return
setter(prev => {
const next = { ...prev }
for (const [k, v] of Object.entries(derived)) {
if (!prev[k] || prev[k] === '') next[k] = String(v ?? '')
}
return next
})
}
function buildEffectiveSlice(raw) {
const dimCols = new Set(colMetaRef.current.filter(c => c.role === 'dimension').map(c => c.cname))
const dateCols = new Set(colMetaRef.current.filter(c => c.role === 'date').map(c => c.cname))
const out = {}
for (const [k, v] of Object.entries(raw)) {
if (dimCols.has(k)) { out[k] = v; continue }
if (dateCols.has(k)) {
const ms = Number(v)
out[k] = isFinite(ms) ? new Date(ms).toISOString().slice(0, 10) : v
}
}
return out
}
function buildPayload(op) {
if (!Object.keys(slice).length) return null
const effectiveSlice = buildEffectiveSlice(slice)
let body = { pf_user: 'admin', slice: effectiveSlice }
if (op === 'scale') {
let vi = null, ui = null
if (scaleMode === 'target') {
const curValue = currentTotals?.total?.value
const curUnits = currentTotals?.total?.units
if (scalePrice !== '' && curUnits != null && curValue != null)
vi = (parseFloat(scalePrice) * curUnits) - curValue
if (scaleValue !== '' && curValue != null)
vi = parseFloat(scaleValue) - curValue
if (scaleUnits !== '' && curUnits != null)
ui = parseFloat(scaleUnits) - curUnits
} else {
if (scaleValue !== '') vi = parseFloat(scaleValue)
if (scaleUnits !== '') ui = parseFloat(scaleUnits)
}
body = { ...body, note: scaleNote || undefined, value_incr: vi, units_incr: ui, pct: scaleMode === 'delta' && scalePct }
} else if (op === 'recode') {
const set = Object.fromEntries(Object.entries(recodeSet).filter(([, v]) => v.trim()))
body = { ...body, note: recodeNote || undefined, set }
} else if (op === 'clone') {
const set = Object.fromEntries(Object.entries(cloneSet).filter(([, v]) => v.trim()))
body = { ...body, note: cloneNote || undefined, set, scale: parseFloat(cloneScale) || 1 }
}
return body
}
function flash(text, type = 'ok') { function flash(text, type = 'ok') {
setMsg({ text, type }) setMsg({ text, type })
if (type !== 'error') setTimeout(() => setMsg(null), 3000) if (type !== 'error') setTimeout(() => setMsg(null), 3000)
@ -791,31 +856,46 @@ export default function Forecast({ sources = [], sourceId, versionId, refreshSou
)} )}
<Row label="Note"><input value={scaleNote} onChange={e => setScaleNote(e.target.value)} placeholder="optional" className={inp} /></Row> <Row label="Note"><input value={scaleNote} onChange={e => setScaleNote(e.target.value)} placeholder="optional" className={inp} /></Row>
<PayloadPreview payload={buildPayload('scale')} />
<Submit onClick={() => submitOp('scale')}>Apply Scale</Submit> <Submit onClick={() => submitOp('scale')}>Apply Scale</Submit>
</>} </>}
{activeOp === 'recode' && <> {activeOp === 'recode' && <>
<p className="text-gray-400">New values for dimensions to replace. Leave blank to keep.</p> <p className="text-gray-400">New values for dimensions to replace. Leave blank to keep.</p>
{dimCols.map(c => ( {dimCols.map(c => (
<Row key={c.cname} label={c.cname}> <Row key={c.cname} label={c.label || c.cname}>
<input value={recodeSet[c.cname] || ''} onChange={e => setRecodeSet(s => ({ ...s, [c.cname]: e.target.value }))} <input
placeholder={slice[c.cname] || '—'} className={`${inp} font-mono`} /> value={recodeSet[c.cname] || ''}
onChange={e => setRecodeSet(s => ({ ...s, [c.cname]: e.target.value }))}
onBlur={c.is_key && c.dim_group
? e => lookupDerivedCols(c.cname, e.target.value, setRecodeSet)
: undefined}
placeholder={slice[c.cname] || '—'}
className={`${inp} font-mono`} />
</Row> </Row>
))} ))}
<Row label="Note"><input value={recodeNote} onChange={e => setRecodeNote(e.target.value)} placeholder="optional" className={inp} /></Row> <Row label="Note"><input value={recodeNote} onChange={e => setRecodeNote(e.target.value)} placeholder="optional" className={inp} /></Row>
<PayloadPreview payload={buildPayload('recode')} />
<Submit onClick={() => submitOp('recode')}>Apply Recode</Submit> <Submit onClick={() => submitOp('recode')}>Apply Recode</Submit>
</>} </>}
{activeOp === 'clone' && <> {activeOp === 'clone' && <>
<p className="text-gray-400">Override dimensions on cloned rows. Leave blank to keep.</p> <p className="text-gray-400">Override dimensions on cloned rows. Leave blank to keep.</p>
{dimCols.map(c => ( {dimCols.map(c => (
<Row key={c.cname} label={c.cname}> <Row key={c.cname} label={c.label || c.cname}>
<input value={cloneSet[c.cname] || ''} onChange={e => setCloneSet(s => ({ ...s, [c.cname]: e.target.value }))} <input
placeholder={slice[c.cname] || '—'} className={`${inp} font-mono`} /> value={cloneSet[c.cname] || ''}
onChange={e => setCloneSet(s => ({ ...s, [c.cname]: e.target.value }))}
onBlur={c.is_key && c.dim_group
? e => lookupDerivedCols(c.cname, e.target.value, setCloneSet)
: undefined}
placeholder={slice[c.cname] || '—'}
className={`${inp} font-mono`} />
</Row> </Row>
))} ))}
<Row label="Scale"><input type="number" step="any" value={cloneScale} onChange={e => setCloneScale(e.target.value)} className={inp} /></Row> <Row label="Scale"><input type="number" step="any" value={cloneScale} onChange={e => setCloneScale(e.target.value)} className={inp} /></Row>
<Row label="Note"><input value={cloneNote} onChange={e => setCloneNote(e.target.value)} placeholder="optional" className={inp} /></Row> <Row label="Note"><input value={cloneNote} onChange={e => setCloneNote(e.target.value)} placeholder="optional" className={inp} /></Row>
<PayloadPreview payload={buildPayload('clone')} />
<Submit onClick={() => submitOp('clone')}>Apply Clone</Submit> <Submit onClick={() => submitOp('clone')}>Apply Clone</Submit>
</>} </>}
</div> </div>
@ -874,3 +954,12 @@ function Submit({ onClick, children }) {
</button> </button>
) )
} }
function PayloadPreview({ payload }) {
if (!payload) return null
return (
<pre className="text-xs font-mono text-gray-400 bg-gray-50 border border-gray-100 rounded p-2 overflow-auto max-h-36 leading-relaxed">
{JSON.stringify(payload, null, 2)}
</pre>
)
}

72
ui/src/views/Setup.jsx
View File

@ -1,4 +1,5 @@
import { useState, useEffect } from 'react' import { useState, useEffect, useRef } from 'react'
import useTheme from '../theme.jsx'
const ROLES = ['ignore', 'dimension', 'value', 'units', 'date', 'filter'] const ROLES = ['ignore', 'dimension', 'value', 'units', 'date', 'filter']
@ -12,6 +13,7 @@ const ROLE_STYLE = {
} }
export default function Setup({ refreshSources }) { export default function Setup({ refreshSources }) {
const { dark } = useTheme()
const [tables, setTables] = useState([]) const [tables, setTables] = useState([])
const [sources, setSources] = useState([]) const [sources, setSources] = useState([])
const [selectedSource, setSelectedSource] = useState(null) const [selectedSource, setSelectedSource] = useState(null)
@ -24,12 +26,26 @@ export default function Setup({ refreshSources }) {
const [saving, setSaving] = useState(false) const [saving, setSaving] = useState(false)
const [generating, setGenerating] = useState(false) const [generating, setGenerating] = useState(false)
const [msg, setMsg] = useState(null) const [msg, setMsg] = useState(null)
const [dimPeriodCols, setDimPeriodCols] = useState([])
const [openPeriodIdx, setOpenPeriodIdx] = useState(null)
const periodDropRef = useRef(null)
useEffect(() => { useEffect(() => {
fetch('/api/tables').then(r => r.json()).then(setTables).catch(console.error) fetch('/api/tables').then(r => r.json()).then(setTables).catch(console.error)
fetch('/api/dim-period/cols').then(r => r.json()).then(setDimPeriodCols).catch(console.error)
loadSources() loadSources()
}, []) }, [])
useEffect(() => {
if (openPeriodIdx === null) return
function handleClick(e) {
if (periodDropRef.current && !periodDropRef.current.contains(e.target))
setOpenPeriodIdx(null)
}
document.addEventListener('mousedown', handleClick)
return () => document.removeEventListener('mousedown', handleClick)
}, [openPeriodIdx])
function loadSources() { function loadSources() {
fetch('/api/sources').then(r => r.json()).then(data => { fetch('/api/sources').then(r => r.json()).then(data => {
setSources(data) setSources(data)
@ -140,7 +156,12 @@ export default function Setup({ refreshSources }) {
async function deleteSource(id, e) { async function deleteSource(id, e) {
e.stopPropagation() e.stopPropagation()
if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return if (!confirm('Deregister this source? Existing forecast tables are not affected.')) return
await fetch(`/api/sources/${id}`, { method: 'DELETE' }) const res = await fetch(`/api/sources/${id}`, { method: 'DELETE' })
if (!res.ok) {
const data = await res.json().catch(() => ({}))
flash(data.error || 'Delete failed', 'err')
return
}
if (selectedSource?.id === id) { setSelectedSource(null); setCols([]); setEditedCols([]) } if (selectedSource?.id === id) { setSelectedSource(null); setCols([]); setEditedCols([]) }
loadSources() loadSources()
} }
@ -281,6 +302,8 @@ export default function Setup({ refreshSources }) {
<th className="px-3 py-1.5 font-medium">column</th> <th className="px-3 py-1.5 font-medium">column</th>
<th className="px-3 py-1.5 font-medium">role</th> <th className="px-3 py-1.5 font-medium">role</th>
<th className="px-3 py-1.5 font-medium text-center">key</th> <th className="px-3 py-1.5 font-medium text-center">key</th>
<th className="px-3 py-1.5 font-medium">group</th>
<th className="px-3 py-1.5 font-medium">period col</th>
<th className="px-3 py-1.5 font-medium">label</th> <th className="px-3 py-1.5 font-medium">label</th>
</tr> </tr>
</thead> </thead>
@ -302,10 +325,53 @@ export default function Setup({ refreshSources }) {
type="checkbox" type="checkbox"
checked={!!col.is_key} checked={!!col.is_key}
onChange={e => updateCol(i, 'is_key', e.target.checked)} onChange={e => updateCol(i, 'is_key', e.target.checked)}
disabled={col.role !== 'dimension'} disabled={col.role !== 'dimension' && col.role !== 'date'}
className="cursor-pointer disabled:opacity-20" className="cursor-pointer disabled:opacity-20"
/> />
</td> </td>
<td className="px-3 py-1.5">
<input
type="text"
value={col.dim_group || ''}
onChange={e => updateCol(i, 'dim_group', e.target.value || null)}
placeholder="—"
disabled={col.role !== 'dimension' && col.role !== 'date'}
className="border border-transparent hover:border-gray-200 focus:border-gray-300 rounded px-1.5 py-0.5 w-full outline-none bg-transparent disabled:opacity-20 disabled:cursor-default"
/>
</td>
<td className="px-3 py-1.5 relative">
{(() => {
const disabled = col.role !== 'dimension' || !col.dim_group
const isOpen = openPeriodIdx === i
return (
<div ref={isOpen ? periodDropRef : null} className="relative">
<button
type="button"
disabled={disabled}
onClick={() => setOpenPeriodIdx(isOpen ? null : i)}
className="text-xs px-1.5 py-0.5 rounded border border-transparent hover:border-gray-200 outline-none bg-white text-gray-700 disabled:opacity-20 disabled:cursor-default font-mono w-full text-left"
>
{col.dim_period_col || '—'}
</button>
{isOpen && (
<ul style={{ backgroundColor: dark ? '#2a2c2f' : '#ffffff', color: dark ? '#c5c9d0' : '#374151', borderColor: dark ? '#3b3f46' : '#e5e7eb' }} className="absolute z-50 left-0 top-full mt-0.5 min-w-full border rounded shadow-lg text-xs font-mono max-h-48 overflow-y-auto">
<li
className="px-2 py-1 cursor-pointer hover:bg-gray-100"
onMouseDown={() => { updateCol(i, 'dim_period_col', null); setOpenPeriodIdx(null) }}
>—</li>
{dimPeriodCols.map(c => (
<li
key={c}
className={`px-2 py-1 cursor-pointer hover:bg-gray-100 ${col.dim_period_col === c ? 'bg-blue-50 text-blue-700' : ''}`}
onMouseDown={() => { updateCol(i, 'dim_period_col', c); setOpenPeriodIdx(null) }}
>{c}</li>
))}
</ul>
)}
</div>
)
})()}
</td>
<td className="px-3 py-1.5"> <td className="px-3 py-1.5">
<input <input
type="text" type="text"