Three changes, one feature: a period with no baseline borrows a shape from
prior year or plan.
The generator handled one date group. sql_generator.js used find(), so a
source with order, requested and ship date groups derived the order one and
copied the other two raw -- shifted dates against unshifted period labels.
Every group now gets its own join, and they are LEFT rather than inner: an
order that has not shipped has no ship date, and an inner join would have
dropped the row from the load entirely rather than leaving its period
columns empty. dateGroupsOf() is now shared, like grainOf, so the routes and
the generator agree on membership and on join aliases.
Clone carries every date column rather than only the primary one, since it
is the operation that moves rows through time, and re-derives the period
dimensions from pf.dim_period against the shifted date instead of copying
them from the row being cloned.
from_logid names the segment to copy from, replacing the exclude clause for
that one entry rather than widening it. Rows written stay pf_iter = 'clone',
so scale applies to them afterwards as a second step.
set is no longer required: copying a segment forward unchanged is a
reasonable thing to ask for.
Needs dim_period_col set in Setup (fisc_year on oseas/rseas/sseas_e,
fisc_month_abbr on omon/rmon/smon_e) and Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_bucket mirrored pf_segment for scale/recode/clone rows, so adjustments
showed as '(adjustment)' -- a banner of their own, sitting outside the
forecast they are adjustments to. Grouping by pf_bucket therefore split the
forecast in two.
An adjustment is always part of the forecast: exclude_iters keeps operations
off the reference segments, so there is no adjustment that is not. It falls
back to 'Forecast' now, and still yields to an explicit bucket on its own
log entry.
pf_segment keeps '(adjustment)', which is right there -- it answers which
segment a row came from, and an adjustment came from none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every question about a part -- what values exist, what attributes go with
one -- was answered by querying the source, and the source is the wrong
place to ask. It is a view over a transaction table, so the query is slow
(76s for one ILIKE against 6.9M rows), it describes only what was
transacted, and it cannot express intent: there is no way to say a part is
discontinued, or to name one that has not sold yet.
pf.dim_member holds the app's own list: one row per key value per group,
siblings in jsonb, keyed on (source_id, dim_group, key_value). Refresh is a
merge rather than a replace, so curation survives it -- members absent from
the source are marked source_seen = false, not deleted. Triggered from
Setup, next to Generate SQL, because it reads the whole source and the
answer only changes when the catalogue does.
A key can carry several attribute sets across history -- 11,290 parts
against 13,662 combinations on osm_skinny -- so the refresh takes the most
recent by the source's date column. That also fixes the sibling autofill,
which used to run a DISTINCT ... LIMIT 2 against the source and silently
fill nothing whenever a part came back ambiguous. A member row is one
definition by construction.
The client fetches each group's list once per source and does both
completion and autofill against it in memory, so neither costs a request.
Columns outside a group, or a group never refreshed, still fall back to the
version's values endpoint.
Run 01_schema.sql to create the table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
55814ee introduced calls to sliceUnits() in the recode and clone routes
without ever defining it, so both have thrown "sliceUnits is not defined"
since that commit. Scale worked only because it inlined the same three lines
rather than calling the helper.
Defined from scale's copy, and scale now calls it too. Recode and clone
passed req.body.apply_mode straight through, where scale normalised it
first, so an absent or unrecognised value would have fallen to the 'each'
branch by accident; they normalise the same way now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pointing completion at the source made every keystroke a 76-second query:
gs.osm_skinny is a plain view over rlarp.osm_stack, so ILIKE '%1601%'
scanned 6.9M rows and read 1.3M buffers off disk, 64s of it I/O. Nothing had
called that endpoint before, so the cost only appeared once a debounced
input was wired to it.
The source is the wrong list anyway. It reaches back over all of history and
would offer parts discontinued years ago; the version holds what was
actually loaded, which is what the forecast is being written against.
So GET /versions/:id/values/:col reads the version's own forecast table --
2.0s for 11,290 parts on fc_osm_skinny_29 -- and holds the result in memory,
keyed on that version's latest pf.log id. Any load, adjustment or undo moves
the id and the next request rebuilds, so nothing has to remember to
invalidate. Filtering happens over the cached array, so typing costs one
small max(id) query.
The column name is interpolated into the DISTINCT, so it is checked against
col_meta first.
The source-side endpoint keeps its new q/limit but no longer has a caller;
its ILIKE path against a view is the expensive one and should stay unused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recoding to a part meant typing a code from memory into a plain text box,
with the only feedback being that the sibling autofill either fired or
silently did nothing.
The values endpoint already existed but returned every distinct value with
no filter and no limit, which for part on osm_skinny is 11,290 rows -- too
slow to open and no easier to read than a short list. It now takes ?q= and
?limit=, so the field fetches matches for what has been typed so far,
debounced 200ms, and offers them through a native datalist.
Only key columns get it, which is the same condition the endpoint already
enforced, and the same one that decides whether the dim_group sibling
lookup runs on blur. So on osm_skinny it is part and customer: type 1601
and the eight parts containing it are offered; pick one and the nine
part-group columns fill themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Filtering the loads out in the dialog left the server still joining the
whole forecast table for them and discarding the answer. ?kind=adjustments
moves the filter into the WHERE, so they never enter the join: on version 29
the join goes from 2,556,821 rows to 25, and the query from 1666ms to 719ms.
Still 719ms, because nothing indexes pf_logid and it stays a sequential scan
of 2.5M rows. New forecast tables now get an index on it. That column is how
every entry-level operation finds its rows -- undo deletes by it, this
aggregate groups by it, and it is part of the grain key -- so the scan was
being paid on all of them.
Existing tables predate the index and still scan; fc_osm_skinny_29 would
want it added by hand.
The route keeps returning everything by default: Baseline.jsx lists the
segments from the same endpoint, and the ledger's tag lookup needs every
entry to label its lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sales figure on its own does not say which of price or volume moved, and
scale was answering silently: it wrote the value increment and left units
alone, so volume stayed flat and price absorbed everything. That is one of
the three behaviours the Excel form offered, picked by default rather than
by the user.
/opt/forecast_api/VBA/fpvt.frm has the semantics. Edit Sales, then plug one
side:
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
So 'volume' moves units in the same proportion as the dollars, holding
price; 'price' leaves units alone, as now. Checked against those formulas:
+100 on 1000/500 gives units 500 -> 550 with price held at 2.0000, and
-400 on 2000/800 gives 800 -> 640 at 2.5000.
Price still defaults, so nothing changes for an existing caller that does
not send `plug`. The control only appears once the edit is dollars-only --
naming units or price has already settled the question. A price target now
also honours a units target alongside it, which is the form's Edit Price
mode where both are inputs and dollars fall out.
Holding price when the selection has no value is refused rather than
divided by zero, the same case the form guarded with "Zero times any number
is zero".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The label on a baseline or reference load ("Open Orders", "Prior Year")
lived only on the pf.log row; the data stream was a straight dump of the
forecast table, so Perspective saw pf_logid and never the name.
The stream now joins pf.log and emits two columns rather than one, because
commingling them makes neither useful: pf_segment names the load a row came
from, and is '(adjustment)' for everything else; pf_note carries the free
text on scale/recode/clone and is null on loads. The operation routes stamp
the same two fields on the rows they push back incrementally.
The tag column those labels belong in is declared in 01_schema.sql but
predates some installs, so the backfill seeding it from note joins the
ALTER already there. Adjustment notes are free text, not labels, and are
left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The server had no authentication: every /api route was open, CORS
allowed any origin, and the identity written to the audit log came from
the request body — the UI sent a hardcoded pf_user: 'admin', which any
client could have set to anything it liked.
Accounts live in pf.app_user with scrypt hashes from node's own crypto,
so there is no native build step and the parameters travel with each
hash. Sessions are express-session over connect-pg-simple in pf.session:
a restart no longer signs everyone out, and a session can be revoked by
deleting its row, which is how disable-user cuts off access immediately
rather than at cookie expiry.
Everything under /api except login/logout/me now requires a session, and
the React app is mounted only once there is one — its load effects call
the API on mount, so a logged-out mount would just fire a burst of 401s.
A session that expires while the app is open lands back on the login
screen: auth.jsx wraps fetch once rather than teaching every call site
to check.
Identity is now read from the session for pf_user, created_by and
closed_by, and the body values are ignored.
Hardened for an internet-facing deployment: trust proxy so req.ip and
secure-cookie detection are right behind TLS termination, httpOnly +
SameSite=Lax + Secure cookies, ten login failures per IP per fifteen
minutes, one error message for unknown, wrong and disabled alike, and a
fresh session id on success. CORS is off entirely unless CORS_ORIGIN
names an origin — a wildcard alongside a session cookie would be CSRF by
construction. The server refuses to boot without SESSION_SECRET rather
than falling back to a guessable default.
pf.sh grows add-user, passwd, list-users, disable-user and enable-user;
passwords are read on stdin and hashed before they reach psql, so no
plaintext in argv or shell history. install.sh generates the secret,
applies 02_auth.sql, and creates the first account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /api/dim-period/cols queries information_schema for pf.dim_period columns
(excluding sdat/edat/drange/ndays) so the UI always reflects actual columns.
Setup col_meta editor now shows a dropdown populated from that endpoint instead
of a free-text field, preventing invalid column names like the cash source had.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL generator no longer requires a units col; recode/clone/scale omit units
expressions when none is configured in col_meta
- Source registration validation drops units from required roles (value + date
are the only hard requirements)
- DELETE /api/sources/:id returns 409 when existing versions reference the source
- Setup.jsx surfaces the 409 error via flash instead of silently failing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta gets dim_period_col field: maps a dimension column to its pf.dim_period counterpart (e.g. year -> cal_year, month -> cal_month)
- When the date column is is_key of a dim_group and any sibling dimension has dim_period_col set, baseline and reference SQL JOIN pf.dim_period on the shifted date instead of copying raw source values
- No dim_period config = identical SQL to before (fully backwards compatible)
- Setup UI: period col input in col_meta editor, enabled for dimension columns with a dim_group set
- Schema migration applied: dim_period_col text null on pf.col_meta
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/sources/:id/lookup?col=X&value=Y — given a key column value, queries the source table for sibling column values in the same dim_group; returns null if no match or ambiguous
- Recode and Clone panels: key columns (is_key + dim_group) trigger lookup on blur and auto-fill sibling inputs that the user hasn't already typed into
- Row labels now use col_meta label field when set, falling back to cname
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PATCH /api/log/:logid — saves note updates to pf.log (was missing, frontend call was silently failing)
- GET /api/versions/:id/log — joins fc_table to return row_count per entry so the change log modal shows rows affected instead of '—'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta: add dim_group field to group related columns (dimension hierarchies, date-adjacent columns); is_key now enabled for date role to mark group parent
- sources.js: upsert includes dim_group
- Setup.jsx: group column in col_meta editor, key checkbox enabled for date role
- gen_dim_period.sql: create and populate pf.dim_period with calendar and fiscal period cuts (monthly grain, 2018-2035)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch server Arrow encoding from tableFromJSON (row objects) to
tableFromArrays (column arrays) — cuts peak Node heap 3-5x for large
datasets by avoiding one JS object per row
- Remove unused pf.log JOIN from data endpoint; forecast rows only
- Load Perspective viewer with direct table reference instead of worker
Server object — fixes "No Table attached" error on large datasets where
named-table registry lookup raced against WASM initialization
- Pre-emptively clean up stale named table in worker registry before
creating, eliminating the "already exists" retry path that silently
swallowed errors (finally ran but flash never fired)
- Strip cfg.table from restore configs since table is loaded by reference
- Throttle progress bar updates to 100ms intervals (was every chunk)
- Persist load errors until dismissed; add console.error for devtools
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Add PayloadPreview component showing the exact JSON that will be POSTed,
live-updating as form fields change (value_incr shown as computed delta)
- buildEffectiveSlice strips expression/system columns and converts
Perspective ms-timestamps to ISO date strings for date-role columns
- fetchCurrentTotals now includes date columns in Perspective view filter
(passing ms number as Perspective expects) so subtotals respect the
clicked date
- Server buildWhere now receives filterCols (dimensions + date cols) so
date values reach the SQL WHERE clause correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forecast falls back to a saved per-source layout when no version-local
layout is cached, so new versions of a source open with a sensible pivot
without each user reconfiguring it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reference segments can now apply a date offset just like baselines.
SQL template gains the {{date_offset}} token; both POST /reference and
PUT baseline/:logid pass it through. Existing sources need to
regenerate SQL to pick up the new template — old stored reference SQL
ignores the token (preserving prior verbatim behavior). The Baseline
form drops the "dates land verbatim" hint and shows the offset
control for both segment types.
Editing a segment now color-codes the source row amber with a ring
and tints the form border + header amber so the active connection is
visually obvious. Header label reads "Edit segment #3 — baseline —
note" instead of just "#43" (the internal log id).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The segment form is now one component rendered in either 'view' or
'edit' mode — the expanded segment row in the list and the
add/edit form below share the same layout, view mode just disables
the inputs. Edit and View are visually identical so toggling between
them feels like enabling fields, not switching tools.
Filters become groups (conditions AND-ed inside, groups OR-ed
between) with + AND condition and + Add OR group affordances. The
compiled WHERE renders live below the groups so you can see what's
being built. A "Switch to manual SQL" toggle flips to a textarea
seeded with the compiled clause; backend baseline POST/PUT and
reference POST accept raw_where alongside filters and store whichever
arrived in pf.log.params for round-tripping.
The Add form is hidden until you click "+ Add segment" at the
bottom of the segments table; Edit also opens it. Cancel/Close
returns the table to its compact state.
/versions/:id/log now also returns value_total, units_total, and the
column names so the segments table can show row count and value sum
inline (header uses the source's actual value column name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds PUT /versions/:id/baseline/:logid that, in one transaction, drops
the segment's rows and log entry and replays the baseline or reference
SQL with new params. The endpoint refuses (409) if any scale, recode,
or clone has been applied — those operations were calibrated against
the old totals and would silently misreconcile.
Baseline view gets an Edit button on each segment (hidden once
forecast operations exist), populating the form with the original
filters, offset, and note. Submit issues PUT in edit mode, POST
otherwise. POST baseline and POST reference now also persist the
structured filters in pf.log.params so edit can reload them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Forecast tables don't carry pf_note; it's joined from pf.log on /data
fetch. RETURNING * from scale/recode/clone INSERTs lacks those fields,
so rows appended via table.update arrived with pf_note null until a
reload re-ran the join. Inject pf_note and pf_op server-side from the
request before responding.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Perspective table is now created with index: 'pf_id'. Delete endpoints
return the pf_ids they removed; the client calls table.remove(pf_ids)
in undoEntry. Avoids the full /data refetch that dominated undo time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
/data now joins pf.log to surface note text and operation type as
pf_note/pf_op so users can pivot/bridge by assumption. Joining at
fetch time avoids storing notes per row and keeps edits live.
/api/tables joined pg_class by name only with namespace filtered in
a separate LEFT JOIN, which cross-producted table names that exist in
multiple schemas. Restructured so namespace participates in the join.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
pg now returns bigint/numeric as JS numbers so Arrow infers Int/Float64
instead of Dictionary<Utf8>. /data accumulates rows and emits a single
record batch to avoid dictionary REPLACEMENT messages that crash
Perspective's WASM reader. Forecast view streams the response body and
shows received/total bytes while loading. Drops stale public/ static
middleware that was shadowing the React build at /.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Server streams rows from a pg cursor in 10k-row batches, building Arrow
record batches incrementally and piping them as chunked HTTP response —
Node.js heap stays bounded regardless of dataset size.
Client fetches as arrayBuffer() and loads directly into Perspective worker
(native Arrow path, no JSON deserialization). X-Row-Count header drives
a non-blocking banner for datasets >= 500k rows. validCols now derived
from col_meta rather than from row keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/versions/:id/log — log entries with row counts via JOIN
- DELETE /api/log/:logid — undo in a transaction (delete fc rows + log entry)
- PATCH /api/log/:logid — update note text
- History button opens a modal: op badge, slice, editable note, row count, Undo per entry
- Undo triggers full Perspective table reload via initViewer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix perspective-click handler to use event filter triples instead of
__ROW_PATH__ — Perspective encodes row position as [col,'==',val] in
detail.config.filter
- buildWhere now skips unrecognised slice keys (e.g. pf_iter) instead of
throwing, so only dimension columns reach the WHERE clause
- Add draggable resize handle on the operation panel (160–480px)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Baseline.jsx: merge Reference section into Add Segment form with baseline/reference toggle; segment rows now clickable to expand stored WHERE clause + timeline; date filter inputs use type="date" for date-role columns
- Timeline.jsx: add type prop ('baseline'|'reference'); reference band uses purple; single-band height shrinks to 52px; canvas uses requestAnimationFrame to fix offsetWidth=0 on mount
- operations.js: reference route now accepts where_clause like baseline (drops date_from/date_to)
- sql_generator.js: reference SQL template uses {{filter_clause}} instead of hardcoded BETWEEN
Note: existing sources need Generate SQL re-run to pick up the new reference template.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ui/: React + Vite + Tailwind app (Setup, Baseline, Forecast views, collapsible sidebar, status bar, canvas timeline)
- server.js: serve built UI from public/app/
- package.json: add build script (cd ui && npm run build)
- routes/sources.js: default new col_meta role to 'dimension' instead of 'ignore'
- .gitignore: exclude public/app/ build output
- pf_spec.md: update tech stack, nav, frontend section, and project status to reflect current implementation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Sources page: left column with stacked DB tables + registered sources panels,
right column as full-height column mapping workbench
- Add compact table search, column search, table preview button, delete source button
- Rename fc_table system columns to pf_ prefix (pf_id, pf_iter, pf_logid,
pf_created_at) to avoid collisions with source table columns like 'id'
- Remove 'filter' col_meta role — any non-ignore column usable in baseline filters
- Replace structured filter row builder with free-form SQL WHERE clause textarea
and clickable column chips for insertion; fully flexible AND/OR logic
- Baseline segment cards now display raw WHERE clause text + offset
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New Baseline nav view replaces the simple Load Baseline modal
- Baseline loads are now additive; each segment is independently undoable
- Filter builder: any date/filter-role column, full operator set
- Timeline preview shows source → projected period bars for date BETWEEN filters
- Clear Baseline action deletes all baseline rows and log entries
- DELETE /api/versions/:id/baseline route
- buildFilterClause() added to sql_generator
- filter role added to col_meta editor
- Reminder: re-run generate-sql for each source after this change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The baseline operation now accepts a date_offset interval (e.g. "1 year",
"6 months") and applies it to every date when inserting rows, shifting
historical actuals into the target forecast period.
SQL: {date_col} + '{{date_offset}}'::interval)::date at insert time.
Route: defaults to '0 days' if omitted so existing calls are unaffected.
UI: year/month spinners with a live before→after month chip preview so
the projected landing period is visible before submitting.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of re-fetching all forecast data after scale/recode/clone/reference,
the routes now return the inserted rows directly. The frontend uses ag-Grid's
applyTransaction to add only the new rows, eliminating the full reload round-trip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
