table-info feeds two things and was scoped for neither: the status bar's
"total rows", and the figure the load progress promises while it waits. So
an account that can see a fraction of the table was told the whole size of
it -- jbukowski's 542k rows reported as 2.8M.
It only showed on the larger territories. The seeded figure is replaced by
X-Row-Count when the headers arrive, and on a small territory the aggregate
returns fast enough that the wrong number barely appears.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
params says what was asked for. That is not enough to explain a surprising
result, because the same intent produces different rows depending on state
the entry does not carry, and because the translation from intent to SQL is
itself a place bugs live.
So both, not one. env records the state that cannot be reconstructed later:
the territory in force, the version's exclude_iters, and when the template
was generated -- all mutable rows elsewhere with nothing remembering what
they were. sql_text records the statement as executed, territory and scope
already resolved into it.
The template generation is a fingerprint rather than a version. It cannot
bring the old template back; it can tell you the entry did not run under the
current one, which is what would otherwise make a comparison quietly wrong.
Generate SQL has overwritten those templates four times today.
The statement is fetched on demand through GET /log/:logid/debug and left out
of the list, which is opened to scan rather than to read SQL. Under an
entry's payload in the change log there is now an "executed SQL" toggle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PATCH /log/:logid had no check, so any account could edit the tag, note,
label and bucket of any entry -- including loads whose rows it cannot see.
That reads as harmless annotation and is not: label and bucket name the
pivot's columns for everyone in the version, so a rep could rename the
company's segments.
Same rule as undo now, author or admin, with the fields shown read-only
rather than editable-then-403 -- in the change log's tag and note cells and
on the Baseline page's label and bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The login query names its columns, and territory and is_admin were not among
them -- so every session carried an empty list, every account scoped to
FALSE, and the forecast page came back with nothing however the grant was
set. The CLI had written it correctly; nothing read it.
The empty case then aborted twice over. First on the index, fixed already.
Then on the layout: an empty table has no schema, so restoring a saved
config asks for the dtype of a column that is not there and the worker dies
-- "Could not get dtype for column `sseas_e`". With no rows there is nothing
to lay out, so nothing is restored, and the saved layout waits in
localStorage for rows to come back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flag was enforced everywhere and settable nowhere but psql, so the one
piece of configuration a second account depends on was invisible.
A radio rather than a checkbox, because exactly one column per source can be
the territory -- the control should say so rather than leaving it to an
error on save. Clicking the chosen one again clears it, which a radio has no
other way to express. The save still refuses two, since the UI is not the
only caller, and two would mean whichever a .find() reached first -- the trap
is_key already fell into.
Restricted to dimension columns: a territory is something rows are divided
by, and scoping on a date or a measure is not a thing to offer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Territory filtering was deferred from v1, so every account saw and could
write every row. With the sales team about to adjust their own territories
that is the thing standing in the way, and it is also why a rep would wait
fifteen seconds to load 2.7M rows to work on a few thousand.
The list lives on pf.app_user.territory with is_admin beside it, and
col_meta.is_territory marks which column of a source the values belong to --
flagged rather than named in code, so a second source can be divided by
something other than a sales rep.
Fail closed: buildTerritoryClause returns FALSE for an empty list or an
unflagged source. An account nobody configured sees nothing, rather than
everything because a column was left null.
Built from the session, never the request. That is what separates it from
`scope`, which the browser sends and should: a filter the user chose belongs
in the payload, a permission cannot come from the thing it restrains. It is
ANDed on last, where nothing in the request can undo it.
Enforced on /data (the cursor and the count behind X-Row-Count), on /agg
before the GROUP BY since the territory column need not be in the grain, on
every operation through sliceUnits, and on the value completion endpoint --
which reads the source table, so without it a dropdown enumerates every
customer and rep in the business to someone shown none of their rows.
Undo is gated by owner rather than territory: it removes an entry's rows
wholesale, so half-undoing one would leave a state nothing describes. Recode
refuses to set the territory column unless you are an admin, since moving a
row between territories is reassignment, not forecasting.
./pf.sh gains set-territory, set-admin and orphan-territory. The last lists
territory values no account owns -- work under one is invisible to everybody
but an admin, which a typo causes easily and nothing in the app reveals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change log joined the whole forecast table on every open to total rows
it had just written -- 2.5M rows to report a few thousand, and the Baseline
page's row and value columns paid the same cost again.
The totals go onto pf.log at write time instead. This is not a cache that
can drift: a log entry's forecast rows never change once written, because
only the operation owning the logid inserts them and the only thing that
removes them is undo, which deletes the log row too. measure_cols records
which columns the figures are denominated in, since the value and units
roles can be reassigned in col_meta and the numbers would otherwise quietly
come to mean something else.
Stamping is best-effort and runs after the commit: a failure to record what
happened must not roll back the thing that happened. The loads return their
new log id to make it possible, the adjustments take theirs from the rows
they return, and apply_mode 'each' writes one entry per slice, so it is a
set rather than a single id.
?recount=1 does it the old way and writes back what it finds. Stored totals
cannot drift on their own, but nothing stops someone deleting forecast rows
by hand, and a stored figure has no way to notice -- so there is a way back,
which doubles as the backfill for entries written before the columns
existed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refusing anything but == was safe and useless: a view bounded to
sseas_e <= 2027 is an ordinary way to scope a forecast, and it has no slice
form at all, a slice being {col: value}. The filter now travels beside the
slices as [col, op, value] triples and is ANDed onto every unit -- not folded
into the slices, since it applies to all of them equally and under
apply_mode 'each' would just repeat itself in every statement.
Operators are Perspective's, since that is where they come from, and the list
is a whitelist: anything outside it is refused rather than ignored, because a
scope silently dropped is a write wider than the panel that authorised it.
The scope goes into the log's params too, so the audit trail records what
bounded the write and not only what was clicked.
The panel prints it above the selection as "within sseas_e <= 2027". It
scopes every figure below it and every row the operation writes while
appearing in none of the slices, so without it the panel showed a selection
wider than the one it was acting on -- which is exactly what made the
ledger's 956,485.13 look plausible against a cell of 921,225.71.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phantom: pf_segment and pf_bucket are computed from pf.log when the rows
are served, so buildWhere had no column to compare and dropped them. Clicking
one bucket's cell and scaling therefore wrote every bucket at that dimension
intersection, while the panel showed only the bucket clicked. On the example
slice that is 350,524.74 displayed against 503,446.08 written.
They resolve exactly, without a new column: the name lives on the log row and
every forecast row carries the pf_logid that points at it, so the predicate is
pf_logid IN (SELECT id FROM pf.log WHERE <the same expression> = ...). Verified
against version 29 -- the clause returns 350,524.74 over 12 rows.
Any other pf_ key is now refused rather than skipped, since skipping is the
mechanism by which a selection silently widens. pf_iter stays exempt: the
client drops it deliberately, two cells differing only by iter band being the
same slice.
Client side they are ordinary columns in the loaded table, so both the
dispatch path and the panel's own totals filter on them directly -- the latter
matters as much, or the ledger reconciles against a wider selection than the
operation writes.
The ledger: excluded rows read "02 - Prior Year · FINAL" in amber rather than
"reference · fixed" -- named by the segment a forecaster recognises instead of
the iter band that happens to exclude it, and coloured because immovable is a
property worth seeing before reading a number. When the whole selection is
immovable it now says so in a sentence, where before it printed a row of zeros
and left the reason to be inferred from the edit rows failing below.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
adjustment_segment, adjustment_bucket and unlabeled_load are columns on
pf.version now, edited under "Fallback names" on the Baseline page, with the
constants in sql_generator left as the built-in for a version that sets none.
Read through a join, not substituted at generation: pf.sql is keyed on
(source_id, operation) and shared by every version of a source, so a baked-in
value could not vary by version and regenerating for one would change the
others. The join costs three more GROUP BY columns on /agg, all functionally
dependent on a version id that is already fixed for the whole query.
The built-ins stay a convention guess -- ADJUSTMENT_BUCKET's "04 - " suits one
numbering -- which is now a default to override rather than the only answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adjustments fell back to a bare 'Forecast' while the loads they adjust read
'04 - Forecast', so the bucket column split in two and the adjustments sat
apart from the rows they came from. The fallback now matches.
The three fallback names are gathered into one DISPLAY DEFAULTS block at the
top of sql_generator, exported, and tabulated in CLAUDE.md, so the answer to
"where did that name come from" is one place rather than a grep. The
incremental row stamps in the operation routes use the constant now instead
of restating the literal, which is how they drifted apart in the first place.
None of this belongs in the source. ADJUSTMENT_BUCKET carries a number that
only suits one convention and changing it changes every version on every
source; the note says what per-version would take.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PUT /versions/:id/baseline/:logid deletes the log row and inserts a fresh one
from the stored template, so every field the form does not send comes back
null. That took the label and the bucket with it, and the tag besides -- the
segment form has no tag input at all, so a tag could not survive an edit made
for any other reason.
The route now hands back what it was not given, reading the row it is about
to replace. `??` rather than `||`: an empty string is the form clearing a
field deliberately, undefined is the form not carrying it.
The load templates gained a tag token to receive it. Stored templates are
per-source and were generated before any of this existed, which is the other
half of why labels vanished -- source 14's had no label column to write to.
Regenerated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bucket_order and log.seq were still settable -- PUT /versions/:id took the
first, PATCH /log/:logid the second -- with nothing left to read either. A
field that only ever gets written is worse than a missing one: the call
succeeds, so the caller has no way to find out it did nothing.
The columns themselves stay, marked vestigial where they are declared.
Dropping a column is not worth a migration to reclaim two that cost nothing.
pf.log.bucket is untouched and stays exactly as it was -- what a row counts
toward, read first by BUCKET_EXPR. It is the bucket *order* that no longer
needs storing, the text having become the order.
Also removes a comment head in Forecast.jsx that survived its function and
had glued itself onto fitColumns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Perspective orders column groups by the value string, so "01 - Actual" is
the only way an arbitrary order can be expressed. That prefix now lives in
pf.log.label, typed by whoever names the segment, rather than being built
from a seq column by client-side expressions.
pf_segment and pf_bucket read label first, and the expressions are shared
between /agg and /data instead of being spelled out in each -- they have to
agree, and they had drifted apart in whitespace already.
The synthetic values lose their parentheses and their ordinals, except the
adjustment fallback: '(adjustment)' sorted *before* '01 - ...', since '(' is
0x28 and digits begin at 0x30, so it becomes '99 - Adjustments' to sit last.
Labelling an adjustment's own log row overrides that, which is how one kind
of adjustment splits out from the rest. '(unlabeled load)' becomes plain
'Unlabeled', which needs no ordinal -- letters already follow digits.
The load routes carry label and bucket onto the log row, so the fields the
segment form has been offering since af9e6de are no longer a silent no-op.
startEdit now reads them back, which it never did: editing a segment for any
other reason blanked both.
Existing sources need Generate SQL re-run -- the load templates are stored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps you hit. There was nowhere to set "counts toward" while defining a
segment -- only in the list afterwards -- and the Edit buttons disappear
entirely once any adjustment exists.
That guard is right in principle and too broad in practice. Editing a
segment's filters or date offset after a scale would silently recalibrate a
distribution that was sized against the old rows, so it stays gated. But the
label and the bucket are presentation: they change what the pivot shows and
what the segment counts toward, never which rows were loaded. Those are now
editable in the list at any time, and settable on the create form.
pf.log.label is new: the segment's display name, falling back to tag then
note. Separate from both because those have jobs already -- tag groups
adjustments into initiatives for the bridge, note is commentary -- and
because the label is where sort order lives. Perspective orders column
groups by the value string, so a leading "01 - " is how ordering gets
expressed, and putting that in the note would put it in every note.
The load routes do not yet carry bucket and label through: they return only
rows_affected, with no log id to attach them to. That comes with the switch
away from computed prefixes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Perspective orders column groups by the value string, and SortDir's col asc /
col desc only reverses that -- so Prior Year -> Plan -> Actual -> Forecast is
expressible as neither, being alphabetical in neither direction. The order
has to be part of the value, as a "01 · " prefix.
Done as Perspective expression columns, named Bucket and Segment, generated
from pf.version.bucket_order and pf.log.seq. The first attempt computed the
prefix in the served SQL (archived on feature/column-sequencing-sql), which
was the wrong layer: the prefix is a pivot-ordering concern, and putting it
in the query put it in every other reader too -- the change log and the
bridge's basis list would both have read "04 · Forecast". It also meant a
Generate SQL to introduce the placeholder, and a reload to see any change.
As expressions it stays in the pivot, travels with saved layouts because it
lives in ViewConfig, and reordering on the Baseline page takes effect
immediately -- the expressions are rebuilt and the pivot re-renders, no
reload.
Kept from the SQL attempt: pf.version.bucket_order and pf.log.seq, which are
needed either way, and the Baseline controls -- a seq column per segment and
a reorderable row of bucket chips. bucket_order is on the version because
the Baseline page is version-scoped; Setup is the only source-level context
and not where anyone would look for this.
Anything unordered falls through to the raw column, so it sorts after the
ordered entries (digits before letters) rather than silently landing first.
The expressions are merged into the live config rather than replacing it, so
a user's own expressions survive, and an expression that stops existing is
dropped from the axes first -- restore() rejects a config that pivots on an
expression it no longer defines.
Needs 01_schema.sql for the two columns. No Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Date offset on a baseline or reference segment was a pair of
type="number" inputs, years and months, both clamped at min 0. So no
characters could be typed, days could not be expressed at all, and a segment
could only ever be shifted forward in whole months -- while the stored value
is a Postgres interval that happily accepts "4 months", "1 year" or
"-90 days". parseOffset only read year and month, so anything else would not
have survived a round trip through the edit form either.
One text field now, with suggestions, matching what clone already offers.
parseInterval parses it far enough to draw the timeline preview; Postgres
stays the authority, and assertInterval -- now shared by the baseline,
reference and clone routes -- rejects what it will not accept, with a
message naming the field rather than a parse error from inside a CTE.
Timeline takes months and days separately rather than years and months,
because the two are not interchangeable: a month shift lands on the same day
of another month, a day shift can cross a month boundary. It adds months
then days, as Postgres does, and its label handles negatives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A selection of five AOP rows cloned nothing, with no explanation. The cause
was exclude_iters, which clone applied along with scale and recode.
It should not. That exclusion exists to stop operations *modifying*
reference rows: scale distributes an increment across its pool, so including
reference would attribute forecast movement to prior-year rows, and recode
writes negative rows that zero the original out. Clone does neither -- it
reads rows and inserts new pf_iter = 'clone' rows, leaving the source
untouched. Copying a plan or a prior year out of reference and into
adjustments is the operation doing exactly what it is for.
So from_logid stops being the only way to reach those rows and becomes what
it should be: a narrowing, for a selection spanning AOP and Prior Year where
only one is wanted.
The "would just duplicate the rows" guard no longer fires when the selection
is entirely non-adjustable, since moving rows from reference into adjustments
changes what they are even at factor 1 with no shift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shifting backwards already worked -- the offset is added as a Postgres
interval, and '-90 days' is a perfectly good one -- but the suggestions only
listed forward shifts, so nothing said so. Pulling a plan back a quarter is
a normal thing to want.
A day-level shift is also the case that most needs the dim_period
derivation: 15 Mar 2027 less 90 days is 15 Dec 2026, which crosses from
2027/10 - Mar into 2027/07 - Dec. Copying the period columns across would
have labelled it March.
The offset is interpolated into the statement, so a typo surfaced as a
Postgres parse error from the middle of a CTE. It is parsed on its own
first, where the failure is cheap and can name the field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes, one feature: a period with no baseline borrows a shape from
prior year or plan.
The generator handled one date group. sql_generator.js used find(), so a
source with order, requested and ship date groups derived the order one and
copied the other two raw -- shifted dates against unshifted period labels.
Every group now gets its own join, and they are LEFT rather than inner: an
order that has not shipped has no ship date, and an inner join would have
dropped the row from the load entirely rather than leaving its period
columns empty. dateGroupsOf() is now shared, like grainOf, so the routes and
the generator agree on membership and on join aliases.
Clone carries every date column rather than only the primary one, since it
is the operation that moves rows through time, and re-derives the period
dimensions from pf.dim_period against the shifted date instead of copying
them from the row being cloned.
from_logid names the segment to copy from, replacing the exclude clause for
that one entry rather than widening it. Rows written stay pf_iter = 'clone',
so scale applies to them afterwards as a second step.
set is no longer required: copying a segment forward unchanged is a
reasonable thing to ask for.
Needs dim_period_col set in Setup (fisc_year on oseas/rseas/sseas_e,
fisc_month_abbr on omon/rmon/smon_e) and Generate SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pf_bucket mirrored pf_segment for scale/recode/clone rows, so adjustments
showed as '(adjustment)' -- a banner of their own, sitting outside the
forecast they are adjustments to. Grouping by pf_bucket therefore split the
forecast in two.
An adjustment is always part of the forecast: exclude_iters keeps operations
off the reference segments, so there is no adjustment that is not. It falls
back to 'Forecast' now, and still yields to an explicit bucket on its own
log entry.
pf_segment keeps '(adjustment)', which is right there -- it answers which
segment a row came from, and an adjustment came from none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to express "these segments together are the forecast".
pf_iter cannot say it: it answers whether operations may write to a row, and
Open Orders is loaded as reference precisely so nothing adjusts it while
still being part of the forecast number. The two questions are independent,
so one cannot be derived from the other.
pf.log.bucket is the second axis. Free text with suggestions -- Forecast,
Prior Year, Prior Prior Year, Plan -- rather than an enum, so another banner
needs no migration. Blank by default, falling back in the pivot to the
segment's own name, so nothing changes until something is labelled.
/data and /agg emit it as pf_bucket beside pf_segment, through the pf.log
join that is already there. Set it per segment in the Baseline list, which is
where loads live now that the change log only shows adjustments.
Needs 01_schema.sql for the column and Generate SQL for /agg to select it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every question about a part -- what values exist, what attributes go with
one -- was answered by querying the source, and the source is the wrong
place to ask. It is a view over a transaction table, so the query is slow
(76s for one ILIKE against 6.9M rows), it describes only what was
transacted, and it cannot express intent: there is no way to say a part is
discontinued, or to name one that has not sold yet.
pf.dim_member holds the app's own list: one row per key value per group,
siblings in jsonb, keyed on (source_id, dim_group, key_value). Refresh is a
merge rather than a replace, so curation survives it -- members absent from
the source are marked source_seen = false, not deleted. Triggered from
Setup, next to Generate SQL, because it reads the whole source and the
answer only changes when the catalogue does.
A key can carry several attribute sets across history -- 11,290 parts
against 13,662 combinations on osm_skinny -- so the refresh takes the most
recent by the source's date column. That also fixes the sibling autofill,
which used to run a DISTINCT ... LIMIT 2 against the source and silently
fill nothing whenever a part came back ambiguous. A member row is one
definition by construction.
The client fetches each group's list once per source and does both
completion and autofill against it in memory, so neither costs a request.
Columns outside a group, or a group never refreshed, still fall back to the
version's values endpoint.
Run 01_schema.sql to create the table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
55814ee introduced calls to sliceUnits() in the recode and clone routes
without ever defining it, so both have thrown "sliceUnits is not defined"
since that commit. Scale worked only because it inlined the same three lines
rather than calling the helper.
Defined from scale's copy, and scale now calls it too. Recode and clone
passed req.body.apply_mode straight through, where scale normalised it
first, so an absent or unrecognised value would have fallen to the 'each'
branch by accident; they normalise the same way now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pointing completion at the source made every keystroke a 76-second query:
gs.osm_skinny is a plain view over rlarp.osm_stack, so ILIKE '%1601%'
scanned 6.9M rows and read 1.3M buffers off disk, 64s of it I/O. Nothing had
called that endpoint before, so the cost only appeared once a debounced
input was wired to it.
The source is the wrong list anyway. It reaches back over all of history and
would offer parts discontinued years ago; the version holds what was
actually loaded, which is what the forecast is being written against.
So GET /versions/:id/values/:col reads the version's own forecast table --
2.0s for 11,290 parts on fc_osm_skinny_29 -- and holds the result in memory,
keyed on that version's latest pf.log id. Any load, adjustment or undo moves
the id and the next request rebuilds, so nothing has to remember to
invalidate. Filtering happens over the cached array, so typing costs one
small max(id) query.
The column name is interpolated into the DISTINCT, so it is checked against
col_meta first.
The source-side endpoint keeps its new q/limit but no longer has a caller;
its ILIKE path against a view is the expensive one and should stay unused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recoding to a part meant typing a code from memory into a plain text box,
with the only feedback being that the sibling autofill either fired or
silently did nothing.
The values endpoint already existed but returned every distinct value with
no filter and no limit, which for part on osm_skinny is 11,290 rows -- too
slow to open and no easier to read than a short list. It now takes ?q= and
?limit=, so the field fetches matches for what has been typed so far,
debounced 200ms, and offers them through a native datalist.
Only key columns get it, which is the same condition the endpoint already
enforced, and the same one that decides whether the dim_group sibling
lookup runs on blur. So on osm_skinny it is part and customer: type 1601
and the eight parts containing it are offered; pick one and the nine
part-group columns fill themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Filtering the loads out in the dialog left the server still joining the
whole forecast table for them and discarding the answer. ?kind=adjustments
moves the filter into the WHERE, so they never enter the join: on version 29
the join goes from 2,556,821 rows to 25, and the query from 1666ms to 719ms.
Still 719ms, because nothing indexes pf_logid and it stays a sequential scan
of 2.5M rows. New forecast tables now get an index on it. That column is how
every entry-level operation finds its rows -- undo deletes by it, this
aggregate groups by it, and it is part of the grain key -- so the scan was
being paid on all of them.
Existing tables predate the index and still scan; fc_osm_skinny_29 would
want it added by hand.
The route keeps returning everything by default: Baseline.jsx lists the
segments from the same endpoint, and the ledger's tag lookup needs every
entry to label its lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sales figure on its own does not say which of price or volume moved, and
scale was answering silently: it wrote the value increment and left units
alone, so volume stayed flat and price absorbed everything. That is one of
the three behaviours the Excel form offered, picked by default rather than
by the user.
/opt/forecast_api/VBA/fpvt.frm has the semantics. Edit Sales, then plug one
side:
plug volume: pchange = fVal/(pVal+bVal); fVol = (pVol+bVol)*pchange
plug price: fVol = pVol + bVol
So 'volume' moves units in the same proportion as the dollars, holding
price; 'price' leaves units alone, as now. Checked against those formulas:
+100 on 1000/500 gives units 500 -> 550 with price held at 2.0000, and
-400 on 2000/800 gives 800 -> 640 at 2.5000.
Price still defaults, so nothing changes for an existing caller that does
not send `plug`. The control only appears once the edit is dollars-only --
naming units or price has already settled the question. A price target now
also honours a units target alongside it, which is the form's Edit Price
mode where both are inputs and dollars fall out.
Holding price when the selection has no value is refused rather than
divided by zero, the same case the form guarded with "Zero times any number
is zero".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the /agg endpoint, col_meta.in_grain, the pf_gkey index and the
append/remove write model that replaces undo's full reload. On the live
2,574,287-row forecast this collapses to 32,411 rows at a
rep/customer/channel/season/month grain, with totals tying exactly
(857,792,111.91 either way) and pf_gkey unique across every group.
Three conflicts, all from work done on this branch after the grain branch
was cut:
- sql_generator exports: union of both sides, adding grainOf.
- Forecast.jsx fetchArrow: the grain branch factored the inline progress
reader into a helper; kept the helper, and the tag/note ledger functions
beside it, since the two were only textually adjacent.
- Forecast.jsx initViewer: took the grain branch's endpoint selection, but
dropped its loadPerspective() -- 99375bb replaced that lazy CDN loader
with a static inline import, so awaiting fetchArrow directly is correct
here.
Carried the segment labels into grain mode as well: /agg now joins pf.log
the way /data does. pf_logid is part of the grain, so the join adds no
rows. Without it the labels would have disappeared exactly when a source
declared a grain -- which is the mode that will actually be used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The label on a baseline or reference load ("Open Orders", "Prior Year")
lived only on the pf.log row; the data stream was a straight dump of the
forecast table, so Perspective saw pf_logid and never the name.
The stream now joins pf.log and emits two columns rather than one, because
commingling them makes neither useful: pf_segment names the load a row came
from, and is '(adjustment)' for everything else; pf_note carries the free
text on scale/recode/clone and is null on loads. The operation routes stamp
the same two fields on the rows they push back incrementally.
The tag column those labels belong in is declared in 01_schema.sql but
predates some installs, so the backfill seeding it from note joins the
ALTER already there. Adjustment notes are free text, not labels, and are
left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The server had no authentication: every /api route was open, CORS
allowed any origin, and the identity written to the audit log came from
the request body — the UI sent a hardcoded pf_user: 'admin', which any
client could have set to anything it liked.
Accounts live in pf.app_user with scrypt hashes from node's own crypto,
so there is no native build step and the parameters travel with each
hash. Sessions are express-session over connect-pg-simple in pf.session:
a restart no longer signs everyone out, and a session can be revoked by
deleting its row, which is how disable-user cuts off access immediately
rather than at cookie expiry.
Everything under /api except login/logout/me now requires a session, and
the React app is mounted only once there is one — its load effects call
the API on mount, so a logged-out mount would just fire a burst of 401s.
A session that expires while the app is open lands back on the login
screen: auth.jsx wraps fetch once rather than teaching every call site
to check.
Identity is now read from the session for pf_user, created_by and
closed_by, and the body values are ignored.
Hardened for an internet-facing deployment: trust proxy so req.ip and
secure-cookie detection are right behind TLS termination, httpOnly +
SameSite=Lax + Secure cookies, ten login failures per IP per fifteen
minutes, one error message for unknown, wrong and disabled alike, and a
fresh session id on success. CORS is off entirely unless CORS_ORIGIN
names an origin — a wildcard alongside a session cookie would be CSRF by
construction. The server refuses to boot without SESSION_SECRET rather
than falling back to a guessable default.
pf.sh grows add-user, passwd, list-users, disable-user and enable-user;
passwords are read on stdin and hashed before they reach psql, so no
plaintext in argv or shell history. install.sh generates the secret,
applies 02_auth.sql, and creates the first account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Forecast operations could only act on one clicked row at a time, and the
panel that drove them separated the numbers you were reading from the
inputs that changed them. This reworks both, and adds initiative tags so
a version's history can be read as a bridge.
Operations
- Accept `slices` (array) alongside the legacy single `slice`, with
apply_mode 'prorate' (one pool) or 'each' (independent per slice).
- buildWhereAny() ORs the slices into one predicate. A union of slices
cannot be flattened into per-column IN lists without over-selecting,
and the result is parenthesised so the appended exclude clause does not
bind wrong.
- resolveIncrs() now resolves each measure independently: target, percent
or change amount per measure, so a target on value and a percent on
units can be submitted together. Replaces the single global `mode`.
- target_basis chooses what a target measures against: only the rows an
operation can write, or everything the pivot shows for the slice.
Excluded iters are visible in the grid but immovable, so a target set
against the visible total previously overshot by their contribution.
Two latent bugs surfaced by the above, both pre-existing:
- A slice naming no filterable column reduced to TRUE and applied the
operation to the entire version. Now rejected on all three operations.
- Prorating across a pool that nets to ~zero multiplies each row's share
by an exploding factor, sending rows to extreme opposite values to hit
the target. Refused when the net falls below 1% of gross.
Tags and the bridge
- pf.log gains a nullable `tag`, written by a follow-up UPDATE rather
than through the generated SQL: those templates are stored per source
in pf.sql, so a {{tag}} token would strand any source that had not
re-run "Generate SQL".
- Tag is editable after the fact in the change log, with completion from
tags already used on the source. PATCH branches on whether a field was
sent, so a tag can be cleared as well as set.
- BridgeView renders the walk from baseline to current as a waterfall,
one step per tag, scoped to the selection, the pivot's filters, or the
whole version. Computed from the loaded Perspective table so the
figures always reconcile with what is on screen; overlapping slices are
deduped by pf_id to match the OR semantics operations use.
- Colour is a polarity job, so it uses the validated diverging pair
(blue/red, CVD dE 21.6) with neutral anchors, not categorical hues.
Every bar is directly labelled and a table view is available.
Panel
- Extracted to OperationPanel; the scale form is one continuous ledger:
baseline, each adjustment, current, then New value / Change / % change
as three interchangeable editable rows. Typing in any one derives the
others, which removes the target/delta/percent mode toggle entirely.
- Dockable bottom, right, or floating (drag to move, grip to resize), and
closable via header, Esc, or the toolbar. Placement persists.
- Controls no longer stretch to the dock width, and text contrast now
clears WCAG AA against white throughout.
Also: the status bar names the physical table writes land in, with live
row counts; and the pivot's expand depth is re-applied when the tab
regains focus, since Perspective rebuilds its view on redraw and a
ROLLUP view with no depth set renders fully expanded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1TQiBYZbbWWkMNoCtUd8M
Ship rows pre-aggregated to the grain the pivot displays instead of raw
forecast rows. This is Path B from pf_perspective_options.md: it keeps
Perspective's native WASM engine — so expand/collapse/depth/sort/filter
all still work — and fixes load time by cutting rows, not transport.
Measured on pf.fc_osm_stack_20 at pending_rep x customer x smon:
534,902 -> 6,154 rows (~87x), pf_gkey unique across all 6,154, and both
measures reconcile exactly to the raw totals.
The grain is static: flagged once per source in Setup and baked into the
stored pf.sql templates, so load and operations agree by construction.
Sources with no flagged column keep the previous raw-row behaviour, so
this is backward compatible.
- pf.col_meta gains in_grain; grainOf() in lib/sql_generator.js is the
single definition of the grain and is reused by routes/log.js.
- New get_agg template + GET /api/versions/:id/agg, generated only when a
grain is defined. Regenerating drops templates no longer produced, so
clearing the grain falls back to /data.
- scale/recode/clone now aggregate their own new rows to grain before
returning. Because pf_logid is part of pf_gkey those keys are always
new, so table.update() appends and the view re-sums — the Excel
pivot-cache pattern, no bucket recomputation.
- Undo reports pf_gkeys (RETURNING cannot take DISTINCT, so the delete
feeds a CTE that reduces to distinct keys); the client removes those
index values and the view re-sums.
- pf_gkey is concat_ws(chr(31), COALESCE(col::text, chr(30)), ...).
The separator and NULL sentinel are load-bearing: plain concat_ws skips
NULLs, so ('a',NULL) and (NULL,'a') would collide and silently merge two
groups into one indexed row.
- Forecast.jsx reads col_meta first to pick /agg vs /data; the Arrow
streaming logic is extracted to fetchArrow() since both share it.
- Setup.jsx gains a grain checkbox and shows the resulting grain.
- 01_schema.sql: move the col_meta ALTERs after its CREATE TABLE — they
referenced the table before it existed on a fresh install.
All six generated statements verified to plan against the real forecast
table; the in_grain column has been added to the dev database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /versions/:id/log, DELETE /log/:logid, and PATCH /log/:logid were
defined in both routes/operations.js and routes/log.js. operations.js is
registered first, so its handlers shadowed log.js entirely (dead code).
Move the authoritative implementations (value/units totals in GET,
closed-version 403 guard in DELETE) into log.js and remove the duplicates
from operations.js, keeping operations.js focused on the forecast ops.
No behavior change — the served handlers were already the operations.js
versions; they are now defined once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /api/dim-period/cols queries information_schema for pf.dim_period columns
(excluding sdat/edat/drange/ndays) so the UI always reflects actual columns.
Setup col_meta editor now shows a dropdown populated from that endpoint instead
of a free-text field, preventing invalid column names like the cash source had.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SQL generator no longer requires a units col; recode/clone/scale omit units
expressions when none is configured in col_meta
- Source registration validation drops units from required roles (value + date
are the only hard requirements)
- DELETE /api/sources/:id returns 409 when existing versions reference the source
- Setup.jsx surfaces the 409 error via flash instead of silently failing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta gets dim_period_col field: maps a dimension column to its pf.dim_period counterpart (e.g. year -> cal_year, month -> cal_month)
- When the date column is is_key of a dim_group and any sibling dimension has dim_period_col set, baseline and reference SQL JOIN pf.dim_period on the shifted date instead of copying raw source values
- No dim_period config = identical SQL to before (fully backwards compatible)
- Setup UI: period col input in col_meta editor, enabled for dimension columns with a dim_group set
- Schema migration applied: dim_period_col text null on pf.col_meta
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GET /api/sources/:id/lookup?col=X&value=Y — given a key column value, queries the source table for sibling column values in the same dim_group; returns null if no match or ambiguous
- Recode and Clone panels: key columns (is_key + dim_group) trigger lookup on blur and auto-fill sibling inputs that the user hasn't already typed into
- Row labels now use col_meta label field when set, falling back to cname
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PATCH /api/log/:logid — saves note updates to pf.log (was missing, frontend call was silently failing)
- GET /api/versions/:id/log — joins fc_table to return row_count per entry so the change log modal shows rows affected instead of '—'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- col_meta: add dim_group field to group related columns (dimension hierarchies, date-adjacent columns); is_key now enabled for date role to mark group parent
- sources.js: upsert includes dim_group
- Setup.jsx: group column in col_meta editor, key checkbox enabled for date role
- gen_dim_period.sql: create and populate pf.dim_period with calendar and fiscal period cuts (monthly grain, 2018-2035)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch server Arrow encoding from tableFromJSON (row objects) to
tableFromArrays (column arrays) — cuts peak Node heap 3-5x for large
datasets by avoiding one JS object per row
- Remove unused pf.log JOIN from data endpoint; forecast rows only
- Load Perspective viewer with direct table reference instead of worker
Server object — fixes "No Table attached" error on large datasets where
named-table registry lookup raced against WASM initialization
- Pre-emptively clean up stale named table in worker registry before
creating, eliminating the "already exists" retry path that silently
swallowed errors (finally ran but flash never fired)
- Strip cfg.table from restore configs since table is loaded by reference
- Throttle progress bar updates to 100ms intervals (was every chunk)
- Persist load errors until dismissed; add console.error for devtools
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Add PayloadPreview component showing the exact JSON that will be POSTed,
live-updating as form fields change (value_incr shown as computed delta)
- buildEffectiveSlice strips expression/system columns and converts
Perspective ms-timestamps to ISO date strings for date-role columns
- fetchCurrentTotals now includes date columns in Perspective view filter
(passing ms number as Perspective expects) so subtotals respect the
clicked date
- Server buildWhere now receives filterCols (dimensions + date cols) so
date values reach the SQL WHERE clause correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forecast falls back to a saved per-source layout when no version-local
layout is cached, so new versions of a source open with a sensible pivot
without each user reconfiguring it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reference segments can now apply a date offset just like baselines.
SQL template gains the {{date_offset}} token; both POST /reference and
PUT baseline/:logid pass it through. Existing sources need to
regenerate SQL to pick up the new template — old stored reference SQL
ignores the token (preserving prior verbatim behavior). The Baseline
form drops the "dates land verbatim" hint and shows the offset
control for both segment types.
Editing a segment now color-codes the source row amber with a ring
and tints the form border + header amber so the active connection is
visually obvious. Header label reads "Edit segment #3 — baseline —
note" instead of just "#43" (the internal log id).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The segment form is now one component rendered in either 'view' or
'edit' mode — the expanded segment row in the list and the
add/edit form below share the same layout, view mode just disables
the inputs. Edit and View are visually identical so toggling between
them feels like enabling fields, not switching tools.
Filters become groups (conditions AND-ed inside, groups OR-ed
between) with + AND condition and + Add OR group affordances. The
compiled WHERE renders live below the groups so you can see what's
being built. A "Switch to manual SQL" toggle flips to a textarea
seeded with the compiled clause; backend baseline POST/PUT and
reference POST accept raw_where alongside filters and store whichever
arrived in pf.log.params for round-tripping.
The Add form is hidden until you click "+ Add segment" at the
bottom of the segments table; Edit also opens it. Cancel/Close
returns the table to its compact state.
/versions/:id/log now also returns value_total, units_total, and the
column names so the segments table can show row count and value sum
inline (header uses the source's actual value column name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds PUT /versions/:id/baseline/:logid that, in one transaction, drops
the segment's rows and log entry and replays the baseline or reference
SQL with new params. The endpoint refuses (409) if any scale, recode,
or clone has been applied — those operations were calibrated against
the old totals and would silently misreconcile.
Baseline view gets an Edit button on each segment (hidden once
forecast operations exist), populating the form with the original
filters, offset, and note. Submit issues PUT in edit mode, POST
otherwise. POST baseline and POST reference now also persist the
structured filters in pf.log.params so edit can reload them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Forecast tables don't carry pf_note; it's joined from pf.log on /data
fetch. RETURNING * from scale/recode/clone INSERTs lacks those fields,
so rows appended via table.update arrived with pf_note null until a
reload re-ran the join. Inject pf_note and pf_op server-side from the
request before responding.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Perspective table is now created with index: 'pf_id'. Delete endpoints
return the pf_ids they removed; the client calls table.remove(pf_ids)
in undoEntry. Avoids the full /data refetch that dominated undo time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
/data now joins pf.log to surface note text and operation type as
pf_note/pf_op so users can pivot/bridge by assumption. Joining at
fetch time avoids storing notes per row and keeps edits live.
/api/tables joined pg_class by name only with namespace filtered in
a separate LEFT JOIN, which cross-producted table names that exist in
multiple schemas. Restructured so namespace participates in the join.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
