setup_sql/01_schema.sql was an idempotent bootstrap script — CREATE TABLE
IF NOT EXISTS plus a tail of ALTER ... ADD COLUMN IF NOT EXISTS. It had no
record of what any given database had applied, which is how a branch could
declare col_meta.in_grain while the running database lacked it, with
nothing able to detect the mismatch. The symptom would have been a
confusing 'column "in_grain" does not exist' inside an unrelated request.
Migrations-only, no hand-maintained current-state file to drift:
- setup_sql/migrations/*.sql applied in filename order, recorded in
pf.schema_version with a checksum. Split along the schema's actual
evolution, so each column is declared exactly once — 01_schema.sql had
grown to declare dim_group, dim_period_col and in_grain twice each.
- lib/migrations.js holds the bookkeeping, shared by the CLI and the boot
check. scripts/migrate.js provides up | status | baseline.
- server.js refuses to start when the database is behind, listing what is
pending. This converts silent drift into a clear boot message, which was
the whole point. PF_SKIP_MIGRATION_CHECK=1 bypasses.
- Four integrity guards, each verified to fire: a migration modified after
being applied, one recorded as applied but missing from disk, one that
would apply out of order, and a re-run when already current.
- No IF NOT EXISTS on new migrations. The bookkeeping already guarantees
one run each, and the guards hide ordering mistakes — that is exactly why
01_schema.sql had ALTERs sitting above the CREATE TABLE they depended on,
broken for anyone installing from scratch. 0004 keeps the guard only
because it was applied by hand before migrations existed.
Verified: replaying all four migrations into a throwaway schema reproduces
the live pf schema exactly, 41 columns, column for column.
schema.generated.sql is a pg_dump snapshot for reading, refreshed by
npm run schema:dump. It excludes the runtime fc_* tables and strips
pg_dump's random \restrict token and version banner, so regenerating an
unchanged schema yields an identical file rather than a spurious diff.
pf.dim_period stays out of migrations — it is a parameterised data load
(fiscal year start month), not a schema change.
The dev database (ubm) has been baselined at all four migrations.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Server streams rows from a pg cursor in 10k-row batches, building Arrow
record batches incrementally and piping them as chunked HTTP response —
Node.js heap stays bounded regardless of dataset size.
Client fetches as arrayBuffer() and loads directly into Perspective worker
(native Arrow path, no JSON deserialization). X-Row-Count header drives
a non-blocking banner for datasets >= 500k rows. validCols now derived
from col_meta rather than from row keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ui/: React + Vite + Tailwind app (Setup, Baseline, Forecast views, collapsible sidebar, status bar, canvas timeline)
- server.js: serve built UI from public/app/
- package.json: add build script (cd ui && npm run build)
- routes/sources.js: default new col_meta role to 'dimension' instead of 'ignore'
- .gitignore: exclude public/app/ build output
- pf_spec.md: update tech stack, nav, frontend section, and project status to reflect current implementation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
