pt/pf_app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace idempotent schema script with tracked forward-only migrations

Browse Source 
setup_sql/01_schema.sql was an idempotent bootstrap script — CREATE TABLE
IF NOT EXISTS plus a tail of ALTER ... ADD COLUMN IF NOT EXISTS. It had no
record of what any given database had applied, which is how a branch could
declare col_meta.in_grain while the running database lacked it, with
nothing able to detect the mismatch. The symptom would have been a
confusing 'column "in_grain" does not exist' inside an unrelated request.

Migrations-only, no hand-maintained current-state file to drift:

- setup_sql/migrations/*.sql applied in filename order, recorded in
  pf.schema_version with a checksum. Split along the schema's actual
  evolution, so each column is declared exactly once — 01_schema.sql had
  grown to declare dim_group, dim_period_col and in_grain twice each.
- lib/migrations.js holds the bookkeeping, shared by the CLI and the boot
  check. scripts/migrate.js provides up | status | baseline.
- server.js refuses to start when the database is behind, listing what is
  pending. This converts silent drift into a clear boot message, which was
  the whole point. PF_SKIP_MIGRATION_CHECK=1 bypasses.
- Four integrity guards, each verified to fire: a migration modified after
  being applied, one recorded as applied but missing from disk, one that
  would apply out of order, and a re-run when already current.
- No IF NOT EXISTS on new migrations. The bookkeeping already guarantees
  one run each, and the guards hide ordering mistakes — that is exactly why
  01_schema.sql had ALTERs sitting above the CREATE TABLE they depended on,
  broken for anyone installing from scratch. 0004 keeps the guard only
  because it was applied by hand before migrations existed.

Verified: replaying all four migrations into a throwaway schema reproduces
the live pf schema exactly, 41 columns, column for column.

schema.generated.sql is a pg_dump snapshot for reading, refreshed by
npm run schema:dump. It excludes the runtime fc_* tables and strips
pg_dump's random \restrict token and version banner, so regenerating an
unchanged schema yields an identical file rather than a spurious diff.

pf.dim_period stays out of migrations — it is a parameterised data load
(fiscal year start month), not a schema change.

The dev database (ubm) has been baselined at all four migrations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-08-19 09:07:39 -04:00
parent 654a368672
commit d1197df7d5
14 changed files with 873 additions and 79 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
CLAUDE.md
View File

@ -16,6 +16,7 @@ Data transport architecture options: `pf_perspective_options.md`
- **Frontend:** React + Vite + Tailwind CSS in `ui/`; built output lands in `public/app/`
- **Pivot:** [Perspective](https://github.com/perspective-dev/perspective) (`@perspective-dev/*` distribution, **not** FINOS `@finos/perspective`) 4.4.0 loaded from CDN at runtime — see `PERSPECTIVE.md` for config/deploy guidance
- **Dev:** `npm run dev` (nodemon) in root; `npm run build` in `ui/`
- **Schema:** forward-only SQL migrations in `setup_sql/migrations/`, applied by `npm run migrate` and tracked in `pf.schema_version`. `server.js` refuses to start when the database is behind. See `setup_sql/README.md`
---
@ -33,7 +34,13 @@ lib/
sql_generator.js buildFilterClause, token substitution helpers
utils.js
setup_sql/
01_schema.sql pf schema DDL — run once to install
README.md migration workflow — read before changing the schema
migrations/ ordered .sql, applied once, tracked in pf.schema_version
gen_dim_period.sql parameterised calendar load (not a migration)
schema.generated.sql pg_dump reference snapshot; generated, never edited
scripts/
migrate.js migration CLI (up | status | baseline)
schema-dump.js regenerates schema.generated.sql
ui/src/
views/
Setup.jsx DB browser, source registration, col_meta editor
@ -55,6 +62,7 @@ ui/src/
- **`pf.log`** — audit log; every write gets one entry; `slice` + `params` stored as jsonb
- **`pf.sql`** — generated SQL templates per source/operation; tokens substituted at request time
- **`pf.dim_period`** — calendar lookup table (2018–2035); one row per month keyed on `sdat` (month start date); provides cal/fiscal year, quarter, and month columns; populated by `setup_sql/gen_dim_period.sql` with a configurable fiscal year start month
- **`pf.schema_version`** — applied migrations (filename + checksum); owned by the migration runner, never edited by hand
### Key token substitution tokens
`{{fc_table}}`, `{{where_clause}}`, `{{exclude_clause}}`, `{{logid}}`, `{{pf_user}}`, `{{value_incr}}`, `{{units_incr}}`, `{{pct}}`, `{{set_clause}}`, `{{scale_factor}}`, `{{date_offset}}`, `{{filter_clause}}`
@ -123,6 +131,7 @@ Theme state lives in `ui/src/theme.jsx` — a React context (`ThemeContext`) wit
- Source/version selection doesn't persist across page reload
- Col_meta / version schema drift: if col_meta roles change after a version's forecast table is created, SQL and DDL go out of sync — workaround is to delete and recreate the version
- Grain drift: changing `in_grain` after a load requires Generate SQL + a page reload, since the loaded table's index and columns are fixed at load time. `routes/log.js` derives the grain from live col_meta, so a grain changed mid-session yields `pf_gkeys` that don't match the loaded table and undo silently removes nothing
- Migrations are forward-only — there are no down migrations. Rolling back a schema change means writing a new migration that reverses it
- Grain is static per source — a dimension left unflagged cannot be pivoted on. Dynamic per-cut grain (intersect the viewer's field set with the eligible set) is the additive next step; see `pf_spec.md` → §Display-grain pre-aggregation
## Deferred (not in v1)

158
lib/migrations.js Normal file
View File

@ -0,0 +1,158 @@
// Migration bookkeeping shared by the runner (scripts/migrate.js) and the
// startup check in server.js.
//
// Migrations are plain .sql files in setup_sql/migrations, applied in filename
// order and recorded in pf.schema_version. They are immutable once applied: the
// runner stores a checksum and refuses to proceed if a file has changed, since
// editing an applied migration means databases silently disagree about what the
// schema is.
//
// pf.dim_period is deliberately not a migration — it is a parameterised data
// load (fiscal year start month), so it stays in setup_sql/gen_dim_period.sql.
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const MIGRATIONS_DIR = path.join(__dirname, '..', 'setup_sql', 'migrations');
// Bookkeeping table has to exist before the first migration can be recorded, so
// the runner creates it rather than a migration doing so.
const BOOKKEEPING_DDL = `
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE IF NOT EXISTS pf.schema_version (
filename text PRIMARY KEY,
checksum text NOT NULL,
applied_at timestamptz NOT NULL DEFAULT now(),
applied_by text
);
`;
function checksum(sql) {
return crypto.createHash('sha256').update(sql).digest('hex').slice(0, 16);
}
// every .sql file on disk, in apply order
function readMigrations() {
if (!fs.existsSync(MIGRATIONS_DIR)) return [];
return fs.readdirSync(MIGRATIONS_DIR)
.filter(f => f.endsWith('.sql'))
.sort()
.map(filename => {
const sql = fs.readFileSync(path.join(MIGRATIONS_DIR, filename), 'utf8');
return { filename, sql, checksum: checksum(sql) };
});
}
async function ensureBookkeeping(client) {
await client.query(BOOKKEEPING_DDL);
}
async function readApplied(client) {
const { rows } = await client.query(
`SELECT filename, checksum, applied_at FROM pf.schema_version ORDER BY filename`
);
return rows;
}
// Compare disk against the database. Returns pending migrations plus any
// integrity problems worth refusing to run on.
async function status(client) {
await ensureBookkeeping(client);
const onDisk = readMigrations();
const applied = await readApplied(client);
const appliedBy = new Map(applied.map(r => [r.filename, r]));
const diskBy = new Map(onDisk.map(m => [m.filename, m]));
const pending = onDisk.filter(m => !appliedBy.has(m.filename));
// an applied file whose contents changed — the schema is now undefined
const modified = onDisk
.filter(m => appliedBy.has(m.filename) && appliedBy.get(m.filename).checksum !== m.checksum)
.map(m => m.filename);
// recorded as applied but no longer on disk — someone deleted or renamed it
const missing = applied.filter(r => !diskBy.has(r.filename)).map(r => r.filename);
// a pending migration ordered before an applied one would apply out of sequence
const lastApplied = applied.length ? applied[applied.length - 1].filename : null;
const outOfOrder = lastApplied ? pending.filter(m => m.filename < lastApplied).map(m => m.filename) : [];
return { onDisk, applied, pending, modified, missing, outOfOrder };
}
// Apply pending migrations, each in its own transaction so a failure leaves
// earlier ones committed and the failing one fully rolled back.
async function migrate(client, { user, log = console.log } = {}) {
const st = await status(client);
if (st.modified.length) {
throw new Error(
`Applied migrations have been modified: ${st.modified.join(', ')}. ` +
`Migrations are immutable — revert the edit and add a new migration instead.`
);
}
if (st.missing.length) {
throw new Error(
`Migrations recorded as applied are missing from disk: ${st.missing.join(', ')}.`
);
}
if (st.outOfOrder.length) {
throw new Error(
`Migrations would apply out of order: ${st.outOfOrder.join(', ')} sort before ` +
`already-applied migrations. Renumber them after the latest applied migration.`
);
}
if (!st.pending.length) {
log('Schema is up to date — no migrations to apply.');
return [];
}
const done = [];
for (const m of st.pending) {
log(`applying ${m.filename} …`);
try {
await client.query('BEGIN');
await client.query(m.sql);
await client.query(
`INSERT INTO pf.schema_version (filename, checksum, applied_by) VALUES ($1, $2, $3)`,
[m.filename, m.checksum, user || null]
);
await client.query('COMMIT');
done.push(m.filename);
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
throw new Error(`${m.filename} failed: ${err.message}`);
}
}
log(`Applied ${done.length} migration(s).`);
return done;
}
// Record migrations as applied without running them — for a database whose schema
// already matches, from before migrations existed.
async function baseline(client, { user, upTo, log = console.log } = {}) {
const st = await status(client);
const target = upTo
? st.pending.filter(m => m.filename <= upTo)
: st.pending;
if (!target.length) {
log('Nothing to baseline — no pending migrations.');
return [];
}
await client.query('BEGIN');
try {
for (const m of target) {
await client.query(
`INSERT INTO pf.schema_version (filename, checksum, applied_by)
VALUES ($1, $2, $3)
ON CONFLICT (filename) DO NOTHING`,
[m.filename, m.checksum, user ? `${user} (baseline)` : 'baseline']
);
}
await client.query('COMMIT');
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
throw err;
}
log(`Marked ${target.length} migration(s) as applied without running them.`);
return target.map(m => m.filename);
}
module.exports = { MIGRATIONS_DIR, readMigrations, status, migrate, baseline, checksum };

6
package.json
View File

@ -7,7 +7,11 @@
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js",
"build": "cd ui && npm run build"
"build": "cd ui && npm run build",
"migrate": "node scripts/migrate.js up",
"migrate:status": "node scripts/migrate.js status",
"migrate:baseline": "node scripts/migrate.js baseline",
"schema:dump": "node scripts/schema-dump.js"
},
"dependencies": {
"apache-arrow": "^21.1.0",

15
pf_spec.md
View File

@ -189,9 +189,22 @@ CREATE TABLE pf.sql (
```
setup_sql/
01_schema.sql -- CREATE SCHEMA pf; create all metadata tables (source, col_meta, version, log, sql)
migrations/ -- forward-only .sql, applied once each, tracked in pf.schema_version
0001_initial_schema.sql CREATE SCHEMA pf + metadata tables
0002_source_default_layout.sql
0003_col_meta_dim_group_period.sql
0004_col_meta_in_grain.sql
gen_dim_period.sql -- parameterised calendar load (fiscal year start month); not a migration
schema.generated.sql -- pg_dump reference snapshot; generated, never edited or applied
README.md -- migration workflow
```
Install with `npm run migrate`, then run `gen_dim_period.sql`. For a database that
already matches the schema, `npm run migrate:baseline` records the migrations as
applied without running them. `server.js` refuses to start when the database is
behind, so drift surfaces at boot rather than as a column-not-found error inside
an unrelated request.
Source registration, col_meta configuration, SQL generation, version creation, and forecast table DDL all happen via API.
---

72
scripts/migrate.js Normal file
View File

@ -0,0 +1,72 @@
#!/usr/bin/env node
// Migration CLI.
//
// npm run migrate apply pending migrations
// npm run migrate:status show what is applied and what is pending
// npm run migrate:baseline record pending migrations as applied WITHOUT running
// them — for a database that already matches
//
// Pass --up-to=<filename> to baseline only through a given migration.
require('dotenv').config();
const os = require('os');
const { Client } = require('pg');
const { status, migrate, baseline } = require('../lib/migrations');
const args = process.argv.slice(2);
const command = args.find(a => !a.startsWith('--')) || 'up';
const upTo = (args.find(a => a.startsWith('--up-to=')) || '').split('=')[1] || null;
function connect() {
return new Client({
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT) || 5432,
database: process.env.DB_NAME,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
ssl: false
});
}
async function showStatus(client) {
const st = await status(client);
console.log(`\n${process.env.DB_NAME} (${st.applied.length} applied, ${st.pending.length} pending)\n`);
for (const m of st.onDisk) {
const hit = st.applied.find(a => a.filename === m.filename);
const mark = !hit ? 'PENDING'
: st.modified.includes(m.filename) ? 'MODIFIED'
: 'applied';
const when = hit ? hit.applied_at.toISOString().slice(0, 19).replace('T', ' ') : '';
console.log(` ${mark.padEnd(9)} ${m.filename.padEnd(40)} ${when}`);
}
for (const f of st.missing) console.log(` MISSING ${f.padEnd(40)} recorded as applied but not on disk`);
if (st.modified.length) console.log(`\n ! modified after being applied: ${st.modified.join(', ')}`);
if (st.outOfOrder.length) console.log(` ! would apply out of order: ${st.outOfOrder.join(', ')}`);
console.log('');
return st;
}
(async () => {
const client = connect();
await client.connect();
try {
const user = `${os.userInfo().username}@${os.hostname()}`;
if (command === 'status') {
const st = await showStatus(client);
process.exitCode = (st.modified.length || st.missing.length) ? 1 : 0;
} else if (command === 'baseline') {
await baseline(client, { user, upTo });
await showStatus(client);
} else if (command === 'up') {
await migrate(client, { user });
} else {
console.error(`Unknown command "${command}" — expected up, status, or baseline.`);
process.exitCode = 2;
}
} catch (err) {
console.error(`\nmigration error: ${err.message}\n`);
process.exitCode = 1;
} finally {
await client.end();
}
})();

50
scripts/schema-dump.js Normal file
View File

@ -0,0 +1,50 @@
#!/usr/bin/env node
// Regenerate setup_sql/schema.generated.sql — a readable snapshot of the current
// pf schema, for reference only. Migrations are the source of truth; this file is
// generated so it cannot drift from the database the way a hand-maintained
// schema file does. Never edit it, and never apply it to create a database.
require('dotenv').config();
const { spawnSync } = require('child_process');
const fs = require('fs');
const path = require('path');
const out = path.join(__dirname, '..', 'setup_sql', 'schema.generated.sql');
const result = spawnSync('pg_dump', [
'--schema-only', '--no-owner', '--no-privileges',
'--schema=pf',
// per-version forecast tables are created at runtime — including them would
// make this file churn every time a version is created or dropped
'--exclude-table=pf.fc_*',
'-f', out
], {
stdio: 'inherit',
env: {
...process.env,
PGHOST: process.env.DB_HOST,
PGPORT: process.env.DB_PORT || '5432',
PGDATABASE: process.env.DB_NAME,
PGUSER: process.env.DB_USER,
PGPASSWORD: process.env.DB_PASSWORD
}
});
if (result.error) {
console.error(`pg_dump failed: ${result.error.message}`);
process.exit(1);
}
if (result.status !== 0) process.exit(result.status);
// Strip the lines pg_dump varies between runs — a random \restrict token and the
// server/client version banner — so regenerating an unchanged schema produces an
// identical file instead of a spurious diff.
const cleaned = fs.readFileSync(out, 'utf8')
.split('\n')
.filter(l => !/^\\(un)?restrict /.test(l))
.filter(l => !/^-- Dumped (from|by) /.test(l))
.join('\n')
.replace(/\n{3,}/g, '\n\n');
fs.writeFileSync(out, cleaned);
console.log(`wrote ${path.relative(process.cwd(), out)} (${cleaned.split('\n').length} lines)`);

35
server.js
View File

@ -33,5 +33,38 @@ app.use('/api', require('./routes/operations')(pool));
app.use('/api', require('./routes/log')(pool));
// Refuse to start against a database whose schema is behind the migrations on
// disk. Without this the mismatch surfaces later as a confusing query error
// ('column "x" does not exist') deep inside an unrelated request.
// Set PF_SKIP_MIGRATION_CHECK=1 to bypass.
async function checkSchema() {
if (process.env.PF_SKIP_MIGRATION_CHECK === '1') return;
const { status } = require('./lib/migrations');
const client = await pool.connect();
try {
const st = await status(client);
const problems = [
st.pending.length && `${st.pending.length} pending: ${st.pending.map(m => m.filename).join(', ')}`,
st.modified.length && `modified after being applied: ${st.modified.join(', ')}`,
st.missing.length && `applied but missing from disk: ${st.missing.join(', ')}`
].filter(Boolean);
if (problems.length) {
console.error('\n Database schema is out of date:');
for (const p of problems) console.error(` - ${p}`);
console.error('\n Run "npm run migrate" (or "npm run migrate:status" for detail).');
console.error(' For a database that already matches, "npm run migrate:baseline".\n');
throw new Error('schema out of date');
}
console.log(`schema up to date (${st.applied.length} migrations applied)`);
} finally {
client.release();
}
}
const port = process.env.PORT || 3010;
app.listen(port, '0.0.0.0', () => console.log(`pf_app started on port ${port}`));
checkSchema()
.then(() => app.listen(port, '0.0.0.0', () => console.log(`pf_app started on port ${port}`)))
.catch((err) => {
if (err.message !== 'schema out of date') console.error('startup failed:', err.message);
process.exit(1);
});

75
setup_sql/01_schema.sql
View File

@ -1,75 +0,0 @@
-- Pivot Forecast schema install
-- Run once against target database: psql -d <db> -f setup_sql/01_schema.sql
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE IF NOT EXISTS pf.source (
id serial PRIMARY KEY,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text NOT NULL DEFAULT 'active', -- active | archived
default_layout jsonb, -- Perspective view config used as the per-source default
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
UNIQUE (schema, tname)
);
-- backfill columns for existing installs
ALTER TABLE pf.source ADD COLUMN IF NOT EXISTS default_layout jsonb;
-- pf.dim_period: run setup_sql/gen_dim_period.sql to create and populate
CREATE TABLE IF NOT EXISTS pf.col_meta (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
cname text NOT NULL,
label text,
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | ignore
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
dim_group text, -- groups functionally dependent columns
dim_period_col text, -- pf.dim_period column this dimension derives from
in_grain boolean NOT NULL DEFAULT false, -- true = column defines the display grain
opos integer,
UNIQUE (source_id, cname)
);
-- backfill columns for existing installs (must follow the CREATE above)
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS dim_period_col text;
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;
CREATE TABLE IF NOT EXISTS pf.version (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
name text NOT NULL,
description text,
status text NOT NULL DEFAULT 'open', -- open | closed
exclude_iters jsonb NOT NULL DEFAULT '["reference"]'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
closed_at timestamptz,
closed_by text,
UNIQUE (source_id, name)
);
CREATE TABLE IF NOT EXISTS pf.log (
id bigserial PRIMARY KEY,
version_id integer NOT NULL REFERENCES pf.version(id) ON DELETE CASCADE,
pf_user text NOT NULL,
stamp timestamptz NOT NULL DEFAULT now(),
operation text NOT NULL, -- baseline | reference | scale | recode | clone
slice jsonb,
params jsonb,
note text
);
-- generated operation SQL per source, stored after col_meta is configured
CREATE TABLE IF NOT EXISTS pf.sql (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
operation text NOT NULL, -- get_data | baseline | reference | scale | recode | clone | undo
sql text NOT NULL,
generated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_id, operation)
);

64
setup_sql/README.md Normal file
View File

@ -0,0 +1,64 @@
# Database setup
## Migrations
`migrations/*.sql` are applied in filename order and recorded in
`pf.schema_version`. They are the **source of truth** for the `pf` schema — there
is no hand-maintained current-state file to drift out of sync.
```bash
npm run migrate # apply pending migrations
npm run migrate:status # what is applied, what is pending
npm run migrate:baseline # record pending as applied WITHOUT running them
```
`server.js` refuses to start when the database is behind, so drift surfaces at
boot rather than as `column "x" does not exist` inside an unrelated request. Set
`PF_SKIP_MIGRATION_CHECK=1` to bypass.
### Fresh database
```bash
npm run migrate
psql -d <db> -f setup_sql/gen_dim_period.sql
```
### Existing database that already matches
Use `baseline` so the runner does not try to re-create tables that exist:
```bash
npm run migrate:baseline
```
To baseline only part of the way — the schema matches through `0003` but not
`0004` — pass `--up-to` and then migrate the rest:
```bash
node scripts/migrate.js baseline --up-to=0003_col_meta_dim_group_period.sql
npm run migrate
```
## Writing a migration
- Name it `NNNN_short_description.sql`, numbered after the highest existing file.
- One concern per file. Keep it forward-only; there are no down migrations.
- **Applied migrations are immutable.** The runner stores a checksum and refuses
to proceed if a file changes after being applied, because editing one means
databases silently disagree about what the schema is. To fix a mistake, add a
new migration.
- No `IF NOT EXISTS` guards on new migrations. The bookkeeping already guarantees
each runs once, and the guards hide ordering mistakes — the reason the old
`01_schema.sql` had `ALTER`s sitting above the `CREATE TABLE` they depended on,
broken for anyone installing from scratch. `0004` is the one exception, since it
was applied by hand before migrations existed.
## Not migrations
- **`gen_dim_period.sql`** — creates and populates `pf.dim_period`. It is a
parameterised data load (configurable fiscal year start month), not a schema
change, so it stays a script you run deliberately.
- **`pf.fc_{tname}_{version_id}`** — per-version forecast tables, created and
dropped at runtime by `routes/versions.js` from `col_meta`. Never migrated.
- **`schema.generated.sql`** — a `pg_dump` snapshot for reading, refreshed with
`npm run schema:dump`. Generated, never edited, never applied.

63
setup_sql/migrations/0001_initial_schema.sql Normal file
View File

@ -0,0 +1,63 @@
-- Initial pf schema: sources, column metadata, versions, audit log, generated SQL.
--
-- This is the schema as it stood before the additive columns in later migrations.
-- Columns added afterwards are declared once, in their own migration — not here.
CREATE SCHEMA IF NOT EXISTS pf;
CREATE TABLE pf.source (
id serial PRIMARY KEY,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text NOT NULL DEFAULT 'active', -- active | archived
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
UNIQUE (schema, tname)
);
CREATE TABLE pf.col_meta (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
cname text NOT NULL,
label text,
role text NOT NULL DEFAULT 'ignore', -- dimension | value | units | date | filter | ignore
is_key boolean NOT NULL DEFAULT false, -- true = usable in WHERE slice
opos integer,
UNIQUE (source_id, cname)
);
CREATE TABLE pf.version (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE RESTRICT,
name text NOT NULL,
description text,
status text NOT NULL DEFAULT 'open', -- open | closed
exclude_iters jsonb NOT NULL DEFAULT '["reference"]'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
created_by text,
closed_at timestamptz,
closed_by text,
UNIQUE (source_id, name)
);
CREATE TABLE pf.log (
id bigserial PRIMARY KEY,
version_id integer NOT NULL REFERENCES pf.version(id) ON DELETE CASCADE,
pf_user text NOT NULL,
stamp timestamptz NOT NULL DEFAULT now(),
operation text NOT NULL, -- baseline | reference | scale | recode | clone
slice jsonb,
params jsonb,
note text
);
-- generated operation SQL per source, stored after col_meta is configured
CREATE TABLE pf.sql (
id serial PRIMARY KEY,
source_id integer NOT NULL REFERENCES pf.source(id) ON DELETE CASCADE,
operation text NOT NULL, -- get_data | get_agg | baseline | reference | scale | recode | clone | undo
sql text NOT NULL,
generated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_id, operation)
);

4
setup_sql/migrations/0002_source_default_layout.sql Normal file
View File

@ -0,0 +1,4 @@
-- Per-source default Perspective view config, applied when a version has no
-- saved layout of its own.
ALTER TABLE pf.source ADD COLUMN default_layout jsonb;

7
setup_sql/migrations/0003_col_meta_dim_group_period.sql Normal file
View File

@ -0,0 +1,7 @@
-- dim_group groups functionally dependent columns (e.g. a date and the year/month
-- dimensions derived from it). dim_period_col maps such a dimension to a
-- pf.dim_period column, so date-adjacent values are derived by JOIN at load time
-- rather than copied raw from the source.
ALTER TABLE pf.col_meta ADD COLUMN dim_group text;
ALTER TABLE pf.col_meta ADD COLUMN dim_period_col text;

8
setup_sql/migrations/0004_col_meta_in_grain.sql Normal file
View File

@ -0,0 +1,8 @@
-- in_grain flags dimension/date columns that define the display grain: the
-- forecast load is pre-aggregated to the flagged columns instead of shipping raw
-- rows. See pf_spec.md -> Display-grain pre-aggregation.
--
-- Already applied by hand on the original dev database before migrations existed;
-- IF NOT EXISTS keeps replay safe there. New migrations should not need the guard.
ALTER TABLE pf.col_meta ADD COLUMN IF NOT EXISTS in_grain boolean NOT NULL DEFAULT false;

384
setup_sql/schema.generated.sql Normal file
View File

@ -0,0 +1,384 @@
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET transaction_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Name: pf; Type: SCHEMA; Schema: -; Owner: -
--
CREATE SCHEMA pf;
SET default_tablespace = '';
SET default_table_access_method = heap;
--
-- Name: col_meta; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.col_meta (
id integer NOT NULL,
source_id integer NOT NULL,
cname text NOT NULL,
label text,
role text DEFAULT 'ignore'::text NOT NULL,
is_key boolean DEFAULT false NOT NULL,
opos integer,
dim_group text,
dim_period_col text,
in_grain boolean DEFAULT false NOT NULL
);
--
-- Name: col_meta_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.col_meta_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: col_meta_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.col_meta_id_seq OWNED BY pf.col_meta.id;
--
-- Name: dim_period; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.dim_period (
sdat date NOT NULL,
edat date NOT NULL,
drange daterange NOT NULL,
ndays integer NOT NULL,
cal_year integer NOT NULL,
cal_quarter integer NOT NULL,
cal_month integer NOT NULL,
cal_month_abbr text NOT NULL,
cal_month_name text NOT NULL,
cal_label text NOT NULL,
fisc_year integer NOT NULL,
fisc_quarter integer NOT NULL,
fisc_quarter_label text NOT NULL,
fisc_month integer NOT NULL,
fisc_month_abbr text NOT NULL,
fisc_month_name text NOT NULL,
fisc_label text NOT NULL,
period_key text NOT NULL
);
--
-- Name: log; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.log (
id bigint NOT NULL,
version_id integer NOT NULL,
pf_user text NOT NULL,
stamp timestamp with time zone DEFAULT now() NOT NULL,
operation text NOT NULL,
slice jsonb,
params jsonb,
note text
);
--
-- Name: log_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.log_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: log_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.log_id_seq OWNED BY pf.log.id;
--
-- Name: schema_version; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.schema_version (
filename text NOT NULL,
checksum text NOT NULL,
applied_at timestamp with time zone DEFAULT now() NOT NULL,
applied_by text
);
--
-- Name: source; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.source (
id integer NOT NULL,
schema text NOT NULL,
tname text NOT NULL,
label text,
status text DEFAULT 'active'::text NOT NULL,
created_at timestamp with time zone DEFAULT now() NOT NULL,
created_by text,
default_layout jsonb
);
--
-- Name: source_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.source_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: source_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.source_id_seq OWNED BY pf.source.id;
--
-- Name: sql; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.sql (
id integer NOT NULL,
source_id integer NOT NULL,
operation text NOT NULL,
sql text NOT NULL,
generated_at timestamp with time zone DEFAULT now() NOT NULL
);
--
-- Name: sql_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.sql_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: sql_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.sql_id_seq OWNED BY pf.sql.id;
--
-- Name: version; Type: TABLE; Schema: pf; Owner: -
--
CREATE TABLE pf.version (
id integer NOT NULL,
source_id integer NOT NULL,
name text NOT NULL,
description text,
status text DEFAULT 'open'::text NOT NULL,
exclude_iters jsonb DEFAULT '["reference"]'::jsonb NOT NULL,
created_at timestamp with time zone DEFAULT now() NOT NULL,
created_by text,
closed_at timestamp with time zone,
closed_by text
);
--
-- Name: version_id_seq; Type: SEQUENCE; Schema: pf; Owner: -
--
CREATE SEQUENCE pf.version_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
--
-- Name: version_id_seq; Type: SEQUENCE OWNED BY; Schema: pf; Owner: -
--
ALTER SEQUENCE pf.version_id_seq OWNED BY pf.version.id;
--
-- Name: col_meta id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta ALTER COLUMN id SET DEFAULT nextval('pf.col_meta_id_seq'::regclass);
--
-- Name: log id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log ALTER COLUMN id SET DEFAULT nextval('pf.log_id_seq'::regclass);
--
-- Name: source id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source ALTER COLUMN id SET DEFAULT nextval('pf.source_id_seq'::regclass);
--
-- Name: sql id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql ALTER COLUMN id SET DEFAULT nextval('pf.sql_id_seq'::regclass);
--
-- Name: version id; Type: DEFAULT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version ALTER COLUMN id SET DEFAULT nextval('pf.version_id_seq'::regclass);
--
-- Name: col_meta col_meta_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_pkey PRIMARY KEY (id);
--
-- Name: col_meta col_meta_source_id_cname_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_source_id_cname_key UNIQUE (source_id, cname);
--
-- Name: dim_period dim_period_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.dim_period
ADD CONSTRAINT dim_period_pkey PRIMARY KEY (sdat);
--
-- Name: log log_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log
ADD CONSTRAINT log_pkey PRIMARY KEY (id);
--
-- Name: schema_version schema_version_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.schema_version
ADD CONSTRAINT schema_version_pkey PRIMARY KEY (filename);
--
-- Name: source source_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source
ADD CONSTRAINT source_pkey PRIMARY KEY (id);
--
-- Name: source source_schema_tname_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.source
ADD CONSTRAINT source_schema_tname_key UNIQUE (schema, tname);
--
-- Name: sql sql_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_pkey PRIMARY KEY (id);
--
-- Name: sql sql_source_id_operation_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_source_id_operation_key UNIQUE (source_id, operation);
--
-- Name: version version_pkey; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_pkey PRIMARY KEY (id);
--
-- Name: version version_source_id_name_key; Type: CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_source_id_name_key UNIQUE (source_id, name);
--
-- Name: dim_period_cal_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_cal_idx ON pf.dim_period USING btree (cal_year, cal_month);
--
-- Name: dim_period_drange_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_drange_idx ON pf.dim_period USING gist (drange);
--
-- Name: dim_period_fisc_idx; Type: INDEX; Schema: pf; Owner: -
--
CREATE INDEX dim_period_fisc_idx ON pf.dim_period USING btree (fisc_year, fisc_month);
--
-- Name: col_meta col_meta_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.col_meta
ADD CONSTRAINT col_meta_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE CASCADE;
--
-- Name: log log_version_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.log
ADD CONSTRAINT log_version_id_fkey FOREIGN KEY (version_id) REFERENCES pf.version(id) ON DELETE CASCADE;
--
-- Name: sql sql_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.sql
ADD CONSTRAINT sql_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE CASCADE;
--
-- Name: version version_source_id_fkey; Type: FK CONSTRAINT; Schema: pf; Owner: -
--
ALTER TABLE ONLY pf.version
ADD CONSTRAINT version_source_id_fkey FOREIGN KEY (source_id) REFERENCES pf.source(id) ON DELETE RESTRICT;
--
-- PostgreSQL database dump complete
--