pipekit/docs/cms_migration.md

CMS (DB2) → pipekit migration

Scope and plan for moving the remaining CMS/DB2 table syncs from the legacy /opt/sync (jrunner + shell + cron) tool onto pipekit modules.

Decisions

• Destination schema: cms.*. All migrated CMS tables land in cms.<table> (pipekit convention), not the legacy lgdat.*. Downstream rlarp.* views that currently read lgdat.* must be repointed to cms.* as each table is cut over.
• Sync tables as they reside on DB2. Column set and types come from fresh introspection of the DB2 source (QSYS2.SYSCOLUMNS), not from the legacy pull.sql / build.sql. Legacy queries are not ported; we regenerate clean.
• One table at a time on cutover. Generate/verify the module, repoint the downstream view, then retire the /opt/sync module.

Current state

• pipekit already covers 39 CMS tables (source_connection: S78030956, FROM LGDAT.*, landing in cms.*).
• /opt/sync/s7830956/ (92 module dirs) holds the legacy CMS syncs. 24 overlap the 39; the rest is the migration backlog below.

The module recipe (pipekit)

A module is two sidecar files under config/modules/, named for the module:

• <name>.json — definition. Required fields: name, source_connection, dest_connection, dest_table (schema-qualified), staging_table (pipekit_staging.<name>), merge_strategy (full | incremental | append), merge_key (required for incremental, else null), enabled, columns[], watermarks[], hooks[]. dest_description optional.
  • columns[] entries: source_name, source_type, dest_name, dest_type, description.
• <name>.sql — the extract SELECT, run on the source connection in its dialect. DB2: RTRIM(col) on char cols, CASE WHEN col IN (DATE('0001-01-01'), DATE('9999-12-31')) THEN NULL ELSE col END on dates, "DOUBLE#QUOTES" for #-bearing identifiers. Watermarks are {name} placeholders substituted at run.

Group assignment + cron live in config/groups.json (add {"module","run_order"} to a group's members). Source connections in config/connections.json (S78030956 = DB2/AS400). pipekit apply hydrates pipekit.db from these files.

Merge semantics: staging recreated LIKE dest each run, then full = TRUNCATE + INSERT; incremental = DELETE by merge_key + INSERT; append = INSERT only.

Introspection method (how drafts are generated)

pipekit exposes introspection two ways:

• HTTP API GET /api/introspect/columns (:8200, Basic Auth).
• Driver layer pipekit.drivers.db2.DB2Driver.get_columns() — same code, in-process, needs only DB2PW from /etc/pipekit/secrets.env.

Drafts use the driver layer, reusing pipekit's own _TYPE_MAP, default_expression (RTRIM / date-sentinel), and quote_identifier (# handling) — so output matches pipekit style exactly, including real column/table descriptions from the DB2 catalog. Type mapping: CHAR/VARCHAR/GRAPHIC/CLOB → text, DECIMAL/NUMERIC(p,s) preserved, DATE → date, TIME → time, TIMESTAMP → text (pipekit convention), FLOAT → double precision, BIGINT/INTEGER/SMALLINT preserved.

Reproduce: drafts/bucket1/_generate.py (reads DB2 read-only, writes .json/.sql).

Bucket 1 — full truncate-reload (~41 tables) — DRAFTED

No watermark; legacy insert.sql was delete-all + reinsert. Simplest module: merge_strategy: "full", merge_key: null.

Status: drafted, not activated. Files in /opt/pipekit/drafts/bucket1/ (41 .json + 41 .sql). Nothing wired to a group; pipekit apply not run.

Tables (source library in parens where not LGDAT): adrs, cret, depts, fresre, resre, ftcstm, ftcstp, ftcstr, glie, icstm, icstp, icstr, iprca, iprcb, iprcc, iprcctn, iprccto, irea, macgrp, majg, methdm, methdo, methdr, mmgp, mmsl, opcode, plnt, punit, sach, sscc, stka, stkmm, stkmp, usrd, vend, color / colorb / colortier / iprcbhc ("CMS.CUSLG"), usrcust (LGPGM), ffpdglr1 (FANALDEV).

Activation (per table, when ready):

1. Review the drafted .json/.sql.
2. mv drafts/bucket1/<name>.{json,sql} config/modules/.
3. Add {"module":"<name>","run_order":N} to the right group in groups.json.
4. pipekit apply, then pipekit run <name> to smoke-test.
5. Repoint any rlarp.* view from lgdat.<name> to cms.<name>.

Post-load hooks needed (legacy refreshed a mat-view; add a hooks[] entry once the downstream view is repointed to cms.*): icstr → rlarp.icstx, ftcstr → rlarp.ftcstx, ffpdglr1 → rlarp.pdglr1, stkmp → CALL rlarp.itemm_ps_build().

Bucket 2 — watermarked incremental (6 tables) — TODO

Need merge_key + a watermark resolver query (SELECT MAX(...) on the dest, substituted into {wm} in the .sql). Templates: config/modules/ocrs.json, gtran.json.

Table	merge_key	watermark
qcrh	dcord#	order # high-water
qcri	ddord#	order #
qtnote	ggkey	key
methh	anpart+anplnt (EXISTS merge)	andate
iprcct	date	tadate
icstt	date	jhdate

Each has a *_full unbounded counterpart in /opt/sync that can be dropped.

Price-list change log (iprcct / iprcctn / iprccto) — DONE 2026-08-05. Append-only log, ~6.4–6.9M rows each, ~100 rows/day. All three: watermark MAX(<date>) - 7 on the dest (guarded <= current_date), source WHERE <date> BETWEEN '{wm}' AND DATE('9998-12-31'), incremental merge keyed on the date alone — the staged set is every row on/after the watermark, so delete-by-date replaces whole day partitions and re-running is a no-op. The 9998-12-31 upper bound matters: the source transform NULLs the 9999-12-31 sentinel, and a NULL merge key never matches the DELETE, so such a row would duplicate on every run. Grain is (plcd, part, unit, date, time) for the header and the same plus voll for the detail tables. All three reconcile exactly to source counts. Members of group 14 (Price Lists), scheduled 0 3 * * * — local time, after Sales Matrix at 02:20. The group is ~3.5 min, nearly all of it iprcc (908k rows, full reload); the three change-log modules are seconds each.

Quote family (qcrh / qcri / qtnote) — DONE 2026-08-05. The order-number watermark sketched in the table above is wrong and was replaced: quotes are edited long after creation (of 644 headers updated in a 30-day sample, 25 were below MAX(dcord#) - 1000), so an order-number window silently drops edits. QCRH shares OCRH's DC* layout, so all three now use the ocrh/ocri pattern — watermark MAX(dcudat) - 7 on cms.qcrh (guarded <= current_date) for all three modules, header WHERE unioning dcudat / dcodat / dccdat, details joined to a changed CTE over lgdat.qcrh. The union is load-bearing: 36,339 rows have dccdat after dcudat, 270 have dcodat bumped without dcudat. Because the detail watermarks resolve off the header table, run order within group 16 must stay header-first and the lookback must exceed the sync interval.

• qcri (492k rows, grain ddord#+dditm#+dddes#) keys on ddord#. Verified the changed-set catches line activity: all 5,807 lines created in a 30-day window (DDCTMS) belong to a header whose dates moved, 0 orphans. DDCTMS is a creation timestamp, not last-changed — unusable as a watermark.
• qtnote (253k rows) has no change signal of its own — no dates, no timestamp. GGKEY is 9-digit quote number (header note) or + 3-digit item (line note); all 50,876 note keys resolve to QCRH, only 6 also exist in OCRH, so it is purely quote notes. Keying on ggkey would leave a cleared note stranded in the dest forever, so a derived tail column cms.qtnote.ggord = SUBSTR(ggkey,1,9) was added (and backfilled) and is the merge key — delete now replaces all notes for a changed quote. Join is string-on-string via DIGITS(dcord#), deliberately avoiding a numeric CAST: one junk key (19 spaces + .) would raise SQL0420 on cast, and staying textual leaves it harmlessly excluded (it survives in the dest with ggord IS NULL, so dest count ties to source exactly at 252,721).

One-time rebaseline was required: a quote-level line-count diff found 273 quotes disagreeing with source (in both directions, back to 2023) plus 22 missing entirely — accumulated debt from the old order-number window. Rebaselined by temporarily pointing each resolver at DATE '1900-01-01' and running, then restoring; a wide date window provably covers all 82,224 headers, and delete-by-key avoids the ACCESS EXCLUSIVE lock a full TRUNCATE would take on tables the 15-min Quote Review group reads. All three now tie exactly to source. Group 16 (Quotes) is scheduled */15 * * * * (~20 s per pass) with explicit run_order 1/2/3 — members sort run_order, name, so header-first was previously only an alphabetical accident, and both detail modules read their changed-set from cms.qcrh.

Residual risk to watch on qtnote: it inherits change detection entirely from the header, so a note edited without the quote header's dates moving is invisible. Worth a periodic wide-window rebaseline (the same widen/run/restore steps) or a reconcile.py check.

icstt (Cost History, module 129) — reviewed 2026-08-05; KNOWN DRIFT ACCEPTED. 3.77M rows, watermark now MAX(jhdate) - 7 guarded <= current_date (the guard matters — source holds 2 future-dated rows out to 2031). Scheduled 30 3 * * * local as the sole member of group 13; before this it ran only when clicked.

Two defects found, one fixed, one deliberately left:

1. Fixed: the resolver had no lookback, so anything back-dated below the high-water was invisible forever.
2. Accepted: the source is purged at row level, and a date-keyed incremental structurally cannot see it — delete-by-jhdate only touches dates in the staged set, and old dates never enter it, so every upstream deletion strands a dest row permanently. Currently +2,241 rows across 149 dates from 2010 onward (0.06% of the table). Verified by key-level diff on one date: 20 dest-only keys, 0 source-only, i.e. dest ⊇ source. This grows over time and silently inflates historical cost aggregates. Only a wide-window pass removes them.

Also 5 source rows dated 1953–1966 (1 row each, data-entry junk) have never synced and are intentionally not synced — a 7-day lookback starts ~2026-07-29 and never reaches them.

Consequence for monitoring: a bare reconcile.py icstt will always report DIVERGED. Use the floor filter so the comparison is apples-to-apples — dates then tie exactly (5,156 = 5,156) and the only mismatch is the purge overage:

PIPEKIT_SECRETS=/etc/pipekit/secrets.env .venv/bin/python reconcile.py icstt \
  --super-quick --source-from "(SELECT * FROM LGDAT.ICSTT WHERE JHDATE >= DATE('2000-01-01')) t"

To rebaseline when there's a ~45 min window (est. from a 417k-row run at 5m06s): point watermark 32 at SELECT DATE '2000-01-01' (that floor stages all 5,156 real dates and clears the orphans while still skipping the 1953–66 junk), pipekit run icstt, restore the MAX(jhdate) - 7 resolver, then re-run the check above — it should read IN SYNC. Delete-by-date, so no TRUNCATE and no ACCESS EXCLUSIVE lock.

Purchase order family (poh 138 / poi 139 / poh_deleted 144) — DONE 2026-08-18. Converted from full (187k rows / 30 s and 529k / 75 s) to incremental: 1,201 rows / 5.5 s and 3,660 rows / 10 s. reconcile.py --quick reads IN SYNC on all 115 / 120 metrics, and a repeat run is a no-op.

• POH carries its own change stamp in its "future" fields — KAFUT12 = Date Created, KAFUT20 = Date Updated, both CHAR(10) ISO YYYY-MM-DD (so watermark compares are lexical and blanks sort below any floor). KAFUT20 is populated on 113k of 187k rows — the gap is history that predates the field, irrelevant to a forward incremental. Watermark poh_wm / poi_wm = MAX(kafut20)::date - 7 off cms.poh, capped at CURRENT_DATE.
• Predicate: KAFUT20 >= wm OR KAFUT12 >= wm OR KACRCM = '1'. KACRCM (1-Current / 2-Complete) leads most of POH's logicals, so the open hot set — only 1,035 of 187k headers, 2,374 of 529k lines — is a keyed read.
• POI has no change stamp at all (every KBFUT* is blank or a business attribute), so it follows the header on a PO-grain join, plus 4 open lines that hang off completed headers. merge_key is kbpo#, so the pull must stay PO-grain — staging only the changed lines would drop the rest of the PO.
• CMS logs this family too, and unlike OCRIT the header log is usable: POHL (4.6M rows, back to 2015) has logical POHLX1 keyed on DQ0TMSP alone; POIL has only POILX1 keyed DQ1PO#, DQ1TMSP. Both were measured as redundant for change detection: over 7 days POHL and the KAFUT20/12 window agree exactly (365 POs each, 0 either direction), and POIL reports 0 POs POHL does not — a line edit always stamps the header. So no log join is in the predicate.
• DQ0ACTN = '3' is the delete action (verified on PO 618973: last event is an action 3, row gone from POH; 1/2 and 4/5 are before/after pairs). This is the one thing the date fields structurally cannot see — an incremental merge only deletes keys present in staging, so a purged PO would sit in the dest forever, the same shape as the accepted icstt drift. Rate is ~88 POs/yr (142,009 all-time since 2015, 140,924 currently absent from POH).
• poh_deleted (144) closes it: DQ0ACTN = '3' AND DQ0TMSP >= '{wm}' AND NOT EXISTS (… LGDAT.POH …), incremental on dq0po#, watermark MAX(SUBSTRING(dq0tmsp,1,10))::date - 7 off its own dest table. The NOT EXISTS is what makes it exact instead of a guess about action ordering — a re-used PO number is back in POH and drops out of the list on its own. Three dest hooks, order matters: (1) DELETE FROM cms.poh_deleted WHERE dq0po# IN (SELECT kapo# FROM cms.poh) clears the tombstone of any PO that is live again before (2) deletes from cms.poh and (3) from cms.poi — without hook 1, hook 2 could remove a row poh had just refreshed. First run backfilled 140,924 rows in 13 s; steady state is a keyed range read of a handful. Hooks need an explicit connection_id or the run fails with hook connection None not found.
• Line-level deletes need no POIL module. Over 60 days, 66 POs had line deletes: 36 still have a header and all 36 are already caught by poi's predicate (the PO-grain delete+reinsert removes the vanished line), and the other 30 are whole-PO purges handled by poh_deleted keyed on the PO number.
• All three are in no group — manual runs for now. Whenever they are scheduled, order must be poh → poi → poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's hook 1 reads cms.poh expecting it to be fresh.
• Note WHERE TRUE cannot be used in these extracts — DB2 for i has no boolean literal and raises SQL0199.

Bucket 3 — special cases — decide, don't port as-is

• Reverse pushes PG→DB2 (osmfs, reprice, reprice_test): go Postgres→AS400, opposite of pipekit's source→PG model. Leave in /opt/sync or redesign separately.
• Stored-proc modules (sb_ud_r2, sb_gj_r1, osm_sync): CALL procs, not table extracts. Convert as hooks or leave out.
• Derived RLARP reads (family, ffterr, qrh, ctqpor): read derived AS400 rlarp.* tables, not raw CMS masters. Confirm the upstream proc still runs first.

Sequencing recommendation

Prioritize by what the marts consume: the cost/pricing cluster (icst*, iprc*, ftcst*, punit) and vendor/plant/method dims are highest value. Explicitly defer or retire the reverse pushes and derived reads rather than porting them.