config/ had drifted from pipekit.db: group memberships and three
schedules existed only in the DB, cjobdr and cjobh had no sidecars at
all, and ab.{json,sql} was left over from a module no longer in the DB
(an apply would have recreated it). pipekit apply --dry-run now reports
that the database matches config.
Two real inconsistencies fixed in ffpdglr1.json: it was missing the
ffpdglr1_hot watermark added with the rollover hot set, so an apply from
files would have deleted the watermark while the committed SQL still
referenced {ffpdglr1_hot}; and its period resolver had been edited in the
UI without an export, so the file carried a stale format('%L', ...)
variant.
Purchasing members were all run_order 0, which happened to sort
poh, poh_deleted, poi alphabetically and satisfy the real constraints by
accident. Now explicit: poh 1, poi 2, poh_deleted 3 -- poi resolves its
watermark off cms.poh, and poh_deleted's first hook reads cms.poh.
Also describes eight GP modules that had no description (dest_description
is the half the sidecar format carries; the module-level description
lives only in the DB).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rm20201 (Applied Open) is 985 rows and stays a full reload. rm30201
(Apply History) was a full pull of 1,149,405 rows taking 19 minutes; it
now moves 661 rows in 6 s and reconcile --quick ties on all 108 metrics.
Neither apply table has DEX_ROW_TS, so rm30101's pattern does not port.
DEX_ROW_ID catches inserts but never moves on an update, so a 7-day
GLPOSTDT/APFRDCDT window carries restated and back-posted applies.
APTODCDT belongs to the apply-to document and lags by weeks; DATE1 has
163 rows dated to year 6201.
The pull is document-grain, not row-grain. merge_key is (aptodcty,
aptodcnm), so the DELETE clears every apply row for a document, and
staging only the row that matched would drop its siblings -- ~16k of
1.13M docs carry more than one. AK1/AK5 lead on those two columns, so
the EXISTS is an index seek.
Uses OPENQUERY pushdown rather than the four-part linked-server name: an
aggregate through the four-part name ran past 2 minutes against 1.3 s
pushed down. Note no apostrophes may appear inside the OPENQUERY string,
comments included.
Unapplies remain uncovered -- they delete the RM30201 row and write one
back to RM20201, and delete-by-key cannot see a row that is gone. The
985-row rm20201 reload is the cross-check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fiscal-period floor advances to the new period as soon as one batch
posts into it, so a late posting into the prior period stopped being
refreshed from that moment. FFPDGLR1 itself has no dates -- only
BTID/ENT#/SEQ# -- so RPRH is the only change signal available: the new
branches follow its NWPDAT/NWUDAT/NWCDAT stamps against a 14-day floor,
and carry the unposted batches too.
Strictly additive. Measured mid-period the hot set is a subset of the
period branch, adding 0 rows -- the run stays at 132,325 rows. It earns
its keep at a period boundary, the only time the two floors disagree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NWPDAT stays 0001-01-01 until a batch posts, so the watermark branch
alone can never see an in-flight batch -- it only appears once posting
stamps a real date. Only 4 rows are unposted, so the whole set now comes
over every run.
Found by reconcile: dest and source tied on count but five numeric sums
differed, which resolved to a single row pair -- an unposted batch
missing from the dest, and one stale row for a batch purged upstream
(deleted separately, since delete-by-key cannot remove it).
The resolver still has no lookback, so a back-dated post below the
high-water stays invisible; posting stamps the current date, so this only
matters for a genuinely back-dated one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
poh and poi were full truncate-reloads: 187k rows / 30 s and 529k / 75 s.
Both now pull only changed POs -- 1,201 rows / 5.5 s and 3,660 / 10 s --
and reconcile.py --quick reads IN SYNC on all 115 / 120 metrics.
POH turns out to carry its own change stamp in its "future" fields:
KAFUT12 is Date Created and KAFUT20 is Date Updated, both CHAR(10) ISO
text. Predicate is those two against the watermark, plus the open hot set
KACRCM = '1' -- 1,035 of 187k headers, and KACRCM leads most of POH's
logicals, so it is a keyed read. POI has no change stamp of its own, so
it joins the header's changed set at PO grain; its merge key is kbpo#, so
staging only the changed lines would drop the rest of the PO.
CMS logs this family (POHL, POIL), but both are redundant for change
detection: over 7 days POHL and the KAFUT20/12 window agree exactly, and
POIL reports no PO that POHL does not, so a line edit always stamps the
header. The logs' one unique contribution is deletes, which an
incremental merge structurally cannot see -- delete-by-key only touches
staged keys, so a purged PO would sit in the dest forever (~88 POs/yr).
poh_deleted closes that: DQ0ACTN = '3' is the delete action, and the
NOT EXISTS against live POH makes the list exact rather than a guess
about action ordering, since a re-used PO number is back in POH and drops
out on its own. Three dest hooks, in order -- clearing the tombstone of
any PO that is live again must happen before the deletes, or the second
hook could remove a row poh had just refreshed.
Whenever these are scheduled, run order must be poh -> poi ->
poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's
first hook reads cms.poh expecting it to be fresh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hold/release rewrites OCRH.DCSTAT and DCHREA without touching any date
column, so no watermark could see it. A full-table row diff of LGDAT.OCRH
vs cms.ocrh (468,823 rows both sides) found 168 drifted rows, 0 missing,
0 extra -- and 159 of them changed with no date moving at all (checked
dcudat/dcodat/dccdat/dcmdat/dchdat/dcxdat/dcsdat/dcmxdt). 155 of the
drifts were DCSTAT, 4 DCHREA.
Every drifted row was still open (DCSTAT in A/N/H/B); none had reached
'C'. Only ~2,044 of 468,823 orders are open, so re-pulling the whole open
set each run is cheap and catches 100% of the observed drift. Applied to
ocrh and to the two modules that read its changed-set, ocri and ocrs.
Result: drift 168 -> 11 rows, DCSTAT 155 -> 3.
No invoice-driven branch: tested against LGDAT.OCRHT (CMS's header change
log, keyed DC3TMSP-leading via logical OCRHTX1) over the live watermark
window -- 1,107 headers changed, 65 missed by the date branches, 0 missed
by dates+open, and a 5- or 30-day invoiced-orders branch added exactly 0.
Closing an order writes dcudat/dccdat, so the date window covers the flip.
Also evaluated LGDAT.OCRIT (the order-line change log) as a replacement
for the open-order branch and rejected it: at the live 8-day watermark it
reports 0 changed orders not already caught, at ~12x the predicate cost,
and its lower row count means it declines to pull 18k lines the open set
would -- safe only if CMS's logging is complete, which is unverified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Full-reload modules for the CMS/DB2 tables migrated off /opt/sync, landing
in cms.*. Columns introspected fresh from DB2 (QSYS2.SYSCOLUMNS). Three large
tables use incremental merge ported from the legacy sync instead of full
reload: iprcctn/iprccto (date watermark on t8date/t7date), ffpdglr1 (fiscal-
period watermark via rprh join, keyed on btid).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five dest_names in the column map were junk from a crude source-name
sanitizer (ext._price, margin__, _hd_grower, ext._cost,
ext._margin_dollar) and didn't match the physical gp table's clean names
(ext_price, margin_pct, hd_grower, ext_cost, ext_margin_dollar). The load
is positional so this was harmless for years — until the new dest-table
preflight's name check started (correctly) flagging the drift and failing
the daily run. Align the map to the table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `pipekit export` / `pipekit apply` (pipekit/config_io.py) to serialise the
DB's config — drivers, connections, modules (+ columns, watermarks, hooks),
groups, schedules — to a git-trackable text tree under config/, and rehydrate
it. SQLite stays runtime state; definitions become diffable/reviewable/
revertible.
- config only: run_log/group_run/settings and per-run state columns excluded
- name-keyed refs (portable across databases); source_query in .sql sidecars;
columns as real JSON arrays for line-by-line diffs
- newline-normalised so CRLF-vs-LF is never a spurious change
- apply is create/update by name; child collections fully synced; top-level
deletes gated behind --prune; --dry-run prints the plan
- round-trip is identity (export -> apply --dry-run == nothing to do)
Commits the current config/ as the first baseline, capturing the freshly
populated columns_json for rm00101/rm00301/iv00101. Documented in SPEC.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
