config_io handled dest_description but never the module's own
description field: it was absent from export_config, from both
repo.create_module and repo.update_module calls in apply_config, and
from the change-detection watch tuple. Descriptions set via the web
form were therefore invisible to the config tree — `pipekit apply`
silently ignored a description in a module's JSON, and `pipekit export`
would have dropped all 127 existing ones on a round-trip.
Adding "description" to the watch tuple means a module file without the
key now reads as a change to NULL, so backfill the key into all 131
existing module files from the DB. Without this, the next apply would
have wiped every description it was meant to preserve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
config/ had drifted from pipekit.db: group memberships and three
schedules existed only in the DB, cjobdr and cjobh had no sidecars at
all, and ab.{json,sql} was left over from a module no longer in the DB
(an apply would have recreated it). pipekit apply --dry-run now reports
that the database matches config.
Two real inconsistencies fixed in ffpdglr1.json: it was missing the
ffpdglr1_hot watermark added with the rollover hot set, so an apply from
files would have deleted the watermark while the committed SQL still
referenced {ffpdglr1_hot}; and its period resolver had been edited in the
UI without an export, so the file carried a stale format('%L', ...)
variant.
Purchasing members were all run_order 0, which happened to sort
poh, poh_deleted, poi alphabetically and satisfy the real constraints by
accident. Now explicit: poh 1, poi 2, poh_deleted 3 -- poi resolves its
watermark off cms.poh, and poh_deleted's first hook reads cms.poh.
Also describes eight GP modules that had no description (dest_description
is the half the sidecar format carries; the module-level description
lives only in the DB).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rm20201 (Applied Open) is 985 rows and stays a full reload. rm30201
(Apply History) was a full pull of 1,149,405 rows taking 19 minutes; it
now moves 661 rows in 6 s and reconcile --quick ties on all 108 metrics.
Neither apply table has DEX_ROW_TS, so rm30101's pattern does not port.
DEX_ROW_ID catches inserts but never moves on an update, so a 7-day
GLPOSTDT/APFRDCDT window carries restated and back-posted applies.
APTODCDT belongs to the apply-to document and lags by weeks; DATE1 has
163 rows dated to year 6201.
The pull is document-grain, not row-grain. merge_key is (aptodcty,
aptodcnm), so the DELETE clears every apply row for a document, and
staging only the row that matched would drop its siblings -- ~16k of
1.13M docs carry more than one. AK1/AK5 lead on those two columns, so
the EXISTS is an index seek.
Uses OPENQUERY pushdown rather than the four-part linked-server name: an
aggregate through the four-part name ran past 2 minutes against 1.3 s
pushed down. Note no apostrophes may appear inside the OPENQUERY string,
comments included.
Unapplies remain uncovered -- they delete the RM30201 row and write one
back to RM20201, and delete-by-key cannot see a row that is gone. The
985-row rm20201 reload is the cross-check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fiscal-period floor advances to the new period as soon as one batch
posts into it, so a late posting into the prior period stopped being
refreshed from that moment. FFPDGLR1 itself has no dates -- only
BTID/ENT#/SEQ# -- so RPRH is the only change signal available: the new
branches follow its NWPDAT/NWUDAT/NWCDAT stamps against a 14-day floor,
and carry the unposted batches too.
Strictly additive. Measured mid-period the hot set is a subset of the
period branch, adding 0 rows -- the run stays at 132,325 rows. It earns
its keep at a period boundary, the only time the two floors disagree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NWPDAT stays 0001-01-01 until a batch posts, so the watermark branch
alone can never see an in-flight batch -- it only appears once posting
stamps a real date. Only 4 rows are unposted, so the whole set now comes
over every run.
Found by reconcile: dest and source tied on count but five numeric sums
differed, which resolved to a single row pair -- an unposted batch
missing from the dest, and one stale row for a batch purged upstream
(deleted separately, since delete-by-key cannot remove it).
The resolver still has no lookback, so a back-dated post below the
high-water stays invisible; posting stamps the current date, so this only
matters for a genuinely back-dated one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
poh and poi were full truncate-reloads: 187k rows / 30 s and 529k / 75 s.
Both now pull only changed POs -- 1,201 rows / 5.5 s and 3,660 / 10 s --
and reconcile.py --quick reads IN SYNC on all 115 / 120 metrics.
POH turns out to carry its own change stamp in its "future" fields:
KAFUT12 is Date Created and KAFUT20 is Date Updated, both CHAR(10) ISO
text. Predicate is those two against the watermark, plus the open hot set
KACRCM = '1' -- 1,035 of 187k headers, and KACRCM leads most of POH's
logicals, so it is a keyed read. POI has no change stamp of its own, so
it joins the header's changed set at PO grain; its merge key is kbpo#, so
staging only the changed lines would drop the rest of the PO.
CMS logs this family (POHL, POIL), but both are redundant for change
detection: over 7 days POHL and the KAFUT20/12 window agree exactly, and
POIL reports no PO that POHL does not, so a line edit always stamps the
header. The logs' one unique contribution is deletes, which an
incremental merge structurally cannot see -- delete-by-key only touches
staged keys, so a purged PO would sit in the dest forever (~88 POs/yr).
poh_deleted closes that: DQ0ACTN = '3' is the delete action, and the
NOT EXISTS against live POH makes the list exact rather than a guess
about action ordering, since a re-used PO number is back in POH and drops
out on its own. Three dest hooks, in order -- clearing the tombstone of
any PO that is live again must happen before the deletes, or the second
hook could remove a row poh had just refreshed.
Whenever these are scheduled, run order must be poh -> poi ->
poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's
first hook reads cms.poh expecting it to be fresh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hold/release rewrites OCRH.DCSTAT and DCHREA without touching any date
column, so no watermark could see it. A full-table row diff of LGDAT.OCRH
vs cms.ocrh (468,823 rows both sides) found 168 drifted rows, 0 missing,
0 extra -- and 159 of them changed with no date moving at all (checked
dcudat/dcodat/dccdat/dcmdat/dchdat/dcxdat/dcsdat/dcmxdt). 155 of the
drifts were DCSTAT, 4 DCHREA.
Every drifted row was still open (DCSTAT in A/N/H/B); none had reached
'C'. Only ~2,044 of 468,823 orders are open, so re-pulling the whole open
set each run is cheap and catches 100% of the observed drift. Applied to
ocrh and to the two modules that read its changed-set, ocri and ocrs.
Result: drift 168 -> 11 rows, DCSTAT 155 -> 3.
No invoice-driven branch: tested against LGDAT.OCRHT (CMS's header change
log, keyed DC3TMSP-leading via logical OCRHTX1) over the live watermark
window -- 1,107 headers changed, 65 missed by the date branches, 0 missed
by dates+open, and a 5- or 30-day invoiced-orders branch added exactly 0.
Closing an order writes dcudat/dccdat, so the date window covers the flip.
Also evaluated LGDAT.OCRIT (the order-line change log) as a replacement
for the open-order branch and rejected it: at the live 8-day watermark it
reports 0 changed orders not already caught, at ~12x the predicate cost,
and its lower row count means it declines to pull 18k lines the open set
would -- safe only if CMS's logging is complete, which is unverified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Full-reload modules for the CMS/DB2 tables migrated off /opt/sync, landing
in cms.*. Columns introspected fresh from DB2 (QSYS2.SYSCOLUMNS). Three large
tables use incremental merge ported from the legacy sync instead of full
reload: iprcctn/iprccto (date watermark on t8date/t7date), ffpdglr1 (fiscal-
period watermark via rprh join, keyed on btid).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five dest_names in the column map were junk from a crude source-name
sanitizer (ext._price, margin__, _hd_grower, ext._cost,
ext._margin_dollar) and didn't match the physical gp table's clean names
(ext_price, margin_pct, hd_grower, ext_cost, ext_margin_dollar). The load
is positional so this was harmless for years — until the new dest-table
preflight's name check started (correctly) flagging the drift and failing
the daily run. Align the map to the table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `pipekit export` / `pipekit apply` (pipekit/config_io.py) to serialise the
DB's config — drivers, connections, modules (+ columns, watermarks, hooks),
groups, schedules — to a git-trackable text tree under config/, and rehydrate
it. SQLite stays runtime state; definitions become diffable/reviewable/
revertible.
- config only: run_log/group_run/settings and per-run state columns excluded
- name-keyed refs (portable across databases); source_query in .sql sidecars;
columns as real JSON arrays for line-by-line diffs
- newline-normalised so CRLF-vs-LF is never a spurious change
- apply is create/update by name; child collections fully synced; top-level
deletes gated behind --prune; --dry-run prints the plan
- round-trip is identity (export -> apply --dry-run == nothing to do)
Commits the current config/ as the first baseline, capturing the freshly
populated columns_json for rm00101/rm00301/iv00101. Documented in SPEC.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
