Paul Trowbridge
d5618e9514
poh and poi were full truncate-reloads: 187k rows / 30 s and 529k / 75 s. Both now pull only changed POs -- 1,201 rows / 5.5 s and 3,660 / 10 s -- and reconcile.py --quick reads IN SYNC on all 115 / 120 metrics. POH turns out to carry its own change stamp in its "future" fields: KAFUT12 is Date Created and KAFUT20 is Date Updated, both CHAR(10) ISO text. Predicate is those two against the watermark, plus the open hot set KACRCM = '1' -- 1,035 of 187k headers, and KACRCM leads most of POH's logicals, so it is a keyed read. POI has no change stamp of its own, so it joins the header's changed set at PO grain; its merge key is kbpo#, so staging only the changed lines would drop the rest of the PO. CMS logs this family (POHL, POIL), but both are redundant for change detection: over 7 days POHL and the KAFUT20/12 window agree exactly, and POIL reports no PO that POHL does not, so a line edit always stamps the header. The logs' one unique contribution is deletes, which an incremental merge structurally cannot see -- delete-by-key only touches staged keys, so a purged PO would sit in the dest forever (~88 POs/yr). poh_deleted closes that: DQ0ACTN = '3' is the delete action, and the NOT EXISTS against live POH makes the list exact rather than a guess about action ordering, since a re-used PO number is back in POH and drops out on its own. Three dest hooks, in order -- clearing the tombstone of any PO that is live again must happen before the deletes, or the second hook could remove a row poh had just refreshed. Whenever these are scheduled, run order must be poh -> poi -> poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's first hook reads cms.poh expecting it to be fresh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
21 lines
999 B
SQL
21 lines
999 B
SQL
-- POs purged from LGDAT.POH. An incremental merge only deletes keys that are
|
|
-- present in staging, so a purged PO would sit in cms.poh / cms.poi forever;
|
|
-- this list drives the hooks below that remove them.
|
|
--
|
|
-- DQ0ACTN = '3' is the delete action (verified 2026-08-18: PO 618973's last log
|
|
-- event was an action 3 and the row is gone from POH). The NOT EXISTS is what
|
|
-- makes this exact rather than a guess about action ordering -- a PO number that
|
|
-- was deleted and later re-used is back in POH, so it drops out of the list on
|
|
-- its own, and hook 1 clears any tombstone left from an earlier run.
|
|
--
|
|
-- DQ0TMSP is CHAR(26) 'YYYY-MM-DD-HH.MM.SS.ffffff', so the watermark compare is
|
|
-- lexical, and POHL has a timestamp-only logical (POHLX1) to range-scan it.
|
|
SELECT
|
|
l."DQ0PO#" AS "dq0po#",
|
|
MAX(l.DQ0TMSP) AS dq0tmsp
|
|
FROM LGDAT.POHL l
|
|
WHERE l.DQ0ACTN = '3'
|
|
AND l.DQ0TMSP >= '{poh_deleted_wm}'
|
|
AND NOT EXISTS (SELECT 1 FROM LGDAT.POH p WHERE p."KAPO#" = l."DQ0PO#")
|
|
GROUP BY l."DQ0PO#"
|