Paul Trowbridge
d5618e9514
poh and poi were full truncate-reloads: 187k rows / 30 s and 529k / 75 s. Both now pull only changed POs -- 1,201 rows / 5.5 s and 3,660 / 10 s -- and reconcile.py --quick reads IN SYNC on all 115 / 120 metrics. POH turns out to carry its own change stamp in its "future" fields: KAFUT12 is Date Created and KAFUT20 is Date Updated, both CHAR(10) ISO text. Predicate is those two against the watermark, plus the open hot set KACRCM = '1' -- 1,035 of 187k headers, and KACRCM leads most of POH's logicals, so it is a keyed read. POI has no change stamp of its own, so it joins the header's changed set at PO grain; its merge key is kbpo#, so staging only the changed lines would drop the rest of the PO. CMS logs this family (POHL, POIL), but both are redundant for change detection: over 7 days POHL and the KAFUT20/12 window agree exactly, and POIL reports no PO that POHL does not, so a line edit always stamps the header. The logs' one unique contribution is deletes, which an incremental merge structurally cannot see -- delete-by-key only touches staged keys, so a purged PO would sit in the dest forever (~88 POs/yr). poh_deleted closes that: DQ0ACTN = '3' is the delete action, and the NOT EXISTS against live POH makes the list exact rather than a guess about action ordering, since a re-used PO number is back in POH and drops out on its own. Three dest hooks, in order -- clearing the tombstone of any PO that is live again must happen before the deletes, or the second hook could remove a row poh had just refreshed. Whenever these are scheduled, run order must be poh -> poi -> poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's first hook reads cms.poh expecting it to be fresh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
58 lines
1.6 KiB
JSON
58 lines
1.6 KiB
JSON
{
|
|
"name": "poh_deleted",
|
|
"source_connection": "S78030956",
|
|
"dest_connection": "usmidsap02",
|
|
"dest_table": "cms.poh_deleted",
|
|
"staging_table": "pipekit_staging.poh_deleted",
|
|
"merge_strategy": "incremental",
|
|
"merge_key": "dq0po#",
|
|
"enabled": 1,
|
|
"dest_description": "POs purged from LGDAT.POH (POHL delete log) -- drives purge cleanup of cms.poh / cms.poi",
|
|
"columns": [
|
|
{
|
|
"id": "c1",
|
|
"source_name": "DQ0PO#",
|
|
"source_type": "NUMERIC(9,0)",
|
|
"dest_name": "dq0po#",
|
|
"dest_type": "numeric(9,0)",
|
|
"description": "Purchase Order Number"
|
|
},
|
|
{
|
|
"id": "c2",
|
|
"source_name": "DQ0TMSP",
|
|
"source_type": "CHAR(26)",
|
|
"dest_name": "dq0tmsp",
|
|
"dest_type": "text",
|
|
"description": "Change TimeStamp of the delete"
|
|
}
|
|
],
|
|
"watermarks": [
|
|
{
|
|
"name": "poh_deleted_wm",
|
|
"connection": "usmidsap02",
|
|
"resolver_sql": "SELECT TO_CHAR(COALESCE(MAX(SUBSTRING(dq0tmsp, 1, 10))::DATE, DATE '1900-01-01') - 7, 'YYYY-MM-DD') FROM cms.poh_deleted",
|
|
"default_value": null
|
|
}
|
|
],
|
|
"hooks": [
|
|
{
|
|
"run_order": 1,
|
|
"connection": "usmidsap02",
|
|
"sql": "DELETE FROM cms.poh_deleted WHERE \"dq0po#\" IN (SELECT \"kapo#\" FROM cms.poh)",
|
|
"run_on": "success"
|
|
},
|
|
{
|
|
"run_order": 2,
|
|
"connection": "usmidsap02",
|
|
"sql": "DELETE FROM cms.poh WHERE \"kapo#\" IN (SELECT \"dq0po#\" FROM cms.poh_deleted)",
|
|
"run_on": "success"
|
|
},
|
|
{
|
|
"run_order": 3,
|
|
"connection": "usmidsap02",
|
|
"sql": "DELETE FROM cms.poi WHERE \"kbpo#\" IN (SELECT \"dq0po#\" FROM cms.poh_deleted)",
|
|
"run_on": "success"
|
|
}
|
|
]
|
|
}
|