reconcile.py was written against the DB2 LGDAT modules and broke four
different ways on the first GP module (sop30200):
- detect_source_from's regex handled at most a two-part name, so
CHG.dbo.SOP30200 truncated to CHG.dbo -> "Invalid object name".
Now matches 1-4 parts, including [bracketed] and "quoted" forms.
- Even fully qualified, that table only exists behind the GPSERVER
linked server. Detect the module's OPENQUERY wrapper and push the
aggregate through it, so the scan runs remotely and one row comes
back instead of 1.6M rows crossing the link.
- OPENQUERY caps its passthrough string at 8000 chars and 291 metrics
overran it. Split the metric list into chunks that fit and CROSS JOIN
them back into one row; alias columns c0..cN since OPENQUERY rejects
unnamed result columns (msg 8155). EXEC(@sql) AT has no such cap but
needs RPC Out, which GPSERVER has disabled.
- Two dialect bugs that would hit any SQL Server source: T-SQL SUM(int)
stays int and overflowed at 2^31 where Postgres promotes to bigint,
and T-SQL has no LENGTH. Added Driver.sum_expression /
length_expression with mssql overrides -- integer types cast to
BIGINT (decimals left alone so scale isn't lost), LENGTH -> LEN.
--super-quick and --quick now run clean end to end. Full depth is
syntax-checked only; it wasn't run against the server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--super-quick compares only COUNT(*) and COUNT(DISTINCT merge_key) —
seconds on multi-million-row tables, enough to catch missing/duplicated
rows but blind to changed values.
Also fix derived merge keys: columns_json source_name can hold a SQL
expression (e.g. SUBSTR(GGKEY,1,9)), which must be emitted verbatim
rather than quoted as an identifier (SQL0206) or re-transformed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reconcile.py compares a module's live source table against its synced dest
using column-wise aggregates (COUNT/SUM/MIN/MAX/SUM(LENGTH)) — arithmetic and
ordering that DB2 for i and Postgres compute identically, so no shared hash or
byte-identical serialization is needed. Re-applies the module's per-column
source transform (default_expression) so aggregates line up when row sets
agree; exits non-zero on any divergence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
