Go to file
Paul Trowbridge 002de48bba Make reconcile.py work against SQL Server / linked-server sources
reconcile.py was written against the DB2 LGDAT modules and broke four
different ways on the first GP module (sop30200):

- detect_source_from's regex handled at most a two-part name, so
  CHG.dbo.SOP30200 truncated to CHG.dbo -> "Invalid object name".
  Now matches 1-4 parts, including [bracketed] and "quoted" forms.

- Even fully qualified, that table only exists behind the GPSERVER
  linked server. Detect the module's OPENQUERY wrapper and push the
  aggregate through it, so the scan runs remotely and one row comes
  back instead of 1.6M rows crossing the link.

- OPENQUERY caps its passthrough string at 8000 chars and 291 metrics
  overran it. Split the metric list into chunks that fit and CROSS JOIN
  them back into one row; alias columns c0..cN since OPENQUERY rejects
  unnamed result columns (msg 8155). EXEC(@sql) AT has no such cap but
  needs RPC Out, which GPSERVER has disabled.

- Two dialect bugs that would hit any SQL Server source: T-SQL SUM(int)
  stays int and overflowed at 2^31 where Postgres promotes to bigint,
  and T-SQL has no LENGTH. Added Driver.sum_expression /
  length_expression with mssql overrides -- integer types cast to
  BIGINT (decimals left alone so scale isn't lost), LENGTH -> LEN.

--super-quick and --quick now run clean end to end. Full depth is
syntax-checked only; it wasn't run against the server.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 02:25:24 -04:00
.archive/pre-rewrite Initial commit: Pipekit rewrite. 2026-04-22 00:38:26 -04:00
bin bin/pipekit auto-detects venv; stop rewriting it in deploy.sh. 2026-04-23 00:34:32 -04:00
config updates 2026-08-06 15:39:22 -04:00
docs Add CMS→pipekit migration plan doc 2026-08-05 15:48:45 -04:00
pipekit Make reconcile.py work against SQL Server / linked-server sources 2026-08-07 02:25:24 -04:00
systemd Add scheduling, harden deploy, and update docs 2026-06-03 21:18:13 -04:00
.gitignore drivers: make staging DDL dialect-aware; stop tracking pipekit.db 2026-06-10 23:33:36 -04:00
CLAUDE.md jrunner: use bulk copy (-b) for Postgres dests too (COPY) 2026-06-18 23:14:15 -04:00
config.yaml web: add session-cookie login for web UI 2026-06-04 13:21:50 -04:00
deploy.sh deploy/cli: fix /etc/pipekit permissions so non-root group members can write secrets 2026-06-04 13:35:06 -04:00
reconcile.py Make reconcile.py work against SQL Server / linked-server sources 2026-08-07 02:25:24 -04:00
requirements.txt web: add session-cookie login for web UI 2026-06-04 13:21:50 -04:00
SPEC.md feat: version-control pipeline definitions via export/apply 2026-07-22 10:56:55 -04:00