Groups allow multiple modules to be run sequentially in a defined order.
Adds full CRUD (repo, engine orchestrator, web routes, templates) for grp,
group_member, and group_run tables that were previously schema-only. Module
index now shows group membership badges per module. Wizard default dest name
now sanitizes source column names with spaces or special characters to valid
identifiers rather than failing at CREATE TABLE time.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Module detail and index Run/Dry run buttons no longer redirect to the
run page. The status cell (index) and recent runs panel (detail) poll
every 3s while running and stop automatically when idle. force_poll
ensures polling starts immediately after clicking Run despite the race
between the HTTP response and the background task setting running=1.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Supports iSeries schema names that contain dots (e.g. CMS.CUSLG).
Strips surrounding double quotes on input so users don't need to worry
about quoting — the driver's quote_identifier handles that when building SQL.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
create_run now sets started_at on INSERT. list_runs computes duration_s
via julianday arithmetic. Both the module detail and runs page show
duration formatted as Xs or Xm Ys. A Jinja2 filter handles formatting.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Watermarks, merge strategy, merge key, and source query are now edited
together in one form on both the module edit page and wizard step 3.
A client-side placeholder warning fires when {name} tokens in the query
don't match the watermark rows on the page. The wizard now shows an
editable source query textarea pre-populated from column picks so WHERE
clauses can be added before module creation. Watermarks submitted via
wm_* arrays are processed by _save_inline_watermarks() in both
module_update and wizard_create.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Web POST /modules/{id}/run now returns immediately (BackgroundTasks)
instead of blocking until the run completes. jrunner.migrate() switches
from subprocess.run to Popen so stdout lines are read as they arrive and
appended to run_log.live_log via repo.append_run_live_log(). The run
detail page embeds an HTMX fragment that polls /runs/{id}/live every 2s
while status=running, showing current status, row count, and live output;
polling stops automatically once the run finishes.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Wire watermark WHERE clause into GL20000 source query ({dex_row_id} placeholder was present but query had no WHERE clause)
- Fix watermark resolver connection for GL20000 (was pointing at AS400, should be postgres dest)
- Resolve watermarks live on dry runs and module detail page load instead of using defaults
- Use status='dry_run' (not 'success') for dry runs so they can be filtered from recent runs UI
- Add exclude_status param to repo.list_runs; module detail excludes dry_run rows
- Expand run_log CHECK constraint to include 'dry_run'; backfill 16 historical records
- Delete SPEC_v1_archive.md (obsolete v1 design doc)
- Update SPEC.md and CLAUDE.md to reflect current engine flow and status values
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds a /wizard/schemas JSON endpoint and a live-filtered schema picker
panel on step 2. Clicking a row fills the schema input; the datalist
also powers browser autocomplete. MSSQL refetches when database or
linked_server qualifiers change. CSS fixes prevent picker tables and
two-col grid items from overflowing their containers.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Base provides a no-op default; drivers opt in by overriding. MSSQL
scopes the lookup to a linked server / database when those qualifiers
are supplied.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Previously the existing-dest check fired on submit and surfaced as a raw
JSON 400. Now step 3 introspects the default dest up front and renders a
yellow banner listing existing columns; submit-time mismatches render
wizard_error.html (409) with missing vs. existing side-by-side and a back
link that re-plumbs the form qvals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New text input + "jump to columns" button skip the full table listing
when you already know what you want. Typing "schema.table" and tabbing
out auto-splits into the schema qualifier + table name. Jump button
stays disabled until the table field has a value.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
If the dest table already exists, introspect its columns and verify the
wizard's picks line up. Missing columns surface a specific error message
naming what's missing instead of the opaque "column X does not exist"
from a failed COMMENT. On match, skip CREATE + COMMENT so existing
schema and comments aren't touched; staging still gets provisioned.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The tracked launcher now checks for .venv/bin/python3 under the repo and
uses it if present, else falls back to system python3. Works pre-deploy
(no venv) and post-deploy (venv exists) without being modified. Deploy
no longer regenerates the file, so `git pull` on a deployed box won't
conflict with the launcher.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cmd_serve now reads api_host from Config with a 127.0.0.1 safe default,
matching the existing api_port pattern. --host/--port CLI flags still
override. Local config is bumped to bind 0.0.0.0:8200.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Without -H, sudo keeps HOME pointed at the invoking user, so pip running
as root tries to write to /home/<user>/.cache/pip and disables caching
with a warning. -H resets HOME to /root while -E preserves the rest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
deploy.sh is the idempotent rollout path: venv + deps, launcher,
/etc/pipekit/secrets.env skeleton (mode 0600), schema init, and
auto-register of every JDBC driver shipped with jrunner. systemd
unit is a template, not auto-installed — user copies it when ready
to cut over.
`pipekit secrets {list,set,unset}` manages /etc/pipekit/secrets.env
with atomic 0600 writes so passwords don't need sudoedit. Prompted
input by default; positional value allowed for scripting.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Registers a driver-table row from the CLI. Kind is validated against
the code-level driver registry; JDBC class names default from a
built-in table (db2, pg, mssql). Refuses to double-register a kind
unless --force is passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging table drift caused silent data loss when dest grew columns but
staging kept the old shape. Fix on three fronts:
- Runner now DROP+CREATEs staging each run instead of CREATE IF NOT
EXISTS, so any drift self-heals.
- Wizard create drop+creates staging right after dest is provisioned,
surfacing DDL errors at create time.
- Module edit drops the (old-name) staging table and re-applies
COMMENT ON TABLE when dest_description changed.
jrunner's query mode uses executeQuery() which raises
"No results were returned by the query" after DDL/DML succeeds; the
stack-trace detector now allowlists that exception so normal
CREATE/TRUNCATE/INSERT runs aren't flagged as failures.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Modules get a full edit form (name, connections, tables, source query,
merge config, description, enabled); reachable via Edit button on the
detail page and the source-query panel.
jrunner catches SQLException and calls System.exit(0) at every failure
site, so pipekit was marking runs success when the migrate phase had
actually errored. query() and migrate() now scan stdout+stderr for a
Java stack-trace signature and raise JrunnerError. runner.py also
captures the failed jrunner output onto run_log so the stack trace is
visible on the run detail page.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Delete button lives in module-detail header, refuses to delete a
running module, and clears run_log history first since it doesn't
cascade from module. Wizard now returns 409 on duplicate name before
touching the destination, so a failed resubmit doesn't redundantly
rerun CREATE TABLE / COMMENT ON on the dest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Orchestration layer around the jrunner Java JDBC CLI, replacing the
previous shell-based sync system in .archive/pre-rewrite. Includes
the FastAPI + Jinja web frontend, per-driver adapters (DB2, MSSQL,
PG), wizard-driven module creation with editable dest types and
source-sourced table/column descriptions, watermark/hook CRUD,
and the engine that runs modules end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
