The wizard previously required picking a single source table; modules whose
entry point is arbitrary SQL (CTEs, joins, computed columns) didn't fit. Add a
"write SQL" path alongside "browse a table":
- Driver.introspect_query_columns + _zero_row_wrap discover a query's result
columns by running it with ~no rows. Generic wrap is a derived table with
WHERE 1=0; DB2 appends FETCH FIRST 1 ROW ONLY (DB2 for i forbids WITH inside
a nested table expression).
- /wizard/sql + POST /wizard/sql/columns seed the column-mapping grid; dest
types default to text (no result-set type metadata over jrunner CSV).
- wizard_step3.html grows a sql_mode branch (array-named inputs, query shown
verbatim, no column unchecking); wizard_create branches on entry_mode.
Verified end-to-end against a live DB2 for i connection, including a top-level
CTE query.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Scheduler now evaluates cron expressions against local time instead of
UTC, so schedules fire at the user's local clock time. All timestamp
displays in templates use a new `localtime` Jinja filter that converts
UTC strings from SQLite to the server's local timezone. Updated CLAUDE.md
to reflect the systemd service setup.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Scheduling: cron-based group runs via a daemon thread (scheduler.py)
started at API startup. Schedules managed inline on the group edit form.
last_fired_at persisted before run to prevent double-fire on restart.
Requires croniter (added to requirements.txt); DB migration adds
last_fired_at column to schedule table.
Deploy: deploy.sh now creates the pipekit system user, chowns the repo,
builds the venv as pipekit, and installs/enables the systemd unit.
systemd/pipekit.service is now a production-ready unit (User=pipekit
uncommented). pipekit secrets set preserves existing file permissions
instead of resetting to 0600. Driver registration is now idempotent
(upsert via get_driver_by_name + update_driver).
Docs: CLAUDE.md and SPEC.md updated to reflect groups, scheduling,
scheduler-in-API-process architecture, TUI deferred (not dropped),
stop-on-failure tradeoff, jrunner as prerequisite, and deploy flow.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Watermarks, merge strategy, merge key, and source query are now edited
together in one form on both the module edit page and wizard step 3.
A client-side placeholder warning fires when {name} tokens in the query
don't match the watermark rows on the page. The wizard now shows an
editable source query textarea pre-populated from column picks so WHERE
clauses can be added before module creation. Watermarks submitted via
wm_* arrays are processed by _save_inline_watermarks() in both
module_update and wizard_create.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Web POST /modules/{id}/run now returns immediately (BackgroundTasks)
instead of blocking until the run completes. jrunner.migrate() switches
from subprocess.run to Popen so stdout lines are read as they arrive and
appended to run_log.live_log via repo.append_run_live_log(). The run
detail page embeds an HTMX fragment that polls /runs/{id}/live every 2s
while status=running, showing current status, row count, and live output;
polling stops automatically once the run finishes.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Wire watermark WHERE clause into GL20000 source query ({dex_row_id} placeholder was present but query had no WHERE clause)
- Fix watermark resolver connection for GL20000 (was pointing at AS400, should be postgres dest)
- Resolve watermarks live on dry runs and module detail page load instead of using defaults
- Use status='dry_run' (not 'success') for dry runs so they can be filtered from recent runs UI
- Add exclude_status param to repo.list_runs; module detail excludes dry_run rows
- Expand run_log CHECK constraint to include 'dry_run'; backfill 16 historical records
- Delete SPEC_v1_archive.md (obsolete v1 design doc)
- Update SPEC.md and CLAUDE.md to reflect current engine flow and status values
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
