The group pages were browser-only, so external schedulers had no way to
trigger a group or read its outcome. Adds /api/groups, /api/group-runs and
their run/list endpoints under Basic auth, following the same async contract
as POST /modules/{id}/run.
Cancellation needed more than an endpoint. run_group deliberately continues
past module failures, so killing the in-flight jrunner alone would just let
the loop march on to the next member. A group-level flag in engine/cancel.py
is now checked before each member, and engine.request_group_cancel sets that
flag *and* signals the running module -- it lives in runner.py rather than
cancel.py because finding the live run needs the DB, and cancel.py is
deliberately DB-free.
Group final status gains 'cancelled', ranked above 'error' since the error is
usually the terminated jrunner process. group_run.status already permitted the
value; no migration needed.
Also adds a "cancel group" button to the group run page (web) and .pill
styling for cancelled/dry_run, neither of which had a color rule.
Note: /runs/{id}/cancel keeps its existing meaning -- cancel one member and
let the group continue -- which is now a deliberate contrast to cancel group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
schema.sql has carried 'dry_run' in the group_run.status CHECK for a while,
but CREATE TABLE IF NOT EXISTS never re-applies a constraint to a database
that already has the table. Any DB created before that value was added kept
the old four-value CHECK, so run_group's `final = "dry_run"` (runner.py:230)
hit an IntegrityError in finish_group_run on every group dry run.
The per-module work was unaffected -- run_log already permits 'dry_run', so
each member recorded its resolved source and merge SQL correctly. Only the
group-level bookkeeping failed, leaving the group_run row stuck at
'running'. Via the web UI the failure was invisible:
_run_group_in_background swallows the exception and writes status='error',
mislabeling a successful dry run.
SQLite has no ALTER for CHECK constraints, so this rebuilds the table, the
same treatment run_log received earlier. Idempotent on the constraint text,
so it runs once. Verified on the live DB: 10,209 rows preserved, no foreign
key violations, sqlite_sequence intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
engine.run_group already sequences a group's enabled members by run_order,
continues past individual failures, and aggregates status -- but nothing
outside the process could reach it. The scheduler called it on cron, and
the web UI exposed it as a form POST that returns 303 + a background task.
Neither is usable from an external orchestrator: driving the web route
means scraping HTML to poll for completion.
This wraps it as a synchronous CLI command, mirroring `pipekit run` for
modules, so callers get an exit code and a summary on stdout.
LockBusy maps to exit 75 (EX_TEMPFAIL) rather than 1, so a caller can
distinguish "another run holds the lock, retry later" from a genuine
failure. This intentionally diverges from `pipekit run`, which returns 1
for both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The groups list only showed last-run info, so you had to open each
group to see whether it was scheduled. Attach each group's schedules
(with the next fire time) in groups_index and render them as new
"schedule" and "next run" columns; multiple schedules stack, disabled
ones are greyed with no next fire.
_schedules_with_next now also computes a relative next_fire_in label
("in 23h 43m"), shown on both the groups list and the group detail
page, and drops the empty %Z from the timestamp.
The groups template tolerates a missing g.schedules so the currently
running process (which auto-reloads templates but not app.py) keeps
rendering until it is restarted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
config/ had drifted from pipekit.db: group memberships and three
schedules existed only in the DB, cjobdr and cjobh had no sidecars at
all, and ab.{json,sql} was left over from a module no longer in the DB
(an apply would have recreated it). pipekit apply --dry-run now reports
that the database matches config.
Two real inconsistencies fixed in ffpdglr1.json: it was missing the
ffpdglr1_hot watermark added with the rollover hot set, so an apply from
files would have deleted the watermark while the committed SQL still
referenced {ffpdglr1_hot}; and its period resolver had been edited in the
UI without an export, so the file carried a stale format('%L', ...)
variant.
Purchasing members were all run_order 0, which happened to sort
poh, poh_deleted, poi alphabetically and satisfy the real constraints by
accident. Now explicit: poh 1, poi 2, poh_deleted 3 -- poi resolves its
watermark off cms.poh, and poh_deleted's first hook reads cms.poh.
Also describes eight GP modules that had no description (dest_description
is the half the sidecar format carries; the module-level description
lives only in the DB).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rm20201 (Applied Open) is 985 rows and stays a full reload. rm30201
(Apply History) was a full pull of 1,149,405 rows taking 19 minutes; it
now moves 661 rows in 6 s and reconcile --quick ties on all 108 metrics.
Neither apply table has DEX_ROW_TS, so rm30101's pattern does not port.
DEX_ROW_ID catches inserts but never moves on an update, so a 7-day
GLPOSTDT/APFRDCDT window carries restated and back-posted applies.
APTODCDT belongs to the apply-to document and lags by weeks; DATE1 has
163 rows dated to year 6201.
The pull is document-grain, not row-grain. merge_key is (aptodcty,
aptodcnm), so the DELETE clears every apply row for a document, and
staging only the row that matched would drop its siblings -- ~16k of
1.13M docs carry more than one. AK1/AK5 lead on those two columns, so
the EXISTS is an index seek.
Uses OPENQUERY pushdown rather than the four-part linked-server name: an
aggregate through the four-part name ran past 2 minutes against 1.3 s
pushed down. Note no apostrophes may appear inside the OPENQUERY string,
comments included.
Unapplies remain uncovered -- they delete the RM30201 row and write one
back to RM20201, and delete-by-key cannot see a row that is gone. The
985-row rm20201 reload is the cross-check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fiscal-period floor advances to the new period as soon as one batch
posts into it, so a late posting into the prior period stopped being
refreshed from that moment. FFPDGLR1 itself has no dates -- only
BTID/ENT#/SEQ# -- so RPRH is the only change signal available: the new
branches follow its NWPDAT/NWUDAT/NWCDAT stamps against a 14-day floor,
and carry the unposted batches too.
Strictly additive. Measured mid-period the hot set is a subset of the
period branch, adding 0 rows -- the run stays at 132,325 rows. It earns
its keep at a period boundary, the only time the two floors disagree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NWPDAT stays 0001-01-01 until a batch posts, so the watermark branch
alone can never see an in-flight batch -- it only appears once posting
stamps a real date. Only 4 rows are unposted, so the whole set now comes
over every run.
Found by reconcile: dest and source tied on count but five numeric sums
differed, which resolved to a single row pair -- an unposted batch
missing from the dest, and one stale row for a batch purged upstream
(deleted separately, since delete-by-key cannot remove it).
The resolver still has no lookback, so a back-dated post below the
high-water stays invisible; posting stamps the current date, so this only
matters for a genuinely back-dated one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
poh and poi were full truncate-reloads: 187k rows / 30 s and 529k / 75 s.
Both now pull only changed POs -- 1,201 rows / 5.5 s and 3,660 / 10 s --
and reconcile.py --quick reads IN SYNC on all 115 / 120 metrics.
POH turns out to carry its own change stamp in its "future" fields:
KAFUT12 is Date Created and KAFUT20 is Date Updated, both CHAR(10) ISO
text. Predicate is those two against the watermark, plus the open hot set
KACRCM = '1' -- 1,035 of 187k headers, and KACRCM leads most of POH's
logicals, so it is a keyed read. POI has no change stamp of its own, so
it joins the header's changed set at PO grain; its merge key is kbpo#, so
staging only the changed lines would drop the rest of the PO.
CMS logs this family (POHL, POIL), but both are redundant for change
detection: over 7 days POHL and the KAFUT20/12 window agree exactly, and
POIL reports no PO that POHL does not, so a line edit always stamps the
header. The logs' one unique contribution is deletes, which an
incremental merge structurally cannot see -- delete-by-key only touches
staged keys, so a purged PO would sit in the dest forever (~88 POs/yr).
poh_deleted closes that: DQ0ACTN = '3' is the delete action, and the
NOT EXISTS against live POH makes the list exact rather than a guess
about action ordering, since a re-used PO number is back in POH and drops
out on its own. Three dest hooks, in order -- clearing the tombstone of
any PO that is live again must happen before the deletes, or the second
hook could remove a row poh had just refreshed.
Whenever these are scheduled, run order must be poh -> poi ->
poh_deleted: poi's watermark resolves off cms.poh, and poh_deleted's
first hook reads cms.poh expecting it to be fresh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--strict means a failure now exits non-zero, so the returncode branch
raises instead of _detect_silent_failure. That branch used raw stderr,
whose first line is always the JVM's "Picked up JAVA_TOOL_OPTIONS: ..."
banner (_subprocess_env sets JAVA_TOOL_OPTIONS to keep jt400 headless).
Every failed run would therefore have recorded that banner in run_log
instead of the actual error — a regression against the old path, which
extracted the exception header.
_error_message strips the banner and prefers the Java exception header:
org.postgresql.util.PSQLException: ERROR: relation "..." does not exist
org.postgresql.util.PSQLException: ERROR: syntax error at or near "SELEKT"
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hold/release rewrites OCRH.DCSTAT and DCHREA without touching any date
column, so no watermark could see it. A full-table row diff of LGDAT.OCRH
vs cms.ocrh (468,823 rows both sides) found 168 drifted rows, 0 missing,
0 extra -- and 159 of them changed with no date moving at all (checked
dcudat/dcodat/dccdat/dcmdat/dchdat/dcxdat/dcsdat/dcmxdt). 155 of the
drifts were DCSTAT, 4 DCHREA.
Every drifted row was still open (DCSTAT in A/N/H/B); none had reached
'C'. Only ~2,044 of 468,823 orders are open, so re-pulling the whole open
set each run is cheap and catches 100% of the observed drift. Applied to
ocrh and to the two modules that read its changed-set, ocri and ocrs.
Result: drift 168 -> 11 rows, DCSTAT 155 -> 3.
No invoice-driven branch: tested against LGDAT.OCRHT (CMS's header change
log, keyed DC3TMSP-leading via logical OCRHTX1) over the live watermark
window -- 1,107 headers changed, 65 missed by the date branches, 0 missed
by dates+open, and a 5- or 30-day invoiced-orders branch added exactly 0.
Closing an order writes dcudat/dccdat, so the date window covers the flip.
Also evaluated LGDAT.OCRIT (the order-line change log) as a replacement
for the open-order branch and rejected it: at the live 8-day watermark it
reports 0 changed orders not already caught, at ~12x the predicate cost,
and its lower row count means it declines to pull 18k lines the open set
would -- safe only if CMS's logging is complete, which is unverified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Credentials no longer reach jrunner on argv. _write_passfile drops a 0600
alias file per call and the existing finally removes it, so the password
flags are gone from the command line — verified via /proc/<pid>/cmdline
that a live jrunner shows only "-sc src --passfile ... --strict".
This was an active exposure, not a hypothetical: a running migration was
observed with both an AS/400 and a Postgres password in plaintext in its
ps output.
Both entry points now pass --strict, so a failed jrunner is a non-zero
exit rather than something to infer. That only became safe once jrunner
stopped raising on statements that return no result set (DDL, TRUNCATE,
INSERT) — without that fix, --strict would have failed every merge.
Row counts come from jrunner's @summary line instead of three regexes
guessing at prose. _parse_row_count and _detect_silent_failure stay as
fallbacks for an older jar and can go once the version is pinned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Summary, install, then a fuller description starting with the tech stack.
Points at SPEC.md for architectural rationale rather than restating it, so
the two don't drift.
Also documents `pipekit export` / `pipekit apply`, which round-trip the
driver/connection/module/group config to text files and were missing from
CLAUDE.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reconcile.py was written against the DB2 LGDAT modules and broke four
different ways on the first GP module (sop30200):
- detect_source_from's regex handled at most a two-part name, so
CHG.dbo.SOP30200 truncated to CHG.dbo -> "Invalid object name".
Now matches 1-4 parts, including [bracketed] and "quoted" forms.
- Even fully qualified, that table only exists behind the GPSERVER
linked server. Detect the module's OPENQUERY wrapper and push the
aggregate through it, so the scan runs remotely and one row comes
back instead of 1.6M rows crossing the link.
- OPENQUERY caps its passthrough string at 8000 chars and 291 metrics
overran it. Split the metric list into chunks that fit and CROSS JOIN
them back into one row; alias columns c0..cN since OPENQUERY rejects
unnamed result columns (msg 8155). EXEC(@sql) AT has no such cap but
needs RPC Out, which GPSERVER has disabled.
- Two dialect bugs that would hit any SQL Server source: T-SQL SUM(int)
stays int and overflowed at 2^31 where Postgres promotes to bigint,
and T-SQL has no LENGTH. Added Driver.sum_expression /
length_expression with mssql overrides -- integer types cast to
BIGINT (decimals left alone so scale isn't lost), LENGTH -> LEN.
--super-quick and --quick now run clean end to end. Full depth is
syntax-checked only; it wasn't run against the server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scope, decisions (dest schema cms.*, fresh DB2 introspection, one table
per cutover), module recipe, and the remaining backlog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--super-quick compares only COUNT(*) and COUNT(DISTINCT merge_key) —
seconds on multi-million-row tables, enough to catch missing/duplicated
rows but blind to changed values.
Also fix derived merge keys: columns_json source_name can hold a SQL
expression (e.g. SUBSTR(GGKEY,1,9)), which must be emitted verbatim
rather than quoted as an identifier (SQL0206) or re-transformed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Export reconciliation: the db has default_dest_schema=gp for the sqlserver
connection (GP tables land in gp.*); bring the config file in line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-reload modules for the CMS/DB2 tables migrated off /opt/sync, landing
in cms.*. Columns introspected fresh from DB2 (QSYS2.SYSCOLUMNS). Three large
tables use incremental merge ported from the legacy sync instead of full
reload: iprcctn/iprccto (date watermark on t8date/t7date), ffpdglr1 (fiscal-
period watermark via rprh join, keyed on btid).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five dest_names in the column map were junk from a crude source-name
sanitizer (ext._price, margin__, _hd_grower, ext._cost,
ext._margin_dollar) and didn't match the physical gp table's clean names
(ext_price, margin_pct, hd_grower, ext_cost, ext_margin_dollar). The load
is positional so this was harmless for years — until the new dest-table
preflight's name check started (correctly) flagging the drift and failing
the daily run. Align the map to the table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The engine built staging as `LIKE dest` and merged into it without ever
checking the destination existed — a missing dest (e.g. a file-based deploy
that never went through the web wizard) failed deep in the staging step with
a raw `relation ... does not exist`.
Add `engine/dest.reconcile_dest()`, called as step 3b of run_module (after the
dry-run return, before staging):
- missing + not create -> DestError telling the user to pass --create-dest
- missing + create -> CREATE SCHEMA + CREATE TABLE from the module's
column map
- exists -> fail loudly if any mapped column is absent (drift);
no auto-ALTER, since the positional load can't
tolerate a mid-table add
Expose via `pipekit run --create-dest`. Default (scheduled groups, API) is
unchanged: validate-and-fail, never silently provision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reconcile.py compares a module's live source table against its synced dest
using column-wise aggregates (COUNT/SUM/MIN/MAX/SUM(LENGTH)) — arithmetic and
ordering that DB2 for i and Postgres compute identically, so no shared hash or
byte-identical serialization is needed. Re-applies the module's per-column
source transform (default_expression) so aggregates line up when row sets
agree; exits non-zero on any divergence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
groups_index attaches last_duration_s from the latest group run; groups.html
renders it as a new right-aligned duration column (— when never run).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Modules:
- add `description` column (schema + migration); create/update/wizard/edit
forms and module detail all carry it
Modules list page:
- columns: name, description, groups, last run, rows, duration, status
(dropped strategy/dest and the run/dry-run buttons)
- every column click-to-sort; description truncates single-line with ellipsis
- compact grid; restore main max-width (now 1400px)
- status pill partial is now a <span> so it sits in its own cell
Startup reliability:
- reconcile_abandoned_runs(): on boot, mark run_log/group_run rows stuck in
`running` (left by a hard kill mid-run) as `error`, complementing the
existing stale module-lock sweep
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `pipekit export` / `pipekit apply` (pipekit/config_io.py) to serialise the
DB's config — drivers, connections, modules (+ columns, watermarks, hooks),
groups, schedules — to a git-trackable text tree under config/, and rehydrate
it. SQLite stays runtime state; definitions become diffable/reviewable/
revertible.
- config only: run_log/group_run/settings and per-run state columns excluded
- name-keyed refs (portable across databases); source_query in .sql sidecars;
columns as real JSON arrays for line-by-line diffs
- newline-normalised so CRLF-vs-LF is never a spurious change
- apply is create/update by name; child collections fully synced; top-level
deletes gated behind --prune; --dry-run prints the plan
- round-trip is identity (export -> apply --dry-run == nothing to do)
Commits the current config/ as the first baseline, capturing the freshly
populated columns_json for rm00101/rm00301/iv00101. Documented in SPEC.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds run cancellation, which had no implementation despite `cancelled`
being a declared run status.
Mechanism: API/web/scheduler runs execute as background tasks in the
service process, so `running_pid`'s PID is the service itself — killing it
is not an option. Instead, engine/cancel.py keeps an in-process registry of
run_id -> live jrunner Popen. The cancel handler (same process) terminates
that subprocess; migrate then fails and the engine records the run as
`cancelled` (not `error`) and releases the lock in its finally. Checkpoints
before migrate and before merge honor a cancel flag set between phases.
- engine/cancel.py: registry (arm/register/unregister/is_cancelled/request_cancel) + RunCancelled
- jrunner.migrate: optional proc_hook to expose the live Popen (no engine dependency)
- runner: arm, register via proc_hook, two checkpoints, classify cancelled, unregister
- POST /api/modules/{id}/cancel (extracts run_id from the lock) and POST /runs/{id}/cancel (web)
- cancel button on run detail + live fragment, shown while running
- api/app lifespan: clear stale locks (dead PID / >24h) at startup via the existing
repo.clear_stale_locks, so a hard service kill no longer wedges `running=1`
Verified: real subprocess terminate -> status=cancelled, lock released;
endpoints return 409 (not running) / 404 (unknown) / 200 (fragment).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Incremental merge interpolated merge-key columns raw, so a key with a
character illegal in a bare identifier (e.g. dcord#) produced
WHERE dcord# IN (...) and Postgres errored at "IN". build_merge_sql now
takes the dest driver and quotes each key via quote_identifier (clean
names stay bare, the rest wrap in the dialect's quotes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the -b wiring to jdbc:postgresql: dests, so DB2->PG (and any PG-dest)
loads use jrunner's COPY FROM STDIN path instead of batched INSERTs. SQL
Server already used -b (SQLServerBulkCopy); DB2 dests stay on INSERT. Update
the CLAUDE.md bulk section accordingly.
Validated DB2->PG COPY with real types (dates -> date col, decimals ->
numeric, char) and null/empty-string fidelity.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Catch CLAUDE.md up to features shipped this cycle:
- post-creation "+ add column" (ALTER ADD at tail, stable column ids)
- auto bulk copy for SQL Server dests (jrunner -b) + the PG streaming note
- HTTPException + masked request-body logging
- live_log progress-tick collapse to a single updating line
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jrunner prints an in-place progress counter (a bare number) every batch;
read with universal newlines, each tick became its own live_log line, so a
large load stacked thousands of numbers (a 1.27M-row INSERT left 5,073
progress lines). append_run_live_log now overwrites a trailing bare-number
line with the new tick instead of appending, keeping a single current count.
Real lines (headers, "N rows written", timestamps) aren't bare numbers and
are preserved. No jrunner change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pass jrunner's -b flag when the dest JDBC URL is jdbc:sqlserver:, so SQL
Server loads stream via TDS bulk copy instead of 250-row INSERT...VALUES
round trips. Non-SQL-Server dests are unchanged. Requires the jrunner -b
support (bulk-copy branch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wizard_create built the generated source query's column aliases with the
DEST driver's quote_identifier, but that query runs on the SOURCE. A pg->SQL
Server module emitted "AS [col]" (SQL Server brackets) into a Postgres query,
which failed with: syntax error at or near "[". The load maps columns by
position, so the alias is cosmetic — quote it with the source dialect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an ASGI middleware that buffers each request body onto the scope and
replays it downstream, so the HTTPException handler can log the submitted
payload alongside the error. Fields whose name looks secret (password/pwd/
secret/token) are masked. Makes failures like the wizard 500 debuggable
against the actual call content.
Covers raised HTTPExceptions; handlers that *return* an error response are
not body-logged yet.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_dest_sql executes DDL via jrunner query mode (executeQuery), which demands
a ResultSet. CREATE SCHEMA/TABLE produce none, so the driver throws, jrunner
logs the trace and exits 0, and _detect_silent_failure flags it as a failure
unless the message is allowlisted. Only PG's wording was listed ("No results
were returned by the query"); SQL Server says "The statement did not return a
result set." — so pg->mssql wizard provisioning died on the first statement.
Add the SQL Server phrasing to the benign list.
Verified against the live SQL Server connection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Failures surfaced via HTTPException (e.g. wizard dest-provisioning errors
raised as HTTPException(500, "dest provisioning failed: …")) were turned
into responses by FastAPI and never logged — only the access line showed,
so the real DB error went to the browser and vanished from the journal.
Register a StarletteHTTPException handler that logs 5xx at ERROR (with
exc_info, capturing the chained cause) and 4xx at WARNING, then defers to
the default handler. Also configure pipekit's logger to emit to stderr so
INFO-level records aren't dropped by uvicorn's last-resort handler.
Unhandled (non-HTTPException) errors were already logged by uvicorn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Modules' column listings were write-once at wizard time — no way to add a
column to an established sync (e.g. an RRN watermark column on a history
table) without hand-editing columns_json and ALTERing the dest by hand.
Phase 0 (groundwork):
- columns_json rows get a stable `id` (c1, c2, …) — the data-movement
identity for future schema reconciliation (the load is positional).
- repo.update_module_columns to persist the listing.
- Driver.build_add_column_sql + Driver.column_inventory.
Phase 1 (append a column):
- "+ add column" on the module detail page -> column_form.html.
- POST /modules/{id}/columns: validates the name isn't already in the
listing or on the table, runs ALTER TABLE … ADD COLUMN (appends at the
tail, rows preserved), applies COMMENT ON COLUMN where supported, and
appends to columns_json. Re-renders with an error on conflict/DDL failure.
Append-only and non-destructive; reorder/retype/drop (which can require a
table rebuild) are out of scope for this phase. Verified end-to-end against
the live PG dest on a throwaway module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wizard previously required picking a single source table; modules whose
entry point is arbitrary SQL (CTEs, joins, computed columns) didn't fit. Add a
"write SQL" path alongside "browse a table":
- Driver.introspect_query_columns + _zero_row_wrap discover a query's result
columns by running it with ~no rows. Generic wrap is a derived table with
WHERE 1=0; DB2 appends FETCH FIRST 1 ROW ONLY (DB2 for i forbids WITH inside
a nested table expression).
- /wizard/sql + POST /wizard/sql/columns seed the column-mapping grid; dest
types default to text (no result-set type metadata over jrunner CSV).
- wizard_step3.html grows a sql_mode branch (array-named inputs, query shown
verbatim, no column unchecking); wizard_create branches on entry_mode.
Verified end-to-end against a live DB2 for i connection, including a top-level
CTE query.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add dialect-aware DDL hooks to the Driver base (create_schema_sql,
drop_table_if_exists_sql, create_like_table_sql, check_dest_table) and
implement DB2/MSSQL overrides so they can serve as merge destinations,
not just Postgres. runner.py now dispatches staging table creation
through the dest driver instead of hardcoding PG syntax.
Also untrack pipekit.db (runtime SQLite state) and add it to .gitignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- deploy.sh: set /etc/pipekit to root:pipekit 0775 and secrets.env to
pipekit:pipekit 0640 so group members can run 'pipekit secrets set'
without sudo
- cli.py secrets set: drop os.chown() on temp file — non-root users
can't chown to the pipekit service user, and os.replace() preserves
the target file's ownership anyway
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- New pipekit/web/auth.py: itsdangerous-signed cookie, 8hr expiry,
auto-generates signing secret in settings table on first use
- GET/POST /login and POST /logout routes (public, no auth dependency)
- All other web routes protected via router-level require_web_auth dep
- Starlette middleware injects request.state.current_user for templates
- Topbar shows logged-in username + logout button when session active
- Reuses existing api_user/api_pass credentials and api_auth_enabled flag
- Add itsdangerous>=2.1 to requirements.txt
- Enable api_auth_enabled in config.yaml
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Size all table columns to fit content (em-based) rather than loose percentages
- Add white-space:nowrap to groups header and last-run cells
- Sticky topbar and panel header so New Module button stays visible while scrolling
- Scope min-width:0 to label.field inputs so they don't blow past two-col grid borders
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Convert all timestamps to local time for display and scheduling
- deploy.sh: detect JAVA_HOME and inject into systemd unit at deploy time
- repo: add duration_s to get_group_run query
The group run detail page was crashing because get_group_run returned no
duration_s field, unlike the list queries. Fixes 500 on /group-runs/{id}.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The pipekit system user has no PATH to java. deploy.sh now detects
JAVA_HOME by searching common locations and injects Environment= lines
into the installed unit file, making deploys portable across machines.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
