drivers: make staging DDL dialect-aware; stop tracking pipekit.db

Add dialect-aware DDL hooks to the Driver base (create_schema_sql,
drop_table_if_exists_sql, create_like_table_sql, check_dest_table) and
implement DB2/MSSQL overrides so they can serve as merge destinations,
not just Postgres. runner.py now dispatches staging table creation
through the dest driver instead of hardcoding PG syntax.

Also untrack pipekit.db (runtime SQLite state) and add it to .gitignore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -2,7 +2,7 @@ __pycache__/
 *.py[cod]
 *.egg-info/
 
-# Local SQLite database — contains connection rows + user state.
+# SQLite database and its WAL/journal artifacts — runtime state, not tracked.
 pipekit.db
 pipekit.db-journal
 pipekit.db-wal

CLAUDE.md

@@ -4,13 +4,22 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Running the Server
 
+Pipekit runs as a systemd service (`pipekit.service`). Common commands:
+
+```bash
+sudo systemctl status pipekit          # check status
+sudo systemctl restart pipekit         # restart after changes
+sudo systemctl stop pipekit            # stop
+sudo journalctl -u pipekit -f          # follow logs
+```
+
+For dev/testing outside the service — never nohup or background it:
+
 ```bash
 pipekit serve                          # default host/port from config.yaml
 pipekit serve --host 0.0.0.0 --port 8080 --reload   # dev mode with auto-reload
 ```
 
-The user runs the server themselves in their own terminal — do not nohup or background it.
-
 ## Other CLI Commands
 
 ```bash
@@ -19,8 +28,10 @@ pipekit doctor                         # health check (config, jrunner, DB)
 pipekit run <module_name>              # run a module synchronously (manual test)
 pipekit set-password <username>        # set HTTP Basic Auth credentials
 pipekit secrets set KEY [VALUE]        # add/update a secret (prompts if value omitted)
+pipekit secrets list                   # show stored secret keys (no values)
 pipekit drivers list                   # show registered driver kinds
-./deploy.sh                            # idempotent: venv, deps, launcher, driver registration
+./deploy.sh                            # full idempotent deploy: user, venv, deps, launcher,
+                                       # secrets.env, schema init, driver registration, systemd unit
 ```
 
 ## Architecture
@@ -30,7 +41,7 @@ Pipekit is a database sync tool. A **module** defines a source query → dest ta
 **Layers (bottom to top):**
 
 1. **SQLite** (`pipekit.db`) — single file, all state
-2. **`repo.py`** — all CRUD for every table; ~1,900 LOC, the only layer that touches the DB
+2. **`repo.py`** — all CRUD for every table; ~725 LOC, the only layer that touches the DB
 3. **`engine/`** — orchestrates a module run: lock acquisition, watermark resolution, jrunner calls, merge SQL, post-run hooks, run_log write, lock release
 4. **jrunner** — external Java CLI; handles all JDBC access. Python never talks to remote DBs directly; it shells out to jrunner
 5. **`api/`** — FastAPI REST endpoints under `/api/*`, HTTP Basic Auth (except `/health`)
@@ -43,24 +54,44 @@ Pipekit is a database sync tool. A **module** defines a source query → dest ta
 - **module** — sync job (source/dest connection, source query, merge strategy, merge key, enabled, running lock)
 - **watermark** — named placeholder with resolver SQL; first column of first row used as opaque string; replaces `{watermark_name}` in source query
 - **hook** — post-merge SQL (run_order, run_on: success/failure/always)
-- **run_log** — immutable history record with resolved SQL, merge SQL, watermark values, stdout/stderr, timing
+- **run_log** — immutable history record with resolved SQL, merge SQL, watermark values, stdout/stderr, timing, live_log (streamed jrunner output written during the run)
+- **grp** — named group of modules; modules belong to a group via `group_member` (with run_order)
+- **group_member** — join table: group → module with run_order; disabled modules are skipped by group runs
+- **group_run** — one execution of a group: status, timing, triggered_by (manual | schedule:{id})
+- **schedule** — cron expression tied to a group; scheduler fires `run_group` when due; tracks `last_fired_at` to survive restarts without double-firing
 
 ## Engine Flow
 
 ```
 run_module(module_id)
   → atomic UPDATE module SET running=1 WHERE running=0  # fail if already locked
-  → for each watermark: run resolver SQL via jrunner → capture first cell
+  → for each watermark: run resolver SQL via jrunner → capture first cell  # always live, even dry runs
   → materialize source query (simple string replace {name} → value)
-  → jrunner create staging table (pipekit_staging.{module_name})
-  → jrunner migrate source → staging
   → build merge SQL (engine/merge.py: full=truncate+insert, incremental=delete by key+insert, append=insert)
+  → if dry_run: write run_log (status='dry_run'), return early — no data movement
+  → DROP + CREATE staging table (pipekit_staging.{module_name})  # self-healing schema drift
+  → jrunner migrate source → staging  # uses Popen; each stdout line appended to run_log.live_log
   → run merge SQL via jrunner
   → run hooks in order
   → write run_log entry
   → UPDATE module SET running=0 (in finally)
+
+run_group(group_id)
+  → create group_run row (status=running)
+  → for each enabled group_member in run_order: call run_module(group_run_id=...)
+  → continues past individual module failures (all members run)
+  → final status: dry_run if all dry_run, error if any errored, success otherwise
+  → finish_group_run(status)
 ```
 
+## Run Statuses
+
+- `running` — in progress
+- `success` — completed, data moved
+- `dry_run` — resolved watermarks + built SQL, no data movement
+- `error` — failed
+- `cancelled` — cancelled mid-run
+
 ## Credentials Pattern
 
 Passwords in connections are stored as `$VAR_NAME` references. At run time they are resolved from `/etc/pipekit/secrets.env` (override with `PIPEKIT_SECRETS` env var). Config path override: `PIPEKIT_CONFIG`.
@@ -73,6 +104,10 @@ Each driver in `pipekit/drivers/` inherits from `base.py::Driver` and implements
 
 Modules store their column mapping as `columns_json` — a JSON list of dicts with keys `source_name`, `source_type`, `dest_name`, `dest_type`. The engine uses this to build the staging CREATE TABLE and the merge INSERT column lists.
 
+## Inline Watermark Editing
+
+Watermarks are managed inline on both the module edit form and wizard step 3 (not just via the standalone `/watermarks/{id}/edit` page). The module edit form (`module_form.html`) renders existing watermarks as editable rows and submits them as parallel arrays (`wm_id[]`, `wm_name[]`, `wm_connection_id[]`, `wm_resolver_sql[]`, `wm_default_value[]`, `wm_deleted_id[]`). The `_save_inline_watermarks(form, module_id)` helper in `web/app.py` processes these arrays — updates existing rows, creates new ones, deletes removed ones. Both `module_update` and `wizard_create` call it after saving the module. A client-side placeholder warning checks that `{name}` tokens in the source query match the watermark names on the page.
+
 ## Merge Key
 
 `merge_key` is stored as a comma-separated string (e.g., `"col1, col2"`). The engine parses it and generates a multi-column DELETE predicate for incremental strategy.
@@ -81,18 +116,25 @@ Modules store their column mapping as `columns_json` — a JSON list of dicts wi
 
 Recreated on every run as `pipekit_staging.{module_name}` (DROP + CREATE, not IF NOT EXISTS). Ephemeral — exists only during the run.
 
+## Scheduler
+
+`pipekit/scheduler.py` runs a single daemon thread (started via FastAPI lifespan in `api/app.py`). It wakes every 60 s, reads all enabled schedules, and fires `run_group` for any whose next cron occurrence has passed since `last_fired_at`. `last_fired_at` is written to the DB before the run thread is spawned — prevents double-fire if a run is slow or pipekit restarts mid-run. Uses `croniter` for cron expression evaluation. Cron expressions are evaluated in **local server time** (not UTC) — `0 4 * * *` fires at 04:00 local.
+
 ## API vs. Web
 
 - `/api/*` — JSON REST, HTTP Basic Auth, consumed by HTMX fragments and external callers
 - `/` and other bare paths — full HTML pages (Jinja2), no auth currently
-- `POST /modules/{id}/run` returns `{run_id}` immediately; run is async; poll `/runs/{id}` for status
+- `POST /modules/{id}/run` returns `{run_id}` immediately (both API and web); run is async via BackgroundTasks
+- `GET /runs/{id}/live` — HTML fragment endpoint; HTMX polls this every 2 s while status=running to show live_log + status
+- `/groups/*` — group CRUD, run, live fragment; `/group-runs/{id}` — group run detail with per-module run links
 
 ## Tech Stack
 
-- Python 3.10+, FastAPI, Uvicorn, Jinja2, PyYAML, SQLite3 (stdlib)
+- Python 3.10+, FastAPI, Uvicorn, Jinja2, PyYAML, SQLite3 (stdlib), croniter
 - `python-multipart` required for HTML form POSTs (not auto-installed as a FastAPI transitive dep)
-- Frontend: HTMX + Alpine.js (CDN), no build step
-- jrunner: separate Java tool, must be on PATH
+- Frontend: HTMX (CDN) + vanilla JS; Alpine.js is NOT loaded despite being listed in older docs
+- jrunner: separate Java tool, must be on PATH; `JAVA_HOME` must also be set — `deploy.sh` detects and injects it into the systemd unit automatically
+- Runs as the `pipekit` system user; venv at `/opt/pipekit/.venv` owned by that user; secrets at `/etc/pipekit/secrets.env` (mode 0640, group pipekit)
 
 ## Full Spec
 

SPEC.md

@@ -6,7 +6,9 @@ design from first principles. The previous version is archived at
 
 ## Status
 
-**Spec is done.** Ready to move to implementation planning.
+**Implementation is underway.** Core module sync, wizard, web UI, groups,
+scheduling, and deploy tooling are built and running. The TUI was dropped
+in favour of a web UI (see Architecture note below).
 
 One item is intentionally deferred: the **migration plan** for bringing
 over the ~90 existing modules from `/opt/sync`. Not needed to start
@@ -72,16 +74,30 @@ jrunner   (Java CLI — bulk JDBC transfer + query mode)
    ↑
 engine    (Python — orchestrates jrunner, watermarks, merge, hooks, run log)
    ↑
-API       (FastAPI — REST, Basic Auth)
+API process  (FastAPI — REST, Basic Auth)
+   ├── scheduler thread  (daemon thread; fires group runs from cron schedules)
+   └── web UI            (Jinja2 templates, HTMX, vanilla JS — served alongside /api/*)
    ↑
-TUI / web UI / curl
+TUI (deferred) / curl
 ```
 
 The engine shells out to jrunner for **everything that touches a database** —
 bulk transfers, watermark resolver queries, hooks. No separate JDBC layer in
 Python. One driver-loading code path, one set of bugs.
 
-The API exists so a web front-end or curl can drive Pipekit, not just the TUI.
+The scheduler is a daemon thread started inside the API process via FastAPI
+lifespan — not a separate process or service. It shares the same SQLite
+connection pool and imports.
+
+The FastAPI app serves both a JSON REST API (`/api/*`) and full HTML pages
+(`/`, `/modules/*`, `/groups/*`, etc.) using Jinja2 templates, HTMX for
+in-place updates, and vanilla JS. The web UI covers all management: modules,
+connections, groups, schedules, run history.
+
+**TUI is deferred, not dropped.** The original TUI design remains valid — it
+would be an HTTP client against the API, never touching SQLite directly. Build
+it when the web UI's limitations become friction (keyboard-heavy workflows,
+SSH-only environments). The API design already accommodates it.
 
 ## Storage: SQLite
 
@@ -185,15 +201,16 @@ hooks later if it gets painful.
 ## Engine flow (per module run)
 
 1. **Acquire lock** atomically: `UPDATE module SET running=1 WHERE id=? AND running=0`. If row count is 0, bail with "already running."
-2. **Resolve watermarks.** For each watermark: shell out to jrunner query mode against the watermark's connection with its resolver SQL. Take first row's first column as a string. Fall back to `default_value` on NULL/empty.
+2. **Resolve watermarks.** For each watermark: shell out to jrunner query mode against the watermark's connection with its resolver SQL. Take first row's first column as a string. Fall back to `default_value` on NULL/empty. Always runs live — even for dry runs.
 3. **Materialize the resolved source query.** Substitute `{name}` placeholders in `source_query`. Store on the module record so the TUI can preview.
-4. **Truncate staging** (`TRUNCATE pipekit_staging.{module_name}`).
-5. **Run jrunner** (migration mode) with the resolved query, target = staging.
-6. **Materialize the merge SQL** based on strategy + merge_key.
-7. **Run merge** against dest connection (also via jrunner, or whatever path the engine uses for SQL execution).
-8. **Run hooks** in order, respecting `run_on`.
-9. **Write `run_log` entry** with everything (see below).
-10. **Release lock** in a `finally` block — always runs, even on error.
+4. **Materialize the merge SQL** based on strategy + merge_key.
+5. **Dry-run exit.** If `dry_run=True`, write `run_log` with status `dry_run` and return — no data movement.
+6. **Recreate staging** (DROP + CREATE … LIKE dest). Self-healing against schema drift.
+7. **Run jrunner** (migration mode) with the resolved query, target = staging.
+8. **Run merge** against dest connection via jrunner.
+9. **Run hooks** in order, respecting `run_on`.
+10. **Write `run_log` entry** with everything (see below).
+11. **Release lock** in a `finally` block — always runs, even on error.
 
 ## Locking
 
@@ -228,14 +245,15 @@ run_log(
     group_run_id,           -- nullable; set when run as part of a group
     started_at, finished_at,
     row_count,
-    status,                 -- running | success | error | cancelled
+    status,                 -- running | success | error | cancelled | dry_run
     error,
     resolved_source_sql,    -- exact SQL that ran on source
     merge_sql,              -- exact merge SQL that ran on dest
     watermark_values_json,  -- {prev_period: "'2610'", ...}
     jrunner_stdout,
     jrunner_stderr,
-    hook_log
+    hook_log,
+    live_log                -- jrunner stdout lines appended in real time during migrate
 )
 ```
 
@@ -263,12 +281,21 @@ grp(id, name)
 group_member(id, group_id, module_id, run_order)
   -- many-to-many; same module can live in multiple groups with different run_orders
 
-schedule(id, group_id, cron_expr, enabled)
+schedule(id, group_id, cron_expr, enabled, last_fired_at)
   -- a group can have 0..N schedules
 ```
 
-**Sequential execution, stop on failure.** Mirrors the `set -e` behavior of
-existing orchestrator scripts.
+**Sequential execution, continue past failures.** Each enabled member runs in
+`run_order` regardless of whether prior members errored. Final group_run status
+is `error` if any member errored, `success` if all succeeded, `dry_run` if all
+were dry runs. Disabled modules are skipped entirely.
+
+This was a deliberate change from the original "stop on failure" design. The
+case for stopping: mirrors `set -e` shell behaviour; avoids running downstream
+modules against stale data if an upstream one failed. The case for continuing:
+full visibility into all failures in one run rather than discovering them
+one-at-a-time; individual module locks already prevent unsafe concurrency.
+A per-group `stop_on_failure` flag would satisfy both and is worth adding.
 
 **Many-to-many membership.** Junction table is needed anyway for `run_order`,
 so many-to-many costs nothing extra. Unique constraint can be added later if
@@ -277,9 +304,14 @@ ever needed.
 **Schedule attaches to groups, not modules.** Matches the user's mental model
 and avoids a huge cron-list. Individual modules can still be run ad-hoc.
 
-**Scheduler.** Background thread inside the API process. Wakes every minute,
-evaluates all enabled schedules, fires any whose cron matches. A scheduled
-fire and a manual fire use the same code path — only `triggered_by` differs.
+**Scheduler.** `pipekit/scheduler.py` — a single daemon thread started via
+FastAPI lifespan on server startup. Wakes every 60 s. For each enabled
+schedule, uses `croniter` to find the next occurrence after `last_fired_at`
+(or 2 minutes ago if never fired); fires if that occurrence has passed.
+`last_fired_at` is written to the DB *before* the run thread is spawned —
+prevents double-fire if a run is slow or pipekit restarts mid-run. A scheduled
+fire and a manual fire use the same `run_group` code path; only `triggered_by`
+differs (`schedule:{id}` vs `manual`).
 
 **Ad-hoc runs:**
 
@@ -330,8 +362,30 @@ Pipekit verifies jrunner exists on startup (configurable path in
 drivers loadable, database accessible, all configured connections testable.
 First thing to run after a `git pull`.
 
-**Packaging.** Start loose-coupled (install jrunner separately, point Pipekit
-at it). Bundle later if/when the two-step gets annoying.
+**`./deploy.sh`** — idempotent full-system deploy. Run once from a fresh clone
+or re-run after any code update. Does:
+
+1. Creates the `pipekit` system user (`useradd --system`) if absent
+2. Chowns `/opt/pipekit` to `pipekit:pipekit`
+3. Creates/repairs the Python venv at `.venv` owned by `pipekit`
+4. Installs deps as `pipekit` user (`pip install -r requirements.txt`)
+5. Installs `/usr/local/bin/pipekit` launcher symlink
+6. Creates `/etc/pipekit/secrets.env` (mode 0640, group pipekit) if absent
+7. Runs `pipekit init` (schema creation + migrations) as `pipekit`
+8. Upserts JDBC driver rows for each jar found in jrunner's lib dir
+9. Installs and enables `systemd/pipekit.service`
+
+**Secrets** live in `/etc/pipekit/secrets.env` as `KEY=VALUE` lines. The
+`password` field on a connection row stores `$KEY`; the engine resolves it at
+runtime from the environment. The systemd unit loads this file as
+`EnvironmentFile=`. Mode 0640, group pipekit — the service account can read it
+but it is not world-readable. Managed with `pipekit secrets set/list/unset`.
+
+**jrunner is a prerequisite, not managed by deploy.sh.** `deploy.sh` checks
+that `jrunner` is on PATH and aborts with a clear message if not. Installing
+jrunner (`/opt/jrunner/deploy.sh`) is a separate step the operator does first.
+Keeping them decoupled means jrunner can be updated independently. Bundle later
+if/when the two-step gets annoying.
 
 ## New module wizard
 
@@ -364,8 +418,12 @@ Most of the time you accept defaults.
 
 Pick dest connection. Dest table defaults to
 `{source_conn.default_dest_schema}.{lowercase_source_table_name}`. Pick
-merge strategy. Pick merge key from a dropdown of dest column names. Add
-zero or more watermarks via a sub-form.
+merge strategy. Pick merge key. When strategy is `incremental`, a
+**Watermarks** panel and an editable **Source query** textarea appear
+inline. The source query is pre-populated from column picks; the user
+edits it to add the WHERE clause with `{placeholder}` references before
+creating the module. Zero or more watermarks can be added in step 3;
+they are created atomically with the module on submit.
 
 **Multiple destinations are real** (e.g. PG → SQL Server). The wizard
 doesn't assume one dest. Each source connection has a
@@ -434,8 +492,8 @@ Defaults are **opinions hardcoded in driver modules** for now. Lift to a
 
 ### Wizard scope (what it does NOT do)
 
-- **No CTE-based queries.** Wizard generates simple `SELECT cols FROM table WHERE watermark`. For complex queries (like `ffsbglr1`), create with the wizard and edit the source query post-creation via `e`.
-- **No multi-watermark wizard.** Single watermark. Add more after.
+- **No CTE-based queries.** Wizard generates a simple `SELECT cols FROM table` skeleton; the user adds WHERE clauses (including watermark placeholders) in the editable source query textarea in step 3, or post-creation via the module edit form.
+- **Multi-watermark supported in wizard.** Add as many watermarks as needed in step 3; they're created atomically with the module.
 - **No hooks in the wizard.** Add hooks from the module detail screen.
 - **No group assignment in the wizard.** Assign separately.
 
@@ -516,7 +574,7 @@ Anything with side effects or that composes multiple steps:
 ```
 POST /connections/{id}/test       run SELECT 1 via jrunner; return ok/fail/elapsed
 GET  /modules/{id}/preview        return next resolved source SQL + merge SQL
-                                  (runs watermark resolvers but does NOT sync)
+                                  (resolves watermarks live, does NOT sync)
 GET  /modules/{id}/columns        parse source query, return column list
 
 POST /modules/{id}/run            start async run; return {run_id} immediately
@@ -554,30 +612,35 @@ GET  /settings
 POST /settings/{key}
 ```
 
-### Async runs + SSE
+### Async runs + live progress
 
 `POST /modules/{id}/run` does NOT block. It atomically acquires the
 module lock, kicks off the sync in a background task, and returns
-`{"run_id": 4892}` immediately.
+`{"run_id": 4892}` immediately (both the API and web POST).
 
 Two ways to watch a run after that:
 
 1. **Polling** — `GET /runs/{id}` returns the run_log row; keep hitting
    it until `status != running`. Simple, works anywhere.
-2. **Streaming** — `GET /runs/{id}/stream` opens a Server-Sent Events
-   connection. The server pushes event lines as things happen — log
-   lines, row-count updates, final status. The TUI uses this for the
-   run watch screen. curl supports it with `-N` (no buffering).
+2. **Live fragment polling** — the web run detail page (`/runs/{id}`)
+   embeds an HTMX fragment that polls `GET /runs/{id}/live` every 2 s
+   while `status == running`. The fragment shows current status,
+   row count, and the `live_log` field as it grows. Polling stops
+   automatically once status leaves `running`.
 
-SSE is plain HTTP with a long-lived connection, not WebSockets. Simpler
-to implement, works in browsers natively (`EventSource` in JS), works in
-curl for debugging.
+**Live log** — during `jrunner.migrate()` the engine uses `subprocess.Popen`
+(not `subprocess.run`) so stdout can be read line-by-line as jrunner
+emits it. Each non-empty line is appended to `run_log.live_log` via
+`repo.append_run_live_log()`. How frequently lines appear depends on
+jrunner's output flushing behaviour; at minimum the final row-count
+summary line appears when the transfer completes.
 
-Splitting `start` from `watch` (two endpoints) means:
+Splitting `start` from `watch` means:
 
 - Cron-triggered runs don't have to watch
 - Curl scripting can fire-and-forget
-- TUI can reconnect to an already-running sync if it crashes mid-run
+- A browser navigating directly to `/runs/{id}` mid-run picks up the
+  live panel automatically
 
 ### Auth
 
@@ -617,7 +680,7 @@ behavior.
 | Hooks | Per-module, post-merge, run_on success/failure/always |
 | Group hooks | Deferred — not needed yet |
 | Group membership | Many-to-many (junction table for run_order anyway) |
-| Group execution | Sequential, stop on failure |
+| Group execution | Sequential, continue past failures; final status = worst of members |
 | Schedules | Attach to groups; multiple schedules per group allowed |
 | Locking | Atomic UPDATE on `module.running`; PID + time-based stale clearing |
 | Credentials | Env var references (`$DB2PW`); resolved at runtime |
@@ -631,6 +694,8 @@ behavior.
 | Linked servers | Not first-class; only affect FROM-clause syntax at author time; not persisted on module |
 | API style | REST, GET for reads, POST for writes, no PUT/DELETE |
 | Run model | Async — POST /run returns run_id immediately; watch via polling or SSE stream |
-| Live output | Server-Sent Events (SSE) — plain HTTP, curl-friendly, browser-native |
+| Live output | HTMX polling every 2–3 s (SSE deferred — polling sufficient for current scale) |
 | Auth | HTTP Basic, single user, creds in settings table |
-| TUI ↔ backend | TUI is an HTTP client; never touches SQLite directly |
+| TUI | Deferred — web UI built first; TUI remains valid future work as an API client |
+| Scheduler | croniter for cron evaluation; last_fired_at persisted before run to prevent double-fire |
+| Service account | Runs as pipekit system user; venv + db owned by that user; secrets 0640 group pipekit |

SPEC_v1_archive.md

@@ -1,485 +0,0 @@
-# Pipekit — ETL Tool Specification
-
-## Overview
-
-A lightweight, JDBC-based ETL tool for syncing tables between source systems and a PostgreSQL destination (or other JDBC destinations). Config-driven, no boilerplate scripts. Managed via TUI, API, or future web UI.
-
-## Architecture
-
-```
-jrunner (JDBC transfer engine — existing Java app)
-    ^
-engine (Python — orchestrates jrunner, manages staging, merge, DDL, logging)
-    ^
-API (FastAPI — REST interface, Basic Auth)
-    ^
-TUI / Web UI / external callers
-```
-
-## Core Concepts
-
-| Concept        | Description                                                                 |
-|----------------|-----------------------------------------------------------------------------|
-| **Connection** | A JDBC source or destination — URL, driver class, credentials               |
-| **Driver**     | A JDBC driver jar registered with the system                                |
-| **Module**     | A sync job — source query + destination table + merge strategy              |
-| **Hook**       | Post-sync SQL action run against the destination (e.g. refresh mat view)    |
-| **Group**      | An ordered list of modules that run together                                |
-| **Schedule**   | A cron expression tied to a group                                           |
-| **Run**        | A single execution — tracked with timing, row count, status, error, SQL     |
-
-## Bootstrap Config (only file on disk)
-
-```yaml
-# /opt/pipekit/config.yaml
-database: /opt/pipekit/pipekit.db    # SQLite — self-contained, no external DB required
-jrunner_path: /usr/local/bin/jrunner
-driver_dir: /opt/pipekit/drivers/
-api_port: 8100
-smtp:                                # optional, for failure notifications
-  host: smtp.example.com
-  port: 587
-  from: etl@example.com
-  to: admin@example.com
-```
-
-Everything else lives in SQLite (`pipekit.db`). No external database dependency for config — destinations can be PostgreSQL, SQL Server, or anything with a JDBC driver.
-
-## Column Identity Model
-
-A module's source query defines column mappings from source to destination. This is the central design constraint — every column has two identities:
-
-| Context | Name | Example | Where used |
-|---------|------|---------|------------|
-| **Source column** | The original column name in the source system | `DCORD#`, `DCODAT` | Source query SELECT, WHERE clauses against the source |
-| **Destination column** | The alias in the SELECT, which becomes the column name in staging and dest tables | `dcord`, `dcodat` | Staging table DDL, merge SQL, destination queries |
-
-### Rules
-
-1. The **source query** maps source → destination: `SELECT "DCORD#" AS dcord ...`
-2. **`merge_key`** references the **destination column name** — it's used in merge SQL that runs against PostgreSQL (e.g. `DELETE FROM dest WHERE dcord IN (SELECT dcord FROM staging)`)
-3. **`watermark_column`** references the **destination column name** — the engine looks up `MAX(watermark_column)` in the destination table, then must translate it back to the source column name to build the WHERE clause against the source
-4. The **watermark WHERE clause** must use the **source column name** — e.g. `WHERE "DCORD#" > 12345`, not `WHERE dcord > 12345` (the source system doesn't know the alias)
-5. The engine maintains a **column mapping** (alias → source expression) parsed from the source query to perform this translation
-
-### Column Mapping Derivation
-
-The source query is parsed to extract the mapping:
-
-```sql
-SELECT
-     "DCORD#"                AS dcord      -- source: "DCORD#",  dest: dcord
-    ,RTRIM(DCOTYP)           AS dcotyp     -- source: DCOTYP,    dest: dcotyp (trimmed)
-    ,DCODAT                  AS dcodat     -- source: DCODAT,    dest: dcodat
-FROM LGDAT.QCRH
-```
-
-From this, the engine derives:
-- `dcord` → `"DCORD#"` (used for WHERE clause on source)
-- `dcotyp` → `DCOTYP` (the unwrapped column, without RTRIM)
-- `dcodat` → `DCODAT`
-
-When building an incremental WHERE clause for watermark column `dcord`:
-1. Query dest: `SELECT MAX(dcord) FROM sync.qcrh` → `12345`
-2. Look up source expression for `dcord` → `"DCORD#"`
-3. Build: `WHERE "DCORD#" > 12345`
-
-### Special Character Handling
-
-Source columns with special characters (`#`, `@`, `$`, spaces) are:
-- **Quoted in the source query** using platform-appropriate syntax: `[DCORD#]` (SQL Server), `"DCORD#"` (DB2/PostgreSQL)
-- **Aliased to safe names** that are valid unquoted PostgreSQL identifiers: `dcord`, `company_name`
-- The alias generation (`_safe_alias`) strips special characters, lowercases, and replaces non-alphanumeric chars with underscores
-
-## Database Schema
-
-All tables in SQLite (`pipekit.db`). Same schema works if migrated to PostgreSQL later.
-
-### connection
-
-| Column           | Type    | Description                                      |
-|------------------|---------|--------------------------------------------------|
-| id               | integer PK | Auto-increment                                |
-| name             | text    | Human-readable label                              |
-| jdbc_url         | text    | JDBC connection string                            |
-| driver_id        | integer | FK to driver                                      |
-| username         | text    |                                                   |
-| password         | text    | Env var reference (e.g. `$DB2PW`) resolved at runtime |
-| supports_deletes | boolean | Whether destination supports DELETE/UPDATE        |
-| created_at       | text    | ISO datetime                                      |
-| updated_at       | text    | ISO datetime                                      |
-
-### driver
-
-| Column       | Type    | Description                                      |
-|--------------|---------|--------------------------------------------------|
-| id           | integer PK | Auto-increment                                |
-| name         | text    | e.g. "SQL Server", "AS/400 DB2"                  |
-| jar_file     | text    | Filename in driver_dir                            |
-| class_name   | text    | JDBC driver class                                 |
-| url_template | text    | e.g. `jdbc:sqlserver://{host};databaseName={db}`  |
-
-### module
-
-| Column              | Type    | Description                                     |
-|---------------------|---------|-------------------------------------------------|
-| id                  | integer PK | Auto-increment                               |
-| name                | text    | Module identifier (unique)                       |
-| source_connection_id| integer | FK to connection                                 |
-| dest_connection_id  | integer | FK to connection                                 |
-| dest_table          | text    | Fully qualified destination (schema.table)       |
-| source_query        | text    | The SELECT query to run against the source       |
-| merge_strategy      | text    | `full`, `incremental`, `append`, `upsert`        |
-| merge_key           | text    | **Destination** column name for merge operations |
-| watermark_column    | text    | **Destination** column name for incremental watermark. If null, falls back to merge_key |
-| key_sync            | boolean | After incremental, reconcile keys and delete orphans |
-| key_sync_query      | text    | Optional custom query to fetch source keys       |
-| full_refresh_cron   | text    | Optional cron for periodic full refresh          |
-| enabled             | boolean | Whether the module is active                     |
-| running             | boolean | Lock flag — set during execution                 |
-| created_at          | text    | ISO datetime                                     |
-| updated_at          | text    | ISO datetime                                     |
-
-### hook
-
-| Column    | Type    | Description                                          |
-|-----------|---------|------------------------------------------------------|
-| id        | integer PK | Auto-increment                                    |
-| module_id | integer | FK to module (CASCADE delete)                        |
-| run_order | integer | Execution order                                       |
-| sql       | text    | SQL to execute against destination                   |
-| run_on    | text    | `success`, `failure`, `always`                        |
-
-### grp (group)
-
-| Column | Type    | Description        |
-|--------|---------|--------------------|
-| id     | integer PK | Auto-increment  |
-| name   | text    | e.g. "pricing"     |
-
-### group_member
-
-| Column    | Type    | Description                |
-|-----------|---------|----------------------------|
-| id        | integer PK | Auto-increment          |
-| group_id  | integer | FK to grp (CASCADE)        |
-| module_id | integer | FK to module (CASCADE)     |
-| run_order | integer | Execution order in group   |
-
-### schedule
-
-| Column    | Type    | Description                         |
-|-----------|---------|-------------------------------------|
-| id        | integer PK | Auto-increment                   |
-| group_id  | integer | FK to grp (CASCADE)                 |
-| cron_expr | text    | Cron expression (e.g. `0 2 * * *`) |
-| enabled   | boolean |                                     |
-
-### run_log
-
-| Column       | Type    | Description                                              |
-|--------------|---------|----------------------------------------------------------|
-| id           | integer PK | Auto-increment                                        |
-| module_id    | integer | FK to module                                             |
-| group_id     | integer | FK to grp (nullable — null if run manually)              |
-| started_at   | text    | ISO datetime                                             |
-| finished_at  | text    | ISO datetime                                             |
-| row_count    | integer |                                                          |
-| status       | text    | `running`, `success`, `error`, `cancelled`               |
-| error        | text    | Error message if failed                                  |
-| source_query | text    | The exact source SQL executed (with resolved WHERE)      |
-| merge_sql    | text    | The exact merge SQL executed against destination         |
-
-### module_history
-
-| Column      | Type    | Description                         |
-|-------------|---------|-------------------------------------|
-| id          | integer PK | Auto-increment                   |
-| module_id   | integer | FK to module (CASCADE)               |
-| source_query| text    | Previous query text                  |
-| changed_at  | text    | ISO datetime                         |
-
-### settings
-
-| Column | Type | Description                   |
-|--------|------|-------------------------------|
-| key    | text PK | e.g. `smtp_host`           |
-| value  | text |                               |
-
-## Merge Strategies
-
-| Strategy      | Behavior                                                              |
-|---------------|-----------------------------------------------------------------------|
-| `full`        | Transfer all rows to staging, TRUNCATE dest, INSERT from staging      |
-| `incremental` | Query dest for MAX(watermark), build WHERE clause using source column name, transfer delta, DELETE matching rows by merge_key, INSERT from staging |
-| `append`      | Transfer, INSERT into dest, no deletes                                |
-| `upsert`      | Transfer, INSERT ON CONFLICT(merge_key) DO UPDATE                     |
-
-### Incremental Sync Flow (detailed)
-
-1. Resolve watermark column: use `watermark_column`, fall back to `merge_key`
-2. Query destination: `SELECT MAX({watermark_col}) FROM {dest_table}`
-3. Parse the result — handle NULL (empty table), numeric values, date/text values
-4. Parse source query to find the source expression for the watermark alias
-5. Build WHERE clause using the **source expression** (not the alias):
-   - Numeric watermark: `WHERE "DCORD#" > 12345`
-   - Date/text watermark: `WHERE DEX_ROW_TS >= '2026-04-01 00:00:00'`
-6. Append WHERE clause to the base source query
-7. Transfer delta rows to staging
-8. Merge: DELETE from dest WHERE merge_key IN (SELECT merge_key FROM staging), then INSERT
-9. Run hooks
-
-**NULL watermark handling**: If `MAX(watermark)` returns NULL (empty dest table or psql null representation like `∅`), skip the WHERE clause entirely — pull all rows.
-
-### Handling Source Deletes
-
-Incremental strategies only detect new/changed rows — not rows deleted from the source. Two mechanisms address this:
-
-**1. Key reconciliation (`key_sync`)** — optional per module. After the incremental load, pull all primary key values from the source (lightweight query), compare against destination, and delete any destination rows whose key is not in the source.
-
-**2. Periodic full refresh (`full_refresh_cron`)** — optional per module. A cron expression that triggers a full refresh on a different cadence than the incremental schedule.
-
-### Destination-Aware Merge
-
-The engine checks `connection.supports_deletes`:
-- If true: DELETE + INSERT merge works normally
-- If false: incremental/upsert fall back to insert-only, relying on the destination's dedup engine (e.g. ClickHouse ReplacingMergeTree)
-
-## Staging Table Management
-
-- Named `pipekit_staging.{module_name}` (persistent across runs)
-- If table exists: TRUNCATE before transfer
-- If table doesn't exist: probe source for column metadata (0-row jrunner transfer), create table with mapped PostgreSQL types
-- Probe always uses the **base source query** (no WHERE clause) to avoid comment/subquery issues
-- Left in place after runs (success or failure) for debugging
-- Schemas `pipekit_staging` and destination schema auto-created if missing
-
-## Source Introspection
-
-The engine can browse source systems via jrunner query mode against INFORMATION_SCHEMA (or equivalent):
-
-- **Table browsing**: list tables/views filtered by schema
-- **Column metadata**: column names, types, positions
-- **Linked server support** (SQL Server): query tables on linked servers via 4-part naming
-- **Cross-database** (SQL Server): specify a different database than the connection default
-- **Auto-propose**: given a source table, generate complete module config:
-  - SELECT query with RTRIM on text columns, safe aliases for special characters
-  - Platform-aware identifier quoting (`[brackets]` for SQL Server, `"double quotes"` for DB2/others)
-  - Destination DDL with mapped PostgreSQL types
-  - Suggested merge strategy, key, and watermark column
-
-### Source Type Detection
-
-Detected from JDBC URL: `as400`, `sqlserver`, `postgresql`, `clickhouse`, `mysql`
-
-### Type Mapping (source → PostgreSQL)
-
-varchar/char/nvarchar/nchar/text → text, int/integer → integer, bigint → bigint, decimal/numeric → numeric, float/double → double precision, date → date, datetime/timestamp → timestamp, bit → boolean, binary/varbinary → bytea, uniqueidentifier → uuid
-
-## API Endpoints
-
-```
-# Auth: HTTP Basic Auth on all endpoints
-
-# Connections
-GET    /connections
-POST   /connections
-GET    /connections/{id}
-PUT    /connections/{id}
-DELETE /connections/{id}
-POST   /connections/{id}/test
-GET    /connections/{id}/tables?schema=
-GET    /connections/{id}/tables/{schema}.{table}/columns
-GET    /connections/{id}/tables/{schema}.{table}/propose
-
-# Modules
-GET    /modules
-POST   /modules
-GET    /modules/{id}
-PUT    /modules/{id}
-DELETE /modules/{id}
-GET    /modules/{id}/preview
-GET    /modules/{id}/dest-columns
-POST   /modules/{id}/run
-POST   /modules/{id}/run/stream
-GET    /modules/{id}/history
-
-# Hooks
-GET    /modules/{module_id}/hooks
-POST   /hooks
-DELETE /hooks/{id}
-
-# Groups
-GET    /groups
-POST   /groups
-GET    /groups/{id}
-DELETE /groups/{id}
-POST   /groups/{id}/members
-DELETE /groups/members/{id}
-POST   /groups/{id}/run
-
-# Runs
-GET    /runs
-GET    /runs/{id}
-
-# Drivers
-GET    /drivers
-POST   /drivers
-DELETE /drivers/{id}
-
-# Schedules
-GET    /schedules
-POST   /schedules
-PUT    /schedules/{id}
-DELETE /schedules/{id}
-```
-
-## TUI
-
-### Main Screen
-
-Module tree grouped by source connection. Icons: `✔` enabled, `○` disabled, `▶` running.
-
-| Key | Action |
-|-----|--------|
-| `i` | Inspect module |
-| `r` | Run selected module |
-| `l` | Module run history |
-| `L` | Global run log (all modules) |
-| `n` | New module wizard |
-| `c` | Manage connections |
-| `/` | Search modules |
-| `j/k` | Navigate |
-| `g/G` | Top/bottom |
-| `F5` | Refresh |
-| `q` | Quit |
-
-### Module Detail Screen (i)
-
-Top section: module info (strategy, merge key, watermark, dest table, staging table, enabled, updated).
-
-Middle section: column table showing source column, destination alias, and whether RTRIM is applied.
-
-Bottom: footer with keybindings. **No SQL visible by default** — all SQL opens in `$EDITOR` (read-only) via keybindings:
-
-| Key | Opens in editor |
-|-----|-----------------|
-| `q` | Next source SQL — the resolved query that would execute on next run (with WHERE clause) |
-| `m` | Merge SQL — the staging-to-dest merge statements |
-| `h` | Post-merge hooks |
-| `b` | Base query template — the stored SELECT before watermark WHERE is appended |
-| `e` | Edit base query (writable) |
-| `s` | Module settings (opens edit screen) |
-| `r` | Run sync |
-| `l` | Run history |
-
-### Module Settings Screen (s)
-
-Full edit form matching the new module wizard layout:
-- Module name, source/dest connections, dest table
-- Merge strategy (radio buttons)
-- Merge key and watermark column (searchable dropdowns populated from source query aliases = destination column names)
-- Enabled toggle
-
-Source query is **not** on this screen — use `e` from the detail screen to edit it in `$EDITOR`.
-
-### New Module Wizard (n)
-
-- Source/destination connection selection
-- Table browser: linked server, database, schema filter fields + Load button
-- Real-time search/filter over loaded tables (DataTable)
-- Auto-propose on table selection (generates query, DDL, strategy suggestions)
-- Merge strategy, key, watermark, dest table fields
-
-### History Screens (l, L)
-
-Run table with status, rows, timing, error. Below: **separate** panels for source query and merge SQL (not combined). Error shown as red text. `v` opens selected run's SQL in editor. `esc` closes.
-
-### Run Screen (r)
-
-Streaming jrunner output via SSE. Shows real-time transfer progress.
-
-## Concurrency Control
-
-Each module has a `running` flag. Before starting a sync:
-1. Check if module is already running — reject if so
-2. Set `running = true`
-3. Execute sync
-4. Set `running = false` on success or failure
-
-## Error Handling
-
-- On module failure: log error to run_log, stop group execution
-- No automatic retries
-- Staging tables preserved for debugging
-- Generated SQL logged to run_log for post-mortem analysis
-
-## Security
-
-- API: HTTP Basic Auth (username/password stored in settings table)
-- Connection passwords: stored as env var references (e.g. `$DB2PW`) resolved at runtime
-
-## Deployment
-
-- Single directory install (`/opt/pipekit/`)
-- Bootstrap config file (`config.yaml`)
-- SQLite database (`pipekit.db`) — created on first run
-- JDBC drivers directory
-- Python dependencies via pip/venv
-- Portable: copy the directory and you've moved the whole install
-
-## Directory Structure
-
-```
-/opt/pipekit/
-  config.yaml           # bootstrap config (only file-based config)
-  pipekit.db            # SQLite — all config, queries, run history
-  drivers/              # JDBC .jar files
-  engine/
-    db.py               # SQLite schema + CRUD operations
-    runner.py           # Sync orchestration (staging, transfer, merge, hooks)
-    introspect.py       # Source browsing, query generation, type mapping
-  api/
-    main.py             # FastAPI app
-  tui/
-    app.py              # Textual TUI
-    client.py           # HTTP client for API
-  requirements.txt
-```
-
-## jrunner Fixes
-
-- **NVARCHAR/NCHAR/NTEXT/NCLOB quoting** — added case labels to jrunner's INSERT builder type switch so Unicode string types get quoted correctly.
-
-## Migration Path from Current Setup
-
-1. Create connections for s7830956, usmidsql01, gpserver, localhost PostgreSQL
-2. Import existing modules — parse shell scripts to extract query, dest table, strategy
-3. Import orchestrators as groups
-4. Set up schedules to match current crontab
-5. Verify runs produce same results
-6. Decommission shell scripts and cron entries
-
-## TODO
-
-- [ ] **Implement column mapping for watermark WHERE clause** — parse source query to build alias → source expression map, use source expression (not alias) in incremental WHERE clauses
-- [ ] **Cancel running sync** — track PID, add cancel endpoint + TUI binding
-- [ ] **Scheduler** — background thread in the API process evaluating cron expressions every minute
-- [ ] **Email notifications** — SMTP on failure
-- [ ] **Upsert + incremental combo** — pull only changed rows, then INSERT ON CONFLICT UPDATE
-- [ ] **Module history — full audit** — expand module_history to track all field changes, store as JSON diff
-
-### Resolved
-
-- **Persistent staging tables** — `pipekit_staging.{name}`, truncated before each run, left in place after
-- **Global run log in TUI** — `L` from main screen
-- **Connection pooling** — not needed at current scale
-- **Scheduler location** — built into the API process (background thread)
-- **module_history scope** — track all field changes
-- **`timestamp_column` renamed to `watermark_column`** — reflects actual purpose (any monotonic value, not just timestamps)
-
-## Known Issues
-
-- **Watermark WHERE clause uses alias instead of source column name** — `WHERE dcord > 12345` should be `WHERE "DCORD#" > 12345`. Blocked on implementing the column mapping (top TODO item).
-- **psql null display** — `MAX()` on empty table can render as `∅` depending on locale. The null check must handle this.
-- **Merge key stored as `dcord#` vs alias `dcord`** — historical data may have source column names stored where alias was intended. Merge key should always be the destination column name.

config.yaml

@@ -1,4 +1,5 @@
 database: /opt/pipekit/pipekit.db
+api_auth_enabled: true
 jrunner_path: /usr/local/bin/jrunner
 driver_dir: /opt/pipekit/drivers/
 api_host: 0.0.0.0

deploy.sh

@@ -1,17 +1,23 @@
 #!/usr/bin/env bash
-# Pipekit deployment — idempotent. Re-run any time.
+# Pipekit deployment — idempotent. Re-run after any code update.
 #
-# Steps:
-#   1. Check prerequisites (python3, jrunner on PATH)
-#   2. Create Python venv at $REPO/.venv and install requirements
-#   3. Install launcher at /usr/local/bin/pipekit (wraps the venv python)
-#   4. Ensure /etc/pipekit/secrets.env exists (mode 0600, placeholder body)
-#   5. Run `pipekit init` to create/upgrade the SQLite schema
-#   6. Register driver rows for every JDBC jar shipped with jrunner
+# What it does:
+#   1. Creates the 'pipekit' system user and group (if absent)
+#   2. Adds the invoking user to the 'pipekit' group
+#   3. Sets ownership of /opt/pipekit to pipekit:pipekit (group-writable)
+#   4. Creates Python venv (as pipekit) and installs requirements
+#   5. Installs /usr/local/bin/pipekit launcher
+#   6. Creates /etc/pipekit/secrets.env (mode 0640, group pipekit)
+#   7. Runs 'pipekit init' to create/upgrade the SQLite schema
+#   8. Registers JDBC driver rows for every jar shipped with jrunner
+#   9. Installs and enables the systemd unit (does not start it)
 #
-# After running:
-#   - Set DB passwords with: sudo pipekit secrets set <KEY>
-#   - See systemd/pipekit.service for a unit file template
+# Usage:
+#   ./deploy.sh          # re-execs itself with sudo if needed
+#
+# After deploy:
+#   sudo pipekit secrets set KEY VALUE   # add connection passwords
+#   sudo systemctl start pipekit
 
 set -euo pipefail
 
@@ -20,78 +26,213 @@ VENV_DIR="$REPO_DIR/.venv"
 LAUNCHER="/usr/local/bin/pipekit"
 CONFIG_DIR="/etc/pipekit"
 SECRETS_FILE="$CONFIG_DIR/secrets.env"
+SERVICE_NAME="pipekit"
+UNIT_SRC="$REPO_DIR/systemd/pipekit.service"
+UNIT_DST="/etc/systemd/system/pipekit.service"
 
+# Capture the invoking user before re-execing as root
+INVOKING_USER="${SUDO_USER:-${USER:-}}"
+
+# Re-exec as root if needed
 if [ "$EUID" -ne 0 ]; then
-    exec sudo -H -E "$0" "$@"
+    exec sudo -E "$0" "$@"
 fi
 
+step() { echo ""; echo "── $* ──"; }
+
 echo "== pipekit deploy =="
 echo "repo:    $REPO_DIR"
 echo "venv:    $VENV_DIR"
 echo "secrets: $SECRETS_FILE"
-echo ""
+echo "user:    $SERVICE_NAME (invoking user: ${INVOKING_USER:-unknown})"
 
+# ── Prerequisites ─────────────────────────────────────────────────────────────
+step "Checking prerequisites"
 command -v python3 >/dev/null || { echo "ERROR: python3 not on PATH"; exit 1; }
+echo "  python3: $(python3 --version)"
 command -v jrunner >/dev/null || { echo "ERROR: jrunner not on PATH — install /opt/jrunner first"; exit 1; }
+echo "  jrunner: $(command -v jrunner)"
+
+# ── 1. System user ────────────────────────────────────────────────────────────
+step "System user"
+if ! id -u "$SERVICE_NAME" >/dev/null 2>&1; then
+    echo "  Creating system user: $SERVICE_NAME"
+    useradd --system --no-create-home --home-dir /nonexistent \
+            --shell /usr/sbin/nologin "$SERVICE_NAME"
+    echo "  Done."
+else
+    echo "  User '$SERVICE_NAME' already exists (uid=$(id -u "$SERVICE_NAME"))."
+fi
+
+# ── 2. Add invoking user to pipekit group ─────────────────────────────────────
+step "Group membership"
+if [ -n "$INVOKING_USER" ] && [ "$INVOKING_USER" != "$SERVICE_NAME" ]; then
+    if id -nG "$INVOKING_USER" | grep -qw "$SERVICE_NAME"; then
+        echo "  $INVOKING_USER is already in the '$SERVICE_NAME' group."
+    else
+        echo "  Adding $INVOKING_USER to the '$SERVICE_NAME' group."
+        usermod -aG "$SERVICE_NAME" "$INVOKING_USER"
+        echo "  Done. Log out and back in for this to take effect."
+    fi
+else
+    echo "  Skipping — could not determine invoking user."
+fi
+
+# ── 3. Ownership ──────────────────────────────────────────────────────────────
+step "File ownership and permissions"
+# Source code stays owned by the invoking user.
+# pipekit needs to own pipekit.db and write to the repo directory
+# (SQLite creates journal files alongside the db file).
+DB_FILE="$REPO_DIR/pipekit.db"
+if [ -f "$DB_FILE" ]; then
+    echo "  $DB_FILE → $SERVICE_NAME:$SERVICE_NAME"
+    chown "$SERVICE_NAME:$SERVICE_NAME" "$DB_FILE"
+else
+    echo "  $DB_FILE not yet created (pipekit init will create it as $SERVICE_NAME)"
+fi
+echo "  $REPO_DIR (directory only) → group $SERVICE_NAME, group-writable"
+chgrp "$SERVICE_NAME" "$REPO_DIR"
+chmod g+w "$REPO_DIR"
+echo "  $VENV_DIR → $SERVICE_NAME (created/managed below)"
+echo "  Done."
+
+# ── 4. Venv + deps ────────────────────────────────────────────────────────────
+step "Python venv"
+if [ -d "$VENV_DIR" ] && [ "$(stat -c '%U' "$VENV_DIR")" != "$SERVICE_NAME" ]; then
+    echo "  Venv exists but is not owned by $SERVICE_NAME — recreating."
+    rm -rf "$VENV_DIR"
+fi
 
 if [ ! -d "$VENV_DIR" ]; then
-    echo "Creating venv at $VENV_DIR"
-    python3 -m venv "$VENV_DIR"
+    echo "  Creating venv at $VENV_DIR"
+    mkdir -p "$VENV_DIR"
+    chown "$SERVICE_NAME:$SERVICE_NAME" "$VENV_DIR"
+    sudo -u "$SERVICE_NAME" HOME=/nonexistent python3 -m venv "$VENV_DIR"
+else
+    echo "  Venv already exists at $VENV_DIR."
 fi
-"$VENV_DIR/bin/pip" install --quiet --upgrade pip
-"$VENV_DIR/bin/pip" install --quiet -r "$REPO_DIR/requirements.txt"
-echo "Python deps installed."
 
-# The in-repo bin/pipekit auto-detects .venv at runtime; no rewrite needed.
+echo "  Upgrading pip"
+sudo -u "$SERVICE_NAME" HOME=/nonexistent \
+    "$VENV_DIR/bin/pip" install --quiet --no-cache-dir --upgrade pip
+
+echo "  Installing requirements"
+sudo -u "$SERVICE_NAME" HOME=/nonexistent \
+    "$VENV_DIR/bin/pip" install --quiet --no-cache-dir -r "$REPO_DIR/requirements.txt"
+echo "  Done."
+
+# ── 5. Launcher ───────────────────────────────────────────────────────────────
+step "Launcher"
 chmod +x "$REPO_DIR/bin/pipekit"
 ln -sf "$REPO_DIR/bin/pipekit" "$LAUNCHER"
-echo "Launcher: $LAUNCHER -> $REPO_DIR/bin/pipekit"
+echo "  $LAUNCHER -> $REPO_DIR/bin/pipekit"
 
-install -d -m 0755 "$CONFIG_DIR"
+# ── 6. Secrets file ───────────────────────────────────────────────────────────
+step "Secrets file"
+install -d -m 0775 "$CONFIG_DIR"
+chown "root:$SERVICE_NAME" "$CONFIG_DIR"
 if [ ! -f "$SECRETS_FILE" ]; then
-    install -m 0600 /dev/null "$SECRETS_FILE"
+    echo "  Creating $SECRETS_FILE (mode 0640, group $SERVICE_NAME)"
+    install -m 0640 /dev/null "$SECRETS_FILE"
+    chown "$SERVICE_NAME:$SERVICE_NAME" "$SECRETS_FILE"
     cat > "$SECRETS_FILE" <<'EOF'
-# pipekit secrets — sourced by the service process (EnvironmentFile=)
-# or by the shell before `pipekit serve`. One KEY=VALUE per line.
-# Connection rows reference these as $KEY (e.g. password: "$DB2PW").
-#
-# This file must stay mode 0600 and out of version control.
-# Use `sudo pipekit secrets set <KEY>` to add entries safely.
+# pipekit secrets — loaded by the systemd unit as EnvironmentFile.
+# Connection passwords are stored as $KEY references in the DB.
+# Add entries with:  sudo pipekit secrets set KEY VALUE
 EOF
-    chmod 0600 "$SECRETS_FILE"
-    echo "Created $SECRETS_FILE"
 else
-    echo "Keeping existing $SECRETS_FILE"
+    echo "  $SECRETS_FILE already exists — keeping contents."
+    chown "$SERVICE_NAME:$SERVICE_NAME" "$SECRETS_FILE"
+    chmod 0640 "$SECRETS_FILE"
+    echo "  Permissions ensured: 0640 owner $SERVICE_NAME."
 fi
 
-"$LAUNCHER" init
+# ── 7. Schema init ────────────────────────────────────────────────────────────
+step "Database schema"
+DB_FILE="$REPO_DIR/pipekit.db"
+if [ ! -f "$DB_FILE" ]; then
+    echo "  Creating $DB_FILE owned by $SERVICE_NAME"
+    touch "$DB_FILE"
+    chown "$SERVICE_NAME:$SERVICE_NAME" "$DB_FILE"
+fi
+echo "  Running pipekit init"
+sudo -u "$SERVICE_NAME" HOME=/nonexistent "$LAUNCHER" init
+echo "  Done."
 
-# Register drivers for each JDBC jar jrunner ships with.
+# ── 8. Driver registration ────────────────────────────────────────────────────
+step "JDBC driver registration"
 JR_LIB="$(dirname "$(readlink -f "$(command -v jrunner)")")/../lib"
+echo "  jrunner lib dir: $JR_LIB"
 register_jar() {
     local kind="$1" pattern="$2"
     local jar
     jar="$(find "$JR_LIB" -maxdepth 1 -name "$pattern" 2>/dev/null | head -1)"
     if [ -n "$jar" ]; then
-        "$LAUNCHER" drivers register "$kind" --jar "$jar"
+        echo "  Registering $kind: $(basename "$jar")"
+        sudo -u "$SERVICE_NAME" HOME=/nonexistent "$LAUNCHER" drivers register "$kind" --jar "$jar"
     else
-        echo "  (no $pattern in $JR_LIB — skipping $kind)"
+        echo "  No $pattern found in $JR_LIB — skipping $kind"
     fi
 }
 register_jar db2   "jt400-*.jar"
 register_jar pg    "postgresql-*.jar"
 register_jar mssql "mssql-jdbc-*.jar"
 
+# ── 9. Systemd unit ───────────────────────────────────────────────────────────
+step "Systemd unit"
+if [ ! -f "$UNIT_SRC" ]; then
+    echo "  WARNING: $UNIT_SRC not found — skipping"
+else
+    # Detect JAVA_HOME — check PATH first, then search each location individually
+    JAVA_BIN="$(command -v java 2>/dev/null || true)"
+    if [ -z "$JAVA_BIN" ]; then
+        for _dir in /opt /usr/lib/jvm /usr/local/lib/jvm; do
+            [ -d "$_dir" ] || continue
+            JAVA_BIN="$(find "$_dir" -maxdepth 3 -name java -type f 2>/dev/null | head -1 || true)"
+            [ -n "$JAVA_BIN" ] && break
+        done
+    fi
+    if [ -z "$JAVA_BIN" ]; then
+        echo "  WARNING: java not found — JAVA_HOME will not be set in unit"
+        JAVA_HOME_DETECTED=""
+    else
+        JAVA_HOME_DETECTED="$(dirname "$(dirname "$(readlink -f "$JAVA_BIN")")")"
+        echo "  Detected JAVA_HOME: $JAVA_HOME_DETECTED"
+    fi
+
+    echo "  Installing $UNIT_DST"
+    if [ -n "$JAVA_HOME_DETECTED" ]; then
+        # Inject Environment lines after WorkingDirectory= using printf to handle newlines portably
+        ENV_BLOCK="$(printf 'Environment=JAVA_HOME=%s\nEnvironment=PATH=%s/bin:/usr/local/bin:/usr/bin:/bin' "$JAVA_HOME_DETECTED" "$JAVA_HOME_DETECTED")"
+        awk -v env="$ENV_BLOCK" '/^WorkingDirectory=/{print; print env; next}1' "$UNIT_SRC" > "$UNIT_DST"
+    else
+        cp "$UNIT_SRC" "$UNIT_DST"
+    fi
+    echo "  Running systemctl daemon-reload"
+    systemctl daemon-reload
+    echo "  Enabling $SERVICE_NAME service"
+    systemctl enable "$SERVICE_NAME"
+    echo "  Done."
+fi
+
 echo ""
-echo "pipekit deployed."
+echo "════════════════════════════════════"
+echo "  pipekit deployed successfully."
+echo "════════════════════════════════════"
 echo ""
 echo "Next steps:"
-echo "  1. Set passwords:  sudo pipekit secrets set DB2PW"
-echo "                     sudo pipekit secrets set PGPW"
-echo "  2. Start the server manually:"
-echo "       set -a; source $SECRETS_FILE; set +a"
-echo "       pipekit serve"
-echo "  3. Or install the systemd unit:"
-echo "       sudo cp $REPO_DIR/systemd/pipekit.service /etc/systemd/system/"
-echo "       sudo systemctl daemon-reload"
-echo "       sudo systemctl enable --now pipekit"
+echo "  1. Add connection passwords:"
+echo "       sudo pipekit secrets set DB2PW <value>"
+echo "       sudo pipekit secrets set PGPW  <value>"
+echo "  2. Start the service:"
+echo "       sudo systemctl start pipekit"
+echo "  3. Check it:"
+echo "       sudo systemctl status pipekit"
+echo "       journalctl -u pipekit -f"
+if [ -n "$INVOKING_USER" ]; then
+    if ! id -nG "$INVOKING_USER" 2>/dev/null | grep -qw "$SERVICE_NAME"; then
+        echo ""
+        echo "  NOTE: $INVOKING_USER was added to the '$SERVICE_NAME' group."
+        echo "        Log out and back in for write access to take effect."
+    fi
+fi

pipekit/api/app.py

@@ -7,6 +7,8 @@ content-negotiation complexity and keeps the API curl-testable.
 
 from __future__ import annotations
 
+from contextlib import asynccontextmanager
+
 from fastapi import FastAPI
 
 from .. import __version__, db, jrunner
@@ -14,8 +16,15 @@ from ..web import mount_web
 from .routes import connections, introspect, modules, runs, system
 
 
+@asynccontextmanager
+async def _lifespan(app: FastAPI):
+    from ..scheduler import start_scheduler
+    start_scheduler()
+    yield
+
+
 def create_app() -> FastAPI:
-    app = FastAPI(title="Pipekit", version=__version__)
+    app = FastAPI(title="Pipekit", version=__version__, lifespan=_lifespan)
     app.include_router(system.router)
     app.include_router(connections.router, prefix="/api")
     app.include_router(introspect.router, prefix="/api")

pipekit/cli.py

@@ -59,13 +59,6 @@ def cmd_drivers_register(args) -> int:
         print(f"error: {e}")
         return 1
 
-    existing = [d for d in repo.list_drivers() if d["kind"] == args.kind]
-    if existing and not args.force:
-        print(f"driver kind {args.kind!r} already registered as "
-              f"{existing[0]['name']!r} (id={existing[0]['id']}). "
-              f"Use --force to add a second row.")
-        return 0
-
     class_name = args.class_name or _DEFAULT_JDBC_CLASSES.get(args.kind)
     if not class_name:
         print(f"error: no built-in JDBC class for kind {args.kind!r}; "
@@ -78,6 +71,16 @@ def cmd_drivers_register(args) -> int:
               f"(registering anyway)")
 
     name = args.name or f"{args.kind}-jdbc"
+    # Match by kind first (the meaningful key), fall back to name.
+    existing_by_kind = [d for d in repo.list_drivers() if d["kind"] == args.kind]
+    existing = existing_by_kind[0] if existing_by_kind else repo.get_driver_by_name(name)
+    if existing:
+        row = repo.update_driver(existing["id"], jar_file=args.jar,
+                                 class_name=class_name,
+                                 url_template=args.url_template)
+        print(f"updated driver id={row['id']} name={row['name']!r} "
+              f"kind={row['kind']!r}")
+    else:
         row = repo.create_driver(
             name=name, kind=args.kind, jar_file=args.jar,
             class_name=class_name, url_template=args.url_template,
@@ -218,7 +221,13 @@ def cmd_secrets_set(args) -> int:
     tmp = path + ".tmp"
     with open(tmp, "w") as f:
         f.writelines(lines)
-    os.chmod(tmp, stat.S_IRUSR | stat.S_IWUSR)  # 0600
+    # Preserve existing permissions/ownership if the file already exists;
+    # otherwise default to 0640 so the pipekit service user can read it.
+    if os.path.exists(path):
+        st = os.stat(path)
+        os.chmod(tmp, stat.S_IMODE(st.st_mode))
+    else:
+        os.chmod(tmp, 0o640)
     os.replace(tmp, path)
     print(f"{'updated' if replaced else 'added'} {args.key} in {path}")
     return 0
@@ -300,8 +309,6 @@ def main(argv: list[str] | None = None) -> int:
                             help="JDBC Driver class (default: built-in per kind)")
     p_drv_reg.add_argument("--url-template",
                             help="optional JDBC URL template for the wizard")
-    p_drv_reg.add_argument("--force", action="store_true",
-                            help="register even if a row for this kind exists")
     p_drv_reg.set_defaults(func=cmd_drivers_register)
 
     p_run = sub.add_parser("run", help="run a module by name (synchronous)")

pipekit/db.py

@@ -38,6 +38,14 @@ def _apply_migrations(conn: sqlite3.Connection) -> None:
     if "dest_description" not in cols:
         conn.execute("ALTER TABLE module ADD COLUMN dest_description TEXT")
 
+    rl_cols = {r[1] for r in conn.execute("PRAGMA table_info(run_log)")}
+    if "live_log" not in rl_cols:
+        conn.execute("ALTER TABLE run_log ADD COLUMN live_log TEXT")
+
+    sc_cols = {r[1] for r in conn.execute("PRAGMA table_info(schedule)")}
+    if "last_fired_at" not in sc_cols:
+        conn.execute("ALTER TABLE schedule ADD COLUMN last_fired_at TEXT")
+
 
 @contextmanager
 def connect(db_path: Path | None = None):

pipekit/drivers/base.py

@@ -66,11 +66,16 @@ class RemoteColumn:
 # identifiers; reject everything else before it reaches a query.
 # ---------------------------------------------------------------------------
 
-_SAFE_IDENT = re.compile(r"^[A-Za-z_][A-Za-z0-9_$#]*$")
+_SAFE_IDENT = re.compile(r"^[A-Za-z_][A-Za-z0-9_$#.]*$")
 
 
 def validate_identifier(value: str, field_name: str = "identifier") -> str:
-    if not isinstance(value, str) or not _SAFE_IDENT.match(value):
+    if not isinstance(value, str):
+        raise ValueError(f"invalid {field_name}: {value!r}")
+    # Strip surrounding double quotes — caller should pass the bare name
+    if value.startswith('"') and value.endswith('"') and len(value) > 2:
+        value = value[1:-1]
+    if not _SAFE_IDENT.match(value):
         raise ValueError(f"invalid {field_name}: {value!r}")
     return value
 
@@ -140,15 +145,42 @@ class Driver(abc.ABC):
 
     def build_create_table_sql(self, qualified_table: str,
                                 columns: list[dict]) -> str:
-        """Generate CREATE TABLE IF NOT EXISTS SQL for a destination table.
+        """Generate CREATE TABLE SQL for a destination table.
 
         ``columns`` is a list of ``{dest_name, dest_type}`` dicts.
-        Default implementation raises — only destination drivers (PG today)
-        need to implement it."""
+        Default raises — destination drivers must implement this."""
         raise NotImplementedError(
             f"driver {self.kind!r} does not implement build_create_table_sql "
             "(not a supported destination)")
 
+    supports_table_comments: ClassVar[bool] = True
+
+    def create_schema_sql(self, schema: str) -> str:
+        """DDL to create a schema if it doesn't already exist."""
+        return f"CREATE SCHEMA IF NOT EXISTS {self.quote_identifier(schema)};"
+
+    def drop_table_if_exists_sql(self, qualified_table: str) -> str:
+        """DDL to drop a table only if it exists."""
+        return f"DROP TABLE IF EXISTS {qualified_table};"
+
+    def create_like_table_sql(self, new_table: str, source_table: str) -> str:
+        """DDL to create new_table with the same columns as source_table."""
+        return f"CREATE TABLE {new_table} (LIKE {source_table} INCLUDING ALL);"
+
+    def check_dest_table(self, conn: dict, schema: str,
+                          table: str) -> "set[str] | None":
+        """Return lowercase column names of an existing dest table, or None
+        if the table does not exist. Default uses INFORMATION_SCHEMA (works
+        for PG and MSSQL); override for dialects that use a different catalog."""
+        sql = (
+            f"SELECT column_name FROM information_schema.columns "
+            f"WHERE table_schema='{schema}' AND table_name='{table}'"
+        )
+        r = self.query(conn, sql)
+        if not r.rows:
+            return None
+        return {row[0].strip().lower() for row in r.rows if row and row[0]}
+
     # ---- Shared helper ----
     def query(self, conn: dict, sql: str) -> jrunner.QueryResult:
         """Run `sql` in jrunner query mode against `conn`."""

pipekit/drivers/db2.py

@@ -133,6 +133,68 @@ class DB2Driver(Driver):
                     f"THEN NULL ELSE {col} END")
         return col
 
+    def drop_table_if_exists_sql(self, qualified_table: str) -> str:
+        # DB2 for i doesn't support DROP TABLE IF EXISTS; check catalog first.
+        parts = qualified_table.strip('"').split(".", 1)
+        if len(parts) == 2:
+            schema_esc = parts[0].strip('"').replace("'", "''").upper()
+            table_esc = parts[1].strip('"').replace("'", "''").upper()
+        else:
+            schema_esc = ""
+            table_esc = parts[0].strip('"').replace("'", "''").upper()
+        where = f"TABLE_NAME='{table_esc}'"
+        if schema_esc:
+            where += f" AND TABLE_SCHEMA='{schema_esc}'"
+        return (
+            f"BEGIN\n"
+            f"  IF EXISTS (SELECT 1 FROM QSYS2.SYSTABLES WHERE {where})\n"
+            f"  THEN DROP TABLE {qualified_table};\n"
+            f"  END IF;\n"
+            f"END"
+        )
+
+    def create_like_table_sql(self, new_table: str, source_table: str) -> str:
+        return f"CREATE TABLE {new_table} LIKE {source_table};"
+
+    def create_schema_sql(self, schema: str) -> str:
+        # DB2 for i: CREATE SCHEMA IF NOT EXISTS is available 7.4+; use the
+        # catalog check for broader compatibility.
+        name = schema.replace("'", "''")
+        q = self.quote_identifier(schema)
+        return (
+            f"BEGIN\n"
+            f"  IF NOT EXISTS (SELECT 1 FROM QSYS2.SYSSCHEMAS WHERE SCHEMA_NAME='{name}')\n"
+            f"  THEN CREATE SCHEMA {q};\n"
+            f"  END IF;\n"
+            f"END"
+        )
+
+    def build_create_table_sql(self, qualified_table: str,
+                                columns: list[dict]) -> str:
+        if not columns:
+            raise ValueError("no columns provided for CREATE TABLE")
+        lines = []
+        for c in columns:
+            name = c["dest_name"]
+            validate_identifier(name, "dest column name")
+            dtype = _normalize_db2_type((c.get("dest_type") or "").strip())
+            if not dtype:
+                raise ValueError(f"column {name!r} has no dest_type")
+            lines.append(f"    {self.quote_identifier(name)} {dtype}")
+        body = ",\n".join(lines)
+        return f"CREATE TABLE {qualified_table} (\n{body}\n);"
+
+    def check_dest_table(self, conn: dict, schema: str,
+                          table: str) -> "set[str] | None":
+        sql = (
+            f"SELECT COLUMN_NAME FROM QSYS2.SYSCOLUMNS "
+            f"WHERE TABLE_SCHEMA='{schema}' AND TABLE_NAME='{table}'"
+        )
+        r = self.query(conn, sql)
+        if not r.rows:
+            return None
+        return {row[0].strip().lower() for row in r.rows if row and row[0]}
+
     def map_type(self, type_raw: str) -> str:
         base = _base(type_raw)
         mapped = _TYPE_MAP.get(base, "text")
@@ -141,6 +203,33 @@ class DB2Driver(Driver):
         return mapped
 
 
+_PG_TO_DB2: dict[str, str] = {
+    "text": "VARCHAR(32000)", "varchar": "VARCHAR(32000)",
+    "integer": "INTEGER", "int": "INTEGER",
+    "bigint": "BIGINT", "smallint": "SMALLINT",
+    "numeric": "NUMERIC", "decimal": "DECIMAL",
+    "real": "REAL", "double precision": "DOUBLE",
+    "boolean": "SMALLINT", "bool": "SMALLINT",
+    "date": "DATE",
+    "timestamp": "TIMESTAMP", "timestamptz": "TIMESTAMP",
+    "time": "TIME",
+    "bytea": "BLOB",
+    "uuid": "CHAR(36)",
+    "json": "CLOB", "jsonb": "CLOB",
+}
+
+
+def _normalize_db2_type(dest_type: str) -> str:
+    """Map PG-style type names to DB2 for i equivalents; pass native types through."""
+    base = dest_type.lower().split("(")[0].strip()
+    native = _PG_TO_DB2.get(base)
+    if native:
+        if "(" in dest_type and base in ("numeric", "decimal"):
+            return base.upper() + dest_type[dest_type.index("("):]
+        return native
+    return dest_type or "VARCHAR(32000)"
+
+
 def _format_type(dtype: str, length: str, prec: str, scale: str) -> str:
     base = dtype.upper()
     if base in ("DECIMAL", "NUMERIC") and prec:

pipekit/drivers/mssql.py

@@ -216,6 +216,31 @@ class MSSQLDriver(Driver):
             return "numeric" + type_raw[type_raw.index("("):]
         return mapped
 
+    supports_table_comments = False  # MSSQL uses sp_addextendedproperty, not COMMENT ON
+
+    def create_schema_sql(self, schema: str) -> str:
+        name = schema.replace("'", "''")
+        q = self.quote_identifier(schema)
+        return f"IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name='{name}') EXEC('CREATE SCHEMA {q}');"
+
+    def create_like_table_sql(self, new_table: str, source_table: str) -> str:
+        return f"SELECT TOP 0 * INTO {new_table} FROM {source_table};"
+
+    def build_create_table_sql(self, qualified_table: str,
+                                columns: list[dict]) -> str:
+        if not columns:
+            raise ValueError("no columns provided for CREATE TABLE")
+        lines = []
+        for c in columns:
+            name = c["dest_name"]
+            validate_identifier(name, "dest column name")
+            dtype = _normalize_mssql_type((c.get("dest_type") or "").strip())
+            if not dtype:
+                raise ValueError(f"column {name!r} has no dest_type")
+            lines.append(f"    {self.quote_identifier(name)} {dtype}")
+        body = ",\n".join(lines)
+        return f"CREATE TABLE {qualified_table} (\n{body}\n);"
+
     # ---- helpers ----
     def _validate(self, linked_server, database, schema):
         if linked_server:
@@ -233,6 +258,33 @@ class MSSQLDriver(Driver):
         return ""
 
 
+_PG_TO_MSSQL: dict[str, str] = {
+    "text": "NVARCHAR(MAX)", "varchar": "NVARCHAR(MAX)",
+    "integer": "INT", "int": "INT",
+    "bigint": "BIGINT", "smallint": "SMALLINT",
+    "numeric": "NUMERIC", "decimal": "DECIMAL",
+    "real": "REAL", "double precision": "FLOAT",
+    "boolean": "BIT", "bool": "BIT",
+    "date": "DATE",
+    "timestamp": "DATETIME2", "timestamptz": "DATETIMEOFFSET",
+    "time": "TIME",
+    "bytea": "VARBINARY(MAX)",
+    "uuid": "UNIQUEIDENTIFIER",
+    "json": "NVARCHAR(MAX)", "jsonb": "NVARCHAR(MAX)",
+}
+
+
+def _normalize_mssql_type(dest_type: str) -> str:
+    """Map PG-style type names to MSSQL equivalents; pass native types through."""
+    base = dest_type.lower().split("(")[0].strip()
+    native = _PG_TO_MSSQL.get(base)
+    if native:
+        if "(" in dest_type and base in ("numeric", "decimal"):
+            return base.upper() + dest_type[dest_type.index("("):]
+        return native
+    return dest_type or "NVARCHAR(MAX)"
+
+
 def _format_type(dtype: str, length: str, prec: str, scale: str) -> str:
     base = dtype.upper()
     if base in ("DECIMAL", "NUMERIC") and prec:

pipekit/engine/__init__.py

@@ -1,3 +1,3 @@
-from .runner import LockBusy, RunOutcome, run_module
+from .runner import GroupRunOutcome, LockBusy, RunOutcome, run_group, run_module
 
-__all__ = ["LockBusy", "RunOutcome", "run_module"]
+__all__ = ["GroupRunOutcome", "LockBusy", "RunOutcome", "run_group", "run_module"]

pipekit/engine/runner.py

@@ -19,7 +19,7 @@ import os
 import traceback
 from dataclasses import dataclass
 
-from .. import jrunner, repo
+from .. import drivers, jrunner, repo
 from . import merge, watermark
 
 
@@ -33,6 +33,13 @@ class RunOutcome:
     merge_sql: str | None
 
 
+@dataclass
+class GroupRunOutcome:
+    group_run_id: int
+    status: str                   # success | error | dry_run
+    module_outcomes: list
+
+
 class LockBusy(RuntimeError):
     """Raised when a module is already running."""
 
@@ -70,9 +77,11 @@ def run_module(module_id: int, *, group_run_id: int | None = None,
         dest_conn = repo.get_connection(module["dest_connection_id"])
         if source_conn is None or dest_conn is None:
             raise ValueError("source or dest connection missing")
+        dest_drv_row = repo.get_driver_row(dest_conn["driver_id"])
+        dest_drv = drivers.get_driver(dest_drv_row["kind"]) if dest_drv_row else None
 
         # 2–3. watermarks + materialised source query
-        wm_values = watermark.resolve_watermarks(module, use_defaults_only=dry_run)
+        wm_values = watermark.resolve_watermarks(module, use_defaults_only=False)
         resolved_sql = watermark.materialise(module["source_query"], wm_values)
         repo.set_next_resolved_query(module_id, resolved_sql)
         repo.log_run_sql(run_id, resolved_source_sql=resolved_sql,
@@ -88,7 +97,7 @@ def run_module(module_id: int, *, group_run_id: int | None = None,
         repo.log_run_sql(run_id, merge_sql=merge_sql)
 
         if dry_run:
-            status = "success"
+            status = "dry_run"
             return RunOutcome(run_id, status, None, None, resolved_sql, merge_sql)
 
         # 4. (re)create staging from dest. DROP+CREATE (not IF NOT EXISTS) so
@@ -96,16 +105,17 @@ def run_module(module_id: int, *, group_run_id: int | None = None,
         # self-healing. Staging is ephemeral per SPEC; nothing of value lives
         # between runs.
         staging_schema, _, _ = module["staging_table"].partition(".")
-        if staging_schema and staging_schema != module["staging_table"]:
+        if staging_schema and staging_schema != module["staging_table"] and dest_drv:
             jrunner.run_dest_sql(
-                dest_conn, f"CREATE SCHEMA IF NOT EXISTS {staging_schema};")
-        jrunner.run_dest_sql(
-            dest_conn, f"DROP TABLE IF EXISTS {module['staging_table']};")
-        jrunner.run_dest_sql(
-            dest_conn,
+                dest_conn, dest_drv.create_schema_sql(staging_schema))
+        drop_sql = (dest_drv.drop_table_if_exists_sql(module["staging_table"])
+                    if dest_drv else f"DROP TABLE IF EXISTS {module['staging_table']};")
+        like_sql = (dest_drv.create_like_table_sql(module["staging_table"], module["dest_table"])
+                    if dest_drv else
                     f"CREATE TABLE {module['staging_table']} "
-            f"(LIKE {module['dest_table']} INCLUDING ALL);",
-        )
+                    f"(LIKE {module['dest_table']} INCLUDING ALL);")
+        jrunner.run_dest_sql(dest_conn, drop_sql)
+        jrunner.run_dest_sql(dest_conn, like_sql)
 
         # 5. migrate source → staging. jrunner does its own `DELETE FROM staging`
         # before loading, so we don't need a separate TRUNCATE.
@@ -113,6 +123,7 @@ def run_module(module_id: int, *, group_run_id: int | None = None,
             source_conn=source_conn, dest_conn=dest_conn,
             sql=resolved_sql, dest_table=module["staging_table"],
             clear=False,
+            progress_cb=lambda line: repo.append_run_live_log(run_id, line),
         )
         row_count = migrate_result.row_count
         repo.log_run_output(run_id, jrunner_stdout=migrate_result.stdout,
@@ -149,6 +160,49 @@ def run_module(module_id: int, *, group_run_id: int | None = None,
         repo.release_module_lock(module_id)
 
 
+def run_group(group_id: int, *, dry_run: bool = False,
+              group_run_id: int | None = None) -> GroupRunOutcome:
+    """Run all enabled members of a group sequentially in run_order.
+
+    Continues past individual module failures so partial progress is visible.
+    Final status: dry_run if all dry_run; error if any errored; success otherwise.
+    """
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise ValueError(f"group id={group_id} not found")
+
+    if group_run_id is None:
+        group_run_id = repo.create_group_run(group_id, triggered_by="manual")
+
+    members = [m for m in repo.list_group_members(group_id) if m["module_enabled"]]
+    outcomes: list[RunOutcome] = []
+
+    for member in members:
+        try:
+            outcome = run_module(
+                member["module_id"],
+                group_run_id=group_run_id,
+                dry_run=dry_run,
+            )
+        except LockBusy as e:
+            run_id = repo.create_run(member["module_id"], group_run_id=group_run_id)
+            repo.finish_run(run_id, status="error", error=str(e))
+            outcome = RunOutcome(run_id, "error", None, str(e), None, None)
+        outcomes.append(outcome)
+
+    if not outcomes:
+        final = "dry_run" if dry_run else "success"
+    elif all(o.status == "dry_run" for o in outcomes):
+        final = "dry_run"
+    elif any(o.status == "error" for o in outcomes):
+        final = "error"
+    else:
+        final = "success"
+
+    repo.finish_group_run(group_run_id, status=final)
+    return GroupRunOutcome(group_run_id, final, outcomes)
+
+
 def _run_hooks(module_id: int, *, fail_fast: bool, run_on_set: set[str]) -> str:
     """Run hooks whose ``run_on`` is in run_on_set. Returns a text log."""
     hooks = [h for h in repo.list_hooks(module_id) if h["run_on"] in run_on_set]

pipekit/jrunner.py

@@ -14,6 +14,7 @@ argv or in the database.
 
 from __future__ import annotations
 
+from collections.abc import Callable
 import csv
 import io
 import os
@@ -21,6 +22,7 @@ import re
 import shutil
 import subprocess
 import tempfile
+import threading
 from dataclasses import dataclass
 from pathlib import Path
 
@@ -144,8 +146,13 @@ def migrate(
     clear: bool = False,
     trim: bool = True,
     timeout: int = 3600,
+    progress_cb: "Callable[[str], None] | None" = None,
 ) -> MigrateResult:
-    """Stream `sql` results from source into `dest_table` via jrunner migration mode."""
+    """Stream `sql` results from source into `dest_table` via jrunner migration mode.
+
+    If ``progress_cb`` is provided it is called with each non-empty stdout line
+    as jrunner produces it, enabling live progress reporting.
+    """
     path = jrunner_path()
     with tempfile.NamedTemporaryFile("w", suffix=".sql", delete=False) as f:
         f.write(sql)
@@ -164,21 +171,50 @@ def migrate(
             argv.append("-t")
         if clear:
             argv.append("-c")
-        r = subprocess.run(argv, capture_output=True, text=True,
-                           timeout=timeout, env=_subprocess_env())
+
+        proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                                text=True, env=_subprocess_env())
+
+        stderr_buf: list[str] = []
+
+        def _drain_stderr() -> None:
+            for line in proc.stderr:  # type: ignore[union-attr]
+                stderr_buf.append(line)
+
+        t = threading.Thread(target=_drain_stderr, daemon=True)
+        t.start()
+
+        stdout_lines: list[str] = []
+        for line in proc.stdout:  # type: ignore[union-attr]
+            stdout_lines.append(line)
+            if progress_cb:
+                stripped = line.rstrip()
+                if stripped:
+                    progress_cb(stripped)
+
+        try:
+            proc.wait(timeout=timeout)
+        except subprocess.TimeoutExpired:
+            proc.kill()
+            proc.wait()
+            raise
+        t.join()
+
+        stdout = "".join(stdout_lines)
+        stderr = "".join(stderr_buf)
     finally:
         os.unlink(sql_path)
 
-    if r.returncode != 0:
-        raise JrunnerError(r.stderr.strip() or r.stdout.strip(),
-                           stdout=r.stdout, stderr=r.stderr)
-    silent = _detect_silent_failure(r.stdout, r.stderr)
+    if proc.returncode != 0:
+        raise JrunnerError(stderr.strip() or stdout.strip(),
+                           stdout=stdout, stderr=stderr)
+    silent = _detect_silent_failure(stdout, stderr)
     if silent:
-        raise JrunnerError(silent, stdout=r.stdout, stderr=r.stderr)
+        raise JrunnerError(silent, stdout=stdout, stderr=stderr)
 
     return MigrateResult(
-        row_count=_parse_row_count(r.stdout + "\n" + r.stderr),
-        stdout=r.stdout, stderr=r.stderr,
+        row_count=_parse_row_count(stdout + "\n" + stderr),
+        stdout=stdout, stderr=stderr,
     )
 
 

pipekit/repo.py

@@ -33,6 +33,27 @@ def create_driver(*, name: str, kind: str, jar_file: str, class_name: str,
         return _row(c.execute("SELECT * FROM driver WHERE id=?", (cur.lastrowid,)).fetchone())
 
 
+def get_driver_by_name(name: str) -> dict | None:
+    with db.connect() as c:
+        return _row(c.execute(
+            "SELECT * FROM driver WHERE name=?", (name,)).fetchone())
+
+
+def update_driver(driver_id: int, *, jar_file: str | None = None,
+                  class_name: str | None = None,
+                  url_template: str | None = None) -> dict:
+    fields, vals = [], []
+    for col, val in (("jar_file", jar_file), ("class_name", class_name),
+                     ("url_template", url_template)):
+        if val is not None:
+            fields.append(f"{col}=?"); vals.append(val)
+    if fields:
+        vals.append(driver_id)
+        with db.connect() as c:
+            c.execute(f"UPDATE driver SET {', '.join(fields)} WHERE id=?", vals)
+    return get_driver_row(driver_id)
+
+
 def list_drivers() -> list[dict]:
     with db.connect() as c:
         return [dict(r) for r in c.execute("SELECT * FROM driver ORDER BY name")]
@@ -404,7 +425,8 @@ def clear_stale_locks(max_age_hours: int = 24, live_pids: set[int] | None = None
 def create_run(module_id: int, *, group_run_id: int | None = None) -> int:
     with db.connect() as c:
         cur = c.execute(
-            "INSERT INTO run_log (module_id, group_run_id) VALUES (?, ?)",
+            "INSERT INTO run_log (module_id, group_run_id, started_at) "
+            "VALUES (?, ?, datetime('now'))",
             (module_id, group_run_id),
         )
         return int(cur.lastrowid)
@@ -442,6 +464,14 @@ def log_run_output(run_id: int, *, jrunner_stdout: str | None = None,
         c.execute(f"UPDATE run_log SET {', '.join(sets)} WHERE id=?", vals + [run_id])
 
 
+def append_run_live_log(run_id: int, text: str) -> None:
+    with db.connect() as c:
+        c.execute(
+            "UPDATE run_log SET live_log = COALESCE(live_log, '') || ? WHERE id=?",
+            (text + "\n", run_id),
+        )
+
+
 def finish_run(run_id: int, *, status: str, row_count: int | None = None,
                error: str | None = None) -> None:
     with db.connect() as c:
@@ -473,17 +503,226 @@ def set_setting(key: str, value: str) -> None:
         )
 
 
+# ---------------------------------------------------------------------------
+# Groups
+# ---------------------------------------------------------------------------
+
+def create_group(*, name: str) -> dict:
+    with db.connect() as c:
+        cur = c.execute("INSERT INTO grp (name) VALUES (?)", (name,))
+        return _row(c.execute("SELECT * FROM grp WHERE id=?", (cur.lastrowid,)).fetchone())
+
+
+def get_group(group_id: int) -> dict | None:
+    with db.connect() as c:
+        return _row(c.execute("SELECT * FROM grp WHERE id=?", (group_id,)).fetchone())
+
+
+def get_group_by_name(name: str) -> dict | None:
+    with db.connect() as c:
+        return _row(c.execute("SELECT * FROM grp WHERE name=?", (name,)).fetchone())
+
+
+def list_groups() -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute("SELECT * FROM grp ORDER BY name")]
+
+
+def update_group(group_id: int, *, name: str) -> dict | None:
+    with db.connect() as c:
+        c.execute("UPDATE grp SET name=? WHERE id=?", (name, group_id))
+    return get_group(group_id)
+
+
+def delete_group(group_id: int) -> bool:
+    with db.connect() as c:
+        cur = c.execute("DELETE FROM grp WHERE id=?", (group_id,))
+        return cur.rowcount > 0
+
+
+# ---------------------------------------------------------------------------
+# Schedules
+# ---------------------------------------------------------------------------
+
+def create_schedule(*, group_id: int, cron_expr: str, enabled: int = 1) -> dict:
+    with db.connect() as c:
+        cur = c.execute(
+            "INSERT INTO schedule (group_id, cron_expr, enabled) VALUES (?, ?, ?)",
+            (group_id, cron_expr, enabled),
+        )
+        return _row(c.execute("SELECT * FROM schedule WHERE id=?", (cur.lastrowid,)).fetchone())
+
+
+def get_schedule(schedule_id: int) -> dict | None:
+    with db.connect() as c:
+        return _row(c.execute("SELECT * FROM schedule WHERE id=?", (schedule_id,)).fetchone())
+
+
+def list_schedules_for_group(group_id: int) -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute(
+            "SELECT * FROM schedule WHERE group_id=? ORDER BY id", (group_id,))]
+
+
+def update_schedule(schedule_id: int, *, cron_expr: str | None = None,
+                    enabled: int | None = None) -> dict | None:
+    fields, vals = [], []
+    for col, val in (("cron_expr", cron_expr), ("enabled", enabled)):
+        if val is not None:
+            fields.append(f"{col}=?"); vals.append(val)
+    if not fields:
+        return get_schedule(schedule_id)
+    vals.append(schedule_id)
+    with db.connect() as c:
+        c.execute(f"UPDATE schedule SET {', '.join(fields)} WHERE id=?", vals)
+    return get_schedule(schedule_id)
+
+
+def delete_schedule(schedule_id: int) -> bool:
+    with db.connect() as c:
+        cur = c.execute("DELETE FROM schedule WHERE id=?", (schedule_id,))
+        return cur.rowcount > 0
+
+
+def list_all_enabled_schedules() -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute(
+            "SELECT s.*, g.name AS group_name "
+            "FROM schedule s "
+            "JOIN grp g ON s.group_id=g.id "
+            "WHERE s.enabled=1"
+        )]
+
+
+def mark_schedule_fired(schedule_id: int) -> None:
+    with db.connect() as c:
+        c.execute(
+            "UPDATE schedule SET last_fired_at=datetime('now') WHERE id=?",
+            (schedule_id,),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Group members
+# ---------------------------------------------------------------------------
+
+def list_group_members(group_id: int) -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute(
+            "SELECT gm.*, m.name AS module_name, m.enabled AS module_enabled "
+            "FROM group_member gm "
+            "JOIN module m ON gm.module_id=m.id "
+            "WHERE gm.group_id=? "
+            "ORDER BY gm.run_order, m.name",
+            (group_id,))]
+
+
+def module_group_map() -> dict[int, list[dict]]:
+    """Return {module_id: [{group_id, group_name}, ...]} for all members."""
+    with db.connect() as c:
+        rows = c.execute(
+            "SELECT gm.module_id, g.id AS group_id, g.name AS group_name "
+            "FROM group_member gm "
+            "JOIN grp g ON gm.group_id=g.id "
+            "ORDER BY g.name"
+        ).fetchall()
+    result: dict[int, list[dict]] = {}
+    for r in rows:
+        result.setdefault(r["module_id"], []).append(
+            {"group_id": r["group_id"], "group_name": r["group_name"]}
+        )
+    return result
+
+
+def set_group_members(group_id: int, members: list[dict]) -> None:
+    """Replace all members for a group. members: list of {module_id, run_order}."""
+    with db.connect() as c:
+        c.execute("DELETE FROM group_member WHERE group_id=?", (group_id,))
+        for m in members:
+            c.execute(
+                "INSERT INTO group_member (group_id, module_id, run_order) "
+                "VALUES (?, ?, ?)",
+                (group_id, m["module_id"], m.get("run_order", 0)),
+            )
+
+
+# ---------------------------------------------------------------------------
+# Group runs
+# ---------------------------------------------------------------------------
+
+def create_group_run(group_id: int, *, triggered_by: str | None = None) -> int:
+    with db.connect() as c:
+        cur = c.execute(
+            "INSERT INTO group_run (group_id, triggered_by) VALUES (?, ?)",
+            (group_id, triggered_by),
+        )
+        return int(cur.lastrowid)
+
+
+def get_group_run(group_run_id: int) -> dict | None:
+    with db.connect() as c:
+        return _row(c.execute(
+            "SELECT gr.*, g.name AS group_name, "
+            "CASE WHEN gr.started_at IS NOT NULL AND gr.finished_at IS NOT NULL "
+            "THEN CAST(ROUND((julianday(gr.finished_at) - julianday(gr.started_at)) * 86400) AS INTEGER) "
+            "ELSE NULL END AS duration_s "
+            "FROM group_run gr "
+            "JOIN grp g ON gr.group_id=g.id "
+            "WHERE gr.id=?",
+            (group_run_id,)).fetchone())
+
+
+def finish_group_run(group_run_id: int, *, status: str) -> None:
+    with db.connect() as c:
+        c.execute(
+            "UPDATE group_run SET finished_at=datetime('now'), status=? WHERE id=?",
+            (status, group_run_id),
+        )
+
+
+def list_group_runs(group_id: int, limit: int = 20) -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute(
+            "SELECT gr.*, "
+            "CASE WHEN gr.started_at IS NOT NULL AND gr.finished_at IS NOT NULL "
+            "THEN CAST(ROUND((julianday(gr.finished_at) - julianday(gr.started_at)) * 86400) AS INTEGER) "
+            "ELSE NULL END AS duration_s "
+            "FROM group_run gr "
+            "WHERE gr.group_id=? ORDER BY gr.id DESC LIMIT ?",
+            (group_id, limit))]
+
+
+def list_runs_for_group_run(group_run_id: int) -> list[dict]:
+    with db.connect() as c:
+        return [dict(r) for r in c.execute(
+            "SELECT r.*, m.name AS module_name, "
+            "CASE WHEN r.started_at IS NOT NULL AND r.finished_at IS NOT NULL "
+            "THEN CAST(ROUND((julianday(r.finished_at) - julianday(r.started_at)) * 86400) AS INTEGER) "
+            "ELSE NULL END AS duration_s "
+            "FROM run_log r "
+            "LEFT JOIN module m ON r.module_id=m.id "
+            "WHERE r.group_run_id=? "
+            "ORDER BY r.id",
+            (group_run_id,))]
+
+
 def list_runs(*, module_id: int | None = None, status: str | None = None,
-              limit: int = 50) -> list[dict]:
+              exclude_status: str | None = None, limit: int = 50) -> list[dict]:
     where, params = [], []
     if module_id is not None:
         where.append("r.module_id=?"); params.append(module_id)
     if status is not None:
         where.append("r.status=?"); params.append(status)
+    if exclude_status is not None:
+        where.append("r.status!=?"); params.append(exclude_status)
     clause = ("WHERE " + " AND ".join(where)) if where else ""
     params.append(limit)
     with db.connect() as c:
         return [dict(r) for r in c.execute(
-            f"SELECT r.*, m.name AS module_name FROM run_log r "
+            f"SELECT r.*, m.name AS module_name, "
+            f"CASE WHEN r.started_at IS NOT NULL AND r.finished_at IS NOT NULL "
+            f"THEN CAST(ROUND((julianday(r.finished_at) - julianday(r.started_at)) * 86400) AS INTEGER) "
+            f"ELSE NULL END AS duration_s "
+            f"FROM run_log r "
             f"LEFT JOIN module m ON r.module_id=m.id "
             f"{clause} ORDER BY r.id DESC LIMIT ?", params)]

pipekit/scheduler.py

@@ -0,0 +1,95 @@
+"""Background scheduler: fires group runs when cron expressions are due.
+
+One daemon thread wakes every 60 s, checks all enabled schedules, and
+spawns a per-run thread for any that are due. ``last_fired_at`` is written
+to the DB before the run starts so a slow or crashing run cannot double-fire
+the same occurrence. Survives server restarts: missed ticks while pipekit was
+down are detected on the next startup check (5 s after start).
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+import time
+from datetime import datetime, timedelta, timezone
+
+log = logging.getLogger(__name__)
+
+
+def _now() -> datetime:
+    return datetime.now()
+
+
+def _check_and_fire() -> None:
+    from . import engine, repo
+    try:
+        schedules = repo.list_all_enabled_schedules()
+    except Exception as e:                           # noqa: BLE001
+        log.error("scheduler: failed to load schedules: %s", e)
+        return
+
+    now = _now()
+    for sched in schedules:
+        try:
+            _maybe_fire(sched, now)
+        except Exception as e:                       # noqa: BLE001
+            log.error("scheduler: error on schedule id=%s: %s", sched["id"], e)
+
+
+def _maybe_fire(sched: dict, now: datetime) -> None:
+    from croniter import croniter, CroniterBadCronError
+    from . import engine, repo
+
+    last_str = sched["last_fired_at"]
+    if last_str:
+        # SQLite stores UTC; convert to local for cron evaluation.
+        last_dt = datetime.fromisoformat(last_str).replace(
+            tzinfo=timezone.utc).astimezone().replace(tzinfo=None)
+    else:
+        last_dt = now - timedelta(seconds=120)
+
+    try:
+        cron = croniter(sched["cron_expr"], last_dt)
+        next_dt = cron.get_next(datetime)
+    except CroniterBadCronError:
+        log.warning("scheduler: invalid cron_expr %r for schedule id=%s — skipping",
+                    sched["cron_expr"], sched["id"])
+        return
+
+    if next_dt > now:
+        return
+
+    log.info("scheduler: firing group_id=%s (schedule id=%s expr=%r)",
+             sched["group_id"], sched["id"], sched["cron_expr"])
+
+    # Mark fired before spawning so a slow run can't double-fire.
+    repo.mark_schedule_fired(sched["id"])
+
+    group_id = sched["group_id"]
+    sched_id = sched["id"]
+
+    def _run() -> None:
+        try:
+            group_run_id = repo.create_group_run(
+                group_id, triggered_by=f"schedule:{sched_id}"
+            )
+            engine.run_group(group_id, group_run_id=group_run_id)
+        except Exception as exc:                     # noqa: BLE001
+            log.error("scheduler: run failed for group_id=%s: %s", group_id, exc)
+
+    threading.Thread(target=_run, daemon=True).start()
+
+
+def start_scheduler() -> None:
+    """Start the background scheduler daemon thread. Safe to call once at startup."""
+
+    def _loop() -> None:
+        time.sleep(5)        # let the app finish initialising
+        while True:
+            _check_and_fire()
+            time.sleep(60)
+
+    t = threading.Thread(target=_loop, daemon=True, name="pipekit-scheduler")
+    t.start()
+    log.info("scheduler: started")

pipekit/schema.sql

@@ -90,7 +90,7 @@ CREATE TABLE IF NOT EXISTS group_run (
     group_id     INTEGER NOT NULL REFERENCES grp(id),
     started_at   TEXT DEFAULT (datetime('now')),
     finished_at  TEXT,
-    status       TEXT NOT NULL DEFAULT 'running' CHECK (status IN ('running','success','error','cancelled')),
+    status       TEXT NOT NULL DEFAULT 'running' CHECK (status IN ('running','success','error','cancelled','dry_run')),
     triggered_by TEXT                                   -- schedule | manual | null
 );
 
@@ -101,14 +101,15 @@ CREATE TABLE IF NOT EXISTS run_log (
     started_at            TEXT DEFAULT (datetime('now')),
     finished_at           TEXT,
     row_count             INTEGER,
-    status                TEXT NOT NULL DEFAULT 'running' CHECK (status IN ('running','success','error','cancelled')),
+    status                TEXT NOT NULL DEFAULT 'running' CHECK (status IN ('running','success','error','cancelled','dry_run')),
     error                 TEXT,
     resolved_source_sql   TEXT,
     merge_sql             TEXT,
     watermark_values_json TEXT,
     jrunner_stdout        TEXT,
     jrunner_stderr        TEXT,
-    hook_log              TEXT
+    hook_log              TEXT,
+    live_log              TEXT
 );
 
 CREATE INDEX IF NOT EXISTS idx_run_log_module    ON run_log(module_id, id DESC);

pipekit/web/app.py

@@ -15,13 +15,19 @@ from __future__ import annotations
 from pathlib import Path
 from urllib.parse import urlencode
 
-from fastapi import APIRouter, FastAPI, HTTPException, Query, Request
+import secrets as _secrets
+from urllib.parse import quote as _quote
+
+from fastapi import APIRouter, BackgroundTasks, Depends, FastAPI, HTTPException, Query, Request
 from fastapi.responses import HTMLResponse, RedirectResponse
 from fastapi.staticfiles import StaticFiles
 from fastapi.templating import Jinja2Templates
+from starlette.middleware.base import BaseHTTPMiddleware
 
 from .. import __version__, drivers, engine, jrunner, repo
 from ..config import get_config
+from .auth import (NotAuthenticated, auth_enabled, clear_session_cookie,
+                   get_session_user, require_web_auth, set_session_cookie)
 from ..engine import watermark
 from ..engine.merge import MergeError, build_merge_sql
 
@@ -29,19 +35,99 @@ _WEB_DIR = Path(__file__).parent
 _templates = Jinja2Templates(directory=_WEB_DIR / "templates")
 
 
+def _fmt_duration(seconds) -> str:
+    if seconds is None:
+        return "—"
+    s = int(seconds)
+    if s < 60:
+        return f"{s}s"
+    return f"{s // 60}m {s % 60:02d}s"
+
+
+def _fmt_localtime(utc_str) -> str:
+    """Convert a UTC datetime string from SQLite to local time for display."""
+    if not utc_str:
+        return "—"
+    from datetime import datetime, timezone
+    try:
+        dt = datetime.fromisoformat(str(utc_str)).replace(tzinfo=timezone.utc)
+        return dt.astimezone().strftime("%Y-%m-%d %H:%M %Z")
+    except (ValueError, TypeError):
+        return str(utc_str)
+
+
+_templates.env.filters["duration"] = _fmt_duration
+_templates.env.filters["localtime"] = _fmt_localtime
+
+
+class _SessionMiddleware(BaseHTTPMiddleware):
+    """Injects request.state.current_user so templates can read it."""
+    async def dispatch(self, request, call_next):
+        request.state.current_user = get_session_user(request)
+        return await call_next(request)
+
+
 def mount_web(app: FastAPI) -> None:
     """Attach HTML pages + /static onto a FastAPI app."""
     app.mount("/static", StaticFiles(directory=_WEB_DIR / "static"), name="static")
+    app.add_middleware(_SessionMiddleware)
+    app.include_router(_public_router)
     app.include_router(_router)
 
+    @app.exception_handler(NotAuthenticated)
+    async def _not_auth_handler(request: Request, exc: NotAuthenticated):
+        return RedirectResponse(f"/login?next={_quote(exc.next_url, safe='')}", status_code=302)
 
-_router = APIRouter(include_in_schema=False)
+
+# Public routes — no auth required.
+_public_router = APIRouter(include_in_schema=False)
+
+# Protected routes — require_web_auth no-ops when auth is disabled.
+_router = APIRouter(include_in_schema=False, dependencies=[Depends(require_web_auth)])
 
 
 def _ctx(**extra) -> dict:
     return {"version": __version__, "flash": None, **extra}
 
 
+# ---------------------------------------------------------------------------
+# Auth — login / logout
+# ---------------------------------------------------------------------------
+
+@_public_router.get("/login", response_class=HTMLResponse)
+def login_page(request: Request, next: str = "/"):
+    if get_session_user(request) or not auth_enabled():
+        return RedirectResponse(next or "/", status_code=302)
+    return _templates.TemplateResponse(request, "login.html", _ctx(next=next, error=None))
+
+
+@_public_router.post("/login")
+async def login_submit(request: Request, next: str = "/"):
+    form = await request.form()
+    username = str(form.get("username") or "")
+    password = str(form.get("password") or "")
+    expected_user = repo.get_setting("api_user") or ""
+    expected_pass = repo.get_setting("api_pass") or ""
+    user_ok = _secrets.compare_digest(username, expected_user)
+    pass_ok = _secrets.compare_digest(password, expected_pass)
+    if user_ok and pass_ok and expected_user:
+        resp = RedirectResponse(next or "/", status_code=303)
+        set_session_cookie(resp, username)
+        return resp
+    return _templates.TemplateResponse(
+        request, "login.html",
+        _ctx(next=next, error="Invalid credentials"),
+        status_code=401,
+    )
+
+
+@_public_router.post("/logout")
+def logout():
+    resp = RedirectResponse("/login", status_code=303)
+    clear_session_cookie(resp)
+    return resp
+
+
 # ---------------------------------------------------------------------------
 # Modules — home page
 # ---------------------------------------------------------------------------
@@ -51,6 +137,7 @@ def home(request: Request):
     modules = repo.list_modules()
     conns_by_id = {c["id"]: c for c in repo.list_connections()}
     drivers_by_id = {d["id"]: d for d in repo.list_drivers()}
+    groups_by_module = repo.module_group_map()
 
     # attach last-run summary to each module
     for m in modules:
@@ -64,6 +151,7 @@ def home(request: Request):
             m["last_run_at"] = None
             m["last_status"] = None
             m["last_row_count"] = None
+        m["groups"] = groups_by_module.get(m["id"], [])
 
     # group by source connection
     grouped: dict[tuple[str, str], list] = {}
@@ -94,7 +182,7 @@ def module_detail(request: Request, module_id: int):
     dest = repo.get_connection(module["dest_connection_id"])
     watermarks = repo.list_watermarks(module_id)
     hooks = repo.list_hooks(module_id)
-    recent_runs = repo.list_runs(module_id=module_id, limit=10)
+    recent_runs = repo.list_runs(module_id=module_id, limit=10, exclude_status='dry_run')
     schema_cols: list[dict] = []
     if module.get("columns_json"):
         try:
@@ -105,7 +193,7 @@ def module_detail(request: Request, module_id: int):
     preview = None
     preview_error: str | None = None
     try:
-        wm_values = watermark.resolve_watermarks(module, use_defaults_only=True)
+        wm_values = watermark.resolve_watermarks(module, use_defaults_only=False)
         merge_sql = build_merge_sql(
             strategy=module["merge_strategy"],
             dest_table=module["dest_table"],
@@ -141,6 +229,7 @@ def module_edit(request: Request, module_id: int):
         request,
         "module_form.html",
         _ctx(module=module, connections=repo.list_connections(),
+             watermarks=repo.list_watermarks(module_id),
              form_action=f"/modules/{module_id}",
              cancel_url=f"/modules/{module_id}"),
     )
@@ -170,10 +259,15 @@ async def module_update(request: Request, module_id: int):
     # fresh from dest on next run). Re-apply dest COMMENT if it changed.
     dest_conn = repo.get_connection(module["dest_connection_id"])
     if dest_conn is not None:
+        dest_drv_row = repo.get_driver_row(dest_conn["driver_id"])
+        _dest_drv = (drivers.get_driver(dest_drv_row["kind"])
+                     if dest_drv_row else None)
         try:
-            jrunner.run_dest_sql(
-                dest_conn, f"DROP TABLE IF EXISTS {module['staging_table']};")
-            if new_description != module["dest_description"]:
+            drop_sql = (_dest_drv.drop_table_if_exists_sql(module["staging_table"])
+                        if _dest_drv else f"DROP TABLE IF EXISTS {module['staging_table']};")
+            jrunner.run_dest_sql(dest_conn, drop_sql)
+            if new_description != module["dest_description"] and (
+                    _dest_drv is None or _dest_drv.supports_table_comments):
                 jrunner.run_dest_sql(
                     dest_conn,
                     f"COMMENT ON TABLE {module['dest_table']} IS "
@@ -195,6 +289,7 @@ async def module_update(request: Request, module_id: int):
         dest_description=new_description,
         enabled=1 if form.get("enabled") == "1" else 0,
     )
+    _save_inline_watermarks(form, module_id)
     return RedirectResponse(url=f"/modules/{module_id}", status_code=303)
 
 
@@ -210,17 +305,64 @@ def module_delete(module_id: int):
 
 
 @_router.post("/modules/{module_id}/run")
-async def module_run_action(module_id: int, request: Request):
+async def module_run_action(module_id: int, request: Request,
+                            background: BackgroundTasks):
     form = await request.form()
     dry = form.get("dry_run") == "1"
     if repo.get_module(module_id) is None:
         raise HTTPException(404, f"module id={module_id} not found")
     run_id = repo.create_run(module_id)
+    background.add_task(_run_in_background, module_id, run_id, dry)
+    if request.headers.get("HX-Request"):
+        hx_target = request.headers.get("HX-Target", "")
+        if hx_target.startswith("module-status-"):
+            return _module_status_pill_response(request, module_id, force_poll=True)
+        module = repo.get_module(module_id)
+        recent_runs = repo.list_runs(module_id=module_id, limit=10,
+                                     exclude_status="dry_run")
+        return _templates.TemplateResponse(
+            request, "_module_live.html",
+            _ctx(module=module, recent_runs=recent_runs, force_poll=True))
+    return RedirectResponse(url=f"/runs/{run_id}", status_code=303)
+
+
+@_router.get("/modules/{module_id}/live-fragment", response_class=HTMLResponse)
+def module_live_fragment(request: Request, module_id: int):
+    module = repo.get_module(module_id)
+    if module is None:
+        raise HTTPException(404, f"module id={module_id} not found")
+    recent_runs = repo.list_runs(module_id=module_id, limit=10,
+                                  exclude_status="dry_run")
+    return _templates.TemplateResponse(
+        request, "_module_live.html",
+        _ctx(module=module, recent_runs=recent_runs, force_poll=False))
+
+
+@_router.get("/modules/{module_id}/status-pill", response_class=HTMLResponse)
+def module_status_pill(request: Request, module_id: int):
+    if repo.get_module(module_id) is None:
+        raise HTTPException(404, f"module id={module_id} not found")
+    return _module_status_pill_response(request, module_id)
+
+
+def _module_status_pill_response(request: Request, module_id: int,
+                                  force_poll: bool = False):
+    module = repo.get_module(module_id)
+    recent = repo.list_runs(module_id=module_id, limit=1)
+    if recent:
+        module["last_status"] = recent[0]["status"]
+    else:
+        module["last_status"] = None
+    return _templates.TemplateResponse(
+        request, "_module_status_pill.html",
+        _ctx(module=module, force_poll=force_poll))
+
+
+def _run_in_background(module_id: int, run_id: int, dry_run: bool) -> None:
     try:
-        engine.run_module(module_id, run_id=run_id, dry_run=dry)
+        engine.run_module(module_id, run_id=run_id, dry_run=dry_run)
     except engine.LockBusy as e:
         repo.finish_run(run_id, status="error", error=str(e))
-    return RedirectResponse(url=f"/runs/{run_id}", status_code=303)
 
 
 # ---------------------------------------------------------------------------
@@ -344,7 +486,7 @@ def wizard_step3(request: Request,
     try:
         for c in drv.get_columns(conn, table, **qvals):
             d = c.to_dict()
-            d["default_dest_name"] = c.name.lower()
+            d["default_dest_name"] = _sanitize_identifier(c.name)
             d["default_dest_type"] = drv.map_type(c.type_raw)
             d["default_description"] = c.description or ""
             columns.append(d)
@@ -357,7 +499,7 @@ def wizard_step3(request: Request,
     drivers_by_id = {d["id"]: d for d in repo.list_drivers()}
     dest_conns = [
         c for c in repo.list_connections()
-        if drivers_by_id.get(c["driver_id"], {}).get("kind") == "pg"
+        if drivers_by_id.get(c["driver_id"]) is not None
     ]
     qualified = drv.qualified_table_name(table, **qvals) if not fetch_error else table
     default_module_name = (table_schema + "_" + table).lower() if table_schema else table.lower()
@@ -370,8 +512,12 @@ def wizard_step3(request: Request,
     if not fetch_error and default_dest_conn_id and default_dest_schema:
         dest_conn_row = repo.get_connection(default_dest_conn_id)
         if dest_conn_row is not None:
+            dest_drv_row = repo.get_driver_row(dest_conn_row["driver_id"])
+            default_dest_drv = (drivers.get_driver(dest_drv_row["kind"])
+                                if dest_drv_row else None)
+            if default_dest_drv is not None:
                 try:
-                existing = _existing_dest_columns(
+                    existing = default_dest_drv.check_dest_table(
                         dest_conn_row, default_dest_schema, default_module_name)
                 except jrunner.JrunnerError:
                     existing = None
@@ -464,7 +610,8 @@ async def wizard_create(request: Request):
         f"{dest_drv.quote_identifier(c['dest_name'])}"
         for c in chosen
     )
-    source_query = f"SELECT\n  {select_list}\nFROM {qualified_source}"
+    source_query_override = (form.get("source_query") or "").strip()
+    source_query = source_query_override or f"SELECT\n  {select_list}\nFROM {qualified_source}"
 
     dest_schema, _, dest_table_bare = dest_table.partition(".")
     if not dest_table_bare:
@@ -481,8 +628,7 @@ async def wizard_create(request: Request):
     # If the dest table already exists, don't clobber it. Verify the picks
     # match its shape and skip CREATE/COMMENT.
     try:
-        existing_cols = _existing_dest_columns(
-            dest_conn, dest_schema, dest_table_bare)
+        existing_cols = dest_drv.check_dest_table(dest_conn, dest_schema, dest_table_bare)
     except jrunner.JrunnerError as e:
         raise HTTPException(500, f"could not introspect dest: {e}")
     dest_exists = existing_cols is not None
@@ -509,29 +655,19 @@ async def wizard_create(request: Request):
             )
 
     try:
-        jrunner.run_dest_sql(
-            dest_conn,
-            f"CREATE SCHEMA IF NOT EXISTS {dest_drv.quote_identifier(dest_schema)};",
-        )
+        jrunner.run_dest_sql(dest_conn, dest_drv.create_schema_sql(dest_schema))
         if not dest_exists:
             jrunner.run_dest_sql(dest_conn, create_table_sql)
+            if dest_drv.supports_table_comments:
                 comment_sql = _build_comment_sql(dest_drv, qualified_dest,
                                                   dest_description, chosen)
                 if comment_sql:
                     jrunner.run_dest_sql(dest_conn, comment_sql)
         # Pre-align staging to dest so first run doesn't surprise us.
         if staging_schema and staging_schema != effective_staging:
-            jrunner.run_dest_sql(
-                dest_conn,
-                f"CREATE SCHEMA IF NOT EXISTS {dest_drv.quote_identifier(staging_schema)};",
-            )
-        jrunner.run_dest_sql(
-            dest_conn, f"DROP TABLE IF EXISTS {effective_staging};")
-        jrunner.run_dest_sql(
-            dest_conn,
-            f"CREATE TABLE {effective_staging} "
-            f"(LIKE {qualified_dest} INCLUDING ALL);",
-        )
+            jrunner.run_dest_sql(dest_conn, dest_drv.create_schema_sql(staging_schema))
+        jrunner.run_dest_sql(dest_conn, dest_drv.drop_table_if_exists_sql(effective_staging))
+        jrunner.run_dest_sql(dest_conn, dest_drv.create_like_table_sql(effective_staging, qualified_dest))
     except jrunner.JrunnerError as e:
         raise HTTPException(500, f"dest provisioning failed: {e}")
 
@@ -547,28 +683,80 @@ async def wizard_create(request: Request):
         columns=chosen,
         dest_description=dest_description,
     )
+    _save_inline_watermarks(form, module["id"])
     return RedirectResponse(url=f"/modules/{module['id']}", status_code=303)
 
 
+def _save_inline_watermarks(form, module_id: int) -> None:
+    """Process wm_* arrays from a module form POST and persist watermarks.
+
+    Deletes IDs listed in wm_deleted_id[], updates rows with a wm_id, and
+    creates new rows (no wm_id). Skips rows with a blank name. No-ops if the
+    form contains no wm_name fields at all (backwards-compat with old POSTs).
+    """
+    wm_names = form.getlist("wm_name")
+    if not wm_names:
+        return
+    for wm_id_str in form.getlist("wm_deleted_id"):
+        if wm_id_str:
+            repo.delete_watermark(int(wm_id_str))
+    wm_ids = form.getlist("wm_id")
+    wm_conns = form.getlist("wm_connection_id")
+    wm_sqls = form.getlist("wm_resolver_sql")
+    wm_defaults = form.getlist("wm_default_value")
+    for idx, name in enumerate(wm_names):
+        name = name.strip()
+        if not name:
+            continue
+        conn_id_str = wm_conns[idx] if idx < len(wm_conns) else ""
+        if not conn_id_str:
+            continue
+        conn_id = int(conn_id_str)
+        sql = wm_sqls[idx] if idx < len(wm_sqls) else ""
+        default_val = (wm_defaults[idx] if idx < len(wm_defaults) else "").strip() or None
+        wm_id_str = wm_ids[idx] if idx < len(wm_ids) else ""
+        if wm_id_str:
+            repo.update_watermark(int(wm_id_str), name=name, connection_id=conn_id,
+                                  resolver_sql=sql, default_value=default_val)
+        else:
+            repo.create_watermark(module_id=module_id, name=name,
+                                  connection_id=conn_id, resolver_sql=sql,
+                                  default_value=default_val)
+
+
+def _schedules_with_next(schedules: list[dict]) -> list[dict]:
+    """Attach a 'next_fire_at' string to each schedule dict."""
+    from croniter import croniter, CroniterBadCronError
+    from datetime import datetime, timezone
+    now = datetime.now()
+    result = []
+    for s in schedules:
+        s = dict(s)
+        try:
+            cron = croniter(s["cron_expr"], now)
+            s["next_fire_at"] = cron.get_next(datetime).strftime("%Y-%m-%d %H:%M %Z")
+        except CroniterBadCronError:
+            s["next_fire_at"] = "invalid expression"
+        result.append(s)
+    return result
+
+
+def _sanitize_identifier(name: str) -> str:
+    """Lower-case a source column name and replace characters that aren't
+    valid in an unquoted identifier with underscores."""
+    import re as _re
+    s = name.lower().strip()
+    s = _re.sub(r"[^a-z0-9_$#.]", "_", s)
+    if s and s[0].isdigit():
+        s = "_" + s
+    return s or "_"
+
+
 def _sql_str(v: str) -> str:
     """SQL string literal — PG-style single-quote escaping."""
     return "'" + v.replace("'", "''") + "'"
 
 
-def _existing_dest_columns(dest_conn: dict, schema: str,
-                           table: str) -> set[str] | None:
-    """Return lowercase column names of an existing PG dest table, or
-    None if it doesn't exist. PG-only; fine while pg is the sole dest."""
-    r = jrunner.run_dest_sql(
-        dest_conn,
-        f"SELECT column_name FROM information_schema.columns "
-        f"WHERE table_schema={_sql_str(schema)} "
-        f"AND table_name={_sql_str(table)}",
-    )
-    if not r.rows:
-        return None
-    return {row[0].strip().lower() for row in r.rows if row and row[0]}
-
 
 def _build_comment_sql(dest_drv, qualified_dest: str,
                         table_description: str | None,
@@ -716,6 +904,20 @@ def run_detail(request: Request, run_id: int):
     )
 
 
+@_router.get("/runs/{run_id}/live", response_class=HTMLResponse)
+def run_live_fragment(request: Request, run_id: int):
+    run = repo.get_run(run_id)
+    if run is None:
+        raise HTTPException(404, f"run id={run_id} not found")
+    module = repo.get_module(run["module_id"])
+    run["module_name"] = module["name"] if module else "?"
+    return _templates.TemplateResponse(
+        request,
+        "_run_live.html",
+        _ctx(run=run),
+    )
+
+
 # ---------------------------------------------------------------------------
 # Watermarks — add/edit/delete forms on module detail
 # ---------------------------------------------------------------------------
@@ -864,3 +1066,219 @@ def hook_delete(hook_id: int):
     module_id = hook["module_id"]
     repo.delete_hook(hook_id)
     return RedirectResponse(url=f"/modules/{module_id}", status_code=303)
+
+
+# ---------------------------------------------------------------------------
+# Groups
+# ---------------------------------------------------------------------------
+
+@_router.get("/groups", response_class=HTMLResponse)
+def groups_index(request: Request):
+    groups = repo.list_groups()
+    for g in groups:
+        members = repo.list_group_members(g["id"])
+        g["member_count"] = len(members)
+        recent = repo.list_group_runs(g["id"], limit=1)
+        if recent:
+            g["last_run_at"] = recent[0]["started_at"]
+            g["last_status"] = recent[0]["status"]
+        else:
+            g["last_run_at"] = None
+            g["last_status"] = None
+    return _templates.TemplateResponse(
+        request, "groups.html",
+        _ctx(groups=groups),
+    )
+
+
+@_router.get("/groups/new", response_class=HTMLResponse)
+def group_new(request: Request):
+    return _templates.TemplateResponse(
+        request, "group_form.html",
+        _ctx(group=None, all_modules=repo.list_modules(), members=[], schedules=[],
+             form_action="/groups", cancel_url="/groups"),
+    )
+
+
+@_router.post("/groups")
+async def group_create(request: Request):
+    form = await request.form()
+    name = form["name"].strip()
+    if repo.get_group_by_name(name) is not None:
+        raise HTTPException(409, f"group name {name!r} already exists")
+    grp = repo.create_group(name=name)
+    _save_group_members(form, grp["id"])
+    _save_group_schedules(form, grp["id"])
+    return RedirectResponse(url=f"/groups/{grp['id']}", status_code=303)
+
+
+@_router.get("/groups/{group_id}", response_class=HTMLResponse)
+def group_detail(request: Request, group_id: int):
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    members = repo.list_group_members(group_id)
+    recent_runs = repo.list_group_runs(group_id, limit=10)
+    group_running = bool(recent_runs and recent_runs[0]["status"] == "running")
+    schedules = _schedules_with_next(repo.list_schedules_for_group(group_id))
+    return _templates.TemplateResponse(
+        request, "group_detail.html",
+        _ctx(group=grp, members=members, recent_runs=recent_runs,
+             group_running=group_running, schedules=schedules),
+    )
+
+
+@_router.get("/groups/{group_id}/edit", response_class=HTMLResponse)
+def group_edit(request: Request, group_id: int):
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    members = repo.list_group_members(group_id)
+    schedules = repo.list_schedules_for_group(group_id)
+    return _templates.TemplateResponse(
+        request, "group_form.html",
+        _ctx(group=grp, all_modules=repo.list_modules(), members=members,
+             schedules=schedules,
+             form_action=f"/groups/{group_id}", cancel_url=f"/groups/{group_id}"),
+    )
+
+
+@_router.post("/groups/{group_id}")
+async def group_update(request: Request, group_id: int):
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    form = await request.form()
+    name = form["name"].strip()
+    if name != grp["name"] and repo.get_group_by_name(name) is not None:
+        raise HTTPException(409, f"group name {name!r} already exists")
+    repo.update_group(group_id, name=name)
+    _save_group_members(form, group_id)
+    _save_group_schedules(form, group_id)
+    return RedirectResponse(url=f"/groups/{group_id}", status_code=303)
+
+
+@_router.post("/groups/{group_id}/delete")
+def group_delete(group_id: int):
+    if repo.get_group(group_id) is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    repo.delete_group(group_id)
+    return RedirectResponse(url="/groups", status_code=303)
+
+
+@_router.post("/groups/{group_id}/run")
+async def group_run_action(group_id: int, request: Request,
+                            background: BackgroundTasks):
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    form = await request.form()
+    dry = form.get("dry_run") == "1"
+    group_run_id = repo.create_group_run(group_id, triggered_by="manual")
+    background.add_task(_run_group_in_background, group_id, group_run_id, dry)
+    if request.headers.get("HX-Request"):
+        members = repo.list_group_members(group_id)
+        recent_runs = repo.list_group_runs(group_id, limit=10)
+        return _templates.TemplateResponse(
+            request, "_group_live.html",
+            _ctx(group=grp, members=members, recent_runs=recent_runs,
+                 group_running=True, force_poll=True))
+    return RedirectResponse(url=f"/group-runs/{group_run_id}", status_code=303)
+
+
+@_router.get("/groups/{group_id}/live-fragment", response_class=HTMLResponse)
+def group_live_fragment(request: Request, group_id: int):
+    grp = repo.get_group(group_id)
+    if grp is None:
+        raise HTTPException(404, f"group id={group_id} not found")
+    members = repo.list_group_members(group_id)
+    recent_runs = repo.list_group_runs(group_id, limit=10)
+    group_running = bool(recent_runs and recent_runs[0]["status"] == "running")
+    return _templates.TemplateResponse(
+        request, "_group_live.html",
+        _ctx(group=grp, members=members, recent_runs=recent_runs,
+             group_running=group_running, force_poll=False))
+
+
+def _run_group_in_background(group_id: int, group_run_id: int,
+                              dry_run: bool) -> None:
+    try:
+        engine.run_group(group_id, dry_run=dry_run, group_run_id=group_run_id)
+    except Exception as e:                                           # noqa: BLE001
+        repo.finish_group_run(group_run_id, status="error")
+
+
+# ---------------------------------------------------------------------------
+# Group run detail
+# ---------------------------------------------------------------------------
+
+@_router.get("/group-runs/{group_run_id}", response_class=HTMLResponse)
+def group_run_detail(request: Request, group_run_id: int):
+    group_run = repo.get_group_run(group_run_id)
+    if group_run is None:
+        raise HTTPException(404, f"group run id={group_run_id} not found")
+    module_runs = repo.list_runs_for_group_run(group_run_id)
+    return _templates.TemplateResponse(
+        request, "group_run_detail.html",
+        _ctx(group_run=group_run, module_runs=module_runs),
+    )
+
+
+@_router.get("/group-runs/{group_run_id}/live", response_class=HTMLResponse)
+def group_run_live_fragment(request: Request, group_run_id: int):
+    group_run = repo.get_group_run(group_run_id)
+    if group_run is None:
+        raise HTTPException(404, f"group run id={group_run_id} not found")
+    module_runs = repo.list_runs_for_group_run(group_run_id)
+    return _templates.TemplateResponse(
+        request, "_group_run_live.html",
+        _ctx(group_run=group_run, module_runs=module_runs),
+    )
+
+
+def _save_group_schedules(form, group_id: int) -> None:
+    """Process sched_* parallel arrays from a group form POST."""
+    from croniter import croniter as _croniter
+    exprs = form.getlist("sched_cron_expr")
+    if not exprs:
+        return
+    ids = form.getlist("sched_id")
+    enableds = form.getlist("sched_enabled")   # select always submits a value
+    for sid_str in form.getlist("sched_deleted_id"):
+        if sid_str:
+            repo.delete_schedule(int(sid_str))
+    for i, expr in enumerate(exprs):
+        expr = expr.strip()
+        if not expr:
+            continue
+        if not _croniter.is_valid(expr):
+            continue
+        enabled = 1 if (i < len(enableds) and enableds[i] == "1") else 0
+        sid_str = ids[i] if i < len(ids) else ""
+        if sid_str:
+            repo.update_schedule(int(sid_str), cron_expr=expr, enabled=enabled)
+        else:
+            repo.create_schedule(group_id=group_id, cron_expr=expr, enabled=enabled)
+
+
+def _save_group_members(form, group_id: int) -> None:
+    module_ids = form.getlist("member_module_id")
+    run_orders = form.getlist("member_run_order")
+    members = []
+    seen = set()
+    for i, mid_str in enumerate(module_ids):
+        if not mid_str:
+            continue
+        try:
+            mid = int(mid_str)
+        except ValueError:
+            continue
+        if mid in seen:
+            continue
+        seen.add(mid)
+        try:
+            order = int(run_orders[i]) if i < len(run_orders) else i
+        except (ValueError, IndexError):
+            order = i
+        members.append({"module_id": mid, "run_order": order})
+    repo.set_group_members(group_id, members)

pipekit/web/auth.py

@@ -0,0 +1,74 @@
+"""Session-cookie auth for the web UI.
+
+Uses itsdangerous to sign a short-lived cookie storing the username.
+Reuses the same api_user / api_pass credentials and api_auth_enabled
+flag as the JSON API — no extra config needed.
+
+Enable with:
+    api_auth_enabled: true   # in config.yaml
+    pipekit set-password <username>
+"""
+
+from __future__ import annotations
+
+import secrets
+
+from fastapi import Request, Response
+from itsdangerous import BadSignature, SignatureExpired, URLSafeTimedSerializer
+
+from .. import repo
+from ..config import get_config
+
+_COOKIE = "pk_session"
+_MAX_AGE = 8 * 3600  # 8 hours
+
+
+def auth_enabled() -> bool:
+    return bool(get_config().get("api_auth_enabled", False))
+
+
+def _signer() -> URLSafeTimedSerializer:
+    secret = repo.get_setting("web_session_secret")
+    if not secret:
+        secret = secrets.token_hex(32)
+        repo.set_setting("web_session_secret", secret)
+    return URLSafeTimedSerializer(secret, salt="pipekit-web")
+
+
+def get_session_user(request: Request) -> str | None:
+    token = request.cookies.get(_COOKIE)
+    if not token:
+        return None
+    try:
+        return _signer().loads(token, max_age=_MAX_AGE)
+    except (BadSignature, SignatureExpired):
+        return None
+
+
+def set_session_cookie(response: Response, username: str) -> None:
+    token = _signer().dumps(username)
+    response.set_cookie(_COOKIE, token, httponly=True, samesite="lax", max_age=_MAX_AGE)
+
+
+def clear_session_cookie(response: Response) -> None:
+    response.delete_cookie(_COOKIE, httponly=True, samesite="lax")
+
+
+class NotAuthenticated(Exception):
+    """Raised by require_web_auth; caught by app-level handler → redirect to /login."""
+    def __init__(self, next_url: str = "/"):
+        self.next_url = next_url
+
+
+def require_web_auth(request: Request) -> str | None:
+    """FastAPI dependency for protected web routes.
+
+    No-ops when api_auth_enabled is false. Otherwise redirects to /login
+    via NotAuthenticated if there is no valid session cookie.
+    """
+    if not auth_enabled():
+        return None
+    user = get_session_user(request)
+    if not user:
+        raise NotAuthenticated(next_url=str(request.url.path))
+    return user

pipekit/web/static/style.css

@@ -5,6 +5,7 @@
    - Layout directs flow; nothing floats. */
 
 :root {
+    --topbar-h: 2.65rem;
     --bg: #111418;
     --surface: #181c22;
     --border: #2a3038;
@@ -39,6 +40,9 @@ header.topbar {
     padding: 0.6rem 1.2rem;
     background: var(--surface);
     border-bottom: 1px solid var(--border-strong);
+    position: sticky;
+    top: 0;
+    z-index: 100;
 }
 header.topbar .brand {
     font-weight: 700;
@@ -62,7 +66,6 @@ header.topbar nav a:hover {
 header.topbar .right { margin-left: auto; color: var(--text-muted); font-size: 12px; }
 
 main {
-    max-width: 1200px;
     margin: 1rem auto;
     padding: 0 1.2rem;
 }
@@ -83,6 +86,9 @@ main {
     display: flex;
     align-items: center;
     gap: 0.6rem;
+    position: sticky;
+    top: var(--topbar-h);
+    z-index: 99;
 }
 .panel > header .subtitle {
     color: var(--text-muted);
@@ -140,6 +146,20 @@ table.grid tr:hover td { background: #1c2128; }
 .pill.disabled               { color: var(--text-muted); }
 .pill.warning                { color: var(--warning); }
 
+/* Group membership tags */
+.tag {
+    display: inline-block;
+    padding: 0.05rem 0.45rem;
+    border-radius: 4px;
+    font-size: 11px;
+    font-weight: 500;
+    background: var(--border);
+    color: var(--text-muted);
+    text-decoration: none;
+    margin-right: 0.2rem;
+}
+.tag:hover { color: var(--accent); }
+
 /* Labeled key-value rows (used in detail views) */
 dl.keyval {
     display: grid;
@@ -238,6 +258,10 @@ label.field {
     margin-bottom: 0.6rem;
 }
 label.field .help { grid-column: 2; color: var(--text-muted); font-size: 12px; }
+/* Inputs inside label.field must not overflow their grid cell — the global
+   min-width:14rem is correct for standalone use but blows past narrow containers. */
+label.field input[type="text"], label.field input[type="number"], label.field input[type="password"],
+label.field select, label.field textarea { min-width: 0; width: 100%; }
 
 /* Step indicator */
 .steps {

pipekit/web/templates/_group_live.html

@@ -0,0 +1,44 @@
+{# Partial: recent group runs panel for group_detail.html.
+   Polls every 3s while a group run is in progress. #}
+
+<div id="group-live"
+     {% if group_running or force_poll %}
+     hx-get="/groups/{{ group.id }}/live-fragment"
+     hx-trigger="every 3s"
+     hx-swap="outerHTML"
+     {% endif %}>
+    <div class="panel">
+        <header>
+            Recent runs
+            <span class="subtitle">last {{ recent_runs|length }}</span>
+        </header>
+        <div class="body tight">
+            {% if recent_runs %}
+            <table class="grid">
+                <thead>
+                    <tr>
+                        <th>id</th>
+                        <th>started</th>
+                        <th>duration</th>
+                        <th>triggered by</th>
+                        <th>status</th>
+                    </tr>
+                </thead>
+                <tbody>
+                {% for r in recent_runs %}
+                    <tr>
+                        <td><a href="/group-runs/{{ r.id }}">#{{ r.id }}</a></td>
+                        <td class="mono">{{ r.started_at | localtime }}</td>
+                        <td class="mono">{{ r.duration_s | duration }}</td>
+                        <td class="mono">{{ r.triggered_by or "—" }}</td>
+                        <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+            {% else %}
+            <div class="empty">No runs yet.</div>
+            {% endif %}
+        </div>
+    </div>
+</div>

pipekit/web/templates/_group_run_live.html

@@ -0,0 +1,49 @@
+{# Partial: module runs table for group_run_detail.html. Polls while running. #}
+<div id="group-run-live"
+     {% if group_run.status == 'running' %}
+     hx-get="/group-runs/{{ group_run.id }}/live"
+     hx-trigger="every 3s"
+     hx-swap="outerHTML"
+     {% endif %}>
+    <div class="panel">
+        <header>
+            Module runs
+            <span style="margin-left:auto">
+                <span class="pill {{ group_run.status }}">{{ group_run.status }}</span>
+                {% if group_run.duration_s is not none %}&nbsp;· {{ group_run.duration_s | duration }}{% endif %}
+            </span>
+        </header>
+        <div class="body tight">
+            {% if module_runs %}
+            <table class="grid">
+                <thead>
+                    <tr>
+                        <th>run</th>
+                        <th>module</th>
+                        <th>started</th>
+                        <th>duration</th>
+                        <th>status</th>
+                        <th>rows</th>
+                    </tr>
+                </thead>
+                <tbody>
+                {% for r in module_runs %}
+                    <tr>
+                        <td><a href="/runs/{{ r.id }}">#{{ r.id }}</a></td>
+                        <td><a href="/modules/{{ r.module_id }}">{{ r.module_name }}</a></td>
+                        <td class="mono">{{ r.started_at | localtime }}</td>
+                        <td class="mono">{{ r.duration_s | duration }}</td>
+                        <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
+                        <td class="mono">{{ r.row_count if r.row_count is not none else "—" }}</td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+            {% elif group_run.status == 'running' %}
+            <div class="empty">Waiting for module runs to start…</div>
+            {% else %}
+            <div class="empty">No module runs recorded.</div>
+            {% endif %}
+        </div>
+    </div>
+</div>

pipekit/web/templates/_module_live.html

@@ -0,0 +1,43 @@
+{# Partial: recent runs panel + out-of-band status pill update.
+   Rendered by both module_detail.html (on load) and the live-fragment endpoint.
+   Polls every 3s while running; stops automatically when idle. #}
+
+<div id="module-live"
+     {% if module.running or force_poll %}
+     hx-get="/modules/{{ module.id }}/live-fragment"
+     hx-trigger="every 3s"
+     hx-swap="outerHTML"
+     {% endif %}>
+    <div class="panel">
+        <header>Recent runs
+            <span class="subtitle">last {{ recent_runs|length }}</span>
+            <span style="margin-left:auto"><a href="/runs?module_id={{ module.id }}">all →</a></span>
+        </header>
+        <div class="body tight">
+            {% if recent_runs %}
+            <table class="grid">
+                <thead><tr><th>id</th><th>started</th><th>duration</th><th>status</th><th>rows</th></tr></thead>
+                <tbody>
+                {% for r in recent_runs %}
+                    <tr>
+                        <td><a href="/runs/{{ r.id }}">#{{ r.id }}</a></td>
+                        <td class="mono">{{ r.started_at | localtime }}</td>
+                        <td class="mono">{{ r.duration_s | duration }}</td>
+                        <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
+                        <td class="mono">{{ r.row_count if r.row_count is not none else "—" }}</td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+            {% else %}
+            <div class="empty">No runs yet.</div>
+            {% endif %}
+        </div>
+    </div>
+</div>
+
+{# Out-of-band: keep the status pills in the page header in sync #}
+<span id="module-pills" hx-swap-oob="true">
+    {% if module.running %}<span class="pill running">running</span>{% endif %}
+    {% if not module.enabled %}<span class="pill disabled">disabled</span>{% endif %}
+</span>

pipekit/web/templates/_module_status_pill.html

@@ -0,0 +1,18 @@
+{# Partial: status cell for one module row on the index page.
+   Swaps itself (outerHTML) every 3s while running; stops when idle. #}
+<td id="module-status-{{ module.id }}"
+    {% if module.running or force_poll %}
+    hx-get="/modules/{{ module.id }}/status-pill"
+    hx-trigger="every 3s"
+    hx-swap="outerHTML"
+    {% endif %}>
+    {% if module.running %}
+        <span class="pill running">running</span>
+    {% elif not module.enabled %}
+        <span class="pill disabled">disabled</span>
+    {% elif module.last_status %}
+        <span class="pill {{ module.last_status }}">{{ module.last_status }}</span>
+    {% else %}
+        <span class="pill">never ran</span>
+    {% endif %}
+</td>

pipekit/web/templates/_run_live.html

@@ -0,0 +1,26 @@
+<div id="run-live"
+  {% if run.status == 'running' %}
+  hx-get="/runs/{{ run.id }}/live"
+  hx-trigger="every 2s"
+  hx-swap="outerHTML"
+  {% endif %}
+>
+  <div class="panel">
+    <header>
+      Progress
+      <span style="margin-left:auto">
+        <span class="pill {{ run.status }}">{{ run.status }}</span>
+        {% if run.row_count is not none %}&nbsp;· {{ run.row_count }} rows{% endif %}
+      </span>
+    </header>
+    <div class="body">
+      {% if run.live_log %}
+      <pre style="margin:0;white-space:pre-wrap">{{ run.live_log }}</pre>
+      {% elif run.status == 'running' %}
+      <span style="color:var(--text-muted)">Waiting for output…</span>
+      {% else %}
+      <span style="color:var(--text-muted)">—</span>
+      {% endif %}
+    </div>
+  </div>
+</div>

pipekit/web/templates/base.html

@@ -5,6 +5,7 @@
 <title>{% block title %}Pipekit{% endblock %}</title>
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <link rel="stylesheet" href="/static/style.css">
+<script src="https://unpkg.com/htmx.org@1.9.12/dist/htmx.min.js" defer></script>
 </head>
 <body>
 <header class="topbar">
@@ -12,9 +13,20 @@
     <nav>
         <a href="/"            class="{% if section == 'modules'     %}active{% endif %}">Modules</a>
         <a href="/connections" class="{% if section == 'connections' %}active{% endif %}">Connections</a>
+        <a href="/groups"      class="{% if section == 'groups'      %}active{% endif %}">Groups</a>
         <a href="/runs"        class="{% if section == 'runs'        %}active{% endif %}">Runs</a>
     </nav>
-    <span class="right">v{{ version }} &middot; <a href="/docs">API docs</a></span>
+    <span class="right">
+        {% if request.state.current_user %}
+        {{ request.state.current_user }}
+        &middot;
+        <form method="post" action="/logout" style="display:inline;margin:0">
+            <button type="submit" class="ghost" style="font-size:12px;padding:0.1rem 0.3rem;border:none;color:var(--text-muted)">logout</button>
+        </form>
+        &middot;
+        {% endif %}
+        v{{ version }} &middot; <a href="/docs">API docs</a>
+    </span>
 </header>
 <main>
     {% if flash %}

pipekit/web/templates/group_detail.html

@@ -0,0 +1,109 @@
+{% extends "base.html" %}
+{% set section = "groups" %}
+{% block title %}{{ group.name }} — Pipekit{% endblock %}
+
+{% block content %}
+<div class="panel">
+    <header>
+        {{ group.name }}
+        <span class="subtitle">group #{{ group.id }}</span>
+        <span style="margin-left:auto; display:flex; gap:0.5rem; align-items:center">
+            <a class="btn ghost" href="/groups/{{ group.id }}/edit">Edit</a>
+            <form class="inline"
+                  hx-post="/groups/{{ group.id }}/run"
+                  hx-target="#group-live"
+                  hx-swap="outerHTML">
+                <button type="submit">Run</button>
+            </form>
+            <form class="inline"
+                  hx-post="/groups/{{ group.id }}/run"
+                  hx-target="#group-live"
+                  hx-swap="outerHTML">
+                <input type="hidden" name="dry_run" value="1">
+                <button type="submit" class="ghost">Dry run</button>
+            </form>
+        </span>
+    </header>
+    <div class="body tight">
+        {% if members %}
+        <table class="grid">
+            <thead>
+                <tr>
+                    <th>order</th>
+                    <th>module</th>
+                    <th>enabled</th>
+                </tr>
+            </thead>
+            <tbody>
+            {% for m in members %}
+                <tr>
+                    <td class="mono">{{ m.run_order }}</td>
+                    <td><a href="/modules/{{ m.module_id }}">{{ m.module_name }}</a></td>
+                    <td>
+                        {% if m.module_enabled %}
+                        <span class="pill success">yes</span>
+                        {% else %}
+                        <span class="pill disabled">disabled — skipped</span>
+                        {% endif %}
+                    </td>
+                </tr>
+            {% endfor %}
+            </tbody>
+        </table>
+        {% else %}
+        <div class="empty">No members yet. <a href="/groups/{{ group.id }}/edit">Add some →</a></div>
+        {% endif %}
+    </div>
+</div>
+
+<div class="panel">
+    <header>
+        Schedules
+        <span style="margin-left:auto"><a class="btn ghost" href="/groups/{{ group.id }}/edit">Edit schedules →</a></span>
+    </header>
+    <div class="body tight">
+        {% if schedules %}
+        <table class="grid">
+            <thead>
+                <tr>
+                    <th>cron expression</th>
+                    <th>next fire</th>
+                    <th>last fired</th>
+                    <th>enabled</th>
+                </tr>
+            </thead>
+            <tbody>
+            {% for s in schedules %}
+                <tr>
+                    <td class="mono">{{ s.cron_expr }}</td>
+                    <td class="mono">{% if s.enabled %}{{ s.next_fire_at }}{% else %}—{% endif %}</td>
+                    <td class="mono">{{ s.last_fired_at | localtime }}</td>
+                    <td>
+                        {% if s.enabled %}
+                        <span class="pill success">yes</span>
+                        {% else %}
+                        <span class="pill disabled">no</span>
+                        {% endif %}
+                    </td>
+                </tr>
+            {% endfor %}
+            </tbody>
+        </table>
+        {% else %}
+        <div class="empty">No schedules — <a href="/groups/{{ group.id }}/edit">add one →</a></div>
+        {% endif %}
+    </div>
+</div>
+
+{% include "_group_live.html" %}
+
+<div class="panel" style="margin-top:1rem">
+    <header>Danger zone</header>
+    <div class="body">
+        <form method="post" action="/groups/{{ group.id }}/delete"
+              onsubmit="return confirm('Delete group {{ group.name }}? Module runs are kept.')">
+            <button type="submit" class="ghost" style="color:var(--danger)">Delete group</button>
+        </form>
+    </div>
+</div>
+{% endblock %}

pipekit/web/templates/group_form.html

@@ -0,0 +1,250 @@
+{% extends "base.html" %}
+{% set section = "groups" %}
+{% block title %}{% if group %}Edit group · {{ group.name }}{% else %}New group{% endif %} — Pipekit{% endblock %}
+
+{% block content %}
+<div class="panel">
+    <header>
+        {% if group %}Edit group &middot; {{ group.name }}{% else %}New group{% endif %}
+        <span style="margin-left:auto"><a href="{{ cancel_url }}">&larr; back</a></span>
+    </header>
+    <div class="body">
+        <form method="post" action="{{ form_action }}" id="group-form">
+            <label class="field">
+                <span>name</span>
+                <input type="text" name="name" required value="{{ group.name if group else '' }}">
+            </label>
+
+            <div class="panel" style="margin-top:1rem">
+                <header>
+                    Members
+                    <span class="subtitle">run in order, lowest first; disabled modules are skipped</span>
+                    <button type="button" class="btn ghost" style="margin-left:auto"
+                            onclick="addMemberRow()">+ add</button>
+                </header>
+                <div class="body tight">
+                    <table class="grid" id="members-table">
+                        <thead>
+                            <tr>
+                                <th style="width:5rem">order</th>
+                                <th>module</th>
+                                <th style="width:5rem"></th>
+                            </tr>
+                        </thead>
+                        <tbody id="members-tbody">
+                            {% for m in members %}
+                            <tr>
+                                <td>
+                                    <input type="number" name="member_run_order"
+                                           value="{{ m.run_order }}" min="0" style="width:4rem">
+                                </td>
+                                <td>
+                                    <select name="member_module_id" style="width:100%">
+                                        {% for mod in all_modules %}
+                                        <option value="{{ mod.id }}"
+                                            {% if mod.id == m.module_id %}selected{% endif %}>
+                                            {{ mod.name }}{% if not mod.enabled %} (disabled){% endif %}
+                                        </option>
+                                        {% endfor %}
+                                    </select>
+                                </td>
+                                <td>
+                                    <button type="button" class="ghost"
+                                            style="color:var(--danger);border:none"
+                                            onclick="this.closest('tr').remove()">remove</button>
+                                </td>
+                            </tr>
+                            {% endfor %}
+                        </tbody>
+                    </table>
+                    {% if not members %}
+                    <div id="members-empty" class="empty" style="margin-top:0.4rem">No members yet.</div>
+                    {% endif %}
+                </div>
+            </div>
+
+            <div class="panel" style="margin-top:1rem">
+                <header>
+                    Schedules
+                    <span class="subtitle">cron expression reference</span>
+                    <button type="button" class="btn ghost" style="margin-left:auto"
+                            onclick="addScheduleRow()">+ add</button>
+                </header>
+                <div class="body tight">
+                    <table class="grid" style="margin-bottom:0.6rem">
+                        <thead>
+                            <tr>
+                                <th>pos 1</th>
+                                <th>pos 2</th>
+                                <th>pos 3</th>
+                                <th>pos 4</th>
+                                <th>pos 5</th>
+                            </tr>
+                        </thead>
+                        <tbody>
+                            <tr>
+                                <td>minute<br>0–59</td>
+                                <td>hour<br>0–23</td>
+                                <td>day of month<br>1–31</td>
+                                <td>month<br>1–12</td>
+                                <td>day of week<br>0=Sun, 6=Sat</td>
+                            </tr>
+                        </tbody>
+                    </table>
+                    <table class="grid" id="sched-table">
+                        <thead>
+                            <tr>
+                                <th>cron expression</th>
+                                <th style="width:6rem">enabled</th>
+                                <th style="width:5rem"></th>
+                            </tr>
+                        </thead>
+                        <tbody id="sched-tbody">
+                            {% for s in schedules %}
+                            <tr>
+                                <input type="hidden" name="sched_id" value="{{ s.id }}">
+                                <td>
+                                    <input type="text" name="sched_cron_expr"
+                                           value="{{ s.cron_expr }}" class="mono"
+                                           style="width:100%" placeholder="0 * * * *">
+                                </td>
+                                <td>
+                                    <select name="sched_enabled" style="width:100%">
+                                        <option value="1" {% if s.enabled %}selected{% endif %}>yes</option>
+                                        <option value="0" {% if not s.enabled %}selected{% endif %}>no</option>
+                                    </select>
+                                </td>
+                                <td>
+                                    <button type="button" class="ghost"
+                                            style="color:var(--danger);border:none"
+                                            onclick="removeScheduleRow(this, '{{ s.id }}')">remove</button>
+                                </td>
+                            </tr>
+                            {% endfor %}
+                        </tbody>
+                    </table>
+                    {% if not schedules %}
+                    <div id="sched-empty" class="empty" style="margin-top:0.4rem">No schedules — group runs manually only.</div>
+                    {% endif %}
+                    <details style="margin-top:0.8rem">
+                        <summary style="cursor:pointer;color:var(--text-muted);font-size:0.85em">cron expression reference</summary>
+                        <div style="margin-top:0.5rem;font-size:0.85em">
+                            <table class="grid" style="font-size:1em;width:auto">
+                                <thead>
+                                    <tr>
+                                        <th class="mono" style="text-align:center">pos 1</th>
+                                        <th class="mono" style="text-align:center">pos 2</th>
+                                        <th class="mono" style="text-align:center">pos 3</th>
+                                        <th class="mono" style="text-align:center">pos 4</th>
+                                        <th class="mono" style="text-align:center">pos 5</th>
+                                    </tr>
+                                </thead>
+                                <tbody>
+                                    <tr>
+                                        <td style="text-align:center">minute<br><span style="color:var(--text-muted);font-size:0.9em">0–59</span></td>
+                                        <td style="text-align:center">hour<br><span style="color:var(--text-muted);font-size:0.9em">0–23</span></td>
+                                        <td style="text-align:center">day of month<br><span style="color:var(--text-muted);font-size:0.9em">1–31</span></td>
+                                        <td style="text-align:center">month<br><span style="color:var(--text-muted);font-size:0.9em">1–12</span></td>
+                                        <td style="text-align:center">day of week<br><span style="color:var(--text-muted);font-size:0.9em">0=Sun, 6=Sat</span></td>
+                                    </tr>
+                                </tbody>
+                            </table>
+                        </div>
+                        <table class="grid" style="margin-top:0.5rem;font-size:0.85em">
+                            <thead><tr><th>expression</th><th>meaning</th></tr></thead>
+                            <tbody>
+                                <tr><td class="mono">0 * * * *</td><td>every hour (on the hour)</td></tr>
+                                <tr><td class="mono">*/15 * * * *</td><td>every 15 minutes</td></tr>
+                                <tr><td class="mono">0 4 * * *</td><td>daily at 04:00</td></tr>
+                                <tr><td class="mono">0 4 * * 1-5</td><td>weekdays at 04:00</td></tr>
+                                <tr><td class="mono">0 4 * * 6,0</td><td>weekends at 04:00</td></tr>
+                                <tr><td class="mono">0 0 * * 1</td><td>every Monday at midnight</td></tr>
+                                <tr><td class="mono">0 6,18 * * *</td><td>twice daily at 06:00 and 18:00</td></tr>
+                                <tr><td class="mono">0 0 1 * *</td><td>first day of each month</td></tr>
+                            </tbody>
+                        </table>
+                        <div style="margin-top:0.4rem;color:var(--text-muted);font-size:0.82em">
+                            all times local &nbsp;·&nbsp; use <code>*</code> for "any", <code>*/n</code> for "every n", <code>a-b</code> for range, <code>a,b</code> for list
+                        </div>
+                    </details>
+                </div>
+            </div>
+
+            <div class="actions" style="justify-content:flex-end;margin-top:0.8rem">
+                <a class="btn ghost" href="{{ cancel_url }}">cancel</a>
+                <button type="submit" class="primary">{% if group %}save changes{% else %}create group{% endif %}</button>
+            </div>
+        </form>
+    </div>
+</div>
+
+<template id="sched-row-template">
+    <tr>
+        <input type="hidden" name="sched_id" value="">
+        <td>
+            <input type="text" name="sched_cron_expr" value="" class="mono"
+                   style="width:100%" placeholder="0 * * * *">
+        </td>
+        <td>
+            <select name="sched_enabled" style="width:100%">
+                <option value="1" selected>yes</option>
+                <option value="0">no</option>
+            </select>
+        </td>
+        <td>
+            <button type="button" class="ghost"
+                    style="color:var(--danger);border:none"
+                    onclick="removeScheduleRow(this, '')">remove</button>
+        </td>
+    </tr>
+</template>
+
+<template id="member-row-template">
+    <tr>
+        <td>
+            <input type="number" name="member_run_order" value="0" min="0" style="width:4rem">
+        </td>
+        <td>
+            <select name="member_module_id" style="width:100%">
+                {% for mod in all_modules %}
+                <option value="{{ mod.id }}">{{ mod.name }}{% if not mod.enabled %} (disabled){% endif %}</option>
+                {% endfor %}
+            </select>
+        </td>
+        <td>
+            <button type="button" class="ghost"
+                    style="color:var(--danger);border:none"
+                    onclick="this.closest('tr').remove()">remove</button>
+        </td>
+    </tr>
+</template>
+
+<script>
+function addScheduleRow() {
+    const tmpl = document.getElementById('sched-row-template');
+    const row = tmpl.content.cloneNode(true);
+    document.getElementById('sched-tbody').appendChild(row);
+    const empty = document.getElementById('sched-empty');
+    if (empty) empty.style.display = 'none';
+}
+
+function removeScheduleRow(btn, schedId) {
+    if (schedId) {
+        const input = document.createElement('input');
+        input.type = 'hidden';
+        input.name = 'sched_deleted_id';
+        input.value = schedId;
+        document.getElementById('group-form').appendChild(input);
+    }
+    btn.closest('tr').remove();
+}
+
+function addMemberRow() {
+    const tmpl = document.getElementById('member-row-template');
+    const row = tmpl.content.cloneNode(true);
+    document.getElementById('members-tbody').appendChild(row);
+    const empty = document.getElementById('members-empty');
+    if (empty) empty.style.display = 'none';
+}
+</script>
+{% endblock %}

pipekit/web/templates/group_run_detail.html

@@ -0,0 +1,69 @@
+{% extends "base.html" %}
+{% set section = "groups" %}
+{% block title %}Group run #{{ group_run.id }} — Pipekit{% endblock %}
+
+{% block content %}
+<div class="panel">
+    <header>
+        Group run #{{ group_run.id }}
+        <span class="subtitle">
+            <a href="/groups/{{ group_run.group_id }}">{{ group_run.group_name }}</a> ·
+            started {{ group_run.started_at | localtime }}
+        </span>
+        <span style="margin-left:auto">
+            <span class="pill {{ group_run.status }}">{{ group_run.status }}</span>
+        </span>
+    </header>
+    <div class="body">
+        <dl class="keyval">
+            <dt>started</dt>    <dd class="mono">{{ group_run.started_at | localtime }}</dd>
+            <dt>finished</dt>   <dd class="mono">{{ group_run.finished_at | localtime }}</dd>
+            <dt>duration</dt>   <dd class="mono">{{ group_run.duration_s | duration }}</dd>
+            <dt>triggered by</dt><dd class="mono">{{ group_run.triggered_by or "—" }}</dd>
+        </dl>
+    </div>
+</div>
+
+<div id="group-run-live"
+     {% if group_run.status == 'running' %}
+     hx-get="/group-runs/{{ group_run.id }}/live"
+     hx-trigger="load, every 3s"
+     hx-swap="outerHTML"
+     {% endif %}>
+    <div class="panel">
+        <header>Module runs</header>
+        <div class="body tight">
+            {% if module_runs %}
+            <table class="grid">
+                <thead>
+                    <tr>
+                        <th>run</th>
+                        <th>module</th>
+                        <th>started</th>
+                        <th>duration</th>
+                        <th>status</th>
+                        <th>rows</th>
+                    </tr>
+                </thead>
+                <tbody>
+                {% for r in module_runs %}
+                    <tr>
+                        <td><a href="/runs/{{ r.id }}">#{{ r.id }}</a></td>
+                        <td><a href="/modules/{{ r.module_id }}">{{ r.module_name }}</a></td>
+                        <td class="mono">{{ r.started_at | localtime }}</td>
+                        <td class="mono">{{ r.duration_s | duration }}</td>
+                        <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
+                        <td class="mono">{{ r.row_count if r.row_count is not none else "—" }}</td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+            {% elif group_run.status == 'running' %}
+            <div class="empty">Waiting for module runs to start…</div>
+            {% else %}
+            <div class="empty">No module runs recorded.</div>
+            {% endif %}
+        </div>
+    </div>
+</div>
+{% endblock %}

pipekit/web/templates/groups.html

@@ -0,0 +1,60 @@
+{% extends "base.html" %}
+{% set section = "groups" %}
+{% block title %}Groups — Pipekit{% endblock %}
+
+{% block content %}
+<div class="panel">
+    <header>
+        Groups
+        <span class="subtitle">{{ groups|length }} total</span>
+        <span style="margin-left:auto">
+            <a class="btn" href="/groups/new">New group…</a>
+        </span>
+    </header>
+    <div class="body tight">
+        {% if groups %}
+        <table class="grid">
+            <thead>
+                <tr>
+                    <th>name</th>
+                    <th>members</th>
+                    <th>last run</th>
+                    <th style="width:9em">status</th>
+                    <th></th>
+                </tr>
+            </thead>
+            <tbody>
+            {% for g in groups %}
+                <tr>
+                    <td><a href="/groups/{{ g.id }}"><strong>{{ g.name }}</strong></a></td>
+                    <td class="mono">{{ g.member_count }}</td>
+                    <td class="mono">{{ g.last_run_at | localtime }}</td>
+                    <td>
+                        {% if g.last_status %}
+                        <span class="pill {{ g.last_status }}">{{ g.last_status }}</span>
+                        {% else %}
+                        <span class="pill">never ran</span>
+                        {% endif %}
+                    </td>
+                    <td style="text-align:right">
+                        <form class="inline" method="post" action="/groups/{{ g.id }}/run">
+                            <button type="submit">Run</button>
+                        </form>
+                        <form class="inline" method="post" action="/groups/{{ g.id }}/run">
+                            <input type="hidden" name="dry_run" value="1">
+                            <button type="submit" class="ghost">Dry run</button>
+                        </form>
+                    </td>
+                </tr>
+            {% endfor %}
+            </tbody>
+        </table>
+        {% else %}
+        <div class="empty">
+            No groups yet.<br>
+            <a class="btn" href="/groups/new" style="margin-top:0.7rem; display:inline-block">Create one</a>
+        </div>
+        {% endif %}
+    </div>
+</div>
+{% endblock %}

pipekit/web/templates/login.html

@@ -0,0 +1,28 @@
+{% extends "base.html" %}
+{% block title %}Login — Pipekit{% endblock %}
+
+{% block content %}
+<div style="max-width:22rem;margin:4rem auto">
+    <div class="panel">
+        <header>Login</header>
+        <div class="body">
+            {% if error %}
+            <div class="flash err" style="margin-bottom:0.8rem">{{ error }}</div>
+            {% endif %}
+            <form method="post" action="/login?next={{ next | urlencode }}">
+                <label class="field" style="grid-template-columns:1fr">
+                    <span>username</span>
+                    <input type="text" name="username" required autofocus>
+                </label>
+                <label class="field" style="grid-template-columns:1fr;margin-top:0.5rem">
+                    <span>password</span>
+                    <input type="password" name="password" required>
+                </label>
+                <div class="actions" style="justify-content:flex-end;margin-top:1rem">
+                    <button type="submit" class="primary">Sign in</button>
+                </div>
+            </form>
+        </div>
+    </div>
+</div>
+{% endblock %}

pipekit/web/templates/module_detail.html

@@ -8,14 +8,22 @@
         {{ module.name }}
         <span class="subtitle">
             module #{{ module.id }}
+            <span id="module-pills">
                 {% if module.running %}<span class="pill running">running</span>{% endif %}
                 {% if not module.enabled %}<span class="pill disabled">disabled</span>{% endif %}
             </span>
+        </span>
         <span style="margin-left:auto" class="actions">
-            <form class="inline" method="post" action="/modules/{{ module.id }}/run">
+            <form class="inline"
+                  hx-post="/modules/{{ module.id }}/run"
+                  hx-target="#module-live"
+                  hx-swap="outerHTML">
                 <button class="primary" type="submit">Run now</button>
             </form>
-            <form class="inline" method="post" action="/modules/{{ module.id }}/run">
+            <form class="inline"
+                  hx-post="/modules/{{ module.id }}/run"
+                  hx-target="#module-live"
+                  hx-swap="outerHTML">
                 <input type="hidden" name="dry_run" value="1">
                 <button type="submit">Dry run</button>
             </form>
@@ -179,31 +187,7 @@
             </div>
         </div>
 
-        <div class="panel">
-            <header>Recent runs
-                <span class="subtitle">last {{ recent_runs|length }}</span>
-                <span style="margin-left:auto"><a href="/runs?module_id={{ module.id }}">all →</a></span>
-            </header>
-            <div class="body tight">
-                {% if recent_runs %}
-                <table class="grid">
-                    <thead><tr><th>id</th><th>started</th><th>status</th><th>rows</th></tr></thead>
-                    <tbody>
-                    {% for r in recent_runs %}
-                        <tr>
-                            <td><a href="/runs/{{ r.id }}">#{{ r.id }}</a></td>
-                            <td class="mono">{{ r.started_at }}</td>
-                            <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
-                            <td class="mono">{{ r.row_count if r.row_count is not none else "—" }}</td>
-                        </tr>
-                    {% endfor %}
-                    </tbody>
-                </table>
-                {% else %}
-                <div class="empty">No runs yet.</div>
-                {% endif %}
-            </div>
-        </div>
+        {% include "_module_live.html" %}
     </div>
 </div>
 {% endblock %}

pipekit/web/templates/module_form.html

@@ -19,7 +19,7 @@
         </div>
         {% endif %}
 
-        <form method="post" action="{{ form_action }}">
+        <form method="post" action="{{ form_action }}" id="module-form">
             <label class="field">
                 <span>name</span>
                 <input type="text" name="name" required value="{{ module.name }}">
@@ -70,29 +70,92 @@
 
             <label class="field">
                 <span>source query</span>
-                <textarea name="source_query" rows="14" required class="mono">{{ module.source_query }}</textarea>
+                <textarea name="source_query" id="source-query" rows="14" required class="mono"
+                          oninput="checkPlaceholders()">{{ module.source_query }}</textarea>
                 <span class="help">free text; <code>{name}</code> placeholders resolved from watermarks at run time</span>
             </label>
+            <div id="placeholder-warning" class="flash warn" style="display:none;margin-top:0.3rem"></div>
 
-            <div class="two-col" style="gap:1rem">
+            <div class="panel" style="margin-top:1rem">
+                <header>Merge &amp; watermarks</header>
+                <div class="body">
+                    <div class="two-col" style="gap:1rem;margin-bottom:0.8rem">
                         <label class="field">
                             <span>merge strategy</span>
-                    <select name="merge_strategy" required>
+                            <select name="merge_strategy" id="merge-strategy" required
+                                    onchange="onStrategyChange(this.value)">
                                 {% for s in ("full", "incremental", "append") %}
                                 <option value="{{ s }}" {% if s == module.merge_strategy %}selected{% endif %}>{{ s }}</option>
                                 {% endfor %}
                             </select>
                         </label>
 
-                <label class="field">
+                        <label class="field" id="merge-key-field"
+                               style="{{ '' if module.merge_strategy == 'incremental' else 'display:none' }}">
                             <span>merge key</span>
                             <input type="text" name="merge_key" value="{{ module.merge_key or '' }}"
-                           placeholder="id  or  (col_a, col_b)">
-                    <span class="help">required for <code>incremental</code>; ignored otherwise</span>
+                                   placeholder="id  or  id,version">
+                            <span class="help">column name(s) for the DELETE predicate</span>
                         </label>
                     </div>
 
-            <label class="field">
+                    <div id="watermarks-section"
+                         style="{{ '' if module.merge_strategy == 'incremental' else 'display:none' }}">
+                        <div style="display:flex;align-items:center;gap:0.5rem;margin-bottom:0.5rem">
+                            <strong>Watermarks</strong>
+                            <span class="help" style="margin:0">— resolver SQL runs before each sync; result substituted as <code>{name}</code> in source query</span>
+                            <button type="button" class="btn ghost" style="margin-left:auto"
+                                    onclick="addWatermarkRow()">+ add</button>
+                        </div>
+                        <table class="grid" id="wm-table">
+                            <thead>
+                                <tr>
+                                    <th>name</th>
+                                    <th>resolver connection</th>
+                                    <th>resolver SQL</th>
+                                    <th>default value</th>
+                                    <th></th>
+                                </tr>
+                            </thead>
+                            <tbody id="wm-tbody">
+                                {% for w in watermarks %}
+                                <tr>
+                                    <input type="hidden" name="wm_id" value="{{ w.id }}">
+                                    <td><input type="text" name="wm_name" value="{{ w.name }}"
+                                               class="mono" style="width:100%" oninput="checkPlaceholders()"></td>
+                                    <td>
+                                        <select name="wm_connection_id" style="width:100%">
+                                            {% for c in connections %}
+                                            <option value="{{ c.id }}"
+                                                {% if c.id == w.connection_id %}selected{% endif %}>
+                                                {{ c.name }}
+                                            </option>
+                                            {% endfor %}
+                                        </select>
+                                    </td>
+                                    <td><textarea name="wm_resolver_sql" rows="2" class="mono"
+                                                  style="width:100%;min-width:16rem">{{ w.resolver_sql }}</textarea></td>
+                                    <td><input type="text" name="wm_default_value"
+                                               value="{{ w.default_value or '' }}"
+                                               placeholder="(fail if null)"
+                                               style="width:100%"></td>
+                                    <td>
+                                        <button type="button" class="ghost"
+                                                style="color:var(--danger);border:none"
+                                                onclick="removeWatermarkRow(this, '{{ w.id }}')">remove</button>
+                                    </td>
+                                </tr>
+                                {% endfor %}
+                            </tbody>
+                        </table>
+                        {% if not watermarks %}
+                        <div id="wm-empty" class="empty" style="margin-top:0.4rem">No watermarks — add one to enable incremental filtering.</div>
+                        {% endif %}
+                    </div>
+                </div>
+            </div>
+
+            <label class="field" style="margin-top:1rem">
                 <span>dest description</span>
                 <textarea name="dest_description" rows="2">{{ module.dest_description or '' }}</textarea>
                 <span class="help">COMMENT ON TABLE value; re-applied on save if changed</span>
@@ -112,4 +175,81 @@
         </form>
     </div>
 </div>
+
+<template id="wm-row-template">
+    <tr>
+        <input type="hidden" name="wm_id" value="">
+        <td><input type="text" name="wm_name" value="" class="mono" style="width:100%"
+                   oninput="checkPlaceholders()"></td>
+        <td>
+            <select name="wm_connection_id" style="width:100%">
+                {% for c in connections %}
+                <option value="{{ c.id }}">{{ c.name }}</option>
+                {% endfor %}
+            </select>
+        </td>
+        <td><textarea name="wm_resolver_sql" rows="2" class="mono"
+                      style="width:100%;min-width:16rem"></textarea></td>
+        <td><input type="text" name="wm_default_value" placeholder="(fail if null)"
+                   style="width:100%"></td>
+        <td>
+            <button type="button" class="ghost" style="color:var(--danger);border:none"
+                    onclick="removeWatermarkRow(this, '')">remove</button>
+        </td>
+    </tr>
+</template>
+
+<script>
+function onStrategyChange(val) {
+    document.getElementById('merge-key-field').style.display = val === 'incremental' ? '' : 'none';
+    document.getElementById('watermarks-section').style.display = val === 'incremental' ? '' : 'none';
+    checkPlaceholders();
+}
+
+function addWatermarkRow() {
+    const tmpl = document.getElementById('wm-row-template');
+    const row = tmpl.content.cloneNode(true);
+    document.getElementById('wm-tbody').appendChild(row);
+    const empty = document.getElementById('wm-empty');
+    if (empty) empty.style.display = 'none';
+}
+
+function removeWatermarkRow(btn, wmId) {
+    if (wmId) {
+        const input = document.createElement('input');
+        input.type = 'hidden';
+        input.name = 'wm_deleted_id';
+        input.value = wmId;
+        document.getElementById('module-form').appendChild(input);
+    }
+    btn.closest('tr').remove();
+    checkPlaceholders();
+}
+
+function checkPlaceholders() {
+    const strategy = document.getElementById('merge-strategy').value;
+    const warn = document.getElementById('placeholder-warning');
+    if (strategy !== 'incremental') { warn.style.display = 'none'; return; }
+
+    const query = document.getElementById('source-query').value;
+    const inQuery = new Set([...query.matchAll(/\{(\w+)\}/g)].map(m => m[1]));
+    const inWm = new Set([...document.querySelectorAll('#wm-tbody [name="wm_name"]')]
+                          .map(el => el.value.trim()).filter(Boolean));
+
+    const noWm = [...inQuery].filter(n => !inWm.has(n));
+    const noPlaceholder = [...inWm].filter(n => !inQuery.has(n));
+    const msgs = [];
+    if (noWm.length) msgs.push('Placeholder(s) in query with no watermark: ' + noWm.map(n => '{'+n+'}').join(', '));
+    if (noPlaceholder.length) msgs.push('Watermark(s) not used in query: ' + noPlaceholder.map(n => '{'+n+'}').join(', '));
+
+    if (msgs.length) {
+        warn.textContent = msgs.join(' · ');
+        warn.style.display = '';
+    } else {
+        warn.style.display = 'none';
+    }
+}
+
+document.addEventListener('DOMContentLoaded', checkPlaceholders);
+</script>
 {% endblock %}

pipekit/web/templates/modules_index.html

@@ -18,13 +18,14 @@
                 <table class="grid">
                     <thead>
                         <tr>
-                            <th style="width:30%">name</th>
-                            <th>strategy</th>
-                            <th>dest</th>
-                            <th>last run</th>
-                            <th style="width:9em">status</th>
-                            <th style="width:7em">rows</th>
-                            <th></th>
+                            <th style="width:12em">name</th>
+                            <th style="width:7em">strategy</th>
+                            <th style="width:14em">dest</th>
+                            <th style="width:13em">groups</th>
+                            <th style="width:11em;white-space:nowrap">last run</th>
+                            <th style="width:8em">status</th>
+                            <th style="width:6em">rows</th>
+                            <th style="width:12em"></th>
                         </tr>
                     </thead>
                     <tbody>
@@ -33,24 +34,25 @@
                             <td><a href="/modules/{{ m.id }}"><strong>{{ m.name }}</strong></a></td>
                             <td class="mono">{{ m.merge_strategy }}</td>
                             <td class="mono">{{ m.dest_table }}</td>
-                            <td class="mono">{{ m.last_run_at or "—" }}</td>
                             <td>
-                                {% if m.running %}
-                                    <span class="pill running">running</span>
-                                {% elif not m.enabled %}
-                                    <span class="pill disabled">disabled</span>
-                                {% elif m.last_status %}
-                                    <span class="pill {{ m.last_status }}">{{ m.last_status }}</span>
-                                {% else %}
-                                    <span class="pill">never ran</span>
-                                {% endif %}
+                                {% for g in m.groups %}
+                                <a href="/groups/{{ g.group_id }}" class="tag">{{ g.group_name }}</a>
+                                {% endfor %}
                             </td>
+                            <td class="mono" style="white-space:nowrap">{{ m.last_run_at | localtime }}</td>
+                            {% with module=m %}{% include "_module_status_pill.html" %}{% endwith %}
                             <td class="mono">{{ m.last_row_count if m.last_row_count is not none else "—" }}</td>
                             <td style="text-align:right">
-                                <form class="inline" method="post" action="/modules/{{ m.id }}/run">
+                                <form class="inline"
+                                      hx-post="/modules/{{ m.id }}/run"
+                                      hx-target="#module-status-{{ m.id }}"
+                                      hx-swap="outerHTML">
                                     <button type="submit">Run</button>
                                 </form>
-                                <form class="inline" method="post" action="/modules/{{ m.id }}/run">
+                                <form class="inline"
+                                      hx-post="/modules/{{ m.id }}/run"
+                                      hx-target="#module-status-{{ m.id }}"
+                                      hx-swap="outerHTML">
                                     <input type="hidden" name="dry_run" value="1">
                                     <button type="submit" class="ghost">Dry run</button>
                                 </form>

pipekit/web/templates/run_detail.html

@@ -8,14 +8,14 @@
         Run #{{ run.id }}
         <span class="subtitle">
             <a href="/modules/{{ run.module_id }}">{{ run.module_name }}</a> ·
-            started {{ run.started_at }}
+            started {{ run.started_at | localtime }}
         </span>
         <span style="margin-left:auto"><span class="pill {{ run.status }}">{{ run.status }}</span></span>
     </header>
     <div class="body">
         <dl class="keyval">
-            <dt>started</dt>   <dd class="mono">{{ run.started_at }}</dd>
-            <dt>finished</dt>  <dd class="mono">{{ run.finished_at or '—' }}</dd>
+            <dt>started</dt>   <dd class="mono">{{ run.started_at | localtime }}</dd>
+            <dt>finished</dt>  <dd class="mono">{{ run.finished_at | localtime }}</dd>
             <dt>rows</dt>      <dd class="mono">{{ run.row_count if run.row_count is not none else '—' }}</dd>
             <dt>watermarks</dt><dd class="mono">{{ run.watermark_values_json or '—' }}</dd>
             {% if run.error %}<dt>error</dt><dd class="mono" style="color:var(--danger)">{{ run.error }}</dd>{% endif %}
@@ -37,6 +37,33 @@
 </div>
 {% endif %}
 
+<div id="run-live"
+  {% if run.status == 'running' %}
+  hx-get="/runs/{{ run.id }}/live"
+  hx-trigger="load, every 2s"
+  hx-swap="outerHTML"
+  {% endif %}
+>
+  <div class="panel">
+    <header>
+      Progress
+      <span style="margin-left:auto">
+        <span class="pill {{ run.status }}">{{ run.status }}</span>
+        {% if run.row_count is not none %}&nbsp;· {{ run.row_count }} rows{% endif %}
+      </span>
+    </header>
+    <div class="body">
+      {% if run.live_log %}
+      <pre style="margin:0;white-space:pre-wrap">{{ run.live_log }}</pre>
+      {% elif run.status == 'running' %}
+      <span style="color:var(--text-muted)">Waiting for output…</span>
+      {% else %}
+      <span style="color:var(--text-muted)">—</span>
+      {% endif %}
+    </div>
+  </div>
+</div>
+
 {% if run.jrunner_stdout or run.jrunner_stderr %}
 <div class="panel">
     <header>jrunner output</header>

pipekit/web/templates/runs.html

@@ -22,7 +22,7 @@
                     <th style="width:5em">id</th>
                     <th>module</th>
                     <th>started</th>
-                    <th>finished</th>
+                    <th>duration</th>
                     <th style="width:8em">status</th>
                     <th style="width:7em">rows</th>
                     <th>error</th>
@@ -33,8 +33,8 @@
                 <tr>
                     <td><a href="/runs/{{ r.id }}">#{{ r.id }}</a></td>
                     <td><a href="/modules/{{ r.module_id }}">{{ r.module_name }}</a></td>
-                    <td class="mono">{{ r.started_at }}</td>
-                    <td class="mono">{{ r.finished_at or '—' }}</td>
+                    <td class="mono">{{ r.started_at | localtime }}</td>
+                    <td class="mono">{{ r.duration_s | duration }}</td>
                     <td><span class="pill {{ r.status }}">{{ r.status }}</span></td>
                     <td class="mono">{{ r.row_count if r.row_count is not none else "—" }}</td>
                     <td class="mono" style="max-width:22rem;overflow:hidden;text-overflow:ellipsis;white-space:nowrap">{{ r.error or '' }}</td>

pipekit/web/templates/wizard_step3.html

@@ -62,10 +62,11 @@
                     </thead>
                     <tbody>
                     {% for c in columns %}
-                        <tr onclick="var cb=document.getElementById('col-{{ loop.index }}'); if(event.target.tagName!=='INPUT') cb.checked=!cb.checked">
+                        <tr onclick="var cb=document.getElementById('col-{{ loop.index }}'); if(event.target.tagName!=='INPUT') cb.checked=!cb.checked; fillQuery()">
                             <td class="pick">
                                 <input type="checkbox" id="col-{{ loop.index }}"
-                                       class="col-check" name="col" value="{{ c.name }}" checked>
+                                       class="col-check" name="col" value="{{ c.name }}" checked
+                                       onchange="fillQuery()">
                             </td>
                             <td class="mono">{{ c.position }}</td>
                             <td class="mono">{{ c.name }}</td>
@@ -150,7 +151,7 @@
                     <label class="field">
                         <span>strategy</span>
                         <select name="merge_strategy" id="merge_strategy"
-                                onchange="document.getElementById('mkf').style.display = this.value==='incremental' ? '' : 'none'">
+                                onchange="onStrategyChange(this.value)">
                             <option value="full">full (truncate + insert)</option>
                             <option value="incremental">incremental (delete by key + insert)</option>
                             <option value="append">append (insert only)</option>
@@ -164,6 +165,46 @@
                 </div>
             </div>
 
+            <div class="panel" id="wm-panel" style="display:none">
+                <header>
+                    Watermarks
+                    <button type="button" class="btn ghost" style="margin-left:auto"
+                            onclick="addWatermarkRow()">+ add</button>
+                </header>
+                <div class="body">
+                    <p class="help" style="margin-bottom:0.6rem">
+                        Add a watermark for each <code>{placeholder}</code> you plan to use in the source query's WHERE clause.
+                        Edit the query below to add the filter after choosing columns.
+                    </p>
+                    <table class="grid" id="wm-table" style="display:none">
+                        <thead>
+                            <tr>
+                                <th>name</th>
+                                <th>resolver connection</th>
+                                <th>resolver SQL</th>
+                                <th>default value</th>
+                                <th></th>
+                            </tr>
+                        </thead>
+                        <tbody id="wm-tbody"></tbody>
+                    </table>
+                    <div id="wm-empty" class="empty">No watermarks added yet.</div>
+                </div>
+            </div>
+
+            <div class="panel" id="query-panel" style="display:none">
+                <header>Source query
+                    <span class="subtitle">auto-generated from picks — edit to add WHERE clause</span>
+                </header>
+                <div class="body">
+                    <textarea name="source_query" id="source-query" rows="10" class="mono"
+                              style="width:100%" oninput="checkPlaceholders()"
+                              placeholder="Leave blank to auto-generate from column picks"></textarea>
+                    <div id="placeholder-warning" class="flash warn" style="display:none;margin-top:0.4rem"></div>
+                    <span class="help">Leave blank to auto-generate. Add <code>WHERE col &gt; {name}</code> for incremental filtering.</span>
+                </div>
+            </div>
+
             <div class="panel">
                 <header>Create</header>
                 <div class="body" style="display:flex;justify-content:flex-end;gap:0.5rem">
@@ -175,9 +216,101 @@
     </div>
 </form>
 
+<template id="wm-row-template">
+    <tr>
+        <input type="hidden" name="wm_id" value="">
+        <td><input type="text" name="wm_name" value="" class="mono" style="width:100%"
+                   oninput="checkPlaceholders()"></td>
+        <td>
+            <select name="wm_connection_id" style="width:100%">
+                {% for c in all_connections %}
+                <option value="{{ c.id }}">{{ c.name }}</option>
+                {% endfor %}
+            </select>
+        </td>
+        <td><textarea name="wm_resolver_sql" rows="2" class="mono"
+                      style="width:100%;min-width:12rem"></textarea></td>
+        <td><input type="text" name="wm_default_value" placeholder="(fail if null)"
+                   style="width:6rem"></td>
+        <td>
+            <button type="button" class="ghost" style="color:var(--danger);border:none"
+                    onclick="removeWatermarkRow(this)">remove</button>
+        </td>
+    </tr>
+</template>
+
 <script>
+    const QUALIFIED_TABLE = {{ qualified_table | tojson }};
+
     function toggleAll(val) {
         document.querySelectorAll('.col-check').forEach(function (cb) { cb.checked = val; });
+        fillQuery();
+    }
+
+    function onStrategyChange(val) {
+        document.getElementById('mkf').style.display = val === 'incremental' ? '' : 'none';
+        document.getElementById('wm-panel').style.display = val === 'incremental' ? '' : 'none';
+        document.getElementById('query-panel').style.display = val === 'incremental' ? '' : 'none';
+        if (val === 'incremental') fillQuery();
+        checkPlaceholders();
+    }
+
+    function fillQuery() {
+        const ta = document.getElementById('source-query');
+        if (!ta) return;
+        const checked = [...document.querySelectorAll('.col-check:checked')];
+        if (!checked.length) { ta.value = ''; return; }
+        const cols = checked.map(cb => {
+            const name = cb.value;
+            const destInput = document.querySelector('[name="dest_name__' + name + '"]');
+            const dest = destInput ? destInput.value.trim() : name.toLowerCase();
+            return '  ' + name + ' AS ' + dest;
+        });
+        ta.value = 'SELECT\n' + cols.join(',\n') + '\nFROM ' + QUALIFIED_TABLE;
+        checkPlaceholders();
+    }
+
+    function addWatermarkRow() {
+        const tmpl = document.getElementById('wm-row-template');
+        const row = tmpl.content.cloneNode(true);
+        document.getElementById('wm-tbody').appendChild(row);
+        document.getElementById('wm-table').style.display = '';
+        document.getElementById('wm-empty').style.display = 'none';
+    }
+
+    function removeWatermarkRow(btn) {
+        btn.closest('tr').remove();
+        const tbody = document.getElementById('wm-tbody');
+        if (!tbody.querySelector('tr')) {
+            document.getElementById('wm-table').style.display = 'none';
+            document.getElementById('wm-empty').style.display = '';
+        }
+        checkPlaceholders();
+    }
+
+    function checkPlaceholders() {
+        const strategy = document.getElementById('merge_strategy').value;
+        const warn = document.getElementById('placeholder-warning');
+        if (!warn) return;
+        if (strategy !== 'incremental') { warn.style.display = 'none'; return; }
+
+        const query = (document.getElementById('source-query') || {}).value || '';
+        const inQuery = new Set([...query.matchAll(/\{(\w+)\}/g)].map(m => m[1]));
+        const inWm = new Set([...document.querySelectorAll('#wm-tbody [name="wm_name"]')]
+                              .map(el => el.value.trim()).filter(Boolean));
+
+        const noWm = [...inQuery].filter(n => !inWm.has(n));
+        const noPlaceholder = [...inWm].filter(n => !inQuery.has(n));
+        const msgs = [];
+        if (noWm.length) msgs.push('Placeholder(s) in query with no watermark: ' + noWm.map(n => '{'+n+'}').join(', '));
+        if (noPlaceholder.length) msgs.push('Watermark(s) not used in query: ' + noPlaceholder.map(n => '{'+n+'}').join(', '));
+
+        if (msgs.length) {
+            warn.textContent = msgs.join(' · ');
+            warn.style.display = '';
+        } else {
+            warn.style.display = 'none';
+        }
     }
 </script>
 {% endif %}

requirements.txt

@@ -1,6 +1,8 @@
 fastapi>=0.115
+itsdangerous>=2.1
 uvicorn[standard]>=0.30
 python-multipart>=0.0.20
 jinja2>=3.1
 pyyaml>=6.0
 httpx>=0.27
+croniter>=2.0

systemd/pipekit.service

@@ -1,15 +1,3 @@
-# Pipekit systemd unit template.
-#
-# Install:
-#   sudo cp pipekit.service /etc/systemd/system/
-#   sudo systemctl daemon-reload
-#   sudo systemctl enable --now pipekit
-#
-# Runs as root by default. For a dedicated service account, create the
-# user and uncomment User=/Group= below:
-#   sudo useradd --system --home-dir /opt/pipekit --shell /usr/sbin/nologin pipekit
-#   sudo chown -R pipekit:pipekit /opt/pipekit/pipekit.db /etc/pipekit
-
 [Unit]
 Description=Pipekit sync engine
 After=network-online.target
@@ -17,11 +5,11 @@ Wants=network-online.target
 
 [Service]
 Type=simple
-# User=pipekit
-# Group=pipekit
+User=pipekit
+Group=pipekit
 WorkingDirectory=/opt/pipekit
 EnvironmentFile=/etc/pipekit/secrets.env
-ExecStart=/usr/local/bin/pipekit serve --host 0.0.0.0
+ExecStart=/usr/local/bin/pipekit serve
 Restart=on-failure
 RestartSec=5s