pipekit/SPEC.md

# Pipekit — Spec

This spec was built from a clean-slate conversation that rederived the
design from first principles. The previous version is archived at
`SPEC_v1_archive.md` for reference.

## Status

**Implementation is underway.** Core module sync, wizard, web UI, groups,
scheduling, and deploy tooling are built and running. The TUI was dropped
in favour of a web UI (see Architecture note below).

One item is intentionally deferred: the **migration plan** for bringing
over the ~90 existing modules from `/opt/sync`. Not needed to start
implementation — Pipekit can be built and tested against new modules
first, and migration can happen later (likely via a parser that walks
`/opt/sync/*/`, extracts `pull.sql` / `insert.sql` / shell wrapper,
infers merge strategy and key, and creates module rows).

## How we got here

Started by asking what was painful about the existing shell-script-based
sync setup. Three things surfaced: authoring new modules is tedious,
observability is poor (no easy way to see what ran, how long, how many
rows, any errors), and there's no central management UI. That framed
Pipekit as an orchestration layer on top of the existing `jrunner` JDBC
tool — not replacing jrunner, wrapping it with the state and UI that
shell scripts can't provide.

Everything in this document was worked out by walking through concrete
examples from the current `/opt/sync` modules (`code`, `qcrh`,
`ffsbglr1`) and asking "what would this look like under the new system?"
When the original spec proposed something that didn't fit (like
"watermark is a single column name"), we redesigned it. The result is a
spec that reflects the actual complexity of real modules, not an
idealized simple-sync model.

---

## Motivation

User has ~90 sync modules in `/opt/sync` today, organized as shell scripts
that wrap `jrunner` (a JDBC bulk-transfer CLI at `/opt/jrunner`). Pain points
that drove this redesign:

- **Authoring is tedious.** Building SQL for new sync modules takes too long —
  hand-writing pull.sql, insert.sql, the .sh wrapper, the import table DDL.
- **No observability.** Hard to answer: how often does each module run, how
  many rows transfer, what SQL was used, when's the next run, how long does
  it take, are tables in a good state, were there errors on the last run and
  for which modules.
- **No central management.** Want a TUI like lazygit for browsing, inspecting,
  running, configuring modules. User browses with nvim today and wants the TUI
  to feel as spatial and navigable as a file tree.

## What jrunner does (and doesn't)

`jrunner` (at `/opt/jrunner`) is a Java CLI that does two things:

1. **Migration mode** — given source connection (`-scu/-scn/-scp`), dest
   connection (`-dcu/-dcn/-dcp`), a SQL file (`-sq`), and a dest table (`-dt`),
   it streams rows from source to dest with batched INSERTs.
2. **Query mode** — same source flags but no dest flags, outputs query results
   to stdout in CSV/TSV. Useful for piping to visidata, less, etc.

It has no merge logic, no scheduling, no state, no awareness of incremental
syncs. It's a dumb pipe. That's the right shape — Pipekit wraps it with the
orchestration layer.

## Architecture

```
jrunner   (Java CLI — bulk JDBC transfer + query mode)
   ↑
engine    (Python — orchestrates jrunner, watermarks, merge, hooks, run log)
   ↑
API process  (FastAPI — REST, Basic Auth)
   ├── scheduler thread  (daemon thread; fires group runs from cron schedules)
   └── web UI            (Jinja2 templates, HTMX, vanilla JS — served alongside /api/*)
   ↑
TUI (deferred) / curl
```

The engine shells out to jrunner for **everything that touches a database** —
bulk transfers, watermark resolver queries, hooks. No separate JDBC layer in
Python. One driver-loading code path, one set of bugs.

The scheduler is a daemon thread started inside the API process via FastAPI
lifespan — not a separate process or service. It shares the same SQLite
connection pool and imports.

The FastAPI app serves both a JSON REST API (`/api/*`) and full HTML pages
(`/`, `/modules/*`, `/groups/*`, etc.) using Jinja2 templates, HTMX for
in-place updates, and vanilla JS. The web UI covers all management: modules,
connections, groups, schedules, run history.

**TUI is deferred, not dropped.** The original TUI design remains valid — it
would be an HTTP client against the API, never touching SQLite directly. Build
it when the web UI's limitations become friction (keyboard-heavy workflows,
SSH-only environments). The API design already accommodates it.

## Storage: SQLite

Everything lives in one SQLite file (`pipekit.db`). Why:

- ~90+ modules already exists; flat files don't scale to "show me all modules
  that errored last night" type queries.
- The SQL itself belongs in the database, not as file references — a module is
  a self-contained unit and splitting it across rows + files means two things
  to keep in sync.
- Single file, copy with `cp`, no server. Schema translates to PostgreSQL later
  if ever needed.

User was uneasy about losing filesystem browsing. Resolution: the **TUI is the
file browser**. Inspecting a module feels like `cat`, editing opens `$EDITOR`,
the module list feels like `ls`. For raw access, `sqlite3 pipekit.db` works.

## Module model

A module = one sync job. Fields:

- `name`
- `source_connection_id`, `dest_connection_id`
- `dest_table`
- `staging_table` (auto-managed: `pipekit_staging.{name}`)
- `source_query` — full SQL text with `{watermark_name}` placeholders. Free-form.
- `merge_strategy` — `full` / `incremental` / `append`
- `merge_key` — destination column(s) used in DELETE before INSERT
- `enabled`
- `running` (lock flag — see locking section)

The source query is **a text blob**. Not split into structured columns. The
TUI offers a column-editor mode that *parses* the SELECT list out of the
stored query, lets you edit it as a table, and *splices the new SELECT list
back in* (preserving CTEs, FROM, WHERE). For queries the parser can't handle
(too complex), the TUI falls back to raw `$EDITOR`. **Raw editing always
works.**

### Merge strategies

Two patterns from existing scripts:

- **full** — TRUNCATE dest, INSERT all from staging
- **incremental** — pull delta via watermark, DELETE rows in dest matching
  merge_key, INSERT from staging
- **append** — INSERT only, no deletes

**No upsert.** The DELETE+INSERT approach already handles row-level changes
without needing column-by-column ON CONFLICT UPDATE SET clauses.

### Watermarks (multi, type-agnostic, resolver SQL)

A module can have **multiple named watermarks**. Real example from user: a
query that needs both `{date}` (max modified-timestamp from one table) and
`{number}` (max order number from another) to build a list of changed orders
to repull.

A watermark =

- `name` — placeholder name in the source query
- `connection_id` — which connection runs the resolver (could be dest, source,
  or a third)
- `resolver_sql` — free-form SQL. Engine runs it via jrunner query mode, takes
  first row's first column as a string.
- `default_value` — used if resolver returns NULL or zero rows

**Type-agnostic.** The engine reads the resolver result as an opaque string and
substitutes it literally. No type coercion. The user controls quoting in the
resolver SQL itself (e.g. wrap in `quote_literal()` if you want `'2610'`,
return raw if you want `2610`).

**Dialect-aware by user.** The user writes the resolver in the connection's
dialect. Engine doesn't translate. Same as today — they already write DB2 in
pull.sql and PG in insert.sql.

**No hidden generation.** Resolved SQL gets **materialized** before each run
and stored on the module record (`next_resolved_query` or similar) so the TUI
can always show "here's exactly what would run next." After the run, the
exact resolved SQL goes into the run_log.

### Hooks

A module can have post-execution hooks for things like
`REFRESH MATERIALIZED VIEW rlarp.cust` or `CALL rlarp.osm_stack_refresh()`.

A hook =

- `module_id`, `run_order`
- `connection_id` — usually dest, but anywhere
- `sql`
- `run_on` — `success` / `failure` / `always`

Hooks run sequentially after the merge. Failures get logged but don't roll
back the merge (it's already committed).

**No group-level hooks for now.** Decision deferred. The `REFRESH MATERIALIZED
VIEW rlarp.cust` at the end of `codes.sh` would attach to whichever module
logically owns that data, even if not strictly the last in order. Add group
hooks later if it gets painful.

## Engine flow (per module run)

1. **Acquire lock** atomically: `UPDATE module SET running=1 WHERE id=? AND running=0`. If row count is 0, bail with "already running."
2. **Resolve watermarks.** For each watermark: shell out to jrunner query mode against the watermark's connection with its resolver SQL. Take first row's first column as a string. Fall back to `default_value` on NULL/empty. Always runs live — even for dry runs.
3. **Materialize the resolved source query.** Substitute `{name}` placeholders in `source_query`. Store on the module record so the TUI can preview.
4. **Materialize the merge SQL** based on strategy + merge_key.
5. **Dry-run exit.** If `dry_run=True`, write `run_log` with status `dry_run` and return — no data movement.
6. **Recreate staging** (DROP + CREATE … LIKE dest). Self-healing against schema drift.
7. **Run jrunner** (migration mode) with the resolved query, target = staging.
8. **Run merge** against dest connection via jrunner.
9. **Run hooks** in order, respecting `run_on`.
10. **Write `run_log` entry** with everything (see below).
11. **Release lock** in a `finally` block — always runs, even on error.

## Locking

The `running` flag on the module is the lock. The atomic UPDATE-with-WHERE
above ensures no race window. Belt-and-suspenders for stuck locks:

- **PID-based.** Store the API process PID/UUID on the lock. On API startup,
  clear locks owned by PIDs that no longer exist.
- **Time-based backstop.** On startup, also clear locks held longer than some
  absurd threshold (e.g. 24h).

Lock is enforced regardless of trigger source — scheduler, group runner,
ad-hoc single-module, ad-hoc group run. All paths hit the same atomic check.

**No separate group lock needed.** If a group runner tries to start a module
that's already locked, it fails on that module and stops the group (per
stop-on-failure rule).

## Run log / observability

Two tables:

```
group_run(
    id, group_id, started_at, finished_at, status, triggered_by
    -- triggered_by: schedule | manual | null
)

run_log(
    id,
    module_id,
    group_run_id,           -- nullable; set when run as part of a group
    started_at, finished_at,
    row_count,
    status,                 -- running | success | error | cancelled | dry_run
    error,
    resolved_source_sql,    -- exact SQL that ran on source
    merge_sql,              -- exact merge SQL that ran on dest
    watermark_values_json,  -- {prev_period: "'2610'", ...}
    jrunner_stdout,
    jrunner_stderr,
    hook_log,
    live_log                -- jrunner stdout lines appended in real time during migrate
)
```

Module history is **independent of group context** — `WHERE module_id=?` shows
every run, scheduled or manual, group or standalone. The `group_run_id` is
just an annotation.

**Run detail screen** (in TUI) shows: timing, status, row count, trigger
context, watermark values, plus keys to open in `$EDITOR`:

- `s` — resolved source SQL
- `m` — merge SQL
- `h` — hook output
- `o` — jrunner stdout/stderr

**Global run log** (`L` from main screen) — sortable, filterable across all
modules and groups. Answer "show me everything that errored in the last 24
hours" in two keystrokes.

## Groups and scheduling

```
grp(id, name)

group_member(id, group_id, module_id, run_order)
  -- many-to-many; same module can live in multiple groups with different run_orders

schedule(id, group_id, cron_expr, enabled, last_fired_at)
  -- a group can have 0..N schedules
```

**Sequential execution, continue past failures.** Each enabled member runs in
`run_order` regardless of whether prior members errored. Final group_run status
is `error` if any member errored, `success` if all succeeded, `dry_run` if all
were dry runs. Disabled modules are skipped entirely.

This was a deliberate change from the original "stop on failure" design. The
case for stopping: mirrors `set -e` shell behaviour; avoids running downstream
modules against stale data if an upstream one failed. The case for continuing:
full visibility into all failures in one run rather than discovering them
one-at-a-time; individual module locks already prevent unsafe concurrency.
A per-group `stop_on_failure` flag would satisfy both and is worth adding.

**Many-to-many membership.** Junction table is needed anyway for `run_order`,
so many-to-many costs nothing extra. Unique constraint can be added later if
ever needed.

**Schedule attaches to groups, not modules.** Matches the user's mental model
and avoids a huge cron-list. Individual modules can still be run ad-hoc.

**Scheduler.** `pipekit/scheduler.py` — a single daemon thread started via
FastAPI lifespan on server startup. Wakes every 60 s. For each enabled
schedule, uses `croniter` to find the next occurrence after `last_fired_at`
(or 2 minutes ago if never fired); fires if that occurrence has passed.
`last_fired_at` is written to the DB *before* the run thread is spawned —
prevents double-fire if a run is slow or pipekit restarts mid-run. A scheduled
fire and a manual fire use the same `run_group` code path; only `triggered_by`
differs (`schedule:{id}` vs `manual`).

**Ad-hoc runs:**

- `POST /modules/{id}/run` — single module
- `POST /groups/{id}/run` — whole group sequentially

Both create normal run_log entries.

## Connections and credentials

```
driver(id, name, jar_file, class_name, url_template)

connection(
    id,
    name,
    driver_id,
    jdbc_url,
    username,
    password,
    default_dest_connection_id,  -- nullable; wizard default when this is source
    default_dest_schema,         -- nullable; wizard default when this is source
    notes,
    created_at, updated_at
)
```

**Credentials = env var references.** The `password` column stores something
like `$DB2PW`. Engine resolves at runtime by reading the env var. Passwords
never live in the database. Matches existing setup
(`/opt/sync/.env` + shell scripts) and keeps `pipekit.db` safe to copy/back-up.

Test-connection: engine runs a trivial query (`SELECT 1` or equivalent) via
jrunner against the connection. Confirms URL, credentials, driver all work.

**jrunner handles all SQL execution** — bulk transfers (migration mode) and
single-value queries for watermark resolvers / hooks (query mode). Trade-off:
~100ms JVM spawn per resolver call, but one tool, one set of bugs, one
driver-loading path.

## Bootstrap / install hygiene

Pipekit verifies jrunner exists on startup (configurable path in
`config.yaml`). If missing, surfaces a clear error pointing at
`/opt/jrunner/deploy.sh`.

**`pipekit doctor`** CLI command — checks jrunner present, jrunner version,
drivers loadable, database accessible, all configured connections testable.
First thing to run after a `git pull`.

**`./deploy.sh`** — idempotent full-system deploy. Run once from a fresh clone
or re-run after any code update. Does:

1. Creates the `pipekit` system user (`useradd --system`) if absent
2. Chowns `/opt/pipekit` to `pipekit:pipekit`
3. Creates/repairs the Python venv at `.venv` owned by `pipekit`
4. Installs deps as `pipekit` user (`pip install -r requirements.txt`)
5. Installs `/usr/local/bin/pipekit` launcher symlink
6. Creates `/etc/pipekit/secrets.env` (mode 0640, group pipekit) if absent
7. Runs `pipekit init` (schema creation + migrations) as `pipekit`
8. Upserts JDBC driver rows for each jar found in jrunner's lib dir
9. Installs and enables `systemd/pipekit.service`

**Secrets** live in `/etc/pipekit/secrets.env` as `KEY=VALUE` lines. The
`password` field on a connection row stores `$KEY`; the engine resolves it at
runtime from the environment. The systemd unit loads this file as
`EnvironmentFile=`. Mode 0640, group pipekit — the service account can read it
but it is not world-readable. Managed with `pipekit secrets set/list/unset`.

**jrunner is a prerequisite, not managed by deploy.sh.** `deploy.sh` checks
that `jrunner` is on PATH and aborts with a clear message if not. Installing
jrunner (`/opt/jrunner/deploy.sh`) is a separate step the operator does first.
Keeping them decoupled means jrunner can be updated independently. Bundle later
if/when the two-step gets annoying.

## New module wizard

The centerpiece for fixing the authoring pain. Goal: from "I want to sync
table X from connection Y" to "module created, query previewed, ready to
test-run" in under a minute.

### Step 1 — Source

Pick source connection. Filter by schema. Search tables incrementally. The
TUI calls jrunner in query mode against the source's INFORMATION_SCHEMA
equivalent (DB2: `SYSIBM.SYSTABLES`, SQL Server / PG: `INFORMATION_SCHEMA.TABLES`).

### Step 2 — Columns

The engine introspects the chosen table. Proposes one row per column with:

- **In/out toggle** (default all on; toggle off the noise like `dcfut*` futures)
- **Default alias** — lowercase, special chars stripped: `DCORD#` → `dcord`
- **Default source expression** — bare column for most types; `RTRIM(col)` for
  char/varchar; `CASE WHEN col IN ('0001-01-01','9999-12-31') THEN NULL ELSE col END`
  for date (sentinel-NULL pattern from existing modules)
- **Default dest type** — mapped from source: `INT`→`integer`, `DECIMAL(15,4)`→`numeric(15,4)`,
  `CHAR(40)`→`text`, `DATE`→`date`, etc.

`e` opens an edit modal for one row to override alias / expression / type.
Most of the time you accept defaults.

### Step 3 — Destination & merge

Pick dest connection. Dest table defaults to
`{source_conn.default_dest_schema}.{lowercase_source_table_name}`. Pick
merge strategy. Pick merge key. When strategy is `incremental`, a
**Watermarks** panel and an editable **Source query** textarea appear
inline. The source query is pre-populated from column picks; the user
edits it to add the WHERE clause with `{placeholder}` references before
creating the module. Zero or more watermarks can be added in step 3;
they are created atomically with the module on submit.

**Multiple destinations are real** (e.g. PG → SQL Server). The wizard
doesn't assume one dest. Each source connection has a
`default_dest_connection_id` + `default_dest_schema` pair that
pre-populate Step 3. Both are nullable; fallback is last-used dest.

### Step 1 — Source (driver-dependent browse form)

Different drivers need different scope fields ("qualifiers") to identify a
table. DB2 needs just `schema`. SQL Server can need up to three:
`linked_server`, `database`, `schema` (any combination — linked server
optional, database optional, schema defaults to `dbo`). This is because
SQL Server can reference tables in other databases on the same server, or
tables on entirely different servers via linked servers — and the FROM
clause syntax changes (`schema.table`, `db.schema.table`,
`[linked].[db].[schema].[table]`).

Each driver exposes:

```python
class Driver:
    def browse_fields(self) -> list[BrowseField]:
        """Qualifier fields for the wizard's Step 1 form."""

    def list_tables(self, **qualifiers) -> list[Table]:
        """INFORMATION_SCHEMA query using whatever qualifiers are set."""

    def get_columns(self, table_name: str, **qualifiers) -> list[Column]:
        """Column lookup for a specific table."""

    def qualified_table_name(self, table_name: str, **qualifiers) -> str:
        """FROM-clause identifier. Wizard-time only."""

    def map_type(self, source_type) -> str: ...
    def default_expression(self, source_type, column_name) -> str: ...
    def quote_identifier(self, name) -> str: ...
```

Textual renders Step 1 dynamically from `browse_fields()`. The wizard
calls `qualified_table_name()` once to bake the FROM clause into the
stored source query. **Linked servers / qualifiers are not first-class in
Pipekit** — they exist only as syntax inside the generated FROM. Nothing
is persisted on the module about how the table was qualified at author
time. If you later need to add a column, you type the expression and
alias by hand in the column editor — no re-browsing needed.

### Step 4 — Preview

Show the generated source query, generated staging DDL, generated merge SQL.
Everything visible. `e` to drop into `$EDITOR` for free-form fixes. `c` to
create — writes the module row, creates the staging table on dest, offers a
test-run.

### Per-driver capability needed

Each driver module (`engine/drivers/db2.py`, etc.) implements:

- `list_tables(schema_filter)` — SQL template for INFORMATION_SCHEMA
- `get_columns(schema, table)` — column name, type, length, nullable
- `map_type(source_type)` → dest type
- `default_expression(source_type, column_name)` → wrap in RTRIM, CASE, etc.
- `quote_identifier(name)` — `"DCORD#"` (DB2/PG) vs `[DCORD#]` (MSSQL)

Defaults are **opinions hardcoded in driver modules** for now. Lift to a
`driver_default` table later if configurability is ever needed.

### Wizard scope (what it does NOT do)

- **No CTE-based queries.** Wizard generates a simple `SELECT cols FROM table` skeleton; the user adds WHERE clauses (including watermark placeholders) in the editable source query textarea in step 3, or post-creation via the module edit form.
- **Multi-watermark supported in wizard.** Add as many watermarks as needed in step 3; they're created atomically with the module.
- **No hooks in the wizard.** Add hooks from the module detail screen.
- **No group assignment in the wizard.** Assign separately.

These are intentional. The wizard handles the 80% case fast. The 20% cases
are post-creation edits where you already have a working module to start from.

## TUI — main screen sketch

```
Pipekit
─────────────────────────────────────────────────
▼ s7830956 (AS/400 DB2)
  ✔ code              full      2m ago    1,204 rows   0.8s
  ✔ name              full      2m ago      892 rows   0.6s
  ✔ qcrh              incr      2m ago    1,031 rows   3.2s
  ✗ qcri              incr      2m ago          —      err
  ○ cust              full      disabled
▼ usmidsql01 (SQL Server)
  ✔ live_quotes       full      2m ago      340 rows   1.1s

Groups
  pricing       9 modules   cron 0 20 2 * * *   next: 2:20am
  codes        26 modules   cron 0  0 2 * * *   next: 2:00am
```

Modules grouped by source connection (mirrors today's directory layout).
Status / strategy / last-run / row-count / duration on each line. Groups at
the bottom with schedules and next-fire times.

`i` inspect, `r` run, `l` history, `L` global log, `n` new module, `c`
connections, `/` search, `j/k` navigate, `q` quit. Should feel like lazygit /
nvim file tree.

### Module detail (i)

Top: module info (strategy, merge key, watermark, dest table, staging table,
enabled, last/next run). Middle: column table (parsed from source query).
Bottom: keybindings.

Keys open things in `$EDITOR` (read-only):

- `q` — next resolved source SQL
- `m` — merge SQL
- `b` — base query template (with placeholders)
- `e` — edit base query (writable)
- `w` — watermarks
- `h` — hooks
- `c` — column editor (parsed from query)
- `r` — run
- `l` — history

## API surface

**REST over HTTP**, FastAPI, HTTP Basic Auth on all endpoints except
`/health`. In practice the API only uses **GET (reads) and POST
(writes)** — PUT/DELETE avoided to keep the mental model simple.

### Resource CRUD

Every core table (connection, driver, module, watermark, hook, group,
group_member, schedule) gets the same URL pattern:

```
GET  /things             list (with filter query params)
GET  /things/{id}        read one
POST /things             create
POST /things/{id}        update
POST /things/{id}/delete delete
```

JSON shape = snake_case matching database columns. ISO 8601 timestamps.
Integer IDs. No transformation layer between SQL and JSON.

### Operation endpoints

Anything with side effects or that composes multiple steps:

```
POST /connections/{id}/test       run SELECT 1 via jrunner; return ok/fail/elapsed
GET  /modules/{id}/preview        return next resolved source SQL + merge SQL
                                  (resolves watermarks live, does NOT sync)
GET  /modules/{id}/columns        parse source query, return column list

POST /modules/{id}/run            start async run; return {run_id} immediately
POST /groups/{id}/run             start async group run; return {group_run_id}
POST /modules/{id}/cancel         cancel running module (release lock, kill jrunner)
POST /groups/{id}/cancel          cancel running group

GET  /runs                        list runs (filter: ?module_id= ?status= ?since=)
GET  /runs/{id}                   run detail (SQL, stdout/stderr, hook output)
GET  /runs/{id}/stream            Server-Sent Events: live log + status
GET  /group-runs                  list group runs
GET  /group-runs/{id}             group run with child module runs
GET  /modules/{id}/runs           shortcut: runs for one module
```

### Introspection endpoints (wizard backend)

```
POST /introspect/tables           body: {connection_id, qualifiers: {...}}
POST /introspect/columns          body: {connection_id, table_name, qualifiers}
POST /introspect/propose          body: {connection_id, table_name, qualifiers}
                                  returns a ready-to-POST module JSON
```

`propose` is curl-able — you can generate a module proposal, tweak the
JSON, then POST it to `/modules` to create. No TUI required.

### System endpoints

```
GET /health     liveness only, no auth required
GET /doctor     full check (jrunner, drivers, db, connections, scheduler)
                powers `pipekit doctor` CLI
GET  /settings
POST /settings/{key}
```

### Async runs + live progress

`POST /modules/{id}/run` does NOT block. It atomically acquires the
module lock, kicks off the sync in a background task, and returns
`{"run_id": 4892}` immediately (both the API and web POST).

Two ways to watch a run after that:

1. **Polling** — `GET /runs/{id}` returns the run_log row; keep hitting
   it until `status != running`. Simple, works anywhere.
2. **Live fragment polling** — the web run detail page (`/runs/{id}`)
   embeds an HTMX fragment that polls `GET /runs/{id}/live` every 2 s
   while `status == running`. The fragment shows current status,
   row count, and the `live_log` field as it grows. Polling stops
   automatically once status leaves `running`.

**Live log** — during `jrunner.migrate()` the engine uses `subprocess.Popen`
(not `subprocess.run`) so stdout can be read line-by-line as jrunner
emits it. Each non-empty line is appended to `run_log.live_log` via
`repo.append_run_live_log()`. How frequently lines appear depends on
jrunner's output flushing behaviour; at minimum the final row-count
summary line appears when the transfer completes.

Splitting `start` from `watch` means:

- Cron-triggered runs don't have to watch
- Curl scripting can fire-and-forget
- A browser navigating directly to `/runs/{id}` mid-run picks up the
  live panel automatically

### Auth

HTTP Basic. Username/password in the `settings` table. Single-user tool
for now; swap to JWT later if multi-user is ever needed, without
breaking URL structure.

### TUI = HTTP client

The TUI never touches SQLite directly. Every screen reads from an
endpoint. This guarantees zero behavioral drift between TUI and any
future web UI, and makes the API the single source of truth for
behavior.

## Open questions still to answer

1. ~~**Wizard defaults match user's mental model?**~~ Confirmed — RTRIM,
   sentinel-date NULL, lowercased aliases are fine for now.
2. ~~**Dest table default?**~~ Resolved — per-source connection
   `default_dest_connection_id` + `default_dest_schema`.
3. ~~**API surface.**~~ Resolved — REST, GET/POST only, async runs, SSE
   for live output, CRUD + operations + introspection mix.
4. **Migration plan.** Deferred. Would involve a parser that walks
   `/opt/sync/*/`, extracts pull.sql / insert.sql / sh wrapper, infers
   merge strategy and key, creates module rows.

## Decisions log (fast reference)

| Decision | Choice |
|---|---|
| Storage | SQLite, single file |
| Where SQL lives | In the database (text blobs), not files |
| Source query shape | Free text with `{watermark}` placeholders |
| Columns | Parsed from query; not separate rows; wizard auto-introspects on create |
| Watermarks | Multiple per module, type-agnostic, free-form resolver SQL |
| Merge strategies | full / incremental / append (no upsert) |
| Hooks | Per-module, post-merge, run_on success/failure/always |
| Group hooks | Deferred — not needed yet |
| Group membership | Many-to-many (junction table for run_order anyway) |
| Group execution | Sequential, continue past failures; final status = worst of members |
| Schedules | Attach to groups; multiple schedules per group allowed |
| Locking | Atomic UPDATE on `module.running`; PID + time-based stale clearing |
| Credentials | Env var references (`$DB2PW`); resolved at runtime |
| SQL execution | Everything via jrunner (migration + query mode) |
| Materialized SQL | Always — resolved source SQL stored before run + after run |
| Install | Loose-coupled to jrunner for now; bundle later |
| TUI feel | Like lazygit / nvim file tree; spatial, keyboard-driven |
| Authoring | Wizard handles 80% case; post-creation editing handles the rest |
| Multiple destinations | Supported. Source conn holds `default_dest_connection_id` + `default_dest_schema` for wizard prepopulation |
| Driver browse fields | Per-driver qualifier set (`schema` for DB2/PG, up to `linked_server`/`database`/`schema` for MSSQL) |
| Linked servers | Not first-class; only affect FROM-clause syntax at author time; not persisted on module |
| API style | REST, GET for reads, POST for writes, no PUT/DELETE |
| Run model | Async — POST /run returns run_id immediately; watch via polling or SSE stream |
| Live output | HTMX polling every 2–3 s (SSE deferred — polling sufficient for current scale) |
| Auth | HTTP Basic, single user, creds in settings table |
| TUI | Deferred — web UI built first; TUI remains valid future work as an API client |
| Scheduler | croniter for cron evaluation; last_fired_at persisted before run to prevent double-fire |
| Service account | Runs as pipekit system user; venv + db owned by that user; secrets 0640 group pipekit |