java data mover using jdbc
Go to file
Paul Trowbridge 94cd89899a Report a machine-readable summary; stop failing on no-result statements
Two coupled changes: the summary is what a caller should read instead of
parsing prose, and the statement fix is what makes a non-zero exit code
usable at all.

Summary: emit one "@summary {json}" line carrying status, row count and
elapsed ms. Deliberately on stderr — migration-mode stdout carries the
progress output pipekit streams to its live log, and query-mode stdout
must stay pure CSV, so structured data on stdout would corrupt one or the
other. Previously the row count reached the caller only because a \r
progress tick and a trailing " rows written" happened to render on the
same line, which pipekit then matched with three different regexes.

Statements: query mode used executeQuery(), which raises when a statement
returns no result set — Postgres "No results were returned by the query",
SQL Server "The statement did not return a result set". DDL, INSERT,
DELETE and TRUNCATE all hit this despite succeeding, so pipekit carries a
_BENIGN_EXCEPTION_SUBSTRINGS allowlist to tell a working TRUNCATE from a
real failure. That made --strict unusable: it would have failed every
merge, staging DDL and hook. Now execute() + getUpdateCount() treats them
as the successes they are and reports rows affected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 09:11:16 -04:00
gradle/wrapper upgrade gradle wrapper to 8.5 for java 20 compatibility 2026-01-06 21:42:36 -05:00
jrunner Report a machine-readable summary; stop failing on no-result statements 2026-08-10 09:11:16 -04:00
.gitattributes initial 2022-10-20 11:12:19 -04:00
.gitignore gitignore accidentally extracted distribution files 2026-01-06 22:20:22 -05:00
CLAUDE.md docs: -b now covers Postgres (COPY) as well as SQL Server 2026-06-18 23:16:00 -04:00
deploy.sh Fix global install to use installDist instead of wiping project dir 2026-03-02 22:01:51 -05:00
gradlew initial 2022-10-20 11:12:19 -04:00
gradlew.bat initial 2022-10-20 11:12:19 -04:00
jrq Add jrq query wrapper script and deploy integration 2026-02-26 06:40:44 -05:00
LICENSE add MIT license 2026-01-06 22:44:47 -05:00
readme.md docs: -b now covers Postgres (COPY) as well as SQL Server 2026-06-18 23:16:00 -04:00
settings.gradle rename app module to jrunner for consistency 2026-01-06 21:53:08 -05:00
setup.sh Add automated setup script for new installations 2026-01-16 11:01:26 -05:00

Quick Start

The easiest way to get started on a new system:

git clone https://gitea.hptrow.me/pt/jrunner.git
cd jrunner
./setup.sh

The setup script will:

  • Check for Java 11+ (offers to install if missing)
  • Verify Gradle wrapper is present
  • Run a test build to ensure everything works
  • Show you next steps

Manual Installation

If you prefer to install dependencies manually:

Install Java JDK

Option 1: Package manager (recommended)

# Ubuntu/Debian
sudo apt update && sudo apt install openjdk-17-jdk

# Fedora/RHEL
sudo dnf install java-17-openjdk-devel

# Arch
sudo pacman -S jdk-openjdk

Option 2: Manual download Download from https://www.oracle.com/java/technologies/downloads/

wget https://download.oracle.com/java/19/latest/jdk-19_linux-x64_bin.tar.gz
tar -xvf jdk-19_linux-x64_bin.tar.gz
sudo mv jdk-19.0.1 /opt/
export JAVA_HOME=/opt/jdk-19.0.1
export PATH=$PATH:$JAVA_HOME/bin

Test: java --version

Install Gradle (optional)

Gradle wrapper (gradlew) is included in the repo, so manual Gradle installation is not required.

build

./gradlew build

deploy

Run the interactive deploy script:

./deploy.sh

The script will prompt you to choose:

  1. Local install - Fast, no sudo required, installs to ./jrunner/build/install/jrunner
  2. Global install - Installs to /opt/jrunner with symlink at /usr/local/bin/jrunner
  3. Custom directory - Prompts for path with tab-completion support

The script builds, extracts to a temporary location, and only updates the target directory after the build succeeds. This ensures your existing deployment stays intact if the build fails.

manual deployment

./gradlew build
sudo unzip jrunner/build/distributions/jrunner.zip -d /opt/
sudo ln -sf /opt/jrunner/bin/jrunner /usr/local/bin/jrunner

Or for local testing:

./gradlew installDist
# Binary at: ./jrunner/build/install/jrunner/bin/jrunner

usage

Query Mode (new in v1.1)

Query mode outputs results to stdout for piping to visidata, pspg, or less. It activates automatically when destination flags are omitted.

Basic query to CSV:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql

Pipe to visidata:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql | visidata -f csv

TSV format:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql -f tsv

SQL Server example:

jrunner -scu "jdbc:sqlserver://hostname:1433;databaseName=mydb" -scn user -scp pass -sq query.sql

PostgreSQL example:

jrunner -scu "jdbc:postgresql://hostname:5432/dbname" -scn user -scp pass -sq query.sql

jrq - Query Mode Wrapper

The jrq wrapper script simplifies query mode usage by storing connection details in a config file, eliminating the need to type credentials repeatedly.

Setup:

Create ~/.jrqrc with your database connection details:

JR_URL="jdbc:as400://s7830956"
JR_USER="myusername"
JR_PASS="mypassword"

Usage:

# Output to stdout
jrq query.sql

# Pipe to visidata
jrq query.sql | vd

# Use TSV format
jrq query.sql tsv

# Pipe TSV to less with column display
jrq query.sql tsv | less -S

Configuration options in ~/.jrqrc:

  • JR_URL - JDBC connection URL (required)
  • JR_USER - Database username (required)
  • JR_PASS - Database password (required)
  • JR_BIN - Path to jrunner binary (default: /opt/jrunner/jrunner/build/install/jrunner/bin/jrunner)
  • JR_FORMAT - Default output format: csv or tsv (default: csv)

Note: You can also set these as environment variables instead of using a config file:

export JR_URL="jdbc:as400://hostname"
export JR_USER="username"
export JR_PASS="password"
jrq query.sql | vd

Migration Mode

Full migration mode with both source and destination:

jrunner -scu jdbc:postgresql://source:5432/sourcedb \
        -scn sourceuser \
        -scp sourcepass \
        -dcu jdbc:postgresql://dest:5432/destdb \
        -dcn destuser \
        -dcp destpass \
        -sq query.sql \
        -dt public.target_table

Command-line flags

Source connection:

  • -scu - source JDBC URL
  • -scn - source username
  • -scp - source password
  • -sq - path to source SQL query file

Destination connection (migration mode only):

  • -dcu - destination JDBC URL
  • -dcn - destination username
  • -dcp - destination password
  • -dt - fully qualified destination table name

Options:

  • -t - trim text fields (default: true)
  • -c - clear target table before insert (default: true)
  • -b - bulk load into the destination instead of batched INSERTs — far faster on large/wide tables. SQL Server: TDS bulk-load via SQLServerBulkCopy. Postgres: COPY FROM STDIN. Other dests (e.g. DB2) fall back to INSERT. (migration mode only)
  • -f - output format: csv, tsv (query mode only, default: csv)