java data mover using jdbc
Go to file
Paul Trowbridge a1c9ea26ce fix: stream PostgreSQL source (migration mode) instead of buffering it all
setFetchSize(10000) is a no-op on the PostgreSQL JDBC driver while autoCommit
is true — the driver loads the entire ResultSet into memory, OOM/GC-thrashing
on large source tables (a pg->SQL Server pull pinned the box: 4GB heap, swap
full, 0 rows written). PG only uses a server-side cursor when autoCommit is
false AND fetchSize > 0.

Set the source connection to manual commit ONLY in migration mode: the
migration source is read-only so never committing is harmless. Query mode is
excluded on purpose — callers (pipekit's run_dest_sql) run committed DDL/DML
through query mode, and autoCommit=false would roll those back on close.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:51:57 -04:00
gradle/wrapper upgrade gradle wrapper to 8.5 for java 20 compatibility 2026-01-06 21:42:36 -05:00
jrunner fix: stream PostgreSQL source (migration mode) instead of buffering it all 2026-06-17 15:51:57 -04:00
.gitattributes initial 2022-10-20 11:12:19 -04:00
.gitignore gitignore accidentally extracted distribution files 2026-01-06 22:20:22 -05:00
CLAUDE.md Document streaming architecture and memory usage 2026-01-25 14:28:26 -05:00
deploy.sh Fix global install to use installDist instead of wiping project dir 2026-03-02 22:01:51 -05:00
gradlew initial 2022-10-20 11:12:19 -04:00
gradlew.bat initial 2022-10-20 11:12:19 -04:00
jrq Add jrq query wrapper script and deploy integration 2026-02-26 06:40:44 -05:00
LICENSE add MIT license 2026-01-06 22:44:47 -05:00
readme.md Add jrq query wrapper script and deploy integration 2026-02-26 06:40:44 -05:00
settings.gradle rename app module to jrunner for consistency 2026-01-06 21:53:08 -05:00
setup.sh Add automated setup script for new installations 2026-01-16 11:01:26 -05:00

Quick Start

The easiest way to get started on a new system:

git clone https://gitea.hptrow.me/pt/jrunner.git
cd jrunner
./setup.sh

The setup script will:

  • Check for Java 11+ (offers to install if missing)
  • Verify Gradle wrapper is present
  • Run a test build to ensure everything works
  • Show you next steps

Manual Installation

If you prefer to install dependencies manually:

Install Java JDK

Option 1: Package manager (recommended)

# Ubuntu/Debian
sudo apt update && sudo apt install openjdk-17-jdk

# Fedora/RHEL
sudo dnf install java-17-openjdk-devel

# Arch
sudo pacman -S jdk-openjdk

Option 2: Manual download Download from https://www.oracle.com/java/technologies/downloads/

wget https://download.oracle.com/java/19/latest/jdk-19_linux-x64_bin.tar.gz
tar -xvf jdk-19_linux-x64_bin.tar.gz
sudo mv jdk-19.0.1 /opt/
export JAVA_HOME=/opt/jdk-19.0.1
export PATH=$PATH:$JAVA_HOME/bin

Test: java --version

Install Gradle (optional)

Gradle wrapper (gradlew) is included in the repo, so manual Gradle installation is not required.

build

./gradlew build

deploy

Run the interactive deploy script:

./deploy.sh

The script will prompt you to choose:

  1. Local install - Fast, no sudo required, installs to ./jrunner/build/install/jrunner
  2. Global install - Installs to /opt/jrunner with symlink at /usr/local/bin/jrunner
  3. Custom directory - Prompts for path with tab-completion support

The script builds, extracts to a temporary location, and only updates the target directory after the build succeeds. This ensures your existing deployment stays intact if the build fails.

manual deployment

./gradlew build
sudo unzip jrunner/build/distributions/jrunner.zip -d /opt/
sudo ln -sf /opt/jrunner/bin/jrunner /usr/local/bin/jrunner

Or for local testing:

./gradlew installDist
# Binary at: ./jrunner/build/install/jrunner/bin/jrunner

usage

Query Mode (new in v1.1)

Query mode outputs results to stdout for piping to visidata, pspg, or less. It activates automatically when destination flags are omitted.

Basic query to CSV:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql

Pipe to visidata:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql | visidata -f csv

TSV format:

jrunner -scu "jdbc:as400://hostname" -scn user -scp pass -sq query.sql -f tsv

SQL Server example:

jrunner -scu "jdbc:sqlserver://hostname:1433;databaseName=mydb" -scn user -scp pass -sq query.sql

PostgreSQL example:

jrunner -scu "jdbc:postgresql://hostname:5432/dbname" -scn user -scp pass -sq query.sql

jrq - Query Mode Wrapper

The jrq wrapper script simplifies query mode usage by storing connection details in a config file, eliminating the need to type credentials repeatedly.

Setup:

Create ~/.jrqrc with your database connection details:

JR_URL="jdbc:as400://s7830956"
JR_USER="myusername"
JR_PASS="mypassword"

Usage:

# Output to stdout
jrq query.sql

# Pipe to visidata
jrq query.sql | vd

# Use TSV format
jrq query.sql tsv

# Pipe TSV to less with column display
jrq query.sql tsv | less -S

Configuration options in ~/.jrqrc:

  • JR_URL - JDBC connection URL (required)
  • JR_USER - Database username (required)
  • JR_PASS - Database password (required)
  • JR_BIN - Path to jrunner binary (default: /opt/jrunner/jrunner/build/install/jrunner/bin/jrunner)
  • JR_FORMAT - Default output format: csv or tsv (default: csv)

Note: You can also set these as environment variables instead of using a config file:

export JR_URL="jdbc:as400://hostname"
export JR_USER="username"
export JR_PASS="password"
jrq query.sql | vd

Migration Mode

Full migration mode with both source and destination:

jrunner -scu jdbc:postgresql://source:5432/sourcedb \
        -scn sourceuser \
        -scp sourcepass \
        -dcu jdbc:postgresql://dest:5432/destdb \
        -dcn destuser \
        -dcp destpass \
        -sq query.sql \
        -dt public.target_table

Command-line flags

Source connection:

  • -scu - source JDBC URL
  • -scn - source username
  • -scp - source password
  • -sq - path to source SQL query file

Destination connection (migration mode only):

  • -dcu - destination JDBC URL
  • -dcn - destination username
  • -dcp - destination password
  • -dt - fully qualified destination table name

Options:

  • -t - trim text fields (default: true)
  • -c - clear target table before insert (default: true)
  • -f - output format: csv, tsv (query mode only, default: csv)