Document streaming architecture and memory usage

Clarify that both query and migration modes use streaming with no array
storage. Query mode streams directly to stdout, while migration mode
streams into a SQL string buffer (250 rows). The 10k fetch size is a
JDBC driver hint for network efficiency, not application-level storage.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-01-25 14:28:26 -05:00
parent ba24b874fc
commit c0f6e3a6e6

View File

@ -122,8 +122,31 @@ Query mode uses dedicated output methods:
- All output goes to stdout; no diagnostic messages in query mode - All output goes to stdout; no diagnostic messages in query mode
- Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting - Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting
### Memory and Streaming Architecture
Both modes use a streaming architecture with no array storage of result rows:
**Query Mode Streaming:**
- Rows are pulled from the ResultSet via `rs.next()` one at a time
- Each row is immediately formatted and written to stdout
- No accumulation in memory - pure streaming from database to stdout
- The only buffer is the JDBC driver's internal fetch buffer (10,000 rows)
**Migration Mode Streaming:**
- Rows are pulled from the ResultSet via `rs.next()` one at a time
- Each row is converted to a SQL VALUES clause string: `(val1,val2,val3)`
- VALUES clauses are accumulated into a single `sql` string variable
- When 250 rows accumulate, the string is prepended with `INSERT INTO {table} VALUES` and executed
- The `sql` string is cleared and accumulation starts again
- Only holds up to 250 rows worth of SQL text in memory at once
**JDBC Fetch Size:**
- Both modes set `stmt.setFetchSize(10000)` (line 190)
- This is a hint to the JDBC driver to fetch 10,000 rows at a time from the database
- The driver maintains this internal buffer for network efficiency
- The application code never sees or stores all 10,000 rows - it processes them one at a time via `rs.next()`
### Batch Size (Migration Mode) ### Batch Size (Migration Mode)
INSERT statements are batched at 250 rows (hardcoded around line 324). When the batch threshold is reached, sql is prepended with "INSERT INTO {table} VALUES" and executed. INSERT statements are batched at 250 rows (hardcoded around line 356). Rows are streamed into a SQL string buffer as VALUES clauses. When 250 rows accumulate in the string, it is prepended with "INSERT INTO {table} VALUES" and executed, then the string is cleared.
### Error Handling ### Error Handling
SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery. SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery.