Document streaming architecture and memory usage

Clarify that both query and migration modes use streaming with no array
storage. Query mode streams directly to stdout, while migration mode
streams into a SQL string buffer (250 rows). The 10k fetch size is a
JDBC driver hint for network efficiency, not application-level storage.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Paul Trowbridge 2026-01-25 14:28:26 -05:00
parent ba24b874fc
commit c0f6e3a6e6

View File

@ -122,8 +122,31 @@ Query mode uses dedicated output methods:
- All output goes to stdout; no diagnostic messages in query mode
- Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting
### Memory and Streaming Architecture
Both modes use a streaming architecture with no array storage of result rows:
**Query Mode Streaming:**
- Rows are pulled from the ResultSet via `rs.next()` one at a time
- Each row is immediately formatted and written to stdout
- No accumulation in memory - pure streaming from database to stdout
- The only buffer is the JDBC driver's internal fetch buffer (10,000 rows)
**Migration Mode Streaming:**
- Rows are pulled from the ResultSet via `rs.next()` one at a time
- Each row is converted to a SQL VALUES clause string: `(val1,val2,val3)`
- VALUES clauses are accumulated into a single `sql` string variable
- When 250 rows accumulate, the string is prepended with `INSERT INTO {table} VALUES` and executed
- The `sql` string is cleared and accumulation starts again
- Only holds up to 250 rows worth of SQL text in memory at once
**JDBC Fetch Size:**
- Both modes set `stmt.setFetchSize(10000)` (line 190)
- This is a hint to the JDBC driver to fetch 10,000 rows at a time from the database
- The driver maintains this internal buffer for network efficiency
- The application code never sees or stores all 10,000 rows - it processes them one at a time via `rs.next()`
### Batch Size (Migration Mode)
INSERT statements are batched at 250 rows (hardcoded around line 324). When the batch threshold is reached, sql is prepended with "INSERT INTO {table} VALUES" and executed.
INSERT statements are batched at 250 rows (hardcoded around line 356). Rows are streamed into a SQL string buffer as VALUES clauses. When 250 rows accumulate in the string, it is prepended with "INSERT INTO {table} VALUES" and executed, then the string is cleared.
### Error Handling
SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery.