From c0f6e3a6e61e215e995d22fd2b99a8dd87a8c51c Mon Sep 17 00:00:00 2001 From: Paul Trowbridge Date: Sun, 25 Jan 2026 14:28:26 -0500 Subject: [PATCH] Document streaming architecture and memory usage Clarify that both query and migration modes use streaming with no array storage. Query mode streams directly to stdout, while migration mode streams into a SQL string buffer (250 rows). The 10k fetch size is a JDBC driver hint for network efficiency, not application-level storage. Co-Authored-By: Claude Sonnet 4.5 --- CLAUDE.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1b84f4e..6f90d6b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -122,8 +122,31 @@ Query mode uses dedicated output methods: - All output goes to stdout; no diagnostic messages in query mode - Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting +### Memory and Streaming Architecture +Both modes use a streaming architecture with no array storage of result rows: + +**Query Mode Streaming:** +- Rows are pulled from the ResultSet via `rs.next()` one at a time +- Each row is immediately formatted and written to stdout +- No accumulation in memory - pure streaming from database to stdout +- The only buffer is the JDBC driver's internal fetch buffer (10,000 rows) + +**Migration Mode Streaming:** +- Rows are pulled from the ResultSet via `rs.next()` one at a time +- Each row is converted to a SQL VALUES clause string: `(val1,val2,val3)` +- VALUES clauses are accumulated into a single `sql` string variable +- When 250 rows accumulate, the string is prepended with `INSERT INTO {table} VALUES` and executed +- The `sql` string is cleared and accumulation starts again +- Only holds up to 250 rows worth of SQL text in memory at once + +**JDBC Fetch Size:** +- Both modes set `stmt.setFetchSize(10000)` (line 190) +- This is a hint to the JDBC driver to fetch 10,000 rows at a time from the database +- The driver maintains this internal buffer for network efficiency +- The application code never sees or stores all 10,000 rows - it processes them one at a time via `rs.next()` + ### Batch Size (Migration Mode) -INSERT statements are batched at 250 rows (hardcoded around line 324). When the batch threshold is reached, sql is prepended with "INSERT INTO {table} VALUES" and executed. +INSERT statements are batched at 250 rows (hardcoded around line 356). Rows are streamed into a SQL string buffer as VALUES clauses. When 250 rows accumulate in the string, it is prepended with "INSERT INTO {table} VALUES" and executed, then the string is cleared. ### Error Handling SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery.