Document streaming architecture and memory usage
Clarify that both query and migration modes use streaming with no array storage. Query mode streams directly to stdout, while migration mode streams into a SQL string buffer (250 rows). The 10k fetch size is a JDBC driver hint for network efficiency, not application-level storage. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
ba24b874fc
commit
c0f6e3a6e6
25
CLAUDE.md
25
CLAUDE.md
@ -122,8 +122,31 @@ Query mode uses dedicated output methods:
|
|||||||
- All output goes to stdout; no diagnostic messages in query mode
|
- All output goes to stdout; no diagnostic messages in query mode
|
||||||
- Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting
|
- Helper methods: `escapeCSV()` and `escapeTSV()` for proper formatting
|
||||||
|
|
||||||
|
### Memory and Streaming Architecture
|
||||||
|
Both modes use a streaming architecture with no array storage of result rows:
|
||||||
|
|
||||||
|
**Query Mode Streaming:**
|
||||||
|
- Rows are pulled from the ResultSet via `rs.next()` one at a time
|
||||||
|
- Each row is immediately formatted and written to stdout
|
||||||
|
- No accumulation in memory - pure streaming from database to stdout
|
||||||
|
- The only buffer is the JDBC driver's internal fetch buffer (10,000 rows)
|
||||||
|
|
||||||
|
**Migration Mode Streaming:**
|
||||||
|
- Rows are pulled from the ResultSet via `rs.next()` one at a time
|
||||||
|
- Each row is converted to a SQL VALUES clause string: `(val1,val2,val3)`
|
||||||
|
- VALUES clauses are accumulated into a single `sql` string variable
|
||||||
|
- When 250 rows accumulate, the string is prepended with `INSERT INTO {table} VALUES` and executed
|
||||||
|
- The `sql` string is cleared and accumulation starts again
|
||||||
|
- Only holds up to 250 rows worth of SQL text in memory at once
|
||||||
|
|
||||||
|
**JDBC Fetch Size:**
|
||||||
|
- Both modes set `stmt.setFetchSize(10000)` (line 190)
|
||||||
|
- This is a hint to the JDBC driver to fetch 10,000 rows at a time from the database
|
||||||
|
- The driver maintains this internal buffer for network efficiency
|
||||||
|
- The application code never sees or stores all 10,000 rows - it processes them one at a time via `rs.next()`
|
||||||
|
|
||||||
### Batch Size (Migration Mode)
|
### Batch Size (Migration Mode)
|
||||||
INSERT statements are batched at 250 rows (hardcoded around line 324). When the batch threshold is reached, sql is prepended with "INSERT INTO {table} VALUES" and executed.
|
INSERT statements are batched at 250 rows (hardcoded around line 356). Rows are streamed into a SQL string buffer as VALUES clauses. When 250 rows accumulate in the string, it is prepended with "INSERT INTO {table} VALUES" and executed, then the string is cleared.
|
||||||
|
|
||||||
### Error Handling
|
### Error Handling
|
||||||
SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery.
|
SQLException handling prints stack trace and exits immediately with System.exit(0). There is no transaction rollback or partial failure recovery.
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user