Summarize Large Documents with an LLM API: A Batch vs Streaming Guide

To summarize large documents with an LLM API for education code reviews, separate the review contract from the transport. A scheduled review of many pull requests wants bounded work and replayable output. A developer waiting on one changed module wants the first useful finding quickly.

Short answer: use a provider-neutral review contract, then choose batch when completion and replay matter, or streaming when a human is waiting. Keep chunking, validation, retries, and provider selection outside that contract. The transport is an implementation detail. The finding schema is the product.

The recommendation is boring: make the normalized request and result identical in both modes. Add a stream of progress events only at the edge. That keeps a provider change from becoming a rewrite of the review workflow.

Portability is not achieved by hiding one SDK behind a function called summarize. That function tends to leak provider-specific roles, finish reasons, token fields, and retry behavior into the rest of the codebase. Later, the adapter is a second application.

Start with a small internal contract. A review request needs a document identifier, chunks or chunk references, an expected output schema, and an idempotency key. A review result needs findings, severity, source spans, and a status. A provider request ID belongs in metadata for tracing, not in the business result.

The trade-off is deliberate: schema version 1.0 and stable chunk IDs add a little bookkeeping now, but they preserve old reports when the adapter or transport changes. Skipping that bookkeeping is faster for one prototype and expensive once stored findings become part of an education team's record.

The useful boundary is a complete operation and an event-producing operation. Both accept the same request. Both produce the same final result. The stream adds progress, never a different meaning.

type Severity = " info " | " warning " | " error "; type ReviewRequest = { schemaVersion: " 1.0 "; documentId: string; chunks: Array < { id: string; text: string } >; idempotencyKey: string; }; type Finding = { severity: Severity; message: string; chunkId: string; }; type ReviewResult = { status: " complete " | " failed "; findings: Finding []; }; type ReviewEvent = | { type: " progress "; completedChunks: number; totalChunks: number } | { type: " finding "; finding: Finding } | { type: " complete "; result: ReviewResult } | { type: " error "; message: string }; interface ReviewRuntime { run ( request: ReviewRequest ): Promise < ReviewResult >; stream ( request: ReviewRequest ): AsyncIterable < ReviewEvent >; } Enter fullscreen mode Exit fullscreen mode This contract gives the test suite something concrete to assert. A fake runtime can return two findings without contacting a model. An adapter test can verify that malformed output becomes a typed failure rather than an accidental prose report.

Batch changes scheduling. Streaming changes observation. Neither should change the review rules.

For batch work, persist the normalized request before submission. Give it an idempotency key derived from the document revision, not from a transient queue message. Store the final structured result and the raw response separately when policy permits. The same input can then be replayed through another adapter.

For streaming work, send events that clients can process independently. A client may disconnect after the first finding. The service still needs a policy for whether the review continues, is cancelled, or resumes from stored state. Do not make a browser reconstruct final JSON by concatenating text fragments. Text is a display channel; structured events are the data channel.

A stream can be buffered at the boundary. This is the runner-up path for teams that need responsive progress but cannot accept partial findings as authoritative. Emit progress to the UI, collect findings in the service, validate the completed object, then publish one final result. The extra memory and latency are easy to reason about. A half-valid report is not.

There is a transport limitation worth naming: a dropped connection is not a reliable job queue. Use durable state for work that must finish and a live connection for notification. Mixing those responsibilities is how a review disappears when a laptop sleeps.

The failure path deserves the same design attention as the happy path. Suppose an instructor opens a review in a browser, receives two valid findings, and loses connectivity before the final event. The client should be able to reconnect with the document revision and retrieve the authoritative result, rather than submit the whole review again because it cannot tell whether the server finished. That requires a persisted state transition such as queued, running, complete, or failed, plus an idempotency key that survives reconnects. Batch naturally supplies this shape; interactive streaming needs to build it deliberately. The trade-off is a few more records and a little more code in exchange for a review history that can be explained to a teammate later.

Chunking is where portability plans quietly fail. A provider-neutral adapter cannot fix a chunk that separates a changed function from its imports, or a rubric from the code it evaluates. The splitter needs stable IDs, document order, and enough context for a finding to make sense.

A 12,000-character target with a 1,000-character overlap can be a starting policy, not a universal truth. The right values depend on document format, context limits, and code density. Record the policy in request metadata so a later result remains explainable.

That policy is not a benchmark claim. It is a testable starting point.

Ask for findings tied to chunkId. If a response reports a concern without an evidence location, mark it for review or reject it according to the schema. A polished paragraph with no source span is not useful to an instructor deciding whether a change is safe.

async function reviewLargeChange ( runtime: ReviewRuntime, request: ReviewRequest, mode: " batch " | " stream " ): Promise < ReviewResult > { if ( mode === " batch " ) return validateResult ( await runtime. run ( request )); const findings: Finding [] = []; let finalResult: ReviewResult | undefined; for await ( const event of runtime. stream ( request )) { if ( event. type === " finding " ) findings. push ( event. finding ); if ( event. type === " complete " ) finalResult = event. result; if ( event. type === " error " ) throw new Error ( event. message ); } return validateResult ( finalResult?? { status: " complete ", findings }); } function validateResult ( result: ReviewResult ): ReviewResult { if (! Array. isArray ( result. findings )) { throw new Error ( " Review result has no findings array " ); } return result; } Enter fullscreen mode Exit fullscreen mode In production, validation should be stricter: check every severity, require a known chunk ID, cap message length, and reject duplicate evidence where duplicates mislead the reader. A JSON Schema or equivalent typed validator is a better boundary than scattered string checks.

Batch is not automatically better for large input. Choose streaming when a person is actively waiting and early findings can change the next action. A reviewer can stop after a clear blocking issue or request a narrower second pass.

Streaming is not automatically more responsive either. If incremental events cannot be validated, or if the client treats the first plausible sentence as truth, streaming increases the risk of a misleading interface. For a grading workflow that records one official report, buffered streaming or batch is easier to audit.

The decision has four useful questions: Is the work allowed to finish after disconnect? Must the result be replayable? Can a partial finding be acted on safely? Does the team need to change providers without changing stored reports? Those answers matter more than a benchmark measured on one prompt.

Measure time to first valid finding, time to final validated result, completion after disconnect, retry count, invalid-output rate, and findings with usable evidence spans. Track them by document size and review mode. One average hides the failure that makes a code-review tool unpleasant.

The simplest portable architecture is a queue-backed batch path for scheduled reviews and an event-backed streaming path for interactive reviews. Both call the same chunker, schema validator, and provider adapter interface. Both persist a final ReviewResult. That separation is enough to change transport without changing the education team's review record.