Streaming LLM output through a Next.js proxy: why you only get stream_end
When you proxy an LLM backend's Server-Sent Events (SSE) through a Next.js route handler, the usual failure is that nothing arrives until the end, and then everything arrives at once, or the client only ever sees the final stream_end . Something between the backend and the browser is buffering . Th
Posted on Nov 5 • Originally published at aliahmed.si
When you proxy an LLM backend's Server-Sent Events (SSE) through a Next.js route handler, the usual failure is that nothing arrives until the end, and then everything arrives at once, or the client only ever sees the final stream_end. Something between the backend and the browser is buffering. The fix is a handful of specific things: the Node.js runtime, dynamic = 'force-dynamic', the right headers ( Cache-Control: no-cache, no-transform and X-Accel-Buffering: no ), passing bytes through as they arrive, and a client that parses SSE frames properly. I learned these building Turbo Alan Refiner, a multi-pass LLM text-refinement app ( frontend source ).
Routing every backend call through /api/* route handlers avoids CORS, keeps tokens server-side, and lets you normalise the request. The cost is that your route is now a hop in a streaming pipeline, and every hop is a chance to buffer.
```ts title="Route segment config" export const runtime = "nodejs"; export const dynamic = "force-dynamic"; export const maxDuration = 300;
* `runtime = "nodejs"` gives full streaming support without edge-runtime limits. * `dynamic = "force-dynamic"` stops Next.js caching or pre-rendering the route. * `maxDuration = 300` lets long refinements outlive the default function timeout. Then return a `ReadableStream` that forwards the upstream body as it arrives, with these headers: ```ts title="Streaming response headers" return new Response(stream, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", Connection: "keep-alive", "X-Accel-Buffering": "no", }, }); Enter fullscreen mode Exit fullscreen mode X-Accel-Buffering: no tells nginx-style proxies not to buffer.
no-transform matters more than it looks. A proxy that gzips the response has to collect bytes before it can compress them, which is exactly the buffering you're trying to avoid.
Something buffered the stream (missing headers, the wrong runtime, a proxy in front), so all the events arrived together and the UI only rendered the last one.
The client parsed chunks, not lines. A network chunk can end in the middle of an event. Keep the incomplete tail in a buffer:
```ts title="Client: keep the incomplete last line" buffer += decoder.decode(chunk, { stream: true }); const lines = buffer.split("\n"); buffer = lines.pop()?? "";