Python requests timeout=60 Let My LLM Stream Run for 38 Minutes

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes. Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

The call looked like this, and I'd guess half the LLM scripts on GitHub look the same:

r = requests. post ( URL, json = payload, stream = True, timeout = 60 ) for line in r. iter_lines (): if line. startswith ( b " data: " ) and line!= b " data: [DONE] ": chunk = json. loads ( line [ 6:]) parts. append ( chunk [ " choices " ][ 0 ][ " delta " ]. get ( " content " ) or "" ) Enter fullscreen mode Exit fullscreen mode I read timeout=60 as "this call dies after a minute." That was wrong.

The Python requests timeout sets two limits: how long to wait for the TCP connection, and how long to wait between bytes on the socket. Passing one number sets both to that value. Passing a tuple like (5, 30) sets them separately. Neither one limits total request time.

The requests docs say this outright. The read timeout is the number of seconds the client will wait for the server to send a byte. Every byte that arrives resets that clock.

For a normal JSON API this almost never matters. The server is silent until it has an answer, so "time between bytes" and "total time" are basically the same number.

Streaming changes that. A server streaming tokens is never silent for long. My model produced a chunk roughly every 70 ms under load, so the read timeout restarted about 14 times a second. A 60-second read timeout couldn't fire unless the model stopped completely.

It ran that long because the model got stuck repeating itself, and nothing told it to stop. I hadn't set max_tokens. vLLM's OpenAI-compatible server then allows generation up to whatever context window is left after the prompt. My context was 32k tokens, the prompts were about 1.5k, and that left roughly 30,000 tokens of room.

At around 15 tokens per second per request under batch load, 30,000 tokens takes about 33 minutes. My worst call ran 38. The numbers line up.

The outputs from those calls looked like what you'd expect:

"summary": "Customer reports login failure. Customer reports login failure. Customer reports login failure. Customer reports login failure.... Enter fullscreen mode Exit fullscreen mode That repeated a few thousand times. Some tickets (very long ones, pasted stack traces, logs full of near-identical lines) push small models into these loops. It's rare, but over 2,400 calls a night it happens.

0.18% sounds harmless until you remember there were only 6 workers. When 2 or 3 runaway calls landed on the same night, a third to half of my capacity spent half an hour generating "Customer reports login failure." The other workers also slowed down, because every runaway sequence took a slot in vLLM's batch and a chunk of KV cache.

On the 3 late nights, 4 or more workers were stuck at once.

The job wasn't failing, so I had no errors to read. What I had:

Then I wrote a 15-line test server that sent one SSE line every 30 seconds and pointed requests.post(..., timeout=60) at it. It sat there for an hour until I killed it. That settled it.

Set a total timeout by enforcing a wall-clock deadline in your own code. Neither requests nor httpx gives you one. Check elapsed time on every chunk, cap generation with max_tokens, and close the connection when you give up so the server stops generating.

import json import time import requests def stream_completion ( payload, deadline_s = 90, read_timeout = 20 ): payload = { ** payload, " max_tokens ": 1024, " stream ": True } start = time. monotonic () parts, finish = [], None with requests. post ( URL, json = payload, stream = True, timeout = ( 5, read_timeout )) as r: r. raise_for_status () for line in r. iter_lines (): if time. monotonic () - start > deadline_s: raise TimeoutError ( f " exceeded { deadline_s } s wall clock " ) if not line. startswith ( b " data: " ) or line == b " data: [DONE] ": continue choice = json. loads ( line [ 6:])[ " choices " ][ 0 ] parts. append ( choice [ " delta " ]. get ( " content " ) or "" ) finish = choice. get ( " finish_reason " ) or finish return "". join ( parts ), finish Enter fullscreen mode Exit fullscreen mode Four things changed, and each one matters:

Yes. With async code, wrap the entire call in asyncio.timeout() (Python 3.11+). It cancels the task at the deadline whether or not bytes are flowing:

import asyncio import httpx async def stream_completion ( client: httpx. AsyncClient, payload ): async with asyncio. timeout ( 90 ): async with client. stream ( " POST ", URL, json = payload ) as r: async for line in r. aiter_lines ():... Enter fullscreen mode Exit fullscreen mode Cancellation unwinds the async with and closes the stream. This is my preferred version. I kept the sync one above because the batch job is threaded and I didn't want to rewrite it at 2 a.m.

If you use an SDK built on httpx, like the OpenAI Python client, read what its timeout argument means for streams in the version you have installed. Don't assume it's a total deadline. Wrap the call in your own deadline either way. It costs one line.

Capping generation doesn't fix the loop. It just makes the loop cheap. In the 14 nights after the fix, 58 calls ended with finish_reason == "length", which in this job always meant a repetition loop.

That's 9 tickets a fortnight for a person to look at, instead of 9 GPU-hours of "Customer reports login failure."

Because the Python requests timeout only limits connecting and the gap between received bytes. It never limits total request duration. A streaming LLM response sends a byte every few milliseconds, so the read timeout keeps resetting and a call can run as long as the model keeps generating: 38 minutes in my case, with no max_tokens set. To actually bound a streaming call, set max_tokens, keep a short read timeout, enforce a wall-clock deadline in your own loop or with asyncio.timeout(), and close the connection when the deadline hits so the server stops generating.

Written by the developer behind Preterview, an interview prep platform.

Expand 오피셜메일 오피셜메일 오피셜메일 Follow 1인기업·1인창업자의 바이브코딩, 브랜딩과 마케팅 실행 과정을 기록합니다. Location 대한민국 Work 오피셜메일 Joined Jul 29, 2026 • Nov 2 Dropdown menu Copy link Hide read timeout과 전체 실행 기한을 나누고, 동기 루프의 실제 상한이 deadline에 read_timeout까지 더해질 수 있다고 짚은 부분이 좋았습니다. 재시도까지 포함한 작업 전체 예산도 별도로 두는지 궁금합니다. 개별 호출은 제한 안에 있어도 재시도가 보고서 마감을 넘길 수 있어 보여서요.