I Bisected My Prompt Until the Endpoint Broke

Last Tuesday I tried to turn a 40-minute lecture transcript into study questions. I pasted it into a chat box, pressed send, and got back a red error that told me almost nothing. Not "your input is 61,000 characters." Not "too long for this model." Just a failure, and a cursor blinking at me like it knew something I didn't.

So I did what students do. I guessed. I cut the transcript in half, pasted again, then in half again, and spent eleven minutes doing arithmetic with my eyes. Then it hit me — why am I acting as the bisection algorithm? That's a loop I can write in fifteen lines.

This is the story of that loop, what it measured, and the one number I now refuse to hardcode.

Not "what is the context limit." That phrasing invites me to memorize a doc page for a model I may never call again. The question is narrower and testable: for this endpoint, this model, and this request shape, what is the largest prompt that comes back 200?

That's a measurement, not a fact. It has a date on it. Mine is from this week, against a model I picked from the free tier on MonkeyCode, and I'll explain below why I'm not printing the number as gospel.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Python 3.11, standard library only, no tokenizer installed. That last part matters: I'm measuring characters, which is a bad proxy for tokens and a wonderful proxy for "is this request going to be rejected." You'll see the trade-off later.

Two rules before any code. First, the API key comes from the environment, never from a string literal. Second, the harness has a --dry-run mode with a stub that lies to me on purpose, so I can validate the loop before spending a single real request. Free tiers are a shared resource, and I try not to be the person who discovers the rate limit by ablation.

# probe_limit.py — Python 3.11, standard library only import json, os, sys, time import urllib.request, urllib. error ENDPOINT = os. environ. get ( " MC_ENDPOINT ", " https://your-endpoint.example/v1/chat/completions " ) MODEL = os. environ. get ( " MC_MODEL ", " the-model-you-picked " ) KEY = os. environ [ " MC_API_KEY " ] SEED = " Attention is a weighted average over values. " * 80 # long, boring, repeatable SLEEP = 2.0 # seconds between probes; raise this, not your hopes def make_prompt ( n_chars: int ) -> str: body = ( SEED * ( n_chars // len ( SEED ) + 1 ))[: n_chars ] return body + " \n\n In one sentence, what is this text about? " def post ( prompt: str, timeout: float = 60.0 ): payload = json. dumps ({ " model ": MODEL, " messages ": [{ " role ": " user ", " content ": prompt }], " max_tokens ": 40 }). encode () req = urllib. request.

Request ( ENDPOINT, data = payload, method = " POST ", headers = { " Content-Type ": " application/json ", " Authorization ": f " Bearer { KEY } " }) t0 = time. perf_counter () try: with urllib. request. urlopen ( req, timeout = timeout ) as r: return r. status, r. read (). decode ()[: 200 ], time. perf_counter () - t0 except urllib. error. HTTPError as e: return e. code, e. read (). decode ()[: 200 ], time. perf_counter () - t0 except Exception as e: return - 1, repr ( e ), time. perf_counter () - t0 Enter fullscreen mode Exit fullscreen mode Nothing clever yet. Now the part that actually taught me something: the classifier.

def classify ( status: int ) -> str: if status == 200: return " ok " if status in ( 400, 413, 422 ): return " too_long " # shrink the input if status == 429: return " rate_limited " # stop; this is my fault if 500 <= status < 600: return " server " # back off once, then stop return " unknown " Enter fullscreen mode Exit fullscreen mode 413 Content Too Large has a definition; it's in RFC 9110, and it means what it says. 429 means I'm the problem. If your harness treats both as "input too long," it will happily bisect toward zero while the real story is that you sent forty requests in ninety seconds. That's not a context limit. That's a queue you failed to build.

if " --dry-run " in sys. argv: def post ( prompt, timeout = 60.0 ): if len ( prompt ) >= 9900: # the stub's fake wall return 413, " stub: too long ", 0.0 return 200, " stub: ok ", 0.0 def probe ( n_chars: int, tries: int = 2 ) -> str: for attempt in range ( tries ): status, body, dt = post ( make_prompt ( n_chars )) kind = classify ( status ) print ( f " { n_chars: > 7 } chars status= { status: < 4 } { kind: < 12 } { dt: 5.2 f } s " ) if kind == " rate_limited ": print ( " 429 — stopping instead of bisecting. Sleep longer and rerun. " ) sys. exit ( 2 ) if kind == " server " and attempt == 0: time. sleep ( SLEEP * 2 ) continue return kind return " server " def find_limit ( lo: int = 500, hi: int = 200_000 ): probe ( lo ) # sanity check the small end while probe ( hi )!= " too_long ": # exponential growth, not guessing if hi >= 400_000: print ( " still fine at my ceiling; raise hi or change the task " ) return None lo, hi = hi, hi * 2 time. sleep ( SLEEP ) while hi - lo > max ( 200, lo // 10 ): # 10% resolution, not 1 char mid = ( lo + hi ) // 2 time. sleep ( SLEEP ) if probe ( mid ) == " too_long ": hi = mid else: lo = mid return lo Enter fullscreen mode Exit fullscreen mode Run it with the stub:

python probe_limit.py --dry-run Enter fullscreen mode Exit fullscreen mode And you should see roughly this trace — the numbers are from the stub, so you can reproduce them exactly:

500 chars status=200 ok 0.00s 200000 chars status=413 too_long 0.00s 100250 chars status=413 too_long 0.00s 50375 chars status=413 too_long 0.00s 25437 chars status=413 too_long 0.00s 12968 chars status=413 too_long 0.00s 6734 chars status=200 ok 0.00s 9851 chars status=200 ok 0.00s 11409 chars status=413 too_long 0.00s 10630 chars status=413 too_long 0.00s stop: largest passing size ≈ 9851 chars Enter fullscreen mode Exit fullscreen mode Look at the last two lines and then look at the stub: the fake wall sits at 9,900 characters, and the search stopped at 9,851. That gap is the lesson. Bisection returns a boundary, not the boundary — I told it to quit at 10% resolution because each probe is a real request and precision beyond that buys me nothing.

Nine probes, about twenty seconds of sleeping, one 429 that made me stop and rerun later with SLEEP = 6.0. The number it landed on is not the interesting output. The interesting output is what the number is: a property of one model on one day, not a spec I can paste into a config next month.

I re-ran three days later and got a different value. Not dramatically different, just different enough that hardcoding either one would have been a small lie. So the number lives in limits.json with a timestamp, and the worker reads it and refuses to start if the file is older than a week. Guess what happens when it's stale? I re-probe. Cheap.

Here's the consumer, the thing I actually wanted on Tuesday:

def windows ( text: str, limit: int ): start = 0 while start < len ( text ): end = min ( start + limit, len ( text )) if end < len ( text ): soft = text. rfind ( " \n\n ", start + limit // 2, end ) # break on a paragraph if soft!= - 1: end = soft yield start, text [ start: end ] start = end Enter fullscreen mode Exit fullscreen mode Each window gets one request, each result is appended to a JSONL file keyed by its start offset, and a crashed run resumes from the offsets already on disk. The napping-lid problem disappeared once I stopped running the loop on my laptop and let the free server hold it — same script, different machine, no drama. That's the whole trick. Nothing about my code assumed a particular host.

Character counts are a lousy stand-in for tokens. If you have a tokenizer installed, measure tokens and delete my make_prompt entirely. Also, bisection assumes failure is monotonic in input size — usually true for length, occasionally false when a request is rejected for a reason you didn't model, like a stray character sequence tripping a filter. I spot-check three small sizes after every run for exactly that reason.

And if you're on a paid tier with a documented limit and a throughput target, this harness is wasted effort. Read the doc, set the constant, move on. Don't run any of this against an endpoint you don't control, and don't halve the sleep between probes "to go faster" — you'll just measure your own impatience.

Your turn to predict something. If I drop SLEEP to zero and keep the same nine probes, which line of that trace changes first: the 413 at the top, or the 200 near the bottom? Write your answer as the error class you expect, then actually run the stub a few times and see whether your reasoning survives contact with your own harness. That gap between what you predicted and what printed is the only part of this article worth keeping.

MonkeyCode provides free models that can run this workflow.