Your Free AI Lane Is Slow in the Queue, Not in the Model

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped.

Nothing in the model got slower. The jobs sat in a queue you never charted.

Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation.

This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff.

Your job record needs five timestamps. Anything less and you cannot separate waiting from working.

From those you get three derived numbers:

Gate on wait_share. A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing.

The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database.

#!/usr/bin/env python3 # two_clock.py — lane verdict from job events (JSON Lines) import json, sys def load ( path ): with open ( path ) as fh: for line in fh: line = line. strip () if line: yield json. loads ( line ) def pct ( values, q ): if not values: return 0.0 s = sorted ( values ) i = min ( len ( s ) - 1, int ( round ( q * ( len ( s ) - 1 )))) return s [ i ] def main ( path, wait_share_max = 0.60, margin_frac = 0.15 ): rows = list ( load ( path )) waits, gen, share = [], [], [] for r in rows: total = r [ " accept_ts " ] - r [ " enqueue_ts " ] if total <= 0: continue wait = r [ " start_ts " ] - r [ " enqueue_ts " ] waits. append ( wait ) gen. append ( total - wait ) share. append ( wait / total ) drain = max ( r [ " accept_ts " ] for r in rows ) first = min ( r [ " enqueue_ts " ] for r in rows ) deadline = max ( r [ " deadline_ts " ] for r in rows ) slack = deadline - drain margin = ( deadline - first ) * margin_frac print ( f " jobs= { len ( rows ) } wait_p50= { pct ( waits, 0.5 ):. 0 f } s wait_p95= { pct ( waits, 0.95 ):. 0 f } s " ) print ( f " gen_p50= { pct ( gen, 0.5 ):.

0 f } s wait_share_p95= { pct ( share, 0.95 ):. 2 f } " ) print ( f " slack= { slack:. 0 f } s margin_budget= { margin:.

0 f } s " ) if pct ( share, 0.95 ) > wait_share_max: print ( " VERDICT: spill — wait dominates, throughput is not your lever " ) elif slack < margin: print ( " VERDICT: spill — projected drain lands inside the safety margin " ) else: print ( " VERDICT: hold — the lane is inside budget " ) if __name__ == " __main__ ": main ( sys. argv [ 1 ]) Enter fullscreen mode Exit fullscreen mode Run it against a finished batch:

python3 two_clock.py lane-free.jsonl Enter fullscreen mode Exit fullscreen mode Expected output shape (illustrative, not a measurement from your system):

jobs=412 wait_p50=38s wait_p95=611s gen_p50=22s wait_share_p95=0.81 slack=1740s margin_budget=1080s VERDICT: spill — wait dominates, throughput is not your lever Enter fullscreen mode Exit fullscreen mode Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second.

wait_share_max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data.

margin_frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits.

Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell.

#!/usr/bin/env bash # fake_lane.sh — synthetic congestion, zero production impact set -euo pipefail HOLD = ${ HOLD_SECONDS:- 90 } DEADLINE = $(( $( date +%s ) + ${ WINDOW_SECONDS:- 3600 } )) while read -r id; do ENQ = $( date +%s ) sleep " $HOLD " echo "{ \" job_id \": \" $id \", \" enqueue_ts \": $ENQ, \" start_ts \": $( date +%s ), \" accept_ts \": $( date +%s ), \" deadline_ts \": $DEADLINE }" done Enter fullscreen mode Exit fullscreen mode Generate job ids, then run the same 40 jobs twice:

seq 1 40 > ids.txt HOLD_SECONDS = 5./fake_lane.sh < ids.txt > lane-fast.jsonl HOLD_SECONDS = 90./fake_lane.sh < ids.txt > lane-slow.jsonl python3 two_clock.py lane-fast.jsonl python3 two_clock.py lane-slow.jsonl Enter fullscreen mode Exit fullscreen mode The fast file should print hold. The slow file should print spill. If both print the same verdict, your gate is dead code and you should fix it before touching production routing.

Read the table top-down. The first row that trips decides. You do not average these signals into one score.

You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload.

One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change.

Keep the escape hatch boring and tested.

If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic.

If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.