Pick Your LLM With Your Own Prompt: A Side-by-Side Model Comparison Tool

Benchmarks and leaderboards are useful, but they answer a different question from the one I usually have.

They tell you how a model did on someone else's test. When I need a model for a real use case, I want to know how it handles my prompt: the one my team will send every day, with our tone, our format rules and our edge cases.

The quickest way I know to answer that is to try the prompt on a few models and read the answers. So I built a small tool that makes that loop fast:

An LLM judge scores each answer and explains why, which helps when the answers are long. It's a proof of concept built with FastAPI and React, and the code is on GitHub: jindalankuraws/ai-evals.

Browser (React) │ POST /api/run (prompt, settings, models, judge) ▼ FastAPI backend ├── settings check every filled-in setting must be supported by every model ├── fan-out one async task per model │ ├── OpenRouter adapter │ ├── OpenAI adapter │ ├── Anthropic adapter │ └── OpenAI-compatible adapter ├── stream all answers return on one server-sent events response └── judge POST /api/runs/{id}/evaluate, once the stream ends Enter fullscreen mode Exit fullscreen mode The backend is Python with FastAPI and httpx. The frontend is React, TypeScript, Vite and Tailwind CSS. Runs are kept in memory, which is enough for a tool you use interactively.

The settings every model receives are one small Pydantic model. Anything left empty is not sent at all, so each model uses its own default.

class SharedParams ( BaseModel ): """ Inference params applied to every model in a run; anything unset uses the model ' s default. """ temperature: Optional [ float ] = Field ( None, ge = 0, le = 2 ) top_p: Optional [ float ] = Field ( None, gt = 0, le = 1 ) max_tokens: Optional [ int ] = Field ( None, ge = 1 ) Enter fullscreen mode Exit fullscreen mode Before a run starts, the backend checks each filled-in setting against what each model says it supports. If one model can't take a setting, the run is rejected with a message naming that model, rather than silently dropping the setting for some models and not others.

for field in set_params: key = PARAM_TO_CATALOG_KEY [ field ] if key not in supported: problems. append ( " %s does not support ' %s '. Clear the param for all models, " " or remove the model. " % ( model_id, key ) ) Enter fullscreen mode Exit fullscreen mode Where the support list comes from depends on the provider. OpenRouter publishes supported_parameters for every model in its catalog, so that check is exact. OpenAI and Anthropic don't list sampling parameters per model, so the app assumes they are supported. That assumption is the first item in the limitations section below.

Every model has a composite id: provider::model, for example openrouter::anthropic/claude-haiku-4.5 or anthropic::claude-haiku-4.5. The backend splits the id to choose the adapter, and each adapter only ever sees the bare model name.

Each model runs in its own asyncio task. Every task pushes its events into one shared queue, tagged with the model id, and the route forwards them to the browser as they arrive. Whichever model produces a token first shows it first.

async def consume ( model_id: str ) -> None: try: adapter = adapter_for ( model_id ) _, bare_model = parse_composite ( model_id ) async for ev in adapter. stream_completion ( bare_model, prompt, system_prompt, params ): await queue. put ( TaggedEvent ( model_id, ev )) except Exception as exc: # never let one model's crash kill the fan-out await queue. put ( TaggedEvent ( model_id, StreamEvent ( kind = " error ", error = str ( exc )))) finally: await queue. put ( None ) # sentinel: this model is finished Enter fullscreen mode Exit fullscreen mode Two details matter here:

On the browser side, the run is a POST with a JSON body. The browser's built-in EventSource only takes a URL and a withCredentials flag, with no way to set the method or send a body ( MDN ). So the frontend reads the response with fetch and a stream reader, and splits the server-sent events itself.

Reading three long answers carefully takes time, so a separate model scores them. Each of the five metrics has a weight and a 1–5 scale, with written descriptions of what a 1 and a 5 look like:

There is one judge call per metric, and each call sees every answer, so the scores are relative to each other. Scores are scaled to 0–1 and combined using the weights. When the top two finish within 0.15 of each other, one extra head-to-head call decides between them.

LLM judges have known biases. The MT-Bench paper describes position bias (favouring an answer because of where it appears) and self-enhancement bias (favouring answers that resemble the judge's own) ( Zheng et al., 2023 ). The tool handles these in three ways.

1. The judge never sees model names. Answers are labelled A, B and C, and the mapping back to models stays on the server.

2. The order is shuffled on every call. Letters are assigned in a random order once per run. Then each metric call shows the answers in a fresh random order, so no model is always first.

rng = random. Random ( seed ) completed = [ r for r in record. results. values () if r. text and not r. error ] rng. shuffle ( completed )... letters = { r. model_id: LETTERS [ i ] for i, r in enumerate ( completed )} responses = [( letters [ r. model_id ], r. text ) for r in completed ]... _call_metric ( adapter, record, m, rng. sample ( responses, len ( responses )), expected_letters, params ) Enter fullscreen mode Exit fullscreen mode The optional seed is there so tests get the same order every time.

3. The judge can't belong to a compared model's family. The family of a model is its vendor prefix on OpenRouter, or the provider for direct OpenAI and Anthropic models. So openai::gpt-4.1-mini and openrouter::openai/gpt-4o count as the same family.

def _family ( model_id: str ) -> str: provider, model = parse_composite ( model_id ) if " / " in model: return model. split ( " / ", 1 )[ 0 ] if provider == " openai-compatible ": return model return provider Enter fullscreen mode Exit fullscreen mode The judge's reply must be JSON with a score and a reason for every letter. The backend asks for a strict JSON schema where the provider supports one. It then validates the reply with Pydantic and checks that every letter appears exactly once. A reply that fails gets one retry, and a metric that still fails is left out. Its weight is shared across the metrics that did score, and the UI says which metric is missing.

Four providers means four slightly different APIs. These details affected the design:

async def _complete ( adapter, model_id, user, system, params, schema ): """ Call the judge, retrying once without temperature if the model rejects it. """ raw = await adapter. complete ( model_id, user, system, params, schema ) if raw. error and params. temperature is not None and " temperature " in raw. error. lower (): params = SharedParams () raw = await adapter. complete ( model_id, user, system, params, schema ) return raw, params Enter fullscreen mode Exit fullscreen mode Errors can arrive partway through a stream. Anthropic can send an error event mid-stream, for example when it is overloaded ( Anthropic ).

OpenRouter does the same with an error field in a chunk ( OpenRouter ). Both adapters treat that as a failed answer, so a half-finished answer is never judged as if it were complete. Try it You need at least one provider key, or a local OpenAI-compatible server.

# backend cd backend python3 -m venv.venv source.venv/bin/activate pip install -r requirements.txt cp.env.example.env # add at least one provider key uvicorn app.main:app --reload --port 8000 # frontend, in a second terminal cd frontend npm install npm run dev Enter fullscreen mode Exit fullscreen mode Then open http://localhost:5173, fill two or three slots, and type your prompt.

The code is at github.com/jindalankuraws/ai-evals. When you pick a model for a real workload, what decides it for you: your own test prompts, a judge model's scores, or a person reading the answers?