Same question, same context: compare LLM responses against answers you can verify

Disclosure: SinapsData develops Sinaps Studio, a commercial Windows x64 desktop application. This article was drafted by an AI assistant under the product owner’s direction; the proposed tests are not reported experimental results.

A fast answer is useful only if it answers the question. Before choosing a model for a workflow, try a smaller question: given identical evidence and instructions, which configured model produces the required result, how long does it take, and what usage does its provider report?

Interface example from Sinaps Studio. This screenshot is not a controlled benchmark: it does not establish identical system instructions or complete configurations. Its timers, counters and percentage labels should not be read as independently validated accuracy results.

For each case below, send exactly the same complete request to every model. Start a fresh conversation. Use this system instruction:

Use only the supplied context. Return only the requested JSON object. Do not infer missing information. Do not use tools.

Keep tools disabled, with no retrieval or additional documents attached. Save the actual system prompt, user message and serialized context alongside every response. Equal text in the visible question box alone does not prove equal effective context.

Record provider, model identifier and revision where available, request date, output limit, temperature and supported seed. For local execution, also record runtime, hardware and quantization. Mark unavailable settings explicitly rather than inventing equivalent defaults.

1. Arithmetic. Context: A: 12 units at 2.50; B: 8 units at 4.00; C: 3 units at 10.00. No tax or discounts. Question: “Return the total as a two-decimal string in the key total.”

2. Extraction. Context: Order AX-17. Dispatch date: 2030-04-08. Customer: Nara. Carrier: not stated. Question: “Return order_id, dispatch_date and carrier; use null for missing values.”

Reference: {"order_id":"AX-17","dispatch_date":"2030-04-08","carrier":null}

3. Missing evidence. Reuse case 2’s context. Question: “Return the stated tracking number in tracking_number, or null if absent.”

These are deliberately small checks, not an intelligence ranking. Parse each response as JSON and compare its keys and values with the reference, ignoring object-key order. Record malformed JSON, incorrect values and unsupported additions separately. This scoring can be performed outside Studio; it is not a claim that Studio automatically grades responses.

Measure end-to-end elapsed time from request submission until the complete response arrives. Record time to first token separately only when instrumentation supports it; a full-response timer is not TTFT. NVIDIA’s metric definitions explain this distinction.

Preserve provider-reported input, output, cached and reasoning token fields when available. Missing usage is unknown, not zero. Tokenization depends on the tokenizer and formatting; Hugging Face’s tokenizer documentation describes this processing. Fewer reported tokens alone do not establish a cheaper request: apply the provider’s applicable pricing and billing categories separately.

Use an empty log like this—these are placeholders, not measurements:

Repeat each case, preserve failures, vary model order, and separate warm/cold runs and cache states where controllable. Keep concurrency fixed; identify provider-side details you cannot observe. Publish the run count and raw records before summarizing any spread.

Sinaps Studio provides a side-by-side workspace for inspecting configured models. Explore the application.