AI Proxy · benchmark

Benchmark — 3 configurations

2026-08-16 10:41 · proxy 0.2.0

Configuration

Long context — what the window actually holds

A model's advertised context is the prompt it will accept. It says nothing about whether the model can still find anything in there. Those are different properties and they part company well before the advertised number: a backend will happily prefill a prompt the model cannot actually read, return a confident answer, and report success at every layer. Nothing in a coding or agentic suite can catch that, because every one of their prompts fits in a few thousand tokens.

So this measures the second property directly. Five facts — short codes — are planted at fixed depths through a haystack of known size, and all five are asked for in a single request. The haystack is deliberately repetitive, enumerated ledger lines that all look alike, which is the hard case for binding a fact to a position: the number below is a lower bound, and a real document of the same size would score better.

Each rung is run five times, and each repeat is a different draw — the filler shifts from the first line and the needles jitter around their depths. Repeating one identical prompt would measure the backend's prompt cache instead, which serves repeats two through five off a prefix it already holds and returns five matching answers that look like consistency.

Read the depth columns before the totals. The first needle sits near the start on every repeat, because a backend that front-truncates an over-long prompt discards the beginning first — so a healthy start column is the evidence that the window is genuinely allocated, and a failing one means the prompt was silently cut rather than badly recalled. When start holds and the middle columns fall, that is a retrieval limit in the model. When start falls, it is a configuration problem in the stack.

Prefill tok/s is the prompt size over the time before the first token returned. At these sizes it is the only throughput figure that means anything — the answer is five short lines, so decode speed is noise next to the cost of reading the haystack.

Rung	Prompt tokens	Recalled	start	25%	50%	75%	end	Prefill tok/s	First token	Wait	Excluded
16k	15,987	75/75	15/15	15/15	15/15	15/15	15/15	2,791	5.7s	11.2s	—
64k	63,983	75/75	15/15	15/15	15/15	15/15	15/15	2,644	24.2s	30.2s	—
128k	128,007	75/75	15/15	15/15	15/15	15/15	15/15	2,332	54.9s	1:01	—
256k	262,197	74/75	15/15	15/15	15/15	14/15	15/15	1,833	2:23	2:30	—
300k	300,076	75/75	15/15	15/15	15/15	15/15	15/15	1,724	2:54	3:03	—
512k	512,144	63/65	13/13	13/13	12/13	13/13	12/13	1,303	6:33	6:35	2
600k	600,176	21/25	4/5	5/5	4/5	4/5	4/5	1,181	8:28	8:30	—
700k	700,187	18/25	4/5	5/5	3/5	3/5	3/5	1,068	10:55	10:57	—
800k	800,245	24/25	5/5	5/5	4/5	5/5	5/5	969	13:45	13:47	—
900k	900,284	17/25	5/5	5/5	2/5	2/5	3/5	895	16:46	16:49	—
all	—	517/540	106/108	108/108	100/108	101/108	102/108	—	—	—	2

2 unit(s) excluded: the reply hit the output cap before it finished answering. A truncated reply is not a recall failure, and scoring it as one measures the token budget rather than the model. Reasoning length here grows with the size of the prompt, not the difficulty of the task.

6 unit(s) are not counted at all: the request failed or never reported a prompt size, so the model never saw the haystack. A request that died is not a recall of zero.

Failures by kind: 11 reported as not found · 7 absent from the reply · 5 misattributed to another needle. These are not interchangeable. A model that writes NAME=MISSING has lost the fact and knows it — it can tell you it does not know. A model that answers with another needle's code has lost it and does not know, and will state the wrong answer with the same confidence as a right one. Both score zero; only one is safe to build on.

Why it failed

Every failed task, classified by the first thing that went wrong. A model that is cut off, silent, unwilling or simply incorrect has four different problems, and only one of them is answered by picking a different model — the others are a token budget, a prompt, and a policy decision.

Scores, standings and charts above describe longctx-lite only. 2 further runs from longctx-v1 are reported in their own sections rather than ranked alongside — a 24-task preference suite and a 119-task correctness suite produce numbers that do not mean the same thing.

Determinism

1 task did not settle: the same model, the same prompt, a different outcome between repeats. Treat these scores as samples, not measurements — and expect the coin to land wrong somewhere inside a long agent run.

Consistency

A single configuration, so there is nothing to rank — what matters here is consistency across the 30 requests. A wide gap between p50 and max means the number is not repeatable.

Metric	min	p50	p90	max
Time to first token ms	5,690	98,945	392,370	393,445
Decode rate tokens/sec	27.8	52.3	80.2	84.1
Total ms	11,133	114,816	510,684	686,881

Correctness by tier

The core tier confirms a model is not broken; it saturates for anything capable, which is exactly why the hard tier exists. Compare two models on the hard row when both score 100% on core.

Per-task correctness

Share of responses that passed every case for that task. A model strong everywhere except one task and a model mediocre throughout can share an overall average.

5 of 6 tasks were fully correct on every run; the ones that were not are listed below.

What the failures actually looked like

The first failing case per configuration: the call that was made, what came back, and what should have — or the compile error or timeout that stopped it. This is what a percentage point of correctness is made of.

longctx_512k — Recall five facts planted through a 512,000-token haystack. · 1 example

• nemotron-3.5-lightning-30b-a3b-ctx1m — hit the token cap mid-answer — the echo code, planted at the end → "PERIWINKLE-2051 does not appear anywhere in the reply", expected "PERIWINKLE-2051"

Hardware

What was tested

Every task asks for one answer in the task's language — Python, JavaScript under node, C and C++ under gcc, Rust, C#, or PHP, each run in a separate process under a timeout with the return value compared against the expected one. HTML and CSS tasks are graded structurally: the answer is parsed and checked against required structure (bindings, attributes, declarations in the right context) — a claim about the markup, not about how a browser renders it. Tasks whose toolchain is absent on the machine are skipped and listed here, never scored as zero. Fully correct counts only responses where every case for that task passed; cases is the share of individual cases that passed, so a near-miss still scores there. A response with no extractable code block scores zero — that measures instruction-following, not coding.

• Grading executes model-written code in a subprocess with a hard timeout, a scratch working directory and a stripped environment. That contains accidents and runaway loops; it is not a sandbox against deliberately hostile code.
• Requests go through this proxy so every one is logged and attributed, and carry x-client-name: ai-proxy-bench.

Cold-start cost

The price of making the model answerable, excluded from every measurement above: booting the server where one has to be booted (container start plus weight load) plus the discarded warm-up request. Counting only the warm-up understated a vLLM start twenty-fold — the weights are already in memory by the time that request arrives.

40,736 ms