LLM Benchmarks
What a 1M-token context window actually holds
1 configuration · longctx-lite · 6 tasks · ollama · NVIDIA GB10
Needles planted through haystacks from 16k to 900k tokens. Recall is flawless to 300k and frays past 512k, the window is genuinely allocated the whole way, and a 900k prompt is 16:46 before the first token.
Full report →Tokens per second doesn't tell you how long you wait
6 configurations · full-v2 · 119 tasks · ollama + vllm · NVIDIA GB10
The most correct model is the slowest by decode rate, the second-fastest takes eight times longer to finish an answer, and one reasoning model spends 15× the tokens per solved task to score lower.
Full report →Concurrency costs latency, not accuracy — with one exception
48 configurations · coding-v3 · 47 tasks · alone vs 4× concurrent · llamacpp + ollama + vllm · NVIDIA GB10
A full cache × concurrency matrix — 96 runs across 48 configurations. The two knobs turn out to be independent, and one model quietly loses accuracy under load.
Full report →43,000 requests, 2.1 billion tokens, 37 cents
usage report 18 days of whatever actually ran · 42,987 requests · 2.1B tokens
Not a benchmark — a rolling snapshot of everything my local models actually did, and what the electricity to produce it came to.
Full report →What a cold cache and a 256k context actually cost
19 configurations · coding-v1 · 18 tasks · cached vs cold · llamacpp + ollama + vllm · NVIDIA GB10
Nineteen configurations measured warm and cold, plus one model held at 32k, 128k and 256k context — two knobs that turn out to cost far less than expected.
Full report →40 local models on a GB10
20 configurations · coding-v2 · 29 tasks · alone vs 4× concurrent · llamacpp + ollama + vllm · NVIDIA GB10
Twenty local models on a 122 GB GB10, each run alone and at 4× concurrency, graded on the coding-v2 suite.
Full report →