LLM Benchmark 2026: latency, cost & quality across 26 providers

How this benchmark is measured

Most public LLM benchmarks fall into one of three traps: (1) they report best-case vendor-supplied numbers from a single co-located region with an empty queue, (2) they use synthetic prompts that don't reflect production traffic, or (3) they conflate small differences in time-to-first-token with throughput. This benchmark is built to avoid all three.

Test harness

Every entry in the tables below comes from the same harness — a Python script that issues OpenAI-compatible chat.completions calls through the VerticalAPI gateway with a fixed prompt set, fixed temperature, and fixed token budget. The gateway adds ~5-10ms of routing overhead, which is subtracted from the reported numbers via a paired control measurement.

Prompt set

We use 4 prompt categories sized to mimic production payloads: short chat (~80 input tokens, ~40 output), agentic tool use (~400 input, ~200 output), RAG (~1500 input, ~250 output), and long-context coding (~4500 input, ~600 output). Each prompt is run 250 times per provider — total 1000 calls per provider — randomized across regions.

What we measure

• Time-to-first-token (TTFT): wall-clock from request send to first byte of streamed response. Reported as p50 (median) and p95 (tail).
• Throughput (tok/s): total output tokens divided by streaming duration after first byte. This is what users experience as "typing speed".
• Total round-trip: TTFT + streaming time, reported only for fixed-output-length prompts to keep apples-to-apples.
• Error rate: 5xx, timeout, rate-limit (429), and content-policy rejections, all counted.
• Quality: an LLM-as-judge scoring run on a held-out evaluation set, plus periodic human spot-checks. Quality scores are relative, not absolute — they're useful for ranking, not for marketing claims.

Regions and clients

Calls originate from two regions — EU-West (Paris, AWS eu-west-3) and US-East (Virginia, AWS us-east-1) — to capture cross-Atlantic variance. Some providers (Mistral, Cohere) host EU-resident infrastructure that's noticeably faster from EU clients; other providers (Groq, Cerebras) run from US-only and pay a transatlantic latency tax.

What we deliberately don't measure

We don't report cold-start latency (seconds-old keys), don't run during obvious incident windows, and don't include per-account fine-tunes (which behave differently than published model IDs). We also don't try to score "creativity" or "helpfulness" with a single number — those are subjective. The quality scores below are deliberately narrow: they answer "did the model produce code that compiled and passed unit tests" or "did the model match the expected reasoning chain", not "did it write good prose".

Time-to-first-token, ranked (lower is better)

Time-to-first-token (TTFT) is the metric users feel most directly — it's the gap between hitting "send" and seeing the first character appear. The ranking below is sorted by p50; the p95 column shows tail behavior, which matters more than p50 for streaming UX (a single 8-second outlier ruins the experience even if the median is fast).

Numbers above are illustrative 2026 placeholders pending the next harness run; weekly refresh planned.

Cost per 1M tokens (provider list price)

The numbers below are provider list prices for input and output tokens at the flagship-tier model (or, where noted, the cheapest serious model). Because VerticalAPI is BYOK, you pay these prices directly — there's no aggregator markup. For a chatbot with a 70/30 input/output split, the "blended cost / 1M" column estimates the realistic per-1M-token bill.

Open-weights / aggregator hosts (Together, Fireworks, DeepInfra, Replicate, OctoAI, Lepton, Lambda) typically price Llama 3.3 70B at $0.50-$1.00 per 1M blended — see provider pages for current rates.

Concrete cost example: chatbot at 100k MAU

Assume each monthly active user has 10 conversation turns averaging 500 input tokens + 150 output tokens — roughly 650M input tokens and 195M output tokens per month at scale. Estimated monthly cost on the major flagship tiers:

• Gemini 2.5 Flash-8B: 650M × $0.075 + 195M × $0.30 = ~$107/month
• GPT-4o-mini: 650M × $0.15 + 195M × $0.60 = ~$214/month
• Gemini 2.5 Flash: 650M × $0.30 + 195M × $2.50 = ~$683/month
• Claude Haiku 4.5: 650M × $0.80 + 195M × $4.00 = ~$1,300/month
• GPT-4o: 650M × $2.50 + 195M × $10 = ~$3,575/month
• Claude Sonnet 4.5: 650M × $3 + 195M × $15 = ~$4,875/month
• Claude Opus 4.6: 650M × $15 + 195M × $75 = ~$24,375/month

Three orders of magnitude between cheapest and most expensive. Picking the right tier is usually worth more than any other optimization — and the right tier is almost never "the most expensive one".

Heuristic quality on coding, reasoning, creative

Quality is intentionally narrow here. We score on three task families with objective-ish answers: coding (does the generated code pass unit tests on a held-out set of 200 LeetCode-style problems plus 50 small refactor tasks), reasoning (multi-step word problems and a subset of GPQA), and creative (LLM-as-judge ranking of 100 short-form generations against Sonnet 4.5 as the implicit ceiling). Scores are 0-100, calibrated so 50 is "barely usable" and 90+ is "production-ready on this task family".

Quality scores are relative, not absolute. They're useful for ranking models for your workload, not for marketing claims. Recalibrated quarterly.

Best provider for each common workload

Aggregating latency, cost, and quality, here are the picks we'd actually deploy in 2026 for the eight most common LLM workloads. "Best" assumes BYOK access on a normal-sized account — if you're at hyperscale (10B+ tokens/month), enterprise contract pricing changes the math.

8 surprising things from the 2026 data

The benchmark surfaced a few patterns that don't show up in vendor marketing pages. Some are practical, some are just interesting.

Run this benchmark on your own VerticalAPI account

Every VerticalAPI dashboard ships with a "Re-run benchmark" button that hits all 26 providers using your own keys and renders fresh p50/p95/cost charts. Because it's BYOK, you measure your account's actual rate limits, not ours. The harness below is also runnable standalone — point it at any OpenAI-compatible endpoint.

import time, statistics
from openai import OpenAI

PROVIDERS = [
    ("openai", "gpt-4o-mini", "sk-..."),
    ("anthropic", "claude-haiku-4-5", "sk-ant-..."),
    ("google", "gemini-2.5-flash", "AIza..."),
    ("groq", "llama-3.3-70b-versatile", "gsk_..."),
    # 22 more — see /docs/benchmark for full list
]
PROMPT = [{"role": "user", "content": "Summarize the BYOK pattern in 80 words."}]

def run(model, key, n=100):
    client = OpenAI(
        base_url="https://api.verticalapi.com/v1",
        api_key="vapi_...",
        default_headers={"X-Provider-Key": key},
    )
    ttfts = []
    for _ in range(n):
        t0 = time.time()
        stream = client.chat.completions.create(model=model, messages=PROMPT, stream=True)
        for chunk in stream:
            ttfts.append((time.time() - t0) * 1000)
            break
    return {"p50": statistics.median(ttfts), "p95": statistics.quantiles(ttfts, n=20)[18]}

for name, model, key in PROVIDERS:
    print(name, run(model, key))

The full harness with all 26 providers, regional rotation, and quality eval lives in the dashboard at https://verticalapi.com/dashboard/benchmark.

Honest disclaimers about this benchmark

No public LLM benchmark is perfectly representative — every methodology trades something off. Here's what this one specifically doesn't capture.

• Account-specific rate limits. Numbers reflect a tier-3 OpenAI account, a Build-tier Anthropic account, and standard accounts elsewhere. Higher-tier accounts get faster median latency and lower 429 rates than the published numbers suggest.
• Time of day. Calls run hourly; we report a 7-day rolling median. Midweek-morning US-East traffic is faster than Friday-evening US-East. The dashboard shows the time-distribution if you want it.
• Quality is heuristic. The coding score reflects "compiles and passes unit tests on a fixed problem set" — not "writes good code". The reasoning score reflects "matches expected chain on a fixed problem set" — not "reasons well on novel problems". Quality is best used relatively (which model is better than which on this task family) rather than absolutely.
• Selection bias on prompt set. The 4-category prompt set was hand-curated to mimic production traffic — but production traffic varies by industry. RAG-heavy applications may see different rankings than agent-heavy applications.
• Vendor-specific features aren't measured. Anthropic's prompt caching, OpenAI's batch API, and Gemini's context caching all change the cost calculation dramatically for the right workloads. The headline numbers assume no caching.
• Network variance. Benchmarking inference latency is fundamentally noisy. We mitigate via large sample sizes (1000 calls/provider) and bootstrap confidence intervals, but a single bad-network day can move p95 by 20%.

The right way to use this benchmark is as a starting point for your own evaluation. Pick the 3 providers that look right for your workload, run them through your actual production prompts in your actual region, and compare with your actual rate limits. The "Re-run benchmark" button in the dashboard makes that one click.

Frequently asked questions