AI EngineerVideoSource Linked

Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

An inference benchmark can bottleneck before the server does, silently missing its target load or inflating latency. Validate the client, workload, and sampling before trusting results.

AI Engineer · Sep 19, 2026

Open Source Open Markdown Open JSON

Source Summary

Google engineers asked benchmark clients for **200 QPS** and saw one deliver only **38 QPS**; another client-side overload added as much as **58 seconds** of apparent latency. Temperature and dataset sampling also changed throughput and token counts across runs.

Practical Implication

When evaluating an agent-serving stack, record planned versus delivered load, client timing, server metrics, dataset transformations, generation settings, and random variables. Replay workloads that resemble production, including multi-turn and agentic request patterns.

Agent-Ready Context

Google engineers asked benchmark clients for **200 QPS** and saw one deliver only **38 QPS**; another client-side overload added as much as **58 seconds** of apparent latency. Temperature and dataset sampling also changed throughput and token counts across runs.

When evaluating an agent-serving stack, record planned versus delivered load, client timing, server metrics, dataset transformations, generation settings, and random variables. Replay workloads that resemble production, including multi-turn and agentic request patterns.

InferencePerf addresses these issues with multiprocess generation, declarative configurations, and client observability, and demonstrated load generation at **5,000 QPS**. That shows harness capacity in the presented setup, not that every published result or production environment becomes comparable automatically.

Connected Context · Feed7 Judgment

This identifies the load generator itself as a benchmark confounder: requested QPS, observed QPS, and apparent latency can diverge before model or server quality is judged. It complements prior task-, judge-, and runtime-integrity failures with a client-capacity layer, making delivered-load telemetry, complete configuration capture, and production-shaped replay prerequisites for comparable serving results.

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like BenchmarksPAIChecker addresses whether benchmark tasks and oracles align, while this signal addresses whether the harness delivers and measures the intended load; trustworthy scores require both layers.Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared EndpointsThe shared-endpoint study shows unstable judgment outcomes, and this signal shows unstable or overloaded load generation; together they require repeatability checks for both evaluator behavior and benchmark execution.Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval BelferThe vLLM failures show that load-dependent runtime corruption can be mistaken for model behavior, making the request IDs and joined client/server telemetry recommended here consequential for diagnosis.Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2iThat talk questions prompts, tests, and verifiers; this signal adds traffic delivery, timing, sampling, and generation settings as independent reasons a leaderboard result may not reproduce.

Context Map

benchmark#benchmark-integrity#agent-evals#observability

Uncertainty

InferencePerf addresses these issues with multiprocess generation, declarative configurations, and client observability, and demonstrated load generation at **5,000 QPS**. That shows harness capacity in the presented setup, not that every published result or production environment becomes comparable automatically.