AI EngineerVideoSource Linked
The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Fixed-rubric judges miss failures that emerge across long, variable agent trajectories. Arize argues for adding agent-based analysis while retaining deterministic and LLM-judge evals.
AI Engineer · Jul 24, 2026
Source Summary
Arize says it processes **over 100 million evals per month**; the average team runs about **12 eval jobs**, while top teams use more than 3,800 evaluators. As agents gained tools, memory, subagents, and long-running loops, their failures became harder to capture with fixed scores.
Practical Implication
Keep deterministic checks and LLM judges for stable criteria, but add trajectory-aware analysis when the path itself matters. An evaluator agent can inspect traces for repeated tool calls, loops, forgotten context, incomplete work, or inefficient execution.
Agent-Ready Context
Arize says it processes **over 100 million evals per month**; the average team runs about **12 eval jobs**, while top teams use more than 3,800 evaluators. As agents gained tools, memory, subagents, and long-running loops, their failures became harder to capture with fixed scores. Keep deterministic checks and LLM judges for stable criteria, but add trajectory-aware analysis when the path itself matters. An evaluator agent can inspect traces for repeated tool calls, loops, forgotten context, incomplete work, or inefficient execution. Agent-as-judge is presented as an additional eval type, not a replacement. The talk introduces Arize Signal as the implementation, but provides no comparative accuracy, cost, latency, or calibration results for deciding when it beats a fixed judge.
Context Map
benchmarkcoding#agent-evals#agent-reliability#multi-agentUncertainty
Agent-as-judge is presented as an additional eval type, not a replacement. The talk introduces Arize Signal as the implementation, but provides no comparative accuracy, cost, latency, or calibration results for deciding when it beats a fixed judge.