# The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Source: [AI Engineer](https://www.youtube.com/watch?v=q2JrUKBMf0w)  
Feed7 permalink: https://feed7.dev/p/the-future-of-evals-from-llm-as-a-judge-to-agent-as-a-judge-aparna-dhina-1fu560o  
Published: 2026-07-24T20:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Fixed-rubric judges miss failures that emerge across long, variable agent trajectories. Arize argues for adding agent-based analysis while retaining deterministic and LLM-judge evals.

## Source Summary

Arize says it processes **over 100 million evals per month**; the average team runs about **12 eval jobs**, while top teams use more than 3,800 evaluators. As agents gained tools, memory, subagents, and long-running loops, their failures became harder to capture with fixed scores.

## Practical Implication

Keep deterministic checks and LLM judges for stable criteria, but add trajectory-aware analysis when the path itself matters. An evaluator agent can inspect traces for repeated tool calls, loops, forgotten context, incomplete work, or inefficient execution.

## Agent-Ready Context

Arize says it processes **over 100 million evals per month**; the average team runs about **12 eval jobs**, while top teams use more than 3,800 evaluators. As agents gained tools, memory, subagents, and long-running loops, their failures became harder to capture with fixed scores.

Keep deterministic checks and LLM judges for stable criteria, but add trajectory-aware analysis when the path itself matters. An evaluator agent can inspect traces for repeated tool calls, loops, forgotten context, incomplete work, or inefficient execution.

Agent-as-judge is presented as an additional eval type, not a replacement. The talk introduces Arize Signal as the implementation, but provides no comparative accuracy, cost, latency, or calibration results for deciding when it beats a fixed judge.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, multi-agent

## Uncertainty

- Agent-as-judge is presented as an additional eval type, not a replacement. The talk introduces Arize Signal as the implementation, but provides no comparative accuracy, cost, latency, or calibration results for deciding when it beats a fixed judge.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.