# Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Source: [AI Engineer](https://www.youtube.com/watch?v=31GUkCBD-Uc)  
Feed7 permalink: https://feed7.dev/p/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-1cqjbe2  
Published: 2026-07-24T22:00:25.000Z  
Trust: Source Linked (source_linked)

## Why Included

Uber’s image-editing agent uses routing, iterative QA, golden-set gates, and production feedback to avoid costly edits, hallucinated food, and quality regressions.

## Source Summary

Uber’s pipeline routes each image to enhance or skip, loops edits through a QA agent, and publishes only passing results. Evaluation combines router precision and recall, **pass at K**, pairwise image comparison, and criteria such as faithfulness, completeness, naturalness, and realism.

## Practical Implication

Start with **end-to-end logging**, then replay production mismatches against human labels. Tune the responsible agent or configuration, benchmark it on a **golden data set**, and gate deployment before using marketplace outcomes such as conversion.

## Agent-Ready Context

Uber’s pipeline routes each image to enhance or skip, loops edits through a QA agent, and publishes only passing results. Evaluation combines router precision and recall, **pass at K**, pairwise image comparison, and criteria such as faithfulness, completeness, naturalness, and realism.

Start with **end-to-end logging**, then replay production mismatches against human labels. Tune the responsible agent or configuration, benchmark it on a **golden data set**, and gate deployment before using marketplace outcomes such as conversion.

More iterations can raise pass rate but also cost compute or reward conservative, trivial edits. A routing miss can degrade an already-good photo or encourage hallucinated items, and the talk does not disclose the proprietary rubric behind its final quality judgment.

## Context Map

- Layer: benchmark
- Domains: image
- Topics: agent-evals, agent-reliability, generative-media

## Uncertainty

- More iterations can raise pass rate but also cost compute or reward conservative, trivial edits. A routing miss can degrade an already-good photo or encourage hallucinated items, and the talk does not disclose the proprietary rubric behind its final quality judgment.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.