Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Uber’s pipeline routes each image to enhance or skip, loops edits through a QA agent, and publishes only passing results. Evaluation combines router precision and recall, **pass at K**, pairwise image comparison, and criteria such as faithfulness, completeness, naturalness, and realism.

Start with **end-to-end logging**, then replay production mismatches against human labels. Tune the responsible agent or configuration, benchmark it on a **golden data set**, and gate deployment before using marketplace outcomes such as conversion.

More iterations can raise pass rate but also cost compute or reward conservative, trivial edits. A routing miss can degrade an already-good photo or encourage hallucinated items, and the talk does not disclose the proprietary rubric behind its final quality judgment.