DolphinBench: Mapping the Pareto Frontier of Agent Memory
DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.
DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it.
Use this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project.
DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it. Use this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project. The material does not report baseline scores or show how representative the personas are of software work. Its history/no-history filter strengthens task validity, but it does not by itself isolate which memory component produced a gain.
This makes memory evaluation more defensible by requiring each task to depend causally on supplied history and by treating accuracy, cost and latency as a joint frontier. It narrows generic recall tests to downstream knowledge work, but does not identify which memory subsystem caused an improvement or show that three synthetic personas represent software-agent workloads.