# DolphinBench: Mapping the Pareto Frontier of Agent Memory

Source: [arXiv](https://arxiv.org/abs/2609.24971v1)  
Feed7 permalink: https://feed7.dev/p/2609-24971v1-0k257sd  
Published: 2026-09-21T17:54:35.000Z  
Trust: Needs Review (needs_review)

## Why Included

DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.

## Source Summary

DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it.

## Practical Implication

Use this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project.

## Agent-Ready Context

DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it.

Use this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project.

The material does not report baseline scores or show how representative the personas are of software work. Its history/no-history filter strengthens task validity, but it does not by itself isolate which memory component produced a gain.

## Connected Context

Feed7 judgment across 843 accumulated Signals:

This makes memory evaluation more defensible by requiring each task to depend causally on supplied history and by treating accuracy, cost and latency as a joint frontier. It narrows generic recall tests to downstream knowledge work, but does not identify which memory subsystem caused an improvement or show that three synthetic personas represent software-agent workloads.

- [Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs](https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz) — Both emphasize long-horizon evaluation, but DolphinBench adds a causal history/no-history check while Vending-Bench adds the warning that simulated and real-world behavior can diverge.
- [QuoteBench: How Matched Scores Can Hide Command-Path Failures](https://feed7.dev/p/2608-13547v1-130h6xd) — QuoteBench shows why DolphinBench’s reported frontier must be attributed to the full memory-and-harness execution path, especially because DolphinBench does not isolate individual memory components.
- [PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks](https://feed7.dev/p/2607-28587v1-0u0uow2) — PAIChecker’s prompt–oracle alignment test parallels DolphinBench’s history-dependence filter: both validate that an instance actually measures the capability it claims before scores are trusted.
- [Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments](https://feed7.dev/p/2607-28591v1-169rlqh) — Change2Task could supply renewable software-work tasks for extending DolphinBench beyond its three personas, while its own alignment and workload-coverage requirements would still need separate validation.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: agent-memory, agent-evals, benchmark-integrity

## Uncertainty

- The material does not report baseline scores or show how representative the personas are of software work. Its history/no-history filter strengthens task validity, but it does not by itself isolate which memory component produced a gain.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.