{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.24971v1",
  "slug": "2609-24971v1-0k257sd",
  "url": "https://feed7.dev/p/2609-24971v1-0k257sd",
  "title": "DolphinBench: Mapping the Pareto Frontier of Agent Memory",
  "why_included": "DolphinBench tests whether agent memory improves real task completion, while requiring accuracy, cost and latency so retrieval quality cannot hide impractical tradeoffs.",
  "summary": "DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it.",
  "practical_implication": "Use this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project.",
  "agent_context": "DolphinBench uses **three knowledge-work personas**, each with roughly **500k tokens** of message history and **200 tasks**. Every task is checked by requiring an agent to pass with the relevant history and fail without it.\n\nUse this shape of evaluation for agent memory: test downstream work rather than prompted recall, verify that history is causally useful, and report **accuracy, total cost and latency** together. The dataset and evaluation code are available from the project.\n\nThe material does not report baseline scores or show how representative the personas are of software work. Its history/no-history filter strengthens task validity, but it does not by itself isolate which memory component produced a gain.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.24971v1",
    "published_at": "2026-09-21T17:54:35.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [],
  "topics": [
    "agent-memory",
    "agent-evals",
    "benchmark-integrity"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The material does not report baseline scores or show how representative the personas are of software work. Its history/no-history filter strengthens task validity, but it does not by itself isolate which memory component produced a gain."
  ],
  "connected_context": {
    "meaning": "This makes memory evaluation more defensible by requiring each task to depend causally on supplied history and by treating accuracy, cost and latency as a joint frontier. It narrows generic recall tests to downstream knowledge work, but does not identify which memory subsystem caused an improvement or show that three synthetic personas represent software-agent workloads.",
    "corpus_size": 843,
    "generated_at": "2026-09-22T09:08:51.561Z",
    "connections": [
      {
        "title": "Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=cO8qC6HBuBg",
        "feed7_url": "https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz",
        "reason": "Both emphasize long-horizon evaluation, but DolphinBench adds a causal history/no-history check while Vending-Bench adds the warning that simulated and real-world behavior can diverge."
      },
      {
        "title": "QuoteBench: How Matched Scores Can Hide Command-Path Failures",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.13547v1",
        "feed7_url": "https://feed7.dev/p/2608-13547v1-130h6xd",
        "reason": "QuoteBench shows why DolphinBench’s reported frontier must be attributed to the full memory-and-harness execution path, especially because DolphinBench does not isolate individual memory components."
      },
      {
        "title": "PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.28587v1",
        "feed7_url": "https://feed7.dev/p/2607-28587v1-0u0uow2",
        "reason": "PAIChecker’s prompt–oracle alignment test parallels DolphinBench’s history-dependence filter: both validate that an instance actually measures the capability it claims before scores are trusted."
      },
      {
        "title": "Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.28591v1",
        "feed7_url": "https://feed7.dev/p/2607-28591v1-169rlqh",
        "reason": "Change2Task could supply renewable software-work tasks for extending DolphinBench beyond its three personas, while its own alignment and workload-coverage requirements would still need separate validation."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-21T17:54:35.000Z",
  "modified_at": "2026-09-21T17:54:35.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-24971v1-0k257sd",
    "json": "https://feed7.dev/p/2609-24971v1-0k257sd.json",
    "markdown": "https://feed7.dev/p/2609-24971v1-0k257sd.md"
  }
}