Open Brain
arXivPaperNeeds Review

A Living Benchmark for Information Retrieval from Electronic Health Records

BRIE generates refreshable EHR retrieval evaluations from longitudinal notes, addressing benchmark staleness and leakage while exposing omissions in multi-document clinical synthesis.

arXiv · Sep 24, 2026
Open Source Open MarkdownOpen JSON
Source Summary

BRIE automatically generates question-answer pairs from longitudinal health records, with its generator validated by **19 clinicians**. The study evaluates **nine LLMs** using **five inference strategies**.

Practical Implication

Builders of retrieval agents can borrow the living-benchmark pattern: validate the generation process, refresh cases to reduce leakage, and allow multiple reference answers where expert reasoning legitimately varies.

Agent-Ready Context
BRIE automatically generates question-answer pairs from longitudinal health records, with its generator validated by **19 clinicians**. The study evaluates **nine LLMs** using **five inference strategies**.

Builders of retrieval agents can borrow the living-benchmark pattern: validate the generation process, refresh cases to reduce leakage, and allow multiple reference answers where expert reasoning legitimately varies.

The evaluated systems frequently omitted important clinical details, especially when answers required synthesis across documents and encounters. The evidence is specific to electronic health records, where omissions carry unusually high stakes.
Connected Context · Feed7 Judgment

This turns clinical retrieval evaluation into a renewable, generator-validated benchmark rather than a fixed question set, while preserving multiple acceptable answers where expert reasoning differs. It reinforces prior evidence that omissions are a distinct high-stakes failure missed by plausible outputs or citation presence, and adds cross-encounter synthesis as a measured retrieval bottleneck rather than treating clinical QA as simple lookup.

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question AnsweringBRIE’s omission findings make claim coverage a necessary complement to citation accuracy and entailment; supported excerpts are insufficient if important clinical details never appear in the answer.Inside 847 Production Clinical AI Notes — Sebastian Fox, ComposoBoth identify consequential omissions in plausible clinical outputs; BRIE adds an automatically refreshed benchmark pattern, while the production study supports retaining expert judgment for failures outside a frozen rubric.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioClinician validation and multiple reference answers operationalize the biology-eval warning that legitimate expert workflows can vary and brittle single-answer graders may reject valid reasoning.Eval awareness in Claude Opus 4.6’s BrowseComp performanceRefreshing generated cases directly addresses the leakage risk illustrated by agents discovering fixed benchmark answers, though generator validation remains necessary to preserve task quality.
Context Map
benchmarkresearchdata#retrieval#agent-evals#benchmark-integrity
Uncertainty
The evaluated systems frequently omitted important clinical details, especially when answers required synthesis across documents and encounters. The evidence is specific to electronic health records, where omissions carry unusually high stakes.