A Living Benchmark for Information Retrieval from Electronic Health Records
BRIE generates refreshable EHR retrieval evaluations from longitudinal notes, addressing benchmark staleness and leakage while exposing omissions in multi-document clinical synthesis.
BRIE automatically generates question-answer pairs from longitudinal health records, with its generator validated by **19 clinicians**. The study evaluates **nine LLMs** using **five inference strategies**.
Builders of retrieval agents can borrow the living-benchmark pattern: validate the generation process, refresh cases to reduce leakage, and allow multiple reference answers where expert reasoning legitimately varies.
BRIE automatically generates question-answer pairs from longitudinal health records, with its generator validated by **19 clinicians**. The study evaluates **nine LLMs** using **five inference strategies**. Builders of retrieval agents can borrow the living-benchmark pattern: validate the generation process, refresh cases to reduce leakage, and allow multiple reference answers where expert reasoning legitimately varies. The evaluated systems frequently omitted important clinical details, especially when answers required synthesis across documents and encounters. The evidence is specific to electronic health records, where omissions carry unusually high stakes.
This turns clinical retrieval evaluation into a renewable, generator-validated benchmark rather than a fixed question set, while preserving multiple acceptable answers where expert reasoning differs. It reinforces prior evidence that omissions are a distinct high-stakes failure missed by plausible outputs or citation presence, and adds cross-encounter synthesis as a measured retrieval bottleneck rather than treating clinical QA as simple lookup.