Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.
A harness evaluates **12 LLMs** on **222 synthetic clinical questions** drawn from four practice guidelines. It checks whether each factual claim has a citation, whether the cited text is verbatim, and whether that text fully supports the claim.
Builders should add claim-level entailment checks instead of treating citation presence or exact copying as proof. Retrieval-agent evaluations need separate measures for coverage, quotation fidelity, and complete substantiation.
A harness evaluates **12 LLMs** on **222 synthetic clinical questions** drawn from four practice guidelines. It checks whether each factual claim has a citation, whether the cited text is verbatim, and whether that text fully supports the claim. Builders should add claim-level entailment checks instead of treating citation presence or exact copying as proof. Retrieval-agent evaluations need separate measures for coverage, quotation fidelity, and complete substantiation. Most models quote source text for over 90% of claims, yet support can remain weak. Claude Opus 5 reaches **98.0% verbatim coverage** but only **37.1% full substantiation**; the clinical, synthetic setup limits direct generalization to other domains.
This replaces citation presence as a sufficient retrieval-eval signal with a three-part acceptance test: claim coverage, exact quotation, and full support. It reinforces prior warnings that apparently strong evaluators can miss domain-specific failures, while adding a directly inspectable evidence-to-claim check. Its clinical synthetic data supports the evaluation decomposition, not a universal performance estimate.