# Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

Source: [arXiv](https://arxiv.org/abs/2609.15964v1)  
Feed7 permalink: https://feed7.dev/p/2609-15964v1-08frwh7  
Published: 2026-09-14T17:53:51.000Z  
Trust: Needs Review (needs_review)

## Why Included

Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.

## Source Summary

A harness evaluates **12 LLMs** on **222 synthetic clinical questions** drawn from four practice guidelines. It checks whether each factual claim has a citation, whether the cited text is verbatim, and whether that text fully supports the claim.

## Practical Implication

Builders should add claim-level entailment checks instead of treating citation presence or exact copying as proof. Retrieval-agent evaluations need separate measures for coverage, quotation fidelity, and complete substantiation.

## Agent-Ready Context

A harness evaluates **12 LLMs** on **222 synthetic clinical questions** drawn from four practice guidelines. It checks whether each factual claim has a citation, whether the cited text is verbatim, and whether that text fully supports the claim.

Builders should add claim-level entailment checks instead of treating citation presence or exact copying as proof. Retrieval-agent evaluations need separate measures for coverage, quotation fidelity, and complete substantiation.

Most models quote source text for over 90% of claims, yet support can remain weak. Claude Opus 5 reaches **98.0% verbatim coverage** but only **37.1% full substantiation**; the clinical, synthetic setup limits direct generalization to other domains.

## Connected Context

Feed7 judgment across 778 accumulated Signals:

This replaces citation presence as a sufficient retrieval-eval signal with a three-part acceptance test: claim coverage, exact quotation, and full support. It reinforces prior warnings that apparently strong evaluators can miss domain-specific failures, while adding a directly inspectable evidence-to-claim check. Its clinical synthetic data supports the evaluation decomposition, not a universal performance estimate.

- [Domain-Specific Hallucination Detection in Large Language Models](https://feed7.dev/p/2609-11878v1-1ltrudx) — Both show that a strong surface evaluation signal can conceal weak biomedical grounding, supporting domain-matched checks rather than reliance on a generic hallucination or citation score.
- [Verifiable Environments for AI in Biology — Kenny Workman, LatchBio](https://feed7.dev/p/verifiable-environments-for-ai-in-biology-kenny-workman-latchbio-1vs6y66) — The claim-level support test supplies a deterministic component for scientific evaluation, while the biology evidence explains why expert review is still needed when valid interpretations exceed brittle graders.
- [Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation](https://feed7.dev/p/2609-01604v1-02vljon) — Claim-level decomposition provides observable failure categories that can help distinguish missed evidence from faulty final integration when auditing an LLM judge.
- [Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning](https://feed7.dev/p/2609-04194v1-0milybu) — Both caution against mistaking legible text for valid evidence: verbatim quotations may not substantiate claims, just as readable reasoning steps may not cause correct answers.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: agent-evals, retrieval, benchmark-integrity

## Uncertainty

- Most models quote source text for over 90% of claims, yet support can remain weak. Claude Opus 5 reaches **98.0% verbatim coverage** but only **37.1% full substantiation**; the clinical, synthetic setup limits direct generalization to other domains.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.