Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses
STRATA grades the evidence and statistical choices behind temporal claims, not just the final verdict. Its experiments find that models often fail when selecting which records to use.
**STRATA** contains **8,735** clinical timeline hypotheses with gold analysis plans, evidence sets, p-values, and verdicts, reproduced by a deterministic oracle. The paper also proposes ART—articulate, retrieve, test—to expose each decision in a staged analysis.
When building agentic data analysis, evaluate whether the system selected the right records and test, not only whether its final conclusion matches. In the study, record selection was the main failure point; ART brought a 31B open model to within **0.01** of frontier models under the same paradigm.
**STRATA** contains **8,735** clinical timeline hypotheses with gold analysis plans, evidence sets, p-values, and verdicts, reproduced by a deterministic oracle. The paper also proposes ART—articulate, retrieve, test—to expose each decision in a staged analysis. When building agentic data analysis, evaluate whether the system selected the right records and test, not only whether its final conclusion matches. In the study, record selection was the main failure point; ART brought a 31B open model to within **0.01** of frontier models under the same paradigm. The benchmark focuses on vaccine adverse-event reports and temporal clinical hypotheses. The reported comparison depends on providing the codebook to frontier models, so it does not show that those models reach similar results without that guidance.
STRATA makes record selection and test choice inspectable in clinical hypothesis analysis, and reports record selection as the main failure point. Its deterministic oracle and scoped vaccine-report setting provide concrete evaluation evidence, while the codebook condition limits the frontier-model comparison.