Tracing the Scientific Discovery Capacity of LLMs to Their Training Data
A study links higher rates of once-seen training facts to more scientific discoveries across two open-weight model families. The result points to training-data composition as a possible driver of discovery.
The study manipulates the rate of **monofacts**—facts observed once in training—and finds that higher rates correspond to more discoveries in materials science and regulatory genomics. The pattern held across two open-weight model families, sampling strategies, and reasoning budgets.
For model training or data curation, track the frequency of individual facts alongside overall dataset diversity. The authors report that diversity alone did not explain the relationship and that transfer across scientific domains was possible when paired with in-domain data.
The study manipulates the rate of **monofacts**—facts observed once in training—and finds that higher rates correspond to more discoveries in materials science and regulatory genomics. The pattern held across two open-weight model families, sampling strategies, and reasoning budgets. For model training or data curation, track the frequency of individual facts alongside overall dataset diversity. The authors report that diversity alone did not explain the relationship and that transfer across scientific domains was possible when paired with in-domain data. The supplied material does not identify the model families, data sizes, or effect magnitudes, so it does not establish a practical target monofact rate.
This narrows data-diversity advice by identifying the frequency of individual facts as a potentially relevant training-data property: the reported discovery pattern was not explained by diversity alone. The data-curation account supports deliberate dataset composition, but neither supplied record establishes a practical target for monofact rates or a universal effect.