# onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Source: [arXiv](https://arxiv.org/abs/2608.02595v1)  
Feed7 permalink: https://feed7.dev/p/2608-02595v1-0l7cc8k  
Published: 2026-08-03T17:58:27.000Z  
Trust: Needs Review (needs_review)

## Why Included

onepot-Bench 0 evaluates chemistry models with three lab-oriented tests, including private experimental data to reduce contamination risk. It is a useful eval design pattern, though results and reproducibility are absent here.

## Source Summary

**onepot-Bench 0** contains **three evaluations** for synthetic chemistry: ChemAbacus tests tool-free cheminformatics and numerical reasoning, SynthRefusal examines safety behavior, and SynthBench uses private lab data for reaction prediction and catalyst selection.

## Practical Implication

Builders evaluating research agents should test distinct failure modes rather than collapse competence, safety, and domain judgment into one score. Private, task-relevant data can also reduce the risk that benchmark examples appeared in model training corpora.

## Agent-Ready Context

**onepot-Bench 0** contains **three evaluations** for synthetic chemistry: ChemAbacus tests tool-free cheminformatics and numerical reasoning, SynthRefusal examines safety behavior, and SynthBench uses private lab data for reaction prediction and catalyst selection.

Builders evaluating research agents should test distinct failure modes rather than collapse competence, safety, and domain judgment into one score. Private, task-relevant data can also reduce the risk that benchmark examples appeared in model training corpora.

The supplied material reports the benchmark design but no model scores, dataset size, protocol detail, or access terms. Because SynthBench relies on proprietary experiments, independent reproduction and contamination auditing remain open questions.

## Connected Context

Feed7 judgment across 340 accumulated Signals:

This narrows benchmark-integrity guidance for scientific agents into three separately scored concerns: computational competence, safety refusal, and lab-grounded judgment. Its private experimental data strengthens contamination resistance, but also trades away some reproducibility and auditability; without scores or protocols, it establishes an evaluation design rather than evidence that any system performs well.

- [Verifiable Environments for AI in Biology — Kenny Workman, LatchBio](https://feed7.dev/p/verifiable-environments-for-ai-in-biology-kenny-workman-latchbio-1vs6y66) — Both argue that scientific-agent evaluation must reflect real experimental analysis and distinct valid paths; the biology case additionally warns that brittle deterministic grading may reject legitimate alternatives.
- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — onepot-Bench’s private lab tasks address the contamination concern behind distrust of leaderboard results, while its undisclosed protocol and proprietary data preserve the candidate’s concerns about inspectability and verifier coverage.
- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — Both use original, non-public task material to reduce training contamination, but each remains limited as a general capability proxy because its domain and task coverage are narrow.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- The supplied material reports the benchmark design but no model scores, dataset size, protocol detail, or access terms. Because SynthBench relies on proprietary experiments, independent reproduction and contamination auditing remain open questions.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.