Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
A study of Jev-style models finds that repeatable answers can still become wrong when option order changes. Stable accuracy alone does not show that a model resists selection bias.
The study shuffled candidate options across **eight presentations** of 1,700 questions. Jev’s accuracy changed by less than two percentage points, yet its cumulative failure set grew **27–29%** compared with the original ordering.
When evaluating models that choose among candidates, test whether answers survive equivalent option permutations. Repeating a fixed order or aggregating votes can miss failures that appear only when presentation order changes.
The study shuffled candidate options across **eight presentations** of 1,700 questions. Jev’s accuracy changed by less than two percentage points, yet its cumulative failure set grew **27–29%** compared with the original ordering. When evaluating models that choose among candidates, test whether answers survive equivalent option permutations. Repeating a fixed order or aggregating votes can miss failures that appear only when presentation order changes. The results cover two Jev versions, 12 reproductions, and six generative LLMs on two benchmarks. Voting helped Jev modestly, but even unanimous answers were often wrong.
The shuffled presentations leave Jev’s average accuracy nearly unchanged while exposing substantially more failures across runs. This shows why a stable score can hide selection sensitivity, and why evaluating candidate selection requires tracking the cumulative failure set as well as accuracy.