{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2610.35826v1",
  "slug": "2610-35826v1-0t6qm4i",
  "url": "https://feed7.dev/p/2610-35826v1-0t6qm4i",
  "title": "Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?",
  "why_included": "A study of Jev-style models finds that repeatable answers can still become wrong when option order changes. Stable accuracy alone does not show that a model resists selection bias.",
  "summary": "The study shuffled candidate options across **eight presentations** of 1,700 questions. Jev’s accuracy changed by less than two percentage points, yet its cumulative failure set grew **27–29%** compared with the original ordering.",
  "practical_implication": "When evaluating models that choose among candidates, test whether answers survive equivalent option permutations. Repeating a fixed order or aggregating votes can miss failures that appear only when presentation order changes.",
  "agent_context": "The study shuffled candidate options across **eight presentations** of 1,700 questions. Jev’s accuracy changed by less than two percentage points, yet its cumulative failure set grew **27–29%** compared with the original ordering.\n\nWhen evaluating models that choose among candidates, test whether answers survive equivalent option permutations. Repeating a fixed order or aggregating votes can miss failures that appear only when presentation order changes.\n\nThe results cover two Jev versions, 12 reproductions, and six generative LLMs on two benchmarks. Voting helped Jev modestly, but even unanimous answers were often wrong.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2610.35826v1",
    "published_at": "2026-10-26T17:48:55.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "research"
  ],
  "topics": [
    "benchmark-integrity",
    "model-selection",
    "reasoning"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The results cover two Jev versions, 12 reproductions, and six generative LLMs on two benchmarks. Voting helped Jev modestly, but even unanimous answers were often wrong."
  ],
  "connected_context": {
    "meaning": "The shuffled presentations leave Jev’s average accuracy nearly unchanged while exposing substantially more failures across runs. This shows why a stable score can hide selection sensitivity, and why evaluating candidate selection requires tracking the cumulative failure set as well as accuracy.",
    "corpus_size": 928,
    "generated_at": "2026-10-27T11:06:55.362Z",
    "connections": [
      {
        "title": "WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.04008v1",
        "feed7_url": "https://feed7.dev/p/2608-04008v1-0d8bqwv",
        "reason": "Both challenge majority voting as a reliability shortcut: the tournament study reports no gain under correlated errors, while this study finds unanimous Jev answers can still be wrong."
      },
      {
        "title": "Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.04001v1",
        "feed7_url": "https://feed7.dev/p/2608-04001v1-1gj91hk",
        "reason": "The permutation protocol is another evaluation condition that should be specified and replayable; the scaling paper likewise argues that results depend on reporting the full inference and evaluation protocol."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-10-26T17:48:55.000Z",
  "modified_at": "2026-10-26T17:48:55.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2610-35826v1-0t6qm4i",
    "json": "https://feed7.dev/p/2610-35826v1-0t6qm4i.json",
    "markdown": "https://feed7.dev/p/2610-35826v1-0t6qm4i.md"
  }
}