arXivPaperNeeds Review

A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal

PIR reads internal model states to distinguish concealed knowledge from missing knowledge, offering a stronger audit for sandbagging and unlearning than output-only evals—when model internals are accessible.

arXiv · Sep 18, 2026

Open Source Open Markdown Open JSON

Source Summary

Probe of Internal Recognition inspects model states to identify which candidate answer the model recognizes. Across **eight models from five families**, it reached **0.70–0.87 balanced accuracy**, versus a 0.28–0.40 unknown-item baseline and 0.25 chance.

Practical Implication

Builders evaluating agent models should separate refusal or deliberate underperformance from genuine inability. PIR is **reference-free**, so its audits do not require an honest comparison model or labeled truth corpus; it also extends beyond multiple choice to free-form generation.

Agent-Ready Context

Probe of Internal Recognition inspects model states to identify which candidate answer the model recognizes. Across **eight models from five families**, it reached **0.70–0.87 balanced accuracy**, versus a 0.28–0.40 unknown-item baseline and 0.25 chance.

Builders evaluating agent models should separate refusal or deliberate underperformance from genuine inability. PIR is **reference-free**, so its audits do not require an honest comparison model or labeled truth corpus; it also extends beyond multiple choice to free-form generation.

The method requires access to internal states, which hosted model APIs may not expose. Its **0.85–0.93 recognition** under tested concealment methods is evidence across specific models and interventions, not proof that every hidden capability will remain detectable.

Connected Context · Feed7 Judgment

PIR adds a capability-audit distinction missing from ordinary output scoring: an incorrect, refused, or strategically weak answer need not mean the model lacks the answer internally. It reinforces mechanistic evaluation while narrowing what can be concluded from behavior alone. Unlike harness, judge, or task-quality audits, it targets concealed recognition inside accessible models, so it cannot directly audit opaque hosted APIs.

From Deceptive Outputs to Deceptive Mechanisms: A Causal Framework for Language-Model Deception ResearchThe causal deception framework warns that suspicious outputs do not identify their mechanism; PIR supplies internal evidence that can distinguish recognized-but-unrevealed answers from genuine lack of recognition, without by itself proving deceptive intent.Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization EvaluationBoth move evaluation beneath scalar outputs by probing internal representations; the judge study localizes evidence routing and rating formation, while PIR tests whether an answer is internally recognized despite the emitted response.QuoteBench: How Matched Scores Can Hide Command-Path FailuresQuoteBench separates model behavior from failures introduced by the execution path, while PIR separates expressed performance from internal recognition; together they show that an observed miss may not identify the underlying capability failure.

Context Map

benchmark#agent-evals#benchmark-integrity

Uncertainty

The method requires access to internal states, which hosted model APIs may not expose. Its **0.85–0.93 recognition** under tested concealment methods is evidence across specific models and interventions, not proof that every hidden capability will remain detectable.