arXivPaperNeeds Review
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.
arXiv
Source Summary
BLOOM-WILT revises an auditor’s multi-turn strategy and reweights target decoding using next-token distributions. Across **4 models and 8 behaviors**, it beat the baseline in **30 of 32 settings**.
Practical Implication
Agent evaluators should treat elicitation as part of the test harness, not a fixed prompt set. Adaptive probing may expose rare failure modes that ordinary sampling misses and can change comparative safety conclusions.
Agent-Ready Context
BLOOM-WILT revises an auditor’s multi-turn strategy and reweights target decoding using next-token distributions. Across **4 models and 8 behaviors**, it beat the baseline in **30 of 32 settings**. Agent evaluators should treat elicitation as part of the test harness, not a fixed prompt set. Adaptive probing may expose rare failure modes that ordinary sampling misses and can change comparative safety conclusions. The method requires access to the target’s next-token distribution, which many hosted APIs do not expose. Results cover eight specified behaviors, so broader reliability and transfer remain open questions.
Context Map
benchmarksecurity#agent-evals#benchmark-integrity#agent-reliabilityUncertainty
The method requires access to the target’s next-token distribution, which many hosted APIs do not expose. Results cover eight specified behaviors, so broader reliability and transfer remain open questions.