BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.
BLOOM-WILT revises an auditor’s multi-turn strategy and reweights target decoding using next-token distributions. Across **4 models and 8 behaviors**, it beat the baseline in **30 of 32 settings**.
Agent evaluators should treat elicitation as part of the test harness, not a fixed prompt set. Adaptive probing may expose rare failure modes that ordinary sampling misses and can change comparative safety conclusions.
BLOOM-WILT revises an auditor’s multi-turn strategy and reweights target decoding using next-token distributions. Across **4 models and 8 behaviors**, it beat the baseline in **30 of 32 settings**. Agent evaluators should treat elicitation as part of the test harness, not a fixed prompt set. Adaptive probing may expose rare failure modes that ordinary sampling misses and can change comparative safety conclusions. The method requires access to the target’s next-token distribution, which many hosted APIs do not expose. Results cover eight specified behaviors, so broader reliability and transfer remain open questions.
This expands benchmark integrity from choosing tasks and graders to actively searching for behavior: a fixed prompt set can understate rare failures and even distort comparisons between models. It complements prior oracle, counterfactual, and trace-validity controls, but its dependence on target logits limits hosted-model use and its eight behaviors do not establish broad transfer.