TTPO: Test-Time Policy Optimization
TTPO adapts a model at test time without labels by treating agreement and disagreement differently, reducing the damage from wrong majority-vote pseudo-labels.
TTPO uses majority votes as pseudo-labels but avoids trusting them uniformly. It distills agreeing rollouts with on-policy self-distillation and penalizes disagreeing rollouts with grouped reinforcement learning, while token selection down-weights converged positions and targets confident errors.
Without labels, **TTPO matched label-supervised OPSD on five benchmarks**. In test-time training it raised **Qwen3-1.7B from 38.0% to 45.2%**, suggesting adaptive inference can improve a small model without an external answer key.
TTPO uses majority votes as pseudo-labels but avoids trusting them uniformly. It distills agreeing rollouts with on-policy self-distillation and penalizes disagreeing rollouts with grouped reinforcement learning, while token selection down-weights converged positions and targets confident errors. Without labels, **TTPO matched label-supervised OPSD on five benchmarks**. In test-time training it raised **Qwen3-1.7B from 38.0% to 45.2%**, suggesting adaptive inference can improve a small model without an external answer key. The evidence is limited to competition-level mathematical reasoning in the supplied abstract. Test-time parameter updates also differ from ordinary agent retries or critique loops, so applicability to coding agents and the operational cost of per-task adaptation remain open questions.
TTPO strengthens disagreement-aware test-time adaptation by assigning different learning rules to agreeing and dissenting rollouts instead of collapsing them into one majority label. Matching a supervised comparator without labels is notable, but the evidence narrows the result to mathematical reasoning. Because TTPO updates weights per task, it occupies a more operationally demanding point in the test-time toolbox than routing, sampling, or frozen-state optimization.