arXivPaperNeeds Review

Score Centering Stabilizes Off-policy Reinforcement Learning

Training–inference mismatch may destabilize LLM reinforcement learning through accumulating score drift. An additive centering correction matched or beat importance sampling in the reported tests.

arXiv · Sep 17, 2026

Open Source Open Markdown Open JSON

Source Summary

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

Practical Implication

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

Agent-Ready Context

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.

Connected Context · Feed7 Judgment

This identifies score drift as a specific, correctable failure mode when off-policy RL uses mismatched rollout and training engines. It makes score centering a lower-cost systems lever to test before enforcing engine parity, including under quantization and staleness, while leaving its generality beyond the reported training settings unresolved.

The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent GloballyBoth address quantized models, but at different stages: the candidate guides precision allocation, while this Signal addresses score drift when quantization contributes to training–inference mismatch.RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward ConstructionRRC changes how comparative judgments become RL rewards; score centering instead stabilizes how off-policy training consumes scores, making the two potentially complementary parts of an RL pipeline.UE5M3 FP4 Block Scaling for Stable Language Model PretrainingThe candidate seeks stable low-precision pretraining through numerical-format design, whereas this Signal reports an additive correction for instability when training and rollout execution differ, including under quantization.

Context Map

model

Uncertainty

The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.