arXivPaperNeeds Review

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Feeding a model partial reasoning from a weaker model during RLVR preserved exploration and improved large-k coverage. This suggests diversity can come from cross-model perturbation, not only regularization.

arXiv · Aug 27, 2026

Open Source Open Markdown Open JSON

Source Summary

The method gives an RLVR target model partial reasoning prefixes from a **smaller, weaker model**. These unfamiliar prefixes interrupt overconfidence, preserve generative diversity, and outperform vanilla RLVR across multiple math benchmarks.

Practical Implication

Model trainers should consider cross-model prefixes when RLVR collapses entropy or narrows pass@k coverage. The intervention requires **no additional SFT**, custom reward design, or elaborate prompting.

Agent-Ready Context

The method gives an RLVR target model partial reasoning prefixes from a **smaller, weaker model**. These unfamiliar prefixes interrupt overconfidence, preserve generative diversity, and outperform vanilla RLVR across multiple math benchmarks.

Model trainers should consider cross-model prefixes when RLVR collapses entropy or narrows pass@k coverage. The intervention requires **no additional SFT**, custom reward design, or elaborate prompting.

Reported gains become larger as **k scales up**, but the supplied abstract contains no effect sizes, model names, or compute costs. Evidence is limited to mathematical benchmarks, so transfer to coding-agent reasoning remains open.

Connected Context · Feed7 Judgment

This adds a low-complexity training intervention for RLVR exploration collapse: weaker-model prefixes can disrupt an overconfident target and recover diversity without new supervision or reward design. It strengthens the case that useful weak-to-strong transfer need not imitate the weaker model, while leaving coding transfer, effect size, and compute tradeoffs unresolved.

Weak-to-Strong Generalization via Direct On-Policy DistillationBoth transfer signal from a weaker model to a stronger one, but Direct-OPD uses the weak model’s policy change as dense reward whereas this method injects weak-model prefixes to broaden exploration.CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and DiversityBoth target diversity lost during post-training; CreativeInstruct learns explicit control spans, while weak-model guidance introduces unfamiliar trajectories during RLVR without additional SFT.Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion ParadigmsFusion improved single-sample accuracy without expanding solution coverage, whereas weak-model guidance specifically reports larger benefits as pass@k grows, making exploration breadth the distinguishing objective.DemoPSD: Disagreement-Modulated Policy Self-DistillationDemoPSD controls weak supervision through token-level disagreement to avoid leakage shortcuts; this method instead uses cross-model mismatch deliberately to interrupt overconfidence and preserve exploration.

Context Map

modelresearch#reasoning

Uncertainty

Reported gains become larger as **k scales up**, but the supplied abstract contains no effect sizes, model names, or compute costs. Evidence is limited to mathematical benchmarks, so transfer to coding-agent reasoning remains open.