Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Three ways to combine RLVR domain experts perform similarly on average but diverge by task. Choose Merge for cheap reuse, Mix RL for training from pooled data, and MOPD for preserving expert gains.
The study compares **Merge**, **Mix RL**, and **MOPD** using shared experts and data. Average performance differs by at most **1.4 points**, while one benchmark shows an **8.6-point** gap; all improve single-sample accuracy.
Choose Merge when trained experts already exist and fusion cost dominates. Use Mix RL to train one model from pooled data, tuning domain proportions; use MOPD when retaining domain-specific gains matters more than exceeding teachers or minimizing total cost.
The study compares **Merge**, **Mix RL**, and **MOPD** using shared experts and data. Average performance differs by at most **1.4 points**, while one benchmark shows an **8.6-point** gap; all improve single-sample accuracy. Choose Merge when trained experts already exist and fusion cost dominates. Use Mix RL to train one model from pooled data, tuning domain proportions; use MOPD when retaining domain-specific gains matters more than exceeding teachers or minimizing total cost. None of the three produced measurable gains in solution coverage, though held-out capabilities were not measurably reduced. Per-domain behavior can depart sharply from the close average, so aggregate scores are insufficient for selection.
This narrows RLVR consolidation from choosing a universal best method to matching deployment constraints: reuse cost favors Merge, pooled training favors Mix RL, and expert retention favors MOPD. Close averages conceal substantial task-level differences, and unchanged solution coverage means fusion should not be treated as evidence of broader reasoning exploration.