arXivPaperNeeds Review

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

DARTS targets representation drift in merged decoder LLMs with entropy-weighted, position-aware correction, adding 0.1% parameters in the reported Llama-2-7B tests.

arXiv · Aug 28, 2026

Open Source Open Markdown Open JSON

Source Summary

DARTS addresses hidden-state drift after merging task-tuned decoder models. It combines an **entropy-weighted L1 loss** for decision-critical tokens with a **per-position additive bias** to handle drift that accumulates under causal attention.

Practical Implication

For builders experimenting with merged local models, the work suggests evaluating token-position effects rather than applying encoder-style correction uniformly. The reported module adds **0.1% of total parameters**.

Agent-Ready Context

DARTS addresses hidden-state drift after merging task-tuned decoder models. It combines an **entropy-weighted L1 loss** for decision-critical tokens with a **per-position additive bias** to handle drift that accumulates under causal attention.

For builders experimenting with merged local models, the work suggests evaluating token-position effects rather than applying encoder-style correction uniformly. The reported module adds **0.1% of total parameters**.

Evidence is limited to **Llama-2-7B** across HumanEval, GSM8K, and AlpacaEval. The abstract reports gains over standard surgery but gives no effect sizes, so generalization to newer model families and agent workloads remains open.

Connected Context · Feed7 Judgment

DARTS adds post-merge representation repair as a distinct customization lever beside training-data, mid-training, and self-distillation interventions. It narrows the repair problem to decoder-specific, position-dependent drift and offers a very small module, but its Llama-2-only evidence and missing effect sizes leave its value for current coding agents and newer open models unresolved.

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context ModelingOctoLong changes representations through structured mid-training data, whereas DARTS intervenes after task-tuned models are merged; together they distinguish capability training from merge-induced drift repair.$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillationβ-OPSD tunes reasoning behavior through self-distillation, while DARTS preserves decision-critical representations after merging, making them interventions at different stages of model customization.Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAIThe data-curation signal argues for improving task fit before adding compute; DARTS supplies a downstream corrective option when independently tuned models are subsequently combined.

Context Map

modelcodingresearch#open-models#reasoning

Uncertainty

Evidence is limited to **Llama-2-7B** across HumanEval, GSM8K, and AlpacaEval. The abstract reports gains over standard surgery but gives no effect sizes, so generalization to newer model families and agent workloads remains open.