arXivPaperNeeds Review

DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

DARTS targets representation drift in merged decoder LLMs with entropy-weighted, position-aware correction, adding 0.1% parameters in the reported Llama-2-7B tests.

arXiv

Open Source Open Markdown Open JSON

Source Summary

DARTS addresses hidden-state drift after merging task-tuned decoder models. It combines an **entropy-weighted L1 loss** for decision-critical tokens with a **per-position additive bias** to handle drift that accumulates under causal attention.

Practical Implication

For builders experimenting with merged local models, the work suggests evaluating token-position effects rather than applying encoder-style correction uniformly. The reported module adds **0.1% of total parameters**.

Agent-Ready Context

DARTS addresses hidden-state drift after merging task-tuned decoder models. It combines an **entropy-weighted L1 loss** for decision-critical tokens with a **per-position additive bias** to handle drift that accumulates under causal attention.

For builders experimenting with merged local models, the work suggests evaluating token-position effects rather than applying encoder-style correction uniformly. The reported module adds **0.1% of total parameters**.

Evidence is limited to **Llama-2-7B** across HumanEval, GSM8K, and AlpacaEval. The abstract reports gains over standard surgery but gives no effect sizes, so generalization to newer model families and agent workloads remains open.

Context Map

modelcodingresearch#open-models#reasoning

Uncertainty

Evidence is limited to **Llama-2-7B** across HumanEval, GSM8K, and AlpacaEval. The abstract reports gains over standard surgery but gives no effect sizes, so generalization to newer model families and agent workloads remains open.