Multi-State Linear Attention with Generalized Second-Order Routing
MERLIN adds mean and variance estimates to how multi-state linear attention routes information from long-context segments. It improves reported RULER and LongBench results on two model variants, including against baselines using four times more memory.
MERLIN tracks the first and second moments of each segment’s keys, letting a query estimate both mean and variance in its alignment with that segment. On **Gated DeltaNet-1.3B** and **Mamba-2-780M**, it improves reported RULER and LongBench results, including over some baselines using **4× more memory**.
Builders exploring long-context models can watch this approach as a way to improve how compressed segment memories are read. It suggests routing quality may matter alongside the amount of memory available.
MERLIN tracks the first and second moments of each segment’s keys, letting a query estimate both mean and variance in its alignment with that segment. On **Gated DeltaNet-1.3B** and **Mamba-2-780M**, it improves reported RULER and LongBench results, including over some baselines using **4× more memory**. Builders exploring long-context models can watch this approach as a way to improve how compressed segment memories are read. It suggests routing quality may matter alongside the amount of memory available. These are results on multi-state variants and specific long-context evaluations. The material does not say how the method behaves in deployed agent workloads or across other model families.
MERLIN adds first- and second-moment routing to compressed segment memories and reports gains on two long-context model families. It suggests that how a model reads stored segments can matter alongside memory capacity, while leaving performance in deployed agent workloads unestablished.