# Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Source: [arXiv](https://arxiv.org/abs/2610.42147v1)  
Feed7 permalink: https://feed7.dev/p/2610-42147v1-1v0l2qi  
Published: 2026-10-30T17:59:34.000Z  
Trust: Needs Review (needs_review)

## Why Included

Latent Sparse Attention uses learned clusters to avoid scoring every key in ultra-long contexts. The reported gains are large at 10M tokens, but apply to one attention layer in the evaluated model.

## Source Summary

Latent Sparse Attention (LSA) groups keys hierarchically in a learned latent space, letting queries score clusters instead of every key. On GLM-5.3-FLASH (QWEN-3.8-27B) at **10M tokens**, Distilled-LSA reports **22.7×** prefill and **4.6×** decode speedups over the model's original sparse attention.

## Practical Implication

For agent runs that accumulate very long histories, this points to attention indexing as a possible bottleneck and optimization target. The paper offers a distilled variant for pretrained models and a native variant trained on next-token loss.

## Agent-Ready Context

Latent Sparse Attention (LSA) groups keys hierarchically in a learned latent space, letting queries score clusters instead of every key. On GLM-5.3-FLASH (QWEN-3.8-27B) at **10M tokens**, Distilled-LSA reports **22.7×** prefill and **4.6×** decode speedups over the model's original sparse attention.

For agent runs that accumulate very long histories, this points to attention indexing as a possible bottleneck and optimization target. The paper offers a distilled variant for pretrained models and a native variant trained on next-token loss.

The speedups are for one attention layer, and the baselines differ: the figures in parentheses compare against full attention. The material says accuracy stayed similar across tasks for Distilled-LSA; Native-LSA's full-attention accuracy match and 1.6× training speedup are reported at 128k tokens.

## Context Map

- Layer: model
- Domains: research
- Topics: None

## Uncertainty

- The speedups are for one attention layer, and the baselines differ: the figures in parentheses compare against full attention. The material says accuracy stayed similar across tasks for Distilled-LSA; Native-LSA's full-attention accuracy match and 1.6× training speedup are reported at 128k tokens.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.