ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.
ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance.
For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100.
ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance. For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100. The method transfers zero-shot to LVBench for an **8.0-point** gain with Qwen3VL-8B, but the abstract provides no latency, token-selection rate, or broader model coverage. The results remain benchmark evidence rather than an application-level reliability claim.
This adds a trained, token-level visual retrieval mechanism to the broader case for selecting evidence before reasoning over long context. Unlike training-free replay or external retrieval, it operates over a pre-filled visual KV cache with one learned target embedding. The benchmark gains support sparse query-relevant selection, but missing latency, selection-rate, model-breadth, and application evidence limit production conclusions.