ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.
ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance.
For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100.
ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance. For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100. The method transfers zero-shot to LVBench for an **8.0-point** gain with Qwen3VL-8B, but the abstract provides no latency, token-selection rate, or broader model coverage. The results remain benchmark evidence rather than an application-level reliability claim.
This extends selective long-context retrieval into the model’s cached visual-token space: a tiny trained target can recover query-relevant evidence without processing every image or video token equally. It reinforces bounded evidence selection as an alternative to expanding context, while leaving operational value uncertain because latency, selection rates, broader model coverage, and application reliability are unreported.