# ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

Source: [arXiv](https://arxiv.org/abs/2607.28627v1)  
Feed7 permalink: https://feed7.dev/p/2607-28627v1-0pc8021  
Published: 2026-07-30T17:59:56.000Z  
Trust: Needs Review (needs_review)

## Why Included

ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.

## Source Summary

ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance.

## Practical Implication

For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100.

## Agent-Ready Context

ReToken trains **one learnable embedding** as a retrieval target, selecting sparse query-relevant visual tokens from a pre-filled KV cache. It is trained on a small image-QA dataset and applied to long visual contexts where distractors otherwise degrade performance.

For visual agents, test retrieval over cached tokens before expanding the context processed by the model. Reported gains reach **13.4 points** for Qwen3VL-8B and **12.4 points** for InternVL3.5 on Visual Haystacks, with training and long-video inference fitting on one H100.

The method transfers zero-shot to LVBench for an **8.0-point** gain with Qwen3VL-8B, but the abstract provides no latency, token-selection rate, or broader model coverage. The results remain benchmark evidence rather than an application-level reliability claim.

## Connected Context

Feed7 judgment across 297 accumulated Signals:

This adds a trained, token-level visual retrieval mechanism to the broader case for selecting evidence before reasoning over long context. Unlike training-free replay or external retrieval, it operates over a pre-filled visual KV cache with one learned target embedding. The benchmark gains support sparse query-relevant selection, but missing latency, selection-rate, model-breadth, and application evidence limit production conclusions.

- [ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning](https://feed7.dev/p/2607-02509v1-11vodps) — Both replay query-relevant evidence instead of processing all long context uniformly, but ReToken learns visual-token selection whereas ReContext is a training-free text harness.
- [RLM: Recursive Language Models for Large Codebases - Shashi, Superagentic AI](https://feed7.dev/p/rlm-recursive-language-models-for-large-codebases-shashi-superagentic-ai-1w21nex) — Both treat large context as evidence to inspect selectively and return in bounded form, extending the same design principle across visual caches and code repositories.
- [AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j](https://feed7.dev/p/ai-on-your-lakehouse-context-comes-in-shapes-not-queries-zach-blumenfeld-0r4s9u5) — The graph-retrieval candidate reinforces that retrieval shape should match the data; ReToken supplies a specialized shape for sparse evidence inside long visual contexts.

## Context Map

- Layer: context
- Domains: image, video
- Topics: retrieval, context-engineering, context-caching

## Uncertainty

- The method transfers zero-shot to LVBench for an **8.0-point** gain with Qwen3VL-8B, but the abstract provides no latency, token-selection rate, or broader model coverage. The results remain benchmark evidence rather than an application-level reliability claim.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.