SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification
SpecCompact proposes parallel draft-and-verify summary slots to reduce agent-history compaction latency. Its reported speedups come with model- and task-dependent downstream trade-offs.
SpecCompact drafts structured summary slots in parallel with a fine-tuned small model, then has the target model verify and repair each line. Across AppWorld and τ³ with **two target models**, it reports **2.04–5.63×** faster compaction than Hermes-style summarization; a Hermes user study reports an **8.68× median speedup**.
For agents that frequently compact long histories, this is a design to evaluate when latency matters: parallelize independent summary slots, then verify them against the source history. Measure task performance as well as compaction time.
SpecCompact drafts structured summary slots in parallel with a fine-tuned small model, then has the target model verify and repair each line. Across AppWorld and τ³ with **two target models**, it reports **2.04–5.63×** faster compaction than Hermes-style summarization; a Hermes user study reports an **8.68× median speedup**. For agents that frequently compact long histories, this is a design to evaluate when latency matters: parallelize independent summary slots, then verify them against the source history. Measure task performance as well as compaction time. Separate downstream evaluations found trade-offs that depended on model and task. The reported gains do not establish that summaries preserve the information every agent workflow needs.
SpecCompact applies parallel drafting and target-model verification to context compaction, with reported speedups across two target models and separate task-dependent downstream trade-offs. It adds a latency-focused design to existing memory strategies, while leaving open whether its summaries preserve what a particular workflow needs.