# CompressionBench: How can your coding agents compress LLMs?

Source: [arXiv](https://arxiv.org/abs/2611.06858v1)  
Feed7 permalink: https://feed7.dev/p/2611-06858v1-1vojzui  
Published: 2026-11-04T18:14:50.000Z  
Trust: Needs Review (needs_review)

## Why Included

CompressionBench tests whether coding agents can compress language models under storage and time limits; verification passed for 29 of 36 submissions, with generation tests catching failures perplexity missed.

## Source Summary

In **CompressionBench**, agents receive a model checkpoint, storage cap, inference environment, and time budget, then produce compressed weights and runnable inference code. **29 of 36 submissions** passed independent verification; the best per-task designs averaged a 3.4% perplexity increase and 77.33% downstream accuracy, versus 79.56% for BF16.

## Practical Implication

For agent-driven optimization, use end-to-end checks alongside local metrics: full-model feedback reversed 17 of 79 local rankings, and generation and inference tests caught failures that perplexity did not. The benchmark treats experimental judgment and capability validation as part of the task.

## Agent-Ready Context

In **CompressionBench**, agents receive a model checkpoint, storage cap, inference environment, and time budget, then produce compressed weights and runnable inference code. **29 of 36 submissions** passed independent verification; the best per-task designs averaged a 3.4% perplexity increase and 77.33% downstream accuracy, versus 79.56% for BF16.

For agent-driven optimization, use end-to-end checks alongside local metrics: full-model feedback reversed 17 of 79 local rankings, and generation and inference tests caught failures that perplexity did not. The benchmark treats experimental judgment and capability validation as part of the task.

The study covers **six agents and six model–size settings**. Its results describe this benchmark setup and do not establish how agents perform on other compression goals or environments.

## Connected Context

Feed7 judgment across 1009 accumulated Signals:

CompressionBench evaluates agent work through runnable compressed models and downstream checks, showing that local metrics such as perplexity can misrank candidate designs. It adds measured evidence for end-to-end acceptance when agents optimize models, while keeping conclusions bounded to six agents and six model–size settings in this benchmark.

- [SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving](https://feed7.dev/p/2609-26777v1-1ylj2uv) — Both benchmarks find local checks can miss failures revealed by end-to-end verification; CompressionBench demonstrates this for model compression, while SWE-Serve tests production inference changes.
- [Can Large Language Models Recover Semantic Optimization Opportunities That Compilers Miss?](https://feed7.dev/p/2608-03983v1-07mv55y) — SeGaBench requires independent correctness and performance validation of agent-generated optimizations; CompressionBench similarly shows that local perplexity is insufficient without full-model and inference checks.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: coding-agents, agent-evals, agent-reliability

## Uncertainty

- The study covers **six agents and six model–size settings**. Its results describe this benchmark setup and do not establish how agents perform on other compression goals or environments.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.