Open Brain
arXivPaperNeeds Review

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

BitDecoding-FP4 co-designs KV-cache quantization and attention kernels, reporting near-BF16 accuracy and 2.4–3.0× peak throughput on NVIDIA B300. Its results are tied to specific formats, models, and hardware.

arXiv · Nov 6, 2026
Open Source Open MarkdownOpen JSON
Source Summary

BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV.

Practical Implication

For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect.

Agent-Ready Context
BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV.

For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect.

The figures are reported for specified models and B300 hardware; the material does not establish the same gains on other GPUs or serving systems. The accuracy numbers are relative to BF16 KV, not a general quality guarantee.
Connected Context · Feed7 Judgment

This makes KV-cache quantization a concrete serving option: co-designed FP4 formats and kernels reportedly raise throughput while keeping accuracy near BF16 on the tested B300 setup. It adds implementation-level evidence to the candidates’ broader claims that cache movement and capacity constrain inference, while the hardware and model limits argue for workload-specific validation.

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMsBoth target KV-cache costs in inference, but Flash-dLLM combines cache kernels with parallel decoding for diffusion models, while this paper focuses on quantized KV formats and fused attention; the gains address related costs and are not directly comparable.What's New in Inference Engineering — Philip Kiely, BasetenThe talk frames cache management and quantization as joint inference-tuning levers; this paper supplies a specific co-designed FP4 implementation and measured accuracy-throughput tradeoff on named hardware.Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI ResearcherThe deep dive identifies KV-cache size as a possible GPU capacity bottleneck for long contexts; this paper tests reducing that burden through FP4 cache formats, with results bounded to its reported models and B300.
Context Map
infra
Uncertainty
The figures are reported for specified models and B300 hardware; the material does not establish the same gains on other GPUs or serving systems. The accuracy numbers are relative to BF16 KV, not a general quality guarantee.