# BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

Source: [arXiv](https://arxiv.org/abs/2611.09825v1)  
Feed7 permalink: https://feed7.dev/p/2611-09825v1-0g2rcid  
Published: 2026-11-06T18:13:54.000Z  
Trust: Needs Review (needs_review)

## Why Included

BitDecoding-FP4 co-designs KV-cache quantization and attention kernels, reporting near-BF16 accuracy and 2.4–3.0× peak throughput on NVIDIA B300. Its results are tied to specific formats, models, and hardware.

## Source Summary

BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV.

## Practical Implication

For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect.

## Agent-Ready Context

BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV.

For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect.

The figures are reported for specified models and B300 hardware; the material does not establish the same gains on other GPUs or serving systems. The accuracy numbers are relative to BF16 KV, not a general quality guarantee.

## Connected Context

Feed7 judgment across 1032 accumulated Signals:

This makes KV-cache quantization a concrete serving option: co-designed FP4 formats and kernels reportedly raise throughput while keeping accuracy near BF16 on the tested B300 setup. It adds implementation-level evidence to the candidates’ broader claims that cache movement and capacity constrain inference, while the hardware and model limits argue for workload-specific validation.

- [Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs](https://feed7.dev/p/2609-26796v1-0t57w0i) — Both target KV-cache costs in inference, but Flash-dLLM combines cache kernels with parallel decoding for diffusion models, while this paper focuses on quantized KV formats and fused attention; the gains address related costs and are not directly comparable.
- [What's New in Inference Engineering — Philip Kiely, Baseten](https://feed7.dev/p/what-s-new-in-inference-engineering-philip-kiely-baseten-1mlen9m) — The talk frames cache management and quantization as joint inference-tuning levers; this paper supplies a specific co-designed FP4 implementation and measured accuracy-throughput tradeoff on named hardware.
- [Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher](https://feed7.dev/p/deep-dive-on-llm-inference-at-scale-harshul-jain-audible-tanmay-sah-inde-04utjkh) — The deep dive identifies KV-cache size as a possible GPU capacity bottleneck for long contexts; this paper tests reducing that burden through FP4 cache formats, with results bounded to its reported models and B300.

## Context Map

- Layer: infra
- Domains: None
- Topics: None

## Uncertainty

- The figures are reported for specified models and B300 hardware; the material does not establish the same gains on other GPUs or serving systems. The accuracy numbers are relative to BF16 KV, not a general quality guarantee.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
