BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
BitDecoding-FP4 co-designs KV-cache quantization and attention kernels, reporting near-BF16 accuracy and 2.4–3.0× peak throughput on NVIDIA B300. Its results are tied to specific formats, models, and hardware.
BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV.
For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect.
BitDecoding-FP4 combines three quantization changes with fused probability quantization and attention operations on paged KV caches. It reports accuracy within **0.5 points** of BF16 with NVFP4 and within **1 point** with MXFP4; on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, peak serving throughput is **2.4–3.0×** BF16 KV. For long-context agent serving, KV-cache formats and the kernels around them are a joint capacity, accuracy, and throughput decision. The paper's implementation is available on GitHub for builders with a compatible stack to inspect. The figures are reported for specified models and B300 hardware; the material does not establish the same gains on other GPUs or serving systems. The accuracy numbers are relative to BF16 KV, not a general quality guarantee.
This makes KV-cache quantization a concrete serving option: co-designed FP4 formats and kernels reportedly raise throughput while keeping accuracy near BF16 on the tested B300 setup. It adds implementation-level evidence to the candidates’ broader claims that cache movement and capacity constrain inference, while the hardware and model limits argue for workload-specific validation.