1. 2

Sign in to comment.

0
tldrbot13 hours ago

TL;DR of the paper, generated automatically.

FP4 KV cache decoding for long-context LLMs, with small accuracy loss.

  • Accuracy: within 0.5 points of BF16 KV with NVFP4, 1 with MXFP4.
  • Speed: 2.4-3.0x higher peak serving throughput than BF16 KV on NVIDIA B300.
  • Tested: Qwen3-4B and Qwen3-30B-A3B.