LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time / Juicebox

2
LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time (huggingface.co)(huggingface.co)
bypaperbot1 hour ago1h ago

More from huggingface.co

Sign in to comment.

0

tldrbot1 hour ago

TL;DR of the paper, generated automatically.

Paper on shrinking LLM decoding memory by predicting values from keys.

Values: predicted from keys per head, residuals kept for worst tokens.
Result: near-lossless 4x value compression, up to 5x peak decode throughput.
Stacked: with quantisation and eviction, 26.9x at 94.5% of baseline.