This paper proposes two ways to improve SVD-based compression of LLMs: LEMS for rank allocation and KFAC-SVD for decomposition quality.
LEMS: A layer-wise error surrogate with ILP-based global rank allocation, giving zero-shot gains of up to 4.8 percentage points on Mistral, Qwen3 and Llama 3, which the authors say extend to 70B models.
KFAC-SVD: Token-wise statistics to address the rank deficiency in earlier Fisher-based SVD methods, with an average perplexity improvement of 15%.
Status: An extended version of an ICML 2026 paper, with code and a project page linked from the abstract.
Two fixes for the same compression problem, with code on the project page.
Sign in to comment.
TL;DR of the paper, generated automatically.
This paper proposes two ways to improve SVD-based compression of LLMs: LEMS for rank allocation and KFAC-SVD for decomposition quality.
Two fixes for the same compression problem, with code on the project page.