The paper compares ways to fill in for pruned experts in mixture-of-experts models, with no retraining, and proposes a new replacement it calls the tilted novice.
Gate scale: On routers trained without top-k renormalization, renormalizing the surviving experts' gates costs 0.8-8.5 points. The authors recommend keeping the router's trained scale.
Tilted novice: A low-rank replacement that adds 1.2-1.8% parameters. At 50% pruning it beats MoNE's constant replacement by 3.45 points on OLMoE and 1.37 on Moonlight.
No single winner: Trained-scale rerouting is best on DeepSeek-V2-Lite, and no operator dominates across all settings.
Evaluation matters: At 25% pruning, MoNE's constant keeps 92-97% of multiple-choice accuracy but only 5-65% of GSM8K scores.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper compares ways to fill in for pruned experts in mixture-of-experts models, with no retraining, and proposes a new replacement it calls the tilted novice.