BYOD converts existing autoregressive language models into diffusion models by training only small LoRA adapters, and the authors have released a Colab notebook for it.
Models: Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B, all with frozen base weights. The Mistral conversion is described as the first reported for that family.
Cost: Adapters are 4.1-5.8% of final parameters. Each of the four primary conversions took 3.1-4.0 hours on one Google Colab GPU.
Quality: BYOD-Gemma averages 50.6% on the nine-task evaluation, against 51.9% for LLaDA 8B Instruct. The authors name low-step generation quality as the main limitation, with perplexity at 8.3-11.1 versus 4.1 for LLaDA.
Access: A notebook, interactive demos for all four models, and standalone code. Quick mode uses smaller models, while paper mode needs a paid GPU.
Sign in to comment.
TL;DR of the paper, generated automatically.
BYOD converts existing autoregressive language models into diffusion models by training only small LoRA adapters, and the authors have released a Colab notebook for it.