A preprint introduces Scale-Free Attention (SFA), an attention mechanism for diffusion transformers that keeps fine local detail and summarizes distant regions at progressively coarser scales.
Speed: The authors report O(N log N) complexity in both forward and backward passes, and a Triton implementation that is 357.3x faster forward and 265.6x faster backward than FlashAttention at 262,144 tokens.
Image quality: After 40 epochs of PixelFlow training from scratch on ImageNet at 256x256, FID-50k improves from 13.25 with full attention to 12.51, and the abstract says the advantage persists through 80 epochs.
Fine-tuning: Per the abstract, Stable Diffusion 3.5 Medium with SFA in place of attention keeps the quality of its full-attention counterpart under matched fine-tuning budgets.
Sign in to comment.
TL;DR of the paper, generated automatically.
A preprint introduces Scale-Free Attention (SFA), an attention mechanism for diffusion transformers that keeps fine local detail and summarizes distant regions at progressively coarser scales.