SpecSlot adds fine image detail to pixel-space Transformers without lengthening the global attention sequence.
Core idea: Each spatial anchor is split into several slots, each holding a separate band of DCT frequency coefficients. Only the lowest-frequency slot per anchor joins global attention, and the other slots interact locally.
ImageNet-256: Final FID is comparable to a FLOPs-matched JiT control, and the model converges faster.
ImageNet-512: FID improves by 2.84 over the control with 3.9% fewer FLOPs. It lands within 0.69 FID of JiT-B/16 at 38.2% lower cost.
Text-to-image: On DPG-Bench, it beats JiT-B/8 by 1.06 at 40.6% lower forward cost.
Sign in to comment.
TL;DR of the paper, generated automatically.
SpecSlot adds fine image detail to pixel-space Transformers without lengthening the global attention sequence.
Code is on GitHub.