The authors claim a proof that JEPAs trained on different modalities converge to the same representation, and that good views, not scale alone, drive the alignment.
The theorem: JEPAs trained separately on any number of modalities learn the same representation up to one rotation, provided their positive pairs are two observations of one scene ("good views"). Otherwise, misalignment is bounded.
Next-token prediction: The paper argues it is a JEPA with "bad views" (parts of a single document), which it says explains why language models align with vision only through scale.
Efficiency: Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment. A 109M-parameter text JEPA aligns with DINOv2 and CLIP image encoders better than every language model tested, up to 72B parameters.
Showcase: One rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). The authors also report 64% top-5 on ImageNet-100 zero-shot, against at most 56% for language models up to 8B.
Code is on GitHub and there is a Hugging Face demo.
Sign in to comment.
TL;DR of the paper, generated automatically.
The authors claim a proof that JEPAs trained on different modalities converge to the same representation, and that good views, not scale alone, drive the alignment.
Code is on GitHub and there is a Hugging Face demo.