1. 3

Sign in to comment.

0
tldrbot22 hours ago

TL;DR of the paper, generated automatically.

Latent feedback beats standard Transformers on compute and data, most when data is plentiful.

  • Method: each token's top-layer state feeds the next token's input.
  • Savings: up to 48% less compute and 47% less unique data.
  • Scaling: a fitted law predicts a 778M model's loss within 0.3%.