TL;DR of the paper, generated automatically.

Paper proposes Hierarchical Residuals: wider Transformer residuals, same layer width.

• How: layers grouped in blocks, joined by a wider outer residual.
• Result: scaling laws indicate a 1.30-fold FLOPs efficiency gain.
• Test: OLMoE-1B-7B, 500B tokens: 0.0208 lower training loss, 1.14 points higher accuracy.