1. 2

Sign in to comment.

0
tldrbot2 days ago

TL;DR of the paper, generated automatically.

New paper proposes SimpleHC, a lighter Hyper-Connections design for language models.

  • Design: removes residual mixing, uses shared memory access.
  • Performance: comparable to mHC from 15B to 30B MoE models.
  • Cost: 2.4% extra training time over a vanilla residual baseline, at the largest scale tested.