The paper introduces Library of Layers (LoL), a transformer that chooses the order of its layers at inference time instead of always running them in the same sequence.
One knob: A single hyperparameter, the size of the global library of transformer blocks, moves the model between parameter reuse and sparsity.
Baselines: The authors say it is competitive with dense, depth-recurrent, and mixture-of-experts models when pretrained on text.
Routing: On sudoku and maze data, different layer orders implement different solution strategies, and the routing can be intervened on to force one strategy.
Safety: Probes fit to the router outputs distinguish safe from unsafe text with high AUROC.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper introduces Library of Layers (LoL), a transformer that chooses the order of its layers at inference time instead of always running them in the same sequence.