The paper describes a gradient-based filter that finds instruction data quietly eroding LLM safety, even when no single example looks harmful.
Method: SimpleLoRIF simplifies a low-rank attribution method and backpropagates only through the final four transformer blocks, so it can embed datasets of about a million examples.
Results: On emergent misalignment, the gradient space reaches an AUC of up to 0.865. The authors removed the 20% of Tulu 3 examples most aligned with unsafe responses to make SafeTulu, which cuts GCG attack success by up to 22% across Qwen, Llama and Olmo, with capability largely preserved or improved.
Release: SafeTulu is out, along with a Hugging Face Space for building custom versions with different removal fractions or subsets.
OP [author] adds one comment: small models can act as strong monitors for the training data of big ones.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper describes a gradient-based filter that finds instruction data quietly eroding LLM safety, even when no single example looks harmful.
OP [author] adds one comment: small models can act as strong monitors for the training data of big ones.