A paper introduces DecisionRank, a decision model trained with item-space on-policy distillation to pick up ranking knowledge from generative LLM teachers.
How it works: The student sorts its document scores into a ranked sequence, and a generative teacher gives feedback on that ranking to supervise the scores. The authors combine this with offline ranking supervision.
Results: The paper reports state-of-the-art performance on TREC DL, BEIR and BRIGHT. At 1.7B parameters, it surpasses its 27B generative teacher on TREC DL19 to 23.
Speed: At 8B, it beats its generative listwise counterpart while running 24.5x faster on TREC DL and 16.7x faster on BRIGHT.
Sign in to comment.
TL;DR of the paper, generated automatically.
A paper introduces DecisionRank, a decision model trained with item-space on-policy distillation to pick up ranking knowledge from generative LLM teachers.