Dual-Sample Training Helps LLM Rerankers Avoid Over-Reliance on Click Data

Large language model rerankers that use behavioral signals like click-through rates and query-success scores can develop shortcut learning, where they follow clicks rather than understanding semantic relevance. This works well for popular queries but causes performance to collapse on cold-start items and rare long-tail queries that lack reliable historical data. A mitigation technique called paired dual-sample training addresses this by presenting each training example twice — once with behavioral features and once without — forcing the model to learn genuine semantic reasoning alongside click-based shortcuts. A weighting parameter controls how much the model leans on behavioral data versus pure semantics, and can be tuned based on query frequency. Supporting measures such as confidence filters, feature bucketing, and log randomization further reduce the risk of models exploiting noisy or misleading behavioral signals.