Source-linked AI summary
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
Wei Yang, Kuang Lu, Peilin Yang, Jimmy Lin
TL;DR
The paper tests whether neural IR gains in limited-data retrieval are genuine or inflated by weak baselines. A Robust04 meta-analysis and strong-baseline reranking experiments find no upward effectiveness trend, while only one of five neural models significantly improves a strong baseline.
Problem
The paper examines whether reported neural IR gains in limited-data retrieval reflect genuine improvements or comparisons against weak baselines.
Method
The authors combine a meta-analysis of Robust04 results with experiments applying five recent neural ranking models to strong baselines.
Results
Only one of five neural models significantly improves the strong baseline; Robust04 effectiveness shows no upward trend, and the best reported result predates recent neural approaches.
Takeaways & Limitations
Neural IR can produce additive gains over a strong baseline, but such gains are not widespread and some reported literature gains appear illusory.
Takeaways & Limitations
The study focuses on a limited-data setting using TREC judgments and does not dispute that neural approaches may be effective with large amounts of behavioral-log data.
Abstract
from arXiv · showhide
Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed skepticism that neural ranking models were actually improving ad hoc retrieval effectiveness in limited data scenarios. He provided anecdotal evidence that authors of neural IR papers demonstrate "wins" by comparing against weak baselines. This paper provides a rigorous evaluation of those claims in two ways: First, we conducted a meta-analysis of papers that have reported experimental results on the TREC Robust04 test collection. We do not find evidence of an upward trend in effectiveness over time. In fact, the best reported results are from a decade ago and no recent neural approach comes close. Second, we applied five recent neural models to rerank the strong baselines that Lin used to make his arguments. A significant improvement was observed for one of the models, demonstrating additivity in gains. While there appears to be merit to neural IR approaches, at least some of the gains reported in the literature appear illusory.
1 INTRODUCTION
The paper rigorously tests whether neural ranking models deliver genuine gains for limited-data ad hoc retrieval. Its evidence indicates persistent weak-baseline comparisons, no recent Robust04 results near the historical best, but a significant improvement from one neural model over strong baselines.
- The study evaluates Lin’s skepticism through a Robust04 meta-analysis and neural reranking experiments against strong baselines.
- No recent paper, neural or otherwise, reports effectiveness close to the best Robust04 results from a decade ago.
- Comparisons against weak baselines remain widespread, supporting Lin’s criticism of current IR evaluation practice.
- One of five neural models significantly improves strong baselines, while four do not significantly beat them.
2 META-ANALYSIS
The meta-analysis examines Robust04 results reported in the literature and finds frequent weak-baseline comparisons alongside no upward effectiveness trend. Neural papers show similar weaknesses, with their best reported AP remaining below leading non-neural results.
- Method: The visualization groups papers by publication year and marks baseline versus best AP with empty versus filled circles.
- Method: The meta-analysis examined 130 papers, using extractable average precision values from 109 papers for its reported results.
- Results: Across 109 papers, only 6 (5.5%) reported scores higher than the TREC best.
- Results: In 36 papers (33%), the baseline was below the TREC median, while 65 papers (60%) reported best results below untuned Anserini RM3.
- Results: The highest AP was 0.3686 in 2009, with no obvious upward effectiveness trend across more than a decade.
- Results: Among neural approaches, 8 of 18 papers (44%) used below-median baselines, and 12 (67%) reported best results below untuned Anserini RM3.
3 EXAMINING ADDITIVITY
The study tests whether neural ranking gains remain additive when applied to strong RM3 baselines, using five MatchZoo models and matched reranking experiments. Only DRMM significantly improves the strong baseline, while most models do not, although the combined result remains below the best known Robust04 score.
- 3 EXAMINING ADDITIVITY: The additivity question asks whether improvements over weak baselines persist when the same innovations are applied to strong baselines.If they do not, gains over weak baselines may be illusory because relevance signals can overlap.
- 3.1 Experimental Setup: The experiments replicate Lin’s settings and use DSSM, CDSSM, DRMM, KNRM, and DUET on top of the strong RM3 baseline.The models were selected because they were designed for ad hoc retrieval and document handling.
- 3.1 Experimental Setup: The reranking setup linearly interpolates neural scores with RM3 scores, allowing the model to retain or disregard the baseline depending on α.The score is defined as score = α · scoreNN + (1 − α) · scoreRM3, with α = 1 ignoring RM3 scores completely.
- 3.2 Results: 0.3152 AP with five-fold DRMM reranking significantly improves the strong baseline, beats 90% of analyzed papers, yet remains below the best known 0.3686 AP.The DRMM difference is statistically significant at p = 0.0032, including after Bonferroni correction.
- 3.2 Results: Only one of five MatchZoo neural models significantly improves Lin’s strong baseline, providing evidence that at least one neural gain is additive.The other four models do not significantly beat the baseline.
- 3.3 Discussion: The study’s scope is limited to five MatchZoo implementations, linear score interpolation, and training with TREC relevance judgments under limited-data conditions.The authors leave open whether other implementations, aggregation methods, or larger behavioral-log datasets would change the findings.
4 CONCLUSIONS
On Robust04, the paper finds that weak-baseline comparisons remain widespread and that recent results have not matched the 2009 high-water mark. Among five MatchZoo neural models, only one significantly improves a well-tuned RM3 reranking baseline, so most gains do not appear additive in this limited-data setting.
- 4 CONCLUSIONS: Only one of five MatchZoo neural ranking models significantly improves a well-tuned RM3 run, while most reported neural gains do not appear additive.The conclusion is explicitly limited to the Robust04 collection and the study’s limited-data scenario.
- 4 CONCLUSIONS: The Robust04 high-water mark was set in 2009, and no reported neural or non-neural result since then comes close.The paper also concludes that weak-baseline comparisons still pervade the literature.