Source-linked AI summary
Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection
Weijie Liu, Running Zhao, Wenhao Yuan, Jinfeng Xu, Zhanfeng Xu, Xiaoxi Zhang, Edith Cheuk-Han Ngai
TL;DR
Paper-code discrepancy detection is increasingly difficult as submissions exceed reviewers’ capacity, while single-agent LLMs have limited context and one-sided detection. Dude addresses these issues with dual-detection agents, granularity-aligned negotiation, and two-stage filtering. Experiments report improvements of up to 22.8% in recall, 9.0% in precision, and 18.7% in F1 score, while the method remains more token-intensive than single-agent frameworks and requires human oversight.
Problem
Paper-code inconsistencies threaten research credibility and reproducibility, while growing submissions exceed reviewers’ capacity and single-agent LLMs provide limited-context, one-sided detection.
Method
Dude uses specialized paper and code agents for dual detection, granularity-aligned negotiation, and two-stage anchor-guided and evidence-based filtering.
Results
Up to 22.8% recall, 9.0% precision, and 18.7% F1 improvements are reported for Dude on real-world paper-code discrepancy datasets.
Takeaways & Limitations
Dude substantially improves recall while preserving precision by addressing one-sided detection and granularity-driven false positives.
Takeaways & Limitations
Dude consumes more tokens than single-agent baselines, was not evaluated across all latest provider backbones, and its outputs require human oversight.
Abstract
from arXiv · showhide
LLM-empowered paper-code discrepancy detection has received growing concern since the scaling of research submissions exceeds the manual review capability. However, the limited context capacity and one-sided discrepancy detection of existing single-agent LLM paradigms lead to an inferior recall performance in detecting discrepancies. In this paper, we propose Dude, the first Dual-Detection Multi-Agent System for paper-code discrepancy detection. We discover that the granularity asymmetry of the paper-language and code-language introduces over-interpretation and over-reporting challenges in a multi-agent system design for discrepancy detection, resulting in increasing false positives. To address this, we propose a granularity-aligned negotiation and a two-stage salience-filtering mechanism in Dude, which effectively prevents agents from falsely reporting discrepancies. Experimental results in real-world paper-code discrepancy datasets showcase Dude's significant recall and precision improvement by up to 22.8%, increasing F1 score by up to 18.7% compared to baseline methods.
1 Introduction
Paper-code discrepancy detection seeks inconsistencies between manuscript claims and code, but single-agent methods face limited context and one-sided detection. Dude uses dual-detection with specialized agents, granularity-aligned negotiation, and filtering to improve discrepancy detection.
- Paper-code inconsistencies can compromise research credibility and reproducibility, while submission growth exceeds reviewers’ capacity for timely assessment.
- Single-agent LLMs struggle with long papers and codebases and primarily perform paper-oriented detection, limiting their ability to identify discrepancies comprehensively.The best-performing models reached only 46% recall on the SciCoQA dataset.
- Dude decomposes detection into paper-side and code-side subtasks assigned to specialized agents, enabling dual-detection while reducing each agent’s context burden.
- Granularity asymmetry can make code agents over-interpret high-level paper claims and falsely report discrepancies when valid implementations differ from inferred details.
- Granularity-aligned negotiation refines claims through multi-round paper–code interaction, while two-stage anchor-guided and evidence-based filtering suppresses over-reporting.
2 Related Work
Prior work applies LLMs to paper and code error detection and introduces datasets for paper-code inconsistencies, but existing discrepancy studies rely on single-agent paradigms. Multi-agent collaboration instead decomposes complex tasks among specialized agents to reduce individual context burden.
- Prior research uses LLMs to identify logical issues, calculation errors, invalid arguments, flawed proofs, experiment-design problems, and code-comment inconsistencies.
- SciCoQA and BioCon study paper-code inconsistencies, but both rely on single-agent discrepancy detection despite single agents’ limited context and one-sided detection.
- Multi-agent LLM systems decompose complex tasks into subtasks assigned to specialized agents, helping alleviate each agent’s context burden.
3 Methodology
Dude decomposes paper-code discrepancy detection into paper-oriented detection, code-oriented detection, and report generation using specialized agents. It combines dual-direction verification with negotiation and salience filtering to produce verified discrepancy claims while reducing over-interpretation and trivial reports.
- System overview: Dude uses four specialized agent types—claimer, verifier, negotiator, and orchestrator—across three stages: paper-oriented detection, code-oriented detection, and report generation.The system assigns paper and code agents to separate understanding and analysis subtasks.
- Paper-oriented detection: In paper-oriented detection, a paper claimer extracts research claims, a code verifier maps them to implementations, and the system records matching results and explanations.The code verifier uses code-search tools and labels each claim as matched, partial, mismatch, or unfound.
- Negotiation process: Granularity-aligned negotiation iteratively refines paper descriptions, revises explanations, and corrects matching results when code agents over-interpret high-level claims.The orchestrator coordinates rounds until explanations are valid or a predefined maximum number of rounds is reached.
- Code-oriented detection: In code-oriented detection, a code claimer extracts notable implementations after inspecting paper-oriented claims, while anchored-guided filtering helps exclude repetitive or trivial claims.Category-wise domain knowledge provides anchors for judging implementation significance.
- Code-oriented detection: A paper verifier applies relevance-based salience filtering before checking consistency, preventing reports based on trivial code implementations.The verifier retains implementations with high relevance to the paper and then returns matching results and explanations.
- Report generation: The orchestrator merges paper-oriented and code-oriented claims and assigns conflict, paper-omission, or code-omission labels from matching results and claim origins.Matched claims are excluded from the final discrepancy report, while unfound claims are classified according to whether they originated from paper- or code-oriented detection.
4 Experiment
Experiments evaluate Dude across datasets, model assignments, ablations, negotiation settings, and discrepancy types. Dude consistently improves detection performance while reducing false positives and maintaining relatively low multi-agent overhead.
- Overall performance: Dude consistently outperforms all baselines under GPT-5.4 and DeepSeek-V4, improving recall by up to 22.8%, precision by 9.0%, and F1 by 18.7%.
- Overall performance: Vanilla-MA gains more recall than Prompt-LM but loses substantial precision, whereas Dude preserves high precision through negotiation and filtering.
- Discrepancy-type recall: Dude exceeds Single-LM across all three discrepancy types and four LLM backbones, with average recall gains from 19.1% to 28.5%.The largest improvement occurs for paper omission, at 28.5%.
- Efficiency: Dude incurs the lowest additional token overhead among multi-agent methods by using structured JSON communication and recording previously searched snippets.
- Robustness: Dude maintains robust F1 performance under heterogeneous paper-side and code-side model assignments, with paper-side replacements affecting performance more than code-side replacements.This indicates a more crucial role for paper-side agent capability.
- Ablation studies: Removing either negotiation or saliency filtering degrades F1; filtering mainly reduces precision, while negotiation reduces both recall and precision.The ablation results support both components as contributors to discrepancy detection.
- Ablation studies: Recall, precision, and F1 improve monotonically with negotiation rounds and plateau at r = 2.Most research claims can be resolved within two discussion rounds, so larger configured limits generally terminate within the first two rounds.
- Real-world evaluation: Dude identifies the second-largest number of valid discrepancies and achieves the highest precision across 20 recent top-tier research works.Vanilla-MA finds more valid discrepancies but also produces the most invalid ones and significantly lower precision.
5 Conclusion
The conclusion presents Dude as a dual-detection framework that addresses the recall limits of single-agent systems and the precision degradation of naive multi-agent designs. It combines iterative granularity alignment with two-stage filtering to improve recall while preserving precision.
- Dude addresses one-sided single-agent detection and multi-agent precision degradation through negotiation and two-stage filtering.
- The negotiation module aligns paper and code representations through iterative agent dialogue, while filtering suppresses over-reported false positives.
- Extensive experiments confirm substantially improved recall while preserving precision in automated paper-code discrepancy detection.
Limitations
Dude improves discrepancy-detection performance but incurs higher token consumption than single-agent frameworks and lacks evaluation across all latest LLM backbones.
- Resource and evaluation limitations: Dude’s multi-agent design consumes more tokens than single-agent baselines, despite using fewer tokens than other multi-agent methods.The paper identifies token efficiency as an open direction for future work.
- Resource and evaluation limitations: High inference costs prevented evaluation of all baseline methods under the latest backbones from every LLM provider.The paper proposes extending evaluation to additional advanced LLM backbones to assess generalizability.
Ethical Considerations
Dude is intended to support researchers and reviewers by summarizing potential paper-code inconsistencies, while human oversight remains necessary because its outputs may be incomplete or incorrect.
- Dude is designed as a self-checking tool for researchers and a discrepancy-detection tool for reviewers assessing research reliability and reproducibility.
- Dude’s outputs should be treated as supportive evidence rather than fully automated decisions because they may remain incomplete or incorrect.
- Final judgments about validity, quality, or reproducibility should remain under human oversight and consider the paper, code, and experimental context.
GenAI Usage Disclosure
The system uses specialized agents and structured claim files to audit paper-code consistency through paper-oriented and code-oriented analysis, followed by verification and discrepancy classification.
- Generative AI tools were used only for typo revision to improve clarity and readability, not for method design or experimental analysis.
- Dude assigns specialized agents to paper-oriented detection, code-oriented detection, and final report generation.
- The workflow inspects paper and code claims by category, removes trivial or redundant claims, and records implementation details for verification.
- Verification compares each code claim with the paper and classifies it as matched, partial, mismatch, or unfound.
- The final discrepancy classification uses matching results to identify paper-code conflicts, code omissions, and paper omissions.
- The system performs both paper-oriented and code-oriented detection by extracting claims from each source and checking the corresponding content in the other.
B.2 Evaluation Setup
Evaluation uses SciCoQA-aligned judging and annotation procedures on real-world paper-code pairs, with manual expert review for ambiguous or conflicting cases.
- The evaluation uses the LLM-as-a-Judge protocol and prompt from SciCoQA to assess whether predicted discrepancies match discovered discrepancies.
- Annotations adopt SciCoQA’s annotation process, validation protocol, and prompt to maintain consistency and enable fair comparison.
- All compared methods are applied to 20 paper-code pairs, after which reported discrepancies are aggregated and validated using Gemini-3.1-Pro.
- Authors with relevant domain expertise manually verify discrepancies when the judge is ambiguous or different LLM outputs conflict.
- Validated discrepancies receive standardized 3–5 sentence descriptions covering the paper statement, code implementation, and location of the discrepancy.
C Token Consumption Analysis
Token analysis shows that Dude’s paper-oriented components and stage consume most of the system’s token budget, reflecting external-material retrieval and multi-round negotiation.
- The paper claimer, paper negotiator, and code verifier account for the main portions of token consumption across agents.
- Higher token consumption by paper-side agents supports the finding that paper-side capability contributes more to overall performance than code-side capability.
- Paper-oriented detection consumes over 73% of total tokens, substantially exceeding code-oriented detection and final report generation.
- Its high cost reflects the paper claimer’s retrieval of external materials and the multi-round interactions required by granularity-aligned negotiation.
D.1 Real-world Research Results
Dude is evaluated on 20 recent papers using recall, precision, F1, and token usage, with recall and F1 computed against 55 verified discrepancies. Results indicate substantially higher recall while maintaining high precision, though recall is a comparative rather than exhaustive measure.
- Recall and F1 are measured against 55 verified discrepancies because the 20 recent papers lack pre-existing discrepancy annotations.
- The evaluation covers 20 recent papers and reports recall, precision, F1, and average token consumption.These papers lack pre-existing discrepancy annotations, so recall and F1 are computed against a reference set of 55 verified discrepancies.
- Dude improves recall substantially while maintaining high precision, rather than merely reporting more discrepancies.
- The reference set may omit discrepancies missed by all methods, so reported recall supports fair comparison rather than an exact count of true discrepancies.
D.2 Evaluation Results on GPT-5.5
On SciCoQA with GPT-5.5, Dude is compared with baseline methods and remains stronger than the Single-LM paradigm. Its evaluation also examines LLM-judge agreement and documents remaining false positives and false negatives.
- Dude improves recall by 18.48% and F1 by 12.76% over Single-LM with GPT-5.5.Single-LM with GPT-5.5 achieves 64.13% overall recall, while Dude remains strong under the newer backbone.
- GPT-5.5 benefits most methods, but replacing the backbone alone does not eliminate single-agent discrepancy-detection limitations.
- Gemini-3.1-Pro agrees with human judgments on 47 of 50 cases, while agreement with Qwen-3.6 is 88%.The authors describe the LLM-as-a-Judge protocol as reasonably reliable while noting that subtle cases still benefit from human evaluation.
- Dude’s remaining false negatives mainly involve implicit discrepancies, while false positives arise from ambiguous engineering code.One documented false-positive pattern involves demonstration notebooks treated as authoritative implementations despite differing from the paper and official code.