Source-linked AI summary
ATIBA: Grounded Integrity and Quality Checking for Research Papers
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, Eray Tüzün
TL;DR
ATIBA targets the manual, repetitive, venue-dependent checking of manuscript references, submission rules, and empirical reporting standards. It combines five evidence-grounded checks that let LLMs judge but not invent supporting evidence, with initial evaluation indicating high perceived usefulness across the workflows.
Problem
Manuscript integrity, venue compliance, and empirical-standards checking are manual, repetitive, inconsistent, and dependent on varying venue requirements.
Method
ATIBA combines five manuscript checks grounded in bibliographic sources, venue-page quotes, and manuscript-verbatim evidence, with user review gates for extracted criteria and checklist results.
Results
The evaluation reported high perceived usefulness across all five checks in real-world writing and internal peer-review workflows.
Takeaways & Limitations
ATIBA establishes an evidence-grounded design for integrated manuscript checking while limiting unsupported or fabricated verdicts.
Abstract
from arXiv · showhide
Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.
1 Introduction
ATIBA addresses fragmented, venue-dependent manuscript checks with five workflows grounded in external or verbatim evidence. Its central safeguard is to prevent unsupported LLM verdicts while providing initial evidence of perceived usefulness.
- Manuscript checking is effortful and inconsistent because requirements span integrity, formatting, scope, and quality while varying across venues.
- Every check grounds LLM judgments in bibliographic records, venue-page quotes, or manuscript-verbatim evidence rather than model output alone.
- Unsupported evidence causes feedback or verdicts to be discarded, or the check to return no verdict or an explicitly low-confidence result.
- ATIBA combines reference verification, venue compliance, empirical-standards checking, multi-mode AI review, and verified citation suggestion in one manuscript-analysis system.
2 Background and Related Work
Prior tools address submission management, empirical standards, reference verification, citation recommendation, or AI review separately. ATIBA is positioned as combining these content-level checks while requiring grounded evidence for its judgments.
- Conference-management platforms route submissions and assign reviewers but leave content-level integrity and compliance checking to people.
- ATIBA automates the ACM SIGSOFT Empirical Standards checklist while grounding every verdict in a manuscript-verbatim evidence quote.
- Reference existence verification is distinct from judging citation appropriateness, and ATIBA assigns these tasks to bibliographic lookup and LLM context assessment respectively.
- Prior citation-recommendation work uses contextual ranking or retrieval, while direct LLM prompting is described as vulnerable to severe hallucination.
- Existing review systems can overlap with human comments or use retrieved evidence, but related systems do not share ATIBA’s same evidence requirement across its workflows.
3 Proposed System: ATIBA
ATIBA implements five manuscript-analysis pipelines spanning reference integrity, venue compliance, empirical standards, AI review, and citation recommendation. Across them, the system separates lookup from judgment, requires traceable evidence, and keeps users involved in reviewable decisions.
- ATIBA’s five pipelines cover reference-integrity verification, venue compliance, empirical standards, three-mode AI review, and verified citation suggestion.
- 3.1 Reference-Integrity Check: The reference-integrity workflow assigns existence, author-match, retraction, and parsing outcomes through an external bibliographic-verification tool, then separately assesses citation context.
- 3.2 Venue/Track Compliance Check: Venue compliance derives criteria from a track-specific call-for-papers page, requires verbatim source quotes, and returns per-criterion manuscript checks after a user evidence gate.
- 3.3 Empirical-Standards Checklist: The empirical-standards workflow classifies methodologies, assigns checklist statuses, discards unlocatable manuscript quotes, and presents editable draft results for user confirmation.
- 3.4 AI Review: The AI-review layer provides venue-specific, formal, and page-anchored modes through one venue-aware prompting layer backed by Azure OpenAI.
- 3.5 Citation Recommendation: Citation suggestion verifies LLM-proposed candidates against CrossRef, Semantic Scholar, and OpenAlex before displaying them, discarding candidates no source confirms.
4 Evaluation
ATIBA was evaluated in a moderated user study with 13 non-author participants who used its grounded checks and AI review on their own manuscripts. Agreement was high across the six survey items, with adoption intent and several workflow-specific assessments reaching 92%.
- 4.1.1 Procedure and Instrument: The moderated study recruited 13 non-author participants with varied research experience, who used ATIBA’s four checks, citation suggestions, and AI review on their own manuscripts.Participants included five undergraduates, five master’s students, two PhD students, and one research staff member.
- 4.1 User Study: 10 of 13 participants spent at least three hours reviewing a paper, whereas 11 of 13 spent at most two hours checking its references.Six participants spent less than one hour on references, and only two spent longer on references than on the rest of the review.
- 4.1.2 Results: 12 of 13 participants (92%) agreed they would use ATIBA in future paper-writing or internal peer-review workflows (Q6, M=4.5).Eight participants strongly agreed.
- 4.1.2 Results: The venue-criteria check and AI-generated review each received 92% agreement (Q3 and Q4, both M=4.4), while the empirical checklist reached 85% (Q5, M=4.2).
- 4.1.2 Results: 77% agreed that ATIBA surfaced quality or integrity issues they might otherwise overlook (Q1, M=4.0), while reference-integrity verification received 69% agreement (Q2, M=4.0, SD=1.2).All six items received majority agreement, with the greatest response variation on reference integrity.
5 Threats to Validity
The evaluation provides evidence about perceived usefulness but is limited by its moderated, single-session design, small non-author sample, and dependence on a particular LLM configuration. These constraints limit conclusions about objective accuracy and generalizability.
- Internal validity: The study measured perceived usefulness rather than objective accuracy in a moderated, single-session evaluation without controlled task comparisons.Moderator presence and the absence of controlled comparisons may have influenced participant responses.
- External validity: The 13 participants came from the authors’ research network, most had up to two years of research experience, and the findings may not generalize broadly.Potentially underrepresented groups include experienced researchers, reviewers, program chairs, and authors from other disciplines or institutions.
- External validity: Observed results may vary across LLM providers, model versions, and future updates because several ATIBA features depend on a particular deployment, prompts, and configuration.The paper calls for broader samples, multiple manuscript domains and LLM configurations, and controlled task-based evaluations.
6 Limitations and Future Work
ATIBA’s evaluation establishes perceived usefulness, while several coverage and judgment boundaries remain; future work targets objective accuracy, broader venue-source collection, and stricter deployment settings.
- Evaluation coverage: Objective precision and recall remain unmeasured, motivating controlled benchmarks with labeled manuscripts and seeded reference defects.Such benchmarks would separate tool accuracy from the particular manuscript used in the moderated study.
- Venue-source coverage: Venue compliance is limited to rules present in the supplied page, excluding requirements contained only in linked PDFs, images, or separate documents.Pasting page text provides a fallback when pages cannot be fetched, but cannot recover rules absent from that text.
- LLM judgment: Grounding improves auditability, but context relevance, rubric-based review, and empirical-standard fulfillment still require model judgment.Displayed evidence and editable criteria keep these judgments open to user verification.
- Reference verification: Database coverage, title-extraction errors, and timeouts can produce not_found results for legitimate publications, so that status prompts inspection rather than proving fabrication.Future evaluation will quantify these cases and their effect on false-positive rates.
- Citation suggestion: Citation verification confirms existence and metadata matching but does not establish that a suggested work best supports a particular claim.Claim–citation relevance remains an additional direction for the suggestion pipeline.
- Manuscript confidentiality: Azure OpenAI processing retains prompts and completions for up to 30 days for abuse monitoring, limiting suitability for stricter data-residency settings.A locally hosted option would extend ATIBA to those settings.
7 Conclusion
ATIBA consolidates fragmented manuscript checking into five complementary workflows, grounding model judgments in independent evidence across the system.
- 7 Conclusion: ATIBA brings together reference-integrity verification, venue and track compliance checking, empirical-standards checking, multi-mode AI review, and verified citation suggestion.These workflows address repetitive checking tasks in the publication process.
- 7 Conclusion: The system is designed to ground model judgments in independent evidence rather than relying solely on generated output.
- 7 Conclusion: ATIBA targets reference verification, venue-specific requirements, and reporting standards within a single manuscript-checking workflow.