PaperlayerView exact revision on arXiv

Source-linked AI summary

Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers

Atsuyuki Miyai, Mashiro Toyooka, Zaiying Zhao, Kenta Watanabe, Toshihiko Yamasaki, Kiyoharu Aizawa

arXiv:2604.01128v1cs.CLcs.AIcs.LG

TL;DR

AI-written papers lack sufficiently rigorous, unified evaluation of presentation quality and factual risks. PaperRecon reconstructs papers from compressed overviews and minimal resources, then compares them with originals across presentation and hallucination. Across 51 recent papers, Claude Code has higher presentation quality but more hallucinations, whereas Codex has fewer hallucinations and lower presentation quality.

  • Problem

    Rigorous evaluation of AI-written scientific papers remains limited, especially for jointly assessing quality and hallucination risks.

  • Method

    PaperRecon reconstructs papers from a structured overview and minimal resources, then compares them with originals using presentation and hallucination evaluations.

  • Results

    Claude Code achieves higher presentation quality than Codex, while Codex produces fewer hallucinations; both improve with model advances.

  • Takeaways & Limitations

    PaperRecon provides a framework and benchmark for quantifying presentation quality, hallucination, and their trade-offs in AI-driven scientific writing.

  • Takeaways & Limitations

    Section-wise evaluation may not fully reflect overall quality because scientific writing styles are diverse and not fully captured by current LLMs.

Abstract

from arXiv · show

This paper introduces the first systematic evaluation framework for quantifying the quality and risks of papers written by modern coding agents. While AI-driven paper writing has become a growing concern, rigorous evaluation of the quality and potential risks of AI-written papers remains limited, and a unified understanding of their reliability is still lacking. We introduce Paper Reconstruction Evaluation (PaperRecon), an evaluation framework in which an overview (overview.md) is created from an existing paper, after which an agent generates a full paper based on the overview and minimal additional resources, and the result is subsequently compared against the original paper. PaperRecon disentangles the evaluation of the AI-written papers into two orthogonal dimensions, Presentation and Hallucination, where Presentation is evaluated using a rubric and Hallucination is assessed via agentic evaluation grounded in the original paper source. For evaluation, we introduce PaperWrite-Bench, a benchmark of 51 papers from top-tier venues across diverse domains published after 2025. Our experiments reveal a clear trade-off: while both ClaudeCode and Codex improve with model advances, ClaudeCode achieves higher presentation quality at the cost of more than 10 hallucinations per paper on average, whereas Codex produces fewer hallucinations but lower presentation quality. This work takes a first step toward establishing evaluation frameworks for AI-driven paper writing and improving the understanding of its risks within the research community.

1 Introduction

AI-written papers require rigorous evaluation because existing review-based methods can reward fabricated content and prior hallucination studies remain limited. PaperRecon addresses this gap by reconstructing papers from compressed inputs, comparing them with originals across Presentation and Hallucination, and introducing a diverse benchmark for measuring agent capabilities and trade-offs.

  • Existing AI-review methods can assign higher scores to papers with more severe fabrications, while hallucination studies have largely focused on surface-level citation errors.
  • PaperRecon reconstructs an original paper from a structured overview and minimal resources, isolating the writing component of AI Scientist systems.
  • PaperRecon separates evaluation into Presentation, which measures preservation of key paper elements, and Hallucination, which uses agentic evaluation grounded in the original source.
  • PaperWrite-Bench contains 51 recent papers from diverse top-tier venues and research domains for systematic evaluation of writing agents.
  • Claude Code achieves higher presentation quality than Codex across scientific-writing sections.
  • Codex produces around 3 hallucinations per paper, compared with more than 10 per paper on average for Claude Code.
  • Writing capability improves with model advances, suggesting PaperRecon can track progress in writing ability.

2 Related Work

Prior work on AI-written content has examined automation and evaluation in several domains, but scientific-paper evaluation has not systematically assessed substantive factual validity. The paper therefore motivates protocols that jointly measure presentation quality and hallucination.

  • AI research systems increasingly automate stages of the research process and, in some cases, end-to-end research pipelines.
  • Evaluation of AI-written articles extends beyond scientific papers to Wikipedia pages and essays generated from GRE writing prompts.
  • Studies of hallucinations in AI-written papers have largely focused on citation errors or individual cases rather than systematic assessment of substantive content.
  • Accurately understanding risks requires moving beyond review-based evaluation toward protocols that directly assess both presentation quality and hallucination.

3 Paper Reconstruction Evaluation

PaperRecon reconstructs papers from controlled inputs and evaluates generated papers against originals through aligned sections, rubric-based presentation scoring, claim-level hallucination verification, and citation analysis.

  • 3.1 Problem Definition: PaperRecon supplies a research overview, figures, tables, references, and optionally code, then compares the reconstructed paper with the original to isolate writing ability.
  • 3.2 Evaluation Overview: Generated and ground-truth sections are classified into seven common categories, with unmapped sections assigned by an LLM and duplicate categories merged.
  • 3.2.2 Rubric Evaluation: Presentation evaluation uses rubrics whose elements specify expected, verifiable content for each section.
  • 3.2.2 Rubric Evaluation: Each non-conclusion section receives a 1–5 score for rubric-element coverage, including text, figure, and table items.
  • 3.2.3 Hallucination Evaluation: Hallucinations are identified by extracting claims, labeling them supported, neutral, or contradictory, and verifying contradictory claims against original-paper resources.
  • 3.2.4 Citation Evaluation: Citation evaluation compares ground-truth and generated citation-key sets using precision, recall, and F1, while also detecting invalid, missing, and extra citations.

4 PaperWrite-Bench

PaperWrite-Bench is a recent, diverse benchmark designed to evaluate paper reconstruction across research domains. Its construction standardizes inputs and section structure while preserving the information needed for faithful reconstruction and consistent comparison.

  • 4.1 Benchmark Overview: PaperWrite-Bench comprises 51 manually curated papers from recent top-tier conferences spanning computer vision, natural language processing, machine learning, and multimedia.
  • 4.1 Benchmark Overview: The benchmark uses recent sources because earlier reproduction benchmarks largely rely on papers published around 2024.
  • 4.2 Benchmark Construction Pipeline: Each research_overview.md is generated with GPT-5, manually checked for faithful reconstruction, and averages 463 words.
  • 4.2 Benchmark Construction Pipeline: The pipeline extracts tables, figures, references, and code-related resources to provide agents with structured reconstruction inputs.
  • 4.2 Benchmark Construction Pipeline: A template preserves each original paper’s section headings while leaving content empty, enabling more accurate and consistent comparisons.

5 Experiments

Experiments evaluate agents under a controlled writing pipeline using presentation, hallucination, citation, and validation analyses. Results show a consistent trade-off between presentation quality and factual reliability, with performance also varying by model generation, overview length, and conference type.

  • Main Results: Claude Code achieves higher presentation scores than Codex, although its best configuration reaches only 3.86.The result indicates substantial room for improvement in reconstructed scientific writing.
  • Main Results: More than 10 hallucinations per paper for Claude Code contrasts with around 3 for Codex, revealing a presentation–hallucination trade-off.Hallucinations are defined as claims classified as major contradictory.
  • Main Results: Claude achieves higher Citation F1 scores, whereas Codex produces substantially fewer hallucinated citations, showing a trade-off between citation coverage and factual reliability.
  • Main Results: Writing quality consistently improves from Claude Sonnet 4 to 4.6 and from GPT-5 to GPT-5.4.PaperRecon therefore tracks gains associated with model advances.
  • Human Validation: Human validation compares rubric rankings against judgments from three experienced reviewers across reconstructed papers from 12 source papers.The study uses 72 generated-paper pairs and 24 reviewed paper pairs.
  • Additional Analyses: Longer research overviews improve presentation scores and reduce hallucinations, while NLP conferences achieve the highest performance.The default overview averages 463 words and the long version 1492 words; the authors associate NLP performance with fewer complex mathematical formulations or methods.

6 Conclusion, Limitations and Future Work

The paper concludes that PaperRecon and PaperWrite-Bench support systematic assessment of AI-generated scientific papers and expose risks in their use. It identifies controlled inputs and limited coverage of diverse writing styles as boundaries for future work, while emphasizing the ethical concern of plausible but misleading or fabricated papers bypassing review.

  • Conclusion: PaperRecon is presented as the first systematic evaluation framework for AI-generated scientific papers, paired with the PaperWrite-Bench benchmark.
  • Future Work: The controlled provision of figures, tables, and references reduces external dependencies and focuses evaluation on core writing ability.The authors identify evaluation with more limited resources or external systems as future work.
  • Limitations: Section-wise evaluation may not fully reflect overall quality because scientific writing is diverse and not fully captured by current LLMs.The paper calls for more robust evaluation methods.
  • Ethics Statement: Advanced AI systems can generate plausible but misleading scientific content, creating risks alongside their measurable writing capabilities.
  • Ethics Statement: Fabricated or low-quality AI-generated papers could bypass standard review processes, motivating robust evaluation methods and safeguards.

B Detailed Prompts

EgoLife pairs a week-long multimodal egocentric dataset and long-context QA benchmark with EgoButler, a personalized system combining clip-level understanding and hierarchical retrieval. Results indicate benefits from personalization, multimodal captions, and long-context retrieval, while highlighting limitations in audio understanding, retrieval reasoning, and deployment.

  • 3. Benchmark Design: EgoLifeQA: EgoLifeQA contains 3,000 multiple-choice questions across five life-assistant capabilities, with certificate lengths indicating look-back requirements.The benchmark is built on six participants living together for seven days, and 2,003 questions require more than two hours of context.
  • 5. Proposed Method: EgoButler: EgoButler integrates EgoGPT for personalized omni-modal clip understanding with EgoRAG for hierarchical retrieval over long-context recordings.EgoGPT uses EgoIT-99K and Day-1 personalization, while EgoRAG stores clip-, hour-, and day-level summaries and retrieves top-k evidence.
  • 7. Analysis & Insights: The analysis identifies personalization, caption quality, and hierarchical retrieval as key performance drivers, while noting overfitting, ASR-centric audio limitations, and single-pass retrieval failures.
  • 8. Takeaway: EgoLife and EgoButler establish a path toward personalized life assistants that reason over week-scale multimodal recordings and long-horizon questions.
  • 6. Key Findings: Personalization improves EgoGPT performance, raising EgoSchema from 73.2 to 75.4, EgoPlan from 32.4 to 33.4, and EgoThink from 61.7 to not specified.
  • 6. Key Findings: EgoRAG improves long-context QA most for extended look-backs, increasing accuracy from 25.0 to 35.4 beyond 24 hours and from 26.8 to 38.9 over 6–24 hours.
  • 6. Key Findings: Human-verified visual-audio captions reach 45.5 average accuracy, outperforming narration-only and transcript-only inputs.Audio-only models underperform, while the best results use joint visual and audio information.
  • 7. Contributions: EgoLife provides a 266-hour, week-long, multiperson egocentric dataset with dense transcripts and visual-audio captions.

1. Introduction

PINO addresses the difficulty of generating physically coherent, semantically consistent interactions for arbitrary group sizes without retraining. It composes groups through sequential pairwise diffusion around a pivot, using noise optimization, physics-based constraints, per-pair prompts, and motion inpainting.

  • 1. Introduction: Multi-person motion must remain jointly plausible, language-consistent, and physically coherent, including avoidance of body penetration artifacts.
  • 1. Introduction: Existing multi-person methods commonly require retraining or substantial redesign when adding characters or introducing new spatio-temporal constraints.
  • 1. Introduction: PINO extends a pretrained two-person diffusion model to arbitrary group sizes through sequential pairwise generation anchored by a shared pivot character.
  • 1. Introduction: PINO optimizes each added character’s initial noise with differentiable penalties for overlap, root position, movement region, facing orientation, and relative positioning.
  • 1. Introduction: Per-pair text prompts enable fine-grained role assignment and avoid the ambiguity of using one shared prompt for an entire group.
  • 1. Introduction: Motion inpainting with boundary smoothness penalties extends interactions over longer durations without retraining.
  • 1. Introduction: PINO achieves near-zero overlap in five-person scenarios, reduces penetration volume, and attains competitive or superior FID scores against InterGen and FreeMotion.
  • 3. Preliminaries: The method builds on InterGen’s two-person diffusion model and diffusion-noise optimization, which imposes differentiable constraints at inference time without modifying model weights.

4. Method

PINO generates arbitrary-sized interactions by sequentially adding pairwise motions around a fixed pivot, while optimizing diffusion noise with physics-aware constraints and pair-specific prompts.

  • Sequential pairwise generation: PINO first generates a two-person interaction, then sequentially adds each person conditioned on person 1 as the pivot.Previously generated pivot motion is preserved through masked conditioning, while each new person uses a separate pair prompt.
  • Physics-aware optimization: For later additions, PINO penalizes the new person against all previously generated people while maintaining pivot anchoring.This extends collision avoidance beyond the current pivot pair.
  • Masked conditioning: The initial noise fixes the pivot-related features and leaves the new person’s noise free for optimization.During denoising, predicted pivot features are replaced by the fixed pivot motion.
  • Physics-aware optimization: The composite objective combines overlap, root-position, movement-region, orientation, and relative-position penalties.These terms respectively prevent collisions, control locations, constrain spatial extent, regulate facing, and guide meaningful arrangements.
  • Long-duration generation: Long-duration generation extends completed sequences through overlapping motion inpainting and a boundary smoothness penalty.The original and newly generated segments are concatenated at the overlap boundary.
  • Prompt control: Distinct pair prompts enable fine-grained role assignment that shared-prompt generation cannot provide.The paper illustrates different interactions for different pivot-person pairs in a five-person group.

5. Experiments

Experiments evaluate PINO on InterHuman against InterGen and FreeMotion using physical and semantic metrics across two-person, multi-person, five-person, long-duration, and ablation settings. Results consistently show reduced physical artifacts while preserving or improving semantic quality, with added inference cost and limitations for higher-order interactions.

  • Main results: PINO reduces overlap and penetration substantially across two-person, multi-person, five-person, and extended-sequence evaluations.For two-person generation, overlap falls from 0.119 to 0.000 and PenVol from 3112.72 cm3 to 275.65 cm3; multi-person overlap falls from 0.766 to 0.000.
  • Main results: Semantic quality remains competitive or improves while physical artifacts decrease.Two-person FID drops from 13.278 to 13.163, while five-person pair (1,5) FID is 16.911 versus 19.501 for InterGen and 25.671 for FreeMotion.
  • Ablation: The ablation study reports incremental control improvements as root, overlap, region, and orientation penalties are added.Lroot reduces positional error to 0.0, Loverlap eliminates overlap, Lregion reduces violations from 0.958 to 0.043, and Lorient reduces orientation error from 1.0 to 0.208.
  • Discussion: The framework’s physical gains are attributed to inference-time constraint optimization that adapts to differentiable spatio-temporal penalties without retraining.The paper presents this as decoupling physical plausibility from semantic quality across evaluated settings.
  • Discussion: PINO’s quality remains bounded by InterGen, and pivot-based pairwise decomposition may be unsuitable for interactions requiring simultaneous higher-order coupling.The paper gives group lifts as an example where bilateral relationships may not capture the interaction adequately.

6. Limitations

The paper identifies limitations in collision modeling, computational cost, pivot conditioning, geometric scope, and memory requirements for long sequences or large groups.

  • Collision modeling: The overlap penalty uses root distances rather than the full 3D body mesh, so slight limb penetrations may remain in highly contorted poses.The limitation concerns the geometric representation used for collision avoidance.
  • Efficiency and memory: Noise optimization is slower than vanilla diffusion because gradients must be backpropagated through the entire denoising chain.Retaining the full computation graph can become a GPU-memory bottleneck for long sequences or large groups.
  • Scalability: As group size grows, conditioning every added person on person 1 may limit motion diversity.The paper proposes alternative pivot-assignment graph topologies as future work.
  • Geometric scope: The fixed XZ ground-plane representation may not capture interaction geometries such as stacked or airborne motions.This constrains the spatial configurations represented by the current framework.
  • Model dependence: The method’s semantic fidelity is bounded by the pretrained InterGen model, especially for underrepresented interactions.Noise optimization is post-hoc constraint satisfaction rather than a learned prior.

1. Introduction

PINO addresses the difficulty of generating coherent, physically plausible interactions for arbitrary-sized groups by sequentially composing pairwise interactions around a shared pivot. It uses a frozen two-person diffusion model with inference-time noise optimization, masked conditioning, and motion-space constraints.

  • Arbitrary-sized group interaction generation remains difficult because additional participants introduce physical conflicts, semantic incoherence, and role ambiguity.
  • PINO decomposes group dynamics into sequential pairwise interactions between a shared pivot person and each newly added participant.
  • PINO optimizes initial diffusion noise while keeping a pretrained two-person model frozen, avoiding retraining for group composition and long-duration extension.
  • Masked conditioning preserves the pivot motion while each new participant is generated from a pair-specific prompt and remains compatible with previously generated people.
  • Pair-specific prompts allow one group to combine distinct relations around the same pivot, unlike shared-prompt approaches that can blur individual roles.

4. Experiments

Experiments evaluate PINO-InterGen for two-person generation, sequential multi-person composition, scaling to larger groups, and motion extension using semantic and physical metrics. Across these settings, PINO substantially improves collision-related measures while preserving or improving semantic quality.

  • PINO-InterGen reduces two-person overlap from 0.119 to 0.000 and penetration volume from 3112.72 to 275.65 while slightly improving FID from 13.278 to 13.163.
  • PINO reduces multi-person overlap from 0.766 to 0.000 and penetration volume from 25215.78 to 785.01, with nearly unchanged maximum acceleration.
  • For pair (1, 5), PINO lowers FID from 19.501 to 16.911 and overlap from 0.977 to 0.069 as the group grows to five people.
  • In motion extension, PINO improves FID from 20.390 to 18.930, overlap from 0.089 to 0.000, foot skating from 0.081 to 0.054, and maximum acceleration from 0.036 to 0.025.
  • The motion-extension benchmark reports improvements across every listed non-semantic metric, including overlap from 0.672 to 0.031 and penetration volume from 15622.16 to 1375.24.

5. Analysis

The analysis attributes PINO’s gains primarily to explicit, composable motion constraints that improve physical feasibility and controllability without fundamentally changing motion style. Qualitative results support clearer personal spacing and more distinct participant roles.

  • Adding the overlap loss removes overlap entirely, while region and orientation losses improve their corresponding control metrics.Region violations decrease to 0.043, and orientation error decreases from 1.0 to 0.208.
  • No single penalty satisfies all control requirements; combining penalties at inference time preserves the pretrained motion prior while enabling targeted constraints.
  • PINO maintains clearer personal space and less entangled body configurations than direct multi-person composition, while prompt switching supports distinct roles around the pivot.
  • Across benchmarks, PINO’s largest gains occur in overlap and penetration volume, with wider margins as composition becomes harder.
  • PINO improves physical plausibility and user control while supporting arbitrary-sized and long-duration generation without retraining.
Loading 2604.01128v1…