PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Yiwen Song, Yale Song, Tomas Pfister, Jinsung Yoon

arXiv:2604.05018v1cs.AIcs.LGcs.MA

TL;DR

Automated research writers struggle to turn unconstrained materials into rigorous manuscripts and often produce shallow literature reviews. PaperOrchestra addresses this gap with a standalone multi-agent pipeline and PaperWritingBench, outperforming autonomous baselines in side-by-side evaluations.

  • Problem

    Existing autonomous writers are coupled to experimental pipelines, while survey-oriented systems cannot transform raw experimental data into full manuscripts and often produce shallow literature reviews.

  • Method

    PaperOrchestra uses specialized agents to transform unconstrained pre-writing materials into LaTeX manuscripts with literature synthesis, generated visuals, and iterative refinement.

  • Results

    50%–68% absolute win rate margins in literature review synthesis and 14%–38% in overall manuscript quality were achieved over autonomous baselines.

  • Takeaways & Limitations

    PaperOrchestra provides a standalone workflow for producing submission-ready manuscripts from preliminary research materials while synthesizing broader literature and visuals.

  • Takeaways & Limitations

    Human researchers must retain accountability for factual accuracy, originality, and validity of claims in generated manuscripts.

Abstract

from arXiv · show

Synthesizing unstructured research materials into manuscripts is an essential yet under-explored challenge in AI-driven scientific discovery. Existing autonomous writers are rigidly coupled to specific experimental pipelines, and produce superficial literature reviews. We introduce PaperOrchestra, a multi-agent framework for automated AI research paper writing. It flexibly transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts, including comprehensive literature synthesis and generated visuals, such as plots and conceptual diagrams. To evaluate performance, we present PaperWritingBench, the first standardized benchmark of reverse-engineered raw materials from 200 top-tier AI conference papers, alongside a comprehensive suite of automated evaluators. In side-by-side human evaluations, PaperOrchestra significantly outperforms autonomous baselines, achieving an absolute win rate margin of 50%-68% in literature review quality, and 14%-38% in overall manuscript quality.

1. Introduction

PaperOrchestra addresses the difficulty of turning unstructured research materials into rigorous manuscripts, overcoming limitations in existing survey and autonomous research-writing systems. It introduces a standalone multi-agent writer, PaperWritingBench, and strong side-by-side evaluation results.

  • Research gap: Existing autonomous frameworks struggle to translate raw ideas and experimental logs into submission-ready manuscripts.Survey systems focus on literature reviews, while full-lifecycle agents remain coupled to their experimental pipelines.
  • Research gap: Existing systems produce shallow, under-cited literature reviews and cannot generate conceptual scientific diagrams from unconstrained materials.Their visual outputs are generally restricted to code-generated data plots.
  • Contributions: PaperOrchestra is a standalone multi-agent framework that authors LaTeX manuscripts from unconstrained pre-writing materials.Specialized agents synthesize literature, generate plots and conceptual diagrams, and iteratively refine technical clarity.
  • Contributions: PaperWritingBench provides a standardized writing benchmark built from reverse-engineered ideas and experimental logs from 200 top-tier AI conference papers.The benchmark isolates the writing task from an experimental pipeline.
  • Results: 50%–68% absolute win rate margins in literature review synthesis and 14%–38% in overall manuscript quality separate PaperOrchestra from autonomous baselines.These results come from side-by-side human evaluations.

2. Related Work

Related work spans end-to-end research agents, manuscript-writing assistants, and automated literature-review systems. These approaches remain limited by structured-input requirements, pipeline coupling, or insufficient contextual targeting of related work.

  • Autonomous research agents: End-to-end research systems automate experimentation and drafting, using code templates, agentic tree search, iterative refinement, or structured knowledge.Examples include AI Scientist-v1, AI Scientist-v2, CycleResearcher, and OmniScientist.
  • Manuscript-writing systems: Earlier and newer writing assistants generally lack end-to-end generation from raw experimental logs.PaperRobot lacked complex data-driven narratives, while Prism typically relies on structured inputs or human guidance.
  • Literature-review systems: Automated literature-review systems use retrieval, outline refinement, or multi-agent workflows to generate long-form surveys.AutoSurvey2, SurveyGen-I, and LiRA represent these approaches.
  • Limitations: Survey-oriented systems often lack the contextual awareness needed to contrast prior work and motivate a new method’s specific research gap.Their focus is not targeted related-work sections within complete manuscripts.
  • Comparison: Existing manuscript generators require structured BibTeX inputs, depend on artifacts from internal pipelines, or omit conceptual scientific diagrams.PaperOrchestra is contrasted with CycleResearcher, AI-Researcher, and AI Scientist-v2 on these dimensions.

3. Task and Dataset

PaperOrchestra formulates paper generation as mapping unconstrained pre-writing materials and conference constraints to a LaTeX source file and rendered PDF. PaperWritingBench operationalizes this task with anonymized materials reverse-engineered from 200 CVPR and ICLR papers.

  • Task formulation: The task maps an idea summary, experimental log, LaTeX template, conference guidelines, and optional figures to a submission package.The package contains source LaTeX and a rendered PDF.
  • Task formulation: When no figures are provided, the pipeline autonomously synthesizes relevant visual assets.Figures are an optional input component in the task definition.
  • Task formulation: The finalized submission package consists of a source LaTeX file and a rendered PDF generated by the framework W.This output is denoted P = (Ptex, Ppdf).
  • Dataset construction: PaperWritingBench contains 200 accepted papers, split evenly between CVPR 2025 and ICLR 2025, to test writing across distinct conference formats.The benchmark uses official venue templates and guidelines.
  • Dataset construction: The benchmark reverse-engineers anonymized, self-contained idea summaries and experimental logs from extracted paper content.Sparse and dense idea variants simulate different drafting effort, while logs preserve experimental setups, baselines, datasets, metrics, and tabular data.

4. PaperOrchestra

PaperOrchestra uses specialized agents to plan, generate visuals and literature reviews, write sections, and iteratively refine a complete LaTeX manuscript. Its evaluations report consistent advantages over autonomous baselines in literature review and overall paper quality.

  • Pipeline overview: The pipeline transforms idea summaries, experimental logs, templates, and guidelines into submission-ready manuscripts through five steps.Outline generation precedes parallel plot generation and literature review, followed by section writing and content refinement.
  • Step 1: Outline Generation: The Outline Agent creates visualization, literature-search, and section-writing plans, including citation requirements.The literature strategy covers macro-level context and micro-level methodology clusters.
  • Step 2: Plot Generation: The Plotting Agent generates conceptual diagrams and statistical plots using closed-loop visual critique and revision.PaperBanana is the default module and also synthesizes context-aware captions.
  • Step 3: Literature Review: The Literature Review Agent combines web search with Semantic Scholar verification, temporal filtering, deduplication, citation registration, and BibTeX generation.Verified citations support drafting of the Introduction and Related Work sections.
  • Step 4: Section Writing: The Section Writing Agent integrates experimental values, tables, citations, and generated figures into the remaining manuscript sections.It authors the abstract, methodology, experiments, and conclusion from the partially filled LaTeX file.
  • Step 5: Iterative Content Refinement: The Content Refinement Agent accepts revisions when review scores improve or satisfy tie-breaker conditions, producing the final LaTeX document and compiled PDF.It reverts changes after a score decrease, negative tie-breaker, or iteration limit.
  • Evaluation: PaperOrchestra significantly outperforms Single Agent and AI Scientist-v2 in literature review and overall paper quality across both judge models.Human (GT) serves as the upper-bound reference.

5. Experiments

PaperOrchestra is evaluated against single-agent and autonomous research baselines using citation, literature-review, holistic-quality, human, and ablation analyses. Across these evaluations, it generally improves manuscript quality, citation coverage, literature synthesis, visual generation, and refinement outcomes.

  • Evaluation Setup: PaperOrchestra is compared with a Single Agent baseline and AI Scientist-v2, with figures standardized when baselines lack diagram generation.The comparison uses ground-truth visual inputs where necessary for fairness.
  • Evaluation Metrics: Citation F1 evaluates precision, recall, and F1 separately for must-cite P0 references, good-to-cite P1 references, and the combined reference set.P0 covers essential contextual citations, while P1 covers valuable but non-essential background.
  • Citation Coverage: PaperOrchestra achieves 45.73–47.98 citations, improves P0 Recall by 2.13%–6.07%, and improves P1 Recall by 12.59%–13.75% over the strongest baselines.The baselines average only 9–14 citations, with near-zero P1 Recall despite competitive Overall F1.
  • Literature Review Quality: PaperOrchestra gains 32.87%–33.25% in Overall Score with Gemini-3.1-Pro and 9.66%–9.85% with GPT5 over the strongest AI baseline.It remains highly comparable to human baselines and leads in Citation Practices and Critical Analysis.
  • Human Evaluation: Human evaluations show 50%–68% absolute win margins in Literature Review and 14%–38% in Overall Quality against AI baselines.PaperOrchestra also achieves a 43% tie/win rate against human ground-truth papers in literature synthesis.
  • Sparse vs. Dense Inputs: Dense inputs produce 43%–56% Overall Paper Quality win rates versus 18%–24% for Sparse inputs, while Literature Review remains near-parity at 32%–40% versus 28%–39%.The results indicate stronger methodology generation with denser inputs and robust literature-review performance with sparse inputs.
  • Visual Generation: Autonomously generated visuals achieve ties or wins in 51%–66% of comparisons against human-authored figures despite lacking the data represented by ground-truth visuals.The Plotting Agent synthesizes coherent visuals from scratch.
  • Content Refinement: Iterative refinement achieves 79%–81% win rates with 0% losses, alongside simulated acceptance gains of +19% for CVPR and +22% for ICLR.Overall scores also increase by +0.88 and +1.61 after targeted clarity and presentation corrections.

6. Conclusion

PaperOrchestra and PaperWritingBench transform unstructured preliminary AI research materials into submission-ready manuscripts and support deep, context-aware literature reviews. The framework generates high-quality papers with competitive runtime, while future work targets richer artifacts and interactive writing environments.

  • PaperOrchestra and PaperWritingBench transform unstructured, preliminary AI research materials into submission-ready manuscripts.
  • The multi-agent framework generates high-quality research papers while synthesizing deep, context-aware literature reviews.The paper reports competitive runtime for this capability.
  • Future work should expand the framework to ingest richer research artifacts and support interactive, dynamic writing environments.The stated goal is seamless human-AI scientific collaboration.

Ethics Statement

PaperOrchestra is positioned as an assistive drafting tool rather than an independent author. Human researchers retain accountability for factual accuracy, originality, validity, and verification of generated claims.

  • PaperOrchestra is an advanced assistive tool for accelerating AI research-paper drafting, not an independent entity capable of claiming authorship.
  • Human researchers must retain full accountability for the factual accuracy, originality, and validity of generated manuscript claims.
  • Users must verify outputs to prevent the propagation of LLM-derived biases or misinformation.The framework includes programmatic safeguards such as API-grounded citation validation, but verification remains the users’ responsibility.

A. Limitations and Future Work

The paper identifies limitations in visual-generation control, human steering, and benchmark contamination risk. Future work targets factual visual verification, human-in-the-loop refinement, and benchmarks that better separate generation from memorization.

  • External visual-generation frameworks limit PaperOrchestra’s direct control over figure hallucinations.The authors suggest targeted Vision-Language Models and dedicated human evaluations to verify visual factuality and layout placement.
  • A human-in-the-loop system could let researchers steer drafts iteratively through natural-language critiques.The current refinement agent instead uses structured, LLM-generated feedback.
  • The framework is intended as an advanced assistive tool rather than a fully independent writing entity.
  • PaperWritingBench carries an inherent risk of foundation-model pre-training data contamination.The paper mitigates this through de-contextualized, anonymized inputs and notes that future benchmarks could use unpublished or autonomously generated materials.

B. Computational Cost

PaperOrchestra uses more LLM calls than AI Scientist-v2 but maintains competitive single-paper latency through parallel candidate discovery and sequential citation verification. Its computational investment supports deeper citations and data-grounded visuals with minimal added latency.

  • 39.6 minutes is PaperOrchestra’s mean processing time, compared with 35.1 minutes for AI Scientist-v2.PaperOrchestra uses approximately 60–70 LLM calls, versus approximately 40–45 for AI Scientist-v2.
  • PaperOrchestra decouples paper discovery and verification to optimize retrieval throughput.Candidate discovery uses 10 concurrent workers, followed by sequential Semantic Scholar verification at 1 query per second.
  • The Outline Agent and Section Writing Agent each use one call, while the Content Refinement Agent uses approximately 5–7 calls.The Section Writing Agent drafts and compiles the complete LaTeX manuscript in one comprehensive multimodal call.
  • The pipeline allocates approximately 20–30 calls each to the Hybrid Literature Agent and Plotting Agent.These agents handle citation discovery and verification, plus visual planning, generation, critique, redraw, and captioning.
  • PaperOrchestra delivers substantial improvements in manuscript quality and citation depth while adding only minimal latency compared with AI Scientist-v2.

C.1. Data Distribution

PaperWritingBench controls input granularity and preserves empirical information while spanning papers with different visual, analytical, and material densities. Its extracted materials include sparse and dense ideas plus experimental logs, with structured context injection used to improve fidelity.

  • Dataset distribution: ICLR papers average roughly twice as many figures and tables as CVPR papers, with 9.19 versus 5.20 figures and 8.13 versus 4.20 tables.ICLR experimental logs are also longer on average, at 2,387 versus 1,530 words.
  • Structured context injection: Structured context injection parses PDFs into markdown and uses visual context to flatten document references and translate visual information into standalone factual observations.The procedure is intended to preserve tabular and mathematical fidelity while requiring downstream agents to reconstruct the empirical narrative.
  • Controlled idea density: Dense materials preserve mathematical formulations, loss functions, and LaTeX variables, whereas sparse materials abstract formal architectures into high-level conceptual narratives.This controlled density separates conceptual descriptions from notation-rich inputs.
  • Dataset construction: PaperWritingBench extracts sparse ideas, dense ideas, and experimental logs from randomly selected CVPR 2025 and ICLR 2025 papers.The supplied sample is drawn from the CVPR 2025 split.

5. Training Objective

TSAM trains end-to-end with a weighted combination of binary cross-entropy and intersection-over-union losses, using equal weighting for the two components. The stated contributions emphasize temporal adaptation and automated multimodal prompting without extensive image-backbone retraining.

  • Loss formulation: Ltotal = LBCE + λ·LIoU combines Binary Cross-Entropy and Intersection over Union losses to compare predicted masks with ground truth.The loss is described as a weighted sum used for end-to-end training.
  • Loss weighting: λ = 1.0 balances the contributions of the two loss components.The weighting is fixed rather than tuned across the reported settings.
  • Stated contributions: TSAM introduces a lightweight Temporal Modeling Branch and converts audio-visual-text correlations into the sparse and dense prompts required by SAM.The framework is presented as adapting SAM without extensive retraining of its image backbone.

1. Experimental Setup

The evaluation uses the Ref-AVS dataset and reports TSAM results, ablations, qualitative analyses, and implementation settings against existing methods. TSAM leads on Seen and Unseen sets but trails EEMC on the Null set.

  • Dataset: The Ref-AVS dataset contains 20,000 text expressions and pixel-level annotations across 4,000 10-second videos.Its object categories include audible objects and static, inaudible objects.
  • Evaluation settings: The evaluation compares methods using Ref-AVS performance tables and ablations, while model access, temperatures, research cutoffs, and excluded baselines are specified for reproducibility.The literature and citation pipeline uses Gemini-3-Flash discovery followed by Semantic Scholar verification.
  • Main results: TSAM improves over EEMC by 9.23% in Jaccard Index on Seen and 5.04% on Unseen test sets.The reported evaluation states that TSAM achieves the highest performance on both sets.
  • Main results: TSAM trails EEMC on the Null test set, with S value 0.017 versus 0.007.The authors attribute this to SAM’s tendency to produce a mask even when no target object is present.
  • Ablations: Removing the temporal branch reduces Seen Jaccard from 43.43% to 33.05%, while removing both prompting modules also decreases segmentation performance.The ablations identify temporal modeling and complementary sparse and dense prompting as important components.
  • Ablations: Cached memory and the adapter module support performance, with adapter removal causing a particularly large drop on the Unseen set.Including IoU loss improves Seen and Unseen results but slightly worsens the Null score.
  • Hyperparameters: The best reported settings use k = 5 audio queries and M = 4 temporal blocks.Lower or higher values were reported to perform worse on generalization or introduce excessive complexity.
  • Qualitative analysis: Qualitative results show high-quality Seen masks, segmentation of novel objects on Unseen scenes, and preservation of SAM’s pretrained knowledge.The analysis includes examples involving textual guidance and previously unseen object categories.

D.5. Human Evaluation

Human and visual evaluations compare manuscripts produced by multiple systems from shared raw materials. PaperOrchestra is reported to outperform AI baselines across evaluation dimensions, while sparse-versus-dense inputs affect methodological detail.

  • Evaluation dimensions: Literature Review Quality evaluates Introduction and Related Work for framing, coverage, synthesis, positioning, and readability.The review criteria focus on the quality of the literature-facing sections rather than the full manuscript.
  • Evaluation dimensions: Overall Paper Quality evaluates the full manuscript on scientific depth, technical execution, logical flow, writing clarity, evidence presentation, and academic style.The rubric instructs annotators to judge the complete papers and ignore templates, metadata, length, and verbosity as sole decision criteria.
  • Human evaluation: Eleven AI researchers completed 180 randomized paired evaluations across 40 sampled papers, covering 120 unique paper pairs.Annotators answered 12 diagnostic questions before making a final holistic judgment.
  • Human evaluation: PaperOrchestra systematically outperforms AI baselines across every reported evaluation metric, although a quality gap remains relative to human-written ground truth.The comparison is presented in the human side-by-side evaluation results.
  • Manuscript comparisons: The visual comparison includes Single Agent, AI Scientist-v2, PaperOrchestra PlotOff, and PaperOrchestra PlotOn manuscripts generated from sparse raw materials.The samples cover CVPR and ICLR papers.
  • Input granularity: Sparse inputs yield conceptual, high-level methodology text, whereas dense inputs include formal notation such as BQ-MDPs and underlying operations.The contrast is shown using an ICLR sample generated by PaperOrchestra.
  • Input granularity: The dense methodology describes shared codebooks, task-specific projections, and mixed attention that integrates edge information into node interactions.These mechanisms are presented as part of the notation-rich methodology comparison.

Multi-Task Imitation Learning Strategy

The framework formulates multi-task combinatorial optimization as sequential decision-making over heterogeneous problem environments, using task-specific adapters and a shared attention backbone. It trains with imitation learning on expert trajectories and decodes solutions autoregressively with efficiency-oriented caching.

  • Problem formulation: GOAL models multi-task optimization as a constructive sequential decision process over a disjoint union of BQ-MDPs.States represent problem instances, while actions represent construction steps such as sequential node selections.
  • Input representation: Task-specific adapters and a shared linear projection map varying input dimensions into a common backbone embedding space.The input adapter compresses each task’s raw inputs to a low-dimensional space before the shared projection produces the backbone representation.
  • Shared reasoning backbone: Multi-Head Mixed-Attention integrates edge features directly into attention scores alongside node features.Edge-derived vectors are added to node-derived query and key vectors before the scalar product, while absent edge features contribute zero.
  • Shared reasoning backbone: Dense multidimensional edge attributes are projected into query and key spaces rather than represented as one-dimensional distance-based scalar biases.The mechanism is presented as distinct from relative positional encodings and other edge-augmented models in how it uses asymmetric costs or capacity constraints.
  • Heterogeneous graphs: A Multi-Type Architecture dynamically composes self-attention and cross-attention operations for heterogeneous graph structures.This avoids padding disparate node types into a single homogeneous tensor and instantiates operations according to valid type pairings.
  • Training and decoding: GOAL uses imitation learning from oracle-derived constructive trajectories and autoregressive decoding with cached graph embeddings.The backbone processes static graph features once, while a lightweight output adapter handles changing states during sequential construction.
Loading 2604.05018v1…