Source-linked AI summary

PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Yiwen Song, Yale Song, Tomas Pfister, Jinsung Yoon

arXiv:2604.05018v1cs.AIcs.LGcs.MA

TL;DR

Turning unstructured research materials into rigorous manuscripts remains difficult because existing autonomous writers are pipeline-bound and produce shallow literature reviews. PaperOrchestra uses a standalone multi-agent framework for manuscript generation and outperforms autonomous baselines by 50%–68% in literature review quality and 14%–38% in overall manuscript quality.

  • Problem

    Existing autonomous research writers are tied to experimental pipelines and struggle to turn unstructured materials into rigorous manuscripts with deep literature reviews and standardized evaluation.

    Source: §1 · p. 1
  • Method

    PaperOrchestra uses specialized agents to transform unconstrained pre-writing materials into submission-ready LaTeX manuscripts with literature synthesis, visuals, and iterative refinement.

    Source: §1 · p. 1, §3.1 · p. 3
  • Results

    50%–68% absolute win rate margins in literature review quality and 14%–38% in overall manuscript quality over autonomous baselines were achieved in side-by-side human evaluations.

    Source: §1 · p. 2
  • Takeaways & Limitations

    PaperOrchestra provides an assistive system for accelerating AI research-paper drafting while synthesizing deep, context-aware literature reviews.

    Source: §6 · p. 11, Ethics Statement · p. 11

    Human researchers must retain accountability for the factual accuracy, originality, and validity of generated manuscript claims.

    Source: Ethics Statement · p. 11

Abstract

from arXiv · show

Synthesizing unstructured research materials into manuscripts is an essential yet under-explored challenge in AI-driven scientific discovery. Existing autonomous writers are rigidly coupled to specific experimental pipelines, and produce superficial literature reviews. We introduce PaperOrchestra, a multi-agent framework for automated AI research paper writing. It flexibly transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts, including comprehensive literature synthesis and generated visuals, such as plots and conceptual diagrams. To evaluate performance, we present PaperWritingBench, the first standardized benchmark of reverse-engineered raw materials from 200 top-tier AI conference papers, alongside a comprehensive suite of automated evaluators. In side-by-side human evaluations, PaperOrchestra significantly outperforms autonomous baselines, achieving an absolute win rate margin of 50%-68% in literature review quality, and 14%-38% in overall manuscript quality.

1. Introduction

PaperOrchestra addresses the challenge of converting unconstrained research materials into rigorous, submission-ready manuscripts, while overcoming shallow literature synthesis and limited visual generation in existing systems. It introduces a standalone multi-agent framework and PaperWritingBench, and outperforms autonomous baselines in human evaluations.

  • Translating raw ideas and experimental logs into rigorous manuscripts remains a critical bottleneck as LLMs become active participants in scientific discovery.This challenge arises despite recent end-to-end autonomous frameworks demonstrating the feasibility of automated research loops.Source: §1 · p. 1
  • Existing survey-oriented RAG frameworks such as AutoSurvey2 (Wu et al., 2025) and LiRA (Go et al., 2025) structure literature review but cannot turn raw experimental data into full-length papers.Earlier LLM-based writing systems also risked factual hallucinations because they relied on parametric memory.Source: §1 · p. 1
  • Full-lifecycle research agents remain coupled to experimental loops and produce shallow, weakly cited reviews, limited code-generated plots, and lack standardized evaluation.Their literature-synthesis deficits are reported by Beel et al. (2025) and Tang et al. (2025b).Source: §1 · p. 1
  • PaperOrchestra autonomously converts unconstrained pre-writing materials into LaTeX manuscripts through specialized agents for literature synthesis, visual generation, and iterative refinement.Its visual capabilities include both plots and conceptual diagrams, alongside refinement for improved technical clarity.Source: §1 · p. 1
  • PaperWritingBench isolates AI paper writing with reverse-engineered ideas and experimental logs from 200 papers published at top-tier AI conferences.The benchmark is presented as the first standardized benchmark for AI research paper writing.Source: §1 · p. 2
  • PaperOrchestra achieves 50%–68% absolute win-rate margins in literature review synthesis and 14%–38% in overall manuscript quality over autonomous baselines.These results come from side-by-side human evaluations.Source: §1 · p. 2

2. Related Work

Prior systems automate research execution, text refinement, or literature-review generation, but remain constrained by structured inputs, internal pipelines, or insufficient contextual synthesis. PaperOrchestra addresses these gaps by transforming unconstrained pre-writing materials into manuscripts with targeted literature synthesis, visuals, and coherent narratives.

2.1. AI Researcher Frameworks

  • End-to-end AI researcher frameworks automate experimentation and drafting, but their writing modules remain rigidly coupled to internal experimental loops and cannot independently synthesize human-provided materials.Examples include AI Scientist-v1, AI Scientist-v2, Cycle Researcher, OmniScientist, InternAgent, and AI Co-Scientist.Source: §2.1 · p. 2

2.2. Automated Writing and Literature Synthesis

  • Automated writing systems generally lack flexible end-to-end manuscript generation from raw, under-specified materials, instead relying on entity graphs, structured inputs, analytical results, or human guidance.PaperRobot lacked complex data-driven narrative synthesis; Prism emphasizes stylistic refinement; data-to-paper translates structured results with backward traceability.Source: §2.2 · p. 2
  • Literature-review systems generate comprehensive surveys through retrieval, outline refinement, or multi-agent drafting, but they do not target related-work sections or selectively motivate a method’s research gap.AutoSurvey2, SurveyGen-I, and LiRA lack the contextual awareness needed to contrast prior work for a specific new method.Source: §2.2 · pp. 2–3
  • PaperOrchestra processes unconstrained pre-writing materials while autonomously synthesizing targeted literature, generating comprehensive visuals, and crafting coherent scientific narratives.Unlike CycleResearcher, AI-Researcher, and AI Scientist-v2, it does not require pre-existing plots, ground-truth references, or artifacts from an internal pipeline.Source: §2.2 · p. 3

3. Task and Dataset

PaperOrchestra maps unconstrained pre-writing materials to a complete submission package, while PaperWritingBench evaluates this process on 200 anonymized papers from CVPR 2025 and ICLR 2025. The task inputs include methodological, experimental, formatting, venue, and optional visual information, with visuals synthesized when none are provided.

3.1. Task Formulation

  • The framework maps an Idea Summary, Experimental Log, LaTeX Template, Conference Guidelines, and optional Figures to a finalized package containing source LaTeX and rendered PDF files.If no figures are provided, the pipeline autonomously synthesizes relevant visuals.Source: §3.1 · p. 3

3.2. Dataset Construction

  • PaperWritingBench is a new dataset comprising 200 accepted papers from CVPR 2025 and ICLR 2025, 100 papers each, with high academic standards and adaptability tested across double-column CVPR and single-column ICLR formats.Source: §3.2 · p. 3
  • The dataset pipeline samples papers from OpenReview and CVF Open Access, extracts structured text and visual entities, and discards incomplete or misparsed samples.MinerU generates mathematically faithful Markdown, while PDFFigures 2.0 extracts visual entities and captions.Source: §3.2 · p. 4
  • To prevent leakage, an LLM reverse-engineers self-contained, anonymized Idea Summaries and Experimental Logs from paper PDFs, removing authors, titles, citations, URLs, and explicit figure or table references.Idea Summaries exclude experimental results and have Sparse and Dense variants, while Experimental Logs retain setups, findings, baselines, datasets, metrics, and tabular data as standalone observations.Source: §3.2 · p. 4

4. PaperOrchestra

PaperOrchestra transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts through a five-step multi-agent pipeline covering planning, parallel visualization and literature work, section drafting, and iterative refinement. On CVPR and ICLR 2025 sparse-idea datasets, it significantly outperforms AI baselines in literature-review and overall paper quality across both judge models.

  • The five-step pipeline converts Idea Summary, Experimental Log, LaTeX Template, and Conference Guidelines into a complete manuscript, with Steps 2 and 3 running in parallel.Its stages are outline generation, plot generation, literature review, section writing, and iterative content refinement.Source: §4 · pp. 4–5
  • The Outline Agent creates a JSON plan for visualizations, targeted macro- and micro-level literature searches, section content, and citation hints for external dependencies.The literature strategy guides citation-map construction for the Introduction and Related Work sections.Source: §4 · p. 5
  • The Plotting Agent generates diagrams and statistical plots through PaperBanana’s VLM-critic refinement loop, while the Literature Review Agent verifies papers, enforces temporal cutoffs, deduplicates citations, and drafts related sections.Verified metadata supports an automatically generated BibTeX file and citation registry.Source: §4 · p. 5
  • The Section Writing Agent extracts experimental values into tables and integrates figures while drafting core sections, after which Content Refinement accepts revisions only when simulated-review scores improve under its tie-break rules.The final output is a LaTeX document and compiled PDF; score decreases, unfavorable ties, or iteration limits stop refinement.Source: §4 · p. 5
  • PaperOrchestra significantly outperforms Single Agent and AI Scientist-v2 in literature-review and overall paper quality across both judge models on CVPR and ICLR 2025 sparse-idea datasets.The Human (GT) baseline is included as an upperbound reference.Source: Figure 2 · p. 6

5. Experiments

PaperOrchestra is evaluated against single-agent and AI Scientist-v2 pipelines using automated, side-by-side, human, and ablation studies. It consistently improves manuscript quality, literature synthesis, citation coverage, and refinement outcomes across the reported settings.

5.1. Baselines

  • Evaluations compare PaperOrchestra with Single Agent and AI Scientist-v2 under sparse ideas, ground-truth visual inputs, and a universal anti-leakage prompt.Autoraters measure citation F1, literature-review quality, holistic quality, and side-by-side manuscript preferences.Source: §5.1 · p. 6, §5.2 · pp. 6–7

5.3. Results

  • PaperOrchestra achieves 88%–99% absolute Literature Review win margins and 39%–86% and 52%–88% Overall Paper Quality margins over AI Scientist-v2 and Single Agent, respectively.Human GT remains the upper bound for Overall Paper Quality.Source: §5.3 · p. 8, Figure 3 · p. 9
  • PaperOrchestra reaches 84% CVPR and 81% ICLR simulated acceptance rates, gains 13% and 9% over the strongest autonomous baseline, and closely tracks Human GT.These results use the ScholarPeer holistic technical-quality framework.Source: Table 2 · p. 7, §5.3 · p. 8
  • PaperOrchestra improves citation coverage with 45.73–47.98 references, P0 Recall gains of 2.13%–6.07%, and P1 Recall gains of 12.59%–13.75% over strongest baselines.Baselines average 9–14 citations and have near-zero P1 Recall, while Human GT writeups contain approximately 59 citations.Source: §5.3 · p. 8, Table 3 · p. 8

5.4. Human Evaluation

  • Human evaluations show PaperOrchestra surpassing AI baselines by 50%–68% in Literature Review and 14%–38% in Overall Quality, with a 43% tie/win rate against Human GT.GPT-5 Overall Quality scores correlate with human preferences at Pearson r=0.6458 and Spearman ρ=0.6355.Source: §5.4 · pp. 9–10

5.5. Ablation Studies

  • Dense ideas dominate Overall Paper Quality at 43%–56% versus 18%–24%, while Sparse ideas remain competitive for Literature Review Quality at 32%–40% versus 28%–39%.The sparse setting demonstrates autonomous targeted searching and research-gap identification without heavy reliance on dense human input.Source: §5.5 · p. 10, Table 5 · p. 10
  • PlotOn remains competitive with PlotOff, securing ties or wins in 51%–66% of SxS matchups despite lacking access to data represented by ground-truth visuals.PlotOn generates coherent visuals from scratch to augment manuscripts and reinforce scientific narratives.Source: §5.5 · p. 10, Table 6 · p. 10
  • AfterRefine manuscripts achieve 79%–81% win rates with 0% losses, while refinement raises simulated acceptance by 19% CVPR and 22% ICLR and overall scores by 0.88 and 1.61.The loop applies targeted clarity and presentation corrections.Source: §5.5 · p. 10, Figure 4 · p. 11

6. Conclusion

PaperOrchestra and PaperWritingBench transform unstructured preliminary AI research materials into submission-ready manuscripts, with experiments showing high-quality papers, competitive runtime, and deep, context-aware literature reviews. Future work will expand artifact ingestion and support interactive human-AI writing environments.

  • PaperOrchestra and PaperWritingBench transform unstructured preliminary AI research materials into submission-ready manuscripts.Source: §6 · p. 11
  • Experiments show that the multi-agent framework generates high-quality research papers with competitive runtime and deep, context-aware literature reviews.Competitive runtime is reported in Appendix B.Source: §6 · p. 11
  • Future work will expand the framework’s ability to ingest richer research artifacts and enable interactive, dynamic writing environments for human-AI scientific collaboration.Additional limitations are discussed in Appendix A.Source: §6 · p. 11

Ethics Statement

The system is positioned as an assistive drafting tool rather than an autonomous author, with human researchers retaining accountability for generated manuscripts and verifying outputs.

  • PaperOrchestra accelerates AI paper drafting as an assistive tool, not an independent entity capable of claiming authorship.Source: Ethics Statement · p. 11
  • Human researchers remain accountable for manuscript factual accuracy, originality, and validity, and must verify outputs to prevent LLM-derived biases or misinformation.Programmatic safeguards, including API-grounded citation validation, help minimize hallucinations and support academic rigor.Source: Ethics Statement · p. 11

A. Limitations and Future Work

PaperOrchestra’s limitations concern visual-generation control, the degree of human interaction, and possible pre-training contamination in benchmark evaluations. Future work targets factual visual verification, human-in-the-loop refinement, and benchmarks using unpublished or autonomously generated materials.

Visual generation

  • External PaperBanana-based visual generation limits direct control over figure hallucinations, motivating targeted Vision-Language Models and human evaluations for factual accuracy and LaTeX placement.These measures would systematically verify generated visual content and its layout.Source: A. Limitations and Future Work · p. 14

Human interaction

  • Interactive human-in-the-loop refinement could let researchers steer drafts through natural-language critiques, positioning PaperOrchestra as an assistive rather than fully independent writing tool.This extends the current refinement agent, which improves papers using structured, LLM-generated feedback.Source: A. Limitations and Future Work · p. 14

Benchmark validity

  • PaperWritingBench evaluations risk pre-training contamination, although de-contextualization, anonymization, and shared language models preserve a fair relative comparison across baselines.Because all baselines use the same underlying language models, potential knowledge leakage applies uniformly.Source: A. Limitations and Future Work · p. 14
  • Future benchmarks could isolate generation from memorization by using unpublished research or autonomously generated raw materials.This addresses the contamination risk inherent in evaluating foundation models on PaperWritingBench.Source: A. Limitations and Future Work · p. 14

B. Computational Cost · C. Dataset Details

PaperOrchestra achieves substantial manuscript-quality and citation-depth improvements with only modest latency overhead versus AI Scientist-v2. Its computational design separates high-concurrency discovery from rate-limited citation verification and distributes work across specialized agents.

B. Computational Cost

  • 39.6 minutes mean processing time versus 35.1 minutes for AI Scientist-v2, despite PaperOrchestra requiring ∼60–70 LLM calls versus ∼40–45.Means were computed on 10 papers—5 CVPR and 5 ICLR—with the highest and lowest runtimes removed.Source: Table 7 · p. 14, B. Computational Cost · p. 14
  • Retrieval is decoupled into parallel candidate discovery using 10 concurrent workers and sequential Semantic Scholar verification limited to 1 query per second.This combines LLM API concurrency with Semantic Scholar’s throughput constraints to prevent quota-induced latency.Source: B. Computational Cost · p. 15
  • The Outline Agent and Section Writing Agent each use 1 call for manuscript planning and comprehensive multimodal LaTeX drafting, respectively.Source: B. Computational Cost · p. 15
  • The Hybrid Literature Agent uses ∼20–30 calls for paper discovery and citation verification, while the Plotting Agent uses ∼20–30 calls for visual generation and critique.Plotting includes few-shot retrieval, visual planning, image generation, VLM-guided critique and redraw cycles, and context-aware captioning.Source: B. Computational Cost · p. 15
  • The Content Refinement Agent uses ∼5–7 calls for score-driven iterative reflection and manuscript revision.Source: B. Computational Cost · p. 15
  • PaperOrchestra delivers substantial improvements in overall manuscript quality and citation depth while adding only minimal latency versus AI Scientist-v2.Source: B. Computational Cost · p. 15

C.1. Data Distribution … B. Dense Prompting Module (DPM)

The paper constructs PaperWritingBench with controlled raw-material distributions and extraction procedures, then presents TSAM as a frozen-SAM extension that adds temporal modeling and automated multimodal prompting for Ref-AVS. Its design combines cached temporal fusion, sparse audio-text queries, dense cross-attended embeddings, and standard SAM decoding.

C.1. Data Distribution

  • ICLR papers average 9.19 figures, 8.13 tables, and 2,387 experimental-log words versus CVPR’s 5.20, 4.20, and 1,530, while dense and sparse inputs remain venue-consistent.Dense idea files average 1,083 CVPR and 1,057 ICLR words; sparse files average 591 and 587, with roughly 59 references per manuscript.Source: §C.1 · p. 15, Figure 5 · p. 15, Figure 6 · p. 16

C.2. Raw Material Extraction

  • PaperWritingBench extraction controls granularity with dense materials preserving mathematical formulations and sparse materials replacing notation with high-level conceptual narratives.Experimental logs use MinerU-parsed markdown and PDFFigures 2.0 visual assets, injecting table and figure images into Gemini-3.1-Pro to preserve fidelity and produce standalone observations.Source: §C.2 · p. 16

C.3. Raw Material Example

  • The raw-material example exposes sparse ideas, dense ideas, and experimental logs for a CVPR 2025 paper whose SAM-based problem involves missing temporal awareness and dependence on explicit prompts.Its hypothesis introduces a temporal branch and data-driven multimodal prompts to adapt frozen SAM knowledge for dynamic audio-visual segmentation.Source: §C.3 · p. 17

1. Temporal Modeling Branch (Context Injection)

  • TSAM freezes the heavyweight SAM image encoder and adds a parallel temporal branch with cached memory and adapters to fuse historical visual states with current audio-text embeddings.The branch is intended to maintain object permanence and temporal consistency while minimizing trainable additions.Source: §1 · p. 17, §C.3 · p. 17

2. Automated Multimodal Prompting

  • TSAM replaces manual clicks with multimodal prompting: audio-text correlation selects sparse queries analogous to points, while cross-modal fusion generates dense spatial embeddings analogous to masks.The sparse module uses relevant audio cues, and the dense module mixes visual features with aligned audio and text through cross-attention.Source: §2 · pp. 17–18

3. Decoding

  • The standard SAM mask decoder produces the final segmentation from synthetic sparse and dense prompts refined by temporally aware auxiliary features.This preserves the original decoder while changing the information supplied to it.Source: §3 · p. 18

Expected Contribution; Core Hypothesis

  • TSAM’s stated contribution is latent prompting: lightweight temporal adapters and cross-modal attention translate semantic audio-text cues into SAM’s geometric point and mask prompts without compromising zero-shot capabilities.The framework targets SAM’s lack of temporal dynamics and explicit user-interaction requirements in Ref-AVS.Source: Expected Contribution · p. 18, Core Hypothesis · p. 18

1. Input Formulation and Feature Extraction; 2. Temporal Modality Fusion and Cached Memory; 3. Temporal Modeling Branch; 4. Multimodal Prompting Modules; B. Dense Prompting Module (DPM)

  • The detailed design samples 1-second video frames, encodes audio with VGGish and text with RoBERTa, projects both into SAM’s visual space, and fuses them with temporal attention and cached memory.A trainable branch parallel to the final M frozen SAM blocks combines cached memory, bottleneck adapters, and encoder outputs; language-guided top-k audio cues form sparse prompts, while audio and text cross-attention followed by feed-forward refinement forms dense embeddings.Source: §1 · p. 19, §2 · p. 19

5. Training Objective · Expected Contribution

The framework is trained end-to-end with a weighted combination of Binary Cross-Entropy and Intersection over Union losses, while its expected contributions address architecture, temporal adaptation, and multimodal prompting. The loss components are balanced with λ=1.0.

5. Training Objective

  • The model is trained end-to-end by comparing predicted masks with ground truth using a weighted sum of Binary Cross-Entropy and Intersection over Union losses.Source: §5 · p. 20
  • The training objective combines Binary Cross-Entropy loss, LBCE, with Intersection over Union loss, LIoU.Source: §5 · p. 20
  • λ=1.0 balances the contributions of the two loss components.Source: §5 · p. 20

Expected Contribution

  • TSAM repurposes the Segment Anything Model for referring audio-visual segmentation without extensive retraining of the image backbone.Source: Expected Contribution · p. 20
  • A lightweight Temporal Modeling Branch enables SAM to capture spatio-temporal interactions across video frames and address its static-image limitation.Source: Expected Contribution · p. 20
  • Automated multimodal prompting converts audio-visual-text correlations into the sparse and dense prompts required by SAM.Source: Expected Contribution · p. 20

1. Experimental Setup … Core Principle

The experiments evaluate TSAM on Ref-AVS and show strong performance on Seen and Unseen data, while PaperOrchestra’s implementation enforces controlled model settings, verified citations, strict knowledge isolation, and anonymized reconstruction from session materials.

1. Experimental Setup

  • The Ref-AVS evaluation uses 20,000 text expressions with pixel-level annotations across 4,000 10-second videos, covering audible and static inaudible object categories.Audible categories include 20 musical instruments, 8 animals, 15 machines, and 5 humans, alongside 3 static inaudible categories.Source: §1 · p. 20
  • TSAM achieved the highest performance on Seen and Unseen test sets, improving over EEMC by 9.23% and 5.04% in Seen and Unseen Jaccard Index, respectively.It slightly trailed EEMC on the Null set, with S values of 0.017 versus 0.007, attributed to SAM producing masks even without a target object.Source: §1 · pp. 22–23
  • Removing the temporal branch reduced Seen Jaccard from 43.43% to 33.05%, while removing prompting, memory, or adapter components also degraded segmentation performance.The temporal branch supports cross-frame generalization; SPM and DPM are complementary, cached memory supports temporal alignment, and the adapter facilitates deep multimodal interaction, especially on Unseen data.Source: §1 · p. 23

D.1. Models and Configuration

  • PaperOrchestra fixes Gemini-3.1-Pro as the manuscript-writing backbone, uses Gemini-3-Flash with Google Search grounding for literature discovery, and aligns research cutoffs with venue deadlines.Automated technical evaluations use temperature 0.75, side-by-side evaluations use 0.0, GPT-5 remains at its default 1.0, and ScholarPeer disables baseline scouting.Source: §D.1 · p. 24

D.2. Baseline Selection

  • Baselines are excluded when they lack reproducible code, cannot generate submission-ready LaTeX, require structured intermediate inputs, or lack mechanisms for full-paper drafting from raw materials.These constraints motivate excluding OmniScientist, PaperRobot, Data-to-Paper, AI-Researcher, CycleResearcher, and generic RAG pipelines.Source: §D.2 · p. 24

D.3. Citation Verification

  • Citation gathering uses discovery followed by Semantic Scholar verification, retaining entities that match titles, provide abstracts, satisfy the research cutoff, and are deduplicated by paper ID.The writing agent is restricted to verified papers and is instructed to actively cite at least 90% of the gathered literature pool in the Introduction and Related Work.Source: §D.3 · p. 25

D.4. Information Leakage Prevention; Strict Knowledge Isolation & Anonymity (Critical); Forbidden Behavior

Anonymity Requirement; Allowed Sources; Core Principle

  • The final manuscript must remain fully anonymized and independently reconstructed using only session materials and logical reasoning derived from them.This requirement supports double-blind review and overrides conflicting instructions, while prohibiting author, affiliation, institution, acknowledgement, email, and corresponding-author metadata.Source: Anonymity Requirement · p. 26, Allowed Sources · p. 26, Core Principle · p. 26

D.5. Human Evaluation … Mixed-Attention Mechanism

The paper evaluates PaperOrchestra through blinded human comparisons, manuscript visualizations, and architectural analyses. Results show consistent gains over AI baselines, while input granularity and shared, edge-aware modules shape technical depth and generality.

D.5. Human Evaluation

  • PaperOrchestra systematically outperforms Single Agent and AI Scientist-v2 across evaluation metrics, although it remains behind human-written ground truth.Eleven AI researchers completed 180 randomized paired evaluations across 40 PaperWritingBench papers, covering literature review synthesis and overall paper quality.Source: Figure 7 · p. 26, §D.5 · p. 27

Literature Review Quality (Focus on Introduction & Related Work)

Overall Paper Quality (Holistic Review)

General Rules

  • Evaluators compared papers independently and ignored templates, metadata, author identities, length, and verbosity when judging intrinsic research and writing quality.Each paper was evaluated independently before pairwise comparison.Source: General Rules · p. 28

E.1. Side-by-Side (SxS) Paper Visualizations

  • Side-by-side visualizations compare Single Agent, AI Scientist-v2, PaperOrchestra (PlotOff), and PaperOrchestra (PlotOn) on CVPR and ICLR samples under sparse ideas.The section displays generated manuscripts from raw materials for two PaperWritingBench samples.Source: §E.1 · p. 29, Figure 9 · p. 29, Figure 10 · p. 30

E.2. Sparse vs. Dense Idea Settings

  • Dense inputs produce methodology grounded in formal mathematical notation, whereas sparse inputs yield a conceptual, high-level architectural overview.The comparison uses an ICLR split sample (Drakulic et al., 2025) and explicitly contrasts definitions such as BQ-MDPs.Source: §E.2 · p. 37

Modular Input/Output via Shared Codebooks

  • Task-specific adapters and shared node and edge codebooks map heterogeneous inputs into compatible backbone subspaces, promoting a common vocabulary and preventing catastrophic interference.The shared codebooks project low-dimensional task features into the backbone’s high-dimensional embedding space.Source: Modular Input/Output via Shared… · p. 37

Mixed-Attention Mechanism

  • Mixed-Attention integrates explicit edge attributes into node attention, while parameter-shared relation-specific blocks process heterogeneous node types without specialized modules for every topology.The backbone uses 9 layers, 8 attention heads, and a feed-forward dimension of 512; shared parameters preserve its generalist nature.Source: Mixed-Attention Mechanism · pp. 37–38

Multi-Task Imitation Learning Strategy … F. Prompts

This section presents strategies and prompts for unifying heterogeneous tasks and generating research materials, including manuscripts, reverse-engineered raw materials, exhaustive experimental logs, and automated citation evaluations, while preserving source fidelity and formatting constraints.

Multi-Task Imitation Learning Strategy

  • GOAL formulates multi-task combinatorial optimization as sequential decision-making over disjoint BQ-MDPs, decoupling task-specific feature mapping from a shared reasoning backbone.Expert trajectories from classical oracle solvers supervise autoregressive solution construction.Source: Multi-Task Imitation Learning Strategy · p. 38

Task Representation and Input Adapters

  • Task-specific adapters compress variable node and edge features before a shared linear projection maps them into the common backbone embedding space.The shared projection is a continuous low-rank mapping rather than a discrete codebook.Source: Task Representation and Input Adapters · p. 38

Multi-Head Mixed-Attention Mechanism

  • Mixed-Attention integrates dense asymmetric edge features directly into query and key vectors, while edge-free inputs reduce to standard attention without added computational overhead.Unlike scalar distance biases, the mechanism projects multidimensional relational features before the attention dot product.Source: Multi-Head Mixed-Attention Mechanism · p. 39
  • For heterogeneous graphs, GOAL instantiates attention blocks only for valid node-type pairings and shares their parameters within each layer to preserve generalist reasoning.This structured composition avoids the quadratic waste of materializing full-sequence tensors for invalid interactions.Source: Multi-Head Mixed-Attention Mechanism · p. 39

Training Protocol and Decoding

  • GOAL canonicalizes symmetric routing trajectories and deterministically orders oracle-selected subsets for unordered tasks before minimizing cross-entropy over valid actions.A dynamically updated action mask produces the policy distribution, while task-specific output projections convert backbone representations into selection logits.Source: Training Protocol and Decoding · p. 40
  • Cached autoregressive decoding processes static graph features once and uses a lightweight output adapter for dynamic states, preventing quadratic backbone recalculations up to 1000-node instances.This design caches node and edge embeddings while recomputing only the next valid action from changing state features.Source: Training Protocol and Decoding · p. 40

F.1. Paper Writing

  • The paper-writing prompt converts methodology, experimental logs, templates, and conference guidelines into a venue-compliant manuscript while requiring exact data, citations, figures, and preserved template content.It also requires completed LaTeX output, faithful figure use, and formal, technically grounded writing.Source: §F.1 · p. 40, Critical Instructions · p. 47

F.2. Raw Material Generation

  • The raw-material prompts reverse-engineer papers into anonymized concept notes or technical proposals, separating conceptual or methodological content from experiments, results, and conclusions.The concept-note version emphasizes sparse high-level intuition, whereas the technical-proposal version preserves equations, variables, architecture, and mathematical detail.Source: §F.2 · pp. 54–55
  • Experimental-log generation extracts raw empirical facts from paper content into a self-contained markdown source of truth for future results writing.The log must be exhaustive, meticulous, and fully accurate with numeric values, while using past-tense descriptions and omitting figure or table references.Source: §F.2 · p. 57, Core Instructions · p. 57

Core Instructions

  • Core instructions require structured extraction of table data, factual observations from captions and textual figure analyses, anonymization, and reproducibility details for datasets, metrics, and baselines.The output should not recreate tables or include citations and URLs; it should instead report raw numeric data and experimental setup details.Source: Core Instructions · p. 57

F.3. Autoraters

  • Autorater prompts evaluate citation quality by asking an expert academic reviewer to partition references into two priority categories.The section introduces Citation F1 through a P0/P1 partition of the provided references.Source: §F.3 · p. 58

Output Format (Strict JSON Only)

  • The citation evaluator must return strict JSON containing citation statistics, six scored evaluation axes, penalties, strengths, weaknesses, improvements, and an overall score.The schema includes citation coverage, relevance, synthesis, novelty, organization, and citation-practice assessments with numerical scores and justifications.Source: Output Format (Strict JSON Only) · p. 62
Loading 2604.05018v1…