PaperlayerView exact revision on arXiv

Source-linked AI summary

Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill

Zhuoyang Qian, Biao Wu, Yiran Wang, Chris D Yan, Desan Dai, Liangwei Zheng, Jin Jiang, Junsheng Zhang, Wenhao Wang

arXiv:2608.11924v1cs.CL

TL;DR

End-to-end research paper generation requires coordinated literature retrieval, experimentation, evidence-based revision, figure production, and long-horizon consistency beyond text generation. Spark-to-Paper implements this process as composable skills inside an existing coding assistant, achieving 99.5% citation validity and 96.4% figure editability across eight controlled topics.

  • Problem

    Existing research agents typically use standalone platforms, while skill-based coding-assistant tools generally stop short of complete research-to-paper generation.

  • Method

    Spark-to-Paper uses thirteen composable skills with preplanned experiments, evidence-guided claim revision, deterministic checks, self-critique, and code-based editable figures.

  • Results

    99.5% citation validity and 96.4% figure editability were achieved across eight controlled research topics, while fabrication detection rose from 14% to 92% with the full review stack.

  • Takeaways & Limitations

    End-to-end research paper generation can run as a lightweight workflow inside an existing coding assistant while keeping experimental evidence central to claim decisions.

  • Takeaways & Limitations

    Long-horizon generation can enter a Self-Refutation Loop in which repeated experimentation and revision continue to yield insufficient evidence for the intended claim.

Abstract

from arXiv · show

Turning a research idea into a complete paper requires more than text generation: the system must retrieve literature, design and execute experiments, revise claims according to evidence, produce publication-ready figures, and maintain consistency across a long generation process. We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant, without requiring a separate agent platform or orchestration service. Spark-to-Paper separates model-based judgment from deterministic operations that can be directly executed and checked. It further separates experiment planning from reporting, so that required evidence is specified before results are observed and manuscript claims are revised according to measured outcomes. To improve reliability over long research trajectories, the system combines deterministic integrity checks with self-critique and bounds a failure mode we call the Self-Refutation Loop, in which repeated experiments continue to reject the original research objective. Spark-to-Paper also produces editable vector figures through programmatic plotting for experimental results and code-based reconstruction for generated method diagrams. Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability. A controlled ablation increases fabrication detection from 14% for a single-pass draft to 92% with the full integrity and review stack, while adversarial review achieves 74% precision. The full system uses 11.9M tokens, costs $8.1 per manuscript, and requires 3.2 hours on average. These results show that end-to-end research paper generation can be implemented as a lightweight, composable workflow inside existing coding assistants while keeping experimental evidence central to how claims are accepted, revised, or abandoned.

1 Introduction

Spark-to-Paper is an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant. It centers experimental evidence through precommitted planning, evidence-guided claim revision, integrity checks, self-critique, and editable figure generation.

  • Scope: The workflow addresses literature retrieval, experiment execution, manuscript writing and review, figure generation, and consistency maintenance across long-running research processes.These capabilities extend beyond ordinary paper drafting, which generally stops short of the complete research-to-paper process.
  • Evidence-centered generation: The system separates experiment planning from reporting by specifying required evidence before observing results and revising manuscript claims according to resulting artifacts.Experiments are executed to resolve missing evidence, making evidence central to whether claims are retained or revised.
  • Reliability mechanisms: Spark-to-Paper combines deterministic integrity checks with long-horizon self-critique and bounded recovery from self-refutation loops.Its evidence-grounded mechanisms include precommitted experiment design, evidence-guided claim revision, and deterministic checks.
  • Evaluation: Across eight controlled research topics, the full system achieves 99.5% citation validity and 96.4% figure editability.The system produces editable figures through programmatic or code-based reconstruction mechanisms described among its contributions.
  • System design: Spark-to-Paper implements end-to-end research paper generation through thirteen composable skills inside an existing coding assistant, without a separate agent platform or orchestration service.Skills communicate through artifacts in a shared project directory, while a lightweight pipeline specifies only the high-level task order.

2 Related Work

Related work divides between standalone systems that automate the research cycle and composable skills embedded in existing coding assistants. Spark-to-Paper also builds on agent methods for tool use and self-critique, while addressing citation and factual fabrication through retrieval and verification.

  • Autonomous research systems: Standalone systems automate ideation, experimentation, writing, and often self-review as an autonomous research cycle.Examples include systems that generate, run, and write up machine-learning experiments with simulated review, alongside self-reinforcing human-in-the-loop pipelines.
  • Autonomous research systems: These systems match or exceed Spark-to-Paper’s breadth but typically ship as standalone applications with dedicated orchestration infrastructure.Examples of additional infrastructure include orchestration servers, graph databases, and cluster schedulers.
  • Composable assistant skills: A second line decomposes literature search, drafting, and review into composable steps inside an existing coding assistant.Other systems in this line automate narrower tasks such as web-grounded report writing or single-GPU experiment iteration.
  • Agent foundations: Spark-to-Paper’s internal design draws on interleaved reasoning and action, external tools, and model-based refinement and review.The cited external tools include web search, a DOI resolver, and a plotting library.
  • Integrity and verification: Retrieval-augmented generation mitigates fabricated citations and claims, while Spark-to-Paper specializes retrieval-then-verification to bibliographic metadata and tabular-result fidelity.The tabular-results rule connects to work on faithfully generating text from structured data, where the risk is an invented source rather than an invented source passage.

3 Spark-to-Paper

Spark-to-Paper is a thirteen-skill workflow embedded in an existing coding assistant, combining model-based judgment with deterministic execution and checks. Its staged pipeline routes inputs, generates and reviews manuscripts and figures, and conditionally incorporates measured experimental evidence.

  • Architecture: Thirteen composable skills run inside an existing coding assistant to handle research planning, literature search, manuscript generation, review, figure production, and experiments.The system reuses the assistant’s abilities to read files, execute code, search for information, and call external tools instead of using a separate agent system.
  • Architecture: Model judgment handles context-dependent decisions, while deterministic tools perform operations with explicit rules that can be mechanically executed and checked.Judgment includes organizing arguments, selecting relevant literature, assessing evidential support, and presenting results.
  • Pipeline: Stage 0 routes inputs and selects Proposal Mode or Data-Aware Mode according to whether measured experimental results are available.Proposal Mode leaves unavailable results unspecified, whereas Data-Aware Mode permits measured values supported by provided data.
  • Pipeline: Stage 8 executes feasible planned experiments, records measurements, regenerates result-dependent artifacts, and revises manuscript claims to match the new evidence.Measured outputs can update result tables, data-dependent figures, and claims in the abstract, introduction, experiments, and conclusion.

4 Experiment Design and Revision

Spark-to-Paper commits the experimental design before execution, then runs only the evidence needed to evaluate manuscript claims. Afterward, it classifies claims by evidential support and revises the manuscript while preserving negative or inconclusive findings.

  • Precommitted experiment design: Before execution, Spark-to-Paper fixes datasets, baselines, metrics, ablations, and required result tables, leaving numerical cells empty until experiments finish.This creates a lightweight form of preregistration and reduces adaptation of the evaluation protocol after results are observed.
  • Evidence-driven execution: The system maps claims to required evidence and executes the minimal experiments needed to resolve gaps rather than exhaustively following a generic template.Paper-type guidance can recommend designs and failure checks, but committed designs remain authoritative; costly runs require claim-specific and resource-based justification.
  • Bounded revision: Seven is the cap on experiment–critique–revision cycles, after which an unresolved trajectory becomes a failure report rather than a successful manuscript.Deterministic gates and review stages challenge verifiable and semantic issues before revision proceeds.
  • Evidence-based revision: After execution, claims are classified as SUPPORTED, PARTIALLY-SUPPORTED, UNSUPPORTED, CONTRADICTED, or NEEDS-CONFIRMATION and are retained, weakened, removed, relocated, or experimentally revisited.Null, negative, and inconclusive results are preserved, with resulting changes propagated across the manuscript.

5 Integrity and Self-Critique

Spark-to-Paper combines deterministic integrity gates with model-based self-critique to detect explicit and semantic errors during long-horizon manuscript generation. It also bounds repeated experiment–critique–revision cycles and records unsupported research trajectories as failure reports.

  • Deterministic integrity gates: Deterministic gates verify the blueprint, citations, manuscript content, figures, compiled LATEX project, notation, required files, compilation, and result integrity.These checks handle properties with explicit correctness criteria through directly verifiable programs rather than model judgments.
  • Model-based critique: Model-based Self-Review checks edited content against the surrounding manuscript, while Adversarial-Review challenges whether arguments remain supported and consistent.These critiques address semantic issues that deterministic checks cannot determine, including terminology drift, redundancy, local inconsistency, and insufficient support.
  • Long-horizon correction: Later evidence can revise earlier claims and writing decisions, including changes to the experiments section, introduction, abstract, or conclusion.The correction loop challenges a decision, attempts to refute the challenge, and revises the manuscript when the challenge remains valid.
  • Self-Refutation Loop: The Self-Refutation Loop arises when successive method, design, or analysis revisions still fail to provide sufficient evidence for the intended claim.The system begins with a hypothesis or design objective, plans experiments, evaluates evidence, and may repeat this process when support remains insufficient.
  • Self-Refutation Loop: Seven experiment–critique–revision cycles bound unresolved loops; unsupported trajectories are terminated and documented in failure reports.Reports record the original idea, attempted methods and experiments, observed results, and reasons the evidence was insufficient.

6 Editable Figure Generation

Spark-to-Paper uses role-aware figure generation: explanatory figures are reconstructed from generated raster targets into editable vector artifacts, while experimental-result figures are plotted directly from measured data. This preserves visual flexibility for method diagrams and strict grounding for quantitative figures.

  • Role-aware figure generation: Spark-to-Paper separates figure generation by role: explanatory figures use raster visual targets and reconstruction, whereas experimental figures use plotting programs driven by measured results.This separation enables generative visual design for explanatory content while keeping quantitative figures grounded in experimental measurements.
  • Explanatory figures: Method figures and system diagrams are generated from manuscript descriptions as initial raster images that specify their intended structure, layout, and presentation.The raster image serves as a visual target rather than the final paper artifact.
  • Explanatory figures: Accepted reconstructions are rendered from HTML to PDF, preserving editable text and vector graphics for publication-ready manuscript figures.The reconstructed HTML/PDF becomes the final artifact used in the manuscript, without manually redrawing the original image.
  • Experimental-result figures: Experimental-result figures are generated without image models by writing plotting code that converts recorded measurements into PDFs for comparisons, ablations, and trend plots.Because these figures correspond directly to measured values, their visual content is produced from experimental outputs.

7 Evaluation Protocol

Spark-to-Paper is evaluated through controlled experiments, retrospective audits, and qualitative case studies spanning artifact quality, efficiency, integrity, and autonomous research capability. The protocol uses preregistered topics, independent checks, and fixed evaluation procedures to compare systems and isolate quality-control contributions.

  • Evaluation design: The evaluation combines controlled experiments, retrospective audits, and qualitative case studies to assess reliable, editable, and efficient end-to-end manuscripts.Comparisons target artifact quality and generation cost, using released papers and artifacts for prior systems when possible.
  • Evaluation design: The protocol fixes topics, model configurations, and criteria before generation, registers the protocol externally, and reports outcomes regardless of whether they favor Spark-to-Paper.Independent procedures are used whenever possible, including external bibliographic metadata services for post-generation citation verification.
  • Artifact comparison: 99.5% citation validity and 96.4% figure editability are achieved, exceeding prior autonomous systems’ 91%–96% citation validity and limited figure editability.Spark-to-Paper results come from eight topics; paired single-pass comparisons use three topics, with human preprints evaluated through the same external procedures.
  • Quality-control ablation: Fabrication detection rises from 14% for a single-pass draft to 69% with gates, 81% with self-review, and 92% with the complete adversarial-review stack.Adversarial review achieves 74% precision, measured from 57 independently assessed issues after excluding three “cannot tell” judgments.
  • Qualitative case studies: Two case studies show autonomous research-gap identification, novel-method synthesis, and proposal refinement, including replacing Accuracy-focused evaluation with multiple metrics for clinical screening.The demonstrations inject two incorrect expectations to test revision capabilities and address miscalibration, interpretability, and data imbalance.

8 Conclusion

Spark-to-Paper is an end-to-end research paper generation system built from thirteen composable skills inside an existing coding assistant. It combines native file interaction, tool use, and code execution with model-based judgment and deterministic operations that support direct verification.

  • Spark-to-Paper comprises thirteen composable skills implemented inside an existing coding assistant.
  • The system reuses the assistant’s native capabilities for file interaction, tool use, and code execution.
  • It separates model-based judgment from deterministic operations that support direct verification.

A Deterministic Gates and Failure Conditions

Spark-to-Paper uses deterministic gates to verify explicit correctness criteria before artifacts move downstream, distinguishing execution-stopping fatal violations from non-blocking warnings. These gates check structural, bibliographic, figure, evidence-grounding, and compilation properties, while semantic judgments remain outside their scope.

  • Gate design: Deterministic gates read stage artifacts, emit structured reports, and return pass-or-fail decisions before downstream processing.Fatal violations stop execution; warnings are recorded without blocking progress, and fatality is reserved for invalid, inconsistent, or unverifiable artifacts.
  • Structural gates: The Template and Blueprint Gates enforce venue and manuscript structural consistency before manuscript generation.They check document requirements and planned sections, contributions, figures, tables, citation types, and title constraints.
  • Integrity gates: The Citation Gate detects bibliographic and citation–claim inconsistencies, with invalid identifiers causing failure and temporary resolution errors producing warnings.Checks include malformed or duplicate entries, unresolved keys, unused references, and optional DOI, URL, or preprint verification.
  • Artifact gates: The Figure and Compilation Gates verify figure existence, representation, generation-path consistency, evidence grounding, editability structure, and successful project compilation.Raster figures remain permissible for inherently raster content or when reliable reconstruction is unavailable.
  • Scope and limitations: The gates verify mechanically checkable properties but do not judge contribution importance, argument persuasiveness, experimental appropriateness, or figure communication.Model-based review handles questions requiring semantic judgment.

B Experiment Execution Details

Spark-to-Paper executes experiments as an evidence-grounded process that derives required evaluations from manuscript claims and the planned design, then verifies measurements before admitting them as evidence. It preserves committed experimental designs and records missing dependencies instead of substituting altered or generated results.

  • Evidence-Grounded Experiment Execution: Spark-to-Paper preserves the experimental design committed during planning and does not replace specified evaluations with easier proxies, altered data, or synthetic observations.When resources are unavailable, it records the missing dependency and leaves the corresponding result unspecified.
  • Evidence-Grounded Experiment Execution: The system identifies manuscript claims requiring evidence, derives corresponding experiments from the design, assesses feasibility, executes feasible experiments, verifies results, and updates tables and figures.Algorithm 1 records configurations, seeds, logs, and measurements before admitting verified results into experiment artifacts.
  • Evidence-Grounded Experiment Execution: A measurement becomes verified evidence only when traceable to source artifacts and, for aggregated results, recomputable from underlying runs.Recorded artifacts include the configuration, dataset, random seed, metric definition, logs, and raw measurements.

C Claim Admission Protocol

Spark-to-Paper admits manuscript claims by matching them to verified evidence and the paper’s contribution type, then revises claims when evidence is weaker, negative, null, or inconclusive. The protocol combines deterministic integrity checks with model-based semantic assessment and propagates approved revisions across the manuscript.

  • Evidence matching: The system identifies the paper’s primary contribution type before assessing claims because different claim types require different evidence.Examples include frameworks, models, benchmarks, datasets, systems, and empirical studies.
  • Claim labels: Each major claim receives one of five labels: supported, partially-supported, unsupported, contradicted, or needs-confirmation.Supported and partially-supported labels require links to specific verified evidence, and the label determines the revision action.
  • Claim revision: Claims of statistical significance, robustness, generalization, or broad empirical superiority are retained only when corresponding evidence has been obtained.When evidence supports a narrower conclusion, the manuscript states that narrower conclusion explicitly.
  • Claim revision: Negative, null, and inconclusive results are preserved, allowing the research narrative to weaken when required by measured evidence.Claim weakening changes a statement’s scope rather than merely reducing its rhetorical strength.
  • Manuscript consistency: Revisions propagate across every occurrence of an affected claim, with the abstract revised last to reflect the completed manuscript’s final claim strength.Needs-confirmation claims must be supported, revised, removed, or explicitly resolved before final delivery.
  • Assessment record: Deterministic programs check evidence and manuscript integrity, while the model assesses whether evidence semantically supports a claim in a structured report.The report records the claim, manuscript location, label, supporting evidence, revision action, and rationale.

D Implementation Details · E Generated Paper Examples

Spark-to-Paper implements research-paper generation through state-aware routing, persistent artifacts, and venue-specific templates. Generated-paper examples illustrate complete outputs across introduction, method, experiments, and analysis without serving as an additional quantitative evaluation.

  • D Implementation Details: The implementation details cover input routing, persistent project artifacts, and venue-specific configuration.These components organize how information and requirements flow through the pipeline.
  • D Implementation Details: The pipeline routes inputs by research state and selects Proposal Mode or Data-Aware Mode based on whether measured experimental evidence is available.The selected mode is stored in project state and propagated to downstream stages.
  • D Implementation Details: Each stage materializes explicit artifacts, from the paper blueprint and verified bibliography to section-level LATEX, figures, and the compiled manuscript.Measured quantitative facts are also stored separately when results are available.
  • D Implementation Details: Venue-specific behavior is controlled through template specifications defining structure, constraints, citation style, figure and table requirements, and assembly conventions.Adding a venue requires a new specification and template resources rather than changes to pipeline logic.
  • E Generated Paper Examples: Generated-paper examples present representative sections from complete end-to-end outputs, including introductions, methods, experimental results, and analysis.The examples demonstrate manuscript structure, detail, and visual form rather than providing another quantitative evaluation.
  • E Generated Paper Examples: Figure 7 shows the demo paper’s introduction, while Figure 8 shows its method section.These screenshots are part of the representative generated-paper examples.
  • E Generated Paper Examples: Figures 9 and 10 show the demo paper’s experiment and analysis sections.Together, they continue the visual examples of the generated manuscript output.
Loading 2608.11924v1…