Source-linked AI summary
ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration
Ruofeng Yang, Yongcan Li, Shuai Li
TL;DR
Long-horizon autonomous research can produce plausible claims that outrun their evidence. ARIS addresses this with cross-model execution, review, and assurance workflows, presenting an open-source harness with early observational deployment experience.
Problem
Long-horizon autonomous research lacks reliable assurance against plausible successes whose claims outrun the evidence or inherit an executor’s framing.
Method
ARIS separates execution from evidence checking through cross-family executor–reviewer collaboration and a three-layer architecture with explicit assurance workflows.
Results
ARIS provides more than 65 reusable skills and an assurance stack spanning evidence checks, scientific editing, mathematical proofs, and visual PDF inspection.
Takeaways & Limitations
The report presents cross-model review and layered assurance as ARIS’s practical response to self-review blind spots in long-horizon research.
Takeaways & Limitations
All reported deployment outcomes are observational and cannot be causally attributed to ARIS alone.
Abstract
from arXiv · showhide
This report describes ARIS (Auto-Research-in-sleep), an open-source research harness for autonomous research, including its architecture, assurance mechanisms, and early deployment experience. The performance of agent systems built on LLMs depends on both the model weights and the harness around them, which governs what information to store, retrieve, and present to the model. For long-horizon research workflows, the central failure mode is not a visible breakdown but a plausible unsupported success: a long-running agent can produce claims whose evidential support is incomplete, misreported, or silently inherited from the executor's framing. Therefore, we present ARIS as a research harness that coordinates machine-learning research workflows through cross-model adversarial collaboration as a default configuration: an executor model drives forward progress while a reviewer from a different model family is recommended to critique intermediate artifacts and request revisions. ARIS has three architectural layers. The execution layer provides more than 65 reusable Markdown-defined skills, model integrations via MCP, a persistent research wiki for iterative reuse of prior findings, and deterministic figure generation. The orchestration layer coordinates five end-to-end workflows with adjustable effort settings and configurable routing to reviewer models. The assurance layer includes a three-stage process for checking whether experimental claims are supported by evidence: integrity verification, result-to-claim mapping, and claim auditing that cross-checks manuscript statements against the claim ledger and raw evidence, as well as a five-pass scientific-editing pipeline, mathematical-proof checks, and visual inspection of the rendered PDF. A prototype self-improvement loop records research traces and proposes harness improvements that are adopted only after reviewer approval.
1 Introduction
ARIS frames autonomous research as a harness-engineering problem whose central risk is plausible unsupported success, motivating modular workflows with independent cross-family assurance. It contributes an assurance stack and three-layer architecture that separate artifact production from evidence and manuscript checking.
- Motivation: Autonomous research can produce plausible unsupported success when claims outrun evidence or readers inherit the executor’s framing.The introduction identifies misreported results, unsupported claims, and silently inherited framing as central risks beyond outright failure.
- Design assumption: ARIS assumes long-term single-agent tasks are unreliable and therefore divides workflows into sub-workflows reviewed independently by cross-family models.The design favors strictness because adversarial review can expose weaknesses that same-model self-review may miss.
- Design requirements: The design requires persistent research state, modular execution, and independent assurance to preserve artifacts, divide trajectories, and examine outputs from sufficiently different perspectives.These three bottlenecks connect evidence and claims across stages while preventing the reviewer from merely continuing the executor’s reasoning.
- Contribution: ARIS treats assurance as a first-class workflow layer, pairing reusable Markdown-defined skills with cross-family executor and reviewer models at experimental and manuscript stages.This separates artifact production from evidence checking, claim mapping, and manuscript review.
- Contribution: Its assurance stack checks evidence support through integrity verification, result-to-claim mapping, and claim auditing, alongside five-pass editing, proof checks, and visual PDF inspection.The architecture is organized into execution, orchestration, and assurance layers.
- Limitation: Human-in-the-loop review can improve final-paper generation quality and help users develop knowledge and research taste.ARIS acknowledges this benefit despite its focus on autonomous research.
2 System Overview
ARIS is a research harness that coordinates five end-to-end workflows through artifact contracts and a three-layer architecture. Its design emphasizes heterogeneous review, modular and composable skills, persistent project memory, and critique-to-action assurance loops.
- Workflow library: Five workflows—idea discovery, experiment bridge, auto-review, paper writing, and rebuttal—span Discovery, Experimentation, Manuscript, and Post-Submission phases through plain-text artifact contracts.Figure 1 also depicts reviewer feedback, GPU-triggered evidence collection, and wiki memory across the workflow library.
- Architecture: Three layers address persistent state, modular execution, and independent assurance through a research wiki, Markdown skills, workflow orchestration, and cross-family executor/reviewer pairing.The system topology connects artifacts, workflows, skills, MCP and tool bridges, assurance, and meta-optimization.
- Design principles: Heterogeneous models are preferred over single-model self-refinement, with Claude-family and GPT-family executor/reviewer pairings recommended by default.Gemini, MiniMax, GLM, Kimi, and DeepSeek can also be configured through dedicated or generic model bridges.
- Design principles: Research capabilities are defined as portable Markdown skill files that can be independently developed, chained into workflows, parameterized per invocation, and recovered through checkpoints across sessions.The same SKILL.md files can be used in Claude Code, Codex CLI, and Cursor with no file-level changes, while project wikis preserve papers, ideas, experiments, and claims.
- Assurance loop: The critique-to-action loop has an executor produce an artifact, a different-family reviewer score it and return action items, and a convergence check accept it or request another round.The default termination condition is a review score above 6/10 with all critical items resolved, or a preset maximum number of rounds; failed experiments receive up to three remediation attempts.
3 Cross-Model Assurance Stack
ARIS adds a strict cross-model assurance stack to counter plausible unsupported success and adversarially misleading executor outputs. The stack combines a three-stage evidence-to-claim audit, manuscript assurance mechanisms, and system-wide controls for audit depth and reviewer routing.
- Stack overview: The assurance stack addresses executor deception and unsupported outputs through evidence-integrity checks, manuscript assurance, effort levels, and reviewer routing.Community debugging identified model-derived references, self-normalized metrics, phantom results, and unsupported claims as failure modes.
- Manuscript assurance: The manuscript assurance layer applies five editing passes, proof verification, visual PDF review, and citation audits covering existence, metadata, and contextual support.Proof checking uses a 20-category taxonomy and severity axes, while visual review compares source content with compiled-PDF presentation.
- System-wide controls: Four effort presets scale search, review, and experimentation while preserving core review invariants, with balanced (1×) as the default and beast (≈5–8×) as the highest setting.Lite is ≈0.4× and max is ≈2.5×; Codex-based reviewer calls retain xhigh reasoning effort across presets.
- System-wide controls: Reviewer requests currently route through GPT-5.4 via Codex MCP, with supported high-stakes reviews optionally routed to GPT-5.4 Pro through Oracle MCP.Alternative reviewer backends can connect through the llm-chat bridge, subject to reviewer-independence constraints.
4 Implementation: Skills, Workflows, and Tools
ARIS implements long-running research through reusable Markdown-defined skills, persistent wiki state, chained workflows, and supporting model and tooling integrations. A prototype meta-optimization loop analyzes usage traces and proposes harness changes subject to reviewer and user approval.
- Skills: More than 65 research-oriented skills are encoded as SKILL.md files with metadata, workflows, quality gates, and failure-handling instructions.Five shared reference documents codify cross-cutting invariants without duplicating rules across skill files.
- Skills: Versionable text files and structured Markdown artifact contracts exchange intermediate outputs between skills, improving auditability, checkpoint recovery, and portability across model backends.Examples include IDEA_REPORT.md, EXPERIMENT_LOG.md, and NARRATIVE_REPORT.md passed between named workflows.
- Research wiki: The research wiki stores papers, ideas, experiments, and claims as canonical Markdown pages linked by eight typed relationships, providing persistent cross-session memory.The /idea-creator skill reads an 8,000-character-capped query_pack.md and retains rejected ideas to avoid repeating dead ends.
- Workflows: Five workflows chain reusable skills into end-to-end pipelines, including an auto-review loop that uses a different-family reviewer, runs permitted follow-up experiments, revises drafts, and stops after four rounds or a configurable score threshold.The paper-writing pipeline chains seven core sub-skills and can invoke proof checking for theory-heavy papers.
- Meta-optimization: The prototype meta-optimization loop logs structured usage events, analyzes overrides, failures, and review plateaus, and proposes targeted skill patches that require GPT-5.4 xhigh review of at least 7/10 before user consideration.ARIS never automatically applies harness changes; the user makes the final decision.
5 Deployment Evidence and Limitations
ARIS’s deployment footprint expanded substantially, and one overnight run operationalized review-driven claim pruning and revision. However, outcomes are observational, and the harness provides advisory safeguards rather than guarantees of correctness, scientific soundness, confidentiality, or unbiased improvement.
- Evidence limits: All reported outcomes are observational, and the run does not establish that cross-family review is superior, that two reviewers are optimal, or that review effects are isolated from expertise, model choice, and task difficulty.The bandit and game-theoretic framing motivates the two-role pattern as a design analogy rather than causal evidence.
- Deployment footprint: Over approximately eight hours, one overnight run completed four review–revise rounds, raised the internal reviewer score from 5.0 to 7.5/10, launched more than 20 GPU experiments, and removed unsupported claims.This was a single trajectory on one paper and is not generalized.
- Assurance limits: ARIS cannot guarantee correctness, novelty, or scientific soundness; cross-model review and citation grounding reduce some failures but do not eliminate hallucinations, methodological gaps, or fabricated references.The three-stage audit cascade catches common integrity failures but remains an advisory safety net, not a formal verification system.
- Human oversight: Review iterations can amplify reviewer-model biases and degrade paper quality beyond diminishing returns, so humans must direct research, validate evidence, and make final submission decisions.Configurable checkpoints can require human approval at each workflow step.
- Security and disclosure: Repository-level review may send source code to external LLM APIs, so sensitive repositories should avoid it without an approved local-only path, which is planned but not yet implemented.ARIS assisted this report’s drafting and review, while the authors manually reviewed, edited, and accepted responsibility for its final content.
6 Related Work
ARIS is positioned among autonomous research, self-refinement, automated-review, and agent-framework efforts, combining their ideas in a research-focused harness. It uses external-model review for iterative development while retaining human peer review as the standard for complete evaluation.
- Autonomous research systems: Prior autonomous research systems span end-to-end idea-to-paper automation, hypothesis generation, human-in-the-loop checkpoints, and annotated-data-to-paper workflows.The cited systems include The AI Scientist, AI Scientist-v2, AI co-scientist, Agent Laboratory, and data-to-paper.
- Self-refinement and multi-agent debate: Self-Refine and Reflexion demonstrate iterative self-feedback, while multi-agent debate explores reasoning gains and the complications of heterogeneous-LLM judging.ARIS draws on these ideas by embedding them in its research workflow.
- Automated reviewing: LLMs can assist targeted review and overlap with human reviewers on some dimensions, but they remain unsuitable as complete substitutes for expert peer review.ARIS therefore uses external-model review as a development tool for iterative writing improvement, not as a replacement for human peer review.
- Harness engineering and agent frameworks: ARIS is a hand-engineered research harness with a prototype outer loop, whereas Meta-Harness formalizes outer-loop search over harness code and other cited systems provide general-purpose agent or software-engineering frameworks.The passage contrasts ARIS’s research focus with frameworks including AutoGen, CAMEL, OpenHands, SWE-agent, MetaGPT, and ChatDev.
7 Conclusion
ARIS frames long-horizon single-agent research as vulnerable to plausible unsupported success and responds by structuring workflows around persistent state, modular execution, and independent assurance. The report acknowledges limited evaluation, outlines future validation and deployment work, and proposes extending its accountability mechanisms beyond manuscripts.
- Conclusion: ARIS addresses plausible unsupported success by decomposing research around persistent research state, modular execution, and independent assurance.The report identifies claims outrunning evidence and readers inheriting the executor’s framing as the central failure mode.
- Conclusion: The main limitations are the absence of controlled evaluation and reliance on observational deployment evidence.
- Conclusion: Future work includes compute-matched comparisons of cross-model heterogeneity, local reviewer models for confidential settings, and researcher-productivity user studies.
- Conclusion: ARIS’s reviewer independence, evidence-to-claim audits, and provenance-aware claim ledgers could oversee model outputs before downstream training-data retention or reward signals.This is presented as a speculative adjacent direction that would complement existing self-improvement approaches with an explicit oversight layer.
A Workflow Internals
ARIS structures research through staged workflows that discover and refine ideas, bridge reviewed implementations to full-scale experiments, and iteratively improve manuscripts through reviewer feedback. These workflows combine cross-model generation, verification, execution checks, and repeated revision.
- Workflow 1: Idea Discovery: Workflow 1 surveys literature, generates ideas through cross-model collaboration, verifies novelty, and iteratively refines the leading proposal through GPT-5.4 review.The workflow focuses on idea discovery and proposal refinement.
- Workflow 1.5: Experiment Bridge: Workflow 1.5 implements scripts, reviews code correctness, sanity-checks execution on one GPU, and then deploys the experiment to the full backend.The experiment bridge connects implementation review with staged execution before broader deployment.
- Workflow 2: Auto Review Loop: Workflow 2 has the reviewer score the manuscript while the executor applies fixes and runs requested experiments, repeating the cycle.This forms an iterative manuscript-review loop involving critique, correction, and additional experimentation.
B Skill Inventory
The current release documents its core framework skills in Table 5.
- B Skill Inventory: Table 5 lists the framework’s core skills.The passage identifies Table 5 as the location of the current release’s core framework skill inventory.
- B Skill Inventory: The inventory covers skills included in the current release.The passage describes the listed skills as belonging to the current release.
- B Skill Inventory: The listed skills are presented as core framework capabilities.The passage characterizes the skills as core to the framework.
C Reviewer Configuration
ARIS configures reviewer behavior along two orthogonal axes: access scope and context policy. These settings comprise three access-scope options and two context-policy options.
- C Reviewer Configuration: Reviewer behavior is configured along two orthogonal axes: access scope and context policy, with three and two settings, respectively.Table 6 lists the access-scope settings, while Table 7 lists the context-policy settings.
D ARIS-Code Details
Aris-Code is a standalone Rust CLI that packages research skills, model access, review, prioritization, and cost tracking into an interactive workflow. Its paper-writing and rebuttal workflows chain specialized stages with review and safety checks.
- Implementation: Aris-Code is a standalone Rust CLI with an interactive REPL, setup wizard, slash-command skills, five LLM providers, native LlmReview, tiered skill priorities, and token-cost tracking.Skill priority follows user > Claude Code > bundled.
- Paper Writing Pipeline: The paper-writing pipeline chains seven core sub-skills from outlining through figure generation, LATEX drafting, claim auditing, compilation, and review.Optional proof checking can be added to the pipeline.
- Rebuttal: The rebuttal workflow has seven phases for parsing reviews and stress-testing, separated by three safety gates.
E Controlled Benchmark Protocol (Future Work)
The proposed future benchmark would evaluate research-review configurations on a compute-matched pool of 12+ preprint-based paper drafts. It would compare five critique conditions using blinded ratings of review quality, efficiency, and reliability.
- 12+ paper drafts from publicly available preprints would form the controlled task pool.
- Five compute-matched conditions would compare single-model self-critique, same-model and cross-model two-agent setups, reversed cross-model routing, and same-model use for the second model.
- Evaluation would measure issue recall, false-positive rate, actionability, downstream revision quality, cost, and latency, with three blinded independent raters and Krippendorff’s α for inter-rater agreement.