PaperlayerView exact revision on arXiv

Source-linked AI summary

The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, David Ha

arXiv:2504.08066v1cs.AIcs.CLcs.LG

TL;DR

End-to-end autonomous scientific discovery systems remain limited in autonomy and broad applicability. The AI Scientist-v2 removes human-authored code templates and uses agentic tree search with VLM feedback, producing three workshop submissions, one of which earned a 6.33 reviewer score and would have been accepted after meta-review if human-generated.

  • Problem

    Existing end-to-end scientific discovery systems remained constrained by reliance on human-authored code templates, limiting their autonomy and broad applicability.

  • Method

    The AI Scientist-v2 combines template-free experimentation, experiment-manager-guided agentic tree search, parallel execution, and VLM feedback for manuscript refinement.

  • Results

    One of three fully autonomous workshop submissions earned an average reviewer score of 6.33 and would have been accepted after meta-review if human-generated.

  • Takeaways & Limitations

    The system demonstrates an early milestone for autonomous scientific research by successfully navigating peer review at a workshop.

  • Takeaways & Limitations

    The system has not yet consistently reached top-tier conference standards, with only one of three submissions accepted at the workshop level.

Abstract

from arXiv · show

AI is increasingly playing a pivotal role in transforming how scientific discoveries are made. We introduce The AI Scientist-v2, an end-to-end agentic system capable of producing the first entirely AI generated peer-review-accepted workshop paper. This system iteratively formulates scientific hypotheses, designs and executes experiments, analyzes and visualizes data, and autonomously authors scientific manuscripts. Compared to its predecessor (v1, Lu et al., 2024 arXiv:2408.06292), The AI Scientist-v2 eliminates the reliance on human-authored code templates, generalizes effectively across diverse machine learning domains, and leverages a novel progressive agentic tree-search methodology managed by a dedicated experiment manager agent. Additionally, we enhance the AI reviewer component by integrating a Vision-Language Model (VLM) feedback loop for iterative refinement of content and aesthetics of the figures. We evaluated The AI Scientist-v2 by submitting three fully autonomous manuscripts to a peer-reviewed ICLR workshop. Notably, one manuscript achieved high enough scores to exceed the average human acceptance threshold, marking the first instance of a fully AI-generated paper successfully navigating a peer review. This accomplishment highlights the growing capability of AI in conducting all aspects of scientific research. We anticipate that further advancements in autonomous scientific discovery technologies will profoundly impact human knowledge generation, enabling unprecedented scalability in research productivity and significantly accelerating scientific breakthroughs, greatly benefiting society at large. We have open-sourced the code at https://github.com/SakanaAI/AI-Scientist-v2 to foster the future development of this transformative technology. We also discuss the role of AI in science, including AI safety.

1. Introduction

The AI Scientist-v2 is an end-to-end autonomous scientific discovery framework that removes human-coded templates, uses agentic tree search and VLM feedback, and demonstrates an AI-generated manuscript passing peer review at a recognized machine learning workshop.

  • Motivation: End-to-end AI discovery frameworks could transform science by autonomously formulating hypotheses, conducting experiments, analyzing results, and authoring manuscripts.Automated scientific discovery has attracted considerable attention in recent years (Cornelio et al., 2023; Gil et al., 2014; King et al., 2009; Kitano, 2021; Wang et al., 2023; Xu et al., 2021).
  • Contributions: The AI Scientist-v2 combines autonomous code generation, agentic tree search, VLM feedback, and parallel experiment execution to improve autonomy, flexibility, and exploration depth.It removes dependence on human-provided code templates and uses an experiment manager to support systematic exploration of complex hypotheses.
  • Methodological advances: Its VLM-based feedback mechanism refines generated figures, captions, and text interpretation for improved quality, clarity, and alignment.The mechanism operates during reviewing and refinement stages.
  • Workshop paper: The accepted paper found that compositional regularization did not significantly improve compositional generalization on synthetic arithmetic expression datasets.The method penalized large embedding deviations between successive time steps in sequence models.
  • Evaluation: The system enables an AI-generated manuscript to pass peer review at a recognized machine learning workshop, marking a milestone for AI science.The evaluation submitted three entirely AI-generated manuscripts to peer review.

2. Background

This section situates automated scientific discovery against AI Scientist-v1’s human-template dependence and short-sighted experimentation, while motivating agentic scaffolding and tree-search exploration. It highlights Reflexion’s iterative self-improvement and AIDE’s code-generation tree search as relevant precedents.

  • Background: AI Scientist-v1 automated code writing, experiments, visualization, and manuscript production from a human-crafted baseline code template, but that template dependence was a key limitation (Lu et al., 2024).It was introduced as the first system to automate scientific discovery and presentation of results entirely.
  • Language Model Agent Scaffolding: Reflexion (Shinn et al., 2024) improves robustness by having models iteratively reflect on previous responses, though this can increase computational overhead and slow inference.The framework uses critical evaluation of past outputs to encourage self-improvement.
  • Tree Search with Large Language Models: AI Scientist-v1 often produced short-sighted experiments, motivating open-ended hypothesis generation, steppingstone collection, and iterative hypothesis refinement.The human-driven scientific process is presented as a contrasting model for exploration and refinement.
  • Tree Search with Large Language Models: AIDE combines LLM-based code generation with tree search, selecting solution states for debugging or refinement according to scalar evaluation scores such as validation accuracy.AIDE achieved state-of-the-art performance on the MLEBench benchmark, and its exploration strategy inspired integration of tree search into automated scientific discovery.

3. The AI Scientist-v2

The AI Scientist-v2 increases autonomy and generalization by replacing template- and code-conditioned research with open-ended idea generation, staged experimentation, and flexible agentic tree search. Its workflow combines explicit experiment-management stages, best-node selection and replication, structured execution, and VLM-guided visualization refinement.

  • Idea generation: The system begins with higher-level, open-ended research directions, hypotheses, and experimental designs rather than incremental changes constrained by an existing codebase.This generalized idea-generation phase is intended to support more novel or foundational ideas and incorporates literature-review tools such as Semantic Scholar.
  • Experimentation: The AI Scientist-v2 eliminates The AI Scientist-v1’s reliance on predefined human-authored template code and sequential code adaptations, improving experimental flexibility and autonomy.The initial generated concept feeds into experimentation without requiring a fixed starting implementation.
  • Experiment management: An experiment progress manager coordinates four stages: preliminary investigation, hyperparameter tuning, research agenda execution, and ablation studies.Stages use explicit stopping criteria, carry forward the best-performing node selected by an LLM evaluator, record checkpoints, and replicate selected experiments for rigor and reproducibility.
  • Agentic tree search: Unlike The AI Scientist-v1’s linear refinement, The AI Scientist-v2 expands multiple tree nodes in parallel using bug-focused debugging, best-first refinement, and specialized hyperparameter and ablation nodes.This looser structure supports systematic exploration across the empirical research cycle while maintaining coherence.
  • Agentic tree search: Each node generates and executes Python code, stores structured experimental outputs, creates visualizations, and uses Vision-Language Model feedback to refine figure clarity and aesthetics.Nodes retain plans, errors, metrics, scripts, generated figures, LLM feedback, VLM feedback, and final buggy or non-buggy status.

4. Human Evaluation of Manuscripts Generated by The AI Scientist-v2

Human reviewers judged one fully autonomous manuscript sufficiently compelling for workshop acceptance, while detailed assessment identified promising negative-result insights alongside citation, rigor, dataset-overlap, and generalization concerns. The accepted paper’s central finding was that temporal/compositional regularization did not reliably improve arithmetic generalization and could hinder training as task complexity increased.

  • Human Evaluation: Scores of 6, 7, and 6 led reviewers to accept the AI-generated manuscript, placing it among the top 45% of submissions and supporting workshop-level research quality.The manuscript was later withdrawn before publication, while the authors noted that community consensus on integrating AI-generated work into the scientific record has not been reached.
  • Limitations: The system occasionally produced citation inaccuracies and lacked the methodological rigor and depth expected for acceptance at leading conferences, despite executing standard experimental pipelines successfully.Reviewers additionally requested clearer intuition, related-work references, theoretical motivation, or visual explanations for why the regularizer should improve compositionality.
  • Paper Assessment: The paper’s temporal consistency regularizer did not produce the expected compositional-generalization gains, and increasing arithmetic-expression complexity worsened generalization regardless of regularization.The findings suggest that enforcing compositional structure through regularization alone may conflict with the primary learning objective.
  • Paper Assessment: Assessors identified approximately 57% training–test dataset overlap, terminology confusion between embedding and hidden states, and an apparent 100% accuracy result that degraded as task complexity increased.They attributed the reported attention-augmented LSTM performance primarily to task simplicity and judged the paper borderline acceptable for the workshop but insufficiently rigorous for a full conference.
  • Human Workshop Reviewers: Reviewers valued the paper’s clear analysis of unexpected negative results, its proposed compositional regularization method, and its demonstration that operator complexity can bottleneck neural-network performance.Reviewers also recommended broader experiments across tasks and architectures before generalizing the findings.

5. Limitations & Ethical Considerations

The AI Scientist-v2’s workshop acceptance should be interpreted cautiously: only one of three submissions was accepted, and workshop review is less selective than leading conference tracks. Major challenges remain in high-impact scientific reasoning, while responsible deployment requires transparency, disclosure norms, and safeguards against gaming peer review.

  • Scope of the achievement: Only one of three AI-generated submissions was accepted at a workshop, where typical acceptance rates are 60–80%, versus 20–30% at leading machine-learning conferences.The accepted result therefore demonstrates workshop-level success rather than main-conference-level validation.
  • Scientific limitations: The system still struggles to formulate genuinely novel, high-impact hypotheses, design truly innovative experiments, and justify choices with deep domain expertise.Future iterations must address these limitations to move beyond preliminary or incremental scientific work.
  • Evaluation and governance: The authors advocate submitting appropriately authorized samples of AI-generated research to ordinary peer review and studying its quality, while acknowledging that future systems may overcome current limitations.Their submitted papers were withdrawn and, by agreement with ICLR workshop organizers, will not appear on OpenReview’s public forum.
  • Ethical considerations: The scientific community needs norms for AI-generated science covering disclosure and timing, with transparency balanced against concerns that early disclosure could introduce bias.The authors will continue engaging the research community to prevent AI systems from gaming peer review or artificially inflating unscrupulous researchers’ CVs.

6. Related Work

Related work spans end-to-end automated research systems, LLM-based scientific idea generation, scientific-task benchmarks, and industry or conceptual frameworks for AI-driven discovery.

  • Related Work: End-to-end systems such as The AI Scientist-v1 (Lu et al., 2024) and AI-Researcher (Data Intelligence Lab, 2025) automate navigation across the research pipeline.
  • Related Work: Human evaluations find that LLM-generated scientific ideas are typically more novel but less feasible than ideas proposed by human experts, while GraphEval (Feng et al., 2025) evaluates ideas graphically.
  • Related Work: MLEBench and Aide benchmark research-engineering capabilities, while METR’s benchmark shows AI superiority on sub-2-hour tasks (Wijk et al., 2024).
  • Related Work: Industry efforts include Google’s AI Research Copilot, also called AI Co-Scientist (Gottweis et al., 2025), while Bengio et al. (2025) distinguishes agentic AI from Scientist AIs.

7. Conclusion

The AI Scientist-v2 advances autonomous scientific discovery by removing human-crafted templates, using agentic tree search with experiment-manager supervision, and refining visualizations through VLM feedback. Its first fully AI-generated paper to navigate peer review marks progress while underscoring ongoing quality and hypothesis-generation challenges and the potential of openly developed AI science to benefit humanity.

  • The AI Scientist-v2 improves autonomy and exploration by eliminating human-crafted templates, adding experiment-manager-supervised agentic tree search, and integrating VLM feedback loops for visualization refinement.
  • The system produced the first fully AI-generated paper to navigate peer review, marking a milestone despite limitations relative to workshop and conference standards.Consistently achieving top-tier quality and generating truly groundbreaking hypotheses remain significant challenges.
  • Openly developing and scaling systems like The AI Scientist-v2 could help AI-generated discoveries contribute to human flourishing and the betterment of humanity.The stated goal includes applications such as curing diseases and expanding knowledge of the laws governing the universe.

Supplementary Material … where T is the length of the input sequence.

The supplementary material specifies The AI Scientist-v2’s runtime and prompting infrastructure, then presents three AI-generated workshop manuscripts and examines one compositional-regularization study whose results contradicted its intended benefit. The highlighted study found that operator complexity worsened generalization difficulties, illustrating both the system’s research workflow and its limitations.

  • A. Hyperparameters: The supplementary hyperparameter material records language- and vision-language-model settings alongside agentic tree-search and experiment-stage progression limits.These settings are provided in Tables 2 and 3.
  • A. Hyperparameters: Generating one paper usually takes several hours and is capped by a 15-hour runtime limit.Runtime depends on problem complexity.
  • B. Prompts: The prompts require novel, feasible proposals grounded in at least one literature search, followed by reflection that incorporates new tool information and refines the idea.The system is instructed to stay within resources an academic lab could afford and to produce automatically parseable structured outputs.
  • B. Prompts; Supplementary Material: The write-up and figure prompts emphasize truthful results, non-hallucinated data, scientifically meaningful plots, accurate citations, and aggressive review-driven figure selection within workshop page limits.The VLM reviews figure content, captions, references, and presentation before revisions, while supplementary material holds additional details and plots.
  • C. AI Generated Papers: The system generated three complete manuscripts and submitted them to the ICLR 2025 ICBINB workshop, with each accompanied by scientific and code-review annotations.Table 4 summarizes the submissions before the manuscripts are presented in full.
  • 1 INTRODUCTION; 3 METHOD; where T is the length of the input sequence: The compositional-regularization study adds an explicit loss term intended to penalize deviations from expected compositional structures and encourage representations that generalize to unseen combinations.The method was evaluated through extensive experiments and analyzed for the effects of operator complexity.
  • COMPOSITIONAL REGULARIZATION: UNEXPECTED OBSTACLES IN ENHANCING NEURAL NETWORK GENERALIZATION: Operator complexity exacerbated models’ compositional-generalization difficulties regardless of regularization, so the proposed approach did not yield the anticipated benefits.The study identifies this unexpected outcome as evidence that enforcing compositionality through regularization is not straightforwardly effective.
  • 3 METHOD; where T is the length of the input sequence: The method uses an LSTM-based model for arithmetic-expression evaluation, with a mean-squared-error task loss combined with a compositional regularizer weighted by λ.The regularizer penalizes large changes between successive hidden states, and experiments varied λ to assess its effect.

4 EXPERIMENTS … B.1 EXPERIMENTAL SETUP

Experiments on synthetic arithmetic expressions found that compositional regularization did not improve generalization, while greater operator complexity substantially worsened it. Supplementary analyses further showed that larger embeddings were insufficient to improve compositional generalization, and an attention-enhanced LSTM was implemented for comparison.

  • 4 EXPERIMENTS: The experiments used 1,000 training expressions and 200 unseen test expressions, with Adam, mean squared error, 30 epochs, and λ = 0.1 by default.Experiments varied hyperparameters and operator complexity, and evaluated test accuracy within a tolerance alongside compositional loss.
  • 4 EXPERIMENTS: 84% test accuracy was reached by the baseline LSTM, but steady compositional loss indicated that it did not inherently develop compositional representations without regularization.The baseline used synthetic arithmetic-expression data and was evaluated on unseen expressions including novel combinations and higher operator complexity.
  • 4 EXPERIMENTS: Higher λ reduced compositional loss but did not improve test accuracy and sometimes decreased it, revealing a trade-off between compositionality enforcement and the primary learning objective.Figure 2 reports the effect of λ on training loss, compositional loss, and final test accuracy.
  • 4 EXPERIMENTS: As operator complexity increased, training and validation losses rose while final validation accuracy decreased significantly for both baseline and regularized models.The results indicate that compositional regularization alone could not handle expressions with higher operator complexity.
  • 5 CONCLUSION: Compositional regularization failed to improve generalization and sometimes hindered learning, while increasing arithmetic-expression complexity exacerbated models’ generalization difficulties.The conclusion attributes possible failure to conflict with the primary learning objective and possible mismatch between the regularizer’s compositionality measure and task-relevant compositionality.
  • SUPPLEMENTARY MATERIAL: The supplementary discussion recommends alternative regularization, refined definitions of compositionality, more complex datasets, and recursive or hierarchical models for higher operator complexity.It also emphasizes rigorous evaluation and openly reporting negative or inconclusive results.
  • A EFFECT OF EMBEDDING DIMENSION: Larger embedding dimensions did not consistently improve test accuracy, suggesting that the bottleneck was compositional-structure capture rather than representational capacity.Figure 4 showed similar training-loss and compositional-loss trends across embedding dimensions 16, 32, 64, and 128.
  • B.1 EXPERIMENTAL SETUP: The attention comparison modified the baseline LSTM by adding an attention layer after the LSTM outputs, forming a hidden-state-based context vector for final prediction.The setup compared the baseline with an enhanced model incorporating attention, which focuses on relevant input-sequence parts.

B.2 RESULTS … C.3 DROPOUT IMPACT

Across the results and additional experiments, attention, compositional regularization, recurrent-unit choice, and dropout produced limited gains in compositional generalization. The findings suggest that the challenge reflects deeper architectural or representational limitations rather than any single training or regularization choice.

  • B.2 RESULTS: Training dynamics improved only slightly with attention, while final test accuracy remained similar to the baseline.The comparable compositional loss further indicates that attention did not substantially enhance compositional representations.
  • C ADDITIONAL EXPERIMENTS: The experiments collectively indicate that compositional generalization difficulties are not solely attributable to attention, compositional weighting, recurrent-unit type, or dropout.The results instead point toward deeper architectural limitations or a need for more sophisticated mechanisms to capture compositionality.
  • B.2 RESULTS: Attention achieved test accuracy similar to the baseline, with comparable training and compositional losses, so it did not significantly improve generalization.The results suggest that focusing on relevant input parts is not the primary limitation in compositional generalization.
  • C.1 ABLATION STUDY ON COMPOSITIONAL WEIGHT: Higher compositional weight λ reduced compositional loss but increased training loss and did not improve, and may decrease, final test accuracy.This indicates interference with the primary learning objective and a trade-off between compositional regularization and generalization.
  • C.2 COMPARISON OF LSTM AND RNN ARCHITECTURES: LSTM produced marginally higher final test accuracy than a simple RNN, but both architectures struggled with compositional generalization.Their similar convergence patterns indicate that recurrent-unit choice does not resolve the underlying challenge.
  • C.3 DROPOUT IMPACT: Increasing dropout did not significantly improve generalization, suggesting that standard regularization is insufficient for learning compositional structures.The experiment evaluated final test accuracy across different dropout rates.

D HYPERPARAMETERS AND TRAINING DETAILS … AUTONOMOUSLY GENERATED BY THE AI SCIENTIST-V2

The reviewed work examines compositional generalization and label-noise effects through controlled experiments, while documenting implementation details, dataset limitations, and reviewer assessments. The materials report limited benefits from the proposed regularizer and calibration mitigations, alongside reproducibility settings and methodological concerns.

  • D HYPERPARAMETERS AND TRAINING DETAILS: Higher dropout rates did not enhance compositional generalization, indicating limited effectiveness of dropout in this context.Tested dropout values were 0.0, 0.2, and 0.5, although the supplied figure passage notes that only the best configuration is shown.
  • Weaknesses; Additional Comments; Scores: The paper was judged technically solid but limited by evaluation scope, imprecise regularizer description, missing references, and uncertainty about whether the regularizer induces compositional representations.Review scores included Soundness 3/5, Presentation 2/5, Contribution 3/5, Workshop 5/10, and Conference 4/10.
  • Inspecting the dataset generation process: The dataset generator permits substantial train-test overlap: a sanity check found about 57% average test-set overlap using 1,000 training and 200 test samples.The generator creates at most 81 ∗ k combinations, where k is the number of operators, making overlap dependent on sample counts and operator choices.
  • Model architecture, loss function, and evaluation function: The model uses embedding states for compositional regularization, regression to match ground-truth digits, and an architecture comprising an embedding layer, single LSTM layer, and linear head.The code review recommends denoting embeddings as e_t rather than h_t because “hidden states” could be confused with LSTM hidden states.
  • Attention-augmented LSTM: 100% test accuracy for the attention-augmented LSTM with numbers [1-9] fell to 56% with [10-19], while the baseline LSTM result was also evaluated under both settings.The supplied passage reports the first two attention-augmented LSTM values but truncates the baseline result.
  • C.2. Unveiling the Impact of Label Noise on Model Calibration in Deep Learning; 1 INTRODUCTION; 3 METHODOLOGY: The label-noise study investigates symmetric and asymmetric noise across levels and benchmark datasets using ECE and reliability diagrams, finding that noise worsens calibration and mitigation techniques offer limited improvements.The stated contribution emphasizes overconfident, miscalibrated predictions, with asymmetric noise having a more detrimental effect.
  • C.2.1. The AI Scientist-v2 Idea; 2 RELATED WORK; AUTONOMOUSLY GENERATED BY THE AI SCIENTIST-V2: The proposed label-noise experiments use CIFAR-10, MNIST, and Fashion-MNIST with ResNet-18, synthetic noise rates from 10% to 50%, and post-hoc temperature scaling.The review flags citation and benchmark-justification problems, including that some claims about prior use of the datasets are not fully accurate.

4 EXPERIMENTS AND RESULTS … A ADDITIONAL EXPERIMENTS AND FIGURES

Experiments show that label noise reduces accuracy and worsens calibration across datasets, while standard mitigation techniques provide limited improvements. The study concludes that robust calibration methods remain necessary for noisy-label settings and dependable uncertainty estimates.

  • 4 EXPERIMENTS AND RESULTS: The experiments use a Cosine Annealing Schedule rather than stepwise decay, a condition relevant to interpreting the reported results.
  • 4 EXPERIMENTS AND RESULTS: Across CIFAR-10, MNIST, and Fashion-MNIST, label noise consistently reduces accuracy; MNIST remains above 90% at higher noise levels but develops increased ECE under asymmetric noise.The cross-dataset analysis was intended to assess generalization, and Figure 2 summarizes accuracy and ECE over training epochs.
  • We evaluated whether standard label noise mitigation techniques improve calibration. Specifically,: Basic CNN models can match ResNet-18 in accuracy but exhibit worse calibration, while the supplied Figure 3 contains no ECE results to support that comparison.
  • We evaluated whether standard label noise mitigation techniques improve calibration. Specifically,: Temperature scaling provided limited improvements, reducing ECE marginally without compensating for label-noise degradation; the cited figure does not include temperature-scaling results.
  • 5 DISCUSSION: The discussion attributes overconfident predictions and pronounced calibration damage to asymmetric noise, but notes that robust loss functions and temperature scaling lack experimental support here.
  • 6 CONCLUSION: The conclusion states that label noise exacerbates miscalibration and that existing mitigation techniques offer limited improvements, motivating novel methods for noisy-label calibration.
  • 6 CONCLUSION: Future work should explore calibration-aware training objectives and robust calibration methods tailored to noisy environments for dependable uncertainty estimates in real-world applications.

A.1 NOISE RATE SENSITIVITY ANALYSIS … 4 EXPERIMENTAL SETUP

The paper reports that label noise steadily reduces accuracy, worsens calibration, and affects asymmetric-noise settings more strongly, while broader experiments examine generalization and deployment challenges. Reviews and code inspection identify missing or unused experiments, duplicated or absent figures, unclear evidence, and reproducibility and presentation weaknesses.

  • A.1 NOISE RATE SENSITIVITY ANALYSIS; C.2.2. AI Scientist Team Review; Strengths: Asymmetric label noise causes higher calibration error than symmetric noise, with accuracy declining steadily as noise increases for both noise types.The analysis used CIFAR-10 and ResNet-18; the decline was more pronounced under asymmetric noise.
  • A.2 CALIBRATION CURVES AND RELIABILITY DIAGRAMS; Weaknesses: ECE increases during training, especially at higher noise rates, while reliability diagrams reportedly indicate increasingly overconfident predictions but were not shown.The paper’s calibration evidence was criticized because reliability diagrams and some claimed calibration-method results were missing.
  • A.3 HYPERPARAMETERS; 0.1 at epochs 50 and 75 Number of Epochs 100 Batch Size 128 Weight Decay 5e-4: Experiments used SGD with Momentum 0.9, initial learning rate 0.1, batch size 128, and cosine annealing, but reported epoch counts and weight-decay usage were inconsistent with the stated table.The passages note 20 or 30 rather than 100 epochs and preliminary-only weight decay, while another figure context lists 100 epochs and weight decay 5e-4.
  • A.4 ADDITIONAL DATASETS; noise, complementing the findings in Section 4: Additional SVHN experiments reportedly reproduced the finding that label noise adversely affects calibration metrics, although the experiment lacked figures.The supplementary material also contained duplicate figures, including the noise-level model comparison.
  • C.2.2. AI Scientist Team Review; Strengths; Scores; Additional Comments: Reviewers considered the question important and the noise-model effect robust, but rated the submission a rejection because displayed results did not sufficiently support its conclusions.Concerns included unclear mechanisms, missing calibration-method results, unreadable or duplicated figures, missing citations, and inadequate reproducibility.
  • C.2.3. AI Scientist Team Code Review; Temperature scaling; Dataset class; Evaluation function: Code review found that temperature scaling was implemented but never used, the dataset class initially omitted symmetric/asymmetric noise support, and the final asymmetric mapping used (i+1) % NUM CLASSES.The ECE implementation was validated against torchmetrics’ MulticlassCalibrationError with norm=‘l1’, matching apart from minor numerical differences.
  • C.2.4. Workshop Reviews: Workshop reviewers agreed that label noise degrades calibration but found the analysis unclear, insufficiently connected to its claims, narrow in metrics, and incomplete regarding temperature scaling.Both reviewers assigned rating 3 and no award; one described the paper as not review-ready because of missing references and unreadable figures.
  • C.3. Real-world Challenges in Pest Detection using Deep Learning: an Investigation into Failures and Solutions; C.3.1. The AI Scientist-v2 Idea; REAL-WORLD CHALLENGES IN PEST DETECTION USING DEEP LEARNING: AN INVESTIGATION INTO FAILURES AND SOLUTIONS; 1 INTRODUCTION; 2 RELATED WORK; 3 METHODOLOGY; 4 EXPERIMENTAL SETUP: The pest-detection study investigates data quality, environmental variability, and model limitations using ResNet-18, learning-rate experiments, augmentation, and multi-dataset training, finding that lower learning rates can improve generalization while robustness remains limited.The Crop Pest and Disease dataset contains 22 classes; experiments also used EuroSAT, MedMNIST, and CIFAR-10, but manual dataset downloading remained necessary.

5 EXPERIMENTS AND RESULTS … SUPPLEMENTARY MATERIAL

Experiments show that lower learning rates improve convergence and stabilize environmental robustness, while performance varies substantially across datasets, especially under domain shifts. The discussion and conclusion emphasize that hyperparameter tuning alone is insufficient for robust real-world pest detection and that specialized, domain-aware strategies are needed.

  • 5 EXPERIMENTS AND RESULTS: Lower learning rates (1e−4 and 5e−4) produce smoother convergence, steadily decreasing training loss, and more stable Environmental Robustness Scores (ERS), indicating better generalization and robustness.These results are reported for ResNet-18 trained on the Crop Pest and Disease dataset.
  • 5 EXPERIMENTS AND RESULTS: EuroSAT and CIFAR-10 achieve high, stable training and validation accuracy, whereas MedMNIST shows erratic accuracy linked to domain discrepancies and weaker generalization.The comparison concerns ResNet-18 models fine-tuned separately on the three datasets, rather than a single model trained jointly across them.
  • 5 EXPERIMENTS AND RESULTS: EuroSAT and CIFAR-10 maintain higher and more stable ERS across epochs, while MedMNIST has low and unstable ERS, indicating greater sensitivity to environmental changes.The ERS analysis evaluates robustness to environmental augmentations applied during testing.
  • 6 DISCUSSION: Hyperparameter optimization improves generalization and robustness to some extent, but models remain vulnerable to environmental variability, so learning-rate tuning alone cannot ensure robust real-world performance.This limitation is particularly relevant to deployment for pest detection in agricultural settings.
  • 6 DISCUSSION: Multi-dataset experimentation reveals that dataset characteristics and domain shifts substantially affect learning, overfitting, and robustness, with MedMNIST performing poorly under these differences.The discussion cautions that comparisons across separately trained models should not be interpreted as naive joint multi-dataset training.
  • 6 DISCUSSION: The discussion recommends realistic data augmentation, robust training methods, and domain-specific model adaptations to address data quality issues and environmental variability.These strategies are proposed as necessary directions for improving practical model performance.
  • 7 CONCLUSION: The conclusion identifies environmental variability, domain discrepancies, and dataset compatibility as continuing deployment challenges, while noting that multi-dataset training can add complexity and inconsistent benefits.Future work should therefore develop advanced techniques tailored to the compatibility and characteristics of the data involved.

A ADDITIONAL FIGURES AND DETAILED RESULTS … Environmental noise implementation

Supplementary results show stable training on EuroSAT and CIFAR-10 but erratic MedMNIST behavior, while implementation details and reviews identify methodological, presentation, and domain-transfer limitations. Code review confirms the environmental-noise augmentations and Environmental Robustness Score calculation but reports that the intended domain-adaptation implementation was not ultimately used.

  • applied.: The experiments use PyTorch 1.9.0, ImageNet-pretrained ResNet-18, Adam with β1 = 0.9 and β2 = 0.999, and multi-dataset training for 30 epochs with batch size 64 and early stopping.The supplied implementation passage also reports weight decay of 1e−4, while its comment says the value should be 0.01.
  • A ADDITIONAL FIGURES AND DETAILED RESULTS: Training loss decreases steadily for EuroSAT and CIFAR-10, whereas MedMNIST shows erratic loss and accuracy, indicating instability with the ImageNet-pretrained model.Figure 4 compares training loss across datasets, and Figure 5 reports erratic MedMNIST accuracy and loss behavior.
  • C.3.2. AI Scientist Team Review: The paper targets pest prediction with deep learning and studies environmental robustness, learning-rate effects, and out-of-distribution testing across non-pest vision datasets.The review describes the topic fit and interest in differential augmentation effects across datasets.
  • Weaknesses: Reviewers criticize the weak connection between EuroSAT, MedMNIST, and CIFAR-10 results and pest prediction, arguing that the experiments do not substantiate the paper’s broader conclusions.They also characterize the evaluation as a simple setup and question the claimed domain-adaptation framing.
  • Scores: The review assigns scores of 2 for soundness, 1 for presentation, 1 for contribution, and 3/10 overall, recommending workshop rejection.The criticisms include technical flaws, inadequate evidence, wrong descriptions, duplicated figures, and weak reproducibility.
  • Additional Comments: Additional comments call for major presentation improvements, missing citations, and correction of misleading descriptions of learning-rate effects and multi-model-single-dataset terminology.The code review further states that the implemented domain-adaptation attempts failed and that the selected final implementation omitted domain adaptation and proper multi-dataset training.
  • Domain Adaptation and Multi-dataset training: Code review found that a domain-discriminator implementation correctly combined single-model training across all three datasets but failed, so the final system used an error-free implementation without that domain-adaptation technique.The reviewed domain-adaptation code is shown in Figure 11.
  • Environmental noise implementation: Testing-time ColorJitter, GaussianBlur, and RandomAffine augmentations simulate environmental changes, and ERS compares augmented-test performance with standard-condition performance.The code review confirms both the augmentation implementation and the stated ERS ratio definition.

C.3.4. Workshop Reviews

Workshop reviewers agreed that the paper addresses important robustness challenges in real-world pest detection and explores hyperparameter tuning and multi-dataset training, but they differed sharply on its acceptance. The main criticisms concerned weak experimental motivation, insufficient methodological detail and evidence, unclear theoretical grounding, and presentation problems.

  • Reviewer consensus: Reviewers reported that the paper studies ResNet-18 pest detection under simulated environmental changes, evaluating learning-rate tuning, augmentation, and multi-dataset training for generalization and robustness.The reported metrics include accuracy, loss, and Environmental Robustness Score (ERS).
  • Reviewer consensus: The reported conclusion was that training on EuroSAT and CIFAR-10 improved generalization, although reviewers noted persistent environmental variability and domain-discrepancy challenges.One reviewer described the analysis of multi-dataset training and hyperparameter tuning as novel and significant.
  • Methodological criticisms: Reviewers criticized unclear and weakly motivated experiments, including the links between augmentations and real-world variability, the choice of ERS, and the learning-rate generalization claim.One review noted that only five learning rates and ten training epochs supported the learning-rate conclusion.
  • Revision requests: One reviewer concluded that insufficient experimental detail weakened the evidence and recommended rejection, while another called for substantial revision and real-world deployment to strengthen the conclusions.The reviews emphasized that the experiments were not conducted in an actual real-world deployment setting.
  • Revision requests: Reviewers requested stronger theoretical grounding, fuller dataset and methodological details, comparisons with recent deep learning models, clearer evaluation presentation, and more readable figures.They also identified missing references, language issues, and BibTeX errors.
Loading 2504.08066v1…