Source-linked AI summary
Insights from Autoresearch for Solar Panel Segmentation
Justinas Lekavicius, Kursat Komurcu, Valentas Gruzauskas, Linas Petkevicius
TL;DR
EO segmentation training configurations are sensitive to data and hardware, making reported settings difficult to transfer. This paper tests AutoResearch on PV segmentation with a fixed architecture and evaluator, retaining training-program edits only when validation IoU improves. Three campaigns improve their one-hour baselines, and the Qwen3-8B configuration matches the reference test IoU without synthetic images.
Problem
EO segmentation training choices interact with class imbalance, ground sampling distance, and GPU memory, while reported configurations may not transfer to later hardware and manual search is time-consuming.
Source: §1 · p. 1Method
Results
Three campaigns improve their one-hour baselines, and the Qwen3-8B configuration matches the reference test IoU without synthetic images.
Source: §4 · p. 4Takeaways & Limitations
The protocol is effective under a declared budget when the evaluator stays frozen, but retained modifications should not be treated as universal defaults.
Source: §4 · p. 4After the first one or two retained changes, later accepted gains are around 10^-3 IoU, within single-run variation that the protocol does not estimate.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
This paper investigates AutoResearch, a protocol in which a coding language model edits a training program under a one-hour GPU budget and retains a change only if validation IoU improves. The protocol is applied to photovoltaic panel segmentation on a frozen real-image split, with DeepLabV3--ResNet-50 held fixed. Three campaigns of 24 experiments, using Gemma~4 12B, Qwen3-8B all improve their one-hour baselines, but retained modifications do not transfer across hardware. The Qwen3-8B configuration, trained on real images only, reaches a test IoU of 0.836 versus 0.833 for the reference GAN-augmented schedule. Research repository https://github.com/VU-AIML/automl4eo-autoresearch-segmentation.
1 Introduction
EO segmentation configurations can be difficult to transfer because training choices interact with data and hardware constraints. The paper adapts AutoResearch to PV segmentation and evaluates it with a frozen architecture and real-image split.
1 Introduction
- AutoResearch lets a coding agent edit the training program, retaining an edit only when validation IoU improves under a fixed budget and frozen evaluator.Unlike a pre-declared search space, proposed edits can be free-form.Source: §1 · p. 1
- The testbed fixes DeepLabV3–ResNet-50 and a 2048/256/256 real-image split, with synthetic images withheld from the one-hour search.The reference study improved test IoU from 0.801 without augmentation to 0.833 using 1228 pix2pix images and up to 100 epochs.Source: §1 · p. 1
- The study reports three independent 24-experiment campaigns across two GPU classes and evaluates the best Qwen3-8B configuration on the held-out test split.That test result is placed alongside the reference Table 5 values of Lekavičius and Gružauskas (2024).Source: §1 · pp. 1–2
2 Method
The method keeps the architecture, data split, and evaluator fixed while allowing edits to the training program. A proposed edit is retained only when it improves validation IoU within the one-hour run budget.
2 Method
- The preparation program owns the real split, preprocessing, and 256-image validation evaluator, while the agent cannot train on validation or test images or edit the evaluator.Mean per-image IoU is the acceptance metric; F1 is logged but does not determine acceptance.Source: §2 · p. 2
- The agent may change training-program components, but DeepLabV3–ResNet-50 remains fixed.Editable components include loss, optimizer, schedule, auxiliary head, gradient clipping, and train-time augmentation.Source: §2 · p. 2
- An edit is committed only if validation IoU after 3600 seconds exceeds that of the current retained program; otherwise, the program is reset, and failed runs count as rejections.Each campaign allows at most 24 logged experiments, including its baseline.Source: §2 · p. 2
3 Experimental study
Three one-hour campaigns search training changes on a frozen real-image split, with the architecture held fixed. The real-only Loop B configuration improves test IoU and F1 over both reference schedules, while retained modifications vary across hardware and later gains approach unmeasured single-run variation.
3.1 Retained modifications
- In Loop A, cosine decay with 5% warmup raises validation IoU from 0.616 to 0.708, and reducing Adam weight decay from 10−4 to 10−5 yields 0.710.Dice, AdamW, auxiliary BCE, gradient clipping, augmentation, polynomial decay, and encoder freezing are rejected; clipping at 1.0 drops IoU to 0.349.Source: §3.1 · p. 2
- Loop B’s largest retained gain is +0.103 from combining soft Dice with BCE, while Loop C also begins with Dice (+0.076) but follows a different sequence.Loop C ends with a Dice-heavy, color-jitter configuration at batch size 16 and validation IoU 0.858; unlike Loop B, it does not retain AdamW, polynomial decay, an auxiliary head, or gradient clipping.Source: §3.1 · pp. 2–3
3 Experimental study
- Three campaigns use the same real-image split, seed, architecture, and one-hour budget; only Loop B is evaluated on the held-out test set.Loop A runs with a 12 GB memory constraint, while Loops B and C use a cached loader on GX10-class hardware.Source: §3 · p. 2, Table 1 · p. 3
3.1 Retained modifications
- Retained changes do not transfer reliably across campaigns: Dice is a large repeated GX10 retention but decreases IoU on the 4070 baseline, while cosine decay is rejected on GX10.The two GX10 campaigns reach validation IoU 0.858 with different retained sequences, consistent with a one-hour plateau rather than a unique default.Source: §3.1 · p. 3
- After the first one or two retained changes, later accepted deltas are about 10−3 IoU, within single-run variation that the protocol does not estimate.Each proposed change uses a single random seed, and the coding model is confounded with GPU class.Source: §3.1 · p. 3, §3.2 · p. 4
3.2 Comparison with the reference schedule
- The reference comparison is not compute-matched: it uses up to 100 epochs with early stopping on an A100, and gan60 adds 1228 synthetic images.Loops A and C are validation-only; the test split is scored once after Loop B.Source: §3 · p. 2, §3.2 · pp. 3–4
- IoU improves by 0.034 and F1 by 0.038 relative to no_aug, and IoU by 0.002 and F1 by 0.011 relative to gan60 on the held-out test comparison.The comparison uses the same 256 test images as the reference, but is not compute-matched; it is specific to this split, backbone, and GPU.Source: §3.2 · pp. 3–4, Table 3 · p. 4
4 Conclusion
The conclusion reports that three AutoResearch campaigns improved one-hour DeepLabV3 baselines on a frozen real split, while retained changes are not universal defaults. The best Qwen3-8B configuration matches the reference test IoU without synthetic images.
4 Conclusion
- The Qwen3-8B configuration matches the reference test IoU without synthetic images.The conclusion cautions that retained modifications should not be treated as universal defaults.Source: §4 · p. 4
- Three campaigns improve one-hour DeepLabV3 baselines on a frozen real split.The conclusion frames effectiveness as conditional on a declared budget and frozen evaluator.Source: §4 · p. 4
- The retained changes differ by setting: a cosine schedule with reduced weight decay under 12 GB and two GX10 sequences reaching validation IoU 0.858.Source: §4 · p. 4
- A direct comparison with a declared-space tuner and a less restricted loop are left for future work.The proposed less restricted loop could edit architecture, preprocessing, or literature-derived ideas.Source: §4 · p. 4