PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval

Junghyun Min, Huseyin Uzunalioglu, Mohamed Trabelsi

arXiv:2609.13073v1cs.AIcs.IRcs.LG

TL;DR

Many successful end-to-end autonomous research implementations have focused on tasks with narrow search spaces; this paper studies their use on telecom ticket retrieval, where data generation and representation are also open choices. The best discovered retriever beats the human-designed fine-tuned retriever but not the internal state of the art, supporting human-agent collaboration as the practical conclusion.

  • Problem

    Many successful end-to-end autonomous research implementations have focused on simpler tasks with fixed training data and small search spaces, leaving open how they apply to open-ended tasks.

    Source: §1 · p. 1

  • Method

    The authors conduct a case study on telecom ticket retrieval, an open-ended task spanning architecture, hyperparameters, training-data generation, and document representation.

    Source: §1 · p. 1

  • Results

    The best discovered single fine-tuned retriever substantially outperforms the human-designed fine-tuned model but does not outperform the internal state-of-the-art model.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The evidence supports combining systematic agent exploration with human direction, since available agents excel at narrow hyperparameter sweeps but lack human-like intuition and creativity.

    Source: §1 · p. 2, §6 · p. 6

    Evidence comes from one industrial task and three selected agents, and the human-led versus agentic comparison was uncontrolled, so it does not establish independent reproduction from scratch.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Recent breakthroughs in LLM-based systems and their abilities in problem solving and coding have allowed progress in the AI for Science paradigm, potentially replacing human roles in machine learning (ML) research. However, while several frameworks of fully autonomous end-to-end ML research have been proposed, successful implementations of them are often limited to problems with narrow search spaces, like language modeling or biomedical ML benchmarks. In this paper, we explore how autonomous research can be adapted to solve open-ended, industry-grade ML problems, by considering a case study: telecom ticket retrieval, an open-ended task with degrees of freedom in representation, architecture, and training data generation. We discover that autonomous research for open-ended problems with commercial and open-source agents shows both promise and limitations: while autonomous research can excel in narrow hyperparameter optimization, it lacks human-like intuition and creativity and requires operational overhead. Even with minimal human supervision, autonomous research can reach $90\%$ of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in a much shorter time period (10 weeks vs. 10 months of human work) at a modest cost (up to \$200 per Cursor campaign). Our empirical evidence recommends that human researchers and autonomous research frameworks work together for best results in ML research.

1 Introduction

The introduction asks whether autonomous research can address open-ended ML tasks rather than only problems with narrow search spaces. It studies telecom ticket retrieval and reports both promise and limits in agent creativity and operational reliability.

1 Introduction

  • Many successful end-to-end autonomous research implementations have focused on simpler tasks with fixed training data and small search spaces, motivating the question of applying them to open-ended tasks.Source: §1 · p. 1
  • The study asks how agent research compares with human research in task performance, time and API cost, and empirical advantages and limitations, while varying campaign agent count, LLM, and access to human-designed solutions.Source: §1 · p. 1
  • The case study uses telecom ticket retrieval, whose degrees of freedom include architecture, hyperparameter optimization, training-data generation, and document representation.Source: §1 · p. 1
  • The best discovered model is a single fine-tuned retriever that substantially outperforms the human-designed fine-tuned model but not the internal state-of-the-art model.The internal model incorporates ensembling, data augmentation, and re-ranking.Source: §1 · p. 1
  • Campaign structure, underlying LLM, and prior knowledge do not seriously affect downstream performance; locally hosted GPT-OSS without prior knowledge outperforms Claude Sonnet 5 agents with prior knowledge.The comparison is reported in Section 4.2, Table 3.Source: §1 · p. 2
  • Available agents excel at deep, narrow hyperparameter sweeps but lack human-like intuition or creativity, and campaigns require human babysitting.The authors recommend human direction and report time savings for modest cost.Source: §1 · p. 2

2 Background and Related Work

The background connects end-to-end autonomous research to telecom ticket retrieval, a document-retrieval problem whose search space extends beyond parameter tuning.

2.1 Autonomous Research

  • End-to-end autonomous research developed from partial automation of research tasks to systems that generate ideas, run experiments, and write papers, though scope, quality, and reliability remain limited.The section situates this study alongside single-agent autoresearch and multi-agent AutoScientists implementations.Source: §2.1 · p. 2

2.2 Telecom Ticket Retrieval

  • Telecom ticket retrieval takes a query document and retrieves a relevant resolution document, using a dense encoder and cosine similarity over preprocessed tickets.The retrieval objective is top-K recall over the evaluation split.Source: §2.2 · pp. 2–3
  • The task is difficult because tickets contain contextual, jargon-heavy, variable-detail language, while its search space includes data generation, loss design, and document representation; the incumbent system combines augmentation, ensembling, and re-ranking.The dataset clusters tickets and related fault and technical analyses, enabling different positive-pair sampling strategies.Source: §2.2 · p. 3

3 Experimental Setup

The setup evaluates autonomous research on a large telecom-ticket retrieval dataset and introduces an operational harness for this open-ended task. Campaigns vary framework structure, underlying agent, and access to human-designed solutions.

3.1 Datasets and Evaluation

  • The dataset contains 250k incident reports, 204k fault analyses, and 89k technical analyses; evaluation uses 7.6k held-out queries and 1.2k held-out gold documents against the entire dataset.A retrieval counts as correct when the query and retrieved document belong to the same incident cluster; performance is reported primarily as Recall@k.Source: §3.1 · p. 3
  • The human-designed comparison systems include BM25, a fine-tuned MPNet retriever, its ensemble, and an internal model with LLM-based augmentation and re-ranking.All comparison models were designed and trained by humans.Source: §3.1 · p. 3

3.2 Harness for Operational Stability

  • The harness targets task and environment documentation, search-space definition, and experimental-loop control to improve operational stability for open-ended retrieval campaigns.The authors report that these components emerged through trial and error and vary in effectiveness.Source: §3.2 · p. 3
  • Search-space controls broaden exploration beyond standard hyperparameters, but single-agent variables can constrain agents to toggling predefined options rather than making novel code changes.Multi-agent teams are assigned representation, training-data-generation, and architecture-and-hyperparameter areas; the authors note that they did not include an implementation intended to mitigate reliance on human-designed techniques.Source: §3.2 · p. 3
  • A deterministic outer bash loop proved more effective for maintaining campaign continuity than agent-managed loops or Python heartbeat controllers, which often stalled or repeated experiments.Task documentation covers the data structure, task scope, agent role, Recall@1 objective, and available hardware; agents still often underuse compute.Source: §3.2 · pp. 3–4

3.3 Autonomous Research Implementations

  • Campaigns vary along three design axes: single- versus multi-agent structure, commercial versus open-weight LLM agent, and uninformed versus informed access to human-designed techniques.The informed setup documents techniques such as data augmentation, re-ranking, and model ensembling; the authors hypothesize that multi-agent, Claude-based, and informed campaigns will perform better.Source: §3.3 · p. 4

4 Results

The best discovered system was a single fine-tuned MPNet retriever that outperformed several comparison systems but remained below Internal SoTA. Campaigns completed faster than human-engineered systems, while results varied little across agent designs and exposed limits in agents’ research intuition.

4.1 Our Best Discovered Model

  • Recall@1 = 0.343 for the best system exceeded BM25 (Recall@1 = 0.072), MPNet (Recall@1 = 0.200), and human-designed ensembles (Recall@1 = 0.251), but not Internal SoTA (Recall@1 = 0.380).The best discovered system was a single fine-tuned MPNet retriever; Table 2 compares it with comparison systems.Source: §4.2 · p. 4, §4.1 · p. 4, Table 2 · p. 5

4.2 Answering RQs

  • A 17-run campaign took 10 weeks end-to-end versus 10 months for human-engineered systems; a 10–20 experiment Cursor campaign cost $150–$200 in API charges.The cost estimate excludes repository setup and harness development.Source: §4.2 · pp. 4–5
  • Autonomous research performed focused optimization in a narrowly fixed search space, but agents changed parameters without using training dynamics or data distributions to guide hypotheses.Examples included increasing training from one to two epochs and sequence length from 256 to 384.Source: §4.2 · p. 5

4.3 Effects of Operational Decisions

  • Framework structure had no substantial impact on peak performance, and the best model came from a single-agent loop; multi-agent discussions rarely proposed effective experiments beyond parameter tuning.The initial hypothesis was that multi-agent discussion would yield wider searches and stronger proposals.Source: §4.3 · p. 5
  • LLM choice did not substantially affect the performance ceiling: half of campaigns reached Recall@1 between 0.30 and 0.34, while Cursor Composer 2.5 produced the top single model at Recall@1 = 0.343.No single agent produced particularly strong downstream performance.Source: §4.3 · p. 5
  • Providing human-designed-system documentation had little effect because agents did not implement re-ranking, data augmentation, or model ensembling, instead defaulting to single-model fine-tuning.The authors identify documentation alone as insufficient to induce outside-the-box architectural changes.Source: §4.3 · p. 5

5 Insights

The case study finds that autonomous research can efficiently optimize within a narrow search space, but its discovered systems miss higher-level strategies and still require human guidance and operational support.

Performance boundaries

  • The study frames autonomous research for open-ended problems as promising, with systematic search complementing human intuition and scientific reasoning.The authors position that combination as a likely fit for today's tools and resources.Source: §5 · pp. 5–6
  • Agent campaigns found single fine-tuned systems that surpassed human-designed baselines but underperformed complex state-of-the-art systems, while rarely exploring strategies such as ensembling, re-ranking, or data augmentation.Agents also made suboptimal follow-up proposals and were conservative about alternate representation encoders; effective deployment still relies on humans to define the search space and implementation specifications.Source: §5 · pp. 5–6

Engineering efficiency

  • The work took 10 weeks versus around 10 months of human engineering, but the comparison is not direct because the efforts were not independent..Source: §5 · p. 6
  • A Cursor campaign was estimated at US$150-200, Claude at 2-3 times more, and GPT-OSS at negligible cost when GPU resources were already available.The Cursor estimate excludes costs for fine-tuning documentation and the harness.Source: §5 · p. 6

Operational bottlenecks

  • Agents often underused GPU memory or went idle, and individual systems showed distinct failures such as duplicate experiments, unreliable sub-agent spawning, or debugging scripts instead of running experiments.Additional harness components partly addressed these struggles, but the authors say they do not cover all areas where guidance may be needed.Source: §5 · p. 6

6 Conclusion

The paper evaluates autonomous research on telecom ticket retrieval, an open-ended machine-learning task. It finds promise alongside operational and epistemic limitations, and argues that agentic search and human reasoning can complement each other.

Conclusion

  • The case study uses telecom ticket retrieval to examine autonomous research on open-ended machine-learning problems.The authors describe the task as open-ended and assess generalizability through this case study.Source: §6 · p. 6
  • The authors conclude that autonomous research for open-ended problems shows promise despite operational and epistemic limitations.Source: §6 · p. 6
  • The paper identifies systematic exploration by agentic frameworks and human intuition and scientific reasoning as complementary.The authors call this combination a likely sweet spot given today's tools and resources.Source: §6 · p. 6

Limitations

The authors qualify their findings by the limited case-study scope, lack of controlled comparisons, and evaluation measures that do not capture the full scientific process.

Scope

  • The study covers one industrial task and three selected LLM agents, so generalization to other problems and agents remains unclear.Other objectives, datasets, architectures, or more recent agents may exhibit different dynamics.Source: Limitations · p. 7

Comparison design

  • The human-led and agentic efforts were not controlled or independent: campaigns began with human-built resources, and a researcher monitored and stopped them.The results therefore are not direct evidence that agents can independently reproduce the entire research process from scratch.Source: Limitations · p. 7

Campaign evidence

  • Limited campaigns and uncontrolled variation make agent-choice effects difficult to isolate, while local-optimization observations are specific to this case study.Source: Limitations · p. 7

Evaluation

  • Evaluation primarily measures retrieval performance and campaign cost, not the quality, novelty, or reproducibility of the scientific process.The authors suggest broader task coverage and larger controlled campaign sets for future evaluations.Source: Limitations · p. 7

A Details on Best Discovered Model

The best discovered retriever combines structured-document preprocessing, cross-document training pairs, and a fine-tuned bi-encoder.

Results

Document Representation and Preprocessing

  • JSON-style documents are converted to dense text with explicit field markers, then cleaned by removing empty fields and recurring boilerplate found across at least five documents.The line filter targets system-header noise that could create spurious lexical overlap.Source: A Details on Best Discovered Model · pp. 9–10

Training data generation

  • Training data includes T-to-FA and T-to-T positive pairs, with T-to-T pairs sampled at twice the weight to yield 760k effective instances.For each positive, one hard negative is mined from outside its cluster using matching product and feature metadata when available, with random cross-category sampling as fallback.Source: A Details on Best Discovered Model · p. 10

Model architecture and hyperparameters

  • The retriever is an MPNet-initialized bi-encoder fine-tuned for two epochs with batch size 32, learning rate 2 × 10^-5, and maximum length 384.It minimizes Multiple Negatives Ranking Loss over cosine similarity with inverse temperature scaling of 20.Source: A Details on Best Discovered Model · p. 10
Loading 2609.13073v1…