PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Assessment of Agentic Systems in Acoustics

Arne Hildenbrand, Philipp Diel, Zhuo Cao, Hanno Scharr, Stefan Pischinger, Steffen Marburg

arXiv:2610.39685v1cs.SDcs.AI

TL;DR

This work asks how an agent’s model and harness shape performance and behavior in acoustic engineering tasks. It varies harness elements in a fault-detection benchmark and skill/agent work division in a metamaterial-design case study. The benchmark separates harness contributions, while the agent achieves the specified design goals in the case study.

  • Problem

    The study examines how an agent’s equipment and model contribute to performance in acoustic applications, where measurement faults are routinely checked.

    Source: §3 · p. 6, §3.6 · p. 13

  • Method

    The authors vary harness elements in a benchmark on synthetic vehicle-interior fault recordings and vary the division of work between skills and the agent in a metamaterial-design case study.

    Source: §3.2 · p. 8, §3.3 · p. 9, §5 · p. 18

  • Results

    The harness shapes workflow and achievement while the model sets the performance level; the agent reliably produced constrained metamaterial designs, most reaching their targets.

    Source: §1 · p. 2, §5 · p. 18, §4.4 · p. 18

  • Takeaways & Limitations

    Harness choices can make runs predictable and faults traceable, while skills can fix known design decisions or leave choices open to the agent.

    Source: §3.6 · p. 13, §4.4 · p. 18, §5 · p. 18

    The study varies accessible harness elements around an existing harness, not context composition or loop design; its tests use single-message interactions, leaving cross-session memory mechanisms unexamined.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

In engineering, applications of artificial intelligence have so far been largely limited to surrogate models and optimization tasks. Large language models changed this, as they can write software from a task given in words. Placed inside a harness, a system of components that lets it search for information and execute software, such a model becomes an agent that can take over recurring engineering tasks. This work measures how the model and the elements of the harness shape the performance and behavior of the agent in two common acoustic applications. In the first, the agent detects and locates faults in audio recordings from a vehicle interior with near-perfect accuracy. A benchmark on this task shows that a weak model sets a limit that no harness design can overcome, and that design choices take effect only once the model is capable enough to use them. In the second, a case study on the design of a metamaterial unit cell, the agent reliably finds designs that achieve a specified insertion loss over a given frequency range. The results aim to help the reader understand what each element of an agent contributes and how to design an agent for acoustic tasks.

1 Introduction

This work measures how an agent’s model and harness shape performance and behavior in two acoustic applications. It extends prior demonstrations of agents in acoustics by examining harness contributions and reporting benchmark and case-study outcomes.

1 Introduction

  • Earlier acoustic-agent studies addressed recording question answering, anomalous-sound synthesis, and aircraft-wing noise prediction.The paper contrasts demonstrations of finished frameworks with measurement of an agent against known outcomes.Source: §1 · p. 2
  • The study measures how harness elements influence agent performance and behavior, using acoustics as the test domain.It targets acoustic engineers considering agents for recurring simulation and measurement tasks.Source: §1 · p. 2
  • The evaluation uses a benchmark for acoustic measurement fault detection and a case study designing a metamaterial.The paper also introduces agent terminology and places its system within that vocabulary.Source: §1 · p. 2
  • The paper aims to explain what each agent element contributes and how to design agents for acoustic tasks.It reports that both applications proved achievable and that the benchmark separates harness-element contributions.Source: §1 · p. 2

2 AI Agents

The section defines agents as a model operating within a harness, then describes the harness elements and multi-agent design choices used in the study. It situates the system and its two acoustic tasks within that framework.

2 AI Agents

  • An agent combines a foundational model, which interprets input and chooses next steps, with a harness that supplies information and executes decisions.The foundational model sets the upper performance limit, while the harness influences how closely the agent approaches it.Source: §2 · p. 2, §2.1 · p. 2

2.1 Agent Harness

  • The harness groups functions into perception, planning, memory, and tool use, with skills serving as executable units that provide domain expertise.This work focuses on tool use and planning; perception and cross-session memory mechanisms are outside its scope.Source: §2.1 · p. 2
  • The ReAct loop assembles a context, reasons over it, executes skills, and returns execution results to the context for another turn.Context includes the prompt, configuration, mounted knowledge, memory, and available skills; configuration must document skills as well as provide them.Source: §2.1 · p. 3, Figure 2 · p. 3
  • Domain knowledge can enter through model fine-tuning, harness-provided retrieval, or agent-directed retrieval, which differ in where knowledge resides and who selects it.Memory can persist within or across sessions, but memory storage and retrieval are not examined in these single-message tests.Source: §2.1 · p. 4
  • The harness elements examined are those accessible around an existing harness; changing context composition or loop design requires harness engineering outside this work’s focus.The execution environment is isolated and reset between runs, supporting comparable benchmark runs.Source: §2.1 · p. 4

2.2 Agentic Systems

  • Multi-agent systems are designed along communication, collaboration, and collaboration-strategy axes, with choices including centralized or distributed communication and cooperative or competitive goals.Agents’ separate contexts support task separation but require text handoffs, which can omit information or propagate errors and make behavior less predictable.Source: §2.2 · pp. 4–5

2.3 Agent Setup

  • The study’s system uses centralized, cooperative, role-based orchestration, with subagents receiving tasks from and returning results to an orchestrator.The subagents do not communicate with one another; the setup uses OpenClaw and keeps the two applications’ skills and knowledge separate.Source: §2.3 · pp. 5–6, Figure 4 · p. 6
  • The system is tested on graded fault detection across eight prompts and on metamaterial design, where the agent searches for parameters to reach a noise-reduction target.For fault detection, tasks range from classifying a recording as faulty or clean to identifying one of nine fault types and its time interval; metamaterial runs vary how much work is assigned to a skill.Source: §2.3 · p. 6

3 Fault Detection in Synthetic Measurement Data

The benchmark tests fault detection against known ground truth while varying the harness, its information, and the model. Added harness elements—especially skills and knowledge—improve performance, but model capability and structural choices constrain what the agent achieves.

3.1 The Benchmark Dataset

  • Synthetic vehicle-interior recordings provide exact fault ground truth and a fixed, balanced set of clean and corrupted examples.Faults are injected into clean recordings, recording their type, interval, and severity; the benchmark contains 50 recordings, evenly split between clean and corrupted.Source: §3.1 · p. 7

3.2 Skill Design

  • The detection skills separate feature measurement from fault judgment, leaving the agent to choose and connect primitives and detectors.Four primitives produce measurement artifacts and nine detectors compare them with thresholds; without supplied thresholds, a detector fails and the agent must devise its own.Source: §3.2 · p. 8

3.3 Experimental Design

  • The three experimental phases isolate harness elements, robustness to conflicting information, and dependence on the language model.Each configuration-prompt combination is run ten times to account for probabilistic paths and variation in answers.Source: §3.3 · pp. 8–9

3.4 Evaluation Metrics

  • The evaluation matches scoring to response type, using binary correctness, balanced accuracy, or F1, with spread and resolution to qualify comparisons.Pooled F1 weights faults rather than recordings, and the resolution indicates the smallest distinguishable difference between run means.Source: §3.4 · pp. 9–10

3.5 Results

  • Adding harness elements improves performance, especially with skills and knowledge, while documentation alone shows no demonstrable gain.The fully equipped configuration scores best on every task with both models and perfectly in almost every run; two shortfalls stem from lost results and abandoned workspace output. Without skills, task 006 reaches 0.89 by ranking recordings within the batch.Source: §3.5 · pp. 10–12
  • With conflicting thresholds, knowledge-first instructions eliminate corrupted picks, while higher reasoning effort changes source selection without improving scores.Under fair instructions, the curated table was selected in 25 of 30 runs and the corrupted table in four; raising reasoning effort increased use of both sources from 1 of 30 to 11 of 30 runs.Source: §3.5 · pp. 11–12
  • On task 008, the weaker model drops from a perfect score to 0.80 in the fully equipped configuration.The weaker model scores lower on task 008 under every configuration; on task 004 it scores significantly lower under baseline and knowledge, while task 006 differences remain within noise.Source: §3.5 · p. 12, Figure 6 · p. 12

3.6 Discussion and Implications for the Practitioner

  • Configuration makes the workflow predictable enough to debug, but model limits and orchestration errors remain beyond configuration’s control.Multi-agent handovers can fail or be misinterpreted, and unintended workspace artifacts can affect results; developers should account for these risks and trace outputs to their sources.Source: §3.6 · p. 13

4 Design of a Metamaterial with a Predefined Insertion Loss

The case study tests an agent that designs periodic acoustic arrays to meet insertion-loss targets, varying the division of work between skills and the agent. Eight of twelve designs reached their targets, and the study examines how strategies and templates shaped the designs.

4.1 The Design Problem

  • The agent simulates a periodic array and evaluates insertion loss on a receiver line behind it, subject to maximum-depth and minimum-opening constraints.The two-dimensional frequency-domain model uses the Helmholtz equation, rigid duct and scatterer surfaces, and absorbing inlet and outlet boundaries.Source: §4 · p. 13, §4.1 · pp. 13–14

4.2 Study Setup

  • The study compares agent-driven optimization, agent-orchestrated search strategies, and agent-drawn designs without templates across four tasks and three difficulty levels.The first phase also has narrowband and broadband versions; the easier target is reachable by a scatterer, while middle and hard targets require a resonator.Source: §4.2 · pp. 14–15

4.3 Results

  • 8 of 12 designs reached their insertion-loss targets, all stayed within parameter limits, and agent-reported values differed from independent reruns by no more than 1.4 dB.The agent’s assessment of whether each target was reached matched the grader’s.Source: Table 7 · p. 16, §4.3 · p. 16
  • Runs generally used few solver evaluations: tasks 101, 102, and 104 used 3–10, while task 103 used 20, 46, and 11.The easy design in task 104 missed by 0.5 dB with 53 evaluations remaining.Source: §4.3 · p. 16
  • Without search strategies, resonator peaks fell below the graded band in tasks 101 and 102; with strategies, both resonators in task 103 peaked within the band.The authors call this a pattern rather than a measurement because there was only one run per cell.Source: §4.3 · p. 17
  • At 1 kHz, three resonators attenuated by 30–39 dB, while the task 103 scatterer attenuated by about 11 dB.The designs shown lie at or just inside the 0.20 m depth limit.Source: §4.3 · p. 17

4.4 Discussion and Implications for the Practitioner

  • The study’s contribution is documenting how the agent reaches designs, not advancing metamaterial design; the model is two-dimensional, lossless, and based on one run per design.The templates are standard cell types.Source: §4.4 · p. 17
  • Search strategies tuned resonators toward the target band, while leaving cell geometry open let the agent explore designs without a template.The authors recommend fixing decisions in skills when the answer is known and leaving decisions open when the design form is unknown.Source: §4.4 · p. 18

5 Conclusion and Future Work

The paper varies harness elements in a fault-detection benchmark and varies the division of work between skills and the agent in a metamaterial-design case study. It finds that harness choices shape how an agent works, while model choice sets how far it can get.

5 Conclusion and Future Work

  • Skills and curated thresholds change what the agent achieves, while workflow descriptions change its route and traceability without changing accuracy.Changing the model shifts the performance level of all configurations, and the fully equipped configuration scores best with either model.Source: §5 · p. 18
  • A skill pipeline trades reliability against exploration: encode known workflows in skills, but leave unknown design choices open to agent exploration.The search strategies and solver interface illustrate fixed decisions; agent-drawn unit cells illustrate open-ended exploration.Source: §5 · pp. 18–19
  • Agentic systems in acoustics remain at an early stage; more sophisticated designs, demanding applications, and complete simulation workflows remain to be examined.The authors call for measuring each added element and how elements work together.Source: §5 · p. 19

Author Declarations

The authors report using generative AI services for coding, data screening, and manuscript preparation, and declare no conflicts of interest.

Use of Generative AI

  • The authors used Claude, GPT 5.5, and DeepL for coding, data screening, formatting, and language editing, then reviewed and edited the content and take responsibility for it.Source: Use of Generative AI · p. 19

Conflict of Interest

A Matching Reported Events

Reported fault names and ground-truth labels are mapped to benchmark fault types before scoring. Event matching uses onset and offset tolerances, with duplicate detections counted as false positives.

A Matching Reported Events

  • The agent’s fault names and ground-truth names are mapped to the benchmark’s nine fault types before scoring.The mapping uses ordered rules given in the supplementary material.Source: A Matching Reported Events · p. 19
  • For each reference event, the grader selects the unmatched same-type prediction with the closest onset; each reference absorbs at most one prediction, so duplicates are false positives.Source: A Matching Reported Events · p. 19
  • A match requires both onset and offset to fall within a tolerance based on event duration and a type-specific floor.The floor prevents grading more finely than the measurement instrument, while a cap limits tolerance for faults persisting through the recording.Source: A Matching Reported Events · pp. 19–20

B Additional Benchmark Results

The appendix pairs run-archive records with achieved results and organizes additional benchmark evidence around the configuration ladder, context conflict, and model exchange. It reports costs, skill use, and performance under context corruption.

B Additional Benchmark Results

  • Most performance drops under context corruption stem from the agent selecting corrupted tables instead of the correct table.Table 10 reports three conditions, including a fair condition repeated with higher model reasoning effort.Source: B Additional Benchmark Results · p. 20, Table 10 · p. 22
  • Some cost cells are skewed because one unusually long or expensive run dominates the mean, so those cells warrant caution.Costs are reported as means with standard deviations and resolution.Source: B Additional Benchmark Results · p. 20
  • Table 9 aggregates skill use across 80 runs per configuration, counting a run as primitives only when it produced a feature artifact without calling a detector.The classification indicates that decisions were made somewhere other than in the detectors.Source: B Additional Benchmark Results · p. 20, Table 9 · p. 22
  • For the weaker model, Table 11 reports the same pattern of skill calls, time, and token use as the terra results in Table 8.The cost tables report means over 10 runs with standard deviation and resolution; tokens and wall-clock time are summed over the main agent and its workers in Table 8.Source: B Additional Benchmark Results · p. 20, Table 8 · p. 21, Table 11 · p. 22

Related papers

Loading 2610.39685v1…