PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Textual Interfaces To Enhance Llm Reasoning On Audio-Visual Question Answering

Yi Wang, Tianzi Wang, Tong Lei, Dahan Wang, Peter Bell

arXiv:2610.20085v1cs.AIcs.CLeess.AS

TL;DR

APO optimizes textual prompts, so applying it to audio-visual reasoning requires interfaces that expose editable instructions and useful feedback. This study compares direct, captioner-reasoner, and agentic interfaces, finding that optimization gains depend on the model and do not reliably transfer to the latter two systems.

  • Problem

    APO's use in multimodal reasoning requires textual interfaces specifying which instructions can be changed and what feedback or intermediate information can guide updates.

    Source: §1 · p. 1

  • Method

    The study compares direct OmniLLM prompting with captioner-reasoner and agentic systems to examine how their textual interfaces interact with APO.

    Source: §2 · p. 2, §2.2 · p. 2, §5 · p. 4

  • Results

    APO substantially improves direct Gemini inference, reaching 79.62% overall accuracy on Daily-Omni, but yields no reliable improvement for captioner-reasoner or agentic systems.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Explicit textual interfaces make multimodal reasoning easier to inspect and steer, but do not necessarily make APO more effective.

    Source: §5 · p. 5

    Caption-instruction APO was limited by brittle instruction-following behavior that produced repetitive, meaningless captions.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Automatic prompt optimization (APO) has shown strong effectiveness for text-based LLMs and LLM systems, but its application to audio-visual understanding systems requires suitable textual interfaces through which multimodal reasoning can be exposed to text-based optimizers. We investigate how the design of textual interfaces affects prompt optimization for multimodal reasoning, on the audio-visual question answering task. We compare three interface designs, including a direct inference system consisting of a single omni-modal large language model (OmniLLM), a captioner-reasoner system, and an agentic architecture in which a primary agent orchestrates subagent OmniLLMs. Experiments with Qwen3-Omni and Gemini show that direct prompt optimization provides model-dependent gains; the captioner-reasoner system and the agentic system expose more interpretable optimization interfaces but are harder to optimize, where error analysis of cases affected by optimization is conducted. Our study highlights textual interface design as an important factor in making prompt optimization effective for omni-modal reasoning systems. Our implementation details are specified in GitHub repo.

1. INTRODUCTION

The paper asks whether explicit textual interfaces make automatic prompt optimization more effective for audio-visual reasoning than optimizing a direct OmniLLM instruction. It compares direct, captioner-reasoner, and agentic systems, treating interface design as central to exposing multimodal reasoning to text-based optimizers.

1. INTRODUCTION

  • OmniLLM outputs can vary with prompt wording, making manual prompt design potentially brittle and motivating automatic prompt optimization.APO uses task feedback to iteratively search for refined instructions.Source: §1 · p. 1
  • Because APO optimizes text prompts, multimodal systems need an interface that specifies editable instructions and available feedback or intermediate information.The optimizer may use task feedback, while the prompts it changes remain textual.Source: §1 · p. 1
  • The study asks whether explicit textual interfaces make APO more effective for audio-visual reasoning than direct OmniLLM prompting.It examines how intermediate reasoning interfaces interact with APO, rather than designing systems primarily to improve task performance.Source: §1 · p. 2
  • The comparison treats captioner-reasoner systems as static evidence externalization and agentic systems as dynamic evidence acquisition, alongside direct OmniLLM inference.AVQA provides a testbed requiring retrieval of sparsely distributed audio-visual evidence and reasoning over it.Source: §1 · p. 2
  • The paper reports a systematic study of textual-interface design for APO in audio-visual reasoning, with controlled experiments and analyses of reasoning trajectories and action traces.The analyses identify distinct limitations in captioner-reasoner and agentic systems.Source: §1 · p. 2

2. METHOD

The method compares three AVQA architectures that expose different textual instructions for optimization, while keeping LLM parameters frozen. APO uses training examples and available evidence to revise instructions, then selects the best validation-performing candidate for held-out testing.

2. METHOD

  • All systems predict an AVQA answer from video, audio, a question, and answer options, keep LLM parameters frozen, and expose role-specific prompts for optimization.The method uses OmniLLMs and, optionally, text-only LLMs.Source: §2 · p. 2

2.1. Direct OmniLLM Inference

  • In direct inference, one OmniLLM jointly receives the video, audio, question, and options, with a single optimizable task instruction.That instruction may specify the AVQA task and optional procedures or rules.Source: §2.1 · p. 2

2.2. Captioner-reasoner System

  • The captioner-reasoner system produces a question-independent audio-visual caption, then uses a text-only reasoner conditioned on the question, options, and caption.The captioner and reasoner have separate optimizable instructions.Source: §2.2 · p. 2

2.3. Agent-Subagent System

  • The agent begins with a caption, then chooses whether to request another caption, query a responder, or answer using its interaction history.Caption and query-response OmniLLM subagents provide global event summaries and targeted evidence; the agent and perception-side prompts are optimizable.Source: §2.3 · pp. 2–3

2.4. Automatic Prompt Optimization

  • APO runs systems on labeled training examples, lets a text-based optimizer revise instructions using evidence and predictions, and selects the highest-validation-accuracy candidate for held-out testing.Available evidence can include responses, reasoning trajectories, or agent-action and subagent-observation traces; QA accuracy is the shared optimization objective.Source: §2.4 · p. 3

3. EXPERIMENTS

The experiments evaluate prompt optimization for AVQA across direct single-OmniLLM inference, captioner-reasoner pipelines, and an agentic system. They use Daily-Omni and compare GEPA with COPRO as a reference optimizer.

3. EXPERIMENTS

  • The study compares direct inference, captioner-reasoner pipelines, and an agentic system for AVQA, focusing on compatibility with GEPA and including COPRO as a reference.Source: §3 · p. 3

3.1. Evaluation Dataset

  • Daily-Omni contains 1,197 multiple-choice AVQA examples across six categories, split into 125 training, 125 validation, and 947 held-out test examples.The split preserves the overall task-category distribution; the held-out test set is the primary measure of optimization effectiveness.Source: §3.1 · p. 3

3.2. Models and Systems

  • Direct inference uses Qwen3-Omni Instruct and Thinking variants or Gemini-2.5-Flash, while both composed systems use Gemini for perception and GPT-4.1 for text-based reasoning or orchestration.Qwen3-Omni variants were not carried forward as captioners because their caption instructions under APO produced repetitive, meaningless captions.Source: §3.2 · p. 3

3.3. APO Algorithms

  • COPRO and GEPA use GPT-4.1 as optimizer, receive equal budgets of 2,500 trials, and select instructions by validation accuracy for held-out evaluation.Source: §3.3 · p. 3

4. RESULTS AND ANALYSIS

Prompt optimization helps direct Gemini inference but not consistently direct Qwen3-Omni, and adding captioner-reasoner or agentic interfaces does not yield reliable held-out gains. The analyses identify interface-specific trade-offs, including caption information loss and agentic perception and reasoning failures.

4.1. APO on the Direct-Inference Interface

  • Gemini test accuracy rose from 75.71% to 79.83% (by 4.12%) with GEPA, while Qwen3-Omni optimizers did not consistently improve direct QA.COPRO also gained 2.54% on Gemini; Qwen3-Omni-Think remained comparable to baseline. Table 3 reports the captioner-reasoner results with direct Gemini inference as a reference.Source: §4.1 · pp. 3–4, Table 3 · p. 4

4.2. APO on the Captioner-Reasoner Interface

  • Both Gemini and GPT-4.1 reasoners using the same Gemini-generated captions scored around 5% below direct Gemini inference.The authors attribute the drop possibly to captions bottlenecking information: captions compress audio-visual input without knowing which details matter to the question. Table 3 covers the captioner-reasoner interface and prompt optimization.Source: §4.2 · p. 4, Table 3 · p. 4
  • Optimizing either the caption or reasoner instruction did not improve held-out accuracy; caption optimization was especially brittle, while reasoner optimization was stable but ineffective.Validation gains did not reliably transfer to test. Table 3 reports the captioner-reasoner interface and prompt-optimization results.Source: §4.2 · p. 4, Table 3 · p. 4

4.3. APO on the Agent-subagent Interface

  • The ReAct agent-subagent system performed comparably to direct Gemini inference, but GEPA did not improve held-out accuracy.Table 4 reports the agentic system and prompt optimization on the text agent workflow instruction.Source: §4.3 · p. 4, Table 4 · p. 4
  • Among direct-correct cases made wrong by ReAct, 61.7% involved conflicting perception evidence and 27.3% involved reasoning errors over useful evidence.ReAct improvements mainly came from retrieving additional useful evidence or richer captions, so active evidence acquisition could recover missing information but also add failure points.Source: §4.3 · p. 4
  • GEPA’s validation gains on the agentic system did not transfer to test, despite reducing query-responder calls by 35.8% and total tool calls by 15.0%.Unlike the direct prompt’s focus on final-decision evidence grounding, the optimized ReAct prompt changed a longer chain of interpretation, querying, evidence acquisition, and integration; Table 5 shows selected instruction excerpts.Source: §4.3 · p. 4, Table 5 · p. 4

5. CONCLUSIONS

The study finds that prompt optimization’s effectiveness depends on the audio-visual system’s textual interface: direct Gemini gains are not shared consistently by Qwen3-Omni, captioner-reasoner, or agentic systems. Explicit interfaces aid inspection and steering, but do not necessarily make optimization more effective.

5. CONCLUSIONS

  • The study compares direct OmniLLM prompting with captioner-reasoner and agentic interfaces, finding that APO is not uniformly more effective when reasoning is externalized.Source: §5 · p. 4
  • Direct prompt optimization substantially improved Gemini inference, reaching 79.62% overall accuracy on Daily-Omni, but produced little gain on Qwen3-Omni.Source: §5 · p. 4
  • Explicit textual interfaces make multimodal reasoning easier to inspect and steer, but do not necessarily make APO more effective.The reported interface-specific failure modes include caption information loss, conflicting perception evidence, and downstream reasoning errors.Source: §5 · pp. 4–5

Related papers

Loading 2610.20085v1…