PaperlayerView exact revision on arXiv

Source-linked AI summary

Opera: A Verbal Critic Framework for Long-horizon Coding Agents

Kai Mei, Zhiyuan Hu, Yutong Dai, Juntao Tan, Yifan Zhang, Dingjie Song, Dimitris N. Metaxas, Silvio Savarese, Ran Xu, Zeyuan Chen

arXiv:2609.33987v2cs.CL

TL;DR

Long-horizon coding agents need useful corrections, but incomplete evidence can make it hard to distinguish mistakes from reasonable intermediate work, and feedback may not resolve the underlying problem. Opera tracks audited corrections as persistent notes through subsequent execution. It improves resolve rates across three benchmarks and produces training rollouts that improve fine-tuning on held-out repositories.

  • Problem

    Critics must decide when to intervene, justify feedback despite incomplete evidence, and verify that the underlying problem is resolved rather than merely followed.

    Source: §1 · p. 1

  • Method

    Opera uses periodic and event-driven reviews, typed operators, evidence audits, and persistent notes that track adherence separately from resolution.

    Source: §1 · p. 1, §3 · p. 2, §3.1 · p. 3

  • Results

    Resolve rates improve by up to 12.4, 15.0, and 8.9 pp on Terminal-Bench 2.1, SWE-Bench Pro, and DeepSWE v1.1, respectively, across four policy models.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Opera-guided rollouts provide approximately on-policy training data, and fine-tuning Qwen3.5-9B on them improves held-out SWE-Bench Pro resolve rate by 10.2 pp.

    Source: §1 · p. 2

Abstract

from arXiv · show

Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is delivered. We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved. Opera decides when to review through periodic and event-driven triggers, diagnoses issues with typed operators, audits feedback against visible evidence before delivery, and tracks the agent's subsequent actions to distinguish mere compliance from actual resolution. As a test-time critic, Opera improves the resolve rate of non-critic agents by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, respectively, across four policy models, and achieves the highest mean resolve rate among competitive critic baselines on all three benchmarks, and also improves policy models when the policy critiques itself. Beyond inference, Opera-guided rollouts provide approximately on-policy training data: fine-tuning Qwen3.5-9B on them improves its resolve rate on held-out SWE-Bench Pro repositories by 10.2 percentage points without a critic at inference time, matching fine-tuning on rollouts from a stronger model, while preserving its performance when switching harness, i.e., from Openhands to Terminus-2, which the latter substantially degrades. Our code is available at: https://github.com/dongyuanjushi/Opera.

1 Introduction

Long-horizon coding agents need timely corrections, but critics must distinguish genuine mistakes from reasonable intermediate steps and verify that feedback resolves the underlying issue. Opera treats corrections as persistent notes, and its test-time and training results show benefits across coding benchmarks and policy models.

1 Introduction

  • Critics must judge unfinished work, where incomplete evidence can blur actual mistakes and reasonable intermediate steps.They also need to decide when to intervene, justify feedback, and determine whether the underlying problem was resolved rather than merely followed.Source: §1 · p. 1
  • Opera manages each correction as a persistent note, delivering it only when justified and tracking it until evidence supports resolution.It combines periodic and event-driven reviews, an operator critic, and audits; it interacts through execution traces and natural-language feedback without updating agent weights.Source: §1 · p. 1
  • Opera’s framework follows corrections through long-horizon runs, while its test-time use guides agent self-reflection.Figure 1 depicts issue tracking across stages such as search, edit, and test, alongside resolve-rate improvements across benchmarks, harnesses, and policy models.Source: §1 · p. 1, Figure 1 · p. 2
  • Opera improves resolve rates by up to 12.4, 15.0, and 8.9 pp on Terminal-Bench 2.1, SWE-Bench Pro, and DeepSWE v1.1, respectively, and outperforms competitive critic baselines.The improvements span four policy models; Opera-guided rollouts also improve Qwen3.5-9B fine-tuning on held-out SWE-Bench Pro repositories by 10.2 pp.Source: §1 · pp. 1–2
  • Opera-guided student rollouts provide approximately on-policy training data and improve Qwen3.5-9B on held-out SWE-Bench Pro repositories by 10.2 pp.The contribution summary reports that this fine-tuning matches distillation from a stronger model while preserving Terminal-Bench 2.1 performance.Source: §1 · p. 2

2 Related Work

Prior work develops feedback and reflection methods for coding agents and test-time scaling approaches. Opera connects additional inference computation to diagnosing and correcting an ongoing coding trajectory.

2.1 Critics and Verifiers for Coding Agents

  • LLM-based rubric verification has substantial noise, motivating explicit quality control over critic feedback.Related coding-agent work includes trajectory critics such as SWE-PRM and patch-evaluation approaches such as Agentic Rubrics.Source: §2.1 · p. 2

2.2 Test-Time Scaling for Coding Agents

  • Opera allocates additional inference computation to diagnosing and correcting an ongoing coding trajectory, connecting review timing with audited interventions.Prior test-time scaling methods allocate computation to exploration, evaluation, refinement, or solver selection.Source: §2.2 · p. 2

3 Opera: Verbal Critic Framework through Persistent Notes

Opera organizes verbal supervision around persistent notes that preserve a diagnosed issue and its resolution criterion across reviews. Hybrid scheduling, typed operators, and audited note management govern when and how it intervenes and tracks outcomes.

3 Opera: Verbal Critic Framework through Persistent Notes

  • A persistent note stores one diagnosed issue, current guidance, and a fixed resolution criterion, allowing Opera to follow up across reviews.The fixed criterion keeps resolution checks tied to the original problem rather than the latest guidance.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Hybrid Review Scheduling

  • Opera combines periodic reviews every k turns with immediate reviews triggered by five execution events, while allowing the critic to remain silent.Events are Idle, Repeat, Error, Claim, and Submission; an event-triggered review resets the periodic timer.Source: §3.1 · p. 3

3.2 Operator-Typed Review

  • Nine typed operators across five repair stages map observed issues to targeted corrections, each governed by a contract specifying evidence, correction, resolution, and exclusions.The operators cover Search, View, Edit, Test, and Submit; each review selects one supported issue and its smallest correction, or returns continue.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3 Audited Note Management

  • Admission audits reject unsupported or redundant proposals, while release audits close notes only when evidence meets the fixed resolution criterion.Opera separately tracks rule-checked adherence and criterion-based resolution; attempting a correction does not establish that the original problem is gone.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Evaluation

Opera is evaluated on three coding benchmarks with four policy models and compared with critic baselines. The evaluation also tests whether Opera-guided student rollouts can train a model without inference-time criticism.

4.1 Benchmarks and Setup

  • Evaluation covers Terminal-Bench 2.1, a 100-task SWE-Bench Pro subset, and DeepSWE v1.1, using four policy models and benchmark-specific harnesses.The experiments use Terminus-2 for Terminal-Bench, OpenHands for SWE-Bench Pro, and mini-swe-agent for DeepSWE.Source: §4.1 · pp. 4–5

4.2 Overall Task Performance

  • Opera achieves the highest mean resolve rate among critic baselines on all three benchmarks, with margins of 1.9 pp on Terminal-Bench 2.1, 1.8 pp on DeepSWE, and 0.6 pp on SWE-Bench Pro.The SWE-Bench Pro non-critic agent is already strong, and all critics yield similar, modest gains.Source: Table 1 · p. 5, §4.2 · p. 5
  • Opera improves every evaluated policy model on every benchmark except Qwen3.5-9B on DeepSWE, where neither the critic nor the unassisted model resolves a task.Qwen3.5-9B gains 12.4 pp on Terminal-Bench 2.1 and 15.0 pp on SWE-Bench Pro; gains generally shrink as non-critic resolve rates rise.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3 Recipes for Training Open-weight Coding Agents

  • Training evaluation splits SWE-Bench Pro by repository into 270 training and 108 held-out tasks, and also tests transfer to Terminal-Bench 2.1.The 67 selected training tasks are rarely solved by base Qwen3.5-9B, while both trajectory sources contain at least one successful trajectory for them.Source: §4.3 · pp. 5–6, Table 2 · p. 6
  • Opera-guided rollouts improve Qwen3.5-9B on held-out SWE-Bench Pro by about 10 pp while preserving Terminal-Bench 2.1 performance, unlike teacher rollouts.Teacher-rollout fine-tuning lowers Terminal-Bench 2.1 resolve rate from 23.6% to 7.9%, whereas Opera-guided rollouts yield 25.8%.Source: §4.3 · p. 6, Table 3 · p. 6

5 Analysis

Opera’s admitted diagnoses concentrate on code editing, with the leading diagnosis varying by policy model. The analysis also examines outcome trade-offs, operator ablations, and critic-model choice.

5.1 Critic Behavior and Utility

  • Edit-stage operators account for most delivered notes, while the leading diagnosis shifts from requirement-contract issues in Qwen3.5-9B to state-transition issues in stronger models.These frequencies reflect admitted diagnoses, not all agent errors; Muse Glimmer-30B has a flatter distribution with more focused-verifier notes.Source: §5.1 · pp. 6–7, Figure 5 · p. 7
  • On trials receiving notes, Opera improves over expected no-critic success for all models, but stronger models experience both more rescues and more regressions.Rescue and regression rates use different denominators; stronger models may act on feedback more readily, amplifying both useful and unnecessary interventions.Source: §5.1 · pp. 7–8, Figure 6 · p. 7

5.2 Operator ablation

  • The full operator set has the highest mean on all three benchmarks, but Edit-only and Non-edit variants retain most of its gain.Non-edit operators, despite accounting for about one fifth of delivered notes, recover over half the full gain on Terminal-Bench 2.1, three quarters on SWE-Bench Pro, and about four fifths on DeepSWE.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Sensitivity to Critic Model Choice

  • Opera improves over no critic in all but one of 33 tested policy–benchmark–critic combinations, and no critic model is best in every setting.Self-critique improves Qwen3.8-27B and DeepSeek-V4-Flash by 6.2 pp each on DeepSWE, while a stronger critic helps more when the policy struggles.Source: §5.3 · p. 8, Figure 7 · p. 9

6 Role of Audit

Removing both audits reduces Opera’s gains on all three benchmarks, with the largest reported drops on Terminal-Bench 2.1 and DeepSWE.

6 Role of Audit

  • Without admission and release audits, gains fall from 7.9 to 2.6 pp on Terminal-Bench 2.1 and from 8.9 to 4.5 pp on DeepSWE.The authors suggest audits filter harmful interventions, particularly where intermediate evidence is easy to misread and misguided corrections are costly.Source: §6 · p. 9

7 Conclusion

Opera treats each correction as a persistent note and follows it until the diagnosed problem is resolved. The conclusion reports gains at inference time and training benefits, while identifying limits of the training evaluation.

7 Conclusion

  • As a test-time critic, Opera improves non-critic agents’ resolve rates by up to 15.0 percentage points across three benchmarks and four policy models, achieving the highest mean among competitive critic baselines with Qwen3.8-27B as policy.Source: §7 · p. 9
  • Fine-tuning Qwen3.5-9B on Opera-guided rollouts matches stronger-model distillation on held-out repositories by +10.2 pp while preserving Terminal-Bench 2.1 performance.The paper describes the rollouts as approximately on-policy and reports that the training results use supervised fine-tuning on one model and one out-of-domain benchmark.Source: §7 · p. 9
  • The training evaluation is limited to supervised fine-tuning on a single model with one out-of-domain benchmark.Scaling the recipe, iterating with the trained model as the new policy, and combining critic feedback with reinforcement learning are identified as future work.Source: §7 · p. 9

A Implementation details of Opera

Opera operates as a proxy that can add at most one critic message without changing the agent’s harness, while combining event-driven and periodic reviews. Its corrections are audited persistent notes, and its baselines are adapted to operate during execution.

A Implementation details of Opera

  • Opera sits between the agent and model endpoint, forwards requests unchanged unless it inserts a critic message, and cannot block the agent.The critic sees only agent-visible content and does not execute commands, access files, or run tests.Source: A Implementation details of Opera · p. 13
  • Reviews occur on finish attempts, apparent stalls, or extended work without a successful edit, with periodic checks between events.The critic sees a bounded rollout along with the open note, its resolution criterion, and earlier notes.Source: A Implementation details of Opera · p. 13
  • Proposed corrections pass filters and a separate admission audit, then persist as one message until a release audit confirms resolution.While a note is open, the critic may update it or continue; the resolution criterion stays fixed as guidance changes.Source: A Implementation details of Opera · p. 13

A.1 Baselines and Adaptations

  • Baselines use Opera’s harness and bounded rollout, and process-critic baselines share its hybrid review schedule.This provides a common setup for comparing intervention approaches.Source: §A.1 · p. 13
  • SWE-PRM checks recent steps against its inefficiency taxonomy, while SWE-Search’s negative-reward alternative-branch feedback is applied to the current trajectory.Both are adapted to shared event triggers, although SWE-PRM was originally periodic.Source: §A.1 · p. 13
  • LLM-as-verifier and Agentic Rubrics are adapted from finished-candidate scoring to in-flight readiness checks that deliver missing or failed requirements.LLM-as-verifier accepts scores of at least 14 on its 1–20 scale; Agentic Rubrics scores up to eight repository-grounded items.Source: §A.1 · p. 13

B.1 Data Preprocessing

The training data comes from selected SWE-Bench Pro training tasks and successful trajectories from either critic-guided Qwen3.5-9B runs or uncriticized Qwen3.8-27B runs. The section describes how turns become SFT samples and how critic feedback is handled for training a model that runs without a critic.

B.1 Data Preprocessing

  • Table 6 lists the shared training configuration and per-source statistics, while Figure 8 presents loss and gradient dynamics for the two trajectory sources.The sources are Opera-guided self-reflection trajectories and Qwen3.8-27B trajectories.Source: Table 6 · p. 14, Figure 8 · p. 14
  • 67 SWE-Bench Pro training tasks meet the selection criteria, with no repositories from the 108 held-out tasks used.The base Qwen3.5-9B resolves each selected task at most once in three uncriticized runs; both a critic-guided run of that model and an uncriticized Qwen3.8-27B run resolve it.Source: §B.1 · pp. 13–14
  • Each agent turn becomes an SFT sample whose context includes prior actions and observations and whose target contains reasoning, visible text, and a tool call.Multi-call turns are split into single-call samples; invalid or rejected tool-call retries and samples longer than 128k tokens are excluded.Source: §B.1 · p. 14
  • Critic notes are removed from contexts because the trained model runs without a critic, while delivered-note diagnoses are rewritten as the model’s own reasoning.This rewriting applies to the 4.9% of turns where an audited note was delivered; references to the critic are also removed or excluded.Source: §B.1 · p. 14

B.2 Training details

The results distinguish an agent’s immediate response to feedback from whether it resolves the diagnosed issue. Edit-stage operators yield positive gains across models, while failure triage is less consistent; the section also identifies the training configuration and displays training dynamics.

B.2 Training details

  • Table 6 shows the hyperparameters used during SFT training, while Figure 8 presents training dynamics—loss and gradient—in the two settings.Source: §B.2 · p. 14

C.1 Behavior responses to the different operators

  • Qwen3.5-9B edits after feedback as often as Muse-glimmer-30B but resolves only about half of its requirement-contract and state-transition notes; Muse-glimmer-30B resolves most issues despite rarely editing immediately.Immediate adherence is therefore a poor proxy for correction, and agents seldom test unprompted; focused-verifier notes are the main trigger for testing.Source: §C.1 · pp. 15–16, Figure 9 · p. 15
  • All three Edit-stage operators yield positive gains for every model, while failure triage helps only Qwen3.5-9B and yields slightly negative gains for the others.The gains are grouped by the first admitted operator; failure triage’s weaker pattern mirrors its low closure rates.Source: Figure 10 · p. 15, §C.1 · p. 16

C.2 Critic behavior of using different critic models

Interventions are selective, and their frequency varies with policy strength and task difficulty. Critics also differ in how proposed notes are filtered before delivery, with filtering bringing self-critique’s delivered intervention rate closer to a stronger critic’s.

C.2 Critic behavior of using different critic models

  • Fewer than a third of reviews produce a candidate in nearly all settings, and weaker policies receive more interventions, especially on harder DeepSWE tasks.Qwen3.5-9B receives the most interventions on Terminal-Bench 2.1 and SWE-Bench Pro; GPT-5.6 generally delivers the most notes, consistent with its larger gains.Source: §C.2 · pp. 16–17, Figure 13 · p. 17
  • Self-critique delivers only a third of its candidates while nearly half are filtered before audit, whereas GPT-5.6 delivers most candidates.The self-critic’s dominant filter is the open-note constraint; for Qwen3.5-9B on Terminal-Bench 2.1 it proposes candidates at more than twice Claude’s rate but delivers notes at a similar rate.Source: §C.2 · p. 17

D Case studies

Across six cases, Opera notes addressed failures at search, view, edit, test, and submit stages. In each, the note remained open until visible evidence met a concrete resolution criterion.

Successful correction (search stage)

  • After 20 turns of unsupported grep searches, a search-stage note supplied a working query that located the callback; the task succeeded, while no-critic runs did not reach the edit/test phase.The agent then identified the missing TLS check, and audited closure followed visible evidence of the relevant callback and a passing test.Source: D Case studies · pp. 17–18

Successful correction (view stage)

  • The view-stage note corrected a type-only enum import and key/value membership error; direct calls to exported getLabelID passed both required cases (0/3 without the critic).The critic rejected a proxy check and required verification through the public function.Source: D Case studies · p. 18

Successful correction (edit stage)

  • An edit-stage note caught a wrong-branch change before testing, then addressed a second error-text mismatch; visible pytest output met the criterion (0/3 without the critic).The pre-existing test still asserted the obsolete format, but the visible run started a nonexistent command and showed the required hint.Source: D Case studies · p. 19

Successful correction (test stage)

  • A green test run missed the unavailable-directory branch; a note supplied a runnable check for returning None, and audited closure followed its passing output (0/3 without the critic).Two later notes in the same trial also closed after the option was registered and the function accepted the version value.Source: D Case studies · pp. 19–20

Successful correction (test stage, failure triage)

  • After one compiler error was fixed while TS2345 remained, a triage note prescribed typing enum values as readonly string[]; filtered tsc output was empty for the helper file.The agent subsequently exercised getPageSizeString and getLabelID end to end before finishing.Source: D Case studies · p. 20

Successful correction (submit stage)

  • A submit-stage note required the skipped end-to-end settings() check after an unrelated setup error; the script passed, though the note did not change the patch.The task succeeded in this trial, while no-critic runs did not.Source: D Case studies · pp. 20–21
Loading 2609.33987v2…