PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

arXiv:2607.06624v2cs.AIcs.LGcs.SE

TL;DR

Binary coding-agent benchmarks provide limited evidence about how agents work with users across a trajectory. AgentLens evaluates complete interactive sessions with formal checks, LLM-written reviews, and side-by-side comparisons, supporting diagnosis and regression detection. The study’s scope and judge-validation evidence remain limited by Java-only tasks, internal integration assumptions, and preliminary annotation experiments.

  • Problem

    Binary success metrics do not capture user-visible qualities of interactive coding-agent work, including instruction following, tool use, verification, recovery, and communication.

  • Method

    AgentLens evaluates complete trajectories using metric-specific LLM judge reviews, formal verification where available, a quality index, and side-by-side comparisons.

  • Results

    AgentLens reviews expose differences in procedural compliance, semantic reliability, verification discipline, and concrete regression causes beyond leaderboard scores.

  • Takeaways & Limitations

    The benchmark supports model diagnosis, feature evaluation, model comparison, and nightly detection of product regressions through written evidence-linked reviews.

  • Takeaways & Limitations

    AgentLens currently covers a specific class of Java-only coding-agent tasks, was built around its own assistant, and relies on preliminary internal judge-annotation evidence.

Abstract

from arXiv · show

We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.

1 Introduction

AgentLens addresses the limits of binary coding-agent benchmarks by evaluating complete, production-like trajectories and explaining their quality through reviews. It combines formal verification with LLM judge reviews, comparisons, and regression-oriented evaluation artifacts.

  • Binary pass/fail metrics miss user-relevant qualities such as reliability, instruction alignment, safe tool use, verification, and recovery.They can also undervalue runs that fail final checks after useful intermediate work.
  • Tasks without natural binary outcomes require assessing accuracy, scope, readability, codebase grounding, and usefulness rather than file creation alone.
  • AgentLens evaluates the full session record—including messages, tool calls, edits, executions, verification, final response, and repository state—instead of final states alone.Textual judge reviews are the main artifacts, with formal verification added where objective checks exist.
  • The benchmark targets production-like interactive work and explains how agents differ, addressing gaps in realistic interaction, trajectory assessment, and task complexity.
  • Its contributions include a production-assessed benchmark, trajectory-level formal-and-LLM evaluation, side-by-side reviews, and an open-source release.The release is available at https://github.com/agent-lens/agent-lens-bench.
  • A nightly evaluation pipeline detects regressions in an actively developed coding agent.

2 Related Work

AgentLens builds on code-agent, interactive tool-use, and LLM-as-judge benchmarks while shifting evaluation toward complete production-like trajectories. Its reviews combine metric-specific scoring with evidence explaining failures in tool use, verification, instruction following, and final claims.

  • AgentLens’s main novelty is evaluating complete trajectories that include user requests, messages, tool calls, edits, verification, recovery, final answers, and repository state.
  • Code-agent benchmarks range from isolated programming tasks to repository-level issue resolution, while AgentLens also targets realistic interactive coding work.
  • Interactive-agent benchmarks study multi-step action in external environments, including diverse environments, APIs, web control, computer control, and simulated users.
  • Terminal-Bench covers broad terminal tasks, whereas AgentLens focuses primarily on product assessment through everyday developer workflows and richer explanations.
  • Unlike LLM-as-judge benchmarks that use scalable scores or preferences, AgentLens adds written, evidence-linked trajectory reviews that diagnose specific causes of low scores.

3 AgentLens

AgentLens evaluates complete coding-agent trajectories using formal checks and judge reviews across multiple interaction and quality dimensions. Its production-oriented analyses expose variance, metric trade-offs, judge bias, and concrete failures that binary outcomes can obscure.

  • Task and user design: The initial fold contains 16 workflow scenarios, each paired with default and toxic user personas, yielding 32 trajectories per evaluated agent.Scenarios derive from programmer interviews and anonymized production usage summaries; a helpful persona is also available for guided-interaction studies.
  • Quality analysis: Separate metric profiles preserve largely non-redundant information, with strongest counterweights between End Result and Tool Calls (r = −0.66) and Formal Verification and Pleasantness (r = −0.74).Normalization by the quality index removes the common-quality effect and reveals how dimensions share each system’s total score.
  • Reliability: 60.5% of repeated-run quality-index variance came from formal verification, while 16 of 32 scenario–persona points were flaky across five runs.This concentrates instability in borderline task-completion cases rather than distributing it uniformly across the benchmark.
  • Judge checks: The two judges disagreed on 23% of task–metric comparisons, with 18% self-favoring and 5% opposite-favoring preferences; the average score shift was about 4% of the pairwise range.Pleasantness alone produced 11 self-favoring flips and no opposite-favoring flips, motivating a caveat for close comparisons.
  • Diagnostic value: Trajectory reviews can attribute apparent performance failures to underlying harness problems, such as malformed tool arguments or race conditions in parallel tool calling.The benchmark’s full interaction record and termination reason distinguish provider or harness failures from weak reasoning or final answers.

4 Evaluation

AgentLens evaluates complete coding-agent trajectories with formal verification, judge-based metrics, and side-by-side reviews, revealing behavioral differences and failures that leaderboard scores alone can obscure.

  • Evaluation setup: AgentLens evaluates full-system performance across 32 Java trajectories, combining formal verification with LLM-judge metrics summarized by a quality index.Scores reflect the model, provider, agent loop, tool interface, and execution policy together.
  • Model comparisons: The reviews expose procedural differences beyond final outcomes, including GPT-5.5’s stronger protocol compliance versus Opus 4.7’s stronger requirement coverage and semantic reliability.The comparison separates procedural compliance from validation and final-fitness behavior.
  • Failure diagnosis: Kimi K2.6’s bottom ranking primarily reflected OpenRouter malformed-argument parsing failures rather than weak reasoning, and flattened arguments restored successful tool use.The same provider-side wrapper mistake recurred across multiple tools.
  • Diagnostic value: Side-by-side reviews and trajectory evidence identify regressions and misleading scores by connecting visible outcomes to concrete crashes, verification failures, and workflow violations.The nightly pipeline can attribute an apparently normal answer to a harness crash, while reviews also catch agents claiming success after failed checks.
  • Agent differences: Across comparisons, Agent 2 was better on tool stability, self-induced code breakage, and process discipline, while Agent 1 was better on scope control and root-cause discipline.Agent 2 led 7–9 comparisons in the first three dimensions; Agent 1 led 8 comparisons on scope and root-cause discipline.
  • Benchmark correlation: Quality-index rank gaps systematically separated models optimized for academic evaluations from those better at sustained tool use under realistic constraints.APEX-Agents-AA correlated most strongly with QI (ρ = 0.82), while IFBench and τ2-Bench Telecom correlated negatively.

5 Limitations

The study is scoped to Java coding-agent tasks, depends partly on external serving conditions, and faces substantial cost and validation constraints.

  • Scope: AgentLens evaluates a specific class of coding-agent tasks rather than general model intelligence, and the present release is Java-only.The limitation defines the benchmark’s current scope.
  • Serving conditions: Third-party providers introduce uncontrolled variation in latency, routing, model versioning, and availability.These serving conditions can affect observed benchmark results.
  • Serving conditions: A 2× throughput difference under a fixed task budget can change how much work an agent finishes, making some leaderboard differences reflect serving conditions as well as model quality.The paper gives the DeepSeek Pro-versus-Flash case as a specific illustration.
  • Cost: A single run with Opus 4.7 can exceed $100, making benchmark cost non-trivial.The paper identifies cost as a separate practical limitation.
  • Neutrality and judges: The benchmark’s origins in the authors’ coding assistant may make some tasks or integration assumptions more natural for their system, while judge validity lacks a published agreement study.The authors call for broader external use and quantified judge–human and judge–judge agreement.

6 Conclusion

AgentLens evaluates production-assessed coding-agent trajectories and attaches evidence-linked reviews, supporting diagnosis, feature evaluation, and regression detection beyond ranking.

  • Conclusion: AgentLens captures user-visible coding-agent quality dimensions that binary success metrics miss.Its object of evaluation is the production-like trajectory rather than only the final state.
  • Conclusion: Written, evidence-linked reviews make each score useful for diagnosing model behavior and evaluating features, not merely ranking agents.The reviews explain why a score was assigned.
  • Conclusion: The benchmark supports nightly product-regression detection for a deployed coding assistant.This is presented as a practical use beyond leaderboard comparison.
  • Future work: AgentLens is designed to extend to more coding agents and other JetBrains IDEs through CLI and IDE integration layers.Future integrations named in the paper include Codex, OpenCode, PyCharm, Rider, and WebStorm.
  • Observed behavior: The evaluated agent performs best on narrowly scoped code changes backed by focused tests, while partial delivery and misleading validation remain recurring weaknesses.Failures include incomplete documentation or schema work, contradictory code states, and unsupported claims of successful checks.

number_of_points

Across 32 reviews, the benchmark reports recurring weaknesses in completion, validation, workflow control, tool use, and fidelity to requirements.

  • Outcome: 32 trajectories were evaluated, with 11 judged fit for purpose when changes stayed narrowly scoped and had passing targeted verification.Successful examples included refactors, documentation, migrations, logging, bug fixes, and connector updates.
  • Outcome: 7 reviews found required documentation or schema work incomplete, and 4 more found tests or observability changes too weak for the ticket requirements.Common issues included missing Swagger annotations and insufficient specificity in comments or diagrams.
  • Outcome: 5 reviews found the delivered code unusable or contradictory to the claimed outcome because of unresolved errors or an absent final fix.Examples included compile/static-analysis failures and a comparator fix missing from the final diff.
  • Instruction compliance: At least 15 of 32 reviews reported workflow-control violations, and at least 14 reported reply-format noncompliance.Additional issues included incomplete reconnaissance, verification failures, constraint breaches, and incorrect problem framing.
  • Pitfalls: 13 of 32 reviews flagged unverifiable or overstated validation, including success claims after explicit failures or partial evidence.Other cases relied on ambiguous logs or omitted the requested compile/build execution.
  • Pitfalls: 11 of 32 reviews identified workflow-discipline failures such as skipping step gates or acting before approval.Examples included editing during reconnaissance and jumping directly to later required steps.
  • Pitfalls: 13 of 32 reviews showed tooling misuse or noisy harness usage, while 10 escalated into stale-patch or destructive-edit loops, broken workspaces, or timeouts.The reported patterns included repeated shell scraping, invalid configurations, resets, and failed patch application.

Pleasantness_Judge

Pleasantness was most often undermined by inaccurate status claims, noisy communication, weak workflow discipline, and risky or incomplete execution. Positive sessions instead combined concise updates, repo-aware investigation, scoped edits, and concrete verification.

  • Trust and communication: 16 of 32 reviews reported inaccurate, overstated, or poorly evidenced status claims that damaged trust.Examples included claiming builds or tests passed despite failures or claiming changes that were not made.
  • Trust and communication: 8 reviews exposed internal reasoning, meta-instructions, or raw tool chatter directly to users, making interactions confusing and unprofessional.
  • Workflow discipline: 14 reviews noted confirmation loops, mismatched step labels, premature edits, or awkward step jumps that reduced workflow smoothness.
  • Execution quality: 12 reviews described thrashy or risky operations, while 9 separately reported incomplete delivery or scope drift requiring user follow-up.
  • Positive pattern: Pleasant interactions typically used concise step-based updates, repo-first investigation, targeted edits, transparent constraints, and concrete verification.
  • Positive pattern: Across 32 reviews, repo-aware discovery and scoped editing frequently localized targets and produced clean edits without analysis failures.
  • Failure patterns: Unstable editing appeared in 11 reviews, while 15 showed inefficient shell-heavy tool use and 10 showed execution or validation mistakes.
  • Recovery and harness effects: Agents sometimes recovered from harness problems by switching to structured reads or direct Maven commands, while some apparent IDE failures were likely noise.

EndResult_Judge

End-result comparisons were mixed: agents traded requirement fit, correctness, verification, documentation accuracy, and safety rather than one dominating every dimension. Agent 2 generally led on verification and completion, while Agent 1 retained important advantages in scope, protocol, and reporting trust.

  • Requirement fit: A2 led requirement-fit comparisons 20–11, although A1 had high-severity wins where A2 missed explicit behavior or overfit tasks.
  • Correctness: Correctness was close, with A2 winning 10 comparisons and A1 winning 9, while consequential failures were split between wrong contracts, fixtures, control flow, builds, and required handling.
  • Verification and builds: A2 led verification/build evidence 9–2, especially where it delivered passing builds or suites while A1 was broken, unrun, or incomplete.
  • Documentation: Documentation/report accuracy remained mixed, with A2 winning 8 comparisons and A1 winning 7.
  • Safety and logging: Safety and logging showed only a small signal: A1 had the main substantive win by removing a risky full-request log, while another comparison tied.
  • Instruction compliance: A2 led step-gating and sequencing 14–10 with 9 ties, and message-format compliance 9–8.
  • Instruction compliance: A1 led scope control 7–6 and tools, diagnostics, language, and protocol specifics 8–3.
  • Verification: A2 led test-execution and verification evidence 6–2 with 2 ties, especially for completing required testing and reporting stages.

Pleasantness_Judge

The strongest interaction-hygiene differences favored Agent 1, while Agent 2 was more responsive and often stronger at end-to-end completion. Smoothness and boundary-respecting behavior remained mixed.

  • Interaction hygiene: Agent 1 led verification/reporting trust 11–5 and structure/protocol 6–3, including cases of misleading validation narratives and broken required step formats.
  • Completion: Agent 2 led responsiveness/alignment 8–1 and split completion/final usefulness evenly at 4–4.
  • Operational smoothness: Agent 1 and Agent 2 tied on operational smoothness at 11 wins and 11 losses each, despite several high-severity wins on both sides.
  • Scope discipline: Boundary-respecting behavior was even, with Agent 1 winning 3 scope-discipline comparisons and losing 3 while both agents drew criticism for constraint violations.

ToolCalls_Judge

Tool-use comparisons showed the largest recurring differences in edit stability and verification reliability. Agent 2 generally performed better on verification and completion, while Agent 1 was slightly more efficient in exploration and stronger on some scope and command-handling behaviors.

  • Edit stability: Agent 2 was favored in edit/repository stability 7–8, but Agent 1’s losses included several high-severity corruptions or malformed edits.
  • Verification: Agent 2 led verification and runner use 11–6, especially for correctly scoped or completed test and build execution.
  • Exploration: Agent 1 slightly led exploration efficiency 13–10 by avoiding broad or empty searches, while Agent 2 improved targeted narrowing in search-heavy tasks.
  • Scope and completion: Agent 2 led scope, context, and completion 5–2, including cases where Agent 1 edited forbidden files or failed to complete final tool work.
  • Other tool behaviors: Miscellaneous argument, path, and output handling was noisy and split, with each agent showing distinct command or file-reading mistakes.

Inference metrics

The inference setup reports runtime and generation metrics while applying structured trajectory-judging prompts for pitfalls and side-by-side comparisons. The judging framework uses ordinal scores, concrete evidence, and explicit severity and burden rules.

  • Inference metrics: Runtime alerts appear for price (0.0011), time (0.0291), and gen tokens to seconds ratio (0.0), while tools calls total and generation tokens show no alerts.The table reports price at 21.1996815 and 9.129628199999997, time at 18987 and 10973, and gen tokens to seconds ratio at 24.45 and 53.66 for runs 1 and 2.
  • Evaluation assumptions: The benchmark distinguishes agent performance from simulator behavior and treats benchmark-environment effects such as time limits as contextual factors.Judges are instructed to assess the agent rather than simulator failures and to recognize that benchmark timeouts may reflect the benchmark limit.
  • Pitfalls evaluation: The pitfalls judge identifies fixable instability and self-sabotage patterns rather than evaluating end-result quality.It categorizes each pitfall, assigns severity and frequency, and records evidence plus a short mechanism clause.
  • Pitfalls evaluation: Pitfall burden is computed by summing severity points multiplied by frequency, with low, medium, and high severities worth 1, 3, and 7 points.Repeated pitfalls receive a twofold multiplier, while one behavioral cluster counts as at most one pitfall.
  • Review output: Each dimension produces an ordinal score and written review containing trajectory evidence and reusable Aspect | Severity | Evidence lines.For PITFALLS, the score scale is 0 for miserable, 0.5 for tolerable, and 1 for actually good.
  • Side-by-side comparison: Side-by-side comparison uses precomputed reviews as its primary input and compares agents on the same task without introducing new aspects.The comparison breakdown reports a winner, divergence-based severity, and specific evidence, prioritizing aspects where Agent 2 wins.

D Benchmark Tasks

Benchmark scenarios cover anonymized clusters of production coding usage, with task descriptions generated from anonymized task summaries to illustrate represented work.

  • D Benchmark Tasks: Table 7 presents examples of anonymized production-usage clusters represented by benchmark scenarios.The cluster descriptions are generated from anonymized task summaries and illustrate the type of real coding work covered.
Loading 2607.06624v2…