PaperlayerView exact revision on arXiv

Source-linked AI summary

HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

Tu Trinh, Mohamed Elfeki, Guangze Luo, Kelvin Luu, Nathan Hunt, Ernesto Hernandez, Nandan Marwaha, Yannis Yiming He, Charles Wang, Fernando Carabedo, Alessa Castillo, Bing Liu

arXiv:2604.09408v4cs.AI

TL;DR

Existing benchmarks rarely test whether coding agents recognize unresolved ambiguity and ask for help, because they emphasize execution under complete specifications. HiL-Bench evaluates selective escalation through progressively discovered blockers and finds a large judgment gap, while shaped Ask-F1 reinforcement learning improves help-seeking and task completion across domains.

  • Problem

    Existing benchmarks provide detailed, unambiguous instructions and reward execution correctness, leaving agents’ judgment about when to ask for help insufficiently measured.

  • Method

    HiL-BENCH evaluates selective escalation on SWE and SQL tasks containing human-validated blockers that emerge through progressive exploration, using Ask-F1 to balance question precision and blocker recall.

  • Results

    75–89% pass rates with complete information collapse to 4–24% when agents must decide whether to ask, while RL on shaped Ask-F1 improves help-seeking and task completion across domains.

  • Takeaways & Limitations

    Help-seeking judgment is a model-level weakness but is trainable: reinforcement learning teaches agents to detect unresolvable uncertainty and act on it rather than use domain-specific heuristics.

  • Takeaways & Limitations

    Performance drops under blocked conditions are designed to be minimally confounded by capability limits after required information is restored to tasks.

Abstract

from arXiv · show

Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.

1 Introduction

HiL-Bench targets selective escalation: recognizing when missing, ambiguous, or contradictory information cannot be resolved independently and asking for help. It addresses benchmarks and agents that reward confident guessing by evaluating progressive help-seeking judgment alongside task performance.

  • Selective escalation: Selective escalation means recognizing mid-task when a gap cannot be resolved through exploration or inference alone and surfacing it for help.The motivation contrasts this judgment with immediately coding from a vague specification.
  • The judgment gap: Current agents often fill unclear specifications with confident assumptions, producing plausible but incorrect outputs without hedging or escalation.Clarification mechanisms exist in Claude Code, Codex, and Cursor, but are rarely invoked at the right time.
  • Benchmark blind spot: Existing SWE, HumanEval, and BIRD-SQL benchmarks reward silent guessing because fully specified tasks cannot distinguish lucky assumptions from appropriate clarification.This creates a feedback loop in which strong benchmark scores coexist with persistent deployment failures.
  • Findings: 75–89% pass@3 with complete information falls to 4–24% when models must judge when to ask, while RLVR shaped by Ask-F1 improves judgment and pass rate.The benchmark’s taxonomy identifies distinct judgment signatures across model families, and training gains transfer beyond a single task type.
  • HiL-Bench design: HiL-Bench uses SWE and SQL tasks modified with human-validated blockers, each assessed against seven strict quality criteria and revealed through progressive discovery.Agents must begin working, encounter an unresolved gap, ask a targeted question, incorporate the answer, and continue.
  • Evaluation: Ask-F1 is the harmonic mean of question precision and blocker recall, preventing question spam while measuring the tradeoff between under-asking and over-asking.The benchmark also reports task pass rate to separate help-seeking judgment from solving ability.

2 Related Work

Prior benchmarks primarily reward autonomous execution under complete specifications, while clarification and interactive benchmarks test ambiguity detection rather than selective escalation. HiL-Bench addresses these gaps by making blockers emerge through exploration and embedding multiple independent information gaps per task.

  • Benchmark paradigms: Agent benchmarks predominantly provide complete specifications and score silent execution across coding, repository repair, and longer-horizon agent tasks.These settings reward confident autonomy rather than decisions about whether to request help.
  • Benchmark paradigms: Clarification and interactive benchmarks evaluate whether models identify ambiguity across dialogue, question answering, constraint satisfaction, simulation, and related settings, not selective escalation.The passage distinguishes ambiguity detection from deciding when and how to ask for help.
  • Structural gaps: HiL-Bench requires progressive discovery because prior benchmarks typically expose ambiguity upfront rather than through exploration or execution.Agents must work to uncover what they do not know instead of receiving visible ambiguity at task start.
  • Structural gaps: HiL-Bench embeds 3–5 independent blockers per task, whereas prior benchmark tasks usually contain at most one information gap.Each blocker demands a separate, targeted question to resolve.

3 HIL-BENCH Design

HiL-Bench converts well-specified SWE and text-to-SQL tasks into progressive judgment challenges by injecting independently validated blockers that agents must discover and resolve through targeted questions. Its reproducible human-oracle interface and Ask-F1 metric measure whether agents ask precisely when execution exposes unresolvable information gaps.

  • Blocker construction: Each task contains 3–5 realistic blockers, with missing, ambiguous, or contradictory information removed or obscured and resolutions stored in a blocker registry.A blocker records its type, information gap, exact resolution, and varied trigger questions.
  • Task domains: Tasks span SWE-Bench Pro and BIRD Text-to-SQL, selected so top models reach approximately 85% pass@3 with full information.This selection reduces confounding from baseline capability limits when evaluating blocked conditions.
  • Progressive discovery: Blockers surface during execution and environment exploration rather than upfront inspection, requiring agents to encounter gaps, ask targeted questions, incorporate answers, and continue.With full environment access, Claude Opus 4.6 achieved 61% blocker recall versus 11% in a spec-only SQL ablation; contradictory blockers dropped least because conflicts often appear in specifications.
  • Human-oracle interface: The ask human(question: str) -> str tool uses a frozen Llama-3.3-70B-Instruct semantic judge to return a registered resolution for targeted questions and “irrelevant question” otherwise.This creates a binary, reproducible signal for evaluating question relevance.
  • Evaluation metric: Ask-F1 combines question precision and blocker recall through a harmonic mean, penalizing question spam while distinguishing under-asking from imprecise escalation.High precision with low recall indicates under-asking, whereas low precision with high recall indicates question spamming.

4 Experiments

Experiments show a large, domain-general judgment gap: models perform strongly with complete information but rarely resolve blockers when they must decide whether to ask. Failure traces reveal stable model-family signatures, while RLVR improves help-seeking quality and task performance across domains.

  • Experimental setup: Models were evaluated in SWE-Agent scaffolding across baseline, full-information, and tool-enabled conditions using pass@3 and Ask-F1.The evaluated models were GPT 5.4, GPT 5.3 Codex, Claude Opus 4.6, and Gemini 3.1 Pro.
  • Judgment gap: 86–91% pass@3 on SQL and 64–88% on SWE under full information fell to 38% on SQL and 12% on SWE when models chose whether to ask.Ask-F1 averaged 40.5% on SQL and 37.4% on SWE, showing judgment—not raw capability—limited performance.
  • Help-seeking patterns: GPT models showed low recall and premature implementation, Gemini traded precision for broad SQL questions, and only Claude achieved reasonable SQL calibration.No model exceeded 50% recall on SWE, where confident best guesses were especially common.
  • Failure analysis: Analysis of 3,600+ failure traces found stable model-family signatures across domains, indicating model-level properties rather than task artifacts.The taxonomy covered tool use, logic, and alignment, including accuracy, self-assessment, strategy, and completion failures.
  • Failure signatures: GPT persisted with wrong beliefs, Claude detected uncertainty without resolving it, and Gemini showed domain-sensitive external correctability.Claude submitted despite recognizing being stuck in 45% of alignment failures; Gemini’s SQL tool-use completion failures dropped 38pp with ask human().
  • RLVR training: Shaped Ask-F1 rewards improved held-out precision, recall, and pass@3, with positive cross-domain transfer, establishing judgment as trainable with verifiable rewards.RLVR shifted models toward calibrated help-seeking, and improved Ask-F1 correlated with improved pass@3.

5 Conclusion … C.3 Blocker Registry Construction

HiL-BENCH exposes a universal judgment gap: frontier agents perform well with complete information but often fail when they must decide whether to ask. Its blocker-construction pipeline uses human validation, structural checks, and targeted registry entries to make help-seeking judgment measurable and trainable.

  • 5 Conclusion: 75–89% pass rates with complete information collapse to 4–24% when agents must decide whether to ask, despite possessing coding and escalation capabilities.The reported gap is attributed to judgment rather than raw capability, and failure analysis characterizes it as model-level.
  • 5 Conclusion: ASK-F1 combines question precision and blocker recall, preventing agents from gaming detection-only evaluations through indiscriminate question spam.Shaped ASK-F1 reinforcement learning also improves a 32B model’s help-seeking calibration and task completion across domains.
  • 5 Conclusion: Selective escalation, rather than full autonomy, is the production goal because agents cannot infer all context held in human or organizational knowledge.HiL-BENCH is presented as the benchmark designed to test whether agents recognize when human input is necessary.
  • A Blocker Taxonomy: The blocker taxonomy defines three information-gap types with representative examples from both SWE and SQL tasks.The taxonomy is documented with definitions and concrete dataset examples.
  • B Blocker Quality Criteria: Every blocker must satisfy seven quality criteria so evaluation measures help-seeking judgment rather than general problem-solving or brute-force search.Criteria include realism, criticality, and a vast search space that prevents resolution through guessing, enumeration, or extensive searching.
  • C.1 Annotation Structure: Trained domain experts inject realistic information gaps into task specifications and environments, while independent auditors and automated checks validate them through repeated review.Each task receives 5–6 layers of human review, with additional automated review as needed.
  • C.2 Structural Invariants: Automated evaluation verifies correct task-environment setup and ground truth, including clean setup patches and safeguards against agents retrieving answers from commit history.These structural invariants apply alongside baseline and full-information pass-rate-based task selection.
  • C.3 Blocker Registry Construction: Each blocker registry entry records its type, standalone information gap, exact resolution, and targeted trigger questions constrained to be self-contained and non-self-referential.Trigger questions must focus on the blocker’s core information gap rather than adjacent concerns.

D Judge Validation · D.1 Partial and Multi-Blocker Questions · D.2 Design Rationale

The ask human() judge is validated as a precise, recall-oriented evaluator, while explicit boundary-case rules and deterministic responses make Ask-F1 assessment reproducible. Its design distinguishes narrowly targeted, partially overlapping, and multi-blocker questions without rewarding vague escalation.

  • D Judge Validation: 97% precision and 91% recall were achieved by the ask human() judge on held-out human-annotated question-blocker pairs.Two annotators labeled relevance independently, with disagreements resolved by a third.
  • D Judge Validation: The benchmark prioritizes precision because false negatives undermine evaluation, whereas false positives only modestly inflate scores.Tasks below 80% per-task recall receive trigger-phrase expansion and re-validation until reaching a final floor of ≥85%.
  • D.1 Partial and Multi-Blocker Questions: Partially overlapping questions are accepted when a helpful human would naturally provide the blocker’s resolution from the exact question asked.The rule follows Ask-F1’s precision incentive for boundary cases.
  • D.1 Partial and Multi-Blocker Questions: Multi-blocker questions match at most one blocker—the most directly targeted—leaving other blockers unaddressed.This preserves the benchmark’s intended precision incentive.
  • D.1 Partial and Multi-Blocker Questions: Overly broad questions are judged irrelevant when they discuss a general topic without isolating a specific information gap.This includes vague or high-level implementation questions when only one parameter is blocking.
  • D.2 Design Rationale: The judge returns the exact resolution for registered blockers and otherwise returns “irrelevant question,” enforcing a strict response structure.This avoids free-form simulation confounds and supports deterministic, reproducible evaluation.
  • D.2 Design Rationale: A frozen open-source model guarantees stability and replicability of the judge’s evaluations.The fixed model complements the strict response format.

E Dataset Statistics · F Failure Taxonomy

The paper reports dataset statistics for HiL-Bench and analyzes more than 3,600 failure traces through a two-axis taxonomy. Failure-mode labeling used an LLM judge with high self-agreement and ongoing human calibration.

  • E Dataset Statistics: Dataset statistics for HiL-Bench are presented in Table 3.
  • F Failure Taxonomy: The failure analysis covers 3,600+ failure traces.
  • F Failure Taxonomy: Failures are classified by the capability dimension in which they occur.
  • F Failure Taxonomy: Failures are also classified by the failure mode within each capability dimension.
  • F Failure Taxonomy: The analysis supports high-level model archetypes presented in Section 4.3.
  • F Failure Taxonomy: An LLM judge, Sonnet-4, assigned the failure modes using independently developed rubrics.
  • F Failure Taxonomy: The judge achieved κ = 0.928 inter-run self-agreement, while random subsets were manually reviewed to maintain human calibration.
  • F Failure Taxonomy: Rubric definitions were iteratively refined until spot-checks found no systematic disagreements with human judgment.

F.1 Capability Dimensions and Failure Modes … F.2.2 Alignment

The paper distinguishes capability failures in tool use, alignment, and logic, then quantifies model-specific SQL failure patterns. Tool-use errors center on incorrect or missing actions, while alignment failures include pursuing the wrong goal, fabricating information, and submitting despite recognized errors.

  • F.1 Capability Dimensions and Failure Modes: Tool-use failures arise when agents select incorrect tools or invoke them with incorrect parameters.These failures span completion, accuracy, and strategy modes.
  • F.1 Capability Dimensions and Failure Modes: Completion failures occur when agents explore but never make the critical execution call, consuming their step budget in analysis loops.The SQL example is failing to call execute sql or its equivalent.
  • F.1 Capability Dimensions and Failure Modes: Accuracy failures involve hallucinated tools or correct tools used with malformed SQL, incorrect columns, or wrong table names.Strategy failures instead use an incorrect tool sequence and are relatively rare.
  • F.1 Capability Dimensions and Failure Modes: Alignment failures occur when agents pursue the wrong objective or fabricate information to complete the task.Examples include silently dropping a requirement and constructing mappings to nonexistent database columns.
  • F.1 Capability Dimensions and Failure Modes: Self-assessment failures occur when agents recognize explicit evidence that their output is wrong yet submit it anyway.Examples include submitting an empty result after reasoning that a SQL task is impossible and double-submitting after failed SWE tests.
  • F.1 Capability Dimensions and Failure Modes: Logic failures arise when agents reason incorrectly from correct premises, including wrong beliefs, poor self-evaluation, incomplete execution, or fundamentally flawed strategies.Agents may declare tasks satisfied despite incorrect reasoning or stall after partial implementation.
  • F.2.1 Tool Use: 72–100% of GPT tool-use failures are Accuracy-dominant, while Claude is Completion-dominant at 82.5% and uses 2-5 times as many tokens as other models.Gemini shifts from 56% Completion at baseline to 82.4% Accuracy with ask human(), indicating that human access prompts action but not necessarily correct action.
  • F.2.2 Alignment: Claude’s baseline Self-Assessment rate is 45%, whereas GPT models and Gemini are more Strategy-dominant at 58–73%, with GPT 5.4 reaching 71–73%.Claude uniquely verbalizes task infeasibility while submitting anyway; GPT models and Gemini decompose tasks around the wrong goal with high confidence.

F.2.3 Logic

SQL logic failures reveal distinct model-level reasoning patterns: GPT models mostly apply incorrect beliefs consistently, Claude explores without committing, and Gemini misjudges adequacy or over-explores until external grounding intervenes.

  • Logic: 76–89% Accuracy characterizes GPT models’ SQL logic failures, indicating consistently applied wrong beliefs rather than planning or exploration breakdowns.The pattern remains stable across conditions, suggesting incorrect premises form upstream of reasoning.
  • Logic: 21–25% Completion in Claude’s logic failures mirrors its tool-use behavior of extensive exploration without execution commitment.The shared pattern suggests the same underlying behavior across both capabilities.
  • Logic: 24.2% Self-Assessment makes Gemini the SQL-logic leader, while Completion falls from 22.2% baseline to 7.2% with ask human() and 0% with full information.This indicates recurring belief in inadequate outputs despite contrary evidence, while external grounding curbs over-exploration.

F.3 Domain Asymmetry: SWE vs. SQL

SWE and SQL expose different facets of the judgment gap while retaining model-level signatures. SQL shows diverse alignment failures, whereas SWE failures homogenize around logic, tool use, and incomplete execution.

  • SQL: SQL blockers arise from task questions, schema descriptions, test queries, and business logic, producing diverse alignment failures across tools, methods, and assumptions.These properties expose model-family differences cleanly.
  • SWE: SWE failure modes homogenize across models and concentrate heavily in logic and tool use.Figure 7 reports that SWE failure distributions congregate toward the same failure subtypes across models.
  • SWE: Under ask human(), SWE traces shift strongly toward Completion, including partial implementations, skipped verification, and incomplete submissions.This suggests models struggle to leverage the ask human() tool effectively.
  • SWE: SWE Self-Assessment is common, especially in GPT 5.3 Codex and Gemini, but agents submit despite failure evidence rather than explicitly stating infeasibility.The passage contrasts SWE’s implicit behavior with SQL’s verbalized infeasibility.

G RLVR Training Details · G.1 Model and Infrastructure · G.2 Reward Formulation

RLVR fine-tunes Qwen3 32B separately on SQL and SWE tasks, using LoRA and SkyRL. Its reward decomposes Ask-F1 into per-step precision-like feedback and terminal recall-like coverage, supplying dense training signal while preventing ask-avoidance.

  • G.1 Model and Infrastructure: Qwen3 32B is fine-tuned with LoRA using SkyRL, separately for SQL and SWE domains.Each domain uses 120 training tasks and 30 held-out evaluation tasks.
  • G.1 Model and Infrastructure: The training setup retains the SWE-Agent framework and tool access used in the frontier-model evaluations.This keeps the training environment aligned with the evaluation infrastructure.
  • G.2 Reward Formulation: Total reward is R = ∑t rstep(qt) + rterminal, combining per-step and terminal components.This decomposition preserves Ask-F1’s precision–recall structure and provides dense gradient signal unavailable from pure terminal Ask-F1 reward.
  • G.2 Reward Formulation: Per-step rewards assign +0.3 when a question targets a registered blocker.The reward is defined for each ask human() invocation.
  • G.2 Reward Formulation: Per-step rewards assign −0.1 when a question is irrelevant or duplicates a resolved blocker.This penalizes noisy or redundant escalation.
  • G.2 Reward Formulation: The asymmetric +0.3 versus −0.1 weighting encourages question-space exploration while penalizing noise, providing a precision-like per-step signal.The asymmetry balances exploration against irrelevant questioning.
  • G.2 Reward Formulation: Terminal coverage rewards blockers in Bdiscovered ⊆ B for which the agent asked at least one relevant question.The gate condition prevents policies that avoid asking, while terminal coverage supplies a recall-like signal.

H Complementary Metrics · I Full Leaderboard

The paper supplements Ask-F1 with measures of questioning and token usage across task conditions, and reports a combined SWE-and-SQL Pass@3 leaderboard. These metrics reveal distinct exploration and escalation styles across models.

  • H Complementary Metrics: Complementary evaluation reports average questions asked per task and average tokens sent per task across baseline, full-information, and tool conditions.These measures supplement Ask-F1 with broader behavioral coverage.
  • H Complementary Metrics: GPT models use the fewest tokens, reflecting fast and overconfident execution, while Gemini is intermediate and Claude uses up to five times as many tokens.The token pattern is consistent with GPT executing quickly, Gemini occupying the middle, and Claude exploring and reflecting extensively.
  • H Complementary Metrics: GPT models ask the fewest questions in both SWE and SQL domains.Question frequency distinguishes GPT from the other model families but does not itself establish task quality.
  • H Complementary Metrics: Gemini tends to over-ask on SQL.Its questioning behavior is domain-dependent rather than uniformly high or low.
  • H Complementary Metrics: Claude occupies a middle position in questions asked, but higher questioning frequency does not necessarily yield better integration of blocker resolutions.Question volume alone is therefore insufficient to characterize effective help-seeking.
  • I Full Leaderboard: The full leaderboard reports Pass@3 with the ask human() tool combined across SWE and SQL.The supplied passage identifies the leaderboard metric and aggregation scope but provides no numerical entries.
Loading 2604.09408v4…