PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask, Relax, or Act? Evaluating Actionable Indeterminacy in LLM Preference Reasoning

Ang Li, Yue Lin, Feifei Kou, Zhan Su, Prayag Tiwari, Wenhao Li, Shuhui Zhu, Hongyuan Zha, Baoxiang Wang

arXiv:2610.03102v1cs.CLcs.AI

TL;DR

The paper asks when uncertainty prevents action rather than merely being present, and formalizes actionable indeterminacy in a solver-grounded benchmark. Models often intervene when action is already justified, while explicit response requirements can improve fully correct responses and change intervention decisions.

  • Problem

    Models may ask when a justified action already exists or seek clarification when constraints instead make a request infeasible, so uncertainty alone does not determine the right next step.

    Source: §1 · p. 1

  • Method

    The paper formalizes actionable indeterminacy and evaluates matched action/intervention pairs across four decision structures using solver-determined labels and fixed response-content checks.

    Source: §1 · p. 2

  • Results

    Models repeatedly intervene when action is already justified, and explicit response requirements substantially improve fully correct responses and can change intervention decisions, even when outputs are parseable.

    Source: §1.1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    Reliable agency requires intervening only when necessary and translating the chosen next step into a verifiable response.

    Source: §6 · p. 9

    The H5 local-cue collection is separately constructed and includes a different allocation-constraint family.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

An LLM agent can recognize uncertainty yet still choose the wrong next step: asking when action is already justified, or seeking clarification when the constraints must change. We formalize actionable indeterminacy: act when an accepted action is shared across all admissible preferences or objectives, clarify when each possibility is feasible but no action is shared, and propose a minimum-cost permitted constraint repair when the request is infeasible. We construct a solver-grounded benchmark spanning object allocation, meeting scheduling, apartment choice, and stable matching. Matched pairs retain the same source while changing whether intervention is necessary, and evaluation separates decision correctness, matched-pair reliability, and fully correct responses. Our findings reveal a recurring difficulty in recognizing when intervention is unnecessary: models can identify situations requiring clarification or repair yet still intervene when a justified action already exists. Correct decision labels also fail to guarantee usable actions, questions, or repairs. Crucially, response requirements shape not only how decisions are expressed but also which decisions are made. Making the required content explicit substantially improves fully correct responses and can change intervention decisions, even when outputs are already parseable. These findings highlight that reliable agency requires more than recognizing uncertainty: it requires intervening only when necessary and translating the chosen next step into a verifiable response.

1 INTRODUCTION

The paper asks when uncertainty actually blocks action, distinguishing it from cases where a justified action remains available. It formalizes actionable indeterminacy and evaluates whether models choose and express the warranted next step.

1 INTRODUCTION

  • The central challenge is deciding whether uncertainty prevents action, because unnecessary questions, premature action, and clarification of infeasible requests require different remedies.Recognizing missing information does not by itself show that it is needed.Source: §1 · p. 1
  • Unresolved preferences or objectives need not block action when all admissible cases share an accepted action; otherwise clarification is needed, while infeasibility calls for constraint repair.The apartment example contrasts shared optimal choices with cases where no action is accepted across possibilities and preference information cannot remedy infeasible constraints.Source: §1 · p. 1
  • The benchmark formalizes ACT, clarification, and minimum-cost permitted repair across four decision structures, using matched pairs that vary whether intervention is necessary.Exact solvers determine warranted labels, and fixed checkers assess required answer content without an LLM judge.Source: §1 · p. 2

1.1 MAIN RESULTS

  • Models can recognize cases requiring clarification or repair yet still intervene when action is already warranted, making both sides of matched pairs important to evaluate.For GPT-6-Astra and Claude-5-Sonnet, most original-prompt decision failures occur on tasks where action is warranted.Source: §1.1 · p. 2
  • Explicit response requirements substantially improve decision success on the same tasks, and explicit content checks largely close the gap between correct labels and accepted responses.The aggregate improvement for GPT-6-Astra cannot be attributed to JSON syntax alone because its outputs are already parseable in every condition.Source: §1.1 · pp. 2–3

1.2 RELATED WORK

  • The related-work discussion positions shared-action evaluation as complementary to confidence-based uncertainty and dialogue-cost approaches, and distinguishes decision selection from satisfying response requirements.The shared-action criterion bases intervention necessity on admissible possibilities; the response-evaluation distinction concerns whether the action, question, or repair meets requirements.Source: §1.2 · p. 3

2 PROBLEM FORMULATION AND METHODOLOGY

The paper defines actionability by considering accepted actions across admissible preferences, objectives, or constraints. It tests this distinction with solver-grounded tasks and separate decision and response-content measures.

2.1 PROBLEM FORMULATION

  • The benchmark covers object allocation, meeting scheduling, apartment choice, and stable matching, with finite action spaces and task-specific accepted-action rules.Accepted actions may include all tied optima, a specified algorithm’s output, or all feasible actions.Source: §2.1 · p. 3
  • Each task varies one uncertainty source—preferences, objectives, or constraints—while holding the remaining preferences and objectives fixed.Complete rankings consistent with observed preferences are denoted Comp(P).Source: §2.1 · p. 3
  • A nonempty shared accepted-action set warrants ACT; if feasible cases share no action, ask a pivotal question, while infeasible constraints require a minimum-cost permitted repair and valid action.The rules treat preference, objective, and constraint sources separately and do not prioritize simultaneous sources.Source: §2.1 · p. 4

2.2 METHODOLOGY AND EXPERIMENTAL SETUP

  • Each V3 group contains three source-matched action/intervention pairs, for six tasks; the common evaluation includes 4,320 prompts across 36 cells.Exact solvers supply labels, while fixed checkers assess responses; local-cue and output-requirement comparisons use distinct sampling designs.Source: §2.2 · p. 4
  • V3 evaluates four models, while the first three also cover H2 and H5; primary comparisons are within model because inference settings and collection conditions differ.Single-turn requests use fresh conversations without tools or private labels; task-completion follow-ups retain conversation history.Source: §2.2 · p. 4
  • Accuracy measures decision-label correctness, IFC requires both decisions in a matched pair to be correct, and FC additionally requires accepted response content.Unparseable completed responses and unresolved OSS truncations count as unsuccessful, with truncations treated as delivery failures rather than observed reasoning errors.Source: §2.2 · p. 4

3 WHEN IS INTERVENTION NECESSARY?

The results distinguish individual decisions, matched-pair reliability, accepted response content, and completion after intervention. Models often struggle to avoid unnecessary intervention, and performance varies by structure, source, and response requirements.

3.1 INDIVIDUAL DECISIONS AND MATCHED-PAIR RELIABILITY

  • 92.57% Accuracy and 85.56% IFC are achieved by GPT-6-Astra on the common 4,320-prompt subset, while all six decisions are correct in 65.00% of groups.The three measures distinguish individual response-category correctness, joint correctness across matched pairs, and group-level success.Source: §3.1 · p. 4
  • 85.28% is Astra’s descriptive independence reference for IFC, close to its observed 85.56%, while matched-pair evaluation measures joint success across action and intervention decisions.The reference is calculated from action- and intervention-member accuracies within each cell and source.Source: §3.1 · p. 4

3.2 ERRORS DEPEND ON STRUCTURE AND SOURCE

  • 303 of Astra’s 321 and 468 of Claude’s 551 decision errors occur on tasks where action is warranted, making unnecessary intervention their dominant difficulty.Both models correctly identify most constructed intervention-needed cases, yet often fail to act when available information already suffices.Source: §3.2 · pp. 4–5
  • Meeting scheduling and apartment choice are handled more reliably than object allocation and stable matching, while source profiles vary across models.Preference pairs have the highest IFC for Astra and Gemini; constraint pairs do for Claude and OSS, and objective pairs are especially difficult for Claude and OSS.Source: §3.2 · p. 5

3.3 CORRECT LABELS DO NOT ESTABLISH EXECUTABLE ANSWERS

  • Correct decision labels can still fail required-content checks; Astra’s 720 RELAX labels are correct, yet none passes the complete repair-and-action check.The responses omit or null the required top-level action, and the original wording does not consistently specify the representation expected by the checker.Source: §3.3 · pp. 5–6

3.4 TASK COMPLETION AFTER INTERVENTION

  • The intervention-completion test separates producing an accepted first intervention from solving the updated task after feedback.It evaluates 540 intervention-needed tasks; a valid intervention receives feedback and a final-action request, while an invalid first response gets no second turn.Source: §3.4 · p. 6
  • Astra completes 99.07% of tasks autonomously, near the 99.81% reference rate; Claude’s failures occur both before accepted intervention and after valid intervention.Gemini’s 501 of 540 first responses are rejected because Markdown code fences wrap the JSON, preventing feedback and a second turn.Source: §3.4 · p. 6

4 DO LOCAL CUES EXPLAIN THE ERRORS?

The authors test whether locally matched action/intervention pairs eliminate errors and whether response requirements affect decisions. Errors persist, while explicit JSON improves decision success across models, with different source-level patterns.

4.1 LOCAL-CUE-MATCHED PAIRS

  • The H5 collection matches selected source-dependent features while changing whether intervention is necessary, preventing a deterministic rule over those identical features from distinguishing pair members.Objective pairs preserve objective menus and per-objective solution counts; allocation-constraint pairs preserve agent and object degrees while differing in whether a complete assignment exists.Source: §4.1 · p. 6
  • Five pairs per source and cell were selected using a fixed seed, producing 540 pairs and 1,080 tasks answered under original and explicit JSON requirements.The four requests per pair were randomized and used fresh conversations; each model contributed 2,160 responses.Source: §4.1 · p. 6

4.2 MATCHED CUES DO NOT MAKE ERRORS FORMAT-INVARIANT

  • Errors persist on locally matched tasks, and their frequency changes markedly with response requirements.The improvement concerns decision-label correctness, not only whether accompanying response content passes fully correct checks.Source: §4.2 · pp. 6–7
  • GPT-6-Astra’s IFC rises from 76.67% to 98.33% with explicit response requirements, while aggregate decision gains also replicate in Claude-5-Sonnet and Gemini-3.1-Pro.Astra’s largest IFC gain is on preference pairs; its constraint decisions are already perfect in both conditions. Claude and Gemini benefit most on constraint pairs, while objective pairs remain difficult.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

5 HOW MUCH DO OUTPUT REQUIREMENTS MATTER?

The paper compares response requirements and answer-construction interventions on fixed task sets. Explicit content requirements substantially improve fully correct responses and can change decisions, while candidate answers and intervention reminders produce distinct, nonuniform effects.

5.1 THREE REQUIREMENTS FOR THE SAME TASKS

  • Explicit requirements specify allowed decision strings, fields, action representations, and source-specific question and correction structures; evidence requests add task-derived witnesses but no gold answers or labels.Evidence is checked separately and does not enter FC; the comparison varies solution-content requirements, not merely punctuation.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 MOST EXECUTION FAILURES DISAPPEAR WITH EXPLICIT REQUIREMENTS

  • For GPT-6-Astra, FC rises from 50.79% to 94.40% with explicit requirements and reaches 97.45% when evidence is additionally requested.Claude and Gemini show the same aggregate pattern; correct-label responses failing content checks become rare for Astra, disappear for Gemini in both explicit conditions, and decrease sharply for Claude.Source: Table 1 · p. 8, §5.2 · p. 8
  • A parseable response can still omit a required action, underspecify a clarification question, or fail to express an accepted repair.All Astra responses are already parseable in every condition, so the improvement addresses a mismatch beyond JSON syntax alone.Source: §5.2 · p. 8

5.3 JUDGMENT ALSO CHANGES, BUT NOT EQUALLY ACROSS SOURCES

  • Across the same tasks, Accuracy and IFC improve with explicit requirements and improve further in aggregate when evidence is requested, showing that response specification affects selected decisions as well as response content.Effects vary by source and model: explicit JSON improves Astra’s preference Accuracy but lowers its constraint Accuracy, while adding evidence reverses those directions in part.Source: §5.3 · p. 8

5.4 SEPARATING ANSWER GENERATION FROM INTERVENTION REMINDERS

  • Candidate-answer selection improves Astra and Gemini in aggregate but worsens Claude; all three improve on constraint FC, while Claude’s preference and objective performance decline and unnecessary interventions nearly double.Intervention reminders add only modest gains after explicit requirements: unnecessary interventions decrease slightly for all three models, while missed interventions decrease for Claude and remain unchanged for Astra and Gemini.Source: §5.4 · pp. 8–9, Table 2 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The conclusion presents actionable indeterminacy as a way to distinguish uncertainty that permits action from obstacles needing clarification or repair. It argues that reliable evaluation must assess decisions, matched-pair consistency, and whether the response content is verifiable.

6 CONCLUSION

  • The solver-grounded benchmark tests actionable indeterminacy through matched action/intervention pairs across four decision structures.The framework distinguishes uncertainty that can remain unresolved from obstacles requiring clarification or constraint repair before action.Source: §6 · p. 9
  • Under original prompts, models can recognize necessary interventions yet still intervene when action is already justified.Explicit response requirements shape both selected decisions and whether accompanying content is accepted, even when outputs are already parseable.Source: §6 · p. 9
  • The authors conclude that decision correctness, matched-pair reliability, and response-content validity should be evaluated separately.Their central lesson is that reliable agency requires selecting the warranted next step and expressing it in a verifiable form.Source: §6 · p. 9

AI USE STATEMENT

The supplied material combines an AI-use statement, related-work positioning, and benchmark construction details. It situates the work among decision-focused elicitation, clarification, repair, ambiguity, and verifiable-response research.

AI USE STATEMENT

  • The authors used generative AI tools for language polishing and code assistance and take responsibility for the final work.Source: AI USE STATEMENT · p. 9

A EXTENDED RELATED WORK

  • The shared-action criterion builds on preference elicitation: an action can remain accepted across all admissible possibilities despite incomplete information.A nonempty intersection need not imply a unique outcome or identical solution sets, and differs from general minimax-regret selection.Source: A EXTENDED RELATED WORK · p. 19
  • The benchmark complements clarification research with finite source-matched pairs and exact-solver labels, making missed and unnecessary interventions separately observable.It tests intervention necessity under specified rules rather than optimizing a general dialogue-cost objective.Source: A EXTENDED RELATED WORK · p. 19
  • The RELAX branch requires a minimum-cost permitted repair with a valid resulting action, distinct from finding a minimal conflict or preference-maximal consistent set.Source: A EXTENDED RELATED WORK · p. 19
  • The benchmark evaluates task-level intervention policy rather than treating confidence estimates as ground truth or IFC as a calibration metric.Recognizing ambiguity alone does not establish that intervention is necessary; labels depend on task information and accepted actions.Source: A EXTENDED RELATED WORK · p. 19

B V3 CONSTRUCTION AND COMPLETE RESULTS

B.1 A WORKED EXAMPLE: SIX MATCHED APARTMENT-CHOICE TASKS

The apartment examples show that intervention is needed only when uncertainty blocks a shared accepted action or hard constraints make the request infeasible. The accompanying benchmark results show that decision correctness does not ensure fully correct responses.

B.1 A WORKED EXAMPLE: SIX MATCHED APARTMENT-CHOICE TASKS

  • Adding egalitarian aggregation changes the shared optimum from A4 to none, so the task changes from ACT to ASK-OBJECTIVE rather than using a majority vote over rules.The clarification asks the user to distinguish utilitarian from egalitarian aggregation.Source: §B.1 · p. 21
  • When rent is at most 10,000 and commute at most 18 minutes, A3 is feasible and ACT is warranted; tightening rent to 7,000 makes the request infeasible and calls for RELAX.The certificate gives two permitted minimal repairs: raise rent to 10,000 and return A3, or raise commute to 52 minutes and return A4; minimality is relative to allowed edits.Source: §B.1 · p. 22
  • A1 remains the ACT choice when the hidden A3/A1 order changes no optimum, whereas hiding A1/A2 yields divergent optima and requires ASK-PREFERENCE.The two omissions concern the same resident, but only the latter prevents a robust choice.Source: §B.1 · p. 21, Table 4 · p. 22

B.2 V3 RESULTS AND GEMINI REPLICATION

  • 91.50% Accuracy, 83.01% IFC, and 51.23% FC on Gemini’s 4,320-task subset; all-six FC was 0.83%.Unnecessary intervention occurred on 11.30% of action tasks, while missed intervention occurred on 5.51% of intervention-needed tasks.Source: Table 5 · p. 23, §B.2 · p. 23
  • The inherited Gemini collection combines response batches assigned by prior completion, so batch differences cannot isolate an access-interface effect.The inherited AGY records lack an independently verifiable served-model version.Source: §B.2 · p. 23

B.3 RESULTS ACROSS GENERATOR SEEDS

  • Across three generator seeds, Accuracy and IFC order Astra > Claude > OSS, but FC orders Astra > OSS > Claude.The aggregate comparisons are stable across seeds; OSS differences also include variation in incomplete deliveries.Source: Table 9 · p. 24, §B.3 · p. 24

B.4 OSS V3: FULL COLLECTION AND MISSING RESPONSES

  • 34.71% is the optimistic FC ceiling under OSS truncation; all-six FC remains zero because every group already contains an observed failure.The bounds describe missing outcomes, not sampling uncertainty or confidence intervals.Source: Table 10 · p. 25, §B.4 · p. 25
  • 149 of 185 OSS truncations occur at n = 8 and 115 in allocation, so the complete-response subset is not an unbiased replacement for the full collection.On the common subset, full-denominator Accuracy, IFC, and FC are 83.54%, 70.37%, and 32.52%, respectively.Source: Table 11 · p. 25, §B.4 · p. 25

C INDEPENDENCE REFERENCE AND ERROR COUNTS

The section contrasts observed matched-pair correctness with a descriptive independence reference, then examines how rejected response content persists despite correct labels. A separate same-task comparison reports large gains in Claude’s fully correct responses under explicit output requirements.

C INDEPENDENCE REFERENCE AND ERROR COUNTS

  • 85.56% observed IFC versus an 85.28% reference for Astra (+0.28 points); Claude’s observed 76.39% is below its 76.48% reference (−0.09 points).The reference is computed descriptively from the same observed responses.Source: C INDEPENDENCE REFERENCE AND ERROR… · p. 26
  • The archived V3 categories include 2,194 FC responses and 309 wrong decisions, alongside rejected actions, questions, and relaxation content despite correct labels.The fixed checker rejects invalid content; question and relaxation categories are affected by original output wording.Source: C INDEPENDENCE REFERENCE AND ERROR… · p. 26

C.1 CLAUDE V3 RESPONSE CONTENT AND H2 COMPARISON

  • Claude’s 4,320 V3 responses include 1,344 FC, 551 incorrect labels, and 2,425 correct labels with rejected content.Rejected content includes 700 action answers, 525 preference questions, 530 objective questions, and 670 relaxation answers; none failed parsing.Source: Table 12 · p. 26, §C.1 · p. 26
  • FC rises from 695/2,160 (32.18%) to 1,882/2,160 (87.13%) and 1,983/2,160 (91.81%) across Claude H2’s original, explicit-JSON, and evidence conditions.The comparison is within H2’s matched tasks; its subset and stable-matching repair acceptance rule differ from V3.Source: §C.1 · p. 26

D LOCAL-CUE CONSTRUCTION AND ADDITIONAL RESULTS

The section tests whether matched local cues and explicit response requirements affect decisions and response validity. Explicit JSON improves Gemini’s aggregate metrics, while the detailed results show gains are not uniform across tasks or models.

D LOCAL-CUE CONSTRUCTION AND ADDITIONAL RESULTS

  • After matching selected local cues, Astra’s decisions shift 142 incorrect-to-correct versus two correct-to-incorrect; pair counts shift 118 versus one.The H5 comparison uses the same logical tasks in original and explicit conditions, with output requirements appended to the latter.Source: D LOCAL-CUE CONSTRUCTION AND… · p. 27
  • Claude’s H5 FC transitions are 662 non-FC-to-FC versus 20 reversals, while decision transitions include 100 incorrect-to-correct and 29 correct-to-incorrect.The difference is not principally recovery from unparseable JSON; FC follows the H5 acceptance rules.Source: D LOCAL-CUE CONSTRUCTION AND… · p. 27

D.1 GEMINI-3.1-PRO REPLICATION

  • Explicit JSON improves Gemini aggregate Accuracy by 2.87 percentage points, IFC by 5.74, and FC by 37.69.The gains are not uniform: objective Accuracy falls by one correct decision, apartment Accuracy is unchanged, and meetings and allocation each lose one correct decision.Source: Table 18 · p. 30, §D.1 · p. 30

E OUTPUT REQUIREMENTS AND SOURCE-SPECIFIC RESULTS

  • Explicit JSON specifies decision, action, question, and relax fields, including the compared alternatives or admissible objectives for clarification.The added evidence condition requests a certificate, but evidence validity is checked separately from FC.Source: E OUTPUT REQUIREMENTS AND… · p. 30, Table 19 · p. 31
  • Astra’s outputs are parseable in every condition and meet checked field structure in both explicit conditions, yet 2,094 of 2,160 evidence responses pass their separate checks.This separates response-content requirements from JSON parseability and keeps evidence checks distinct from FC.Source: E OUTPUT REQUIREMENTS AND… · pp. 30–31

E.1 GEMINI-3.1-PRO SAME-TASK OUTPUT REQUIREMENTS

The Gemini same-task replication compares response requirements while checking recorded answers and fixed scoring rules. Its findings distinguish response-content failures from parsing problems and define separate measures for decision, pair, and content success.

E.1 GEMINI-3.1-PRO SAME-TASK OUTPUT REQUIREMENTS

  • The replication contains 6,480 answers to 2,160 fixed logical tasks under three requirements, with each condition covering 1,080 matched pairs.The original task text and system instruction were checked against the fixed collection; accepted responses were nonempty, ended normally, and contained no tool calls.Source: §E.1 · p. 31
  • FC improves by 44.95 percentage points with explicit JSON and 46.16 points with JSON plus evidence; correct decisions with invalid required content fall from 891 to zero in both conditions.Parsing succeeds for 2,159, 2,160, and 2,160 answers across the three conditions, so the original FC gap is not mainly a JSON-parsing failure.Source: §E.1 · p. 31, Table 21 · p. 31

F RESPONSE-CHECKER SCOPE

  • The fixed V3/H5 stable-matching RELAX checker accepts the stored repair–action pair, but may reject other valid solutions.Fully correct evaluates compliance with both task requirements and the specified response interface.Source: F RESPONSE-CHECKER SCOPE · p. 32

F.1 METRIC DEFINITIONS

  • Accuracy measures decision-label correctness, Fully correct additionally requires accepted response content, and Intervention Flip Correctness requires both decisions in a matched pair to be correct.All-six accuracy requires all six decision labels in a V3 group to be correct, without requiring all six responses to pass Fully correct.Source: §F.1 · p. 32
  • Scoring uses fixed checks, reports descriptive percentages and counts, and evaluates evidence validity separately from Fully correct.Unresolved OSS truncations remain in the full denominator as unsuccessful deliveries, rather than observed reasoning errors.Source: §F.1 · p. 32

G EARLIER MULTI-MODEL EXPERIMENTS

These earlier experiments provide supplementary comparisons from a smaller, three-structure collection and from independently generated OSS task sets. Their results include near-zero source–structure independence deviations and modest variation across replications.

G EARLIER MULTI-MODEL EXPERIMENTS

  • The earlier collection contains 150 six-task groups and 900 prompts from three structures at n = 4, rather than the four-structure V3 design.Its results are supplementary comparisons, not scores for the expanded OSS or Claude runs.Source: G EARLIER MULTI-MODEL EXPERIMENTS · p. 32, Table 22 · p. 33
  • Source–structure independence deviations are −0.5, −0.1, and +0.9 percentage points; objective pairs receive intervention responses on both members in 44.0%, 45.3%, and 20.7% of pairs, respectively.The small deviations distinguish joint success from additional pair dependence.Source: G EARLIER MULTI-MODEL EXPERIMENTS · p. 32
  • Across three independently generated 900-task OSS collections, Accuracy is 75.3%, 75.6%, and 75.1%, while IFC is 58.9%, 60.4%, and 60.2%.These are new-instance replications within a fixed setting, not repeated decoding of the same tasks.Source: G EARLIER MULTI-MODEL EXPERIMENTS · p. 33

H EARLIER OUTPUT AND INFEASIBILITY COMPARISONS

The historical comparisons examine output requirements and budgets, feasibility and task properties, and global-intersection judgments. Across these settings, better delivery or decision scores do not always translate into fully correct responses or reliable intervention choices.

H EARLIER OUTPUT AND INFEASIBILITY COMPARISONS

  • Raising the OSS completion cap from 2,048 to 3,072 tokens increases Accuracy by 7.8 points, IFC by 9.4, and Fully correct by 1.9 points.The earlier 360-task output comparison also reports fewer no-final responses and length finishes at the higher cap.Source: H EARLIER OUTPUT AND INFEASIBILITY… · p. 33, Table 24 · p. 33

H.1 CLAUDE FIRST-RESPONSE OUTPUT BUDGET

  • A larger Claude first-response cap raises Accuracy by 5.56 percentage points and IFC by 3.89 points, while Fully correct changes from five successes to four.The comparison uses the same 360 historical tasks under 2,048- and 3,072-token caps; the larger cap reduces cap termination and empty answers without improving Fully correct in this sample.Source: §H.1 · pp. 33–34, Table 25 · p. 34

I OTHER HISTORICAL ANALYSES

  • The sampled tie-free observations have 94.7% raw Accuracy versus 36.5% for tie-rich observations, an association within the sampled tasks.Equal weighting of 18 model–source–structure groups gives a difference of −51.6 points.Source: I OTHER HISTORICAL ANALYSES · p. 34
  • Across additional checks, earlier OSS analysis-first additions yield small Accuracy gains but lower Fully correct, while the 480-task Terra test gets every designed group correct.The Terra result bounds a particular explanation within that sample; Claude’s separate intersection replication finds unnecessary preference clarification on nonempty-intersection tasks.Source: I OTHER HISTORICAL ANALYSES · p. 34

I.1 CLAUDE REPLICATIONS OF HISTORICAL COMPARISONS

  • At n = 6 and 8, Claude selects RELAX on nearly every feasible task as well as every infeasible task, yielding approximately 50% Accuracy rather than successful feasibility discrimination.The panel crosses two fixed seeds with 20 feasible–infeasible pairs per seed–size cell; twelve responses are unparseable.Source: §I.1 · pp. 34–35, Table 27 · p. 35
  • Analysis-first does not improve any aggregate metric over decision-first, although switching to decision-first improves 19 decisions and worsens six.All responses are parseable; the FC transition counts are 23 improvements and eight reversals.Source: §I.1 · p. 35, Table 28 · p. 35

I.2 GLOBAL INTERSECTION VERSUS MAJORITY AND PAIRWISE OVERLAP

  • The 480-task intersection comparison supplies each possible world’s complete admissible-action set and requires ACT for a nonempty global intersection, otherwise ASK-OBJECTIVE.The abstract set comparison avoids deriving admissible actions from natural-language decision problems and is separate from the four-structure V3 collection.Source: §I.2 · p. 35
  • Overall Accuracy and Fully correct are both 93.96% (451/480), but all 29 errors occur on nonempty-intersection tasks where Claude unnecessarily selects ASK-PREFERENCE.All 240 empty-intersection tasks are correct; the errors occur when multiple common actions remain.Source: §I.2 · p. 35, Table 29 · p. 36

J TASK COMPLETION AFTER INTERVENTION

The completion experiments test whether models can turn a valid intervention into a final action, while the answer-selection comparison tests choosing among supplied candidates. Completion outcomes vary across models, and selection improves aggregate scores on some task sources.

J TASK COMPLETION AFTER INTERVENTION

  • The autonomous route requires a valid intervention before feedback and a final action afterward; an invalid first response receives no corrective hint, second turn, or retry.The reference route supplies predetermined feedback in a separate conversation, and relaxation routes can use different permitted changes.Source: J TASK COMPLETION AFTER INTERVENTION · pp. 36–37
  • Astra completes 535/540 tasks autonomously (99.07%) after intervention feedback, with 535/537 completions conditional on a valid first intervention.The reference route succeeds on 534 tasks jointly with the autonomous route, on five alone, and the autonomous route alone succeeds on one; all six failed route responses are parseable JSON.Source: J TASK COMPLETION AFTER INTERVENTION · p. 37, Table 30 · p. 37
  • Claude has 408/540 autonomous completions (75.56%) versus 516/540 reference completions (95.56%), with 408/427 completions after a valid first step.The 20.00-percentage-point aggregate difference is descriptive because routes can differ in history and permitted constraint changes.Source: J TASK COMPLETION AFTER INTERVENTION · p. 37, Table 32 · p. 38
  • Gemini has 6/540 autonomous completions (1.11%) and 29/540 reference completions (5.37%), with six successes among 39 valid first interventions.All 501 invalid first responses contain JSON inside Markdown code fences, which the bare-JSON parser rejects before feedback.Source: J TASK COMPLETION AFTER INTERVENTION · p. 38, Table 34 · p. 39

K.1 GENERATING VERSUS SELECTING AN ANSWER

  • Each selection menu contains two ACT responses and two interventions, with exactly one fully correct candidate; the model returns a choice that is scored for decision and content.Correct answer positions are balanced globally, and distractors vary in difficulty; supplied candidates reduce the construction burden while providing answer content.Source: §K.1 · p. 39
  • In the selection comparison, 25 tasks improve and 12 reverse for both Accuracy and Fully correct, a net gain of 13; unnecessary interventions fall from 45/540 to 35/540.Preference scores are unchanged in aggregate, while objective and constraint Fully correct gain three and ten tasks, respectively.Source: Table 36 · p. 40, §K.1 · p. 41

K.2 NEUTRAL WORDING VERSUS AN INTERVENTION REMINDER

This part compares neutral wording with an intervention reminder under unchanged H2 explicit-JSON prompts, documenting the task, scoring, prompt-construction, and benchmark rules. It uses archived model answers and a two-turn example to examine how decisions relate to unnecessary interventions, invalid actions, and distinct feedback routes.

K.2 NEUTRAL WORDING VERSUS AN INTERVENTION REMINDER

  • Gemini-3.1-Pro Accuracy and FC rise from 95.28% to 95.93%; Astra and Claude each show both corrected and reversed decisions under the reminder.Astra changed 16 incorrect decisions to correct and 12 the reverse; Claude changed 47 and 37, respectively. Gemini unnecessary interventions fell from 50 to 43, while missed interventions remained at one.Source: §K.2 · p. 41, Table 38 · p. 42

L INFERENCE SETTINGS AND FAILURE ACCOUNTING

  • The evaluation checks expected task coverage, prompt hashes, completion records, and recovery mappings; completed wrong answers remain included, while parsing failures count as incorrect.These behavioral denominators differ from service-reliability denominators that count failed requests; archived prompts, checker versions, settings, timestamps, and recovery mappings support reproduction.Source: L INFERENCE SETTINGS AND FAILURE… · pp. 43–44

M EXPERIMENTAL DESIGN AND VALIDATION

  • Selection and wording compare 1,080 tasks sampled from five H2 groups per cell; sampling uses no model responses or correctness, and H5 uses repeated-task pairs.The comparisons are scored for Astra, Claude, and Gemini; completion separately samples 540 intervention-needed tasks.Source: M EXPERIMENTAL DESIGN AND VALIDATION · p. 46

M.1 CONSTRUCTION AND CHECKING PROCEDURES

  • Task constructors compute accepted actions across preference completions or admissible objectives, test feasibility and minimum repairs, then validate each task’s oracle response.ACT requires a nonempty intersection of accepted-action sets; evaluation separates decision Accuracy, pair-level IFC, and answer-content FC.Source: §M.1 · p. 47

N REPRESENTATIVE EVALUATION PROMPTS

  • The representative prompt templates show selected instruction strings, while public task data vary and archived task sets remain authoritative for full prompts.Private labels and verification certificates are excluded from the public task sets.Source: N REPRESENTATIVE EVALUATION PROMPTS · p. 48

N.1 SHARED INSTRUCTIONS AND AN ORIGINAL TASK

  • The original apartment example asks whether a missing resident preference permits action or requires clarification; H5 adds an explicit domain-specific JSON output clause.For RELAX, the apartment clause specifies an integer threshold for the minimum correction.Source: §N.1 · p. 48

N.2 CANDIDATE SELECTION AND WORDING

  • Candidate selection replaces answer generation with a choice among four complete answers, while the neutral and reminder conditions append different wording to the same H2 prompt.The menu has two ACT candidates and two source-appropriate interventions, with one correct answer whose position is balanced across letters.Source: §N.2 · p. 49

N.3 A TWO-TURN PREFERENCE-CLARIFICATION EXAMPLE

  • The two-turn example requires a valid first-turn intervention without an action, then supplies feedback before execution; objective questions must list all admissible objectives.Its apartment example asks about Resident4’s missing Apartment1/Apartment4 preference under a rank-maximal objective.Source: §N.3 · p. 50
  • A specified Resident4 preference (Apartment4 > Apartment1) requires an ACT response with a valid action and null question and repair fields.The autonomous route retains the first response; the reference route starts a separate conversation with predetermined feedback, and invalid first interventions remain failures.Source: §N.3 · p. 50

O.1 ACTIONS, OBJECTIVES, AND PREFERENCE COMPLETIONS

  • Preference tasks use exactly two worlds differing only in one hidden adjacent comparison; unspecified comparisons are not indifference, and matching uses the singleton A-proposing outcome.The first three structures use ordinal rank objectives and retain all optimum ties.Source: §O.1 · p. 51

O.2 ALLOWED REPAIRS, MINIMALITY, AND CHECKER BOUNDARIES

  • RELAX changes only a permitted stated constraint; these fixed tasks need one elementary edit, and the response must pair an accepted repair with an action accepted under it.Edit counts are minimized, not a shared numerical distance across different constraint types.Source: §O.2 · p. 52, Table 43 · p. 52, §O.3 · p. 53

O.3 DECISION AND REPAIR DEFINITIONS

  • ACT requires a shared accepted action; clarification requires every case feasible but no shared action, while RELAX applies when the original constraints are infeasible.Full correctness also checks the requested action, pivotal comparison or objective clarification, and—after repair—a valid resulting action.Source: §O.3 · pp. 52–53

P.1 UNNECESSARY CLARIFICATION

  • In an audited Claude-5-Sonnet example, the model unnecessarily asks about T3 versus T4 even though both completion-optimum sets share T2, so ACT with T2 is accepted.The examples replayed the archived H2 checker and checked decisive witnesses; the unnecessary question fails both decision and full correctness.Source: §P.1 · p. 54, P VERIFIED EXAMPLES FROM ARCHIVED… · p. 54

P.2 ACTING BEFORE A NECESSARY CLARIFICATION

  • For Resident4’s missing Apartment1-versus-Apartment3 comparison, the two worlds uniquely select Apartment4 and Apartment3, so asking is required; the model instead acts with Apartment4.Apartment4 is optimal in only one compatible world, making the response a premature commitment that fails decision and full correctness.Source: §P.2 · p. 54

P.3 A VALID REPAIR FOLLOWED BY AN INFEASIBLE ACTION

  • After a valid minimum repair removes A1’s O1 requirement, the model assigns O1 to both A2 and A5, violating one-to-one allocation despite its correct RELAX decision.Replacing A5’s object with O5 yields a certified valid action; the returned action is invalid after repair.Source: §P.3 · p. 55

P.4 CORRECT ACT DECISION BUT A SUBOPTIMAL ACTION

  • The model correctly chooses ACT and returns a bijection satisfying A2’s O1 requirement, but its rank-count vector (1, 1, 1, 1) is below the accepted (1, 1, 2, 0).The action is therefore not rank-maximal, so full correctness fails despite the correct decision.Source: §P.4 · p. 55

Related papers

Loading 2610.03102v1…