PaperlayerView exact revision on arXiv

Source-linked AI summary

Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu

arXiv:2610.17757v1cs.AIcs.SE

TL;DR

Initial instructions can omit requirements that are necessary for completing a task, even when those requirements are discoverable from the project. This paper introduces TiGanEval to measure whether agents infer them, and finds that partial specifications produce large success-rate losses that feedback and plan mode do not close.

  • Problem

    In 100 SWE-chat trajectories, 56% of initial instructions omitted later-needed requirements, and 52% of useful follow-up requirements could have been inferred from the project.

    Source: §1 · p. 1

  • Method

    TiGanEval benchmarks software engineering and research tasks with project-discoverable true requirements that the simulated user initially only partly knows.

    Source: §1 · pp. 2–3

  • Results

    53 to 67 points of success rate are lost across eight frontier models moving from full specifications to partial instructions, and consequence feedback across four submissions does not close the gap.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Outcome alignment remains broadly missing from frontier models, and most failures mishandle evidence already offered by the project.

    Source: §1 · p. 2, §6 · p. 9

    The simulator generates later instructions from authored observations and interaction history, so it does not capture more complex user behaviors such as partial expression of beliefs or proactive incremental disclosure.

    Source: A Limitations and Future Work · p. 15

Abstract

from arXiv · show

Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incomplete. In our analysis of 100 user-agent trajectories from SWE-chat, 56\% of initial instructions omit requirements that later emerge as necessary, and 52\% of the useful follow-up requirements could have been inferred from the project. A user states requirements from their own understanding of the task, and these \textbf{stated requirements} often differ from the \textbf{true requirements} that completing the task imposes. An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them. We call this ability \textbf{outcome alignment}. In this work, we construct \textbf{TiGanEval}, a benchmark of software engineering and research tasks. In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work. These properties together make outcome alignment measurable. Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap. Furthermore, a plan-mode harness cannot compensate for the missing ability of the model itself, which places outcome alignment among the abilities the next generation of models should be developed for.

1 Introduction

The paper frames outcome alignment as inferring task requirements from both the user's instruction and the project, then introduces TiGanEval to measure this ability. Experiments find a substantial gap between full and partial specifications that feedback and plan mode do not close.

1 Introduction

  • In 100 SWE-chat trajectories, 56% of initial instructions omitted requirements later needed, and 52% of useful follow-up requirements could have been inferred from the project.The paper calls the ability to infer true requirements from the instruction and environment, and satisfy them before a violation reaches the user, outcome alignment.Source: §1 · p. 1
  • Prior settings do not measure outcome alignment when missing requirements are recoverable from the user, unverifiable outside the user, or already fully specified to the agent.The paper distinguishes stated, inferred, and true requirements to make specification gaps an explicit capability to evaluate.Source: §1 · p. 2
  • TiGanEval covers software engineering and research tasks where true requirements are fixed in advance and discoverable from the project, while the simulated user initially knows only part of them.TiGanTrans converts fully specified tasks into partial forms and has potential to scale evaluation and training environments.Source: §1 · pp. 2–3
  • 53 to 67 points of success rate are lost across eight frontier models when moving from full specifications to partial instructions, and neither four rounds of consequence feedback nor plan mode closes the gap.Prompting models to treat instructions as evidence recovers 16 to 34 score points over literal execution but still does not reach the full-information level.Source: §1 · p. 2

2 Related Work

The paper compares prior benchmarks by how they represent specification gaps and evaluate agent behavior. TiGanEval is presented as meeting all six criteria for measuring outcome alignment.

2 Related Work

  • Existing benchmarks leave specification gaps only partly open: some keep all requirements with the user, some hide them in a simulator, and others provide the full goal.Table 1 compares benchmarks on specification structure and evaluation protocol; its checks indicate feature presence, not that every task covers a feature.Source: §2 · p. 3, Table 1 · p. 3
  • Prior evaluation protocols include user interaction or consequence feedback, but none opens a gap the user cannot state while retaining verifiable task requirements.Interactive Task Alignment uses intent match, while Senior SWE-bench uses tests written by an agent for each submitted solution; TiGanEval freezes requirements before rollout.Source: §2 · p. 3
  • TiGanEval is the first benchmark in the comparison to meet all six criteria, including user-unavailable requirements resolvable from the project, consequence feedback, frozen verification, and a full-specification control.The cited comparison describes the criteria as applying to specification structure and evaluation protocol.Source: §2 · p. 3, Table 1 · p. 3

3 Problem Formalization

The formalization separates project-fixed true requirements from the user's partial stated requirements and the agent's evolving estimate. Outcome alignment requires the agent to use environmental evidence to satisfy true requirements before user feedback supplies them.

3 Problem Formalization

  • The environment E comprises project states, actions, transitions, and observations, while true requirements R are predicates fixed by the project before the episode begins.The outcome is one exactly when every requirement holds.Source: §3 · pp. 3–4
  • The user's stated requirements represent a partial, possibly mistaken understanding of R, while the agent maintains an inferred estimate initialized from the instruction and updated through interaction.The specification gap is the difference between the user's initial understanding and the true requirements.Source: §3 · p. 4
  • After a submission violates a requirement, the user sees its visible consequence and revises their understanding for the next instruction, providing a user-mediated route toward the true requirements.The task judges submissions privately against fixed requirements.Source: §3 · p. 4, Figure 2 · p. 4
  • Outcome alignment means delivering a state that satisfies all true requirements while the user's stated requirements remain partial, which requires the agent to infer missing requirements from the environment.Unlike the user-mediated route, the agent-driven route can move its estimate toward R through exploration and experiments before any consequence occurs.Source: §3 · p. 4

4 TiGanEval

TiGanEval pairs partial and full instructions over the same task environment, where true requirements remain discoverable. TiGanTrans constructs and reviews these executable task pairs for evaluation.

4.1 Task Structure and Evaluation Protocol

  • Each task pairs a project workspace, simulated user, private evaluator, and observation registry; partial and full conditions differ in what the user knows.True requirements are machine-checkable predicates, and the private evaluator checks them. When a submission fails, the user receives an authored observation describing a concrete consequence.Source: §4.1 · pp. 4–5

4.2 Task Construction with TiGanTrans

  • TiGanTrans converts fully specified source tasks into executable task packages with partial and full initial instructions.A Planner designs the user’s partial understanding and discovery paths; a Builder implements the package, an Auditor checks it, and a Calibrator replays full, partial, and literal rollouts.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Human experts review every generated task for consistency, plausible instructions, discoverable requirements, and absence of requirement leaks.Tasks failing any of these criteria are repaired.Source: §4.2 · p. 5

4.3 Statistics and Scalability

  • TiGanEval contains 40 software engineering and 24 research tasks, each with five true requirements; 39% of partial instructions cover none of them.The research tasks span instrumented studies, machine-learning experiments, and online systems; no partial instruction covers more than two requirements.Source: §4.3 · p. 6

5 Evaluation and Analysis

Across 64 tasks, frontier models perform substantially worse with partial than full instructions, and feedback or plan mode does not close the gap. Audits attribute most failures to mishandling evidence, while replication and simulator checks support the measurement’s reliability.

5.2 Outcome Alignment Across Models

  • Each frontier model loses 53 to 67 points of Success@1 under Partial versus Full@1 across the 64 tasks.Table 4 reports outcomes over all 64 tasks; each Partial value’s subscript gives its difference from Full@1 in percentage points.Source: §5.2 · p. 6, Table 4 · p. 7
  • Partial scores exceed Literal scores for every frontier model at every submission, but no trajectory reaches Full@1.The Partial-to-Literal margin is 15.7 to 34.1 Score points at the first submission and 7.2 to 24.1 after four submissions.Source: §5.2 · p. 6, Figure 4 · p. 7

5.3 When Requirements Are Resolved

  • Models resolve at most 17.6% of remaining unstated requirements ahead of feedback, while 37.0% to 58.7% of reported requirements pass within the round.Across frontier models, 45% of reported requirements pass at the next submission, 15% at the second, and 40% in neither round; most unreported requirements also remain unresolved.Source: §5.3 · pp. 7–8, Table 5 · p. 7

5.4 Can a Plan Mode Buy Outcome Alignment?

  • Plan mode never raises Success@1; it lowers Score@1 for Kimi-K3 and DeepSeek-V4.1-Flash and raises GPT-5.6-Luna’s Score@1 by only 4.3 points.Trajectory analysis finds that planning often stops before the agent infers the complete requirements, leaving execution with a partial plan.Source: Table 6 · p. 8, §5.4 · p. 8

5.5 Failure Mode Analysis and Case Study

  • Evidence mishandling accounts for 72 of 100 failed Partial episodes; behavioral collapse occurs in 14 of 18 Qwen and Gemma episodes.The five evidence-use failure modes include substituting an unsupported rule, ignoring a replayed violation, and failing to generate a decisive probe.Source: §5.5 · p. 8, Figure 5 · p. 8

5.6 Reliability of the Measurement

  • Across three replicates, seed-to-seed deviation stays within 4.3 Score points; 96.0% of 150 regenerated simulator messages convey exactly licensed information.In 28 of 30 checkpoints, five differently worded resamples convey the same incident and press for the same change.Source: §5.6 · pp. 8–9, Table 7 · p. 8

6 Conclusion

TiGanEval measures whether agents infer true task requirements from the user’s stated requirements and the environment. The paper concludes that this ability is broadly missing from frontier models, and that feedback and plan mode do not close the gap.

6 Conclusion

  • TiGanEval measures outcome alignment by keeping requirements discoverable in the environment while the simulated user lacks some of them.The benchmark separates requirements inferred by the agent from those the user adds after seeing delivered work.Source: §6 · p. 9
  • Frontier models broadly lack outcome alignment; neither consequence feedback nor plan mode closes the gap.Most failures mishandle evidence already offered by the project.Source: §6 · p. 9
  • The authors hope TiGanEval guides development of agents that deliver what users need rather than only what they wrote.Source: §6 · p. 9

A Limitations and Future Work

The task set was constrained by cost, and the user simulator omits some more complex user behaviors.

A Limitations and Future Work

  • Cost constraints limited the task-set scale and prevented exploration of model training to improve outcome alignment.Future work could address these limitations through environment synthesis, model training, and broader exploration of harness designs.Source: A Limitations and Future Work · p. 15
  • The simulator generates follow-up instructions from authored observations and interaction history, excluding partial expressions of user beliefs and proactive, incremental requirement disclosure.Future work could incorporate these behaviors while preserving evaluation verifiability and reliability.Source: A Limitations and Future Work · p. 15

B Preliminary Study and Annotation

A preliminary study sampled 100 SWE-chat trajectories and used two-stage human annotation to assess whether initial instructions covered later requirements and whether follow-ups were inferable. More than half of initial instructions were incomplete, and about half of useful follow-up requirements could have been inferred from prior project context.

Data Sampling

  • The study randomly sampled 100 SWE-chat trajectories, averaging 17.11 user turns each.The study examined whether stated requirements differed from those ultimately necessary and how discrepancies related to interaction and task-solving efficiency.Source: B Preliminary Study and Annotation · p. 15

Annotation

  • Annotators first judged whether the initial instruction covered requirements that later emerged as necessary, including scope, implementation, verification, or delivery requirements.Figure 6 presents the human annotation interface for assessing initial-instruction completeness and follow-up inferability.Source: B Preliminary Study and Annotation · p. 15, Figure 6 · p. 15

Results

  • 56% of sampled trajectories had incomplete initial instructions, and 52.49% of useful follow-up requirements in those trajectories were inferable from preceding context.That context included project state, implementation results, and errors available before the requirement was stated.Source: B Preliminary Study and Annotation · pp. 15–16, Figure 7 · p. 16

Annotation

  • For incomplete trajectories, annotators filtered non-informative turns and judged whether substantive follow-up requirements could reasonably have been inferred from the preceding trajectory.Potential evidence included project state, implementation results, errors, and established task context.Source: B Preliminary Study and Annotation · p. 15

C Benchmark Comparison Criteria

The comparison criteria distinguish whether protocols represent user knowledge gaps, potentially misleading stated requirements, and requirements resolvable from project evidence. They also assess feedback, verification, and matched full-specification controls.

Comparison criteria

  • A check in Table 1 means a protocol explicitly supports a feature, not that every task covers it; a cross means the feature is not explicitly included.The comparison evaluates protocol support rather than per-task prevalence.Source: C Benchmark Comparison Criteria · p. 16

Gap beyond user knowledge

  • TiGanEval models a user who initially lacks relevant knowledge or fully formed intent, while separating that knowledge from fixed true requirements.The compared protocols model different gaps, including unexpressed preferences, evolving intent, or miscalibrated repository beliefs.Source: C Benchmark Comparison Criteria · p. 16

Stated may mislead

  • TiGanEval includes mistaken initial proposals that agents can revise using project evidence within the authorized scope.The comparison contrasts this with protocols featuring contradictory specifications or user edits that interfere with completion.Source: C Benchmark Comparison Criteria · p. 16

Resolvable from the project

  • A requirement is project-resolvable when workspace, public-record, or permitted-experiment evidence can reveal it before the first submission, without a user oracle.The passage distinguishes evidence being present from repository-tool access alone.Source: C Benchmark Comparison Criteria · p. 16

Consequence feedback

  • TiGanEval returns an authored failure observation selected using private measurements, without giving the simulated user scores or requirement text.The criterion concerns whether evaluation produces observations for subsequent revision.Source: C Benchmark Comparison Criteria · p. 16

Verifiable requirements and matched control

  • TiGanEval fixes five binary requirements and records each submission's results with executable checks; its matched full-specification control holds the project and evaluation setup fixed while changing instruction and user knowledge.A checkable artifact or behavior is required, and the control compares the same tasks under complete and incomplete initial specifications.Source: C Benchmark Comparison Criteria · p. 16

D Evaluation Protocol Algorithm

The appendix describes the evaluation episode, the TiGanTrans task-construction workflow, and the prompts defining agent and simulated-user behavior. Together, these procedures specify how tasks are built, audited, calibrated, and run under different instruction conditions.

D Evaluation Protocol Algorithm

  • Under the partial condition, the episode privately judges each submitted artifact, returns an authored consequence for the top violated requirement, and continues until success or budget exhaustion.The algorithm updates the user's instruction after a failed submission and evaluates the final workspace if the budget ends.Source: D Evaluation Protocol Algorithm · p. 17

E TiGanTrans Details

  • TiGanTrans starts from a fully specified source task; its Planner fixes machine-checkable true requirements, user knowledge, discovery paths, failure observations, and leakage risks, while its Builder creates the executable package.The Builder retains evidence while removing statements of the design, then checks the gold, starter, and literal baseline against specified conditions.Source: E TiGanTrans Details · p. 17, §E.1 · p. 17, §E.2 · p. 18

E.3 Auditor

  • The Auditor checks consistency, leaks, discovery paths, and whether the full instruction suffices, then returns pass, required fixes, or redesign.It reviews the package read-only and quotes evidence for its findings.Source: §E.3 · p. 18

E.4 Calibrator

  • The Calibrator runs reference and literal-baseline rollouts to distinguish environment defects from agent limitations and to check whether partial-condition successes come from legitimate project-evidence discovery rather than evaluation gaming.The criteria include silent or ambiguous specifications, evaluator contradictions, misleading workspaces, and hard-coded behavior keyed to checks.Source: §E.4 · p. 18

F.1 Partial and Full

  • Partial and Full share the agent prompt, which treats owner messages and project artifacts as evidence; they differ in the initial instruction's information.The prompt directs agents to inspect the project, replay records, run experiments, and validate before delivery.Source: F Agent and User Simulator Prompts · p. 19, §F.1 · p. 19

F.2 Literal

  • The Literal prompt treats the initial instruction and later owner messages as the complete specification and bars inferring additional requirements from project evidence.It directs agents to implement only explicitly requested changes and validate only explicitly requested behavior.Source: §F.2 · p. 19

F.3 Plan Mode

  • Plan mode withholds editing and submission during read-only investigation, then restores tools for execution of the written plan under a shared budget.The plan records file-level changes, reasons, and validation methods; execution has one submission.Source: §F.3 · p. 20

F.4 User Simulator

  • The simulated user responds in first person from its persona, prior messages, and latest observation or engineer message, without inventing unobserved requirements or diagnosing unsupported causes.For observations, it reports what happened and what needs to differ; for questions beyond its knowledge, it directs the engineer to investigate the project.Source: §F.4 · p. 20

G Simulator Stability Samples

The simulator stability samples show that replayed requests preserve incident, cost, and demand while varying wording; one resample is rejected for adding details the owner had not observed. The model setup uses a shared scaffold, tools, and budgets across configurations.

G Simulator Stability Samples

  • One resample was rejected for asserting unobserved consequences and resolving a question the owner had explicitly left open.Four other resamples preserved the owner's uncertainty about whether the incomplete write shared a path with an earlier tracking problem.Source: G Simulator Stability Samples · p. 21
  • Across five replays, the checkpoint requests preserve the same incident, cost, and demand while varying only their wording.The samples include a condition that cleared and later returned, which the delivered build attached to an earlier ledger record.Source: G Simulator Stability Samples · p. 21

H Model Details

  • The twelve models use the same agent scaffold, tools, and budgets under all three configurations.Eight models use serving APIs; four small open-weight models run with vLLM on one NVIDIA A100 server.Source: H Model Details · p. 22

I Per-Domain Results

Feedback reports consequences rather than requirements, so converting a symptom into the governing rule requires inference from the project. The section also presents domain-specific outcomes and a case task whose unstated requirements are discoverable from project evidence.

I Per-Domain Results

J Feedback Form and Conversion Failures

  • In audited failures, 11% repair only the exhibited path, 25% defend an agent-authored rule, and 21% ignore a replay that prints the violation.In the 11% of cases, validation covers the same path, so the unmet rule survives the agent's own testing.Source: J Feedback Form and Conversion Failures · p. 23
  • A symptom shows one failure path, while the full streaming-response rule covers three termination paths, reading after close, and re-iteration.The observation describes a response ending without a close; the full instruction specifies closing on completion, failure, or abandonment after a partial read.Source: J Feedback Form and Conversion Failures · pp. 22–23

K Case Study: One Task in Full

  • TiGanEval's case task has five true requirements, only one stated initially, with the others discoverable from a protocol module and recorded runs.The task evidence includes a README and four console runs; some requirement details appear only in the README and replay output.Source: K Case Study: One Task in Full · p. 23, §K.1 · p. 23

K.2 Score Trajectories on This Task

On this case task, Luna reaches a score of 1.0 in one submission with full information, while its partial-instruction trajectory improves across four submissions. The rollout shows unstated requirements being inferred from project evidence and user feedback.

K.2 Score Trajectories on This Task

  • Luna scores 1.0 in one submission with full information, but starts at 0.2 under Partial and remains at 0.0 under Literal.Under Partial, Luna converts each observation within its round and reaches 1.0; Literal stores raw observed values instead of canonicalized values.Source: §K.2 · pp. 23, 25
  • Sol remains at 0.4 after removing a reported symptom, then resubmits an unchanged workspace twice without reaching identity's accounting and integrity requirements.The observations do not reach those requirements behind record identity.Source: §K.2 · p. 25

K.3 Rollout Excerpt: Luna Under Partial

  • Luna's four partial-instruction submissions progress from 0.2 to 0.6, 0.8, and 1.0 as further requirements pass.The rollout associates identity and lifecycle with submission 2, accounting with submission 3, and integrity with the accepted fourth submission.Source: §K.3 · p. 25
  • Three unstated requirements convert within the round of their observation, while lifecycle passes at submission 2 without being named in a user message.The identity repair adopts the protocol's record semantics instead of merely patching the reported duplicate.Source: §K.3 · p. 25

Related papers

Loading 2610.17757v1…