PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Owns the Simulation, Jev Owns the Evaluation

Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang

arXiv:2610.01834v1cs.AIcs.LG

TL;DR

The paper asks which decisions judgment models can be trusted to make, then tests Jev across reflection tests, games, ALFWorld, and robot control. It finds a boundary between judging described options and predicting missing information, and shows that code-supplied simulation supports stronger control.

  • Problem

    It is unclear which decisions judgment models can be trusted with, and benchmark scores alone do not identify the decisions at which they succeed or fail.

    Source: §1 · p. 1

  • Method

    The paper tests Jev on reflection tests, one-shot matrix games, ALFWorld, and robot control, including conditions where code supplies missing predictions or simulated outcomes.

    Source: §1 · pp. 1, 3

  • Results

    Across four settings, Jev succeeds when the input contains sufficient information but fails when the right option depends on simulation; code-supplied predictions raise ALFWorld solved games from 31% to 87%.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The results support dividing the work so code makes predictions or simulations and Jev evaluates the resulting options.

    Source: §1 · p. 3

    The study tests one model and API version, and its ALFWorld and Stacker simulations use exact environment copies that a real agent may not have.

    Source: §8 · p. 10

Abstract

from arXiv · show

Judgment models such as \jev{} return, in a single call and without reasoning text, a probability for each described option. This makes them attractive as an agent's action-selection layer, but it is unclear which decisions they can be trusted with. We test \jev{} on reflection tests, one-shot matrix games, the text game ALFWorld and robot control, and find a sharp boundary. \jev{} succeeds when the right option can be judged from what the input describes, which we call \emph{evaluation}. Specifically, it solves 99\% of the counterintuitive Cognitive Reflection Test questions. However, it fails when the right option depends on \emph{simulation} (i.e., predicting something not in the input), such as the opponent's action or the subgoal that must come first. In games, \jev{} plays suboptimally as if its rational opponent acted at random, because the opponent's action is not given. In ALFWorld, \jev{} favors commands that mention an object or place named in the task description. For example, given the task ``put a clean knife in the drawer'', \jev{} carries an unwashed knife straight to the drawer instead of first washing it at the sink. Surprisingly, many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, \jev{} usually answers correctly, and it responds well given the opponent's action. It fails when one call must both perform the simulation and evaluate based on it. This suggests letting code make the prediction or simulation. When code supplies it, such as a lookahead in ALFWorld and physics simulation in robot control, \jev{} becomes an expert controller through its general evaluation ability.

1 Introduction

Jev evaluates options well when the input contains the information needed to judge them, but struggles when a decision requires predicting missing consequences. Supplying those predictions through code supports stronger performance across games, ALFWorld, and robot control.

1 Introduction

  • Jev is attractive as an action-selection layer because it returns probabilities for described options in one call without reasoning text or task-specific training.It also requires no output parsing.Source: §1 · p. 1
  • Jev solves 99% of counterintuitive Cognitive Reflection Test questions, illustrating success when the answer can be checked against the input.The listed prices let it check candidate answers against the question’s conditions.Source: §1 · p. 1
  • In ALFWorld, Jev is accurate 99% of the time when task-mentioned commands are correct, but 36% when they are wrong.For a clean butterknife task, it carries an unwashed butterknife to the drawer instead of washing it at the sink.Source: §1 · p. 2, Figure 1 · p. 2
  • Many failures reflect a single call having to simulate and evaluate together, rather than an inability to predict or respond to an opponent’s action.Asked separately, Jev predicts the opponent’s action correctly 90% of the time and plays the best response 98% of the time when given that action.Source: §1 · p. 2
  • Code-supplied predictions raise one-shot game accuracy from 33% to 98% in specified misleading cases and ALFWorld solved games from 31% to 87%.When given a separately elicited opponent prediction, Jev chooses the best action in 94% of cases.Source: §1 · p. 2
  • The controller further generalizes to new task variants, succeeding in 22 of 40 episodes against 3 for the comparison condition.Source: §1 · p. 3

2 Evaluation, Simulation, and How We Label Decisions

The section defines evaluation as judging options from what the input describes, and simulation as obtaining missing information through prediction. It also specifies how surface shortcuts and decision procedures are identified and tested.

2 Evaluation, Simulation, and How We Label Decisions

  • A decision is a state, a set of described candidate options, and the correct options; evaluation judges correctness from that input, while simulation requires a prediction absent from it.Whether a decision needs simulation depends on how the task is presented, not on whether the model can make the prediction.Source: §2 · p. 3
  • A decision that initially requires simulation needs only evaluation when code adds the command’s possible outcome to the option.For example, the option can state that going to the sink makes cleaning the butterknife possible.Source: §2 · p. 3
  • A surface shortcut is a fixed input-only rule for selecting relevant-looking options; its picks are lures when they are wrong in a simulation-dependent decision.A shortcut can accidentally select the correct option, so aligned and misleading cases are distinguished.Source: §2 · p. 3
  • The shortcut selects highest-average-payoff actions in games and task-mentioned commands in ALFWorld; errors are classified as correct, lure, or other.Concentrated lure errors indicate attraction to the shortcut rather than errors spread across other wrong options.Source: §2 · p. 3
  • The predictions test whether Jev handles evaluation, aligned and misleading simulation decisions, supplied missing predictions, and predictions written back from a separate call.The separate-call test distinguishes inability to predict from difficulty combining prediction with choice in one call.Source: §2 · p. 4
  • Because candidate order shifts Jev’s probabilities, each decision is averaged over several orders, while decision labels are computed from the environment.The evaluation, simulation, aligned, and misleading labels do not depend on Jev’s choices.Source: §2 · p. 4

3 Jev Owns the Evaluation: Cognitive Reflection

On the Cognitive Reflection Test, Jev usually rejects the intuitive but wrong answer by checking the listed options against the question’s conditions.

3 Jev Owns the Evaluation: Cognitive Reflection

  • 99% of CRT questions were answered correctly and 1% with the intuitive answer; the 95% interval for correctness on lure questions was [0.97, 1.00].The table covers 150 questions in four option orders.Source: Table 1 · p. 4, §3 · p. 4
  • Jev answered correctly on 96% of control questions, where the intuitive answer was correct, while people gave the intuitive answer on 55% of the same questions in open form.These controls indicate Jev does not simply avoid intuitive-looking options.Source: §3 · p. 4
  • Because the correct answer is listed among the options, Jev can check each candidate against the question’s stated conditions.For the bat-and-ball example, checking the prices yields $0.05 for the ball rather than the intuitive $0.10.Source: §3 · p. 4

4 Jev Does Not Own the Simulation: One-Shot Games

In one-shot games, Jev’s accuracy depends on whether the surface shortcut identifies the equilibrium action, not on reasoning depth. Supplying the opponent’s action restores strong performance, showing that the failure lies in combining prediction and evaluation in one call.

4.1 Failures follow the surface shortcut, not depth

  • In misleading games, Jev reaches only 28–37% accuracy and chooses the lure in 53–71% of games; stating Column’s action raises accuracy to 97–99%, while reasoning instructions leave performance at chance.In aligned games, accuracy remains 77–93% at every depth; the reported regression finds no detectable depth effect after accounting for alignment.Source: Figure 2 · p. 5, §4.1 · p. 5
  • The experiment varies reasoning depth and whether the surface shortcut selects the equilibrium across 210 random 3×3 games, comparing plain, reason, and told conditions.In the told condition, code adds Column’s equilibrium action to the state.Source: §4.1 · p. 5

4.2 What Jev computes

  • A softmax over actions’ average payoffs best fits Jev’s choices, which resemble a quantal level-1 player that treats the rational opponent as random.Adding whether an action is the equilibrium barely improves the fit; level 1 involves no reasoning about the opponent and is distinct from depth D=1.Source: §4.2 · p. 5

4.3 Knowing is not using

  • Jev’s choice does not use its prediction: separating simulation and evaluation by writing the predicted action into the state raises accuracy in misleading games from 33% to 94%.When Jev’s prediction is wrong, it still plays the best response to the written action in 24 of 26 calls.Source: §4.3 · p. 6

5 Code Owns the Lookahead: Text Games

In ALFWorld, Jev handles decisions when the input makes the right command judgeable, but often follows a task-word lure when it must infer an intermediate step. Code-run lookahead or explicit subgoals supply the missing information and improve decisions.

5 Code Owns the Lookahead: Text Games

  • ALFWorld tests Jev on sequential household text-game tasks, with choices among admissible commands at each step; Jev alone solves 31% of 134 unseen games.The single-decision study replays expert trajectories and classifies steps by whether commands complete a task part, enable a later step, or require exploring for an unseen object.Source: §5 · p. 6, §5.1 · p. 6

5.1 Where Jev fails

  • On seen games, acceptable-command accuracy is 93% on E steps and 99% on aligned S-known steps, but 36% on misleading S-known steps, where Jev chooses the lure 56% of the time.Unseen games show a similar pattern: 96%, 95%, 43%, and 51%, respectively.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 What Jev knows

  • On goal-form tasks, Jev predicts the next subgoal incorrectly 67% of the time; code-provided subgoals raise decisions to 92%, while rewriting tasks into ordered form raises them from 6% to 73%.Under the ordered form, Jev predicts the next subgoal correctly 100% of the time, yet single-call decisions are right only 68%; writing its answer into the input raises them to 86%.Source: §5.2 · p. 7

5.3 Code owns the simulation

  • Lookahead runs each admissible command in a copy of the game and adds its predicted outcome and newly possible commands to the option; at misleading S-known steps, accuracy rises from 36% to 79%.The rate of choosing the lure falls from 56% to 21%.Source: §5.3 · p. 7
  • On 134 unseen games, Jev alone solves 42; adding the game’s next print raises this to 79, and adding newly possible commands raises it to 116 (87%), while a word-overlap rule with the same lookahead solves 18.The result supports the role of Jev’s evaluation of predicted outcomes, not lookahead information alone.Source: §5.3 · pp. 7–8

6 Code Owns the Physics: Robot Control

Robot control tests the same boundary between evaluation and simulation: raw torques give Jev no predicted outcome, while code-simulated skills support control and generalization to new instructions.

Without physics simulation

  • With raw torque options, Jev never picked up a box and solved none of the 20 native-task episodes; random torques also solved none.The input described the arm and joint angles but not the outcomes of candidate moves.Source: §6 · p. 8

With physics simulation

  • Code implements each skill using inverse kinematics and joint PD control, so Jev can evaluate options whose simulated outcomes it can see.The skill library was tuned on the native task.Source: §6 · p. 8

Generalization to new instructions

  • 22 of 40 episodes were solved by Jev on new instructions, versus 3 by the hand-written controller and 0 by random choice.The skills were shared across the five new instructions and the 40 starting-condition episodes.Source: §6 · p. 8

With physics simulation

  • With code-simulated manipulation skills, Jev solved 9 of 20 native-task episodes, compared with 8 for the hand-written controller.The table compares success counts with and without physics simulation performed by code.Source: §6 · p. 8, Table 3 · p. 9

7 Related Work

Related work uses language models to score candidate actions, plan or verify outcomes, and probe cognitive or strategic behavior. This paper instead identifies which decisions require evaluation versus simulation.

Language models as policies over candidate actions

  • Prior action-scoring work reports agent success, while this paper asks which decisions the scorer gets right based on whether they require evaluation or simulation.The distinction is determined from the environment, not from the model.Source: §7 · p. 9

Who simulates and who evaluates

Cognitive and strategic probes

  • Cognitive and strategic probes include CRT error scoring and direct knowledge probes, while this work varies reasoning depth and surface alignment and adds a condition stating the opponent’s action.The passage situates these choices alongside prior work on cognitive tasks, heuristics, and behavioral game theory.Source: §7 · p. 9

8 Discussion

Jev’s failures cluster at decisions that require information absent from the input, even when separate probes show it can make the needed prediction. Code can supply that simulation, but the approach and analysis have stated limits.

Discussion

  • In misleading games, 90% of Jev’s errors choose the highest-average-payoff action, and at misleading ALFWorld steps, 89% choose a task-mentioning command.Asked separately, Jev predicts the opponent correctly in 90% of misleading games and chooses correctly in 98% when code supplies the prediction, but it is right in only 33% when one call must do both.Source: §8 · p. 9
  • Jev follows the order in which a task mentions actions, even when the word after indicates the opposite; in the example, it puts the item first although cleaning must come first.Jev decides correctly in 24% of cases with the wording that mentions putting the item first, versus 69% when the task mentions cleaning first.Source: §8 · p. 10

When can code make the prediction or simulation?

  • When consequences can be simulated, code can compute them directly; otherwise, a separate Jev prediction call can be written into the next input if Jev can predict from that input.If Jev cannot simulate the environment, the passage suggests another source, such as a learned world model or larger reasoning model.Source: §8 · p. 10

Limitations

  • The study covers one model and API version, and its ALFWorld and Stacker simulations use exact environment copies that real agents may not have.The ALFWorld lookahead is one command deep; searching deeper when consequences take several commands increases cost quickly, and the prediction analysis is partly post hoc.Source: §8 · p. 10

9 Conclusion

The conclusion presents Jev as effective when it evaluates described options, but unreliable when it must simulate missing consequences. It argues for code to provide predictions or simulated outcomes so Jev can evaluate them.

Conclusion

  • Jev solves counterintuitive CRT questions but carries an unwashed butterknife to the drawer rather than first washing it at the sink.The conclusion contrasts an answer checkable against the input with a step whose necessary consequence must be simulated.Source: §9 · p. 10
  • Jev can often predict an opponent’s action when asked separately, yet fails when one call must both simulate and evaluate.Source: §9 · p. 10
  • The authors conclude that code should make predictions or simulate outcomes, leaving Jev to evaluate; they report this division supports robot control on new instructions that a hand-written controller cannot follow.Source: §9 · p. 10

Reproducibility Statement

The appendices document the Jev interface and the construction, evaluation, and reproducibility procedures for the CRT, game, and ALFWorld studies. They report the model’s tendency to follow a level-1 game rule and describe reproducible run and analysis artifacts.

Reproducibility Statement

  • The authors store every model call and provide scripts to recompute CRT, game, and ALFWorld results, while DMC results and Figure 4 frames come from logged runs and replay scripts.They state that they will release code, raw runs, and scripts; Jev’s public API outputs may change across versions.Source: Reproducibility Statement · p. 10

A The Jev Interface and Prompts

  • Jev receives a JSON state, an instruction, and described options, then returns a chosen key, option probabilities, and confidence without reasoning text.The appendix specifies domain-specific prompts, including ALFWorld lookahead options that add predicted outputs and newly possible commands.Source: A The Jev Interface and Prompts · p. 13

B CRT Questions

  • After clarifying ambiguous CRT controls, control accuracy rose to 96%, while Jev answered 99% of lure questions correctly in both runs.Type 2 controls rose from 78% to 92%, and type 3 controls from 85% to 96%.Source: B CRT Questions · p. 15

C Games: Generation and Behavioral Models

  • In plain game calls, a level-1 rule based on average payoff against a random opponent best fits Jev’s choices, predicting 77%; adding equilibrium barely improves the fit.The games are generated as 3 × 3 payoff matrices and retained when one outcome survives iterative elimination.Source: C Games: Generation and Behavioral… · p. 15

D ALFWorld Details

  • ALFWorld labels replayed expert trajectories using tracked task state and subgoals; lookahead improves closed-loop outcomes, gaining 41 games over Jev alone in v1 and 38 over v1 in v2.The lookahead forks the game for each admissible command, and its predicted outputs match the real outputs at every step.Source: D ALFWorld Details · p. 16

E DMC Details

The DMC appendix describes skill-based and raw-torque control setups for Stacker, along with new instruction variants. Without simulator-derived state and skills, Jev’s raw-torque controller never picks up a box in the tested episodes.

Stacker skills

  • For skill control, code uses inverse kinematics and physics simulation to select solutions that minimize motion of the other box and joint movement.Skills use a 100 Hz PD controller; place skills carry, release, and wait for the box to settle.Source: E DMC Details · p. 17

New instructions

  • The five new Stacker instructions vary stacking, returning a box to its start, moving only one box, and selecting the farther box for placement.Initial scenes are redrawn until the task is feasible, with eight scenes retained for each instruction.Source: E DMC Details · p. 17

Stacker without simulation

  • Across 20 raw-torque episodes, Jev never held a box; its hand approached no closer than 25 cm on average, versus 16 cm with random torques.Two seeds were excluded because a box already sat at the target at episode start.Source: E DMC Details · p. 17

F Knowledge Probes

The knowledge probes test whether Jev can predict game actions and ALFWorld subgoals separately from choosing a decision. Results show that ALFWorld choices and subgoal predictions shift with task wording, while game-probe performance varies by prediction and elimination task.

Game questions

  • In game probes, Jev correctly identified a dominated Column action in 19% of steps, but with correct elimination notes misleading-game decisions reached 94%.Jev predicted Column’s highest-average-payoff action in 88% of calls, but named the equilibrium action in only 55% of 17 misleading cases where those differed.Source: F Knowledge Probes · p. 18

ALFWorld questions

  • At misleading ALFWorld steps, Jev named the next subgoal correctly in 63% of calls, and every error named putting the object; supplying the correct subgoal yielded 100% on clean and heat steps and 70% on cool steps.The probe asks which subgoal must happen next, while the decision question remains unchanged across told and write-back conditions.Source: F Knowledge Probes · p. 18

ALFWorld task wording

  • Swapping only ALFWorld task templates shifted decision accuracy from 6% to 73% and from 66% to 11%, with subgoal answers flipping alongside them.A third template that says cleaning comes first but mentions putting first yielded 24% accuracy.Source: Figure 5 · p. 18, F Knowledge Probes · p. 19

G Error Outcomes

Across CRT, games, and ALFWorld, errors with a lure overwhelmingly select that lure; supplying missing predictions moves many decisions toward the correct option. The appendices also report compute costs and the authors’ use of coding assistants.

G Error Outcomes

  • When Jev errs on lure-bearing decisions, errors concentrate on the lure: 90% in misleading games and 89% at misleading ALFWorld steps, while code-supplied predictions reduce lure errors.In the CRT, the 1% of errors all select the intuitive answer; stating Column’s action leaves 2% errors in games, and ALFWorld lookahead cuts the lure rate from 56% to 21%.Source: G Error Outcomes · p. 19, Figure 6 · p. 19

H Compute and Cost

  • Reported runs used about 94M input tokens and cost about $4 at the listed September 2026 API price.The two closed-loop ALFWorld runs account for about $2.2, and Stacker runs for about $0.9.Source: H Compute and Cost · p. 19, Table 7 · p. 20

I Use of Large Language Models

  • The authors say they conceived and directed the research and take responsibility for the content, while Claude Code implemented most experiment and analysis code.OpenAI’s Codex reviewed drafts for logic and consistency.Source: I Use of Large Language Models · p. 20

Related papers

Loading 2610.01834v1…