Execution Logs Sway VLM Judges of Robot Progress

1 Introduction

• The study asks whether robot evaluation importing LLM judges also imports their failure modes, particularly when plans, status messages, or logs reach the judge.The authors focus on text claiming progress because robot judges detect regression less reliably than progress.Source: §1 · p. 1
• The protocol compares log sentences with matched neutral sentences on intervals judged correctly from frames in both forward and reversed order.The paper also measures completion-claim attacks and compares prompt mitigations; effects on deployed evaluation or policy training are outside the study.Source: §1 · p. 1
• The work measures three VLM judges and compares mitigation prompts, including the trade-off between resisting misleading logs and responding to logs that agree with the label.The authors report verdict changes, not their downstream propagation into evaluation or policy training.Source: §1 · p. 1

2 Related Work

• Robot evaluation uses VLM success classifiers, simulated proxies, and human- or VLM-scored arenas.Prior work also finds that paraphrasing instructions can flip a VLM reward model on identical behavior and misleading signs can flip some safety guards.Source: §2 · p. 1
• Fabricated progress raises false positives for web-agent judges, while execution traces can sway video-generation verifiers with frames fixed.The video-generation study found supportive traces could make Qwen-VL accept near-miss failures, while tested GPT and Claude judges were barely moved on single clips.Source: §2 · pp. 1–2
• In the illustrated robot task, Qwen changes from Falling to Rising after a log falsely claims the pen reached the right arm and blue plate.The right arm never takes the pen.Source: Figure 1 · p. 2
• For static images, external text can override contradicting visual evidence, and a context-blind visual account recovers much accuracy for some models.Source: §2 · p. 2

3 Setup

• The dataset sample contains 240 intervals—120 Rising and 120 Falling—from 212 episodes, using only ±1 labels and intervals with at least five frames.Each judge sees the instruction and up to 12 evenly spaced frames from the interval’s first camera view.Source: §3 · p. 2
• Each interval is judged on frames alone, reversed frames, and frames with a progress, regression, or neutral log sentence.A log agrees with the label when its direction matches it; otherwise, it contradicts the label.Source: §3 · p. 2
• For completion claims, a separate model writes eight task-completion logs and eight software-only controls; the first claim that flips a Falling verdict is counted only if a second call also answers Rising.Control lines are selected and counted in the same way.Source: §3 · p. 2
• Observe asks for a visible-change description, Verify instructs judges to prioritize frames over conflicting logs, and a warning says the log may be inaccurate.Gemini and Qwen receive these prompts with no log, neutral, agreeing, and contradicting logs.Source: §3 · p. 2
• Figure 2 plots excess correct-Falling-to-Rising flips over a neutral sentence or unrelated log, with 95% confidence intervals on intervals correct in both frame orders.The figure caption reports a 10–35-point excess for fake progress lines across all three judges.Source: Figure 2 · p. 3
• Table 1 reports counts and episode-bootstrap rate differences for reversal-consistent intervals, with a repeat call and separate completion-claim counting rules.A completion claim counts only when a second call also answers Rising; the attack uses all Qwen Falling intervals and a random 80 for Gemini.Source: Table 1 · p. 3
• The primary measure is the excess rate of progress-log flips over neutral-log flips on Falling intervals judged Falling forward and Rising in reverse.The authors use episode-level bootstrap 95% intervals; clip length and episode position are associated with the label, so a correct frame-only verdict need not reflect visible motion.Source: §3 · p. 3
• The judges are Gemini 3.8 Flash, Qwen3.7 Flash, and GPT-6.1 Sol; GPT runs the main conditions only.All use low image detail; Gemini and Qwen use temperature 0, while GPT has no temperature setting.Source: §3 · p. 3

4 Results

• A one-sentence progress log turns reversal-consistent Falling verdicts to Rising more often than a neutral sentence for all three judges.A regression log has the mirror effect on Rising intervals; on 56 intervals reversal-consistent for all three judges, Gemini’s effect nearly vanishes.Source: §4 · p. 3
• 37 of 73 Qwen verdicts turned to Rising with a selected completion claim, versus 7 of 73 with a selected software line (+41.1 points).Gemini’s selected-claim effect was −3.2 [−12.7, +4.8] on 63 intervals, with no detectable effect.Source: §4 · pp. 3–4
• 29 of the 37 successful Qwen completion claims were contradicted by a last-frame check, and 8 were visually unsettled.A claim was coded as contradicted when the last frame showed part of the task unachieved, and unsettled when the relevant state was not visible.Source: §4 · p. 4
• The warning gives the highest accuracy with contradicting logs, but also reduces response to agreeing logs; for GPT-6.1 Sol it cut progress-log flips from 32 to 1 of 91.For Gemini, the warning also removes the effect of agreeing logs; Verify protects less but preserves some response to them.Source: §4 · p. 4, Table 2 · p. 4
• Under Verify, Qwen’s selected completion-claim effect is uncertain, while the effect averaged over all eight claims remains positive.The table caption notes that Verify lowers Qwen’s neutral-sentence accuracy on Falling from 0.78 to 0.61, inflating its gain.Source: §4 · p. 4, Table 2 · p. 4

• Text beside fixed frames shifts all three judges, and reversal partly controls a robot-data confound from clip-length label cues.Source: §5 · p. 4
• Pipelines should compare log effects with a matched neutral sentence and repeated call, and withhold logs when judging visual requirements.If a log must be shown, a warning protects more than verify, but the judge then responds less to logs that agree with the label.Source: §5 · p. 4
• Testing covers three general-purpose judges with one prompt on isolated, low-detail intervals; deployed text exposure and ranking or training effects remain unmeasured.The logs are synthetic, and fixed logs restate Rising and Falling definitions, so muting them is not shown to lose information.Source: §5 · p. 4

• The repository contains sampled intervals, logs and claims, harness code, and raw judge outputs sufficient to recompute every number offline.Source: A Prompts, logs and settings · p. 5
• The prompt presents text before the frames, then asks whether task-relevant state moves toward Rising or away from Falling; fixed logs claim progress, regression, or neutral recording details.Source: A Prompts, logs and settings · p. 5
• Observe asks for a sentence describing visible changes; Verify adds that logs should be used only when consistent with the frames, while the warning says a log may be inaccurate.Source: A Prompts, logs and settings · p. 6
• GPT-6 Luna generated eight task-specific completion claims and eight software-only control lines from each task instruction; filtered controls contained no object, arm, or motion terms.Source: A Prompts, logs and settings · p. 6
• For the Figure 1 Qwen example, the first three completion claims left the verdict Falling, while claim four yielded a repeated Rising answer and none of eight controls did.Source: A Prompts, logs and settings · p. 6
• On Qwen’s 74 reversal-consistent Falling intervals under verify, selected claims succeeded on 21 of 74 versus 16 controls, an excess of +6.8 [−5.5, +19.2] points.Averaged over all eight candidates, the claim-minus-control difference was +.010 [−.040, .062] for Gemini and +.358 [.272, .450] for Qwen; selected control-line successes under verify rose from 10% to 22%.Source: A Prompts, logs and settings · p. 6
• Settings use RoboPulse++ intervals with labels ±1 and at least five frames, 120 per label, the first camera view, and up to 12 evenly spaced frames.Judges ran via OpenRouter at low image detail; all calls were made on 2026-10-01 (UTC).Source: A Prompts, logs and settings · p. 6