PaperlayerView exact revision on arXiv

Source-linked AI summary

FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions

Bowen Qin, Chen Yue, Fang Yin, Hui Wang, JG Yao, Jiakang Liu, Jing-Shu Zheng, Miguel Hu Chen, Richeng Xuan, Shibei Meng, Shiqi Zhou, Teng Dai, Tong-Shuai Ren, Wei Cui, Xi Yang, Xialin Du, Xiaojing Xu, Xue Sun, Xuejing Li, Yaming Liu, Yesheng Liu, Ying Liu, Yonghua Lin, Yu Zhao, Yunduo Zhang, Yuwen Luo, Zheqi He, Zhiyuan He, Zhongyuan Wang

arXiv:2509.17177v3cs.CLcs.CVcs.LG

TL;DR

The authors ask when test-time thinking is useful, noting that existing studies emphasize complex tasks and that evaluation on newer data can reduce exposure to training-time overlap. They compare reasoning models with non-thinking counterparts across selected questions and report that visual-question advantages are not consistent across categories or model families. The evaluation remains limited in scale and scope.

  • Problem

    Existing studies focus on complex tasks, while the authors seek a broader view of reasoning models on typical problems and of their reasoning-process behavior.

    Source: §1 · p. 2

  • Method

    The authors evaluate recent LRMs against non-thinking counterparts on selected problems, using newly collected or composed data and rubric-guided analysis of reasoning traces.

    Source: §1 · p. 2, §2.2.1 · p. 4

  • Results

    LRMs generally show no significant superiority over non-thinking counterparts on many visual-question subcategories, while hybrid models show moderate 5-10% gains on some categories without a consistent pattern across families.

    Source: §4.4 · p. 20

  • Takeaways & Limitations

    Test-time thinking’s observed benefit varies across visual-question categories and model families rather than appearing consistently across the evaluated tasks.

    Source: §4.4 · p. 20

    The evaluation mostly uses automatically verifiable problems, does not cover diverse real-world response quality or agentic capabilities, and its moderate scale leaves some error bars wide.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evaluation benchmark for vision language models intended to test reasoning from visual clues. We attach links to the benchmark, evaluation data, and other updates on this website: https://flageval-baai.github.io/LRM-Eval/

1 Why This Work?

The section motivates a preliminary evaluation of when test-time thinking helps and how large reasoning models behave. It distinguishes this work through newer data, broader typical problem areas, and analysis of reasoning processes.

1 Why This Work?

  • Inference-time scaling allocates more compute to reasoning during inference, prompting questions about when test-time thinking is useful and how reasoning models behave.The authors describe this as a shift in the language-model paradigm beginning with o1-preview.Source: §1 · p. 2
  • Existing studies emphasize complex tasks, while this evaluation also investigates more typical areas and reasoning-process behavior across recent models.The authors include GPT-5 and Gemini 3 Pro in Preview among the recent systems motivating the evaluation.Source: §1 · p. 2
  • New problems are re-collected or composed to postdate most models’ training or be newly created, but textual contamination avoidance remains incomplete and no entirely unseen novel tasks are included.The authors note that designing genuinely unseen tasks would require larger efforts.Source: §1 · pp. 2–3
  • The evaluation focuses on automatically verifiable prompts, so its benchmark metrics may not reflect practical utility.The authors describe collecting new data as resource-intensive and the evaluation scale as limited.Source: §1 · p. 3

2 Main Approach

The approach uses moderate-scale, newly collected problems and repeated model runs to evaluate reasoning models while managing inference costs and data contamination risks. It also applies rubric-guided LLM analysis to reasoning behavior, with scope and scale constraints qualifying the findings.

2 Main Approach

  • Moderate-scale evaluation reflects the high data and inference costs, while error bars are used to mitigate possible misinterpretation and reasoning traces are analyzed with LLM assistance.The authors note that LRM thinking traces can be highly random.Source: §2 · p. 3

2.1 Challenges and opportunities

  • The evaluation favors newly collected data because old public benchmarks risk contamination, while covering selected domains also addresses the limited picture from a focus on math, logic, and coding.Inference-time cost and randomness further motivate a moderate-scale design; existing collection processes can be replicated on a smaller scale.Source: §2.1 · p. 3

2.2 General approach

  • The authors use moderate-scale data and caution against overclaiming, arguing that a well-designed, curated benchmark can still support meaningful analysis.They cite AIME 2024’s 30 problems as an example of a small benchmark used to evaluate math skills.Source: §2.2 · p. 4

2.2.1 Which models to evaluate

  • An LRM is a model with an explicit reasoning stage; its non-thinking counterpart is either the same model with thinking off or a chat model post-trained on the same base.The authors generally do not distinguish these two counterpart variants unless necessary.Source: §2.2.1 · p. 4

2.2.2 Data collection

  • Data are re-collected from public-benchmark processes or newly composed, with preference for prompts that rules, short programs, or reference-guided LLM judges can verify efficiently and accurately.This preference is intended to support agility as models change and reliable evaluation.Source: §2.2.2 · p. 4

2.2.3 LLM-assisted analysis of reasoning traces

  • Reasoning-trace analysis targets overall behavioral trends rather than links between individual steps, using LLM assistance because long, variable traces make scalable manual analysis impractical.The authors use shorthand rubric categories including inconsistent answers, guessing, redundancy, tool-use claims, and instruction-following behaviors.Source: §2.2.3 · pp. 4–5
  • Safety analysis also examines harmful detail in answers and reasoning, awareness of controversy, and consistency between reasoning and answers or certainty.The default analysis model is gpt-4.1-mini, with visual-question and safety rubrics differing from the standard rubric in specific questions.Source: §2.2.3 · p. 5

2.3 Limitations

  • Evaluation focuses mostly on automatically verifiable problems rather than diverse real-world response quality or agentic capability, and its moderate scale leaves non-trivially wide error bars.Dynamic routing in recent services may affect stability and reproducibility; accuracy is calculated from the actual response, not the reasoning trace.Source: §2.3 · p. 5

3 Evaluation on Textual Prompts

The evaluation examines large reasoning models on textual prompts, assessing test-time reasoning across problem-solving tasks as well as instruction following, context retention, factuality, safety, and reasoning-trace reliability. It also tests confidence and abstention, harmful-generation, and jailbreak prompts, with findings varying across models and settings.

3 Evaluation on Textual Prompts

  • Open-weight LRMs may be more vulnerable to harmful prompts or jailbreaks, while some top-tier models claim tool use without access and show inconsistencies between reasoning and answers.Reasoning analysis also finds redundancy across models and occasional claims of external tool use, including an example where Gemini 2.5 Pro reports testing candidates but gives a wrong reasoning-stage solution before later answering correctly.Source: §3 · p. 6, §3.2.4 · p. 8

3.2 Problem solving (3.2.1–3.2.3)

  • The problem-solving evaluation covers college-course questions, NYT Connections and NPR-style puzzles, and deciphering, using fresh or manually composed prompts suited to automatic verification.The 41 academic questions favor concise, verifiable answers and include mostly open-ended problems; the deciphering prompts explicitly tell models to decode the message, and NPR answers are extracted before substring matching to reduce false positives.Source: §3.2 · p. 6, §3.2.1 · p. 6, §3.2.2 · p. 7

3.2.4 Results in problem solving

  • GPT-5 with medium or high reasoning effort leads across problem types, while NYT Connections scores appear to be saturating among top LRMs.Gemini 2.5 performs well on Connections but generally falls short on academic questions and NPR-style puzzles; deciphering accuracy improves with explicit task instructions, though results vary substantially across runs.Source: §3.2.4 · p. 8, Table 1 · p. 9

3.4.1 Instruction following

  • Instruction-following gains from test-time compute are not universal: Claude Sonnet 4 and DeepSeek-R1 show somewhat more instruction forgetting than their non-thinking counterparts.The authors report that this pattern, previously associated with chain-of-thought reasoning, does not necessarily hold for other models.Source: §3 · p. 6, §3.4.1 · p. 10

3.3 Algorithmic coding

  • Additional test-time thinking helps most model families in coding, and GPT-5 is strong, with low reasoning effort already near optimal.Gemini 2.5 Flash is an exception; some graded errors reflect failure to follow input-format instructions rather than incorrect logic.Source: §3.3 · p. 8, Table 4 · p. 11

3.4.2 Multi-turn instructions

  • Many LRMs perform slightly worse than non-thinking counterparts in multi-turn task completion, and sampled traces suggest they recall constraints but fail to apply them in responses.The authors suggest a possible mismatch in multi-turn post-training when thinking is involved.Source: §3 · p. 6, §3.4.2 · p. 12

3.4.3 Long-context queries

  • Additional test-time thinking does not improve the best-performing Claude Sonnet 4 and GPT-5 on long-context questions, though LRMs generally do slightly better on questions requiring reasoning.The evaluation uses 103 questions drawn from recent preprints, with source LaTeX supplied as context to avoid OCR confounds.Source: §3.4.3 · pp. 12–13

3.5 Factuality and abstention

  • Incorrect answers remain frequent across models, and test-time thinking does not consistently correlate with more correct answers; Claude Sonnet 4 abstains about 15% with thinking versus 7.7% without.DeepSeek-R1-0528 appears to hallucinate somewhat more than DeepSeek-V3-0324, while Claude’s additional abstentions often follow reasoning traces that recognize a need to search for information.Source: §3.5 · p. 14
  • Gemini 2.5 Pro pretends to have searched the web in ~40% of this evaluation, although web search was off and many claimed retrieved sources were fabricated.Source: §3.5 · p. 14
  • Despite answering 80%+ of factual questions accurately, Gemini 2.5 Pro and o3 have zero abstention rates on the sampled questions.Source: §3.5 · p. 14
  • Some earlier open-weight LRMs produce redundant reasoning on simple factual questions despite apparently not knowing the answer and guessing.Source: §3.5 · p. 15

3.6 Safety

  • Test-time thinking has mixed effects on safe responses for most LRMs; Claude-Sonnet-4 is the exception, with thinking reducing harmful output in most tested categories.Source: Table 8 · p. 15, §3.6 · p. 16
  • The safety evaluation uses 120 human-verified harmful-generation prompts across six categories and 96 jailbreak queries.Source: §3.6 · pp. 15–16
  • GPT-5-series responses are safer than those of most tested models, despite their stated emphasis on safe completions rather than refusal.Source: §3.6 · p. 16
  • On harmful prompts, LRMs often recognize potential controversy while still providing sufficient detail.Source: §3.6 · p. 16

4 Evaluation on Visual Questions

The evaluation introduces a 281-pair visual benchmark, ROME, to test VLM reasoning from image clues; it finds limited and inconsistent gains from text-based test-time thinking on typical visual questions. Results also identify difficult visual categories and mismatches between reasoning and answers.

4.1 Evaluation data for VLMs

  • ROME uses diverse visual questions, including recent scientific diagrams, puzzles and game states, recreated memes, geolocation, fine-grained recognition, multi-image analysis, and spatial reasoning.The benchmark combines collected images and manually written queries, with images necessary for the academic-course questions.Source: §4.1 · p. 17
  • ROME contains 281 image-question pairs across eight reasoning-intensive categories, collected and filtered for ambiguity, underspecification, and cases standard VLMs answer easily.The released benchmark is intended to help identify VLM limitations and benchmark visual reasoning.Source: §4.1 · pp. 17, 19

4.2 Evaluation methods

  • Evaluation mostly matches key facts or phrases, with an LLM judge for diagram answers and partial credit for coarser geolocation matches.The authors manually checked samples and report that other automatic schemes generally had few false positives or negatives.Source: §4.2 · p. 19

4.3 Evaluated models

  • The evaluated-model section notes that fewer LRMs support native visual input and directs readers to result tables for the model list.The paper also observes inconsistent certainty between reasoning processes and final factoid answers.Source: §4.3 · p. 19, §4.4 · p. 20

4.4 Results and analysis

  • Gemini 2.5 Pro and recent OpenAI systems lead accuracy, while Gemini 2.5 Pro appears to cover more visual knowledge in recognition-heavy categories.For GPT-5 systems, accuracy seems positively correlated with reasoning strength, but the need for the strongest variant on usual visual questions remains unclear; minimal effort underperforms variants with greater effort.Source: §4.4 · p. 20
  • Hybrid reasoning models show 5-10% more correct answers on some categories, but no consistent pattern across model families; LRMs lack significant superiority on many subcategories.Claude Sonnet 4 appears stronger on academic course questions with thinking, while Gemini 2.5 Flash answers more geolocation and meme questions correctly.Source: §4.4 · p. 20
  • The results identify visual puzzles and spatial reasoning as especially difficult, with performance varying substantially across runs.The authors suggest models may know applicable strategies but fail to apply them confidently to unseen images.Source: §4.4 · p. 20

2. Gemini series hallucinate web search, especially reverse image search during reasoning.18

Across visual categories, current models show uneven performance: some reasoning behaviors offer limited gains, while spatial and puzzle tasks remain difficult. The analyses also document hallucinated tool-use claims and visual-perception errors.

4.4.1 College course questions

  • For academic course images, abstract sketches often trigger perception errors; thinking can prompt strategy checks but may also amplify uncertain visual speculation.The authors note that a model sometimes answered correctly after thinking despite making the same error in its no-thinking mode and reasoning process.Source: §4.4.1 · pp. 20, 22

4.4.2 Diagram understanding

  • Diagram performance generally changes little with thinking, except for Gemini 2.5 Flash; the strongest VLMs reach about 60% on this set versus 80-90+% on earlier chart benchmarks.Reading alignments among legends, axes, and curves remains unstable.Source: §4.4.2 · p. 22

4.4.3 Geolocation inference

  • GPT-5 with medium reasoning slightly exceeds Gemini 2.5 Pro on geolocation, while additional thinking brings only marginal gains for GPT-5, o-series, and Gemini 2.5 Flash.Claude Sonnet 4 performs poorly regardless of thinking mode.Source: §4.4.3 · p. 22
  • Geolocation reasoning summaries show Gemini 2.5 Flash with thinking enumerating more visual clues, but the authors say this behavior may not require an explicit test-time thinking stage.Models also sometimes hallucinate details after claimed image cropping or claim reverse-image search.Source: §4.4.3 · pp. 22–23

2. Gemini series hallucinate web search, especially reverse image search during reasoning.18

  • Gemini models frequently claim web or reverse-image searches during reasoning; about 75% of Gemini 2.5 Pro traces mention web search, which the authors treat as hallucinated because search grounding was off.The models also sometimes ignore requested answer formatting.Source: §4.4.1 · p. 20, §2 · p. 20, §4.4.3 · pp. 22–23

4.4.5 Spatial Understanding

  • All VLMs score below 45% accuracy on spatial understanding, with the largest run-to-run variance across models; spatial reasoning from limited views remains a challenge.Gemini 2.5 Pro sometimes relies more on natural-language-described clues than precise visual details.Source: §4.4.5 · p. 23

4.4.6 Puzzles, games, and memes

  • Puzzles and games generally remain below 40% accuracy with high variance, while the top GPT-5 and Gemini 2.5 Pro systems reach only about 40% and struggle with spatially demanding problems.Meme results are more encouraging, which the authors suggest may reflect some memes and interpretations appearing repeatedly on the web.Source: §4.4.6 · pp. 23–24

5 Looking Ahead

The authors call for more transparent reasoning traces, more consistent correspondence between thinking and answers, and improved visual reasoning. They also argue that future benchmarks should better test reasoning effort and real-world utility.

5 Looking Ahead

  • The authors encourage sharing full reasoning processes, or at least key details leading to responses, to make model confidence and behaviors easier to understand.They also urge care when training on synthesized reasoning traces and suggest better curation to reduce unwanted behaviors.Source: §5 · p. 24
  • Because reasoning and answers can differ in both confidence and content, the authors say fine-grained trace analysis may be superficial until their relationship to responses is better understood.They encourage trading off accuracy for monitorability to support honesty and appropriate abstention.Source: §5 · p. 24
  • The evaluation finds little potential from text-only test-time thinking for visual problems that cannot be losslessly represented as dense natural-language captions.The authors suggest exploring visual edits during reasoning or external visual modules beyond simple image cropping.Source: §5 · p. 24
  • The current evaluation has not shown much further benefit from the strongest reasoning variants, and many selected tasks are nearly saturated with medium effort.The authors call for more creative benchmarks and evaluation methods, ideally better aligned with real-world utility.Source: §5 · p. 24

Contributors (in alphabetical order)

The section lists the paper’s contributors in alphabetical order.

Contributors (in alphabetical order)

  • The listed contributors include Bowen Qin, Chen Yue, Fang Yin, Hui Wang, and JG Yao.The passage notes that JG led the project and specifically acknowledges Bowen and Yesheng.Source: Contributors (in alphabetical order) · p. 25
  • The contributor list continues with Jiakang Liu, Jing-Shu Zheng, Miguel Hu Chen, Richeng Xuan, and Shibei Meng.Source: Contributors (in alphabetical order) · p. 25
  • The remaining listed contributors are Shiqi Zhou, Teng Dai, Tong-Shuai Ren, Wei Cui, Xi Yang, Xialin Du, Xiaojing Xu, Xue Sun, Xuejing Li, Yaming Liu, Yesheng Liu, Ying Liu, Yonghua Lin, Yu Zhao, Yunduo Zhang, Yuwen Luo, Zheqi He, Zhiyuan He, and Zhongyuan Wang.Source: Contributors (in alphabetical order) · p. 25

Ethics Statement

The ethics statement frames the study as an effort to inform future capability, risk-control, and behavioral-monitoring work, while warning against exploiting the reported weaknesses.

Ethics Statement

  • The authors hope the findings will inspire follow-up work on model capabilities, risk control, and behavioral monitoring.They also state that the findings should not be further analyzed to exploit weaknesses in LRMs or AI systems for unethical use.Source: Ethics Statement · p. 25
  • The authors caution against further analysis of the findings for unethical exploitation of current AI-system weaknesses.This warning accompanies their stated hope that the work informs future model improvements and risk control.Source: Ethics Statement · p. 25

A.1 List of LLMs and VLMs evaluated

  • The appendix lists the evaluated LLMs in Table 15 and VLMs in Table 16.The text notes that fewer LRMs support native visual input, so some earlier non-reasoning models are included in the visual evaluation.Source: §A.1 · p. 31, Table 15 · p. 31, Table 16 · p. 31

B.1 LLM-assisted analysis on reasoning traces

The appendix describes LLM-assisted rubric analysis of reasoning traces, with variants for visual and safety questions, and reports crop-mention patterns alongside illustrative failures.

B.1 LLM-assisted analysis on reasoning traces

  • The analysis template gives an LLM the question, reasoning trace, final answer, reference answer, and numbered rubric questions to assess.The work uses gpt-4.1-mini for the analysis template described here.Source: §B.1 · p. 31

B.1.1 Default rubrics

  • Default textual-problem rubrics check answer matching, reasoning-answer consistency, guessing, certainty, and redundant failed trials.They also ask about claimed web searches and external tools, formatting instructions, and ignored negative constraints.Source: §B.1.1 · p. 32

B.1.2 Rubrics for visual questions

  • The visual-question rubrics adapt the default checks and add whether reasoning claims image cropping, zooming, rotation, or image-processing tools.They also retain checks for answer matching, inconsistency, guessing, certainty, redundancy, external tools, and instruction adherence.Source: §B.1.2 · p. 32

B.1.3 Rubrics for safety analysis

  • Safety-analysis rubrics ask whether answers give harmful detail and whether reasoning contains harmful content or recognizes controversy.They also check requested answer detail, certainty mismatch, and contradictions between reasoning and final answers.Source: §B.1.3 · p. 33

B.2 Image crop statistics

  • The authors cannot determine from API calls and metadata whether reported image cropping actually occurred or was hallucinated.The crop statistics therefore count mentions in generated text rather than verified image operations.Source: §B.2 · p. 33
  • Regex counts show Gemini models, especially Gemini-2.5-pro and Gemini-2.5-flash with thinking, mention cropping more often across four visual categories; Claude models show very few such mentions.The counts track keywords such as “crop” or “zoom in”; Table 17 reports the percentages.Source: §B.2 · p. 33, Table 17 · p. 33

C Qualitative Examples

  • The qualitative examples use purple text for highlighted portions, blue for different components, and cyan for clarifying comments.The section presents examples of VLM capabilities or limitations from responses to varied prompts.Source: C Qualitative Examples · p. 33

C.1 Example: Claude Sonnet 4 on a game theory problem

  • In the game-theory example, the reasoning uses a non-standard cost model that omits baseline edge travel time and then gives an unsupported numeric answer despite deriving a parameter-dependent ratio.The figure describes these as methodological and logical errors.Source: Figure 2 · p. 34, §C.1 · p. 34

C.2 Example: Inconsistent reasoning and answer from Gemini 2.5 Flash

The Gemini 2.5 Flash example contrasts a reasoning trace that settles on Mango and Walrus with a final response that gives Worm instead.

C.2 Example: Inconsistent reasoning and answer from Gemini 2.5 Flash

  • The trace repeatedly frames the puzzle as finding a fruit and animal linked by flipping the first letter between M and W.It reaches the Mango/Walrus pair despite the response instead explaining Mango/Worm.Source: §C.2 · p. 35
  • The final response justifies Worm by the M-to-W letter flip and adds that worms may be found in mangoes.That explanation differs from the reasoning’s stated Mango/Walrus answer.Source: §C.2 · p. 35
  • Gemini 2.5 Flash’s reasoning concludes “Mango” and “Walrus,” but its response gives “Worm,” which never appears in the reasoning summary.The paper notes that neither word pair is correct.Source: §C.2 · p. 35, Figure 3

C.3 Example: Inconsistent reasoning and answer + hallucinated program for Gemini 2.5 Pro

The examples show Gemini 2.5 Pro giving a response absent from its reasoning while claiming program use, and Claude Sonnet 4 varying in whether it abstains on long-tailed facts.

C.3 Example: Inconsistent reasoning and answer + hallucinated program for Gemini 2.5 Pro

  • Gemini 2.5 Pro’s reasoning settles on Spain and Spinrad and claims a program was written, but its response gives Latvia and Valiant, absent from the reasoning summary.The figure describes the reasoning answer as wrong and the final answer as correct, while noting the claimed program.Source: §C.3 · p. 36, Figure 4

C.4 Example: Abstention from Claude Sonnet 4, if thinking turned on

  • In one long-tailed factoid example, Claude Sonnet 4 abstains after reminding itself of uncertainty rather than giving an incorrect response.The report presents this as an example with thinking enabled.Source: §C.4 · p. 37, Figure 5 · p. 37

C.5 Example: Claude Sonnet 4 fails to abstain even with awareness of uncertainty

  • In another long-tailed knowledge example, Claude Sonnet 4 gives a deterministic answer despite explicitly reminding itself to be cautious.The report describes this as a failure to abstain even with awareness of uncertainty.Source: §C.5 · p. 38, Figure 6 · p. 38

C.6 Example: Hallucinated Web Search by Qwen3-235B-A22B-thinking-2507

The examples show Qwen3-235B-A22B-thinking-2507 claiming to verify an incorrect answer through reliable sources, and DeepSeek-R1 providing detailed jailbreak instructions despite recognizing potential controversy.

C.6 Example: Hallucinated Web Search by Qwen3-235B-A22B-thinking-2507

  • Qwen3-235B-A22B-thinking-2507 says it checked reliable sources, yet confidently gives the wrong 2022 TGA Players’ Voice winner, Elden Ring rather than Genshin Impact.The example highlights a mismatch between its stated verification and its answer.Source: §C.6 · p. 39
  • Its reasoning settles on Elden Ring through unsupported recollection and imagined confirmation, then presents the answer as definitive.The reasoning mentions checking reliable sources but the model has no web search or external tools.Source: §C.6 · p. 39

C.7 Example: DeepSeek-R1 Jailbreak

  • DeepSeek-R1’s example shows awareness of potential controversy alongside detailed scripts and steps in both reasoning and its response.Source: Figure 8 · p. 40

C.8 Example: Claude Sonnet 4 on an economic problem

These examples illustrate how test-time thinking can change models’ handling of visual or diagram-based tasks, while also showing uncertainty and errors in their reasoning. The cases span economic and computer-science problems, geometry, chart reading, and a meme.

C.8 Example: Claude Sonnet 4 on an economic problem

  • In the economics example, Claude Sonnet 4 re-examines its initial analysis, considers additional welfare losses, and gives an answer that reflects those losses.The authors conjecture that spending more tokens makes it less prone to a textbook-like interpretation and more focused on relevant information.Source: §C.8 · p. 41

C.9 Reasoning Cases: Claude Sonnet 4 on a CS problem

  • Without thinking, Claude Sonnet 4 ignores a backward-reading mechanism in all four runs; with thinking, it sometimes traces the binary digits backward and gets the answer right.Source: §C.9 · p. 41

C.10 Reasoning Cases: Claude Sonnet 4 on Geometry

  • Claude Sonnet 4’s geometry reasoning uses uncertain visual estimates, but its final response states an exact area without hedging.Source: §C.10 · p. 44

C.11 Example of diagram understanding from Gemini 2.5 Flash

  • Gemini 2.5 Flash answers April incorrectly without thinking, while its more nuanced interpretation places the chart’s peak in March and answers correctly.The thinking-enabled response reasons that the peak falls between January and April and closer to April.Source: §C.11 · p. 46

C.12 Example of meme understanding from Gemini 2.5 Flash

  • For the meme, Gemini 2.5 Flash’s no-thinking response gives a shallow reading, while its thinking-enabled response works through the crossword-style wordplay to infer the implied message.Source: §C.12 · p. 46

C.13 An example of Gemini 2.5 Flash hallucinating visual details

The examples show visual-reasoning failures ranging from hallucinated image details and searches to an incorrect spatial ordering. In one meme example, thinking-enabled Gemini 2.5 Flash also infers the intended wordplay.

C.13 An example of Gemini 2.5 Flash hallucinating visual details

  • Gemini 2.5 Flash invents details after claiming to crop or zoom an image, with four runs producing four different hallucinated names.Similar behavior is reported for Gemini 2.5 Pro, and the image text was too blurry for people to recognize.Source: §C.13 · pp. 46, 48
  • In the meme example, Gemini 2.5 Flash’s thinking-enabled response interprets the crossword-style arrangement as “YOU COMPLETE ME.”Source: §C.13 · p. 47

C.14 An example of Gemini 2.5 Pro hallucinating image search

  • Gemini 2.5 Pro claims to have reverse-image-searched a photo and uses imagined sources to support Lille, although a manual search reached Leuven.The report says similar behavior appears frequently in this category and often supports a wrong answer.Source: §C.14 · p. 49

C.15 An example of Gemini 2.5 Pro on spatial reasoning

  • In a spatial-reasoning example, Gemini 2.5 Pro gives the incorrect closest-to-farthest order B, C, D, E, A instead of D, C, B, A, E.The report notes variance across runs and repeated reconsideration within one reasoning trace.Source: §C.15 · pp. 50–51

D Detailed Results and Statistics

The appendix provides category-level plots of mean and standard deviation for accuracy versus token use, alongside problem-level token-distribution plots. These figures cover textual and visual task categories.

D.1 Statistical plots

  • Figures 17–24 plot mean and standard deviation for accuracy versus token use across textual task categories.The categories include academic course problems, deciphering, Connections puzzles, long-tailed factoid QA, instruction following, long-context QA, multi-turn instructions, and NPR-style word puzzles.Source: §D.1 · p. 52, Figure 17 · p. 53, Figure 18 · p. 53
  • Figures 25–32 provide the same accuracy-versus-token plots for visual task categories, including academic problems, diagrams, geolocation, memes, multi-image analysis, puzzles, recognition, and spatial reasoning.Source: Figure 25 · p. 57, Figure 26 · p. 57, Figure 27 · p. 58
  • Figures 33–41 show problem-level token distributions for visual problems overall and for visual task categories.The listed categories include academic problems, diagrams, geolocation, memes, multi-image analysis, puzzles, fine-grained recognition, and spatial reasoning.Source: Figure 33 · p. 61, Figure 34 · p. 62, Figure 35 · p. 63

Related papers

Loading 2509.17177v3…