PaperlayerView exact revision on arXiv

Source-linked AI summary

WordPolo: Evaluating Language Models Through Iterative Semantic Feedback

Tyler McDonald, Ali Emami

arXiv:2609.19006v1cs.CLcs.AI

TL;DR

Conventional benchmarks measure whether language models answer correctly but provide limited evidence about how they reason. WORDPOLO makes semantic search iterative and observable, and its evaluations show that progression can reveal distinctions that accuracy misses.

  • Problem

    Benchmarks commonly measure answer correctness without showing the quality or faithfulness of the reasoning process, limiting what outcome-only evaluation reveals.

    Source: §1 · pp. 1–2

  • Method

    WORDPOLO has players find a target through iterative guesses and semantic-rank feedback, making their reasoning progression directly observable.

    Source: §1 · pp. 1–2, §2.1 · p. 2

  • Results

    Evaluations show that accuracy alone can conceal meaningful progress differences, while reasoning models may underthink or overthink and successful players use adaptive exploration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WORDPOLO supports process-aware evaluation by exposing reasoning traits that traditional metrics may hide.

    Source: §7 · p. 8

    Human evaluations are comparative rather than a performance ceiling, and the authors do not claim humans are WORDPOLO’s gold standard.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) and Large Reasoning Models (LRMs) are typically evaluated on challenging benchmarks through dataset accuracy alone, providing no insight into the quality or faithfulness of their reasoning processes. We present WordPolo, a word-finding task where participants must discover an unknown target word using semantic similarity feedback. Players start with zero knowledge, make guesses, and receive distance scores (1 = correct, higher = further away). Success requires interpreting scores to navigate semantic space and systematically narrow the search. This design makes iterative reasoning and adaptive search strategies both directly observable and necessary for success. We evaluate recent LLMs (GPT-4.1, Llama 4, Claude 3.5 Haiku, Qwen 3), LRMs (o4-mini, Deepseek-R1), humans, and a novel heuristic on 1,500 puzzles. Beyond solve rates (which range from 4% to 62%), we introduce progression-based metrics that reveal models often make meaningful progress, insights that accuracy alone would miss. Our analysis shows how reasoning models can be hindered by overthinking and underthinking, while successful models exhibit human-like strategies. WordPolo demonstrates the need for benchmarks that test both reasoning process and outcomes, providing holistic measurements of model capabilities. Our code and dataset can be found at https://wordpolo-demo.vercel.app/.

1 Introduction

WORDPOLO addresses the limits of benchmarks that measure answer correctness without showing how models reason. It makes semantic search iterative and observable, and its evaluations show that accuracy alone can miss both reasoning faults and meaningful progress.

1 Introduction

  • Answer-only benchmarks cannot distinguish genuine problem-solving from pattern matching, while even some process-oriented tasks can be solved without true reasoning.The authors motivate benchmarks in which each step depends on interpreting earlier results.Source: §1 · p. 1
  • WORDPOLO requires players to interpret semantic-rank feedback and refine their search at every step, making progress depend on iterative reasoning.A guess receives rank 1 if correct and higher ranks as it moves farther from the target; players use changes in rank to guide subsequent guesses.Source: §1 · p. 1
  • On the “teacher” puzzle, o4-mini needs 80+ guesses, while GPT-4.1 progresses through “mother” in under 30 and the human follows an equally effective path through “school.”The authors describe this as a counterintuitive contrast between reasoning-token use and performance.Source: §1 · p. 2, Figure 1 · p. 2
  • WORDPOLO makes guess progressions automatically verifiable and lets evaluators observe ignored insights, repetitive patterns, and successful use of discoveries.The task uses single-word nouns, requires no expert annotation, and is described as scalable and challenging regardless of model scale.Source: §1 · p. 2
  • On 1,500 puzzles, progression metrics reveal cases where low-accuracy models make better progress than high-accuracy ones, while reasoning models can underthink or overthink.Successful players exhibit adaptive exploration, sustained improvement, and effective use of semantic discoveries.Source: §1 · p. 2

2 WORDPOLO

WORDPOLO turns semantic word search into a task where players use rank feedback to explore and narrow a search. Its 1,500-puzzle dataset combines archived puzzles with selected supplementary nouns and uses GloVe embeddings for similarity.

2.1 Game Description

  • Players receive semantic-similarity ranks from 1 for the target to higher values for farther words, and must adjust their guesses when ranks improve.The game moves from exploratory guesses toward exploitation of promising semantic directions.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 Dataset Creation

  • The dataset contains 1,500 puzzles: 979 from a public solution archive and 521 single-word nouns from the MsFit Crossword Dataset’s core split.The supplementary terms were selected based on average semantic similarity to ensure suitable difficulty.Source: §2.2 · pp. 2–3
  • Semantic similarities are computed with GloVe 6B embeddings and Faiss for efficient nearest-neighbor search, using a vocabulary of approximately 317,000 valid English words.Source: §2.2 · p. 3

3 Heuristic Baseline

The heuristic approximates human search by combining information from its best-ranked guesses into a next-guess direction. It adds noise to encourage exploration and repeats nearest-word search until solving or reaching the guess limit.

3 Heuristic Baseline

  • The heuristic starts with two random seeds from 12 common words and uses k = 5 and α = 0.01, a grid-searched configuration intended to balance exploitation and exploration.The authors say this configuration approximates novice human performance.Source: §3 · pp. 3–4
  • From the third guess onward, the heuristic selects the top-ranked guesses in its history and combines their embedding vectors into a weighted centroid.Guess vectors receive weights inversely proportional to rank; the centroid is perturbed with noise to promote exploration.Source: §3 · p. 3
  • The next guess is the nearest unguessed word to the perturbed centroid in the GloVe vocabulary, and the process continues until the target is found or 100 guesses are reached.For example, “work” and “person” can lead to the guess “employee.”Source: §3 · p. 4

4 Experimental Setup

The experimental setup evaluates language models, reasoning models, humans, and baselines under specified guess limits. It reports solve and progression metrics to capture both success and movement toward solutions.

4.1 Large-Scale LLM Evaluation

  • Four LLMs are evaluated on all 1,500 puzzles, with a heuristic and random-guess baseline; each model receives 100 guesses per puzzle.Invalid guesses and duplicates count toward the limit but receive no score.Source: §4.1 · p. 4

4.2 Human and LRM Comparison

  • A 50-puzzle comparison includes three human annotators with varying puzzle experience, three LRMs, and the same four LLMs under identical rules and guess limits.Qwen 3 is evaluated both without reasoning tokens in the large-scale evaluation and with reasoning tokens enabled in this comparison.Source: §4.2 · p. 4

4.3 Evaluation Metrics

  • Solve rate measures the fraction solved, average guesses counts guesses on solved puzzles, and proximity measures closeness on unsolved puzzles using best rank on a logarithmic scale.A best rank of 10 scores 0.80, whereas a rank of 1,000 scores 0.60.Source: §4.3 · p. 4
  • Mean progression records whether each guess improves rank, while weighted progression rewards improving guesses according to the new rank and assigns zero to regressions.The weighting gives rank 50 a score of 0.73 and rank 25,000 a score of 0.45, emphasizing late-game refinement.Source: §4.3 · p. 4

5 Results

Across the results, solve rates vary widely and LRMs show no systematic advantage; progression metrics also expose meaningful progress and distinct failure patterns. Successful players adapt across semantic domains, use feedback, and sustain improvement.

5.1 How well do LLMs & LRMs perform?

  • 62% solve rate made o4-mini the leader, but LRMs showed no systematic advantage; the heuristic scored 50%, near Annotator A’s 48%.The reported model solve rates ranged from 6% for Claude 3.5 to 62% for o4-mini.Source: Table 1 · p. 5, §5.1 · p. 5
  • o4-mini scored .478/.306 and GPT-4.1 .466/.295 on Mean/Weighted Progression; disabling Qwen 3 reasoning raised solve rate 12%→36% and proximity .595→.658.The progression metrics capture consistent improvement and give extra weight to late-game refinement.Source: §5.1 · p. 5
  • Llama 3.1 solve rate rose 0%→8%→36% across 8B/70B/405B, while proximity peaked at 70B; alternate embeddings preserved tested models’ rankings.The alternate embedding tests covered GPT-4.1, Claude 3.5 Haiku, and Llama 4.Source: Table 2 · p. 5, §5.1 · p. 5

5.2 Why do reasoning models underperform?

  • Reasoning failures included overthinking (response-limit and irrelevant-term errors), underthinking (fixation and selective history), and output errors (unfaithful guesses or instruction violations).Output errors can cost credit despite sound reasoning.Source: §5.2 · pp. 5–6, Figure 4 · p. 6
  • Qwen 3 primarily hit response limits while Deepseek-R1 more often failed exploration; one overthinking episode wastes 5–10 guesses, versus 50+ for exploration failures.Requiring explicit explanations lowered o4-mini’s solve rate from 62% to 48%; regex-based stripping of Markdown errors had mixed results and did not fully explain performance gaps.Source: §5.2 · p. 6

5.3 How do successful players differ?

  • Humans favor categorical opening guesses, while models follow prompt exemplars and switch to new exemplars under altered prompts; stronger models explore multiple categories early.Claude 3.5’s “water” and “ocean” guesses appeared in 569/1500 puzzles.Source: §5.3 · p. 6, Figure 5 · p. 7, Figure 7 · p. 7
  • For unsolved puzzles, o4-mini improved across 100 guesses, while Qwen 3 with reasoning improved for ∼20 guesses before plateauing near rank 6000.o4-mini’s progression mirrored the human average; Qwen 3 with reasoning had a 12% solve rate.Source: §5.3 · p. 6, Figure 6 · p. 7

6 Related Work

Prior multi-step benchmarks often report final success without diagnosing reasoning quality, while process-evaluation approaches may require post-hoc analysis or human judgment. WORDPOLO instead makes search strategies observable through guess progress and automatically scored semantic proximity.

Multi-Step Reasoning & Puzzle-Inspired Tasks

  • Many puzzle and multi-step benchmarks provide little insight into reasoning quality, whereas WORDPOLO combines an unconstrained search space with immediate, noisy feedback.Its guess progression makes strategic reasoning observable and essential to success.Source: §6 · p. 7

Metrics for Process Evaluation

  • WORDPOLO uses objective semantic-proximity scores to evaluate reasoning automatically, including cases where models make progress but fail to solve the puzzle.Low ranks can reveal strong reasoning that accuracy alone would miss.Source: §6 · p. 8

Systematic Failures in Reasoning Models

  • WORDPOLO exposes overthinking, analysis paralysis, and underthinking through guess progressions, where these faults may prevent guesses or cause early plateaus.Overthinking or analysis paralysis may prevent guesses from being made; underthinking may limit exploration beyond initial categories.Source: §6 · p. 8

7 Conclusion

The paper presents WORDPOLO as a semantic-feedback task for evaluating adaptive exploration and language-model performance. Its findings show that accuracy alone misses reasoning faults and that successful players use adaptive exploration, sustained improvement, and semantic discoveries.

7 Conclusion

  • WORDPOLO evaluates adaptive exploration through semantic feedback and compares the performance of language models and reasoning models.The authors report that accuracy alone does not fully capture performance.Source: §7 · p. 8
  • Reasoning models may exhibit overthinking, underthinking, and unfaithfulness, while successful players use adaptive exploration, sustained improvement, and semantic discoveries.The authors argue that process-based insights reveal traits traditional metrics may hide.Source: §7 · p. 8

Limitations

The authors acknowledge limits in resisting benchmark gaming, real-world applicability, embedding coverage, and the interpretation of human comparisons. They describe these as scope boundaries and identify possible directions for future benchmark variants.

Limitations

  • Sufficiently motivated actors could game WORDPOLO by training on guess sequences rather than only answers.The authors suggest dynamic puzzle generation or adversarial modifications as possible ways to resist optimization.Source: Limitations · pp. 8–9
  • WORDPOLO has limited real-world application, and building domain-specific process benchmarks would require expertise and resources the authors lack.They suggest code debugging, medical diagnosis, and customer-service dialogues as possible domains for future benchmarks.Source: Limitations · p. 9
  • GloVe may encode biases and lacks modern slang, recent concepts, and nuanced cultural meanings.Alternative embeddings could create new variants, but model-specific representations could complicate cross-model comparison.Source: Limitations · p. 9
  • The authors treat human evaluations as strategy comparisons, not as a performance ceiling, because human approaches vary in expertise.They use comparisons to identify effective strategies rather than declare winners.Source: Limitations · p. 9

A Data Collection

The appendix describes the puzzle data and evaluation setup, a heuristic configuration and algorithm, and two studies of prompt examples and guess-history presentation. The prompt study reports changed solve rates and progression metrics, while the presentation study finds core results largely stable.

A Data Collection

  • The 1,500-puzzle dataset combines a public Contexto solutions archive with the MsFit Crossword Dataset.The section outlines the collection and filtering steps for these two component datasets.Source: A Data Collection · p. 12

A.1 WORDPOLO Solutions

  • The WORDPOLO solutions archive supplied 979 entries, retained without filtering by criteria such as difficulty, spanning September 18, 2022, to May 24, 2025.The authors plan monthly updates to the collected solutions.Source: §A.1 · p. 12

A.2 MsFit Crossword Dataset

  • The authors added 521 single-word nouns from MsFit, selected from its core split by frequency after filtering and lemmatization.The 42,831-entry core split was filtered to nouns, then wordfreq rankings selected the top 521 unique terms.Source: §A.2 · p. 12

A.3 Human Evaluation Subset

  • The human-evaluation subset contains 50 puzzles across five categories, with category information obscured from evaluators.The categories are person, organism, location, object, and abstract concept; the puzzles were shuffled before evaluation.Source: §A.3 · p. 12

A.4 Heuristic Seed Words

  • The heuristic starts with two randomly selected guesses from 12 seed words drawn from a Contexto solution archive.The seed list covers broad concepts such as person, animal, thing, nature, home, and technology.Source: §A.4 · p. 12

B.1 Heuristic Grid Search

  • The heuristic configuration was selected through a grid search across the full 1,500-puzzle dataset, with Figure 9 showing weighted solve rates.The passage does not specify the grid values or identify a best configuration.Source: §B.1 · p. 13

C.1 Altered Prompt Study

  • Claude 3.5 solve rate rose from 0.06 to 0.08 and Llama 4 from 0.16 to 0.26 with the altered prompt, while proximity scores generally decreased.Guess counts and weighted progression fluctuated; weaker LRMs improved on proximity score and mean progression, while LRM solve rates and average guess counts remained relatively unchanged.Source: Table 3 · p. 13, §C.1 · p. 13

D.1 Trial-Order vs. Rank-Order

  • Trial-order versus rank-order presentation left GPT-4.1’s solve rate unchanged; other model results showed minor fluctuations, and core findings were largely stable.Claude 3.5 Haiku’s proximity score was an exception, dropping under rank-ordered histories.Source: §D.1 · p. 13, Table 4 · p. 14

D.2 Full Rank-Order vs. Truncated Rank-Order

Rank-order versus trial-order presentation had little effect, but truncating guess histories split model responses: stronger models benefited, while Claude 3.5 Haiku may have lost useful signals. The analysis links effective reasoning to balancing promising feedback against exploration of new directions.

D.2 Full Rank-Order vs. Truncated Rank-Order

  • The n = 50 comparison tested full and top-10 truncated rank-ordered guess histories for GPT-4.1, Llama 4, and Claude 3.5 Haiku.Table 5 reports solve rate, average guesses, proximity score, and weighted progression.Source: §D.2 · p. 14, Table 5 · p. 14
  • Rank-order versus trial-order presentation had little to no effect, while truncation benefited GPT-4.1 and Llama 4 but may have cost Claude 3.5 Haiku useful signals.The authors suggest that removing low-ranked guesses reduces noise for stronger models, while weaker models may lose useful accumulated signals.Source: §D.2 · p. 14, Table 5 · p. 14
  • Effective reasoning requires balancing exploration of promising feedback with exploration of new directions.Source: §D.2 · p. 14

E Linear vs. Logarithmic Proximity Score

A linear proximity score compresses differences between low-ranked guesses, whereas logarithmic scaling produces more interpretable separation. In the reported comparison, GPT-4.1 and Claude 3.5 Haiku differ by 0.003 on the linear score but by 0.099 on the logarithmic score.

E Linear vs. Logarithmic Proximity Score

F Statistical Significance & Confidence Intervals

The supplementary analyses test confidence intervals, embedding changes, Markdown handling, and required explanations. They report preserved model rankings across embeddings, mixed results for Markdown stripping, and reduced o4-mini performance when explanations were required.

F Statistical Significance & Confidence Intervals

G Embedding Generalization

  • Across substantially different embedding spaces, model rankings remained GPT-4.1, then Llama 4, then Claude 3.5 Haiku.The embedding comparison also reports an inverse relationship between solve rate and weighted progression; the authors hypothesize that embedding density may contribute to the trade-off.Source: G Embedding Generalization · p. 15, Table 8 · p. 15
  • On GloVe Dolma, solve rates were worse as weighted progression rose, while on fastText weighted progression worsened as solve rates rose.Source: G Embedding Generalization · p. 15, Table 8 · p. 15

H Markdown Handling

  • Markdown stripping produced mixed results: Qwen 3's proximity score rose from 0.595 to 0.660 as its solve rate fell, while Deepseek-R1's metrics remained largely unchanged.The authors conclude that formatting errors do not fully explain performance gaps.Source: H Markdown Handling · p. 16, Table 9 · p. 16

I Explanation Study

  • Requiring o4-mini to explain before each guess reduced solve rate from 62% to 48% and proximity score from 0.726 to 0.693.The authors suggest added deliberation may be redundant with existing reasoning or conflict with training objectives.Source: I Explanation Study · p. 16, Table 10 · p. 16
Loading 2609.19006v1…