WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot

1. Introduction

• Existing evaluations often provide task-relevant premises, leaving open evidence identification and how information presentation guides judgment outside their central scope.Longer texts or added distractions do not by themselves test how presentation affects evidence judgment.Source: §1 · pp. 1–2
• WORTH asks models to select inferential sentences from unmarked narratives and judge participants where cooperative and adversarial information coexist.Its mystery materials are professionally designed to conceal reasoning cores through narrative techniques.Source: §1 · p. 2
• The evaluation quantifies key-clue selection and participant attribution, while using natural-language explanations diagnostically rather than scoring them independently.Models read the complete story, select a limited quota of key clue sentences, and identify participants from candidate characters.Source: §1 · p. 2
• The pilot reports three observable patterns: clue and participant performance can diverge, models can converge on the same incorrect characters, and suspicion information is favored over counter-evidence.The paper presents these as behavioral structures and does not infer internal cognitive mechanisms.Source: §1 · pp. 2–3
• This is a full-story pilot, not a mature leaderboard or evidence for a universal model mechanism.The paper proposes a diagnostic perspective for cases where models must find where reasoning should begin without pointed-out evidence.Source: §1 · p. 3

2.1 Common Premises of Existing Reasoning Evaluations

• Many reasoning benchmarks organize the task, response space, and premises in advance, and generally do not make evidence identification or narrative presentation a core variable.Increasing retrieval difficulty does not necessarily test whether models independently judge what counts as evidence.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Attempts to Partially Alter These Premises

• Related approaches cover long-context retrieval, artificial distraction, evidence detection, abductive reasoning, and explanation evaluation, but differ from WORTH in task design or output.For example, HotpotQA and DetectBench quantify evidence-related outputs, while WORTH emphasizes original narratives combining evidence layout and obfuscation.Source: §2.2 · pp. 4–5

2.3 Evaluations Related to Mystery Fiction and Detective Narratives

• Mystery-fiction evaluations commonly ask for conclusions, but often do not require itemized selection of key evidence from unmarked narratives.DetectiveQA uses pre-written questions; WHODUNIT asks for the culprit from public-domain novels, with a stated risk of plot contamination.Source: §2.3 · pp. 5–6

2.4 Positioning of WORTH

• WORTH’s positioning combines autonomous clue selection, author-designed informational obfuscation, and separate scoring of clues and participant judgments.The paper describes these as a combined framework rather than claiming to pioneer any single condition.Source: §2.4 · p. 6
• The authors qualify their literature positioning by noting that they have not exhaustively covered the relevant research domains.Source: §2.4 · p. 6

3.1 Autonomous Evidence Discovery and the Target Capability Chain

• WORTH separates clue localization, analysis and reasoning, and participant attribution; current scores quantify clue overlap and participant judgments, not analysis itself.Clue selection uses a quota, while participant attribution rewards correct selections and penalizes incorrect ones.Source: §3.1 · p. 7

3.2 Not-Fully-Cooperative Informational Relationships

• In WORTH, adversariality describes truthful information whose placement, salience, or framing impairs correct judgment, rather than falsehood or random noise.The episode example presents a left-leg injury paired with crutch use on the left side, a designed anomaly obscured by a plausible causal sequence.Source: §3.2 · pp. 7–8

3.3 Why Professionally Crafted Mystery Narratives Are Appropriate Experimental Vehicles

• Professionally authored mystery narratives are designed to contain sufficient clues while delaying revelation through narrative techniques.WORTH treats this as an intrinsic narrative structure rather than post-hoc artificial noise.Source: §3.2 · p. 8, §3.3 · p. 8
• Original materials let authors establish clue function and salience during composition, while reducing—but not eliminating—prior-exposure risk.Published novels may not reveal which details were intentionally planted as critical clues; original materials cannot guarantee models have never encountered the stories.Source: §3.3 · p. 8
• The material-production approach requires narrative design, evaluation-methodology, and diagnostic-analysis capabilities, whose integration lacks standardized workflows in this project.Model findings can be mapped back to story design to calibrate later evidence structures, but formalizing that process remains future work.Source: §3.3 · p. 9

3.4 Current Implementation Scope and Subsequent Verification Conditions

• Only the full-story condition is implemented; evidence-cueing and evidence-modification are proposed verification conditions, not empirical evidence in this paper.Cueing marks author-designated clues for comparison, while modification changes key evidence and observes whether judgments change.Source: §3.4 · p. 9

4.1 Task Materials

• Each episode presents an original Chinese mystery as numbered natural sentences; Episode 01 has 120 sentences and 6 candidates, while Episode 02 has 134 sentences and 9 candidates.The episodes also differ in key-clue counts and weights, reflecting evaluation iteration rather than an intended difficulty gradient.Source: §4.1 · p. 10

4.2 Design Status of Key Clue Sentences

• Key clues are author-designated inferential hubs set during story design, but the set is not exhaustive, so scoring measures overlap with those hubs rather than all valid evidence.Unannotated sentences may provide auxiliary inferential value, and multiple reasoning paths may reach the correct conclusion.Source: §4.2 · p. 10

4.3 Response Protocol

• After reading the complete story, subjects select up to 12 or 20 clue sentences, choose any number of candidate participants, and provide natural-language reasoning.The response protocol requires structured fields and identifier checks; Episode 02 defines participants to include planners, executors, and critical helpers.Source: §4.3 · p. 11

4.4 Scoring Information Disclosure

• Subjects learn general scoring rules but not differentiated clue values or incorrect-character penalties, to reduce strategic behavior unrelated to narrative reasoning.The paper describes this as controlling reasoning-irrelevant strategic factors, not measuring pure reasoning ability.Source: §4.4 · pp. 11–12

4.5 Scoring Rules

• Clue hits earn points without deductions for non-key selections, whereas incorrect participant choices incur penalties; neither total nor clue scores comprehensively measures reasoning capability.Episode 01 uses uniform clue weights and character penalties; Episode 02 differentiates both clue weights and incorrect-character penalties.Source: §4.5 · p. 12

4.6 Models and Run Configurations

• Each model was run five times, producing 70 formal runs; Episode 02 had 37 content-scorable runs after three invalid selections were excluded rather than scored as zero.The experiments used fresh sessions and single-turn complete prompts; model configurations and versions were not uniform across episodes.Source: §4.6 · pp. 12–13, §4.7 · p. 13

4.8 Criteria for Human Player Data

• Human records remain uncleaned, and human players’ incremental game play differs from models’ single-turn prompts, so this version reports no human-model comparison.Episode 02 participant judgments also have a construct boundary: the prompt does not delimit which homicide case is in scope, and “participant” retains borderline ambiguity.Source: §4.1 · p. 10, §4.8 · p. 13, §4.9 · p. 14

4.10 Cross-Episode Reporting Principles

• The two episodes use different stories, evidence structures, candidate sets, and scoring parameters, so raw scores are reported separately rather than compared as difficulty measures.Cross-episode comparisons are limited to structural patterns observable in both episodes.Source: §4.10 · p. 14

5.1.1 Finding Clues Does Not Imply Identifying Participants

• EP01 clue hits and participant judgments diverged: GPT-5.6 Sol averaged 3.4/12 clue hits and 0.8/3 correct participants, while DeepSeek-V4-Flash averaged 2.2/12 and 1.6/3.DeepSeek-V4-Flash had zero incorrect selections across five runs; GPT-5.6 Sol averaged 0.6 incorrect participants per run.Source: §5.1.1 · pp. 15–16

5.1.2 Case Observations of Erroneous Judgments

• EP01 case observations included selecting two non-correct characters, unsupported speculation about tampering, and no selection of the correct first-person narrator across 30 runs.Gemini 3.1 Pro’s five total scores took only two values—three runs at 6 and two at 19—while its confidence field was 100 in every run; the authors caution that this does not reveal how it interprets confidence.Source: §5.1.2 · p. 16, §5.1.3 · p. 16

5.2 Episode 02

• In EP02, Kimi K3’s response03 hit 10 key clues but selected no correct participants and misselected five characters; GPT-5.6 Sol’s response01 hit five clues and selected correct participants without incorrect ones.GPT-5.6 Sol’s response01 scored 16/47, the episode high; it selected Brother Lei and Geng Baishun but missed Geng Baili.Source: §5.2 · p. 17

5.2.1 Common Attractor: Convergent Errors Across Models

• 31/37 content-scorable EP02 runs selected the same incorrect trio: Father, Jia Changgeng, and Director Huang.The trio appeared across all eight models; Jia was a high-salience suspect, Director Huang a distractor, and Father belonged to EP01’s participant set but not EP02’s.Source: §5.2.1 · pp. 17–18

5.2.2 Evidence Asymmetry: S112 vs. S113

• 25/37 runs selected suspicion-oriented S113, while 0/37 selected adjacent exoneration-oriented S112.The reported contrast concerns selection frequencies; the sentences’ adjacency does not establish equal processing weights or a uniform rationale for selecting S113.Source: §5.2.2 · p. 18

5.2.4 The S124 Phenomenon: Boundaries Between Puzzle Premise and Scored Clues

• 37/37 EP02 runs selected S124, the core puzzle premise, although it was outside the author’s scored key-clue set.Its universal selection distinguishes failing to hit a scored clue from selecting information that lacks inferential utility.Source: §5.2.4 · pp. 18–19

5.2.6 Predominance of Negative Scores and Score Distribution

• 28/37 EP02 runs had negative gameplay-equivalent scores, as penalties for incorrect participants exceeded clue-hit points in most runs.The intended participant chain required integrating evidence dispersed across low-salience positions; 13 of the 15 runs identifying at least one correct participant also selected Jia Changgeng.Source: §5.2.6 · p. 19, §5.2.7 · p. 19

5.3 Structural Patterns Shared Across Both Episodes

• Both episodes showed clue-hit/participant-selection dissociation and suspicion-versus-exoneration asymmetry, but the authors limit these patterns to the pilot’s mystery narratives and designs.EP01’s two cited exoneration sentences were each selected 0/30 times, while suspicion-oriented S094 was selected 22/30 times; the cross-episode comparison concerns structural patterns, not raw scores.Source: §5.3.1 · p. 20, §5.3.2 · p. 20, §5.3 · p. 20

6.1 Process Dissociation: Phenomena Manifested Under Autonomous Evidence Discovery

• Separate outputs exposed process dissociation: in EP01, GPT-5.6 Sol hit more key sentences but identified fewer characters, while DeepSeek-V4-Flash showed the inverse pattern.The authors caution that these outputs do not isolate internal reasoning steps and that 70 runs across two narratives are insufficient for stable estimates.Source: §6.1 · p. 21

6.2 Common Attractor: Empirical Metric of Narrative Design Effectiveness

• In EP02, 31/37 runs selected the same incorrect participant trio, a convergence aligned with the story’s intended suspicion-steering direction.The authors cannot disentangle insufficient model vigilance, potent story salience, or their joint effect; subsequent controls are proposed.Source: §6.2 · pp. 21–22

6.3 Selection Boundaries Between Puzzle Premise and Evidence

• All runs selected S124, the death-method and footprint-constraint sentence, although it was not among the author-designated key clues.Its selection alone does not establish whether models distinguished puzzle premises from clues, and does not by itself warrant changing the scoring set.Source: §6.3 · pp. 22–23

6.4 Evidence Asymmetry and Epistemic Vigilance

• S113’s suspicion cue was selected in 25/37 runs, while adjacent S112’s qualified alibi evidence was selected in 0/37.A similar suspicion-versus-exoneration asymmetry appears across both episodes, but the evidence is limited to mystery narratives and requires independent verification elsewhere.Source: §6.4 · p. 23

6.6 Design Justification and Pilot Feedback

• The author-designed key-clue set K comprises reasoning hubs planted during story design, not sentences annotated as important after the fact.Five EP02 clues with zero hits corresponded to inferential steps for ruling out false hypotheses, though their discriminatory power cannot be validated in this sample.Source: §6.6 · p. 24
• The prompt disclosed reward and penalty directions but concealed specific scoring weights; no disclosure-control experiment has tested the effect of that choice..Source: §6.6 · p. 24

6.7 Human Data

• Human player records remain uncleaned, so this version reports no human scores or human-model comparisons.Once verifiable data are available, player responses could help distinguish item effects from model-specific behaviors.Source: §6.7 · p. 24

7.1 Scope of Applicability of Results

• The pilot covers two episodes and 70 model runs, so model means summarize this execution rather than establish general capability rankings.Observed patterns require additional stories and runs to test replicability.Source: §7.1 · p. 25

7.2 Content Covered by Current Scoring

• Current scoring quantifies clue and participant selections, while explanations are retained for diagnosis but not independently scored or tested for inter-rater reliability.Neighboring sentence selections may convey information without matching designated clues, motivating future functional coding of selections and explanations.Source: §7.2 · p. 25

7.3 Subsequent Verification

• Proposed verification includes highlighting key clues and modifying evidence while preserving story coherence; neither control has yet been implemented.Human records also remain uncleaned, and future work proposes same-task comparisons plus additional stories and evidence-modification conditions.Source: §7.3 · p. 25

• WORTH asks models to select key clues, identify case participants, and explain their reasoning from complete professional narratives, separately recording evidence selection and participant attribution.Source: §8 · p. 25
• Across two episodes, 70 runs included 67 content-scorable runs; results show that more clue hits do not guarantee more accurate participant judgments, alongside shared distractor selections and suspicion–exoneration selection asymmetry.Source: §8 · pp. 25–26
• Separately observing evidence selection and final judgment can reveal behavioral differences obscured by a single total score, but further stories and evidence-cueing or evidence-modification tests are needed.Source: §8 · p. 26

• Each run supplied the complete Chinese story, neutral sentence IDs, candidate characters, and instructions; models returned up to the clue quota, selected participants, and completed seven structured fields.Clue interpretations had to correspond one-to-one with selected sentence IDs; runs used fresh conversations with external tools, memory, and personalization disabled.Source: §A.1 · p. 26

A.2 Episode-Level Scoring Summary

• EP01 allows up to 12 clue selections from 120 sentences, with 12 key sentences worth 3 points each (36 total).Source: §A.2 · p. 26
• EP01 has six participant candidates and three correct participants, scored +7, +5, and +5, respectively.Source: §A.2 · p. 26
• EP02 awards 1–4 points for each of 20 key sentences (41 total) and scores correct participants +2 each and incorrect participants −2 to −8 by character.Source: §A.2 · p. 26
• In both episodes, omitted key sentences and selections outside the author-designated set K earn zero clue points, and episode scores are neither summed nor treated as comparable difficulty measures.Source: §A.2 · p. 27

A.3 Model and Run Configurations

• Recorded tier and mode labels should not be interpreted as standardized compute levels across vendors.Source: §A.3 · p. 27
• EP01 runs were conducted on 2026-08-14 using ChatGPT web, Antigravity, Zhipu web, TraeWork, Kimi’s website, and Antigravity IDE interfaces.Source: §A.3 · p. 27
• EP02 runs were conducted from 2026-08-30 to 2026-08-31 using ChatGPT web, Claude.ai, Antigravity IDE, Zhipu website, TraeWork, and WorkBuddy interfaces.Source: §A.3 · p. 27
• EP01 GLM-5.2 is reported as High with Deep Thinking enabled; the EP02 DeepSeek tier was unrecorded, and MiniMax-M3 had no applicable interface tier label.Source: §A.3 · p. 27

A.4 Run Validity and Exclusions

• Three EP02 runs were excluded because two selected nonexistent sentence identifiers and one used illegal selection-field expressions that prevented recovery of valid identifier arrays.The excluded runs were not assigned zero scores.Source: §A.4 · p. 28
• Invalid overall JSON did not necessarily prevent scoring: readable selection arrays kept all 30 EP01 runs scorable and four of five such EP02 responses scorable.Source: §A.4 · p. 28
• One EP01 Codex run was retained only as an exploratory auxiliary control, outside the formal model set and 70 reported formal runs.Source: §A.4 · p. 28