PaperlayerView exact revision on arXiv

Source-linked AI summary

KWBench: Measuring Unprompted Problem Recognition in Knowledge Work

Ankit Maloo

arXiv:2604.15760v1cs.AIcs.GT

TL;DR

KWBench addresses whether language models can recognize the governing structure of a professional situation before attempting to solve it, a step largely omitted by saturated extraction and task-completion benchmarks. It evaluates this ability using unlabeled, realistic scenarios and finds that recognition varies sharply across models even when conditional execution quality converges.

  • Problem

    KWBench addresses the gap between executing a framed task and recognizing the right problem from a professional situation alone.

  • Method

    The benchmark evaluates 223 unlabeled professional scenarios using decomposed binary rubric criteria, including mandatory conditions that make outputs unreliable when missed.

  • Results

    The best model passes 27.9% of tasks, while conditional quality scores converge around 83%, showing that models differ more in recognition than execution after passing.

  • Takeaways & Limitations

    KWBench supports evaluating knowledge work through recognition of strategic structure, rather than relying only on execution quality after the problem has been framed.

  • Takeaways & Limitations

    The study has no human baseline and no direct ablation comparing unlabeled tasks with versions containing explicit game-theoretic hints.

Abstract

from arXiv · show

We introduce the first version of KWBench (Knowledge Work Bench), a benchmark for unprompted problem recognition in large language models: can an LLM identify a professional scenario before attempting to solve it. Existing frontier benchmarks have saturated, and most knowledge-work evaluations to date reduce to extraction or task completion against a specification. KWBench targets the step before that: recognizing the governing structure of the situation from raw inputs alone. The benchmark contains 223 tasks sourced from practitioners across acquisitions, contract negotiations, clinical pharmacy, organizational politics, fraud analysis, and incentive design. Each task encodes a formal game-theoretic pattern (principal-agent conflict, signaling, mechanism design failure, strategic omission, coalitional dynamics, strategic interdependence) and carries structured ground truth recording the expert reading of the situation and the anticipated failure modes. Models receive raw data and a task prompt with no indication of problem type. Scoring is a three-tier rubric gated by a mandatory conjunctive check. Mandatory criteria encode the predicted wrong paths. We evaluate 16 models. The best model passes on 27.9% of tasks. The top two models agree on only 31.7% of their passes. Among the top 8, 44 tasks are solved by exactly one model; routing across the top 8 covers 50.7% of the benchmark, nearly double the best single model. Conditional on passing, quality scores converge (approx 83% across models); unconditional scores do not. Same models articulate the relevant game-theoretic concept correctly when asked, then fail to apply it unprompted. We release KWBench to shift how frontier models are evaluated on knowledge work, scoring them on whether they recognize the right problem from the situation alone, not only on how well they execute once the problem has been framed for them.

1 Introduction

KWBench targets the unprompted recognition step that existing, increasingly saturated benchmarks largely omit: framing a professional situation before solving it. Models often produce polished execution on the wrong problem, and recognition—not conditional execution quality—is where performance varies.

  • Benchmark motivation and findings: Existing benchmarks largely test extraction or task completion after the problem has already been framed, rather than the practitioner’s binding framing step.Professional work requires identifying the governing game, counterparty incentives, and applicable framework from the situation itself.
  • Benchmark motivation and findings: 27.9% of tasks are passed by the best model on KWBench’s mandatory recognition gate.The rest receive zero despite potentially polished, confident, and internally consistent output.
  • Benchmark design: The benchmark supplies raw professional scenarios without labels or hints about the underlying problem type, inferring recognition from the response alone.Its design principle is to measure what models recognize without instructing them to apply adversarial reasoning.
  • Benchmark results: 82.6% and 84.1% are the conditional quality scores for the two reported model groups, while the top-8 average gate-pass rate is 17.8%.The reported variance lies in what models recognize, not how well they execute after passing.
  • Benchmark results: Models can fail the mandatory recognition criteria while still producing thorough, well-structured output and passing roughly half of non-mandatory criteria.The evaluation therefore separates execution quality from whether the underlying problem was correctly identified.
  • Benchmark results: 31.7% Jaccard overlap separates the top two models’ pass sets, while routing across the top 8 covers 50.7% of the benchmark.Among the top 8, 44 tasks are solved by exactly one model, indicating distributed recognition ability.

2 Knowledge Work as Imperfect Information Games

Professional knowledge work is modeled as imperfect-information, strategic interaction rather than a fully specified problem-solving exercise. KWBench instantiates recurring game-theoretic patterns in realistic tasks and tests recognition, framework application, and execution from raw situations.

  • Knowledge Work as Imperfect Information Games: Knowledge work differs from saturated mathematical reasoning, coding, and factual-recall benchmarks because professionals act without full access to the state.Professional decisions involve hidden intentions, capabilities, constraints, and objectives of other strategic parties.
  • Knowledge Work as Imperfect Information Games: Imperfect-information games formalize the hidden types, private signals, and strategic interaction under uncertainty that characterize many professional situations.The paper contrasts this setting with benchmarks built around fully visible states and specifications.
  • The Game-Theoretic Structure of Professional Work: KWBench covers six recurring patterns: signaling, principal-agent problems, mechanism-design failures, coalitional dynamics, strategic interdependence, and strategic omission.These patterns connect formal game structures to concrete professional manifestations across domains.
  • The Game-Theoretic Structure of Professional Work: Strategic omission makes absent information informative: missing procurement, legal, or security contacts changes the interpretation of an otherwise enthusiastic deal summary.The benchmark requires models to treat omissions as data rather than relying only on stated information.
  • Why Models Should Be Tested on This: The benchmark’s deployment relevance follows from professional work being imperfect-information by default, with hidden types, strategic counterparties, and structurally carried signals.Misframed analyses can read as competent and influence decisions without triggering ordinary numerical or computational error checks.
  • Benchmark operationalization: KWBench presents raw data and an unlabeled deliverable, requiring models to recognize the pattern, decode signals or incentives, and produce an analysis accounting for adversarial structure.Existing benchmarks commonly isolate execution after the problem has been correctly framed, whereas KWBench tests all three steps.

3 Benchmark Design

KWBench presents professional scenarios cold, using raw inputs and neutral prompts to test whether models recognize the governing reasoning pattern before solving. Its benchmark construction combines practitioner-validated task formalization, structured annotations, and evaluation findings showing recognition failures across categories.

  • Task Design: 223 tasks present raw professional inputs without hints about the problem type, framework, or applicable vocabulary.The benchmark separates training from evaluation by measuring whether models apply the right reasoning pattern from the situation alone.
  • Task Design: 185 tasks originate from real professional incidents spanning acquisitions, board disputes, compensation negotiations, and incentive-driven process failures.
  • Task Design: Each scenario is formalized as a recognized game-theoretic pattern linking raw signals to expert interpretations and rubric-tested implications.
  • Task Design: Practitioners validate scenario realism, calibrated difficulty, and alignment between documented reasoning and experienced professional practice.
  • Ground Truth and Annotations: Structured annotations decompose 223 tasks into roughly 5,800 expert-extracted signals, predicted common errors, and pre-rubric evaluation criteria.
  • Ground Truth and Annotations: 74% of tasks include a mandatory criterion explicitly testing whether the model avoids a specific wrong answer.
  • Observed Recognition: 27.9% is the best model’s recognition pass rate, while category rates range from 0% for Mechanism Design to 39% for Principal-Agent.
  • Observed Recognition: 64% of Judgment Under Pressure tasks pass, compared with below-40% rates for Adversarial Counterparty, Information & Signaling, and Temporal & Commitment.

4 Evaluation Framework

KWBench uses a three-tier rubric whose mandatory criteria gate the score on problem-recognition requirements, while additional tiers measure depth and excellence. Independent binary judging prevents tradeoffs between criteria and combines the results into a gated score.

  • Three-Tier Rubric: Each task rubric contains 15 criteria: 5 mandatory, 5 good-to-have, and 5 ideal.
  • Three-Tier Rubric: Mandatory criteria test problem recognition, good-to-have criteria test thoroughness, and ideal criteria test practitioner-level excellence.
  • Rubric Design: Criteria are binary yes/no questions that require mechanism explanations and include at least one test of the insight separating correct from incorrect analysis.
  • The Mandatory Gate: If any mandatory criterion fails, the task scores zero; otherwise, the score is 0.40 + 0.35 · ¯g + 0.25 · ¯i.Passing scores range from 0.40 when only mandatory criteria pass to 1.0 when every criterion passes.
  • The Mandatory Gate: The conjunctive gate treats a missed core failure mode as making the work product unreliable despite narrative quality or formatting.
  • Judging: A single judge evaluates each response against one rubric criterion at a time and returns pass or fail.
  • Judging: The judge runs all 15 criteria independently and combines their binary results without allowing success on one criterion to offset failure on another.
  • Evaluation Protocol: All models use the same judge, prompt, and rubric, while each model’s result is selected from its best of three evaluated runs.

5 Related Work

KWBench differs from established reasoning, agent, robustness, and professional benchmarks by evaluating recognition of the appropriate problem structure rather than execution of a known method. Its cross-domain design and wrong-path-focused rubrics target realistic failure modes before task completion.

  • Knowledge and Reasoning Benchmarks: Existing knowledge benchmarks verify correct answers, whereas KWBench makes difficulty depend on recognizing what the problem is before applying a solution method.
  • Agent and Tool-Use Benchmarks: Agent and tool-use benchmarks test execution of complex workflows, while KWBench tests identification of which workflow is appropriate.
  • Adversarial and Robustness Evaluation: Robustness benchmarks examine realistic failure modes but generally evaluate single-turn question answering rather than multi-faceted professional recognition.
  • Professional Evaluation: KWBench is cross-domain rather than field-specific and uses rubrics focused on avoiding dangerous wrong answers.
  • Evaluation Methodology: Its mandatory gate relates to hard-constraint approaches that use selected criteria as binary filters.

6 Results

KWBench results show low absolute recognition pass rates, while conditional execution quality converges across models. The benchmark’s top ranks therefore separate recognition more than execution.

  • Evaluation set: 16 models from 10 organizations were evaluated, with downstream analysis using the top 8 by coverage.The top 8 are described as the smallest set spanning all 113 currently solvable tasks.
  • Absolute performance: 27.9% is Claude Opus 4.6’s mandatory gate pass rate, while the top-8 average is 17.8%.The mandatory gate zeroes tasks when models miss the framing, producing zeros on most tasks.
  • Conditional performance: 71.7%–84.1% is the conditional-score range across 12 models, with a 3.8-point standard deviation.Once models clear the gate, quality is high and relatively model-invariant.
  • Rank structure: Ranks 4–9 cluster within a 2.2-point band, whereas ranks 1–3 span 7.5 points.Unprompted problem recognition has a visible head and a flat middle.

7 Analysis

Model pass sets are highly complementary rather than nested: different models recognize different task structures, and routing across them expands benchmark coverage. Many tasks remain unsolved or are passed by only one model.

  • 7.1 No Model Is a Superset: Opus 4.6 contributes 35 unique passes, while GPT-5.4 contributes 21 and both pass 26 tasks.Opus skews toward adversarial structure; GPT-5.4 contributes more organizational navigation and strategic judgment.
  • 7.1 No Model Is a Superset: 31.7% is the top-two models’ Jaccard overlap, far below the 0.77 overlap expected from a simple pass-count ratio.Opus 4.6 and GPT-5.4 share 26 tasks while each uniquely passes many others.
  • 7.1 No Model Is a Superset: 29.3% is the mean pairwise Jaccard overlap among the top 8 models, with no pair exceeding 45%.Every model has a distinct recognition profile.
  • 7.1 No Model Is a Superset: Opus peaks at 50% in Principal-Agent tasks and GPT-5.4 at 55% in Strategy, with each weak where the other peaks.The capability fingerprints indicate complementary domain profiles.
  • 7.2 Coverage: 44 tasks are passed by exactly one of the top 8 models, and every top-8 model contributes at least two such tasks.The unique passes are distributed across all eight rather than concentrated in one model.
  • 7.2 Coverage: 54% of solvable tasks are covered by Opus 4.6 alone, 73% after GPT-5.4, and 84% after three models.The top 8 cover 113 tasks, while the remaining eight models add three more and 107 remain unsolved overall.
  • 7.4 Task Difficulty Distribution: 110 tasks are unsolved by the top 8, while exactly one model passes 44 of the 113 solved tasks.Exactly-one passes are the modal outcome and represent 39% of solvable tasks.
  • 7.5 Execution Is Decoupled from Recognition: Models scoring zero still pass roughly 40–60% of non-mandatory criteria on gated-out tasks.They often extract data and produce professional outputs while missing the framing decision.

8 What the Gate Measures

The mandatory gate is designed to isolate problem recognition from execution quality. Convergent passing scores and verifiable trap criteria support that interpretation rather than a generic strictness or style penalty.

  • 8. What the Gate Measures: The mandatory gate prevents partial execution credit from concealing failures to identify the governing problem.Without the gate, polished outputs on the wrong problem would accumulate aggregate score.
  • 8.1 The Gate Is Not a Noisy Filter: 76.6%–84.1% is the conditional-score range across the top 8 models, with a 3.8-point standard deviation.On the 26 tasks both leaders pass, their mean scores are 0.816 and 0.819.
  • 8.2 Criteria Test Verifiable Traps, Not Quality: 74% of tasks contain a mandatory criterion testing avoidance of a predicted catastrophic wrong answer.The criteria verify structural or textual traps rather than asking judges to assess general quality.
  • 8.3 The Gate Is Not a Generic Strictness Penalty: 25–38% are the pass rates for game-theoretic recognition tasks, compared with 52–75% for domain execution tasks.The gate therefore distinguishes adversarial reasoning from standard execution rather than uniformly suppressing all task types.

9 Failure Modes

Across domains, models default to cooperative, single-agent task completion instead of recognizing strategic incentives, hidden information, counterparties, or conflicts in the stated frame. They often know the concepts but fail to apply them unprompted.

  • 9.1 The Cooperative Default: The cooperative default is the most consistent of six recurring failure patterns.Models solve imperfect-information games as if they are the only player.
  • 9.7 Recognition, Not Reasoning: The paper characterizes the gap as recognition failure rather than reasoning limitation because models solve game-theoretic concepts when prompted directly.The proposed explanations include posttraining incentives, cooperative training data, instruction pressure, and computational difficulty.
  • 9.1 The Cooperative Default: 19 tasks are scored zero by every evaluated model because models adopt a cooperative or helpful framing in adversarial situations.The failures reflect misapplied stance rather than absent knowledge.
  • 9.2 Missing Counterparty Modeling: Models miss strategic omissions, such as treating a flawless diligence room as verification material rather than a selective signal.The benchmark’s examples show models attending to presented facts while failing to ask what is omitted.
  • 9.3 Blindness to Mechanism Design: Models treat incentive problems as process problems, proposing new interventions after prior process improvements failed.The intended diagnosis is mechanism-design failure caused by structural payoff misalignment.
  • 9.4 Ignoring Strategic Interdependence: Models ignore strategic interdependence when another agent’s outcome changes the relevant tie, tiebreaker, or subgame.They eliminate a dominated strategy but fail to solve the resulting deeper interaction.
  • 9.5 Accepting Stated Constraints: Models accept stated constraints and conflicts of interest instead of challenging the requester’s frame.Examples include mandated cost cuts and reorganizations that consolidate power under the proposer.
  • 9.6 Verbosity as a Substitute for Insight: Models can produce 3,000-word analyses that satisfy formatting and extraction criteria while missing the adversarial frame.Verbosity becomes a substitute for insight when length serves as a proxy for quality.

10 Discussion

KWBench’s discussion argues that recognition is distributed across models and that current evaluation lacks a human baseline, recognition ablation, and multi-judge validation.

  • Triangulation and Disjoint Recognition: Recognition profiles are fundamentally disjoint, with model-specific blind spots appearing during rubric construction and later measurement.The ensemble was needed because individual rubric generators missed different structural traps.
  • Architectural Implications for Agents: No single model spans the domain space: routing across the top eight expands coverage from 54% for the best model to 100% of the solvable set.The authors present this as an implication for autonomous workflow agents requiring cross-domain recognition.
  • The Alignment Tax on Adversarial Reasoning: Current models are described as structurally blind to adversarial framing, motivating training signals for adversarial counterparty modeling rather than only larger models or instruction tuning.The paper labels this mismatch between cooperative defaults and adversarial reasoning an alignment tax.
  • Methodological Limitations: The benchmark has no human baseline, so it measures what models miss without formally establishing human parity or calibrating the mandatory gate against experts.The authors identify expert baselines as necessary for bounding interpretation.
  • Methodological Limitations: A direct recognition ablation with explicit game-theoretic hints is absent, leaving recognition and subgame capability formally unisolated.The proposed ablation would compare identical tasks with and without explicit framing cues.
  • Methodological Limitations: The evaluation relies on a single judge, and contaminated scenarios cannot be ruled out because training corpora are opaque.The authors also note that the dataset reflects Western professional norms and corporate structures.

11 Conclusion

KWBench separates execution from problem recognition by using a mandatory gate, revealing polished analysis that addresses the wrong question and exposing the need for heterogeneous model coverage.

  • 11 Conclusion: KWBench separates execution from recognition: models can pass data extraction, formatting, and professional-tone criteria while scoring zero on the underlying task.The benchmark treats recognition of the adversarial dynamic as distinct from auxiliary execution quality.
  • 11 Conclusion: No single frontier model dominates; the top two agree on fewer than a third of passed tasks, while every top-eight model solves tasks all others miss.The conclusion states that diverse knowledge work therefore requires an ensemble.
  • 11 Conclusion: A mandatory gate prevents compensatory scoring from allowing execution quality to masquerade as domain mastery.Missing the critical structural trap imposes a strict binary penalty.
  • 11 Conclusion: The benchmark is released with documented rubric-construction principles and a public dataset.The paper also illustrates how weak criteria are transformed into tests of mechanisms, data, and concrete language.
  • 11 Conclusion: The rubric requires mechanism explanations, specific data grounding, proposed actions, tradeoff awareness, and nonredundant single-test criteria.Its principles also promote core insights to mandatory status and require constructive criteria alongside diagnosis.
  • 11 Conclusion: Rubrics are calibrated across response-quality levels so that surface-level responses fail mandatory criteria while competent responses pass them.The calibration compares B+, C, and D response profiles and adjusts criteria when thresholds are misaligned.

B Detailed Task Examples

The detailed examples show how KWBench embeds strategic traps in realistic professional scenarios, gives models only task materials, and evaluates whether they recognize the governing structure.

  • B Detailed Task Examples: KWBench examples span compensation, compliance, sports scheduling, crisis response, process incentives, enterprise sales, boards, layoffs, sunk costs, and references.The appendix presents scenarios whose surface tasks conceal strategic or structural reasoning requirements.
  • B Detailed Task Examples: In compensation negotiation, an above-market competing-offer claim is treated as cheap talk requiring credibility probes rather than an immediate concession.The correct approach preserves the employer’s maximum and tests the candidate’s actual priorities.
  • B Detailed Task Examples: The task interface supplies a prompt and reference file without problem-type hints or extra system guidance beyond being thorough and specific.The appendix’s full PIP example makes this unprompted setting explicit.
  • B Detailed Task Examples: The PIP example contains behavioral incidents, single-owner critical systems, prior litigation, and statements indicating knowledge hoarding and perceived persecution.These signals make the document’s legal and organizational constraints central to the task.
  • B Detailed Task Examples: The PIP trap is treating the document as an improvement plan rather than a legal document whose goals must withstand adversarial scrutiny.Naive technical goals can invite disputes over ambiguity, obstruction, bias, and documentation.

C.4 Results: 16 Models, Zero Passes

The 16-model PIP evaluation shows a uniform recognition failure: models produced professional plans but missed the litigation and knowledge-hoarding constraints encoded in the case.

  • C.4 Results: 16 Models, Zero Passes: Every model scored zero on the PIP task because all 16 were gated out by mandatory criteria.They produced standard technical goals while treating the PIP as an improvement tool rather than a legal document.
  • C.4 Results: 16 Models, Zero Passes: Models proposed goals dependent on teammate cooperation and used subjective criteria vulnerable to disputes over hostility or managerial bias.These failures reflect the case’s adversarial interpersonal context.
  • C.4 Results: 16 Models, Zero Passes: Models treated documentation as a nice-to-have instead of using it to break sole-owner knowledge hoarding.The missing mechanism concerned the bus factor created by exclusive system ownership.
  • C.4 Results: 16 Models, Zero Passes: Models mentioned the prior lawsuit as background rather than recognizing litigation risk as the constraint shaping every goal.The reference file supplied the lawsuit, the employee’s statement, and sole system ownership as connected signals.
  • C.4 Results: 16 Models, Zero Passes: Without the mandatory gate, these responses would have scored 40–60% on good-to-have and ideal criteria despite missing the decisive constraint.The models extracted data, structured professional PIPs, and addressed behavioral incidents, but not lawsuit survivability.

C.5 Commentary

The commentary presents the PIP scenario as a test of recognizing legal, operational, and incentive constraints rather than applying a standard improvement template. It emphasizes documentation, explicit wrong-path criteria, and the need to balance knowledge transfer with eventual separation.

  • Problem recognition: The critical failure is treating a protected whistleblower’s prior lawsuit as background rather than the constraint governing PIP design.The relevant signals are explicit, including the prior lawsuit and sole ownership of a system with Bus Factor: 1.
  • Problem recognition: Standard PIP templates produce the wrong answer because their cooperative improvement framing conflicts with this scenario’s legal and strategic context.The paper notes that a non-specialist may miss the danger even though an HR consultant would recognize it immediately.
  • Mechanism design: Requiring system documentation simultaneously creates a binary PIP goal, reduces Alex’s knowledge leverage, and builds an HR paper trail.The intervention aligns legal documentation requirements with operational continuity needs.
  • Operational tradeoff: The response must preserve both constraints: firing now may delay a critical project, while accommodating Alex may neglect the legal constraint.The PIP is described as buying time for knowledge transfer while building the legal record for eventual separation.
  • Evaluation design: The annotations record practitioners’ tacit pattern recognition by mapping mandatory criteria to concrete failure modes and independently testable consequences.The rubric evaluates whether responses avoid predictable wrong paths rather than requiring one specific PIP wording.
Loading 2604.15760v1…