Source-linked AI summary
Classroom Final Exam: An Instructor-Tested Reasoning Benchmark
Chongyang Gao, Diji Yang, Shuyan Zhou, Xichen Yan, Luchuan Song, Shuo Li, Kezhen Chen
TL;DR
CFE-Bench addresses the need for realistic, discriminative evaluation of advanced STEM reasoning. It builds a multimodal benchmark from instructor-tested university problems, applies structured variable-based verification, and diagnoses reasoning flows. Gemini-3.1-pro-preview reaches only 59.69% overall question accuracy, while diagnostics show that models often execute individual steps correctly but fail to maintain correct intermediate states over longer derivations.
Problem
Widely used benchmarks are increasingly saturated, while advanced scientific and technical problems still challenge models requiring deep domain knowledge and multi-step reasoning.
Method
The paper constructs CFE-Bench from authentic instructor-verified STEM materials, evaluates answers with variable-based verification, and decomposes reference solutions into structured reasoning flows.
Results
59.69% overall Question Accuracy is achieved by Gemini-3.1-pro-preview, while models often execute individual steps correctly but fail to derive and preserve correct intermediate states across long derivations.
Takeaways & Limitations
CFE-Bench provides a realistic, diagnostic testbed for measuring domain-grounded reasoning and analyzing failures beyond aggregate accuracy.
Abstract
from arXiv · showhide
We introduce CFE-Bench (Classroom Final Exam), a multimodal benchmark for evaluating the reasoning capabilities of large language models across more than 20 STEM domains. CFE-Bench is curated from repeatedly used, authentic university homework and exam problems, paired with reference solutions provided by course instructors. CFE-Bench remains challenging for frontier models: the newly released Gemini-3.1-pro-preview achieves 59.69% overall accuracy, while the second-best model, Gemini-3-flash-preview, reaches 55.46%, leaving substantial room for improvement. Beyond aggregate scores, we conduct a diagnostic analysis by decomposing instructor reference solutions into structured reasoning flows. We find that while frontier models often answer intermediate sub-questions correctly, they struggle to reliably derive and maintain correct intermediate states throughout multi-step solutions. We further observe that model-generated solutions typically contain more reasoning steps than instructor solutions, indicating lower step efficiency and a higher risk of error accumulation. Data and code are available at https://github.com/Analogy-AI/CFE_Bench.
1 INTRODUCTION
CFE-BENCH addresses saturated benchmarks and persistent difficulty in advanced, multi-step STEM reasoning with a realistic, instructor-verified testbed. It combines stricter variable-based evaluation with diagnostics of atomic and compositional reasoning failures.
- Benchmark: CFE-BENCH contains 449 instructor-verified problems across text-only and multimodal settings and more than 20 STEM subjects.The benchmark includes 305 text-only and 144 multimodal questions, with strong representation from physics, mathematics, and engineering.
- Benchmark: The benchmark is curated from repeatedly used university materials and filtered for well-posed, objectively verifiable, nontrivial questions.Items avoid trivial yes/no or multiple-choice formats and do not require physical experiments.
- Evaluation: 59.69% overall accuracy is achieved by Gemini-3.1-pro-preview under the stricter variable-based evaluation, leaving substantial room for improvement.The protocol targets answer variables rather than comparing entire long-form responses directly.
- Overall contribution: CFE-BENCH is designed as a reliable, unsaturated testbed for measuring domain-grounded reasoning under realistic academic standards.Its contributions combine classroom-based benchmark construction, structured evaluation, and reasoning-flow diagnostics.
- Evaluation and diagnosis: The paper introduces variable-based verification to improve evaluation accuracy and unit-based analyses to separate atomic execution from compositional failures.The diagnostic framework also identifies intermediate results associated with end-to-end success.
2 RELATED WORK
Prior reasoning benchmarks either use synthetic or newly annotated settings or rely on authentic educational materials with limited long-form reasoning verification. CFE-BENCH is positioned against saturation and outcome-based evaluation limits in existing resources.
- Existing reasoning benchmarks: Existing benchmarks include competition-style mathematics and targeted modality tasks, but they do not collectively establish classroom-grounded, multi-domain reasoning evaluation.Examples include MATH, AIME, OmniDocBench, and CharXiv.
- Benchmark construction: Synthetic or newly annotated benchmarks prioritize freshness, difficulty, or short-answer factuality, often providing limited verification of long-form compositional reasoning.The passage contrasts SimpleQA, FACTS, and HLE with the need to assess extended reasoning processes.
- Authentic educational benchmarks: Authentic educational benchmarks improve pedagogical relevance but predominantly use outcome-based metrics and are approaching performance saturation.Detailed explanations are available for only a small fraction of questions, and available rationales may be poor.
3 CFE BENCHMARK
CFE-BENCH is assembled from authentic, instructor-tested STEM materials, reviewed through a structured expert workflow, and evaluated with variable-level verification. The resulting protocol targets false positives and distinguishes partial from fully correct solutions.
- Collection: CFE-BENCH draws on instructor-authored exams, quizzes, and homework selected for nontrivial, objectively verifiable multi-step reasoning.Problems are reviewed by domain experts for correctness, clarity, and realistic difficulty.
- Collection: The dataset contains 305 text-only and 144 multimodal questions, with Physics and Mathematics dominating the text split and Physics dominating the multimodal split.The subject distribution spans additional engineering and other STEM domains.
- Expert annotation protocol: Expert annotation uses a standardized interface to filter trivial items, confirm well-defined answers, verify reasoning flows, and assess image dependency.The workflow involved 17 graduate-level expert annotators and retained each question with a human-verified reasoning flow.
- Variable-based evaluation: Long-to-long evaluation can produce false positives because partial correctness, extended context, and fluent rationales confound holistic judging.These issues motivate structured verification based on concrete target variables.
- Variable-based evaluation: Short-to-Short structured verification extracts typed target variables from model responses and compares every predicted value with its ground truth.A response is correct only when all annotated variables are verified as correct; annotations include names, descriptions, values, and types.
- Variable-based evaluation: The evaluation compares S2S with Long-to-Short and Long-to-Long settings, using variable accuracy for partial progress and question accuracy for all-variable correctness.The protocol is validated against expert labels across automatic evaluation settings.
4 MODEL PERFORMANCE ON CFE-BENCH
CFE-Bench evaluates diverse models with variable-based verification across text-only and multimodal STEM problems. Performance remains substantially below saturation, with multimodal tasks generally more difficult and the strongest combined result reaching 59.69% Question Accuracy.
- 0.65 question accuracy is the strongest text-only result, achieved by Gemini-3.1-Pro-Preview.
- The multimodal split is more challenging for all models, with Gemini-3 leading and Qwen 3.5 reaching 0.45 question accuracy.
- 59.69% Question Accuracy is the strongest combined text-and-multimodal result, achieved by Gemini-3.1-pro-preview.The best open-weight model, Qwen3.5-397B, reaches 47.44% Question Accuracy.
- Variable Accuracy typically exceeds Question Accuracy by approximately 5–7 points, indicating frequent partial progress without a fully correct solution.
5 DECONSTRUCTING THE FRONTIER MODEL PERFORMANCE GAP
The diagnostic framework decomposes instructor solutions into ordered, verifiable reasoning units and tests atomic execution, multi-step composition, and intermediate-state interventions. Models usually execute specified units well, but struggle to derive and preserve correct intermediate answers, especially through difficult middle steps, while generating longer-than-ground-truth reasoning flows.
- 5.1 FORMALIZING THE REASONING FLOW: Reasoning flows represent reference solutions as ordered question–answer units, enabling separate tests of atomic and compositional deficits.Human annotators review candidate decompositions so each sub-question is unambiguous and each target answer is objectively checkable.
- 5.2 Q1: ATOMIC REASONING CAPABILITY: ∼0.8–0.9 mean accuracy is typical for text-subset unit execution, with similar multimodal performance and modest dips.This suggests many end-to-end failures are not caused by inability to execute individual steps once the sub-question is specified.
- 5.3 Q2: REASONING PROGRESSION CAPABILITY: Providing unit answers consistently outperforms providing only unit sub-questions across both modalities.The comparison tests reasoning prefixes and single-unit injections under answer-present and questions-only conditions.
- 5.3 Q2: REASONING PROGRESSION CAPABILITY: The largest performance gap occurs at mid-range unit indices, where transformations combine prior results with identities, theorems, and multi-step algebraic or numerical operations.Correct intermediate answers bridge these difficult transitions, while early units are more direct and later units require final consolidation.
- 5.4 Q3: CRITICAL INTERMEDIATE UNITS: A single unit with its answer can be nearly as effective as a full reasoning prefix without answers, despite using substantially less context.Single-Unit Injection improves over Questions Only, whereas Single-Unit Injection (Question Only) remains low.
- 5.5 REASONING EFFICIENCY: 12.20 versus 10.73 steps in text and 13.86 versus 11.72 in multimodal responses show longer model reasoning flows than ground truth.These correspond to approximately 14% and 18% longer reasoning, respectively, indicating lower step efficiency and more opportunities for error accumulation.
- 5.6 TAKEAWAYS AND IMPLICATIONS FOR STRONGER FUTURE MODELS: The findings motivate supervision and inference strategies that verify intermediate states, check constraints, and condition models on validated intermediate values.
6 CONCLUSION
CFE-Bench combines realistic text-and-multimodal STEM evaluation with variable-based verification and structured reasoning-flow diagnostics. Results show strong unit execution alongside unreliable intermediate-state derivation and inefficiently long reasoning, motivating verifiable intermediate supervision.
- CFE-Bench is a text-and-multimodal benchmark with variable-based evaluation that reduces false positives in long-form answer matching.
- Strong models often execute individual reasoning steps correctly but fail to derive and preserve correct intermediate states over long derivations.
- Models generate less efficient, longer reasoning than instructor solutions, creating more opportunities for intermediate errors to accumulate.
- The benchmark is intended to support models, training objectives, and inference strategies emphasizing verifiable intermediate supervision and efficient reasoning.
A VARIABLE VALUE EXTRACTION AND VERIFICATION PROMPTS
The appendix specifies prompts for extracting target variable values from model responses and verifying them against reference answers. The protocol enforces ordered one-to-one extraction, preserves missing values as null, and checks mathematical, unit, and structural equivalence.
- The extraction prompt obtains values for a specified variable list from a response using types and reference descriptions.
- The output must preserve one-to-one variable mapping, exact cardinality, and variable order.
- Numeric values omit units, formulas use dollar-delimited LaTeX, other values retain exact text, and missing variables become null.
- Verification determines whether extracted values are mathematically or logically equivalent to reference values.
- The verifier accepts equivalent formatting, fractions, percentages, powers, square roots, and convertible units, while rejecting fundamentally different units.
- Verification also checks expressions, vectors or coordinates component-wise, unordered set contents, and reduced fractions, returning correctness, reasoning, and confidence.
B REASONING FLOW CONSTRUCTION PROMPT
The prompt instructs annotators to decompose reference solutions into ordered, traceable reasoning steps, each pairing an answer-agnostic instruction with a verifiable result. Steps must obey strict dependencies and use a JSON structure that supports intermediate verification.
- Purpose: The task decomposes each reference answer into an ordered chain of strictly logical, verifiable reasoning steps.Each step should act as a clear instruction or question that yields a concrete result.
- Step design: Each step follows a question/answer format: the step poses an instruction, while verifiable_answer records its concrete solution or result.Examples include formulas, numbers, Boolean values, and extracted labels.
- Step design: The Anti-Leaking Rule requires every step to be answer-agnostic and prevents embedding the solution in the instruction.For example, a step should extract a force value from the question rather than state that value directly.
- Traceability: The Strict Dependency Rule limits each step to the original question or images and fields established by previous steps.Unestablished information from the reference answer cannot be used prematurely.
- Output format: The output is a JSON object containing an ordered sub_steps list, with each sub-step specifying an identifier, instruction, and verifiable formula, number, or Boolean answer.The prompt also requires returning only the specified JSON object.
- Example: The worked example progresses from identifying glassware and stating Newton’s Second Law to rearranging the formula and calculating acceleration.It substitutes F=10N and m=2kg into a=F/m to obtain 5.0.
C REASONING FLOW EXAMPLE
The example represents a reference solution as an ordered sequence of atomic, verifiable reasoning units, with each unit linked to an objectively checkable target answer. Its later steps combine intermediate terms and state the resulting current.
- Reasoning flow: A reference solution is decomposed into an ordered sequence of atomic, verifiable reasoning units.The sequence is represented as R = [u1, u2, ..., un].
- Reasoning flow: Each reasoning unit is paired with an objectively checkable target answer.This pairing makes intermediate results directly evaluable.
- Step sequence: Step 10 algebraically combines the two terms established in Step 9.The example preserves dependency between the intermediate expression and the subsequent operation.
- Step sequence: Step 11 states the resulting total current i(t) through the short circuit.It follows the algebraic combination performed in the preceding step.