Source-linked AI summary
CL-bench: A Benchmark for Context Learning
Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, Tao Gui, Zuxuan Wu, Xipeng Qiu, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Di Wang, Shunyu Yao
TL;DR
Real-world tasks require models to learn and apply new knowledge from task-specific contexts, a capability that existing evaluations have largely overlooked. The paper introduces CL-bench to measure this ability across expert-crafted, context-dependent tasks, and finds that frontier models solve only a small fraction of them. The benchmark also identifies difficult categories and highlights scope boundaries that affect interpretation of its results.
Problem
Real-world tasks often require models to learn and correctly apply new knowledge from complex contexts beyond pre-training.
Method
CL-bench evaluates context learning with 500 contexts, 1,899 tasks, and 31,607 verification rubrics containing new knowledge across four categories.
Results
17.2% of tasks are solved on average across ten evaluated LMs, while GPT-5.1 achieves 23.7%.
Takeaways & Limitations
CL-bench exposes context learning as a major gap between current LM capabilities and practical context-dependent task demands.
Takeaways & Limitations
The benchmark does not exhaustively cover all real-world domains and knowledge types, especially highly specialized or emerging fields.
Abstract
from arXiv · showhide
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
1 INTRODUCTION
Context learning is the ability to acquire and apply new knowledge from complex contexts, a capability needed for realistic, context-dependent tasks but largely overlooked in current research. CL-bench evaluates this ability and finds that frontier models remain weak at it.
- Motivation: Context learning requires models to acquire new knowledge from complex contexts rather than rely only on pre-trained knowledge.It supports tasks involving unseen documentation, ongoing conversations, and laws inferred from experimental data.
- Benchmark introduction: CL-bench contains 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics crafted for systematic evaluation.Tasks require applying context-provided knowledge, including fictional legal systems and complex product manuals, without external retrieval.
- Benchmark design: The benchmark emphasizes realistic, contamination-free, and challenging instances built from new or niche knowledge and reviewed by domain experts.Some contexts may conflict with pre-training, requiring models to follow the provided material instead.
- Results: 17.2% of tasks are solved on average across ten state-of-the-art LMs, while GPT-5.1 solves 23.7%.Models perform especially poorly on categories requiring laws induced from experimental data or complex sandbox simulation, averaging 11.8%.
- Conclusion: CL-bench indicates that effective context learning remains a major unresolved capability for frontier LMs and a critical testbed for future progress.The authors connect progress in this capability with models that learn from and adapt to evolving contexts.
2 RELATED WORK
Prior LM research emphasizes prompt reasoning, context preparation, and separate capabilities such as long-context reasoning or instruction following. CL-bench instead targets whether models can learn and apply new knowledge from complex contexts.
- Prompt and context paradigms: Prompt engineering and in-context learning mainly use instructions, demonstrations, and pre-trained knowledge to solve relatively simple tasks.They focus primarily on task formats or shallow heuristics rather than acquiring complex new knowledge.
- Prompt and context paradigms: Context engineering organizes and optimizes task-relevant information but has largely overlooked whether models can learn from the provided context.The paper positions context learning as the foundation for effectively leveraging that context.
- Benchmark gap: Existing benchmarks assess reasoning, general task solving, and agentic abilities but primarily rely on static knowledge.They therefore largely overlook learning and applying new knowledge supplied in context.
- Related capabilities: Long-context reasoning and instruction following are related to context learning because complex contexts contain long inputs and intricate constraints.The paper treats these capabilities as necessary but not sufficient for context learning.
- CL-bench distinction: CL-bench is presented as a benchmark for the largely overlooked capability of learning and applying new knowledge from context.Its intended scope is distinct from evaluating retrieval, context preparation, or simple task-pattern induction alone.
3 CL-BENCH: A BENCHMARK FOR CONTEXT LEARNING
CL-bench evaluates context learning through realistic, context-dependent tasks spanning diverse knowledge, rules, procedures, and empirical systems. Its construction uses expert-designed contexts, sequential tasks, taxonomy-based coverage, contamination controls, and rubric-based verification.
- 3.1 OVERVIEW: CL-bench provides new or niche knowledge in context so models can solve tasks without external retrieval.The benchmark includes materials such as documents, reports, experimental data, code repositories, and operational manuals.
- 3.1 OVERVIEW: 51.1% of tasks are sequential and depend on solutions from earlier interaction turns.This multi-turn design is intended to increase difficulty and reflect real-world usage scenarios.
- 3.2 CONTEXT TAXONOMY: CL-bench organizes contexts into four categories and 18 subcategories based on how people encounter, learn from, and use real-world contexts.The categories cover domain knowledge reasoning, rule system application, procedural task execution, and empirical discovery & simulation.
- 3.2 CONTEXT TAXONOMY: Empirical discovery & simulation is the most challenging category because it requires inductive discovery of patterns or laws from evidence.The other three categories emphasize more deductive application of learned knowledge, rules, or procedures.
- 3.3 BENCHMARK CONSTRUCTION: Experts construct contexts containing unavailable, fictional, modified, or niche knowledge, then design tasks requiring genuine learning from those contexts.The benchmark applies quality control and contamination-free construction to limit reliance on pre-training.
- 3.3 BENCHMARK CONSTRUCTION: Less than 1% of tasks are solved in the context-free ablation, supporting the benchmark’s context dependence.The reported ablation is used to confirm that models cannot solve the tasks reliably without the provided context.
- 3.3 BENCHMARK CONSTRUCTION: Evaluation uses expert-authored binary task rubrics, while assessment of 100 sampled solutions reports accuracy exceeding 90%.Rubrics cover factual, computational, judgment, procedural, completeness, and format dimensions.
4 MAIN RESULTS
Across ten frontier LMs, context-learning performance remains low and varies sharply by category and subcategory. Models struggle most when tasks require inductive discovery, extended-context coherence, or strict adherence to contextual constraints.
- 17.2% average task-solving rate leaves frontier models below 30% overall, with GPT-5.1 best at 23.7%.
- Domain knowledge reasoning is most tractable at 25.3%, while empirical discovery and simulation fall to approximately 11%.
- Legal and regulatory rule application exceeds 29% for all models, whereas mathematical formalism leaves most models below 15%.
- Specific knowledge domains and structural characteristics strongly influence how effectively models acquire and apply contextual knowledge.
- Inductive categories score approximately 6% below deductive categories, with observational data and simulation especially difficult.
- GPT-5.2 underperforms GPT-5.1 by 5.6% overall, alongside failures to maintain causal chains and obey contextual constraints.
5 FURTHER ANALYSIS
CL-bench analysis shows that models struggle with context learning because they often misuse, neglect, or lose track of information, especially in longer and more complex contexts. Increased reasoning effort offers limited and model-dependent gains.
- Reasoning effort: GPT-5.1 improves from 21.2% to 23.7% with higher reasoning effort, a 2.5% gain.The reported improvement is modest even for the best-performing model.
- Error patterns: Context misuse and context neglect are dominant failure modes, with context-misused rates exceeding 60% for all models.Higher-solving models tend to neglect context less, but even capable models struggle to interpret and apply it correctly.
- Error patterns: Format errors remain substantial, exceeding 35% for GPT-5.1 and 40% for Claude-Opus-4.5.These failures reflect violations of explicit formatting instructions supplied in the context.
- Reasoning effort: Higher reasoning effort generally improves context learning, but gains vary by model and category.GPT-5.1 gains 5.9% on both management and experimental data, whereas GPT-5.2 shows negligible or negative gains in several subcategories.
- Context length: All models show lower solving rates as context length increases, regardless of reasoning effort.The context constitutes most of the total input, and degradation appears across GPT-5.1, Claude-Opus-4.5, Kimi-K2, HY-2.0, and Gemini-3-Pro.
- Knowledge type: Models perform substantially differently on legal tasks despite sharing the same domain, with Qwen 3 Max differing by more than 25%.The disparity is attributed to differences in the knowledge types learned from context and how they are applied.
- Qualitative analysis: A drone-delivery case shows that Gemini-3-Pro can reject an undocumented function yet still omit required safety operations and task parameters.The model passed only two of four rubrics, omitting Safety_request_airspace() and parameters such as D-998 and Sector 4.
- Qualitative analysis: Across 16 examples, models frequently overlook task requirements and execution conditions, with longer contexts worsening this loss of relevant information.These patterns link context learning to both instruction following and long-context reasoning.
6 DISCUSSION
The discussion positions context learning as the ability to acquire genuinely new knowledge from provided context and apply it to novel tasks. It outlines applications, training and architectural directions, and limitations of the current benchmark.
- Significance: Context learning bridges static parametric knowledge and the dynamic demands of real-world applications by acquiring new knowledge from provided contexts.Unlike in-context learning, it uses new contextual knowledge while relying on the model’s existing reasoning capabilities.
- Significance: Context learning could provide immediate domain specialization without parameter modification, unlike computationally expensive fine-tuning or continual learning.The comparison also notes that those traditional approaches risk catastrophic forgetting.
- Future directions: Proposed directions include context-aware training data, progressive curriculum learning, synthetic rubric generation, and architectural changes for context utilization.These directions target faithful context use, long-context processing, instruction following, scalable feedback, and memory or iterative refinement.
- Limitations: CL-bench does not exhaustively cover all domains and knowledge types encountered in real-world applications.Specialized or emerging fields may have characteristics absent from its 18 subcategories.
- Limitations: The evaluation focuses on single-turn tasks and short task sequences rather than extended dialogues with iterative refinement and feedback.Longer interactions may require models to consolidate, revise, and transfer contextual knowledge over time.
- Limitations: CL-bench evaluates textual contexts but not multimodal knowledge combining images, audio, video, and text.Extending evaluation to multimodal settings is identified as a future challenge and opportunity.
- Limitations: The paper does not establish human baselines because suitable unbiased participants and comparison procedures remain difficult to design.Controlled learning experiments with domain novices are suggested as one possible future approach.
7 CONCLUSION
The paper defines context learning as acquiring and correctly applying knowledge beyond pre-training, then introduces CL-bench to evaluate it. Results show a substantial gap between current model capabilities and practical context-dependent tasks.
- Conclusion: CL-bench contains 500 contexts, 1,899 tasks, and 31,607 verification rubrics across four context-learning categories.Instances are designed to be realistic, contamination-free, and challenging, requiring models to learn and apply new knowledge.
- Conclusion: GPT-5.1 solves only 23.7% of CL-bench tasks, exposing a significant gap between current capabilities and practical application demands.The paper presents CL-bench as a testbed for developing models that effectively utilize context.
- Conclusion: The appendix provides additional subcategory experiments and case studies examining specific causes of context-learning failures.These materials supplement the main evaluation with detailed model performance and qualitative analysis.
A RESOLVING TASKS IN CL-BENCH REQUIRES LEARNING FROM CONTEXT
The context-removal experiment tests whether CL-bench tasks can be solved from pre-trained knowledge alone. GPT-5.1’s performance collapses without the supplied contexts, supporting the benchmark’s intended measurement of context learning.
- Context dependence: CL-bench contexts contain novel or niche knowledge, making models that rely only on pre-trained knowledge almost incapable of solving its tasks.The benchmark is constructed so that task-relevant knowledge must be learned from the provided context.
- Context dependence: 0.9% is GPT-5.1’s task-solving rate on 1,000 sampled tasks after their corresponding contexts are removed.The sharp drop indicates that nearly all sampled tasks require learning from context.
B PERFORMANCE OF MODELS ACROSS SUBCATEGORIES
Across subcategories, frontier models perform unevenly on CL-bench: they achieve their strongest results on domain knowledge reasoning and procedural execution, but struggle with empirical discovery and simulation and vary sharply by knowledge domain.
- 17.2% average solving rate across 19 models, with GPT-5.1 achieving 23.7% on CL-bench.
- 11.8% average solving rate on empirical discovery & simulation, approximately 6% below other categories.This category requires inductive reasoning from extensive experimental data or complex simulations.
- Rule system application varies by subcategory: legal & regulatory exceeds 29% for all models, while mathematical formalism falls below 12% for most.The results show that both the knowledge domain and the type of contextual knowledge influence model performance.
- Figures 10–13 report model performance across subcategories with reasoning enabled and with reasoning disabled or reduced.
C IMPACT OF REASONING ON CONTEXT LEARNING
Higher reasoning effort generally improves context learning, although the effect is not universal across models.
- The comparison covers nine frontier LMs using highest versus lowest adjustable settings or reasoning versus non-reasoning variants.
- Higher reasoning effort facilitates more effective context learning for the majority of models.A few models show no improvement when reasoning effort increases.
- 5.7% average performance gap separates Kimi-K2’s two reasoning settings, the largest improvement reported.
- Figures 14 and 15 compare model performance under different reasoning effort settings.
D IMPACT OF CONTEXT AND INPUT LENGTH ON CONTEXT LEARNING
Context length strongly affects context-learning performance: solving rates decline as contexts grow, while higher reasoning effort becomes increasingly advantageous for longer contexts.
- The analysis evaluates how context and input length affect model performance on CL-bench using Figures 16 and 17.
- Context-length and input-length analyses show nearly identical trends because context constitutes the dominant proportion of total input.
- Solving rates generally fall from approximately 25–35% at 0–4K tokens to 5–10% at 32K+ tokens.The decline occurs regardless of reasoning effort.
- 16.2% solving rate is maintained by GPT-5.1 at 32K+ tokens, substantially higher than other LMs.
- Higher reasoning effort provides minimal advantage at 0–4K tokens but benefits more models significantly at longer context lengths.
F IN-DEPTH ANALYSIS OF CONTEXT LEARNING SUCCESSES AND FAILURES
Qualitative case studies examine frontier-model successes and failures, showing that models often neglect or misapply contextual information when solving complex tasks.
- 16 qualitative cases, four per category, illustrate diverse frontier-model performance.
- Gemini-3-Pro successfully extracts combinatorial reasoning from a video transcript and explains the formula 2×21×13+1=547.
- The case studies primarily focus on GPT-series failure cases to explore the capability boundaries of current frontier models.
- The qualitative analyses corroborate that frontier LMs frequently neglect or misapply contextual information, producing erroneous solutions.Long-context reasoning and instruction-following limitations further exacerbate failures.
F.1 EMPIRICAL DISCOVERY & SIMULATION
The examples test whether models can learn domain-specific procedures and quantitative relationships from context, then apply them in simulations and structured outputs. Results show both correct task execution and failures involving transparency, evidence use, and required distinctions.
- Empirical discovery: The electron-angle task requires finite-difference velocity extraction, perpendicular and parallel component calculation, arctangent-based angle computation, and a final answer of 27.0°.The rubric specifies treating the z-axis as the parallel direction and deriving the entry angle from velocity components.
- Empirical discovery: GPT-5.2 computes the correct electron entry angle from helical-motion data but omits the rationale for magnetic-field orientation and the position-variable units.The example illustrates numerically correct output without full logical transparency or adherence to scientific conventions.
- Simulation: Kimi-K2 successfully initializes the gallium simulation with the 2023 start year, four required stocks and flows, numerical anchors, consistent units, and explicit assumptions.The example uses resource-in-place, extraction/production, demand, and recycling as the required categories.
- Rule application: GPT-5.1 identifies “the Corporation” as TSSA using external knowledge despite the regulation excerpt lacking the definition, violating the task’s evidence-based requirement.The response also fails to provide required direct quotations and to distinguish the Corporation’s duties from those of the director.
- Rule application: A motion-summary task captures several constants and the mass/weight variant but omits required direction and object-type variants and adds unsupported force-magnitude variation.The output also had to satisfy strict formatting requirements for headings, numbering, bullets, bold headers, and plain text.
F.2 PROCEDURAL TASK EXECUTION
The procedural-task cases show models often satisfy visible constraints while missing the deeper task reformulation or constraint composition required for full compliance.
- Recipe generation: GPT-5.2 refuses the dairy-based request but fails to reformulate it into a valid dairy-free oxtail recipe.The case required dairy-free, peanut-free ingredients, cooking steps, and structural requirements.
- Recipe generation: The recipe case illustrates that refusing a prohibited ingredient does not by itself satisfy a constrained task requiring a usable reformulation.The model offers alternative directions but does not provide the requested compliant recipe.
- Course scheduling: 17 credits, not 18, is the correct maximum semester load after combining the annual and per-semester credit limits.The student had already completed 15 credits, while the annual maximum was 32 and the semester range was 13–18.
- Game procedure: GPT-5.1 identifies a plausible low-turn Wingspan strategy but violates several reporting and procedural rubrics.The proposed strategy generates three eggs and spends one egg plus one worm per bird, but the evaluation found additional rubric violations.
F.3 RULE SYSTEM APPLICATION
The EZLang case shows that runnable code can still violate the task’s precise timing and stopping rules when the model substitutes convenient heuristics.
- Constraint fidelity: Gemini-3-Pro generates runnable EZLang logging code but replaces the specified interval and stopping conditions with developer-friendly heuristics.The case concerns checking at true 30-minute intervals and stopping at 5:30 P.M.
Context (continued)
The cases examine models applying learned rules and procedures across programming, game, scientific, and domain-specific contexts, with failures often involving omitted details or nonstandard execution.
- Programming procedures: Gemini-3-Pro produces syntactically valid EZLang code and logs timestamps but does not implement the required true interval and stopping behavior.The task required 30-minute checks, termination at 5:30 P.M., and printing recorded times.
- Game-rule application: The game-rule summary follows formatting constraints but omits required combat, elemental, condition, area-of-effect, and monster-AI mechanics.The model mentions some systems without supplying the underlying rules required by the rubric.
- Paper-to-code reproduction: The LightGTS reproduction provides runnable code but leaves gaps in standardized benchmarking practices and omits requested next steps.The response uses ad-hoc data preparation and manual train/test splits rather than standard benchmark datasets such as ETTh1.
- Model context: The source context describes LightGTS against larger time-series foundation models whose generalization depends on massive pre-training data and large parameters.The comparison reports Chronos at 700M parameters and Time-MoE at 453M in Figure 1.
Model Generated Solution (continued)
The examples include extracted implementation components and a successful combinatorial explanation, while also showing that runnable or numerically correct outputs may not meet all task requirements.
- LightGTS implementation: The LightGTS implementation identifies periods with FFT and dynamically resizes projection weights to match the input period.Its Flex Projection layer uses linear interpolation to adapt weights from a reference patch size.
- LightGTS evaluation: Gemini-3-Pro’s runnable LightGTS prototype omits requested next steps and does not follow standardized benchmark preparation.The evaluation notes ad-hoc dataset preparation and manual train/test splits.
- Combinatorial reasoning: The combinatorial explanation correctly reconstructs the full calculation and motivates the leading factor of 2.The task requires labeling each factor in the formula and explaining how the representable numbers are counted.
- Combinatorial reasoning: 547 is computed as 2×21×13+1 by combining sign, main-body, suffix, and zero options.The leading 2 represents positive or negative values; the final 1 accounts for zero.
F.4 DOMAIN KNOWLEDGE REASONING
The section presents domain-knowledge reasoning tasks that require models to follow strict formats, apply specialized source material, and satisfy detailed rubrics. Results show failures in formatting, completeness, safety sequencing, and evidence-based reasoning across varied domains.
- Document organization: GPT-5.2 omitted required documents and failed to use the mandatory table format when organizing Genfanad development materials.The task required exhaustive inclusion and categorization of the provided documents.
- Safety-constrained healthcare: Gemini-3-Pro gave healthcare advice before confirming the user’s age and omitted SpartanAI’s stated purpose for women aged 18–70.These omissions violated the rubric’s required safety sequence and persona description.
- Environmental reasoning: The monitoring chapter did not support definitive ten-year extinction predictions because it lacked species-specific population viability data and extinction forecasts.The task instead required recognizing documented monitoring gaps, including absent systematic surveys for several taxa.
- Literary comprehension: Kimi-K2 correctly supplied historical context for Vortigern and Rowena but failed requirements for exact character coverage, dialogue-practice suitability, and conversational formatting.The task combined factual extraction with strict stylistic and content constraints.
- Legal reasoning: Legal & Regulatory tasks emphasize quote-accurate extraction, whereas Legal Advisory tasks require applying a fictional criminal code under structured reasoning and formatting constraints.The contrast separates evidence-text-driven reasoning from rule-system application.