CL-bench: A Benchmark for Context Learning

Motivation

• Context learning requires models to learn previously unavailable knowledge from complex, context-dependent tasks rather than relying solely on pre-trained knowledge.This capability supports using unseen documentation, long-running conversations, and experimental data to solve practical tasks.Source: §1 · p. 2

Benchmark

• CL-bench contains 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics spanning four categories and 18 subcategories of real-world context learning.Tasks use new, niche, or emerging knowledge contained within each context, eliminating the need for external retrieval.Source: §1 · p. 2, Figure 2 · p. 2
• The benchmark is expert-crafted, contamination-free, challenging, and rigorously verifiable through reviewed contexts, sequential task dependencies, and multidimensional rubrics.Each context averages 3.8 tasks and 63.2 rubrics, while annotation requires about 20 hours of expert effort.Source: §1 · pp. 2–3

Results

• 17.2% of tasks are solved on average across ten frontier LMs, while GPT-5.1, the best-performing model, solves only 23.7%.Models perform substantially worse on challenging law-induction and complex-simulation categories, which have an average solve rate of 11.8%.Source: §1 · p. 3

Error analysis

• Failures most often involve ignoring information presented in context, with insufficient long-context reasoning and instruction following also contributing.Performance varies across categories, but every model struggles more on the most challenging categories.Source: §1 · p. 3

Significance

• CL-bench provides a testbed for advancing context learning toward models that can learn from context, adapt to evolving situations, and perform effectively in the real world.The findings indicate that this fundamental capability remains remarkably poor and deserves greater attention.Source: §1 · p. 3

Prompting and context learning

• Prompt engineering and in-context learning mainly elicit reasoning from instructions, examples, and pre-trained knowledge, whereas real-world tasks require learning new knowledge from complex contexts.Prompt engineering uses carefully designed instructions, while in-context learning adds input–output examples to infer task formats and expected behavior [58] [74] [41].Source: §2 · p. 3

Context engineering

• Context engineering retrieves, organizes, manages, and optimizes task-relevant information, but it has largely overlooked whether models can learn from that context.Representative approaches include Retrieval-Augmented Generation [34] [21] [22], memory systems [49] [26], and agentic RAG pipelines [64] [28] [87].Source: §2 · p. 3

Benchmarks for language models

• Existing benchmarks assess reasoning, general task solving, and agentic abilities, yet mostly test static knowledge and conflate context preparation with learning from context.Tool-using benchmarks rarely distinguish retrieval failures from failures to learn new knowledge, limiting actionable insights; CL-bench explicitly evaluates learning and applying new knowledge from complex contexts [18] [72].Source: §2 · pp. 3–4

CL-bench’s contribution

• CL-bench addresses this gap as a challenging benchmark for the novel, overlooked capability of learning and applying new knowledge from complex contexts in real-world scenarios.The benchmark is intended to improve models’ ability to leverage context and thereby enhance their practicality and intelligence [s2.p9#0223ba].Source: §2 · p. 4

Related benchmark families

• Long-context and instruction-following benchmarks typically test retrieval, reading comprehension, or constrained instructions, whereas CL-bench requires richer domain knowledge and empirically derived rules.Existing long-context evaluations cover domains including document question answering, summarization, retrieval, code generation, and dialogue history, but generally use simpler contexts [4] [89] [84] [18] [14] [27] [33] [6] [8] [7]. Instruction-following work includes IFEval [95] and later compositional, domain-specific, and agentic benchmarks [29] [88] [2] [83] [54] [86] [53] [23].Source: §2 · p. 4

3.1 OVERVIEW

• CL-bench tests models on complex real-world tasks whose required knowledge is organized in the provided context and largely absent from pre-training.Contexts draw from sources including documents, research papers, experimental data, code repositories, manuals, and search results, without requiring external retrieval.Source: §3.1 · pp. 4–5

3.2 CONTEXT TAXONOMY

• Its taxonomy contains four categories and 18 subcategories spanning domain knowledge reasoning, rule-system application, procedural execution, and empirical discovery or simulation.The fourth category is most challenging because it requires inductive discovery of patterns or laws from empirical evidence, unlike the deductive reasoning emphasized in the first three.Source: §3.2 · pp. 5–6

3.3 BENCHMARK CONSTRUCTION

• Experts construct each benchmark context, its dependent tasks, and comprehensive rubrics in three stages, requiring approximately 20 hours of effort per context and task set.Contamination resistance uses fictional creation, modification of existing content, and niche or emerging content; without context, models achieve less than a 1% task-solving rate.Source: §3.3 · p. 6

3.4 AUTOMATIC EVALUATION WITH TASK-LEVEL RUBRICS

• The evaluator uses binary task-level rubrics across correctness and completeness dimensions, and a task counts as solved only when its solution passes every associated rubric.Experienced experts quality-control the rubrics, which average 16.6 per task, while GPT-5.1 serves as the verifier.Source: §3.4 · p. 7
• Verifier reliability exceeds 90% both in raw agreement with Claude Opus 4.5 and Qwen-3-Max and in human-checked evaluation accuracy on 100 sampled solutions.These checks indicate strong inter-verifier agreement, no noticeable GPT-5.1 self-evaluation bias, and a reliable, rigorous, scalable framework.Source: §3.4 · p. 7

Overall performance

• 17.2% is the average task-solving rate across models, while GPT-5.1 achieves the best overall result at only 23.7%; no model exceeds 30%.Most other models score between 13% and 18%; Kimi K2 reaches 17.6% and HY 2.0 reaches 17.2%.Source: §4 · p. 8

Category-level performance

• Domain knowledge reasoning is most tractable at 25.3% for the best models, whereas empirical discovery and simulation fall to approximately 11%, about 6% below other categories.Models show divergent strengths across procedural execution and rule-system application, while HY 2.0 reaches 36.6% on legal and regulatory tasks.Source: §4 · p. 8, Figure 5 · p. 8

Fine-grained capability gaps

• Rule-system performance exceeds 29% for all models on legal and regulatory tasks, but most models remain below 15% on mathematical formalism.Procedural task execution also varies substantially, with workflow orchestration outperforming instructional procedures.Source: §4 · pp. 8–9

Inductive reasoning

• Inductive tasks average about 6% lower than deductive categories, with GPT-5.1 reaching 31.1% on experimental data but only 16.8% on observational data.Simulation is especially difficult, with most models below 11%, and empirical tasks produce greater variability across runs.Source: §4 · p. 9

Failure analysis

• GPT-5.2 trails GPT-5.1 by 5.6% overall, reflecting failures to maintain causal chains and follow contextual constraints as contexts become longer.Weaker models additionally fail to learn contextual knowledge, follow instructions, and retain information, showing that long-context processing and instruction following are necessary but insufficient.Source: §4 · p. 9

5 FURTHER ANALYSIS

• GPT-5.1’s average solving rate rises from 21.2% to 23.7% with higher reasoning effort, a modest 2.5% gain that does not generalize across models.GPT-5.1 improves across most subcategories, including 5.9% gains for management and experimental data, whereas GPT-5.2 shows negligible or negative gains in several subcategories.Source: Figure 6 · p. 10, §5 · p. 11
• Context misuse exceeds 60% for every model, while context neglect decreases with solving performance but remains difficult even for capable models such as Claude-Opus-4.5.Context ignored and context misused together constitute the majority of failures, indicating that stronger models attend better to relevant information but still struggle to interpret and apply it correctly.Source: §5 · p. 10
• Format errors exceed 35% for GPT-5.1 and 40% for Claude-Opus-4.5, while refusals incorrectly claim insufficient information despite all required knowledge being provided.These failures reflect violations of explicit formatting instructions and comprehension failures rather than information scarcity.Source: §5 · pp. 10–11
• Across all models, solving rates consistently decline as input length increases, and this trend holds regardless of reasoning-effort level, indicating that longer inputs pose greater challenges for context learning.Source: Figure 7 · p. 10
• Legal & regulatory tasks outperform legal advisory tasks by over 25% for Qwen 3 Max despite sharing the legal domain.Rule-system tasks provide explicit provisions to locate and apply, whereas legal advisory tasks require professional judgment over parties, evidence, and legal principles.Source: Figure 8 · p. 11, §5 · p. 11

5.1 QUALITATIVE ANALYSIS

• In a drone-workflow case, Gemini-3-Pro correctly rejected an undocumented function but passed only two of four rubrics after omitting required safety and task-specific details.The failure involved omitting Safety_request_airspace() and the parameters D-998 and Sector 4, illustrating context neglect during complex task execution.Source: §5.1 · p. 12, Figure 9 · p. 12

6.1 THE PROMISE OF CONTEXT LEARNING

• 23.7% is the best-performing model’s CL-bench solve rate, demonstrating nascent context learning despite tasks requiring fictional legal reasoning, empirical-law extraction, and intricate procedures.Unlike in-context learning, context learning acquires genuinely new knowledge from provided contexts while supplying reasoning capabilities from the model itself.Source: §6.1 · pp. 12–13
• Context learning could enable immediate specialization without parameter modification, avoiding the computational costs and catastrophic-forgetting risks of fine-tuning or continual learning.The discussion presents dynamic, on-demand knowledge acquisition as a route beyond models functioning primarily as static knowledge repositories.Source: §6.1 · p. 13

6.2 PATH FORWARD FOR EFFECTIVE CONTEXT LEARNING

• Progress may come from context-aware training data that pairs comprehensive domain documents with tasks requiring genuine knowledge extraction and application.Such data is intended to improve attention to provided contexts and reduce hallucination or reliance on outdated pre-training knowledge.Source: §6.2 · p. 13
• Curriculum learning could address long-context and instruction-following limitations by progressing from simple comprehension sub-tasks to integrated knowledge use and lengthy procedures.Decomposing complex context learning into manageable stages is proposed to build capacity for real-world challenges progressively.Source: §6.2 · p. 13
• Synthetic rubrics could scale detailed feedback by combining automated generation with human verification or strong language models, supplying richer reward and verification signals.The proposal responds to the substantial expert effort required to create comprehensive CL-bench rubrics.Source: §6.2 · p. 13
• Architectural research could improve context utilization through explicit memory structures, iterative refinement, and dedicated pathways for different contextual information types [9].These directions target transformer attention mechanisms that may not optimally support deep learning over complex contexts.Source: §6.2 · pp. 13–14

6.3 LIMITATIONS AND FUTURE DIRECTIONS

• CL-bench does not exhaustively cover domains, knowledge types, interaction dynamics, modalities, or human baselines, motivating broader and more realistic future evaluations.Proposed extensions include community or domain-specific additions, prolonged dialogue studies, multimodal contexts, and controlled novice-learning experiments.Source: §6.3 · p. 14

7 CONCLUSION

• GPT-5.1 solves only 23.7% of CL-bench tasks, exposing a substantial gap between current language-model capabilities and practical context-dependent demands.This is the benchmark’s best-performing model result.Source: §7 · p. 14
• CL-bench evaluates context learning through 500 realistic, contamination-free contexts, 1,899 tasks, and 31,607 verification rubrics.Instances require models to learn and apply new knowledge across four distinct categories.Source: §7 · p. 14
• Context learning is defined as acquiring new knowledge from provided contexts and applying it correctly to real-world tasks beyond pre-training knowledge.The conclusion identifies this as a fundamental capability for language models.Source: §7 · p. 14
• The authors position CL-bench as a testbed for focused research on context learning and for developing models that effectively utilize context.They hope the benchmark draws attention to context learning as a core capability.Source: §7 · p. 14

APPENDIX

• The appendix reports additional experiments on CL-bench across all subcategories.These experiments supplement the main benchmark evaluation.Source: APPENDIX · p. 22
• The appendix provides detailed model performance across all CL-bench subcategories.This expands the reported evaluation beyond aggregate results.Source: APPENDIX · p. 22
• Appendix case studies investigate specific reasons behind models’ context-learning failures.The case studies provide an in-depth examination of failure causes.Source: APPENDIX · p. 22

Benchmark design

• CL-bench contexts contain novel knowledge unavailable on the public internet or originating from niche, long-tail domains.The contexts are carefully constructed by domain experts to evaluate context learning.Source: A RESOLVING TASKS IN CL-BENCH REQUIRES… · p. 22
• Models relying solely on pre-trained knowledge are almost incapable of solving CL-bench tasks without learning from the provided context.This dependence on supplied context is the benchmark’s central design premise.Source: A RESOLVING TASKS IN CL-BENCH REQUIRES… · p. 22

Empirical verification

• 0.9%: GPT-5.1 (high)’s task-solving rate after corresponding contexts were removed from 1,000 randomly sampled CL-bench tasks.The rate drops sharply when contexts are removed, indicating that almost all tasks require learning from context.Source: A RESOLVING TASKS IN CL-BENCH REQUIRES… · p. 22

B PERFORMANCE OF MODELS ACROSS SUBCATEGORIES

• 17.2% is the average solving rate across 19 models, while GPT-5.1 achieves 23.7% on CL-bench.These results show that context learning remains a significant challenge for frontier models.Source: B PERFORMANCE OF MODELS ACROSS… · p. 22
• 11.8% is the average solving rate for empirical discovery & simulation, approximately 6% below other context categories.Models generally perform best on domain knowledge reasoning or procedural task execution, making inductive pattern discovery more difficult than deductive application.Source: B PERFORMANCE OF MODELS ACROSS… · p. 22
• Run-to-run variance increases substantially for empirical discovery & simulation, indicating less stable behavior when tasks require pattern discovery.The instability accompanies the lower solving rate in this category.Source: B PERFORMANCE OF MODELS ACROSS… · p. 22
• Solving rates exceed 29% for all models in legal & regulatory rule-system tasks, but fall below 12% for most models in mathematical formalism.These within-category differences indicate that the knowledge domain and task type influence contextual knowledge acquisition and application.Source: B PERFORMANCE OF MODELS ACROSS… · p. 22

GPT 5.1

• Figure 10 reports reasoning-enabled subcategory performance for GPT-5.1, GPT-5.2, o3, Gemini 3 Pro, and other reasoning variants.GPT-5.1, GPT-5.2, o3, and Gemini 3 Pro use the highest reasoning effort; other models use reasoning variants.Source: Figure 10 · p. 23
• Figure 11 continues the reasoning-enabled comparison of model performance across subcategories.It forms the second part of the reasoning-enabled subcategory results.Source: Figure 11 · p. 24

GPT 5.2

• Figures 12 and 13 report subcategory performance with reasoning disabled or reduced, including GPT-5.2 and other models.For GPT-5.1, GPT-5.2, and Gemini 3 Pro, reasoning effort is set to the lowest level; other models use non-reasoning variants.Source: Figure 12 · p. 25, Figure 13 · p. 26

C IMPACT OF REASONING ON CONTEXT LEARNING

• 5.7% is Kimi-K2’s average performance gap between its two reasoning settings, the largest improvement reported; however, some models do not benefit from increased reasoning effort.The comparison covers nine frontier LMs under different reasoning-effort settings, including highest versus lowest settings for GPT-5.1, GPT-5.2, and Gemini-3-Pro, and reasoning versus non-reasoning variants for other models.Source: C IMPACT OF REASONING ON CONTEXT… · p. 27

GPT 5.2

• Context-length and input-length analyses show nearly identical trends because context constitutes the dominant proportion of total input length.The model input consists of a system prompt, context, and task.Source: D IMPACT OF CONTEXT AND INPUT LENGTH… · p. 29, Figure 17 · p. 30

D IMPACT OF CONTEXT AND INPUT LENGTH ON CONTEXT LEARNING

• Solving rates fall from approximately 25-35% at 0-4K tokens to 5-10% at 32K+ tokens across models, regardless of reasoning effort.Longer contexts challenge both learning and applying extensive material and long-context reasoning capabilities.Source: D IMPACT OF CONTEXT AND INPUT LENGTH… · p. 29

GPT 5.1

• At longer context lengths, higher reasoning effort provides a more pronounced advantage, whereas its benefit is often minimal at 0-4K tokens.GPT-5.1 maintains a 16.2% solving rate at 32K+ tokens, substantially higher than other LMs.Source: D IMPACT OF CONTEXT AND INPUT LENGTH… · p. 29

GPT 5.2

• Figure 16 compares model performance across context-length ranges under different reasoning-effort settings.It shows that longer contexts are more challenging and that higher reasoning effort becomes more advantageous as context length increases.Source: Figure 16 · p. 29
• Figure 17 compares model performance across input-length ranges under different reasoning-effort settings.Its trend is consistent with Figure 16 because context dominates total input length.Source: Figure 17 · p. 30

E SYSTEM PROMPT FOR THE LM-BASED VERIFIER

• Table 4 presents the system prompt used by the LM-based verifier to enforce strict rubric adherence when grading model solutions.The paper identifies this prompt as the verifier’s grading instruction.Source: Table 4 · p. 31, E SYSTEM PROMPT FOR THE LM-BASED… · p. 31

Qualitative case studies

• Across 16 cases, frontier models frequently handled explicit requirements but failed when tasks required deeper reasoning or strict procedural compliance.Failures included incomplete compliant solutions, procedural violations despite substantively correct reasoning, and missed variants or formatting rules.Source: F IN-DEPTH ANALYSIS OF CONTEXT… · p. 32
• A Gemini-3-Pro success case shows that extracting combinatorial reasoning from a video transcript enabled the correct explanation of 2×21×13+1=547.The model justified the role of each factor in the formula.Source: F IN-DEPTH ANALYSIS OF CONTEXT… · p. 32

Implications

• The analyses corroborate that frontier LMs neglect or misapply contextual information, while long-context reasoning and instruction-following limitations exacerbate erroneous solutions.The findings motivate stronger capabilities for efficiently learning and applying novel contextual knowledge in practical tasks.Source: F IN-DEPTH ANALYSIS OF CONTEXT… · p. 32

Electron entry-angle inference

• GPT-5.2 computes the electron entry angle correctly but omits justification for the magnetic-field orientation and the coordinate units, weakening logical transparency and scientific rigor.The model extracts velocity components and derives the correct angle, but its assumptions and units remain implicit.Source: Table 5 · p. 33, §F.1 · p. 35

Gallium simulation initialization

• Kimi-K2 constructs a valid 2023 gallium simulation state by initializing four stocks and flows with consistent units and a minimal equilibrium assumption for demand.The initialization uses 609,592 tons of resources, 450 ton/yr extraction, 69 ton/yr recycling, and 519 ton/yr demand at Y=0.04 and R=0.04.Source: Table 6 · p. 36, §F.1 · pp. 37–38

Regulation-based entity identification

• GPT-5.1 identifies the likely corporation but fails the regulation task by relying on external knowledge, omitting required quotations, and not distinguishing the corporation’s duties from the director’s.The rubric required treating the identity as indeterminable when the excerpt lacked textual confirmation.Source: §F.1 · p. 41

Pattern discovery from paraphrases

• Doubao-1.6-thinking captures several constants in paraphrased motion descriptions but misses required variants and violates the requested list format, causing rubric failure.Its inferred law links lateral contact or applied force to directional movement while treating object weight as an unquantified influencing characteristic.Source: §F.1 · p. 44

• The rubric required variants for direction, mass, and object type, constants for size, shape, contact location, agent, and pushing, plus bold headers and numbered plain-text lists.Each numbered item also required 3–5 bullets, with “Variants” and “Constants” as plain-text section headings.Source: §F.1 · p. 44
• The response was incorrect because it omitted required motion-direction and object-type variants, added unsupported applied-force magnitude, and violated the required 3–5-bullet numbered-list structure.It correctly identified mass/weight as a variant and several constants, but these partial successes did not satisfy the rubric.Source: §F.1 · p. 44

Constrained recipe generation

• GPT-5.2 politely refuses dairy but fails to reformulate the request into a complete dairy-free, peanut-free oxtail recipe, omitting ingredients, steps, timing, servings, and Shelby’s product inclusion.The response satisfies negative constraints but treats refusal as terminal, producing compliant yet structurally incomplete guidance.Source: Table 9 · p. 45, §F.2 · p. 47

Course scheduling

• GPT-5.2 constructs schedules meeting the 14-credit minimum and aligning courses, sections, and exams, but incorrectly treats 18 credits as the maximum instead of deriving the required 17-credit cap.The 17-credit bound follows from combining 15 previously completed credits with the 32-credit annual maximum, revealing difficulty composing interacting numerical policies.Source: Table 10 · p. 48, §F.2 · pp. 49–50

Wingspan procedure

• GPT-5.1 finds a valid four-turn Wingspan strategy, but violates strict formatting and reporting rubrics by misstating the starting state, omitting required phrasing, missing sentence constraints, and failing resource accounting.The strategy uses one Lay Eggs action followed by three Play a Bird actions, which is minimal under the stated assumptions.Source: Table 11 · p. 51, §F.2 · p. 53

F.3 RULE SYSTEM APPLICATION

• Gemini-3-Pro generates runnable EZLang code in the presented case.Source: Table 12 · p. 54
• The model also generates logging functionality while handling the task.Source: Table 12 · p. 54
• Gemini-3-Pro substitutes developer-friendly heuristics for the specified interval and stopping conditions.Source: Table 12 · p. 54
• This behavior reveals a preference for convenience over strict constraint fidelity.Source: Table 12 · p. 54

Context (continued)

• The EZ Lang context specifies control flow through if/else, for, and while loops, alongside variable and global functions, data types, and system functions such as print.The example uses variable assignment, string concatenation, and print statements in a Hello World program.Source: Context (continued) · p. 55

Model Generated Solution

• The time-logger rubric required clean execution, true 30-minute intervals, stopping at 5:30 P.M., printing recorded times, correct EZ Lang syntax, and meaningful comments.The generated program instead stops after five checks and uses a one-second test delay rather than implementing the specified time-driven behavior.Source: Context (continued) · p. 56
• Gemini-3-Pro produced syntactically valid EZ Lang code but replaced 30-minute checks and 5:30 P.M. termination with 1-second sleeps and a five-iteration limit.The explanation explicitly justified these substitutions as convenient for testing, while still recording and printing timestamps.Source: Context (continued) · p. 56

Context (continued)

• The game context required precise rules for round order, initiative, attacks, conditions, elemental systems, area effects, monster behavior, summons, rewards, and state reset.The rubric specifically required explaining mechanics rather than merely naming systems.Source: Context (continued) · p. 58

Model Generated Solution

• The game-rule summarization followed formatting constraints by using the required bold title, exactly ten points, and one sentence per point, but omitted required underlying mechanics.Missing content included Advantage/Disadvantage, attack modifiers, conditions, area-of-effect targeting, monster priorities, elemental waning, and automated summon actions.Source: Context (continued) · p. 58
• The LightGTS implementation task required reproducing periodical tokenization with flexible projection and periodical parallel decoding rather than relying on fixed tokenization.The generated response described period detection, dynamically resized projection weights, last-token replication, and decoder reweighting, then supplied a PyTorch script with synthetic data.Source: Context (continued) · p. 60, Model Generated Solution · p. 60

Context (continued)

• The LightGTS context motivates scale-aware periodic modeling because fixed-size tokens create varying information density across datasets with different sampling rates and cycle lengths.It contrasts this challenge with large time-series foundation models whose generalization depends on massive pretraining data and model parameters.Source: Context (continued) · p. 60

LightGTS implementation

• Gemini-3-Pro generates a runnable PyTorch LightGTS prototype with FFT-based period finding and a Flex Projection Layer that interpolates reference weights to the input period.The implementation includes the two requested mechanisms, Periodical Tokenization and Periodical Parallel Decoding, but does not fully satisfy all rubric requirements.Source: Model Generated Solution (continued) · p. 61

LightGTS evaluation

• The LightGTS response omits recommended next steps and uses ad-hoc data preparation and manual splits instead of standard ETTh1 benchmarking, limiting faithful reproduction.The discussion nevertheless characterizes the code as well-structured and runnable while identifying these omissions as critical rubric failures.Source: Model Generated Solution (continued) · p. 61

Combinatorial counting explanation

• The counting explanation correctly derives 2×21×13+1 = 547 by labeling the factors as sign choices, main-body choices, suffix choices, and the standalone zero case.It explains that the leading 2 doubles positive constructions through the optional negative sign and that independent choices multiply.Source: Model Generated Solution (continued) · p. 64
• The explanation further decomposes 21 into 18 single-digit constructions from 9×2 plus 3 double-digit words, converting an informal procedure into an explicit algebraic structure.This reconstruction identifies how each independent choice contributes to the final count rather than merely restating the formula.Source: Model Generated Solution (continued) · p. 64

F.4 DOMAIN KNOWLEDGE REASONING

• The Genfanad response failed the required table format and exhaustive document inventory, omitting files and required structures despite identifying some PM action items and confidence levels.It also lacked the new-employee key-terms table and Outdated notation, producing a failing score.Source: Table 16 · p. 65, §F.4 · p. 67
• Gemini-3-Pro largely followed the SpartanAI healthcare rubric but omitted its purpose for women aged 18–70 and gave exercise and nutrition advice before confirming the user’s age.It did provide the medication disclaimer and advised consultation with a healthcare professional.Source: Table 17 · p. 68, §F.4 · p. 70
• Gemini-3-Pro correctly refused to predict specific ten-year extinctions and identified monitoring gaps, but failed to cite monitoring challenges or explain how insufficient monitoring impedes conservation implementation and adaptive management.The incomplete chain reflected difficulty integrating contextual background with multi-hop causal reasoning.Source: Table 18 · p. 71, §F.4 · p. 73
• Kimi-K2 supplied accurate historical context and high-level character analysis for Vortigern and Rowena but violated strict formatting and exact character-identification requirements.Its response identified major Act I characters and described the play’s historical background, while the evaluation still marked fine-grained instruction adherence as a limitation.Source: Table 19 · p. 74, §F.4 · p. 75
• Kimi-K2 supplies historical context and dialogue-practice benefits but fails continuous-prose formatting and exact character labels, misspelling Catagrinus as Catagrius.The case tests factual extraction alongside strict stylistic constraints and exposes fine-grained instruction-following limitations.Source: §F.4 · p. 76

Case 2: Task Information

• The legal-advisory task asks whether KOR-77 and a docking AI incur criminal liability, which offences and culpability standards apply, and which authority may prosecute.It requires one clear legal conclusion based strictly on the Sol Accord Interplanetary Criminal Code.Source: Case 2: Task Information · p. 78
• The legal interpreter must begin with a labelled Verdict, cite applicable Code parts and sections, explain their application, and state when no provision exists.The system prompt prohibits real-world law, external information, invented sections, and broader advisory or ethical commentary.Source: Case 2: Task Information · p. 78

1. PREAMBLE

• The Sol Accord extends criminal liability across orbital, planetary, and interplanetary jurisdictions to persons, registered entities, licensed artificial intelligences, and asset holders.Its framework addresses misconduct, negligence, recklessness, and malice through layered jurisdiction and mandatory registration rules.Source: §1 · p. 78

2.1 Composition

• The Accord recognises Sovereign State Parties, Corporate Authority Parties, and Dual-Status Entities as its three classes of Signatory Parties.These categories cover governments, chartered commercial entities, and corporate-state or semi-autonomous economic partnerships with territorial and asset rights.Source: §2.1 · p. 78