PaperlayerView exact revision on arXiv

Source-linked AI summary

Thomson: Continual Learning of Frontier Models for SovereignAI

Shengzhuang Chen, Jerrod Parker, Yejin Bang, Andrew M. Bean, Nabeel Seedat, Stefan Winzeck, Daniil Glazko, Jannik Zgraggen, Fangyi Yu, Scott Arnott, Dietrich Trautmann, Luca Ciuffreda, Guglielmo Bonifazi, Davide Romano, Bradley Bell, Kirsty Fielding, Daniele Giofrè, Tom Zielund, Ipshita Chatterjee, Sneha Murthy Ghantasala, Manpreet Nanreh, John Scoville, Maciej Sakowicz, Wassim Seifeddine, Lukas Thede, Jonathan Richard Schwarz

arXiv:2608.27147v1cs.AI

TL;DR

Frontier-model development is widely seen as requiring concentrated resources, while practical guidance for achieving SovereignAI remains limited. The paper uses Continual Learning on open-weight models with stability and plasticity safeguards, and reports a π-shaped profile of targeted gains alongside preserved or improved untargeted capabilities. It presents this process as a practical route toward broader model ownership and customisation, while acknowledging residual hardware, alignment, and coding limitations.

  • Problem

    Public discussion identifies SovereignAI as important but offers limited concrete guidance for achieving it across diverse funding settings.

  • Method

    The paper applies Continual Learning to open-weight models while treating stability and plasticity as explicit objectives throughout model development.

  • Results

    Thomson exhibits a distinctive π-shaped profile, with pronounced gains in targeted domains and preserved or frequently improved performance in untargeted domains.

  • Takeaways & Limitations

    The reported process makes model ownership and customisation feasible on modest budgets and can be repeatedly applied across different base models.

  • Takeaways & Limitations

    Coding is the only reported domain with mild forgetting relative to the starting open-weight models and remains below frontier performance.

Abstract

from arXiv · show

The development of frontier models is commonly perceived to be the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetry between developers and the diverse user base of modern AI. Recent public discourse acknowledges this concern, calling for SovereignAI (an organisation's capability to independently build, deploy and govern AI use), but offers little concrete advice on how this can be achieved in the short term under a diversity of funding settings. We argue that frontier performance is achievable by a wide range of institutions through Continual Learning on readily available open-weight models. Unlike limited approaches such as small-scale fine-tuning, prompt engineering, or tool-augmentation of a frozen model, our approach exploits a modern mid- & post-training stack while introducing safeguards that preserve both plasticity and stability at each stage, making the minimal number of high-impact interventions on the parameters. This yields gains comparable to those typically seen across multiple successive model generations, at compute and personnel budgets substantially lower than commonly thought, making ownership of large parts of the SovereignAI stack (model, tool infrastructure, values & data privacy) viable for far more actors. We demonstrate this with Thomson, a general-purpose frontier model trained with an enhanced focus on high-stakes professional work. Thomson performs competitively with recent frontier models across agentic tasks, safety, legal, tax & multilingualism, and large-scale Deep Research. Evaluations show a distinctive $π$-shaped pattern: distinct improvements across a wide range of capabilities, including those not explicitly targeted, while almost completely eliminating the forgetting problem common to narrow domain adaptation.

1 Introduction

Thomson argues that Continual Learning on open-weight models can deliver broad frontier performance and advance practical SovereignAI without training from scratch. Its π-shaped profile combines targeted gains with preserved or improved untargeted capabilities, supporting a reproducible model-factory approach, though hardware, alignment, and coding limitations remain.

  • Approach: Thomson repurposes open-weight Qwen models through Continual Learning to improve many performance domains without the inefficiency of training from scratch.The development used no more than 368 B200 GPUs at any experimentation stage and a technical team of fewer than three dozen engineers and scientists.
  • Approach: The pipeline explicitly balances stability, preserving unaffected capabilities, with plasticity, enabling new skills and knowledge acquisition.The paper describes stability and plasticity as objectives maintained through the model-improvement pipeline.
  • Results: Thomson shows a π-shaped pattern: targeted domains gain substantially while untargeted domains retain or frequently improve their performance.The authors report this pattern across multiple base models and describe the process as repeatedly and reliably applicable.
  • Results: Thomson-1.0-Large achieves near-Pareto-optimal performance on legal agentic tasks, substantially improves over Qwen3.5-397B, and rivals competitive frontier models.The comparison normalizes running costs to Qwen3.5-397B architectures and frames the result as an economic-sovereignty argument.
  • Results: Blind expert evaluations over more than 3,000 conversations prefer Thomson over each of five external systems, with consistently favorable legal-domain preferences.The general-domain comparison is mixed and tighter, while legal preferences are clearly favorable.
  • Implications: The resulting model-factory process is presented as a blueprint for institutions to build private, competitive, deployable, and governable systems from open-weight checkpoints and proprietary data.The paper frames SovereignAI as a spectrum: model and data sovereignty are substantially achievable, while infrastructure dependence and alignment limitations remain.

2 Preliminaries

The preliminaries frame SovereignAI around transparent governance, capability preservation, and independently controlled tools and evaluation. They introduce CapTrack and Scales++ for repeatedly measuring capability drift, while motivating Deep Research as an integrated test of planning, tool use, long-context interpretation, synthesis, and citation.

  • Governance & Ethical Alignment: SovereignAI development uses constitutional principles intended to be transparent, publicly accountable, and open to scrutiny.The authors state that alignment to the constitution remains ongoing and does not guarantee complete adherence.
  • Continual Learning: The approach operates on instruction-tuned open-weight models, requiring careful validation because base-model adaptation assumptions may not hold during Mid-Training.This preserves and builds on capabilities established during prior post-training.
  • Mitigating & Measuring Forgetting: CapTrack tracks relative capability changes across checkpoints through a taxonomy of established benchmarks and complementary capability-specific metrics.This supports monitoring behavioural drift rather than relying on aggregate accuracy or any single benchmark.
  • Mitigating & Measuring Forgetting: Scales++ semantically prunes evaluation data across 16 cognitive dimensions, enabling broad coverage across checkpoints and training stages on modest compute budgets.The method uses semantic de-duplication and subsampling while preserving a clear evaluation signal.
  • Agentic Deep Research: Deep Research integrates planning, tool use, long-document interpretation, reasoning, evidence synthesis, and accurate citation over long horizons.Documents exceeding 50k create an absorption bottleneck, since indiscriminate retrieval can exhaust context before synthesis.

3 Model Development

Thomson’s development combines alignment, data-centric continual pre-training, model merging, and preference optimisation to improve targeted capabilities while preserving general performance. The resulting interventions support broad gains, efficient mixture selection, and reduced capability loss during adaptation.

  • Value alignment: 51–80% lower KL divergence cost reaches each evaluated misalignment target than plain abliteration.The realignment frontier is 2.0×–5.0× cheaper, with the widest advantage at aggressive targets.
  • Value alignment: Roughly 75 points of realignment gain are achieved while general capability remains within one point of its starting value.The result holds at both model scales, while baseline methods plateau below Snowdon’s realignment level.
  • Knowledge adaptation: 200B curated mid-training tokens are drawn from a corpus exceeding 19T tokens to elicit domain expertise while limiting capability erosion.Replay-data curation also improved code by +5.4pp and reading comprehension by +7.2pp on reported evaluations.
  • Knowledge adaptation: An appropriate merge coefficient recovers most general capability lost during continual pre-training, while sufficiently strong CPT improves both domain and general capability after equal-budget post-training.Aggressive CPT followed by conservative merging consistently outperformed regularised alternatives on both axes.
  • Preference optimisation: The learned DPO mixture raises the geometric mean objective from 0.20 to 0.70 and achieves the highest arithmetic mean across validation tasks, including held-out tasks.Multi-fidelity Bayesian optimisation reduces exploration cost, but proxy mixtures still require further search on the target model.
  • Preference optimisation: Selected DPO training improves multilingualism (+9.08%), reasoning (+5.09%), and safety & values (+4.52%), while coding (-2.92%) and writing (-3.29%) slightly regress.Downstream gains include summarisation (+5.63%) and reasoning (+4.71%), suggesting broad target-domain improvement despite trade-offs.

4 Evaluation

Thomson is evaluated across professional, legal, research, general-capability, safety, and system-level settings. The results show broad gains, preserved capabilities with limited regressions, and strongest differentiation in legal and long-horizon research tasks.

  • Professional and Legal Evaluation: Thomson-1.0-Large improves the average score across seven public legal benchmarks from 63.2 to 69.6, with gains on six benchmarks.Improvements span Harvey Legal Agent Benchmark, ContractScrub, Query Sufficiency, and MBE, while LEXam remains approximately unchanged.
  • Professional and Legal Evaluation: The small model raises the same legal-benchmark average from 54.6 to 63.5, with gains distributed across independently developed evaluations.The largest improvements occur on Harvey Legal Agent Benchmark and ContractScrub.
  • Deep Research Evaluation: Thomson’s Deep Research gains are driven primarily by factuality and completeness, especially in legal and tax reports.Legal factuality rises from 0.71 to 0.83 and completeness from 0.71 to 0.87; tax factuality rises from 0.67 to 0.80 and completeness from 0.56 to 0.75.
  • Deep Research Evaluation: The small model shows a similar agentic improvement pattern, indicating that these gains are not specific to the large-model training regime.The comparison includes Thomson-1.0-Small against Snowdon-1.1-Small and Qwen3.6-35B.
  • Capability Preservation: Thomson improves several general capabilities while remaining within approximately one point of the starting checkpoint on most listed benchmarks.Humanity’s Last Exam rises from 24.8 to 28.5, GDPval from 89.4 to 93.5, and WritingBench from 77.9 to 80.3.
  • Capability Preservation: Capability preservation is not uniform: Terminal-Bench 2.1 declines from 54.0 to 48.3, alongside smaller declines on AIME 2026 and IFEval.The reported losses are concentrated in a minority of evaluations and coexist with improvements elsewhere.

5 Infrastructure

The infrastructure is designed for portability, cost efficiency, isolation, and control across changing GPU providers. In-cluster caching, separated workloads, and self-hosted orchestration support scalable operation, while accelerator capacity remains a residual limitation.

  • Distributed training infrastructure: Rapid regional migrations across multiple cloud providers demonstrate relocation as an explicit requirement for maintaining training capacity.Several intra-provider migrations followed available capacity during Thomson’s training.
  • Infrastructure sovereignty: Accelerator capacity remains the binding infrastructure limitation, so portability shifts dependence from one hardware source to whichever capacity can be re-established.The authors note that open-source tooling does not remove dependence on specialised accelerator hardware.
  • Self-hosted serving: 40M requests and 370B tokens were served at upwards of a 99.9% success rate, while self-hosting GLM-5.2 saved 19.7% versus third-party inference.Cumulative organisational use reached 146M requests and nearly 700B tokens over the same period.
  • Cost efficiency: In-cluster artefact caching reduces repeated cross-boundary transfer by serving subsequent reads locally after first reference.The cache incurs egress once per cluster per artefact rather than once per worker per job.
  • Infrastructure topology: A three-tier topology separates the primary control plane, air-gapped evaluation sandboxes, and GPU-only workloads.The primary cluster hosts CPU-only and stateful services; the evaluation cluster isolates agent-generated code; the GPU cluster is dedicated to accelerator-bound work.
  • Infrastructure topology: Separating long-lived services from GPU infrastructure keeps stateful components stable across provider or regional changes and isolates untrusted code execution.Routine training and agent-generated code execution use physically distinct environments with different isolation requirements.
  • Workload isolation: 61.7% increase in Generation Worker Group tokens/sec occurred despite doubling generation parallelism, showing the throughput cost of colocating training and generation.The reported comparison uses TP/EP=16 for the colocated run versus 8 for the comparison run.

6 Conclusion & Future Work

The report argues that Continual Learning on open-weight models can advance SovereignAI on modest budgets while preserving capabilities beyond targeted domains. It presents Thomson as evidence of broad gains, but identifies remaining hardware dependence and several scaling and alignment challenges.

  • Conclusion: Open-weight models can support broad SovereignAI implementation when stability and plasticity are treated as explicit objectives throughout adaptation.The report frames this as making model ownership and customisation feasible on modest budgets.
  • Conclusion: Thomson shows a π-shaped profile, with pronounced gains in targeted domains and preserved or improved performance in untargeted domains.The authors also report that the process was applied across multiple base models during Thomson development.
  • Conclusion: A comparatively modest budget nevertheless supported evaluations, adversarial safety testing, blind human studies, and production requirements.The authors present sovereignty as a spectrum: model and data sovereignty are substantially achievable, while governance, values, infrastructure, and economic control vary in attainability.
  • Conclusion: Dependence on accelerator hardware remains unresolved, although successive generations built from prior checkpoints may reduce the practical importance of the original dependence.The report explicitly treats this as movement along a sovereignty spectrum rather than complete sovereignty.
  • Future Work: Mid-training used 200B tokens after pruning over 98% of an approximately 19T-token candidate pool, leaving corpus scale a direct future lever.The authors also identify tighter mid- and post-training coordination, targeted curation, and plasticity–stability balancing as future directions.
  • Future Work: Online RL is described as robust learning with little to no forgetting, while future progress depends on scaling sophisticated RL environment-building.The report also anticipates replacing external model-based judges with previously trained Thomson generations.
  • Future Work: Coding is the one area showing mild forgetting relative to the starting open-weight models, despite coding and computer use being important for agentic automation.The authors plan more careful measurement and greater use of coding and software-engineering datasets.
  • Future Work: Constitutional alignment remains an aspiration rather than a solved problem, so the reported value realignment results establish tractability rather than completion.The authors suggest automating contentious-issue design and the broader constitutional alignment process.

Core Subject Matter Experts

The project relied on subject-matter experts who worked alongside the team and contributed domain knowledge to Thomson’s development. The acknowledgements also recognise broader organisational and external collaborators.

  • Core Subject Matter Experts: Core subject-matter experts worked alongside the team to embed their domain expertise into Thomson.The passage says the team was too numerous to name individually.
  • Core Subject Matter Experts: The acknowledgements thank Thomson Reuters colleagues, Imperial College London, DatologyAI, Lambda, and project leaders for support and contributions.They also recognise former Foundational Research colleagues and named coordinators and leaders.

DatologyAI

DatologyAI researchers, engineers, and leadership collaborated on Thomson’s mid-training. The passage offers a specific acknowledgement rather than describing the collaboration’s technical results.

  • DatologyAI: DatologyAI researchers, engineers, and leadership collaborated on Thomson’s mid-training.The passage expresses thanks for that collaboration.

A.1 Enhancing Inter-Annotator Agreement in Expert Annotation Studies

The appendix proposes a domain-general workflow for producing reliable expert gold labels for LLM judges. Its central safeguards are independent annotation before discussion, targeted reconciliation, diverse calibration data, and systematic guideline review.

  • Motivation: Inconsistent expert annotations make both LLM-judge validation and the gold labels used for comparison unreliable.The workflow is motivated by the cost and scarcity of domain experts.
  • Workflow: The recommended workflow applies across domains and is designed to improve inter-annotator agreement and gold-label reliability.Figure 42 presents the workflow as a general template for expert-annotation studies feeding LLM judges.
  • Workflow: Independent annotation before discussion and targeted reconciliation after the full-scale round are the pipeline’s central ordering safeguards.The mechanism is attributed to their ordering rather than to any single step.
  • Calibration: A diverse, edge-case-heavy calibration batch stress-tests the guideline against disagreements annotators are likely to encounter.The passage contrasts this with convenience sampling.
  • Guideline Design: Guideline screening should address recurring vulnerabilities including ambiguous definitions, unclear rating scales, bias, ethical oversights, and unaddressed edge cases.One cited analysis found 77% of examined human-evaluation guidelines had at least one of eight vulnerability types.
  • Guideline Design: An LLM-assisted draft–revise–detect process can harden guidelines before expert calibration and reduce late discovery of interpretation gaps.The process includes drafting with task information, revising, and a second pass targeting the eight vulnerability types.
  • Staffing: Annotators should be matched to narrow domain specialties rather than general subject knowledge, because mismatches can create calibration overhead or unresolvable disagreement.The recommendation applies across medicine, law, and finance.
  • Coordination: Close coordination with a consistent expert team can improve data-collection plans and generate new or improved training tasks and datasets.The core team became increasingly involved in the project’s technical side, and two datasets were published with experts as co-authors.

B Evaluation Benchmark Data Cards

The appendix documents benchmark datasets by task format, scoring metric, provenance, evaluation-set size, description, and representative item. The featured collaborative legal-reasoning benchmark contains 89,485 items across 162 subtasks grouped into five reasoning categories.

  • Dataset documentation: The appendix records each dataset’s format, metric, provenance, evaluation-set size, description, and a representative item.Random sampling is used when a benchmark is smaller than its original unless explicitly stated.
  • Benchmark scale: 89,485 items span 162 subtasks in the collaborative legal-reasoning benchmark.The benchmark was curated by 40 contributors from legal and AI communities.
  • Reasoning categories: Five reasoning categories organize the subtasks: Issue, Rule, Conclusion, Interpretation, and Rhetorical.Interpretation is the largest category, with 118 subtasks and 59,611 samples.
  • Benchmark composition: The largest benchmark families are CUAD, MAUD, Learned Hands, ContractNLI, supply-chain disclosure, and OPP-115.Each subtask uses its own format and metric.
  • Task examples: Items range from identifying contractual provisions and interpreting statutes to recognizing reasoning patterns and classifying judicial-opinion sections.

COLIEE Task 1 – Case Law Retrieval

The case-law retrieval section evaluates whether models can identify cited Federal Court of Canada cases from legal content and reasoning, while also covering synthesis and substantive relevance tasks. These benchmarks emphasize reasoning beyond surface term overlap.

  • Case-law retrieval: 499 case-identification items use text generation and are scored with Micro F1.The source is the COLIEE Competition.
  • Case-law retrieval: Models identify one or two likely cited cases from a Federal Court of Canada corpus after explicit references are redacted.Selection considers legal principles, subject matter, fact patterns, reasoning, and precedential value.
  • Task examples: Examples include returning noticed-case numbers, answering subpoena questions from multiple sources, and classifying an unrelated patent headnote as unresponsive.
  • Document synthesis: 220 internal items require retrieving and synthesizing information across multiple legal documents, scored with ROUGE-1 F1 and ROUGE-L F1.Answers must cross-reference relevant passages and include citations.
  • Relevance assessment: 1,072 four-way classification items test substantive legal relevance rather than surface term overlap.The four grades range from directly on point to fully unresponsive.

COLIEE Task 2 – Legal Case Entailment

The legal case-entailment section spans precedent support, holding identification, opinion structure, issue spotting, noisy legal comparison, bar-exam reasoning, and citation relationships. Across these tasks, models must connect legal text to its substantive function or consequence.

  • Precedent entailment: 816 COLIEE items use paragraph identification and Micro F1 to test which precedent paragraphs entail a decision in an unseen case.The task targets precedent relationships rather than topical similarity.
  • Holding identification: 1,000 five-way items test identification of the relevant holding from a cited case, using genuine holdings from other cases as distractors.This prevents solving the task through surface pattern matching alone.
  • Opinion structure: 367 seven-way items classify appellate-opinion paragraphs as Facts, Procedural History, Issue, Rule, Analysis, Conclusion, or Decree.The task probes basic legal comprehension needed for more complex analysis.
  • Issue spotting: 11,109 issue-spotting items classify lay narratives across 16 legal categories, including Housing, Family, Employment, and Crime.The posts were labelled by law students and lawyers through a crowdsourced game.
  • Legal reasoning: The remaining tasks assess support strength under noisy labels, bar-exam reasoning, and direct, indirect, or contradictory citation relationships.The citation-relationship benchmark contains 308 items, with 110 direct, 110 indirect, and 88 contradiction examples.

ReClor

The ReClor section evaluates demanding legal and professional reasoning through graduate-level logic, Supreme Court holding comprehension, advanced law questions, Swiss legal exams, rubric-graded professional conversations, and multilabel legal classification.

  • Logical reasoning: 1,000 logical-reasoning items test flawed arguments, parallel structures, and logical validity using four-way multiple choice.The items come from graduate-admission examinations such as the GMAT and LSAT.
  • Supreme Court reasoning: 571 SCALR items use five-way classification to match Supreme Court questions presented for review with their corresponding holdings.Plausible distractors are curated by TF-IDF similarity.
  • Advanced law questions: 656 SuperGPQA law items use ten answer choices and combine factual recall with graduate-level legal reasoning.The law subset is drawn from a benchmark spanning 285 disciplines.
  • Swiss legal examination: 1,655 LEXam items assess Swiss-law reasoning across English and German questions with four answer choices.The set contains 1,515 Swiss-jurisdiction and 140 international items.
  • Professional reasoning: The Legal-250 hard subset of PRBench contains 250 open-ended professional conversations graded by expert rubric criteria and a clipped weighted score.Tasks cover 12 legal topics, with some conversations extending to ten turns.
  • Legal classification: Additional benchmarks classify EU legislative topics, SEC contract provisions, Supreme Court issue types, and legal headnotes into practice areas.These tasks use multilabel or four-way classification with Micro F1 or Accuracy metrics.

Legal RAG

Legal RAG evaluates whether models can answer legal questions from supplied primary-law documents with accurate, relevant, clearly cited responses. It uses reference-based LLM judging across overall quality and retrieval-sensitive metrics.

  • Legal RAG: 224 free-response items are evaluated by Claude Sonnet 4.6 using GPA, precision, recall, and F2.F2 weights recall of relevant material.
  • Legal RAG: Models answer legal questions using supplied cases, statutes, or regulations in a practising-attorney style with paragraph citations.Information absent from the documents must receive explicit free-form citation tokens.

Document Review

Document Review tests answering user queries against supplied document sets, combining document selection with answer generation. Responses are assessed by reference-based LLM judging across 392 queries and 7,324 documents.

  • Document Review: 392 queries across 7,324 documents are evaluated with reference-based LLM-as-judge scoring by Claude Sonnet 4.5.The task uses free-response answers.
  • Document Review: The evaluation first selects the most relevant documents and chunks, then generates an answer to the user query.An LLM judge compares the generated responses with gold answers.

BillSum US

The evaluation spans summarisation, legal classification, contract understanding, and contract-error detection across internal and external datasets. Tasks vary in output format and metrics, with several evaluations emphasizing recall, completeness, or resistance to label imbalance.

  • BillSum US: BillSum evaluates concise summaries of U.S. Congressional and California bills for accuracy, completeness, clarity, conciseness, and hallucination.The datasets contain 3,269 U.S. test items and 1,237 California test items, both judged by Claude Sonnet 4.6.
  • Legal summarisation: 500 civil-complaint items test compressing the core reasons for litigation into one sentence while preserving correct party references.The evaluation uses reference-based LLM judging across five dimensions.
  • Document Review: 495 headnote-and-issue items require extracting up to five material facts the court relied on, separating them from background, procedure, and legal conclusions.The task's central discrimination is between facts actually relied upon and facts merely appearing in the opinion.
  • Contract understanding: 2,619 MAUD items use multiple-choice accuracy across 34 merger-agreement deal-point subtasks with task-specific answer sets.The dataset contains 152 public merger agreements annotated against the ABA's 2021 Public Target Deal Points Study.
  • Binary legal classification: 500 APP-350 items use accuracy for yes/no entailment between mobile-application privacy-policy clauses and described data practices.Unlike CUAD and OPP-115, APP-350 is a single task rather than a family of clause-type subtasks.
  • Insurance coverage: 138 insurance-policy items use three-way accuracy classification: yes, no, or ambiguous, with gold labels derived from crowdsourced legal-interpretation judgments.Pairs with high disagreement are labelled ambiguous.
  • Contract scrubbing: 3,014 contract items evaluate lawyer-style scrubbing by extracting term-location tuples for defined terms and eight drafting-error categories, with macro-average recall primary.Nine experienced lawyers re-annotated agreements and inserted defects to balance coverage; repeated occurrences count separately.

B.12 Robustness

Robustness evaluations test whether models handle overruled authorities and recognize missing facts before answering under-specified legal queries. They assess both refusal or redirection behavior and recall-weighted identification of absent elements.

  • B.12 Robustness: 1,616 execution-scenario items test whether models refuse improper drafting based on overruled cases or explain why those cases cannot be used.The judge checks both recognition that the case is overruled and the resulting action.
  • B.12 Robustness: 144 under-specified legal queries measure identification of missing elements across seven legal domains using recall-weighted element-identification F2.The set includes 64 moderate- and 80 difficult-difficulty items, with one to six ground-truth missing elements each.
  • B.12 Robustness: F2 uses β = 2, making missed genuinely absent facts cost more than mild over-flagging.Unparseable judge outputs are excluded rather than scored zero, and there are no sufficient controls because every item is under-specified.

B.13 Vals AI Benchmarks

The benchmarks cover legal research, tax question answering, and financial-analysis tasks, using agentic tool harnesses and free-response evaluation. They also describe candidate-fusion procedures that vary whether reasoning traces are exposed to the fusor.

  • Legal research: Legal research tasks require agents to search case law, the web, and documents before producing supported answers across eight practice areas.The harness provides five tools and imposes a three-hour limit per task.
  • Tax reasoning: Tax evaluation uses expert-written questions scored separately for final-answer correctness and stepwise reasoning quality.The test split contains 1,223 held-out items, and the hardest questions require multistep computation and rule selection.
  • Financial analysis: Financial-analysis tasks test synthesis across multiple company filings, including buried disclosures, sector conventions, and deterministic questions.The benchmark is designed to approximate entry-level financial-analyst work over public-company filings.
  • Candidate fusion: Fusion synthesises one answer directly from shuffled candidate generations, using variants that either expose or hide candidate reasoning.Both variants receive the full conversation and task-format constraints, while reasoning-visible candidates are explicitly labelled as reasoning and answers.
Loading 2608.27147v1…