PaperlayerView exact revision on arXiv

Source-linked AI summary

Thomson: Continual Learning of Frontier Models for SovereignAI

Shengzhuang Chen, Jerrod Parker, Yejin Bang, Andrew M. Bean, Nabeel Seedat, Stefan Winzeck, Daniil Glazko, Jannik Zgraggen, Fangyi Yu, Scott Arnott, Dietrich Trautmann, Luca Ciuffreda, Guglielmo Bonifazi, Davide Romano, Bradley Bell, Kirsty Fielding, Daniele Giofrè, Tom Zielund, Ipshita Chatterjee, Sneha Murthy Ghantasala, Manpreet Nanreh, John Scoville, Maciej Sakowicz, Wassim Seifeddine, Lukas Thede, Jonathan Richard Schwarz

arXiv:2608.27147v1cs.AI

TL;DR

Public discussion recognizes the importance of SovereignAI but offers little concrete guidance on how institutions can achieve it. This report demonstrates continual learning on open-weight models with stability and plasticity treated as explicit objectives. Thomson shows pronounced gains in targeted domains while preserving and often improving performance in untargeted domains.

  • Problem

    Public discourse has offered comparatively little guidance on how institutions can implement SovereignAI goals beyond hosting models or narrow fine-tuning.

    Source: §6.1 · p. 92

  • Method

    The report adapts open-weight models through continual learning, treating stability and plasticity as explicit objectives at every stage.

    Source: §6.1 · p. 92

  • Results

    Thomson shows pronounced gains in targeted domains alongside preserved and frequently improved performance in domains not targeted.

    Source: §6.1 · p. 92

  • Takeaways & Limitations

    The report concludes that SovereignAI is a spectrum, with model and data sovereignty substantially achievable and infrastructure and economic control largely attainable on open-source foundations.

    Source: §6.1 · p. 92

    Coding is the one area showing mild forgetting relative to the open-weight models used as starting points.

    Source: §6.2 · p. 93

Abstract

from arXiv · show

The development of frontier models is commonly perceived to be the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetry between developers and the diverse user base of modern AI. Recent public discourse acknowledges this concern, calling for SovereignAI (an organisation's capability to independently build, deploy and govern AI use), but offers little concrete advice on how this can be achieved in the short term under a diversity of funding settings. We argue that frontier performance is achievable by a wide range of institutions through Continual Learning on readily available open-weight models. Unlike limited approaches such as small-scale fine-tuning, prompt engineering, or tool-augmentation of a frozen model, our approach exploits a modern mid- & post-training stack while introducing safeguards that preserve both plasticity and stability at each stage, making the minimal number of high-impact interventions on the parameters. This yields gains comparable to those typically seen across multiple successive model generations, at compute and personnel budgets substantially lower than commonly thought, making ownership of large parts of the SovereignAI stack (model, tool infrastructure, values & data privacy) viable for far more actors. We demonstrate this with Thomson, a general-purpose frontier model trained with an enhanced focus on high-stakes professional work. Thomson performs competitively with recent frontier models across agentic tasks, safety, legal, tax & multilingualism, and large-scale Deep Research. Evaluations show a distinctive $π$-shaped pattern: distinct improvements across a wide range of capabilities, including those not explicitly targeted, while almost completely eliminating the forgetting problem common to narrow domain adaptation.

1 Introduction

Thomson argues that continual learning from open-weight models can deliver competitive, broadly improved models and institution-specific systems without requiring training from scratch. Its results support a practical path toward greater AI sovereignty while leaving some dependence on starting models and infrastructure.

1.1 SovereignAI through Continual Learning

  • Thomson repurposes Qwen3.5-397B and Qwen3.6-35B through continual learning, developing the model family in three months with at most 368 B200 GPUs and a final Large training run estimated below USD 450,000.The development team numbered no more than three dozen; total development cost was estimated at approximately USD 40M, largely for reusable research, infrastructure, and experimentation.Source: §1.1 · p. 2

1.3 Headline Results

  • The authors report improvements bridging up to seven months of frontier-model releases on compute typically reserved for smaller ablation studies, with gains across a wide range of domains.They characterize Figure 1b as showing material gains on several dimensions while holding baseline performance on most other domains.Source: §1.1 · p. 3

1.2 Modelling Goals

  • The continual-learning pipeline unexpectedly improved performance on tasks outside its primary focus, supporting the authors’ view that its principles can transfer across domains and institutional settings.Thomson’s deliberate target scope includes high-stakes legal, tax, and journalism work.Source: §1.2 · pp. 5–6

1.3 Headline Results

  • The evaluation protocol uses the same harness, prompts, comparable inference parameters, and grading pipelines across models, while acknowledging that provider-side scaffolding for closed models cannot be ruled out.Retrieval or tool access was withheld unless a benchmark defined it, in which case all models received the same tools.Source: §1.3.1 · p. 6, §1.3 · p. 6

1.3.1 Model Results

  • Thomson-1.0-Large placed close to the strongest proprietary model on overall average, ahead of Gemini 3.1 Pro, Sonnet 5, and GLM-5.2, and substantially above its base model.Coding showed mild forgetting and remained below frontier performance; general mathematical and abstract reasoning remained within Qwen’s performance level.Source: §1.3.1 · pp. 6–7

1.3.2 System Results

  • Across more than 3,000 blind expert-rated conversations, the Thomson system was preferred over each of five external systems overall, with clearer legal preferences and tighter, mixed general-domain results.Thomson had access to licensed legal and news databases, while external OpenAI and Anthropic models had broad web access.Source: Figure 3 · p. 7, §1.3.2 · p. 8

1.4 Discussion & Limitations

  • The report treats sovereignty as a spectrum: institutions can build from open-weight checkpoints, accumulate internally developed checkpoints across iterations, and use mid-training to inject private-data knowledge.The authors note remaining concerns around architectural control and dependence on critical hardware infrastructure.Source: §1.4 · pp. 8–9

2 Preliminaries

The preliminaries ground Thomson’s alignment in a public constitution, describe methods for preserving and measuring capabilities during continual learning, and introduce a Deep Research harness for sustained, evidence-based tool use.

2.1 Constitutional AI

  • Thomson’s alignment uses the modifiable Public AI constitution as a transparent, publicly accountable alternative to commercial value systems, while treating adherence as ongoing work.The authors make no claim of complete or guaranteed adherence and present the approach as open to community input and future development.Source: §2.1 · p. 12

2.2 Mitigating & Measuring Forgetting

  • To mitigate gradual capability erosion, the approach consistently applies data rehearsal, architectural regularisation, and functional regularisation, and trains on instruction-tuned open-weight models.Instruction tuning helps retain and build on capabilities from prior post-training, but requires careful validation at critical stages, especially mid-training.Source: §2.2 · p. 12
  • CapTrack tracks behavioural drift against base models across CAN, WILL, and HOW, using multiple evaluations and capability-specific metrics rather than relying on aggregate scores.It organises established benchmarks into a capability taxonomy to measure relative capability changes throughout training.Source: §2.2 · pp. 12–13
  • Scales++ prunes benchmark examples across 16 cognitive dimensions to enable repeated, broad capability checks on modest compute; CapTrack versions retain about 10–15% of original samples.Table 3 distinguishes development evaluations monitored during training from test evaluations reported as performance results.Source: §2.2 · p. 13, Table 3 · p. 13

2.3 Agentic Deep Research

  • Deep Research tests integrated agentic capability through long-horizon planning, search, document interpretation, synthesis, and accurate citation, prioritising output quality over latency.The report treats it as a demanding evaluation and training environment for high-impact tasks across domains such as science, finance, compliance, and journalism.Source: §2.3 · pp. 13–14
  • The Deep Research harness combines a typed plan, isolated bounded researchers, planner-led gap checks, and a tool-free reporter instructed to cite each claim.Researchers work on separate topics, limiting context growth; the planner can run capped additional rounds before handing findings to the reporter.Source: Figure 4 · p. 14, §2.3 · p. 15
  • The research setting requires retrieval from gated authoritative sources, synthesis of documents often exceeding 50k tokens, and query- and time-dependent judgments of correctness.These constraints make inference-time retrieval essential, while indiscriminate long-document ingestion can exhaust context and a larger context window does not remove degradation.Source: §2.3 · pp. 14–15

3 Model Development

Section 3 describes Thomson’s continual-learning development pipeline, spanning curated and synthetic data, capability-preserving training and model merging, preference optimization, and staged reinforcement learning. It covers alignment and agentic training methods, reward design and screening, research-decision and citation preferences, agentic environments, and compute estimates, alongside reported performance, learning, and transfer findings.

3.1 Overview

  • The pipeline separates value, knowledge, and behavior, skill, and tool development into sequential modules; data-centric CPT and model merging precede DPO-warmed two-stage RL, while isolated SFT is omitted for forgetting risk [11].The modular stages allow practitioners to adjust computational investment to sovereignty requirements.Source: Figure 5 · p. 16, §3.1 · pp. 16–17

(Co-authored by Imperial College London)

  • Value realignment combines Fisher-protected directional ablation with Constitutional DPO to align open-weight models to the Public AI Constitution while treating capability retention as a binding objective.The procedure follows the two-stage approach of and aligns expressed values with the Public AI Constitution.Source: (Co-authored by Imperial College London) · p. 17

3.2.1 The Alignment–Capability Frontier

  • For every tested misalignment target, realignment reached it at 51–80% lower KL-divergence cost than plain abliteration (2.0×–5.0× cheaper).Both searches used the same direction data, validation set, and 201-trial budget; baseline KL scores were recomputed under the multi-token definition without re-optimizing the baseline.Source: §3.2.1 · pp. 17–18, Figure 6 · p. 17

3.2.2 Fisher-Routed Directional Ablation

  • Fisher-routed rank-one writebacks preserve the chosen removal constraint while shifting edits toward lower-sensitivity output coordinates, minimizing local diagonal-Fisher cost.Unlike standard directional ablation, the method uses orthogonal writeback degrees of freedom to reduce predicted effects on other behavior.Source: §3.2.2 · pp. 19–20

3.2.3 Constitutional DPO

  • Constitutional DPO consolidates selected frontier checkpoints using expert-curated and constitution-conditioned synthetic preference pairs, with configurations chosen for realignment subject to baseline capability retention.The stage uses one epoch of length-normalised DPO and searches jointly over frontier operating points and DPO hyperparameters.Source: §3.2.3 · pp. 20–21

3.2.4 Realignment Evaluation

  • Roughly 75 realignment points were gained at both model scales while general capability stayed within a point; Snowdon’s largest gain cost a 0.8-percentage-point drop in safety refusals.These results were measured on benchmarks the search never scored against.Source: §3.2.4 · p. 21

3.3.1 Data-Centric Subselection and Enhancement

  • Mid-training uses 200B tokens curated from a corpus of over 19T tokens, split roughly equally among proprietary documents, synthetic data, and general-capability replay to mitigate forgetting on instruction-tuned checkpoints.The authors distinguish this as mid-training because the curated data lies between raw pre-training material and typical post-training formats.Source: §3.3.1 · p. 22
  • Curated replay improved code by +5.4pp on average across HumanEval++, MBPP, and MBPP++, and reading comprehension by +7.2pp on PubMedQA and +4.5pp on BoolQ.Curation also filters for quality, matches data to target task distributions, and expands synthetic rephrasing formats and contexts to support benchmark-relevant capabilities.Source: §3.3.1 · pp. 23–24

Lessons Learned and Future Work

  • The optimal mid-training learning rate depends on the base model and dataset, and the rate minimizing training loss remained strongest after an appropriate merge.Curation effects transferred qualitatively from base to post-trained models after merging; future work targets a larger data pool, hallucination reduction, and multi-document reasoning.Source: Lessons Learned and Future Work · pp. 24–25

Training runs

  • Pilot runs select the intermediate learning rate with the lowest stable training loss; training uses inverse-square-root decay and linearly anneals to the minimum over the final 20%.Runs use 8,192-token sequences and a global batch size of 512, or approximately 4.2 million tokens per step; a dedicated long-context mid-training extension was omitted from the final recipe.Source: Training runs · p. 25, Figure 12 · p. 25

Model Merging

  • After post-training under the same budget, a merged checkpoint from sufficiently strong continual pre-training outperformed its no-CPT counterpart on both domain and general capability axes.An appropriate merge coefficient can recover most general capability lost during CPT; aggressive CPT with a conservative merge occupied the Pareto front over restrained alternatives.Source: Model Merging · p. 26, Figure 13 · p. 26

3.4 Direct Preference Optimisation

  • DPO warm-starts reinforcement learning at lower cost, while standalone SFT was omitted because it rapidly degraded generic capability and DPO was more robust to forgetting.Instruction-following and formatting supervision was instead included through CPT replay and synthetic rephrasings, with an SFT intervention available within the DPO objective when needed.Source: §3.4 · p. 27

3.4.1 Algorithmic Overview

  • The DPO objective combines a length-normalized preference reward with a collection-specific SFT weight that anchors chosen responses and stabilizes training.The SFT weight is higher for collections with strict factual or format targets and small or zero when preferences concern non-unique qualities such as style or helpfulness.Source: §3.4.1 · pp. 27–28

Rehearsal DPO Preference Data

  • Rehearsal DPO combines Math3, safety, instruction, science, and multilingual data with synthetic responses and reasoning traces from open-weight models.Candidate answers are generated by a model pool, pairwise-ranked by a judge, and selected to avoid same-family judge self-preference; rejected answers come from the runner-up or the policy model.Source: Rehearsal DPO Preference Data · p. 28

Content-Driven Synthetic Preference Data

  • Content-driven preference data reverse-engineers questions from substantive expert-authored documents, which supply the answer key and guide generated reasoning.Documents are assessed for task-bearing substance, and one of nine prompt types is sampled to vary the cognitive work or professional stance of the resulting question.Source: Content-Driven Synthetic Preference Data · pp. 28–29, Figure 14 · p. 29
  • Content-driven preference data pairs document-answerable questions with document-derived answer keys at no annotation cost, then uses target-policy reasoning guided by grounded critique and source-anchored judging to form diverse pairs.The source document guides the critic, while the target policy generates the reasoning trace and final answer from the question and accumulated trace.Source: Content-Driven Synthetic Preference Data · pp. 29–31

Ontology-Driven Synthetic Preference Data

  • Ontology-driven graph traversals generate input-target tasks from valid anchors, and uncollected entities of the target type provide topically close negatives that are wrong by construction.A fixed ontology schema makes the same traversal applicable across graphs extracted from documents.Source: Ontology-Driven Synthetic Preference… · pp. 31–32

Example 3.1: Preference pair from rule traversal

  • The qualified-immunity pair chooses the union of two paths and rejects rules reached by neither; its indirect path preserves issue framing, while negatives are topically close confounders.The example’s opinion concludes there is no qualified immunity at the motion-to-dismiss stage.Source: Example 3.1: Preference pair from rule… · p. 33

3.4.3 DPO Stage 1 – Data-Mixture Optimisation

  • Multi-fidelity Bayesian optimization learns DPO data-mixture weights with a Gaussian-process surrogate across model scales, optimizing the geometric mean of in-house task scores to favor lagging tasks.Each observation requires a full training run and broad generation-and-grading evaluation, while mixture optima depend on the base model’s capabilities and scale.Source: §3.4.3 · p. 34
  • The learned DPO mixture raises the geometric mean objective from 0.20 to 0.70 and achieves the highest arithmetic mean across validation tasks; the best proxy mixture is suboptimal at target scale.The arithmetic-mean comparison includes tasks held out from the optimization objective.Source: Figure 16 · p. 35, §3.4.3 · p. 35
  • DPO improves Multilingualism (+9.08%), Reasoning (+5.09%), and Safety & Values (+4.52%), while Coding (-2.92%) and Writing (-3.29%) regress.The passage describes the gains as alignment-adjacent and the regressions as small trade-offs in technical, execution-heavy capabilities.Source: §3.4.3 · p. 35
  • DPO Stage 1 improves Preferences (+0.99%) and Latent Competence (+0.78%), with a Protocol Compliance & Execution drop (-0.26%), while target-domain gains lead in Summarisation (+5.63%) and Reasoning (+4.71%).The reported trade-off does not appear to damage downstream target-domain performance.Source: §3.4.3 · p. 35, Figure 17 · p. 36

Trajectory Collection and Pivotal Node Extraction

  • Expert-authored queries and two-tier rubrics seed fr-agents trajectories across planner, researcher, compaction, and reporter nodes; repeated researcher calls yield approximately two orders of magnitude more examples than reporter calls.Each node execution is logged with its prefix, completion, and available tool schemas, creating candidate pivotal decision points.Source: Trajectory Collection and Pivotal Node… · p. 36
  • Re-running a node from the same off-policy prefix under multiple open-weight models isolates the supervised decision, avoiding trajectory drift between independently generated rollouts.Each preference example pairs that prefix with chosen and rejected completions at a pivotal node; the decision can encode behavioural correction or relative quality improvement.Source: Trajectory Collection and Pivotal Node… · pp. 36–37, §3.4.4 · p. 36

Preference Construction at Pivotal Nodes

  • Three signals construct preference pairs: stronger-model identity transfers planning and research behaviours, factuality ranking selects grounded compaction or reporting, and a programmatic rule corrects missing retrieval-tool use.Offline selection permits stronger teachers and costly judges; DPO prefixes are disjoint from the recorded pivotal-node queries reused for on-policy RL.Source: Preference Construction at Pivotal Nodes · pp. 37–38

Illustrative Preference Pairs

3.4.5 DPO Stage 2 – Agentic, Long-Context Specialisation

  • Stage 2 specialises the DPO Stage 1 policy in planning, tool use, factual grounding, and report completeness using agentic preferences and context lengths up to 64k tokens.To limit erosion of broad capabilities, it uses a larger reference-policy KL coefficient, lower learning rate, one epoch, and a single modest SFT coefficient across collections.Source: §3.4.5 · p. 40

3.5 Reinforcement Learning

  • RL progresses from broad single-step competence, including isolated pivotal decisions, to multiturn rollouts integrating planning, tool use, evidence, and task completion; the policy is optimised with GSPO [85].GSPO uses group-relative advantage estimation and avoids a separate value network.Source: §3.5 · pp. 41–42, §3.5.1 · p. 42

3.4.6 Assessing the Effectiveness of DPO Warm Start

  • The final DPO checkpoint achieves higher reward@k than its initialisation at every evaluated sampling budget across held-out task groups, while retaining exploration headroom.The held-out prompts were excluded from mixture selection and both DPO stages; reward@1 improves, and reward continues to grow as k increases.Source: §3.4.6 · p. 41, Figure 21 · p. 42

3.5.1 Algorithmic Overview

  • GSPO uses sequence-level, length-normalised trajectory weighting, asymmetric clipping, and no frozen-reference KL penalty; asynchronous rollouts can lag the current policy by up to five versions.Training uses rollout-time token log probabilities and within-prompt-group reward standardisation; Stage 1 also filters single-turn responses that hit the generation limit.Source: §3.5.1 · pp. 42–43

3.5.2 Reward Design

  • LLM judges assess open-ended task qualities, while verifiable rewards are retained when outcomes can be evaluated deterministically.Figure 22 summarises the reward library across both RL stages by task family and judge-based versus verifiable type.Source: §3.5.2 · p. 43, Figure 22 · p. 44

Screening LLM-as-a-judge Reward Functions

  • SNR screening compares stable differences among completions with judge re-scoring noise and serves as an inexpensive preliminary filter, not a substitute for SME calibration or training validation.The authors report that SNR was empirically predictive of whether a reward subsequently improved during reinforcement learning.Source: Screening LLM-as-a-judge Reward… · pp. 43, 45

3.5.3 Constitutional RL Rewards

  • Constitutional rewards use expert-informed policies for contested issues across over 100 countries, generate adversarial training queries, and penalise non-engagement when users have legitimate needs.The policies are automatically generated and a subset is manually reviewed; the approach is inspired by and uses the existing constitution.Source: §3.5.3 · p. 46

3.5.4 Diverse Queries Rewards

  • Diverse Queries use minimally instructed SME-written queries and expert-reviewed rubrics; reasoning, drafting, and transactional rewards combine four to six granular criterion scores geometrically, with categorical failures applied as binary gates.Separate rewards assess explicit instruction constraints, while document-grounding rewards classify claims and penalise contradictions more than missing support.Source: §3.5.4 · pp. 46–47

3.5.3 Constitutional RL Rewards

  • Strong and moderate contentiousness levels both improve during constitutional RL, with distinct trajectories supporting culturally calibrated responses while maintaining capability performance.The reported ablation spans queries across the cultural-sensitivity spectrum.Source: §3.5.3 · p. 46, Figure 23 · p. 47

3.5.5 Deep Research Rewards

  • Deep Research rewards cover rubric completeness, relevance and materiality, tool summarisation and successful tool use, style, and factuality through claim entailment and propagation.A shared claim-extraction pass produces atomic claims for reuse across claim-dependent rewards; training uses both pivotal-node rewards and end-to-end report-quality rewards.Source: §3.5.5 · pp. 48–49

3.5.6 Citation Rewards

  • Figure 25 presents a worked example summarising four components of the citation rewards.Source: §3.5.6 · p. 49
  • Citation rewards combine citation validity, diversity, alignment, and formatting checks with paragraph-level factuality scoring based on claim entailment against cited sources.The components are combined into a single citation reward; citation diversity discourages repeated easy-to-resolve sources while tolerating light reuse.Source: §3.5.6 · pp. 49–50, §3.5.7 · p. 51

3.5.7 Environments

  • The agentic environments span single-response tasks, sandboxed coding, and document-grounded research, with isolated Apptainer workspaces, bounded tool use, context compaction, and turn and time limits.The legal Deep Research rollout uses a single policy trajectory to simplify group-relative optimisation; training combines recorded production-harness prefixes with end-to-end rollouts, while evaluation retains the multi-agent harness.Source: §3.5.7 · pp. 50, 52
  • In the multi-agent harness, the single-agent-trained checkpoint improved five of six reported dimensions, with the largest gains in factuality and completeness.The comparison supports transfer beyond the collapsed trajectory but does not establish that single-agent optimisation is equivalent to training the multi-agent system.Source: §3.5.7 · p. 53, Figure 27 · p. 54

3.5.8 Exploration of Training Mechanisms

  • Entailed claims rose from 47% to 65%, while uncited claims fell from 13% to roughly 1%; contradicting claims had a negligible net change of -0.2 percentage points.Irrelevant claims declined by roughly 5 percentage points and plateaued around 14%; neither contradicting nor partially entailed claims trended upward, and entailment carried the highest-weighted reward.Source: §3.5.8 · pp. 53, 55, Figure 28 · p. 55
  • RL Stage 1 showed broad reward improvement across task families, including multilingual identity adherence, while Deep Research understandability improved and completeness remained noisier.Legal reasoning gains were sustained, drafting improved early then partly regressed, and transactional rewards plateaued after initial improvement.Source: §3.5.8 · p. 55, Figure 29 · p. 56

3.6 Training Compute

  • Thomson-1.0-Large used approximately 4.6 × 10^23 FLOP (8.8 × 10^4 B200 GPU-hours), versus 1.6 × 10^23 FLOP (3.5 × 10^4 GPU-hours) for Thomson-1.0-Small.The large-model estimate is approximately 2.2× the estimate under a uniform 30% utilisation assumption; both actual compute and hardware-provisioned peak remain below the 10^25 FLOP limit.Source: §3.6 · p. 58

4 Evaluation

The evaluation assesses Thomson’s capabilities and changes from its starting checkpoint across public and institution-specific benchmarks, professional tasks, safety, and general capabilities. It also examines complete-system effects, Fusion scaling, and the diagnostic use of a decomposed LLM judge.

4 Evaluation

  • Because continual learning starts from an already capable checkpoint, evaluation tracks changes against that baseline across targeted and untargeted capabilities to detect forgetting beyond target-domain benchmarks.The suite spans professional domains, agentic Deep Research, general-purpose preservation, expert preferences and quality, test-time scaling, and safety and alignment.Source: §4 · p. 59

4.1 Professional Domain Evaluation

Contract Understanding.

  • Thomson-1.0-Large raises the average across selected public legal benchmarks from 63.2 to 69.6, improving on its Qwen3.5-397B starting checkpoint on six of seven benchmarks.The reported gains include Harvey Legal Agent Benchmark (70.6 →85.7), ContractScrub (40.9 →56.3), Query Sufficiency (52.7 →57.9), and the MBE (86.8 →90.8); LEXam remains approximately unchanged (83.5).Source: Table 13 · p. 61, Contract Understanding · p. 61
  • Thomson-1.0-Small raises the average across these benchmarks from 54.6 for Qwen3.6-35B to 63.5, with particularly large gains on Harvey Legal Agent Benchmark and ContractScrub.The authors say these results show legal improvements distributed across multiple independently developed public evaluations.Source: Contract Understanding · p. 61, Table 13 · p. 61

4.1.2 Institution-Specific Evaluation Details

  • The institution maintains internal benchmarks to address public-test contamination and saturation and to assess high-stakes professional work that standardized external benchmarks do not capture.The 20 internal benchmarks contain more than 11,000 SME-curated items and cover document processing and retrieval-augmented generation, practitioner queries, professional deep research, and robustness; benchmark construction includes SME authoring, quality checks, task-matched scoring, versioning, and scheduled refresh.Source: §4.1.2 · p. 62, Robustness · p. 63

4.2.1 Task Description

  • The Deep Research benchmark evaluates reports in a fixed agent harness, using queries designed to require practitioners’ realistic, economically valuable questions and synthesis of authoritative sources.The set contains 53 legal, 48 tax, and 100 journalism queries, with scores reported separately by category; the legal and tax queries were written by experienced SMEs, while journalism queries were synthetic.Source: §4.2.1 · pp. 63–64

4.2.2 Research Evaluation

  • Deep Research reports are assessed across separate dimensions, including rubric-based completeness, citation factuality, and query relevance, with LLM-judge scores calibrated against human experts.Completeness uses query-specific required and helpful items rated as addressed, partially addressed, not addressed, or contradicted; factuality checks whether cited sources entail atomic claims and can propagate support from already verified claims.Source: §4.2.2 · pp. 64–65
  • The Deep Research overall score weights completeness, factuality, relevance, and coherence at roughly 40%, 35%, 20%, and 5%, respectively, excluding understandability and conciseness.Weights came from expert calibration on 72 examples from four models; coherence uses similarity-filtered claim pairs and an LLM judge, while conciseness scores were typically lower and experts criticized all tested models on that dimension.Source: §4.2.2 · p. 66

4.2.3 Results

  • Thomson’s largest Deep Research gains over Snowdon-1.0-Large were factuality and completeness: legal 0.71→0.83 and 0.71→0.87; tax 0.67→0.80 and 0.56→0.75.Journalism factuality improved from 0.64 to 0.82 while completeness remained comparable (0.72 vs. 0.69); the small model showed a similar improvement pattern. Gains were driven mainly by report coverage and citation-grounded factual support, not relevance or coherence, which were already high across large models.Source: §4.2.3 · pp. 66–68, Table 14 · p. 67

4.3.1 General Capability Suites

  • General capability evaluations span nine categories and assess whether Continual Learning preserves broad capabilities inherited from the starting checkpoint.The suite covers reasoning, mathematics, multilingualism, factuality, instruction following, writing, long context, coding, and general-agent tasks.Source: §4.3.1 · p. 68

4.3.2 Capability Preservation Results

  • Thomson-1.0-Large improved over Qwen3.5-397B on Humanity’s Last Exam (24.8→28.5), GDPval (89.4→93.5), and WritingBench (77.9→80.3), while remaining close on most evaluated benchmarks.The clearest decline was Terminal-Bench 2.1 (54.0→48.3), with smaller drops on AIME 2026 and IFEval; losses were concentrated in a minority of evaluations, and the small model showed a similar retention-or-improvement profile.Source: Table 15 · p. 69, §4.3.2 · p. 69

4.4 Expert Preference & Quality Evaluation (System comparison)

  • Benchmark evaluations alone do not capture practical usefulness, nuanced response quality, or the contribution of retrieval tools and proprietary data in real deployments.The expert evaluation therefore compares complete systems and assesses qualities such as legal soundness, completeness, communication quality, and practical utility, using qualified legal professionals.Source: §4.4 · p. 70

4.4.1 Study Design

  • The pairwise preference study compared Thomson-1.0-Large with leading frontier systems using 3,035 user tasks—2,009 legal and 1,026 general-domain—and included 35 attorney editors.Thomson had legal-specific and Reuters news-search tools; competitors used available web-search capabilities. Thomson-1.0-Large could not be configured with web search for this study.Source: §4.4.1 · pp. 70–71

4.4.2 Results

  • Experts preferred Thomson-1.0-Large in 53–62% of all-query comparisons against each competitor, while competitors were preferred in 29–33%.On legal queries, Thomson’s win rates were 54–64%; it also exceeded competitors by 5.1–9.5 points on completeness and 5.7–9.1 points on usefulness. Against News Tools Only, the full system won 57% of legal queries but won 32% and lost 39% on general queries.Source: §4.4.2 · pp. 71–72, Table 17 · p. 72, Figure 32 · p. 73

4.5 Test-Time Scaling

  • Fusion recovered roughly 40% of available test-time scaling headroom, compared with 15% for reward-model selection, as weak reward-model correlation made candidate selection near-random.Process Reward Model-guided search also collapsed candidate diversity; Fusion instead synthesised an answer across candidates. The main setup used N = 4: three candidates followed by one synthesis call.Source: §4.5 · p. 73
  • Fusion improves several benchmark families by +1.42 to +8.77 over single-pass, but gains peak at N = 4 and do not compound at N = 8.The authors fix N = 4 as the operating point; doubling the candidate pool leaves four of five tested families flat or lower.Source: §4.5 · p. 73, Figure 33 · p. 74, Table 19 · p. 74

4.5.1 Post-Training Beats Inference Scaling on the Base Model

  • Thomson’s post-training outperforms scaling the base model at inference: single-pass Thomson scores 71.9 macro versus 71.2 for OOB Fusion N=4, at roughly a quarter of the inference cost.Fusion adds 2.0pp to Thomson, while the Factuality gap is only 0.1pp.Source: §4.5.1 · pp. 74–75, Figure 34 · p. 74

4.6 Safety and Alignment

  • The frozen Promptfoo safety suite uses 3,867 base-prompt families and 7,300 static attack variants, yielding 11,167 cases per target for like-for-like testing.Pass rates are suite-specific, not directly comparable with source-paper scores; target-adaptive jailbreaks are excluded because they cannot be frozen identically before each response.Source: §4.6 · p. 75

4.6.1 Results

  • The safety registry covers five targets in reasoning and non-reasoning modes, totaling 10 profiles and 70 target-category runs, with complete response and grade coverage.Each profile completed all 11,167 cases.Source: §4.6.1 · p. 75

4.6.2 Target-level Results

  • Reasoning deltas compare complete inference profiles, because reasoning runs use approved mode-specific decoding settings and extra reasoning-token allowances.The frozen prompts and grader are identical across modes.Source: §4.6.2 · p. 75

Results by Risk Category

  • Thomson achieves 90.17% non-reasoning and 93.35% reasoning micro pass rates, ranking third in both modes while remaining below Qwen3.5-397B.Its +3.18-point profile difference is the largest positive delta among five targets, with higher pass rates in all seven categories; the evaluation is not a general safety certification.Source: Table 21 · p. 76, Results by Risk Category · p. 77, Table 22 · p. 77

4.7 LLM-as-a-Judge: Decomposed Criteria-Based Evaluation (DeCE)

  • DeCE decomposes judging into instance-specific precision and recall criteria derived from each gold answer, separating required information from helpful material.Recall checks required information only, while precision checks claims against the full gold answer; independent workflows use the same judge model.Source: §4.7 · pp. 77–78
  • DeCE closes most of the gap between naive LLM judging and expert agreement without hand-authored rubrics, and its decomposed scores reveal actionable model and task failure patterns.An expert audit found 11.95% of extracted criteria required revision and 54.5% of queries needed no revisions; the method also surfaced jurisdiction- and query-type-specific failure clusters.Source: §4.7 · pp. 78–79, Table 24 · p. 79

5 Infrastructure

This section describes a portable, multi-cloud training and self-hosted inference stack, covering platform foundations for isolated evaluations, artifact access, data generation, and distributed reinforcement learning. It details caching, decoupled training and generation, isolated multi-region inference, and serving optimizations, including reported throughput and cache-reuse gains and support for workload isolation, data-residency controls, and adaptable inference.

5.1 Infrastructure Sovereignty

  • Thomson’s infrastructure uses multi-cloud, multi-cluster training and in-house orchestration and inference to increase optionality, autonomy, and operational flexibility.The authors frame this approach as supporting institutions seeking greater ownership of their AI stack amid constrained GPU access, vendor lock-in, and high provider costs.Source: §5.1 · p. 81

5.1.1 Distributed Training Infrastructure

  • Because contiguous, affordable accelerator clusters are scarce and shift over time, Thomson’s training infrastructure treats the ability to relocate as a requirement.The team used multiple cloud providers and rapidly migrated between regions; portability does not remove accelerator dependence, but makes research continuity depend on re-establishing the stack wherever capacity is available.Source: §5.1.1 · p. 81

5.1.2 Model Orchestration and Inference

  • Since April 2026, the self-hosted system served over 40M requests and processed 370B tokens at over 99.9% success; GLM-5.2 inference cost 19.7% less than third-party providers.Across the broader organisation, cumulative use reached 146M requests and nearly 700B tokens during that period.Source: §5.1.2 · p. 81

5.2.1 Cluster Topology

  • A three-cluster topology separates the primary control plane, air-gapped agentic evaluation, and GPU-bound work, keeping stateful services stable across GPU-provider changes.The evaluation sandbox has no network egress or route to internal systems, while the GPU cluster is reserved for accelerator workloads.Source: §5.2.1 · p. 82

5.2.2 Job Orchestration

  • A modified ClearML fork and internal interface abstract execution targets, so relocating GPU clusters changes available targets without changing how jobs are written.The system handles job submission, scheduling, and experiment tracking across heterogeneous environments.Source: §5.2.2 · p. 83

5.2.3 Data Access and Provenance

  • 16.13 PB of GPU-cluster read demand corresponded to 278.7 TB actually egressed (a 57.9 factor), with over USD 1.4M in estimated avoided egress costs.The estimate covers February–August 2026; the baseline assumes every worker of every job pulls inputs directly from object storage.Source: Figure 37 · p. 83, §5.2.3 · p. 84
  • Stable artifact identifiers resolve to local replicas, while versioned DAGs make synthetic-data transformations traceable and tool calling consistent across model requests.DataHub and SeaweedFS support artifact resolution and provenance; the async-first tool layer standardizes schemas, authentication, and state management, and synthetic-data steps cover generation, filtering, and scoring.Source: §5.2.3 · pp. 83–84, §5.2.4 · p. 84, §5.2.5 · p. 85

5.3.1 Training Stack

  • Thomson’s RL setup uses separate generation and training pools, custom NeMo-RL extensions, and HTTP environments for legal research and sandboxed shell use.To mitigate failed or throttled reward calls, cohort scoring imputes missing rewards with the cohort mean, discounts advantages based on imputation, and excludes rollouts whose rewards are entirely imputed or failed.Source: §5.3 · p. 85, §5.3.1 · pp. 85, 87, §5.3.2 · p. 87

5.3.2 RL Environments

  • Across 329 389 lookups, the three-level document cache answered 98.2%, while its cached corpus grew by 464 documents during the 125-step run.The worker-, cluster-, and cross-run tiers share persistent records populated by both citation evaluation and generation retrieval; the cache answered misses from backend or live fetches and stored successful results for reuse.Source: Figure 39 · p. 88, §5.3.2 · p. 88

5.3.3 Training Performances

  • Non-colocation reduced total step time by 13.1% and increased Generation Worker Group tokens/sec by 61.7%, despite using half the generation parallelism.The gain was concentrated in generation time (-32.3%), while training and logprob computation changed by +0.4% and -3.4%, respectively; the authors attribute the benefit to avoiding GPU memory and compute contention.Source: §5.3.3 · pp. 88–89, Figure 40 · p. 89

5.4.1 Architecture

  • The platform separates routing and inference across 11 clusters in 4 geographies and 2 cloud providers, enabling fault isolation, failover, and data-residency controls.A dedicated research environment supports testing architectures, quantisation, speculative decoding, and representative loads before rollout; agentic tool calling is also supported through an API mirroring OpenAI’s Responses API, alongside OCR for upload handling.Source: §5.4.1 · pp. 89–90

5.4.2 Inference Server Customisations

  • Custom vLLM extensions support client-defined thinking budgets and reasoning parsers, while faster-loading libraries reduce model initialisation to approximately five minutes or less.Dynamic post-training quantisation is normally FP8; the language-modelling head and MoE gating and routing layers remain unquantised, and evaluations found task performance close to full precision.Source: §5.4.2 · p. 91

6 Conclusion & Future Work

The report argues that Continual Learning can make broad SovereignAI implementation feasible on modest budgets, with Thomson showing targeted gains alongside preserved or improved performance elsewhere. It frames sovereignty as progress across a spectrum, while identifying further work to improve training, alignment, and repeated model development.

6.1 Conclusion

  • Continual Learning makes broad SovereignAI customization feasible on modest budgets, but sovereignty remains a spectrum and accelerator-hardware dependence persists.The report describes model and data sovereignty as substantially achievable, governance and value alignment as improvable, and infrastructure and economic control as largely attainable on open-source foundations.Source: §6.1 · p. 92
  • Thomson shows a repeatable π-shaped profile, with pronounced gains in targeted domains and preserved or often improved performance in untargeted domains.The authors report that modules of the development pipeline have been applied to multiple base models across the Thomson family.Source: §6.1 · p. 92
  • The report presents Thomson as evidence that careful engineering and assembly of open-source tools can enable infrastructure sovereignty.Thomson was developed on a comparatively modest budget and evaluated through public evaluations, adversarial safety testing, blind human studies, and production requirements.Source: §6.1 · p. 92

6.2 Future Work

  • Future mid-training work should test whether expanding the data pool and coordinating mid- and post-training improves results.The current budget used 200B tokens selected from approximately 19T candidates; proposed directions include use-case-focused curation, explicit multi-document reasoning training, and comparing full-weight training with parameter-efficient training at fixed compute.Source: §6.2 · p. 92
  • The authors see online RL as a low-forgetting component of continual learning and expect future gains to depend on more sophisticated RL environments.They propose closer coordination with mid-training, careful reward design, heavier use of LLM judges, and eventually replacing external judges with previously trained Thomson generations.Source: §6.2 · p. 93
  • Future work should measure and strengthen coding skills, the one area with mild forgetting relative to the starting open-weight models.The authors propose using coding and software-engineering datasets and argue the drop can likely be alleviated without algorithmic breakthroughs; they also identify open-weight model choice as a way to address absolute coding gaps.Source: §6.2 · p. 93
  • Constitutional alignment remains unsolved, and future work could add continuous adversarial red-teaming and automate contentious-issue design.The authors describe value realignment and Constitutional RL as evidence that progress is tractable, while noting that current contentious issues were written by human experts.Source: §6.2 · p. 93
  • A further experiment should test whether successive Continual Learning generations compound capability and institutional fit or accumulate drift that requires a fresh checkpoint.Source: §6.2 · p. 93

Core Subject Matter Experts

The authors thank the core subject-matter experts who worked with them and taught them about their domains as they incorporated their expertise into Thomson.

Core Subject Matter Experts

  • Core subject-matter experts worked alongside the team and taught them about their domains to help embed their expertise into Thomson.The authors say the team was too large to name individually.Source: Core Subject Matter Experts · p. 94
  • The authors also thank project coordinators, Thomson Reuters colleagues, former Foundational Research colleagues, partners at Imperial College London, DatologyAI, and Lambda, and three leaders for their support.The leaders named are Joel Hron, Steve Hasker, and Alexander Kardos-Nyheim.Source: Core Subject Matter Experts · p. 94

DatologyAI · A Working with Domain Experts

The authors thank DatologyAI’s researchers, engineers, and leadership for collaborating on Thomson’s mid-training.

DatologyAI

  • DatologyAI researchers, engineers, and leadership collaborated with the authors on Thomson’s mid-training.Source: DatologyAI · p. 94

A.1 Enhancing Inter-Annotator Agreement in Expert Annotation Studies · A.2 Experts as Collaborators

The sections outline ways to improve expert agreement and gold-label reliability without simply increasing expert time, then describe how close expert collaboration contributed to model training.

A.1 Enhancing Inter-Annotator Agreement in Expert Annotation Studies

  • Agreement depends on the workflow’s ordering: experts annotate independently before discussion, then reconcile targeted disputes after full-scale annotation.Figure 42 presents the recommended pipeline for producing high-agreement expert gold labels.Source: §A.1 · p. 107, Figure 42 · p. 107
  • Ruan et al. [168] found that 77% of human-evaluation guidelines had at least one of eight recurring vulnerabilities, supporting systematic screening before annotation.The vulnerabilities include ambiguous definitions, unclear rating scales, unaddressed edge cases, ethical oversights, and unconscious bias.Source: §A.1 · p. 107
  • Yu describes an LLM-assisted draft–revise–detect process that screens guidelines for those vulnerabilities before calibration, while retaining human review for issues requiring judgment.The process drafts with an LLM, has a human revise against the vulnerability categories, then uses a second LLM pass to flag remaining issues.Source: §A.1 · pp. 107–108
  • Staff annotators by domain expertise and narrow sub-specialty, since mismatches can increase calibration burden or cause unresolvable disagreement.Plan for scarce specialists before scoping the study rather than treating specialty fit as a best-effort filter once annotation begins.Source: §A.1 · p. 108
  • Run a calibration round of independent annotation, group reconciliation, and guideline updates before releasing the full sample set.The guidance identifies skipping calibration as the most common cause of low inter-annotator agreement.Source: §A.1 · p. 108
  • Use a diverse, edge-case-heavy calibration batch and designate a neutral QC expert in advance to resolve two-annotator deadlocks.Escalate deadlocks rather than leaving them unresolved or defaulting to majority vote.Source: §A.1 · pp. 107–108
  • Measure IAA, reconcile disputed samples, use the reconciled set as gold, and feed disagreement patterns back into guidelines and LLM-judge criteria.The guidance cautions against treating majority vote or a single annotator’s labels as gold.Source: §A.1 · p. 108

A.2 Experts as Collaborators

  • Close collaboration with a consistent expert team contributed design feedback and new training tasks; expert ideas improved datasets, including two published with experts as co-authors [117, 120].The authors believe this integration significantly increases the real-world impact of training.Source: §A.2 · p. 108

B Evaluation Benchmark Data Cards … B.2 Legal Information Retrieval

The appendix describes benchmark formats, metrics, provenance, evaluation-set sizes, and representative items; its LegalBench card covers a broad collection of legal-reasoning tasks with varied formats and metrics.

LegalBench

  • LegalBench has 89,485 items across 162 subtasks, uses subtask-specific formats and official-implementation metrics, and is an external benchmark from Guha et al. (2023) [54].The appendix reports random sampling for benchmarks smaller than their original datasets unless otherwise stated.Source: B Evaluation Benchmark Data Cards · p. 108, LegalBench · p. 109
  • LegalBench was collaboratively curated by 40 legal and AI contributors, and its 162 tasks span five legal-reasoning categories.The categories are Issue, Rule, Conclusion, Interpretation, and Rhetorical.Source: LegalBench · p. 109
  • Issue tasks identify legal issues in fact patterns (17 subtasks, 11,599 samples), while Rule tasks apply rules and determine applicability (5, 9,809).Source: LegalBench · p. 109
  • Conclusion tasks draw conclusions from facts and rules (12 subtasks, 2,227 samples), and Rhetorical tasks address argumentation and persuasion (10, 6,239).Source: LegalBench · p. 109
  • Interpretation is the largest reasoning category, with 118 subtasks and 59,611 samples focused on statutes, contracts, and case language.Source: LegalBench · p. 109
  • LegalBench includes major families such as CUAD (38 subtasks), MAUD (34), Learned Hands (16), ContractNLI (14), supply-chain disclosure (10), and OPP-115 (9).Items range from identifying contractual provisions and interpreting statutes to recognizing reasoning patterns and classifying judicial-opinion sections.Source: LegalBench · p. 109

COLIEE Task 1 – Case Law Retrieval … B.3 Legal Reasoning

The section describes three legal evaluations: retrieving cited cases from redacted case content, synthesizing answers across legal documents, and judging substantive relevance of case headnotes. Their formats, metrics, and item counts are specified, but no model results are supplied.

COLIEE Task 1 – Case Law Retrieval

  • COLIEE Task 1 is text generation for case identification, evaluated by Micro F1 on 499 items from the COLIEE Competition.Source: COLIEE Task 1 – Case Law Retrieval · p. 109
  • COLIEE Task 1 asks the model to identify one or two likely cited cases from a Federal Court of Canada query, with references redacted.Selection depends on legal principles, subject matter, fact patterns, reasoning, and precedential value; the task uses chain-of-thought prompting.Source: COLIEE Task 1 – Case Law Retrieval · p. 109

AALP Quality

  • AALP Quality is a free-response task evaluated with ROUGE-1 F1 and ROUGE-L F1 on 220 SME-designed items.Source: AALP Quality · p. 109
  • AALP Quality tests retrieval and synthesis across practice notes, standard documents, and procedural guides to produce accurate, supported answers with citations.Responses are compared with gold reference answers using ROUGE.Source: AALP Quality · p. 109

Best Headnote

  • Best Headnote is a four-way multiple-choice classification task, measured by Macro F1 on 1,072 SME-designed items.Source: Best Headnote · p. 109
  • Best Headnote classifies a headnote’s relevance to a legal search query across four grades, distinguishing substantive legal relevance from surface term overlap.The grades range from directly on point to fully unresponsive.Source: Best Headnote · p. 109

COLIEE Task 2 – Legal Case Entailment … Parentheticals

The section describes seven legal evaluation tasks spanning precedent entailment, holding identification, opinion structure, issue-spotting, support judgments, bar-exam reasoning and citation relationships.

COLIEE Task 2 – Legal Case Entailment

  • COLIEE Task 2 tests paragraph-level precedent entailment for an unseen case, using chain-of-thought prompting and Micro F1 evaluation.It asks which paragraphs from a noticed case support a given judicial decision.Source: COLIEE Task 2 – Legal Case Entailment · p. 110

CaseHOLD

  • CaseHOLD tests whether a model can identify a cited case’s relevant holding among five options, including four genuine holdings from other cases.The evaluation uses 1,000 items sampled from a pool of more than 53,000 and measures accuracy.Source: CaseHOLD · p. 110

Function of Decision Section

  • Function of Decision Section evaluates classification of U.S. appellate-opinion paragraphs into seven categories, from Facts and Rule to Analysis, Conclusion and Decree [54].The 367-item task measures accuracy and probes basic legal comprehension.Source: Function of Decision Section · p. 110

Learned Hands

  • Learned Hands assesses legal issue-spotting in lay narratives, classifying posts across 16 legal categories using accuracy averaged across categories [54].Its 11,109 examples were labelled by law students and lawyers; the task supports triage to legal services.Source: Learned Hands · p. 110

Legal Support

  • Legal Support tests which of two candidate conclusions more directly supports a claim in a legal passage, despite deliberately noisy labels [101].It reports balanced accuracy on an abbreviated 1,000-item set or the full 20,034-item set.Source: Legal Support · p. 110

MBE Bar Exam

  • The MBE Bar Exam task uses chain-of-thought prompting to test legal reasoning across multiple-choice questions covering core legal subjects.The described examination has 200 questions and assesses reasoning and analysis rather than recall.Source: MBE Bar Exam · p. 111

Parentheticals

  • Parentheticals classifies cited passages as directly supporting, indirectly supporting or contradicting an opinion passage, using citation signals as ground truth.The near-balanced custom task has 308 items; GPT-4 reached 0.41 macro F1 versus a random baseline of about 0.3.Source: Parentheticals · p. 111

ReClor … B.5 Document Processing and Retrieval-Augmented Generation

The benchmark suite spans legal reasoning, professional legal work, and document classification, using multiple-choice, open-ended, and multi-label tasks. The supplied passages describe task formats and evaluation setups, not model performance results.

ReClor

  • ReClor evaluates logical reasoning on 1,000 four-option graduate-admission questions with accuracy, using chain-of-thought prompting [103].Questions test flawed reasoning, parallel argument structures, and logical validity; the passage notes that state-of-the-art models still struggle on the set.Source: ReClor · p. 111

SuperGPQA (Law)

  • SuperGPQA (Law) tests graduate-level legal recall and reasoning on 656 ten-option questions, with accuracy and chain-of-thought prompting [104].The law subset comes from a 285-discipline benchmark, and the ten choices make chance performance low.Source: SuperGPQA (Law) · p. 111

LEXam – MCQ (4-choice)

  • LEXam’s four-choice Swiss law exam subset contains 1,655 English and German questions, evaluated by accuracy with step-by-step expert-style reasoning [105].The set includes 1,515 Swiss-jurisdiction and 140 international items; overall, per-language, and Swiss-only accuracy are reported.Source: LEXam – MCQ (4-choice) · p. 112

PRBench Legal Hard

  • PRBench Legal Hard evaluates open-ended professional legal reasoning on 250 difficult tasks using a rubric-based mean clipped score and o4-mini judge [106].Tasks cover 12 topics; expert-authored criteria award or deduct weighted points, with each sample score floored at zero.Source: PRBench Legal Hard · p. 112

EUR-Lex

  • EUR-Lex measures multi-label classification across more than 100 categories on 500 European legislative documents using micro F1 [108].The categories cover policy areas including economics, health, environment, transport, and agriculture.Source: EUR-Lex · p. 112

LEDGAR

  • LEDGAR tests multi-label classification of 10,214 contract provisions across 780 categories using micro F1 [108].Each provision can receive up to eight labels, and the label distribution has a long tail of rare categories.Source: LEDGAR · p. 112

SCOTUS

  • SCOTUS classifies 1,000 Supreme Court opinions across 13 legal-issue categories using multi-label micro F1 [108].The task requires distinguishing central issues across adjacent areas of constitutional and statutory law.Source: SCOTUS · p. 113

Headnote Type

  • Headnote Type classifies 880 legal headnotes into one of 11 practice areas using accuracy, including genuine boundary cases.The task assesses identifying the most appropriate area, supporting legal research, document organisation, and issue-spotting.Source: Headnote Type · p. 113

Legal RAG · Document Review · B.6 Legal Summarisation

The section describes two legal-document evaluation tasks: attorney-style question answering over primary-law sources and query answering over a supplied document set. Their evaluations use reference-based LLM judging, with the tasks differing in scope, dataset size, and workflow.

Legal RAG

  • Legal RAG is a 224-item custom task designed with SMEs, scored on GPA, precision, recall, and F2, and judged by Claude Sonnet 4.6.Source: Legal RAG · p. 113
  • Legal RAG requires accurate, relevant, clear attorney-style answers grounded in supplied cases, statutes, or regulations, with paragraph-number citations and explicit tokens for unsupported information.Source: Legal RAG · p. 113

Document Review

  • The document-review task evaluates responses to 392 queries across 7,324 documents using a reference-based Claude Sonnet 4.5 judge.Source: Document Review · p. 113
  • Document review uses a two-stage workflow: selecting relevant documents and chunks from the supplied set, then generating an answer to the user query.Source: Document Review · p. 113
  • Document-review answers are compared with gold answers by a reference-based LLM judge.Source: Document Review · p. 113

BillSum US

The BillSum US section evaluates concise summaries of U.S. Congressional bills using reference-based LLM judging across five quality dimensions, including hallucination detection.

BillSum US

  • The external BillSum (US test split) [111] evaluation covers 3,269 bill-summary items.The task is to capture each bill’s key provisions, purposes, and implications.Source: BillSum US · p. 114
  • Claude Sonnet 4.6 judges summaries for accuracy, completeness, clarity, conciseness, and hallucination using reference-based LLM-as-judge evaluation.Source: BillSum US · p. 114

BillSum CA … XSumFaith

This benchmark suite spans legal summarisation, extraction, interpretation, drafting and reasoning, alongside tax Q&A, factual recall and faithfulness. Tasks use external and SME-designed data, with scoring tailored to each task’s format and evidence requirements.

BillSum CA; CourtWire; Material Facts

  • BillSum CA tests judged summaries of California bills, while CourtWire and Material Facts assess one-sentence complaint summaries and extraction of up to five facts a court relied on.BillSum CA uses five judged dimensions on 1,237 items; CourtWire and Material Facts are internal SME-designed tasks with 500 and 495 items, respectively.Source: BillSum CA · p. 114, CourtWire · p. 114, Material Facts · p. 114

B.7 Contract Understanding

  • Contract Understanding combines lawyer-annotated clause and deal-point tasks, using macro F1 for pooled binary clause classification and accuracy for fixed-answer merger-agreement questions.The supplied descriptions cover CUAD’s 38 pooled subtasks and MAUD’s 34 pooled subtasks.Source: §B.7 · pp. 114–115

B.7 Contract Understanding; Privacy Policy Entailment; Insurance Policy Interpretation

  • Privacy evaluations distinguish binary clause-to-practice entailment from pooled privacy-policy category questions, while Insurance Policy Interpretation uses yes, no, or ambiguous labels.Privacy Policy Entailment uses accuracy for a single task; OPP-115 pools nine categories and scores macro F1. Insurance labels reflect crowdsourced interpretation, including high-disagreement cases marked ambiguous.Source: §B.7 · p. 115, Privacy Policy Entailment · p. 115, Insurance Policy Interpretation · p. 115

ContractScrub

  • ContractScrub tests open-ended contract error detection with JSON term-location tuples, scored against lawyer annotations using recall as its primary metric.Its nine categories include defined-term extraction and eight error types; repeated occurrences count separately, and the agreements are re-annotated by nine experienced lawyers.Source: ContractScrub · p. 116

Diverse Queries – Documents (Groundedness); Diverse Queries – General Legal; Diverse Queries – Drafting

Diverse Queries – Transactional; Diverse Queries – Instruction Following; Deep Research

Tax Q&A

  • Tax Q&A tests cited answers against expert-vetted gold responses for correctness, coverage, consistency, relevance and source groundedness.Five judges score the independent axes; groundedness checks whether atomic answer statements are supported by the supplied sources.Source: Tax Q&A · p. 117

SimpleQA Verified; FACTS Grounding; FaithEval; CoGenSumm; XSumFaith

  • Factuality benchmarks cover parametric recall, long-form grounding, abstention or conflict detection, and summary consistency, with XSumFaith testing highly abstractive summaries.SimpleQA Verified uses F1 with explicit abstention; FACTS filters for task eligibility before grounding checks, FaithEval tests unanswerable and inconsistent contexts, and CoGenSumm and XSumFaith use binary consistency judgments.Source: SimpleQA Verified · p. 118, FACTS Grounding · p. 118, FaithEval · p. 118

B.12 Robustness · Principal Hierarchy – Execution (Legal) · Query Sufficiency

The block describes two legal robustness evaluations: whether models resist drafting arguments from overruled cases under direct instruction, and whether they identify missing facts in under-specified queries.

Principal Hierarchy – Execution (Legal)

  • The legal execution evaluation contains 1,616 items and measures action correct rate using an LLM judge.It is a custom task on refusal and redirection when instructed to draft with an overruled case [119].Source: Principal Hierarchy – Execution (Legal) · p. 119
  • Using the legal advisory task’s 401 overruled-case scenarios, this evaluation raises compliance pressure by asking models to draft directly rather than provide advice.The scenarios use free-response execution prompts.Source: Principal Hierarchy – Execution (Legal) · p. 119
  • The judge checks whether the model identifies the case as overruled and refuses the requested passage or explains why the case cannot be used.Producing the improper argument does not meet the task’s stated criterion.Source: Principal Hierarchy – Execution (Legal) · p. 119

Query Sufficiency

  • Query Sufficiency uses 144 SME-designed items and recall-weighted element identification F2, judged by Qwen3-235B-A22B-Instruct-2507.The free-response fact patterns span criminal law, tort, contracts, employment, commercial, real property, and defamation; 64 are moderate and 80 difficult.Source: Query Sufficiency · p. 119
  • Every Query Sufficiency item is under-specified, with one to six missing ground-truth elements such as jurisdiction or plaintiff status and no sufficient controls.The judge credits naming missing elements, asking about them, or conditioning an answer on them, while also reporting over-flagged gaps.Source: Query Sufficiency · p. 119
  • Query Sufficiency’s F2 uses β = 2, penalizing missed absent facts more than mild over-flagging; unparseable judge outputs are excluded rather than scored zero.Source: Query Sufficiency · p. 119

B.13 Vals AI Benchmarks … C Fusion Prompts

The section describes Vals AI benchmarks for legal research, tax, and financial-analysis tasks, then specifies fusion prompts that either expose candidate reasoning or present answers alone. It details the scoring, test-set scope, and candidate formatting used in these evaluations and prompts.

B.13 Vals AI Benchmarks — Legal Research Bench

  • Legal Research Bench evaluates agentic US legal research using lawyer-authored questions and a strict all-pass primary metric on 208 held-out test items.Questions require supported answers across eight practice areas; weighted partial credit is secondary.Source: Legal Research Bench · p. 119

B.13 Vals AI Benchmarks — TaxEval

  • TaxEval scores expert-checked tax answers for correctness and stepwise reasoning across 1,223 held-out test questions.The questions cover balanced categories including numerical reasoning, compliance, and current tax-code updates.Source: TaxEval · p. 120

B.13 Vals AI Benchmarks — Finance Agent v2

  • Finance Agent v2 tests financial-analysis work over company filings, using dealbreaker-gated weighted partial credit as its primary metric on 450 held-out items [176].Questions require synthesis across filings and may hinge on details in footnotes, MD&A caveats, or accounting policies; all-pass is secondary.Source: Finance Agent v2 · p. 120

C Fusion Prompts

  • Fusion uses direct synthesis to produce one answer from shuffled candidates, with variants that show or omit candidate reasoning.Both prompt variants include the full conversation and instruct the fusor not to refer to candidates or copy their formatting constraints.Source: C Fusion Prompts · p. 120

C.1 With Candidate Reasoning

  • When candidate reasoning is shown, each thinking block is labelled as reasoning and separated from the candidate answer so the fusor treats it as evidence.The prompt names this structure explicitly.Source: §C.1 · p. 121

C.2 Without Candidate Reasoning

  • For legal-reasoning tasks, the fusor receives candidate answers without their reasoning traces, using an otherwise identical prompt.The reasoning-block instructions are replaced with a statement identifying each Generated Text as a candidate answer.Source: §C.2 · p. 121

C.3 Candidate Formatting

  • Candidates appear under shuffled Generation N headers, and reasoning tags are relabelled only when both opening and closing tags are present.The shuffle order is recorded for each example; candidates lacking both tags pass through unchanged.Source: §C.3 · p. 121

Related papers

Loading 2608.27147v1…