Enhancing Legal Reasoning in Pre-trained Language Models via Knowledge Graph-Guided Multi-Task Pre-training

1 Introduction

• Existing legal domain-adaptive pre-training approaches do not explicitly incorporate formal semantic relationships and hierarchical constraints during pre-training.Lawformer, SAILER, and Legal-BERT rely on implicit structural modeling through long-range attention, structural headers, or domain-specific masked language modeling.Source: §1 · p. 1
• The approach combines NSP for document coherence, KG-guided clustering for relational structure, and KG-aware paragraph classification for task-aligned reasoning.It is presented as a multi-task approach for semi-structured regulatory text, validated on the German Road Traffic Regulations (StVO).Source: §1 · p. 1
• The proposed approach integrates knowledge-graph structures into language-model latent representations rather than relying on text concatenation.It is framed as a unified pre-training approach for semi-structured regulatory text.Source: §1 · p. 1
• TransE-derived legal paragraph embeddings serve as soft pre-training targets, transferring relational legal knowledge without hard classification constraints.This allows the model to learn a continuous range of legal concepts.Source: §1 · pp. 1–2
• A differentiable gated KG-aware attention mechanism retrieves from paragraph embeddings in real time and weights external legal context against internal representations.Unlike traditional retrieval-augmented approaches, it does not require discrete external database lookups during inference.Source: §1 · p. 2
• The synthetic German traffic-law QA dataset contains 12,554 examples with paragraph-level citation grounding and strict paragraph-level splits to prevent data leakage.It is presented as a benchmark for citation-accurate legal reasoning.Source: §1 · p. 2

2 Related Work

• Lawformer, SAILER, and Legal-BERT use long-range dependencies, structure-aware retrieval, or legal-domain pre-training, but do not explicitly incorporate structured relational knowledge during pre-training.LexGLUE standardizes evaluation across seven English legal tasks, while LegalBench provides English legal QA but lacks German regulatory coverage and paragraph citation sourcing.Source: §2 · p. 2
• KG-BERT treats triples as text sequences, while GraphGPT aligns graph patterns with natural language through instruction tuning and Chain-of-Thought distillation.The related-work discussion also describes arguments for ontology-enriched representations because standard embeddings can overlook schematic information and logical axioms.Source: §2 · p. 2
• TransE was selected for CLU soft targets after preliminary comparisons found it more robust downstream than DistMult, ComplEx, and RotatE.The paper motivates TransE by its efficiency in preserving hierarchical topology as soft targets and avoiding computational costs associated with Transformer-based graph encoders.Source: §2 · p. 2, Table 1 · p. 2
• The paper adapts knowledge distillation to transfer KG topology into soft pre-training targets and combines text modeling with graph understanding as synergistic objectives.It presents this as extending beyond traditional text augmentation toward embeddings aligned with legal-graph hierarchies.Source: §2 · p. 2

3 Methodology

• The architecture joins a KG pipeline that maps parsed StVO relations into TransE paragraph embeddings with an LM pipeline that receives KG-guided pre-training before evaluation.This design aims to inject regulatory structure into model representations.Source: §3 · pp. 2–3

3.1 Problem Formulation

• The task learns to judge whether an answer is legally correct given a question and its source paragraph, using NSP, CLU, and CLS objectives.The learned function maps a (question, answer, paragraph) triple to a binary judgment.Source: §3.1 · p. 3

3.2 TransE Paragraph Embeddings Generation

• TransE embeddings of paragraph nodes in a 554-node, 945-edge StVO graph serve as 500-dimensional soft clustering targets that preserve relational structure.The graph represents statutory categories, paragraphs, sub-paragraphs, sentences, tables, sections, and signs across five relation types.Source: §3.2 · p. 3

3.3 Hierarchical Document Encoding

• A structure-aware segmenter splits legal text at statutory markers and sentence boundaries, then a shared segment encoder and document encoder produce a max-pooled document representation.The two-level design is intended to preserve intra-document context while remaining computationally efficient for long statutes.Source: §3.3 · p. 3

3.4 Multi-Task Pre-Training Objectives

• NSP, KG-guided clustering, and KG-aware paragraph classification jointly train legal coherence and relational representations, with paragraph-level KG context used during training without graph traversal at inference.Clustering pulls a segment representation toward its source paragraph embedding and separates it from unrelated paragraphs; gated fusion combines retrieved KG context with the input representation.Source: §3.4 · pp. 3–4

4 Experimental Settings

• The research questions examine cross-language transfer, generalization to foreign driving regulations, and transfer to law-related tasks beyond traffic.The questions also ask how domain-specific linguistic features affect cross-language transfer and how results vary with regulatory overlap.Source: §4 · p. 4

4.1 Datasets

• The synthetic StVO benchmark contains 12,554 balanced QA pairs; paragraph-level splitting yields 10,245 train and 2,309 test pairs with zero paragraph overlap.Only pairs passing the quality threshold Q ≥ 0.6 are retained, and 17% of paragraph identifiers are reserved for testing.Source: §4.1 · p. 5
• Out-of-domain evaluation covers Irish and Austrian driving questions and seven English legal NLP tasks in LexGLUE.After image-dependent items are excluded, the Irish and Austrian benchmarks contain 2,500 and 3,852 test samples, respectively.Source: §4.1 · p. 5

4.2 Models and Training

• Experiments use nine Transformer encoders across 324 configurations, focusing on encoder-only binary judgments because bidirectional encoders suit the task; decoder-only extensions are future work.The configurations vary the weights of the three objectives under a unit-sum constraint.Source: §4.2 · pp. 5–6

4.3 Evaluation Metrics

• Evaluation reports Macro-F1, Micro-F1, precision, recall, balanced accuracy, and Macro-F1 gain over each base model.Balanced accuracy is included to account for potential class imbalance in the synthesized data.Source: §4.3 · p. 6

5.1 Impact of KG-aware Pre-Training on StVO Legal Reasoning

• 5/5 legal-domain encoders exceeded 0.8 m-F1 after pre-training; bert-base-echr reached 0.864, while bert-base-eurlex gained +0.524 from a 0.338 base.The reported gains are largely decoupled from base performance; legal-domain models outperform general-purpose models after pre-training, subject to sufficient capacity and language compatibility.Source: §5.1 · p. 6

5.2 Cross-Lingual Transfer (RQ1)

• Five models gained +0.094 to +0.499 zero-shot from German NSP+CLU; three transferred poorly but recovered after fine-tuning, with all models converging to 0.862–0.878.bert-base-eurlex transferred most strongly (0.338→0.837, +0.499) and gained only +0.032 from further fine-tuning.Source: §5.2 · pp. 6–7

5.3 In-Domain Traffic-Law Transfer (RQ2)

• All tested models gained +0.041 to +0.158 on Austrian zero-shot exams; Austrian gains generally exceeded Irish gains, with Legal-heBERT alone at −0.049 on Irish.The authors attribute larger Austrian gains to greater regulatory overlap with StVO and conclude that transfer is modulated more by linguistic distance than traffic-law distance.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Out-of-Domain Legal Transfer (RQ3)

• StVO pre-training improved over published baselines on four of seven English legal tasks: ECtHR-A, ECtHR-B, UNFAIR-ToS, and CaseHOLD.The authors interpret this as evidence of transfer beyond the StVO domain.Source: §5.4 · p. 7

5.5 Task synergy and interference

• Full NSP+CLU+CLS outperformed partial subsets for 7/9 models; CLS was dominant, while bert-base-contracts favored NSP+CLS (0.845) over ALL (0.797).For bert-base-echr, all three objectives raised m-F1 from 0.587 (CLS only) to 0.864 (ALL), a +0.277 gain; the authors associate exceptions with semantic mismatch, possible CLU interference, and capacity limits.Source: §5.5 · pp. 7–8, Figure 3 · p. 8

6 Conclusion and Future Work

• Knowledge-graph-guided multi-task pre-training encodes StVO legal structure through three complementary objectives and yields compounding benefits beyond domain-adaptive single-objective training.The objectives are NSP, CLU, and CLS.Source: §6 · p. 9
• Across evaluated models, legal-domain models outperform general-purpose ones, but model capacity and linguistic distance can override that advantage.The authors report cross-lingual transfer to English, gains on Austrian and Irish driving regulations, and improvements over established LexGLUE baselines on several English legal benchmarks.Source: §6 · p. 9
• Future work will replace random CLU negative sampling with hard-negative sampling and extend the approach to other European regulatory corpora.Source: §6 · p. 9

• The approach is evaluated only on German traffic regulations; generalization to other legal systems, languages, and regulatory domains remains empirically unvalidated.Source: Limitations · p. 9
• Synthetic QA pairs may not reflect authentic examination distributions, especially for ambiguity and multi-hop reasoning, and shared generators limit the gains' interpretation to synthetic QA.Training and evaluation pairs share the same generators.Source: Limitations · p. 9
• Random CLU negatives can pair semantically related paragraphs, potentially interfering with out-of-domain bases such as bert-base-contracts; hard-negative sampling remains future work.Source: Limitations · p. 9
• The KG is parsed from an LLM-translated English StVO, and resulting translation and TransE embedding noise is not quantified.Source: Limitations · p. 9
• CLS attention over all paragraph embeddings is tractable for 59 StVO paragraphs, but its linear cost on larger codes is untested; approximate nearest-neighbor retrieval is proposed as mitigation.Source: Limitations · p. 9

Ethical Considerations

• The models are research prototypes, not legally binding advice or substitutes for qualified counsel; safety-critical driver assessment without human oversight is discouraged.Source: Ethical Considerations · p. 9
• LLM-generated corpora may retain legal inaccuracies, no held-out data was validated by legal experts, and inherited base-encoder biases were not systematically audited.Transfer to Austrian and Irish driving regulations was evaluated zero-shot and showed variable performance; use beyond German traffic law requires further validation.Source: Ethical Considerations · pp. 9–10

A Knowledge Graph Topology and Visualization

• The translated and parsed StVO KG contains 554 nodes and 945 directed edges representing regulatory elements and their dependencies, hierarchies, and cross-references.Figure 4 shows a partial view of the graph.Source: A Knowledge Graph Topology and… · p. 12, Figure 4 · p. 12

B Hyperparameter Configuration

• The appendix reports that all stages use seed 42 and one NVIDIA H100 with 80 GB and CUDA 12.1.Source: B Hyperparameter Configuration · p. 12

B.1 TransE Embedding Training

• TransE is trained on 554 nodes, 945 edges, and five relations to produce 500-dimensional paragraph embeddings used as CLU soft targets.It uses margin-ranking loss with uniformly corrupted head/tail negatives, and selects the checkpoint by validation loss.Source: §B.1 · p. 12

B.2 KG-Guided Multi-Task Pre-Training

• The multi-task setup combines NSP, CLU, and CLS with loss weights summing to one, yielding 36 valid configurations per model.The grid sweeps λNSP and λCLS in increments of 0.1, with λCLU set to 1 − (λNSP + λCLS).Source: §B.2 · p. 12

• Removing CLS causes the largest mean m-F1 drop (0.277), compared with drops of 0.105 for NSP and 0.056 for CLU.Source: C Full Ablation Results · pp. 12–13
• Paired tests across models find significant contributions for CLS [t(8)=4.35, p<0.01] and NSP [t(8)=2.52, p<0.05], but not CLU [t(8)=1.76, p=0.11].Source: C Full Ablation Results · p. 13
• CLU adds gains for top-performing models such as bert-base-echr, but its contribution is architecture-dependent and can interfere with some general-purpose base models.Source: C Full Ablation Results · p. 13

D LexGLUE Full Results

• Table 8 spans 9 models × 7 task configurations and defines ∆(w/o CLS) as ALL − NSP+CLU.Source: Table 8 · p. 14
• Figure 5 reports that CLS dominates the ablation, with an average drop of 0.277, while CLU interference is negative for bert-base-contracts and google-bert-base.The figure describes task synergy as architecture dependent.Source: Figure 5 · p. 14
• StVO-pretrained models improve over published baselines on four LexGLUE tasks—ECtHR-A, ECtHR-B, UNFAIR-ToS, and CaseHOLD—but show negative transfer on SCOTUS, EUR-LEX, and LEDGAR.Table 9 covers all seven tasks and reports micro- and macro-F1 scores.Source: D LexGLUE Full Results · p. 13, Table 9 · p. 14