Source-linked AI summary
Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillation
Mohamed Elaraby, Ahmed Elhady, Diane Litman
TL;DR
Small LLMs struggle to retain sparse, salient argumentative content in long legal opinions, while expert annotations are costly to obtain. The paper uses sequence-level distillation from teacher-generated traces and summaries, finding consistent gains over expert-summary tuning and structured planning, with most gains from about 10 examples and only marginal benefit from adding reasoning traces to summary supervision.
Problem
Small models struggle to preserve salient argumentative content dispersed across long legal opinions, while expert-written supervision is costly and difficult to scale.
Method
The paper trains small students with sequence-level distillation using teacher-generated summaries, reasoning traces, or both.
Results
Distillation consistently surpasses expert-summary tuning and structured planning across student sizes, with most gains emerging from roughly 10 training documents.
Takeaways & Limitations
Summary-only teacher supervision is sufficient for improvement, while reasoning-chain distillation provides only marginal additional benefit when combined with summaries.
Takeaways & Limitations
The conclusions are scoped to CANLII, ARCscore, the Qwen3 student family, two teachers, LoRA tuning, and access to unlabeled in-domain opinions.
Abstract
from arXiv · showhide
We show that sequence-level distillation from a capable long-context teacher model is a simple, annotation-free, and data-efficient strategy for improving argument saliency coverage in long legal opinion summarization, where small LLMs often struggle to retain the most salient argumentative content. Across student model sizes, distillation consistently surpasses tuning on expert-written summaries in our legal-opinion setting. We further demonstrate that most gains are achieved with as few as ~10 training summaries, highlighting the strong data efficiency of teacher-generated supervision. Finally, we find that summary distillation is sufficient for improvements: reasoning-chain distillation remains competitive with summary-only distillation, but provides marginal benefit when combined with summary supervision.
1 Introduction
Long legal opinions distribute sparse but critical argumentative content across thousands of tokens, and smaller models often fail to preserve it. The paper studies sequence-level distillation as an annotation-free alternative to costly expert-summary tuning.
- Motivation: Long legal opinions require summaries to preserve sparse argumentative roles that are critical to include.Important argumentative content may be dispersed across thousands of tokens.
- Motivation: Smaller models face especially strong difficulty prioritizing salient information in long legal opinion summarization.This limits efficient and reliable summarization systems for high-stakes legal settings.
- Motivation: Sequence-level distillation transfers generation behavior through teacher-generated sequences without requiring expert annotations.Expert-written summaries are costly and difficult to scale.
- Research design: The study compares summary-only, reasoning-trace-only, and combined reasoning-trace-plus-summary supervision from proprietary and open-weight teachers.It also compares distillation with expert-summary tuning and structured inference-time planning.
- Findings: Across student sizes, distillation consistently surpasses expert-summary tuning and structured planning, with most gains emerging from roughly 10 training documents.Teacher-generated summaries alone are sufficient, while reasoning-trace supervision adds only marginal benefit.
2 Related Work
Prior legal summarization work identifies faithfulness and coverage problems, while existing coverage methods rely on external labeling or reranking. This paper extends distillation to argument saliency in small LLMs for long legal opinions.
- Legal summarization: Coverage gaps in salient argumentative content can exceed faithfulness problems in legal opinion summarization.Long opinions make salient content sparse and dispersed across thousands of input tokens.
- Prior approaches: Existing legal coverage approaches use external argument labeling or reranking pipelines.The paper instead empirically studies distillation for improving argument saliency in small LLMs.
3 Distillation Setup
A teacher generates a reasoning trace and summary for each prompt and legal opinion, while students learn a selected target sequence through supervised fine-tuning. The study isolates whether reasoning traces add supervision beyond summaries.
- Teacher supervision: Given a prompt and legal opinion, the teacher generates a native reasoning trace followed by a summary.The resulting pair is used to construct supervision sequences for student training.
- Student training: Student models are trained with supervised fine-tuning on a target sequence G.The setup varies only the supervised sequence.
- Supervision settings: Summary-only distillation sets G to the teacher summary, reasoning distillation sets G to the reasoning trace, and combined distillation concatenates both.The reasoning-only condition masks the final summary from the loss.
- Supervision settings: The reasoning-only setting tests whether teacher traces transfer intermediate saliency selection rather than merely summary-output imitation.This distinguishes reasoning supervision from direct supervision on generated summaries.
4 Experimental Setup
Experiments use CANLII legal opinions, Qwen3 students, GPT-5-mini and Qwen3-14B teachers, and ARCscore-based argument-coverage evaluation. Distillation is compared with zero-shot, expert-tuning, and Chain-of-Arguments baselines.
- Dataset: CANLII contains legal opinions paired with expert annotations of argumentative roles: Issues, Reasons, and Conclusions.A sentence-level annotated subset enables evaluation of argument saliency coverage.
- Dataset: The test split contains 1,049 annotated opinions, while training uses 1,000 nonoverlapping unannotated opinions truncated to 4,096 tokens.The full dataset contains approximately 28k legal opinion–summary pairs.
- Models: Teachers are proprietary GPT-5-mini and open-weight Qwen3-14B, while Qwen3 students range from 0.6B to 8B parameters.Reasoning-trace distillation is used only with Qwen3-14B because proprietary models do not expose traces.
- Baselines: Baselines include zero-shot generation, expert-summary tuning on the same 1,000 documents, and Chain-of-Arguments prompting.The comparisons separate teacher-generated supervision from expert supervision and structured prompting.
- Evaluation: Table 1 reports ARCscore mean±std. across supervision setups, with B, E, CoA, and S denoting zero-shot, expert-tuning, Chain-of-Arguments, and summary distillation.Teacher rows provide standalone ARCscore reference values for distilled students.
- Evaluation: ARCscore measures the proportion of expert-annotated arguments supported by a generated summary using an LLM-based atomic verifier.It was reported to correlate most strongly with expert coverage judgments among the compared coverage metrics.
5 Results and Discussion
Distillation improves argument saliency coverage across student models and generally outperforms expert-summary tuning and structured planning. Most gains appear with few training documents, while summary-only supervision is usually sufficient and missing errors decline consistently.
- Main distillation results: Distillation significantly improves saliency coverage across student model sizes and consistently outperforms expert-summary tuning and CoA.At 0.6B, Qwen3-14B distillation is statistically comparable to expert tuning; for larger students, paired tests confirm the advantage.
- Supervision strategies: Summary-only supervision is sufficient, while combining reasoning traces with summaries yields only marginal gains of +0.001–+0.011 over summary-only distillation.Reasoning-trace-only supervision remains competitive with summary-only supervision.
- Error analysis: 55.87% → 31.31%: Qwen3-14B summary distillation reduces Missing Errors for the 0.6B student, while Factual Errors increase from 10.20% to 12.22%.For the 8B student, both Missing Errors and Factual Errors decrease: 30.01% →17.40% and 11.86% →9.05%, respectively.
- Data scaling: Argument coverage improves significantly with as few as 10 training documents across proprietary and open-weight teachers (Mann-Whitney U test, p < 0.05).Scaling beyond 100 documents yields diminishing returns for Qwen3-14B.
- Data scaling: Scaling distillation data produces diminishing and sometimes inconsistent gains, with near-saturation at 1,000 documents.GPT-5-mini coverage drops at 100 documents before partially recovering at 1,000.
6 Conclusion and Future Work
The paper concludes that sequence-level distillation is a simple, data-efficient approach for improving argument saliency in long legal summarization. Its conclusions are bounded by evaluation on one dataset, one primary coverage metric, and the Qwen3 student family.
- Conclusion: Sequence-level distillation improves argument saliency in long legal summarization and outperforms expert-supervised tuning with as few as 10 CANLII training examples.The paper characterizes the strategy as annotation-free and data-efficient.
- Conclusion: Summary-only supervision is sufficient, while reasoning-chain distillation provides only marginal additional gains.The conclusion identifies teacher-generated summaries as adequate supervision for the studied setting.
- Scope and future work: The findings are established on a single legal dataset, one primary coverage metric, and the Qwen3 student family.The paper identifies generality across datasets, metrics, and model families as an important direction.
Limitations
The study identifies limitations in its optimization strategy and empirical scope, while outlining directions for stronger and broader validation.
- Optimization scope: The experiments use standard SFT with parameter-efficient LoRA rather than stronger optimization objectives or full-parameter fine-tuning.Contrastive learning, preference optimization, and full-parameter fine-tuning remain unexplored in this study.
- Domain and evaluation scope: The conclusions are scoped to CANLII, ARCscore, the Qwen3 family, and two teacher models.The study does not test transfer across legal subdomains, jurisdictions, alternative metrics, or other model families.
- Data assumptions: The approach assumes access to unlabeled in-domain legal opinions for teacher distillation.That assumption may not hold in specialized high-stakes settings.
Ethical Statement
The paper describes its legal-argument representation, training procedures, prompts, and ARCscore-based evaluation, while warning that generated summaries are not professional legal analysis.
- Ethical risks and intended use: Generated summaries may omit important details or contain unsupported claims, so they should not substitute for professional legal analysis.The approach is not intended for legal decision-making or autonomous legal advice.
- Argument representation: The study uses CANLII annotations of Issues, Reasons, and Conclusions to evaluate argumentative coverage.Figure 3 illustrates these roles and adds argumentative relations for illustration.
- Training setup: Students are trained with QLoRA using 4-bit quantization and LoRA adapters applied to attention and linear layers.The selected configuration uses rank r = 32 and scaling factor α = 64 for three training epochs.
- Distillation strategies: The study compares summary-only, reasoning-trace-only, and combined supervision sequences for sequence-level distillation.Reasoning-trace-only training retains the standard summarization prompt while applying SFT loss only to reasoning-trace tokens to avoid degeneration.
- Evaluation: ARCscore measures the proportion of expert-annotated salient argument-role facts covered by a generated summary.The final score averages role-level coverage, while verifier labels distinguish missing errors from factual errors.
E Scaling Training Context Length
The study tests whether longer training contexts improve argument coverage and finds that distillation itself contributes more than scaling context length beyond 4k tokens.
- Context-length effects: The jump from zero-shot to 4k training contexts substantially outweighs gains from further context scaling.The comparison uses summary-only distillation from Qwen3-14B with Qwen3-1.7B and Qwen3-4B students.
- Context-length effects: Beyond 4k tokens, longer training contexts provide only marginal additional improvements, particularly for the 4B student in non-thinking inference mode.This indicates that longer contexts can modestly improve coverage, but their effect is secondary to distillation.
- Significance testing: Distillation improves significantly over zero-shot performance for every student under both teachers, with all p < 0.001.The statistical comparison uses paired Wilcoxon signed-rank tests on the same test documents.
- Baseline comparisons: Against expert tuning, Qwen3-14B distillation is significantly better for Qwen3-1.7B and larger students, while Qwen3-0.6B is statistically comparable.For Qwen3-0.6B, the reported difference is +0.022 with p = 0.09.
- Baseline comparisons: Against Chain-of-Arguments, Qwen3-14B distillation is significantly better for all students, whereas GPT-5-mini is competitive for Qwen3-0.6B.GPT-5-mini exceeds Chain-of-Arguments for larger students but not for Qwen3-0.6B.
G Perplexity Analysis: Explaining Teacher Impact
Perplexity analysis links stronger distillation signals to greater compatibility between teacher-generated summaries and student model distributions, while stressing that this relationship is correlational rather than a complete causal explanation.
- Perplexity patterns: Qwen3-14B summaries had the lowest median perplexity across all student sizes, followed by GPT-5-mini and then expert-written summaries.The medians were 9.7–19.3 for Qwen3-14B, 13.6–28.8 for GPT-5-mini, and 34.1–76.5 for expert-written summaries, whose distribution had a heavy tail extending to thousands.
- Perplexity patterns: The perplexity ordering directly mirrored the distillation-performance ranking, with more student-compatible teacher summaries producing stronger coverage gains.
- Student alignment: Qwen3-14B summary perplexity fell from 19.3 at 0.6B to 9.7 at 8B, indicating increasing alignment within the Qwen3 student family.
- Caveat: The analysis is correlational and does not provide a complete causal account of teacher impact.A fuller explanation would control for teacher quality and lexical or semantic similarity between teacher outputs and expert references.
- Cross-family validation: Cross-family experiments indicate that teacher-generated targets, rather than shared model family alone, account for the observed distillation gains.Qwen3-14B distillation improved ARCscore for Qwen2.5-3B and Llama-3.2-3B, while expert tuning reduced it below zero-shot performance.
I Relative Change in Factual and Missing Errors
Summary-only distillation reduces missing errors immediately, while factual-error improvements depend on student capacity. Gains are largely realized with about 10 documents, and increasing supervision or adding reasoning traces provides limited additional benefit.
- Relative error changes: −48% missing errors occurred for the 1.7B student with Qwen3-14B summary-only distillation, the largest relative reduction reported.
- Relative error changes: Missing errors decreased at every student size for both teachers, whereas factual errors initially increased at 0.6B before declining after a capacity threshold.The factuality threshold occurred between 0.6B and 1.7B for Qwen3-14B and between 1.7B and 4B for GPT-5-mini.
- Teacher comparison: Qwen3-14B outperformed GPT-5-mini on both factual and missing errors at every student size, with the factual-error gap widening as students grew.
- Per-fact analysis: ARCscore gains in the illustrated legal opinion primarily came from reducing missing errors from 10 to 5 to 3, while factual errors stayed low at 1 to 2 to 1.
- LoRA capacity: Increasing LoRA rank from 32 to 64 or 128 yielded no significant improvement, with performance often flat or slightly worse under thinking-mode inference.All reported significance tests had p > 0.05.
- Few-shot efficiency: Most few-shot improvement emerged with 10 training documents, while scaling to 100 or 1,000 produced diminishing and sometimes inconsistent gains across student sizes.The same saturation trend held across two random seeds, with gains largely realized by 10 documents and saturating by 100.
- Supervision strategies: Reasoning-only and summary-plus-reasoning supervision closely tracked summary-only distillation, indicating no consistently stronger signal from reasoning traces alone.Reasoning traces added only marginal benefit when combined with summary supervision.