Source-linked AI summary
sui-1: Grounded and Verifiable Long-Form Summarization
Benedikt Droste, Jan Philipp Harries, Maximilian Idahl, Björn Plüster
TL;DR
Large language models often produce summaries with unsupported or misattributed claims, making verification laborious. sui-1 uses a 24B model trained on verified synthetic citation-grounded examples, and achieves 0.842 overall versus 0.427–0.556 for baselines. The authors report improved format compliance and release the model and resources, while noting limits from German-heavy data and LLM-as-a-judge evaluation.
Problem
Large language models frequently generate unfaithful summaries with fabricated or misattributed claims that users must laboriously verify against source documents.
Method
sui-1 combines a 24B model with synthetic citation-grounded training data generated by a teacher model and checked through automated verification and quality filtering.
Results
0.842 overall score substantially exceeds baselines at 0.427–0.556 and approaches the reference model at 0.891.
Takeaways & Limitations
Task-specific training improves format compliance and instruction following for citation-grounded summarization, and the trained model and resources are released.
Takeaways & Limitations
Training data is primarily German-language, potentially limiting generalization to other languages and specialized domains, while LLM-as-a-judge evaluation may be systematically biased.
Abstract
from arXiv · showhide
Large language models frequently generate plausible but unfaithful summaries that users cannot verify against source text, a critical limitation in compliance-sensitive domains such as government and legal analysis. We present sui-1, a 24B parameter model that produces abstractive summaries with inline citations, enabling users to trace each claim to its source sentence. Our synthetic data pipeline combines chain-of-thought prompting with multi-stage verification, generating over 22,000 high-quality training examples across five languages from diverse sources including parliamentary documents, web text, and Wikipedia. Evaluation shows sui-1 significantly outperforms all tested open-weight baselines, including models with 3x more parameters. These results demonstrate that task-specific training substantially outperforms scale alone for citation-grounded summarization. Model weights and an interactive demo are publicly available.
1 Introduction
Large language models can produce unfaithful summaries whose claims are difficult to verify. sui-1 addresses this problem with citation-grounded summarization and synthetic training data with automated verification.
- Citation-grounded summarization attributes claims to source text, allowing users to assess reliability and trace claims to their origins.
- Standard datasets lack citation annotations, while manual annotation is expensive and the task requires coordinated content generation and source referencing.
- Synthetic data generation with a capable teacher model and automated verification addresses the training-data challenges of citation-grounded summarization.
- sui-1 is a 24B parameter model that achieves 84% overall accuracy versus 43–56% for baselines on LLM-as-a-judge evaluation.
2 Related Work
Prior work addresses long-document constraints, attribution, and synthetic instruction tuning, but the cited approaches differ in their reasoning and citation capabilities. sui-1 is positioned within this broader development of verifiable language-model outputs.
- Long-document summarization remains difficult because modern language models can hallucinate despite improved context windows.
- PEGASUS and Longformer address length constraints through sparse attention or hierarchical processing but do not produce verifiable citations.
- ALCE established evaluation standards for attributed text generation, while LongCite generates fine-grained citations from retrieved chunks.
- RAG systems such as Self-RAG and WebGPT cite retrieved external documents but rely on complex inference-time processes.
- Synthetic instruction-tuning work shows that smaller models can learn complex reasoning from larger teachers, motivating the teacher-based approach here.
3 Approach
sui-1 is trained from diverse long-form sources using tagged sentences, teacher-generated summaries, citation verification, and multi-stage quality filtering. The pipeline also supports varied user instructions and long documents through chunking and merging.
- Data and Preprocessing: The corpus combines German parliamentary documents, Common Crawl texts, and multilingual Wikipedia, with documents ranging from short announcements to reports exceeding 50,000 words.
- Data and Preprocessing: Sentences receive deterministic, language-agnostic XML identifiers that enable precise citation tracking without verbatim quotes.
- Generation Pipeline: The five-stage pipeline tags sentences, constructs prompts, generates summaries with a frontier model, verifies citations, and filters quality.
- Generation Pipeline: Documents under 30K tokens are processed in one pass, while longer documents are chunked, summarized independently, and merged while preserving citations.
- Custom Instructions: Custom instructions cover positive, adversarial, and formatting requirements, while relaxed training constraints reduce dependence on rigid deployment prompts.
- Quality Control: Quality control verifies citation existence, evaluates reasoning and citation distribution, penalizes generic filler, and filters examples with poor citation spacing.
4 Training
The training setup uses a multilingual 24B instruction model with parameter-efficient fine-tuning and long-context infrastructure. Documents and training sequences are designed to accommodate extended inputs and diverse lengths.
- Mistral-Small-3.2-24B-Instruct is selected as the base model for its multilingual capabilities, particularly in German and other European languages.
- The model is fine-tuned with LoRA, rank 16, for two epochs on sequences up to 100K tokens.
- The training documents have a long-tail token distribution, truncated at 50K tokens for visibility, with a maximum length of 179K tokens.
- Training uses four NVIDIA H100 GPUs with context parallelism, Flash Attention, and gradient checkpointing for memory-efficient long-sequence processing.
- FP8 quantization reduces memory requirements by 50% while maintaining generation quality.
5 Results
Evaluation compares sui-1 with several baselines across five criteria and shows stronger overall performance, especially in format compliance. Qualitative examples likewise show more specific, mechanically verifiable citations than baseline outputs.
- Quantitative evaluation: sui-1 achieves 0.842 overall, outperforming baselines at 0.427–0.556 and approaching the reference model at 0.891.
- Quantitative evaluation: Format compliance reaches 0.895 for sui-1, compared with 0.137–0.411 for baselines.The evaluation scores binary checks across the test set.
- Quantitative evaluation: sui-1 records 0.600 coverage, highest among open-weight models, while the reference model reaches 0.705.Its factual accuracy is 0.905 and specificity is 0.979 under constrained formats.
- Qualitative analysis: Baseline examples are generic and uncited, whereas sui-1 produces specific claims with verifiable XML tags.A representative output places inline tags immediately after supported claims.
- Qualitative analysis: sui-1’s structured reasoning identifies document themes before generating summaries with citations that can be mechanically verified.Baseline models often omit citations or produce malformed tags that do not match source sentences.
6 Release
The project releases sui-1 model weights and the associated dataset for reproducibility and practical use. Full-precision and FP8 variants support different deployment priorities.
- Model release: The release includes sui-1 model weights in full-precision and FP8-quantized variants.
- Model release: The FP8 variant reduces memory requirements and is recommended when deployment prioritizes memory efficiency.
- Model release: Both model variants use the Apache 2.0 license, enabling commercial use.
- Dataset release: The training dataset contains source documents with XML tags, generated summaries, reasoning traces, custom instructions, and quality annotations.
- Usage: The model uses the Mistral-Small-3.2 chat template with a summarizer system prompt, tagged source document, and optional custom instruction.
7 Conclusion
The paper presents sui-1 as a citation-grounded summarization model trained through synthetic data generation. Its evaluation supports improvements in format compliance and instruction following, while the authors identify language, domain, and evaluator limitations.
- Conclusion: sui-1 is a 24B parameter model trained to produce verifiable inline citations through synthetic data generation.
- Conclusion: The pipeline produces 22K verified training examples across five languages through multi-stage quality filtering.
- Conclusion: Format compliance and instruction following reach 0.895, compared with 0.137 to 0.411 for baselines.
- Conclusion: The 24B model approaches the reference model with scores of 0.842 versus 0.891.
- Limitations: Training data derives primarily from German-language sources, which may limit generalization to other languages and specialized domains.The LLM-as-a-judge evaluation may also exhibit systematic biases compared with human evaluation.
A.2 XML Tag Generation
The appendix describes how source sentences receive compact XML identifiers and how prompts use those tags to construct citation-grounded summaries. It also covers prompt variants for single-window, iterative, merged, and customized summarization.
- XML tag generation: Each source sentence receives an identifier by hashing its UTF-8 text with MD5 and taking the first eight hexadecimal characters.
- XML tag generation: The hash is wrapped in opening and closing XML tags around the sentence.For example, a3f5e823 tags “The budget was approved.”
- XML tag properties: The tagging scheme is deterministic, language-agnostic, and compact, while eight hexadecimal characters provide approximately 4.3 billion possible tags.
- Verification: Verification distinguishes citation tags from ordinary HTML tags by checking tag contents against valid source-document tags.
- Prompting: The oneshot prompt requires selecting significant, distributed XML tags and placing each reference immediately after its supported claim.It also requires every selected tag to appear exactly once and forbids reproducing the original sentence text.
- Long-document prompting: Iterative prompting summarizes chunks of longer documents with consistent 300–600-word outputs before merging partial summaries while preserving citations.
C Training Configuration
The training configuration combines specialized hardware and memory-saving techniques with custom-instruction examples spanning legal and policy documents. The examples demonstrate citation-grounded outputs, including liability rules, exemptions, enforcement powers, and regulatory debates.
- Training infrastructure: Training uses the Axolotl framework with Flash Attention 2, gradient checkpointing, four NVIDIA H100 GPUs, and context parallelism.Examples use ChatML with a system prompt, tagged document, optional instruction, and assistant response containing reasoning and citations.
- Training infrastructure: FP8 quantization reduces model size from approximately 48GB to 24GB while maintaining generation quality on most metrics.
- Custom-instruction examples: A German parliamentary example summarizes operator liability for prevention, damage limitation, and remediation costs, subject to specified exemptions.Exemptions cover third-party causes despite safety precautions and permitted conduct without intent or negligence.
- Custom-instruction examples: The example also covers agricultural and forestry exemptions, authority enforcement powers, and removal of a duplicated administrative-justification requirement.Authorities may act upon application or ex officio, while justification duties are covered by the general Administrative Procedure Act.
- Custom-instruction examples: The summary highlights a debate over whether the Federal Council should immediately use its authorization to create a financial security regulation.
D.2 Example 2: Expert Focus (German)
A German custom instruction asks sui-1 to focus on federal-budget effects and business bureaucracy costs in a Family Care Leave Act report. The output reports annual public spending, administrative staffing, per-case business costs, and citizen time requirements.
- Federal budget effects: The proposed employer-refinancing credit line creates long-term annual federal expenses of up to 8 million euros.The expenses must be offset in Einzelplan 17 and cover average labor and interest costs plus risk.
- Implementation workload: Administrative implementation is handled by the Federal Office for Family Affairs and Civil Society Functions using capacity released by the end of compulsory civilian service.
- Bureaucracy costs: Business information duties cost employers around 32.50 euros per case and insurers around 10 euros per case, with up to 44,000 employer cases annually.Affected citizens face approximately 20 minutes of time expenditure per person.
D.3 Evaluation Platform
The evaluation platform uses an LLM-as-a-judge interface to report aggregate and criterion-level results. It also provides detailed views for inspecting individual test samples and explanations.
- Evaluation interface: The Elluminate interface provides per-criterion pass rates, aggregated overall scores, and detailed per-sample explanations.
- Aggregate evaluation: Figure 6 reports an 84.2% overall score with a per-criterion breakdown for sui-1.
- Sample-level evaluation: Figure 7 displays individual criterion scores and explanations for a single test example.