BOE-XSUM: Extreme Summarization in Clear Language of Spanish Legal Decrees and Notifications

1 Introduction

• Spanish summarization resources are scarce, especially for administrative and legal documents, motivating a new domain-specific resource.The introduction reports 963 English summarization datasets versus 75 in Spanish, while Spanish domain-specific resources are described as virtually non-existent.Source: §1 · p. 2
• BOE-XSUM provides 3,648 manually curated, verified, clear-language summaries of BOE entries alongside their original texts.It is presented as the first dataset of its kind in Spanish, addressing the challenge of translating legal and administrative language into everyday language.Source: §1 · p. 2
• The paper evaluates whether generative models can preserve complex legal meaning while adapting it into clear language comparable to human-expert summaries.The dataset and experiments are intended to address a gap in Spanish NLP resources and assess models on extreme summarization.Source: §1 · p. 2
• Current systems still struggle to produce high-quality extreme summaries, although some generated outputs resemble human-written summaries in qualitative analysis.The paper’s contributions are a publicly available dataset and experiments with generative text models.Source: §1 · p. 2

2 Related work

• Extreme summarization generates highly concise, often single-sentence summaries that capture a document’s core meaning through abstractive synthesis.The cited work distinguishes it from extractive methods and discusses its use in areas including science and news.Source: §2 · p. 3
• Prior summarization approaches span extractive methods based on frequency, heuristics, linguistics, supervised learning, and reinforcement learning.The paper situates extreme summarization as an abstractive form of single-document summarization.Source: §2 · p. 3
• Existing multilingual summarization datasets can include formats that make matching source texts to summaries easy, risking memorization rather than meaning abstraction.Examples include summaries presented as titles before the text or followed by links to the full text.Source: §2 · p. 3
• NoticIA is described as the only available Spanish extreme-summarization dataset, pairing 850 news articles with human-written single-sentence summaries.Related work also notes the challenge of recasting specialized legal or scientific discourse in accessible language while preserving meaning and nuance.Source: §2 · p. 3

3.1 Sources

• The dataset derives from a journalist’s BOE-focused social media posts and pairs each entry with original and edited summaries; experiments use the edited versions.After manual review, the final collection contains 3,648 BOE entries with article-category labels.Source: §3.1 · pp. 3–4

3.2 Data extraction

• BOE-document links were separated from links to the journalist’s external website, and BOE PDFs were converted to plain text while retaining article identifiers.The conversion process supported downloading article content for the dataset.Source: §3.2 · p. 4

3.3 Manual data review

• Cosine-distance analysis found 1,154 posts (a third of the dataset) above 90% similarity between original and edited versions, while approximately 160 were below 10%, spanning minor adjustments to full rewrites.Source: §3.3 · p. 4

3.4 Categorization

• Categories follow explicit mentions in BOE texts, with some related types combined and very low-frequency categories grouped as OTROS ANUNCIOS.BANCO DE ESPAÑA is retained despite its low article count because of its relevance.Source: §3.4 · p. 5

3.5 Content Analysis

• A mortgage-debt decree illustrates the contrast between a lengthy BOE text and a concise edited summary listing banks adhering to good-practice codes.The edited summary highlights protections for mortgage debtors, debt restructuring, and social rental housing.Source: §3.5 · p. 5
• 3,648 BOE texts contain 13,304,989 words; documents average 3,396 words, summaries average 17 words, and the reported compression rate is 0.005%.More than 64% of documents have fewer than 1,000 words, while 2.65% exceed 25,000 words.Source: §3.5 · p. 5, Figure 1 · p. 6
• The dataset uses train, development, and test splits adjusted from an initial 80/10/10 allocation to balance annotated categories.Its documents range from straightforward appointments to complex, lengthy court decisions and agreements.Source: §3.5 · pp. 5–6

4 Experiments

• BOE-document inputs are summarized in outputs capped at 280 characters, and experiments use manually edited summaries rather than original posts.The character limit parallels the limit of the social-media platform where the source summaries appeared.Source: §4 · p. 6

4.1 Evaluation Metrics

• BLEU, ROUGE, METEOR, and BERTScore evaluate n-gram overlap, sequence overlap, flexible unigram matching, and contextual semantic similarity, respectively.The experiments use common automatic evaluation metrics.Source: §4.1 · p. 6

4.2 Fine-tuning experiments

• The fine-tuning study compared BERTIN GPT-J 6B with BOLETIN and tested full fine-tuning alongside LoRA configurations in several precision formats.LoRA updates low-rank attention matrices and reduces computational and memory requirements.Source: §4.2 · p. 6
• Full-precision fine-tuning outperformed every LoRA configuration, while most fine-tuned models achieved similar outcomes.The test-set table compares model variants, bit-depths, and the number of trained layers.Source: Table 1 · p. 7, §4.2 · p. 7
• Truncation meant some training examples likely lacked critical context, potentially contributing to vague or abruptly cut summaries.The method retained the initial portion of long documents, but the examples were not systematically excluded or marked as truncated.Source: §4.2 · p. 7
• After three epochs of BOE pretraining, BOLETIN did not improve plain-language extreme summarization, suggesting domain adaptation may mismatch the target register.The adapted model was based on BERTIN GPT-J 6B and pretrained on BOE content from 1988–2023.Source: §4.2 · pp. 6, 8

4.3 Prompting experiments

• 0.335 BERTScore for DeepSeek R1 led zero-shot BOE-XSUM results, followed by ChatGPT 4o at 0.327 and Llama 3 70b Instruct at 0.285.The zero-shot experiments tested open-source and proprietary models of varied sizes, using a prompt selected after limited manual trials.Source: §4.3 · p. 8

5 Discussion and Future Work

• BOE-XSUM fills a Spanish legal-domain resource gap and is intended to support training extreme-summary and categorization models.The dataset contains categorized BOE summaries; the discussion describes it as enabling these uses.Source: §5 · p. 8
• The authors retained sparsely represented categories because they considered them highly relevant to the dataset.Source: §5 · p. 9
• Fine-tuned models up to 8-bit outperform zero-shot models except ChatGPT 4o and DeepSeek, though the comparison is not symmetrical.The fine-tuned models have 6B parameters, while DeepSeek R1 has 671B; fine-tuned models receive task-specific training, whereas zero-shot models use a single static prompt.Source: §5 · p. 9
• Extreme summarization of long BOE legal texts remains challenging, and the relative difficulty of compression versus plain-language translation is unresolved.Source: §5 · p. 10
• Standard automatic metrics may miss clarity, usefulness, and non-specialist expectations, while the paper's human evaluation was limited.The authors report a strong negative correlation of -0.82 between summary length and BERTScore and call for more robust human-centered validation.Source: §5 · p. 10
• Future work includes stronger category classification, task-specific evaluation metrics, expanded human evaluation, and prompt optimization for general-purpose models.Source: §5 · p. 10

6 Conclusions

• The dataset is designed to train and evaluate Spanish legal extreme-summary models in clear language, including summaries suited to microblogging platforms.Source: §6 · p. 10
• Preliminary experiments favor even small fine-tuned systems over prompted frontier models, but the authors say automatic metrics still need validation for this task.Source: §6 · p. 10
• The authors propose using the categorized dataset to classify daily BOE entries and generate summaries for entries of broad public interest.Source: §6 · p. 10

A Editor server

• The interface displays the original tweet and BOE document identifier, and provides an automated tweet-cleanup editor.Source: A Editor server · p. 14
• The tool supports checking BOE text against its associated tweet, correcting inconsistencies, and editing records during review.Source: A Editor server · p. 14
• Users can categorize and save data items, navigate with Back and Forward controls, and track the current item and its registration status.Source: A Editor server · p. 14

• Table 5 lists categories annotated for the text categorization task.Source: Table 5 · p. 15
• The table also provides examples alongside the annotated categories.Source: Table 5 · p. 15
• The examples illustrate the categories used in the text categorization task.Source: Table 5 · p. 15

• Table 6 shows the distribution of annotated text categories in BOE-XSUM.Source: Table 6 · p. 16
• The dataset columns include BOE metadata, article text, an original tweet, and a cleaned extreme summary.Other listed fields include category and department information, classification codes, and related BOE articles.Source: C Additional Information about the… · p. 16
• Table 7 lists the dataset columns.Source: Table 7
• Table 8 reports cosine-similarity ranges for pairs of original and modified tweets.Source: Table 8 · p. 16

• One edited example replaces a vague tweet about Oracle with the agency, service, and contract amount stated in the BOE article.The amount is 1,232,070.40 euros.Source: D Examples of edited summaries · pp. 16–17
• The dataset statistics cover item counts, document and summary word counts, their averages, and compression rate across the train, development, and test sets.Source: Table 9 · p. 17
• Another edit removes references to people and entities not present in the BOE article.Source: D Examples of edited summaries · p. 17

• Table 10 reports test-set results using the journalist’s unmodified social media posts.Source: Table 10 · p. 17
• Figure 4 compares Boletin and Bertin across bit widths and training configurations.Source: Figure 4 · p. 18
• Table 12 presents summaries, F1 scores, and differences.Source: Table 12 · p. 18

F English Version of BOE example with original and edited summaries

• The edited example replaces a sarcastic, vague comment with a direct description of the bank-adhesion list and its protective purpose.The original calls the codes “not very effective”; the edited version names protection, debt restructuring, and social rent.Source: F English Version of BOE example with… · p. 19

G Annotation Guidelines; 1. General Principles

• Summaries should be factual, concise at 15–22 words, nonredundant, and clear while preserving critical legal details.The guidance also calls for simple but precise language to make legal information more accessible.Source: §1 · p. 19, G Annotation Guidelines · p. 19

2. Formatting Rules

• Use neutral third-person language, retain necessary legal terms, simplify jargon when meaning is preserved, and avoid uncommon abbreviations.Source: §2 · p. 19

3. Information Hierarchy

• When space is limited, prioritize the main legal action, involved parties, consequences, then critical dates or locations.Drop lower-priority information first.Source: §3 · p. 19

Missing Critical Information; Overly Complex Legal Language; Misleading Summaries

• Examples contrast a subjective characterization of a new law with a factual account, and replace an unspecified contract-signing statement with one naming the Ministry of Health, AC3.5M amount, and hospital equipment.Source: §3 · p. 20, Missing Critical Information · p. 20

5. Handling Difficult Cases A) Summarizing Lengthy Legal Texts

• For lengthy BOE texts, focus on the main action in the opening paragraphs rather than trying to summarize everything.Source: §5 · p. 20

B) Appointments & Dismissals; C) Court Rulings

• For appointments, name the person, position, and institution; for court rulings, name the court, outcome, and key details.Source: B) Appointments & Dismissals · p. 20, C) Court Rulings · p. 20

D) Large Financial Figures; 5. Final Checklist for Annotators

• Round large financial figures where possible, then check length, neutrality, legal action, important details, jargon, and accessibility to non-lawyers.Source: D) Large Financial Figures · p. 20, §5 · p. 20