Source-linked AI summary
Reinforcement Learning for improving Large Language Models' Catalan text simplification capabilities
Arnau Ayguadé Domingo, Stefan Bott, Horacio Saggion
TL;DR
ATS remains difficult to optimize for low-resource languages such as Catalan despite its accessibility value. This paper post-trains IberianLLM with GRPO using a SARI-based reward augmented by copying and length penalties. English ASSET training improved performance on Catalan ASSET and iDEM while suppressing over-generation, whereas translated-data variants did not significantly improve the out-of-domain benchmark.
Problem
ATS needs models optimized for low-resource languages such as Catalan, while RL has received limited attention despite its potential to directly encode simplification objectives.
Method
The paper post-trains IberianLLM with GRPO using a normalized SARI reward supplemented by copying and output-length penalties.
Results
English ASSET post-training improved SARI performance on Catalan ASSET and iDEM and suppressed over-generation, while Catalan and Spanish variants did not significantly improve iDEM.
Takeaways & Limitations
GRPO with the proposed reward function improved simplification performance within the study’s limited experimental setting, particularly after training on original English ASSET.
Takeaways & Limitations
The study is limited by ASSET data quality and SARI, including possible reference-pair leakage and incomplete reduction of output diversity.
Abstract
from arXiv · showhide
Although automatic text simplification (ATS) is critical for accessibility, its progress has not matched the rapid evolution of broader natural language processing techniques. This paper investigates the application of reinforcement learning (RL) to improve the quality of ATS for low-resource languages using Large Language Models (LLMs). The paper introduces a novel reward function, designed to guide LLMs toward a targeted simplification style with Group Relative Policy Optimization (GRPO), that combines the SARI metric with specific penalty components. The effectiveness of GRPO with this reward function is motivated and demonstrated by post-training IberianLLM-7B-Instruct on the ASSET dataset. After post-training on the English ASSET, the model's ATS performance improves on two curated Catalan benchmarks while also successfully suppressing previously observed negative behaviors. Cross-lingual transfer learning is explored by translating ASSET into Catalan and Spanish and post-training the model on each version, but these fail to show a significant improvement on the out-of-domain benchmark.
1 Introduction
Automatic text simplification improves accessibility by converting text into simpler language while preserving meaning, yet optimized ATS models remain scarce for low-resource Catalan. The paper applies GRPO with a targeted reward function and evaluates English, Catalan, and Spanish training data for Catalan simplification.
- ATS transforms text into simpler versions while retaining meaning for target audiences, supporting access for people with cognitive disabilities, low literacy, and second-language learners.
- LLMs are widely used for ATS, but task-aligned models specifically optimized for simplification remain scarce, especially for Catalan.
- RL directly optimizes a chosen reward and may provide more consistent outputs, but it has received limited attention in ATS.
- The paper introduces a GRPO reward function combining language-independent SARI with constraints that address metric limitations.
- Post-training IberianLLM-7B-Instruct on English ASSET improved alignment with ASSET and iDEM references while suppressing over-generation; translated-data variants were also evaluated.
2 Datasets
The study uses iDEM as an expert-curated Catalan evaluation corpus and ASSET as a larger crowdsourced training corpus. Translated Catalan and Spanish ASSET versions support the cross-lingual transfer experiments, with filtering applied to improve sentence-level and semantic alignment.
- The Catalan iDEM subset contains 380 source sentences with corresponding expert-curated simplifications.
- iDEM includes document-level and explanatory transformations, so BLEURT filtering removes pairs below 0.5 to favor sentence-level pairs with semantic similarity.
- ASSET contains 2,000 English sentences, each paired with ten crowdsourced simplifications, and was selected for post-training because of its large sample count.
- ASSET simplifications are generally milder and less controlled than iDEM’s expert-curated simplifications, often changing only a few words.
- ASSET was automatically translated into Catalan and Spanish, retaining pairs with BLEURT scores from 0.7 to 0.9 for cross-lingual experiments.
- The translated corpora may have altered sample quality, creating a limitation for interpreting cross-lingual training results.
3 GRPO Methodology
The methodology post-trains IberianLLM with GRPO using SARI-based rewards and penalties for copying and excessive length. Models trained independently on English, Catalan, and Spanish ASSET are evaluated on Catalan ASSET and iDEM.
- GRPO: GRPO normalizes rewards among multiple responses to the same prompt, avoiding a separate value-function estimator and favoring relatively higher-reward outputs.
- Experimental setup: IberianLLM was post-trained independently on English, Catalan, and Spanish ASSET, producing three model variants evaluated on two Catalan benchmarks.
- Reward function: SARI scores the words added, deleted, and retained during simplification against golden references and was used to optimize performance.
- Reward penalties: Pcopy penalizes candidates that copy the source sentence instead of producing a distinct simplification.
- Reward penalties: Plen applies an exponentially decaying penalty when generated output exceeds the reference length by more than five words, with α = 4.
4 Results
GRPO post-training rapidly adapted IberianLLM to the simplification style, improved Catalan ASSET performance across all training-language variants, and produced a statistically significant iDEM gain only for English ASSET.
- SARI scores increased rapidly during approximately the first 10,000 training steps before stabilizing for the remainder of training.The same rapid-adaptation pattern was observed for the Catalan and Spanish ASSET runs.
- Table 1 reports average SARI scores for the Catalan ASSET test set and filtered iDEM benchmark before and after GRPO post-training on Catalan, Spanish, and English ASSET.
- All three post-trained models improved Catalan ASSET SARI scores by 2–2.7 points over the original IberianLLM and no longer suffered from over-generation.
- Only English ASSET post-training significantly improved the filtered Catalan iDEM benchmark and outperformed the Catalan and Spanish variants.Significance was assessed with a bootstrap test on paired per-item score differences from the same 304 sentences.
5 Conclusions
The experiments indicate that GRPO with the proposed reward function can improve simplification performance in the tested setting. English ASSET post-training suppressed over-generation and improved both benchmarks, whereas translated-data gains did not transfer to iDEM, leaving translation effects unresolved.
- GRPO with the proposed reward function improved simplification performance within the limited experimental setting considered.
- English ASSET post-training rapidly suppressed over-generation and improved performance across both evaluation benchmarks.The gain on the more challenging iDEM benchmark occurred despite ASSET generally containing milder simplifications.
- Translated-ASSET models improved in-domain but did not carry those gains over to the iDEM benchmark.
- Transferable simplification behavior depends strongly on the training data because rewards are computed against that corpus’s reference simplifications.The reference properties therefore define the optimization target.
- The English-to-Catalan transfer result rests on a single benchmark and a single training run, while reference-free reward metrics remain future work.
6 Limitations
The study’s main limitations concern the ASSET dataset, SARI-based rewards, and limited computational exploration. These constraints affect dataset reliability, metric validity, and confidence in the training comparisons.
- Dataset limitations: The ASSET dataset may not reduce output diversity as intended and may introduce data leakage because source-reference pairs, rather than unique sources, were split.Each input has ten reference simplifications, and the same source sentence could appear in both partitions with different references.
- Dataset limitations: BLEURT filtering may discard valid sentence pairs when simplification involves substantial content deletion.
- Metric limitations: SARI has known limitations and correlates poorly with human judgments, while the absence of a Catalan reference-free metric keeps the reward tied to references.This constrains the practical advantage of RL over supervised fine-tuning.
- Experimental limitations: Each language variant was trained in only one GRPO run because of limited computational resources, leaving potentially better training configurations unexplored.
Appendix A Prompt Engineering
Prompt engineering compared minimal, role-separated, few-shot, and rule-based prompts using SARI. The rule-based configuration achieved the highest score and was selected for post-training.
- Prompt strategies: Four prompting strategies were evaluated with SARI, sampling four simplifications per input for a more reliable score estimate.
- Prompt strategies: The strategies ranged from a minimal instruction and role-separated zero-shot prompt to few-shot examples and explicit simplification rules.The few-shot prompt used three manually selected iDEM examples, while the rule-based prompt drew on European Easy-to-Read guidelines.
- Selection: The retained rule-based configuration combined the highest-scoring rule set with few-shot examples in the system prompt.
- Selected prompt: The selected instructions required Catalan simplification while preserving exactly the original meaning.
- Selected prompt: The rules emphasized active voice, repeated important information when needed, accessible vocabulary, clear explanations of difficult words, and avoiding metaphors.
Appendix B iDEM Statistical Significance
Appendix B evaluates pairwise SARI differences on iDEM with paired bootstrap confidence intervals. Significance is limited to comparisons involving the English-trained model, while translated variants appear equivalent within benchmark resolution.
- Statistical procedure: Table 2 reports pairwise mean SARI differences with 95% paired bootstrap confidence intervals over 304 shared iDEM sentences.Significance is determined when the interval excludes zero, corresponding to p < 0.05.
- Significant comparisons: Only comparisons involving the English-trained model reach statistical significance, although the English-versus-Original improvement may be modest.
- Translated variants: The Spanish-versus-Catalan interval is narrow enough to exclude differences larger than roughly one SARI point in either direction.
- Translated variants: Within this benchmark’s resolution, the translated variants appear equivalent, with separation occurring between translated and untranslated corpora rather than between languages.The experiments do not establish why this pattern occurs.