PaperlayerView exact revision on arXiv

Source-linked AI summary

Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric

Sami Shames El Deen, Mariette Awad

arXiv:2608.20964v1cs.CLcs.AI

TL;DR

Arabic extractive summarization requires models and evaluation methods suited to Arabic text and document-level meaning. The paper introduces SAraBERT and Semantic Siamese Similarity, and reports that SAraBERT with RNN encoding layers achieved the best performance among evaluated models while SSS closely correlated with BertScore.

  • Problem

    Arabic extractive summarization has limited research coverage, and syntactic metrics may not capture contextual similarity between summaries and references.

  • Method

    The paper enhances AraBERT with inter-sentence transformer layers, scores sentence representations with interchangeable encoders, and proposes Semantic Siamese Similarity using contextualized embeddings and exact matching.

  • Results

    SAraBERT with RNN encoding layers achieved the best performance, while Semantic Siamese Similarity correlated 0.97 with BertScore and was stricter in evaluation.

  • Takeaways & Limitations

    The study reports effective Arabic extractive summarization with SAraBERT and a similarity metric that evaluates semantic as well as syntactic similarity.

Abstract

from arXiv · show

In this research, we introduce SAraBERT, an enhanced version of AraBERT which proposes inter-sentence transformer layers for extractive summarization tasks. To ensure that the summaries generated by SAraBERT achieve a high coverage of the document's main ideas, we propose Semantic Siamese Similarity, a novel evaluation metric that measures the level of similarity between two text inputs. We validated using BLEU, ROUGE, and Semantic Siamese similarity on Sarabert and published related models. Simulation results showed the effectiveness of our proposed model and motivate follow on research.

I. INTRODUCTION

The paper addresses limited Arabic summarization research by introducing SAraBERT for extractive summarization and Semantic Siamese Similarity for evaluating semantic and syntactic similarity.

  • Arabic summarization remains underrepresented, while Arabic NLP presents added challenges from rich morphology, absent diacritics, and no capitalization.
  • SAraBERT extends AraBERT with inter-sentence transformer layers for Arabic extractive summarization.
  • The paper evaluates SAraBERT and related models using BLEU, ROUGE, and Semantic Siamese Similarity.
  • Semantic Siamese Similarity is a hybrid metric combining semantic embeddings with exact matching to measure similarity between two texts.
  • The study uses an Arabic corpus translated from English for extractive summarization and Question Answering, with availability upon request.

II. LITERATURE REVIEW

Earlier Arabic summarization research used statistical, machine-learning, and sequence-to-sequence approaches, while prior work identified dataset coverage as an ongoing need.

  • Classical Approaches: Statistical summarization methods predominantly used relevance scores, Bayesian classifiers, word frequency, and TF-IDF before 2017.
  • Machine Learning Approaches: Extractive summarization can be formulated as binary sentence classification using statistical features, semantic features, or both.
  • Sequence-to-Sequence Models: Sequence-to-sequence and attention-based architectures were applied to headline generation and article summarization, including Arabic summarization.
  • Sequence-to-Sequence Models: Prior Arabic sequence-to-sequence work required broader datasets covering more articles.

C. Other Approaches

The section reviews related Arabic summarization approaches and describes SAraBERT’s sentence-level architecture for extractive summarization. SAraBERT represents sentence boundaries and uses sentence embeddings with an encoder to produce sentence scores.

  • Other Approaches: Arabic summarization research has explored sequence-to-sequence, encoder-decoder, and hybrid transformer approaches, including extractive selection followed by abstractive generation.A cited hybrid approach used AraBERT sentence embeddings for extraction and an Arabic MT5 transformer for abstraction.
  • Other Approaches: 53% precision, 55% recall, and 49% F1 were reported for the cited hybrid approach on the ESAC dataset.The work used 153 articles and 700 summarizations, and its pretrained models were not fine-tuned specifically for summarization.
  • SAraBERT: SAraBERT formulates extractive summarization as assigning each document sentence a binary label indicating whether it belongs in the summary.AraBERT’s masked language modeling and next sentence prediction objectives support the underlying sentence representations.
  • SAraBERT: SAraBERT inserts sentence-boundary tokens, alternates sentence segment embeddings, and adds a summarization layer that derives sentence scores from [CLS] embeddings.The model differs from the original BERT architecture through these sentence-level modifications.
  • SAraBERT: The [CLS] embeddings are passed to an encoder H, which outputs a scoring vector with one score for each sentence.The encoder H is interchanged among MLP, RNN, and Transformer models to compare scoring criteria.

B. Siamese Semantic Similarity (SSS)

The proposed Semantic Siamese Similarity metric combines lexical overlap with contextual embedding similarity for comparing candidate and reference summaries. It further accounts for embedding distance and word-frequency differences to address limitations of individual similarity measures.

  • Motivation: ROUGE measures syntactic similarity but does not capture contextual coverage when summaries express similar content with different wording.Contextualized language-model embeddings provide a fixed-size representation for semantic comparison.
  • Metric: Embedding distance is placed in the denominator because smaller distance should produce a higher similarity score.A square root slows the growth of the distance term as the difference increases, while adding 1 prevents division by zero.
  • Metric: The ROUGE component averages ROUGE-1, ROUGE-2, and ROUGE-L to retain attention to grammar.ROUGE remains part of the formula alongside semantic similarity.
  • Metric: The frequency term penalizes differences in word-frequency distributions because embeddings do not account for redundancy and coverage.This addresses cases where two sentences are contextually similar but differ in redundancy.
  • Metric: SSS combines cosine similarity between AraBERT embeddings with ROUGE-based overlap, embedding distance, and frequency-difference terms.The algorithm encodes candidate and reference texts, computes ROUGE components, and combines the resulting quantities into a similarity score.

1) CNN/ Daily Mail:

The CNN/Daily Mail dataset provides large-scale English news articles with author-written highlights, and the study translates these documents into Arabic for SAraBERT training. Readability analyses compare English, Arabic, and diacritized Arabic, finding no measurable effect on model output.

  • Dataset: The CNN/Daily Mail dataset contains over 300,000 unique English news articles from CNN and Daily Mail journalists.Each instance includes an identifier, article body, and author-written highlights.
  • Dataset: The dataset supports training for both extractive and abstractive summarization.
  • Arabic preparation: The study evaluates translated Arabic documents and separately examines whether diacritics affect readability.
  • Readability analysis: Readability differences between original English and translated Arabic documents do not affect model output or correlate with ROUGE.This conclusion is based on experiments over 1,000 translated samples.

V. RESULTS

The study compares summarization models using BLEU, ROUGE, and Siamese Similarity, with human-summary oracle scores serving as a comparison reference. SAraBert with an RNN encoder performs best across the three evaluation metrics.

  • Evaluation results: SAraBert+RNN is the best-performing model across BLEU, ROUGE, and Siamese Similarity.BLEU averages four n-gram evaluations, while ROUGE averages ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.
  • Evaluation metrics: BLEU, ROUGE, and Siamese Similarity evaluate model summaries, while Oracle Score evaluates human summaries for comparison.
  • Evaluation results: The results are reported for different models under several evaluation metrics.

VI. DISCUSSION

Discussion attributes the strongest observed encoder performance to BiLSTM outperforming the transformer encoder, while identifying document length as an unresolved modeling constraint. The conclusion reiterates that SAraBert with RNN encoding achieves the best performance and that SSS evaluates semantic similarity.

  • Encoder comparison: BiLSTM performs better than the transformer encoder in the reported comparison.The authors suggest transformers may require more data to train their attention heads and layers.
  • Limitation: Very large documents cannot be fed into the models as a single global lookup without segmenting passages and losing linked contexts.
  • Conclusion: SAraBert with RNN encoding achieves the best performance on the summarization task for Modern Standard Arabic.
  • Conclusion: The proposed similarity metric evaluates similarity at the semantic level rather than only the syntactic level.

APPENDIX

The appendix tests Semantic Siamese Similarity against BertScore on candidate summaries and presents sample outputs from multiple encoders. It also documents the experimental basis for evaluating the metric.

  • Siamese Semantic Similarity Experiments: The metric is tested on multiple candidate cases to visualize and evaluate its overall performance.
  • Siamese Semantic Similarity Experiments: SSS is applied to candidate summaries against a reference, with BertScore and intermediate computations reported for comparison.
  • Siamese Semantic Similarity Experiments: SSS and BertScore have a Pearson correlation of 0.97, although SSS is stricter in its evaluations.
  • Sample summaries: The appendix presents sample summaries generated by SAraBert using MLP, RNN, and Transformer encoders.
  • Sample summaries: Additional examples show good and bad summarization produced by SAraBert with a BiLSTM encoder.

C. Translation Quality

This section presents sample Arabic and English passages with readability scores and examines whether translation readability affects ROUGE results. It also identifies the tables organizing reference/candidate sentences, correlations, and document averages.

  • Translation Quality: Translations were assessed through sample Arabic and English passages paired with Osman and Flesch readability scores.The examples include Arabic scores of 66.73 and 88.01 and English scores of 83.69 and 11.38.
  • Translation Quality: Table X lists one reference sentence and candidate sentences numbered 2 through 19.
  • Translation Quality: No readability metric highly affects ROUGE across 1000 sampled documents.The study computed Arabic and English readability metrics alongside the ROUGE value of each generated summary.
  • Translation Quality: Table XIII presents correlations between different features, while Table XII reports average document values over 1000 samples.
Loading 2608.20964v1…