Source-linked AI summary
Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study
Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh, Mohsin Altaf Wani, Abid Hussain Wani, Hilal Ahmad Khanday, Niyaz Ahmad Wani
TL;DR
The available Hindi benchmarks use reference styles that strongly reward article openings, leaving semantic content selection difficult to assess on its own terms. This study adapts a distributional-semantics extractive method to Hindi and evaluates it on XL-Sum and ILSUM; the equal-weight system trails Lead-3 on both, while tuning never surpasses it.
Problem
Whether semantic extractive selection can be evaluated on its own terms remains unsettled because the available Hindi benchmarks reward article openings in their references.
Source: §5 · p. 6Method
The study adapts a seven-feature distributional-semantics extractive summariser to Hindi and evaluates three-sentence extracts on Hindi XL-Sum and ILSUM with validated ROUGE scoring.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors conclude these benchmarks cannot distinguish good content selection from a positional heuristic and motivate Hindi summaries that draw on the article beyond its lead.
Source: §5 · p. 6The evidence is limited to XL-Sum Hindi and ILSUM 2.0 Hindi at a three-sentence budget under instrument-validated ROUGE, and does not establish that distributional extraction lacks value generally.
Source: §5 · p. 6
Abstract
from arXiv · showhide
We replicate the distributional-semantics extractive summarisation method of Mohd, Jan and Shah (2020) and adapt it to Hindi, substituting a Devanagari-appropriate component at every language-specific step. The system is evaluated on two independent corpora --- the Hindi portion of XL-Sum and FIRE ILSUM 2.0 Hindi --- under a Devanagari-aware ROUGE implementation validated against the XL-Sum authors' own multilingual scorer, with all comparisons drawn as 1000-resample paired bootstraps. In its published equal-weight configuration the replicated system is significantly worse than a three-sentence lead baseline on both corpora, trailing Lead-3 by 0.042 ROUGE-1 Fon XL-Sum and by 0.265 on ILSUM. A feature ablation shows that sentenceposition is the only feature that contributes: position alone reproduces the lead baseline exactly, removing position gives the weakest configuration,and a validation-tuned weighting can at best equal Lead-3 and never exceed it. TextRank fails identically, making this a class-level rather than an implementation-level result. A selection analysis shows the remaining features steer extraction towards long, entity-dense body sentences while the references reuse the article lead.Current Hindi benchmarks therefore cannot reward non-lead content selection, motivating purpose-built evaluation resources.
1 Setup
The study replicates a seven-feature distributional-semantics extractive summariser and adapts its language-specific steps for Hindi. It evaluates three-sentence extracts on two Hindi corpora using a validated Devanagari-aware ROUGE protocol.
1 Setup
- The replicated pipeline embeds sentences, clusters them with k-means, ranks them by seven equally weighted features, then selects sentences round-robin across clusters.The features cover length, position, TF–IDF mass, part-of-speech counts, proper-noun count, document-level embedding cosine, and cue phrases.Source: §1 · p. 1
- Hindi-specific processing uses NFC normalisation, danda-aware sentence splitting, Hindi stop words, a light suffix stemmer, Stanza Hindi, and 300-dimensional fastText Common Crawl vectors.Part-of-speech features remain enabled throughout the reported results, so “our system” denotes the full seven-feature configuration.Source: §1 · p. 2
- On Hindi XL-Sum and FIRE ILSUM 2.0, systems produce three-sentence extracts; tuning uses validation data separate from testing, and 95% confidence intervals use 1,000-resample document bootstraps with paired shared-resample comparisons.ROUGE-1 F is the primary significance quantity because it is least sparse at a three-sentence budget. Scores are reported for ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-SU4 F-measure.Source: §1 · p. 2
- The Devanagari-aware ROUGE implementation matches the independent XL-Sum scorer to 0.0000 mean absolute difference on ROUGE-1, ROUGE-2, and ROUGE-L F across 20 documents.The comparison covers three systems; the external scorer does not compute ROUGE-SU4.Source: §1 · p. 2
2 Main result
On both Hindi corpora, the published equal-weight system significantly trails Lead-3, and TextRank shows the same direction of failure. Paired comparisons support the shortfalls, which the authors interpret as a class-level result rather than a defect specific to their reimplementation.
2 Main result
- 0.186 ROUGE-1 F with clustering and 0.185 without clustering on XL-Sum, and 0.256 on ILSUM, trail Lead-3’s 0.228 and 0.522, respectively.The XL-Sum figures are from the ablation set; the ILSUM figure is from the same equal-weight configuration.Source: §2 · p. 2, Table 1 · p. 3, Table 3 · p. 3
- 0.042–0.043 ROUGE-1 F on XL-Sum and 0.265 on ILSUM are the equal-weight system’s paired shortfalls against Lead-3, with intervals bounded away from zero.The paired-gap tables report the comparisons; the prose states that the ILSUM interval does not approach zero.Source: §2 · p. 2, Table 2 · p. 3, Table 4 · p. 3
- TextRank also significantly trails Lead-3, by 0.025 on XL-Sum and 0.254 on ILSUM, supporting a class-level failure on these corpora rather than an implementation-specific defect.The authors also point to agreement with the XL-Sum authors’ scorer as evidence for this interpretation.Source: §2 · pp. 2, 4
3 Ablation
Validation-tuned weighting recovers the equal-weight system’s lost ground but never surpasses Lead-3. Position alone reproduces the lead baseline, while removing position produces the weakest configuration.
3 Ablation
- 0.228 ROUGE-1 F for Ours-position-only matches Lead-3, while Ours-minus-position scores 0.182, 0.046 below Lead-3.Position alone selects the article opening; the full equal-weight system scores 0.185–0.186 as position’s advantage is averaged across seven features.Source: §3 · p. 4
- On XL-Sum, seven-feature ROUGE-1 F rises from 0.191 at unit position weight to 0.235 at weight sixteen; position+TF–IDF rises from 0.211 to 0.239.On ILSUM, position+TF–IDF rises from 0.347 at unit weight to 0.560 at weight sixteen; the equal-weight setting is the worst point on each curve.Source: §3 · p. 4, Table 5 · p. 4
- 0.231 versus Lead-3’s 0.231 on XL-Sum and 0.516 versus 0.522 on ILSUM leave tuned position+TF–IDF tied or slightly behind, never ahead.The XL-Sum paired gap is −0.000 (95% interval [−0.002, +0.001]); ILSUM’s is −0.005 ( [−0.011, −0.000]).Source: §3 · p. 4, Table 6 · p. 5
4 Mechanism
Selected-sentence positions show that position-only extraction tracks the article opening, while the other features favor later, lexically dense content. In one example, the nonposition features select squad lists rather than the lead’s match framing.
4 Mechanism
- Position-only and Lead-3 each draw 0.71 of selected sentences from the first decile, with a mean normalised position of 0.070; minus-position averages 0.501.The equal-weight systems average roughly 0.39, with 0.20 of selections in the first decile, compared with the lead’s 0.71.Source: §4 · p. 5, Table 7 · p. 5
- In document 179, Lead-3 selects the opening match framing, whereas minus-position selects two squad lists near the article’s end.The lists’ length, proper nouns, recurring names, TF–IDF mass and embedding cosine align with the features rewarded by the system.Source: §4 · pp. 5–6
- The nonposition features favor long, entity-dense body and tail sentences, which can carry little of the reference’s summary-worthy content.The passages identify length, proper-noun count, TF–IDF mass and aggregate embedding cosine as features that reward the squad lists in the example.Source: §4 · pp. 5–6
5 Diagnosis
The diagnosis is that these Hindi benchmarks favor lead reproduction over non-lead content selection, while the evidence is bounded to two corpora and a three-sentence budget. The paper proposes a purpose-built Hindi resource with longer, multi-reference summaries drawing across articles and sentence-level faithfulness labels.
5 Diagnosis
- ILSUM Lead-3 reaches ROUGE-2 F of 0.451 and ROUGE-L F of 0.499, while XL-Sum references are single lead-style sentences, favoring lead reproduction over non-lead content selection.The paper argues that these reference designs leave the benchmarks unable to reward correctly selected salient non-lead content.Source: §5 · p. 6
- The conclusion is limited to XL-Sum Hindi and ILSUM 2.0 Hindi at a three-sentence budget under instrument-validated ROUGE; other summary lengths are untested.The authors also caution that lexical n-gram overlap can compound lead-reference bias by rewarding token reuse over paraphrase.Source: §5 · p. 6
- A proposed Hindi resource would use longer human-written summaries drawing from across articles, multiple references per document, and sentence-level faithfulness labels.The paper proposes scoring such a resource with complementary measures such as ChrF and BERTScore alongside human references, rather than ROUGE alone.Source: §5 · p. 6