PaperlayerView exact revision on arXiv

Source-linked AI summary

When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation

Muhammad Ashad Kabir, Kawsar Ahmed, Md. Osama

arXiv:2608.15879v1cs.CL

TL;DR

Bengali news headline generation remains sensitive to how context and prompts are selected and presented, especially for long articles. This study evaluates selective context conditioning and prompting strategies across LLMs, finding that relevant lead paragraphs can match or exceed full-document conditioning while prompt and few-shot benefits vary by model.

  • Problem

    The interaction of selective context conditioning and multilingual prompting strategies remains underexplored for low-resource Bengali headline generation.

  • Method

    The study analyzes headline–paragraph semantic alignment and evaluates conditioning on the first one to five paragraphs alongside prompting strategies and few-shot demonstrations.

  • Results

    Selected lead paragraphs can match or outperform full-document conditioning, while XLP and few-shot prompting show model-dependent benefits.

  • Takeaways & Limitations

    Effective Bengali headline generation depends more on contextual relevance and prompt design than on increasing input length.

  • Takeaways & Limitations

    The study covers few models, and prompting effectiveness is sensitive to formulation, demonstration choices, ordering, and random variation.

Abstract

from arXiv · show

Large language models (LLMs) have shown strong performance in text generation tasks, yet their effectiveness on headline generation remains sensitive to how input context is selected and presented. In this work, we investigate Bengali news headline generation as a document-level generation task that requires effective selection and presentation of salient contextual information from long-form articles. Using Gemini-2.0-Flash, Llama-3.3-70B, and GPT-4o, we systematically study the effects of context selection, prompting strategies, and in-context learning (i.e., few-shot) on the quality of headline generation. Our experiments show that providing the full article does not necessarily improve performance; instead, using selected lead paragraphs of the article can maintain, and in some cases improve, headline generation quality. We further compare Bengali Native Prompting (BNaP) and Cross-Lingual Prompting (XLP), and examine how each interacts with context-enriched prompt templates incorporating auxiliary contextual cues. Results demonstrate that prompting strategies substantially influence generation quality: XLP often yields stronger performance, particularly when combined with contextual enrichment, but its benefits are model-dependent. Additionally, few-shot prompting substantially improves Gemini, with most of the gain obtained from a single demonstration, whereas Llama shows limited benefit from additional examples. Overall, our findings highlight that effective Bengali news headline generation depends more on context relevance and prompt design than on increasing input length, offering practical insights for multilingual and low-resource LLM applications.

1 Introduction

The introduction frames Bengali news headline generation as sensitive to context-window limitations, context selection, and prompting design. It motivates selective use of lead paragraphs and systematic comparison of prompting paradigms and demonstration settings.

  • Motivation: LLMs may lose essential information or degrade on long documents because of context-window limits, loss of focus, semantic dilution, and capacity overflow.The introduction also situates this issue within position bias, where models disproportionately prioritize information from certain document locations.
  • Context Selection: Semantic-similarity analysis finds that opening article paragraphs consistently contain the most informative content relative to gold headlines.The analysis uses sentence-transformer-based representations to compare gold headlines with individual article paragraphs.
  • Context Selection: The proposed selective conditioning strategy prompts LLMs with the first one through five paragraphs instead of the full document to reduce unnecessary contextual noise.This setup tests whether relevance-focused context selection can improve headline generation quality while using less input.
  • Prompting Strategies: The study compares Bengali Native Prompting and Cross-Lingual Prompting with Baseline and enriched MultiGen templates containing sentiment, aspect, category, and definition-based cues.The prompting paradigms are evaluated using news content alone or with additional contextual information.
  • Contributions: Selective lead-paragraph conditioning can match or outperform full-document conditioning, while few-shot gains and sensitivity to demonstrations vary across LLMs.The introduction identifies context relevance, in-context learning, and model-dependent prompting effects as central findings.

2 Related Work

Prior work established transformer and instruction-tuned LLM approaches for headline generation, while prompting, context length, and multilinguality shape generation quality. Bengali headline generation remains under-explored, particularly regarding selective context conditioning and enriched multilingual prompts.

  • Model foundations: BART and T5 achieved strong abstractive summarization performance and were benchmarked for headline generation and news summarization across diverse datasets.
  • Prompting strategies: Instruction-tuned LLMs have demonstrated strong generalization in zero-shot and few-shot headline generation through demonstration-based prompting.
  • Prompting strategies: Prompt enrichment with sentiment, category, and aspect metadata has been explored to improve relevance and controllability in news-related generation tasks.
  • Context selection: Excessively long or redundant prompts can dilute attention over salient information and degrade generation quality, motivating selective context conditioning.
  • Bengali headline generation: Bengali headline generation remains under-explored, while prior Bengali studies combined auxiliary information with full article content rather than testing simple position-based lead context.

3 Methodology

The study uses two Bengali news benchmarks and investigates relevance-focused context selection, prompting paradigms, contextual enrichment, and in-context learning for headline generation. Experiments evaluate selected multilingual LLMs with standardized decoding and multiple headline-quality metrics.

  • Datasets: BeliN contains 2,520 curated Bengali religious news articles with six fields, while Shironaam provides diverse Bengali news with category, topic words, and image-caption context.BeliN is the primary dataset; Shironaam is the secondary benchmark.
  • Context Selection: Paragraph relevance is estimated by banBERT1 cosine similarity between headline and paragraph embeddings, motivating conditioning on initial paragraphs instead of full articles.The most semantically aligned paragraph is identified per article, and early paragraphs show the strongest alignment with reference headlines.
  • Context Selection: Partial context varies k initial paragraphs to test whether relevance-focused inputs can outperform full-document conditioning.Here, k ∈ {1, 2, 3, …} denotes the number of supplied initial paragraphs.
  • Prompting Strategy: The prompting study crosses Bengali Native Prompting and Cross-Lingual Prompting with Baseline and MultiGen templates, producing four configurations.MultiGen adds concise definitions and Category, Aspect, and Sentiment metadata, whereas Baseline contains only article content.
  • In-Context Learning: In-context learning is tested with zero-shot, one-shot, two-shot, and three-shot prompts, each containing complete article-headline demonstrations.Demonstrations are randomly sampled from training data and kept consistent across models and prompting strategies for each shot setting.
  • Evaluation: Generated headlines are assessed with BERTScore, METEOR, and ROUGE-L using means, standard deviations, and 95% BCa bootstrap confidence intervals from 10,000 resamples.Main experiments use the official BeliN test set of 505 articles, with Shironaam evaluation on 15,012 test articles.

4 Results and Discussion · 4.1 Impact of Selective Context Conditioning

Headline-related information is concentrated in the opening paragraphs, and selective context can preserve or improve generation quality while substantially reducing input length. Model responses differ: Gemini benefits from very short lead context, whereas Llama remains stable across broader context scopes.

  • 4.1.1 Semantic Alignment Between Headlines and Article Paragraphs.: 56.48% of articles place the paragraph with highest headline similarity first, reflecting positional concentration of salient information.The analysis uses banBERT embeddings on 2,015 BeliN training articles and aligns this pattern with the inverted-pyramid structure.
  • 4.1.1 Semantic Alignment Between Headlines and Article Paragraphs.: 68.29% of articles contain their highest-similarity content within the first two paragraphs, rising to 75.98% within the first three.After the third paragraph, the likelihood of finding the most semantically aligned content decreases substantially.
  • 4.1.1 Semantic Alignment Between Headlines and Article Paragraphs.: The first paragraph has both the highest frequency and median similarity to reference headlines, followed by declining similarity through Paragraph 3.Similarity distributions stabilize at lower levels after Paragraph 3, although some articles contain salient later information.
  • 4.1.2 Contextual Salience vs. Information Volume.: Gemini achieves its best result with paragraph1, scoring 0.727 BERTScore and improving over full-article input by 0.0078 points.The improvement is statistically significant, with 95% CI: 0.003–0.012 and p= 0.0011.
  • 4.1.2 Contextual Salience vs. Information Volume.: 52.6% reduction lowers average input length from 1011.60±665.89 to 479.35±398.29 tokens when using the first three paragraphs.This comparison covers the full BeliN test set, including 126 articles with three or fewer paragraphs.
  • 4.1.2 Contextual Salience vs. Information Volume.: 65.1% reduction lowers average input length from 1089.44±683.51 to 379.82±267.93 tokens for articles exceeding three paragraphs.The first-three-paragraph setting maintains comparable Llama performance, while Gemini benefits from an even shorter first-paragraph context.
  • 4.1.2 Contextual Salience vs. Information Volume.: Llama changes only 0.001 BERTScore points from paragraph1→3 (0.755) to the full article (0.754), whereas Gemini shows no consistent gain from added paragraphs.The results indicate model-specific sensitivity to contextual salience and suggest that longer inputs can add redundant or less relevant information.

4.2 Effect of In-Context Learning · 4.3 Effect of Prompting Strategies

In-context learning benefits Bengali headline generation unevenly across models, with Gemini gaining substantially from one demonstration while Llama remains comparatively stable. Prompt language and contextual enrichment also interact in model-dependent ways, with XLP especially effective for Gemini and Llama when paired with MultiGen.

  • 4.2 Effect of In-Context Learning: The ICL evaluation used BNaP+Baseline with partial article input (paragraph1→3) across zero-shot, one-shot, two-shot, and three-shot settings.The comparison focused on Gemini and Llama.
  • 4.2 Effect of In-Context Learning: 0.04 BERTScore points was Gemini’s statistically significant mean paired improvement from zero-shot to three-shot prompting.BERTScore rose from 0.72 (95% CI: 0.715–0.725) to 0.76 (95% CI: 0.754–0.766), with 𝑝< 0.001.
  • 4.2 Effect of In-Context Learning: 0.761 was Llama’s highest BERTScore under one-shot prompting, while its zero-shot-to-one-shot improvement was not statistically significant.The change was Δ BERTScore = 0.0056, 95% CI: −0.0024 to 0.0138; 𝑝= 0.252.
  • 4.2 Effect of In-Context Learning: Neither model consistently benefited from increasing demonstrations beyond one, indicating model-dependent ICL effectiveness.Gemini improved substantially from zero-shot to one-shot, while Llama declined slightly after its one-shot peak.
  • 4.3 Effect of Prompting Strategies: Prompting-strategy comparisons fixed three-shot prompting and used paragraph1→3 context across Gemini, Llama, GPT-4o, and prompt variants.Three-shot prompting was selected as a controlled common setting rather than as an optimal configuration for every model.
  • 4.3 Effect of Prompting Strategies: XLP produced higher BERTScores than BNaP for both Llama and Gemini across Baseline and MultiGen variants.Under Baseline, Llama increased from 0.754 (95% CI: 0.749–0.760) to 0.758 (95% CI: 0.752–0.764), while Gemini increased from 0.760 (95% CI: 0.754–0.766) to 0.767 (95% CI: 0.760– 0.773).
  • 4.3 Effect of Prompting Strategies: 1.5% BERTScore gains occurred for both Llama and Gemini when XLP replaced BNaP under MultiGen.The gains were statistically significant: Llama Δ BERTScore = 0.012, 95% CI: 0.0036–0.02; 𝑝= 0.0038, and Gemini Δ BERTScore = 0.011, 95% CI: 0.004–0.018; 𝑝= 0.003.
  • 4.3 Effect of Prompting Strategies: Contextual enrichment helped under XLP but not BNaP, while GPT-4o performed slightly better with BNaP than XLP.For Gemini, XLP+MultiGen reached 0.769 (95% CI: 0.763–0.776), improving over 0.767 (95% CI: 0.760–0.773) with Δ BERTScore = 0.0025, 95% CI: 0.0023–0.0074, 𝑝= 0.0148.

4.4 Comparison with Fine-Tuned Models

Prompting-based in-context learning with Gemini outperforms the best reported fine-tuned BanglaT5 model on the Shironaam benchmark across all evaluation metrics, without task-specific training. The approach relies on prompt design and in-context examples rather than supervised fine-tuning.

  • Comparison setup: The study compares general-purpose LLM prompting with fine-tuned mT5, mBART, and BanglaT5 models from prior work.Prior models used supervised fine-tuning and MultiGen contextual enrichment, whereas this approach uses prompt design and in-context examples.
  • Overall comparison: Gemini with XLP+MultiGen and three-shot prompting scores higher than the best-performing fine-tuned BanglaT5 model across all evaluation metrics.The comparison is conducted on the Shironaam benchmark.
  • Metric gains: BERTScore improves from 0.751 to 0.769, ROUGE-L from 0.242 to 0.292, and METEOR from 0.167 to 0.244.The BERTScore change corresponds to a relative improvement of 2.4%.

4.5 Evaluation on a Second Benchmark · 4.6 Limitations

Evaluation on the Shironaam benchmark shows that fine-tuned BERT2BERT outperforms Gemini on several automatic metrics, while prompting-based adaptation remains a competitive training-free alternative. The study is limited by model, prompt, reproducibility, evaluation, and dataset constraints.

  • 4.5 Evaluation on a Second Benchmark: 15,012 articles comprise the Shironaam test dataset used to assess transferability beyond BeliN.The comparison uses Gemini with three-shot XLP+MultiGen prompting without task-specific parameter updates.
  • 4.5 Evaluation on a Second Benchmark: BERT2BERT achieves a BERTScore of 0.831, compared with 0.810 for Gemini.BERT2BERT also scores higher on ROUGE-L and METEOR.
  • 4.5 Evaluation on a Second Benchmark: Fine-tuned encoder–decoder models remain stronger for maximizing task-specific performance, whereas selective context conditioning and in-context learning offer a flexible alternative without additional training.The comparison reflects supervised task-specific optimization and exposure to labeled training data.
  • 4.5 Evaluation on a Second Benchmark: Carefully designed prompting strategies transfer across datasets, although supervised fine-tuning remains stronger on several metrics.This conclusion summarizes the complementary strengths observed between supervised fine-tuning and prompting-based adaptation.
  • 4.6 Limitations: The analysis covers only a small set of proprietary and open-source LLMs, so trends may vary across other multilingual models and prompting configurations.Prompt effectiveness also depends on formulation, demonstration selection, ordering, and random variation across repeated runs.
  • 4.6 Limitations: Prompt sensitivity and run-to-run variation may affect reproducibility and generalizability across datasets and tasks.These factors include prompt formulation, demonstration selection, and demonstration ordering.
  • 4.6 Limitations: Evaluation relies primarily on automatic metrics without large-scale human assessment of factual consistency, fluency, or editorial quality.Proposed directions include human-centered evaluation, adaptive context selection, and automated prompt optimization.
  • 4.6 Limitations: The study should be extended to larger, more diverse datasets and multimodal or retrieval-augmented settings.Such extensions would strengthen understanding of selective context conditioning for multilingual headline generation.

5 Conclusion

The study finds that Bengali headline generation benefits more from context relevance and prompt design than from simply increasing input length. Selective lead-paragraph conditioning and model-specific prompting can support competitive performance in a low-resource setting without task-specific fine-tuning.

  • Context selection: Selected lead paragraphs can maintain comparable headline-generation performance to full-document conditioning and improve results for some models.The findings link this advantage to the semantic alignment of initial article paragraphs with headline content.
  • Prompting strategies: Prompting strategies substantially affect headline quality, with XLP often improving Gemini and Llama while remaining model-dependent.BNaP benefits from concise, minimally structured prompts.
  • In-context learning: A single demonstration captures most of Gemini’s observed in-context-learning improvement, whereas additional examples provide limited benefit for Llama.The contribution of in-context learning is strongly model-dependent.
  • Overall implications: Carefully designed prompting and selective context conditioning can enable competitive headline generation for low-resource languages without task-specific fine-tuning.The conclusion emphasizes contextual salience, prompt design, and model-specific behavior in multilingual document-level generation.

A Headline-Paragraph Semantic Alignment

The analysis identifies each article paragraph most semantically aligned with its reference headline using sentence-embedding scores. This training-set characterization motivates position-based context settings that select the first k article paragraphs independently of test reference headlines.

  • Role in context selection: Algorithm 1 is used only for training-set semantic-alignment analysis, not for selecting paragraphs during test-time headline generation.Its output characterizes where headline-relevant information occurs in the training data.
  • Role in context selection: The resulting distribution motivates position-based settings that independently select the first k article paragraphs during evaluation.Selection is based on paragraph position rather than the test reference headline.
  • Alignment procedure: The most-aligned paragraph is selected as the paragraph with the maximum similarity score, with its index and score recorded.The procedure applies arg max over paragraph similarity scores and stores the resulting index, paragraph, and score.

B Prompt Templates

The section presents baseline prompt templates for XLP and BNaP, alongside enriched multi-generation templates that add contextual cues such as category, topic words, image captions, articles, and headlines. The templates cover both one-shot prompting with Shironaam examples and corpus-specific variants for BeliN and Shironaam.

  • Baseline templates: Tables 8 and 9 define separate baseline prompt templates for XLP and BNaP.These templates establish the two prompting-language variants used in the study.
  • Context-enriched templates: The XLP+MultiGen+1-shot template incorporates Shironaam contextual cues and includes a demonstration with category, topic words, image caption, article, and headline fields.The demonstration uses example-specific placeholders before the target news input.
  • Context-enriched templates: The target portion of the enriched template retains fields for category, topic words, image caption, article, and headline generation.The headline field is left open for model completion.
  • Corpus-specific variants: Additional BNaP+MultiGen and XLP+MultiGen templates incorporate contextual cues from the BeliN corpus.These corpus-specific variants are listed in Tables 11 and 12.

Ethics and Privacy Statement

The study uses existing Bengali benchmark datasets without human-subject experiments, user interaction, or new personal-data collection. It urges privacy and licensing compliance in deployment and warns of large-scale misleading or manipulative headline generation.

  • Data and privacy: The study uses previously released Bengali news benchmarks and involves no human-subject experiments, user interaction, or collection of new personal data.It also does not identify individuals, infer sensitive attributes, or generate user-specific products.
  • Deployment safeguards: Real-world deployment should comply with dataset licenses, data-use conditions, and applicable privacy regulations, particularly for non-public or user-provided documents.Model outputs are generated only from the provided article context.
  • Misuse risks: The technology could be misused to generate misleading, clickbait-style, or politically manipulative headlines at scale.The statement therefore recommends safeguards for headline-generation systems.
Loading 2608.15879v1…