PaperlayerView exact revision on arXiv

Source-linked AI summary

PosterSum: A Multimodal Benchmark for Scientific Poster Summarization

Rohit Saxena, Pasquale Minervini, Frank Keller

arXiv:2502.17540v1cs.CVcs.AIcs.CL

TL;DR

Scientific poster summarization requires integrating complex visual and textual information, a challenge for which specialized datasets have been lacking. The paper introduces POSTERSUM and benchmarks MLLMs, then proposes a hierarchical method that outperforms current MLLMs on automated metrics. The dataset and results offer a resource for future research on poster summarization.

  • Problem

    Scientific posters combine complex layouts, technical text, tables, and figures, while poster summarization has been under-explored for lack of specialized datasets.

    Source: §1 · p. 1

  • Method

    POSTERSUM pairs 16,305 posters with abstracts, and SEGMENT & SUMMARIZE segments posters, summarizes regions locally, and composes a global abstract.

    Source: §1 · p. 2

  • Results

    24.18 ROUGE-L is achieved by SEGMENT & SUMMARIZE, outperforming both open-source and closed-source models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    POSTERSUM is presented as a resource for developing and evaluating MLLMs that process information-dense scientific content.

    Source: §8 · p. 8

    The method assumes posters can be meaningfully decomposed into spatial regions, which may not hold for complex cross-referencing or interdependent visual elements.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Generating accurate and concise textual summaries from multimodal documents is challenging, especially when dealing with visually complex content like scientific posters. We introduce PosterSum, a novel benchmark to advance the development of vision-language models that can understand and summarize scientific posters into research paper abstracts. Our dataset contains 16,305 conference posters paired with their corresponding abstracts as summaries. Each poster is provided in image format and presents diverse visual understanding challenges, such as complex layouts, dense text regions, tables, and figures. We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on PosterSum and demonstrate that they struggle to accurately interpret and summarize scientific posters. We propose Segment & Summarize, a hierarchical method that outperforms current MLLMs on automated metrics, achieving a 3.14% gain in ROUGE-L. This will serve as a starting point for future research on poster summarization.

1 Introduction

Scientific poster summarization is challenging because models must integrate complex layouts, technical text, tables, and figures, and the task has been under-explored without specialized datasets. The paper introduces POSTERSUM and benchmarks MLLMs, then proposes a hierarchical summarization method that outperforms existing models.

1 Introduction

  • Poster summarization remains under-explored because its layouts, technical terminology, and interplay among text, tables, and figures make accurate summarization challenging.Information may be distributed across a poster, requiring navigation and integration of diverse elements.Source: §1 · p. 1
  • POSTERSUM pairs 16,305 scientific posters with their corresponding research paper abstracts as summaries.The posters come from ICLR, ICML, and NeurIPS and cover a broad range of scientific disciplines.Source: §1 · pp. 1–2
  • GPT-4o, the best-performing closed-source model in the experiments, scores 22.30 ROUGE-L on posters with figures and tables.The authors use this result to illustrate MLLMs’ limitations on this task.Source: §1 · p. 1
  • 24.18 ROUGE-L is achieved by SEGMENT & SUMMARIZE, outperforming both open-source and closed-source models on scientific poster summarization.The method segments posters, summarizes regions locally, and combines those summaries into a cohesive abstract.Source: §1 · p. 2
  • The paper also demonstrates POSTERSUM’s utility for fine-tuning MLLMs, with promising improvements over zero-shot results.The contribution summary also describes a hierarchical method that segments posters into coherent regions, extracts their text, and composes a final summary.Source: §1 · p. 2

2 Related Work

The paper positions scientific poster summarization as a gap among existing multimodal and scientific summarization work, then presents POSTERSUM, a dataset of poster images paired with abstracts. Its creation and analysis describe the dataset’s scope, summary characteristics, and visual-text alignment.

2 Related Work

  • Existing scientific summarization benchmarks address videos, slide decks, surveys, and research papers, while poster summarization remains unexplored.Existing layout-analysis methods largely target standard documents or individual elements rather than posters’ rich multimodal structure.Source: §2 · p. 3

3 The POSTERSUM Dataset

  • POSTERSUM contains 16,305 academic poster images paired with their corresponding research paper abstracts.The posters vary in layout, content, and visual complexity, from text-heavy presentations to those emphasizing charts, graphs, and figures.Source: §3 · p. 3

3.1 Dataset Creation

  • The dataset was collected from ICLR, ICML, and NeurIPS websites and includes posters from 2022 to 2024 with available poster images.Abstracts were collected from paper pages or manually extracted from PDFs when missing from webpages.Source: §3.1 · pp. 3–4

3.2 Dataset Statistics and Analysis

  • The 16,305 poster-summary pairs provide a resource for evaluating abstractive summarization of academic research posters.Topic labels generated with GPT-4o identified 137 distinct topics; the summaries average 224 word-piece tokens, and the dataset was split into training, validation, and test sets.Source: §3.2 · p. 4
  • A significant portion of summaries contained novel content relative to OCR-extracted poster text, particularly in 3-gram and 4-gram categories; mean poster-image/summary CLIP score was 29.08, computed at sentence level and averaged across the dataset.Source: §3.2 · p. 4

4 Multimodal Poster Summarization

The task is to generate a textual summary from a poster image, integrating its varied visual and textual content. The paper evaluates multimodal and OCR-based baselines and introduces a hierarchical method that segments posters before composing a global summary.

4.1 Task Formulation

  • The task takes a poster image as input and generates a textual summary that captures its key points and essential content.The model must abstract across text, charts, diagrams, and equations to produce a coherent and informative summary.Source: §4.1 · p. 5

4.2 Baselines

  • The evaluation covers open- and closed-source MLLMs, OCR-based methods, zero-shot and Chain-of-Thought settings, and parameter-efficient fine-tuning of selected open-source models.OCR baselines extract and concatenate poster text, with one variant using a text-based LLM to summarize the extracted text.Source: §4.2 · p. 5

4.3 SEGMENT & SUMMARIZE

  • SEGMENT & SUMMARIZE divides posters into coherent regions, clusters them into fewer groups, and summarizes each region independently before global composition.Clustering reduces redundancy while aiming to ensure complete poster coverage.Source: §4.3 · pp. 5–6
  • SEGMENT & SUMMARIZE outperforms both open-source and closed-source models, achieving the best results across all reported metrics on the POSTERSUM test set.Table 3 reports ROUGE, BERTScore, SacreBLEU, and METEOR scores as percentages.Source: Table 3 · p. 6

2. Localized

The method generates localized summaries for poster regions and combines them into a cohesive global summary. It uses frozen models and requires no additional training or fine-tuning.

2. Localized

  • An MLLM extracts and interprets text, figures, and tables within each region to generate a localized summary.Processing regions separately also helps with high-resolution poster images.Source: §2 · p. 6
  • A text-based LLM combines the localized summaries into one well-structured output representing the poster’s overall content.The global step is intended to maintain logical flow and coherence.Source: §2 · p. 6
  • The hierarchical pipeline uses frozen models and requires no additional training or fine-tuning.The authors describe localized summaries as capturing fine-grained details that may be overlooked by a global approach.Source: §2 · p. 6

5 Experimental Details

The experiments evaluate poster summarization with standard text-generation metrics and a hierarchical segmentation-and-summarization pipeline. Final results are measured on the test set.

Experimental setup

  • Experiments generate at most 768 new tokens; open-source models use beam size 4 with greedy decoding, while closed-source models use platform defaults.Fine-tuning runs for 10 epochs with batch size 4.Source: §5 · pp. 6–7
  • Figure 4 depicts posters being segmented, locally summarized by MLLMs, and then merged by a text-based large language model.The pipeline produces a single coherent summary from the localized summaries.Source: Figure 4 · p. 7
  • SEGMENT & SUMMARIZE uses SAM segmentation with k-means clustering (k=8), MiniCPM-Llama3-V-2.5 for local summaries, and Llama 3.1-8B-Instruct for global summarization.The training set is used for fine-tuning and the validation set for hyperparameter tuning; final results are evaluated on the test set.Source: §5 · p. 7

Evaluation metrics

  • Accuracy is evaluated with ROUGE F1, SacreBLEU, METEOR, and BERTScore.The ROUGE variants are R-1, R-2, R-L, and LSum.Source: §5 · p. 7

6 Results

SEGMENT & SUMMARIZE leads the reported test-set comparisons across metrics, while the results also show limitations in MLLM and OCR-based approaches. CoT prompting and fine-tuning improve some baselines, but do not displace the proposed method’s reported overall results.

Closed-source models

  • GPT-4o records ROUGE-1/2/L scores of 44.98/13.12/22.30, while Claude-3.5 Sonnet records ROUGE-L 19.51.GPT-4o is described as relatively strong among closed-source models.Source: §6 · p. 7

OCR baselines

  • ROUGE-L rises from 12.73 to 15.49 when raw OCR text is combined with Llama-3.1, while the OCR methods still outperform some multimodal models.The authors attribute low OCR-baseline scores to concatenating raw OCR text without using other visual elements.Source: §6 · p. 7

Open-source models and CoT

  • MiniCPM-Llama3-V-2.5 leads zero-shot open-source models at ROUGE-1/L 39.88/20.14, and CoT prompting raises its ROUGE-1/L/METEOR scores to 41.50/21.04/26.34.CoT also raises mPLUG-DocOwl2 to ROUGE-1/L 37.04/19.71; the authors report similar gains for LLaVA-NeXT and Qwen2-VL-7B.Source: §6 · p. 7

Fine-tuned models

  • LoRA substantially boosts both fine-tuned MLLMs, but Llama-3.2-11B-Instruct does not surpass the best CoT variants of mPLUG-DocOwl2 and MiniCPM-Llama3-V-2.5.The authors suggest those variants may benefit from pre-training on multimodal scientific data.Source: §6 · pp. 7–8

Additional comparisons

  • Table 4 compares SEGMENT & SUMMARIZE with and without clustering, and Table 5 compares using mPLUG-DocOwl2 as the local summarizer with using that model alone.The supplied captions state that clustering yields more accurate results and that applying SEGMENT & SUMMARIZE improves results over using mPLUG-DocOwl2 itself.Source: Table 4 · p. 8, Table 5 · p. 8

SEGMENT & SUMMARIZE

  • ROUGE-1/2/L scores of 46.68/15.73/24.18 and BERTScore-F1 of 61.37 make SEGMENT & SUMMARIZE the top method across all reported metrics, with a 3.14% ROUGE-L gain over open-source models.The authors report that it outperforms both open-source and closed-source models.Source: §6 · pp. 7–8

7 Ablation Studies and Analysis

The ablations examine clustering and the local vision summarizer, alongside evaluation constraints for scientific summaries. The authors also note that human assessment is difficult and existing factuality metrics perform poorly on scientific text.

Effect of clustering

  • Removing clustering lowers ROUGE-1/2/L by 4.43/1.43/1.42 relative to the clustered approach.The non-clustered baseline selects the top eight segments by region size; the authors hypothesize clustering reduces redundancy and improves context aggregation.Source: §7 · p. 8

Effect of local vision summarization

  • With mPLUG-DocOwl2 as the local summarizer, the hierarchical method reaches ROUGE-1 42.48 and METEOR 26.72, but does not outperform the MiniCPM-based version.The authors report improvement over mPLUG-DocOwl2 in the CoT setting and conclude that segmentation and summarization improve results over using a poster as a single input.Source: §7 · p. 8

Evaluation challenges

  • Human evaluation is costly and logistically complex, and SummaC Conv and FActScore perform poorly on scientific text.The authors therefore rely on automatic metrics and note the need for improved factuality evaluation methods for multimodal scientific data.Source: §7 · p. 8

8 Conclusions

The paper presents POSTERSUM as a 16,305-pair benchmark and SEGMENT & SUMMARIZE as a hierarchical approach for poster summarization. Its experiments show that MLLMs still struggle with scientific posters, while the proposed method outperforms them in zero-shot and fine-tuned settings.

Contributions

  • POSTERSUM comprises 16,305 scientific poster–abstract pairs for poster summarization.The paper presents it as a multimodal benchmark.Source: §8 · p. 8

Conclusions

  • SEGMENT & SUMMARIZE breaks summarization into localized segments before producing a cohesive abstract, and outperforms MLLMs in both zero-shot and fine-tuned settings.The conclusion says substantial room remains to improve multimodal understanding of complex scientific documents such as posters.Source: §8 · p. 8
  • The authors position POSTERSUM as a resource for developing and evaluating MLLMs that process information-dense scientific content.They describe it as capturing the diverse and heterogeneous nature of academic posters.Source: §8 · p. 8

Limitations

The paper identifies limits in dataset scope and representativeness, automated topic labeling, and the assumptions behind its summarization method. It also flags risks from generated summaries and from treating abstracts as ground truth.

Limitations

  • The dataset covers only machine learning conference posters from 2022–2024, and GPT-4o topic labeling may bias or misstate the topic distribution.The method also assumes posters can be meaningfully divided into spatial regions, which may not fit cross-referenced or interdependent visual elements; the poster may differ from its paper.Source: Limitations · p. 9
  • Suboptimal initial segmentation can produce fragmented or redundant local summaries.The method relies heavily on segmentation quality.Source: Limitations · p. 9

Ethics Statement

  • The dataset uses publicly accessible conference posters and abstracts, and the authors sought permission from conference website contacts for research use.Source: Ethics Statement · p. 9
  • MLLMs may exhibit biases, hallucinate, or generate non-faithful text; generated summaries should be filtered automatically or manually verified before release.The authors specify this verification to ensure accuracy and reliability.Source: Ethics Statement · p. 9
  • The dataset may underrepresent poster styles, languages, and disciplines, potentially biasing models trained on POSTERSUM toward the included poster types.The authors recommend expanding the dataset across academic fields and visual formats.Source: Ethics Statement · p. 9

A Factuality Evaluation

The paper finds that current factuality metrics are poorly suited to scientific text and examines how poster text amount relates to summarization performance. Performance tends to improve with more text, but the relationship is weak.

A Factuality Evaluation

  • SummaC Conv scores were substantially low and FActScore extremely high, with both metrics showing failures on scientific text and motivating new factuality measures for multimodal scientific documents.The evaluation treated the reference abstract, rather than the poster image, as ground truth.Source: A Factuality Evaluation · p. 13

Prompt Template for Zero-Shot

C Effect of Poster Text Content on Summarization Performance

  • Pearson r = 0.213 and Spearman r = 0.210 indicate a weak relationship between OCR-extracted poster text length and ROUGE-L, despite performance tending to improve with more text.Low performance on posters with minimal text underscores the need for stronger understanding of figures, charts, equations, and tables.Source: C Effect of Poster Text Content on… · p. 13, Figure 5 · p. 13

D Selecting the Number of Clusters

The cluster analysis selects eight clusters for the final Segment & Summarize experiments, with the maximum tested count limited to manage inference time. Additional passages report model and fine-tuning setup details.

D Selecting the Number of Clusters

E Additional Experiment Details

  • Fine-tuning used LoRA with rank r = 8, α = 8, dropout 0.1, and a learning rate of 1 × 10−4 with Adam and a cosine schedule.Closed-source images were scaled to a maximum width of 2048 while preserving aspect ratio; training used two A100 GPUs with 80GB memory.Source: E Additional Experiment Details · p. 14

F Dataset Examples with Model Summaries

  • The dataset-example table captions identify samples from Chen et al. (2022) and Abreu et al. (2022), each paired with a gold reference and model-generated summaries.Source: Table 8 · p. 15, Table 9 · p. 16

Related papers

Loading 2502.17540v1…