Source-linked AI summary
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Tsu-Jui Fu, William Yang Wang, Daniel McDuff, Yale Song
TL;DR
Scientific document-to-slide generation requires summarizing multimodal content and arranging it into concise, visually structured presentations. DOC2PPT addresses this with a hierarchical sequence-to-sequence model, paraphrasing, text-figure matching, layout prediction, and a large paired dataset. The approach outperforms the other evaluated methods in human ratings and supports topic generalization, while its setup includes assumptions about figure references and the scope of figures.
Problem
Existing document summarization, retrieval, and multimodal methods do not jointly produce concise, visually organized slide decks from scientific documents.
Method
DOC2PPT uses a hierarchical sequence-to-sequence architecture with section- and slide-level structure, paraphrasing, text-figure matching, and layout design modules.
Results
Human evaluators rated the approach significantly higher than the other two methods on text similarity, figure/text matching, and figure-to-text matching.
Takeaways & Limitations
The paper establishes document-to-slide generation as a multimodal benchmark with paired data, evaluation metrics, and an approach that produces content-rich slides with aligned imagery.
Takeaways & Limitations
The setup treats images, graphs, charts, and tables as figures, and allows each section to reference any figure.
Abstract
from arXiv · showhide
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner. Can machines learn to emulate this laborious process? We present a novel task and approach for document-to-slide generation. Solving this involves document summarization, image and text retrieval, slide structure and layout prediction to arrange key elements in a form suitable for presentation. We propose a hierarchical sequence-to-sequence approach to tackle our task in an end-to-end manner. Our approach exploits the inherent structures within documents and slides and incorporates paraphrasing and layout prediction modules to generate slides. To help accelerate research in this domain, we release a dataset about 6K paired documents and slide decks used in our experiments. We show that our approach outperforms strong baselines and produces slides with rich content and aligned imagery.
Introduction
DOC2PPT introduces automatic generation of presentation slides from scientific documents, combining multimodal summarization, retrieval, and structured layout prediction. The paper contributes a paired document–slide dataset and a hierarchical end-to-end approach designed to produce concise, visually aligned slide drafts.
- Motivation: DOC2PPT defines slide generation from scientific documents as a multimodal task involving concise summarization, visual-semantic understanding, and stylized slide layout.Slides differ from manuscripts through bullet-style text, fixed screen space, and visual-centric organization.
- Approach: The hierarchical recurrent sequence-to-sequence model reads documents section by section and generates slides while using structural context to decide when to advance sections or slides.Section and slide embeddings support hierarchical decisions about the generated presentation structure.
- Approach: A paraphrasing module converts document-style sentences into concise slide phrases, while text-figure matching encourages related content to share slides.The paper also compares template-based and learning-based layout design quantitatively and qualitatively.
- Contributions: The broader objective is to generate editable slide drafts that enhance human productivity rather than remove humans from the presentation-making process.The paper evaluates the approach quantitatively with proposed metrics and qualitatively through human evaluation.
Related Work
DOC2PPT extends several research areas by requiring summarization, image–text alignment, and structured slide generation together. Unlike prior work in each area, it must produce a concise multimodal presentation while generating text and selecting related visual content.
- Document Summarization: Document summarization methods produce concise text but generally do not handle figures or multimodal understanding, whereas DOC2PPT requires both abstractive and extractive summarization.
- Visual-Semantic Embedding: Visual-semantic embedding methods learn text–image similarity, but DOC2PPT must generate the text and retrieve related images simultaneously rather than match images to known text.
- Multimodal Summarization: Multimodal summarization produces text summaries with related images, while DOC2PPT additionally organizes the summary into a structured slide format.
Approach
DOC2PPT generates slide decks from multimodal documents through a hierarchical, modular architecture that organizes sections, slides, and objects. It combines document reading, progress tracking, object placement, paraphrasing, and content- and structure-aware training.
- Document Reader: The Document Reader projects sentence and figure representations into a shared space so the Object Placer can select either modality as an object.Sentence and figure embeddings are combined into section representations, with figures available across sections because multiple sections may reference them.
- Architecture: A hierarchical recurrent sequence-to-sequence model reads documents and generates slide decks through section-, slide-, and object-level decisions.The Progress Tracker models the section-slide-object hierarchy and learns when to advance at each level.
- Object Placer: The Object Placer selects sentences or figures and predicts each chosen object’s slide location and size within the generated deck.Figures are resized while preserving aspect ratio, whereas sentences are paraphrased and font-adjusted to fit their bounding boxes.
- Paraphraser: The Paraphraser converts selected document sentences into concise slide-style text because uncompressed sentences are too verbose for presentations.It uses a sequence-to-sequence model with a copy mechanism and conditions on the current object state.
- Training: The training objective combines structural similarity with content similarity, covering object selection, concise phrasing, placement, and text-figure relevance.The content objective evaluates whether selected objects match the ground truth, text is presentation-like, objects are correctly located, and co-occurring objects are relevant.
- Paraphraser: Document sentences average 17.3 words versus 11.6 words in slides, supporting explicit paraphrasing for presentation-style compression.The reported difference is statistically significant (p = 0.0031).
Dataset
The dataset pairs scientific documents with presentation decks from computer vision, natural language processing, and machine learning venues. Its processing pipeline extracts, matches, and cleans text, figures, slides, and document sections, with human annotation used for the test set.
- Collection: The corpus contains paired documents and slide decks from computer vision, natural language processing, and machine learning proceedings.The source venues include CVPR, ECCV, BMVC, ACL, NAACL, EMNLP, ICML, NeurIPS, and ICLR.
- Annotation: Training and validation correspondences are extracted automatically, while the entire test set receives human image extraction and matching for reliability.The test-set procedure uses Amazon Mechanical Turk annotations.
- Processing: The pipeline extracts document sections and figures, slide text and figures, then matches slides to sections, sentences to document sentences, and figures by embedding similarity.It uses ScienceParse, PDFFigures, Azure OCR, border following, RoBERTa, and MobileNet across these stages.
- Cleaning: Near-duplicate animated slides are removed when a succeeding slide covers more than 80% of the preceding slide’s text or visual content.The more complete succeeding slide is retained.
- Processing: Figures absent from the corresponding document are discarded when their highest visual embedding similarity is below θI = 0.8.This threshold is applied during figure matching.
Experiments
DOC2PPT evaluates slide generation with automatic metrics, ablations, topic-aware tests, and human judgments. The final approach improves perceived similarity across text, figures, and text-figure alignment, while component studies clarify contributions from paraphrasing, text-figure matching, post-processing, and layout design.
- Evaluation Metrics: The evaluation combines ROUGE-SL for slide text, LC-FS for figure correctness and ordering, TFR for text-figure relevance, and mIoU for layout overlap.The study reports ablations under these automatic metrics and supplements them with human evaluation.
- Ablation Results: The text-figure matching loss improves figure quality, especially LC-FS precision, while moderately reducing recall and improving ROUGE-SL.The loss constrains the multimodal embedding space and improves text selection.
- Figure Post-Processing: Test-time figure post-processing further improves the image-related LC-FS and TFR metrics by adding missing or removing unnecessary figures.The method uses the multimodal projection head to perform this correction.
- Layout Prediction vs. Template: Predicted layouts achieve higher mIoU with ground-truth layouts, whereas template-based layouts produce more consistent visual designs.The comparison contrasts learned placement from layout loss with fixed title, text, and figure placement.
- Topic-Aware Evaluation: Within-topic training and testing performs best, while training on all topics achieves the second-best performance and generalizes across CV, NLP, and ML.NLP-trained models also generalize well on the text metric despite using the smallest topic dataset.
- Human Evaluation: The final approach received significantly higher human ratings than the other two methods for text similarity, figure-text matching, and text-figure matching.Ratings used a 1–7 scale; no significant difference was found between the other two methods.
Conclusion
DOC2PPT introduces a multimodal document-to-slide generation task and approach, releasing paired data and evaluation metrics to advance vision-and-language understanding.
- DOC2PPT frames slide generation as understanding and summarizing documents containing text and figures into a structured presentation.
Details of the Data Processing Steps
The dataset pipeline extracts and matches textual and visual content between scientific documents and slide decks, while manually labeled data evaluates its reliability.
- The pipeline automatically extracts text and figures, then matches them between documents and slide decks.
- Text Extraction: Azure OCR extracts slide sentences, with results described as quite reliable because slides contain text.
- Slide Stemming: 93% accuracy is achieved for slide stemming, which removes preceding slides with at least 80% content overlap as redundant.Ghosted slides from sequential animations are removed while retaining the final slide.
- Slide-Section Matching: 82% accuracy is achieved for slide-section matching using RoBERTa text embeddings and cosine similarity to assign each slide to its closest document section.
- Figure Matching: Figure matching uses morphological transformation, border following, and MobileNet visual embeddings with a cosine-similarity threshold.Precision, recall, and F1 are evaluated on human-labeled test data; annotators label matches and slide bounding boxes.
Settings of Approach
The approach settings examine paraphrasing with object-state context and tune post-processing thresholds using validation-set performance.
- Paraphrasing Module: Applying paraphrasing improves Rouge-L in all cases, with hobj achieving 32.27 versus 31.95 without it.The object state supplies contextual information that helps generate paraphrases more relevant to document content.
- Post-Processing: Post-processing removes figures below θR and adds figures above θA, with θR = 0.8 and θA = 0.9 achieving the highest LC-F1.The thresholds are tuned on a 100-example labeled validation set.
Inference Flow
During inference, the model generates a slide deck from an academic paper and post-processes it to refine figures and slide-style text.
- The inference flow first generates slides, then removes unrelated figures, adds relevant ones, and paraphrases sentences into slide-style text.
Human Evaluation
Human evaluation compares generated slides with ground-truth decks across text quality, figure extraction, and text–figure relevance. The evaluation interface and inference/design figures document the assessment setup and presentation workflow.
- Human Evaluation: Human annotators compare ground-truth and generated slides for text similarity, figure similarity, and figure–text relevance.The task also allows annotators to indicate that no figures are present.
- Human Evaluation: The evaluation interface presents the ground-truth deck alongside the generated slides for side-by-side assessment.
- Human Evaluation: The workflow includes an inference-flow visualization and an example of applying PowerPoint Design Ideas to generated decks.The supplied figure captions identify these components without specifying additional quantitative outcomes.
- Human Evaluation: The reported evaluation covers text quality, figure extraction, and the relevance of figures to the slide text.
Qualitative Examples
The paper presents qualitative generated slide decks and describes using PowerPoint Design Ideas to improve template-based outputs. These examples include additional results and failure cases available on the project webpage.
- Qualitative Examples: The approach produces qualitative generated slide decks, with additional results and failure cases provided on the project webpage.
- Applying PowerPoint Design Ideas: Template-based generation fixes content size and position, so PowerPoint Design Ideas can automatically produce a more attractive layout.The generated deck is presented as a draft that humans can build upon.
- Qualitative Examples: The qualitative examples include generated decks associated with Luo et al. (2020), Liu et al. (2020), Dai et al. (2020), Jang et al. (2019), and Chaudhary, Schütze, and Gupta (2020).