Source-linked AI summary
Paper2Video: Automatic Video Generation from Scientific Papers
Zeyu Zhu, Kevin Qinghong Lin, Mike Zheng Shou
TL;DR
Creating academic presentation videos is labor-intensive and requires coordinating dense paper content across slides, subtitles, speech, and presenters. Paper2Video introduces a benchmark and PaperTalker, a multi-agent generation framework, which outperforms human-made presentations by 10% in PresentQuiz accuracy and achieves comparable user-study ratings.
Problem
Manual academic presentation video production takes hours and requires coordinating paper content across slides, subtitles, recordings, and editing.
Method
Paper2Video benchmarks 101 papers with videos, slides, and speaker metadata, while PaperTalker generates presentations through coordinated multi-agent modules.
Results
PaperTalker outperforms human-made presentations by 10% in PresentQuiz accuracy and achieves comparable user-study ratings.
Takeaways & Limitations
Automated generation of ready-to-use academic presentation videos is feasible while reducing production time by 6 times.
Abstract
from arXiv · showhide
Academic presentation videos have become an essential medium for research communication, yet producing them remains highly labor-intensive, often requiring hours of slide design, recording, and editing for a short 2 to 10 minutes video. Unlike natural video, presentation video generation involves distinctive challenges: inputs from research papers, dense multi-modal information (text, figures, tables), and the need to coordinate multiple aligned channels such as slides, subtitles, speech, and human talker. To address these challenges, we introduce Paper2Video, the first benchmark of 101 research papers paired with author-created presentation videos, slides, and speaker metadata. We further design four tailored evaluation metrics--Meta Similarity, PresentArena, PresentQuiz, and IP Memory--to measure how videos convey the paper's information to the audience. Building on this foundation, we propose PaperTalker, the first multi-agent framework for academic presentation video generation. It integrates slide generation with effective layout refinement by a novel effective tree search visual choice, cursor grounding, subtitling, speech synthesis, and talking-head rendering, while parallelizing slide-wise generation for efficiency. Experiments on Paper2Video demonstrate that the presentation videos produced by our approach are more faithful and informative than existing baselines, establishing a practical step toward automated and ready-to-use academic video generation. Our dataset, agent, and code are available at https://github.com/showlab/Paper2Video.
1 INTRODUCTION
Paper2Video addresses the labor-intensive production and evaluation of academic presentation videos by introducing a benchmark with author-created materials and metrics, alongside PaperTalker, a multi-agent generation framework. PaperTalker integrates slide creation, subtitling, cursor grounding, speech synthesis, and talking-head rendering to automate presentation production.
- Motivation: Producing a 2 to 10 minute academic presentation video manually can require several hours of slide design, subtitle writing, recording, and editing.Academic presentation videos are widely used in research communication and are required by some conferences.
- PaperTalker: PaperTalker is a multi-agent framework that integrates slide generation, subtitling, cursor grounding, speech synthesis, and talking-head video rendering.Its modules include tree search visual choice for fine-grained slide generation and GUI grounding coupled with WhisperX.
- Paper2Video: Paper2Video provides a benchmark of 101 papers with author-recorded presentation videos, slides, speaker metadata, and four evaluation metrics.The metrics are Meta Similarity, PresentArena, PresentQuiz, and IP Memory.
2 RELATED WORKS
Related work spans AI4Research systems and video-generation methods, but existing approaches remain limited in long-form, multi-modal, and fully synchronized presentation video generation. Recent agents address some of these challenges, though PresentAgent lacks personalization.
- Video Generation: Video diffusion models improve natural-video length, quality, and controllability but struggle with several-minute videos, multiple shots, multi-image conditioning, and aligned audio.These limitations create a gap between natural-video generation and real-world presentation-video applications.
- Video Generation: MovieAgent enables long-form movie generation through hierarchical chain-of-thought planning and LLM-simulated roles for directing, screenwriting, storyboarding, and location management.Its multi-agent design supports natural-video production by coordinating distinct planning roles.
- Presentation Video Generation: PresentAgent targets presentation videos by combining PPTAgent with text-to-speech, but lacks personalization such as mechanical speech adaptation.The supplied passage identifies PresentAgent as a presentation-video method with limited personalization.
- AI for Research: AI4Research supports the scholarly workflow across text, static visuals, and dynamic video, alongside efforts in academic writing, literature surveying, and end-to-end agent benchmarking.These efforts aim to improve research efficiency across multiple scholarly tasks.
3 PAPER2VIDEO BENCHMARK
Paper2Video establishes a benchmark for generating academic presentation videos from research papers and evaluates them by how effectively they communicate scholarly information. It curates diverse author-created presentations with aligned media and introduces evaluation centered on the relationship between generated videos, papers, and human-made presentations.
- Benchmark Motivation: Academic presentation videos require coordinated slides, subtitles, speech, and presenter rendering to communicate research clearly rather than merely reproduce visual appearance.Slides should organize expressive figures and tables with concise descriptions, while subtitles and speech remain semantically synchronized.
- Benchmark Statistics: The benchmark spans diverse topics, with presentations containing 4–28 slides and lasting 2–14 minutes.These characteristics provide varied inputs for automatic generation and evaluation of academic presentation videos.
- Dataset Construction: Paper2Video curates 101 peer-reviewed conference papers with author-created presentation videos, slides, and speaker-related metadata for academic video generation.The dataset prioritizes papers with relatively complete presentation metadata and supplements missing presenter images from authors’ websites.
- Evaluation Metrics: The evaluation framework measures the relationship between generated videos, source papers, and human-made presentations, including information coverage through multiple-choice quizzes.PresentQuiz asks whether viewers can recover paper knowledge from the presentation video.
4 PAPERTALKER AGENT
PaperTalker is a multi-agent framework that generates academic presentation videos directly from papers by coordinating slide creation, layout refinement, aligned narration and cursor grounding, and presenter-video synthesis. Its design combines visual tree search, GUI-grounded cursor alignment, and parallel slide-wise generation to improve flexibility and efficiency.
- Overview: PaperTalker automatically generates presentation videos from academic papers through a scalable multi-agent pipeline that decouples specialized generation roles.The pipeline includes builders for slides, narration alignment, cursor grounding, and presenter-video synthesis.
- Slide Builder: Beamer slide generation compiles paper-derived code, uses diagnostics to repair errors, and preserves the paper’s content in grammatically correct slides.Compilation warnings also identify slides requiring layout refinement, including overflow caused by unsuitable font sizes or figure parameters.
- Slide Builder: Tree Search Visual Choice refines problematic layouts by combining rule-based candidate proposals with VLM scoring instead of directly tuning numeric parameters.The method addresses models’ insensitivity to small changes in font sizes, margins, and figure scales.
- Spatial-Temporal Grounding: The system links sentence-level subtitles and visual-focus prompts to cursor positions and word-level timestamps for precise spatial-temporal narration alignment.UI-TARS grounds cursor locations, while WhisperX extracts timestamps for cursor-segment timing.
- Presenter Video: PaperTalker synthesizes author-preserving presenter videos with lip-audio synchronization using F5-TTS, Hallo2 or FantasyTalking, and parallel slide-wise generation.Separate slide generation is appropriate because slide transitions are hard scene changes and do not require presenter continuity across adjacent slides.
5 EXPERIMENTS
Experiments compare PaperTalker with end-to-end and multi-agent baselines using alignment, efficiency, qualitative, and cursor-grounding evaluations. PaperTalker achieves the highest Meta Similarity scores and lowest generation cost, while qualitative examples show stronger content, grounding, and layout quality than competing methods.
- 5.1 Experimental Setup: The evaluation compares end-to-end video generators, multi-agent frameworks, and PaperTalker variants using GPT-4.1 as the VLM and Gemini-2.5-Flash as the VideoLLM.End-to-end methods generate presentation videos directly from paper-derived prompts, whereas multi-agent frameworks combine slide and text-to-speech generation.
- 5.1 Quantitative Analysis: PaperTalker attains the highest speech and content similarity scores, aligning most closely with human-authored presentations.Meta Similarity evaluates generated slides, subtitles, and speech against corresponding human-authored materials.
- 5.2 Efficiency Analysis: PaperTalker achieves the lowest generation cost through Beamer-based slide creation, lightweight visual-choice post-processing, and parallel talking-head generation.PresentAgent incurs higher token costs because of frequent refinement queries during slide editing.
- 5.2 Ablation Study: The cursor ablation evaluates whether visible spatial cues improve content grounding for vision-language models through a subtitle-conditioned localization question-answering task.The study is motivated by the hypothesis that cursors help audiences locate relevant regions and facilitate VLM content grounding.
- 5.3 Qualitative Analysis: Qualitative comparisons show PaperTalker producing rich slide content, accurate cursor grounding, and an engaging talker, unlike competing methods’ blurred text or overfull layouts.Veo3 shows blurred text and incomplete coverage, while PresentAgent exhibits text-heavy slides, overfull layouts, and inaccurate information.
6 CONCLUSIONS · Appendix
Paper2Video introduces a benchmark and evaluation metrics for academic presentation videos, while PaperTalker demonstrates feasible automated generation of ready-to-use videos. The appendix illustrates how tree-search visual choice refines slide layouts.
- 6 CONCLUSIONS: PaperTalker shows that automated generation of ready-to-use academic presentation videos is feasible and effective.The framework is presented as addressing the bottleneck of presentation video generation through agent automation.
- 6 CONCLUSIONS: Paper2Video provides a comprehensive benchmark and metrics for evaluating presentation-video quality, knowledge coverage, and academic memorability.The work frames these resources as enabling rigorous evaluation of presentation videos.
- 6 CONCLUSIONS: The framework targets a long-standing bottleneck in presentation video generation through agent automation.
- 6 CONCLUSIONS: The benchmark and metrics are designed to assess both the quality and informational value of academic presentation videos.
- Appendix: Tree-search visual choice refines slide layouts by transforming initial slide results into corresponding refined slides.Figure 8 contrasts slides before layout refinement with their corresponding slides after refinement.
- Appendix: Figure 8 presents paired before-and-after examples of slide layout refinement.The first row contains slides before refinement, and the second row contains the corresponding refined slides.
A Evaluation Metrics · A.1 IP Memory
IP Memory evaluates whether viewers retain a presentation well enough to recall the work and pose a relevant question. It operationalizes this through shuffled video–question matching by a VideoLLM, with accuracy quantifying performance.
- A Evaluation Metrics: IP Memory measures whether an audience member can recall a presented work and pose a relevant question after viewing several presentation videos.The metric is motivated by real-world conference interactions with authors.
- A Evaluation Metrics: The metric targets audience retention rather than only immediate presentation understanding.It evaluates recall after watching several presentation videos and preparing for interaction with the author.
- A.1 IP Memory: Video–question pairs are constructed by sampling a five-second clip from each presentation video and pairing it with a corresponding PresentQuiz understanding-level question.The questions are drawn from PresentQuiz.
- A.1 IP Memory: A VideoLLM receives four randomly sampled video–question pairs whose videos and questions are shuffled.This setup requires matching questions to the relevant presentation content.
- A.1 IP Memory: The VideoLLM also receives an image of one speaker as the query while evaluating which question should be posed.The speaker image is provided alongside the shuffled video–question pairs.
- A.1 IP Memory: Matching accuracy quantifies IP Memory performance in the audience-proxy evaluation.The model is asked to identify the relevant question to pose, and accuracy is used as the resulting measure.
B Experiment · B.1 Video Results
The video results are provided in the supplementary materials.
- B.1 Video Results: Video results are reported in the supplementary materials.
B.2 Results of Tree Search Visual Choice · C Prompts
Tree search visual choice resolves overfull slides and substantially improves layout quality. The prompts specify structured Beamer generation, compilation-safe correction, visual selection, cursor-grounded scripting, video evaluation, and evidence-based question answering.
- B.2 Results of Tree Search Visual Choice: Tree search visual choice resolves overfull slides and substantially improves slide quality, demonstrating its crucial role in layout adjustment.Figure 8 compares slides before and after refinement.
- C Prompts: The Beamer-generation prompt requires chapters covering motivation, related work, method, innovation, experiments, results, ablations, limitations, future work, and an end slide.It also requires clear chapter titles and content, with method components introduced separately.
- C Prompts: Slide-generation instructions emphasize concise academic layouts, highlighted key terms and mathematics, correctly sourced images and tables, roughly 10 slides, and compilable English LaTeX.They require real source assets and preservation of core methods, experimental data, and conclusions when summarizing.
- C Prompts: The correction prompt applies minimal edits to make Beamer files compile while preserving slide content, assets, layout semantics, theme, colors, and structure.Required fixes include packages, environments, braces, special characters, math delimiters, labels, paths, options, and undefined commands, with Tectonic compilation as the target.
- C Prompts: The layout judge disqualifies overfull options, then selects the greatest-coverage remaining slide or the first non-overfull option scanned from A through D.The prompt defines overfull as clipped, outside-frame, overlapped, or hidden content and describes risk and coverage trends across A–D.
- C Prompts: The cursor-grounded script prompt requests concise first-person academic narration for each slide, with one ordered cursor description per sentence, a 50-word limit, and ### slide delimiters.Scripts must explain the current slide clearly and avoid unrelated content.
- C Prompts: The question-answering prompt requires answering every multiple-choice question solely from the presentation video, selecting an option only when supported and providing a brief evidence reference in strict JSON.It prohibits outside context and invalid timestamps.