PaperlayerView exact revision on arXiv

Source-linked AI summary

RAP: A Dataset for Paper-to-Poster Generation via Declarative Markup from Human-Crafted Exemplars

Haonan Han, Yukang Lin, Minliang Lin, Haoxian Tan, Jun-Yan He, Rui Yang, Zhi-Qi Cheng, Jie Hu, Guanglu Wan, Xiu Li

arXiv:2610.23230v1cs.CV

TL;DR

Paper-to-poster generation is a visual-coding task, but existing resources offer limited scale, renderable markup, and human-grounded design references. RAP provides verified paper-poster-markup triplets and a held-out benchmark, using the corpus for retrieval-based generation and model training; PosterRAG leads baselines in diversity and human-likeness.

  • Problem

    Paper-to-poster generation combines summarization, visual design, and code implementation, but existing benchmarks are limited in scale and markup depth and often rely on subjective evaluation.

    Source: §1 · p. 2

  • Method

    RAP builds verified paper-poster-markup triplets through automated derendering and uses them for PosterRAG template retrieval and an SFT+RL training recipe.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    PosterRAG achieves state-of-the-art diversity and distributional human-likeness against strong baselines by retrieving and refilling human-crafted designs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Human-crafted posters serve both as reusable design references and as verified supervision for training a dedicated generation model.

    Source: §1 · p. 3

Abstract

from arXiv · show

Making a conference poster is visual coding in miniature: an entire paper must be compressed onto a single page, which exercises summarization, visual design, and code implementation at once, using nothing beyond the paper itself. Yet benchmarks for this task in the context of visual coding remain scarce: existing resources are small and annotated only with layout boxes, so evaluation must fall back on VLM-as-judge or human checklists rather than objective render-and-compare, and the generation methods they support design layouts from scratch, leaving human-crafted posters unusable as either a retrievable design library or as verified supervision. We introduce RAP (Reconstruct A Poster), a dataset and benchmark for paper-to-poster generation as a visual coding task. RAP contains 25K+ verified paper-poster pairs from various machine learning conferences held between 2022 and 2026, annotated with a three-level content taxonomy, plus a held-out benchmark of 150 test papers each paired with its human poster. A fully automated derendering pipeline turns each poster into styled HTML/CSS of typed, refillable slots, so every sample is a paper-poster-markup triplet verifiable by rendering against the source poster. The corpus serves both inference and training: PosterRAG, a training-free multi-agent pipeline, retrieves an exemplar and refills its slots without touching layout or styling, while an SFT+RL recipe trains a dedicated model from the same triplets. An online demo is available at https://reconstruct-a-poster.github.io/.

1 Introduction

RAP frames paper-to-poster generation as visual coding and addresses gaps in benchmark scale, renderable markup, and reuse of human-crafted designs. Its verified corpus supports both exemplar-based generation and model training.

1 Introduction

  • Paper-to-poster generation combines paper summarization, layout and visual design under space and aspect-ratio constraints, and code implementation using the paper alone.Existing resources are limited in scale, often lack renderable markup, and commonly use subjective evaluation or generate layouts from scratch.Source: §1 · p. 2
  • RAP contains 25,435 verified paper-poster pairs, three-level content annotations, and a held-out benchmark of 150 test papers with human posters.The taxonomy covers 15 domains, 20 tasks, and 39 methods.Source: §1 · pp. 2–3
  • PosterRAG retrieves and refills human-designed templates, while SFT followed by reinforcement learning trains a dedicated paper-to-poster model from the same triplets.PosterRAG leaves retrieved layout and styling untouched; the training recipe uses paper–markup pairs and verifiable rewards.Source: §1 · pp. 2–3
  • An automated pipeline converts raster posters into styled HTML/CSS with addressable, typed, refillable slots, enabling the corpus to serve as verified supervision.The paper-poster-markup triplets support both inference-time retrieval and training.Source: §1 · p. 3

2 Related Work

Prior visual-coding benchmarks address screenshots, webpages, or charts rather than condensing full papers into dense posters. Existing poster resources lack renderable reconstructions and human-grounded design supervision, leaving a trade-off between editable content and design grounding.

2 Related Work

  • Visual-coding benchmarks target screenshots, webpages, websites, plots, or charts, whereas paper-to-poster generation must condense a full paper into a space-constrained design and implement it as code.RAP adapts the benchmark setting with full-paper inputs, styled HTML/CSS outputs, and render-and-compare verification.Source: §2 · p. 3
  • Paper2Poster [1] and P2PEval pair papers with only a few hundred poster images, while SciPostGen scales up but supplies layout-level annotation rather than renderable markup.PosterSum uses abstracts, and SciPostLayout [4] and SciPostLayoutTree annotate layout features without reconstructing posters as markup.Source: §2 · p. 3
  • Text-to-image methods produce non-editable posters with weak dense-text fidelity, while LLM pipelines often generate layouts from scratch without human-grounded HTML supervision.P2PInstruct uses model-synthesized HTML targets; PPTAgent retrieves manually curated slide templates but can fail to adapt content appropriately.Source: §2 · p. 3

3 RAP: Large-Scale Poster Derendering

RAP builds a large paper-poster collection from three machine-learning conferences and converts each poster into styled HTML with typed, fillable slots. Its derendering pipeline combines layout detection, design description, and HTML reconstruction.

3 RAP: Large-Scale Poster Derendering

  • The dataset contains 25,435 verified pairs from NeurIPS, ICLR, and ICML papers accepted between 2022 and 2026.Each paper also receives a taxonomy of 15 domains, 20 tasks, and 39 methods, plus a distilled introduction summary.Source: §3 · p. 3
  • The four-stage derendering pipeline detects poster elements, estimates text font sizes, generates a structured design caption, and reconstructs HTML.Layout detection assigns bounding boxes and semantic categories; design captioning describes global and per-region visual attributes.Source: §3 · pp. 3–4
  • HTML reconstruction embeds cropped image-like and decorative regions, while text-like regions become empty slots with OCR-estimated font sizes and pixel-exact geometry.An MLLM then fills text slots from the poster and synthesizes CSS styling using the image and design caption.Source: §3 · p. 4
  • The output is a complete HTML file whose styled, addressable content blocks reproduce the source poster’s layout and visual appearance.Source: §3 · p. 4

4 Method

RAP supports paper-to-poster generation with retrieval and post-training applications. PosterRAG retrieves and refills human-crafted templates, while the post-training pipeline constructs paper and instruction inputs and uses supervised fine-tuning followed by rendering-aware reinforcement learning.

4.1 PosterRAG: Training-free retrieval-augmented generation

  • PosterRAG parses each paper, retrieves a template through a three-level cascade, then maps paper content into its slots and iteratively reviews the rendered poster.Retrieval filters by category, matches narrative, and selects by aesthetic scores; slot filling preserves the template’s layout and styling.Source: §4.1 · p. 5

A Paper Information Extraction

  • Paper-side inputs combine section-organized paper text with selected assets, including tables, figures, and equations matched to poster slots.Text is drawn from method, experiments, and conclusion sections; uncertain asset matches are left unresolved.Source: B User Instruction Construction · p. 6

B User Instruction Construction

  • Instruction-side inputs pair a natural-language design request with a layout specification that captures structure without exposing poster text or HTML.The design description covers palette, typography, composition, and visual emphasis; layout recovers columns, section order, geometry, and asset assignments.Source: B User Instruction Construction · p. 6
  • Post-training uses supervised fine-tuning followed by rendering-aware reinforcement learning, with rule-based and VLM-based rewards weighted 0.4 and 0.6.Rule checks cover layout, density, and assets; VLM scoring assesses quality and similarity to the reference poster.Source: B User Instruction Construction · p. 6

5 Experiments

RAP’s retrieval-based HTML guidance leads the reported diversity and human-likeness comparisons, while post-training improves a base model across five metrics. Ablations favor HTML scaffolds and show that layout, OCR, and design-caption components contribute to reconstruction quality.

5.1 Baselines and Dataset

  • The held-out test set contains 150 papers, with 10 sampled from each of RAP-25K’s 15 research domains; comparisons include Nano Banana, P2P, Paper2Poster, and PosterGen.LLM-based agent calls across methods are replaced with Claude-Opus4.6 for a fair comparison.Source: §5.1 · p. 6

5.2 Evaluation Metrics

  • Diversity is measured with LPIPS, DINOv2-CosDist, and DISTS, while FID and FDiD measure distributional fidelity to human-crafted designs.Higher diversity scores indicate more varied generations; FDiD uses DINOv2 rather than InceptionV3 features and is sensitive to visual structure and layout geometry.Source: §5.2 · p. 7

5.3.1 Quantitative Comparison

  • FDiD is 18.7% lower for RAP (HTML Guidance) than P2P (377.25); RAP also leads all five metrics, with FID/FDiD of 23.75/306.79 and DINOv2-CosDist of 0.4443.P2P scores 0.2847 and Paper2Poster 0.1138 on DINOv2-CosDist.Source: §5.3.1 · p. 7, Table 2 · p. 7
  • Five metrics improve with RAP-based SFT over base Qwen3.8-27B; RL reduces FID/FDiD from 28.00/326.24 to 23.42/306.94 and raises LPIPS and DISTS, with a small DINOv2-CosDist decrease.The RL variant has the highest LPIPS and DISTS and second-lowest FID/FDiD among compared models, behind Claude-Opus-5.Source: §5.3.1 · p. 7, Table 2 · p. 7

5.3.2 Qualitative Comparison

  • In the same-paper examples, PosterRAG follows a retrieved template’s palette, typography, and section organization, while post-trained models produce complete three-column layouts unlike the incomplete base-model output.The PosterRAG comparison contrasts with Paper2Poster’s conventional columns and NanoBanana Pro’s stylized raster poster; the post-trained layouts follow the design specification and resemble the human reference.Source: §5.3.2 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

5.4 Ablation Studies

  • HTML Guidance beats Image Guidance on all five metrics, reducing FID/FDiD from 77.11/1267.84 to 23.75/306.79.DINOv2-CosDist rises from 0.0877 to 0.4443, LPIPS from 0.5584 to 0.6471, and DISTS from 0.2246 to 0.3490; the HTML scaffold constrains layout and styling while allowing content replacement and refinement.Source: §5.4 · p. 7
  • Claude-Opus-4.6 and Gemini-3.1-Pro outperform GPT-4o and Qwen-3.5; Gemini leads SSIM and DISTS, while Claude leads layout and distributional fidelity.Claude-Opus-4.6 is adopted as the backbone for the other experiments.Source: §5.4 · p. 9, Table 3 · p. 9
  • Adding Layout raises mIoU from 0.0652 to 0.5236; OCR further improves layout accuracy and distributional fidelity, while Design Caption yields smaller, consistent gains across six metrics.The full pipeline achieves the best reconstruction scores, supporting complementary geometric, typographic, and design constraints.Source: §5.4 · p. 9

6 Conclusion

RAP provides verified paper-poster pairs and a derendering pipeline that converts posters into refillable HTML/CSS. Its triplets support retrieval and post-training applications, and the authors release the dataset and tools for future research.

6 Conclusion

  • RAP contains 25,435 verified paper-poster pairs from NeurIPS, ICML, and ICLR (2022–2026), with a three-level taxonomy and a held-out benchmark of 150 papers.The fully automated derendering pipeline produces typed, refillable HTML/CSS, with SSIM 0.79 and layout mIoU 0.69, enabling render-and-compare fidelity evaluation.Source: §6 · p. 9
  • Paper-poster-markup triplets support training-free PosterRAG, which retrieves and refills human-designed templates, and an SFT+RL recipe that trains a dedicated model against the same references.PosterRAG leads every baseline on diversity and human-likeness; the authors characterize human designs as reusable design knowledge, not only evaluation references.Source: §6 · p. 9

AI use statement

  • Claude-Opus-4.6 powered derendering, RAG, and agentic baselines, while AI assistants supported manuscript drafting and language polishing; the authors retain responsibility for research decisions and submitted content.The post-trained model was Qwen3.8-27B, using Gemini-3.8-Flash for VLM-based rewards.Source: AI use statement · p. 9

A.1 Dataset Overview and Taxonomy Distribution

RAP-25K organizes papers with a three-level domain, task, and method taxonomy and includes varied human-crafted poster designs. Its conference and year breakdown notes that 2026 coverage currently includes only ICLR.

A.1 Dataset Overview and Taxonomy Distribution

  • RAP-25K’s three-level taxonomy comprises 15 domains, 20 tasks, and 39 methods; Table 4 lists domain categories with counts and proportions.The domain taxonomy is presented in Table 4.Source: §A.1 · p. 12, Table 4 · p. 12
  • The poster mosaic illustrates variation in layout, color palette, typography, and content density, forming the reference pool for retrieval and post-training.Figure 7 provides the visual overview of human-crafted posters in RAP.Source: §A.1 · p. 12, Figure 7 · p. 12

A.2 Per-Conference Breakdown

B Implementation Details

This section details PosterRAG’s template retrieval and slot-filling workflow, the prompts and programmatic operations for derendering, baseline generation, and visual refinement, and the post-mapping steps that hide leftover template imagery and resize text to fit.

B.1.1 POSTERRAG: TRAINING-FREE RETRIEVAL-AUGMENTED GENERATION

  • PosterRAG parses papers into content, taxonomy labels, and an introduction summary, then matches templates by categorical fit and narrative similarity.A final aesthetic-selection stage scores content density, visualization ratio, and scientific aesthetics, defaulting to the neutral midpoint unless user preferences shift the weighting.Source: §B.1.1 · p. 13
  • The MappingAgent profiles typed template slots, maps adapted paper content to them, and programmatically updates text, image, or rendered-equation/table assets rather than regenerating HTML.It preserves the template’s CSS and layout; equations and tables are rendered to PNG, while figure paths are resolved through the registry before image replacement.Source: §B.1.1 · pp. 13–14, §B.3 · p. 18
  • The ReviewerAgent renders posters in Chromium and applies localized CSS patches from screenshot-based VLM feedback, without changing layout geometry or hiding or truncating content.Review stops at a score of at least 8/10, after three rounds, or when no patches are proposed; permitted fixes target local presentation issues under safety constraints.Source: §B.1.1 · p. 14, §B.2.8 · pp. 17–18

B.2.1 Prompt 1: Design Caption in Derendering

  • The derendering design-caption prompt asks an MLLM to infer 3–7 semantic regions from the poster image and describe their design tokens, palette, and holistic visual style.It requires four observed hex-color roles and concrete notes for each dimension, without layout annotations or simply restating token names.Source: §B.2.1 · pp. 14–15

B.2.2 Prompt 2: HTML Reconstruction in Derendering

  • The HTML reconstruction prompt combines the reference image, detected layout and OCR, design specification, and CSS and font-size hints to generate a faithful HTML/CSS reproduction.It locks block geometry, requires exact text and measured font sizes, and specifies that design tokens take precedence when they conflict with the model’s interpretation.Source: §B.2.2 · p. 15

B.2.3 Prompt 3: Paper Parser (ParserAgent)

  • The ParserAgent prompt condenses parsed paper sections into factual poster-ready summaries while preserving technical details, results, conclusions, and coverage of core paper sections.Input sections are capped at 3,000 characters and subsections at 1,500; each section is condensed to 50–200 words with an importance label.Source: §B.2.3 · pp. 15–16

B.2.4 Prompt 4: Poster Planner (PlannerAgent)

  • The baseline-only PlannerAgent selects poster sections, assigns figures and space priorities, and plans a layout; the GeneratorAgent then produces self-contained HTML from that plan.These prompts support from-scratch generation and are not used in the template-filling pipeline.Source: §B.2.4 · p. 16, §B.2.5 · p. 16

B.2.7 Prompt 7: Content Adapter (AdapterAgent)

  • The Content Adapter may edit only poster text to shorten overflow, expand sparse slots, improve formatting, and verify author details, while preserving CSS, layout, DOM, and image placeholders.It operates on a filled HTML poster with base64 images temporarily stripped, then restores the image data after refinement.Source: §B.2.7 · p. 17

B.3 Programmatic Filling Details

  • Two safety passes hide unmapped image slots and images still referencing template-local assets, preventing placeholder or residual template content from appearing.Unmapped image-like slots receive display:none, and <img> elements whose src still references assets/* are hidden.Source: §B.3 · p. 19
  • Text-slot font sizes are dynamically adjusted according to new content length and slot capacity to avoid overflow or excessive whitespace.The adjustment aims to make text fill each slot comfortably.Source: §B.3 · p. 19

B.4 CSS Safety Guard Implementation

The CSS safety guard screens proposed fixes before application, using property restrictions and block-aware rules to avoid destructive changes.

B.4 CSS Safety Guard Implementation

  • The guard checks each proposed CSS property–value pair against a forbidden list before applying it.Table 6 summarizes the CSS safety guard rules.Source: §B.4 · p. 19, Table 6 · p. 19
  • For font-size fixes, it retrieves the template’s original size and rejects any decrease.The check prevents proposed fixes from shrinking text below the original template size.Source: §B.4 · p. 19
  • The guard blocks text-hostile properties only for text blocks, while image blocks may use overflow:hidden to contain images.It also distinguishes text-like from image-like blocks when applying the property restrictions.Source: §B.4 · p. 19

B.5 Post-Training Dynamics

The post-training curves show improving SFT objectives followed by rising RL reward, while completion length falls; these dynamics are distinct from held-out poster-quality evaluation.

B.5 Post-Training Dynamics

  • Validation loss and accuracy approach 0.24 and 92.2%, respectively, as SFT learning rate decays.Training and validation loss decrease while next-token accuracy improves during SFT; Figure 8 plots loss, accuracy, and learning rate.Source: §B.5 · p. 19, Figure 8 · p. 19
  • Mean total reward rises from approximately 0.40 to 0.69 over 300 RL steps, while completion length falls from roughly 5.1k to 4.2k tokens.The model-based reward increases more, while rule-based reward remains high after an initial rise; Figure 9 plots reward components and completion length.Source: §B.5 · p. 19, Figure 9 · p. 20
  • The curves characterize training-objective optimization, not held-out poster quality, which is evaluated separately in the main experiments.Figures 8 and 9 summarize the two-stage post-training dynamics.Source: §B.5 · p. 19, Figure 8 · p. 19, Figure 9 · p. 20

B.6 Ablation on RL Rewards

The reward ablation favors the combined rule-based and model-based objective on most reported measures, with a slight reduction in DINOv2-CosDist versus model-based reward alone.

B.6 Ablation on RL Rewards

  • DINOv2-CosDist decreases slightly from 0.2784 to 0.2735 with combined reward versus model-based reward alone.The combined objective improves distributional human-likeness and two of the three diversity measures.Source: §B.6 · p. 20, Table 7 · p. 20
  • Combined reward reduces FID from 25.58 to 23.42 and FDiD from 313.34 to 306.94 versus model-based reward alone.Table 7 compares the rule-based, model-based, and combined configurations.Source: Table 7 · p. 20, §B.6 · p. 20
  • The combined configuration outperforms both single-reward variants on LPIPS, DISTS, FID, and FDiD.Table 7 reports the ablation of RL reward components on Qwen3.8-27B.Source: Table 7 · p. 20, §B.6 · p. 20

C Evaluation Details

The evaluation measures diversity through pairwise distances across generated posters and human-likeness through feature-distribution distances to real human-created designs. Metric variants capture different visual properties, and lower Fréchet distances indicate closer distributions.

C.1.1 Diversity

  • Higher average pairwise distances across the generated set indicate greater structural and perceptual diversity.The evaluation averages each metric over all poster pairs.Source: §C.1.1 · p. 20
  • LPIPS measures perceptual distance, while DINOv2-CosDist captures semantic and stylistic variation between generated posters.Higher mean pairwise LPIPS and larger DINOv2-CosDist indicate more visually distinct posters and broader design styles, respectively.Source: §C.1.1 · p. 20
  • DISTS measures structural and textural dissimilarity using multi-scale VGG features.Higher mean pairwise DISTS indicates greater deep-feature diversity.Source: §C.1.1 · p. 21

C.1.2 Human-Likeness

  • Fréchet distances compare generated and real-poster feature distributions; lower values indicate generated posters are distributionally closer to human designs.FID uses InceptionV3 features and is relatively insensitive to document-specific layouts, whereas FDiD uses DINOv2 features and is sensitive to visual structure and layout geometry.Source: §C.1.2 · p. 21

Related papers

Loading 2610.23230v1…