PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization

1 Introduction

• Automatic document-to-poster transformation remains less studied and involves both content summarization and template generation with harmonization.Content summarization selects and paraphrases key material; template generation sets layout and design elements based on that content.Source: §1 · p. 1
• A poster must fit a multi-page document onto one page while covering important aspects, limiting repetition, and aligning images with text.The multimodal interpretation of coverage and diversity is not well understood.Source: §1 · p. 1
• Direct LLM summarization is unsuitable for long multimodal inputs because models may lack multimodal output, struggle with lengthy context, lose global document view, incur high computation, or hallucinate.The passage also notes that long-context performance can drop.Source: §1 · p. 1
• PostDoc proposes an end-to-end pipeline for generating posters from long multimodal documents, combining content selection, LLM paraphrasing, and aesthetically designed templates.Its content-selection objective explicitly targets coverage, diversity, and alignment between text and images; the work evaluates content and design through automated and human evaluations.Source: §1 · pp. 1–2

2.1 Multimodal Summarization

• Earlier multimodal summarization methods have limitations including modality bias, selection of only one image, treating text and images separately, or focusing on text and video.The paper positions its work in the setting where both input and output are multimodal.Source: §2.1 · p. 2

2.2 Layout Generation

• Prior layout approaches include recursive division, non-conditional generation, or computationally expensive methods that may not produce non-overlapping boxes; PostDoc uses poster-focused heuristics and content-based design elements.The design elements include colors, fonts, and backgrounds.Source: §2.2 · p. 2

2.3 Document Transformation

• Existing document-to-poster work relies on retrieved templates and is limited to research papers, while automatic slide generation often requires user-provided outlines or section summaries.Source: §2.3 · p. 2

2.4 Background on Submodular Functions

• Deep submodular functions are trainable variants of submodular functions, whose diminishing-returns property supports greedy selection with an approximation guarantee.The paper identifies a gap: existing deep submodular approaches for text summarization do not incorporate multimodal coverage, diversity, and image-text alignment.Source: §2.4 · p. 2, §2.1 · p. 2

3 Problem Statement and Solution Approach

• PostDoc extracts multimodal document content with Adobe Extract API and represents text and non-text elements using BLIP embeddings in a shared 768-dimensional space.The embeddings are shifted to positive coordinates and L1-normalized because text and image embeddings may be on different scales.Source: §3 · p. 2
• The method selects at most K elements from the ordered document, treating each element as a normalized BLIP embedding, and trains selection using documents paired with ground-truth summaries.At inference, the selected subset is paraphrased for a poster-friendly format.Source: §3 · p. 2
• The generated poster template uses design elements conditioned on the document content and selected material, then receives the paraphrased content.Source: §3 · pp. 2–3

3.1 Multimodal Extractive Summarization

• The selection objective seeks a subset of at most K elements that provides document coverage, within-summary diversity, image-text alignment, and adaptability to ground-truth summaries.Selected images should correspond to selected text and vice versa; training data may capture additional summary properties.Source: §3.1 · p. 3
• The objective weights embedding dimensions with nonnegative trainable parameters, and the model chooses the subset A that maximizes f(A).The weights are intended to capture the importance of each embedding dimension.Source: §3.1 · p. 3
• The objective’s first similarity term rewards selected elements that are similar to elements across the document, capturing coverage; its second term penalizes similarity among selected elements, capturing diversity.Source: §3.1 · p. 3
• A cross-modal term compares selected image elements with selected text sentences to encode multimodal alignment.The objective also includes a final term for a mathematical property; it is constant when all dimension weights are initialized to 1/d.Source: §3.1 · p. 3
• Training uses a hinge loss to keep the model’s objective value for its predicted summary close to that of the ground-truth summary.The loss includes an L2 regularizer on the weights, with λ set to 0.1 based on validation performance.Source: §3.1 · p. 3

3.2 Training and Optimization

• Training alternates between maximizing over fixed-cardinality subset A and minimizing over nonnegative weights w, repeating until validation loss converges.The optimization stops when the loss converges on the validation set used in the experiments.Source: §3.2 · p. 3, §3.2.2 · p. 4

3.2.1 Maximization w.r.t. A

• With w fixed, maximizing f becomes subset selection for each document Di, a combinatorial problem addressed by the greedy algorithm.The method adds elements from D\A according to their marginal gain.Source: §3.2.1 · pp. 3–4
• Theorem 3.1 establishes that f is monotone submodular, enabling greedy maximization with a (1−1 e) factor approximation of the optimum.At each step, greedy adds the element with the largest marginal gain until |A|=K.Source: §3.2.1 · pp. 3–4

3.2.2 Minimization w.r.t. w

• With A fixed, projected stochastic gradient descent updates w and projects each weight to the nonnegative range.One training update has complexity O(NKd + Kd), while inference has complexity O(KNd).Source: §3.2.2 · p. 4

• ChatGPT (GPT-3.5-turbo) paraphrases the extractive text because the full document is too long for a direct call, so K is chosen to fit the summary in one API call.The authors cite its paraphrasing performance across use cases [6].Source: §3.3 · p. 4
• The prompt groups and rephrases content into 5 to 8 ordered topics, each with a title and at least three bullet sentences, without adding content.Source: §3.3 · p. 5
• The poster uses the paraphrased text together with images selected in the multimodal extractive summary.Source: §3.3 · p. 5

3.4 Template Generation

• A poster template combines font, colors, and layout, including the positions of content elements.Source: §3.4 · p. 5

3.4.1 Font Selection

• The font-selection model uses poster titles and a fine-tuned MiniLM encoder with a two-layer fully connected network, dropout, and LeakyReLU.It is trained on title-font pairs from the Let me choose dataset, which covers 10 fonts.Source: §3.4.1 · p. 5
• The font-selection model is trained for 20,000 epochs on the dataset’s train section.Source: §3.4.1 · p. 5

3.4.3 Layout Generation

• Figure 2 shows a sample poster generated by PostDoc for a research paper.Source: Figure 2 · p. 5

3.4.2 Color Selection

• Color generation uses TPNSmall, based on the TPN architecture from Text2Colors, trained for 7,000 epochs on the Text2Colors dataset.Source: §3.4.2 · p. 5
• A question-answering model derives poster intent from the title; TPNSmall then outputs a palette, whose dominant color sets the background and complement sets box fill.Text is black or white according to contrast with the box fill, and the background color conditions Firefly background generation.Source: §3.4.2 · p. 5

3.4.3 Layout Generation

• The heuristic layout places images on the left and text boxes on the right, adjusts image-box width to image count, and estimates text-box height from content length.It creates a text box for each topic and an image box for each image, with captions when available.Source: §3.4.3 · p. 5

4.1 Datasets Used

• The MSMO evaluation trains on 9,000 samples and tests on 1,000, while NJU-Fudan provides 76 paper-poster pairs used only for out-of-domain testing.For MSMO, samples come from the 10,261 test examples with image annotations; the NJU-Fudan pairs are filtered to retain at least one image.Source: §4.1 · p. 5

4.2 Baseline Methods

• Because replicable code for existing document-to-poster systems was unavailable, PostDoc’s modules are compared with corresponding baselines rather than through an end-to-end comparison.The summarization baselines use text summarizers with images added afterward; font selection and layout generation are evaluated against separate baselines.Source: §4.2 · p. 6

4.3.1 Multimodal Summarization

• Multimodal summarization is assessed with ROUGE, coverage, diversity, image precision and recall, and average inference time.Coverage and diversity use BLIP embeddings of document content and generated-summary elements.Source: §4.3.1 · p. 6

4.5 Model Ablation Study

• In ablations, full PostDoc achieves the best ROUGE performance, while the other three metrics show no consistent pattern across variants.The ablation study is reported on MSMO and NJU-Fudan.Source: §4.5 · p. 7, Table 3 · p. 7

• PostDoc is an end-to-end pipeline that combines deep submodular functions with a single ChatGPT call to generate posters from long multimodal documents.Source: §5 · p. 7
• PostDoc achieves better or comparable performance than direct calls to the same LLM, which are expensive in computation and cost.Source: §5 · pp. 7–8
• Its current performance is limited for non-natural images such as flowcharts and neural diagrams, and for structured elements such as tables.Source: §5 · p. 8

A.1 Color Selection

• Text2Colors’ TPN model generates a candidate palette for the poster.Source: §A.1 · p. 9
• The dominant palette color becomes the background, its complement fills text boxes, and black or white text is selected for contrast.Source: §A.1 · p. 9
• The background color is named for a Firefly prompt that generates a background grounded in that color.Source: §A.1 · p. 9

A.2 Layout Generation

• The heuristic creates one text box per topic and one image box per image, then divides remaining space between image and text regions.Source: §A.2 · p. 9
• Text-box widths are fixed, image widths vary with image count, and image heights follow their aspect ratios.Source: §A.2 · p. 9
• For fewer than three images, the layout elongates bottom text boxes toward the image side and reduces dh2 for visual balance.Source: Figure 3 · p. 9
• After estimating box heights and widths, the method calculates top-left coordinates; X coordinates are interpolated from box widths and the two region widths.Source: §A.2 · pp. 9–10

B Additional Results for Model Ablation Study

• PostDoc performs best against the ablations on the MSMO test dataset in terms of ROUGE-1 and ROUGE-2 scores.Source: B Additional Results for Model… · p. 10
• PostDoc is reported as best among the ablations on MSMO test in terms of ROUGE-1.Source: B Additional Results for Model… · p. 10
• PostDoc is reported as best among the ablations on MSMO test in terms of ROUGE-2.Source: B Additional Results for Model… · p. 10

• NGOMetrics combines equilibrium, padding, density, and overlap to score layout aesthetic properties.Source: C Details of NGOMetrics · p. 10
• Equilibrium measures the distance between the combined bounding boxes’ center of mass and the layout center; padding measures space to topmost and rightmost box edges.Source: C Details of NGOMetrics · p. 10
• Density is the proportion of the frame covered by boxes, while overlap sums the areas where bounding boxes intersect.Source: C Details of NGOMetrics · p. 10
• With more text and image boxes, LayoutDM layouts show increased overlap and clutter near the document center; the authors report their algorithm scales better on density and overlap scores.Source: C Details of NGOMetrics · p. 11
• Future work includes fine-tuning LayoutDM on posters and developing conventions for ordering image and text boxes.Source: C Details of NGOMetrics · p. 11