mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model

1. Introduction

• Scientific paper-writing assistance remains limited by weak diagram analysis, while a copilot should understand multiple diagram types and formats, connect them to preceding text, and remain controllable by users.The introduction names these as three critical abilities for diagram-analysis assistance.Source: §1 · pp. 1–2
• M-Paper aligns diagrams in image and LaTeX formats with captions and paragraph analyses, supporting multimodal captioning, diagram analysis, and outline recommendation tasks.The dataset uses preceding text as context and concise key points as outline controls for analysis.Source: §1 · p. 2
• PaperOwl is an instruction-tuned DocOwl baseline trained on data from the three tasks, and experiments test context, outline, and vision-encoding strategies.The reported ablations include increasing image resolution and improving correlation across multiple diagrams.Source: §1 · p. 2

2. Related Work

• Text-only paper-understanding tools handle tasks such as extraction, classification, summarization, or citation recommendation, but lack the multimodal abilities needed for diagram analysis.The cited review also describes multimodal document work on charts, tables, documents, infographics, and scientific figures.Source: §2 · p. 2
• Existing multimodal systems are described as producing short answers or single-image descriptions rather than comprehensive analysis that correlates context with multiple figures or tables.The passage notes that prior work includes varied image-understanding tasks and high-resolution document processing.Source: §2 · p. 2
• M-Paper and PaperOwl are presented as a dataset and tuned model aimed at strengthening multimodal diagram analysis and moving toward a paper-writing copilot.The paper specifically frames this as a step toward, rather than a completed realization of, that copilot.Source: §2 · p. 3

3.1. Paper Collection

• The collection contains 48,000 arXiv IDs from 2012–2023 across 15 categories, selected through PapersWithCode and downloaded as LaTeX sources.The study focuses on computer science papers.Source: §3.1 · p. 3

3.2. Paper Parse

• LaTeX parsing aligns diagrams with captions and paragraphs; tables are rendered as images by compiling each table on its own page and cropping its detected bounding box.Captions are removed before rendering to avoid leaking them into table images; GPT-3.5 generates outlines for analysis samples.Source: §3.2 · p. 3

3.3. Task Definition

• Captioning summarizes figures or tables, while analysis produces a paragraph from multiple diagrams and uses an outline to specify key points.Captioning inputs pair diagrams with preceding context; analysis adds the outline as an explicit control signal.Source: §3.3 · p. 4
• CIDErgpt multiplies CIDEr by F1gpt to combine detailed n-gram matching with semantic matching of key points.The example contrasts a prediction with lower CIDEr but higher CIDErgpt because it describes semantics more accurately despite mentioning fewer ground-truth n-grams.Source: §3.3 · p. 4
• Outline Recommendation generates outlines from context and instructions, optionally with diagrams, and evaluates them with captioning metrics.Source: §3.3 · p. 4

3.4. Statistic

• More than 25% of Multimodal Diagram Analysis samples use multiple diagrams, compared with a much smaller share in captioning; the test set also balances diagram types.M-Paper spans figures and tables in vision or LaTeX form, and the outline is far shorter than the analysis it guides.Source: §3.4 · p. 5

4. mPLUG-PaperOwl

• PaperOwl uses mPLUG-DocOwl as its base model for instruction-tuning on M-Paper.Source: §4 · p. 6

4.1. Model Architecture

• A cropping module splits each 448x448 image into four 224x224 sub-images before independent vision encoding.Source: §4.1 · p. 6
• The Vision Abstractor uses six transformer layers and 64 learnable query tokens to aggregate or filter visual semantics into condensed features.Source: §4.1 · p. 6
• The language decoder follows LLaMA-7B, with rank-8 LoRA used to adapt it to vision-and-language tasks and alleviate catastrophic forgetting.Source: §4.1 · p. 6

4.2. Model Training

• Training combines 702,247 samples from diagram captioning, diagram analysis, and outline recommendation; some analysis samples omit outlines to strengthen vision understanding.Source: §4.2 · p. 6
• During instruction-tuning, the Vision Encoder and raw LLaMA-7B parameters are frozen, while the Vision Abstractor and language-decoder LoRA are updated.Source: §4.2 · p. 6

5.1. Comparison with SOTA MLLMs.

• After fine-tuning, PaperOwl performs much better across diagram captioning, diagram analysis, and outline recommendation than the zero-shot models.Source: §5.1 · p. 6, Table 3 · p. 7

5.2. Ablation Study

• Providing context at inference improves captioning but hurts analysis unless context is also included during training.Source: §5.2 · p. 7, Table 4 · p. 7
• Adding an outline at inference improves analysis, and training with outlines further improves performance; even model-generated outlines help.Source: §5.2 · p. 7, Table 5 · p. 7
• Fine-tuning the Vision Abstractor matters for analysis, while increasing image resolution alone brings no significant improvement under a frozen Vision Encoder.Source: §5.2 · p. 8, Table 6 · p. 8
• Cropping improves diagram captioning more than analysis; its many visual tokens make multimodal understanding harder for the language decoder.Source: §5.2 · p. 8, Table 6 · p. 8

5.3. Qualitative Results

• With preceding text and a simple outline, PaperOwl follows the outline and describes diagram details, but still makes mistakes about concrete figure numbers.Source: §5.3 · p. 8, Figure 7 · p. 8

6. Conclusion

• M-Paper aligns diagrams from high-quality paper LaTeX files with captions and paragraph analysis, and supports captioning, analysis, and outline recommendation tasks.Source: §6 · p. 8
• PaperOwl is a baseline trained on combined task data; experiments support context and outline inputs and identify image resolution and balancing multimodal information as improvement concerns.Source: §6 · p. 9

Supplementary Material

• Text cleaning replaces citation tags with a special token and drops paragraphs containing equations longer than 40 characters.Source: §A.1 · p. 11

A.2. Outline Construction

• Two outline formats are constructed: a highly concise summary and multiple content-related key points.GPT-3.5 prompts and in-context demonstrations are used for both.Source: §A.2 · p. 11

A.3. Statistic

• Figure 8 presents the category distribution of the 48,688 academic papers in M-Paper.The supplied passages do not specify category counts.Source: Figure 8 · p. 11, §A.3 · p. 11

A.4. Task Instruction

• Instructions vary across the three tasks to strengthen the model’s instruction-following ability.Source: §A.4 · p. 11

• F1gpt uses GPT-extracted key points and pairwise match judgments to calculate precision, recall, and F1 for diagram analyses.During extraction, GPT processes the prediction and ground truth together to capture similarities and differences.Source: B. GPT-based Metric · p. 11

C.1. Influence of Table Format

• Latex tables are easier to caption than image tables, but do not significantly improve diagram analysis and can lower CIDEr.With table Latex, models tend to mention rare proper nouns or numbers that are unnecessary for discussion and absent from the ground-truth analysis.Source: §C.1 · p. 11, Table 7 · p. 11

C.2. More Qualitative Results

• Figures 9 and 10 show additional diagram-captioning examples, including tables and figures with multiple or single sub-images.Figure 9 covers image- and Latex-format tables; Figure 10 covers multiple-sub-image and single-image figures.Source: §C.2 · p. 11, Figure 9 · p. 13, Figure 10 · p. 14
• Without an outline, PaperOwl can produce diagram-related analysis that differs from author intent; a concise outline yields better analysis with minor errors.Figure 11 marks wrong descriptions in red and diagram-conforming descriptions inconsistent with ground truth in blue.Source: §C.2 · p. 12, Figure 11 · p. 15
• A failure case shows PaperOwl making only small revisions to an outline and providing uninformative diagram analysis.The authors identify following the outline while adding diagram details as a remaining challenge.Source: §C.2 · p. 12, Figure 12 · p. 16