Source-linked AI summary
PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents
Bihui Yu, Xinglong Xu, Junjie Jiang, Jiabei Cheng, Caijun Jia, Siyuan Li, Conghui He, Jingxuan Wei, Cheng Tan
TL;DR
Compilable LaTeX can still produce visually flawed PDFs, while code-level tools and text-only editing cannot reliably assess two-dimensional layout. PaperFit addresses this with vision-in-the-loop diagnosis and constrained repair, substantially outperforming all baselines on visual quality and page-budget compliance.
Problem
Compilable LaTeX does not guarantee visual quality, and source code or compilation logs cannot fully support two-dimensional typesetting judgments.
Method
PaperFit iteratively renders pages, diagnoses categorized defects, and applies constrained source revisions, evaluated on PaperFit-Bench with 200 papers across 10 venues and 13 defect types.
Results
PaperFit substantially outperforms all baselines in visual quality and page-budget compliance, including a naive multi-round visual agent.
Takeaways & Limitations
The results establish Visual Typesetting Optimization as a critical missing stage in document automation.
Takeaways & Limitations
PaperFit enforces content preservation, compilation and rendering, and page-budget requirements as hard constraints, prioritizing them over visual quality and edit minimality.
Abstract
from arXiv · showhide
A LaTeX manuscript that compiles without error is not necessarily publication-ready. The resulting PDFs frequently suffer from misplaced floats, overflowing equations, inconsistent table scaling, widow and orphan lines, and poor page balance, forcing authors into repetitive compile-inspect-edit cycles. Rule-based tools are blind to rendered visuals, operating only on source code and log files. Text-only LLMs perform open-loop text editing, unable to predict or verify the two-dimensional layout consequences of their changes. Reliable typesetting optimization therefore requires a visual closed loop with verification after every edit. We formalize this problem as Visual Typesetting Optimization (VTO), the task of transforming a compilable LaTeX paper into a visually polished, page-budget-compliant PDF through iterative visual verification and source-level revision, and introduce a five-category taxonomy of typesetting defects to guide diagnosis. We present PaperFit, a vision-in-the-loop agent that iteratively renders pages, diagnoses defects, and applies constrained repairs. To benchmark VTO, we construct PaperFit-Bench with 200 papers across 10 venue templates and 13 defect types at different difficulty. Extensive experiments show that PaperFit outperforms all baselines by a large margin, establishing that bridging the gap from compilable source to publication-ready PDF requires vision-in-the-loop optimization and that VTO constitutes a critical missing stage in the document automation pipeline.
1 Introduction
Structural formatting has advanced, but existing tools cannot judge the two-dimensional visual quality of compiled documents or reliably repair it. PaperFit addresses this gap with a vision-in-the-loop sense–act–verify agent and a benchmark spanning diverse templates, papers, defects, and difficulty levels.
- Motivation: Structural formatting tools transform document representations, reconstruct LaTeX from PDFs, or generate LaTeX frameworks from natural descriptions.The paper collectively refers to these capabilities as structural formatting.
- Limitations: Existing approaches provide incomplete observability because code-level signals cannot assess visual overflow, figure placement, reading flow, or page white-space distribution.Typesetting quality is inherently a two-dimensional spatial judgment, while rule-based tools and compilation logs provide one-dimensional signals.
- Approach: Vision-language models enable a closed compile–inspect–edit loop, but effective automation also requires structured diagnosis, constrained repair, and gated validation.Naive multi-round visual prompting can introduce new defects or ignore page-budget constraints.
- PaperFit: PaperFit closes the sense–act–verify loop by integrating source, log, PDF, and page-image evidence into defect records and enforcing constrained repair operations.Its policy specifies permitted operations, forbidden pseudo-fixes, and protected content.
- Benchmark and results: PaperFit-Bench contains 10 venue templates, 200 papers, and 13 defect types across three difficulty levels, with six baselines covering rule-only through multi-round visual repair.PaperFit achieves perfect compilation and rendering success, the highest visual quality and page-budget compliance, and substantially outperforms all baselines.
2 Related Work
Related work has advanced document understanding, visual-to-code conversion, and agentic document optimization, but most systems still treat LaTeX as a static target or lack a reliable evaluation-optimization loop. Iterative visual feedback begins to address this limitation, motivating PaperFit’s visual closed-loop approach.
- Document Understanding: Document layout analysis progressed from structural formatting toward multimodal recognition using visual features, textual information, and handling of complex perturbations and large-scale layouts.Representative systems include VTLayout, LayoutLM, DocFormer, and DONUT.
- Visual-to-Code Conversion: VLMs increasingly map document visuals to structured markup, with Nougat, Im2Latex-100K, A2R2, Math2LaTeX, and Vision-RWKV supporting academic, mathematical, and structural recognition.Nougat uses a Swin Transformer to convert academic PDFs into markup language, while Im2Latex-100K provides a benchmark for image-to-LaTeX conversion.
- Visual-to-Code Conversion: Most visual-to-LaTeX models treat LaTeX as a static translation target rather than an iteratively optimized rendered document.LATTE introduced visual-feedback refinement for tables and formulae, while other work explored reinforcement learning for high-fidelity conversion of complex table images.
- Agentic Document Optimization: Multi-agent systems automate document parsing, presentation and poster generation, illustration editing, stateful LaTeX editing, structured translation, and domain-specific review.Examples include PaperTalker, Paper2Poster, AutoFigure-Edit, LaTeXAgent, and multi-agent systems for structured translation and review feedback.
- Agentic Document Optimization: A reliable evaluation-optimization loop remains challenging, although Seeing is Improving uses visual rewards for iterative text-layout refinement and ReLook applies vision-grounded reinforcement learning to web coding.These methods address readability or coding issues that are difficult to observe from source code alone.
3 The PaperFit-Benchmark
PaperFit-Bench evaluates automated LaTeX layout repair as visual restoration from systematically perturbed sources, using original compilable papers as ground truth. Its 200 mixed-disturbance instances span five defect categories, three empirical difficulty tiers, and workflows designed for rendered-page and iterative full-document evaluation.
- Benchmark Design: PaperFit-Bench pairs systematically perturbed LaTeX sources with original compilable versions for deterministic visual layout-restoration evaluation.The benchmark covers five defect categories and three difficulty tiers.
- Data Collection: Its corpus draws on published arXiv papers across multiple AI subfields and 10 venue templates spanning single- and double-column formats with 7–14-page limits.This diversity is intended to reduce evaluation bias toward any single typesetting style.
- Assembly and Quality Control: 200 instances remain after controlled compilation testing, visual verification, and final quality assurance confirm that perturbations are realized.Samples failing compilation or requiring private macro packages are excluded, and each sample contains at least three figures and two tables.
- Perturbation Design: Thirteen perturbation strategies are organized into five taxonomy-aligned categories: space utilization, float placement, table width, overflow, and cross-template migration.These categories correspond to Classes A–E in the VTO defect taxonomy.
- Perturbation Design: The benchmark prioritizes realistic mixed disturbances in academic paper projects rather than isolated one-defect examples, grouping cases into empirical easy, medium, and hard buckets.Difficulty buckets are empirical groups rather than deterministic recipes.
- Benchmark Comparison: PaperFit-Bench uniquely combines systematic perturbation injection, rendered-page visual evaluation, multimodal evidence integration, and iterative full-document repair workflows.These capabilities target evaluation of modern AI-powered LaTeX layout optimization agents.
4 Method
PaperFit formulates Visual Typesetting Optimization as constrained, iterative search that minimizes rendered visual defects while preserving scientific content, compilation, and page-budget requirements. It combines source, log, PDF, and page-image evidence with risk-aware repairs, protected-object checks, rollback, and full-document re-verification after every edit.
- Problem formulation: VTO minimizes residual visual defects while penalizing source edits under hard constraints for compilation, content preservation, and any specified page budget.Constraint priority is content preservation > compilation/rendering > page budget > visual quality > edit minimality.
- Evidence and diagnosis: PaperFit fuses source, log, PDF, and rendered-page evidence because no single layer reliably captures all defect types.Page images expose two-dimensional defects, while source and logs localize repairs and diagnose executable-state failures.
- Repair policy: A repair preference profile prioritizes layout-native fixes, restricts spacing manipulation, and forbids pseudo-fixes that distort typography, violate templates, or shift defects.Preferred actions include re-anchoring floats, splitting equations, restructuring tables, and normalizing figure widths; forbidden examples include resizebox, newpage, scalebox, and content deletion.
- Safety checks: Before each repair, PaperFit snapshots protected objects and automatically rolls back if any object is deleted, displaced across section boundaries, or has its caption altered.Protected objects include figures, tables, captions, labels, citations, and bibliography entries.
- Verification loop: After every edit, PaperFit recompiles, re-renders, and re-inspects the complete document through a six-step loop before a gatekeeper determines the next action.The loop parses deterministic compile signals, builds structured defect records, applies constrained category-specific repairs, and validates the updated output.
5 Experiment … 5.3 Capability Boundary Comparison
PaperFit is evaluated against rule-based, text-only, and visual baselines using programmatic and visual metrics on PaperFit-Bench. It achieves the strongest combined performance, while the capability comparison shows that external systems cover only partial segments of the full-paper visual repair chain.
- 5.1 Experimental Setting: PaperFit-Bench gives every method the same LaTeX project and page budget, then evaluates compiled, rendered outputs with programmatic checks and VLM-based visual assessment.The benchmark protocol combines technical validity with rendered-page quality.
- 5.1.1 Baselines: The baselines span rule-based repair, text-only source and log editing, and single- or multi-round visual agents, isolating feedback and iteration capabilities.VisualMR is the closest baseline because it uses source, logs, and page images over fixed rounds, but lacks PaperFit’s structured diagnosis and validation machinery.
- 5.1.2 Evaluation protocol: The evaluation reports compile success, render success, Page hit, and Win rate alongside Program and VLM composite scores.Program is a 0–5 non-visual execution-and-fidelity score, while VLM is a gated 0–5 visual-quality score.
- 5.1.2 Evaluation protocol: The VLM score combines absolute quality, repair quality, and final-paper aesthetics, then caps candidates violating rendering, compilation, page-budget, reference, or defect constraints.The raw score is VLMraw = 0.35 Sabs + 0.40 Srepair + 0.25 Sfinal.
- 5.2 Main Quantitative Results: RuleLog, TextST, TextMR, and VisualST remain below a VLM score of 2.19 and a Win rate of 0.43, showing that logs, text feedback, or single-turn visual edits are insufficient.Text/log methods miss two-dimensional failures, while single-turn visual editing often fails on non-local cascades.
- 5.2 Main Quantitative Results: 0.975 compile and render success: VisualMR removes most execution failures, but reaches only 0.549 Page hit and 0.650 Win rate.Its fixed-round visual process lacks explicit planning, constrained repair, and gatekeeper validation.
- 5.2 Main Quantitative Results: 1.000 compile and render success: PaperFit achieves the best VLM score of 3.391, Win rate of 0.895, and Page hit of 0.805, with a Program score of 4.579.All methods maintain content embedding similarity above 0.97, indicating that the gains come from layout-structure repair rather than semantic drift.
- 5.3 Capability Boundary Comparison: External system families cover local capability segments, but none combine multi-source input, LaTeX editing, execution feedback, full-paper page-image diagnosis, page-level repair, and page-budget or gatekeeper constraints.PaperFit’s contribution is integrating these capabilities into a full-paper visual typesetting optimization system rather than providing one isolated parser or editor.
5.4 Model Backend Comparison … 5.7 Error Analysis
Across backend comparisons, PaperFit’s gains are stable across models, align closely with human judgments, and are illustrated by successful repairs of diverse layout defects. Error analysis nevertheless shows that passing compilation and page-count checks can still leave global budget violations or visually invalid output.
- 5.4 Model Backend Comparison: All backends achieve VLM scores of 3.52–3.66, win rates of 90–100%, and near-perfect compile/render reliability, with only a 0.14-point overall VLM spread.The backend spread is much smaller than the 0.59-point PaperFit–VisualMR gap reported in Table 5.
- 5.4 Model Backend Comparison: MiMo-v2.5 leads repair-oriented dimensions, while GPT-5.4 leads new-defect avoidance and DeepSeek-V4 leads several visual-balance dimensions.MiMo-v2.5 leads defect resolution (3.90), visual quality (3.85), and publication readiness (3.80); GPT-5.4 leads new-defect avoidance (4.30); DeepSeek-V4 leads space utilization (3.50), float placement (3.90), and visual balance (3.20).
- 5.4 Model Backend Comparison: Difficulty-split VLM scores remain within a spread of ≤0.14, with GPT-5.4 leading easy and medium cases and DeepSeek-V4 Pro leading hard cases.No single backend dominates all difficulty levels, indicating crossover rather than systematic backend advantage.
- 5.5 Human–VLM Evaluation Correlation: The automated VLM score closely tracks human judgments, reaching a Spearman correlation of r = 0.8571 across methods.This correlation is reported between VLM scores and average human ratings.
- 5.6 Qualitative Case Study: In qualitative cases, PaperFit realigns displaced tables and figures while satisfying a 10-page CVPR/ICCV budget.VisualMR leaves referenced visual objects far from their semantic anchors, whereas PaperFit restores Tables 3–4 and Figure 3 near their references.
- 5.6 Qualitative Case Study: PaperFit also corrects page-budget shifts, footer and reference-layout defects, and template-migration mismatches across IJCAI, IEEE, AAAI→ICLR, and ICLR→CVPR cases.It reduces the IJCAI case to 8 pages, returns the IEEE case to 16 pages, and adapts figure dimensions and float placement during template conversion.
- 5.6 Qualitative Case Study: VisualMR produces renderable output but often misses underlying defects and page constraints, whereas PaperFit uses defect records, constrained repairs, and acceptance gates.The qualitative evidence supports an organized closed loop rather than visual feedback alone.
- 5.7 Error Analysis: Error cases expose failures in both global page-budget control and visual validation: outputs reach 16/10 or 20/19 pages, retain a cropped figure at 10/10, or render grayed pages at 13/13.These cases show that compilation and page-count metadata can pass even when the layout remains defective or visually invalid.
6 Conclusion
The paper defines Visual Typesetting Optimization as a missing document-automation stage and introduces PaperFit, which bridges compilable and publication-ready LaTeX through vision-in-the-loop repair and validation. On PaperFit-Bench, PaperFit achieves perfect compile success, the highest VLM score, and an 80.5% page-budget hit rate.
- Conclusion: PaperFit addresses Visual Typesetting Optimization through multi-source evidence integration, constrained repair policies, and checklist-gated validation.The approach targets the gap between compilable and publication-ready LaTeX.
- Conclusion: 200 papers, 10 templates, and 13 defects comprise PaperFit-Bench, the benchmark used to evaluate PaperFit.These benchmark dimensions are stated alongside the reported outcomes.
- Conclusion: 80.5% page-budget hit rate is achieved by PaperFit, alongside perfect compile success and the highest VLM score.The conclusion reports these as PaperFit-Bench outcomes.
Appendix · A Benchmark Papers
The PaperFit-Bench corpus is fully cataloged in Tables 9–16, which organize the benchmark papers by conference across the appendix.
- A Benchmark Papers: PaperFit-Bench papers are cataloged in Tables 9–16 with complete benchmark-corpus details.The appendix identifies these tables as the source for the corpus catalog.
- A Benchmark Papers: Table 9 organizes benchmark papers by conference.
- A Benchmark Papers: Table 10 continues the conference-organized benchmark-paper listing.
- A Benchmark Papers: Table 11 continues the conference-organized benchmark-paper listing.
- A Benchmark Papers: Table 12 continues the conference-organized benchmark-paper listing.
- A Benchmark Papers: Table 13 continues the conference-organized benchmark-paper listing.
- A Benchmark Papers: Tables 14–16 continue the conference-organized benchmark-paper listing.
B Prompt Records
Prompt records make model-facing baselines reproducible and auditable by fixing templates, saving run artifacts, and defining each method’s evidence boundary. PaperFit’s prompt uniquely encodes the VTO taxonomy, repair constraints, priorities, and quality-gate workflow.
- Prompt records: Fixed prompt templates and saved run artifacts support reproducibility, while RuleLog remains a deterministic rule/log baseline without an LLM prompt.TextST, TextMR, and VisualST use versioned templates; VisualMR and PaperFit record actual per-case prompts in reports.
- Prompt records: TextST uses source-only repair, TextMR adds compile-log feedback, VisualST adds rendered pages for one edit turn, and VisualMR uses fixed-round visual-agent instructions.These input boundaries distinguish the baselines by available evidence and iteration scope.
- Prompt records: PaperFit’s prompt includes the VTO taxonomy, forbidden operations, repair priority, and quality-gate workflow, unlike baseline prompts that exclude its structured artifacts.The records prevent hidden prompt differences from being treated as implementation details.
- Prompt records: Runtime prompt fields include the main TeX filename, target page count, maximum rounds, source window, compile-log excerpt, and rendered page images.These case-specific fields are filled during execution rather than fixed entirely in the core templates.
C Reproducibility Notes
The evaluation preserves case-level artifacts and derives aggregate results from those reports. Missing, non-compilable, and non-renderable outputs remain included under common failure-accounting rules, with compilation and rendering performed locally.
- Artifact Recording: Each method-case evaluation records generated source, compile logs, rendered pages when available, programmatic metric outputs, and VLM reports.These artifacts support inspection of both execution and visual results.
- Aggregation: Aggregated tables are computed from case-level reports rather than hand-entered summary values.This preserves a traceable link between reported aggregates and individual evaluations.
- Failure Handling: Missing, non-compilable, and non-renderable outputs are handled under the same failure-accounting rules used for all methods.Such outputs are not silently dropped from evaluation accounting.
- Execution Environment: All LaTeX outputs are compiled with the local TeX toolchain and rendered into pages when available.The procedure uses the local compilation and rendering environment for evaluation artifacts.
D Limitations
PaperFit remains limited by VLM sensitivity to subtle layout defects and reduced page-budget compliance on highly complex cases, while its scope and computational efficiency are constrained.
- D Limitations: Approximately 70% page-budget hit rate is achieved on hard cases with 5–8 co-occurring perturbations, showing that highly complex multi-defect scenarios remain challenging.Subtle microtypographic defects and font-level kerning errors may also be missed by current vision models.
- D Limitations: PaperFit is currently limited to LaTeX projects and has been evaluated only on English-language academic papers.Coverage of other document languages is left to future work.
- D Limitations: Multi-round recompilation and re-rendering incurs higher computational cost than single-pass methods, motivating overhead reduction while preserving repair quality.Reducing this overhead is identified as an important practical direction.