PaperlayerView exact revision on arXiv

Source-linked AI summary

FigAct: Turning Scientific Figures into Active Canvases for Explanation

Shishi Xiao, Zichao Wang, Alexa Siu, David H. Laidlaw, Jennifer Healey

arXiv:2609.36190v1cs.AI

TL;DR

MLLMs typically explain scientific figures with text that readers must map back to the visual. FigAct turns figures into question-conditioned presentations by pairing narrations with grounded evidence and visual actions; experiments report improvements across MLLM backbones and lower SVG token use.

  • Problem

    MLLMs typically answer questions about scientific figures with text, leaving readers to map the explanation back to the figure.

    Source: §1 · p. 1

  • Method

    FigAct plans short narration beats, grounds each beat in figure elements, and applies visual actions to present that evidence.

    Source: §3 · p. 3

  • Results

    FigAct improves grounding, action generation, and end-to-end presentation quality across open-weight and proprietary MLLMs while reducing SVG token usage by approximately 40–50×.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Human evaluation indicates that grounded visual evidence and presentation actions make scientific figures easier to follow.

    Source: §8 · p. 10

    On real-world figures, one reported failure is imprecise focus: the selected region can be too broad and retain distracting neighboring content.

    Source: §E.2.2 · p. 24

Abstract

from arXiv · show

Scientific figures are designed to communicate information visually, yet MLLMs typically explain them by translating their visual content back into text. This requires readers to manually map the resulting explanations back to the figure. Inspired by how people present visual information, we introduce FigAct, a framework that transforms static scientific figures into question-conditioned visual presentations by acting directly on their existing graphical elements. Like a human presenter, FigAct generates a sequence of short narrations, grounds each narration in the corresponding visual evidence, and applies visual actions to guide the viewer's attention. We develop a hierarchical search strategy for efficient element localization, reducing token usage by approximately 40$\times$. We further train FigAct-8B using three task-specific rewards for grounding accuracy, search efficiency, and rendering quality. We further build a human-verified benchmark from figures in real-world scientific papers to evaluate the ability of MLLMs to generate grounded visual explanations. Our results demonstrate the effectiveness of FigAct and show that treating scientific figures as presentation canvases makes explanations clearer and easier to follow.

1 INTRODUCTION

FigAct turns static scientific figures into question-conditioned visual presentations, pairing short narrations with grounded evidence and visual actions. Its experiments show gains across MLLM backbones and reduced SVG token use.

Motivation and approach

  • FigAct responds to questions with a sequence of visual beats, each pairing a short narration with its visual evidence and attention-guiding actions.This approach addresses the need for readers to map text-based answers back to dense figures.Source: §1 · pp. 1–2

Framework

  • The Plan–Ground–Act framework organizes what to communicate, what evidence to show, and how to present that evidence.Plan sequences the explanation, Ground identifies relevant figure elements, and Act selects visual transformations.Source: §1 · p. 2
  • FigAct uses SVG elements and a geometry-based forest for coarse-to-fine evidence search, then maps high-level presentation actions to rendered elements.This design addresses SVGs with thousands of low-level elements and separates presentation decisions from visual execution.Source: §1 · p. 2

Experiments

  • FigAct improves grounding, action generation, and end-to-end presentation quality across open-weight and proprietary MLLMs while reducing SVG token usage by approximately 40–50×.FigAct-8B also transfers to real-world figures, and human evaluation confirms benefits from grounded highlighting and visual actions.Source: §1 · p. 2

2 RELATED WORK

Prior figure-reasoning systems generally produce text answers, while figure-transformation systems often treat figures as static content or create new ones. FigAct instead progressively augments the original figure across presentation beats.

Figure reasoning

  • Unlike prior scientific-figure question-answering work, including SPIQA, FigAct produces a visual presentation rather than a text-based answer.Related work on mathematical figures, charts, and diagrams also generates text-based answers.Source: §2 · p. 2

Figure transformation

  • Prior work transforms papers into slides, posters, videos, or interactive agents, but largely treats figures as static content or generates new figures.FigAct instead augments the original figure across beats to communicate an answer.Source: §2 · p. 2

Plan–Ground–Act framework

  • FigAct’s Plan–Ground–Act overview links explanation planning, visual-evidence grounding, and presentation actions.The stages form a sequence from text explanation to grounded elements and then a temporal visual presentation.Source: Figure 2 · p. 3

Structured figure representations

  • SVG provides element-level addressability for fine-grained grounding and editability for manipulating selected evidence.FigAct builds on structured visual languages such as SVG and TikZ used to reconstruct or create editable figure representations.Source: §2 · p. 3

3 PROBLEM FORMULATION

Visual Question Presentation answers a question about a scientific figure through a reader-facing presentation made of explanatory beats.

Task definition

  • Visual Question Presentation (VQP) takes a figure and user question and aims to communicate the answer through narration, grounded evidence, and visual actions.A beat combines a short narration, supporting figure elements, and actions applied to those elements.Source: §3 · p. 3

Stage-wise generation

  • PLAN generates the narration sequence, GROUND identifies corresponding visual evidence, and ACT selects actions conditioned on both.These stages produce the presentation beat sequence.Source: §3 · p. 3

Presentation structure

  • Each beat is an explanatory step organized around a coherent purpose, with evidence drawn from addressable elements of the figure.The paper borrows the notion of a beat from film theory.Source: §3 · p. 3

4 PLAN–GROUND–ACT FRAMEWORK

FigAct models a human presenter’s process through planning, grounding, and acting. It turns explanations into ordered narrations, locates supporting SVG elements, and applies actions to present them.

4 PLAN–GROUND–ACT FRAMEWORK

  • FigAct divides figure explanation into planning what to communicate, grounding its evidence, and acting to determine how that evidence is presented.The stages model how a presenter directs audience attention to relevant parts of a figure.Source: §4 · p. 3

4.1 PLANNING: WHAT SHOULD BE COMMUNICATED?

  • PLAN decomposes an answer into temporally ordered short narrations, each defining one beat before the presentation moves to the next step.Each narration determines what should be communicated at that point.Source: §4.1 · p. 4

4.2 GROUNDING: WHAT SHOULD BE SHOWN?

  • GROUND selects SVG elements supporting each narration, using addressable, inspectable, and editable elements despite figures containing many low-level primitives.A geometry-based forest groups primitives into spatially nested regions, while context and isolated views support candidate inspection.Source: §4.2 · p. 4
  • At each search turn, the model keeps evidence nodes, expands nodes for finer inspection, or deletes irrelevant branches until no node is expanded.The final evidence comprises the SVG elements covered by retained nodes.Source: §4.2 · p. 4

4.3 ACTING: HOW IT SHOULD BE PRESENTED?

  • ACT chooses ordered actions for grounded evidence, spanning view control, attention control, relational guidance, and information augmentation.A deterministic renderer maps these high-level decisions to low-level SVG transformations.Source: §4.3 · pp. 4–5

5 TWO-STAGE TRAINING

FigAct-8B is initialized with structured Plan–Ground–Act supervision, then optimized with stage-aware reinforcement learning rewards for grounding correctness, search efficiency, and rendered presentation quality.

5.1 STAGE 1: SFT WARM-UP

  • SFT initializes FigAct-8B with the complete Plan–Ground–Act workflow in a unified JSON format for subsequent RL training.Source: §5.1 · p. 5

5.2 STAGE 2: RL TRAINING

  • RL samples Ground–Act trajectories under a fixed plan and evaluates evidence correctness, search efficiency, and whether the rendered presentation communicates the narration.Ground receives grounding, search, and rendering signals; Act receives the rendering signal.Source: §5.2 · pp. 5–6
  • Grounding correctness compares predicted and reference terminal SVG element sets using element-level precision and recall.Expanded elements are recursively replaced with their terminal descendants before comparison.Source: §5.2 · p. 5
  • Search efficiency rewards reaching target evidence with lower cost, measured by the number of child nodes revealed during expansion.The grounding term prevents a short but inaccurate search from receiving a high reward.Source: §5.2 · p. 5
  • Rendered beats receive a semantic-alignment reward against their corresponding narrations using a frozen video–text encoder.The reward is averaged across presentation beats.Source: §5.2 · p. 5
  • Rewards are normalized independently within rollout groups and assigned by stage to preserve distinct signals for Ground and Act.Ground aggregates grounding, search, and rendering signals; Act is evaluated by rendering quality.Source: §5.2 · p. 6

6 DATASET

The paper constructs a real-world scientific-figure benchmark and a larger synthetic supervision set, using semantic SVG structure to derive Plan–Ground–Act training trajectories.

6.1 REAL-WORLD SCIENTIFIC DATA COLLECTION

  • The real-world collection contains 1,666 scientific SVGs, of which 400 figures are curated for evaluation through LLM-assisted annotation reviewed by a human designer.The figures come from 2025–2026 papers under CC BY 4.0; only 25.5% of raw SVGs fit the context window, rising to 73.4% after raster removal and path simplification.Source: §6.1 · p. 6

6.2 CONSTRUCTING FIGACT SUPERVISION

  • Semantic SVG labels and DOM hierarchies support synthetic supervision, yielding 32,319 presentation samples from 12,509 figures across six question types.The semantic structure is removed before training; target elements are projected onto a geometry-based forest to create multi-turn Keep, Expand, and Delete trajectories.Source: §6.2 · pp. 6–7

7 EXPERIMENTS

On a 400-pair real-world benchmark, FigAct improves MLLM grounding, actions, and presentation quality while reducing SVG token use; training gains persist across task complexity, and human evaluation favors visual presentations.

7.1 EXPERIMENTAL SETUP

  • The benchmark comprises 400 question–figure pairs with one question per scientific figure, and evaluation covers Plan, Ground, Act, and final presentation quality.Baselines include proprietary and open-weight MLLMs, evaluated under the same multi-turn protocol.Source: §7.1 · pp. 7–8

7.2 MAIN RESULTS

  • FigAct improves grounding, action generation, and presentation quality across proprietary and open-weight MLLMs while reducing SVG tokens; Gemini 3.5 Flash gains +46.27 Elem. F1 and +18.46 execution success.Pres-Q rises by +0.02 to +0.09 across backbones; proprietary models use over 50× fewer SVG tokens, while Qwen3.5-9B and Qwen3-VL-8B use approximately 40× fewer.Source: §7.2 · p. 8, Table 1 · p. 8

7.3 ABLATION ANALYSIS

  • RL raises Elem. F1 from 52.67 to 62.27 over SFT alone, and combining all three rewards gives the strongest Elem. F1, NVA, and Pres-Q.Removing Rrender causes the largest NVA degradation; Rground also contributes to grounding accuracy and downstream presentation quality.Source: §7.3 · pp. 8–9, Table 2 · p. 8
  • As presentation length, SVG complexity, and search depth increase, FigAct-8B consistently outperforms the base model across Elem. F1, Search-E, and NVA, generally degrading more mildly.Search-E is weighted by grounding F1, so efficiency depends on reaching correct evidence.Source: §7.3 · p. 9, Figure 5 · p. 9

7.4 HUMAN EVALUATION Protocol.

  • Human-study comprehension accuracy rises from 66.7% for text-only explanations to 75.6% with highlighting and 83.3% with full FigAct; visual clarity reaches 5.53 versus 4.87.The study involved 18 graduate students evaluating 20 question–figure pairs each.Source: §7.4 · p. 9

8 CONCLUSION

FigAct turns static scientific figures into question-conditioned visual presentations through a Plan–Ground–Act pipeline. The conclusion reports gains across MLLMs, reduced SVG token usage, and human-evaluation evidence that the presentations are easier to follow.

8 CONCLUSION

  • FigAct turns static scientific figures into question-conditioned visual presentations through a Plan–Ground–Act pipeline.Source: §8 · p. 9
  • The framework improves overall performance across proprietary and open-weight MLLMs while substantially reducing SVG token usage.Source: §8 · p. 9
  • FigAct-8B is trained on a large-scale synthetic dataset with a stage-aware GRPO objective targeting grounding correctness, search efficiency, and rendering quality.Source: §8 · pp. 9–10
  • Human evaluation indicates that linking explanations to visual evidence and presentation actions makes scientific figures easier to follow.Source: §8 · p. 10

A ADDITIONAL METHOD DETAILS

The additional methods construct a geometry-based hierarchy for SVG search, use executable actions to present grounded evidence, and derive synthetic training trajectories from annotated figures.

A.1 GEOMETRY-BASED SVG FOREST CONSTRUCTION

  • Text primitives are grouped by OCR-region overlap when coverage exceeds 0.30, with unmatched primitives clustered by text-only spatial proximity.Primitives assigned to one OCR region are merged; proximity grouping never joins text to visual elements.Source: §A.1 · p. 15
  • Visual primitives are merged only under three high-confidence same-object relations: coincident fill–outline layers, layered raster renderings, and marker–stroke pairs.Marker–stroke grouping attaches each marker only to its nearest eligible stroke.Source: §A.1 · p. 15, Figure 6 · p. 15
  • The shared geometric forest assigns the smallest enclosing valid parent, requires at least 0.85 containment, and restricts visual parents to avoid spurious bounding-box containment.The resulting relations are deterministic and cycle-free while preserving original SVG drawing order.Source: §A.1 · p. 16
  • Grouping reduces the median primitive count by 54.9% and yields 84.3% fewer top-level entry points than the original primitive count, while retaining nested groups.Across 50 sampled figures, medians were 266 primitives, 104 groups, and 30 roots; median maximum depth was 2.Source: §A.1 · p. 16

A.2 ACTION SPACE AND RENDERING

  • FigAct’s action space includes tint, deemphasize, reemphasize, camera, arrow, infer_relation, annotation, and abstraction, with actions ordered within each beat.Source: §A.2 · p. 16, Table 4 · p. 17
  • A deterministic renderer maps semantic actions to element-specific SVG operations, while stateful execution preserves relevant effects and reverts inactive ones between beats.This avoids redundant visual resets and supports smooth transitions while keeping the original SVG state recoverable.Source: §A.2 · pp. 16–17

B.1 SYNTHETIC TRAINING DATA

  • Training trajectories are reconstructed from final grounding targets by labeling nodes Keep, Expand, or Delete and progressively refining only branches containing targets.The reconstruction assumes targets are mutually non-ancestral and matches the inference-time grounding protocol.Source: §B.1 · p. 18

A Simplified Training Sample

B.2 REAL-WORLD EVALUATION DATA

The real-world evaluation set comprises converted scientific figures from recent papers, whose complex SVG structure motivates hierarchical grounding. A model-assisted human process produces verified presentation annotations.

B.2 REAL-WORLD EVALUATION DATA

  • The evaluation collection contains 1,666 SVG figures converted from method figures in 2025–2026 papers from six research venues.The figures were drawn from CC BY 4.0 arXiv source packages and converted from PDF using Inkscape’s Poppler-based importer.Source: §B.2 · p. 19
  • Real-world figures average 302.8 SVG paths, 26.5 embedded raster images, and 461.1 text glyphs, with long-tailed distributions that motivate hierarchical grounding.Text often appears as individual glyph paths, and complex components may be decomposed into many primitives.Source: §B.2 · p. 19
  • Because full-SVG annotation is unreliable and creating a presentation can take up to 1.5 hours, annotators review and iteratively correct model-generated candidates.The resulting human-verified set contains 400 real-world presentation examples produced through one week of annotation.Source: §B.2 · pp. 19–20

C TRAINING DETAILS

The training and evaluation details specify a figure-disjoint data split, stage-aware reinforcement learning, and metrics for each pipeline stage and human evaluation.

C TRAINING DETAILS

  • Synthetic data are split by figure into supervised fine-tuning, reinforcement-learning, and evaluation sets, and FigAct is initialized from Qwen3-VL-8B-Instruct for three-epoch SFT.The splits contain 3,391, 7,869, and 1,249 figures, respectively; LoRA is applied to language-model linear layers while the vision encoder and multimodal aligner are frozen.Source: C TRAINING DETAILS · p. 20
  • Stage-aware GRPO optimizes Ground and Act while keeping Plan fixed, with separate reward channels for grounding correctness, search efficiency, and rendering quality.Ground tokens receive all three rewards; Act tokens receive only rendering quality, and advantages are normalized independently by reward channel.Source: C TRAINING DETAILS · p. 20

D.1.1 PLANNING EVALUATION

  • Planning evaluation measures narration factuality by checking claims against the figure and coverage by testing probe-question answers against target answers.Both metrics are normalized to [0, 1], with higher scores indicating better quality.Source: §D.1.1 · pp. 20–21

D.1.2 VISUAL GROUNDING EVALUATION

  • Visual grounding evaluation measures beat completeness, element-level F1 against human-curated terminal element sets, and search efficiency as minimal required cost divided by actual cost.Element F1 is macro-averaged across beats; search cost counts unique child nodes exposed during hierarchical search.Source: §D.1.2 · p. 21

D.1.3 ACTING EVALUATION

  • Acting evaluation counts the fraction of samples producing successfully rendered, error-free videos after action-schema and grounding-ID checks.Scripts that violate structural or rendering constraints are excluded from subsequent NVA evaluation.Source: §D.1.3 · p. 21

D.1.4 END-TO-END PRESENTATION QUALITY

  • End-to-end presentation quality is judged across content quality, visual effectiveness, and temporal coherence using a rubric-based MLLM evaluation of rendered videos.The reported score is the mean of the three dimension scores.Source: §D.1.4 · p. 22

D.2 HUMAN EVALUATION PROTOCOL

  • In the human study, participants view an explanation, answer a comprehension question, and rate their experience; question–figure pairs are counterbalanced across conditions.Subjective ratings cover comprehension ease, visual clarity, and visual appeal on a 7-point Likert scale.Source: §D.2 · pp. 22–23

E.1 ABLATION STUDIES

The ablations examine grounding views, reward optimization, rollout size, and transfer to real-world figures; further analysis describes action use and failure modes. The paper also identifies limits involving SVG binding, visual scope, and missing document context.

E.1 ABLATION STUDIES

  • Using both isolated and context views gives the best grounding accuracy for both models, while isolated views outperform context-only views, especially on GS@0.75.GS@0.75 measures the proportion of beats with element-level F1 of at least 0.75.Source: §E.1 · p. 23, Table 5 · p. 23
  • Stage-aware GRPO raises Elem. F1 from 61.41 to 62.27, NVA from 70.83 to 72.25, and Pres-Q from 81.53 to 82.85, but has slightly lower search efficiency.It independently normalizes reward channels and routes advantages to Ground and Act; both RL variants improve over SFT.Source: §E.1 · p. 23, Table 6 · p. 23
  • Increasing rollout group size from 4 to 8 yields consistent gains, while increasing it to 16 brings only marginal changes and NVA peaks at G = 8.The effect is not monotonic across all metrics.Source: §E.1 · p. 23, Table 7 · p. 24
  • SFT and RL improve grounding on both synthetic and real-world figures, with RL further improving grounding and action quality on real-world figures.The datasets differ in distribution and difficulty, so their absolute scores are not directly comparable.Source: §E.1 · p. 24, Table 8 · p. 24

E.2.1 ACTION DISTRIBUTION.

  • FigAct-8B shifts from 94.0% tint use in Qwen3-VL-8B to 16.2%, with more varied action use resembling proprietary models.InternVL3.5-8B assigns 73.3% of its actions to tint.Source: §E.2.1 · p. 24, Table 9 · p. 25

E.2.2 FAILURE CASES AND LIMITATIONS.

  • Real-world failures include near-match SVG identifiers that do not exist at runtime, leaving a beat visually unchanged, and overly broad focus that retains distracting neighbors.The authors say identifier confusions are more likely in real-world SVGs, where identifiers can differ by small changes in element type, suffix, or index.Source: §E.2.2 · pp. 24–25, Figure 10 · p. 25
  • Reliable walkthroughs require both exact element binding and visual scope calibrated to the level of detail being presented.The cited failures distinguish finding the right region from selecting the right level of visual detail.Source: §E.2.2 · pp. 24–25
  • FigAct uses figure content without surrounding document context, and raster-only figures need vectorization or element decomposition before processing.Questions depending on main-text or broader scientific context can yield less reliable responses.Source: §E.2.2 · p. 25

F QUALITATIVE RESULTS

This section presents a complete FigAct example and qualitative model comparisons, then gives prompts for planning, grounding, recursive refinement, and visual action. The prompts specify how each stage structures its output and constrains its choices.

F QUALITATIVE RESULTS

  • The qualitative examples include a complete presentation and model comparisons showing successive beats for the same figure and question.The comparison captions describe examples involving AU localization, song identification, and streaming-video response generation.Source: F QUALITATIVE RESULTS · p. 26, Figure 11 · p. 26, Figure 12 · p. 27

Prompt 1: Plan — Atomic Walkthrough Planning

  • The Plan prompt organizes a figure explanation into atomic beats, each aligning brief narration, a concrete visual focus, and one takeaway.It asks for explanatory logic grounded in visible evidence rather than an inventory of objects.Source: Prompt 1: Plan — Atomic Walkthrough… · p. 31

Prompt 2: Ground — Top-Level Selection

  • The top-level Ground prompt selects focal SVG evidence and only the context needed to preserve identity, containment, relationships, orientation, or visual integrity.Candidates can be kept, expanded when too coarse, or kept while their children are inspected.Source: Prompt 2: Ground — Top-Level Selection · pp. 31–32

Prompt 3: Ground — Recursive Parent Expansion

  • Recursive Ground refines only requested beats within an opened parent, selecting direct children and retaining only context needed to keep focal evidence clear.Children can be kept, expanded, or retained while their own children are inspected.Source: Prompt 3: Ground — Recursive Parent… · p. 32

Prompt 4: Act — Executable Visual Presentation

  • The Act prompt converts grounded beats into executable visual actions, using the smallest clear action set while preserving information-bearing colors and coherence across beats.It limits camera actions to three per walkthrough and combines deemphasis, tint, and camera only when all are essential.Source: Prompt 4: Act — Executable Visual… · pp. 33–34

Related papers

Loading 2609.36190v1…