PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding

Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, Cha Zhang

arXiv:2501.05452v1cs.CVcs.CL

TL;DR

Structured image understanding demands selective, multihop visual reasoning, which current multimodal models largely replace with text-only intermediate reasoning. REFOCUS adds iterative, code-driven image editing as visual thought, improving table and chart performance and producing stronger supervision data than QA pairs or CoT.

  • Problem

    Structured image understanding requires multihop selective attention, but current multimodal models typically extract image content into text and do not revisit the image.

  • Method

    REFOCUS has multimodal LLMs generate Python code that iteratively masks, highlights, or boxes image regions, returning edited images for further reasoning.

  • Results

    REFOCUS improves performance over GPT-4o without editing, with average gains of 11.0% on table tasks and 6.8% on chart tasks.

  • Takeaways & Limitations

    REFOCUS visual chain-of-thought data provides stronger supervision than standard QA pairs and CoT in the reported finetuning experiments.

Abstract

from arXiv · show

Structured image understanding, such as interpreting tables and charts, requires strategically refocusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, we introduce ReFocus, a simple yet effective framework that equips multimodal LLMs with the ability to generate "visual thoughts" by performing visual editing on the input image through code, shifting and refining their visual focuses. Specifically, ReFocus enables multimodal LLMs to generate Python codes to call tools and modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas, thereby enhancing the visual reasoning process. We experiment upon a wide range of structured image understanding tasks involving tables and charts. ReFocus largely improves performance on all tasks over GPT-4o without visual editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth analysis of the effects of different visual edits, and reasons why ReFocus can improve the performance without introducing additional information. Further, we collect a 14k training set using ReFocus, and prove that such visual chain-of-thought with intermediate information offers a better supervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over CoT.

1. Introduction

Structured image understanding requires multistep selective attention, but current multimodal models primarily reason textually after extracting image content. REFOCUS addresses this gap through iterative image editing, improving table and chart performance and enabling visual chain-of-thought supervision.

  • Motivation: Structured image questions require multiple reasoning steps that selectively connect relevant information while ignoring distractions.The paper illustrates this with identifying Belgian teams, locating their wins, and summing those values.
  • Problem: Current multimodal models typically extract image information into text and then reason without returning to the image.The paper identifies this as a limitation of textual-only intermediate reasoning.
  • Approach: REFOCUS lets multimodal LLMs generate Python-based visual edits, including boxes, highlights, and masks, to direct selective attention.The edited visual artifact becomes an intermediate input for subsequent reasoning.
  • Approach: REFOCUS iteratively executes model-generated edits and feeds each modified image back to the model until it reaches an answer.Examples include masking irrelevant columns and highlighting important rows.
  • Results: 8.0% average gain over QA-pair supervision and 2.6% over CoT supervision was obtained using REFOCUS visual chain-of-thought data.The collected training data includes focus-area bounding boxes and reasoning processes.

2. Related Works

Prior structured-image methods commonly convert images into text or augment VQA training, while programming-based reasoning usually remains textual. REFOCUS instead edits the input image to support multistep visual refocusing without relying on external visual-tool information.

  • Structured Image Understanding: OCR-based approaches convert scientific charts and tables into text before performing textual reasoning.Other methods enhance end-to-end VQA using augmented training data.
  • Visual Reasoning through Programming: Python-programming approaches facilitate chain-of-thought reasoning but typically operate at the text level while leaving images unchanged.This limits their ability to perform visual reasoning through iterative image transformation.
  • Visual Reasoning through Programming: Visual Sketchpad creates visual artifacts with computer-vision tools, but its gains mainly come from external expert information and its tools target object-centric natural images.The paper states that it cannot solve text-rich structured-image problems.
  • Visual Prompting: Visual prompting can improve abilities such as visual grounding, but prior work does not involve multistep reasoning and refocusing processes.The cited comparison concerns visual prompts such as circles around target objects and OCR-derived answer coordinates.

3. REFOCUS

REFOCUS implements iterative visual chain-of-thought by having multimodal LLMs edit structured images with masks, boxes, and highlights. The framework targets tables and several chart types, using coordinate acquisition and task-specific editing tools to refocus model attention.

  • REFOCUS: REFOCUS iterates between multimodal reasoning and Python-based image editing until a final answer is produced.The edited image serves as the new input for the next reasoning step.
  • Tabular Problems: Table experiments use TableVQA, including VWTQ, VWTQ_syn, and VTabFact visual question-answering datasets.VWTQ contains 750 pairs, while VWTQ_syn and VTabFact each contain 250 pairs.
  • Coordinate Acquisition: Coordinate acquisition uses OpenCV contours and structural elements to identify table rows, columns, and chart subplots.For CharXiv, the method supplies the top k longest contours to the LLM, with k = 10 in the experiments.
  • Chart Problems: Chart experiments cover CharXiv multi-subplot reasoning questions and ChartQA horizontal- and vertical-bar subsets.The selected CharXiv subset contains 143 pairs, while the horizontal- and vertical-bar subsets contain 444 and 382 pairs, respectively.
  • Visual Editing Tools: REFOCUS combines masking, box drawing, and highlighting across columns, rows, subplots, and bars selected by x or y values.Tabular tools focus on rows and columns, while chart tools focus on subplots or bars.
  • Visual Editing Tools: The model receives visual-tool function names in its prompt, generates pseudocode, and receives the executed modified image as new input.The actual editing functions run only when the model chooses to perform an edit.

4. Experiments and Analyses

Experiments show that REFOCUS consistently improves structured-image understanding, while analyses examine editing choices, editing frequency, and how visual edits support grounding and OCR.

  • Results: REFOCUS + GPT-4o consistently surpasses baseline models, including vanilla GPT-4o without editing abilities.The strongest gains occur on VWTQ, VWTQ_syn, Horizontal Bar, and Vertical Bar tasks, with 5-10% improvements over GPT-4o.
  • Results: REFOCUS often outperforms GPT-4o given gold text and figure inputs, approaching performance associated with gold text access.The comparison uses direct question answering, whereas Table 1 reports conversational performance without visual editing.
  • Results: REFOCUS can improve other multimodal models when they receive GPT-4o-edited images, despite not being fine-tuned on visual prompts.Consistent improvements are reported for VWTQ, VWTQ_syn, CharXiv, and Vertical Bar tasks.
  • Analysis: Visual edits appear to improve GPT-4o’s visual grounding and OCR while reducing hallucinations, without adding information.Examples describe correcting country selection and character recognition after irrelevant regions are masked or important columns highlighted.
  • Analysis: Experiments compare draw-box, highlight-color, and mask-out tools on VWTQ and VWTQ_syn, finding similar performance across tools.The tool-type comparison is reported in Table 4.
  • Analysis: GPT-4o edits more than 85% of VWTQ and CharXiv images, while editing approximately 40-55% of images in the other datasets.The analysis counts images for which GPT-4o chooses to perform a visual edit.

5. Finetune with REFOCUS data

The paper investigates whether REFOCUS visual reasoning can be distilled into multimodal models as supervision. It constructs a 14k Visual CoT dataset and compares it with QA-only and text-only CoT training.

  • Data collection: A 14k training set is collected by applying REFOCUS with GPT-4o to ChartQA training data.The pipeline processes 15,059 ChartQA examples and retains correct processes with textual CoT, editing code, and focus bounding boxes.
  • Finetuning setup: REFOCUS Visual CoT training includes textual thoughts, refocus bounding boxes, and answers, unlike standard QA training.The Visual CoT input format is image, question, thought1, bounding box, thought2, and answer; QA uses image, question, and answer.
  • Results: 8.0% higher accuracy is achieved by REFOCUS Visual CoT than by the same model trained on the same QA data.The comparison uses Phi-3.5-Vision with identical training-data size and matched hyperparameter search procedures.
  • Results: 2.6% higher accuracy is achieved by REFOCUS Visual CoT than by CoT data without refocus-area bounding boxes.The result isolates the contribution of bounding-box information within the Visual CoT data.
  • Results: REFOCUS Visual CoT finetuning outperforms the base Phi-3.5-Vision model by 3.2% in accuracy.The paper interprets the findings as evidence that visual reasoning data provides stronger supervision than QA pairs and CoT data.

6. Conclusion

The conclusion presents REFOCUS as a Python-based visual-editing framework for structured image understanding and reports a 14k Visual CoT dataset with superior supervision over standard QA data.

  • Conclusion: REFOCUS enhances multimodal LLMs’ structured-image interpretation by incorporating Python-based visual editing of input images.The framework is described as simple yet effective and improves performance on table and chart tasks.
  • Conclusion: The 14k REFOCUS Visual CoT training dataset provides superior supervision to standard QA data.The conclusion identifies the dataset as a further contribution alongside performance gains on structured-image tasks.

A.1. Visual Editing Tools for Charts

REFOCUS provides chart-specific Python tools that highlight relevant bars, mask irrelevant bars, or draw boxes around bars requiring attention.

  • Editing methods: Chart editing uses three visual-editing methods: mask out, draw box, and highlight color.These methods are implemented through Python code.
  • Highlight tools: Highlight Bar at X overlays light red on a selected bar in a vertical bar chart using its x-value.The tool focuses attention on the chosen vertical bar.
  • Highlight tools: Highlight Bar at Y overlays light red on a selected bar in a horizontal bar chart using its y-value.The tool focuses attention on the chosen horizontal bar.
  • Mask tools: Mask Bar at X or Y places a white mask over unneeded bars in vertical or horizontal bar charts.The x-variant applies to vertical charts, while the y-variant applies to horizontal charts.
  • Box tools: Draw Bar at X or Y overlays a solid red bounding box on bars requiring focus in vertical or horizontal charts.The x-variant applies to vertical charts, while the y-variant applies to horizontal charts.

A.2. Experiment Details

The experiments use deterministic evaluation with GPT-4 judging prediction correctness, and report GPU resources for open-source multimodal models.

  • Temperature is set to 0 for all experiments.
  • GPT-4 evaluates predictions by comparing each prediction with the gold answer and deciding correctness.
  • Section 4 inference uses four NVIDIA Quadro RTX 8000 GPUs for several open-source multimodal LLMs.The models include LLaVA-NeXT-7B, LLaVA-NeXT-13B, LLaVA-NeXT-34B, and Phi 3 vision (4B).
  • The Section 4 experiments cost around 40 GPU hours.

B.1. REFOCUS Dataset Statistics

The dataset section provides detailed REFOCUS statistics and points readers to an example input.

  • Listing 1 provides an example input data instance.
  • Detailed REFOCUS dataset statistics are presented in Table 6.

B.2. Finetune Experiment Details

The fine-tuning experiments use full training on Phi-3.5-vision with hyperparameter search over learning rate, epochs, and edited-image inclusion.

  • Fine-tuning uses 8 NVIDIA RTX A6000 GPUs with 48GB RAM per GPU, following Phi-3.5-vision instructions.
  • The experiments use full training and search hyperparameters to report the best-performing model.
  • The hyperparameter search varies learning rate, epoch number, and whether edited images are included in training input.

B.3. SFT Result Analyses

REFOCUS fine-tuning consistently improves chart performance, while the prompt examples show how visual edits select relevant chart elements and table rows or columns before answering.

  • SFT Result Analyses: REFOCUS data consistently improves chart problems in supervised fine-tuning.
  • SFT Result Analyses: 5.4% improvement is reported on Vertical Bar problems using 14k training data.
  • SFT Result Analyses: Fine-tuning with de facto QA data impairs chain-of-thought ability relative to the original model.
  • Data Format: REFOCUS data pairs generated code and answers based on edited images, while vcot_input serves as the target answer.
  • Table Prompts: For a table question, the example draws attention to the Wins column and Els's row before answering 2.
  • Chart Prompts: For a horizontal bar chart, the example focuses on y values whose blue bars exceed 20 before averaging them.
  • Chart Prompts: The selected chart values are Accessibility, Speed, Flexibility, and Willingness to take risks, producing a final average of 24.75.

D. SFT Qualitative Examples

Phi-3.5-vision finetuned on REFOCUS visual chain-of-thought data outputs textual descriptions with bounding-box coordinates identifying areas to focus on. These areas are visualized as red boxes in examples from horizontal and vertical bar-chart sets and compared with REFOCUS plus GPT-4o prompting.

  • Phi-3.5-vision finetuned on REFOCUS visual CoT data outputs text containing bounding-box coordinates for areas to focus on.The coordinates are drawn as red boxes for illustration.
  • The qualitative examples come from the Horizontal Bar and Vertical Bar sets.
  • Figure 8 compares the finetuned Phi-3.5-vision focus areas with the output produced by REFOCUS plus GPT-4o prompting.
Loading 2501.05452v1…