DOMINO: A Dual-System for Multi-step Visual Language Reasoning

1 INTRODUCTION

• DOMINO addresses the challenge of combining multi-step reasoning, visual text or number extraction, and arithmetic or logical operations for chart and plot questions.End-to-end models answer in one step, while chart-to-text pipelines may lose or include irrelevant information.Source: §1 · p. 1
• DOMINO alternates System-1 visual extraction with System-2 deliberate reasoning, which decomposes questions and requests image information as needed.System-2 uses extracted intermediate results to continue reasoning or derive an answer.Source: §1 · pp. 1–2
• System-1 is based on a pretrained chart-understanding vision-language model, while System-2 uses an LLM adapted through few-shot prompting or fine-tuning.The paper also creates template-generated training examples for atomic chart operations.Source: §1 · p. 2
• With 100 training examples, DOMINO outperforms the best fully supervised method by 5.7% accuracy on ChartQA.The paper also reports that prompting without fine-tuning outperforms few-shot pipeline methods on in- and out-of-distribution data.Source: §1 · p. 2

2 RELATED WORK

• Chart reasoning combines information-dense image understanding with multi-step arithmetic or logical reasoning, and prior work studies both supervised end-to-end and pipeline approaches.The cited research includes chart and plot question-answering work.Source: §2 · pp. 2–3
• Supervised models fuse visual and language inputs but can struggle with complex reasoning, whereas DOMINO uses an LLM for multi-step reasoning.Some approaches incorporate OCR, which adds computational cost and can fall short when charts lack explicit text or numbers.Source: §2 · p. 3
• Pipeline systems extract chart information—sometimes as a textual table—then apply a question-answering or reasoning model; DOMINO alternates extraction and reasoning instead.Prior pipelines use methods ranging from OCR and heuristics to deep models, followed by language or code models.Source: §2 · p. 3

3 DOMINO

• DOMINO differs from single-step end-to-end models and whole-chart table pipelines by retrieving chart information one reasoning step at a time.This design allows more interaction between textual and visual modalities.Source: §3 · p. 3

3.1 SYSTEM-1

• System-2 issues chart queries to System-1 or derives the final answer from intermediate query-result pairs.System-1 encodes the chart image and uses the query to guide its text decoder in generating an intermediate result.Source: §3 · p. 3, §3.1 · pp. 3–4
• System-1’s atomic operations include Describe, which returns key chart elements to help System-2 ground subsequent queries.The paper makes Describe the first reasoning step because it is vital for valid follow-up queries.Source: §3.1 · p. 4
• Extract-Point retrieves one specified chart value, while Extract-Group retrieves all values for a specified group.Extract-Point avoids extracting every chart value when only one is needed.Source: §3.1 · p. 4

3.2 SYSTEM-2

• System-2 combines atomic operations to collect information, then uses chain-of-thought reasoning to synthesize the results into an answer.The paper explores few-shot prompting and prompting plus fine-tuning; fine-tuning uses at most 100 examples and targets System-2-generated query and answer text.Source: §3.2 · pp. 4–5

4.1 DATASETS

• Evaluation and fine-tuning use ChartQA and PlotQA, while DVQA and FigureQA provide additional datasets not used for fine-tuning.ChartQA includes machine-generated and human-written questions; PlotQA has extractive-focused v1 and more numerical-reasoning-focused v2 sets.Source: §4.1 · p. 5

4.2 TRAINING DETAILS

• System-1 uses DePlot fine-tuned on synthetic operation examples, and System-2 uses LLaMA-2 70B with at most 100 expert-annotated examples.The paper generates 774,019 examples for System-1 from ChartQA and PlotQA training data.Source: §4.2 · p. 5

4.3 BASELINE MODELS AND EVALUATION METRICS

• The evaluation compares DOMINO with supervised end-to-end models and chart-to-table pipelines using relaxed accuracy, which allows 5% tolerance for numeric answers.The pipeline baselines include DePlot paired with several LLMs and prompting configurations.Source: §4.3 · pp. 5–6

• Without fine-tuning, DOMINO scores 75.8% versus PaLI-X with OCR's 72.3% on ChartQA, while supervised methods outperform both DePlot and DOMINO on PlotQA.The authors attribute the ChartQA result to its more diverse and complex questions, and PlotQA's supervised advantage to its template-based questions and training-data bias.Source: §5 · p. 6
• With self-consistency, DOMINO beats the best fully supervised method by 5.7% in accuracy on ChartQA and raises PlotQA-V2 from 71.3% to 80.7%.Source: §5 · p. 7
• DOMINO outperforms DePlot with GPT-3, LLaMA-2 (70B), or FlanPaLM (540B) on ChartQA and PlotQA, and DePlot with GPT-4 on ChartQA2.The authors associate this with greater interaction between language and vision and less redundant information than DePlot's chart-to-table conversion.Source: §5 · p. 7
• On OOD DVQA and FigureQA, DOMINO trails supervised methods but generalizes better than DePlot among few-shot approaches; Table 2 reports results on 10K random evaluation samples.The authors attribute the supervised advantage to synthetic datasets and baseline fine-tuning on their training partitions; they suggest a more advanced vision module as future work.Source: §5 · p. 7, Table 2 · p. 7

Effectiveness of image description in addressing hallucination

• Removing Describe generally lowers DOMINO’s performance, except on PlotQA-V2, where synthetic questions provide more context than flexible natural wording.Describe provides initial context for System-2 to generate appropriate queries, particularly when questions do not supply enough information for reasoning.Source: §6 · p. 7

Limitations

• GPT4 was not evaluated on the larger PlotQA sets because of cost and usage limits.Source: §6 · p. 7

Skills learnt from fine-tuning System-2

• Fine-tuning System-2’s decomposition and answer-deduction skills has different effects: answer-only training hurts ChartQA more, while PlotQA results vary by split and supervision target.On PlotQA-V2, fine-tuning only answer deduction or only decomposition can improve performance; the authors relate this to its restricted question types.Source: §6 · p. 8, Table 3 · p. 8

Robustness in handling complex charts

• DOMINO, with either frozen or fine-tuned System-2, consistently outperforms DePlot as underlying chart-table length increases.The authors attribute this robustness to avoiding redundant information from converting the whole chart into a table; Figure 2 groups performance by table length and shows question ratios by interval.Source: §6 · p. 8, Figure 2 · p. 8

Data efficiency of reasoning-based fine-tuning

• Fine-tuned DOMINO generally outperforms fine-tuned DePlot across training-set sizes and needs more than 50 PlotQA-V1 or 20 PlotQA-V2 examples to beat the frozen LLM.Figure 3 compares performance across training-example counts; at zero examples, neither method’s LLM is fine-tuned.Source: §6 · p. 8, Figure 3 · p. 9

Error Analysis

• In PlotQA-V2, fine-tuned DOMINO reduces errors by 45% to 68% versus DePlot on reasoning templates, while improving across all template types.The largest reductions are for arithmetic, compound, comparison, and min-max questions.Source: §6 · p. 9

Case studies on ChartQA-human

• Table 4 contrasts DePlot’s correct chart table but wrong value extraction with DOMINO’s invalid query, which fine-tuning helps System-2 correct.In the DOMINO example, System-2 fails to use an earlier reasoning step identifying two chart groups before generating its query.Source: §6 · p. 9, Table 4 · p. 9

7 CONCLUSION

• DOMINO alternates System-1 targeted image extraction with System-2 task decomposition and answer generation.The paper compares its performance with supervised and pipeline approaches across chart and plot question-answering datasets.Source: §7 · p. 9
• A general chart description helps System-2 perform task decomposition.Source: §7 · p. 9
• DOMINO is more robust when handling complex charts.Source: §7 · p. 9
• System-2 fine-tuning is data-efficient, but benefits differ according to the skills acquired during fine-tuning.Source: §7 · p. 9

ETHICS STATEMENT

• Step-by-step reasoning provides intermediate steps that the authors hope will build transparency and trust and ease bug-fixing.Source: ETHICS STATEMENT · p. 10
• Chart question-answering models could potentially be abused to mislead the public about chart content and implications.Source: ETHICS STATEMENT · p. 10
• The authors cannot guarantee that model outputs will always be correct, despite comparable or state-of-the-art results on evaluated datasets.Source: ETHICS STATEMENT · p. 10
• The authors report sharing hyperparameter settings for reproducibility and state that they will open-source the code.Source: ETHICS STATEMENT · p. 10

A.1.1 STATISTICS

• Test-set statistics are reported for the datasets used in the paper.Source: §A.1.1 · p. 13
• Evaluation randomly samples 10K examples from each synthetic dataset: PlotQA, DVQA, and FigureQA.Source: §A.1.1 · p. 13
• The statistics table is organized by dataset, number of charts, and number of QA pairs.Source: §A.1.1 · p. 13

A.1.2 TEMPLATES FOR DATA GENERATION FOR SYSTEM-1

• Templates generate System-1 training data using ChartQA and PlotQA training sets, with examples of atomic-operation data shown in Tables 5 and 6.Source: §A.1.2 · p. 13, Table 5 · p. 13, Table 6 · p. 14

A.1.3 EXAMPLE OF ANNOTATED DATA FOR SYSTEM-2

• Annotated System-2 examples show staged chart description, targeted extraction, and reasoning, including finding a minimum across years or counting values above a threshold.Source: §A.1.3 · p. 14

A.2 EVALUATION PROMPTS

• The evaluation prompts include a DePlot one-shot prompt with a table and five question-answer pairs, and a five-shot prompt for DOMINO on ChartQA.Source: §A.2 · pp. 14, 16
• Further prompt examples use table data to find a minimum, sum selected values, compare groups, and identify the group with the lowest rates.Source: §A.2 · p. 15
• Prompt examples illustrate answering chart questions through value lookup, subtraction, threshold counting, and comparisons across categories or years.Source: §A.2 · pp. 15–16

A.3 ERROR EXAMPLES

• PlotQA V2 error examples are organized by template type, with parenthetical counts denoting the evaluated examples for each type.Source: Table 7 · p. 17
• The shown arithmetic and compound-question examples include discrepancies between listed predictions and calculations from extracted values; some examples note that values require checking against the y-axis.Source: §A.3 · pp. 18–19