PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modals

Fanglong Yao, Changyuan Tian, Jintao Liu, Zequn Zhang, Qing Liu, Li Jin, Shuchao Li, Xiaoyu Li, Xian Sun

arXiv:2308.06207v1cs.CL

TL;DR

CoT’s linear, step-by-step reasoning is not designed for expert-like high-order multi-hop reasoning and multimodal comparative judgment. The paper proposes multimodal HoT, using textual and visual hypergraphs with cross-modal interaction. On ScienceQA, HoT-enhanced models outperform CoT-based GPT-3.5 and ChatGPT.

  • Problem

    CoT’s linear, sequential reasoning is limited for expert-like high-order multi-hop reasoning and multimodal comparative judgment.

    Source: §1 · p. 2

  • Method

    HoT represents textual reasoning with triplet-based thoughts and multi-hop paths, and interacts with visual HoT through cross-modal co-attention graph learning.

    Source: §1 · p. 2, §3.1.1 · p. 5

  • Results

    On ScienceQA, HoT-enhanced foundation models outperform CoT-based GPT-3.5 and ChatGPT.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The ScienceQA results indicate that HoT-enhanced foundation models can support high-order multi-hop reasoning and multimodal comparative judgment.

    Source: §4.4 · p. 8, §5 · p. 10

    The authors say current multimodal HoT has not achieved as excellent performance as they imagined.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reasoning ability is one of the most crucial capabilities of a foundation model, signifying its capacity to address complex reasoning tasks. Chain-of-Thought (CoT) technique is widely regarded as one of the effective methods for enhancing the reasoning ability of foundation models and has garnered significant attention. However, the reasoning process of CoT is linear, step-by-step, similar to personal logical reasoning, suitable for solving general and slightly complicated problems. On the contrary, the thinking pattern of an expert owns two prominent characteristics that cannot be handled appropriately in CoT, i.e., high-order multi-hop reasoning and multimodal comparative judgement. Therefore, the core motivation of this paper is transcending CoT to construct a reasoning paradigm that can think like an expert. The hyperedge of a hypergraph could connect various vertices, making it naturally suitable for modelling high-order relationships. Inspired by this, this paper innovatively proposes a multimodal Hypergraph-of-Thought (HoT) reasoning paradigm, which enables the foundation models to possess the expert-level ability of high-order multi-hop reasoning and multimodal comparative judgement. Specifically, a textual hypergraph-of-thought is constructed utilizing triple as the primary thought to model higher-order relationships, and a hyperedge-of-thought is generated through multi-hop walking paths to achieve multi-hop inference. Furthermore, we devise a visual hypergraph-of-thought to interact with the textual hypergraph-of-thought via Cross-modal Co-Attention Graph Learning for multimodal comparative verification. Experimentations on the ScienceQA benchmark demonstrate the proposed HoT-based T5 outperforms CoT-based GPT3.5 and chatGPT, which is on par with CoT-based GPT4 with a lower model size.

1 Introduction

The paper argues that linear Chain-of-Thought (CoT) is limited for expert-like high-order multi-hop reasoning and multimodal comparative judgment. It proposes Hypergraph-of-Thought (HoT), combining textual and visual hypergraphs, and reports that HoT-based T5 outperforms CoT-based GPT-3.5 and ChatGPT on ScienceQA.

1 Introduction

  • CoT’s linear steps make leaping reasoning difficult and inadequately represent concurrent steps and conflicts, motivating a paradigm for expert-like reasoning.The paper characterizes expert reasoning by high-order multi-hop reasoning and multimodal comparative judgment; hyperedges can connect arbitrary numbers of vertices to model higher-order relationships.Source: §1 · p. 2
  • HoT models expert reasoning as a hypergraph and can be adjusted into CoT, ToT, or GoT by changing its paths or edges.The figure describes CoT as one path without hyperedges, ToT as multiple non-intersecting paths, and GoT as ordinary two-thought edges.Source: §1 · p. 2, Figure 1 · p. 2
  • On ScienceQA, HoT-based T5 outperforms CoT-based GPT-3.5 and ChatGPT.The authors describe this result as demonstrating HoT’s potential to enhance foundation-model reasoning.Source: §1 · p. 3
  • Textual HoT uses triplets as thoughts and multi-hop walking paths as hyperedges to represent higher-order relations and support multi-hop inference.AllSet Transformer performs alternating thought-to-hyperedge and hyperedge-to-thought updates.Source: §1 · p. 3
  • Visual HoT interacts with textual HoT through Cross-modal Co-Attention Graph Learning for multimodal comparative verification.The paper presents this interaction as a component of its multimodal HoT reasoning paradigm.Source: §1 · p. 3

2 Related work

Related work reviews CoT methods spanning prompting, fine-tuning, and multimodal reasoning, alongside vision-language instruction-following models. It also describes a two-stage multimodal CoT framework that generates rationales before answers.

2.1 CoT

  • Fine-tuning-based CoT uses rationale-generating smaller models, including approaches distilled from large-model reasoning samples [22] [23] [24].The review contrasts these methods with prompt-based CoT, which it says relies on extensive models with billions of parameters.Source: §2.1 · p. 3
  • CoT work includes few-shot and zero-shot prompting [14, 21], as well as extensions to multimodal inference and graph-based reasoning [25] [26].MM-CoT generates rationales first and then infers the final answer; GoT adds graphs to model human thinking processes.Source: §2.1 · p. 3
  • The illustrated multimodal CoT framework encodes text and images separately, fuses textual, visual, and cross-modal information to generate rationales, then uses rationale and context to produce answers.The text and visual foundation-model parameters are frozen during training.Source: Figure 2 · p. 4

2.2 Vision-Language LLMs

  • Vision-language instruction-following work includes LLaVA [27], modality adapters in MMA [28], rationale teaching signals in T-SciQ [29], and scientific instruction tuning in SciTune [30].The review describes these approaches as using generated or human-provided multimodal or scientific instruction data to tune models.Source: §2.2 · p. 4

3.1 Hypergraphof-Thought Construction

The section constructs separate text and image hypergraphs to model higher-order semantic relationships, then aligns information across modalities. Text hyperedges represent multi-hop reasoning paths, while image hyperedges are formed from clusters of image patches.

3.1 Hypergraphof-Thought Construction

  • The construction models higher-order relationships in text and images with separate hypergraphs, then aligns semantic information between the modalities.Hyperedges can connect multiple objects, supporting the section's use of hypergraphs for higher-order relationships.Source: §3.1 · pp. 4–5

3.1.1 Textual Hypergraph-of-Thought Construction

  • Text hyperedges are multi-hop reasoning paths over thoughts, built by random walks whose basic unit is a triplet.The section describes both one-hop and k-hop walks, with each successive walk starting at the previous endpoint.Source: §3.1.1 · p. 5

3.1.2 Visual Hypergraph-of-Thought Construction

  • Swin produces image-patch representations, and K-means forms the image hypergraph by treating each cluster as a hyperedge.The number of clusters, m, determines the number of image hyperedges.Source: §3.1.2 · p. 6

3.2 Textual and Visual Hypergraph-of-Thought Encoding

The AllSet Transformer encodes both textual and visual hypergraphs through node-to-hyperedge and hyperedge-to-node functions. Text-node embeddings come from text-encoder outputs, while image-node representations come from the Swin encoder.

3.2.1 Hypergraph Encoder

  • The AllSet Transformer serves as the hypergraph encoder, using transformer-parameterized node-to-hyperedge and hyperedge-to-node functions.Its described update includes layer normalization and an MLP.Source: §3.2.1 · p. 6

3.2.2 Textual Hypergraph-of-Thought Encoding

  • Text thoughts are marked in a formatted token sequence, and text-encoder outputs at the marker positions initialize node representations before AllSet encoding.The text hypergraph and initial node embeddings are the inputs to the AllSet Transformer encoder.Source: §3.2.2 · pp. 6–7

3.2.3 Visual Hypergraph-of-Thought Encoding

  • Image nodes use Swin-derived patch representations directly, without an additional embedding layer, before AllSet encoding.Source: §3.2.3 · p. 7

3.3 Cross-HoT Interaction

The cross-modal module interacts through text and image hyperedge representations. It uses their representations to construct a co-attention graph and obtain a fused representation.

3.3 Cross-HoT Interaction

  • The text and image hyperedge representations are formed from vectors representing the nodes in each hyperedge.For image hyperedges, the represented nodes are image patches.Source: §3.3 · p. 7
  • A vector inner product between text and image hyperedge representations constructs the Cross-Modal Co-Attention Graph, yielding a final fused representation.Source: §3.3 · p. 7

4 Experiments

Across ScienceQA experiments, HoT-T5 outperformed previous methods and even surpassed human performance. The experiments also report gains for paired-image questions and contributions from each HoT component.

4.1 Dataset

  • ScienceQA contains 21,000 multiple-choice questions across three subjects, 26 topics, 127 categories, and 379 skills, split into training, validation, and test sets.The respective split sizes are 12,726, 4,241, and 4,241 examples.Source: §4.1 · p. 7

4.2 Implementation Details

  • Experiments used T5-base and T5-large initialized from UnifiedQA, with DETR visual features, 20 fine-tuning epochs, and a 5e-5 learning rate.The setup used six NVIDIA Tesla V100 32G GPUs; maximum input length was 512, and batch sizes were 8 and 4 for base and large models.Source: §4.2 · pp. 7–8

4.4 Main Results

  • HoT-T5 outperformed previous methods and surpassed human performance; Table 1 reports overall baseline comparisons on ScienceQA.The authors attribute the improvement to Textual HoT, Visual HoT, and Cross-HoT Interaction.Source: Table 1 · p. 8, §4.4 · p. 8

4.3 Baselines

  • Baselines included VQA models, text-to-text language models, and CoT-prompted text-to-text LLMs, with the latter using generated image captions as vision semantics.The listed text-to-text models include UnifiedQA and GPT-3.5.Source: §4.3 · p. 8

4.4 Main Results

  • Among eight question classes, HoT-T5Large significantly improved on questions accompanied by paired images, compared with caption-based vision semantics.The cited caption-based methods are UnifiedQA and GPT-3.5.Source: §4.4 · p. 8
  • The two-stage HoT-T5Large framework was reported to contribute to its superior results.The authors connect this finding to leveraging multimodal inputs and image features.Source: §4.4 · p. 8

4.5 Ablation Study

  • Removing Textual HoT, Visual HoT, or Cross-HoT Interaction reduced performance; the authors link these components to text semantics, visual understanding, and cross-modal semantic combination.The authors qualify the proposed explanation for the Cross-HoT Interaction drop as a possibility.Source: Table 2 · p. 9, §4.5 · p. 9

4.6 Case Study

  • In two ScienceQA examples, CoT produced wrong rationales and answers in Figure 4, while HoT produced correct results; in Figure 5, HoT used correct rationales more effectively.For Figure 5, CoT produced correct rationales but wrong answers.Source: §4.6 · p. 9, Figure 4 · p. 9

5 Conclusion

The paper proposes multimodal Hypergraph-of-Thought (HoT), combining textual and visual hypergraphs to support high-order multi-hop reasoning and multimodal comparative judgement. On ScienceQA, HoT-enhanced models outperform CoT-based GPT-3.5 and ChatGPT, though the current approach falls short of the authors’ expectations.

5 Conclusion

  • The proposed multimodal HoT aims to move beyond CoT by supporting expert-like high-order multi-hop reasoning and multimodal comparative judgement.The paper describes this as a new reasoning paradigm for foundation models.Source: §5 · pp. 9–10
  • On ScienceQA, HoT-enhanced foundation models outperform CoT-based GPT-3.5 and ChatGPT.The conclusion also notes that current multimodal HoT has not achieved performance as strong as imagined.Source: §5 · p. 10
  • Textual HoT uses triplets and multi-hop walking paths, while visual HoT interacts with it through cross-modal co-attention graph learning.The visual HoT is built from image features from a visual foundation model.Source: §5 · p. 10
  • The current multimodal HoT has not achieved performance as strong as the authors imagined.The authors present it as a potential new paradigm for instruction learning in foundation models.Source: §5 · p. 10

Related papers

Loading 2308.06207v1…