PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models

Yao Yao, Zuchao Li, Hai Zhao

arXiv:2305.16582v2cs.CL

TL;DR

Human reasoning's non-linear structure is not captured by sequential CoT chains. GoT represents thoughts as a graph and integrates that representation into two-stage language-model reasoning; it reports improved accuracy on AQUA-RAT and ScienceQA.

  • Problem

    CoT methods represent reasoning as sequential chains and overlook the structural information and non-linear connections among thoughts.

    Source: §1 · p. 1

  • Method

    GoT models thought units as graph nodes and their connections as edges, encoding the graph and fusing it with language-model representations in a two-stage framework.

    Source: §1 · p. 2

  • Results

    GoT improves over traditional CoT on both AQUA-RAT and ScienceQA, raising ScienceQA accuracy from 85.19% to 87.59%.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    Results support thought-graph features as a more effective approach than methods using only text or vision features on these reasoning tasks.

    Source: §4.1 · p. 7

    Compared with CoT, GoT may incur additional computational costs and slightly slower training times.

    Source: §4.2.2 · p. 9

Abstract

from arXiv · show

With the widespread use of language models (LMs) in NLP tasks, researchers have discovered the potential of Chain-of-thought (CoT) to assist LMs in accomplishing complex reasoning tasks by generating intermediate steps. However, human thought processes are often non-linear, rather than simply sequential chains of thoughts. Therefore, we propose Graph-of-Thought (GoT) reasoning, which models human thought processes not only as a chain but also as a graph. By representing thought units as nodes and connections between them as edges, our approach captures the non-sequential nature of human thinking and allows for a more realistic modeling of thought processes. GoT adopts a two-stage framework with an additional GoT encoder for thought graph representation and fuses the graph representation with the original input representation through a gated fusion mechanism. We evaluate GoT's performance on a text-only reasoning task (AQUA-RAT) and a multimodal reasoning task (ScienceQA). Our model achieves significant improvement over the strong CoT baseline on the AQUA-RAT test set and boosts accuracy from 85.19% to 87.59% using the T5-base model over the state-of-the-art Multimodal-CoT on the ScienceQA test set.

1 Introduction

Because human thought can be non-linear and connected beyond sequential chains, the paper proposes GoT to represent reasoning as a graph as well as a chain. It combines this representation with language-model inputs in a two-stage framework and reports gains on reasoning benchmarks.

  • Prior CoT methods model reasoning as sequential chains and omit structural information, while Tree-of-thoughts represents thoughts as a tree.Source: §1 · p. 1
  • GoT represents thoughts as nodes and connections as edges, aiming to capture linear and nonlinear cognition beyond chains or trees.Source: §1 · pp. 1–2
  • GoT generates rationales before answers, constructing thought graphs with an Extract-Cluster-Coreference process and encoding them with graph attention before gated fusion.Source: §1 · p. 2
  • For multimodal reasoning, GoT integrates visual features alongside text and thought-graph features.Source: §1 · p. 2
  • GoT is reported to surpass ChatGPT by 9.28% and Multimodal-CoT by 2.40%, and to exceed human performance while performing strongly on AQUA-RAT and ScienceQA.Source: §1 · p. 2

2 Graph-of-Thought

GoT extends two-stage rationale and answer generation with a constructed thought graph, encoded alongside text and optionally image features. Its pipeline builds graph nodes from extracted relations, encodes graph structure, and fuses aligned features for prediction.

2 Graph-of-Thought

  • GoT generates rationales first, then predicts the answer using the input and predicted rationales, incorporating thought-graph features in both stages.For multimodal tasks, the framework also encodes images; text-only tasks omit image encoding and fusion.Source: §2 · pp. 2–3, §2.1 · p. 4

2.2 GoT Encoding and Integration

  • Separate encoders represent text, thought graphs, and, for multimodal tasks, images; cross-attention aligns text with graph nodes and image patches.The image pathway is optional for text-only reasoning.Source: §2 · p. 3, §2.2 · p. 4

2.1 GoT Construction

  • ECC constructs thought graphs by extracting subject-verb-object triplets, resolving coreference, and reconnecting nodes into a graph whose adjacency matrix marks edges.The graph is intended to support deductions such as inferring x→z from x→y and y→z.Source: §2.1 · pp. 3–4

2.2.1 Base Encoder

  • A Transformer encodes input text into token representations, while the optional vision encoder projects image-patch features to match the text representation's shape.For text, the representation uses hidden states from the Transformer's last layer.Source: §2.2.1 · p. 4

2.2.2 GoT Encoder

  • The GoT encoder applies graph attention over neighboring nodes, concatenates multi-head outputs, and uses a feed-forward network to produce the graph embedding.Attention weights represent the importance of one node's features to another's and are computed only between neighboring nodes.Source: §2.2.2 · p. 5

2.3 Feature Fusion

  • Single-head attention aligns text with image and graph representations, then gated fusion combines those outputs with text features for the decoder.The decoder uses the fused features to predict rationales or the final answer; image features are optional.Source: §2.3 · pp. 5–6

3 Experiments

Experiments evaluate GoT on text-only AQUA-RAT and multimodal ScienceQA, using T5-base and T5-large models and a ViT-large vision encoder.

3 Experiments

  • GoT is evaluated on text-only AQUA-RAT and multimodal ScienceQA.The passage places dataset statistics in Appendix B.Source: §3 · p. 6
  • Experiments use T5-base and T5-large, initialized from the fine-tuned FLAN-Alpaca checkpoint, with ViT-large for vision.Models were fine-tuned for 100 epochs at a 5e-5 learning rate on four NVIDIA A800 80G GPUs.Source: §3 · p. 6
  • The model setup follows Zhang et al. (2023) for fair comparison and uses separate text and vision components.T5 was the base architecture; detailed training parameters are in Appendix C.Source: §3 · p. 6

4 Results and Discussion

GoT outperforms traditional CoT on AQUA-RAT and ScienceQA, while ablations and analyses examine thought-graph structure, performance patterns, and computational cost.

4.1 Main Results

  • 30.09%→32.09% in AQUA-RAT and 85.19%→87.59% in ScienceQA: GoT beats traditional CoT on both tasks.On ScienceQA, GoT also reaches 87.59% accuracy, 2.40% above Multimodal-CoTbase with a similar parameter count.Source: Table 1 · p. 6, Table 2 · p. 7, §4.1 · p. 7
  • On AQUA-RAT, GoTbase improves rationale-generation ROUGE-L by 0.78 and answer accuracy by 2.00%; GoTlarge improves accuracy by 0.75%.GoT-large uses a 738M-parameter backbone and comes close to Manual-CoT’s results.Source: §4.1 · p. 6
  • ScienceQA accuracy reaches 87.59%, 2.40% above Multimodal-CoTbase; GoT also achieves state-of-the-art results across subjects and grades.The rationale-generation stage records ROUGE-L 94.39, outperforming Multimodal-CoTbase by 1.15.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2.1 Ablation Study

  • Random thought graphs reduce AQUA-RAT accuracy by 1.78%, while triplet concatenation reaches 31.20%, a 0.89-point gap.Coreference injection reduces accuracy by 1.77%; the authors suggest replacement can lose coherence or introduce OpenIE noise.Source: §4.2.1 · pp. 7–8, Table 3 · p. 8
  • On ScienceQA, GoT beats a parameter-matched enlarged Multimodal-CoT by 2.04%; randomly constructed graphs perform below Multimodal-CoT.The authors report greater gains on smaller models and say weaker language and vision components increase reliance on GoT.Source: Table 4 · p. 8, §4.2.1 · p. 8

4.2.2 Analysis

  • Across ScienceQA subjects and question classes, GoT is more consistent and improves social-science accuracy by 5.25%.The largest class improvement is on questions involving images; the authors hypothesize that combining text, image, and graph features better aligns textual and visual information.Source: §4.2.2 · pp. 8–9, Figure 5 · p. 9
  • As grade level rises, Multimodal-CoT accuracy decreases by 1.78, compared with 0.56 for GoT; the gap widens on harder questions.The authors caution that grade 1 and grades 11–12 have at most 130 questions each, contributing to greater accuracy fluctuation.Source: Figure 6 · p. 9, §4.2.2 · p. 9
  • Compared with CoT, GoT may incur additional computational costs and slightly slower training.The paper points to Appendix G for detailed limitation analysis.Source: §4.2.2 · p. 9

5 Conclusion

GoT models non-sequential human thinking for language models and reports strong results on both text-only AQUA-RAT and multimodal ScienceQA.

5 Conclusion

  • GoT establishes a new state of the art on ScienceQA with fewer parameters, surpassing ChatGPT, GPT-4, and human performance.The conclusion reports a similar result to GPT-3 on AQUA-RAT while using significantly fewer parameters.Source: §5 · p. 9
  • GoT models the non-sequential nature of human thinking for language models and is presented as providing a more realistic representation of thought processes.The conclusion describes GoT as enhancing language models with deductive reasoning abilities.Source: §5 · p. 9
  • The authors report that case studies and ablation studies provide evidence of GoT's effectiveness in reasoning tasks.Source: §5 · p. 9

A Related Works

The related work contrasts sequential Chain-of-Thought methods with approaches that add demonstrations or visual inputs, motivating GoT's graph representation of reasoning.

A Related Works

  • Most Chain-of-Thought methods either prompt for reasoning or fine-tune models to generate it alongside final answers.Chain-of-Thought uses intermediate natural-language rationales, and prompting approaches include zero-shot and few-shot methods.Source: A Related Works · p. 13
  • Tree-of-Thought models nonlinear thought explicitly as a tree search, while GoT adds nonlinear structures to Chain-of-Thought to represent both linear and nonlinear reasoning.The passage describes GoT as leveraging nonlinear structures to assist Chain-of-Thought reasoning.Source: A Related Works · p. 13
  • Few-shot Chain-of-Thought uses multiple input-output demonstrations, while Auto-CoT generates demonstrations by clustering problems and sampling representatives.Auto-CoT uses a heuristic to generate a reasoning chain for each sampled representative problem.Source: A Related Works · p. 13
  • Prior fine-tuned Chain-of-Thought work incorporated visual inputs through gated fusion, while this work extends thought-chain modeling to thought graphs.The stated aim is to represent deduction reasoning more comprehensively for answer generation.Source: A Related Works · p. 13

B Dataset

The appendix describes AQUA-RAT as an algebraic reasoning dataset and ScienceQA as a multimodal science-question benchmark with three subject areas.

B Dataset

  • AQUA-RAT contains about 100,000 algebraic word problems with natural-language rationales.The model is trained to reason through steps and generate a final answer on AQUA-RAT.Source: B Dataset · p. 13
  • ScienceQA contains 21,000 questions across natural science, language science, and social science, with natural-language context and optional images.Source: B Dataset · p. 14
  • On ScienceQA, the model is trained to explain its reasoning in natural language while selecting an answer from a set of options.Source: B Dataset · p. 14

C Training Parameters

The appendix identifies Table 7 as presenting GoT's training parameters.

C Training Parameters

  • Table 7 is identified as presenting training parameters for GoT.Source: Table 7 · p. 14

D Rationale Generation Results

Rationale-generation gains are limited when graph construction uses only questions and choices. Case studies describe both more accurate answers and robustness to noisy rationales, while attention visualizations show which thought nodes receive greater or lower weight.

D Rationale Generation Results

  • Rationale-generation impact is limited because graphs built from questions and choices provide constrained information for deduction.Answer generation must relate rationales to questions and choices.Source: D Rationale Generation Results · p. 14

E Case Study

  • Figures 8 and 9 show GoT understanding rationales better and generating more accurate answers than CoT.Source: E Case Study · p. 14
  • In Figure 10, GoT answers correctly despite a wrong rationale, whereas Figure 11’s low-context common-sense question stumps both GoT and CoT.Source: E Case Study · p. 14

F Representation Visualization

  • In the Figure 10 visualization, “blue,” “color,” and “common” receive the greatest attention weights, while “a hard object” receives a lower weight.The passage links this lower weight to avoiding selection of an incorrect answer from a disruptive node.Source: F Representation Visualization · p. 14

G Limitation

Compared with Multimodal-CoT, GoT may increase computational costs and slightly slow training; the reported parameter increase is 0.2%.

G Limitation

  • Compared with Multimodal-CoT, incorporating GoT may add computational costs and slightly slow training.Source: G Limitation · p. 14
  • GoT requires 0.2% more parameters than Multimodal-CoT.Table 9 presents training parameters and inference times for the different models.Source: G Limitation · p. 15, Table 9 · p. 15

Dataset

The dataset visuals include examples from AQUA-RAT and ScienceQA, as well as a representation visualization.

Dataset

Related papers

Loading 2305.16582v2…