PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò, Fabrizio Silvestri

arXiv:2610.31572v1cs.LG

TL;DR

Graph soft prompts are intended to convey graph information, but G-Retriever also verbalizes the same graph, making graph-token use difficult to identify from accuracy alone. The paper audits token content and LLM use, finding near-identical tokens and proposing a two-stage procedure. The procedure yields non-collapsed tokens, while the paper shows that variation alone does not establish graph use.

  • Problem

    Because G-Retriever supplies graph information through both a graph token and verbalized text, answer accuracy alone cannot establish whether the LLM relies on the graph token.

    Source: §1 · pp. 1–2

  • Method

    The paper combines token geometry, graph-free controls, fixed-weight interventions, likelihood measures, and decoding probes, and proposes a two-stage pipeline that aligns and freezes the graph encoder before training separate graph-token and prompt-token pathways.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    G-Retriever tokens are nearly identical across inputs, and graph-free controls and token interventions match the original model’s performance; the two-stage pipeline mitigates collapse, but distinct tokens do not guarantee graph use or improved accuracy.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Evaluating token variation, graph content, and downstream use separately provides a practical way to test graph-token alignment rather than infer it from task accuracy.

    Source: §1 · pp. 2–3

    The detailed analysis covers G-Retriever on three QA benchmarks, while the cross-method audit is limited to WebQSP, runnable implementations, and controlled variants using G-Retriever’s retrieved subgraphs.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We also examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model uses graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not use. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://github.com/difra100/Token_Collapse_KGQA.

1 INTRODUCTION

The introduction questions whether G-Retriever’s graph token carries graph-specific information when the same graph is also verbalized. It reports token collapse and proposes a two-stage pipeline and evaluation protocol to distinguish token variation, graph content, and LLM use.

1 INTRODUCTION

  • Because G-Retriever supplies the graph both as a soft prompt and verbalized text, answer accuracy alone cannot identify which channel the model relies on.The answer objective does not explicitly require the graph-derived token to encode graph-specific information when text suffices.Source: §1 · pp. 1–2
  • The study tests token content and LLM use with token-space analyses, fixed-weight substitutions, gold-answer likelihood, decoding probes, and graph-free controls.Substitutions include the training-set mean, another example’s token, and tokens recomputed from corrupted or different graphs.Source: §1 · p. 2
  • Across tested encoders, benchmarks, LLM families, and tuning regimes, G-Retriever tokens are nearly input-invariant, and a learned constant token matches or surpasses the full model.The collapsed token functions primarily as a soft prompt; likelihood sensitivity alone does not establish graph-conditioned use.Source: §1 · p. 2
  • The authors associate collapse with projector variation suppression and an answer objective that need not preserve variation, then propose a two-stage pipeline that produces non-collapsed tokens.The pipeline first aligns and freezes the graph encoder, then trains separate graph-token and prompt-token pathways; the proposed factors are not isolated as sole causes.Source: §1 · p. 2
  • Non-collapsed tokens can carry question, answer-prior, or graph information, but graph information may be unused; the protocol tests alignment rather than inferring it from task accuracy.The token’s content and its downstream use are distinct questions.Source: §1 · pp. 2–3

2 RELATED WORK

Prior graph soft-prompting methods differ in encoders, token counts, retrieval, and LLM adaptation. This paper tests whether their projected tokens preserve and use graph-specific information, complementing work that questions graph–language integration objectives and benchmarks.

2 RELATED WORK

  • Graph soft prompting projects graph-encoder outputs into continuous LLM prompt tokens; surveyed methods vary in encoders, token counts, retrieval, and LLM adaptation.The paper tests whether G-Retriever and subsequent methods preserve and use graph-specific information in those tokens.Source: §2 · p. 3
  • Related work questions whether existing objectives and benchmarks require graph–language integration, while this study audits the graph-token–LLM interface.The cited studies report text-dominant autoregressive training, competitive single-modality and raw soft-prompt baselines, and CLEGR for compositional graph–text reasoning.Source: §2 · p. 3

3 METHOD

The method describes G-Retriever’s graph-token pipeline and separates token variation, graph-specific content, and downstream use in its evaluation. It then uses a two-stage training procedure designed to preserve input-specific variation while providing a separate prompting pathway.

3.1 PRELIMINARIES: GRAPH SOFT PROMPTING IN G-RETRIEVER

  • When a graph exceeds the context or token budget, question-to-node and edge similarities guide retrieval of a compact connected subgraph; otherwise the full graph is used.The retrieved subgraph is denoted G_i.Source: §3.1 · p. 3
  • G-Retriever encodes retrieved subgraph features, mean-pools node representations, and projects the pooled representation into the LLM embedding space as a graph token.The graph encoder receives the question only through retrieval; the subgraph is also verbalized and supplied with the question.Source: §3.1 · p. 3

3.2 THE GRAPH TOKEN COLLAPSE PHENOMENON

  • The evaluation separates token variation, LLM use, and token content through geometry measures, fixed-weight interventions, answer likelihood, and decodability probes.MEAN and SWAP replace the token, GSWAP recomputes it from another graph while preserving question conditioning, and NOTXT removes the verbalized graph.Source: §3.2 · p. 4
  • Graph-free CONST, MLPQ, and raw prompt-tuning controls distinguish graph-input contributions from projector and token-parameterization effects.CONST uses a learned graph-independent vector through the projector; MLPQ replaces the graph encoder with a question-input MLP, while PT does not use the projector.Source: §3.2 · p. 4

3.3 PREVENTING COLLAPSE: A TWO-STAGE PIPELINE

  • The authors associate collapse with a sigmoid projector that suppresses encoder variation and end-to-end answer training that does not require preserving it.The ablations implicate both factors without isolating either as the sole cause.Source: §3.3 · p. 5
  • Stage 1 trains the graph encoder without the LLM using InfoNCE to align pooled representations with sample-specific targets, then freezes the encoder.Targets are answers, or verbalized graphs when the answer space is too small to distinguish samples; the question conditions each node representation.Source: §3.3 · p. 5
  • Stage 2 jointly trains a GELU projector over the frozen encoder and a separate graph-independent prompt token using the original answer loss.Freezing protects Stage 1 variation, while the separate pathway gives the shared prompting function its own token.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments find graph-token collapse across tested configurations, then show that token geometry, graph-specific use, and task performance are distinct. A two-stage pipeline mitigates collapse, but what its tokens encode and whether the LLM uses them varies by benchmark and training configuration.

4.1 SETUP

  • The setup tests four questions using three GraphQA benchmarks, multiple encoders and LLM configurations, and accuracy or Hit@1 with stability-aware thresholds.Answer-level effects require more than two points and p < .05; gold-answer likelihood requires |∆NLL| > 0.03 and a bootstrap interval beyond the replicate floor.Source: §4 · p. 5, §4.1 · p. 5

4.2 RQ1: DO GRAPH TOKENS COLLAPSE, AND HOW GENERAL IS COLLAPSE?

  • Token cosine is 0.985–0.998 across tested configurations, and the same collapse appears across encoders, LLM families, tuning regimes, and later single- and multi-token methods.Encoder representations are more variable, with cosine values of 0.423–0.693; near-identity appears after projection.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 RQ2: WHAT ROLE DOES THE COLLAPSED TOKEN PLAY?

  • Mean, swapped, and corrupted-graph token substitutions have no resolved answer-level effect, while removing the verbalized graph costs 19–51 points and CONST matches or exceeds G-Retriever.Replacing the token with zero costs 59.64–81.95 points; small WebQSP likelihood shifts remain below the 0.03 criterion.Source: Table 2 · p. 6, §4.3 · p. 6

4.4 RQ3: WHAT IS COLLAPSE ASSOCIATED WITH, AND CAN IT BE PREVENTED?

  • A Sigmoid projector suppresses encoder variation, while GELU alone does not prevent collapse under end-to-end answer training; the ablations associate both factors without isolating either as the sole cause.With the aligned frozen encoder, GELU prevents collapse, but the single-token model scores 57.76 versus 58.8 for the comparison reported.Source: §4.4 · p. 7
  • Token cosine falls from above 0.99 to 0.099–0.142 with Llama-2 and 0.103–0.200 with Qwen2.5-3B, while performance is retained or improved in most tested settings.On SceneGraphs, however, the Llama-2 score falls from 81.35 to 72.50; on Qwen2.5-3B, the pipeline scores 3.40 points below G-Retriever there.Source: §4.4 · p. 7, Table 4 · p. 8

4.5 RQ4: WHAT DOES A NON-COLLAPSED TOKEN CARRY?

  • On WebQSP, question and answer are strongly recoverable from the standard token, while graph-specific use resolves only after removing the verbalized text; training inputs can shift the token toward priors or graph information.The standard token retrieves the question and answer at ranks 65 and 60 against a control of 407; removing question conditioning produces resolved graph use.Source: §4.5 · p. 8, Table 5 · p. 9
  • On SceneGraphs, the pipeline’s graph-specific token use lowers accuracy by 5.30 points, yet remains less accurate than G-Retriever; G-Retriever’s collapsed token has recoverable graph information but no resolved graph use.The no-entity model nearly recovers its accuracy while eliminating graph use, further separating performance from token use.Source: §4.5 · p. 9, Table 5 · p. 9
  • On ExplaGraphs, graph information is more recoverable from the pipeline token (rank 41 against control 139), but GSWAP has no resolved effect with or without the verbalized graph.Across benchmarks and training inputs, non-collapsed tokens may encode question or answer priors, unused graph information, or graph information that changes the answer.Source: §4.5 · p. 9, Table 5 · p. 9

5 DISCUSSION

The discussion argues that accuracy alone cannot establish graph-token use and recommends evaluating token geometry, graph-free controls, and fixed-weight interventions together. It distinguishes graph use from task performance and states boundaries on the evidence and recipe.

5 DISCUSSION

  • A minimal audit combines token geometry, a graph-free control using the same projector, and fixed-weight MEAN, SWAP, and GSWAP interventions.Raw prompt tuning changes parameterization, while no-text retraining changes the learning problem; the controlled WebQSP audit also finds collapse in query-conditioned and multi-token approaches.Source: §5 · p. 9
  • Graph-specific token variation can affect answers without improving accuracy, so alignment claims should report graph use and performance separately.The discussion notes that textual inputs may be easier to exploit and continuous graph tokens may be lossy or difficult for the LLM to interpret; Qwen2.5-3B results show the trade-off is not inevitable.Source: §5 · p. 9
  • The detailed analysis covers G-Retriever on three QA benchmarks, while the cross-method audit is limited to WebQSP, runnable implementations, and controlled variants using retrieved subgraphs.Mechanism evidence is associative and does not establish either the projector or end-to-end training as a sole cause.Source: §5 · p. 9
  • The audited recipe trains from scratch with answer cross-entropy, and collapse may differ with pretrained encoders or explicit contrastive, reconstructive, or counterfactual objectives.GSWAP tests graph–question pairing rather than fact-level reasoning, and decodability measures recoverability rather than use; the pipeline is described as a diagnostic proof of concept.Source: §5 · pp. 9–10

6 CONCLUSION

The conclusion presents token collapse as a recurring graph-prompting failure and argues that token variation, recoverable graph information, downstream use, and task performance need separate evaluation. The appendices provide further experimental, protocol, and diagnostic details.

6 CONCLUSION

  • Graph-free controls and fixed-weight interventions show that the collapsed token can function primarily as a graph-independent soft prompt while the verbalized graph supplies the dominant graph-information channel.The two-stage pipeline mitigates collapse, but distinct tokens do not guarantee graph use or improved accuracy.Source: §6 · p. 10

APPENDIX OVERVIEW

  • The appendices document implementation and dataset settings, evaluation and statistical protocols, collapse measurements, causal interventions, content probes, and pipeline diagnostics.Appendix A covers implementation and hyperparameters, B covers datasets and retrieval, and later appendices organize evidence by diagnostic topic.Source: REPRODUCIBILITY STATEMENT · p. 10, APPENDIX OVERVIEW · p. 12

A IMPLEMENTATION DETAILS

The implementation section describes a pinned G-Retriever reproduction, its default model configuration, and the sequential two-stage pipeline. It also cautions that performance differences between some controls cannot be attributed to projector parameterization alone.

A IMPLEMENTATION DETAILS

  • The baseline uses a four-layer, four-head graph transformer, a two-layer projector, frozen Llama-2-7B, and a verbalized graph truncated to 512 tokens.The implementation uses the authors’ public code at a pinned commit; generation is limited to 32 tokens.Source: A IMPLEMENTATION DETAILS · p. 13
  • Stage 1 trains the encoder and question-conditioning layer with InfoNCE; Stage 2 freezes them, replaces the graph projector’s Sigmoid with GELU, and jointly trains separate graph-token and prompt-token pathways.Interventions modify only the graph token, not the prompt token.Source: A IMPLEMENTATION DETAILS · p. 13
  • CONST improves over raw prompt tuning by 19 points, but the comparison cannot isolate projector parameterization from capacity; CONST and G-Retriever share that parameterization for the controlled graph-input comparison.The larger Stage-2 model scores 1.35 points below CONST, illustrating that accuracy does not track capacity in these comparisons.Source: A IMPLEMENTATION DETAILS · p. 13

B DATASETS

The paper describes GraphQA’s textual-graph datasets, graph encoders, and protocols for measuring token geometry, interventions, and decodability. It also bounds what the decodability measure can establish.

B DATASETS

  • Retrieval reduces average verbalisation from 100,627 to 610 tokens on WebQSP and from 1,396 to 235 on SceneGraphs; ExplaGraphs graphs fit in the prompt and are used in full.SceneGraphs retrieval keeps the top 3 nodes and edges; WebQSP keeps the top 3 nodes and 5 edges.Source: B DATASETS · p. 13

C GRAPH ENCODERS

  • Replacing the graph transformer with GAT or GCN changes ExplaGraphs accuracy, but not the encoders’ token behaviour.GCN loses 6.7 points to the graph transformer on ExplaGraphs.Source: C GRAPH ENCODERS · p. 14

D PROTOCOL DETAILS

  • Fixed-weight interventions compare original tokens with mean, swapped, or graph-corrupted alternatives, while additional conditions alter token geometry or remove verbalised text.The protocol also uses gold-answer likelihood and decodability probes alongside accuracy.Source: D PROTOCOL DETAILS · p. 14
  • The graph-column decodability test is lossy and bounded: successful decoding supports recoverability, but failure does not prove information is absent.The sentence encoder truncates some WebQSP and SceneGraphs verbalised subgraphs; a single 1024-dimensional vector also compresses each subgraph.Source: D PROTOCOL DETAILS · p. 15

E COMPLETE COLLAPSE STATISTICS AND PER-ENCODER INTERVENTIONS

  • The appendix provides the remaining token statistics and per-encoder interventions for the GAT and GCN variants.It directs readers to the appendix for the other measured conditions as well.Source: E COMPLETE COLLAPSE STATISTICS AND… · p. 15

F BEYOND G-RETRIEVER: TOKEN COLLAPSE IN SUBSEQUENT GRAPH-PROMPTING METHODS

The audit finds token collapse across the tested subsequent graph-prompting configurations, including multi-token methods, while interventions provide little evidence of token-specific effects. Its scope is limited to a single-seed audit without likelihood measurements.

F BEYOND G-RETRIEVER: TOKEN COLLAPSE IN SUBSEQUENT GRAPH-PROMPTING METHODS

  • The audit covers WebQSP methods with runnable public implementations that construct continuous graph prompts, excluding methods outside those criteria or unavailable for reproduction.The tested methods are GRAG, GRETPool, and GraSP.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 16
  • The audit holds retrieval fixed across methods, making its adapted GRAG-PCST and GraSP-PCST configurations controlled comparisons rather than complete reproductions.All methods use the same WebQSP test split and G-Retriever PCST subgraphs.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 16
  • Multi-token diagnostics mean-pool tokens only for geometry measurement, while interventions preserve each method’s token cardinality.For methods with multiple tokens, the audit also measures within-graph pairwise cosine.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 16
  • 0.998–1.000 inter-sample cosine across all six configurations and tested cardinalities shows collapse; multi-token methods are also internally almost collinear.GraSP reaches inter-sample cosine 0.9978, within-example cosine 0.9995, and residual ratio 0.039 despite using a Tanh projector and one token per retained node.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 16
  • With verbalised text present, token swaps, graph swaps, mean-token replacements, and node randomisation produce no resolved effects under the study’s joint threshold.The reported marginal edge-rewiring effects do not meet both the effect-size and p-value criteria.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 16
  • Without verbalised text, the reported effects remain limited and inconsistent, and none meets both the effect-size and p-value criteria.Removing the verbalised graph lowers performance by 25.6–43.2 points, but that comparison does not test token specificity.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · pp. 16, 18
  • The recurrence finding is limited to the tested configurations: the audit is single-seed and lacks likelihood measurements, so it does not establish universal collapse or that graph information is unnecessary.Its evidence supports recurrence rather than the stronger conclusions drawn for G-Retriever in the main experiments.Source: F BEYOND G-RETRIEVER: TOKEN COLLAPSE… · p. 18

G THE TOKEN SPACE OF EVERY MODEL

  • Figures 3–5 show near-collapsed G-Retriever tokens across benchmarks and encoders, while the two-stage and projector-only variants occupy different token-space regimes.Figure 6 contrasts a constant token, a question-derived token, GELU, and Stage 1 with a Sigmoid projector as diagnostics of collapse.Source: Figure 3 · p. 19, Figure 4 · p. 19, Figure 5 · p. 20

H MECHANISM DIAGNOSTICS

The mechanism diagnostics associate collapse with projector behavior and answer training, while the intervention and control results qualify what token variation and performance can establish. The paper also examines token magnitude and gold-answer likelihood.

H MECHANISM DIAGNOSTICS

  • An untrained projector produces nearly identical tokens for fixed-norm random inputs unless their norm is two orders of magnitude above trained encoder outputs.The authors present a local Sigmoid approximation as one possible explanation, not a complete account; GraSP remains collapsed with a zero-centred Tanh projector.Source: H MECHANISM DIAGNOSTICS · p. 21

I FULL FIXED-WEIGHT INTERVENTIONS

  • The intervention tables use REAL as the reference, or NOTXT-REAL when verbalised text is removed, and require an effect above 2.0 accuracy points regardless of p-value.The tables report absolute accuracy, differences from reference, and exact McNemar p-values.Source: I FULL FIXED-WEIGHT INTERVENTIONS · p. 21

I.1 EXPLAGRAPHS

  • ExplaGraphs intervention tables cover G-Retriever, encoder variants, graph-free controls, and the paper’s proposed configurations on 554 test questions.The supplied table captions identify the configurations and test-set size, not their outcomes.Source: Table 14 · p. 22, Table 18 · p. 24, Table 19 · p. 24

I.2 SCENEGRAPHS

  • −4.65 points in each of two further SceneGraphs derangements confirms the SWAP effect is not confined to one pairing.Both results use the same 2,000 questions; the −3.40-point effect in Table 4 is the weakest of the three.Source: §I.2 · p. 32

I.3 WEBQSP

J THE TOKEN’S FORM: MAGNITUDE AND RESIDUAL CONDITIONS

K COMPLETE GRAPH-FREE CONTROLS

  • Accuracy does not track parameter capacity: PT to CONST multiplies trainable parameters by 256 and adds 19 WebQSP Hit@1, while CONST to Stage 2 doubles parameters and removes 1.35.The CONST–PT comparison remains confounded by differences in parameterisation and capacity.Source: K COMPLETE GRAPH-FREE CONTROLS · p. 45

L GOLD-ANSWER LIKELIHOOD

  • On SceneGraphs, the graph-swap likelihood effect is seven times the token-swap effect; on ExplaGraphs, neither model’s likelihood moves under the reported conditions.On SceneGraphs, the mean token lowers likelihood, making an empty token better than a wrong graph.Source: L GOLD-ANSWER LIKELIHOOD · pp. 45–46

M COMPLETE TOKEN-CONTENT AND USE RESULTS

The Qwen replication distinguishes whether information can be recovered from a token from whether the LLM uses it. It also shows that geometric collapse does not necessarily imply complete insensitivity to token changes.

M COMPLETE TOKEN-CONTENT AND USE RESULTS

  • On SceneGraphs, the Qwen model has the study’s largest graph-replacement effect (−6.50) despite a graph rank close to the constant control (450 versus 500); on ExplaGraphs, graph rank is 62 versus 139 but replacement has no resolved effect.The result illustrates that a probe can miss information affecting the LLM or recover information the LLM ignores.Source: M COMPLETE TOKEN-CONTENT AND USE RESULTS · p. 46
  • Qwen G-Retriever on ExplaGraphs remains geometrically collapsed yet shows a resolved −4.51-point GSWAP effect, so collapse does not establish complete insensitivity to residual variation.For collapsed baselines, nearly identical SWAP and GSWAP likelihood effects indicate generic sensitivity rather than identified graph-specific use.Source: M COMPLETE TOKEN-CONTENT AND USE RESULTS · p. 46
  • The decodability results are bounded by truncation, lossy graph-vector compression, and a linear fitted map; successful decoding supports recoverability, but failed decoding does not prove absence.The constant-token control is at the median rank by construction.Source: M COMPLETE TOKEN-CONTENT AND USE RESULTS · p. 46

N VALIDATION OF THE DESIGN CHOICES

The design studies test how question conditioning and separate graph and prompt pathways affect token information and use. Its ablations show that the pathways must be trained together, while cheaper alternatives can leave tokens closer to collapse.

Question conditioning in Stage 1

  • Removing question conditioning drops Stage 1 top-1 from 0.433 to 0.084, while the graph gap indicates its learned representations are not merely question copies.The graph gap is defined only for SceneGraphs, which asks multiple questions about some graphs.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Stage-1 targets

  • The Stage 1 target is the gold answer on WebQSP, the full answer sentence on SceneGraphs, and the verbalised graph on ExplaGraphs.ExplaGraphs uses the graph because its two answer labels cannot identify individual samples.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Freezing the encoder in Stage 2

  • A trainable encoder re-collapses during Stage 2 to cosine 0.9139, whereas the frozen encoder reproduces the intended graph token within 1.2 × 10^-7.This recollapse occurs even with a GELU projector.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Parameterisation of the prompt token

  • A learned constant prompt passed through a Sigmoid projector matches G-Retriever on all three datasets, while raw prompt tuning scores 11–25 points lower.The raw-prompt comparison does not isolate parameterisation from capacity.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Must the two tokens be trained together?

  • Adding a prompt token only after the projector converges leaves accuracy at 63.64, with the best epoch already first.By then the model has settled into using the graph token for both roles, leaving the new token nothing to take over.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Is a second token needed at all?

  • Projector dropout reaches 64.56 accuracy but leaves one token doing both jobs and closer to collapse: cosine 0.7563 versus 0.1778 for the joint model.The comparison is between the dropout alternative and the joint model.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

Could the redundancy be removed from the data instead of the architecture?

  • Hiding 40% of verbalised edges during training does not force graph use; with complete test text, substitute tokens match or beat the real token, and other graphs’ node features add 2.03 points.The +2.03-point effect has p = 0.03.Source: N VALIDATION OF THE DESIGN CHOICES · p. 48

O PIPELINE AND ROBUSTNESS ABLATIONS

The ablations show that token dependence and resolved graph-specific use can diverge, including when the verbalised graph is removed. Cross-fitting, initialization checks, and the verbalised-graph test further qualify what performance and interventions establish.

O PIPELINE AND ROBUSTNESS ABLATIONS

  • Across both LLMs and all three benchmarks, removing entity features preserves MEAN/SWAP sensitivity but leaves no resolved GSWAP effect; removing question conditioning yields resolved graph use on WebQSP, but not ExplaGraphs.On Qwen SceneGraphs, GSWAP is significant but does not clear the two-point criterion; removing entity features also changes task performance differently by benchmark.Source: O PIPELINE AND ROBUSTNESS ABLATIONS · p. 49

O.1 EVALUATION WITHOUT THE VERBALISED GRAPH

  • With the verbalised graph removed, Llama-2 shows resolved graph-specific effects for the standard pipeline on WebQSP and SceneGraphs, but none on ExplaGraphs.The question-free SceneGraphs variant also shows a resolved effect.Source: §O.1 · p. 49
  • With Qwen2.5-3B and text removed, many MEAN-sensitive rows establish token dependence but not graph use; standard ExplaGraphs shows no resolved token effect despite improved score and decodable graph information.MEAN alone does not identify the token’s information as graph-specific; GSWAP changes the graph while preserving other inputs.Source: §O.1 · p. 49

P CROSS-FITTING: IS THE TOKEN’S USEFULNESS STAGE-1 MEMORISATION?

  • Cross-fitting shows token reliance is partly memorisation: disjoint Stage 1 and Stage 2 training halves reduce intervention effects while accuracy rises toward the same-half baseline.Rewiring edges costs the cross-fitted joint model 1.95 points in one SceneGraphs run and nothing in the other.Source: P CROSS-FITTING: IS THE TOKEN’S… · p. 51

Q INITIALISATIONS AND EVALUATION NOISE

  • Across three WebQSP initialisations, model-score spread exceeds the difference between the joint model and G-Retriever, yet pooled discordant questions show only the joint model’s token is used.The pooled evaluation covers n = 4,884 discordant questions.Source: Q INITIALISATIONS AND EVALUATION NOISE · p. 53

R THE VERBALISED GRAPH IS NOT ENOUGH

  • On ExplaGraphs, arguments alone score 61.0% and the complete explanation graph raises this to 64.3%, but the graph fixes 68 questions and breaks 50.A support bias accounts for 451 of 554 answers and 161 of the 198 errors; Figure 7 shows a graph that explicitly states the opposition can still mislead the model.Source: R THE VERBALISED GRAPH IS NOT ENOUGH · p. 53, Figure 7 · p. 53

S EXTENDED RELATED WORK

  • Prior work identifies implicit graph–text alignment and benchmark baselines as reasons graph use is not guaranteed; this paper instead tests token variation and use directly.The authors distinguish representation alignment from evidence that projected tokens vary across graphs or that the LLM uses graph-specific content.Source: S EXTENDED RELATED WORK · p. 54
  • Graph soft prompting is intended to preserve input-specific graph information, unlike shared prompt-tuning vectors, but the paper finds it can collapse into a nearly input-independent prompt.The related-work discussion notes that LoRA is used only as a robustness check.Source: S EXTENDED RELATED WORK · p. 55

Related papers

Loading 2610.31572v1…