PaperlayerView exact revision on arXiv

Source-linked AI summary

CHORD: Cross-Model Hallucination Detection via Relational Graph Discrimination

Yongxin Deng, Zhen Fang, Guansong Pang, Sharon Li, Ling Chen

arXiv:2610.38314v1cs.AI

TL;DR

Cross-model hallucination detection addresses the difficulty of transferring hallucination detectors to unseen LLMs when hidden representations differ semantically across models. CHORD builds joint relational graphs over layers, tokens, and features and trains structure-aware features with relation-aware attention and meta-learning. Experiments report improved cross-model generalization over representative baselines across models, scales, and benchmarks.

  • Problem

    Representation-based hallucination detectors are often model-specific, while cross-model semantic inconsistency limits their transfer to unseen LLMs.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    CHORD represents hidden states as joint relational graphs and uses relation-aware attention and meta-learning to learn structure-aware transferable features.

    Source: §1 · pp. 2–3

  • Results

    Across eight LLMs from three model families and seven benchmarks, CHORD consistently improves cross-model generalization and outperforms representative baselines.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Layer-token-feature relational structures provide transferable hallucination signals across the evaluated models, scales, and task types.

    Source: §1 · pp. 2–3

    CHORD requires access to hidden states and cannot be directly applied to purely black-box APIs that expose only generated text or token probabilities.

    Source: C Limitations · p. 37

Abstract

from arXiv · show

Hallucination detection is critical for deploying large language models (LLMs) in real-world applications. Due to strong empirical performance, internal representation-based methods have emerged as the prevailing direction for detecting hallucinations, yet they remain largely model-specific and often fail to generalize to unseen LLMs. In this paper, we study an important yet underexplored problem, termed cross-model hallucination detection (CMHD), which aims to train hallucination detectors on source LLMs while ensuring performance on unseen target LLMs. The core challenge of CMHD lies in cross-model representation heterogeneity: hidden states from different LLMs exhibit severe semantic inconsistency, which limits the transferability of classical representation-based detectors. Through analysis and empirical validation, we show that relational structures over layers, tokens, and features capture transferable detection signals despite semantic inconsistency. Based on this observation, we propose cross-model hallucination detection via heterogeneity-oriented relational discrimination (CHORD), a relational graph discrimination framework that represents hidden states as joint relational graphs over layers, tokens, and features. It feeds these graphs into relation-aware attention to obtain structure-aware features, and uses meta-learning to shape these features toward transferable detection signals. Extensive experiments show that CHORD outperforms representative detection methods in cross-model generalization. Code is available at https://github.com/MiracleworkerD/CHORD.

1 Introduction

The introduction frames cross-model hallucination detection as a response to representation-based detectors that often fail on unseen LLMs. It motivates CHORD by arguing that relational structures can transfer across heterogeneous models, then introduces a graph-discrimination framework and reports broad cross-model evaluation.

1 Introduction

  • Training-based detectors often fail on unseen LLMs, limiting their practical utility and potentially requiring retraining or redesign for each new model.The challenge is severe semantic inconsistency across heterogeneous model representations.Source: §1 · p. 2
  • Relational structures show cross-model alignment and detection transfer despite semantic inconsistency, with joint structures attaining the highest cross-model AUROC among the relational features studied.The joint structures combine cross-layer token, within-token feature, and layer-token-feature relations.Source: §1 · p. 2
  • CHORD represents hidden states as joint layer–token–feature relational graphs and uses relation-aware attention with meta-learning to learn transferable detection signals.The method uses graph pooling to obtain structure-aware features.Source: Figure 1 · p. 2, §1 · p. 3
  • Experiments span eight LLMs from three model families and seven benchmarks across factual and conversational QA, reasoning, coding, and agent tasks; CHORD consistently improves generalization over representative baselines.The paper studies detectors trained on source LLMs and evaluated on unseen target LLMs.Source: §1 · p. 3

2 Learning Setups

The learning setup defines how answer labels are assigned and formalizes CMHD as training on multiple source models and datasets before testing on an unseen target model. The available source models may differ in architecture, scale, and training procedure.

2 Learning Setups

  • The setup considers multiple training LLMs that may vary in architecture family, parameter scale, and training procedure.These variations may induce systematically different generation behaviors.Source: §2 · p. 3
  • An answer is labeled truthful when it aligns semantically with a reference answer or explicitly abstains, and hallucinated when it is not aligned.The labels are assigned to generated answers for questions sampled from the truthful-answer domain.Source: §2 · p. 4
  • CMHD trains a detector using datasets from multiple source LLMs and evaluates whether it generalizes to an unseen target model.At test time, the target model is not among the source models used to learn the detector.Source: §2 · p. 4

3 Analysis of Hallucination Signals: What Transfers Across Models?

The analysis asks which hidden-state relations transfer across models whose raw representations have inconsistent semantics. It evaluates token-level relations and broader layer, feature, and joint structures for cross-model stability and hallucination-detection transfer.

3 Analysis of Hallucination Signals: What Transfers Across Models?

  • Raw representation-based detectors often generalize poorly across models, a pattern attributed to cross-model semantic inconsistency.The analysis therefore seeks signals that remain stable across heterogeneous representations.Source: §3 · p. 4

3.1 Stable Cross-Model Signals from Token-Level Relations

  • Token-level relations weight token-pair relations by representation similarity and positional distance, then aggregate selected tokens into a graph-level feature.The token-pair relation uses a position-based decay factor controlled by τtok; the graph is fully connected without self-loops and uses relation-weighted aggregation and mean readout.Source: §3.1 · p. 4
  • Token-level relational features raise mutual kNN alignment from 0.03–0.11 for raw representations to 0.17–0.30 across five transfer pairs.Their cross-model AUROC reaches 0.62–0.67, versus approximately 0.47–0.52 for raw representations; the token-level result matches PPL at about 0.63 but remains below Semantic Entropy at about 0.75.Source: §3.1 · pp. 5–6, Figure 3 · p. 5

3.2 From Token-Level to Joint Relational Structures

  • The broader analysis constructs cross-layer token, within-token feature, and joint layer-token-feature relations, using selected layers and feature blocks as graph nodes.Joint relational structures have near-highest cross-model stability and the highest AUROC among the four relational features, even with fixed handcrafted edge weights.Source: §3.2 · pp. 6–8
  • Joint relational structures achieve the highest cross-model AUROC at around 0.69–0.73; three of the four relational features match or exceed PPL (0.63), and joint structures further close the gap to Semantic Entropy (0.75).Within-token feature relations reach 0.64–0.67, token-level relations 0.62–0.67, and cross-layer token relations 0.56–0.60.Source: §3.2 · p. 7

4 CHORD: Relational Attention for Cross-Model Hallucination Detection

CHORD builds a unified graph over hidden-state layers, tokens, and feature blocks, then uses relational attention and meta-learning to learn detection signals transferable to unseen LLMs.

4 CHORD: Relational Attention for Cross-Model Hallucination Detection

  • The framework combines a unified layer-token-feature graph with position- and content-aware edges, relation-aware attention, and meta-learning to learn transferable detection signals.The ablation study compares CHORD with variants that remove relation-aware attention or replace meta-learning with standard training.Source: §4 · p. 8, Table 2 · p. 9

Hidden Representation Restructuring

  • CHORD maps hidden representations with different dimensions into a unified space using scattered padding, then divides the padded dimension into equal-size blocks.The unified dimension is at least the largest source dimension.Source: §4 · p. 8

Graph Construction and Relation-Aware Encoding

  • Each graph node represents a layer-token-block triple, and the fully connected graph carries learned positional and content relations on its edges.Node features combine the corresponding block feature with learned layer, token, and block embeddings; edge features combine index differences with content similarity signals.Source: §4 · p. 8
  • Relation-aware attention propagates edge information into updated node representations, which attention pooling then collapses into a graph-level vector for detection.The propagation incorporates positional and content relations into node representations, so subsequent stages need not access edges directly.Source: §4 · p. 8

Meta-Learning Training

  • CHORD uses meta-learning rather than ordinary empirical risk minimization to reduce reliance on shortcuts specific to training LLMs.Feature-Critic holds out one training LLM as a pseudo-unseen model and favors updates that transfer to it; the resulting detector is used on unseen LLMs.Source: §4 · p. 9

5 Experiments

Across held-out model families, CHORD leads the evaluated methods overall and in every model–benchmark evaluation. Results vary by task and target, and ablations and shift comparisons characterize where its advantage holds.

Overall Cross-model Hallucination Detection Performance

  • 0.8042 AUROC overall for CHORD versus 0.7152 for InterrogateLLM, the strongest baseline by average; CHORD ranks first in all 56 model–benchmark evaluations.MM, SEP, SAPLMA, and LapEigvals average 0.4976–0.5177; matching feature dimensions alone does not resolve their weak transfer.Source: §5 · p. 9

Dataset-level Differences

  • 0.8564 target-averaged AUROC on xLAM and 0.8298 on GSM8K for CHORD, versus 0.7421 and 0.7297 for InterrogateLLM, respectively.Across TruthfulQA, HaluEval, TriviaQA, and CoQA, CHORD averages 0.8023 versus 0.7102 for InterrogateLLM; on MBPP, the margin is smaller, 0.7337 versus 0.6940.Source: §5 · pp. 9–10

Variation across Target Models

  • 0.0793–0.0942 is CHORD’s benchmark-averaged margin over InterrogateLLM across eight targets; results describe the evaluated models, not a general scaling law.Within Llama, CHORD improves from 0.7941 on 3B to 0.8299 on 70B, while Qwen targets range from 0.8034 to 0.8088.Source: §5 · p. 10

Ablation Study

  • Removing relation-aware attention lowers overall AUROC from 0.8042 to 0.7078, while replacing meta-learning with ERM lowers it to 0.7268.These are drops of 9.64 and 7.74 AUROC percentage points, respectively; on MBPP, both variants fall below InterrogateLLM’s 0.6940, while full CHORD reaches 0.7337.Source: §5 · p. 10

Contributions of Individual Relation Channels

  • 0.7592, 0.7629, and 0.7607 are the overall AUROCs after removing token, layer, and intra-block feature edges, respectively.The channel whose removal produces the largest task-specific loss varies: token edges on xLAM, layer edges on GSM8K, and intra-block feature edges on MBPP.Source: §5 · p. 10

Transfer across Model Scales

  • 0.7664–0.7854 is CHORD’s average AUROC across six directed transfers between evaluated scale groups; large-to-small transfer exceeds the reverse direction, 0.7854 versus 0.7664.The groups differ in model-family composition and source-model count, so the result does not establish that larger source models always provide better supervision.Source: §5 · p. 10

Distinguishing Model Shift from Benchmark Shift

  • 0.7800 overall AUROC for CHORD versus 0.7152 for InterrogateLLM when the LLM is fixed and one benchmark is held out.Training-based baselines recover to 0.6366–0.6797 rather than near chance; differing training pools mean this comparison supports, but does not prove, the representation-mismatch explanation.Source: §5 · p. 10

6 Conclusion

The paper studies detection transfer from source LLMs to unseen targets and identifies layer-token-feature relations as transferable signals. CHORD combines relational graphs, relation-aware attention, and meta-learning, and experiments show it outperforms representative baselines.

6 Conclusion

  • CHORD represents hidden states with layer-token-feature relational structures and learns structure-aware features through relation-aware attention and meta-learning.The paper reports experiments across model families, scales, and benchmarks, with ablations of core modules.Source: §6 · p. 10

A.1 Pilot Transfer-failure Analysis

A pilot study tests whether standard training-based hallucination detectors transfer across three 8B-scale LLMs. Their strong in-model AUROC falls close to random guessing on cross-model evaluations.

A.1 Pilot Transfer-failure Analysis

  • All three training-based methods achieve strong in-model AUROC but drop close to random guessing when transferred across models.The pilot evaluates MM, SEP, and SAPLMA on Llama3.1-8B, Qwen3-8B, and Ministral3-8B using diagonal entries for in-model performance and off-diagonal entries for transfer.Source: §A.1 · p. 16, Figure 4 · p. 16

A.2 Token–Layer Window Size and Position

The window sweeps compare fixed relational diagnostics with the full CHORD detector, finding that CHORD reaches a stable region with a smaller token–layer window. Late generated tokens and middle layers are the preferred positions for both settings.

Evaluation settings

  • The fixed diagnostic uses hand-designed edges and a source-trained MLP, whereas the method setting evaluates full CHORD under leave-one-family-out transfer.The diagnostic probes transferable relational structure; the method setting measures end-to-end detector performance.Source: §A.2 · p. 16

Count sweeps

  • CHORD reaches a stable region around T = 3, L = 3, while the observational diagnostic is strongest around T = 7, L = 5.The diagnostic curve largely saturates beyond its stronger region; CHORD gains little from larger windows.Source: Figure 5 · p. 17, §A.2 · p. 17

Position sweeps

  • Late generated tokens outperform earlier or middle tokens, and middle-layer windows perform best or near-best across the position sweeps.The authors relate late-token usefulness to hallucination evidence becoming clearer as more of an answer is generated.Source: §A.2 · p. 17

Resulting choices

  • The selected windows are T = 7 late tokens and L = 5 middle layers for diagnostics, versus T = 3 and L = 3 for CHORD.The compact CHORD window avoids unnecessary graph expansion while retaining most performance.Source: §A.2 · p. 17

A.3 Feature-Block Granularity

This section examines feature-block granularity, local-neighborhood alignment, graph construction, and relation-aware attention. It supports a moderate block count and describes a shared layer–token–feature graph whose edges encode both position and content.

A.3 Feature-Block Granularity

  • B = 64 lies near the strongest performance region and balances localized feature structure against graph size; very large B can create low-information nodes and reduce transfer AUROC.At B = 1024, each block has eight dimensions and the graph is an order of magnitude larger than at the default.Source: Figure 7 · p. 19, §A.3 · p. 19

A.4 Why We Use Mutual k-Nearest-Neighbor Alignment

  • Mutual kNN alignment emphasizes preservation of local example neighborhoods, which is relevant to transfer even when heterogeneous models do not share globally identical representation geometry.The score averages neighborhood agreement and ranges from 0 to 1.Source: §A.4 · p. 20

A.5 Graph Construction Details

  • Graph construction maps hidden states to a common dimension, partitions them into feature blocks, and represents each selected layer–token–block triple as a node.Learnable index embeddings indicate each node’s position in the layer-token-feature grid.Source: §A.5 · pp. 20–21

A.6 Details of Relation-Aware Attention

  • CHORD uses a directed fully connected graph whose edges combine relative layer, token, and block offsets with content similarity and distance.Relation-aware attention uses edge information to modify both attention scores and propagated values.Source: §A.5 · p. 21, §A.6 · pp. 22–23

A.7 Details of Meta-Learning Training

The appendix details CHORD’s leave-one-source-model-out meta-training and the diagnostic procedures used to study alignment, transfer, and handcrafted relational features. In meta-training, a held-out source model serves as a pseudo-target for learning an auxiliary training signal.

A.7 Details of Meta-Learning Training

  • Each meta-episode holds out one source LLM as a pseudo-target, and the critic is trained to produce auxiliary gradients that improve graph-encoder performance on it.The classifier is fixed during virtual updates, and the critic is discarded at inference; a final consolidation stage trains the detector on all source-model data.Source: §A.7 · pp. 24–25

A.8 Details for the Cross-Model Alignment, Transfer, and Score-Distribution Diagnostics

  • Diagnostics use matched benchmark examples, mutual kNN overlap for structural alignment, and source-trained detectors evaluated directly on target features for transfer AUROC.Figure 3 visualizes target score distributions for factual and hallucinated examples and includes training-free Perplexity and Semantic Entropy references.Source: §A.8 · p. 26

A.9 Observation Pipeline for Relational Features

  • The observation pipeline tests fixed handcrafted token, cross-layer, feature-block, and joint relations with a lightweight MLP, separately from CHORD’s learned attention and meta-learning.The raw-state baseline uses the same selected window and common hidden-state mapping, controlling for representation size and token-layer coverage.Source: §A.9 · pp. 27–29

A.10 Baseline Implementation Details

This section standardizes baseline evaluation for cross-model use and describes benchmark labels, data, preprocessing, task inputs, and accepted answers. It covers benchmarks for coding, agent actions, factual and context-grounded QA, and reasoning, and situates CHORD within hallucination detection, cross-model generalization, and relational representation learning.

A.10 Baseline Implementation Details

  • Cross-model adaptations make training-based probes and attention detectors executable across architectures, while training-free baselines run directly on target models.MM, SEP [6], and SAPLMA [57] use padded hidden-state mappings; LapEigvals [11] maps attention into fixed-dimensional spectral features. These are compatibility operations, not improvements to the original methods; EigenScore [52] operates within each target model without cross-model mapping.Source: §A.10 · pp. 30–32

A.11 LLM-as-Judge Labeling Protocol

  • The labeling protocol uses three-way GPT-4o judgments, mapping correct and abstention to 1 and hallucination to 0 for binary detection.Separating abstention from hallucination is intended to avoid treating explicit uncertainty as factual error; deterministic decoding and structured output parsing support consistent evaluation.Source: §A.11 · pp. 32–33

A.12 Dataset Details and Preprocessing

  • Seven benchmarks are shuffled and capped at 10,000 items, split by question or task into 80/10/10 partitions before generation, and converted to a shared schema.The schema records task type, instructions, context, question, references, and available incorrect answers; references and traps are withheld from generation prompts, and each item retains its original row for traceability.Source: §A.12 · p. 33, §A.12.1 · p. 33

A.12.2 TruthfulQA

  • TruthfulQA is treated as open-ended factual QA, preserving accepted-answer variants and known incorrect answers for semantic evaluation.The incorrect alternatives remain evaluation-side signals, not model inputs.Source: §A.12.2 · p. 34

A.12.3 HaluEval

  • HaluEval uses knowledge-grounded QA, retaining its right answer as the reference and its hallucinated answer as an evaluation-only trap.The generated response is prompted to rely strictly on the provided context.Source: §A.12.3 · p. 34

A.12.4 TriviaQA

  • TriviaQA uses no-context open-ended QA and preserves answer aliases as references, falling back to the canonical answer when aliases are unavailable.It has no dataset-provided incorrect answers.Source: §A.12.4 · p. 34

A.12.5 CoQA

  • CoQA uses the first question-answer turn from each story to create one context-grounded example per story and reduce correlated examples across splits.The story supplies context, and its first answer is the reference.Source: §A.12.5 · p. 35

A.12.6 GSM8K

  • GSM8K tests reasoning-oriented hallucination detection, using the problem as the question and retaining the reference solution for semantic or normalized numeric matching.The reference may include both reasoning and the final answer; no dataset-provided incorrect answers are available.Source: §A.12.6 · p. 35

A.12.7 MBPP

  • MBPP tests functional code-generation correctness, where plausible programs can fail edge cases or violate requirements; all available items are retained, making its smaller training set a potentially fragile setting for training-based detectors.The natural-language programming requirement is mapped to the question, assertion tests are included in the prompt, and reference code and tests support evaluation.Source: §A.12.7 · pp. 35–36

A.12.8 xLAM Function Calling

  • xLAM tests agent-action hallucination detection, where correctness depends on selecting an appropriate available tool and producing an action with semantically equivalent arguments.Tool-call JSON is canonicalized when possible before evaluation; no dataset-provided incorrect tool calls are used.Source: §A.12.8 · p. 36

A.12.9 Summary of Dataset Roles

  • The seven benchmarks cover adversarial truthfulness, context-grounded QA, open-domain recall, conversational QA, mathematical reasoning, code generation, and structured agent actions to test transfer across task formats.This diversity is intended to help assess whether detectors capture model-general hallucination signals rather than task-specific artifacts.Source: §A.12.9 · pp. 36–37

B Related Work

  • CHORD addresses cross-model hallucination detection by modeling hidden states as joint layer-token-feature graphs and using relation-aware representations with meta-learning to encourage transfer to unseen LLMs.Prior detection methods use uncertainty, response consistency, hidden states, or attention structures, but are often designed for fixed LLMs or rely on output-level proxies.Source: B Related Work · p. 37

C Limitations

CHORD requires access to hidden states, limiting direct use with black-box APIs. The accompanying sections describe the experimental hardware setup and compliance with asset licenses.

C Limitations

  • CHORD is intended for white-box or partially transparent models with accessible hidden states, so it cannot be directly applied to black-box APIs that expose only generated text or token probabilities.The paper notes that the requirement is less restrictive in controlled enterprise or platform settings where providers can access activations or instrument inference.Source: C Limitations · pp. 37–38

D Computational Infrastructure

  • Experiments used four NVIDIA H100 GPUs throughout for consistency, although detector training and evaluation could in principle run on a consumer-grade GPU after hidden-state extraction.The paper cites substantial inference memory needs, especially for the 70B model, and additional memory use for hidden-state extraction.Source: D Computational Infrastructure · p. 38

E Licenses for Existing Assets

F.1 Alignment and Transfer Are Related but Decoupled

This section examines how relational views relate to cross-model transfer and detection utility, showing that preserved structure supports transfer but alignment alone does not determine detection utility. Diagnostics of variation in relational-channel importance and transfer stability motivate adaptive rather than fixed aggregation.

F.1 Alignment and Transfer Are Related but Decoupled

  • Alignment and transfer rankings are positively but moderately correlated, so greater local-neighborhood agreement does not guarantee stronger detector transfer.Alignment measures unlabeled structural consistency, whereas AUROC reflects whether transferred features separate factual from hallucinated examples.Source: Figure 11 · p. 42, §F.1 · p. 43
  • Relational views yield stronger cross-model neighborhood agreement than raw hidden states, while raw-state detectors remain near chance on most source-to-target pairs.This pattern holds across the full model suite and model families and scales.Source: Figure 8 · p. 39, Figure 9 · p. 40, §F.1 · p. 43
  • Joint relational structures are the most consistently useful fixed view for transfer, producing clearer target score separation than raw states and outperforming often well-aligned token relations.The joint view combines layer, token, and feature-block relations; the diagnostic results motivate learnable, input-dependent relation aggregation.Source: Figure 10 · p. 41, §F.1 · pp. 42–43, Figure 12 · p. 43

F.2 Handcrafted Relation Priors Help but Saturate

  • Refined handcrafted edges generally improve alignment and transfer: token-state jumps add transition patterns, layer-evolution edges encode transformation directions, and sparse feature edges suppress weak activations.The feature-edge improvement appears in several settings; benefits vary across relation families and transfer pairs.Source: Figure 13 · p. 44, §F.2 · pp. 45–46
  • Handcrafted refinements provide modest, nonuniform gains over the larger move from raw states to relational views, supporting learned input-dependent relation aggregation over a fixed edge formula.The comparisons retain the same node sets and readout while changing edge relations.Source: Figure 13 · p. 44, §F.2 · pp. 45–46

F.3 Relation-Channel Importance Is Input-Dependent

  • Factual and hallucinated examples receive different channel mixtures, and channel weights also vary substantially across individual examples.Some examples emphasize token relations, while others emphasize feature-block, layer, or joint relations.Source: §F.3 · p. 48
  • Channel preferences differ across source–target pairs, so a fixed channel mixture may be suboptimal across heterogeneous LLMs.The diagnostic probe's weights describe its allocation across channels and should not be interpreted as causal explanations.Source: §F.3 · pp. 47–48
  • The probe combines multiple channels with nonuniform, input-dependent preferences, motivating adaptive relation aggregation instead of a fixed view or mixture.The analysis supports relation-aware attention that combines positional and content-based signals adaptively.Source: §F.3 · pp. 48–49

F.4 Additional Robustness and Transfer Diagnostics

  • Relational views improve transfer in both scale directions, while large-source-to-small-target transfer is only slightly stronger than the reverse direction.The scale-direction gap is modest compared with the difference between raw hidden states and relational views.Source: §F.4 · p. 49
  • Relational views yield more stable detector performance across random seeds than raw hidden states, which show noticeably higher seed variance.This suggests raw-state transfer may depend on less stable decision boundaries.Source: §F.4 · p. 49

G Additional Experimental Analysis

Under leave-one-family-out evaluation, CHORD achieves the best AUROC across target models and seven benchmarks, spanning QA, reasoning, coding, and agent tasks. Analyses of baselines, datasets, model scale, and ablations examine where its advantage is strongest and which components support transfer.

G Additional Experimental Analysis

  • CHORD achieves the best AUROC across all target models and seven benchmarks in leave-one-family-out evaluation.The benchmarks span factual and conversational QA, mathematical reasoning, coding, and agent tasks.Source: G Additional Experimental Analysis · pp. 49–50
  • Training-free baselines remain below CHORD in averaged results, while several training-based baselines are often near random guessing under cross-model transfer.The analysis associates training-based brittleness with source-specific regularities and cross-model representation mismatch.Source: G Additional Experimental Analysis · p. 50
  • 0.8564 average AUROC on xLAM and 0.8298 on GSM8K mark CHORD’s largest benchmark gains.The analysis links these results to agent-oriented and reasoning errors that may produce structured internal deviations.Source: G Additional Experimental Analysis · p. 50
  • CHORD consistently outperforms training-free and training-based baselines on TruthfulQA, HaluEval, TriviaQA, and CoQA.Its advantage on TriviaQA and CoQA is described as complementary to lexical or consistency-based output signals.Source: G Additional Experimental Analysis · p. 50
  • Larger target models often yield stronger detection, but local reversals mean scale is not a strictly monotonic predictor.The trend is reported within Llama, Qwen, and Ministral settings, with family, version, architecture, and generation behavior also affecting results.Source: G Additional Experimental Analysis · pp. 50–51
  • Removing relation-aware attention usually causes a larger performance drop than removing meta-learning, while both ablations fall below full CHORD.The ablation table reports AUROC; relation-aware attention propagates edge information, and meta-learning simulates held-out model transfer.Source: Table 4 · p. 51, G Additional Experimental Analysis · p. 52
  • Both ablated variants exceed the strongest training-free baseline on GSM8K and xLAM, but MBPP has smaller margins and remains the most challenging benchmark.The analysis attributes MBPP’s difficulty to fewer than one thousand problems and execution-dependent code correctness.Source: G Additional Experimental Analysis · pp. 50, 52

H.1 Scale-Group Transfer

The scale-group analysis tests transfer between coarse model-size groups rather than isolating parameter count. CHORD remains strongest across transfer directions, though direction and source-group composition affect absolute results.

H.1 Scale-Group Transfer

  • The scale groups are not family-balanced at every size, so results reflect source composition as well as scale and do not isolate parameter count causally.The small group contains more models and families than the medium and large groups.Source: §H.1 · pp. 52–53
  • CHORD remains the strongest method across scale-group transfer directions, including shifts between substantially different source and target sizes.The table reports AUROC averaged over target models in each target group.Source: §H.1 · p. 53, Table 5 · p. 53
  • 0.7854 for Large→Small exceeds 0.7664 for Small→Large, but the direction effect reverses for other group pairs.Small→Medium scores 0.7760 versus 0.7688 for Medium→Small; Medium→Large scores 0.7826 versus 0.7803 for Large→Medium.Source: §H.1 · p. 53
  • Ordinary training-based baselines perform better than in leave-one-family-out evaluation but remain below CHORD.The scale-group sources can include multiple model families, producing a less severe distribution shift than holding out a full family.Source: §H.1 · p. 53
  • xLAM and GSM8K show strong CHORD gains, while MBPP remains more fragile in scale-group transfer.The analysis relates MBPP’s smaller gap to execution semantics and fine-grained program correctness.Source: §H.1 · p. 54

H.2 Fine-Grained Edge Ablation

Fine-grained ablations remove one edge channel at a time while retaining the rest of CHORD. Each channel contributes, but its relative importance varies by benchmark and target model.

H.2 Fine-Grained Edge Ablation

  • Removing any single edge channel weakens CHORD on average, while full CHORD performs best across the table.The table reports AUROC under cross-family evaluation; each ablated variant retains the other channels and meta-learning.Source: §H.2 · p. 54, Table 6 · p. 55
  • Removing token-level edges tends to hurt xLAM, CoQA, and HaluEval, where answer trajectories and token-level consistency matter.Token-level relations describe how hidden states vary across generated answer positions.Source: §H.2 · p. 54
  • Removing layer-level edges has a more visible effect on GSM8K and TruthfulQA, where representation changes across Transformer depth can carry useful signals.The analysis connects these layerwise dynamics with reasoning and semantic consistency.Source: §H.2 · p. 54
  • Removing intra-block feature edges produces a more pronounced drop on MBPP and also affects some agent and knowledge-intensive settings.These edges capture fine-grained relations among feature blocks within the unified hidden representation.Source: §H.2 · p. 54
  • No edge type dominates every setting; token, layer, and feature-block relations correspond to different patterns of hallucination errors.The analysis concludes that combining the channels benefits from their complementary signals.Source: §H.2 · pp. 54–55

H.3 Cross-Benchmark Transfer

The cross-benchmark test trains on six benchmarks and evaluates on a held-out seventh for the same LLM, separating benchmark shift from model shift. CHORD remains strongest across target models and held-out benchmarks.

H.3 Cross-Benchmark Transfer

  • The protocol trains each detector on six benchmarks from one LLM and evaluates it on the remaining benchmark for that same LLM.Each table column represents a held-out benchmark; training-free methods serve as source-independent reference baselines.Source: §H.3 · pp. 56–57
  • Cross-benchmark transfer is less severe than cross-model transfer because source and test features share the same LLM representation space.It still tests generalization to a different task distribution rather than the training benchmark.Source: §H.3 · p. 57
  • CHORD achieves the strongest performance across target models and held-out benchmarks in leave-one-benchmark-out evaluation.This tests transfer across benchmarks for a fixed LLM, alongside the paper’s separate leave-one-family-out test of transfer across LLMs.Source: §H.3 · p. 57
  • Ordinary training-based baselines often exceed 0.5 AUROC and can approach strong training-free methods when training and evaluation use the same LLM.The analysis attributes this to reuse of model-specific representation coordinates across benchmarks.Source: §H.3 · p. 57
  • QA benchmarks transfer relatively smoothly, GSM8K also performs strongly, MBPP remains fragile, and xLAM is comparatively favorable to CHORD.The analysis links MBPP’s smaller gap to execution-dependent code correctness and its distance from natural-language QA.Source: §H.3 · p. 57

NeurIPS Paper Checklist

  • The paper checklist answers yes that the abstract and introduction claims are supported by the methodology and experimental results.The checklist also says those claims should reflect the paper’s assumptions, limitations, and supported generalization scope.Source: NeurIPS Paper Checklist · p. 58

2. Limitations

The paper points readers to Appendix C for its limitations discussion and characterizes the work as foundational methodology without a direct path to specific applications or societal harms. It also reports its asset, code, and model-use disclosures.

2. Limitations

  • The paper identifies Appendix C as the location of its limitations discussion.Source: §2 · p. 58
  • The work is foundational machine-learning methodology without a direct path to specific applications or negative societal harms.Source: §2 · p. 61

12. Licenses for existing assets

  • Existing dataset and code creators are credited, their licenses are detailed in Appendix E, and the paper introduces no new foundational models or large-scale datasets.Source: §12 · pp. 62–63
  • The paper describes its use of LLMs in Section 4 because response generation and evaluation are central to its cross-model transfer methodology.Source: §12 · p. 64

Related papers

Loading 2610.38314v1…