Source-linked AI summary
Can LLMs Use Relational Transformer Embeddings?
Francisco Galuppo Azevedo, Clarissa Lima Loures
TL;DR
The paper asks whether frozen relational embeddings can reliably augment an LLM for multi-table prediction without lossy serialization. It builds and trains an RT–Qwen3.5-4B soft-token hybrid, then finds that the hybrid generally fails to outperform standalone RT and is fragile across design and training choices. The authors conclude that reliable fusion requires stronger alignment and schema-aware design.
Problem
The paper investigates whether LLMs can use injected relational embeddings to overcome their difficulty with multi-table structure while avoiding lossy serialization.
Method
The study projects frozen RT embeddings into Qwen3.5-4B, adapts the LLM with LoRA, and trains it with SFT followed by GSPO across 10 tasks and four supervision regimes.
Results
Across 10 tasks and four regimes, the hybrid does not consistently outperform standalone RT, is often below random, and remains sensitive to serialization and training choices.
Takeaways & Limitations
Soft-token fusion is not a plug-and-play route to better relational prediction and appears to require stronger alignment objectives and schema-aware design.
Takeaways & Limitations
The evaluation is limited to binary classification, one 4B-parameter LLM, 100 GSPO samples per task–regime combination, and single-seed results.
Abstract
from arXiv · showhide
Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.
1. Introduction
The paper motivates injecting frozen relational embeddings into an LLM to combine structure-aware encoding with language reasoning, then tests this fusion across diverse RelBench settings. The results are largely negative: the hybrid is unstable and does not reliably surpass standalone RT.
- Relational encoders capture multi-table structure, whereas LLMs struggle with lossy serialization, context limits, and typed foreign-key semantics.
- The proposed fusion injects frozen RT embeddings as soft tokens into an LLM, aiming to bypass serialization while enabling semantic reasoning and cross-schema generalization.
- The study evaluates MLP-projected RT embeddings with LoRA-adapted Qwen3.5-4B after SFT and GSPO across 10 tasks, 6 databases, and four supervision regimes.
- The hybrid does not consistently outperform standalone RT, is frequently below random, and is sensitive to serialization format, token budget, and initialization.
- The paper contributes a concrete fusion architecture, broad supervision-regime evaluation, and failure analysis spanning serialization, token budget, masking, and initialization.
2. Background
The background contrasts cross-schema relational foundation models with LLM approaches that serialize relational data or inject learned graph embeddings. It also motivates the paper’s two-stage SFT-to-GSPO training recipe for teaching an LLM to use injected relational information.
- 2.1. Relational Transformer: RT is a cross-schema relational foundation model that encodes bounded-width, temporally constrained ego-graphs with structured attention and supports transfer to unseen schemas and tasks.
- 2.2. LLMs over Relational Data: Serialized-JSON LLM prompting is competitive but sensitive to prompt design and context length, while Rel-LLM uses projected GNN embeddings with a frozen LLM and database-specific pretraining.
- 2.2. LLMs over Relational Data: The proposed work differs by combining a cross-schema encoder with LoRA training that teaches the LLM to interpret injected relational tokens.
- The training recipe applies SFT on chain-of-thought traces before GSPO with a binary correctness reward, adapting a two-stage multimodal-reasoning strategy to relational prediction.
3. Method
The method encodes each entity’s relational ego-graph with RT, projects its embeddings into Qwen3.5-4B, and inserts them alongside text tokens. Training uses SFT-generated neighborhood descriptions followed by task-specific GSPO.
- 3.1. Problem Setup: The task is binary node classification over RelBench entities, using natural-language descriptions and temporally constrained ego-graphs built by bounded-width BFS.
- 3.1. Problem Setup: Flat serialization places projected RT embeddings in a continuous prefix, whereas hierarchical json interleaves them with field, entity, and punctuation tokens.
- 3.2. Multimodal Architecture: The architecture encodes 256 RT embeddings, projects them from dimension 256 to 2560 with a two-layer MLP, and trains only the projection and LoRA adapters while freezing RT.
- 3.2. Multimodal Architecture: The mixed sequence uses a standard causal mask over text and relational tokens, while the alternative hybrid mask permits bidirectional attention within the relational span.
- 3.3. Training Pipeline: SFT uses up to three teacher-generated style variations per sample to train neighborhood descriptions conditioned on injected RT embeddings before task-specific learning.
- 3.3. Training Pipeline: GSPO runs independently for each task–regime combination, sampling four completions and rewarding correct labels plus valid reasoning and answer blocks.
4. Experiments
The experiments evaluate hybrid RT–LLM models across supervision regimes and design choices. Standalone RT remains strongest on average, while hybrid performance is fragile to masking, serialization, token budget, and training configuration.
- Main results: The study compares SFT, GSPO, and SFT+GSPO hybrids with standalone RT using AUROC across ST, WD, CD, and ALL regimes.RT has no WD or ALL results by design.
- Ablation study: Performance is highly sensitive to design choices that should be irrelevant if the LLM robustly used injected relational tokens.The ablation covers attention masking, serialization, and relational-token context size.
- Ablation study: Causal attention improves rel-f1 by +0.047 but hurts rel-trial by −0.028, showing task-dependent effects rather than a universally superior mask.The opposite directions across two tasks indicate no stable attention strategy.
- Ablation study: JSON serialization improves reward by +0.632 on rel-f1 and +0.359 on rel-trial over flat tokens, the largest ablation effect.This sensitivity suggests the model relies heavily on textual tokens rather than injected soft tokens.
- Ablation study: Increasing relational-token length from 64 to 256 yields +0.070 on rel-f1 and +0.144 on rel-trial, while L=1024 adds little despite higher cost.The selected L=256 is presented as a practical compromise.
- Main results: Standalone RT is strongest in every reported regime, and no hybrid configuration closes the average AUROC gap.In ST, SFT scores 48.3, GSPO 51.1, and SFT+GSPO 50.7; in CD, RT retains 69.7 while hybrids fall well short.
- Main results: The hybrid exceeds RT only on user-churn and study-outcome, its two weakest tasks, with GSPO margins of 64.8 vs 63.3 and 55.9 vs 54.6.These margins do not appear on RT’s strong tasks.
- Main results: Format reward saturates early across configurations, so task-reward differences reflect prediction quality rather than format-following failures.Training curves are reported in Appendix Figure 4.
5. Conclusion
Across 10 tasks and 6 databases, the hybrid does not consistently outperform RT and is frequently below random. The evaluation is limited to binary classification, one 4B-parameter LLM, 100 GSPO samples per task–regime combination, and single-seed results.
- Across 10 tasks, 6 databases, and four supervision regimes, the hybrid does not consistently outperform RT and is frequently below random.
- The hybrid is highly sensitive to serialization format, while the evaluation uses a single 4B-parameter LLM and single-seed results.
- The study uses 100 GSPO training samples per task–regime combination and evaluates binary classification only.
A. Hyperparameters
This section provides the SFT-stage and GSPO-stage hyperparameter tables.
- Table 4 reports the hyperparameters used during the SFT stage.
- Table 5 reports the hyperparameters used during the GSPO stage.
B. SFT Data Generation
Training data are generated from structured ego-graphs, varied through teacher-model text generation, filtered for noise, and stored in chat format for SFT.
- Each training example begins with a structured ego-graph containing entity attributes and reachable rows from related tables.The ego-graph preserves table names, column names, and type values.
- A teacher model generates descriptions in narrative, analytical, and concise styles from the ego-graph.
- Four noise types are detected and removed: YAML echo, refusal, meta comment, and too short.
- Each retained example uses a system, user, and assistant turn, with SFT loss computed only over the assistant turn.
C. Prompts
The prompts define the assistant’s role, provide structured database context and task information, and require exact answer formatting alongside reasoning-trace instructions.
- The system prompt assigns the assistant to analyze database entities and predict labels from relational context.
- The prediction prompt supplies structured instances, known labels, a task description, and possible label options.
- The prompt asks the model to predict the final instance and return the answer in an exact format.
- The structured input includes center-node attributes and related-table rows with a total edge count and row limit.
- The SFT reasoning-trace prompt requires a step-by-step deduction inside <think> tags before the final description.
- The instructions prohibit text before the opening reasoning tag and request natural prose grounded in the structured data.
- The prompt template places ego-graph documentation and the ego-graph dump inside separate documentation and data fields.
D. Full Results
The section presents averaged training curves and AUROC results, with visual encodings distinguishing performance levels and highest task values.
- Figure 4 reports averaged training curves for rollout length, task reward, and format reward.
- Table 6 presents the main results using AUROC as the evaluation metric.
- Table 6 uses colors to distinguish below-random, near-random, moderate, and strong performance, with bold marking each task’s highest value.
- Table 6 identifies rel-amazon with † and rel-hm with ‡.