Source-linked AI summary
Investigating Knowledge Transfer Across Interactive Dialogue Games
Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda, Pavel Merinov, Raffaella Bernardi, Oswald Lanz, Alessandro Suglia, Alessandro Torcinovich
TL;DR
Dialogue games require models to coordinate language-based goals and rules while exercising varied cognitive skills, but systematic evidence about transfer across games is limited. The paper adapts performance-based transfer graphs and task-vector analysis to clembench games, finding that visuospatial games generalize especially well while weight similarity captures game-role relationships rather than transferability. The analysis is exploratory and limited to mostly collaborative clembench games and positive transfers.
Problem
Systematic evidence is missing on how knowledge transfers across a broad variety of dialogue games, despite evidence of positive transfer on limited task sets.
Method
The paper finetunes LLMs on clembench games, models performance as a directed task-transferability graph, and compares task vectors from finetuned models with their baselines.
Results
Visuospatial games show superior generalization, while similarity-based task-vector metrics are insufficient to recover transferability from weight space.
Takeaways & Limitations
Transfer direction must be measured through gameplay because weight similarity records what was trained rather than how knowledge transfers.
Takeaways & Limitations
The exploratory analysis is limited to clembench games, which are mostly collaborative, and focuses only on positive transfers.
Abstract
from arXiv · showhide
Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.
1 Introduction
The paper investigates whether knowledge transfers across dialogue games, where language encodes goals and rules and gameplay requires multiple interacting cognitive skills. It combines performance-based transfer graphs with task-vector analysis to characterize these relationships.
- Dialogue games require language understanding, rule-following, reasoning, turn-level contributions, and global strategy to reach game goals.
- Systematic evidence about transfer across diverse dialogue games remains limited despite prior evidence of positive knowledge transfer.
- The study adapts Taskonomy to clembench, finetuning LLMs on specific games and constructing graph representations of knowledge-transfer relationships.
- Finetuning on another game can sometimes improve performance more than finetuning on the target game itself.
- Visuospatial games, including exploration games, transfer knowledge to purely verbal games such as Taboo, with some transfers occurring asymmetrically.
- Task-vector similarity does not explain transfer relationships, instead providing fingerprints of the game and player role.
- Weight similarity records what was trained, whereas transfer direction must be measured through gameplay.
2 Related Work
Related work positions dialogue games as interactive probes of model capabilities and connects this study to structured task transfer and parameter-space analysis. The paper extends these lines by examining internal capability organization and transfer across a broad game suite.
- Dialogue Games for LLM Evaluation: Dialogue games provide multi-turn, goal-oriented settings with multiple roles, revealing capabilities that static benchmarks may not surface.
- Dialogue Games for LLM Evaluation: The paper uses performance across a structured collection of dialogue games to study how model capabilities are internally organized and transferred across tasks.
- Interactive and Collaborative Games as Assessment Probes: Interactive and serious games have been used to assess collaborative problem solving, communication, teamwork, strategy, and perspective taking from naturalistic interaction traces.
- Transfer Learning Across Tasks: Prior NLP transfer studies show that transfer relationships are often asymmetric and difficult to predict from surface task similarity, motivating the adapted Taskonomy framework.
- Parameter Space Analysis and Mechanistic Interpretability: Task-vector research examines task-specific adaptations and their composition in weight space, while related work suggests such adaptations often occupy low-dimensional subspaces.
- Parameter Space Analysis and Mechanistic Interpretability: Fine-tuning may enhance preexisting internal mechanisms rather than introduce new ones, potentially explaining why some game adaptations generalize more broadly.
3 Investigating Knowledge Transfer
The study models transfer among role-specific dialogue-game tasks using finetuned specialists and a budgeted graph-optimization framework. It complements performance-based transfer analysis with task-vector comparisons in model weight space.
- Dataset and game setup: The dataset contains 15 role-specific tasks from 9 clembench games after filtering transcripts and excluding tasks with fewer than 800 samples.Two-player transcripts are split into independent conversations reflecting player roles, and models are finetuned with a standard autoregressive loss.
- Dataset and game setup: The selected games are grouped into verbal games operating over words and visuospatial games operating over grids and spatial configurations.The taxonomy coarsely characterizes the primary cognitive demands of each game.
- Taxonomy modeling: The task-transferability graph represents tasks as vertices and directed source-to-target edges weighted by performance obtained after training on the source task.The collective transferability problem selects a source set under budget γ to maximize collective transfer performance.
- Taxonomy modeling: The optimization imposes source-selection, incoming-transfer, and source-budget constraints, and its optimum determines the searched graph structure.Each target has at most one incoming source task, while the source set contains at most γ tasks.
- Transfer measurement: Each specialist is evaluated on every task using clembench’s mean quality score, which separates game-playing competence from rule-following.Scores are baseline-adjusted, and entries where specialists underperform the baseline are removed from the transfer matrix.
- Weight-space analysis: Task vectors quantify finetuning changes as the difference between specialist and base-model parameters, enabling complementary weight-space comparisons.The weight-space analysis compares updates induced by training on different tasks.
4 Experimental Evaluation
The evaluation compares transfer taxonomies, task-vector similarity, and robustness against random taxonomies. Visuospatial tasks transfer broadly, whereas weight similarity captures game-role structure but does not reliably predict transfer.
- 4.2 Transferability: Visuospatial tasks prevalently transfer better across all budget regimes.
- 4.2 Transferability: At γ = |T | = 15, visuospatial tasks are self-sufficient, while verbal tasks generally benefit from visuospatial transfer.Self-sufficient tasks achieve their best performance without transfer, indicated by looping edges.
- 4.2 Transferability: Adventuregame is identified as a strong baseline specialist because it teaches planning for navigating rooms and interacting with objects.
- 4.3 Robustness of the Identified Taxonomies: The identified transfer sets significantly outperform 1000 random taxonomies under the same supervision-budget constraint.The comparison evaluates collective transfer performance.
- 4.4 Weight Analysis: Task-vector similarity does not determine transfer: taboo_P1/P2 is similar but transfers in neither direction, while asymmetric transfers have lower similarity.Similarity metrics are symmetric, whereas transferability can be asymmetric.
5 Conclusions
The paper adapts Taskonomy to study transferability between dialogue games and complements this analysis with task vectors. Visuospatial games generalize better than verbal games, while similarity metrics are insufficient to recover transferability from weight space.
- The study adapts Zamir et al.’s Taskonomy methodology to analyze transferability between dialogue games.
- Visuospatial games show superior generalization performance compared with verbal games, including across different game families.
- Task-vector similarity metrics are insufficient for recovering dialogue-game transferability from finetuning-induced weight updates.
6 Limitations
The study is exploratory and limited mainly to clembench’s mostly collaborative games, with experiments focused on positive transfer.
- The exploratory findings require more experiments, other LLMs, broader games including competitive settings, and analyses of negative transfer.The authors also propose nonsymmetrical operators and curriculum-learning applications as future directions.