Source-linked AI summary
Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
Emilio Jorge, Mikael Kågebäck, Fredrik D. Johansson, Emil Gustavsson
TL;DR
The paper addresses how language can be grounded in the world rather than learned from text alone. It trains two agents to communicate while playing image-based Guess Who? and reports that they learn image-grounded words and multi-step dialogue. It also reports benefits from increasing communication noise and using orthogonal messages of arbitrary dimension.
Problem
Language learned from text alone lacks interaction with the world that would ground concepts in what the words describe.
Source: §1 · p. 1Method
Results
The agents learn multi-step dialogue that uses prior conversational state, and increasing communication noise improves training speed and learning capacity while retaining discrete-symbol learning.
Source: §1 · p. 2Takeaways & Limitations
The experiments show that agents can develop words grounded in image characteristics and adapt their communication to earlier dialogue.
Source: §8 · pp. 8–9
Abstract
from arXiv · showhide
Acquiring your first language is an incredible feat and not easily duplicated. Learning to communicate using nothing but a few pictureless books, a corpus, would likely be impossible even for humans. Nevertheless, this is the dominating approach in most natural language processing today. As an alternative, we propose the use of situated interactions between agents as a driving force for communication, and the framework of Deep Recurrent Q-Networks for evolving a shared language grounded in the provided environment. We task the agents with interactive image search in the form of the game Guess Who?. The images from the game provide a non trivial environment for the agents to discuss and a natural grounding for the concepts they decide to encode in their communication. Our experiments show that the agents learn not only to encode physical concepts in their words, i.e. grounding, but also that the agents learn to hold a multi-step dialogue remembering the state of the dialogue from step to step.
1. Introduction
The paper argues that language needs interaction with the world for concepts to be grounded, and investigates whether agents can develop a shared language through situated play. In Guess Who?, the agents learn image-grounded words and multi-step dialogue while using an end-to-end reinforcement-learning model.
1. Introduction
- The motivation is that language detached from interaction with the world lacks grounding, whereas interactive communication provides feedback on utterances.The paper cites Jim’s delayed language acquisition without supervision or feedback as an illustration.Source: §1 · p. 1
- The approach uses Deep Recurrent Q-Networks to let two agents invent a language while playing Guess Who? with images as a grounding environment.The game requires words to represent image characteristics relevant to solving the task.Source: §1 · p. 1
- Prior work used continuous communication for multi-agent tasks (Sukhbaatar et al., 2016) and one-bit messages with DIAL (Foerster et al., 2016a;b); this paper extends the message dimensions and dialogue setting.It contrasts its learned image representations and multi-step dialogue with Lazaridou et al. (2016), which used a pretrained classifier and one-step, one-symbol questions.Source: §1 · p. 2
- The proposed model learns a near-optimal Guess Who? strategy without shared parameters or a predefined communication protocol.The agents invent their communication while learning the task end to end.Source: §1 · p. 2
- The agents learn to use information from earlier time steps to guide a multi-step conversation.The paper’s language analysis connects words to characteristics visible in the game images.Source: §1 · p. 2
- The authors report that increasing communication-channel noise improves training speed and learning capacity compared with constant noise while preserving discrete-symbol learning.They also report that orthogonal messages of arbitrary dimension improve system performance and interpretability.Source: §1 · p. 2
2. Background
The section introduces reinforcement learning, deep Q-networks, recurrent Q-networks, and differentiable inter-agent learning as foundations for the paper. These methods address value estimation, partial observability, and learning through messages between agents.
2.1. Reinforcement learning
- In cooperative multi-agent RL, agents can receive a shared team reward, while partial observability leaves the global state hidden and provides correlated observations instead.The value of a state-action pair is the expected return under a policy, and the optimal value function obeys the Bellman optimality equation.Source: §2.1 · p. 2
2.2. Deep Q-Networks
- DQN approximates the Q-function with a neural network because storing and updating values for every state-action pair can be computationally intractable.A target network is held fixed for a number of iterations during optimization.Source: §2.2 · p. 3
- DRQN uses a recurrent network and hidden state to aggregate observations over time when agents have partial observability.The hidden state is supplied as an input to the network.Source: §2.2 · p. 3
2.3. Differentiable inter-agent learning
- DIAL trains agents together but evaluates them separately, allowing continuous messages during training so gradients can pass between agents.A discretise/regularise unit helps reduce discretisation error at evaluation; noise during training pushes messages toward near-discrete values.Source: §2.3 · p. 3
- DIAL backpropagates recipient error through the message to the sender, and is reported to improve learning speed and ability over independent Q-learning.The model is presented as related to earlier multi-agent communication work.Source: §2.3 · p. 3
3. Guess Who?
The paper studies a cooperative variation of Guess Who? in which one agent asks questions about a hidden image and another answers. The setup uses a fixed number of rounds and evaluates games with small image subsets, as well as a larger CelebA image pool.
3. Guess Who?
- The asking agent receives a subset of two or four images, while the answering agent sees only the correct image; a correct final guess gives both agents reward 1.The agents specialize as asker and answerer and collaborate rather than compete.Source: §3 · pp. 3–4
- Players communicate through discrete questions and yes/no answers in a fixed schedule: one question-answer round for two images, or two rounds for four images.The asking agent then chooses an image as its guess.Source: §3 · pp. 3–4
- With two images chosen from 24 classes and two available question words, the stated maximum average reward is 0.89 because some images remain indistinguishable.For four images and two rounds, the equivalent score is 0.71; recurrent networks may allow later questions to have different meanings, raising that maximum.Source: §3 · p. 4
- To evaluate the proposed framework on a larger and more realistic image set, the authors also use CelebA (Liu et al., 2015), containing 202,599 face images of 10,177 unique celebrity identities, with large pose variations and background clutter.Source: §3 · p. 4
4. Model
Each agent is a recurrent neural network that processes incoming information and messages, updates its internal state, and produces actions and outgoing messages. The model uses one-hot messages, with training and evaluation message processing handled differently.
4. Model
- Each agent has an input network, a two-layer GRU, and an output network, with the RNN unrolled over the game’s time steps.The architecture’s time dimension corresponds to question-answer rounds.Source: §4 · p. 4, Figure 2 · p. 5
- The model separately embeds incoming information and the previous message, then adds those embeddings element-wise before passing them through the GRU.The resulting joint embedding is processed together with the incoming recurrent state.Source: §4 · p. 4
- The top-layer GRU state becomes the output embedding, which the output network uses to generate Q-values and a message.The Q-values generate the agent’s action.Source: §4 · p. 4
- During training, a noisy DRU variant produces one-hot message encodings; evaluation instead uses an argmax-based encoding.The model also applies batch normalization to image embeddings and regularized incoming messages, using running averages during testing.Source: §4 · pp. 4–5
- Messages use one-hot encodings, which the authors say improve results and make individual message effects easier to study than binary encodings.The authors note that one-hot encoding scales less well because n symbols require length n rather than a binary code’s length sufficient for 2^n possibilities.Source: §4 · p. 5
5. Increasing noise
Increasing communication noise is scheduled to balance rapid early learning against over-encoding that harms discretized testing. The schedule raises σ linearly over training.
5. Increasing noise
- Fixed noise creates a trade-off: high σ risks preventing learning, while low σ can yield poor testing after over-encoding.The resulting gap is attributed to discretization error.Source: §5 · p. 5
- The proposed schedule linearly increases σ over epochs, allowing quick early learning while increasingly penalizing over-encoding.Source: §5 · p. 5
- The authors compare the schedule to language learners moving from near-noiseless surroundings to understanding speech in noisier settings as proficiency grows.Source: §5 · p. 5
6. Experiments
The experiments evaluate Guess Who? performance, communication properties, and the effect of increasing noise, using image pools of different sizes and multiple vocabulary settings.
6. Experiments
- The evaluation compares increasing noise with fixed noise to assess its effectiveness.Source: §6 · p. 5
- Episodes sample two or four of 24 down-sampled Guess Who? images; two-image games use one question and answer, while four-image games use two.The model is also tested on four-image CelebA games drawn from a pool of 202,599 images, with vocabularies of eight or sixteen words.Source: §6 · p. 5
- Guess Who? tests use two, four, or eight asking-agent words, and the CelebA experiment uses eight or sixteen; answers are yes or no.Source: §6 · p. 5
- Communication analysis samples messages for history-dependent adaptation and context-sensitive word meaning, and compares performance after zeroing the answering GRU's incoming state and previous action.Source: §6 · pp. 5–6
- Unless noted otherwise, σ rises linearly from 0.1 to 1; experiments average repeated runs and use RMSProp with a 5×10^-4 learning rate.The setup also uses ε = 0.05, γ = 1, batch size 32, and target-network updates every 100 epochs.Source: §6 · p. 6
7. Results
Across the reported tests, performance improves with larger vocabularies and remains similar across several image settings. The analyses also find image-grounded questions, history-sensitive dialogue, and better results with increasing noise.
7.1. Performance
- On four-image games, performance is similar across question counts and on CelebA; a 90%/10% train-test split also yields very similar scores.With two images, more words improve performance; four and eight words surpass the two-word theoretical bound of 0.89.Source: §7.1 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7
7.2. Understanding the questions
- With a two-word vocabulary, some distinct images receive identical answers to both words, making them impossible for the asking-agent to distinguish.The analysis interprets one word as possibly asking whether the top of the person’s head is very light or dark; similar images also tend to receive similar answers in the t-SNE projection.Source: §7.2 · p. 7, Figure 6 · p. 7, Table 3 · p. 7
7.3. Interactive conversation
- In CelebA games, the second question differs in 93% of cases depending on the first answer, consistent with dialogue that adapts to communication history.Context-dependent homographs allow 8 × 2 × 8 possible question sequences rather than 8 × 8.Source: §7.3 · p. 8, Figure 8 · p. 8
7.4. Inreasing noise
- With four images and two available words, increasing σ learns faster and achieves better performance than the tested constant-noise settings.The comparison includes constant σ = 0.5 and σ values of 0, 0.1, and 1.Source: §7.4 · p. 8
8. Conclusions
The paper concludes that DRQNs can learn Guess Who? while grounding words in image characteristics and adapting communication to dialogue history. It also reports that noise regulation affects training speed and final performance.
8. Conclusions
- A DRQN learns to play Guess Who? and connects agents’ words to characteristics of the game images.Source: §8 · p. 8
- The agents create interactive conversations in which communication adapts to what has previously been said.Source: §8 · p. 9
- Regulating communication-channel noise can substantially affect training speed and final system performance.The reported comparison finds increasing σ outperforms constant-noise models.Source: §8 · p. 9, Figure 9 · p. 9
A. Derivations for optimal scoring with two words
The derivation estimates correct-image guessing by separating cases where the target images are separable from those requiring a guess. It also considers a four-image setting with two question phases.
A. Derivations for optimal scoring with two words
- For 24 images and one of two available questions, the calculation divides outcomes into separable and non-separable image pairs.Images are separable when the target and incorrect image fall in different question-defined partitions.Source: A. Derivations for optimal scoring… · p. 11
- ≈0.89: for 24 images split into four equal partitions, P(Correct) = 1 · 18/23 + 1/2 · 5/23.When the images are inseparable, the agent must guess randomly.Source: A. Derivations for optimal scoring… · p. 11
- For four images, the calculation assumes two question phases reuse the same two available questions, with one question asked in each phase.It accounts for cases where multiple images remain in the correct partition and distinguishes non-separability with one, two, or three images.Source: A. Derivations for optimal scoring… · p. 11