Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

• Descriptions can shift with the distractors: speakers use different basic color terms for the same target, and may choose “blue” based on assumed shared preferences and color prototypicality.Source: §1 · p. 1, Table 1 · p. 1
• The model replaces fixed-message semantic agents with learned recurrent neural networks and recursively uses their speaker and listener predictions for pragmatic reasoning.The pragmatic speaker uses the literal RNN listener, and the pragmatic listener interprets utterances based on the pragmatic speaker’s behavior.Source: §1 · pp. 1–2
• The new corpus comprises real-time dyadic color reference games and elicits context-dependent, compositional, vague, and ambiguous language, including negations, comparatives, and metaphor.Source: §1 · p. 2
• The combined pragmatic model improves interpretation accuracy over the basic RNN listener, with the largest improvement coming from adding speaker-trained predictions despite poor speaker-only performance.Source: §1 · p. 2
• Pragmatic reasoning helps most when colors are very similar or when few referring expressions adequately identify the target.Source: §1 · p. 2

• The study recruited 967 unique participants to play 1,059 games, each with 50 rounds.Source: §2 · p. 2
• The task paired speakers and listeners in a chat-enabled game, where the speaker described a highlighted target among three color patches and the listener selected a patch.Source: §2 · p. 3
• Trials were balanced across close, split, and far conditions that varied how near the distractors were to the target in color space.Close trials had all colors within distance θ; split trials had one nearby distractor; far trials had both distractors farther than θ.Source: §2 · p. 3
• After exclusions, the corpus contained 53,365 speaker utterances across 46,994 rounds in 948 games, with the three difficulty conditions equally represented.Source: §2 · p. 3

3 Behavioral results

• The corpus was designed to assess models through listener classification accuracy and how closely speaker production resembles human participants’ language use.The behavioral findings inform the model assessment.Source: §3 · p. 3

3.1 Listener behavior

• Listener accuracy fell from 97% in far contexts to 90% in split and 83% in close contexts.These human results calibrate expectations for computational performance.Source: §3.1 · p. 3

3.2 Speaker behavior

• Human speakers used more words in split and close contexts than in far contexts.The section reports similar results for character counts.Source: §3.2 · p. 3
• Human speakers used comparatives and superlatives more when distractors were close, with comparatives favored in split and superlatives in close contexts.The patterns were significantly different across conditions.Source: §3.2 · p. 4
• Human speakers used negative constructions more when one or both distractors were close than in far contexts.“Not” was counted as the most frequent negation in the corpus.Source: §3.2 · p. 4
• High-specificity messages, defined by WordNet color-term depth greater than 7, were more likely in split and close than far contexts.The increases were small but reliable.Source: §3.2 · p. 5

4 Models

• RSA’s scalability obstacles include hand-specified message sets and interpretation functions, plus intractable normalization over potentially unbounded utterances.The model replaces literal speaker and listener agents with RNNs and samples alternative utterances to approximate normalization.Source: §4 · p. 6

4.1 Base listener

• L0 uses an LSTM encoder to map a description to a Gaussian distribution over colors in a transformed representation space.Color representations are derived by Fourier-transforming RGB vectors; the quadratic form is not guaranteed to be negative definite, though it is for more than 95% of inputs.Source: §4.1 · p. 6

4.2 Base speaker

• S0 uses an LSTM context encoder and description decoder, and supplies utterances for constructing pragmatic listeners.The context is reordered to place the target last, and the decoder predicts description tokens.Source: §4.2 · p. 7

4.3 Pragmatic agents

• L2 reasons about an S1 speaker derived from L0, using sampled S0 utterances as alternatives to avoid enumerating the full utterance space.The method uses m = 8 alternative samples per target and averages over n = 8 sampled sets.Source: §4.3 · p. 7
• L1 derives a listener from S0; weighted blends combine it with L0 and L2, with weights allowing pragmatic or base predictions to be emphasized or subtracted.The authors motivate blending partly because L1 alone is expected to be less accurate, while its speaker-trained perspective may complement listener-based models.Source: §4.3 · pp. 7–8

4.4 Training

• Training separates dyads across train, development, and test splits, and tunes model and pragmatic-reasoning hyperparameters on held-out data.The final ensemble uses a negative βb weight, amplifying the difference between L0 and L2.Source: §4.4 · p. 8

5.1 Speaker behavior

• The pragmatic speaker, like humans, produces longer utterances when colors are split or close rather than far apart.Its utterance length increases significantly in both split and close contexts compared with far contexts.Source: §5.1 · p. 8
• The pragmatic speaker uses more comparatives and superlatives in split or close contexts, but misses humans’ peak comparative use in split contexts.The reported increases versus far contexts are z = 14.45 for comparatives and z = 16 for superlatives.Source: §5.1 · p. 8
• Its negation use and high-specificity word use also increase with closer colors, matching the reported human patterns.The reported negation effects are z = 8.55 and z = 16.61; specificity effects are z = 2.65, p = 0.008 and z = 2.1, p = 0.036.Source: §5.1 · pp. 8–9

5.2 Listener accuracy

• On the test set, the final ensemble significantly improves over the base listener on both accuracy and perplexity, and is the most effective listener model.Blending speaker- and listener-based perspectives yields improvement; L2 also significantly outperforms L0, while the Lb-over-L2 gain is not significant.Source: Table 3 · p. 9, §5.2 · p. 9
• Pragmatic listener gains occur primarily in close and split contexts, where similar colors make basic color terms insufficiently distinguishing.The analysis of changed examples identifies these contexts as the primary source of improvement from including pragmatic models.Source: §5.2 · p. 9, Figure 4 · p. 9

6 Model analysis

• When sampled descriptions fail to identify a referent, L2 treats it as difficult to describe and increases its probability, which helps when that referent is the target.This reasoning can instead overweight a distractor and lead to an incorrect choice.Source: §6 · pp. 9, 11
• For the blue example, only two alternative utterances give appreciable probability to the target, so choosing “blue” supports the target over a cyan with many good description options.The reasoning reflects how a speaker’s choice among alternatives can convey information about the intended referent.Source: §6 · p. 10
• Combining listener- and speaker-based agents improves performance, with the speaker-based listener sometimes overriding incorrect predictions from both listener-based agents.The speaker can use relationships among context colors when scoring utterances, whereas the listener predicts color from the utterance a priori.Source: §6 · pp. 9, 11
• In the drab-green example, L0 honors the low-saturation, green, non-blue constraints but places its probability distribution near the yellowish-green distractor.Figure 6 visualizes L0’s color prediction as a density marginalized over HSV value.Source: §6 · p. 11, Figure 6 · p. 11
• In that context, S0 learns that “drab” better describes the grayish-green target than the yellowish distractor, enabling blends to prefer the target.Figure 5 presents conditional probabilities for two development examples, including the human utterance “drab green not the bluer one.”Source: Figure 5 · p. 10, §6 · p. 11

7 Related work

• Much prior work applied pragmatic reasoning to speaker-side generation, including models of spatial reference, robot assistance requests, and color descriptions.Several approaches required enumerating possible utterances, which is infeasible for the varied utterances in this dataset.Source: §7 · pp. 11–12
• Andreas and Klein (2016) combined neural speaker and listener models for generation; this paper shows pragmatic techniques also help listener-side understanding.Listener-side approaches either derive a listener from a speaker or add another recursive reasoning step, and both contribute to an effective listener.Source: §7 · p. 12
• The paper’s listener-side extension addresses a gap in prior neural pragmatic work, which had largely centered on generation.The cited prior approaches include pragmatic speaker models and neural reference-game generation models.Source: §7 · pp. 11–12

8 Conclusion

• A pragmatic agent incorporating neural listener and speaker models interprets color descriptions in context better than the listener alone.The paper also presents a newly collected corpus of color descriptions from reference games.Source: §8 · p. 12
• The approach relies on enumerating possible referents, a challenging requirement for tasks with intractably large referent spaces.The listener’s Gaussian representation of utterance semantics may also limit performance.Source: §8 · p. 12
• Applying the model to other referents, such as images, is possible in principle, but listener performance may be limited by its Gaussian utterance-semantics representation.Source: §8 · p. 12
• Multiturn dialogue remains a next step, requiring long-term planning, memory of earlier utterances, and decisions about when to clarify or commit to a referent.More than half of dyads had at least one two-way interaction in the reference-game task.Source: §8 · p. 12