PaperlayerView exact revision on arXiv

Source-linked AI summary

Anti-efficient encoding in emergent communication

Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, Marco Baroni

arXiv:1905.12561v4cs.CLcs.AIcs.LGcs.MA

TL;DR

The paper addresses limited understanding of the basic properties of neural networks’ emergent codes and how they compare with human language. It studies a speaker-listener signaling game and finds anti-efficient message lengths, which shift toward ZLA when longer messages are penalized.

  • Problem

    Basic properties of emergent neural communication codes, including how their message lengths compare with human language, remain insufficiently understood.

    Source: §1 · p. 1

  • Method

    The authors train speaker and listener networks to communicate inputs in a signaling game, then examine message lengths and test the effects of penalizing long messages.

    Source: §1 · p. 2

  • Results

    The networks develop anti-efficient codes, with frequent inputs receiving longer messages and messages clustering near the maximum length; penalizing long messages yields a distribution that starts to respect ZLA.

    Source: §4 · pp. 8–9

  • Takeaways & Limitations

    The findings support controlled analysis of emergent communication and illustrate that different speaker and listener pressures can produce successful codes unlike human language.

    Source: §4 · p. 9

    The experiments cap message length at 30, although natural languages have no rigid upper bound, and replication with transformer or CNN architectures remains future work.

    Source: §3.1 · p. 5, §4 · p. 9

Abstract

from arXiv · show

Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human language. One fundamental characteristic of the latter, known as Zipf's Law of Abbreviation (ZLA), is that more frequent words are efficiently associated to shorter strings. We study whether the same pattern emerges when two neural networks, a "speaker" and a "listener", are trained to play a signaling game. Surprisingly, we find that networks develop an \emph{anti-efficient} encoding scheme, in which the most frequent inputs are associated to the longest messages, and messages in general are skewed towards the maximum length threshold. This anti-efficient code appears easier to discriminate for the listener, and, unlike in human communication, the speaker does not impose a contrasting least-effort pressure towards brevity. Indeed, when the cost function includes a penalty for longer messages, the resulting message distribution starts respecting ZLA. Our analysis stresses the importance of studying the basic features of emergent communication in a highly controlled setup, to ensure the latter will not strand too far from human language. Moreover, we present a concrete illustration of how different functional pressures can lead to successful communication codes that lack basic properties of human language, thus highlighting the role such pressures play in the latter.

1 Introduction

The paper examines whether neural networks in a simple signaling game develop message-length patterns resembling human language. It finds an anti-efficient code and relates it to differences in speaker and listener pressures.

1 Introduction

  • The study tests message-length patterns in a simple signaling game, addressing a basic property of emergent codes that earlier analyses of more complex setups had not established.The inputs are sampled according to a power-law distribution, with frequency skewed like natural language.Source: §1 · p. 1
  • Zipf’s Law of Abbreviation (ZLA) describes an inverse nonlinear relation between word frequency and length in human language.The passage presents this pattern as an efficient encoding strategy and notes that random-typing distributions can also exhibit it.Source: §1 · pp. 1–2
  • The networks develop an anti-efficient code: frequent inputs receive the longest messages, while messages overall cluster near the maximum length.When longer messages are penalized, the emergent distribution starts to respect ZLA.Source: §1 · p. 2

2 Setup

The setup uses a power-law signaling game between neural-network speakers and listeners, with message length bounded by a maximum. It compares learned codes with constructed efficiency references and natural-language word distributions.

2.1 The game

  • A speaker maps one of 1,000 inputs sampled from a power-law distribution to a message, and a listener succeeds by reconstructing that input.Messages use an alphabet with an end-of-sequence token and are capped at max_len.Source: §2.1 · p. 2

2.2 Architectures

  • Both agents are single-layer LSTMs; the speaker generates discrete symbols and the listener predicts an input from the full message.The listener processes the end-of-sequence token and uses a softmax output over the 1,000 inputs.Source: §2.2 · pp. 2–3

2.3 Optimization

  • Training minimizes the listener’s cross-entropy loss using a gradient estimator for discrete messages, with listener backpropagation and a REINFORCE-like speaker update.The authors use an Adam optimizer and train for 2,500 episodes, then evaluate accuracy with equal weight for each input.Source: §2.3 · p. 3

2.4 Reference distributions

  • The reference distributions include an optimal code, a monkey-typing process, and word-length distributions from natural-language corpora.The language comparison uses the top 1,000 words from English, Arabic, Russian, and Spanish frequency lists.Source: §2.4 · p. 3, §2.4.3 · p. 4

2.4.1 Optimal code

  • The optimal code assigns shorter messages to more frequent inputs to guarantee the shortest average message length under the alphabet and end-token constraints.For an alphabet of size three, the example has one message of length one, two of length two, and four of length three.Source: §2.4.1 · pp. 3–4

2.4.2 Monkey typing

  • Monkey typing assigns unique random messages subject to an end token and a maximum length; its distribution can resemble ZLA when the alphabet is small and max_len is large.The analysis attributes this pattern to the finite number of distinct messages available at each length.Source: §2.4.2 · p. 4

2.4.3 Natural language

  • The natural-language reference uses cleaned top-1,000-word lists, with comparisons made only in settings suited to the available alphabet sizes and length constraints.The listed alphabets range from 30 characters for English to 59 for Arabic.Source: §2.4.3 · p. 4

3 Experiments

Across successful runs, the emergent code is anti-efficient: more frequent inputs receive longer messages, often at the maximum length. Untrained-speaker and listener analyses suggest a listener-side preference for discriminable messages, while adding a length penalty produces a ZLA-like distribution; trained messages also show skewed symbol and bigram frequencies.

3.1 Characterizing the emergent encoding

  • More frequent inputs receive longer messages, with the most frequent reaching max_len; emergent lengths exceed reference distributions except for MT with a=1000.The reported analysis focuses on max_len=30, and the anti-efficient pattern also appears with uniformly distributed inputs.Source: §3.1 · p. 5, Figure 1 · p. 5

3.2 Causes of anti-efficient encoding

  • The proposed explanation is an imbalance: listeners favor message discriminability, while speakers lack an evident pressure to minimize production effort.The authors predict that listener-side bias drives the anti-efficient pattern.Source: §3.2 · p. 6

3.2.1 Untrained Speaker behavior

  • Untrained Speakers show no prior for long messages; when messages must be unique, their length distribution matches MT and would obey ZLA if the vocabulary is not too large.The comparison uses messages generated by 90 untrained Speakers and two input-message association procedures, one of which enforces uniqueness.Source: §3.2.1 · p. 6, Figure 2 · p. 6

3.2.2 Untrained Listener behavior

  • Emergent messages yield the greatest average pairwise distance in untrained Listeners’ hidden representations, making them a priori easier for those Listeners to distinguish.Natural-language messages are not easy for these Listeners to process; the authors suggest ad-hoc pressures may be needed for natural-language-like emergence.Source: §3.2.2 · pp. 6–7, Figure 3 · p. 7

3.2.3 Adding a length minimization pressure

  • With a penalty for longer messages, the emergent distribution follows ZLA, but remains less efficient than natural language and optimal codes.In the max_len=30, a=40 setting with α=0.5, lengths ascend across the 40 most frequent inputs; regularization also slows convergence and reduces successful runs, with fewer as α increases.Source: §3.2.3 · p. 7, Figure 4 · p. 8

3.3 Symbol distributions in the emergent code

  • Training produces a highly skewed symbol-frequency distribution, more extreme than natural language’s, and a bigram distribution more skewed than controls matched for mean length and unigram frequencies.Repeated-character bigrams contribute to the skew: for each of the 10 most probable symbols, P(s_r,s_r) exceeds P(s_r)^2 in more than 97.5% of runs.Source: §3.3 · pp. 7–8, Figure 5 · p. 8

4 Discussion

The networks develop an anti-efficient code, with frequent inputs assigned longer messages near the maximum length. The pattern is stable across tested network and task hyperparameters, and penalizing long messages yields a ZLA distribution.

4 Discussion

  • Frequent inputs are significantly associated with longer messages, and messages cluster near the allowed maximum length.The finding holds in a communication task with inputs sampled from a power-law distribution.Source: §4 · p. 8
  • The anti-efficient pattern is stable across tested network and task hyperparameters, though other architectures remain untested.The authors leave replication with transformers or CNNs to future work.Source: §4 · p. 9
  • Penalizing longer messages produces a ZLA distribution; the authors present this as a proof of concept and call for less ad hoc conditioning methods.They also emphasize controlled analyses and power-law inputs when studying language emergence.Source: §4 · p. 9

A.1.1 Hyperparameters

The hyperparameter study uses single-layer LSTMs, with four tested hidden-size pairings and four random-seed runs per setting.

A.1.1 Hyperparameters

  • Both agents are single-layer LSTMs, tested with hidden-size pairs (100, 100), (250, 100), (250, 250), and (500, 250).Only pairings with Speaker hidden size greater than or equal to Listener hidden size were tested, reflecting task asymmetry.Source: §A.1.1 · p. 11
  • The study uses Adam with learning rate 0.001 and Speaker entropy-regularization coefficients of 1, 1.5, or 2.Source: §A.1.1 · p. 11
  • Each hyperparameter setting is run four times with different random seeds.Source: §A.1.1 · p. 11

A.1.2 Monkey typing

The Monkey typing benchmark is adapted with a maximum message length, and its resemblance to ZLA depends on alphabet size.

A.1.2 Monkey typing

  • With the maximum message length constraint, Monkey typing produces a ZLA-like distribution only when the vocabulary size is small.Source: §A.1.2 · p. 11
  • As alphabet size increases, the Monkey typing distribution departs further from a ZLA pattern.Source: §A.1.2 · p. 11
  • Figure A1 plots mean message length by input-frequency rank across 50 Monkey typing simulations.Source: Figure A1 · p. 12

A.1.3 Natural language distributions

The studied natural-language distributions follow ZLA and approximate optimal encoding, despite differences in alphabet size.

A.1.3 Natural language distributions

  • Across the natural languages considered, word lengths follow ZLA and approximate optimal encoding despite differing alphabet sizes.Source: §A.1.3 · p. 12
  • Figure A2 compares word lengths by word-frequency rank for natural languages, emergent code, and optimal encoding at max_len = 30 and a = 40.Source: Figure A2 · p. 12
  • For readability, natural-language distributions in Figure A2 are smoothed using sliding averages of 10 consecutive lengths.Source: Figure A2 · p. 12

A.1.4 Anti-efficient emergent language

Figure A3 presents mean message length by input frequency rank across successful runs, alongside reference distributions; the text says these results align with the main paper’s Section 3.1 finding.

A.1.4 Anti-efficient emergent language

  • Figure A3 plots mean message length across successful runs by input frequency rank, with reference distributions.Natural-language distributions are smoothed as in Figure A2.Source: Figure A3 · p. 13, §A.1.4 · p. 13

A.1.5 Emergent language with uniform input distribution

With uniformly distributed inputs, agents’ messages remain longer on average than MT messages.

A.1.5 Emergent language with uniform input distribution

  • Agents’ average message length significantly exceeds MT messages with uniform inputs (t-test, p < 10−9).The comparison is shown in Figure A4.Source: §A.1.5 · p. 13

A.1.6 Randomization test

The randomization test finds anti-efficiency in emergent messages for smaller alphabets, while length regularization can produce near-optimal coding; the appendix also examines initial length distributions, repetition, and symbol entropy.

A.1.6 Randomization test

  • The test compares mean encoding length E with random permutations of messages across input types, using 10^5 permutations and a 0.005 significance threshold.A significantly large E corresponds to anti-efficiency.Source: §A.1.6 · pp. 13–14
  • Emergent messages are significantly anti-efficient for a ≤40; at a = 1000, all lengths are approximately max_len with no frequency-rank relation.Optimal coding, natural languages, and length-regularized emergent language are significantly more efficient than chance across the considered settings.Source: §A.1.6 · p. 14, Table A.1 · p. 14

A.1.7 Speaker initial length distribution

  • Figure A5 compares untrained Speakers’ message lengths by input frequency rank with MT, across settings that succeeded during training.The text says these results align with the main paper’s Section 3.2.1 finding.Source: §A.1.7 · pp. 14–15, Figure A5 · p. 15

A.1.8 The effect of length regularization

  • With α = 1 in the max_len = 30, a = 1000 setting, emergent messages approximate optimal coding; larger α values prevented successful communication training.The authors report the same pattern for other max_len and a choices, and describe accurate communication and efficiency as competing pressures.Source: §A.1.8 · p. 15

A.2 Repetition in emergent messages

  • Repetition of consecutive identical symbols increases with max_len in emergent messages, unlike in natural-language messages.The authors link this repetition to very skewed bigram distributions.Source: §A.2 · p. 15, Figure A7 · p. 16

A.2.1 Entropy of symbol distributions in different codes

Loading 1905.12561v4…