PaperlayerView exact revision on arXiv

Source-linked AI summary

Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language

David L. Chen, Joohyun Kim, Raymond J. Mooney

arXiv:1405.7711v1cs.CL

TL;DR

Annotated language data is costly, and the paper explores learning language grounded in perceptual context without language-specific prior knowledge. It trains a system on ambiguous commentary and extracted events to parse and generate sportscasts in English and Korean. Evaluations found accurate language-task performance and commentary sometimes similar in quality to human-produced sportscasts within this limited domain.

  • Problem

    Annotated corpora are difficult and expensive to produce, motivating language learning from linguistic input paired with perceptual context instead.

    Source: §1 · p. 1

  • Method

    The system learns to align commentary with possible game events, build parsing and generation models, and select events worth describing from ambiguous supervision.

    Source: §1 · p. 3

  • Results

    Human evaluations found generated sportscasts in some cases similar in quality to human-produced commentary in this limited domain.

    Source: §1 · p. 3, §12 · p. 33

  • Takeaways & Limitations

    The work demonstrates a system that learns to interpret and generate sportscasts in English and Korean without language-specific prior knowledge.

    Source: §12 · p. 32

    The system sometimes misses significant plays such as scoring or corner kicks, which occur less frequently and often coincide with other events.

    Source: §9 · p. 28

Abstract

from arXiv · show

We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.

1. Introduction

The paper addresses the cost of annotated language data and the lack of learned systems that jointly align, parse, and generate language grounded in perceptual context. It presents a bilingual RoboCup sportscaster trained from ambiguous commentary and game-event pairs, with evaluations showing accurate language tasks and human-comparable commentary in some cases.

1. Introduction

  • The work targets the expense of annotated corpora and the gap between grounded-language systems focused on raw perception or word meanings and complete sentence-level language learning.It studies alignment, semantic parsing, and natural-language generation together in a simulated soccer environment.Source: §1 · pp. 1–2
  • The system learns to interpret and generate language from ambiguous commentary paired with game events, and learns which events are worth describing.Its strategic generation selects what to say, while tactical generation determines how to say it.Source: §1 · p. 2
  • Human evaluations found that generated sportscasts were in some cases similar in quality to human-produced commentary in this limited domain.Automatic and human evaluations also assessed disambiguation, semantic parsing, and generation.Source: §1 · p. 3
  • Training the system in English and Korean without language-specific prior knowledge demonstrates its language independence within the RoboCup sportscasting task.The authors describe the language as restricted compared with handcrafted commercial systems and frame feasibility as the goal.Source: §1 · p. 3

2. Background

The background reviews semantic parsing and generation methods, focusing on systems that can learn from ambiguous sentence-to-meaning pairs. KRISPER extends KRISP for ambiguity, while WASP and WASP−1 use a learned grammar for parsing and generation.

2.1 KRISP and KRISPER

  • KRISP learns substring classifiers for meaning-representation grammar productions, then composes their predictions into a sentence-level semantic parse.Its string-kernel SVM classifiers were reported as robust to noisy training data.Source: §2.1 · p. 4
  • KRISPER iteratively selects the most likely meaning representation for each sentence and retrains KRISP on the resulting pairs.It was designed for ambiguous training data but had previously been tested only on artificially corrupted or generated data.Source: §2.1 · p. 4

2.2 WASP and WASP−1

  • WASP adapts statistical machine translation to translate between natural-language sentences and meaning-representation language.It learns from parallel examples rather than requiring each sentence to be manually annotated with its meaning representation.Source: §2 · p. 3, §2.2 · p. 4
  • WASP aligns natural-language words with meaning-representation grammar productions, then uses synchronous CFG rules to support parsing and generation.A probabilistic chart parser selects a synchronous derivation for parsing, while WASP−1 generates with a noisy-channel model and an n-gram language model.Source: §2.2 · p. 6

3. Sportscasting Data

The data pairs human English and Korean soccer commentaries with symbolic events automatically extracted from simulated games. Each comment is paired with recent possible events, producing ambiguous supervision that can also include comments with no correct event match.

3. Sportscasting Data

  • Game events were extracted from simulator traces and represented as timestamped predicate-logic facts, with a manually developed CFG defining the meaning-representation language.The extracted events include ball actions and game states such as kickoff, offside, and corner kick.Source: §3 · p. 6
  • Human commentators produced English and Korean commentary, and each comment was paired with events occurring no more than five seconds earlier.The meaning-representation tokens were treated as arbitrary concepts whose connection to either language had to be learned.Source: §3 · p. 7
  • The corpus contains four RoboCup finals from 2001–2004, with synonyms and polysemes presenting language challenges.The 2001 final had almost twice as many events as the other games because it went into double overtime.Source: §3 · p. 7
  • About one fifth of English comments and one tenth of Korean comments had no correct match among the extracted events.Some comments described concepts absent from the meaning-representation language, while others lacked a match because of incomplete event detection or the five-second window.Source: §3 · p. 9

4. Learning Tactical Generation from Ambiguous Supervision

The section introduces systems that use different criteria to disambiguate ambiguous NL–MR training pairs before training a language generator. WASPER-GEN performs best across matching, parsing, and generation, though generation rankings differ from the other tasks.

4.1 WASPER

  • WASPER selects NL–MR pairs by maximizing the parser’s probability of assigning an MR to each sentence, then trains a WASP parser and generator.Its EM-like retraining extends WASP to ambiguous data.Source: §4.1 · p. 9

4.2 KRISPER-WASP

  • KRISPER-WASP uses KRISPER to disambiguate the data, then trains WASP on the resulting pairs to produce a parser and generator.The approach combines KRISPER’s handling of noisy training data with WASP’s generation capability.Source: §4.2 · p. 11

4.3 WASPER-GEN

  • WASPER-GEN selects NL–MR pairs using a generation-based score rather than a parsing-based score.It scores each pair by the NIST score between its sentence and the best sentence generated from its MR.Source: §4.3 · p. 11

4.4 Experimental Evaluation

  • Evaluation compares four systems against random-pairing and gold-matching WASP baselines across matching, parsing, and generation.Matching is evaluated on training data; parsing and generation are tested on held-out games, with results averaged over train/test combinations.Source: §4.4 · pp. 11–12
  • WASPER-GEN performs best on matching in English and Korean, and on semantic parsing overall and tactical generation.It outperforms KRISPER on matching in both datasets; parsing results largely track matching, while relative system performance changes for generation.Source: Figure 3 · p. 13, §4.4.1 · p. 13, Figure 4 · p. 13

4.4.2 SEMANTIC PARSING

  • For parsing, WASPER-GEN is best overall on both languages, and results generally track matching performance because cleaner training pairs reduce noise.KRISPER is comparatively robust to noise, yielding a comparable English parser despite noisier training pairs.Source: Figure 4 · p. 13, §4.4.2 · p. 14

4.4.3 GENERATION

  • For generation, WASPER-GEN leads, while even random-matching WASP outperforms KRISPER-WASP on both datasets and WASPER on Korean.Generation tolerates training noise more than parsing; the authors suggest WASPER-GEN may benefit from fewer MR patterns and a softer score, while BLEU does not capture commentary variety.Source: §4.4.3 · pp. 14–15, Figure 5 · p. 14

5. Learning for Strategic Generation

The section introduces IGSL, which learns event-mention probabilities from ambiguous training data to decide what a sportscast should describe. It outperforms methods using inferred matchings and approaches gold-matching performance, though that comparison uses predicates alone.

5. Learning for Strategic Generation

  • IGSL estimates event-mention probabilities from ambiguous training data by iteratively treating sentence-to-meaning-representation matchings as hidden variables.Its iterations estimate matching probabilities from current event priors, then update event priors from those matchings.Source: §5 · p. 15

5.1 Experimental Evaluation

  • Unlike estimates based only on inferred matchings, IGSL uses events that occur without any associated training comment.This helps it assign low mention probabilities to rarely commented events such as ballstopped and kick.Source: §5 · p. 15, §5.1 · pp. 17–18

5. Learning for Strategic Generation

  • At each timestep, the generator samples an occurring event using normalized probabilities, then stochastically decides whether to comment on it.The second decision helps avoid commenting on every event, especially events rarely mentioned.Source: §5 · p. 16, Figure 6 · p. 16

5.1 Experimental Evaluation

  • IGSL performs best among the evaluated methods and approaches gold-matching performance, although the gold baseline is limited to event predicates.Evaluation compares how often predicted event descriptions coincide with those chosen by human commentators.Source: §5.1 · p. 17, Figure 7 · p. 17

6. Using Strategic Generation to Improve Matching

The section incorporates IGSL event-mention probabilities into sentence-to-meaning matching, combining strategic and tactical generation scores. This improves matching and parsing over WASPER-GEN, but not tactical generation.

6.1 WASPER-GEN-IGSL

  • WASPER-GEN-IGSL multiplies the NIST sentence-generation score by IGSL’s event-mention probability to select sentence–meaning pairs.The strategic score favors meanings whose event types are more likely to be discussed, particularly when tactical generation scores cannot distinguish candidates.Source: §6.1 · p. 18

6.2 Variant of WASPER-GEN Systems

  • NIST is unnormalized, and short commentaries often lack higher-order n-gram matches, motivating consideration of alternative metrics.METEOR is normalized between 0 and 1 and emphasizes word-to-word matches; English shows no clear metric winner, while Korean has a slight METEOR edge.Source: §6.2 · pp. 18–19, §6.3 · p. 19

6.3 Experimental Evaluation

  • Both WASPER-GEN-IGSL variants outperform WASPER-GEN on matching and semantic parsing for English and Korean, while strategic information does not improve tactical generation.The tactical-generation variants come close to, but do not beat, WASPER-GEN.Source: §6.3 · pp. 19, 21, Figure 8 · p. 20, Figure 9 · p. 20
  • The matching gains are clear, but tactical generation remains near WASPER-GEN’s upper-baseline performance, leaving little room for improvement.The paper suggests a ceiling effect as a possible explanation.Source: §6.3 · pp. 19, 21, Figure 10 · p. 21

7. Using a Generative Alignment Model

The section uses Liang et al.’s generative alignment output to train or initialize the paper’s systems. Initialization generally improves performance across matching, parsing, and generation, and combined systems often surpass direct training on those alignments.

7. Using a Generative Alignment Model

  • Liang et al.’s model addresses alignment but not semantic parsing or language generation; its English alignments improve downstream systems, while its Korean matching results are much worse than the paper’s best system.Using those alignments as WASP training data produced good semantic parsers and tactical generators overall.Source: §7 · pp. 21–22
  • Using Liang et al.’s disambiguated pairs to initialize iterative alignment methods, rather than training initially on all ambiguous pairs, is the tested initialization strategy.The reported results indicate that combining this initialization with the paper’s disambiguation techniques often exceeds direct use of the alignments.Source: §7 · pp. 22–23
  • Initializing with Liang et al.’s alignments almost always improves matching, semantic parsing, and tactical generation, and usually outperforms training WASP directly on those alignments.The exception is semantic parsing on English; for Korean, WASPER-GEN-IGSL-METEOR with initialization leads on matching and parsing, while WASPER-GEN with initialization leads on generation.Source: Table 5 · p. 22, Table 6 · p. 22, Table 7 · p. 22

8. Removing Superfluous Comments

The section addresses comments with no corresponding extracted event and estimates their rate through internal cross-validation. It then evaluates how their removal affects matching, parsing, and generation.

8. Removing Superfluous Comments

  • About one fifth of English comments and one tenth of Korean comments are superfluous because they refer to no currently extracted event in the potential meaning representations.Such comments can mention past events or information outside the representation, or arise from perception and event-timing errors.Source: §8 · p. 23

8.1 Estimating the Superfluous Rate Using Internal Cross Validation

  • Internal cross-validation selects a pruning threshold by testing parsers trained with different thresholds on held-out ambiguous data, then retrains the selected parser on all training data.The approach assumes most sentences have their correct meaning representation among the potential MRs associated with them.Source: §8.1 · p. 24

8.2 Experiments

  • Removing superfluous comments marginally improves parsing, decreases tactical generation performance, and is evaluated for matching in Figure 11.The supplied Figure 11 caption identifies a comparison but does not state its outcome.Source: Figure 12 · p. 25, Figure 13 · p. 25, Figure 11 · p. 25

9. Human Subjective Evaluation

Human judges rated machine- and human-produced English and Korean sportscasts on fluency, semantic correctness, and sportscasting ability. English machine scores were higher on average but not significantly; Korean human scores were higher, while machine commentaries were still judged acceptable.

Evaluation setup

  • Judges assessed eight clips from four RoboCup finals, each paired with a human and system commentary; English clips averaged 36 ratings and Korean clips averaged seven.The system used IGSL for event selection and WASPER-GEN for tactical generation, with the test game excluded from training.Source: §9 · p. 26

Human evaluation results

  • English machine commentaries scored higher than human commentaries on average across categories, but the differences were not statistically significant.The comparison used an unpaired t-test with p > 0.05.Source: §9 · pp. 26, 28
  • For Korean, humans scored higher in every category, but the largest score gap was 0.8 and judges mistook machine commentaries for human ones one third of the time.The absolute scores indicated at least acceptable quality; the authors attributed some performance difference to Korean commentaries describing events absent from the extracted meaning representations.Source: §9 · p. 28

Qualitative evaluation

  • Judges generally found the generated commentaries good and accurate, but reported missed significant plays, long gaps, and insufficient detail.The event detector focused on ball possession rather than player positions or elapsed time.Source: §9 · p. 28

10. Related Work

The related work spans semantic parsing, language generation, and grounded language learning. The paper contrasts its learning from ambiguous perceptual-context data with prior approaches that rely on annotations, hand-built rules, restricted language, or narrower tasks.

10.1 Semantic Parsing

  • Grounded semantic parsing matters because syntactic annotation or prior syntactic knowledge is not directly available from perceptual context; the framework can use parsers that provide confidence scores.The paper's atomic-formula meaning representations can also be represented as frames and slots.Source: §10.1 · pp. 28–29

10.2 Natural Language Generation

  • Prior RoboCup sportscasters used hand-built templates and rules, while this work learns generation components from ambiguous data; strategic-generation research also includes classifiers, reinforcement learning, and joint decisions.The paper's setting lacks the reinforcement available in the cited game-based approach.Source: §10.2 · p. 29

10.3 Grounded Language Learning

  • Grounded-language systems often learn individual words or short expressions with restricted language or pre-coded knowledge, whereas this work targets fuller language and actions using statistical grammars and syntax-based translation.The paper also distinguishes its task from text-event alignment work that does not address semantic parsing or generation.Source: §10.3 · pp. 30–31

11. Future Work

Future work proposes expanding the perceptual representations and improving how the system models and integrates information. The authors also identify real games and instruction-following as extensions.

11. Future Work

  • Extending extracted meaning representations to cover events such as dribbling could address commentary about actions the current perceptual system misses.Adding score or turnover statistics could make commentaries more natural and engaging, though they may be difficult to learn correctly.Source: §11 · p. 31
  • Learning higher-level concepts such as “sloppiness” from patterns across events could reduce reliance on predefined concepts, but extracting many such predicates would increase training ambiguity.The current system assumes the needed concepts are already available and representable in meaning representations.Source: §11 · pp. 31–32
  • Treating each meaning representation independently misses relations among events; a graphical representation could encode entities, events, and their relationships.The paper gives a pass preceded by a kick and a bad pass followed by a turnover as examples.Source: §11 · p. 32
  • Tactical and strategic generation are loosely coupled, and tighter integration could yield stronger results across tasks.Initializing with Liang et al. (2009) produced somewhat better results, but component interaction remained loose.Source: §11 · p. 32
  • The authors propose applying the approach to real RoboCup games using symbolic traces from a real perceptual system, and to spoken-language input through speech recognition.They are also exploring learning to interpret and generate navigation instructions in virtual environments.Source: §11 · p. 32

12. Conclusion

The system learns to interpret and generate sportscasts from human commentaries paired with automatically extracted game events, without explicitly annotated training data. Evaluation found accurate parsing and generation, with sportscasts competitive with human-produced ones.

12. Conclusion

  • Training on paired commentaries and automatically extracted game events supports sportscast generation without explicitly annotated training data, in both English and Korean.Source: §12 · p. 32
  • When initial training data is very noisy, selecting NL–MR pairs by generation evaluation yields better results than selecting them by semantic parsing scores.Source: §12 · p. 32
  • Estimating which event types evoke commentary supports strategic generation, and using that information to disambiguate training data improves results.Source: §12 · p. 32
  • Initializing with alignments from another system achieves better results than either system alone.Source: §12 · p. 32
  • Experimental evaluation found that the system accurately parses and generates comments and produces sportscasts competitive with human-produced ones.Source: §12 · p. 33

Appendix A. Details of the meaning representation language

The appendix describes detected events and gives the meaning-representation grammar, whose derivations begin at the root symbol *S.

Appendix A. Details of the meaning representation language

  • Table 10 provides explanations of the different events detected by simulated perception.Source: Appendix A · p. 33, Table 10 · p. 33
  • The meaning-representation grammar starts derivations at *S and includes productions for play modes and event forms such as passes, steals, and blocks.Source: Appendix A · p. 33
  • The grammar enumerates play-mode values, including restarts and goals, and player symbols for pink and purple team members.Source: Appendix A · p. 34
Loading 1405.7711v1…