Source-linked AI summary
Probabilistic Frame Induction
Jackie Chi Kit Cheung, Hoifung Poon, Lucy Vanderwende
TL;DR
Manually built frames limit their use, and existing automatic approaches can be difficult to diagnose or extend. ProFinder induces frames with a probabilistic model of events, participants, and transitions, using split-merge learning. It reports state-of-the-art end-to-end extraction results while reducing engineering effort.
Problem
Frames support information extraction and other language tasks, but have often been manually constructed, while recent induction methods use ad hoc procedures that are difficult to diagnose or extend.
Source: §1 · pp. 1–2Method
ProFinder jointly models frames, events, participants, and their transitions as latent structure, using a split-merge method to determine the number of events and slots.
Source: §1 · p. 2, §3 · p. 3, §3.4 · p. 5Results
Takeaways & Limitations
The probabilistic approach substantially reduces engineering effort compared with frame induction procedures involving multiple processing steps and customized metrics.
Source: §1 · p. 2, §4 · p. 6The evaluation does not yet assess event clustering in addition to slots and entities.
Source: §6 · p. 9
Abstract
from arXiv · showhide
In natural-language discourse, related events tend to appear near each other to describe a larger scenario. Such structures can be formalized by the notion of a frame (a.k.a. template), which comprises a set of related events and prototypical participants and event transitions. Identifying frames is a prerequisite for information extraction and natural language generation, and is usually done manually. Methods for inducing frames have been proposed recently, but they typically use ad hoc procedures and are difficult to diagnose or extend. In this paper, we propose the first probabilistic approach to frame induction, which incorporates frames, events, participants as latent topics and learns those frame and event transitions that best explain the text. The number of frames is inferred by a novel application of a split-merge method from syntactic parsing. In end-to-end evaluations from text to induced frames and extracted facts, our method produced state-of-the-art results while substantially reducing engineering effort.
1 Introduction
Frames represent related events, their prototypical transitions, and shared participant slots, but have often been manually constructed and difficult to extend. ProFinder introduces a probabilistic approach to inducing frames and reports state-of-the-art end-to-end extraction results with reduced engineering effort.
1 Introduction
- Frames comprise related events with prototypical transitions and slots for common participants, supporting representations used in information extraction, dialogue, and generation.The paper illustrates how events and participants can connect across a larger scenario.Source: §1 · p. 1
- Manual frame construction limits applicability to relatively few domains and slots, and extraction also requires effort to annotate examples or design supervised features.The paper notes that this paradigm can hinder generalization across tasks and may suffer from annotator bias.Source: §1 · pp. 1–2
- Earlier automatic frame induction by Chambers and Jurafsky (2011) relied on ad hoc clustering, customized similarity metrics, and external-corpus retrieval, making replication and adaptation harder.The paper also identifies the lack of a coherent model as an obstacle to incorporating linguistic insights and prior knowledge.Source: §1 · p. 2
- ProFinder achieved state-of-the-art end-to-end entity extraction on standard MUC and TAC datasets while reducing engineering effort and requiring no external data.The approach models frame and event transitions, event-slot correlations, and surface realizations in one probabilistic framework.Source: §1 · p. 2
2 Related Work
Related approaches reduce manual effort or induce semantic structure, but differ in whether they require predefined frames or capture discourse-level dependencies. ProFinder combines probabilistic semantic and discourse modeling, while its notion of frames differs from several established representations.
2 Related Work
- Bootstrapping still requires prespecified frames and seed words, while Open IE uses domain-independent patterns but extracts noisy, non-canonicalized atomic factoids within sentences.These approaches address manual effort through different constraints and output scope.Source: §2 · p. 2
- Unsupervised semantic induction and slot-induction methods model distributional contexts, but focus on independent sentences and do not capture discourse-level dependencies.Examples include semantic parsing, semantic role labeling, and slot induction from web search logs (Poon and Domingos, 2009; Swier and Stevenson, 2004; Lang and Lapata, 2011; Cheung and Li, 2012).Source: §2 · p. 2
- ProFinder combines sequential topic models for frames and events with modeling of dependencies among events, slots, and surface text (Gruber et al., 2007; Poon and Domingos, 2009).Its transition modeling is also inspired by topic-model applications in summarization (Barzilay and Lee, 2004; Daumé III and Marcu, 2006; Haghighi and Vanderwende, 2009).Source: §2 · p. 2
- ProFinder is closest in aim to Chambers and Jurafsky (2011), but its probabilistic approach is described as easier to modify or extend, including with linguistic features or labeled examples.The paper contrasts this with Chambers and Jurafsky's earlier narrative event-and-argument work.Source: §2 · pp. 2–3
- ProFinder's frames relate to frames, schemata, and scripts, but do not model Minsky-style inheritance; FrameNet emphasizes semantic roles rather than event or frame transitions, while scripts are manually defined (Minsky, 1974; Rumelhart, 1975; Schank and Abelson, 1977; Fillmore, 1982; Baker et al., 1998).The paper notes that split-and-merge learning produces intermediate event and slot hierarchies, which it currently discards.Source: §2 · p. 3
3 Probabilistic Frame Induction
ProFinder models documents and word-to-frame assignments jointly, learning frame, event, and slot structure from text. Its split-merge procedure dynamically determines event and slot counts, while learned assignments support event and entity extraction.
3 Probabilistic Frame Induction
- ProFinder defines a joint generative model over documents and word-to-frame assignments, learning the number of frames, events, and slots by maximizing document likelihood.The most probable assignments can also be used for event and entity extraction.Source: §3 · p. 3
3.1 Base Model
- Frame assignments are shared within clauses and Markov-linked across neighboring clauses, while events transition within frames and emit heads and argument slots.Slots generate argument heads and dependency labels independently of the event.Source: §3.1 · pp. 3–4
3.2 Model refinements
- Refinements add a background frame for generic content, a stickiness prior for neighboring frame transitions, and caseframes pairing event heads with dependency relations.The caseframe representation is intended to provide a stronger signal for slot identity.Source: §3.2 · pp. 4–5
3.3 Full generative story
- The generative story draws model distributions and then generates each clause’s background status, frame, event, event head, slots, and argument emissions.The model uses additive smoothing with uniform Dirichlet priors for its multinomials.Source: §3.3 · p. 5
3.4 Learning and Inference
- Inference uses forward-backward and Viterbi algorithms, while incremental EM learns parameters; split-merge addresses model-size selection and initialization challenges.The paper notes that EM can suffer from local optima, particularly with many hidden states and poor initialization.Source: §3.4 · pp. 5–6
- Split-merge learning starts with one event and two slots per frame, then splits and merges elements to adapt their number to the data.Starting small reduces local optima and supports coarse-to-fine slot refinement.Source: §3.4 · pp. 5–6
4 MUC-4 Entity Extraction Experiments
ProFinder is evaluated on MUC-4 entity extraction against Chambers and Jurafsky (C&J), with document filtering and learned frame examples also examined. It outperformed C&J while using a single probabilistic model and MUC-4 data alone.
Evaluation
- MUC-4 contains 1,300 training and development documents and 200 test documents on South American terrorism, with evaluation focused on four slots across four templates.The evaluated slots are perpetrator, physical target, human target, and instrument.Source: §4 · p. 6
- For each MUC template, the best-mapping induced slot was selected using development-set F1, then induced slots were merged across templates for final evaluation.Correctness was based on matching head words; optional MUC slots were excluded from recall.Source: §4 · p. 6
Results
- ProFinder outperformed C&J on MUC-4 entity extraction despite using only MUC-4 data and a single probabilistic model.C&J used additional Gigaword text; Table 1 reports the comparison.Source: §4 · p. 6, Table 1 · p. 7
- Filtering to gold-standard relevant documents substantially improved precision, suggesting room for improvement through better document classification.The system also lacked the coreference information used by C&J.Source: §4 · p. 7
Learned frames
- ProFinder learned events beyond those annotated in MUC, including a discussion event, and identified additional arrest, peace-agreement, and authorities structures.Its background frame captured reporting verbs such as say, continue, add, and believe, but missed report.Source: §4 · p. 7
5 Evaluating Frame Induction Using Guided Summarization Templates
The TAC guided-summarization evaluation tests frame induction on more diverse frames and slots than MUC-4. ProFinder outperforms C&J on recall and F1, though not precision, including when compared with C&J's five-to-one mapping.
Evaluation motivation
- The evaluation addresses MUC-4's limited template and entity variety by testing on TAC, which spans diverse frames and topics.The authors frame the task as extracting structured information from source text, linking summarization and information extraction.Source: §5 · p. 7
Data preparation
- TAC 2010 provides 46 clusters across five domains, each with 20 source documents and eight model summaries annotated with template slots.The evaluation uses the 368 model summaries and 920 source texts.Source: §5 · pp. 7–8
Task definition
- Because TAC lacks source-text entity annotations, the task extracts entities from annotated model summaries, where important domain slots occur more frequently.This choice also avoids the effort of training annotators to label source text reliably.Source: §5 · p. 8
Entity representation
- TAC contributors are normalized to maximal-noun-phrase head lemmas, with matches requiring the same head and summary; the heuristic can add noise when meaning depends on a larger phrase.The authors leave denoising this process to future work.Source: §5 · p. 8
Method and experiments
- Because some TAC slots are more general than MUC-style slots, the evaluation maps the N-best induced slots to TAC slots, with N up to 5.The models are compared on the same source texts without external data for C&J; relevance classification is omitted because summary sentences are expected to be relevant.Source: §5 · p. 8
Results
- On TAC, ProFinder beats C&J on recall and F1 but not precision; ProFinder with 1-to-1 mapping also outperforms C&J with 5-to-1 mapping.Overall results are poorer than on MUC-4, and the task is harder given the greater diversity of frames and slots.Source: §5 · p. 8, Table 2 · p. 8
6 Conclusion
The paper presents the first probabilistic approach to frame induction and reports state-of-the-art end-to-end entity extraction on standard MUC and TAC data sets. It identifies evaluation of event clustering, alongside slots and entities, as future work.
Conclusion
- The proposed probabilistic frame-induction approach achieves state-of-the-art end-to-end entity extraction results on standard MUC and TAC data sets.The authors describe the model as easy to extend.Source: §6 · p. 8
Future work
- The authors identify evaluating event clustering, in addition to slots and entities, as future work.Source: §6 · p. 9