Source-linked AI summary
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning
Karthik Narasimhan, Adam Yala, Regina Barzilay
TL;DR
The paper asks how information extraction can improve when annotated training data is scarce and difficult values are expressed ambiguously. It uses reinforcement learning with a deep Q-network to retrieve external articles, extract values, reconcile them, and stop when evidence is sufficient. Across shooting and food-adulteration domains, the system outperforms traditional extractors and a meta-classifier baseline.
Problem
The paper targets information extraction in domains with limited annotated data, where varied linguistic expressions can make entity values difficult to recover.
Method
A reinforcement-learning framework uses a deep Q-network to select queries and reconciliation actions while incorporating extracted evidence from additional sources.
Results
The model outperforms traditional extractors by 7.2% and 5% on average across two domains and gains up to 7% over a meta-classifier.
Takeaways & Limitations
Sequentially acquiring and consolidating external evidence improves extraction accuracy when training data is limited.
Takeaways & Limitations
The reported experiments found no significant differences from adding linear layers or changing the Sigmoid/ReLU nonlinearity.
Abstract
from arXiv · showhide
Most successful information extraction systems operate with access to a large collection of documents. In this work, we explore the task of acquiring and incorporating external evidence to improve extraction accuracy in domains where the amount of training data is scarce. This process entails issuing search queries, extraction from new sources and reconciliation of extracted values, which are repeated until sufficient evidence is collected. We approach the problem using a reinforcement learning framework where our model learns to select optimal actions based on contextual information. We employ a deep Q-network, trained to optimize a reward function that reflects extraction accuracy while penalizing extra effort. Our experiments on two databases -- of shooting incidents, and food adulteration cases -- demonstrate that our system significantly outperforms traditional extractors and a competitive meta-classifier baseline.
1 Introduction
The paper addresses information extraction when annotated data is scarce by acquiring external articles that express difficult values more explicitly. It formulates retrieval, extraction, and reconciliation as a sequential reinforcement-learning problem and reports gains over extractors and a meta-classifier.
- Motivation: Scarce annotated data may not cover the varied linguistic expressions needed to identify entities such as perpetrators and victim counts.A shooting article can refer to a shooter indirectly and express four victims as “A couple and four children.”
- Motivation: External articles about the same event can state sought entities in forms that basic extractors handle more easily.The strategy uses information redundancy across news coverage to resolve ambiguities in the source article.
- Problem: The central challenge is sequentially retrieving relevant event-related articles and reconciling their extracted values as evidence accumulates.Search can return tangential incidents, while one retrieval-and-reconciliation pass may leave confidence low.
- Approach: The proposed reinforcement-learning approach jointly selects query and reconciliation actions using contextual information and a reward tied to accuracy and extra effort.The DQN predicts querying and reconciliation choices, while the framework can use other base extractors.
- Results: 7% average accuracy improvement over the meta-classifier is reported in the Shootings domain, while the final RL model outperforms basic extractors and the baseline in both domains.The evaluation uses mass-shooting and food-adulteration databases with limited available training data.
2 Related Work
The paper distinguishes its task from related web-based extraction, multi-document extraction, knowledge-base completion, and search-based information extraction. Its distinctive focus is finding and reconciling additional sources for a specific event.
- Open Information Extraction: Unlike open information extraction, this work extracts information from additional sources about a specific event rather than collecting relation instances independently of context.The resulting novel challenge is event coreference across retrieved articles.
- Entity Linking, Multi-Document Extraction and Event Coreference: Multi-document extraction links mentions across documents that are already provided, whereas this task must search for additional sources and decide their relevance.The comparison separates cross-document linking from source acquisition.
- Knowledge Base Completion and Online Search: Traditional knowledge-base completion increases recall without requiring the system to search for additional information sources.The paper therefore separates database completion from web acquisition and evidence reconciliation.
- Knowledge Base Completion and Online Search: Prior search-and-extraction systems address document retrieval or query reformulation, while this approach learns to query and consolidate sources rather than using predefined rules.The related faculty-directory system uses reinforcement learning to reduce computational actions, not to optimize extraction accuracy.
3 Framework
The framework models external-evidence extraction as an MDP in which an agent repeatedly reconciles values from retrieved articles and chooses further queries or termination. States encode confidence, value matches, contextual words, and document similarity, while rewards favor accuracy with fewer steps.
- States: The MDP state combines current and new entity values with document similarity and contextual information, enabling dynamic reconciliation and retrieval decisions.Its features include confidence scores, one-hot value matches, unigram/tf-idf context counts, and source-document similarity.
- Actions: Each action jointly chooses a reconciliation decision and a query, allowing the agent to accept selected values, accept all, reject all, or stop.The episode ends when the stop decision is selected.
- Rewards: The reward favors improvements in final extraction accuracy while penalizing longer episodes and additional queries.The accuracy component compares the current and previous entity-value sets, and each step receives a negative reward.
- Queries: Query templates combine an article title with context words associated with entity types, and search results older than one month are filtered.The templates target entities such as ShooterName, NumKilled, NumWounded, and City.
- Transitions: An episode begins with entity extraction from one source article, then repeatedly retrieves articles, extracts new values, forms states, and applies the chosen transition until stopping.The transition incorporates the reconciliation decision and values from the next queried article.
4 Reinforcement Learning for Information Extraction
The paper frames external-evidence acquisition as a sequential reinforcement-learning problem, using a DQN to select queries and reconciliation actions that balance extraction accuracy against effort.
- 4 Reinforcement Learning for Information Extraction: Q-learning updates action values from rewards using the Bellman equation, with γ discounting future rewards.The reward is defined over state-action transitions, and the expectation covers possible transitions.
- 4 Reinforcement Learning for Information Extraction: The agent uses a DQN to approximate action values in a continuous state space and jointly predict query and reconciliation decisions.The network uses separate output layers for the two decision types.
- 4 Reinforcement Learning for Information Extraction: Training follows an ϵ-greedy DQN procedure that explores random actions, stores transitions in replay memory, and updates parameters from sampled mini-batches.A periodically updated target network provides stable Q-value targets during learning.
5 Experimental Setup
The experiments use limited-data shooting and food-adulteration datasets, a maximum-entropy base extractor, external search results, and baselines that separate aggregation from sequential decision-making.
- 5 Experimental Setup: The study evaluates extraction on shooting and food-adulteration datasets whose recorded incidents are insufficient for large-scale information extraction systems.The datasets annotate shooting entities such as shooter, casualties, and city, and adulteration entities such as product, adulterant, and location.
- 5 Experimental Setup: For each source article, the system downloads the top 20 Bing results using automatically generated queries and trains models on source and downloaded articles.Final evaluation uses the combined train and development data and the complete test set with downloaded articles.
- 5 Experimental Setup: The base system is a maximum-entropy word tagger that assigns entity types and aggregates extracted values by mode.A CRF with the same features was also tested but performed worse empirically.
- 5 Experimental Setup: Evaluation reports corpus-level average accuracy by entity type, using exact matches except that either annotated name can count for ShooterName.Final accuracies average three independent runs, with each run averaged over 20 epochs after 100 training epochs.
- 5 Experimental Setup: Baselines include basic extractors, confidence and majority aggregation, a meta-classifier, and an oracle separating reconciliation and querying effects.The RL variants separately test reconciliation, querying, and the full combination of both decisions.
6 Results
RL-Extract improves extraction accuracy over basic extractors, aggregation baselines, and a meta-classifier across both domains, while joint querying and reconciliation and selected design choices further improve performance.
- RL-Extract improves accuracy over basic extractors on every entity type across both domains, with gains up to 11.5% on Adulteration Location.It is 11.4% better than Maxent on Shootings City and 7.1% better on Shootings NumKilled.
- RL-Extract outperforms Confidence and Majority aggregation by 7.2% on Shootings and 5% on Adulteration averaged over all entities.
- RL-Extract exceeds the meta-classifier by 7.0% on Shootings over all entities, supporting sequential decision-making over aggregating all retrieved documents.The meta-classifier includes noisy and irrelevant documents from the entire retrieved set.
- The full RL-Extract model outperforms RL-Basic and RL-Query on both domains, indicating gains from jointly selecting queries and reconciling values.
- Reward and entity accuracy increase together across training epochs, and successful cases often use articles with prototypical or readily extractable language.Table 4 presents examples where RL-Extract succeeds while Maxent fails.
- Analysis: Replace reconciliation outperforms alternative schemes by 2–6% on all entities, while tf-idf context and step-based rewards also improve performance.Step-based rewards are over 10% better on average than a single delayed reward, and the best system uses 9.4 steps per episode.
7 Conclusions
The paper studies external-evidence acquisition for information extraction with limited training data, using reinforcement learning to sequence querying, extraction, and reconciliation. Its deep Q-network outperforms traditional extractors and gains up to 7% over a comparable meta-classifier.
- The paper addresses improving information extraction by acquiring and incorporating external evidence when training data is limited.
- A deep Q-network learns action sequences for querying, extracting from new sources, and reconciling values while maximizing accuracy and penalizing extra effort.
- The model improves average accuracy by 7.2% and 5% across two domains and gains up to 7% over a meta-classifier.
Reinforcement Learning: Supplementary Material
The supplied supplementary-material passage provides only an arXiv date and no substantive reinforcement-learning content.
- The supplementary-material passage contains only the document identifier and date, arXiv:1603.07954v3, dated 27 Sep 2016.
1 Framework and Model
The supplied framework-and-model passages identify the paper’s DQN architecture and Maximum Entropy classifier features, but provide no further model description.
- The framework-and-model material identifies Figure 1 as the DQN architecture and describes unigram features for the Maximum Entropy classifier.
- The listed classifier features include capitalization, word length, digits, number words, ordinal words, and city-name indicators.