PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-lingual Annotation Projection for Semantic Roles

Sebastian Pado, Mirella Lapata

arXiv:1401.5694v1cs.CL

TL;DR

The paper addresses the shortage of FrameNet-style semantic role resources for new languages. It proposes projecting English annotations through constituent-level semantic alignments formulated as bipartite-graph optimization, and finds that constituent-based models—especially restrictive perfect matchings—support high-precision German projections.

  • Problem

    FrameNet-style role-semantic resources are scarce outside English, while creating them manually is expensive and existing word alignments are inadequate for long role spans.

  • Method

    The framework projects English annotations through lexical and syntactic information, modeling constituent alignments as constrained optimization over bipartite graphs.

  • Results

    Constituent-based models outperform word-based alternatives by a large margin, with perfect matchings performing most reliably and achieving the highest precision across conditions.

  • Takeaways & Limitations

    Parallel corpora can help relieve the lexical acquisition bottleneck and induce high-quality semantic role projections for new languages.

  • Takeaways & Limitations

    Perfect-matching models assume strong equivalence between the constituent structures of the two languages, an assumption not met by all observed alignments.

Abstract

from arXiv · show

This article considers the task of automatically inducing role-semantic annotations in the FrameNet paradigm for new languages. We propose a general framework that is based on annotation projection, phrased as a graph optimization problem. It is relatively inexpensive and has the potential to reduce the human effort involved in creating role-semantic resources. Within this framework, we present projection models that exploit lexical and syntactic information. We provide an experimental evaluation on an English-German parallel corpus which demonstrates the feasibility of inducing high-precision German semantic role annotation both for manually and automatically annotated English data.

1. Introduction

Semantic roles abstract over surface syntax and support broad NLP applications, but role-semantic resources remain scarce outside English. The paper proposes projecting English FrameNet annotations onto new languages using parallel corpora and graph-based semantic alignments.

  • Motivation: Semantic roles describe predicate–argument relations independently of surface syntactic configuration.The same role can appear in different syntactic positions across sentences.
  • Motivation: Semantic role labeling supports applications including information extraction, textual entailment, categorization, question answering, and machine translation.
  • Resource gap: English FrameNet is substantially larger than role-semantic resources for other languages, making annotation costly to reproduce cross-lingually.English FrameNet contains roughly 800 frames and around 150,000 annotated tokens, while other resources are often two to three times smaller.
  • Approach: The proposed approach uses parallel corpora and an existing English FrameNet to acquire frame elements and their syntactic realizations for new languages.It aims to reduce human effort through annotation projection.
  • Approach: The framework addresses noisy word alignments by constructing constituent-level semantic alignments and optimizing them in a bipartite graph.Constituent-based models outperform word-based alternatives by a large margin.

2. Annotation Projection and Semantic Correspondence

The paper evaluates whether English semantic annotations can transfer to German by measuring semantic correspondence and designing a carefully filtered annotation study. The results show substantial frame and role agreement, supporting projection when predicates evoke corresponding frames.

  • Projection and correspondence: Annotation projection transfers analyses from resource-rich languages through word alignments, but translational divergences prevent perfect cross-lingual correspondence.Divergences may be structural or semantic.
  • Projection and correspondence: The expanded article adds novel experiments and a comprehensive error analysis beyond earlier versions of the work.
  • Sample selection: The study uses professionally translated English–German Europarl sentences, with about 25 million words on each side.The corpus is restricted in genre but covers a wide range of topics.
  • Sample selection: The final sample filters predicate pairs using GIZA++ alignments, FrameNet and SALSA coverage, and intersecting frame lists.The resulting corpus contains 83 frame types and 696 lemma pairs.
  • Annotation: The annotation protocol assigns one frame per sentence and roles to constituents after syntactic correction, using training, calibration, and production phases.The production dataset contains 1,000 bi-sentences.
  • Annotation: Annotators show high agreement across frames, roles, and role spans, with Span Match reaching 0.84.The study uses Frame Match, Role Match, and Span Match rather than Kappa because spans and categories vary by predicate.
  • Evaluation: About 72% of English–German predicate pairs evoke the same frame, while role agreement reaches an F1-Score of 91%.Frame divergence often reflects one language using a single predicate and the other using complex predication; role mismatches often involve passivization or infinitival constructions.

3. Modeling Semantic Role Projection with Semantic Alignments

Word alignments are inadequate for projecting semantic roles over long spans because they can be noisy or incomplete. The proposed model instead aligns syntactic constituents, enabling more linguistically meaningful target spans.

  • Problem: Semantic roles may span arbitrary-length constituents, so word-level projection can produce incorrect role spans.
  • Problem: Noisy or incomplete word alignments can project a role onto only part of the correct target phrase.The example incorrectly aligns the German Message role to a partial span.
  • Constituent-based projection: Constituency information biases projection toward linguistically meaningful spans and can recover a target constituent when only part of its yield is word-aligned.

3.1 Framework Formalization

The framework models semantic annotation projection as an optimization problem over bilingual linguistic units, using similarities and structural constraints to select alignments and transfer roles.

  • Projection maps source role labels onto the union of target units aligned with source units bearing each role.A semantic alignment links source and target units treated as semantically equivalent.
  • The framework seeks an alignment maximizing the product of bilingual similarities between source and target units.Similarity is assumed to range from 0 for minimal similarity to 1 for maximal similarity.
  • Constituent alignment is formulated as bipartite graph optimization, enabling structural constraints that discourage linguistically implausible mappings.The graph partitions source and target linguistic units, with edges representing possible links and weights representing alignment costs.
  • Finding an optimal alignment is equivalent to identifying a minimum-weight subgraph satisfying specified structural constraints.The edge costs correspond to link similarities through the optimization formulation.
  • Figure 2 represents a bi-sentence as an edge-weight matrix in which high similarities produce low edge weights and zero similarities produce effectively infinite weights.The example uses a complete bipartite graph over sentential constituents; high-similarity alignments appear along the matrix diagonal.
  • The framework supports word- and constituent-based models with corresponding similarity functions and alignment constraints.Complete projection models require choices of linguistic units, similarity functions, and admissible alignment constraints.

3.2 Word-based Projection

The word-based model treats individual word tokens as alignment units and uses existing word alignments as semantic alignments, enabling projection without external corpora but exposing errors from noisy alignments.

  • The first projection model represents source and target sentences as sets of word tokens and links individual words.Its binary similarity function treats word-aligned pairs as semantically aligned.
  • The word-based model uses a binary similarity function that assigns 1 to word-aligned pairs and 0 otherwise.This function turns an existing word alignment into an optimal semantic alignment.
  • Word-alignment constraints may be imposed heuristically or arise from the translation model used to compute the alignments.IBM models permit one-to-many alignments in one direction, and the experiments use alignments induced by GIZA++.
  • The word-based labeling function transfers a role to target words that are GIZA++-aligned with source words assigned that role.This corresponds to implicit labeling functions in other word-based annotation projection models.
  • Automatically induced word alignments are often noisy, causing projection errors in function words and multi-word expressions.These cases are systematically misaligned in the word-based approach.

3.3 Constituent-based Projection

The constituent-based projection model represents bilingual sentences as constituent sets and seeks semantic alignments through graph optimization. It compares alignment constraints ranging from bijective matching to highly permissive source-to-target links.

  • Graph formulation: The model represents admissible semantic alignments as weighted bipartite graphs whose nodes are source and target constituents.Edges encode cross-lingual constituent similarity, while constraints regulate the number of links incident to each constituent.
  • Constituent representation: Constituent similarity estimates shared semantic content using bidirectional word-alignment overlap between source and target constituents.The model uses Jaccard-based overlap in both directions and averages the two scores.
  • Empirical correspondence patterns: Gold alignments contain 67% one-to-one constituent correspondences, 32% one-to-many or many-to-one correspondences, and 1% untranslated constituents.These proportions motivate comparing alignment families with different structural assumptions.
  • Alignment constraints: Perfect matchings enforce bijective correspondences, edge covers allow one-to-many links in either direction, and total alignments impose only source-side coverage.Perfect matchings are restrictive; edge covers occupy an intermediate position; total alignments are the most permissive class.
  • Perfect matchings: Perfect matchings cannot represent one-to-many constituent matches, although their edge competition can correct some word-alignment errors.They are computationally efficient and equivalent to the linear assignment problem, with cubic-time solution methods.
  • Total alignments: Total alignments can leave target constituents unaligned and are computed by independently linking each source constituent to its most similar target constituent.This local optimization is globally optimal and has quadratic complexity, O(|Us||Ut|).

3.4 Noise Reduction

The framework reduces projection noise with filters that remove unreliable words or unlikely argument candidates. These filters target different error sources and trade recall against precision.

  • Motivation: The constituent-based model is expected to compensate for alignment errors through syntactic information, whereas the word-based model relies solely on automatic word alignments.Consequently, the word-based model is expected to be more error-prone.
  • Word Filtering: Word filtering removes non-content words or words left unaligned by automatic word alignment.The filters aim to distinguish genuine alignments from noisy ones and reduce computation.
  • Argument Filtering: Argument filtering retains constituents likely to serve as predicate arguments, excluding nodes that dominate the predicate or cross a sentence boundary.The heuristic can cover long-distance dependencies and coordination.
  • Argument Filtering: The argument filter reduces a full target parse tree to likely argument positions, such as the example nodes NP Kim and S pünktlich zu kommen.This filter applies only to constituent-based models defined over full parse trees.

3.5 Discussion

The discussion frames projection as weighted bipartite-graph optimization over constituent similarities and alignment constraints. The models trade structural flexibility against robustness to alignment errors and computational cost.

  • Framework: The framework includes word-based and constituent-based projection models, with constituent-based models optimizing semantic alignments as weighted bipartite subgraphs.Nodes represent bilingual syntactic constituents and weighted edges represent their cross-lingual similarity.
  • Alignment families: Perfect matching enforces bijective English-German constituent correspondences, edge cover requires coverage without one-to-one enforcement, and total alignment leaves target constituents optional.These constraints define different admissible subgraph structures.
  • Trade-offs: Total alignments model structural changes more appropriately but are more sensitive to alignment errors than restrictive perfect matchings.Perfect matching is described as linguistically implausible because it assumes no structural divergence.
  • Noise reduction and comparison: Filtering techniques reduce the impact of alignment errors, while the model families differ in linguistic units, similarity measures, correspondence assumptions, graph structure, and computational complexity.These dimensions are summarized in Table 4 and evaluated empirically.

4. Experiments

The experiments evaluate English-to-German role projection under gold and automatically obtained resources, comparing constituent-based models with a word-based baseline. Performance is measured against German gold annotations using exact labeled span matches.

  • Experimental design: Two experiments project semantic roles from English sentences onto German translations and evaluate them against German gold-standard role annotations.The first uses gold syntactic and semantic annotations to isolate projection potential from other error sources.
  • Data: The parallel corpus is randomly divided into development and test sets, each containing 50% of the data.Table 5 reports token, sentence, frame, and argument statistics for English and German.
  • Alignment resources: Automatic word alignments use GIZA++ trained on the full English-German Europarl bitext and symmetrized with the intersection heuristic.Manual word alignments were also produced for the sentences used in evaluation.
  • Model evaluation: Sixteen model variants were evaluated on development data, and the best-performing models were subsequently validated on the test set.Practical runtime was dominated by XML input/output rather than graph optimization.
  • Baseline: The word-based model serves as a simple baseline because it uses only word alignments, requires no syntactic analysis, and has linear time complexity.Constituent-based models are compared against this baseline in both experiments.
  • Evaluation metrics: Exact Match counts a prediction as correct only when both the semantic-role label and projected span match the German gold annotation.Performance is reported with labeled Precision, Recall, and F1, with statistical significance assessed by stratified shuffling.
  • Upper bound: Inter-annotator agreement was 0.84 for Span Match, providing a reasonable within-language upper bound for automatic semantic role labeling.The projection task has an additional ceiling imposed by bilingual divergence.

4.2 Experiment 1: Projection on Gold Standard Data

On gold-standard English roles and syntax, constituent-based projection substantially outperformed word-based projection, with filtering improving several constituent models. The strongest systems achieved high precision but recall remained limited, especially for short roles and alignment-dependent cases.

  • Development-set performance: 48.9% F1 was obtained by the word-based WordBL model without filtering, while constituent-based models increased F1 by approximately 20%.All differences between WordBL and constituent-based models were significant at p < 0.01.
  • Filtering effects: 9.8% F1, 6.9% F1, and 9.2% F1 were gained by PerfMatch, EdgeCover, and Total, respectively, under the NA Filter.PerfMatch and Total were the best-performing models in this condition.
  • Filtering effects: Filtering combinations generally failed to improve results further because removing many words also removed good alignments.The paper reports this effect for combinations of non-aligned, content-word, and non-argument filters.
  • Model trade-offs: 88.8% precision and 56.2% recall characterized PerfMatch with its strict one-to-one correspondence constraint.PerfMatch abstains when no direct correspondence is identified, trading recall for precision.
  • Limitations: Around 70% recall was achieved at best, below the 83% upper bound because short roles such as pronouns often lacked intersective alignment links.Projection cannot proceed when the relevant word alignment contains no links.
  • Test-set evaluation: 1%–3% F1 was gained by constituent-based models with manually corrected alignments, and EdgeCover approached the human upper bound under noise-free conditions.Cleaner alignments mainly improved recall, including approximately 5% for EdgeCover.

4.3 Experiment 2: Projection with Automatic Roles

With automatically produced syntactic and semantic analyses, constituent-based projection remained stronger than the word-based baseline but suffered substantial performance losses. Parsing errors, alignment noise, and incomplete role coverage constrained recall and overall F1.

  • Setup: The experiment used automatic syntactic parses and semantic roles assigned by a FrameNet-trained shallow semantic parser.The parser identified and labeled frame elements while treating frames as given.
  • Results: 55–56% F1 was obtained by both PerfMatch with NA Filter and EdgeCover with Arg Filter under automatic analysis.These results were approximately 25 F1 points below manual-annotation results.
  • Results: 41.6% F1 was obtained by WordBL, while constituent-based models still outperformed it by more than 10% F1.The difference was significant at p < 0.01.
  • Results: Automatic annotations reduced precision by 15% F1 and recall by 30%, including about 5% nominal roles outside the shallow parser’s scope.The models retained overall high precision but low recall.
  • Error analysis: Parsing errors were a major source of problems, particularly for German verb phrases affected by free word order and morphological ambiguity.The parser produced an implausible analysis for an example with ambiguous finite and infinitival forms.
  • Error analysis: PerfMatch generally sacrificed recall when a source constituent lacked a matching German constituent, whereas EdgeCover sacrificed precision by producing a wrong alignment.The example arose because the incorrect parse prevented the English complement from matching a single German constituent or constituent combination.

5. Related Work

Related work covers projection of short linguistic annotations, resource-based induction of frame-semantic information, and constituent alignment for translation. The paper positions its approach as graph-based optimization over constituent semantic alignments and notes applications beyond role projection.

  • Annotation projection: Earlier projection work primarily addressed POS tags, NP chunks, dependencies, and word senses rather than long semantic-role spans.These annotations generally span shorter linguistic units than frame elements.
  • Resource-based induction: Fung and Chen’s Chinese approach maps FrameNet entries to HowNet concepts and labels arguments in monolingual sentences, but depends on resources likely unavailable for many languages.Their method does not require a parallel corpus.
  • Constituent alignment: This paper creates semantic alignments between constituents and casts the search for the best alignment as a bipartite-graph optimization problem.The framework uses parallel-corpus information about semantic correspondences.
  • Constituent alignment: Many constituent-alignment methods require perfect word alignment or use greedy search without guaranteed optimality.The paper contrasts these limitations with its optimization-based framework.
  • Broader applications: The proposed models may also support syntactic reordering in statistical machine translation, although the paper evaluates them only for semantic-role projection.The authors specifically connect constituent alignment to translation systems using syntactic information.

6. Conclusions

The paper presents annotation projection as a framework for inducing FrameNet annotations in new languages from English resources and parallel corpora. Experiments emphasize constituent-based alignment, noise filtering, and scope boundaries for the approach.

  • Annotation projection uses parallel corpora and constituent information to obtain FrameNet annotations automatically for new languages.The framework is formulated as a general projection approach that exploits English resources and constituent alignment.
  • Constituent information substantially improves projections over word alignments, with perfect matchings performing most reliably and achieving the highest precision across conditions.Word-based models remain a starting point for low-density languages without parsers, but are too noisy and fragmentary for long-span semantic roles.
  • Removing non-aligned words or non-arguments produces the best noise-reduction results, with the former balancing precision and recall and the latter favoring high precision over recall.The choice depends on projection goals; aggressive non-argument filtering is presented as desirable when training target-language shallow semantic parsers requires high-quality projections.
  • The framework may extend beyond FrameNet to PropBank roles, discourse structure, named entities, machine translation, and constituent reordering.These are proposed applications rather than evaluated extensions in this article.
  • The projection models use a simple word-alignment-based semantic similarity measure, while bilingual dictionary or vector-model enhancements remain possible.The paper identifies more sophisticated lexical similarity modeling as an unexplored improvement.
  • The experiments assume preserved frames in parallel sentences, avoiding translational-divergence modeling during direct role transfer.A broader framework could introduce mappings during projection, analogous to transfer rules in machine translation.
Loading 1401.5694v1…