Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language

1 Introduction

• Urdu’s rich morphology and complex case, predicate, and word-order properties pose NLP challenges, and it remains under-resourced despite resource-development efforts.Phrase structure represents hierarchical organization but lacks grammatical relations, whereas dependency structure captures predicate–argument relations.Source: §1 · pp. 1–2
• The paper converts CLE-UTB phrase structure to dependency structure using a language-specific head-word model, phrase-to-dependency label mapping, and post-conversion rules.It proposes learning constituency and dependency structures together through cross-structure representations.Source: §1 · p. 2
• A shared BLSTM sequence-labeling framework represents both syntactic parsing tasks, while single-task experiments first evaluate them independently.The framework incorporates character embeddings, convolutions, POS tags, and pretrained word representations.Source: §1 · p. 2
• Single-task CLE-UTB parsing reached constituency F-score 90.94 and dependency LAS 85.36 using POS tags and contextualized word representations.The constituency model used third-layer ELMo representations; the dependency model used Word2Vec and ELMo embeddings.Source: §1 · p. 3
• F-score 91.39 and labeled attachment score 85.69 were achieved by multi-task parsing, improving CLE-UTB constituency results by 3.29 points and dependency results by 1.49 points.The multi-task constituency parser also improved function-labeling accuracy by 3.53 points.Source: §1 · p. 3

2 Related Work

• The two Urdu treebanks differ in structure, annotation scheme, depth, genre coverage, and size, motivating analysis of their suitability for multi-task parsing.The cited sample from Urdu.KON-TB illustrates a multi-clause sentence with case phrases and a complex verbal construction.Source: §2 · p. 3
• Urdu.KON-TB has 1,400 sentences, 26 phrase labels, and deeper linguistic annotation, but prior conversion produced incorrect dependency structures and judged it unsuitable for dependency parsing.Its labels encode morphological information, and its complex hierarchy includes verbal categories and duplicate question labels.Source: §2 · pp. 3–4
• CLE-UTB’s flat structure accommodates flexible word order and marks grammatical relations, including genitive and copula constructions and complex predicates.Arguments and predicates can be annotated without restricting their positions in the parse tree.Source: §2 · pp. 4–5
• The CLE-UTB has 7,854 sentences across fifteen genres and a flat annotation compatible with dependency structure; this paper converts it and performs multi-task parsing.Its annotation includes functional labels and supports flexible word order; some Urdu-specific POS and phrase labels extend Penn Treebank-compatible labels.Source: §2 · p. 5
• Urdu resources and methods have also advanced in semantic parsing, machine translation, POS tagging, text classification, and dynamic-label multi-label classification.Reported work includes UrduMASD summarization, the UQA question-answering corpus, conversational generation, and clickbait detection.Source: §2 · pp. 5–6
• Urdu NLP work also covers UrduLLaMA, speech recognition and synthesis datasets and models, and the SAATHI conversational assistant.The speech-recognition discussion notes persistent accent-variability and text-normalization challenges.Source: §2 · pp. 6–7

3 PS to DS Conversion

• The first stage uses a head-word model to generate an intermediate dependency representation containing dependency arcs, phrase labels, and POS tags.This intermediate representation is the input to phrase-to-dependency label mapping.Source: §3 · p. 7
• The second stage maps phrase labels to dependency labels to transform the intermediate representation into dependency structure.This is the label-conversion step in the three-stage process.Source: §3 · p. 7
• The final stage applies post-conversion rules to produce the dependency treebank.The section also evaluates the resulting conversion.Source: §3 · p. 7

3.1 Urdu Head-word model

• The proposed CLE-UTB head-word model refines an existing algorithm using language-dependent rules to select the dominant lexical item in each phrase.The model identifies head-words from clauses; Table 1 gives label-wise details.Source: §3.1 · p. 7, Table 1 · p. 7
• Verbal constructions are searched left to right, prioritizing main verbs over auxiliaries so the main verb becomes the phrase head.The model’s directions refer to the left-to-right internal Unicode sequence, despite Urdu being written right to left.Source: §3.1 · pp. 7–8
• Post-positional phrases are headed by the noun phrase that precedes the case marker, with head-finding proceeding left to right.The NP has the highest priority in the phrase’s head-finding list.Source: §3.1 · p. 8
• Noun phrases are headed from right to left, including phrases with genitive case and internal noun phrases.Adjective and quantifier phrases also place their head modifier on the right.Source: §3.1 · p. 8
• Adverbial phrases generally contain a single adverb, while an adverbial phrase containing an infinitive clause remains right-headed.The account also notes that Urdu prepositional phrases are infrequent in corpora.Source: §3.1 · p. 9
• Demonstrative phrases are searched right to left; foreign fragments are treated as left-headed, and clauses take the main verb as head.Hierarchical and parental annotations provide context for clause head-finding and phrase-to-dependency mappings.Source: §3.1 · p. 9

3.2 PS to DS Label Mappings

• Phrase labels and POS tags both contribute dependency labels: phrase labels encode head relations, while POS tags encode non-head relations.The refined POS inventory expands from 35 to 44 tags, and dependency labels use Universal Dependencies V-2.0; the mapping also analyzes HUTB.Source: §3.2 · p. 10
• Parental clause labels and verbal-construction levels distinguish attachment contexts and help identify sentence roots for accurate mapping.For example, clauses receive labels such as S^NP or S^PP, while VC^1 marks a first-level verbal construction.Source: §3.2 · pp. 10–11

3.2.1 Core Arguments

• Core-argument mappings assign dependency labels to subjects, objects, subordinate clauses, nominal obliques, and infinitive clausal objects.Table 2 maps CLE-UTB phrase labels to UD labels; nominal obliques map to iobj, while infinitive clausal objects map to xcomp.Source: Table 2 · p. 11, §3.2.1 · p. 11

3.2.2 Non-Core Dependents

• Non-core dependents map OBL to obl, adverbial phrases and adjuncts to advmod, and predicate-link constructions to cop.The mapping also assigns advel to non-core finite clauses and conjunctive participles, and mark to subordinate clitics.Source: §3.2.2 · pp. 11–12, Table 3 · p. 12

3.2.3 Nominal Dependents

• Nominal mappings assign genitive PP-G phrases to nmod, cardinals to nummod, adjectival modifiers to amod, and case markers to case.Determiners include demonstrative phrases, particles, fractions, ordinals, quantifiers, multiplicatives, and several pronoun types.Source: §3.2.3 · pp. 12–13, Table 4 · p. 13

3.2.4 Other Dependency Relations

• Other mappings cover coordination, multi-word expressions, punctuation, roots, and unspecified dependencies; multiword compound nouns use the rightmost word as head and flat relations for other words.The table’s scope is these other dependency relations; conjunctions map to conj or cc, and punctuation maps to punct.Source: §3.2.4 · pp. 13–14, Table 5 · p. 14

3.3 Post-Conversion Rules

• Secondary objects and recipients annotated as PP-OBL map to iobj when followed by the clitic kO; specified nominal obliques also map to iobj.The second rule identifies recipient or beneficiary pronouns listed in Table 6.Source: §3.3 · p. 15, Table 6 · p. 15
• Infinitive-headed S or S-OBJ constructions map to xcomp, while internal clitic constructions map to fixed and conjunction-linked phrases to conj.Conjunctions are identified using CC or comma with the PU-C POS tag.Source: §3.3 · p. 15
• Additional rules align case markers after infinitive verbs with HUTB’s mark label and adjust spacio-temporal noun annotations using an NST noun list.The rules also specify how adjacent case markers attach and receive case labels.Source: §3.3 · pp. 15–16
• After label mapping and post-conversion rules, the final tree marks the predicate link as root and the copula hE as its cop dependent.The generated dependency treebank uses 27 unique dependency labels.Source: §3.3 · p. 16, Figure 4 · p. 16, Table 7 · p. 16

3.4 Conversion Evaluation

• The agreement evaluation used LAS, UAS, label accuracy, and Cohen’s Kappa on a 100-sentence manually annotated reference corpus.The reference sentences came from the Urdu development set of the Hindi-Urdu treebank project and were converted to dependency structure for comparison.Source: §3.4 · p. 17, Table 8 · p. 17
• Head and label agreement was almost 0.80 (strong), while labeled attachment agreement was near 0.70 (moderate).The reference corpus comprised 100 manually annotated sentences, and the authors note that this method may not completely evaluate conversion.Source: §3.4 · p. 17, Table 8 · p. 17, Table 9 · p. 17
• The generated dependency labels cover major core and non-core arguments but are limited in representing all dependency relations.The comparison treebank was annotated manually, whereas this dependency treebank was generated from a phrase-structure treebank using functional labels.Source: §3.4 · p. 17

4 Sequential Representations

• Both phrase-structure and dependency trees can be transformed into sequence labels and accurately reconstructed as hierarchical structures.The shared sequence-labeling representation aligns parsing with a common modeling approach.Source: §4 · p. 18

4.1 PS Tree Representation

• The proposed phrase labels encode common ancestors and the lowest common phrase; unary branches are handled separately, and entropy measures label randomness.The scheme builds on absolute labeling, while the relative alternative predicts unary labels and relative phrase labels in two passes.Source: §4.1 · pp. 18–19
• The proposed phrase-structure labeling outperformed relative labeling on CLE-UTB parsing scores and had lower average entropy.The authors describe the CLE-UTB as relatively flat and propose a labeling scheme intended to generate fewer labels.Source: §4.1 · p. 19
• The proposed labeling reduced the tag set to 143 labels, compared with 180 labels under relative labeling; integers were removed from VC and DMP labels and recovered after parser output.NP label patterns decreased parsing scores, so the researchers used the four aforementioned patterns.Source: §4.1 · p. 20

4.2 DS Tree Representation

• Dependency parsing uses a three-part relative POS-based encoding: head position relative to the head’s POS tag, the head POS tag, and the dependency label.A plus or minus sign indicates whether the head lies to the right or left; the encoding was used in both single-task and multi-task experiments.Source: §4.2 · pp. 20–21

5 Multi-Task Parsing

• The framework compares independent constituency and dependency parsing with multi-task learning that incorporates cross-representations to improve both parsing results.Both models are based on LSTM networks.Source: §5 · p. 21

5.1 Model

• Bidirectional LSTMs model the linearized parse labels and POS tags using forward and backward context.The described architecture has one LSTM layer in each direction.Source: §5.1 · p. 21

5.1.1 Single-Task Learning Model

• The single-task parser trains constituency and dependency parsing independently using token, POS, and sequence-label inputs with concatenated vector features.The experiments include character embeddings, POS tags, and pretrained word representations.Source: §5.1.1 · pp. 21–22

5.1.2 Multi-Task Learning Model

• The multi-task model feeds learned phrase-structure encodings to dependency parsing and dependency encodings to phrase-structure parsing.Separate softmax layers predict each structure’s labels, with the cross-task encodings and contextual representations passed through a shared BLSTM layer.Source: §5.1.2 · p. 22

5.2 Transfer Learning

• The authors trained context-free and contextualized word representations on a 220-million-token Urdu corpus collected from the web across multiple genres.The corpus includes news, business, culture, health, and sports text.Source: §5.2 · pp. 22–23

5.2.1 Context-Free Word Representations

• The context-free Word2Vec embeddings use skip-gram training, a 125,000-token vocabulary, 100 dimensions, and a window size of five.Word2Vec embeddings are one of the transfer-learning representations used in the parsing experiments.Source: §5.2.1 · p. 23

5.2.2 Contextualized Word Representations

• ELMo vectors represent each token using context and are computed from a two-layer bidirectional language model with character-level CNN input.The final representation is a weighted sum of raw vectors and vectors from the two intermediate layers; the parsing experiments use the last layer.Source: §5.2.2 · p. 24

6.1 Dataset

• The CLE-UTB test set contains 973 sentences and 18,470 tokens; conversion also inflates dependency obl counts through mappings for ADJ annotations and spacio-temporal nouns.The converted treebank has over 11,000 obl occurrences, compared with 2,321 oblique constructions in the constituency treebank.Source: §6 · p. 25, §6.1 · p. 26

6.2 Evaluation Metrics

• Constituency evaluation uses labeled recall, precision, and F-score, while dependency evaluation reports UAS, LAS, and label accuracy.LAS counts tokens with both correct heads and labels; label accuracy ignores head correctness.Source: §6.2 · p. 27

6.3 Results

• Single-task models establish strong baselines: the CNN-BLSTM reaches 90.94 F-score without functional labels, and the BLSTM dependency parser reaches 85.36 LAS.POS tags and ELMo representations are part of the reported high-performing configurations.Source: §6.3 · pp. 28, 31
• Parsing is less effective on rare constructions: csubj has ten evaluation samples, fixed has none, and the authors attribute weak rare-label predictions to data sparsity.Constituency F-scores gradually decline as sentence length increases, whereas dependency performance is described as more uniform across lengths.Source: §6.3 · pp. 30–32, Figure 15 · p. 33
• 91.39 F-score for constituency parsing is the best multi-task result without functional labels, up from 90.94; dependency parsing reaches 85.41 LAS in that setting.The single-task dependency parser reaches 85.36 LAS, while the multi-task model with functional labels reaches 85.47 LAS.Source: §6.3 · pp. 31, 33, Table 19 · p. 34, Table 20 · p. 34
• Cross-structure encodings improve frequent constituency and dependency labels, while multi-task learning reduces functional-label errors by 12.4%.The nsubj–obj dependency confusion falls from 82 to 37, and disagreement counts among the top 36 dependency labels decline from 901 to 861.Source: §6.3 · pp. 34, 36, Table 21 · p. 35, Table 22 · p. 35

• Multi-task learning further enhanced parsing results by learning cross-structure representations, which the authors report are useful for improving Urdu syntactic parsing.Source: §7 · p. 37
• The work converts the CLE-UTB phrase-structure treebank to dependency structure using a language-dependent head-word model, label mapping, and post-conversion rules.Source: §7 · p. 37
• Single-task learning with contextualized word representations produced state-of-the-art constituency and dependency parsing results for CLE-UTB; character-based representations can learn Urdu morphology and syntax.Source: §7 · p. 37