Source-linked AI summary
POSPAN: Position-Constrained Span Masking for Language Model Pre-training
Zhenyu Zhang, Lei Shen, Yuming Zhao, Meng Chen, Xiaodong He
TL;DR
Existing span-level masking methods focus on span length and assume masked spans are uniformly positioned, despite dependencies among spans. POSPAN combines span-length and position-constraint distributions; experiments report consistent gains over span-length-only counterparts with suitable constraints.
Problem
Existing span-level masking methods focus on span length and assume masked-span positions are uniform, despite dependencies among phrases and entities.
Source: §1 · p. 1Method
POSPAN combines span-length and position-constraint distributions to control span length and inter-span distance, unifying existing span-level masking methods.
Source: §1 · pp. 1–2Results
POSPAN consistently outperforms previous span-length-only counterparts when combined with suitable position constraints, and achieves the best performance across different tasks.
Source: §1 · p. 2, §3.2 · p. 4Takeaways & Limitations
The analysis identifies dependent span positions as relevant to contextual modeling, with position constraints critical in some cases and improper settings able to harm NLU performance.
Source: §2.2.1 · p. 3, §3.3 · p. 4The analysis assumes that, given R_ij, unmasked tokens in I_ij are independent of tokens in spans S_i and S_j.
Source: §2.2.2 · p. 3
Abstract
from arXiv · showhide
Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.
1 INTRODUCTION
Existing span-level masking assumes masked spans are uniformly positioned, despite dependencies among phrases and entities. POSPAN combines span length and position constraints to represent these dependencies and unify existing span-masking methods.
1 INTRODUCTION
- Vanilla MLM masks individual tokens, overlooking phrase compositional semantics and entity inter-token dependencies that matter for NLU [25].Source: §1 · p. 1
- Existing span-level methods assume masked-span positions are uniform, although semantic flows and inter-span interactions can create dependencies across phrases and entities [3].The paper argues that this assumption may lead to sub-optimal span selection during pre-training.Source: §1 · p. 1
- POSPAN combines span-length and position-constraint distributions to control span length and inter-span distance, unifying existing span-level methods through uniform position constraints.Existing methods use different span-length distributions while deriving their position constraints from the same uniform distribution.Source: §1 · pp. 1–2
2 APPROACH
POSPAN combines span-length and inter-span position distributions, extending span masking to model dependencies among masked spans. Its analysis motivates position constraints as a way to represent semantic dependencies that span-length-only methods leave unmodeled.
2.1 Span Masking with Two Distributions
- POSPAN models span lengths with F_M and inter-span distances with F_D, while random span placement induces a polynomial F_D.The position constraint is the distance between consecutive spans.Source: §2.1 · p. 2
2.2 Theoretical Analysis of POSPAN
- The analysis represents span-pair semantics with latent R_ij, allowing independence or either span to serve as the other’s premise.These cases distinguish span pairs that can be predicted independently from pairs with directional semantic dependencies.Source: §2.2.1 · p. 2
2.2.1 Latent Semantic Dependency.
- Prior methods focus on span-length terms and ignore latent dependencies; when spans are semantically linked, an improper F_D can harm NLU masking models.Source: §2.2.1 · p. 3
2.2.2 Position Constraint as Prior Knowledge.
- POSPAN infers span-pair dependency using intervening tokens and decomposes pretraining into span-length and span-dependency losses governed by F_M and F_D.The analysis assumes that, given R_ij, unmasked intervening tokens are independent of the two spans.Source: §2.2.2 · p. 3
2.3 POSPAN Algorithm
- The algorithm samples lengths and position constraints, alternates them while traversing token positions, then trims selected spans to meet the masking-rate target.It re-implements prior strategies and supports combinations of Normal, Geometric, Uniform, and Poisson distributions; the distribution hyperparameters are grid-searched.Source: §2.3 · p. 3, Table 1 · p. 3
3 EXPERIMENTS
Experiments compare POSPAN masking strategies on NLU benchmarks using post-training, then examine overall performance and the effects of position-constraint distributions.
3.1 Experimental Settings
- Experiments cover four NLU task types and use DeBERTaV3 post-trained on about 1.5M sentences and 250M tokens, with shared post-training and fine-tuning corpora for comparisons.The evaluated task types include named entity recognition, sentence-pair classification, question answering, and machine reading comprehension.Source: §3.1 · pp. 3–4
3.2 Main Results
- POSPAN achieved the best performance across tasks, surpassing previous best baselines by 1.8% on COPA, 0.9% on BoolQ, and 0.6% on RACE.All span-level methods also substantially improved over single-token masking, and all post-training models improved over DeBERTaV3.Source: Table 2 · p. 4, §3.2 · p. 4
3.3 Discussions
- Position-constraint choice substantially affects performance: Pois-Pois improves on Pois by 0.6%, whereas Geo-Geo decreases Geo's score by 0.6%.Across tested variants, Geo position constraints harm the original span-masking methods, while other distributions boost them to varying extents.Source: §3.3 · p. 4
- The authors argue that dependent span positions help model contextual dependencies; random selection and Geo constraints promote small inter-span distances that hinder dependency modeling.Pois performs relatively better than the other continuous distributions in these experiments.Source: §3.3 · p. 4
4 CONCLUSION AND FUTURE WORK
POSPAN combines span-length and position-constraint distributions into a general framework for span-level masking. Experiments and theoretical analysis support the framework's effectiveness and the role of both distributions.
4 CONCLUSION AND FUTURE WORK
- POSPAN uses span-length and position-constraint distributions to mask tokens and unifies existing span-level masking methods.The paper reports experiments on NLU tasks and theoretical analysis of the two distributions' rationality and necessity.Source: §4 · p. 5
- The analysis links the two distributions to learning span-level semantics and contextual dependencies.Source: §4 · p. 5
- Future work will explore masking strategies by designing better span-length and position-constraint distributions.Source: §4 · p. 5