Source-linked AI summary
MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
Shuai Peng, Ke Yuan, Liangcai Gao, Zhi Tang
TL;DR
Mathematical formula understanding remains challenging because formulas have complex structure and implicit semantics, while existing approaches do not jointly train structural and contextual information. MathBERT jointly trains formulas, contexts, and Operator Trees, with masked-substructure prediction to capture formula structure. It achieves state-of-the-art performance on three downstream tasks and is reported to capture semantic-level structure qualitatively.
Problem
Math-related tasks remain challenging because formulas vary in representation, have complex structure, and contain ambiguous implicit semantics, while existing approaches do not jointly train structural and semantic information.
Source: §1 · p. 1Method
MathBERT jointly trains mathematical formulas, their contexts, and Operator Trees, including a task that predicts masked formula substructures.
Source: §1 · p. 2, Masked Substructure Prediction · p. 4Results
MathBERT achieves state-of-the-art performance on mathematical information retrieval, formula topic classification, and formula headline generation, and is reported to capture formula structure qualitatively.
Source: §5 · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Large-scale pre-trained models like BERT, have obtained a great success in various Natural Language Processing (NLP) tasks, while it is still a challenge to adapt them to the math-related tasks. Current pre-trained models neglect the structural features and the semantic correspondence between formula and its context. To address these issues, we propose a novel pre-trained model, namely \textbf{MathBERT}, which is jointly trained with mathematical formulas and their corresponding contexts. In addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula substructures extracted from the Operator Tree (OPT), which is the semantic structural representation of formulas. We conduct various experiments on three downstream tasks to evaluate the performance of MathBERT, including mathematical information retrieval, formula topic classification and formula headline generation. Experimental results demonstrate that MathBERT significantly outperforms existing methods on all those three tasks. Moreover, we qualitatively show that this pre-trained model effectively captures the semantic-level structural information of formulas. To the best of our knowledge, MathBERT is the first pre-trained model for mathematical formula understanding.
1 Introduction
Math-related tasks remain challenging because formulas vary in representation, have complex structures, and carry implicit semantics; prior customized approaches did not jointly train structural and semantic information. MathBERT jointly trains formulas, contexts, and Operator Trees, and is reported to outperform existing methods on three downstream tasks.
1 Introduction
- Formula understanding is challenging because mathematical representations vary, formula structures are complex, and implicit semantics can be ambiguous.Prior customized models use formula structure or topical correspondence with context, but do not jointly train structural and semantic information.Source: §1 · p. 1
- Masked Substructure Prediction uses Operator Trees to predict masked formula substructures and capture semantic-level structure.The paper describes the substructures as an operator together with its parent and child nodes.Source: §1 · p. 2
- MathBERT achieves state-of-the-art performance on mathematical information retrieval, formula topic classification, and formula headline generation.The paper also reports qualitative evidence that it captures formulas’ semantic-level structural information.Source: §1 · p. 2
- MathBERT is presented as the first pre-trained model for mathematical formula understanding, jointly trained with formulas, contexts, and Operator Trees.The paper also introduces a new pre-training task to capture formulas’ semantic-level structural information.Source: §1 · p. 2
- The authors construct a formula-topic-classification dataset containing formulas and their corresponding contexts, which they say would be open soon.The paper also reports training MathBERT on more than 8.7 million formula-context pairs extracted from arXiv articles.Source: §1 · p. 2
2 Related Work
The related work reviews pre-trained models for general and specialized tasks, then discusses formula representations and the choice of Operator Trees for MathBERT.
2.1 Pre-Trained Models
- Specialized pre-trained models include CodeBERT for code synthesis and LayoutLM for document understanding, alongside general-purpose models such as BERT and XLNet.LayoutLM jointly uses text, image, and layout information.Source: §2.1 · p. 2
2.2 Mathematical Formula Representation
- Formula representations include symbol sequences and tree structures such as Symbol Layout Trees and Operator Trees; MathBERT selects OPTs for their mathematical syntax and semantics.The paper also notes that some formula layout information in SLTs is already represented in LaTeX codes.Source: §2.2 · p. 2
3 MathBERT
MathBERT combines formula tokens, context, and Operator Tree (OPT) information in a BERT-based model, with pre-training tasks targeting text, formula-context relationships, and formula structure. The section describes its architecture, training objectives, dataset construction, and training setup.
3.1 Model Architecture
- MathBERT concatenates formula LaTeX tokens, context, and OPT operators, and modifies Transformer attention using OPT structure.The architecture is built on a multi-layer bidirectional Transformer modified from vanilla BERT.Source: §3.1 · pp. 2–3
3.2 Pre-Training Tasks
- The pre-training tasks target text representations, formula-context relationships, and semantic-level formula structure.These three information targets correspond respectively to the model's three pre-training tasks.Source: §3.2 · p. 3
Masked Language Modeling
- Masked Language Modeling randomly samples 15% of formula and context tokens, then trains the model to predict the original masked tokens.Among selected tokens, 80% are replaced with [MASK], 10% with arbitrary tokens, and 10% remain unchanged.Source: Masked Language Modeling · p. 3
Context Correspondence Prediction
- Context Correspondence Prediction replaces context in 50% of examples and trains the model to identify whether it corresponds to the formula.The task targets the latent semantic relation between a formula and its context.Source: Context Correspondence Prediction · p. 4
Masked Substructure Prediction
- Masked Substructure Prediction samples 15% of OPT nodes, cuts their parent and child connections, and predicts those neighboring nodes.A substructure is an operator together with its parent and child nodes; the three task losses are added for the total loss.Source: Masked Substructure Prediction · p. 4
3.3 Pre-Training Data
- The pre-training corpus uses arXiv articles to pair single-line display formulas with at least 400 characters of surrounding context, yielding 8.7 million formula-context pairs with OPTs.The formula is replaced in the context with the special token [MATH].Source: §3.3 · p. 4
3.4 Pre-Training Details
- MathBERT is initialized from BERT base and trained on 8.7 million examples for about 10,000,000 iterations over two weeks.Training used four NVIDIA TITAN X 12GB GPUs, batch size 48, maximum input length 256, and learning rate 2e-5.Source: §3.4 · p. 4
4 Experiment
Experiments evaluate MathBERT on retrieval, topic classification, and headline generation, with ablations and qualitative analysis examining its inputs and learned formula structure.
4 Experiment
- The experiment evaluates output embeddings on NTCIR-12 retrieval, uses a collected 10-topic dataset for classification, and tests generation on EXEQ-300K.NTCIR-12 contains over 590,000 Wikipedia formulas and 20 queries; topic classification uses 132,302 arXiv formula-context pairs.Source: §4.1 · pp. 4–5, §4.2 · p. 5, §4.3 · p. 5
4.1 Mathematical Information Retrieval
- On NTCIR-12 retrieval, MathBERT ranks highest in partial and harmonic-mean bpref, but remains below TangentS and Approach0 on full bpref.Reordering TangentCFT’s top 1,000 results with MathBERT embeddings beats its original ranking on all metrics; MathAPP combines MathBERT and Approach0 retrieval scores.Source: Table 1 · p. 4, §4.1 · p. 5
4 Experiment
- MathBERT achieves state-of-the-art performance on topic classification and headline generation, while retrieval results lead on partial and harmonic-mean bpref but not full bpref.Combining MathBERT retrieval scores with Approach0 produces MathAPP, which achieves state-of-the-art performance.Source: §4.2 · p. 5, §4.3 · p. 5, §4.1 · p. 5
4.2 Formula Topic Classification
- MathBERT achieves state-of-the-art performance on all TopicMath-100K metrics and significantly outperforms vanilla BERT.The dataset contains 132,302 formula-context pairs across 10 selected topics, split into train, validation, and test sets.Source: §4.2 · p. 5
4.3 Formula Headline Generation
- For formula headline generation, MathBERT outperforms the compared models on all evaluation metrics, especially Transformer and BERT-fused.MathBERT representations are fused with each Transformer encoder and decoder layer through attention.Source: §4.3 · pp. 5–6
4.4 Ablation Study
- Using only formulas for pre-training gives the lowest results; OPTs matter more for structure-sensitive retrieval, while context matters more for topic classification.The ablation compares settings with formula, context, and OPT inputs and their associated pre-training tasks.Source: §4.4 · p. 6
4.5 Qualitative Analysis
- Qualitative rankings suggest complete MathBERT captures semantic relationships and layout structure, including similarities between equivalent expressions and formulas sharing substructures.Without OPT input, embeddings retain some semantic information, while the complete model also gives higher similarity to examples sharing layout or substructure.Source: §4.5 · p. 6
5 Conclusion
The conclusion presents MathBERT as a pre-trained model jointly trained on formulas, contexts, and their Operator Trees. It reports state-of-the-art performance across three downstream tasks and qualitative evidence of semantic-level structural learning.
5 Conclusion
- MathBERT is presented as the first pre-trained model for mathematical formula understanding, jointly trained with formulas, contexts, and their corresponding OPTs.The conclusion reports state-of-the-art performance on retrieval, topic classification, and headline generation; ablation and qualitative analyses support its training settings and structural-information capture.Source: §5 · p. 6
- The conclusion reports state-of-the-art performance on mathematical information retrieval, formula topic classification, and formula headline generation.It also states that ablations show the pre-training settings could contribute to improving downstream performance.Source: §5 · p. 6
- Qualitative analysis is reported to show that MathBERT captures semantic-level structural information in mathematical expressions.Source: §5 · p. 6