Source-linked AI summary
HTLM: Hyper-Text Pre-Training and Prompting of Language Models
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, Luke Zettlemoyer
TL;DR
Language-model pretraining often strips web-page markup, despite HTML carrying document-level signals and enabling structured prompts. HTLM trains on simplified HTML with BART-style denoising and uses HTML templates and size hints for prompting. It transfers strongly across tasks, improves zero-shot summarization, and finds hyper-text prompts more data-efficient for HTLM than plain-text prompts are for existing LMs.
Problem
Web-page markup is usually discarded in language-model pretraining, although HTML carries document-level signals and standard BART prompting cannot control generated span length.
Source: §1 · p. 1, §2.3 · p. 3Method
HTLM pretrains on simplified HTML with BART-style denoising and uses HTML-structured prompts, including noisy size hints to guide generated span length.
Source: §1 · p. 2, §2.3 · p. 3Results
HTLM matches or exceeds comparably sized text-only LMs on zero-shot prompting and full fine-tuning for GLUE, while improving zero-shot summarization by up to 8 ROUGE-1 points.
Source: §1 · p. 2Takeaways & Limitations
Hyper-text prompts are more data-efficient for HTLM than plain-text prompts are for existing LMs, and auto-generated HTML prompts sometimes rival or surpass manually engineered prompts.
Source: §1 · p. 2, §8 · p. 10
Abstract
from arXiv · showhide
We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling title tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.
1 Introduction
HTLM treats HTML markup as useful pretraining material because it carries document-level and task-adjacent signals. It combines HTML-structured prompting with BART-style denoising, achieving strong transfer across tasks and supervision levels.
1 Introduction
- HTML provides document-level signals: title elements can summarize page bodies, while class and id attributes can encode document categories and proxy for NLP tasks.Such supervision varies with what website authors choose to present.Source: §1 · p. 1
- Structured prompts use HTML semantics and size hints to specify tasks, including zero-shot summarization by infilling a page’s title tags.The scheme provides more fine-grained control over task specifications.Source: §1 · pp. 1–2
- HTLM matches or exceeds comparably sized text-only LMs on zero-shot prompting and full fine-tuning for GLUE, and improves zero-shot summarization by up to 8 ROUGE-1 points.The paper also reports few-shot learning on tasks such as table-to-text generation.Source: §1 · p. 2
- The model is trained on 23TB of simplified Common Crawl HTML with a modified BART denoising objective that reconstructs masked spans.The objective includes noisy size hints for more fine-grained task-specific length priors.Source: §1 · p. 2
- Hyper-text prompts are effectively equivalent to up to a thousand extra training examples for HTLM, while plain-text prompts provide less data efficiency for existing LMs.The analysis follows the methodology of Le Scao and Rush (2021).Source: §1 · p. 2
- HTLM can auto-prompt by formatting available examples in HTML, sometimes rivaling or surpassing manually engineered prompts.The model generates likely hyper-text formatting that can be templatized as a task prompt.Source: §1 · p. 2
2 HyperText Language Model (HTLM)
HTLM models simplified HTML with BART-style span denoising, using the retained markup both as document context and as a structure for prompting. Its size hints address the difficulty of controlling generated span length.
2 HyperText Language Model (HTLM)
- HTLM trains on simplified HTML from Common Crawl with a BART-style denoising autoencoder and span masking, augmented with size hints.The model is trained on a large-scale corpus of automatically extracted MHTML.Source: §2 · p. 2
2.1 Minimal HTML
- MHTML filtering removes 94% of raw-page characters, fits close to 85% of documents within 1,024 BPE tokens, and yields 23TB from the January 2021 Common Crawl snapshot.The filtering targets markup that provides little pretraining supervision, including much JavaScript and CSS.Source: §2.1 · pp. 2–3
2.2 Model
- BART-style denoising reconstructs arbitrary masked substrings from the rest of a document, supporting both output masks in prompts and automatic generation of HTML prompt formatting.The model uses BART-Large architecture and initialization, with roughly 400 million parameters.Source: §2.2 · p. 3
2.3 Size Hints
- Standard BART prompting cannot reliably control each generated mask’s length, even with decoding length penalties.During BART reconstruction, a mask may be replaced by multiple tokens and its length is predicted implicitly.Source: §2.3 · p. 3
- Size hints append a noisy estimate of span length after a mask, letting prompts request output of roughly a specified length or omit the hint to leave length implicit.Hints are given to 80% of masks with noisiness ε = 0.1.Source: §2.3 · p. 3
3 HTML-based Prompting
HTLM prompts encode task structure in HTML templates with output masks, chosen manually or generated by the model. Optional size hints help control output length and make templates more reusable, while auto-prompting is less effective for classification.
3 HTML-based Prompting
- HTML templates instantiate task inputs and output masks; the model reconstructs the full input, and heuristics extract the generated output around each mask.Templates may be selected manually or generated by HTLM.Source: §3 · p. 4
3.1 Generation Prompting Policies
- Without size hints, templates must match both task semantics and average target length, making them brittle and requiring careful manual design.Hints decouple generation length from the prompt and improve template reuse across related tasks.Source: §3.1 · p. 4
- For manual generation prompts, HTLM uses the average target length from a training subset as the size hint, then tries up to five alternatives if output extraction fails.It selects the valid output with lowest perplexity and falls back to an auto-template if needed.Source: §3.1 · p. 4
3.2 Auto-Prompting
- Auto-prompting masks each independent data block and asks HTLM to recover the document markup, producing a structured prompt template.The approach uses learned document semantics to avoid manually engineering prompts.Source: §3.2 · p. 4
- HTLM auto-prompting was less effective for classification; the authors hypothesize that generative targets carry more information than binary target tokens.For generation experiments, Auto-S uses size hints and Auto-NS does not.Source: §3.2 · p. 4
4 Zero/One-Shot Prompting
HTLM’s structured prompts support zero-shot generation and classification, as well as one-shot table-to-text tasks. In summarization, manual prompts with size hints improve on prior zero-shot results across four datasets, while auto-prompts with size hints beat PEGASUS on three.
4 Zero/One-Shot Prompting
- Manual prompts were drawn from related work or developed using at most 50 training examples to limit development-data tuning.Prompts were manually searched using a maximum of 50 training examples for evaluation.Source: §4 · p. 4, §4.1 · p. 6
4.1 Generation
- E2E contains about 50K restaurant-domain examples with eight fields; WebNLG spans 15 DBPedia domains, and DART uses Wikipedia tables.The WebNLG results are reported for Seen, Unseen, and All subsets.Source: §4.1 · p. 6
- More than 8 ROUGE-L F1 improvement on Gigaword is reported for HTLM-Manual, while HTLM-Auto-S outperforms PEGASUS on three of four datasets, including by roughly 6 ROUGE-L points on Gigaword.Manual prompts with size hints improve prior zero-shot results on all four datasets without tailored pretraining.Source: §4.1 · p. 6, Table 2 · p. 7
- HTLM’s HTML-based prompting provides control over dataset-specific attributes such as output length and style.The summarization experiments compare manual and auto-prompt variants with and without size hints using ROUGE-1, ROUGE-2, and ROUGE-L.Source: §4.1 · p. 6, Table 2 · p. 7
- For one-shot table-to-text generation, prompting required a single training example, and the authors report these as the first such results to their knowledge.These tasks use structured tabular inputs, for which text-only model prompting is not straightforward.Source: §4.1 · p. 6
- Table-to-text results compare GPT-2 Medium, GPT-2 Large, and HTLM on E2E, WebNLG, and DART.The table layout places the datasets in left, middle, and right panels, respectively.Source: Table 3 · p. 7
4.2 Classification
- HTLM prompting outperforms GPT-3 Medium on most zero-shot classification tasks and approaches GPT-3 Large, outperforming it on RTE despite the latter having roughly twice as many parameters.Classification selects the candidate whose instantiated template has the lowest perplexity.Source: §4.2 · pp. 6–7
5 Fine-tuning Experiments
HTLM’s learned representations improve over existing pretraining methods in full fine-tuning, and prompting examples during fine-tuning can further improve classification performance. For table-to-text fine-tuning, HTLM consistently outperforms both GPT-2 variants.
5 Fine-tuning Experiments
- GLUE comparisons include RoBERTa, BART, and T5; fine-tuning uses prompts derived from prior work rather than simple sentence concatenation.The study also reports results using R3F fine-tuning.Source: §5 · p. 7
- Overall, HTLM improves over existing pretraining methods on GLUE, and placing examples into prompts while fine-tuning the classification head can further improve performance.The prompting improvements do not adversely affect fine-tuning, supporting structured pretraining as a viable alternative for this setting.Source: §5 · p. 7
- HTLM fine-tuned on table-to-text generation consistently outperforms both GPT-2 variants.The experiments place all NLG samples into prompts during fine-tuning.Source: §5 · p. 7
6 Prompt Data Efficiency
The section measures prompt data efficiency by comparing training points needed to reach matched fine-tuning performance. HTLM’s HTML prompts show greater training advantage than natural-language prompts for other pretrained models, while removing meaningful verbalizers reduces that advantage.
6 Prompt Data Efficiency
- The section frames prompt training-point value as a comparison between structured prompts and classification heads, and between meaningful and nonsensical verbalizers.Tables 6 and 7 report these respective comparisons.Source: §6 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 8
- The analysis measures prompt efficacy in training points by matching end fine-tuning performance across classification-head, meaningful-verbalizer, and nonsensical-verbalizer settings.The prompt comparisons use PET prompts, wrapped in HTML for HTLM, across MNLI, BoolQ, CB, RTE, and WiC.Source: §6 · pp. 8–9
- HTLM prompts provide greater training-point advantage than natural-language prompts across the compared pretrained models, including nearly triple RoBERTa-Large’s advantage on CB and double on RTE.On WiC, HTLM is the only compared model with a positive training advantage from prompts.Source: Table 6 · p. 8, §6 · p. 9
- For small datasets such as CB, all of HTLM’s prompt advantage over a random-name verbalizer comes from the pattern itself.The meaningful-verbalizer versus random-name comparison loses much of the advantage seen in the prompt-versus-classification-head setting.Source: Table 7 · p. 8, §6 · p. 9
- The authors regard these results as evidence that structured, document-level pretraining and prompting are a viable alternative to purely natural-language approaches.Source: §6 · p. 9
7 Related Work
Related work covers language-model prompting, task-specific summarization methods, parameter-efficient prompt optimization, and structured-input architectures. HTLM differs by pretraining a generative masked language model directly over HTML and learning its structure from the data.
7 Related Work
- GPT-3 associated prompting with large-scale pretraining, while PET and follow-up work showed prompting can also work with smaller models and without unlabeled data.HTLM instead uses hyper-text prompting with generative masked language models pretrained directly over HTML.Source: §7 · p. 9
- PEGASUS and Wiki-Transfer target zero-shot summarization, whereas HTLM uses mask-size hints and varied prompts to control length and style without dataset-specific augmentation or fine-tuning.Source: §7 · p. 9
- Other hybrid prompting work optimizes few parameters, but this paper focuses on full fine-tuning or zero-shot prompting.Examples include continuous prompt optimization (Li and Liang, 2021) and low-rank projection optimization (Aghajanyan et al., 2020b).Source: §7 · p. 9
- Unlike architectures that encode structural priors for structured inputs, HTLM learns structure directly from HTML without adding such priors to the model.The cited structured-input architecture is discussed by Ainslie et al. (2020).Source: §7 · pp. 9–10
8 Conclusion
The conclusion presents HTLM as a language model pretrained on simplified HTML, with structured prompting and improved transfer. It also reports gains from auto-prompting and data efficiency, while identifying scaling laws as future work.
8 Conclusion
- HTLM models simplified HTML with a BART-like objective and uses HTML task representations for structured zero-shot prompting, including summarization results that exceed prior best performance by a wide margin.The conclusion also reports improved full fine-tuning performance relative to pretrained models that model only natural language.Source: §8 · p. 10
- HTLM can generate HTML formatting from training samples for auto-prompts, and its HTML prompts offer greater data efficiency than natural-language prompts for the compared baselines.Auto-prompts on Gigaword and CNN/DM outperformed previous state-of-the-art zero-shot approaches.Source: §8 · p. 10
- The authors identify scaling laws for structured pretraining and prompting as future work, noting that model size and compute affect prompting performance.Source: §8 · p. 10
A.1 Finetuning Hyper-Parameters
The appendix identifies task-specific GLUE hyperparameters and a separate set for R3F experiments on GLUE.
A.1 Finetuning Hyper-Parameters
- The appendix section concerns the hyperparameters used for GLUE-related experiments.Source: §A.1 · p. 13
- Table 8 lists task-specific hyperparameters for GLUE experiments.Source: Table 8 · p. 14
- Table 9 lists hyperparameters for R3F experiments on GLUE.Source: Table 9 · p. 14