Source-linked AI summary
Ad Headline Generation using Self-Critical Masked Language Model
Yashal Shakti Kanungo, Sumit Negi, Aruna Rajan
TL;DR
Creating attractive, brand-building ad headlines at e-commerce scale is difficult because catalogs are large and retail product content is variable. The paper uses a self-critical RL policy-gradient method with a Transformer-based MLM conditioned on multiple products, and reports the highest overlap, quality-audit, and grammar-audit scores against baselines and human submissions. The training procedure changes without impacting inference latency, supporting a practical headline-generation system.
Problem
Millions of products and variable retail titles make scalable manual headline creation difficult, while templates are less expressive and may not build shopper excitement or brand identity.
Method
The method trains a Transformer-based masked language model with self-critical reinforcement learning and jointly conditions headline generation on multiple products.
Results
The method achieves the highest scores in overlap metrics, quality audits, and grammar audits compared with baselines and human-submitted headlines.
Takeaways & Limitations
RL changes only the training procedure while preserving inference latency, and the resulting system supports practical programmatic generation of multi-product advertising headlines.
Takeaways & Limitations
A 30-epoch fine-tuning sub-experiment takes over 20 days on an Nvidia V100 and consumes approximately 150 kWh of energy.
Abstract
from arXiv · showhide
For any E-commerce website it is a nontrivial problem to build enduring advertisements that attract shoppers. It is hard to pass the creative quality bar of the website, especially at a large scale. We thus propose a programmatic solution to generate product advertising headlines using retail content. We propose a state of the art application of Reinforcement Learning (RL) Policy gradient methods on Transformer based Masked Language Models. Our method creates the advertising headline by jointly conditioning on multiple products that a seller wishes to advertise. We demonstrate that our method outperforms existing Transformer and LSTM + RL methods in overlap metrics and quality audits. We also show that our model-generated headlines outperform human submitted headlines in terms of both grammar and creative quality as determined by audits.
1 Introduction
The paper addresses scalable generation of attractive, brand-building headlines for campaigns spanning multiple retail products. It proposes an MLM-based NLG system trained with RL to improve headline quality while reducing manual effort and preserving low inference latency.
- Motivation: E-commerce catalogs can contain millions of products, making manual headline creation costly and motivating programmatic generation.Existing workarounds include padding keywords, repasting catalog content, and using templates.
- Motivation: Templates reduce creative effort but fail to create shopper excitement or brand identity.The paper links higher-quality headlines with more attractive advertisements and stronger value propositions.
- Generation Challenges: Retail titles vary from poorly structured fragments to long lists of product features, complicating headline generation.Generated headlines must preserve retail information while remaining distinct and attractive.
- Generation Challenges: Campaign headlines must generalize shared characteristics across related products rather than describe only one product.A single headline is shared across all products in an advertising campaign.
- Contributions: The proposed MLM generates advertisement headlines conditioned on multiple products simultaneously and is evaluated through test-set metrics, quality audits, and grammar audits.The stated contributions include outperforming baselines and human-submitted headlines in quality and grammar.
- Contributions: RL training directly optimizes headline quality metrics without changing the inference setup or latency, while the system reduces manual headline-writing effort.The authors also state that the method has low latency and can extend to other NLG tasks.
2 Related Work
Prior NLG work includes autoregressive language models and RL-based headline generation, but advertising systems had not combined RL with pretrained Transformer-based MLMs in this setting. The paper extends BERT-based generation with self-critical policy gradients and multi-product conditioning.
- Language-Model Generation: BERT introduced masked-language-model pretraining within a Transformer architecture, while GPT2 and BART use autoregressive word-by-word generation.Related work also combines pretrained language-model representations with encoder-decoder systems.
- Reinforcement Learning: RL-based NLG treats generation quality as a reward, and SCST uses the model’s inference-time algorithm as the reward baseline.This normalizes rewards without relying on a separately learned baseline.
- Advertising Headline Generation: Advertising headline systems combining LSTM pointer networks with RL improved results but did not exploit pretrained Transformer models.The paper positions this limitation as motivation for its Transformer-based approach.
- Paper Positioning: The proposed method extends BERT-based generation with self-critical policy gradients and simultaneous conditioning on multiple products.This combines pretrained MLMs with campaign-level product inputs.
- Paper Positioning: The approach can optimize inference-time metrics such as BLEU, Rouge, and Readability, including metrics that are typically nondifferentiable.The cited design connects RL training with direct optimization of these evaluation objectives.
3 Self-Critical Masked Language Model
The model adapts BERT-style masked language modeling for autoregressive, multi-product headline generation by combining masked attention with Self-Critical reinforcement learning. It jointly optimizes token likelihood and headline-quality rewards while using beam search at inference.
- 3.1 Masked Language Model: The model represents multiple product titles and a headline with sub-token, positional, segment, and optional category embeddings.Product titles are separated with a special token, distinguishing products and source-target subsequences.
- 3.1 Masked Language Model: Only headline tokens are randomly masked during training, and Transformer layers predict the masked tokens with a softmax vocabulary output and cross-entropy loss.The input contains a headline and one or more product titles, while source-token masking is treated as an experimental variant.
- 3.3 Generation using Self-Critical Masked Language Model: Masked attention prevents training-time headline predictions from accessing future headline tokens unavailable during autoregressive inference.The attention mask sets allowed positions to 0 and disallowed positions to −∞.
- 3.3 Generation using Self-Critical Masked Language Model: The reinforcement-learning formulation addresses exposure bias and optimizes reward functions for headline quality rather than only masked-word log-likelihood.Rewards may measure overlap with approved headlines or be predicted by another model, and REINFORCE computes the policy gradient.
- 3.3 Generation using Self-Critical Masked Language Model: Self-Critical training compares a sampled headline with a greedy inference-time headline from the same MLM and uses their reward difference to update the model.The resulting loss combines headline reward with the likelihood of correct words.
- 3.4 Inference: During inference, the model generates headlines left-to-right with beam search, using modified length normalization and regular-expression post-processing.Length normalization addresses beam-search bias against longer headlines, while post-processing removes extra spaces around symbols.
4 Experiments
Experiments use a large Amazon advertising-campaign dataset and compare the proposed system with LSTM-based and ablated alternatives. The study also examines training objectives, product conditioning, model scale, and inference modifications, while noting substantial computational cost.
- 4 Experiments: The dataset contains over 500,000 Amazon ad campaigns with related products and associated headlines.Campaigns were filtered to English content, deduplicated by product-headline pairs, and split into train, validation, and test sets.
- 4 Experiments: The experiments use pre-trained Transformer BERT Large models, first fine-tuned with MLM loss and then with Self-Critical MLM gradients.Rouge L F1 overlap with approved headlines supplies the headline-quality reward.
- 4 Experiments: A single 30-epoch fine-tuning sub-experiment takes over 20 days on an Nvidia V100 and was estimated to consume approximately 150 kWh.The computational cost limited the study to essential experiments.
- 4 Experiments: The comparison includes a Pointer Network bi-LSTM with intra-decoder and temporal attention, trained with Self-Critical methods for fairness.This baseline represents prior advertisement-headline generation approaches.
- 4 Experiments: Ablations remove MLM pre-training or Self-Critical loss while preserving the proposed architecture, parameter count, and input.These variants isolate the contributions of the two training components.
- 4 Experiments: Additional experiments use only the primary product, omit length normalization and post-processing, or replace BERT Large with BERT Base.The single-product condition is motivated partly by reduced training time and inference latency for cohesive campaigns.
5 Results
The proposed Self-Critical MLM achieves the strongest overlap, quality, and grammar results among evaluated models and human-submitted headlines. Audits use approved human headlines, blind quality ratings, and independent grammar judgments.
- 5.1 Overlap with Approved Headlines: ‘Proposed Self-Critical MLM’ achieves the best scores across all overlap metrics.The comparison includes no-pretraining, proposed MLM, BERT Large, and alternative masking configurations.
- 5.1 Overlap with Approved Headlines: Multi-product conditioning outperforms using only one product title, although single-title inference is 8% faster.The single-title model produces correct headlines but lacks sufficient input to make them attractive.
- 5.2 Quality and Grammar Audits: The proposed SC-MLM model achieves the highest average quality rating and the most perfectly rated headlines.The quality audit compares model-generated and human-submitted headlines on a 3-point scale using a random blind test set of approximately 5,000 examples.
- 5.2 Quality and Grammar Audits: 98.13% of SC-MLM headlines had correct grammar, compared with 93.14% of human-submitted headlines.The grammar-specific audit covered approximately 10,000 judgments.
- 5.2 Quality and Grammar Audits: SC-MLM produces more descriptive headlines, while both evaluated models abstract product qualities.The sample headlines come from blind-test campaigns and include subsets rated 3, 2, and 1.
6 Conclusion
The paper addresses difficult, weakly expressive template-based ad headline generation by proposing an NLG method for multiple products. Masked Language Models and reinforcement learning improve evaluation outcomes while preserving inference latency and supporting broader NLG use.
- 6 Conclusion: Varying retail product attributes make ad headline generation difficult, while template-based methods are often not expressive.Retail attributes vary substantially, complicating the creation of attractive headlines.
- 6 Conclusion: The proposed NLG method generates headlines for multiple products and achieves the highest overlap, quality-audit, and grammar-audit scores.The comparison is against baselines and human-submitted headlines.
- 6 Conclusion: Masked Language Models were relatively unexplored for ad headline generation, and the paper demonstrates their utility.The model's performance is further extended through reinforcement learning.
- 6 Conclusion: Reinforcement learning changes only training and does not affect inference latency.The conclusion presents this as relevant to practical business applications.
- 6 Conclusion: The approach can also be used for other natural language generation tasks.The paper explicitly mentions broader NLG applicability.