Source-linked AI summary
Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior
Ashmit Khandelwal, Aditya Agrawal, Aanisha Bhattacharyya, Yaman K Singla, Somesh Singh, Uttaran Bhattacharya, Ishita Dasgupta, Stefano Petrangeli, Rajiv Ratn Shah, Changyou Chen, Balaji Krishnamurthy
TL;DR
Communication models have made progress in transmitting symbols and understanding meaning, but predicting and optimizing messages for receiver behavior remains an open effectiveness problem. This paper trains LCBMs on content and behavior together, reports capabilities across behavior-related and content tasks, and releases the Content Behavior Corpus to support further research.
Problem
Modern language models are not conventionally designed to predict or optimize communication for desired receiver behavior, leaving the effectiveness level largely untouched.
Source: §1 · pp. 3–4Method
The authors introduce receiver behavior tokens into language-model training and instruction-tune multimodal LCBMs to predict behavior from content and content from behavior.
Source: §1 · p. 4, §2.4 · p. 10Results
LCBMs show good performance across behavior-related tasks, alongside content understanding, and report behavior simulation, content simulation, behavior understanding, and behavior domain adaptation capabilities.
Source: §1 · p. 4, §3 · pp. 11–12, §4 · p. 12Takeaways & Limitations
The work makes initial strides toward modeling communication effectiveness and releases CBC to spur research on large content and behavior models.
Source: §4 · pp. 12–13The authors could not compare this model with GPT-3 because the data are private.
Source: §2.4 · p. 10
Abstract
from arXiv · showhide
Shannon and Weaver's seminal information theory divides communication into three levels: technical, semantic, and effectiveness. While the technical level deals with the accurate reconstruction of transmitted symbols, the semantic and effectiveness levels deal with the inferred meaning and its effect on the receiver. Large Language Models (LLMs), with their wide generalizability, make some progress towards the second level. However, LLMs and other communication models are not conventionally designed for predicting and optimizing communication for desired receiver behaviors and intents. As a result, the effectiveness level remains largely untouched by modern communication systems. In this paper, we introduce the receivers' "behavior tokens," such as shares, likes, clicks, purchases, and retweets, in the LLM's training corpora to optimize content for the receivers and predict their behaviors. Other than showing similar performance to LLMs on content understanding tasks, our trained models show generalization capabilities on the behavior dimension for behavior simulation, content simulation, behavior understanding, and behavior domain adaptation. We show results on all these capabilities using a wide range of tasks on three corpora. We call these models Large Content and Behavior Models (LCBMs). Further, to spur more research on LCBMs, we release our new Content Behavior Corpus (CBC), a repository containing communicator, message, and corresponding receiver behavior (https://behavior-in-the-wild.github.io/LCBM).
1 INTRODUCTION
Communication research has advanced technical transmission and semantic understanding, but effectiveness—whether a message elicits desired receiver behavior—remains largely unsolved. The paper introduces LCBMs, which model content and receiver behavior together, and releases a corpus and benchmark for studying them.
1 INTRODUCTION
- LCBM outperforms GPT-3.5 and GPT-4 on behavior simulation, content simulation, and behavior understanding, while GPT-3.5 performs better on content understanding.The comparison also notes that GPT-3.5 and GPT-4 are at least 12 times larger than LCBM-13B.Source: Figure 2 · p. 2
- Communication has technical, semantic, and effectiveness levels; effectiveness concerns whether received meaning induces desired conduct and remains largely unsolved.The paper frames effectiveness as designing messages to fulfill communicators’ objectives, such as explaining concepts or informing choices.Source: §1 · pp. 2–3
- The paper proposes representing receiver actions alongside content as text, so a language model can simulate, optimize for, and reason about those actions.Typical corpora omit actions such as email opens, read-time, replies, likes, shares, and forum replies.Source: §1 · pp. 3–4
- LCBMs connect a visual encoder with an LLM and use behavior instruction tuning to predict behavior from content and content from behavior.The model also supports content understanding and behavior understanding across a range of tasks.Source: §1 · pp. 4–5, Figure 4 · p. 5
- The authors release CBC and a benchmark, including data from over 40,000 public-domain YouTube videos and 168 million Twitter posts.The data pairs content with receiver behavior, including video engagement signals and Twitter likes.Source: §1 · p. 4
2 SETUP
The setup brings communicator, message, and receiver behavior into a shared text-to-text modeling framework, then evaluates content and behavior capabilities across five tasks.
2.1 THE CONTENT BEHAVIOR CORPUS (CBC)
- CBC combines YouTube and Twitter communicator, message, and receiver-effect data, verbalizing multimodal content and numeric behavior for text-to-text modeling.For videos, replay values are grouped to accommodate longer durations, with scene content represented using frame captions and speech recognition.Source: §2.1 · pp. 5–6
2.2 MODEL
- The model pairs Vicuna-13B with EVA-CLIP visual encoding, temporal aggregation, and Q-Former language tokens, using visual alignment followed by behavior-instruction training.Scene captions are included alongside visual tokens and replay information.Source: §2.2 · p. 6
2.3 CONTENT BEHAVIOR TEST BENCHMARK
- The benchmark covers behavior simulation, content simulation, behavior understanding, content understanding, and behavior domain adaptation across video, social-media, and email tasks.Tests include replay and like/view prediction, sentiment reasoning, advertisement understanding, email click-through prediction, and LVU behavior prediction.Source: §2.3 · pp. 6–9
2.4 BEHAVIOR INSTRUCTION FINE-TUNING (BFT)
- BFT first teaches visual modality and then fine-tunes the LLM to predict behavior from content, with a reverse task for simulating content from behavior.The behavior-prediction inputs include visual representations, scene descriptions, replay rates, views, likes, and communicator information; the visual encoder remains frozen during behavior tuning.Source: §2.4 · pp. 9–11
- Behavior instruction tuning trains the model on content-behavior pairs with an autoregressive objective, predicting behavior tokens from content tokens.The setup also trains the reverse direction, simulating content given behavior.Source: §2.4 · pp. 10–11
3 RESULTS AND DISCUSSION
LCBM outperforms content-only models on the reported behavior-related tasks despite being much smaller, while retaining content-understanding capability and showing behavior-domain adaptation. The authors also report that behavior fine-tuning improves many content-understanding tasks over the base LLM.
3 RESULTS AND DISCUSSION
- 10x smaller than GPT-3.5 and GPT-4, LCBM performs better on all reported behavior-related tasks, while GPT models often perform comparably to or worse than random baselines.The authors also report a substantial LCBM advantage on content simulation and little performance difference between 10-shot and 2-shot GPT-4 or between GPT-3.5 and GPT-4.Source: §3 · p. 11
- Behavior fine-tuning improves most tested content-understanding tasks over the base LLM, while LCBM exceeds Vicuna and VideoChat but not GPT-3.5.The authors suggest behavior modality may carry additional information about content.Source: §3 · p. 12
- LCBM shows behavior-domain adaptation across email, Twitter, and YouTube-related tasks; models trained on combined YouTube and target-domain data outperform the reported single-domain alternatives.The authors also report that LCBM can explain replay and comment behavior, though its explanations are verbose.Source: §3 · p. 12
- Figure 5 presents examples of LCBM explaining audience sentiment and compares its outputs with Vicuna and GPT-3.5.The authors report that LCBM explains behavior well while being verbose.Source: Figure 5 · p. 12, §3 · p. 12
4 CONCLUSION
The paper reports initial progress on the communication effectiveness problem: training LLMs with behavior tokens supports behavior-related tasks, and the authors introduce the Content Behavior Corpus to encourage further research.
4 CONCLUSION
- Training LLMs on behavior tokens supports good performance across behavior-related tasks, alongside content understanding.The paper frames these capabilities as initial progress on predicting and optimizing communication for desired receiver behavior.Source: §4 · pp. 12–13
- Figure 6 gives examples of LCBM explanations of scene replayability alongside human-provided explanations.Source: Figure 6 · p. 13
- The paper introduces the Content Behavior Corpus (CBC) to spur research on large content and behavior models.Source: §4 · p. 13
A.1 VERBALIZATION LISTINGS
The verbalization listings show how video, email, and Twitter content and behavior are represented as task inputs and outputs.
A.1 VERBALIZATION LISTINGS
- For video behavior understanding, the listing represents scenes with ASR, captions, and replay values and shows a scene-description output.Its example also gives video title, view count, and like percentage.Source: §A.1 · p. 17
- The email behavior-domain-adaptation example masks click-through rate for an email described by its content, audience, and sending context.The displayed output gives a rate of 0.037%.Source: §A.1 · p. 17
- A separate video listing demonstrates predicting scene content given behavior information, with one scene's ASR masked and replay value shown.Source: §A.1 · p. 17
- For video content simulation, the listing masks scene text while providing replay values, then presents candidate completions and an output.Source: §A.1 · p. 17
- The video behavior-simulation listing masks replay values for scenes and asks the model to predict them from video content.The example also asks for views and likes as a percentage of views.Source: §A.1 · p. 17
- For Twitter behavior simulation, the listing supplies tweet and media verbalization and asks whether the post will receive high or low likes.The example output is “low likes.”Source: §A.1 · pp. 17–18
- For Twitter content simulation, the listing asks for a tweet given media verbalization and its low-like behavior label.The example output supplies a tweet text.Source: §A.1 · p. 18
A.2 OTHER RELATED WORK
The related work contrasts content-only language models with behavior-modeling approaches, arguing that existing approaches often omit important parts of communication or broader text-to-text capabilities.
A.2 OTHER RELATED WORK
- The paper defines communication as involving sender, message, receiver, and an intended effect, and says LLMs are trained on messages while ignoring other available factors.Source: §A.2 · p. 18
- Agent-based models can neglect communication content, may assume rational reward-maximizing agents, and often rely on simulated rather than real human behavior data.The passage identifies scarcity of large repositories of human actions as a training challenge.Source: §A.2 · p. 18
- The authors contrast conventional behavior predictors and LLMs with classification or regression heads, arguing these approaches lack LCBM's described text-to-text content generation and explanation capabilities.They also describe LCBMs as supporting behavior simulation and domain adaptation.Source: §A.2 · p. 18