Source-linked AI summary
You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong, Siying Hu, Haichang Li, Ziying Wang, Zhicong Lu
TL;DR
Existing evaluations leave open whether models can distinguish plausible readings of naturally occurring comments in context. The paper builds a human-validated diagnostic benchmark and evaluates LLMs as question writers and solvers; the strongest model reaches 81.42% leave-writer-out accuracy, while models can still miss key aspects of the interpretation.
Problem
Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models distinguish plausible readings of a naturally occurring comment in its particular exchange.
Source: §1 · p. 2Method
The benchmark pairs naturally occurring target comments with reconstructed context and plausible misreadings, using LLM-generated questions that humans validate and models solve across writers.
Source: §1 · p. 2Results
81.42% leave-writer-out accuracy for the strongest model, while case analysis finds that models may recognize broad irony or playfulness yet misidentify the target, stance, mechanism, or interactional move.
Source: §1 · p. 2Takeaways & Limitations
Question writing and question solving provide complementary views of pragmatic understanding, and interpreting comments as detached text snippets can misrepresent stance or the force of a playful remark.
Source: §6 · p. 9The purposive, topic-guided collection targets information-rich cases and does not support claims about the prevalence of such comments across Chinese social media as a whole.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.
1 Introduction
Existing evaluations leave open whether models can distinguish plausible interpretations of naturally occurring comments in context. The paper introduces a human-validated benchmark and evaluates models’ ability to write and solve diagnostic questions.
1 Introduction
- Existing benchmarks often use predefined phenomena or task schemas, leaving unresolved whether models distinguish competing plausible readings of what a comment does in context.Two readings can share a broad label such as sarcasm or humor yet differ in target, speaker alignment, operative expression, or social action.Source: §1 · p. 2
- The benchmark contains 4,735 human-validated items drawn from more than 200,000 public Chinese social-media interaction records, pairing target comments with reconstructed context and plausible misreadings.The data and code are publicly available.Source: §1 · p. 2
- Eight LLMs generate diagnostic questions about likely misunderstandings, while models also solve human-validated questions written by other models.Human annotators filter and revise items, confirm gold answers, and write rationales; question writing and solving jointly operationalize situated pragmatic understanding.Source: §1 · p. 2
- 81.42% leave-writer-out accuracy for the strongest model, versus 68.70% mean accuracy across eight models and 90.8% human accuracy.Case analysis finds that models may recognize irony or playfulness but misidentify the target, stance, mechanism, or interactional move.Source: §1 · p. 2
2 Related Work
Related work covers nonliteral meaning, pragmatic and discourse understanding, and human-in-the-loop diagnostic benchmark construction. The paper distinguishes its focus on integrated readings of comments within particular interaction episodes.
2.1 Situated Interpretation for LLM benchmarks
- Phenomenon-based resources test labels such as sarcasm or implicit abuse, while newer benchmarks add dimensions such as stance, targets, or explanations; this benchmark asks for a coherent situated interpretation.Recognizing a category alone is insufficient because the interpretation must jointly account for dimensions of the comment in its exchange.Source: §2.1 · p. 3
2.2 Context in Pragmatic and Discourse Understanding
- Prior work uses context to resolve predefined judgments, whereas this benchmark treats the reconstructed interaction episode as the interpretation unit and tests dependence on preceding interaction.Component ablations examine evidence from the source post, direct parent, and local reply sequence.Source: §2.2 · p. 3
2.3 Diagnostic Benchmark Construction
- Diagnostic benchmarks use adversarial questions and plausible distractors, but situated social-pragmatic interpretation lacks a direct external anchor, motivating grounded episode construction followed by cross-model question generation and human validation.The procedure separates human construction of grounded episodes from generation and validation of diagnostic contrasts.Source: §2.3 · pp. 3–4
3 Formative Study
Researchers jointly reviewed more than 600 candidate comments with their posts and surrounding interactions to examine situated meaning, contextual evidence, and likely model misunderstandings. Their observations shaped a context-centered, human-led benchmark construction process.
3 Formative Study
- Four researchers reviewed more than 600 candidate comments case by case, discussing each comment’s function, supporting context, interpretive stability, indirectness or playfulness, and plausible misunderstandings.The topics encountered ranged from regional discrimination and education to economic issues; the list was not an exhaustive taxonomy or predefined sampling strata.Source: §3 · p. 4
- Many relevant comments appeared ordinary, vague, exaggerated, or nonsensical without the source post and preceding replies, so context became a required part of every benchmark instance.Platform choice also affected the kinds of indirect expression encountered, informing collection from Zhihu, Tieba, Douban, and Xiaohongshu.Source: §3 · p. 4
- LLM probing revealed variation in which interactional details models noticed, but many generated questions reduced the task to generic sentiment or sarcasm detection.This led to a two-stage process in which models propose diagnostic questions while humans construct episodes, gold labels, and final questions.Source: §3 · p. 4
4 Methodology
The methodology turns public Chinese social-media interactions into human-validated diagnostic questions and evaluates models in a cross-writer setting. It combines purposive collection, human screening, model-written questions, and human review.
4.1 Multi-platform Interaction Collection
- Researchers collected more than 200,000 public interaction records from four platforms and reconstructed available preceding context, including posts, thread material, parent comments, and local reply sequences.Collection used purposive, topic-guided searches for active interactions and potentially context-dependent replies.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
4.2 Human Filtering of Interaction Episodes
- Human screening applied context-dependence, recoverability, nonliteral-expression, and plausible-misunderstanding criteria to prioritized candidates, producing 3,200 interaction episodes.The review queue was prioritized with a lightweight supervised classifier trained on 5,000 human preference judgments; a second annotator’s exact agreement on 200 sampled episodes was 89.5% (179/200), with unweighted Cohen’s κ ≈0.77.Source: §4.2 · p. 5
4.3 Question Generation and Human Review
- Eight LLMs each wrote questions for assigned episodes, with two independently written questions per episode; human annotators filtered or revised candidates, confirmed gold answers, and wrote optional rationales.The same eight models were evaluated as solvers using leave-writer-out accuracy, which excludes questions proposed by the solver.Source: §4.3 · pp. 5–6, §4.4 · p. 6
- 4,735 of 6,395 analyzable candidate questions were retained or repairable (74.0%); 4,392 were retained without substantive modification (68.7%).A second annotator’s exact agreement on a random sample of 200 candidate questions was 91.0% (182/200), with unweighted Cohen’s κ ≈0.81.Source: §4.3 · p. 6
5 Results and Analysis
Cross-writer performance shows that the benchmark remains challenging, with accuracy varying by solver and question writer. Human answerability, context ablations, and case analyses further reveal difficult revised items, the value of preceding context, and several ways models misread situated comments.
5.1 Cross-writer Solver Performance
- GPT 5.5 scored 97.34% on its own questions but 62.95% leave-writer-out, while writer-source difficulty varied across models.Harder writer sources are not automatically better: difficulty may reflect sharper contrasts or less stable item formulation, and writer models surface different misunderstandings.Source: §5.1 · pp. 6–7
- 81.42% leave-writer-out accuracy was achieved by Qwen 3.5 9B; Llama 4 Maverick, Mistral Medium 3, and DeepSeek V4 Pro followed at 72.59%, 72.18%, and 71.77%.MiniMax M2.5, Kimi K2.6, GPT 5.5, and Claude Opus 4.7 ranged from 59.70% to 65.89%.Source: §5.1 · p. 6, Table 1 · p. 7
5.2 Human Answerability
- On the random 300-item audit, human accuracy was 90.8% (κ ≈0.85), versus 64.14% (κ ≈0.53) across all 343 human-revised items.Models scored 30.49%–39.63% on the human-revised subset, which the authors characterize as harder and less stable.Source: §5.2 · p. 7, §5.1 · p. 7
5.3 Context Ablations
- Mean leave-writer-out accuracy fell from 68.70% with full context to 29.63% with only the target comment, a 39.07-point decrease.For GPT 5.5, Llama 4 Maverick, and Qwen 3.5 9B, removing post-level context caused larger losses than removing the direct parent; all single-layer losses were smaller than deleting all preceding context.Source: §5.3 · p. 7
5.4 Case Failure Analysis
- Models sometimes sanitize uncomfortable meanings, recasting a comment’s cynical, ableist framing as praise for the achievement.The situated reading treats the news as socially absurd rather than praiseworthy.Source: §5.4 · pp. 7–8
- Recognizing wordplay does not ensure the right mechanism: Kimi’s plausible homophone reading misses the idiom 干柴烈火 invoked in context.The comment decomposes the idiom into 柴 and 火 to tease about the birth.Source: §5.4 · p. 8
- GPT overstates the force of 我佛只渡有钱人: the reply primarily invites amused recognition of an odd fit among religion, immigration, and wealth.The passage allows mild criticism but distinguishes it from a stronger moral critique.Source: §5.4 · p. 8
6 Discussion and Conclusion
The benchmark finds that question writing and solving offer complementary views of pragmatic understanding, while current models still miss situated meanings in authentic online discourse. Its protocol is designed to be rerun as online language changes.
6 Discussion and Conclusion
- Good diagnostic-question writing and question-solving ability do not coincide, making item writing a complementary probe that is harder to game than accuracy alone.Some models that write highly discriminative items are only middling solvers, and vice versa.Source: §6 · p. 8
- Treating comments as detachable snippets can lead agents to sanitize discriminatory or cynical stances, invent wordplay, or overstate light playfulness.Such pragmatic mistakes can produce misleading summaries, moderation decisions, or interactionally mismatched responses.Source: §6 · p. 9
- Separating human context reconstruction, cross-model question generation, and human validation makes the protocol rerunnable as memes and platform slang change.Human validation remains necessary because these answers are not objectively verifiable like math or coding questions.Source: §6 · p. 9
- Models may recognize irony or playfulness yet misidentify the target, stance, mechanism, or interactional move.The conclusion presents question writing and solving as complementary views of pragmatic understanding.Source: §6 · p. 9
Limitations
The benchmark is designed to collect challenging diagnostic cases, not to estimate how common such comments are or permanently rank models.
Limitations
- The purposive, topic-guided collection strategy finds information-rich cases but cannot establish the prevalence of such comments across Chinese social media.Platform and topic distributions reflect the goal of constructing challenging diagnostic items.Source: Limitations · p. 9
- The benchmark excludes video-based interaction and focuses on text-based and screenshot-mediated Chinese online discourse.The authors identify multimodal platforms as a direction for extending the protocol.Source: Limitations · p. 9
- The exact-overlap audit cannot detect indirect exposure to related content, so it does not establish that evaluated models were entirely free from contamination.The audit queried target-comment strings against Dolma and suggests large-scale verbatim memorization alone is unlikely to explain overall performance or model ranking.Source: Limitations · p. 9
- Reported results are tied to particular model versions and access dates, so the benchmark is a diagnostic testbed under specified conditions rather than a permanent capability ranking.Source: Limitations · p. 9
Ethical Considerations
The study uses public, socially sensitive material and a staged annotation workflow with trained annotators to construct and review diagnostic items.
Ethical Considerations
- The study used publicly visible posts and threads without participant intervention, private messages, or profile information, while treating socially sensitive content as research data rather than endorsed speech.Source: Ethical Considerations · p. 10
A.1 Annotator background
- Seven trained annotators were native Chinese speakers, frequent social-media users, and experienced annotators; collectively, they covered four source platforms.The annotators were anonymized as A1–A7.Source: §A.1 · p. 14
A.2 Calibration
- Annotators calibrated on 30 shared examples, including borderline cases where context did not support a stable interpretation.Source: §A.2 · p. 14
A.3 Preference labeling for triage
- Five annotators labeled 5,000 examples as preference data solely to train the review-ordering classifier.Positive labels marked candidates likely to contain context dependence, indirect or playful language, social positioning, or plausible model misunderstandings.Source: §A.3 · p. 14
A.4 Human screening of interaction episodes
- After classifier-assisted triage, annotators screened more than 12,000 candidate comments and retained 3,200 human-built interaction episodes.Retained cases needed a recoverable context-dependent pragmatic reading, nonliteral expression, and a plausible route of model misunderstanding; uncertain cases were discussed by the group.Source: §A.4 · p. 14
A.5 Diagnostic question review
- Seven annotators reviewed 6,400 model-written questions for context dependence, gold-answer fit, plausible distractors, and a single best answer.Questions could be accepted, revised, or rejected; rejection reasons included context-free answerability, hallucinated details, ambiguity, unsupported answers, or weak distractors.Source: §A.5 · p. 14
A.6 Stage-level reliability audits
The appendix reports agreement at three annotation stages, describes data collection across four platforms, and explains classifier-assisted triage.
A.6 Stage-level reliability audits
- Agreement was 86.0% (κ ≈0.71) for preference labeling, 89.5% (κ ≈ 0.77) for episode screening, and 91.0% (κ ≈0.81) for candidate-question validation.Each of the three reliability audits used N = 200 records.Source: §A.6 · p. 15
B Platform Collection
- Collection covered Zhihu answer-level discussions, Tieba public threads, Douban topics, and Xiaohongshu notes, retaining public posts and comment threads visible at collection.The logs report 149 Zhihu questions, 155 Tieba threads, 158 Douban topics, and 252 Xiaohongshu notes.Source: B Platform Collection · p. 15
C Classifier Assisted Triage
- A supervised classifier ranked cleaned comments into high-priority, uncertain, and low-priority review pools to prioritize likely candidates and remove obvious low-value cases.It used character-level TF-IDF features from target comments and reconstructed context, plus surface and interaction features; held-out discussions reduced leakage between comments under the same source discussion.Source: C Classifier Assisted Triage · pp. 15–16
D Model Prompts and Evaluation Settings
The appendix specifies how models generate diagnostic questions and answer them under full-context and comment-only conditions, along with prompt settings and answer extraction.
D Model Prompts and Evaluation Settings
- The appendix provides English translations of Chinese prompts; writer calls use temperature 0.2 and solver calls 0.0, with one completion per item and no reranking or multi-sample decoding.Both stages use top-p 1.0; their maximum outputs are 2,048 tokens for writers and 64 for solvers.Source: D Model Prompts and Evaluation Settings · p. 16
D.1 Candidate Question Writer Prompt
- Writer models receive an interaction episode and target comment, identify a likely misunderstanding, then write a diagnostic question.The question should test implied meaning, interactional function, or expressive effect in context.Source: §D.1 · p. 16
- Writer instructions require context-dependent questions with one best answer and plausible misreadings, rather than generic sarcasm or sentiment judgments.The prompt targets such distinctions as criticism, mockery, alignment, humor mechanism, and interactional function.Source: §D.1 · p. 16
- The requested writer output includes a misunderstanding point, a five-choice question, an answer, and a short rationale for the selected option.The rationale explains why the selected option is best.Source: §D.1 · pp. 16–17
D.2 Full-Context Solver Prompt
- The full-context solver receives preceding context, the target comment, question, and options, and returns one option letter without explanation.The solver is asked to identify the comment’s implied meaning, interactional function, or expressive effect.Source: §D.2 · p. 17
D.3 Comment-Only Solver Prompt
- The comment-only ablation removes preceding context but retains the target comment, question, and options for the solver.The solver answers based only on the target comment and returns one option letter.Source: §D.3 · p. 17
D.4 Answer Extraction
- A deterministic extraction rule normalizes outputs, accepts a single valid option letter or an explicitly marked final answer, and counts unresolved outputs as incorrect.Outputs with no valid option or conflicting unmarked letters are counted incorrect.Source: §D.4 · p. 17
E Detailed Context-Ablation Results
The context ablations compare full-context and comment-only performance, then examine the sensitivity of selected models to removing individual context layers.
E Detailed Context-Ablation Results
- 39.07 percentage points was the mean LWO accuracy loss when context was removed; every model lost at least 32.91 points.The comparison is full context versus the target comment alone; the table reports LWO accuracy in percent and absolute percentage-point drop.Source: E Detailed Context-Ablation Results · p. 17, Table 4 · p. 17
- The paired leave-one-component-out analysis reports net correct-answer losses against full-context predictions on the same 4,735 items for three selected models.GPT 5.5, Llama 4 Maverick, and Qwen 3.5 9B were selected to span three model families and a range of primary LWO scores.Source: E Detailed Context-Ablation Results · p. 17, Table 5 · p. 18
- Larger post-level losses indicate greater sensitivity under that deletion, while smaller direct-parent losses do not show that parent comments are dispensable.The context layers contain overlapping information, so a remaining layer can compensate for one that is removed.Source: E Detailed Context-Ablation Results · pp. 17–18
F Exact-Overlap Audit
The audit checks exact target-comment overlap with Dolma as one possible explanation for model rankings.
F Exact-Overlap Audit
- 1.59% of queried target comments had any exact match in Dolma, falling to 0.26% among successfully queried comments with at least six Chinese characters.The audit queried 3,200 episodes; 3,198 queries succeeded, and the restricted subset had 8 matches among 3,095 comments.Source: F Exact-Overlap Audit · p. 18, Table 6 · p. 18