Source-linked AI summary
Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness
Yihang Peng, Yi Jing, Dewen Liu, Weihan Meng, Ying Liu
TL;DR
Whether a recurring expression is a model tic depends on whether the context calls for it, a question that frequency-based research has not resolved. The paper applies a decision tree to Mandarin corrective negation and finds that optional uses are far more common than defective ones. It also reports that evaluators and posttraining can favor or increase use of the construction.
Problem
Frequency-based studies of recurring expressions cannot distinguish appropriate from inappropriate uses, leaving contextual appropriateness in model production largely unexamined.
Source: §1 · pp. 1–2Method
A theory-based decision tree classifies Mandarin corrective-negation uses as substantive, optional, or defective according to contextual licensing and informative contribution.
Source: §3 · p. 3Results
Only 1.30% of 307 annotated uses were defective and 52.8% optional, while five commercial models used the construction 2.5 to 9.2 times as densely as a human reference.
Source: §1 · p. 2Takeaways & Limitations
Because suppressing the form would also remove appropriate uses, reducing such tics requires evaluation and training signals that register whether an expression is called for.
Source: §5 · p. 8The study assesses appropriateness through linguistic criteria, not reader perception; effects on annoyance or comprehension require a separate user study.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Recurring expressions in model-written text are often treated as signatures of machine authorship, yet most of them serve legitimate functions. Whether such an expression is a tic depends on whether the model knows when it is called for. This judgment of appropriateness, central to communicative competence, remains largely unexamined for language models. We study it through the Mandarin corrective construction bushi X, ershi Y ("not X, but Y"). Drawing on Gricean pragmatics and corrective negation, we introduce a decision tree that classifies uses as substantive, optional, or defective by testing whether the context makes X a live alternative and whether rejecting X is informative. Five commercial models use the construction 2.5 to 9.2 times as densely as a human reference. Of 307 annotated uses, only 1.30% are defective while 52.8% are optional, that is, correct but unneeded, so models fall short in appropriateness far more often than in correctness. Several reward models favor the construction even over quality-matched variants, an LLM judge favors it even where the context does not license it, and both supervised fine-tuning and preference optimization tend to raise its frequency in open-weight models. We hope this work encourages the development of models attentive to the appropriateness of their expressions.
1 Introduction
A recurring expression is a tic not simply because it is frequent, but when a model uses it without contextual need. The paper studies this distinction through Mandarin corrective negation, finding that optional uses are much more common than defective ones.
1 Introduction
- Frequency-based analysis can conflate appropriate and inappropriate uses, while suppressing a legitimate expression removes both kinds.The paper frames contextual appropriateness as distinct from grammatical possibility and production frequency.Source: §1 · pp. 1–2
- The diagnostic separates substantive, optional, and defective uses by asking whether X is a live alternative and whether rejecting it adds information.A correction can be well formed but unneeded, or defective because it rejects something true or misattributes a view.Source: §1 · p. 2
- Corrective negation can correct a misconception, but the rejected X may also be a position nobody has raised, making contextual licensing central.The construction is assessed for whether X is accessible as an alternative and whether rejecting it contributes information not already established.Source: §1 · p. 2
- Only 1.30% of 307 annotated uses were defective, while 52.8% were optional; models used the construction 2.5 to 9.2 times as densely as the human reference.The authors report that shortfalls concern appropriateness more than correctness.Source: §1 · p. 2
- The study formalizes stylistic tics as failures of communicative appropriateness and proposes diagnosing how evaluators and training stages shape preferences for them.Its framework is grounded in linguistic theory and distinguishes uses that are correct from those that are needed.Source: §1 · p. 2
2 Related Work
Prior work studies model pragmatics, characteristic writing, corrective negation, and evaluator bias largely in separate strands. This paper draws on those strands to assess when a recurring expression is contextually appropriate.
2 Related Work
- The paper’s related-work framing links overproduction research, linguistic accounts of corrective negation, and evaluator-bias research as separate foundations for its analysis.These lines respectively identify recurring expressions, specify when the construction is appropriate, and suggest where preferences may originate.Source: §2 · p. 2
2.1 Language Models as Speakers
- Research on model speakers measures overrepresented expressions and proposes suppression, while listener-side pragmatic tests examine comprehension rather than open-ended production.These approaches establish what models produce too often but do not directly assess whether a use is called for.Source: §2.1 · pp. 2–3
2.2 Corrective Negation
- Accounts of English and Mandarin corrective negation describe correction of alternatives, including pseudo-denial and metalinguistic uses, and specify conditions for appropriate use.The cited work distinguishes corrective meaning from contrast and adversative relations.Source: §2.2 · p. 3
2.3 Evaluator Bias
- Reward models and LLM judges have documented preferences for length, formatting, and style, while preference optimization can narrow output variety.Released intermediate checkpoints make it possible to trace changes across posttraining stages.Source: §2.3 · p. 3
3 Methods
The methods assess contextual appropriateness rather than frequency alone, using a theory-based decision tree and analyses of generation, evaluation, and training. The framework classifies each use and pairs those labels with measures of construction frequency and evaluator preference.
3 Methods
- The framework labels each occurrence substantive, optional, or defective, then pairs those labels with frequency, evaluator comparisons, and checkpoint analyses.Its decision tree is derived from corrective negation and Gricean pragmatics.Source: §3 · p. 3
3.1 Analytical Criteria
- Appropriateness is tested through corrective meaning and discourse contribution: alternatives must share a recoverable contrast, and the correction must advance the open question.Separate tests address contextual licensing and informative contribution, with interpretability also considered.Source: §3.1 · p. 3
3.2 An Evidence-Based Diagnostic Tree
- The decision tree first screens for defects, then tests corrective function; the first positive defect test yields defective, while a failed function test yields optional.The defect tests cover factual conflict, misattributed positions, and inappropriate rejection or disrupted interpretation; function tests cover rejection-and-replacement, contextual motivation, and advancing the open distinction.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 Frequency and Function Measurement
- Frequency measurement separates activation, intensity, and log frequency, with occurrence rates normalized by answer length and missing answers excluded.Human–model comparisons average activation equally across four shared broad dimensions; full generation and checkpoint analyses average activation and intensity across four benchmarks.Source: §3.3 · p. 4
3.4 Experimental Framework
- Evaluator experiments compare four minimally edited variants and use functional sensitivity to contrast preferences for substantive uses with unlicensed or optional uses.Quality matching retains texts scored equally on applicable dimensions except construction clarity; checkpoint comparisons follow parent–child relations, and preference-data audits compare context-matched chosen and rejected answers.Source: §3.4 · pp. 4–5
4 Experiments
Across four shared task dimensions, the experiments compare construction use in five API models with a human reference, assess evaluator preferences, and examine training-stage patterns and case studies. Models use the construction more densely, while many uses are optional; evaluators also favor it in contexts where it is unlicensed.
4.1 Frequency and Function
- 2.5–9.2 times the human-reference density, with 52.8% of 307 annotated model uses optional and only four defective.Among human uses, 63.6% were substantive and 33.3% optional, compared with 45.9% and 52.8% in model answers.Source: Figure 3 · p. 5, §4.1 · pp. 5–6
4.2 Evaluator Preference
- 68.3–76.8% of non-tied optional-use comparisons favored the original order for FsfairX, Skywork, and OffsetBias.ArmoRM showed a weaker preference, while InternLM2 favored the reordered version; the three reward models’ ordering preference persisted at 73.1–78.3% under quality-score matching.Source: Figure 4 · p. 6, §4.2 · pp. 6–7
- 0.630 and 0.563 were Claude Sonnet 5.5’s mean preferences for unlicensed originals over affirmative-only and reordered answers, respectively.GPT-6.1-sol showed weaker, directionally consistent preferences, and neither judge’s preference varied with corrective function.Source: Figure 4 · p. 6, §4.2 · p. 7
4.4 Case Study
- In the Messi example, the judge acknowledged the correction was “slightly odd logically” yet preferred it for reading smoothly and passionately.The judge favored the original in every judgment for both case-study answers except one tie; FsfairX and Skywork also scored the originals above the reordered versions in both cases.Source: Table 1 · p. 7, §4.4 · p. 8
4.3 Training Stages and Data Attribution
- 0–1.41 occurrences per 10,000 Han characters was the range for open checkpoints, versus 0.942 for humans and 2.38–8.68 for API models.Ten of twelve open checkpoints were below 0.500.Source: §4.3 · p. 7, Figure 5 · p. 8
- All five families had higher construction density at SFT than Base, and both available DPO checkpoints exceeded their SFT predecessors.Chosen answers also showed higher relative activation than rejected answers in both preference datasets: +14.7% in Tülu 3 and +26.5% in OLMo 2; the enrichment’s magnitude depends on corpus composition and repetition.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
5 Discussion and Conclusion
The discussion separates frequency from appropriateness: many model uses are correct but not called for, and evaluators do not reliably supply the missing judgment. The authors argue that evaluation and training signals should register contextual appropriateness, and propose a controlled follow-up to test attribution.
5 Discussion and Conclusion
- Frequency and appropriateness are separate properties, and suppressing the form would remove appropriate uses along with the rest.The authors conclude that reducing such tics requires evaluation and training signals that register whether an expression is called for; the diagnostic can also apply to English “it’s not X, it’s Y” and other context-dependent expressions.Source: §5 · p. 8
- A controlled follow-up would hold initialization, source composition, update budget, and prompts fixed while intervening on matched construction-bearing preference pairs across seeds.Measuring activation, repetition, and local function would distinguish changes in selection frequency from changes in contribution.Source: §5 · p. 8
Limitations
The study’s estimates are bounded by its sampled models, domains, and annotated uses, and its training comparisons are observational. It assesses linguistic appropriateness for one Mandarin construction, not reader reactions or other constructions.
Limitations
- The analysis covers five commercial models, a limited set of task domains, and 307 annotated uses; broader models, domains, and human data would strengthen the estimates.Source: Limitations · p. 8
- Checkpoint and preference-data comparisons are observational, so they show frequency rising along training pipelines but not which stage or data causes the rise.Controlled retraining is needed to test that attribution.Source: Limitations · pp. 8–9
- The study evaluates appropriateness through linguistic criteria, not reader perception; whether unneeded uses cause annoyance or affect comprehension requires a user study.Source: Limitations · p. 9
- The analysis covers only Mandarin bushi X, ershi Y; although the decision tree is designed to transfer to English and other expressions, it has not been applied beyond this construction.Source: Limitations · p. 9
Ethical Considerations
The paper describes potential misuse of its diagnostic framework and frequency measures, while limiting its intended application to aggregate appropriateness analysis. It also reports research-use licensing, privacy, and AI-assistance practices.
Ethical Considerations
- The decision tree could help make model text harder to identify, while frequency measures could be misused to judge human writers; aggregate analyses are not validated for individual authorship attribution.Source: Ethical Considerations · p. 9
- Benchmarks, checkpoints, reward models, and preference mixtures are publicly released for research and used under their licenses; commercial models are accessed under providers’ terms.Materials are used solely for scientific research.Source: Ethical Considerations · p. 9
- The human reference uses public datasets with personally identifiable information removed; annotators participated voluntarily, and no personal information beyond annotations was collected.Source: Ethical Considerations · p. 9
- The framework is intended for contextual-appropriateness analysis and evaluator or training-stage study, not authorship detection or evaluation of individual writers.Source: Ethical Considerations · p. 9
- The authors will release the decision-tree code and annotation guidelines, documenting their scope and procedures.Source: Ethical Considerations · p. 9
- Cursor assisted code development, and an LLM helped refine the paper’s language; LLM annotators and judges are described as part of the method.Source: Ethical Considerations · p. 9
A Measurement and Annotation
The measurement setup compares model and human construction rates across shared tasks, using explicit corpus, detector, and averaging choices. The annotation sample is supplemented by uncertainty estimation and audits of training data and checkpoints.
A Measurement and Annotation
- Human references combine community answers, examination responses, student essays, and role-play dialogue from Chinese-language datasets and examination sources.Source: A Measurement and Annotation · p. 13
- The main human baseline averages Han-character density equally over four shared dimensions, yielding 0.942 occurrences per 10,000 characters; pooling all eight dimensions yields 0.664.Source: A Measurement and Annotation · p. 13
- The five-model panel has 5,285 valid answers to 1,061 prompts drawn from four benchmarks; the human corpus contains 572,412 Han characters and 38 occurrences across eight dimensions.Source: A Measurement and Annotation · p. 13
- The fixed detector uses literal matches with specified punctuation and span constraints; it can miss nested starts and comma-free forms and produce spurious detections.Source: A Measurement and Annotation · p. 13
- Density is averaged equally across the four shared tasks, and within a corpus decomposes into activation, occurrences per activated answer, and mean answer length.Full-panel uncertainty uses 5,000 prompt-cluster bootstraps stratified by benchmark; between-model tests use common prompts and Holm correction.Source: A Measurement and Annotation · p. 13
A.1 Functional Annotation and Human Comparison
Functional annotation classifies uses by corrective function and contextual contribution, with most non-substantive cases failing contextual licensing or issue advancement. Human-authored uses are more often substantive, while fine-grained annotation agreement is moderate.
A.1 Functional Annotation and Human Comparison
- The human-reference table defines IH as occurrences per 10,000 Han characters and uses the four-shared-dimension macro average of 0.942 as the main comparison baseline.The all-eight-dimension pooled estimate is 0.664.Source: Table 2 · p. 14
- The five-model table reports valid-response counts, benchmark-averaged activation and Unicode-character density, and four-dimension Han-character density relative to the human baseline.Source: Table 3 · p. 14
- Of 307 annotated cases, 141 were substantive, 162 optional, and four defective; 160 of 166 non-substantive uses failed contextual licensing or issue advancement.The sample excludes one occurrence after an annotation-service refusal and should not be treated as a precise population estimate.Source: §A.1 · p. 15, Table 4 · p. 15
- Two independent linguistics-trained annotators showed moderate κ agreement, with greater difficulty at the seven-label terminal level, especially distinguishing lack of licensing from no discourse advancement.Source: §A.1 · p. 15, Table 5 · p. 15
- Among 33 eligible human-authored occurrences, 21 were substantive, four unlicensed, seven redundant, and one defective; these labels were automated and did not receive the model-sample review.Five quoted or metalinguistic occurrences were excluded before classification.Source: §A.1 · p. 15
- Substantive corrections appeared earlier than unlicensed ones (median positions 0.341 versus 0.544, p < .001), while redundant corrections appeared latest at 0.862.These are descriptive associations and do not enter evaluator tests.Source: §A.1 · p. 15
B Evaluator Design and Supplementary Results
The evaluator analyses compare construction-preserving variants with edited alternatives using reward models and pairwise LLM judgments. Reward models show positive preference within optional and unlicensed uses, while judge preferences show little functional sensitivity.
Variants and reward models
- The comparisons use original, reordered, and affirmative-only variants; the cross-sentence variant is excluded from reported comparisons.Reward scorers receive the question and full answer without functional or variant labels; coverage ranges from 252 complete groups to 305, depending on scorer.Source: B Evaluator Design and Supplementary… · p. 15
General-quality matching
- Quality matching retains unchanged texts with equal scores on every applicable general-quality dimension, excluding construction clarity, and does not condition on functional category.Repeated scores have 78.5% exact agreement, 96.2% agreement within one point, and weighted κ = .822.Source: B Evaluator Design and Supplementary… · p. 16
Pairwise judgment
- Pairwise judges assess correctness, helpfulness, relevance, concision, completeness, and applicable creativity, averaging ordinal preferences across two presentation orders.Strong/weak preference, tie, and counterpart preference map to 1, .75, .5, .25, and 0 after variant identity is restored.Source: B Evaluator Design and Supplementary… · p. 16
- The judge analyses cover 301 eligible groups, excluding defective, relation-failure, and unresolved cases, with bootstrap intervals for preference and functional-sensitivity estimates.Reward-model win rates use Wilson 95% intervals; judge preference and sensitivity intervals use 10,000 item bootstraps.Source: B Evaluator Design and Supplementary… · p. 16
Functional sensitivity of the judges
- Claude and GPT prefer retaining unlicensed corrections over affirmative-only answers at 0.630 (95% CI [0.603, 0.656]) and 0.519 ([0.505, 0.535]), respectively.Substantive-minus-unlicensed and substantive-minus-pooled-optional differences range from −0.002 to 0.006 across edits, and all eight 95% intervals include zero.Source: B Evaluator Design and Supplementary… · p. 16
Functional differences across reward models
- Reward-model functional sensitivity is .00194 for reordering (95% interval [−.00924, .0125]) and .0154 for affirmative-only editing ([.000439, .0301]).The affirmative-only comparison removes corrective content as well as its realization; the main claim rests on positive preference within optional and unlicensed uses.Source: B Evaluator Design and Supplementary… · p. 16
C Checkpoint and Preference-Data Protocols
The checkpoint and preference-data analyses describe construction density across training stages and chosen/rejected answer pairs. Their interpretation is bounded by retained-output and corpus-specific protocols, and density changes can reflect length and repetition as well as construction selection.
Checkpoint panel
- The checkpoint panel uses 12 Base, SFT, and DPO checkpoints on the same 1,061 prompts as the API panel, yielding 12,518 valid answers.Base checkpoints use a neutral question–answer frame; chat checkpoints use native templates without an added system message.Source: C Checkpoint and Preference-Data… · p. 16
Checkpoint protocol
- Checkpoint density averages valid-answer density equally over four benchmarks, retaining nonempty truncated answers and excluding empty or overlength records.Decoding temperatures are .1 for 725 prompts and .7 for 336; the context cap is the smaller of the native limit and 32,768 tokens.Source: C Checkpoint and Preference-Data… · p. 16
Checkpoint results
- Figure 6 plots benchmark-macro log-frequency, LFC = log10(1 + IC), by family; lines connect available stages and panel vertical scales differ.Table 8 reports paired parent–child density changes on common successful prompts, which need not equal differences between marginal checkpoint points.Source: Figure 6 · p. 17, Table 8 · p. 17
- Density reflects completion length and repeated use as well as construction selection, and excluding truncated answers reverses the OLMo Base–SFT direction.Concentrated repetition contributes substantially to MiMo’s density; the displayed trajectories describe the retained-output protocol.Source: C Checkpoint and Preference-Data… · p. 18
Preference-data protocol
- The preference-data audit uses final assistant answers from 648,868 valid context-matched chosen/rejected pairs, and chosen answers show higher construction activation in both datasets.The pairs comprise 271,958 Tülu and 376,910 OLMo examples; the reported activation rates are corpus-pooled, not benchmark-macro estimates.Source: C Checkpoint and Preference-Data… · p. 18
Preference-data results
- Paired conditional-logit models give construction odds ratios of .947 for Tülu (95% CI [.673, 1.332]) and 1.040 for OLMo ([.839, 1.289]).One Tülu chosen answer supplies 41 of 145 occurrences; removing its prompt cluster reduces the intensity difference by 73.8%, while OLMo chosen answers repeat it less within activated answers: 1.213 versus 1.360 instances.Source: C Checkpoint and Preference-Data… · p. 18
D Implementation Details
The implementation used local GPU runs for checkpoint generation and reward-model scoring, while commercial models and judges were accessed through provider APIs. The preference-data table reports higher construction activation in chosen answers in both datasets.
Implementation
- Local checkpoint and reward-model work used 8.35 A800 GPU hours and 17.07 H800 GPU hours; commercial models, the annotation model, and judges were accessed through provider APIs.Checkpoint generation used vLLM 0.11.0; reward scoring used Transformers 4.56.1, PyTorch 2.8.0, and CUDA 12.8.Source: D Implementation Details · p. 18
Preference-data results
- Chosen answers consistently show higher construction activation than rejected answers in both public preference datasets, with a larger gap in OLMo 2.Table 9 uses complete context-matched pairs and final assistant responses; its rates are corpus-pooled rather than benchmark-macro estimates.Source: Table 9 · p. 19