CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing

1 Introduction

• Existing evaluations have examined downstream tasks and generation quality, but contextually appropriate, functionally motivated switching remains comparatively underexplored.Prior work includes input-oriented NLP evaluations, fluency and fidelity assessments, and comparisons with naturally occurring patterns.Source: §1 · p. 2
• Creative writing makes individual switches and their broader narrative roles observable through characters’ resources, relationships, circumstances, and developing interactions.A switch can be judged by its fit with a character’s established abilities, the exchange, and its contribution to the narrative.Source: §1 · p. 2
• CSWBench evaluates context-sensitive code-switching in creative writing through Focused Narrative, emphasizing local choices, and Extended Narrative, emphasizing multilingual organization across a text.Its source material combines spontaneous and controlled Chinese–English records transformed into structured prompts with interactional grounding and limited narrative scaffolding.Source: §1 · p. 2
• Models differ especially in multilingual literary integration, multilingual-focused dimensions show stronger difficulty effects than general narrative dimensions, and more salient switches tend to have stronger contextual and functional realization.Event-level code-switching quality is also moderately associated with whole-text performance.Source: §1 · p. 2
• CSWEval separates event-level Licensing, Contextual Appropriateness, and Functional Gain from task-specific whole-text quality.Whole-text assessment includes literary and narrative quality in Focused Narrative and multilingual organization in Extended Narrative.Source: §1 · p. 3

2 Related Work

• Input-oriented evaluations cover downstream prediction and understanding, while LLM studies extend to translation, reasoning, and comprehension.Generation-oriented work considers controllability, fluency, semantic fidelity, human judgments, and correspondence with naturally occurring switching patterns.Source: §2 · p. 3
• Recent code-switching studies examine language-choice conditions through bilingual minimal pairs, pragmatic cues, and community-specific conventions.This work extends that focus to contextual and narrative roles in creative writing.Source: §2 · p. 3
• Creative-writing evaluation measures coherence, engagement, creativity, literary quality, writing quality, character consistency, event retention, and cross-scene continuity.This study examines local switching functions and multilingual development across characters, relationships, and narrative states.Source: §2 · p. 3
• Sociolinguistic accounts connect code-switching to discourse position, speaker alignment, social distance, identity, and stance, while literary work treats it as a staged narrative resource.Literary perspectives link switching with characterization, relationships, and textual organization.Source: §2 · p. 3

3.1 Data Source and Seed Construction

• CSWBench combines naturally occurring Chinese–English conversations and a controlled code-switching corpus, screening records for interpretable context and attributable switching.Of 3,500 sampled records, screening and sensitive-content filtering retained 3,009 for Seed construction; validated Seeds were selected for contextual diversity.Source: §3.1 · pp. 3–4

3.2 Theoretical Foundations of CSWEval

• The framework grounds code-switching evaluation in interactional and literary theories and distinguishes event-level constructs from whole-text multilingual orchestration.Its theory review retained 107 works and consolidated 70 candidate concepts into six preliminary constructs.Source: §3.2 · pp. 4–5

3.3 Task and Prompt Construction

• The two settings use coverage-first sampling of validated Seeds and Seed-conditioned prompts, with Extended adding narrative-development guidance.Focused targets about 700 characters to establish a situation; Extended targets about 4,000 and requires evolving characters, relationships, narrative states, and multilingual behavior.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.2 Prompt Validation

• Two independent human reviews, with third-rater adjudication for disagreements, retained 807 of 855 candidate prompts: 407 Focused and 400 Extended.Review agreement was Gwet’s AC1 = .975 for Focused and .838 for Extended.Source: §3.3.2 · p. 5

3.4 CSWEval: Evaluation Framework

• CSWEval scores Licensing, Contextual Appropriateness, and Functional Gain at event level, alongside task-specific whole-text quality.Licensing uses binary checks aggregated as 5× the pass rate; the other dimensions use 1–5 ordinal rubrics.Source: §3.4 · p. 5, §3.4.1 · p. 5, §3.4.2 · p. 5

3.4.1 Code-Switching Event Annotation

• Event annotation fixes candidate-span boundaries before quality scoring and records character, language, relationship, narrative-world, and Licensing evidence for Extended narratives.Focused scores applicable Contextual Appropriateness and Functional Gain subdimensions per event; Extended uses them as rubric guidance for an overall dimension score.Source: §3.4.1 · p. 5, §3.4.2 · p. 5

3.4.2 Task-Specific Multi-Level Evaluation

• Whole-text rubrics differ by task: Focused assesses literary and narrative quality, while Extended scores multilingual organization, character language development, and general narrative dimensions.Extended includes multilingual narrative-system stability, two focal-character profiles, three general narrative dimensions, and multilingual literary integration.Source: §3.4.2 · p. 6

3.4.3 Narrative Salience and Score Aggregation

• Contextual Appropriateness and Functional Gain are aggregated across events using narrative-salience weights of 1, 2, and 3; whole-text scoring is computed independently.Within each setting, the reported composite adds D1, D2, D3, and Whole with equal weights.Source: §3.4.3 · p. 6

4 Experiment Setup

• The evaluation covers twelve contemporary LLMs from proprietary and open-weight model families, using temperature 0.7, top-p 0.7, and a 30,000-token output limit.The full roster and generation-side reasoning configurations are reported in Appendix M.Source: §4 · p. 6
• The test set contains 407 Focused Narrative prompts and 400 Extended Narrative prompts, with one narrative generated per model and prompt.Generation-model identities were hidden from evaluators; each narrative was independently assessed by GPT-5.6-Sol and DeepSeek-V4-Flash, and their article-level Licensing scores were averaged for model-level reporting.Source: §4 · p. 6

5.1.1 Human Validation

• Human ratings aligned with dual-judge scores: Kendall’s W = .718 (Focused) and .810 (Extended), with ICC(A,1) = .864/.816 and Spearman’s ρ = .863/.804.The ratings covered 180 Focused and 144 Extended outputs.Source: §5.1.1 · p. 6
• Audits yielded candidate-classification F1 .963/.960 and event-grouping F1 .944/.998 (Focused/Extended); prompt-constructor and annotator changes broadly preserved rankings.Source: §5.1.1 · p. 6

5 Results and Analysis

• Σ = 15.537 for GPT-5.5 in Focused Narrative, while GPT-5.6-Sol leads Extended Narrative at Σ = 17.767.Both remain top-ranked under evaluator-family exclusion and alternative aggregations.Source: §5 · p. 6, Table 2 · p. 7

5.1.2 Construct Sensitivity

• HIGH-D3 events more often favored original code-switching, whereas LOW-D3 events produced more ties against human-reviewed Chinese-only alternatives.Source: Table 3 · p. 7, §5.1.2 · p. 7
• At the strongest swap level, Whole-text scores fell from 3.702 to 2.901 under CS swaps and 3.137 under Chinese-span swaps.CS swaps primarily disrupted multilingual organization; Chinese-span swaps had relatively greater effects on general narrative execution.Source: §5.1.2 · p. 7

5.2 Multilingual Integration as a Major Axis of Model Differentiation

• Multilingual Literary Integration had the greatest variation across model means (SD = 0.742), and multilingual-focused dimensions declined more on harder prompts than general narrative dimensions.Source: §5.2 · p. 7

5.3 Narrative Salience and Event-Level Quality

• Salience correlated with D2/D3 at r = .642/.496 (Focused, 4,884 articles) and .768/.789 (Extended, 4,800), with larger correlations in Extended.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 From Local Code-Switching Quality to Whole-Text Performance

• In Extended Narrative, D2 and D3 correlated with Whole-text performance at r = .254 and .255; CS and multilingual-organization dimensions associated more strongly than general narrative dimensions.All Whole-text subdimensions remained positively associated with event-level code-switching quality.Source: §5.4 · p. 8

• CSWBench evaluates context-sensitive code-switching in creative writing, while CSWEval assesses events and task-specific whole-text quality.Source: §6 · p. 8
• Models differ substantially in sustaining multilingual choices within an evolving narrative, beyond producing locally plausible switches.Source: §6 · p. 8
• Individual switch quality is positively associated with narrative importance and moderately associated with whole-text performance.The paper argues that event-level and whole-text evaluation provide complementary and convergent evidence.Source: §6 · pp. 8–9

• CSWBench covers Chinese-dominant narratives with Latin-script English insertions, not English-dominant text or other language pairs; targets of about 700 and 4,000 non-whitespace code points exclude long-form writing.Source: Limitations · p. 9
• GPT-5.6-Sol contributes to prompt construction, event annotation, and judging; GPT-side judges rate GPT-family Focused outputs about 0.5 points (/20) more favorably relative to human ratings.Changing the event annotator shifts Extended scores by a pooled mean of +0.745 points (/20), despite broadly stable rankings.Source: Limitations · p. 9
• About 82% of final prompts (661 of 807) derive from synthetic SwitchLingua; generalization to naturally occurring interaction requires separate investigation.Seeds are expanded with model-generated narrative scaffolding, including participant profiles and language backgrounds.Source: Limitations · p. 9

• CSWBench is designed for generated Chinese–English creative writing, not for evaluating real bilingual speakers’ linguistic competence.Source: Ethical Considerations · p. 9
• Constructed narratives and LLM-based judgments may reflect limited sociolinguistic norms or cultural stereotypes.Source: Ethical Considerations · p. 9
• Source screening excludes records flagged for personal identifiers or sensitive content; the benchmark is not intended for profiling or evaluating individual speakers.Source: Ethical Considerations · p. 9

A Details of Experimental Setup

• Twelve models were evaluated, with one intended narrative per model and prompt; generation identities were hidden, and GPT-5.6-Sol and DeepSeek-V4-Flash were the primary judges.The roster includes GPT, Gemini, Qwen, DeepSeek, and MiniMax models.Source: A Details of Experimental Setup · pp. 11–12

A.1 Generation Configuration

• Generation used temperature 0.7, top-p 0.7, a 30,000-token output budget, and target lengths of about 700 and 4,000 non-whitespace code points for Focused and Extended narratives.Generation-side reasoning was active for all models, with model-specific configurations.Source: §A.1 · p. 12

A.2 Scoring Configuration

• Judges applied the CSWEval rubrics; article-level scores were averaged across the two judges before model-level aggregation.For Licensing, article-level D1 is five times the proportion of valid events passing both participant- and world-level checks.Source: §A.2 · p. 12

B.1 Units and Source Normalization

• ASCEND provides spontaneous spoken Chinese–English interactions, while SwitchLingua provides controlled synthetic text; both are normalized into one record pool.A source row, conversational turn, window, Seed, and task Prompt are distinct units.Source: §B.1 · p. 12

B.2 Final-Panel Coverage Audit

• The 807 final prompts link to 546 distinct Seeds from 596 validated Seeds; the tasks share 261 Seeds, with 146 Focused-only and 139 Extended-only Seeds.The shared Seed overlap has a Jaccard value of .478.Source: §B.2 · p. 12
• Focused omits one recorded scene domain and three relation categories, while Extended omits one relation category; both retain all recorded interaction-pattern and style categories.Table 5 reports retained coverage and the largest cross-task prevalence gap for each item.Source: §B.2 · p. 12

C.1 Operational Source Screening

• Of 3,500 source records, 3,401 pass coherence screening, 3,309 satisfy semantic code-switching criteria, and 3,009 remain after sensitivity screening.Retained records require interpretable discourse context and meaningful English use attributable to a current speaker or voice; borrowings or English words alone do not suffice.Source: §C.1 · pp. 12–13

C.2 Seed Construction and Validation

• Task-specific prompt sets are selected from 596 validated Seeds to prioritize coverage of scene domain, participant relation, interaction pattern, and preferred style, with deterministic tie-breaking and source-group constraints.Source: §C.2 · p. 13

D.2 From Concepts to Constructs

• The theory review retained 107 sources and consolidated 70 concepts into six preliminary constructs, of which four remain quality constructs distinguishing permission, local fit, functional contribution, and whole-text organization.Eligible theory records must expose an evaluative distinction and textual evidence applicable to generated narratives.Source: §D.1 · p. 15, §D.2 · p. 15

E.1 Actual Seed Representation

• Each Seed separates source-grounded information from explicitly labeled narrative scaffolding; DeepSeek-V4-Pro checks source fidelity and consistency, yielding 596 validated Seeds.Scaffolding may add plausible relations, interaction patterns, and narrative pressure, but inconsistent additions are discarded.Source: §E.1 · p. 15

E.3 Task Templates and Representative Samples

• Each task template combines fixed paragraphs with Seed-conditioned paragraphs; representative Focused and Extended Prompt Cards built from the same Seed allow direct comparison of task mappings.Focused generation also used a fixed system message requiring narrative-body-only output.Source: §E.3 · p. 15, §E.4 · p. 15

F.1 LLM Event Discovery and Boundary Fixing

• A semantic event is a locally coherent English choice tied to a speaker or narrative voice and discourse action; its boundaries are fixed before quality scoring.Several mechanical Latin-script spans may form one event, and judges cannot alter the finalized event manifest.Source: §F.1 · p. 15

H.1 Licensing (D1)

• Licensing checks whether both the participant or voice and the narrative world permit a switch; an event passes only if both checks pass.In Extended Narrative, licensing evidence is restricted to the prompt and article units preceding the event.Source: §H.1 · p. 16, §F.2 · p. 16

H CSWEval Rubrics and Aggregation

• CSWEval scores local contextual fit and functional contribution at events, alongside overall literary and narrative quality in Focused and seven independently scored whole-text dimensions in Extended.The event-level ordinal scale runs from 1 to 5; Extended D3 is a direct event score, while Focused D3 aggregates five subdimensions.Source: §H.3 · p. 17, §H.4 · p. 17, §H.5 · p. 17

I.2 Recorded Screening Counts

• Focused and Extended screening retained 407 and 400 prompts, respectively, after applying grounding, compatibility, and code-switching validity checks.Among 415 Focused and 440 Extended candidates, screening agreement was Gwet’s AC1 = .975 and .838, respectively; Focused review also checked eight criteria.Source: §I.1 · pp. 18–19, §I.2 · p. 19, §I.3 · p. 20

J.1 Set Coverage and Scoring

• Models remain close on general narrative execution but can differ substantially on multilingual-integration dimensions.The scored corpus contains 4,884 Focused and 4,800 Extended articles, with all twelve models evaluated in both settings.Source: §J.1 · pp. 20–21

K.1 Judge Ablation and Ranking Robustness

• Focused GPT-side rankings equal Dual, while other judge comparisons have ρ = .993; Extended DeepSeek-side rankings equal Dual, while GPT-side comparisons have ρ = .986.Mean rank shifts are .167 for Focused comparisons and .333 for Extended comparisons, with maximum shifts of one in both settings.Source: §K.1 · p. 21

K.2 Constructor and Annotation Sensitivity

• Across four constructor–annotator configurations, ranking agreement ranges from ρ = .888 to .979, although annotator choice affects Extended scores.In Extended Narrative, event-annotator choice corresponds to a pooled +0.745-point difference on the /20 scale; in Focused Narrative, estimated effects span zero.Source: §K.2 · p. 21, §K.3 · p. 21

K.4 Human Validation

• Human ratings show substantial concordance, and audited Extended evidence records have 98.40% factual accuracy and 99.26% temporal accuracy.The same three annotators rated outputs; Focused Kendall’s W = .718 and Extended W = .810. Independent bilingual annotators also validated event discovery and grouping.Source: §K.4 · pp. 21–23, §K.5 · p. 24

K.6 Evaluator–Generator Self- and Family Preference

• The Focused GPT evaluator signal reflects family-level affinity, but evaluator-overlap checks leave model ordering unchanged in both settings.GPT-5.5 remains highest in Focused Narrative and GPT-5.6-Sol highest in Extended Narrative under family-excluded judging; the exact-self and same-family residual groups are statistically indistinguishable.Source: §K.6 · pp. 24–25

K.7 Position Swaps

• At eight swapped event pairs, Chinese-minus-CS differences are +0.236 whole-text, +0.594 multilingual, and −0.241 general.The paired analysis uses 36 Extended narratives and compares event-span swaps with Chinese-only clause swaps.Source: §K.7 · p. 25

K.8 Aggregation Sensitivity and Within-Setting Model Comparisons

• Removing or half-weighting D1 leaves model orderings unchanged in both settings; rank averaging retains ρ = .977 with Extended baseline rankings.The top-ranked models also remain unchanged: GPT-5.5 in Focused Narrative and GPT-5.6-Sol in Extended Narrative.Source: §K.8 · p. 25
• The appendix also examines reciprocal cross-judge D3-stratified Functional Gain judgments and position swaps on a 36-narrative holdout, alongside Chinese-span controls.Source: Figure 9 · p. 26, Figure 10 · p. 26, Table 21 · p. 26
• Pairwise differences use Seed-clustered bootstrap intervals over 546 clusters and 10,000 replicates, with Holm correction across 66 comparisons per setting.Comparisons use dual-judge article means underlying within-setting Σ scores.Source: §K.8 · p. 26
• In Focused Narrative, 63 of 66 comparisons remain significant after Holm correction; in Extended Narrative, all 64 comparisons do.The corresponding percentile-bootstrap intervals exclude zero for 65 of 66 Focused Narrative pairs and 64 of 66 Extended Narrative pairs.Source: §K.8 · p. 26
• Percentile-bootstrap intervals exclude zero for 65 of 66 Focused Narrative pairs and 64 of 66 Extended Narrative pairs.Source: §K.8 · p. 26
• A small number of closely scoring pairs do not support stronger pairwise ordering after multiplicity correction.Table 29 lists the five comparisons that are not significant after Holm correction.Source: §K.8 · p. 26
• GPT-5.6-Sol versus GPT-5.5 and MiniMax-M2.5 versus MiniMax-M2.7 in Focused Narrative fall just above .05 after correction.Their unadjusted bootstrap intervals already exclude zero.Source: §K.8 · p. 26

K.9 Source and Seed Dependency Robustness

• Rank agreement stays high across source and Seed comparisons (ρ = .986–.993), with at most one-position shifts; source-balanced rankings are identical for Extended and differ by one adjacent swap for Focused.Source provenance shifts absolute scores for some models, whereas none of the shared-versus-task-specific Seed contrasts has an interval excluding zero.Source: §K.9 · p. 27

K.10 Uncertainty and Ranking Robustness

• Source-group resampling leaves rank intervals and top-model retention unchanged, while sampling standard deviations rise 3.3–3.7%.The analysis resamples source groups while retaining each group’s prompts.Source: §K.10 · p. 27
• M-F3.4 has the largest standardized easy-to-hard decline (1.336), followed by M-F2B (1.216); the mean raw decline across M-F3.1–M-F3.3 is 0.088.The M-F3.4 raw decline is 0.3025, and omitting any one Whole-text dimension keeps every difficulty gap positive.Source: §K.10 · pp. 27–28

K.11 Switching Behavior and Coverage

• Event count and code-switching density have weak associations with Σ, distinguishing switching frequency from the event-level qualities D1–D3 assess.The reported correlations are .032 and −.052 for event count in Extended and Focused, and −.123 and −.159 for density, respectively.Source: §K.11 · p. 28

L Representative Instruction Excerpts and Samples

• Representative appendix materials show Seed construction, comparable Focused and Extended Prompt Cards, event-discovery instructions, Extended evidence construction, and quality-judge instructions.The Prompt Cards are built from the same Seed, enabling direct comparison of the two task mappings.Source: L Representative Instruction Excerpts… · p. 32, Figure 11 · p. 32, Figure 14 · p. 33

M Model Provenance

• All models use temperature 0.7, top-p 0.7, and a 30,000-token output limit; reasoning settings vary by provider and model.Qwen thinking was enabled by default and explicitly requested in the released configuration; MiniMax thinking required explicit activation.Source: Table 32 · p. 38