PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Cantonese-Adapted Language Models Better Predict Cantonese Reading? A Cross-Model Eye-Tracking Evaluation

Ziqi Zhang, Emmanuele Chersoni, Mohammad Momenian

arXiv:2609.02163v1cs.CL

TL;DR

Whether Cantonese-specific training improves psycholinguistic alignment remains unclear, especially given mixed evidence from Cantonese NLP evaluations. The study compares two within-family adaptation contrasts on naturalistic Cantonese eye-tracking data using four information-theoretic predictors. CantoneseLLM-7B shows the strongest lexical-surprisal and joint-model fit, while entropy reduction favors CKIP, indicating measure-dependent rankings.

  • Problem

    It remains unclear whether Cantonese-specific training improves probabilistic prediction of human reading beyond conventional predictors.

  • Method

    The study compares two within-family model-adaptation contrasts on Cantonese MCFIX eye-tracking data using lexical surprisal, POS surprisal, entropy before, and entropy reduction.

  • Results

    CantoneseLLM-7B outperforms Qwen2.5-7B for lexical surprisal and the joint four-metric model, whereas entropy reduction shows the strongest gains for CKIP.

  • Takeaways & Limitations

    Psycholinguistic alignment depends on adaptation strategy and the aspect of predictive uncertainty being modeled, rather than Cantonese training alone.

  • Takeaways & Limitations

    The model comparisons do not isolate Cantonese training because paired checkpoints differ in vocabulary, pretraining, instruction tuning, and other properties.

Abstract

from arXiv · show

Information-theoretic measures derived from autoregressive language models are widely used to characterize the expectations that shape human reading, but whether language-variety-specific training improves such psycholinguistic alignment remains unclear. This question is still open for Cantonese, where recent NLP evaluations reported mixed benefits from Cantonese-specific training relative to Mandarin-oriented or general-purpose models. Using naturalistic Cantonese eye-tracking data, we compare two within-family adaptation contrasts: CKIP GPT-2 Tiny versus its lightly Cantonese-adapted JED351 derivative, and Qwen2.5-7B versus CantoneseLLM-7B, which underwent substantially more extensive Cantonese continued pretraining and instruction tuning. From each model, we derive lexical surprisal, POS surprisal, entropy before the target, and entropy reduction. Lexical surprisal and the joint four-metric model consistently favor CantoneseLLM-7B, followed by Qwen2.5-7B, CKIP, and JED351, whereas entropy reduction favors CKIP. These results suggest that more extensive Cantonese-specific training can be associated with stronger predictive fit, while model rankings also depend on the information-theoretic measure being evaluated.

1 Introduction

This study asks whether Cantonese-specific language-model training improves prediction of human Cantonese reading behavior. It compares within-family adaptation contrasts using information-theoretic predictors and naturalistic eye-tracking data.

  • Cantonese remains comparatively underexplored in eye-tracking prediction, despite widespread use and scarce textual resources relative to Mandarin.
  • Existing evaluations provide mixed evidence about whether Cantonese-specific adaptation is needed or beneficial.
  • The study uses Cantonese NR and TSR eye-tracking measurements from the MCFIX translation of The Little Prince.
  • It compares CKIP GPT-2 Tiny with JED351, and Qwen2.5-7B with CantoneseLLM-7B, across four information-theoretic predictors.
  • The research questions concern incremental predictive value, adaptation intensity, eye-tracking measure, and reading-task differences.
  • The contributions include matched within-family comparisons, joint evaluation of four metrics, robustness analysis, task comparison, and predictor-block ablations.

2 Background

The paper uses four complementary measures to represent uncertainty and expectation before, at, and after a target word. These measures distinguish lexical and syntactic prediction from broader anticipatory uncertainty and post-target updating.

  • Together, the metrics characterize pre-target anticipation, target-level expectation, and post-target probabilistic updating without assuming discrete cognitive stages.
  • Lexical surprisal measures prediction error for the observed word, with low-probability continuations producing higher surprisal.
  • POS surprisal measures how expected the target’s syntactic category is in context rather than merely recording its observed category.
  • Entropy before measures dispersion in the next-token distribution before the target and captures anticipatory uncertainty.
  • Entropy reduction measures the signed change in uncertainty over next-token continuations after the current word is added to context.
  • The Cantonese MCFIX corpus contains natural reading and task-specific reading conditions with word-level FFD, SFD, and TFD measurements.

3 Data and Models

The study uses traditional-character Cantonese eye-tracking data and compares two model pairs designed as within-family adaptation contrasts. The contrasts differ in model scale and the extent of Cantonese-specific training.

  • The traditional-character Cantonese corpus contributes 10,011 task-level rows after filtering and intersecting observations across all four models.
  • The data combine NR and TSR observations, include reading task as a predictor, and report both overall and task-specific performance.
  • The small-model contrast pairs CKIP GPT-2 Tiny Chinese with its Cantonese-adapted JED351 derivative.
  • JED351 adds Cantonese characters to the tokenizer and embedding matrix and receives ten epochs of fine-tuning on approximately 50 MB of Cantonese Wikipedia text.
  • The large-model contrast pairs Qwen2.5-7B with CantoneseLLM-7B, which receives substantially more extensive Cantonese-specific training.

4 Methods

The analysis extracts four model-based predictors for each target word and evaluates them with CatBoost regression against conventional linguistic predictors. Cross-validation uses shared word-level folds and multiple seeds.

  • Metric extraction: For each target, the study derives lexical surprisal, POS surprisal, entropy before, and entropy reduction from preceding within-sentence context.
  • Metric extraction: The study evaluates autoregressive models in two within-family Cantonese-adaptation contrasts while keeping architectures fixed.
  • Metric extraction: POS surprisal uses a common Cifu candidate inventory, making its construction directly comparable across models.
  • Regression design: CatBoost regression uses a no-LLM baseline containing task, lexical, positional, syntactic, dependency, frequency, syllable, and neighbourhood predictors.
  • Regression design: Models add individual LLM metrics, all four metrics jointly, or remove predictor blocks from the full model.
  • Regression design: Word-level five-fold cross-validation uses shuffled split seeds 42–46 and identical row assignments across language models.
  • Regression design: Single-metric improvement is evaluated relative to the baseline, while leave-one-out contribution compares a reduced model with the full four-metric model.
  • Regression design: Baseline-normalized improvement helps prevent comparisons across FFD, SFD, and TFD from being driven only by outcome scale.

5 Results

LLM-derived measures improve eye-tracking prediction, with the largest and most consistent gains for CantoneseLLM-7B and the joint four-metric model. However, entropy reduction favors CKIP, task effects vary by outcome and model, and absolute gains remain modest.

  • Incremental predictive value: Every model–metric combination improves FFD and TFD on average, while SFD gains are smaller and entropy reduction is approximately null for JED351 and Qwen2.5-7B.Entropy reduction is clearly positive for CKIP and slightly positive for CantoneseLLM-7B on SFD.
  • Robustness and magnitude: Gains are stable across split seeds but modest: even the largest joint improvement is below 2% of baseline MAE after normalization.The authors characterize these effects as incremental predictive contributions rather than large changes in overall accuracy.
  • Metric-specific comparisons: CantoneseLLM-7B produces the largest lexical-surprisal MAE reductions, followed by Qwen2.5-7B, CKIP, and JED351 across all three outcomes.For TFD, the gains are 0.683, 0.599, 0.249, and 0.229 ms, respectively.
  • Metric-specific comparisons: CKIP’s entropy reduction yields the largest mean improvement among the four models for FFD (0.183 ms), SFD (0.104 ms), and TFD (0.362 ms).This differs from CKIP’s weaker position for lexical surprisal.
  • Joint modeling: The joint four-metric model outperforms every single-metric addition for every model and outcome, with CantoneseLLM-7B reducing MAE by 0.729 ms for FFD, 0.267 ms for SFD, and 1.251 ms for TFD.Mean joint-model improvements follow CantoneseLLM-7B > Qwen2.5-7B > CKIP > JED351.
  • Outcome and task effects: Relative four-metric improvement is largest for FFD in all four models (1.118–1.991%), although TFD has the largest raw reductions for the two 7B models.Thus, outcome-level gain magnitude depends on whether error is expressed in milliseconds or relative to baseline MAE.
  • Outcome and task effects: CantoneseLLM-7B gives the largest joint improvement in five of six task–outcome combinations, while TSR improves FFD more across all models but SFD and TFD show model-specific patterns.For CKIP, joint SFD and TFD improvements are 0.257 and 0.778 ms in NR versus 0.138 and 0.523 ms in TSR.
  • Unique contributions: LLM-derived measures contribute less conditionally than in single-metric additions, and their partial overlap means they share information without being interchangeable.Entropy before and entropy reduction show the strongest cross-model association (ρ = .55–.67).

6 Discussion

Information-theoretic predictors add modest information beyond conventional linguistic predictors, while their predictive value varies with adaptation strategy, eye-tracking outcome, and reading task.

  • Beyond conventional predictors: Each information-theoretic metric improves prediction in some model–outcome combinations, and the joint four-metric model outperforms every corresponding single-metric model.The largest baseline-normalized joint improvement remains below 2%, so these are incremental gains.
  • Beyond conventional predictors: Current syllable count and relative word position remain the strongest predictor blocks, while LLM-derived measures generally rank in the middle.Their conditional contributions are smaller than their single-metric additions, consistent with partial overlap among the four measures.
  • Cantonese-specific training: CantoneseLLM-7B yields the largest mean lexical-surprisal and joint-model improvements across all three eye-tracking outcomes, whereas JED351 does not generally outperform CKIP.The large-model pair favors CantoneseLLM-7B, while the lighter adaptation contrast is negative for the joint model across FFD, SFD, and TFD.
  • Cantonese-specific training: CantoneseLLM-7B exceeds Qwen2.5-7B by 0.132, 0.066, and 0.430 ms for the joint model on FFD, SFD, and TFD, respectively.These positive contrasts hold across all five split seeds; entropy reduction contrasts remain close to zero.
  • Eye-tracking outcomes and tasks: After baseline normalization, joint four-metric improvement is largest for FFD in every model, despite larger raw improvements often occurring for TFD.FFD indexes the first fixation, whereas SFD records a second fixation and may emphasize later structural demands.
  • Eye-tracking outcomes and tasks: All four models show larger mean joint-model FFD improvements in task-specific reading than natural reading, but this pattern does not extend uniformly to SFD or TFD.The predictive value of LLM-derived measures may therefore depend jointly on reading objective and eye-tracking measure.

7 Conclusion

The study finds that Cantonese-specific training is not uniformly advantageous for predicting Cantonese reading behavior. Predictive alignment depends on adaptation strategy and the aspect of uncertainty being modeled.

  • Conclusion: LLM-derived predictors provide consistent but modest improvements beyond conventional linguistic predictors, while Cantonese adaptation does not yield a uniform advantage.JED351 does not outperform CKIP overall, whereas CantoneseLLM-7B outperforms Qwen2.5-7B for lexical surprisal and the joint four-metric model.
  • Conclusion: Psycholinguistic alignment depends on both adaptation strategy and the aspect of predictive uncertainty being modeled.Better prediction of the upcoming word does not necessarily imply better modeling of how linguistic input updates uncertainty during reading.

Limitations

The comparisons constrain causal attribution because model pairs differ in multiple training components, architecture, and scale, while the data and tokenization setup also limit scope.

  • Model-comparison scope: The model comparisons do not isolate the effect of Cantonese training because each checkpoint pair differs in multiple adaptation components.JED351 differs from CKIP in vocabulary, embeddings, and Wikipedia fine-tuning; CantoneseLLM-7B differs from Qwen2.5-7B in continued pretraining and instruction tuning.
  • Model-comparison scope: The results reflect broader adaptation strategies rather than any single training component, with the two pairs also differing in architecture and scale.Accordingly, the findings should not be attributed to one isolated intervention.
  • Data and measurement scope: The extraction pipeline imposes a separate lexical boundary from natural concatenated tokenization on 246 of 5,072 Qwen-family rows.The study is also limited to one literary translation and participant sample, and entropy values are not directly comparable across model–tokenizer systems with different continuation spaces.

A Reproducibility Details

The regression analysis uses fixed split and boosting settings, with code planned for release upon acceptance.

  • Reproducibility details: Regression uses split seeds 42–46, five shuffled folds per seed, CatBoost internal seed 10, and 1,000 boosting iterations.Code will be released upon acceptance.

B Supplementary Analyses

The supplementary analyses assess robustness, baseline-normalized gains, task-stratified effects, within-family adaptation contrasts, metric correlations, and conditional predictor contributions across five split seeds.

  • Five-seed robustness: Table 3 summarizes pooled single-metric and joint-model additions as mean ∆MAE ± SD and counts of positive effects across five split seeds.The positive-effect count is descriptive stability information, not a significance test.
  • Normalized improvements: Table 4 reports baseline-normalized percentage improvements for each LLM-derived metric, computed separately within each split seed.Positive values indicate improvement over the baseline.
  • Task-stratified analyses: Table 5 provides complete NR task-stratified additions from pooled models, evaluated on NR out-of-fold observations from jointly trained NR+TSR models.These are not results from separately trained NR-only models.
  • Model and metric comparisons: Figure 4 displays seed-level ∆MAE distributions for JED351, CKIP, Qwen2.5-7B, and CantoneseLLM-7B across four LLM-derived metrics.Each box contains five split-seed values, shown alongside individual points; positive values indicate improvement over the no-LLM baseline.
  • Model and metric comparisons: Figure 5 compares Cantonese-adapted models with their within-family base counterparts using paired ∆MAE differences across five split seeds.Negative values favor the base model, while positive values favor the Cantonese-adapted model.
  • Metric relationships: Figure 6 reports within-model Spearman correlations among the four LLM-derived metrics over 5,007 unique canonical text positions.Duplicated NR and TSR observations were collapsed, with full-precision coefficients supplied separately.
Loading 2609.02163v1…