PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

Florian Braun

arXiv:2608.14896v1cs.CLcs.LG

TL;DR

Japanese pragmatic competence in small bilingual language models is poorly captured by translation and JGLUE-style evaluations. This paper introduces a minimal-pair benchmark and combines behavioral evaluation with probing to locate pragmatic contrasts, finding that honorifics are cleanly represented while other phenomena emerge during generation or expose probe–behavior mismatches.

  • Problem

    Small Japanese–English bilingual models are primarily evaluated through translation quality and JGLUE-style tasks rather than isolated pragmatic phenomena.

  • Method

    The paper combines a minimal-pair benchmark with residual-stream probing and behavioral evaluation to locate representations of four Japanese pragmatic phenomena.

  • Results

    Honorifics are linearly decodable, while implicit subject and group reference show behavioral scenario effects without prompt-final linear signals, and indirect refusal reveals a probe–behavior mismatch.

  • Takeaways & Limitations

    The findings distinguish pragmatic features represented in residual activations from those computed during generation or confounded by the benchmark.

  • Takeaways & Limitations

    Only honorifics satisfy the study’s single-position linear-decoding assumption, while the other features require methods that track or model generation more fully.

Abstract

from arXiv · show

Large language models work well on English and behave in poorly understood ways on languages typologically far from it. Japanese is a clean example, where evaluation still leans on translation quality and JGLUE-style benchmarks, which roll lexical, syntactic and pragmatic competence into a single score. The phenomena on which general-purpose models fail Japanese users are pragmatic: honorifics, in-group and out-group reference, context-sensitive politeness, zero anaphora. I introduce J-PragEval-v0, a minimal-pair benchmark isolating four such phenomena from surface fluency, and combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B (28 layers, hidden size 1536) the corresponding contrasts live. The four features split three ways. Honorific register sits cleanly in the residual stream: 0.96 balanced accuracy at layer 15, and the model flips its preferred continuation with the scenario on 93 percent of items. Implicit subject and in-group reference are not linearly decodable at the final prompt token (0.48 and 0.38), yet flip rates are 0.77 and 0.79, so the contrast is worked out during generation rather than stored at the prompt. Indirect refusal is the negative case: 0.95 probe accuracy collapsing to a 0.43 flip rate under length-normalised teacher forcing, because the current minimal pairs conflate politeness with continuation length. I also specify Pragmatic Representation Steering, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies. Feasibility is argued indirectly rather than demonstrated: the contrastive activation addition baseline, the same geometry the method would inject, recovers probe accuracy within one to two points of logistic regression wherever a linear signal exists. Scaling to Llama-3.1-Swallow-8B is the next step.

1 Introduction

The introduction argues that Japanese pragmatic and cultural competence remains underexplored internally despite strong evaluation of bilingual models on translation and JGLUE-style tasks. It presents J-PragEval-v0, probing and behavioural analyses of TinySwallow-1.5B, and Pragmatic Representation Steering as steps toward probing, locating, and editing these representations.

  • Motivation: Existing evaluations reveal outputs but say almost nothing about internal representations of Japanese pragmatics, including honorifics, implicit subjects, and in-group/out-group reference.The paper frames cultural competence as a representational property rather than only a behavioural outcome.
  • Research questions: The paper asks where Japanese pragmatic features live, how to evaluate them independently of surface fluency, and whether lightweight inference-time intervention can improve them without full fine-tuning.The questions target layers, attention heads, residual-stream subspaces, evaluation contamination, and preservation of English capability.
  • Benchmark: 56 adjudicated items per feature, 224 in total, define J-PragEval-v0 as a minimal-pair suite covering honorific register, implicit subjects, in-group/out-group reference, and indirect refusal.Items are constructed so the correct answer is determined pragmatically rather than semantically.
  • Empirical findings: The study separates four features into cleanly decodable, behaviourally scenario-sensitive, and dataset-artefact cases using per-layer probes and length-normalised teacher-forced evaluation.Honorifics are cleanly decodable; implicit subject and in-group/out-group effects are behaviourally present but not linearly visible at the prompt-final token; indirect refusal is an apparent probe signal that fails behavioural evaluation.
  • Intervention and scope: Where a probe signal exists, the class-mean-difference direction used by PRS recovers accuracy within a couple of points of logistic regression, providing a feasibility argument for steering.Full behavioural PRS evaluation, transfer to Llama-3.1-Swallow-8B, and capability-preservation measurement remain part of the continuing programme.

2 Background and Related Work

Japanese cross-lingual evaluation must distinguish lexical and syntactic competence from pragmatic judgments such as register, perspective, and in-group/out-group reference. This paper positions mechanistic interpretability as a complement to behavioural cultural-alignment evaluation, using Japanese-adapted small language models as its setting.

  • Cross-lingual transfer: Cross-lingual transfer uses zero-shot transfer, continual pretraining, or instruction tuning, with Swallow relying on large-scale Japanese text and parallel corpora.Continual pretraining extends the Llama family with Japanese web text, Wikipedia, and parallel corpora, whereas instruction tuning adds a smaller target-language instruction set.
  • Evaluation gap: Standard benchmarks collapse Japanese lexical, syntactic, and pragmatic competence into one score, so strong JGLUE performance can coexist with socially inappropriate or pragmatically incoherent output.The collapsed dimensions include plausible word choice, topic-marking, zero anaphora, agreement, register, perspective, and in-group/out-group distinctions.
  • Japanese small language models: TinySwallow-1.5B’s strong Japanese benchmark results motivate treating small language models as first-class research objects rather than pragmatic compromises.TAID produces the 1.5B-parameter Japanese model, which beats much larger baselines on Japanese benchmarks.
  • Interpretability methods: Mechanistic interpretability offers probes for correlational decoding, activation patching for causal analysis, and representation engineering for gradient-free inference-time behavioural steering.Activation patching overwrites corrupt-pass activations with clean-pass activations and measures recovered behaviour; representation engineering edits activations without gradient updates.
  • Japanese pragmatics: Japanese pragmatics targets grammaticalised contrasts in honorific register, subject-drop and uchi/soto reasoning, perspectival alignment, and indirect refusal.Honorific choice depends on speaker–addressee relations and verb referents; subject-drop is unmarked, while expressions such as chotto can function as polite declines invisible in translation.
  • Research gap: The paper complements behavioural cultural-alignment work with a mechanistic account that asks where failures occur inside the model’s computation.Most cultural-alignment research evaluates behaviourally rather than opening up the model’s internal computation.

3 Preliminaries and Notation

The framework models pragmatic reasoning in a decoder-only bilingual language model through residual-stream activations and next-token distributions. It evaluates finite pragmatic features using minimal pairs whose surface and syntactic properties remain constant.

  • Model notation: A decoder-only bilingual model M is defined with L transformer layers, hidden dimension d, vocabulary V, and residual activation h^(ℓ)_t at layer ℓ and position t.Its next-token distribution is p_M(· | x).
  • Pragmatic features: The feature set F contains honorific register, implicit subject resolution, in-group/out-group reference, and indirect refusal.These are denoted hon, isr, io, and ref, respectively.
  • Minimal-pair datasets: For each feature, the dataset uses paired inputs differing only in that pragmatic feature, with a gold pragmatically appropriate continuation.Minimality holds surface fluency, topic content, and syntactic structure constant across each pair.

4 Methodology

The methodology links a four-part benchmark, layerwise linear probing, and teacher-forced behavioral evaluation to distinguish representational decodability from features that drive generation. It also specifies activation patching for future causal analysis and Pragmatic Representation Steering as a parameter-free inference-time intervention.

  • Methodological design: The methodology uses probing to locate sub-spaces, benchmarking to measure behavior, and steering to test whether probed sub-spaces drive benchmark-scored behavior.These components are designed to feed into each other.
  • Probing: Linear probes estimate feature decodability at each layer and token position using logistic regression with stratified 70/30 splits averaged over five seeds.Balanced accuracy on the held-out split is the decodability measure; high decodability alone is correlational, not sufficient for a causal role.
  • Causal behavioral evaluation: Teacher-forced evaluation scores length-normalised log probabilities for correct and incorrect continuations and reports the scenario-flip rate.A high flip rate supports a feature driving generation, whereas high probe accuracy with a low flip rate indicates a possible dataset confound.
  • Benchmark: J-PragEval-v0 contains four sub-tasks covering honorific register, implicit subject resolution, in-group/out-group reference, and context-sensitive refusal.The released version has 56 items per feature, while a future batch adds native-speaker annotation, adversarial filtering, and English-paired counterparts.
  • Pragmatic Representation Steering: Pragmatic Representation Steering edits the residual stream with class-mean-difference directions, using a held-out development split to choose α and additive composition for multiple features.The method does not modify weights or require training, adding one vector per generation step at a single layer.

5 Experiments … 5.3 Metrics

Experiments use TinySwallow-1.5B-Instruct and J-PragEval-v0 to evaluate probing and teacher-forced behavior, with specified baselines and a follow-on capability-preservation analysis. The intended 8B reference and several benchmark controls remain outside this study’s compute or evaluation scope.

  • 5.1 Models: TinySwallow-1.5B-Instruct is the primary model, built on Qwen2.5-1.5B with 28 transformer layers and hidden size 1536.
  • 5.1 Models: Llama-3.1-Swallow-8B-Instruct-v0.3 is the intended cross-model reference, but its 32-layer, hidden-size-4096 evaluation exceeded the study’s compute budget.The extraction pipeline, probe suite, and steering procedure are model-agnostic; English-first baselines and the Sakana Namazu reference remain future work.
  • 5.2 Data: 224 items comprise J-PragEval-v0, with 56 items per feature in a minimal-pair set used for both probing and teacher-forced evaluation.Each item pairs scenario-plus and scenario-minus prompts with pragmatically correct and incorrect continuations while holding content, topic, and register-neutral wording constant.
  • 5.2 Data: 0.97–0.99 is the mean annotator agreement range per feature, while a handful of implicit-subject items reach the 0.60 minimum and remain contested.A larger adjudicated release with independent native-speaker annotation is designated batch 02 and falls outside this paper’s scope.
  • 5.3 Metrics: Balanced accuracy is reported at each feature–method–position peak layer, averaged across five stratified 70/30 train/test splits with distinct seeds.The logistic probe is compared with a class-mean-difference direction thresholded at the class-means midpoint and a random unit vector thresholded at the training-set median.
  • 5.3 Metrics: Teacher-forced log-probability scores compare correct and incorrect continuations under both scenarios, reporting correct-preference fractions and scenario-flip rates.Scenario-flip rate is the fraction of items whose preferred continuation changes when the scenario changes.
  • 5.3 Metrics: Length-normalised log-probabilities divide by token count before accuracy and flip-rate calculations when paired continuations differ systematically in length.The capability-preservation ratio is specified as J-PragEval gain divided by changes on MMLU and HellaSwag, but is deferred to the follow-on programme.

5.4 Probing Results (RQ1)

Probing residual-stream activations separates the four pragmatic features into three groups: honorific register is linearly decodable, indirect refusal appears decodable but is misleading, and implicit subject resolution plus in-group/out-group reference are not linearly decodable before continuation. Pre-continuation probing is treated as the primary localization signal because post-continuation scores can reflect token-level continuation differences.

  • Probe setup: Probes use residual activations at the final prompt token (“pre”) and final prompt-plus-continuation token (“post”), with post probing serving as a robustness check.Results are peak-layer balanced accuracies on TinySwallow-1.5B across five stratified 70/30 splits; probes include logistic regression, CAA, and a random-direction baseline.
  • Honorific register: 0.959 ± 0.039 balanced accuracy: honorific register is linearly decodable at layer 15 of 28, while CAA reaches 0.947 at layer 27.The logistic result exceeds the random-direction baseline of 0.641, and CAA has a tighter spread.
  • Indirect refusal: 0.953 balanced accuracy: indirect refusal appears decodable at layer 17, but the behavioral evaluation shows this probe result is misleading.The post-continuation column reaches 1.000 ± 0.000 at layer 1 for indirect refusal, a ceiling artifact caused by token differences between continuations.
  • Implicit and referential phenomena: 0.476 and 0.376 peak logistic accuracies: implicit subject resolution and in-group/out-group reference are at or below chance at the pre-continuation token.Post-continuation probing rises to 0.606 and 0.524, above the matched random baseline but not decisively; sub-chance balanced accuracy is not an inverted real signal.

5.5 Behavioural evaluation (RQ2) · 5.6 Pragmatic Representation Steering: feasibility and planned evaluation (RQ3)

Behavioural evaluation shows that honorifics align both linearly and behaviourally, while implicit subject and in-group/out-group reference emerge during generation. PRS feasibility is supported where linear signals exist, but full evaluation and better data design remain future work.

  • 5.5 Behavioural evaluation (RQ2): Length-normalised teacher-forced log-probabilities are used because summed log-probability favours systematically shorter continuations.Indirect continuations are substantially longer than direct ones, biasing the raw comparison.
  • 5.5 Behavioural evaluation (RQ2): 93% scenario-flip rate accompanies strong linear decodability for honorifics, making it the clean positive behavioural result.The model’s preferred continuation flips when the scenario changes.
  • 5.5 Behavioural evaluation (RQ2): 0.77 flip rate for implicit subject resolution and 0.79 for in-group / out-group reference show behavioural scenario sensitivity despite no pre-continuation linear direction.These contrasts appear to be computed during generation rather than stored in a static final-token prompt representation.
  • 5.5 Behavioural evaluation (RQ2): Indirect refusal is the cautionary case: its probe axis exists in activations but does not drive generation because current pairs introduce surface leakage and length confounds.The proposed remedy is data redesign, beginning with the batch 02 release.
  • 5.6 Pragmatic Representation Steering: feasibility and planned evaluation (RQ3): CAA uses the same class-mean-difference direction that PRS would inject at inference time.This contrastive activation addition baseline supplies PRS’s raw feasibility evidence.
  • 5.6 Pragmatic Representation Steering: feasibility and planned evaluation (RQ3): CAA offers nothing for implicit-subject and in-out-group cases, so PRS must target the layer or token position where those features surface behaviourally.Finding that intervention location is the next methodological question.
  • 5.6 Pragmatic Representation Steering: feasibility and planned evaluation (RQ3): Full PRS evaluation requires an (ℓ, α) grid search on held-out J-PragEval, English-capability controls on MMLU and HellaSwag, and transfer to Llama-3.1-Swallow-8B.These evaluations lie outside the present probing budget and define the continuing programme.

5.7 Ablations and controls · 5.8 Reproducibility

Ablations support three distinct probing regimes, while length normalization is necessary to separate scenario sensitivity from continuation-length preference. Reproducibility is supported by containerized execution, fixed configuration, and released artifacts.

  • 5.7 Ablations and controls: 0.959 honorifics logistic accuracy and 0.947 honorifics CAA accuracy exceed the null, whereas isr and io logistic scores do not.This pattern confirms the three-regime reading.
  • 5.7 Ablations and controls: 1.000 accuracy at layers 1–4 for hon/post and ref/post reflects lexical separability of continuation tokens rather than mid-network pragmatic representation.Therefore, pre-continuation is the primary probe position.
  • 5.7 Ablations and controls: 0.000 acc(+) and mean raw ∆= −34 nats arise without length normalization because indirect continuations are longer than direct ones.After normalization, per-token margins collapse near zero, revealing no genuine scenario flip behind the probe’s 0.95 accuracy.
  • 5.7 Ablations and controls: Cross-layer transfer, cross-model transfer, activation patching, and attention-head attribution remain planned PRS-evaluation ablations.They are named in the study design but left to the continuing programme.
  • 5.8 Reproducibility: All experiments run in a single Docker image with a poetry-locked Python environment, using fixed recorded random seeds and YAML-committed hyperparameters.These controls define the reproducible execution and configuration setup.
  • 5.8 Reproducibility: Training logs, probe weights, per-chunk results, cached residual-stream activations, and future PRS steering deltas are released on Hugging Face Hub with an available Japanese academic mirror.The PRS steering deltas will accompany the follow-on evaluation described in Section 5.

6 Discussion

The study extends pragmatic probing to a bilingual Japanese–English small language model and finds that the four tested features divide into three groups rather than one linearly decodable category. It also shows that behavioral checks expose confounds hidden by pre-continuation probes, while releasing J-PragEval-v0 and probe scripts for reuse.

  • Core findings: The four pragmatic features fall into three groups rather than a single linearly decodable category.The study extends existing English-focused pragmatic probing to a bilingual Japanese–English small language model.
  • Core findings: 0.959 probe performance places honorifics cleanly in the residual stream at the final prompt token.
  • Core findings: More than three-quarters of items show scenario-dependent continuation flips for implicit subject and in-group/out-group reference despite probes at or below chance.These features therefore support a generation-time rather than storage-time account.
  • Methodological implications: Behavioral checks reveal that indirect-refusal probes can collapse because of length-induced bias and prompt leakage rather than model internals.This warns that pre-continuation residual probes on small minimal-pair datasets can hide dataset confounds.
  • Resources and deployment: J-PragEval-v0 and its probe scripts are released so other groups can evaluate their own models and datasets.The benchmark is also designed for future Swallow releases.

7 Limitations and Ethical Considerations

The study’s probing assumptions, benchmark design, indirect PRS validation, model coverage, and sample size limit the strength and generality of its conclusions. It also notes annotator bias and the risk that steering could produce inappropriate personas, recommending safety controls for deployment.

  • Methodological limitations: Only honorifics satisfy the single-position linear-decoding assumption cleanly; implicit subject and in-group/out-group reference require richer methods because their computation appears during generation.Suggested approaches include non-linear probes, dictionary-learning feature extraction, and sparse autoencoders.
  • Benchmark limitations: The indirect-refusal probe is healthy but its behavioural flip rate is not, indicating confounded continuation length and scenario framing rather than model incompetence.The passages identify ref-item redesign, rather than a different probe, as the remedy.
  • Methodological limitations: PRS is validated only indirectly through the CAA baseline; a full held-out evaluation with layer and α search was outside the paper’s experimental budget.The proposed next step is behavioural evaluation on a held-out J-PragEval split.
  • Scope and data limitations: The experiments cover one open-weight bilingual SLM, use n = 56 per feature, and cannot probe or steer API-only frontier Japanese models.Llama-3.1-Swallow-8B is next, while batch 02 aims to bring per-feature counts into triple digits with independent native-speaker annotation.
  • Ethical considerations: J-PragEval labels may carry dialectal, regional, and generational bias because they reflect a finite annotator pool.Mitigations include documenting demographics, reporting inter-annotator agreement, releasing disaggregated scores, and recording genuinely contested items.
  • Ethical considerations: PRS could push models toward inappropriate personas as well as culturally appropriate behaviour, so downstream deployments should pair it with an output content-safety layer.The PRS specification explicitly calls out this risk.

8 Conclusion

The paper contributes J-PragEval-v0 and shows that Japanese pragmatic features in TinySwallow-1.5B differ in whether they are linearly represented or behaviorally expressed. It also proposes Pragmatic Representation Steering, with further behavioral evaluation, transfer, and capability-preservation studies remaining.

  • Benchmark: J-PragEval-v0 contributes 224 adjudicated minimal-pair items separating four Japanese pragmatic phenomena from surface fluency.The benchmark targets culturally and pragmatically relevant behavior rather than general surface fluency.
  • Findings: 0.959 honorific probe accuracy at layer 15 marks honorifics as the cleanly linearly decodable feature.The study contrasts this case with features whose scenario effects are behavioral but not linearly visible at the prompt-final token.
  • Findings: 0.77 and 0.79 flip rates show scenario effects for implicit subject and in-group / out-group reference despite limited prompt-final-token linear visibility.These features are behaviorally expressed even though they are not linearly visible at the prompt-final token.
  • Findings: 0.953 indirect-refusal probe accuracy versus 0.429 flip rate exposes the apparent probe signal as a dataset artefact.Behavioral evaluation reveals the mismatch between the apparent probe signal and scenario-sensitive behavior.
  • Intervention and outlook: 1–2 points is the CAA-baseline gap from the logistic probe wherever a linear signal exists, supporting PRS feasibility while leaving full evaluation and scaling future work.The continuing programme includes full behavioral PRS evaluation, transfer to Llama-3.1-Swallow-8B, and capability-preservation measurement on English benchmarks.
Loading 2608.14896v1…