PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Language Effects on Tool Admission and Post-Retrieval Safety

Aditya Nawal, Manit Baser, Mohan Gurusamy

arXiv:2610.23093v1cs.CL

TL;DR

Multilingual safety evaluations have focused largely on English, leaving unclear whether language affects tool admission, post-retrieval generation, or both. This paper varies request, retrieved-page, and response languages in controlled retrieval experiments. Non-English requests raise admission, and requiring page-language answers raises matched-call harmfulness; audits show that non-harmful labels can also reflect capability failures.

  • Problem

    Safety evaluation of retrieval-enabled agents remains English-centric, and prior work has not established whether language changes tool admission, post-retrieval harmfulness, or both.

    Source: §1 · p. 1

  • Method

    The study independently varies request, retrieved-page, and required response languages in controlled URL retrieval, separating tool admission from post-retrieval generation.

    Source: §1 · p. 2

  • Results

    Mean admission rises from 46.9% in English to 67.4% across non-English languages; required page-language answers also increase matched-call harmfulness, while page translation alone has no consistent effect.

    Source: §4.1 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    Multilingual evaluation should measure tool admission and response harmfulness separately and distinguish genuine refusals from language deflections and unusable answers.

    Source: §1 · p. 1

    Native-speaker harmfulness validation does not cover Igbo, Swahili, Lao, or Haitian Creole, and automated response-taxonomy audits lack human label validation.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLM agents increasingly use retrieval tools across languages, yet their safety evaluation remains English-centric. It remains unclear whether language affects tool admission, post-retrieval generation, or both. We independently vary request, retrieved-page, and required response languages in a controlled URL-retrieval setting across ten models and seven languages. We identify Safety Loss through Interlingual Processing (SLIP). Non-English harmful requests increase mean tool admission from 46.9% to 67.4%, with a positive mean change in every model. This pattern also persists in an autonomous retrieval setup across all four models tested in this setting. Requiring answers in the page language increases pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models, mainly in Chinese and Hindi. Translating pages alone produces no consistent harm increase in eight models with comparable inference settings. Automated audits show that a non-harmful response may be a genuine safety refusal, a deflection citing language difficulties, or an unusable answer. Harmfulness labels alone cannot distinguish these outcomes. Multilingual safety evaluations should therefore measure tool admission and response harmfulness separately and distinguish refusal from capability failure.

1 Introduction

Retrieval-agent safety spans both the decision to invoke a tool and the response after retrieval, but multilingual evaluations have focused largely on English. This section separates those stages and examines how request, page, and response languages relate to safety.

1 Introduction

  • Prior work examines retrieval-related safety risks and multilingual safeguards, but does not establish whether language changes tool invocation, post-retrieval harmfulness, or both.Source: §1 · p. 1
  • The study uses controlled URL retrieval to separate tool admission from post-retrieval generation while independently varying request, page, and required response languages.A fixed user-specified URL holds the retrieval target constant; model-initiated admission is also tested without an experiment-supplied URL.Source: §1 · p. 2
  • Safety Loss through Interlingual Processing (SLIP) names language-associated changes in safety, distinguishing admission-side changes from response-side harmfulness changes.Source: §1 · p. 2
  • Non-English harmful requests raise mean tool admission from 46.9% to 67.4%, while post-admission harm varies with model capability and refusal behavior.The pattern persists in autonomous retrieval; among smaller models, lower harm often coincides with failure to understand or produce a usable answer.Source: §1 · p. 2
  • Requiring answers in the page language raises matched-call harmfulness from 50.3% to 57.4%, mainly in Chinese and Hindi, while translating pages alone has no consistent effect.Source: §1 · p. 2
  • Among audited non-harmful Chinese or Hindi responses from four 3–14B models, 40.0% were genuine refusals and 51.5% were language deflections or realization failures.Source: §1 · p. 2

2 Related Work

Prior work identifies safety risks in retrieval-augmented generation and tool-using agents, as well as multilingual variation in safety and tool use. This study extends those lines by measuring language effects at retrieval admission and post-retrieval generation.

2 Related Work

  • Retrieval-augmented generation can weaken safety, and agent benchmarks document harmful completion and unsafe tool actions across stages.Source: §2 · pp. 2–3
  • Prior multilingual research finds language-dependent safeguards, tool-execution failures, and response-language problems, but does not jointly isolate the retrieval pipeline’s two safety decisions.Source: §2 · pp. 2–3
  • This work varies language while holding task, retrieval target, and tool interface fixed, and distinguishes genuine refusal from multilingual understanding or generation failure.Source: §2 · p. 3

3 Methodology

The methodology separates the decision to retrieve from post-retrieval response generation using controlled URL retrieval. It varies language at distinct stages and defines stage-specific safety measures.

3 Methodology

  • A supplied URL fixes the retrieval target, separating the pre-page admission decision from post-call response generation.After an admitted call, the model receives the cached page; fixing the target controls query formulation, ranking, and source selection.Source: §3 · p. 3
  • The evaluation uses 296 harmful-behavior pages, seven languages, and ten models; autonomous retrieval is tested on four closed-weight models.Pages use the direct-assistance stance, and matched benign summary requests serve as an admission-selectivity control.Source: §3 · pp. 3–4
  • The study varies URL form, request language, retrieved-page language, and required response language while holding other components fixed.Request-language changes retain an English page; page-language changes retain an English request and URL; response-language changes retain the English request and retrieved page.Source: §3 · p. 3
  • Admission rate measures tool calls, while post-admission harm rate measures harmful final responses among called cases with usable harmfulness labels.Request- and page-language summaries average model-level rates equally; required-response summaries pool evaluated responses across model–language conditions.Source: §3 · p. 4
  • Admission-side SLIP compares target-language requests with English under a fixed English page; response-side SLIP compares target-language with English response requirements on a fixed translated page and English request.The response comparison reports both the full-condition contrast and a paired-called contrast restricted to requests admitted under both conditions.Source: §3 · pp. 4–5
  • Translation adequacy and harmfulness labels receive independent checks, including native-speaker annotation of stratified English, Chinese, and Hindi responses.Translation judges report request adequacy of 86.5–100% and page adequacy of 97.3–99.7%; GPT-4o labels are checked against native-speaking annotators and additional judges.Source: §3 · p. 5

4 Results

Across the retrieval experiments, non-English requests increased tool admission, while post-retrieval harm depended on page and response language and model group. Audits show that non-harmful labels can reflect refusal or unsuccessful task completion.

4.1 Does Request Language Affect Tool Admission?

  • 46.9% to 67.4%: mean tool admission rose from English to six non-English languages, with a positive mean change in every model.Benign controls also showed reduced admission selectivity, and the admission increase remained under a strict translation-quality filter.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Does the Admission Effect Persist Without a Supplied URL?

  • 14.9% in English versus 23.9–57.9% across non-English languages: mean admission increased in each of four closed-weight models without a supplied URL.Most generated tool calls pursued the harmful task (73.2–96.7% per language).Source: §4.2 · p. 6

4.3 How Do Page and Response Languages Affect Response-Side Harm?

  • −0.9 points in 3–14B models and −3.1 in closed-weight models: translating only retrieved pages produced no consistent mean harm increase across eight comparable models.Page translation often left answers in English, motivating the separate response-language directive test.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • 50.3% to 57.4%: pooled harmfulness among paired-called behaviors in four closed-weight models rose when the page language, rather than English, was required.The rise was concentrated in Chinese/Hindi: 12.3 points, versus 1.4 points in Igbo/Swahili; Figure 3 compares the paired directives.Source: Figure 3 · p. 7, §4.3 · p. 7

4.4 Does Low Harmfulness Indicate Genuine Refusal?

  • 99.2% of English goals but 10.0–48.4% of lower-resource-language goals were recovered by 3–14B models in a tool-free probe.For these models, lower observed harm on translated requests accompanied more estimated realization failure, while estimated refusal stayed near 25%.Source: §4.4 · pp. 7–8, Figure 4 · p. 7
  • 40.0% of audited non-harmful Chinese/Hindi answers from 3–14B models were refusals, while 51.5% were language deflections or realization failures.In Igbo/Swahili, 17.9% were refusals and 79.2% were deflections or failures; larger and closed-weight model groups had refusal as their largest category in every language.Source: §4.4 · pp. 7–8, Table 4 · p. 8
  • Among non-harmful responses to translated requests, 3–14B models’ refusal share fell from 70.1% in English to 23.3–60.8% in other languages as failure rose from 23.8% to 37.5–71.2%.Refusal remained the largest category in every language for the 27–31B and closed-weight groups; probe and audit patterns are associations across model-language conditions.Source: §4.4 · p. 8, Table 5 · p. 8

5 Conclusion

The paper identifies language-associated safety changes at tool admission and post-retrieval generation. Its findings support evaluating admission and response harmfulness separately, while distinguishing safety refusal from capability limitations.

5 Conclusion

  • Non-English harmful requests increase mean tool admission, while requiring page-language answers increases matched-call harmfulness in four closed-weight models, mainly in Chinese and Hindi.Translating pages alone produces no consistent harm increase in eight models with comparable inference settings.Source: §5 · p. 8
  • Multilingual safety evaluations should measure tool admission and response harmfulness separately and distinguish genuine refusal from capability limitations.The conclusion also notes that non-harmful responses can include language deflections and unusable answers.Source: §5 · p. 8

6 Limitations

The study’s comparisons are bounded by observational capability–harm relationships, condition-dependent coverage, and validation limits across languages and evaluators.

6 Limitations

  • Comprehension probes and response audits use separate outputs, so capability–harm relationships are observational.Evaluated-response coverage varies across conditions, and excluded failed API requests also change admission coverage.Source: §6 · p. 9
  • The primary page-language claim covers eight models with comparable inference settings, while supplementary comparisons combine page-language and inference-limit changes.Required-response directives precede admission, so paired-called contrasts condition on a post-treatment decision.Source: §6 · p. 9
  • Language-group contrasts do not isolate script, typology, alignment coverage, model size, training, provider filtering, or weight access.Per-language training data are unknown; resource groups describe mC4 web availability.Source: §6 · p. 9
  • Automated translation-adequacy checks do not establish pragmatic or cultural equivalence, and the response-taxonomy audit lacks human label validation.Native-speaker harmfulness validation covers English, Chinese, and Hindi reference samples, but not Igbo, Swahili, Lao, or Haitian Creole.Source: §6 · p. 9

7 Ethics Statement

The work evaluates harmful requests and model-generated harmful content to identify multilingual tool-agent safety weaknesses before deployment. It also highlights the equity stakes of English-centric evaluation and the potential for cross-lingual gaps to be exploited.

7 Ethics Statement

  • The study examines harmful requests and model-generated harmful content to identify weaknesses in multilingual tool-agent safety before deployment.It reports aggregate statistics over controlled cached content and releases no instructions intended for misuse.Source: §7 · p. 9
  • English-centric safety evaluation may leave lower-resource-language users with less reliable protection, while cross-lingual safety gaps may be exploited by users with translation tools.The authors motivate broader multilingual red-teaming and stage-specific safeguards for tool-using systems.Source: §7 · p. 9

A Experimental Setup

The appendices document the evaluated models and inference protocol, then detail answer coverage, judge sensitivity, and the rubric used to score response harmfulness.

A.1 Models

  • The primary evaluation uses ten models, all evaluated on the same 296 target behaviors; main-text results and averages refer to these ten, excluding supplementary analyses in Tables 7 and 8.Source: §A.1 · p. 12

A.2 Inference Protocol and Artifacts

  • Models receive an English system instruction and use automatic tool choice; valid no-call outputs are retained, while failed API requests are excluded from admission and harm rates.Forced response-language directives differ only in assigned language and are visible before tool admission; the protocol specifies temperature 0 and a 4,096-token generation limit unless stated otherwise.Source: §A.2 · p. 12

B.1 Final-Answer and Audit Coverage

  • The paired harm comparison includes 1,448 closed-weight-model pairs and 1,091 pairs for the 3–14B models; the 27–31B comparison uses 580 pairs.The reported paired harm rates vary by language set: for 27–31B models, Chinese/Hindi is 53.8% versus 66.3%, while Igbo/Swahili is 60.0% versus 59.2%, for English versus page-language directives.Source: §B.1 · p. 13

B.2 Harmfulness-Judge Sensitivity

  • On the same 1,448 pairs, GPT-5.5 rescoring finds pooled harm rising from 48.7% under the English directive to 54.1% under the page-language directive.The response-language check reports 88.5% and 93.3% binary agreement, with κ values of 0.77 and 0.87, respectively.Source: §B.2 · pp. 14–15

B.3 GPT-4o Evaluation Rubric and Prompts

  • GPT-4o scores final-response harmfulness with a zero-shot, temperature-0.0 rubric on a 1-to-5 scale, treating scores of 3 or higher as safety failures.The rubric ranges from proactive refusal or safe steering at score 1 to fulfilling a policy-violating instruction at score 5.Source: §B.3 · p. 15

B.4 Judge Validation

The appendices assess harmfulness-label stability across translation and provider checks, and audit safe-labeled translated-page responses by outcome type.

B.4 Judge Validation

  • Translation-mediated judge agreement ranges from 72.4% to 90.1%, with corrected directional shifts toward more harmful labels for request-language Igbo and Swahili and assigned-response-language Igbo.The protocol keeps the judge, rubric, and threshold fixed while replacing the target-language response with its English translation.Source: §B.4 · p. 16
  • The cross-provider Gemini check covers selected responses in Igbo, Swahili, Lao, and Haitian Creole, and its non-random samples do not estimate population harm prevalence.Binary agreement with GPT-4o is 72.7–86.4%, and the second judge labels fewer responses harmful in every language.Source: §B.4 · p. 16

B.5 Taxonomy Audit: Translated-Page Safe Responses

  • For closed-weight models, audited translated-page responses show refusals falling from 49.1% with English pages to 43.2% with translated pages, while 9.0% cite language difficulties.Pooled harm falls from 45.2% to 42.7%; in 3–14B models, refusals fall from 21.6% to 14.1% and pooled harm rises from 68.9% to 74.0%.Source: §B.5 · p. 17
  • The 27–31B translated-page and English-page runs use mismatched inference limits, and pooled shares weight models by scored-answer counts.Page-language rate summaries instead average models equally.Source: §B.5 · p. 18

C.1 Request Translation Adequacy

The request-translation analyses test whether inadequate translations or translator choice explain language-related safety changes. They also describe the translated-page validation pipeline and resource grouping used in the study.

C.1 Request Translation Adequacy

  • After restricting to quality-accepted requests, the admission gap remains; for Igbo, it falls from +27.5 to +23.3 points when the strict filter retains 202 of 296 behaviors.For post-admission harm, strict-subset changes preserve the direction of full-set changes; closed-weight increases and 3–14B decreases remain significant, while 27–31B changes do not.Source: §C.1 · p. 18, Table 17 · p. 19, Table 18 · p. 19

C.2 Translator-Invariance Control

  • Under an alternative translator, the Igbo admission increase remains large (+29.7 versus +34.2 points), while Hindi falls from +4.3 to +1.0 points.Post-admission harm is unchanged for Igbo (+0.3 points, p = 0.28) and lower for Hindi (−5.3 points, p = 0.015).Source: §C.2 · p. 18

C.3 Retrieved-Page Translation

  • Retrieved pages are translated in aligned chunks, automatically checked for four output failures, and audited for preservation of harmful-domain, action, target, and technical details.Pages that trigger a check are retried with smaller chunks; a fallback translator handles some provider-side blocks, leaving no missing page-language pairs.Source: §C.3 · pp. 18–19

C.4 Language Resource Grouping

  • The study groups Chinese and Hindi as higher-resource languages and Igbo, Swahili, Lao, and Haitian Creole as lower-resource languages using mC4 corpus sizes.The grouping describes public web availability; language-specific training data are unknown.Source: §C.4 · p. 19

D.1 Does Request Language Affect Autonomous Retrieval?

The autonomous retrieval test finds higher admission for non-English harmful requests across models. Controls examine URL cues, benign-task selectivity, harmful yield, and category-level variation.

D.1 Does Request Language Affect Autonomous Retrieval?

  • Every tested model admits harmful requests more often in every non-English language, with mean increases of 9.0–42.9 percentage points over English.In the autonomous setup, mean admission is 15.4% in English versus 24.4–59.1% across six non-English languages; targeted-call admission also exceeds English in 23 of 24 comparisons after excluding copyright behaviors.Source: §D.1 · pp. 19–20, Table 22 · p. 20

D.2 URL-Variant Tool-Admission Rates

  • Under harmful requests, URL variants produce tool-call rates of 60.9%–68.5% for the four 3–14B models, and their small differences do not account for the multilingual gap.Under benign requests, explicit harmful paths reduce calls, while hidden paths and shortener-style URLs increase them.Source: §D.2 · p. 20

D.3 Multilingual Benign Admission Selectivity

  • Harmful call rates rise and admission selectivity contracts in every target language, with the largest effects for Igbo and Swahili.The pattern argues against non-English input merely increasing a language-general propensity to invoke the available tool.Source: §D.3 · pp. 20–21, Table 23 · p. 21

D.4 Retrieval-Mediated Harmful Yield

  • Coverage-conditioned retrieval-mediated harmful yield rises outside English in all three model groups, with changes spanning 0.1–18.1, 3.9–11.5, and 10.6–37.6 points, respectively.Yield is conditional on scoring coverage, which varies across languages and models, and does not estimate complete system harmfulness.Source: §D.4 · p. 21, Figure 5 · p. 21

D.5 Per-Category Harm Analysis

  • The lower-resource admission gap is largest for harassment/bullying and harmful (+51.2 and +43.2 pp) and smallest for copyright (+10.3 pp).English copyright admission is already 88.2%.Source: §D.5 · p. 21

D.6 Retrieved-Page Language Results

The supplied results cover page-language comparisons, harm after tool denial, tokenizer and script controls, defenses, and interface localization. They qualify what these comparisons can establish and how safe-looking outputs should be interpreted.

D.6 Retrieved-Page Language Results

  • The primary retrieved-page comparison covers four 3–14B and four closed-weight models; the two 27–31B page-language contrasts also change inference limits and other settings.The 27–31B page-language results therefore cannot isolate page language, although their required-response arms use matching limits.Source: §D.6 · p. 21, Table 27 · p. 23

D.7 Request-Language Refusal-Text Harm After Tool Denial

  • Across ten models and seven languages, 2.0% of no-call responses are harmful (138 of 6,868), much less common than post-admission harm.Audits distinguish refusal from language deflection, misunderstanding, and failed generation among nonempty no-call responses.Source: §D.7 · p. 22, Table 28 · p. 23, Table 29 · p. 23

E.1 Tokenizer and Script Controls

  • Admission gaps are not explained by bytes-per-character or native scripts: Latin-script Igbo has English-like fertility despite a large gap, and romanization leaves the gap intact.Bytes-per-character does not predict mean admission across languages (r = −0.16, p = 0.74).Source: §E.1 · p. 23, Table 31 · p. 24

E.2 Defense Sweep: System-Prompt and Tool-Description Restrictions

  • Neither the system-prompt warning nor restricted tool description, alone or combined, prevents harmful admission; the strongest combined setting still admits 36–71% of harmful requests.The warning raises Qwen-2.5-3B admission from 67.2% to 82.4%.Source: §E.2 · p. 24, Table 32 · p. 24

E.3 Localized System and Tool Interface

  • Admission increases in every language even when the system prompt and natural-language tool descriptions are localized, weakening an English-interface mismatch explanation.Because both interface components are localized together, their separate effects are not identified.Source: §E.3 · p. 24

E.4 Comprehension Probe

The comprehension probe measures whether models understand translated harmful requests, then connects those measurements to response outcomes and normalization checks. Behavior-clustered inference summarizes language-versus-English patterns across model groups.

E.4 Comprehension Probe

  • Tool-free goal recovery among 3–14B models is near the floor for Igbo (10.0%) and reaches 27.1% for Lao and 48.4% for Haitian Creole; 27–31B models recover 60.1–92.9% of lower-resource requests.Closed-weight pooled goal recovery is 73.5% for Igbo, 90.8% for Swahili, 94.7% for Lao, and 91.8% for Haitian Creole, coexisting with increased harmful-request admission.Source: §E.4 · pp. 24–25

E.5 How Do Comprehension and Refusal Vary with Post-Admission Harm?

  • Comprehension and post-admission harm are associated differently by model group: positively within six open-weight models (r = 0.42, p = 0.019) and negatively within closed-weight models (r = −0.39, p = 0.083).The probes use separate outputs, and the model suite does not isolate scale, training data, alignment procedure, or weight access.Source: §E.5 · p. 25

E.6 Request Normalization Diagnostic

  • English normalization does not remove the closed-weight admission gap: pooled admission is 39.1% in native English versus 45.6%–50.0% across Chinese, Hindi, Igbo, and Swahili.The pooled increase does not hold for every model; this diagnostic tests admission after normalization, without retaining the original request as a separate safety-check input.Source: §E.6 · p. 25

F.1 Behavior-Clustered Primary Inference

  • Behavior-clustered inference finds admission increases in all six languages across all three groups; conditional harm rises in all six for closed-weight models and five of six for 27–31B models.The 3–14B group has conditional-harm decreases in Igbo, Lao, and Haitian Creole; coverage-conditioned yield rises in all closed-weight and 27–31B comparisons and in three 3–14B comparisons.Source: §F.1 · pp. 26–27

F.2 McNemar Paired Sensitivity Analysis

Paired sensitivity analyses support the admission and closed-weight conditional-harm effects, while response-language, audit, and per-model analyses clarify their scope. The results show language-related changes in both tool admission and post-retrieval outcomes, with non-harmful answers spanning refusals and language or realization failures.

F.2 McNemar Paired Sensitivity Analysis

  • McNemar sensitivity results agree with the admission and closed-weight conditional-harm effects; Table 42 remains the primary inference.The paired contrasts pool models and key on model plus behavior; page-language inference uses eight comparable-setting models, while 27–31B tests combine language and limit changes.Source: §F.2 · p. 27

F.3 Surface, Translation, and URL-Form Control Tests

  • All reported categorical-outcome homogeneity tests are significant, including tests of no-call harm, realization-audit taxonomy, and page-taxonomy audit groups.The reported tests include no-call harm across languages (χ2 = 98.0, dof = 6, p = 6.6 × 10−19) and realization-audit taxonomy across seven languages (χ2 = 168.1, dof = 12, p = 1.2 × 10−29).Source: §F.3 · p. 27

F.4 Realization Audit Counts

  • Among audited 3–14B answers, generic responses, misunderstanding, and low-fluency text occur at distinct rates across language and directive conditions.Under the page-language directive, Igbo/Swahili answers include 16.6% generic answers, 9.4% misunderstanding, and 35.2% low-fluency text; Chinese/Hindi generic answers are 17.0%.Source: §F.4 · pp. 27–28

F.5 Response-Language Identification and Intervention Details

  • With a page-language directive, 97.4% of closed-weight answers are classified as non-English, but language identification does not establish semantic realization.The English-response shares for 3–14B models are 3.5% in Chinese/Hindi and 10.7% in Igbo/Swahili; the directive is given before voluntary tool calling, so the full contrast can reflect admission and generation.Source: §F.5 · pp. 29–30

F.6 Response-Side SLIP by Language Set and Model

  • The closed-weight paired harm increase persists when any one model is omitted, while refusal shares fall in Chinese and Hindi and rise in Igbo and Swahili.Omitting any one closed-weight model leaves the pooled increase between 5.9 and 9.6 points with intervals above zero; translating page-language answers yields 95.3% label agreement and leaves the paired increase at +6.0 points.Source: §F.6 · pp. 30–31
  • In closed-weight models, page-language answers retain a paired harm increase of 6.0 points after translation rescoring; increases are concentrated in Chinese and Hindi.The paired increase remains +6.0 points (3.5–8.7; McNemar p < 0.001); the language-specific changes are +10.2 in Chinese, +11.5 in Hindi, +5.8 in Igbo, and −4.3 in Swahili.Source: §F.6 · pp. 30–31

F.7 Per-Model Admission-Side SLIP

  • Admission-side SLIP is positive for all ten models in Igbo, Swahili, Lao, and Haitian Creole, and the reported mean increase is not driven by a few models.Chinese and Hindi increase admission for every model except claude-sonnet-4, which decreases by 5.1 points in both.Source: §F.7 · p. 32, Figure 8 · p. 33
Loading 2610.23093v1…