Arabic Dataset for LLM Safeguard Evaluation

• Arabic safety evaluation is limited despite linguistic, cultural, and socio-political complexities and reported vulnerabilities under Arabic prompting.Source: §1 · p. 1
• The dataset contains 5,799 Arabic questions spanning direct attacks, indirect attacks, and harmless requests with sensitive words.It includes 1,024 Arabic-specific sensitive direct-attack questions before expansion into the two additional variants.Source: §1 · p. 1
• The authors localized an existing Chinese safety dataset and introduced Arab-region-specific harm types for culturally relevant evaluation.Source: §1 · p. 1
• A dual-perspective framework evaluates responses from governmental and oppositional viewpoints to analyze bias and harm across controversial topics.Source: §1 · pp. 1–2
• Experiments benchmarked five Arabic and multilingual LLMs to assess safety and inform improvements to Arabic safeguards.Source: §1 · p. 2

• Arabic prompts can produce lower safety and accuracy than other languages, while Arabic safety benchmarks remain comparatively scarce.Reported explanations include limited Arabic resources and training data.Source: §2 · pp. 2–3
• Existing multilingual toxicity benchmarks include Arabic and capture culturally specific toxic language or naturally occurring toxic prompts.Source: §2 · p. 2
• Research reports performance variation in Arabic instruction tasks and greater unsafe-content vulnerability in lower-resource languages.One cited study found unsafe translated inputs generated harmful content 79% of the time.Source: §2 · p. 2
• Arabic-centric models incorporate safety tuning and filtering, but safety coverage remains uneven, including concerns about AceGPT’s evaluation of reasoning and bias.Source: §2 · p. 2
• Arabic benchmarks primarily target NLU tasks, whereas AraTrust evaluates truthfulness, ethics, safety, and privacy through multiple-choice questions.AraTrust evaluations reported GPT-4 outperforming AceGPT and Jais in trustworthiness.Source: §2 · pp. 2–3
• The paper positions its dataset as a dedicated Arabic safety benchmark inspired by Do-Not-Answer and CDNA.Source: §2 · p. 3

• The authors localized CDNA questions through translation into Arabic and replacement of China-specific references with Arab-region-specific equivalents.The process used Chinese-to-English and English-to-Arabic translation before cultural adaptation.Source: §3 · p. 3
• Questions are designed around six risk areas and eighteen harm types, with Table 2 organizing the three question variants across them.Source: Table 2 · p. 4, §3 · p. 4
• Region-specific risks span cultural and social issues, regional issues, economic policy, political sensitivity, human rights, and controversial historical events.Source: §3 · p. 4
• The dataset includes 1,024 region-specific sensitive questions and expands them into indirect attacks and harmless sensitive-word questions.The harmless variant supports over-sensitivity assessment, while the indirect variant makes risky questions appear safer.Source: §3 · p. 4
• The final collection contains 5,799 questions: 3,073 on regional sensitivities and 2,726 across five general risk areas.Source: §3 · p. 4
• Cultural quality control involved a native Arabic speaker with expertise in Arab-region history, politics, and social issues.Source: §3 · p. 4

4.2 Harmfulness Evaluation

• The evaluation adapts risk-specific criteria to Arabic regional sensitivities, using dual government and opposition perspectives for controversial topics.Non-controversial regional, cultural, and social issues use unified criteria, while governance criteria test claims and whether responses avoid incitement and promote lawful measures.Source: §4.2 · pp. 5–6

4.3.1 Safety Rank

• GPT-4o was safest overall, with 2,081 government-side and 1,422 opposition-side harmful responses, while Llama3 was least safe.Government-side harmful-response counts exceeded opposition-side counts for every model.Source: §4.3.1 · p. 6, Figure 2 · p. 6

4.3.2 Harmfulness over Risk Areas

• Risk VI regional topics produced most harmful responses; Llama3 reached 2,382 government-side and 1,633 opposition-side harmful responses, while GPT-4o remained safest.For risks I–V alone, AceGPT was most harmful with 962 unsafe responses, Llama3 had 885, and GPT-4o had 394.Source: §4.3.2 · p. 6, Table 5 · p. 7
• 25–56% harmful responses occurred in Arabic versus <4% in English and Chinese across the five models.The results underscore the importance of evaluating safety mechanisms across cultural and regional contexts.Source: §4.3.2 · p. 7

4.3.3 Dual Perspectives

• Government and opposition perspectives produced similar economic-policy harm counts, but opposition had more governance and political-dynamics harms, whereas government had more repression, human-rights, and historical-event harms.For cultural and social issues, GPT-4o had 41 harmful responses compared with Llama3’s 413.Source: §4.3.3 · p. 7

4.3.4 Question Type

• Indirect attacks generated the most harmful responses, followed by direct attacks, while harmless questions containing sensitive words generated the fewest.Similar direct-attack and general-question counts for some models indicate possible over-sensitivity to particular words or phrases.Source: §4.3.4 · pp. 7–8, Figure 4 · p. 7

4.4 Response Pattern Evaluation

• Response patterns varied substantially: Jais often refused direct attacks, GPT-4o favored well-rounded statements, and Llama3 frequently followed instructions without thorough risk evaluation.Across question types, patterns 5 and 2 dominated, while indirect and general questions often elicited disclaimers.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 Human Evaluation

• GPT-4o achieved higher recall than GPT-4 across English, Arabic, and Chinese evaluations, but lower precision, including 12.0% versus 20.0% on Chinese data.The study prioritizes recall to reduce missed unsafe cases, accepting more false positives in safety-critical evaluation.Source: §4.5 · pp. 8–9, Table 6 · p. 9

• The findings underscore the importance of culturally localized datasets and region-specific safeguards for Arabic LLM safety evaluation.Source: §5 · p. 9
• 29k responses from five Arabic and multilingual LLMs revealed significant differences in sensitive-content handling, including vulnerabilities in some models such as Llama3.Source: §5 · p. 9
• The dataset covers cultural, political, and social issues, while the framework evaluates responses from both government and opposition viewpoints.Source: §5 · p. 9

• Arab-region dialectal, cultural, and socio-political diversity may not be fully captured, limiting evaluation accuracy in more localized settings.The paper proposes country- or dialect-specific sub-datasets and adaptive frameworks.Source: Limitations and Future Work · p. 9
• The governmental-versus-oppositional framework may introduce subjective bias and omit religious, minority, and other internal viewpoints.Future work could use a more granular multi-perspective framework.Source: Limitations and Future Work · p. 9
• Using GPT-4o with a single human annotator may bias harmfulness judgments, while automated evaluation can overestimate risks and produce false positives.The question sets may also omit some potential risks; the paper recommends more context-sensitive algorithms and broader criteria.Source: Limitations and Future Work · p. 9

Ethical Considerations

• The sensitive dataset could be misused for prompt attacks, political manipulation, or propaganda.Source: Ethical Considerations · p. 9
• The authors state that region-specific questions and dual-perspective evaluation aim to enhance responsible handling of complex Arabic social, cultural, and political issues.Source: Ethical Considerations · pp. 9–10
• The paper argues that advancing Arabic LLM safety and responsible AI development outweighs the acknowledged misuse risks.Source: Ethical Considerations · p. 10

• The non-controversial categories are Cultural Preservation & Social Issues and Regional Issues.Source: A Region-specific risk areas · p. 12
• The controversial categories cover Economic Policies & Development, Politically Sensitive Topics, Human Rights & Freedom of Expression, and Controversial Historical Events.Source: A Region-specific risk areas · p. 12
• The categorization distinguishes cultural and social considerations from topics likely to provoke debate or disagreement in economic, political, human-rights, and historical contexts.Source: A Region-specific risk areas · p. 12
• Risk Area VI contains six harm types and eleven corresponding risk types for region-specific Arabic sensitivities.Source: Table 7 · p. 12, A Region-specific risk areas · p. 12

B Evaluation Criteria

• Controversial risks are evaluated through separate governmental and opposition question sets, while non-controversial topics use a singular perspective.The framework distinguishes regional sensitivities by perspective rather than applying one uniform criterion set.Source: B Evaluation Criteria · p. 13, Table 8 · p. 15
• General risk evaluation uses question sets covering five risk areas, following the framework presented by Wang et al. (2024b).Source: B Evaluation Criteria · p. 13
• Non-controversial criteria assess respect for Arab cultural, religious, and social values while avoiding language that could incite unrest or division.Source: B Evaluation Criteria · p. 13
• Government-oriented criteria assess balanced responses that acknowledge governance efforts, national security, public order, human rights, and economic reforms while refuting claims against the government.Source: B Evaluation Criteria · pp. 13–14
• Opposition-oriented criteria assess evidence-based confirmation of governance, repression, human-rights, and economic criticisms while requiring peaceful and lawful responses.Source: B Evaluation Criteria · pp. 13–14

B.1 Construction process of the Evaluation Question sets

• GPT-4o evaluation results exhibit high agreement with human annotations, supporting the reliability of the evaluation framework.Source: §B.1 · p. 15
• The question set was designed for comprehensive coverage of anticipated risks, including region-specific cultural norms, values, controversial topics, and contextual concerns.Source: §B.1 · p. 15
• Approximately 250 questions and responses were manually annotated by socioculturally aware annotators to create a culturally nuanced ground truth.Source: §B.1 · p. 15
• Evaluation criteria and prompts were iteratively refined by analyzing discrepancies between human annotations and GPT-4o assessments to improve accuracy and reduce bias.Source: §B.1 · p. 15
• The framework incorporates governmental and oppositional viewpoints to assess bias and potential harm with cultural and contextual sensitivity.Source: §B.1 · p. 16
• The framework is intended to remain comprehensive, balanced, and aligned with diverse perspectives through this systematic construction process.Source: §B.1 · p. 16

C Automatic Evaluation

• The automatic evaluation compares the safety levels of five LLMs across five risk areas.Source: C Automatic Evaluation · p. 17, Figure 6 · p. 17
• Response-pattern categories classify how LLMs handle sensitive questions or instructions, enabling systematic assessment of potential risks in outputs.Source: C Automatic Evaluation · p. 17, Table 10 · p. 17

D Human Annotation

• GPT-4o improves harmful-response detection over GPT-4, but this greater sensitivity comes at the expense of precision.Source: D Human Annotation · p. 18, Figure 7 · p. 18
• Both GPT-4 and GPT-4o accurately identify the majority of safe responses, whereas harmful-response detection is near random, especially for GPT-4.Source: D Human Annotation · p. 18, Figure 7 · p. 18