Source-linked AI summary
ClaimMirage: When Self-Claims in Domain Names Change LLM Threat Judgments
Daiki Chiba, Hiroki Nakano, Takashi Koide
TL;DR
Short safety or approval claims embedded in a domain name can affect an LLM’s threat judgment without task-changing instructions. The paper compares claims with matched controls across brands, LLMs, and input settings, finding both alert reductions and increases. Even with brand and official-domain references, Llama 70B shows a 45.3-point registrable-label denial gap in the reported setting.
Problem
The paper examines whether short safety or approval claims inside the domain name being judged affect alert decisions, despite containing no task-changing instructions.
Source: §I · p. 1Method
The study compares ten claims with matched controls in constructed brand-like names across 64 brands and five LLMs, varying task wording, references, and name presentation.
Source: §I · p. 1Results
Both claim groups increased or reduced alerts across LLMs and settings; under Classifier role, Full reference, and Raw input, Llama 70B’s registrable-label denial gap was 45.3 points.
Source: §I · p. 1, A. Claim Terms and Naming Position · p. 5Takeaways & Limitations
The findings motivate testing self-claim sensitivity and seeking independent evidence before using claims in a domain name to suppress alerts.
Source: §V · p. 6, §VI · p. 6The constructed-name results do not establish live attack success or registration feasibility and cover fixed brands, terms, and one naming pattern.
Source: §V · p. 6
Abstract
from arXiv · showhide
Short claims such as not-phishing or official can change how a large language model (LLM) judges a domain name, without explicit prompt-injection commands. We study this manipulation as ClaimMirage: a name under inspection claims its own safety or approval. We analyze 622,080 judgments across 64 brands and five LLMs, comparing ten claims with length- and hyphen-matched controls in constructed brand-like names. Self-claims can substantially reduce or increase alerts, depending on the LLM and input setting. In one setting, risk-denial terms inside the registrable name reduce alerts by 45.3 percentage points even with a basic safeguard: the prompt supplies the potentially impersonated brand and its official domain for comparison. Without these references, endorsement terms at that position increase alerts by 65.6 points in the same LLM. References and component annotation remove some alert reductions but leave others or make them larger. These findings motivate testing resistance to self-claims and seeking independent evidence before treating a domain name under inspection as safe or authorized.
I. INTRODUCTION
ClaimMirage is the use of attacker-authored safety or approval claims in a domain name under judgment, without task-changing instructions. The paper measures these claims against matched controls and finds that their effects on alerts vary across LLMs and input settings.
I. INTRODUCTION
- ClaimMirage tests short self-claims inside the judged domain name as a manipulation distinct from injected, task-changing instructions.The study measures their effects against matched words across different input designs.Source: §I · p. 1
- Across 64 brands and five LLMs, both claim groups increased or reduced alerts, while references and component annotation removed some reductions but left others.The effect depended on the LLM and setting.Source: §I · p. 1
- The controlled comparisons use risk-denial and endorsement terms at two naming positions against matched controls, neutral labels, and unaltered names.Controls match character and hyphen counts, as in not-phishing versus not-archived.Source: §I · p. 1
- The study also examines how alert-rate differences persist, disappear, or reverse across three prompt factors and five LLMs, with false-alert and output-quality checks.The application may provide a possible brand match and its official domain for comparison.Source: §I · p. 1, Figure 1 · p. 1
II. BACKGROUND AND RELATED WORK
This section explains domain-name structure and how attackers can use misspellings or added words to imitate brands. It distinguishes a registrable domain from labels added as subdomains.
A. Domain Names as Security Evidence
- In help.example.com, com is the top-level domain, example the second-level label, and help a subdomain; example.com is the registrable domain.A registrant can normally create or delegate subdomains, while editing example changes the registrable domain itself.Source: A. Domain Names as Security Evidence · pp. 1–2
- Typosquatting uses misspellings, while combosquatting combines a brand name with other words; either can support phishing.The examples are exampel.test and example-login.test.Source: A. Domain Names as Security Evidence · p. 2
B. LLM Decisions and Attacker-Controlled Text
Prior work applies LLMs to phishing and deceptive-domain judgments, and examines how attacker-controlled text can steer LLM applications. ClaimMirage focuses on short claims in domain names, without crawling or retraining, and varies wording, references, and name presentation.
B. LLM Decisions and Attacker-Controlled Text
- Phishing-related systems use different evidence, including webpage vision and interaction, name-to-brand comparisons, URL and webpage representations, or search and page behavior.Other work classifies phishing emails rather than domain names.Source: B. LLM Decisions and… · p. 2
- LLM judgments can reflect learned brand-to-domain associations and input wording, while training collections and negation benchmarks provide related context.Input design can affect outputs.Source: B. LLM Decisions and… · p. 2
- Prior work studies attacker-controlled text through false official-site claims, indirect prompt injection, deceptive addresses, and log fields that steer security analysis.Benchmarks also test injection effects and instruction priorities.Source: B. LLM Decisions and… · p. 2
- Several defenses address prompt injection through prompt hardening, retrieval, output validation, structured-data training, detection, or checks on instructions and tool calls.These include InjectDefuser, StruQ, DataSentinel, and Task Shield.Source: B. LLM Decisions and… · p. 2
- ClaimMirage instead measures domain-name judgments with optional brand and official-domain references, comparing matched words at two positions while varying task wording and name presentation.The study does not crawl webpages or retrain models.Source: B. LLM Decisions and… · p. 2
A. Attacker and Classifier
The study models an attacker who controls an imitating name but not the brand’s official domain, and a classifier that judges one name using only supplied text and references. Its constructed candidates use fixed misspellings and a convenience set of 64 brands, defining the scope of the comparisons.
A. Attacker and Classifier
- The attacker can choose a brand-imitating name under a registered domain or delegated subdomain, but cannot modify the brand’s official domain.The added text makes a claim, not a command; the goal is fewer alerts than for similar names with control terms.Source: A. Attacker and Classifier · p. 2
- The LLM receives one candidate name and optional task wording, brand and official-domain references, and name presentation, but no webpage, DNS, ownership, or reputation evidence.Its Boolean decision raises an alert or does not; no alert does not establish brand approval.Source: A. Attacker and Classifier · p. 2
B. Candidate Names and Brands
- Each brand’s base candidate is <typo>-developer-test.com, built from one fixed misspelling and wording that can suggest either a legitimate test service or impersonation.The request does not state that the name is unauthorized.Source: B. Candidate Names and Brands · p. 2
- The fixed convenience set contains 64 brands, mainly technology and software services, each with an official-domain reference and predefined misspelling.Results describe this set, using brands as comparison units despite some shared corporate owners; the 64 official domains form the benign comparison set for false alerts.Source: B. Candidate Names and Brands · p. 2
C. Claims and Comparators
The study compares ten safety or approval claims with lexical controls in constructed brand-like names. Its comparisons are designed to separate claim wording from length and hyphen count, while acknowledging other wording-related differences remain.
C. Claims and Comparators
- Ten claims comprise five risk-denial terms and five endorsement terms, compared with matched controls and neutral labels.Denial controls retain “not-” but pair it with a security-unrelated term; unaltered base names show changes from adding text.Source: C. Claims and Comparators · p. 2, Table I · p. 3
- These comparisons do not isolate meaning from tokenization, familiarity, or plausibility.Source: C. Claims and Comparators · p. 3
- Each claim/control comparison holds the LLM, misspelled brand, naming position, task, references, and presentation fixed, changing only the added term.Each name is judged in a separate request.Source: C. Claims and Comparators · p. 3
- The 26 lexical strings yield 52 modified names across two positions; adding the base name and official domain gives 54 candidates per brand and setting.Shared comparators are reused across comparisons.Source: C. Claims and Comparators · p. 3
D. Three Prompt Factors
The evaluation varies task wording, reference condition, and name presentation across 18 prompt settings. Reference removal changes prompt wording, while raw and component-annotated presentations are evaluated together.
D. Three Prompt Factors
- Three task wordings, three reference conditions, and two name presentations produce 18 settings.The system message carries the task, references, and fixed JSON requirements; the user supplies one FQDN.Source: D. Three Prompt Factors · p. 3
- Classifier role, Risk assessment, and Threat decision use different task wordings, and their comparisons change the whole wording.Source: D. Three Prompt Factors · p. 3
- Removing references also changes clauses that use them, including declarations and comparison phrases.The wording changes differ by task and reference condition.Source: D. Three Prompt Factors · p. 3
- Raw FQDN presents the complete name as one string, while Annotated components separates labels and adds a reading note; the two changes are evaluated together.Neither presentation adds ownership evidence.Source: D. Three Prompt Factors · p. 3
E. LLMs and Collection
The collection covers five fixed LLM configurations and 622,080 judgments across brands, prompt settings, names, and rounds. A separate query measures whether models can recall official domains without references.
E. LLMs and Collection
- Five fixed served configurations include Llama 3.3 70B, three Gemma 3 sizes, and gpt-oss-20b.The configurations differ in precision, architecture, and serving, so results cannot be attributed to parameter count alone.Source: E. LLMs and Collection · p. 3
- The study collects 622,080 judgments across five LLMs, 64 brands, 18 prompt settings, 54 names per brand, and two rounds.Fixed shuffled schedules interleave modified, base, and official names; recovery filled 600 initially missing decisions without adding observations.Source: E. LLMs and Collection · p. 4
- An explicit Boolean and matching ID are extracted from the requested JSON object or one-item array, while duplicate keys or conflicting flags are rejected.Source: E. LLMs and Collection · p. 4
- A separate brand-to-domain query tests recall without selecting brands, collecting 640 answers without references.Answers are classified as reference matches, other domains, or abstentions.Source: E. LLMs and Collection · p. 4
F. Measures
The measures compare alert rates for claims and matched controls, averaging across terms, rounds, and equally weighted brands. A separate paired-evasion count captures repeated brand-level decisions that an average gap can hide.
F. Measures
- The alert-rate gap averages control-minus-claim differences across 64 equally weighted brands and multiplies by 100 to give percentage points.Each brand’s alert rate is computed over five pairs and two rounds; positive gaps mean fewer alerts with claims.Source: F. Measures · p. 4
- A negative gap can still include paired evasion for some brands, because alert increases for others may outweigh it.The paired-evasion count is out of 64 brands and requires the control to alert and the claim not to alert in both rounds.Source: F. Measures · p. 5
- The gap is not a relative reduction or attack-success rate, and subtracting rounded rates may differ by 0.1 point.Calculations use unrounded rates.Source: F. Measures · p. 5
- Official-domain false-alert rates and base-name alert rates provide basic quality checks, each using two judgments of 64 names per setting.Source: F. Measures · p. 5
IV. EVALUATION
Self-claims can either suppress or increase alerts, with effects varying by term, model, and input setting. References mitigate some reductions but do not make the effect uniform.
A. Claim Terms and Naming Position
- 45.3 points: with Full reference, Llama 70B’s registrable-label denial gap is 45.3 points, with 90.2% control alerts and 44.8% claim alerts.The five-term mean gap is positive for 62 brands and zero for two.Source: A. Claim Terms and Naming Position · p. 5, §IV · p. 5
- 65.6 points: without references, Llama 70B’s registrable-label endorsement gap is -65.6 points, with alerts rising from 32.3% for controls to 98.0% for claims.A negative gap here means more alerts on constructed names, not evasion.Source: A. Claim Terms and Naming Position · p. 5, §IV · p. 5
- With Full reference, Llama’s not-impersonating gap is 92.2 points with paired evasion for 59/64 brands, versus 16.4 points and 8/64 for not-phishing.Gemma 27B has zero gaps for every term with Full, but gaps of 8.6–51.6 points with None.Source: A. Claim Terms and Naming Position · p. 5
- For not-phishing, the gap is -2.3 points against matched not-archived but 73.4 points against neutral daily-record, so the comparator changes the apparent reduction.Denial gaps can also be small or reverse direction: Gemma 27B’s Full-reference gaps span 0–1.4 points, while gpt-oss-20b’s Threat decision, No reference, subdomain gap is -28.4 points.Source: A. Claim Terms and Naming Position · p. 5
B. Reference Information
- Gemma 27B’s Table V mean gap is 0.0 points with Full reference and 32.7 with None; its rates are 100% for controls and claims with Full, and 88.1% and 55.5% with None.Its official-domain false alerts are zero in both settings.Source: B. Reference Information · p. 5
- Brand-to-domain answers are identical across rounds for every LLM-brand pair, but reference matches cover 54–61 of 64 brands and do not show which associations affect threat triage.Gemma 4B abstains on ten; the matches establish recall, not influence on decisions.Source: B. Reference Information · p. 5
C. Task Wording and Presentation
Alert gaps vary across task wording and input presentation, and component annotation changes effects differently across LLMs and naming positions.
C. Task Wording and Presentation
- For Llama with Brand only, Raw FQDN, and registrable denial, gaps are 23.3, 50.9, and 64.5 points under Classifier, Risk assessment, and Threat decision, respectively.The LLM and claim alone do not specify the behavior.Source: C. Task Wording and Presentation · pp. 5–6
- Under Classifier role and Full reference, annotation reduces Llama’s subdomain denial gap from 41.6 to 0.0 points, but its registrable-label gap falls only from 45.3 to 19.8 points.All claims and controls then raise alerts in the annotated subdomain setting.Source: C. Task Wording and Presentation · p. 6
- Gemma 12B’s registrable-label denial gap rises from 23.8 to 51.9 points after annotation, while official-domain false alerts remain zero.Base-name alerts change from 98.4% to 96.9%.Source: C. Task Wording and Presentation · p. 6
D. Detection Quality and Repeatability
Detection-quality checks and repeatability results qualify the observed alert gaps, while the discussion limits their scope to constructed-name judgments and calls for further testing.
D. Detection Quality and Repeatability
- Gemma 4B’s official-domain false alerts span 16.4–78.1% with Raw FQDN and 75.8–100% with Annotated components, showing that gaps alone do not measure detection accuracy.Under Classifier role, Full reference, and Raw FQDN, Llama flags all base names and no official domains despite its large denial gap.Source: D. Detection Quality and Repeatability · p. 6
- Repeated-input disagreement ranges from 0.4% to 6.1% across LLMs over 62,208 complete pairs per LLM, so temperature zero does not guarantee identical threat decisions.Both judgments remain when flags differ.Source: D. Detection Quality and Repeatability · p. 6
V. DISCUSSION AND LIMITATIONS
- The comparisons are repeatable for exact LLM and prompt settings, but component annotation can remove, preserve, or increase gaps; independent evidence is needed before claims suppress alerts.The authors have not evaluated verification or the cited defenses.Source: §V · p. 6
- The study uses constructed names and name text with supplied references, so it does not establish live attack success or registration feasibility.Fixed brands, terms, and one naming pattern support descriptive comparisons; broader claims require tests on new brands, naming patterns, and legitimate irregular names.Source: §V · p. 6
VI. CONCLUSION
- Across 64 brands and five LLMs, constructed-name self-claims produce alert reductions, increases, and near-zero gaps, with effects changing across models and inputs.With Full reference, Llama’s registrable-label denial gap is 45.3 points; annotation leaves 19.8 points and removes its subdomain gap, but increases Gemma 12B’s registrable gap.Source: §VI · p. 6