PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

Teng-Ruei Chen

arXiv:2610.09981v1cs.CRcs.CL

TL;DR

Routing metadata can expose the chosen model and caller even when content logging is off, yet prior work did not examine that post-routing record as a privacy channel. We measure the channel on real traffic and evaluate defenses, finding that routing shifts vary by category and router.

  • Problem

    Prior work did not examine the logged routing decision as a privacy channel, although systems can record the chosen model and caller without content and routes may reveal sensitivity.

    Source: §I · p. 1

  • Method

    We measure routing metadata on real traffic, survey systems’ logging, and test post-processing defenses.

    Source: §I · p. 2, A. System model · p. 4

  • Results

    At matched length, shifts varied by category and router: RouteLLM sent harassment and self-harm 19 points less often to the strong model on unseen prompts, but sexual requests 10 points more often.

    Source: A. The shift’s direction depends on… · p. 9, §XI · p. 16

  • Takeaways & Limitations

    Length-matched per-category parity removes the real-traffic gap when labels are accurate, at a cost of at most 0.2 accuracy points on RouterBench.

    Source: §I · p. 2

    Medical results are exploratory because the judges failed both pre-registered label gates.

    Source: §X · p. 15

Abstract

from arXiv · show

LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off. We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts. We run pre-registered studies on 1.7 million real requests (WildChat-1M, LMSYS-Chat-1M) with two cost/quality routers and a domain router, survey eleven systems' logging, and test post-processing defenses. At matched length, the shift's direction depends on category and router. For RouteLLM at the 50% operating point, harassment and self-harm requests reach the strong model 19 points less often than comparable ones on prompts unseen in exploration, medical requests (exploratory: LLM labels failed their gate) 31 points less often on distinct prompts (both post hoc), and sexual requests 10 points more often (secondary); the other router's four are negative. Twenty RouteLLM decisions separate frequent medical askers with AUC 0.71, exploratory and below the pre-registered primary endpoint's 0.75 (domain router: 0.92, an upper estimate). Per-category length-matched parity with accurate labels removes the gap on real traffic, costing at most 0.2 accuracy points on RouterBench (post hoc), where routers' gaps on sensitive subjects (13-42 points, pre-registered) exceed those of an oracle routing by realized accuracy gain (1-11, post hoc). Per-conversation stickiness, per-user budget bands, and pooled parity fail, the last as categories' shifts differ in size or sign. A post hoc exact per-user rate hides only even-prefix strong counts and forfeits most self-assessed routing value; it preserves odd-position decisions, from which a post hoc log attack reaches AUC 0.73 after 20 RouteLLM requests (exploratory).

I. INTRODUCTION

Routers make content-dependent model choices that can remain visible in gateway metadata even when content logging is off. This paper measures what those choices reveal on real traffic and evaluates defenses, addressing a gap in prior routing and privacy work.

I. INTRODUCTION

  • The study treats logged routing decisions as a privacy channel and measures leakage per request and across users without access to request content.Its measurement accounts for length, noisy labels, repeated prompts, and the choice of router, dataset, and operating point.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

I. INTRODUCTION

  • At matched length, RouteLLM’s shifts vary by category: harassment and self-harm reach the strong model less often, while sexual requests reach it more often.Medical requests also show a negative shift, but their exploratory labels failed the validity gate.Source: §I · p. 2

A. Cost/quality and domain routing

  • Prior work studies router cost and quality, category-dependent decisions, privacy-preserving routing, and other selection side channels, but not logged model-router decisions on real user traffic.The paper identifies its gap as measuring what those records reveal per request and per user, and what closing the channel costs.Source: §I · p. 1, A. Cost/quality and domain routing · p. 3

B. Privacy-preserving routing

Prior work protects routing inputs or studies other selection side channels, while this paper examines model choices recorded in operational metadata. It surveys who can access those records and measures their leakage and defense costs on real traffic.

C. Side channels of adaptive inference and LLM traffic

D. Defense lineage

  • The paper applies fairness post-processing to routing and does not claim a new theory or inherit RAPPOR’s differential-privacy guarantee.Its exact per-user-rate defense requires persistent per-user state.Source: D. Defense lineage · p. 4

A. System model

  • In at least six of eleven surveyed systems, per-request records can link a served model and time to a caller without content; metadata-only logging is default in OpenRouter and LiteLLM.Five routing products also return or record the routing decision as a named field.Source: A. System model · p. 4, §III · p. 4

B. Observers

  • The main adversary is a gateway insider who can read each user’s ordered routing decisions without enabling content logging.Other observers include organization members, security teams, finance, hosted vendors, and subprocessors, with access varying by system.Source: B. Observers · p. 4

C. Attacker capabilities and assumptions

The threat model centers on observers who see routing metadata but not request content, including insiders and finance observers. The paper frames this as a privacy concern rather than a vulnerability in systems that retain metadata for billing or monitoring.

C. Attacker capabilities and assumptions

  • The attacker sees routing decisions and message token counts but never content, and needs no router internals.The measured leakage is what routing reveals beyond request length; the assumed attacker has labels for some users and is fitted on half of them.Source: C. Attacker capabilities and assumptions · p. 4

D. Why not enable content logging

  • Metadata can matter when content logging is prohibited, when it is retained longer than content, or when observers cannot enable content logging.The paper describes possible employer inference about employees’ health or self-harm requests and risks from retained metadata.Source: D. Why not enable content logging · pp. 4–5

E. Framing

  • The paper does not treat metadata retention as a vulnerability; its narrower claim is that disabling content logging alone does not hide topics when routing depends on content.Latency and timing channels are outside the study’s scope.Source: E. Framing · p. 5

IV. MODEL AND METHODOLOGY

  • The original measurements are taken or recomputed from released result files without alteration; the odd-position follow-up is reported separately.The supplied passage identifies Table IX as the location of the follow-up results.Source: §IV · p. 5

A. System and observer model

The section defines the router’s recorded decision and the observer’s view, then specifies how to measure category-related leakage beyond request length. It also describes the data and labeling constraints that shape those measurements.

A. System and observer model

  • The observer sees the caller, time, routing decision, input-token count, and cost, but not request text; the strong-model rate is set at a threshold calibrated on comparison requests.For the two score routers, the threshold makes fraction q of comparison requests reach the strong model; q defaults to 0.5.Source: A. System and observer model · pp. 5–6

B. Data, routers, and labels

  • The study uses WildChat-1M, LMSYS-Chat-1M, RouterBench, two cost-quality routers, and a domain router, with medical labels failing the pre-registered validity gate.Medical-label precision was 0.75 on LMSYS and 0.70 on WildChat, below the pre-registered 0.85 threshold.Source: B. Data, routers, and labels · p. 6

C. Per-request leakage

  • For a single routing decision, AUC is 1/2 + 1/2|p_c − p_O|; on WildChat medical requests, the cross-validated AUC is 0.651.The corresponding absolute strong-rate difference is 0.308, giving a formula-based AUC of 0.654.Source: C. Per-request leakage · p. 6
  • Per-request leakage is assessed with length-matched gaps and a specified attacker’s AUC gain over token counts, not as a distribution-free information bound.The specified logistic-regression attacker adds a routing coefficient to a model using quantile bins of log length.Source: C. Per-request leakage · p. 6

D. Estimation and replication

The estimation procedure accounts for user dependence and repeated prompts, then recomputes the leakage measures under resampling. The replication protocols and their deviations are tracked alongside the estimates.

D. Estimation and replication

  • WildChat intervals resample users, while LMSYS intervals resample requests because user identifiers are unavailable; both analyses also account for repeated prompts through fingerprint clusters.The fingerprint is a proxy for text identity, not a text-identity test.Source: D. Estimation and replication · p. 7
  • The procedure sets the comparison-group threshold, computes length-matched gaps and privacy loss, resamples to form intervals, and estimates AUC gain over token counts.The comparison set is held fixed for the held-out replication’s fingerprint-cluster resampling.Source: D. Estimation and replication · p. 7
  • The studies were pre-registered, but WildChat studies reused one dataset and each was frozen after its predecessor’s results were known.The odd-position follow-up was post hoc and exploratory, not a new pre-registered endpoint.Source: D. Estimation and replication · p. 7

E. User-level attackers

The section defines user-level attackers that infer category membership from routing records, then describes post-processing defenses and the limits of what they guarantee. Its results establish design properties and caveats, rather than empirical defense performance.

E. User-level attackers

  • The user-level attacker classifies frequent category askers from prefixes of their routing records, using either a strong-decision count or itemized decisions and length.Users with at least 10 requests are included; positive users have at least 20% of requests in the category, while users with none are negative.Source: E. User-level attackers · p. 7
  • The post hoc A2-odd attacker estimates and scores users using only odd-position decisions from each ordered prefix.For total prefix lengths of 5, 10, 20, and 50 requests, it uses 3, 5, 10, and 25 decisions, respectively.Source: E. User-level attackers · p. 8
  • With user-specific routing rates, the strong-count attacker’s AUC can saturate below one when user-level heterogeneity leaves positive and negative rates overlapping.Without heterogeneity, its limiting AUC is one when the class-specific mean rates differ in the direction used by the attacker.Source: E. User-level attackers · p. 8

F. Defenses

  • The listed post-processing options include mixing, length-matched parity, per-user budget bands, and reusing an initial decision across a conversation or user.Per-category parity requires no per-user state or router retraining; its online version needs calibrated category-by-length cutoffs and does not inherit the batch guarantee under distribution shift.Source: F. Defenses · p. 8
  • Per-category parity assigns each category and the comparison set the same strong rate within each global length bin, preserving each bin’s budget up to rounding.The stated guarantee is within-bin parity; the reported matched-length gap can retain a residual because it uses different bins and comparison sets.Source: F. Defenses · p. 8
  • Pooled parity constrains only weighted-average rates, so categories may retain opposite-signed gaps when category score distributions differ.Their post-parity directions are not determined by their undefended gaps alone.Source: F. Defenses · p. 8
  • Under an iid Bernoulli input model, boundary occupancy can favor one side depending on the input rate, but this does not guarantee leakage for arbitrary correlated sequences.The remark identifies a possible boundary leak, not a general result for all request sequences.Source: F. Defenses · p. 8
  • The exact-rate rule fixes every even-prefix strong count, but its ordered transcript preserves the odd-position input decisions verbatim.The information identity does not require independence; an observer that uses position can therefore retain information absent from the even-prefix count.Source: F. Defenses · p. 8

G. Real-unit evaluation

This section defines a post hoc oracle router using realized accuracy gain and compares its leakage gaps with those of per-category parity. The oracle is one quality-optimal routing choice, not the only one.

G. Real-unit evaluation

  • The oracle selects prompts by realized accuracy gain, sending the budgeted share to the strong model and breaking ties at random.The passage characterizes this as one quality-optimal router, not the only one.Source: G. Real-unit evaluation · p. 9
  • With per-category parity, the post hoc oracle keeps its accuracy while its largest gap falls to 0.031 or less.The oracle routes the budgeted number of prompts with the largest realized strong-versus-weak accuracy gains.Source: G. Real-unit evaluation · p. 9

V. PER-REQUEST LEAKAGE

The section examines per-request routing differences on real traffic, their relationship to repeated prompts, and how they compare with an accuracy oracle.

  • The section measures how each sensitive category is routed relative to comparable real-traffic requests.Source: §V · p. 9
  • It tests whether the observed differences persist after accounting for repeated prompts.Source: §V · p. 9
  • It compares the routing differences with those of an accuracy oracle.Source: §V · p. 9

A. The shift’s direction depends on the category and the router

On real traffic, per-request routing shifts vary by category and router, and some remain on prompts unseen during exploration. The section also examines what routing adds beyond token counts and compares router gaps with an accuracy oracle.

B. Repeated prompts inflate the replication

  • 61% of held-out harassment rows repeat an exploration prompt, making the nominal replication less independent; RouteLLM’s harassment and self-harm shifts persist on never-seen prompts.On never-seen prompts, notdiamond’s self-harm interval includes zero; WildChat medical ∆adj is −0.312 on distinct prompts versus −0.331 on all rows.Source: B. Repeated prompts inflate the… · p. 9

C. What routing adds to token counts

  • Routing adds ∆AUC of +0.11 for medical requests, +0.05 to +0.06 for self-harm, and about +0.01 for harassment and sexual requests beyond a flexible length model.The frozen log-linear model’s +0.18 harassment gain was withdrawn; a 20-bin model measured +0.014, while length alone had AUC 0.78 for harassment and 0.75 for sexual requests.Source: C. What routing adds to token counts · p. 9

A. The shift’s direction depends on the category and the router

D. The routers’ gaps far exceed an accuracy oracle’s

  • Routers’ sensitive-subject gaps are 13–42 points, versus oracle gaps of 1–11 points; most observed differentiation is not explained by realized accuracy gain.The oracle comparison is post hoc and uses realized per-prompt correctness; the strong model’s accuracy gain is 0.05–0.12 lower on these subjects than on non-sensitive prompts.Source: D. The routers’ gaps far exceed an… · p. 9, Figure 3 · p. 11

VI. USER-LEVEL INFERENCE

Using WildChat medical requests, the paper tests what a user’s routing sequence reveals and whether post-processing defenses reduce per-request and user-level leakage. The first 20 RouteLLM decisions miss the pre-registered AUC threshold, while aggregate bills retain most of the signal and defenses leave important leakage paths.

A. Accumulation over the evaluated horizons

  • AUC 0.707 [0.671, 0.740] from a user’s first 20 RouteLLM requests missed the pre-registered 0.75 threshold; medical-label results are exploratory.Length alone gives 0.54 on the same users, while count plus length gives 0.72.Source: A. Accumulation over the evaluated… · p. 11
  • The topic-only null predicts AUC 0.83 at T=20 and 0.90 at T=50, above observed 0.707 and 0.736; eligibility changes across horizons.Routing rates vary within label classes (ρ = 0.35 among negative users and 0.21 among positive users), so the curve does not establish an asymptotic ceiling.Source: A. Accumulation over the evaluated… · p. 11, Figure 4 · p. 11

B. An aggregate bill carries most of the signal

  • A strong-model count over the first 20 requests reaches AUC 0.707, while hiding per-request records therefore offers little protection.A stylized total cost gives 0.65 at either tested strong-model price ratio, and total tokens give 0.57.Source: B. An aggregate bill carries most of… · p. 11

A. Per-request gaps close cheaply; user-level inference persists

  • Per-category parity with perfect labels removes the per-request gap (−0.001) and retains 0.99 of router skill, but user-level AUC remains 0.60.With the cheap classifier, parity removes only half the gap (−0.151); the primary defense combining it with a ±2 budget band fails all four criteria.Source: A. Per-request gaps close cheaply;… · p. 12

B. Budget bands leak through their boundaries

  • A two-sided budget band still leaks: after 20 requests, the count identifies frequent medical askers with AUC 0.67–0.71 across slacks.The post hoc exact rate hides the strong count only at even request counts, preserves odd-indexed decisions, and has skill 0.23.Source: B. Budget bands leak through their… · p. 12

C. Odd positions retain user-level signal

The exact per-user rate suppresses even-prefix strong-model counts but leaves ordered routing decisions exploitable. Per-category parity performs better in real units than pooled parity, while tested user-level defenses remain incomplete.

C. Odd positions retain user-level signal

  • At T=20, odd-position attacks reach AUC 0.730 under RouteLLM’s exact rate and 0.667 with cheap-classifier parity before the exact rate.The attack uses ten routing decisions; the follow-up is exploratory and inherits the failed medical-label gate.Source: C. Odd positions retain user-level… · p. 12
  • Even-prefix strong counts remain at chance under the exact rate, but odd-position routing preserves user-level signal.The attack uses a different observer, and its achievable performances are not upper bounds on inference.Source: C. Odd positions retain user-level… · p. 12

D. Real accuracy and dollars

  • In RouterBench, for actual routers, per-category parity changes accuracy by at most 0.2 points and lowers cost by 2–4%.The post hoc oracle routes by realized accuracy gain, with ties broken at random, and is an upper bound at the router’s budget rather than a deployable router.Source: D. Real accuracy and dollars · pp. 12–13
  • RouterBench tests router self-assessment: RouteLLM is 1.30 points worse than random routing at the same budget, and its “skill kept” reflects the router’s own belief, not measured quality.Source: D. Real accuracy and dollars · p. 12
  • Pooled parity can leave category gaps because their sizes differ or their directions oppose; per-category parity is needed to address those differences.For RouteLLM on RouterBench, pooled parity moves the medical gap from −0.24 to +0.14 while psychology remains at −0.14; on LMSYS, sexual and harassment shifts offset each other.Source: D. Real accuracy and dollars · p. 12

A. Recommendations

  • Developers are advised to use length-matched parity for a single category or disjoint partition only with validated labels and calibrated cutoffs.With accurate labels it equalizes strong rates within global length deciles; a cheap candidate flag moves the medical gap only from −0.331 to −0.151.Source: A. Recommendations · pp. 13–14
  • Gateway operators should not treat per-conversation stickiness or two-sided per-user budget bands as privacy controls.Stickiness lowers user-level AUC only from 0.72 to 0.69; under a ±1 band, 60.5% of frequent medical askers end at the lower edge versus 34.4% of users with no medical requests.Source: A. Recommendations · p. 14
  • No tested option fully addresses user-level inference: the exact rate hides strong counts only at even request counts and preserves exploitable ordered decisions.It keeps 0.23 of the router’s self-assessed skill; the post hoc odd-position attack reaches AUC 0.793 at T=50.Source: A. Recommendations · p. 14

B. What the pre-registration record shows

The pre-registration record includes missed endpoints and failed label gates, and the held-out replication was less independent than intended because many prompts repeated. The ethics disclosure details dataset constraints, handling departures, and maintainer notifications.

B. What the pre-registration record shows

  • Both judge-label gates failed, medical results remain exploratory, and no pre-registered defense endpoint was met.The user-level primary endpoint was 0.707 against 0.75; parity with a budget band failed all four criteria, and pooled parity failed its RouterBench leakage criterion.Source: B. What the pre-registration record… · p. 14
  • Although both held-out RouteLLM endpoints were met, 61% of held-out harassment rows repeated an exploration prompt.On never-seen prompts, the post hoc gaps were −0.193 and −0.186 and still passed; the authors treat these as robust estimates.Source: B. What the pre-registration record… · p. 14

IX. ETHICS AND RESPONSIBLE DISCLOSURE

  • The study used public datasets under distinct constraints, including an LMSYS license that forbids identifying individuals and inferring covered sensitive personal data.The authors say the license does not clarify whether classifying message topic counts as such inference, and they do not claim their use is covered.Source: §IX · p. 14
  • Earlier studies departed from handling rules four times, including sending snippets and E-5 reference-label lines to an AI coding assistant.The assistant assigned E-5 labels non-blind, and agreement may be overestimated; after the audit, conversation-text reading was moved off the assistant.Source: §IX · p. 15
  • The authors notified RouteLLM, Not Diamond, and vLLM Semantic Router before submission and do not classify the finding as a software vulnerability.They attribute the leak to content-dependent routing combined with standard metadata logging, rather than a defect to patch.Source: §IX · p. 15

X. LIMITATIONS

The evidence is bounded by noisy labels, repeated prompts, limited router and user coverage, and evaluation choices. The conclusion reports category-dependent routing shifts and defenses, while noting that exact-rate defenses leave ordered decisions exposed.

X. LIMITATIONS

  • Medical judges failed both pre-registered label gates, so every medical result is exploratory and its magnitude is uncertain.The LMSYS comparison used one human annotator; the WildChat reference labels came from an AI coding assistant non-blind, and agreement may be overestimated.Source: §X · p. 15
  • Prompt fingerprints are only a duplicate proxy: distinct texts can share one, while near-duplicates can have different fingerprints.The held-out replication is therefore less independent than intended, and LMSYS fingerprint-cluster intervals hold the comparison set fixed.Source: §X · pp. 15–16
  • The study covers two open cost/quality routers and one domain router; other architectures and hosted commercial routers were not measured.The domain-router health label is treated as the decision, not mapped to models under its default configuration.Source: §X · p. 16
  • User-level evidence covers only medical requests with exploratory labels on WildChat, the sole used dataset with user identifiers.The primary endpoint was 0.707 against 0.75 at T=20; inference for harassment, self-harm, and sexual requests is unmeasured.Source: §X · p. 16
  • RouterBench’s academic MMLU subjects and recorded 2023–2024 prices limit how its real-unit results transfer to users’ own chats.The oracle comparison and real-unit cost of per-category parity are post hoc; user-level defense costs remain in router units.Source: §X · p. 16
  • Length-decile parity can leave residual gaps, up to 0.064 on RouterBench for RouteLLM law questions.Parity was tested against the original observers, not the odd-position attacker, and observers using exact token counts or other fields may recover residual signal.Source: §X · p. 16
  • Leakage depends on the operating point: RouteLLM’s WildChat medical gap ranges from −0.35 at 55–60% strong calls to −0.14 at 20%.The authors report the whole operating-point range; the package-default uncalibrated RouteLLM configuration sent every task to the weak model.Source: §X · p. 16

XI. CONCLUSION

  • At matched length, RouteLLM’s medical requests reach the strong model 31 points less often on distinct WildChat prompts, but those labels are exploratory.On RouterBench, per-category parity leaves a largest gap of at most 0.064 for at most 0.2 accuracy points; the exact-rate defense still preserves odd-position decisions.Source: §XI · pp. 16–17

APPENDIX SUPPORTING TABLES

The appendix supplies supporting tables for leakage, user-level inference, defenses, and odd-position attacks, and documents the released research artifacts and limits on what is shared.

APPENDIX SUPPORTING TABLES

  • The appendix reports full numbers for RouterBench leakage, WildChat user-level inference and defenses, and all 16 post hoc odd-position configurations.Tables VI–VIII provide the first three analyses; Table IX includes paired differences and uncertainty.Source: APPENDIX SUPPORTING TABLES · p. 17

ARTIFACT STATEMENT

  • The artifact excludes the draft E-7 label-error simulation protocol, which was never frozen, while releasing the aggregate simulation results.The artifact statement also describes the private repository history and the text-free per-request data exclusions.Source: ARTIFACT STATEMENT · p. 17
  • The artifact releases study records, analysis code, aggregate results, router revisions and hashes, and gold-set labels, while withholding text-free per-request tables.The per-request tables are excluded because router scores with token counts can link rows to source data, and dataset licenses restrict redistribution or are unverified.Source: ARTIFACT STATEMENT · p. 17
  • Table VII defines medical-user positivity as at least 20% medical requests versus none, and distinguishes aggregate-count, length-adjusted, and itemized-log attacks.The domain-router estimate is an upper estimate because its labels helped select what judges read.Source: Table VII · p. 18
  • RouteLLM’s pre-registered primary endpoint was A1 at T=20 on the E-5 split, 0.707 [0.671, 0.740], below 0.75; the medical-label analysis was exploratory after its validity gate failed.The endpoint is identified as P2.Source: ARTIFACT STATEMENT · p. 18
  • For the post hoc exact-rate defense, the strong count is fixed and AUC is 0.500 only at even T; the reported attack assumes the complete ordered prefix and starting phase are observable.The attack AUCs are achievable performance, not upper bounds on inference; at T=50, both notdiamond difference intervals include zero.Source: ARTIFACT STATEMENT · pp. 18–19

Related papers

Loading 2610.09981v1…