PaperlayerView exact revision on arXiv

Source-linked AI summary

RouteProfile: Graph-Based Profiling for Cold-Start LLM Routing

Jingjun Xu, Hongji Pu, Tao Feng, Haozhen Zhang, Jiaxuan You, Ge Liu

arXiv:2605.00180v2cs.NIcs.CL

TL;DR

Cold-start routing struggles to integrate new LLMs without interaction data, while existing profiles are coarse or costly. RouteProfile builds heterogeneous graph profiles from public model signals and finds that structured profiles improve cold-start routing, with profile–router co-design important for new-LLM integration.

  • Problem

    New LLMs lack interaction data for router training, while available public signals are sparse, heterogeneous, and only partially comparable for profile construction.

  • Method

    RouteProfile organizes model families, descriptions, benchmark scores, and domains into heterogeneous graph profiles across structural, representational, aggregation, and learning dimensions.

  • Results

    Structured profiles improve training-free cold-start routing, model family metadata is more reliable than benchmark domains, and new-LLM integration benefits from profile–router co-design.

  • Takeaways & Limitations

    Profile design is important for adapting routing systems to newly released models and evolving model ecosystems.

  • Takeaways & Limitations

    Public signals are incomplete and inconsistently reported, potentially introducing reporting bias, and experiments do not cover all routers, model families, or deployment scenarios.

Abstract

from arXiv · show

LLM routing is increasingly important for selecting suitable models under diverse user needs and deployment constraints, but its practical effectiveness depends on continual adaptation to emerging queries and newly released models. New-LLM integration is particularly challenging, as newly released models lack the query-response-reward interactions required for router training and cannot be profiled as directly as new queries via semantic embeddings. Existing profiles are limited: LLM-generated descriptions are often coarse, while interaction-based embeddings are costly to construct. To address this problem, we propose RouteProfile, a graph-based profiling framework that constructs LLM profiles from public signals in technical reports or model cards, including model family, model description, reported benchmark scores, and benchmark domains. RouteProfile organizes these heterogeneous signals into a graph and studies profile construction along four dimensions: organizational form, representation type, aggregation depth, and learning configuration. We evaluate RouteProfile in training-free cold-start routing and new-LLM integration settings. Experiments show that: (1) structured profiles outperform flat baselines in training-free cold-start routing; (2) model family metadata is more reliable than benchmark domain information; and (3) effective new-LLM integration requires profile-router co-design. Overall, our findings highlight the importance of profile design for enabling routing systems to adapt to the evolving model ecosystem.

1 Introduction

Cold-start LLM routing must adapt to new models without query–response–reward interactions or direct semantic profiling, relying instead on sparse, heterogeneous public signals. RouteProfile addresses this challenge with a graph-based profile design that organizes these signals and evaluates construction choices across routing settings.

  • Problem: New-LLM integration is difficult because newly released models lack router-training interactions and cannot be directly profiled through semantic embeddings.Conventional integration consequently requires large-scale inference, data collection, and router retraining.
  • Problem: Cold-start profiles infer model capabilities from sparse, heterogeneous public signals such as family, descriptions, benchmark scores, and benchmark domains.These signals are only partially comparable because models are evaluated differently.
  • Prior limitations: Existing profiles are limited because one-hot index vectors lack semantic information, LLM-generated descriptions are coarse and incomplete, and benchmark summaries discard structured relationships.These limitations motivate profiles that are informative and structurally grounded.
  • RouteProfile: RouteProfile organizes heterogeneous public signals into a structured graph and studies profile construction through organizational form, representation type, aggregation depth, and learning configuration.The framework explicitly examines whether profiles leverage graph structure and how profile information is represented and aggregated.
  • Evaluation: The evaluation covers training-free cold-start routing with SimRouter and new-LLM integration with MLPRouter and GraphRouter without retraining the existing router.In the integration setting, the newly released model is added using only its public profile.

2 Related Work

Prior LLM-routing work frames routing as inference-time model assignment and primarily advances router design under quality, cost, or latency constraints. Separate profiling studies characterize model capabilities, while model-side signals are usually treated as auxiliary rather than as a standalone design problem.

  • LLM Routing: LLM routing assigns each query to a model at inference time under quality, cost, or latency constraints.
  • LLM Routing: Existing routing methods emphasize preference-trained, reward-guided, contrastive, and graph-based router designs.
  • LLM Routing: Model-side signals, including benchmark statistics, metadata, and structured benchmark–query–model relations, are typically auxiliary inputs rather than a standalone design problem.
  • LLM Profiling: Profiling studies explicitly characterize model capabilities through natural-language groups, latent skills, capability trees, and skill-based profiles for cost-aware routing.

3 Public Signals as a Heterogeneous Graph for Cold-start Profiling

Under cold-start constraints, RouteProfile constructs LLM profiles from coarse public signals rather than query-response-reward data. These heterogeneous sources are organized as a typed graph whose aggregated node representation defines each model profile.

  • Public Signals: Cold-start profiles rely on four public sources: model family metadata, model descriptions, reported benchmark scores, and benchmark domain information.Model family encodes a model’s structural prior, while domain coverage characterizes competence specialization across benchmark areas.
  • Graph Construction: Because public sources are sparse, heterogeneous, and only partially comparable, the framework represents them as a heterogeneous graph G = (V, E).Node and edge types are assigned through mapping functions ϕ : V →C and ψ : E →D.
  • Feature Initialization: Node features use type-specific initialization, including tailored descriptions generated by an additional LLM for model, domain, and benchmark nodes.The descriptions are generated from publicly available information, with all generated descriptions reported in Appendix A.2.
  • Profile Definition: A model profile pm is defined from the aggregated representation ˆxvm of its model node using an information aggregation function f over the heterogeneous graph.This formulation makes graph-based information aggregation the basis of the final LLM profile.

4 RouteProfile: Profile Design for Cold-Start LLM Routing

RouteProfile is a graph-based framework for designing LLM profiles from coarse public model signals for cold-start routing. It organizes profile construction around four dimensions that govern how heterogeneous information is represented, aggregated, and learned.

  • Profile inputs: RouteProfile constructs LLM profiles from model family metadata, model descriptions, reported benchmark scores, and benchmark domain information.These heterogeneous public signals are organized into a graph for profile construction.
  • Design dimensions: Profile design varies along organizational form, representation type, aggregation depth, and learning configuration.The framework emphasizes dimensions that materially affect routing while excluding overly task-specific choices such as particular LLMs or GNNs.
  • Organizational form: Organizational form determines whether graph structure is used, with structured profiles modeling relations through GNNs and flat profiles concatenating information into text or one vector.This dimension distinguishes relational aggregation from direct information concatenation.
  • Representation and aggregation: Representation type controls information fusion between textual descriptions and embeddings, while aggregation depth determines whether direct or higher-order neighborhoods contribute.Aggregation depth is represented by hop values from 0 through 4.
  • Learning configuration: Learning configuration specifies whether the aggregation function is training-free or trainable, with trainable variants potentially optimized through graph self-supervised learning.The formal configuration uses ℓ ∈ {Training-free, Trainable}.

5 Experimental Setup

The experimental setup evaluates how LLM profile design affects cold-start routing through heterogeneous graph construction, profile configurations, datasets, candidate models, tasks, and routers. It covers training-free routing and new-LLM integration using public-signal profiles and complementary router configurations.

  • Upstream profile construction: The heterogeneous graph spans 15 datasets across four capability domains and includes 25 LLMs from five model families, with eight serving as routing candidates.The remaining LLMs are auxiliary nodes that improve graph connectivity and evidence diversity.
  • Downstream routing evaluation: Downstream evaluation uses 12 datasets with 50 instances per dataset and a fixed pool of eight LLMs from five families spanning 3B to 176B parameters.The candidate families are Qwen2, Llama, Gemma2, Mistral, and Mixtral.
  • Evaluation settings: Training-free cold-start routing has no candidate query-response-reward interactions and uses SimRouter to isolate the effect of profile design.Performance is measured by average response performance across queries, with Oracle and Single-Best reported as reference points.
  • Evaluation settings: New-LLM integration adds Mistral-Small-24B-Instruct-2501 as a graph node using public information, then applies a fixed router without parameter updates.MLPRouter and GraphRouter are trained only on old-LLM query-response-reward data, and NCIR jointly captures new-LLM selection frequency and correctness.

6 Experimental Results

Profile design substantially affects training-free cold-start routing and new-LLM integration. Structured profiles improve cold-start performance, while effective unseen-model integration depends on embedding-based representations and alignment between profiles and routers.

  • Training-Free Cold-Start Routing: Structured profiles outperform flat PlainText in training-free routing, with TextGNN-4hop at 0.580 and EmbGNN-4hop at 0.577 exceeding Single-Best at 0.547.PlainText scores 0.532, below Single-Best’s 0.547, while sufficient aggregation depth closes the gap without router training.
  • Training-Free Cold-Start Routing: TrainGNN-1hop and TrainGNN-2hop both reach 0.613, surpassing TextGNN-4hop at 0.580 and approaching Oracle performance at 0.679.TrainGNN performance falls from 0.613 at 2 hops to 0.600 at 3 hops and 0.555 at 4 hops, consistent with over-smoothing.
  • Public Data Sources: Model family metadata consistently improves routing, including EmbGNN-3hop at 0.551 versus 0.500 and TextGNN-3hop at 0.552 versus 0.509.TrainGNN-3hop also improves from 0.502 to 0.519 when family information is added.
  • New-LLM Integration: PlainText yields NCIR = 0.000 for both routers, while TextGNN reaches at most NCIR = 0.007 with MLPRouter for new-LLM integration.Textual profiles can retain competitive average performance but provide little signal for integrating newly introduced LLMs.
  • New-LLM Integration: EmbGNN-3hop achieves NCIR = 0.411 with GraphRouter and 0.272 with MLPRouter, while delivering average performance of 0.613 and 0.624, respectively.TrainGNN-1hop reaches NCIR = 0.400 with GraphRouter but 0.000 across all hops with MLPRouter; EmbGNN-3hop is best with GraphRouter, whereas MLPRouter’s best EmbGNN result is 0.283 at 1 hop.

7 Conclusion

RouteProfile is a graph-based profiling framework for cold-start LLM routing that builds model profiles from public signals and organizes them into a heterogeneous graph. It supports systematic analysis of profile design across four dimensions and evaluation in training-free cold-start routing.

  • RouteProfile is proposed as a graph-based profiling framework for cold-start LLM routing.
  • Model profiles combine model family metadata, model descriptions, reported benchmark scores, and benchmark domain information from technical reports or model cards.
  • The framework organizes these public signals into a heterogeneous graph for systematic profile-design analysis.
  • RouteProfile studies profile design through organizational form, representation type, aggregation depth, and learning configuration, including training-free cold-start routing evaluation.

Limitations

RouteProfile is limited by incomplete, inconsistently reported public signals and by experiments that do not exhaustively cover all possible routers and model configurations.

  • Public-signal limitations: Public signals vary in evaluation protocols, benchmark coverage, and reporting granularity, potentially introducing reporting bias into constructed LLM profiles.The signals are drawn from technical reports or model cards.
  • Evaluation coverage: The experiments cover representative profile designs and router architectures but do not exhaustively evaluate all possible routers and model configurations.

A Appendix · A.1 Implementation Details

Implementation used one NVIDIA A6000 GPU, with profile construction requiring approximately 0.1–0.3 GPU hours depending on the GNN and candidate-LLM inference requiring approximately 15 hours via the NVIDIA API.

  • A.1 Implementation Details: Experiments were conducted on 1 NVIDIA A6000 GPU.
  • A.1 Implementation Details: Text-based GNN profile construction required approximately 0.2 GPU hours.The construction used iterative LLM calls through vLLM.
  • A.1 Implementation Details: Emb-based GNN profile construction required approximately 0.1 GPU hours in total.
  • A.1 Implementation Details: Trainable GNN profile construction experiments required approximately 0.3 GPU hours in total.
  • A.1 Implementation Details: Text-based GNN profile construction involved iterative LLM calls using vLLM.
  • A.1 Implementation Details: Candidate-LLM inference outputs were obtained via the NVIDIA API, requiring approximately 15 hours in total.

A.2 Data Sources for LLM Profile Construction … A.5 LLM Statistics

RouteProfile constructs interaction-graph profiles from heterogeneous public signals and initializes node types with textual, metadata-based, or encoded query features. LLM-based propagation then synthesizes local-neighborhood information, while datasets and models are separated by their roles in profiling and routing.

  • A.2 Data Sources for LLM Profile Construction: The interaction graph covers model family, model, benchmark, domain, and query nodes, using publicly available datasets and models for academic research.
  • A.2.1 Model Family Nodes: Model family nodes use natural-language descriptions of architectural design, training methodology, and general capabilities, with examples including Qwen2, Gemma2, and LLaMA.
  • A.2.2 Model Nodes: Model nodes combine inherited family descriptions with parameter count, instruction-tuning status, and available model-card information.
  • A.2.3 Benchmark Nodes: Benchmark nodes use natural-language benchmark descriptions, while domain nodes represent six high-level capability categories with descriptive text features.
  • A.2.4 Domain Nodes: The defined domains include Knowledge, Reasoning, QA, and Math, covering factual knowledge, inference, question answering, and mathematical problem solving.
  • A.2.5 Query Nodes: For each benchmark dataset, up to 1,000 queries are randomly sampled as query nodes and initialized by encoding raw query text with a pretrained language model.
  • A.3 Prompts for Text-based GNN: At every propagation hop, an LLM updates each node by synthesizing information from its local neighborhood through node-type-specific prompt templates.
  • A.4 Dataset Statistics: The datasets are divided between evidence-graph construction and routing evaluation, while the LLMs are divided into routing candidates and auxiliary graph-context models.
Loading 2605.00180v2…