Source-linked AI summary
Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models
Surya Narayanan Hari, Matt Thomson
TL;DR
Users face context-dependent model selection across workflows and domains, with additional goals such as computational cost, recency, and security. Tryage predicts expert-model performance on individual prompts and routes them using user constraints; on Pile masked language modeling, it identified the ideal model more accurately than GPT 3.5 Turbo and Gorilla.
Problem
Selecting and optimizing models for context-dependent workflows and domains is difficult, while users may also need to balance accuracy against goals such as model size, recency, and security.
Source: §1 · p. 2Method
Tryage uses a language-model perceptive router to predict expert-model accuracy on individual prompts and make routing decisions that incorporate user constraints.
Source: §1 · p. 2, Oracle Router Model · p. 5Results
50.8% accuracy for Tryage versus 23.6% for GPT 3.5 Turbo and 10.8% for Gorilla in identifying the ideal model for Pile masked language modeling prompts.
Source: Tryage outperforms Gorrila on… · p. 7Takeaways & Limitations
Users can incorporate constraints such as model size into routing and explore trade-offs between accuracy and other goals.
Source: §1 · p. 2, Oracle Router Model · p. 5The oracle formulation assumes complete per-prompt expert-model loss knowledge, which the paper describes as impractical.
Source: Oracle Router Model · p. 5
Abstract
from arXiv · showhide
The introduction of the transformer architecture and the self-attention mechanism has led to an explosive production of language models trained on specific downstream tasks and data domains. With over 200, 000 models in the Hugging Face ecosystem, users grapple with selecting and optimizing models to suit multifaceted workflows and data domains while addressing computational, security, and recency concerns. There is an urgent need for machine learning frameworks that can eliminate the burden of model selection and customization and unleash the incredible power of the vast emerging model library for end users. Here, we propose a context-aware routing system, Tryage, that leverages a language model router for optimal selection of expert models from a model library based on analysis of individual input prompts. Inspired by the thalamic router in the brain, Tryage employs a perceptive router to predict down-stream model performance on prompts and, then, makes a routing decision using an objective function that integrates performance predictions with user goals and constraints that are incorporated through flags (e.g., model size, model recency). Tryage allows users to explore a Pareto front and automatically trade-off between task accuracy and secondary goals including minimization of model size, recency, security, verbosity, and readability. Across heterogeneous data sets that include code, text, clinical data, and patents, the Tryage framework surpasses Gorilla and GPT3.5 turbo in dynamic model selection identifying the optimal model with an accuracy of 50.9% , compared to 23.6% by GPT 3.5 Turbo and 10.8% by Gorilla. Conceptually, Tryage demonstrates how routing models can be applied to program and control the behavior of multi-model LLM systems to maximize efficient use of the expanding and evolving language model ecosystem.
1 Introduction
The expanding model library makes prompt- and workflow-specific model selection difficult, while aggregate leaderboards do not capture performance on individual prompts. Tryage routes prompts to expert models using predicted performance and user constraints, and reports stronger model selection than Gorilla and GPT-3.5 Turbo.
1 Introduction
- Leaderboard averages are only rough guides for mixed-task, mixed-domain prompts, and leaderboard-based selection requires time-consuming testing and integration.Maintaining multiple models for one task can also be prohibitively expensive.Source: §1 · p. 2
- Users can trade off accuracy against goals such as model size, recency, security, and verbosity, including exploring a Pareto front as constraints or operating conditions change.The introduction gives reducing model size by 50% for a 2% accuracy change as an example trade-off.Source: §1 · p. 2
- Tryage uses a language-model perceptive router to predict downstream expert-model accuracy on individual prompts, then combines predictions with user flags to choose a model dynamically.The router contrasts with frameworks such as Gorilla and Langchain that route using model-card analysis.Source: §1 · p. 2
- Tryage achieves expert-level prompt performance across domain-specific datasets, exceeds RoBERTa by as much as 17.9% on specific domains, and outperforms Gorilla and GPT-3.5 Turbo in model selection.It also achieves aggregate performance superior to any individual model.Source: §1 · p. 2
Tryage model architecture and mathematical framework
Tryage selects expert models by analyzing individual prompts rather than relying on domain labels, combining estimated model performance with weighted user constraints. Its predictive router estimates downstream losses, while the routing objective makes the final model choice.
Preliminaries and Notation
- Tryage targets prompt-specific model selection because models differ across domains, while prompts may mix domains and often lack labels.The routing system combines downstream-model performance with user constraints in a single objective.Source: Tryage model architecture and… · p. 3, Preliminaries and Notation · p. 4
- The router's Q-function estimates each expert's loss for a prompt, and a routing function combines those estimates with weighted user goals to select a model.The Q-function is implemented as a language model trained to predict downstream losses.Source: Preliminaries and Notation · p. 5
Oracle Router Model
- The Oracle Router minimizes prompt loss plus weighted model constraints, with constraint weights allowing users to adjust trade-offs such as model size or recency.The oracle assumes complete knowledge of the Q table, which the paper describes as theoretically useful but impractical to construct for incoming prompts.Source: Oracle Router Model · p. 5
Predictive Router model
- Instead of evaluating every expert on each incoming prompt, the predictive router is trained on prompts and expert losses to estimate downstream performance dynamically.The router is a language model that outputs estimated losses for the models in the library.Source: Predictive Router model · p. 5
- Router models approximate masked-language-modeling loss within ϵ = .1 of true loss.The authors also note that router and downstream experts can be trained end-to-end, though they do not explore that implementation.Source: Predictive Router model · p. 6
Experiments with Tryage
The experiments evaluate Tryage on masked language modeling across Pile domains and an 11-model library, then examine model selection, domain routing, latent representations, and size–accuracy trade-offs. Across these analyses, Tryage selects models more accurately than GPT 3.5 Turbo and Gorilla, routes domain prompts to specialists, and can reduce compute by trading off accuracy.
Tryage outperforms Gorrila on unconstrainted model selection for MLM on Pile despite being 1000x smaller
- 50.8% model-selection accuracy for Tryage, versus 23.6% for GPT 3.5 Turbo and 10.8% for Gorilla; Tryage also achieved near-expert performance across the Pile test set.The authors note that the GPT and Gorilla estimates are overestimates because their outputs were credited when they contained any evidence for the best model, rather than only evidence for it.Source: Tryage outperforms Gorrila on… · p. 7, Figure 3 · p. 8
Tryage achieves interpretable routing of data to models
- 78% of Github examples, 98% of DM Mathematics examples, and 98.5% of USPTO Backgrounds examples were routed to the reported code or patent specialists.Tryage also routed PubMed Abstracts to patent models and CommonCrawl and OpenWebText2 to general English models.Source: Tryage achieves interpretable routing… · p. 7
Tryage matches expert level performance on mission critical, domain specific datasets
- 17.9% was Tryage’s largest reported performance gain over RoBERTa, with gains also reported for Freelaw, Stackexchange, DM Mathematics, BookCorpus2, and USPTO Backgrounds.The reported gains were 7.7%, 6.5%, 10%, 12.3%, and 5.02%, respectively; Figure 3 presents performance across subsets.Source: Tryage matches expert level… · p. 7, Figure 3 · p. 8
Tryage generates latent separation of data into domains
- Tryage’s UMAP showed domain clustering across data sources, while GPT-2 embeddings were diffuse and not clustered.The authors describe this unsupervised latent separation as interpretable because it indicates learned separation between data subsets.Source: Tryage generates latent separation of… · p. 9, Figure 4 · p. 9
Tryage enables routing and model selection along Pareto front
- A size penalty let users trade accuracy for compute: the reported setting reduced combined accuracy by 5% while saving over 50% of compute.As λ increased, allocations shifted from mostly large models through a mixture of sizes to predominantly smaller models.Source: Tryage enables routing and model… · p. 9