Source-linked AI summary
Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems
Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas, Iraklis Anagnostopoulos
TL;DR
Function-calling LLMs are often sent to powerful cloud models despite their energy and carbon costs, while smaller edge models have query-dependent accuracy limits. The paper predicts per-query performance and routes across edge and cloud tiers using grid carbon intensity. It reports 4× average carbon reduction while maintaining competitive accuracy and near-cloud inference delay.
Problem
Function-calling inference is often routed to powerful cloud models despite their energy and carbon costs, while smaller edge models have query-dependent accuracy gaps.
Source: §I · p. 1Method
A lightweight k-NN predictor estimates edge-tier success, delay, and power, which a router combines with real-time grid carbon intensity to select a qualifying tier.
Source: §I · p. 2, §II · p. 2Results
The framework maintains competitive accuracy and inference delay close to cloud levels while reducing emissions by 4× on average and up to 8× for simpler queries.
Source: §IV · p. 4Takeaways & Limitations
The results present query-level heterogeneity as a practical path to sustainable agentic AI.
Source: §IV · p. 4
Abstract
from arXiv · showhide
Large Language Models (LLMs) with function-calling capabilities are becoming critical for modern agentic AI systems. Nevertheless, current deployments typically route inferences to powerful cloud-based models, incurring significant energy use and carbon emissions. We address this sustainability challenge with a carbon-aware routing framework that distributes function-calling queries across a three-tier edge-cloud architecture, combining edge and cloud LLMs on heterogeneous hardware. At its core, a lightweight k-NN predictor operating in a unified semantic-lexical embedding space estimates query-specific accuracy, delay, and power consumption on each edge tier. These predictions are then combined with real-time grid carbon intensity to route every query to the lowest-emission tier capable of executing it successfully. Evaluated on state-of-the-art function-calling benchmarks and LLM families, our framework matches cloud-level accuracy while reducing operational carbon emissions by $4\times$ on average.
I. INTRODUCTION
Function-calling agents are often routed to powerful cloud models, while edge models use less energy but have uneven accuracy across queries. The paper proposes three-tier carbon-aware routing using per-tier predictions and grid intensity, reporting average carbon reductions of 4×.
I. INTRODUCTION
- Inference accounts for 80-90% of cloud ML compute demand and 60% of ML-related energy consumption.ChatGPT alone is reported to consume approximately 850 megawatt-hours per day; the paper notes that an always-cloud default amplifies the trend.Source: §I · p. 1
- On BFCL V2, Llama3.1-70B reaches 97.8% success versus 56.5% for Llama3.1-8B and 13.6% for Llama3.2-1B, while edge models use approximately 25× less energy per query than 70B counterparts.Table I is scoped to BFCL V2 success rate, per-query energy, and estimated carbon emissions.Source: §I · p. 1, Table I · p. 1
- Existing LLM routers balance accuracy against dollar cost but ignore hardware, power, and grid carbon intensity.The paper frames function-calling inference as a Performance–Power–Carbon constrained routing problem.Source: §I · p. 1
- The framework spans 1–4B edge-SoC, 7–12B edge-accelerator, and >100B cloud models, using k-NN predictions to select a lowest-emission tier meeting accuracy constraints.The predictor estimates per-tier correctness, delay, and carbon footprint from historical executions.Source: §I · p. 2
- 4× average carbon reduction, and up to 8× for simpler queries, while preserving cloud-level success rates.The routing policy uses performance predictions and real-time grid carbon intensity to choose a tier that meets accuracy requirements.Source: §I · p. 2
II. METHODOLOGY
The method predicts how each query will perform on edge tiers, then routes it to the lowest-carbon eligible tier. If no edge tier meets the accuracy threshold, it uses the cloud tier.
II. METHODOLOGY
- The three tiers pair 1–4B models with resource-constrained edge devices, 7–12B models with edge infrastructure, and >100B models with datacenters.The tiers offer different accuracy and carbon-efficiency trade-offs, with Tier 3 delivering state-of-the-art accuracy at the highest environmental cost.Source: §II · p. 2
- Offline calibration records each query’s semantic-lexical embedding, binary success against ground truth, inference delay, and power on target hardware.Function calling uses binary success because invoking the wrong tool or missing a required one makes the response fail.Source: §II · p. 2
- At runtime, cosine-distance k-NN uses similar calibration queries to predict success probability, delay, and power for each edge tier.The same neighborhood is used for all three predictions; the non-parametric model requires no training and adds negligible runtime overhead.Source: §II · p. 2
- The predicted carbon footprint combines each tier’s real-time grid carbon intensity with predicted inference delay and power.Grid intensity is measured in gCO2/kWh; delay and power come from the performance predictor.Source: §II · p. 2
- The router selects the minimum-carbon edge tier whose predicted success probability meets the threshold, and uses Tier 3 if none qualifies.This policy reserves the highest-emission tier for queries that no edge tier can handle at the required accuracy.Source: §II · pp. 2–3
III. EVALUATION
The framework is evaluated on BFCL V2 and GeoEngine against routing baselines, with query allocations and success, carbon, and delay compared across tiers. It approaches cloud-only success while reducing carbon, with the size of the savings varying by benchmark.
Experimental Setup
- The evaluation uses BFCL V2 and GeoEngine, an 80–20 calibration/evaluation split, and a 0.9 accuracy threshold across three hardware tiers.Tier 1 uses Jetson Orin Nano, Tier 2 Jetson AGX Orin, and Tier 3 GPT-OSS-120B on H100 GPUs; four model families are tested on Tiers 1 and 2 against five routing baselines.Source: §III · p. 3
Results Discussion
- On simpler BFCL V2, Qwen3 and Gemma3 send almost 70% of queries to Tier 1 and only 4.3% to Tier 3, unlike TensorOpera’s 67.4–73.9% cloud share.On GeoEngine, Qwen3 routes 42.4% to Tier 3, 47.8% to Tier 2, and 9.8% to Tier 1; TensorOpera sends 67.4–91.3% to Tier 3.Source: §III · p. 3, Figure 2 · p. 3
- On GeoEngine, our framework achieves 55.3–59.6% success across families versus cloud-only’s 60.9%, while reducing carbon footprint by approximately 1.39× on average.Edge-only has the lowest emissions but unacceptably low success on this complex sequential-function benchmark.Source: §III · p. 3, Figure 3 · p. 4
- On BFCL V2, our framework reaches 89.1–95.7% success versus cloud-only’s 96.7%, with 6× average carbon reduction and up to 8× for Falcon3.Figure 3 compares success, normalized carbon footprint, carbon-per-success, and normalized inference delay; normalized metrics use Cloud Only as the reference.Source: §III · p. 3, Figure 3 · p. 4
- Inference delay stays close to cloud-only, while Gemma3 is consistently slowest on edge hardware across both benchmarks.At comparable accuracy, the framework uses less carbon per correct answer than any baseline.Source: §III · p. 4
IV. CONCLUSION
The paper presents carbon-aware routing for function-calling LLMs across heterogeneous edge-cloud tiers. Its predictor guides queries to tiers meeting accuracy requirements, and experiments report competitive accuracy and near-cloud delay with lower emissions.
Conclusion
- A lightweight k-NN predictor estimates per-query success, delay, and power to guide routing to the lowest-emission tier meeting accuracy requirements.The framework operates across heterogeneous edge-cloud tiers.Source: §IV · p. 4
- Across BFCL V2, GeoEngine, and four LLM families, the framework maintains competitive accuracy and near-cloud inference delay while reducing emissions by 4× on average and up to 8× for simpler queries.Source: §IV · p. 4
- The conclusion presents query-level heterogeneity, when exploited, as a practical path to sustainable agentic AI.Source: §IV · p. 4