PaperlayerView exact revision on arXiv

Source-linked AI summary

Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

Delong Li, Xu Wang, Haochen Gong, Rui Lang, Guangsheng Yu

arXiv:2609.22753v2cs.DCcs.NI

TL;DR

Natural-language interpretation consumes part of an edge request’s response budget, creating a challenge for timely service completion. The paper integrates decision models into a shared edge-admission pipeline and compares them with hosted LLMs. Jev reduces median decision latency relative to the fastest LLM, with contract width limiting the substitution.

  • Problem

    Natural-language edge requests need interpretation and placement decisions, but interpretation consumes part of the response budget available for service completion.

    Source: §1 · p. 1

  • Method

    The paper integrates Jev and other decision models into a shared four-to-eight-field admission pipeline and compares them with hosted LLMs on verified requests and a live service path.

    Source: §1 · p. 2, §7 · p. 19

  • Results

    Jev-1.13.0’s median decision latency is 22.7–64.5% below that of DeepSeek-V4.1-Flash, while the accuracy gap exceeds 30 points at eight fields.

    Source: §7 · p. 19

  • Takeaways & Limitations

    The conclusion supports Jev-1.13.0 for latency-bound admission on short contracts, with caching for recurring requests and a generative interpreter for wide contracts.

    Source: §7 · p. 19

    The service-layer evaluation uses one arrival trace per cell and a real service with three workers and one service family.

    Source: §6.3 · p. 19

Abstract

from arXiv · show

Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion. The integration extracts four to eight bounded intent fields and applies a shared validator, admission policy, and scheduler, accounting for decision waiting throughout the request timeline. We compare Jev, two self-hosted decision models, and three hosted large language models (LLMs) on 8,280 verified requests and on a live admission path with modeled execution and a real optical character recognition service. Across 33 test conditions, Jev reduces median decision latency by 22.7-64.5% relative to the fastest LLM. This latency barely moves with input size, contract width, or catalog size. On four-field contracts, Jev's API fees per correct decision are 59.7-80.9% lower at a cost of a few exact-match points, while wide contracts mark the limit of the substitution. Receiving the service catalog with each request, Jev names unseen services as accurately as known ones. On the live admission path, Jev keeps 0.91-0.95 of requests exact and on time at loads where the LLMs fall below 0.1. Since caching repeated descriptions gives the interpreters nearly the same latency, Jev's gain lies in fresh decisions. These results support decision-model substitution for latency-bound admission on bounded contracts.

1. Introduction

Natural-language edge requests require semantic interpretation and placement decisions, but interpretation itself consumes time from the request’s response budget. This paper tests whether bounded decision models can reduce that overhead while preserving correct, on-time service completion.

1. Introduction

  • The study asks how input scale, wording, contract width, catalog changes, workload, and caching affect interpretation latency, correctness, cost, and end-to-end service completion.The fifth question examines when faster interpretation helps under workload conditions and how caching repeated requests changes its value.Source: §1 · p. 2
  • The evaluation compares three decision models with three hosted LLMs on 8,280 verified requests and a live admission path ending in a real three-node OCR service.Additional baselines include a rule parser, a self-hosted generative model, a sentence-embedding reranker, and a trained service classifier.Source: §1 · p. 2
  • The admission system uses a shared typed intent contract of four to eight fields, validator, and scheduler, and evaluates interpreters against deadlines and service completion.The system accounts for interpretation, queueing, communication, and execution on one timeline.Source: §1 · p. 2
  • The paper tests Jev and two self-hosted decision models against three hosted LLMs across 33 conditions, measuring latency, API fees per correct decision, and accuracy.Further tests examine unseen catalog services and separate fresh interpretation latency from repeated-request reuse under a shared cache policy.Source: §1 · p. 2

2. Related Work

Prior work connects natural-language interpretation to networking and service orchestration, while related work also explores bounded models, serving, structured output, cost, and caching. This paper compares decision and generative interpreters under a shared contract and downstream policy, evaluating service completion as well as interpretation.

2. Related Work

  • Seven of fourteen network-intent and orchestration studies report interpretation latency, while none varies offered load; across 22 studies, none reports more than six of 11 measured dimensions.GLiNER2 alone evaluates unseen options and InferLine alone sweeps request rate; this study measures all 11 dimensions for the same interpreters.Source: §2 · p. 2

2.1. Intent Interpretation and Service Orchestration

  • Prior systems connect interpretation to configuration and orchestration, and this study compares decision and generative interpreters while holding the intent contract and downstream policy fixed.The comparison accounts for decision waiting, case-sensitive OCR correctness, and caching under the same policy for each backend.Source: §2.1 · p. 2

2.2. Bounded Decisions and Small Models

  • Bounded-task research includes lightweight semantic representations, discriminative extraction, and specialized networking components; catalog-fed decision models avoid retraining when services are added.The catalog comparison uses a DistilBERT classifier and a MiniLM reranker as references.Source: §2.2 · pp. 2–3

2.3. Serving, Structured Output, and Cost

  • Serving configuration affects measured self-hosted model latency, while structured-output validity differs from task correctness and caching offers a way to reuse interpretation results.The paper compares fixed interpretation backends under a shared admission policy and reuses valid parsing results for identical normalized intent text.Source: §2.3 · p. 3

3. Service Admission Architecture

The admission architecture maps each request to a bounded intent contract, validates interpretations, and applies shared caching and admission rules. A common scheduler then selects a feasible worker using predicted finish time, with actual completion assessed against the request’s requirements.

3.1. Request and Interpretation Contract

  • Each request supplies a description, arrival time, deadline, origin, and payload metadata; only the description reaches the interpreter, while four core contract fields can expand to six or eight.With K catalog services, the four-field contract admits 27(K + 1) tuples; added fields map to scheduler actions, and missing requirements remain unspecified.Source: §3.1 · p. 3, Table 2 · p. 4
  • Decision models answer one choice question per contract field, while generative backends return the same fields in structured form; the catalog is passed as service options with each request.A new service can therefore reach the interpreter without retraining, subject to each model’s option limit.Source: §3.1 · p. 4

3.2. Validation, Caching, and Admission

  • A shared validator rejects missing or out-of-set values before scheduling, and every backend may reuse its own validated interpretation for identical normalized request text.Cached results may be semantically incorrect, and repeated descriptions can still require separate service execution; interpretation waiting and remote response time consume the arrival-to-deadline budget.Source: §3.2 · p. 4

3.3. Shared Scheduling and Execution

  • In the real service, locality defaults to local execution unless remote placement is explicitly permitted, and urgent requests receive priority without interrupting active work.The scheduler observes local, edge, and cloud workers with their own link delay and bandwidth.Source: §3.3 · pp. 4–5
  • The scheduler chooses the eligible node with the smallest predicted finish time before the deadline, breaking ties locally and rejecting requests with no feasible node.The estimate can be invalidated by execution-time variation or later higher-priority arrivals; the evaluator checks actual service, node, tier, priority, output, and deadline.Source: §3.3 · p. 5

4. Evaluation Methodology

The evaluation compares six primary interpreters under shared contracts and downstream policies, using a verified benchmark and a service path that combines live decisions with modeled or real execution. It defines separate measures for interpretation correctness, service completion, latency, cost, and statistical uncertainty.

4.1. Comparison and Evidence Design

  • Three decision models and three hosted LLMs use the same intent contract, validator, admission limits, execution policy, and cache; references and diagnostic systems are evaluated separately.The comparison also includes a fixed parser for RQ1–RQ3 and added classifiers and a reranker for RQ4.Source: §4.1 · p. 5

4.2. The EdgeIntent v1 Benchmark

  • EdgeIntent v1 has 8,280 cases across 23 generated conditions (300 test and 60 development each); generated text is retained only when a blind verifier matches its label tuple on every field.Source: §4.2 · p. 5

4.3. Service Workloads and Matched Comparisons

  • Part A combines live interpretation timing with modeled execution across load, deadline, topology, and cache/reuse conditions; Part B runs a real three-node OCR service under matched workloads.Part B varies steady or bursty arrivals, repeated or changing descriptions, and caching; its OCR uses Tesseract on IIIT5K images, while unsupported requests are rejected.Source: §4.3 · pp. 5–6

4.4. Correctness and Completion

  • Exact semantic correctness requires every interpreted field to match; operational completion requires a supported request to meet its deadline and service, locality, tier, and priority requirements.Strict completion additionally requires exact field equality; unsupported requests and correct rejections are reported separately so they do not inflate service completion.Source: §4.4 · pp. 6–7

4.5. Latency and API Cost

  • Decision latency measures client initiation to the full interpretation response, while full request latency includes admission waiting, interpretation, queueing, transfer, and execution.Paired latency savings use requests completed correctly by both backends; API cost per completion divides interpretation fees by the applicable completion count.Source: §4.5 · p. 7
  • Reported API fees exclude computation, communication infrastructure, energy, and maintenance.Self-hosted models have no API fee, so their energy per decision is reported instead.Source: §4.5 · p. 7

4.6. Statistical Analysis

  • Hypotheses and decision rules were fixed before runs, with bootstrap confidence intervals, paired tests, Holm correction, and deployment-class-specific confirmatory latency contrasts.RQ5 uses a block bootstrap over contiguous arrivals; contrasts count as resolved only when the 95% interval excludes the null and adjusted p is below 0.05.Source: §4.6 · p. 7

4.7. Implementation and Settings

  • Hosted calls use strict structured outputs and pinned providers; self-hosted models run on an H100, while both service-path parts use shared admission controls and a normalized-text cache.The real OCR setup uses identical worker containers and calibrated service-duration estimates; cached decisions do not replace service execution.Source: §4.7 · pp. 7–8

5. Results

Across the interpreter tests, Jev-1.13.0 maintained low decision latency as inputs and catalogs grew, while wider contracts reduced its exact match. On the modeled admission path, Jev retained high exact, on-time completion as offered load increased; caching repeated descriptions narrowed latency differences.

5.1. RQ1: Input Scale

  • Jev-1.13.0’s median decision latency rose from 0.274 s to 0.418 s across input lengths up to 16,384 tokens and remained faster than every LLM at each length.With eight requests bundled, Jev’s message latency rose from 0.272 to 0.290 s, while the LLMs rose from 0.379–1.302 s to 0.819–3.785 s.Source: §5.1 · p. 8, Figure 3 · p. 9, Table 4 · p. 9

5.2. RQ2: Input Quality

  • Jev-1.13.0’s exact match stayed at 0.920–0.950 across seven wording conditions, but default-baiting reduced it to 0.610.Jev made no unsafe decisions in seven conditions and 4.3% under default-baiting; hosted LLMs stayed at or below 5.0%.Source: §5.2 · pp. 8–9, Figure 4 · p. 10, Table 6 · p. 11

5.3. RQ3: Task Difficulty

  • Jev-1.13.0’s exact match fell from 0.923–0.967 with four fields to 0.527–0.577 with eight, while DeepSeek-V4.1-Flash kept 0.900–0.930.Jev’s latency stayed at 0.264–0.280 s across the task-difficulty cells; at eight fields DeepSeek was 32.3–40.3 points more accurate for about 0.2 s of additional latency.Source: §5.3 · pp. 9, 11, Figure 5 · p. 10, Table 8 · p. 13

5.4. RQ4: Dynamic Service Catalog

  • With request-time catalogs, Jev-1.13.0 and the LLMs named unseen services with 0.995–1.000 accuracy at 128 and 254 services and under both churn levels.Jev’s latency stayed at 0.266–0.335 s across catalogs of 4–254 services, but its fee exceeded DeepSeek-V4.1-Flash from 64 services; self-hosted interfaces rejected some larger catalogs.Source: §5.4 · p. 12, Figure 6 · p. 13, Table 9 · p. 14

5.5. Tail Latency, Jitter, and Energy

  • On clean requests, 0.7% of Jev-1.13.0 decisions exceeded 0.5 s, compared with 19.0% for DeepSeek-V4.1-Flash, 98.0% for GLM-5.3-Flash, and 100% for Qwen3.8-Flash.SemIf-Qwen3.5-4B and Laya used 22.5 J and 6.1 J per decision, respectively, versus 123.3 J for Qwen3.5-4B-JSON on the same accelerator.Source: §5.5 · p. 13, Figure 7 · p. 15, Figure 8 · p. 15

5.6. RQ5: End-to-End Service Outcome

  • In the real OCR service, Jev-1.13.0’s correct completion was 0.511 on changing text and 0.500 on repeated text, matching Tesseract’s recognition accuracy.It dispatched all 180 OCR requests in each condition to a permitted node and tier, and every request finished before its deadline.Source: §5.6 · pp. 14–15
  • Caching repeated descriptions narrowed median-latency gaps and closed completion gaps where they existed; Jev-1.13.0’s advantage therefore lay in fresh decisions.At 16 requests/s, Jev’s fees were 0.038 USD per 1,000 correct completions, while the LLMs reached 0.58–2.13 USD; with repeated descriptions cached, all four hosted interpreters fell to 0.001–0.009 USD.Source: §5.6 · pp. 15–16, Figure 9 · p. 16
  • Jev-1.13.0 completed 0.907–0.953 of requests exactly and on time across 1–16 requests/s, while GLM-5.3-Flash fell below 0.1 from 8 requests/s and Qwen3.8-Flash from 4.DeepSeek-V4.1-Flash fell to 0.603 at 8 requests/s and 0.093 at 16.Source: §5.6 · pp. 13, 16, Figure 9 · p. 16

6. Discussion

Jev’s latency advantage remains visible on the real OCR path, but its accuracy trade-off bounds substitution to short contracts. The discussion connects these results to service-path evaluation and identifies scope limits and next steps.

6.1. Implications for Service Orchestration

  • Jev-1.13.0’s decision time barely moves with input length, bundling, contract size, or catalog size, but its accuracy gap exceeds 30 points at eight fields.The discussion recommends Jev for short, latency-bound contracts and a generative interpreter for wide contracts.Source: §6.1 · p. 17
  • 0.63–1.42 s: Jev-1.13.0’s 95th-percentile request latency stays this far below the LLMs under steady arrivals with changing text.The real OCR evaluation includes image transfer, worker queueing, and recognition.Source: §6.1 · pp. 16–17, Figure 10 · p. 18
  • With eight recurring descriptions, caching answers nearly every request, and every interpreter except Laya completes 0.489–0.500, the recognizer’s accuracy.Backend choice depends on how many requests need a new decision and how much time remains for service execution.Source: §6.1 · p. 18

6.2. Evaluating the Complete Service Path

  • The shared scheduler checks placement, tier, priority, and predicted completion time; separate measures of intent, execution compliance, and output trace effects through the service path.The evaluation supports choosing an interpreter using completion, response time, and fees together.Source: §6.2 · p. 19

6.3. Scope and Next Steps

  • The benchmark wording may favor language-model parsing; each case runs once per interpreter, the service layer uses one arrival trace per cell, and the real service has three workers and one family.Hosted timings include provider and network paths, while API fees exclude local compute and communication costs.Source: §6.3 · p. 19
  • Next steps include independently authored requests across days, services, and network conditions, with deadlines drawn from application requirements.Optimized local serving, a bounded extractor, and decision models with larger option limits would extend deployment choices.Source: §6.3 · p. 19

7. Conclusion

The conclusion reports Jev-1.13.0’s latency, cost, accuracy, and live-path results against decision models and hosted LLMs. It recommends Jev for short latency-bound contracts, caching recurring requests, and routing wide contracts to a generative interpreter.

7. Conclusion

  • 22.7–64.5%: Jev-1.13.0’s median decision latency is this much below DeepSeek-V4.1-Flash and barely moves with input length, bundling, contract size, or catalog size.On four-field contracts, API fees per correct decision are 59.7–80.9% lower, with a few exact-match points lost; at eight fields, the accuracy gap exceeds 30 points.Source: §7 · p. 19
  • On the live admission path, Jev-1.13.0 keeps 0.91–0.95 of requests exact and on time at up to 16 requests/s, while the LLMs fall below 0.1.Caching repeated descriptions gives interpreters nearly the same latency, confining Jev’s advantage to fresh decisions.Source: §7 · p. 19
  • The conclusion recommends Jev-1.13.0 for latency-bound admission on short contracts, caching for recurring requests, and a generative interpreter for wide contracts.The comparison covers EdgeIntent v1 and a live admission path, alongside two self-hosted decision models and three hosted LLMs.Source: §7 · p. 19

Related papers

Loading 2609.22753v2…