Source-linked AI summary
Can You Check That? The Checkability Boundary for Local LLM Network Automation
Maleeha Masood, Momina Nofal
TL;DR
Network automation can expose sensitive operational artifacts to third-party frontier LLMs, while local SLM outputs can be error-prone. Touchstone uses task-specific intrinsic checks to screen local candidates and escalate unresolved inputs; it performs strongly on conflict detection and intent translation but falls below the frontier baseline on TeleQnA.
Problem
Frontier LLM inference can require exporting sensitive operational artifacts, while local SLMs are more prone to hallucinations and have weaker reasoning than LLMs.
Source: §1 · p. 1Method
Touchstone uses seven off-the-shelf SLMs to generate candidates, applies task-specific deterministic checks for necessary correctness conditions, and escalates unresolved inputs to a frontier LLM.
Source: §1 · p. 1Results
Touchstone reaches 98.6% end-to-end accuracy on conflict detection with 16% escalation and 93.8% accuracy on intent translation with 17%, but remains below the frontier baseline on TeleQnA.
Source: §1 · p. 2Takeaways & Limitations
Keep inference local when task semantics support precise, low-cost checks; escalate tasks without such checks.
Source: §7 · p. 6Knowledge-based tasks such as TeleQnA expose no intrinsic-check property that distinguishes correct from incorrect answers, limiting this approach’s check-based screening there.
Source: §2 · p. 3
Abstract
from arXiv · showhide
Sending every network-automation input to a third-party frontier LLM exports sensitive artifacts such as production configurations, topologies, and logs. Querying small language models (SLMs) locally avoids this egress, but SLM outputs can be error-prone for direct use. This work introduces checkability as a criterion for determining which tasks are suitable for local inference. A task is checkable when it exposes a cheap, deterministic test - an intrinsic check - that rejects outputs violating a necessary correctness condition. We instantiate this idea in Touchstone, a local-first pipeline that uses seven off-the-shelf SLMs (1-8B parameters) to generate candidates, uses task-specific intrinsic checks to reject responses, and escalates unresolved inputs to a frontier LLM. On conflict detection and intent translation tasks, Touchstone reaches 98.6% and 93.8% end-to-end accuracy while escalating only 16% and 17% of inputs, respectively. On TeleQnA, a knowledge-only control that has no task-specific intrinsic checks, Touchstone is unable to match the accuracy of the frontier baseline. Our results support a simple deployment rule: keep inference local when task semantics support precise, low-cost checks; escalate the rest.
1 INTRODUCTION
The introduction proposes checkability as a boundary for deciding when network-automation inference can stay local, and presents Touchstone as a local-first pipeline that checks SLM candidates before selective frontier escalation. Its reported results vary by task: Touchstone performs strongly on conflict detection and intent translation, but not on the knowledge-only control.
1 INTRODUCTION
- Frontier LLM inference can require exporting sensitive operational artifacts and incur substantial costs, while on-premise inference requires significant GPU infrastructure and operational expertise.SLMs can run inside an operator’s environment, but are more prone to hallucinations and have weaker reasoning and smaller context windows than LLMs.Source: §1 · p. 1
- Examples of intrinsic checks include rejecting contradictory reachability rules, detecting entities absent from an intent request, testing whether a log template regenerates its line, and executing routing code on topologies.These checks are inexpensive and do not compare against a reference answer or invoke a verifier or LLM.Source: §1 · p. 1
- Touchstone uses seven heterogeneous off-the-shelf SLMs with 1–8B parameters to generate candidates, applies task-specific checks, then selects passing candidates or escalates unresolved inputs.The SLMs provide diversity, checks determine eligibility, and frontier escalation handles the unresolved residue.Source: §1 · p. 1
- The evaluation covers four structured networking tasks—conflict detection, intent translation, log parsing, and routing-code generation—and the telecommunications knowledge dataset TeleQnA.The paper quantifies the end-to-end accuracy–escalation tradeoff.Source: §1 · pp. 1–2
- 98.6% end-to-end accuracy on conflict detection with 16% escalation, and 93.8% accuracy on intent translation with 17% escalation; Touchstone remains below the frontier baseline on TeleQnA.Conflict detection had zero false accepts.Source: §1 · p. 2
- Checkability sets the local-deployment boundary: tasks with cheap, deterministic checks should remain local, while tasks without them should be escalated to a frontier LLM.The paper defines intrinsic checks as deterministic tests that reject candidates when the input or network state refutes a claim.Source: §1 · pp. 1–2
2 INTRINSIC CHECKS
Intrinsic checks are deterministic necessary-condition tests that reject candidates refuted by the input or network state, but passing does not establish correctness. The section illustrates checks for structured networking tasks and explains why knowledge-only tasks lack the structure needed for such checks.
Intrinsic Checks
- A check is a necessary, not sufficient, condition for correctness: failure refutes a candidate, while passing only means it survived refutation.Its false-accept rate—the fraction of accepted candidates that are wrong—must be measured.Source: §2 · pp. 2–3
Intent Translation
- Intent-translation checks test whether generated entities are grounded in the request and whether rule coverage roughly matches its requirement clauses.These checks can reject hallucinated subnets or dropped requirements, but cannot prove that a grounded, complete-looking specification captures the operator’s intent.Source: §2 · p. 2
Conflict Detection
- Conflict detection parses reachability claims into location–prefix pairs and rejects “no conflict” when a pair is asserted both reachable and unreachable.If no contradiction is extracted, the check does not prove that the request is conflict-free.Source: §2 · p. 2
Log Parsing
- Log-template invertibility checks whether replacing wildcards with concrete substrings can regenerate the original log line.Over-general templates can still pass, so this check has a larger false-accept rate.Source: §2 · pp. 2–3
Routing Code Generation
- Routing-code execution checks run generated functions against task test cases, an example of intrinsic checking already standard in code tasks.A passing candidate may still be wrong when its error leaves no detectable witness.Source: §2 · p. 3
Uncheckable Tasks
- Knowledge-based tasks such as TeleQnA expose no task-intrinsic property that distinguishes a correct answer from a wrong but well-formed option.The checks are written once per task by system designers, who identify sound necessary conditions; prototypes range from 28 to 49 lines of code.Source: §2 · p. 3
3 THE TOUCHSTONE WORKFLOW
Touchstone separates local candidate generation from admission: seven off-the-shelf SLMs propose answers, task-specific checks filter them, and confidence-gated escalation sends unresolved inputs to a frontier LLM. Its aggregation and confidence rules vary with the task’s output structure and check strength.
Workflow
- Touchstone admits a local candidate only after generation, task-specific checking, and a confidence decision; otherwise it escalates the input to a frontier LLM.Escalation is a single per-input decision made after aggregation.Source: §3 · pp. 3–4
Offline Profiling
- Seven off-the-shelf SLMs contribute candidates, with voting weights based on task-specific development accuracy above chance.Development splits contain 100–150 examples per task; subtracting chance prevents near-random models from dominating, while the exponent emphasizes models reliably above chance.Source: §3 · p. 4
Local Generation and Aggregation
- For conflict detection and log parsing, Touchstone selects the highest skill-weighted vote, subject to checks that can override conflict labels or exclude non-invertible templates.Routing code instead retains every generated function that passes the execution tests, since multiple programs may be correct.Source: §3 · p. 4
- Intent translation pools rule atoms across models and retains atoms supported by at least half of total ensemble weight.Touchstone compares the assembled rules with the individual specification scoring highest on grounding and coverage, then keeps the higher-scoring one to guard against incoherent recombination.Source: §3 · p. 4
Confidence-Gated Escalation
- Confidence combines weighted model-family support with task-check evidence, and inputs escalate when confidence falls below 0.6.Conflict detection sets confidence to 1 when a contradiction is established; routing code sets it to 1 if any function passes and 0 if none does.Source: §3 · pp. 4–5
- Escalation prompts include diagnostic evidence—such as an ungrounded entity or failed execution test—as a repair hint, and the frontier response becomes the final output.For log parsing, only templates passing invertibility are eligible; intent defects can discount agreement and trigger escalation.Source: §3 · pp. 4–5
4 EVALUATION
Touchstone’s accuracy and egress tradeoffs depend on whether local candidates can be reliably checked: strong checks approach frontier accuracy with limited escalation, while weak or absent checks leave errors or require substantial fallback. Compared with the best single SLM, the pipeline improves accuracy most on checkable tasks.
4 EVALUATION
- The evaluation measures accuracy, escalation to GPT-5.5, and false-accept rates across four structured networking tasks and a knowledge-only control.The false-accept rate measures the fraction of locally accepted items that were actually wrong.Source: §4 · p. 5
4.1 Accuracy, Egress and False-accept Rates
- 98.6% accuracy versus GPT-5.5’s 100% on conflict detection, with 16% escalation and no false accepts; intent translation reaches 93.8% versus 99.2% with 17% escalation and a 5.6% false-accept rate.Intent checks catch hallucinated entities and omitted clauses but do not prove semantic correctness.Source: §4.1 · p. 5
- Invertibility is necessary but weakly selective: false-accept rates are 27.7% on OpenSSH and 30.0% on Proxifier; Touchstone escalates 37% and 42% of inputs, respectively.Over-general templates can regenerate the original log line and pass the check, so accuracy comes largely from frontier fallback rather than the check itself.Source: §4.1 · p. 5
- 91.7% HDFS accuracy with 7% escalation trails the frontier model’s 98.7%, alongside a 14.4% false-accept rate.Low egress is not a success when incorrect outputs are admitted locally.Source: §4.1 · p. 5
- Routing-code checks reject incorrect programs, but only 1 of 8 inputs for the best single model and 2 of 8 across all models pass, yielding 75% escalation and the frontier model’s 50% accuracy; TeleQnA reaches 78.2% accuracy with 16% escalation, below the frontier baseline.TeleQnA has no intrinsic check, and even the frontier model solves only 84.2% of its questions.Source: §4.1 · p. 5
4.2 Improvement over the best single SLM
- Touchstone raises conflict-detection accuracy from 84.0% to 98.6% and intent-translation accuracy from 74.0% to 93.8% versus the best single SLM.Log-parsing accuracy rises from 74.3% to 99.7% on OpenSSH, 86.3% to 91.7% on HDFS, and 65.0% to 87.7% on Proxifier; TeleQnA changes from 78.0% to 78.2%.Source: §4.2 · pp. 5–6
5 INTRINSIC CHECK AS A BOUNDARY
The paper frames local deployment as a task property: deterministic checks make some SLM outputs admissible, while tasks lacking checkable structure fall outside that boundary. Checkability varies with the strength of the available checks.
5 INTRINSIC CHECK AS A BOUNDARY
- Local deployment depends on whether task semantics expose deterministic structure for rejecting bad outputs, not only on model strength.Tasks requiring external knowledge or judgment that cannot be checked from input and local state are unsafe for the same local model.Source: §5 · p. 6
- The checkability boundary ranges from strong contradiction checks, through partial grounding and coverage checks and permissive invertibility checks, to knowledge-only tasks with no deterministic correctness witness.Intent checks cannot prove policy equivalence, while log-template invertibility can admit over-general templates.Source: §5 · p. 6
6 RELATED WORK
Touchstone uses deterministic checks to decide whether locally generated answers are admissible, distinguishing its purpose from verification systems, synthesis methods, and model cascades. Checks gate candidates but do not establish full correctness.
6 RELATED WORK
- Prior networking work establishes that language models can produce useful artifacts, while often relying on frontier models, tool-rich workflows, or model-specific adaptations [12] [19] [21] [23] [29].Touchstone instead asks when inference can remain local without exposing sensitive operator data.Source: §6 · p. 6
- Touchstone applies cheap deterministic admission tests after local generation; passing means only that a candidate has not violated the tested necessary condition.This contrasts with approaches that use syntax or semantic verifiers, human input, ambiguity resolution, or an SMT solver during generation [21],,.Source: §6 · p. 6
- Network verification and synthesis systems check properties or compile configurations, whereas Touchstone uses per-query necessary-condition tests to gate outputs before downstream automation [5] [7].The paper describes Touchstone as weaker by design than a verification system.Source: §6 · p. 6
- Unlike model cascades that route by confidence or aggregate generations [6] [14] [22] [28], Touchstone accepts candidates based on external network constraints and escalates cases local checks cannot resolve.Model confidence or majority agreement is not the admission criterion.Source: §6 · p. 6
7 CONCLUSION
The conclusion argues that local-first network automation requires checks, not SLM accuracy or model agreement alone. Touchstone admits locally generated candidates that pass task-specific checks and escalates unresolved inputs, supporting local inference when task semantics permit precise, low-cost checks.
7 CONCLUSION
- Local-first network automation cannot rely on SLM accuracy or model agreement alone; it depends on cheap deterministic checks that can reject erroneous outputs.When checks have low false-accept rates and local models generate valid candidates, Touchstone keeps most requests on premises while approaching frontier-model accuracy.Source: §7 · p. 6
- Touchstone generates local candidates, admits only those surviving task-specific intrinsic checks, and escalates the remainder to a frontier LLM.The conclusion positions checks as the admission criterion for local inference.Source: §7 · p. 6
- The supported deployment rule is to keep inference local when task semantics support precise, low-cost checks and escalate the rest.Source: §7 · p. 6