Source-linked AI summary
Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agents
Vasileios Rizeakos, Georgios Paisios, Alexandros Machairas, Michael Birbas, Athanasios Bachoumis
TL;DR
Factory workers need on-premise access to machine documentation, but capable models must run on constrained industrial hardware. The paper uses weight-shared structural compression, hardware-measured sub-network selection, and retrieval-grounded adaptation, finding that selected models retain about 95% of unpruned judged RAG quality across heterogeneous edge devices.
Problem
Factory documentation assistants must operate on shop-floor controllers or embedded boxes, where capable models may not fit and documents or camera feeds may need to remain on premises.
Method
The paper combines a weight-shared supernetwork with sandwich-style in-place distillation, hardware-aware quality-throughput selection under capability and memory constraints, and retrieval-grounded distillation.
Results
Application adaptation breaks size-based quality ordering, while selected sub-networks retain ∼95% of the unpruned model’s judged RAG quality across three heterogeneous edge tiers.
Takeaways & Limitations
Deployment should select one adapted sub-network per device using measured judged quality and throughput rather than optimizing size, speed, or quality alone.
Takeaways & Limitations
Judged quality depends on a fixed, unmeasured retriever and a single LLM judge; the 80% ARC-Easy floor is defensible but arbitrary and can change selections.
Abstract
from arXiv · showhide
On-premise assistants can give factory workers conversational access to machine documentation, but models capable of the task rarely fit shop-floor hardware. We show that after structural compression and retrieval-grounded adaptation, model size is no longer a reliable predictor of adapted answer quality: general capability falls almost linearly with parameter count, while judged retrieval-augmented answer quality does not. We therefore treat deployment as a post-adaptation selection problem, committing one sub-network per device on judged answer quality and measured on-device throughput under a configurable general-capability floor and memory budget; rules that optimize size, speed, or quality alone each give up capability or throughput. A weight-shared supernetwork trained with sandwich-style in-place distillation keeps this selection inexpensive. In a manufacturing-manual case study, extraction costs 13.7 percent of the unpruned model's judged quality and retrieval-grounded distillation returns it to within 4.6 percent, recovering two thirds of the loss, and the same assistant runs across three heterogeneous edge tiers at 1.3 to 5 watts standby.
I. INTRODUCTION
Factory assistants must bring reliable document-grounded answering onto constrained, privacy-sensitive shop-floor hardware. This paper addresses the problem with a multi-stage compression, adaptation, and hardware-aware selection pipeline.
- Factory workers consult machine documentation under time pressure, motivating conversational access at the point of use.
- 2–8 GB edge hardware conflicts with the billions of parameters carried by models that answer technical questions reliably.
- Structural pruning alone deteriorates the general capability required for reliable answering.
- The pipeline combines a weight-shared supernetwork, hardware-aware per-device selection, retrieval-grounded distillation, and tool-augmented deployment.
- The paper evaluates whether compressed candidates retain application-specific RAG quality, whether measured constrained selection beats simpler rules, and whether deployment spans heterogeneous edge hardware.
- Selection commits one sub-network using judged RAG quality and measured throughput under a configurable general-capability floor.
II. RELATED WORK
Prior work reduces LLM inference cost through compression, distillation, adaptation, quantization, and weight-shared supernetworks. The paper identifies a remaining selection gap: deployment choices have commonly relied on pre-adaptation accuracy predictors.
- Structural pruning, knowledge distillation, low-rank adaptation, and quantization are established approaches for reducing LLM inference cost.
- Weight-shared supernetworks jointly train many sub-networks and support train-once, specialize-per-device deployment.
- Train-once pipelines have selected using pre-adaptation accuracy predictors, whereas this paper argues selection must follow application adaptation.
- Retrieval-augmented generation grounds answers in retrieved document passages rather than parametric memory alone.
- Retrieval-aware finetuning mixes distractor passages into context, and this paper adopts it inside a distillation objective.
C. LLM agents in industrial scenarios
The paper adapts compact language models into on-premise industrial agents that coordinate retrieval and vision tools. Its deployment pipeline compresses, specializes, quantizes, and serves candidates across heterogeneous hardware.
- Existing manufacturing agent frameworks generally assume large, typically cloudhosted models, leaving compact on-premise industrial deployment under-addressed.
- The proposed pipeline trains a weight-shared supernetwork, extracts candidates under a deployment budget, and selects one per target using hardware-grounded evaluation.
- Retrieval-grounded distillation specializes the extracted model to factory documents, after which it is exported to quantized GPU or Arm CPU formats.
- Elastic depth and per-layer MLP size create an 11-point candidate grid spanning 2.20–3.21 billion parameters on the 3B base and 0.90–1.24 billion on the 1B base.
B. Sandwich-style in-place distillation
Sandwich-style in-place distillation trains the full supernetwork and sampled candidates together, transferring full-model behavior through shared weights. Importance-calibrated sampling selects candidates from a calibrated grid rather than the raw search space.
- Each optimization step trains the full supernetwork with M=3 sampled sub-networks using the full network’s detached output distribution as teacher.
- Importance-calibrated sampling prepermutes channels and draws sub-networks uniformly from the 11-point calibrated grid.
- The objective combines ground-truth token loss with forward KL divergence from the full network’s detached distribution, balancing them with α=0.8 and β=0.2.
- Gradients from all forwards accumulate in shared weights, transferring large-capacity behavior to smaller candidates without a separate teacher pass.
- Candidates are materialized from an 11-point elastic-dimension grid that retains the best WikiText-perplexity configuration per bin plus the smallest and full models.
D. Hardware-grounded sub-network selection
Deployment selection follows application adaptation: each candidate is evaluated on judged retrieval quality and measured device performance, then filtered by capability and memory constraints. A three-anchor benchmark makes this measurement-driven process practical for new devices.
- Hardware-grounded sub-network selection: Selection uses proxy metrics for orientation, real evaluation of every adapted rank, and final deployment choice based on measured performance.Evaluation covers routing accuracy, judged RAG quality, throughput, latency, memory, and energy.
- Hardware-grounded sub-network selection: The objective blends normalized judged RAG quality and measured throughput while enforcing an ARC-Easy capability floor and memory ceiling.The reported configuration uses w=0.5 and ϕ=0.8; ties favor higher throughput.
- Hardware-grounded sub-network selection: 2.5-3.1% mean absolute error: a three-anchor throughput predictor matches fully measured Pareto fronts on both candidate grids.The predictor is fitted per device on three ranks spanning the parameter range.
- Hardware-grounded sub-network selection: 6.0 hours instead of 28.8: commissioning a new device requires benchmarking only the three throughput anchors.The reduced benchmark covers the selected device’s throughput prediction rather than the full candidate grid.
- Hardware-grounded sub-network selection: The application corpus is a 187-page mill operator manual divided into 1,460 overlapping 100-token chunks for grounded adaptation.Training uses 686 generated question-answer pairs and a 633-question out-of-sample set.
Objective:
The selected compact model is adapted with low-rank distillation and deployed as the reasoning core of an on-premise tool-augmented assistant. It routes queries to retrieval or vision tools, synthesizes grounded answers, and keeps documents and images on site.
- Objective:: The extracted sub-network is adapted with rank-8 LoRA on attention and MLP projections while base weights remain frozen.The student minimizes the distillation objective on answer tokens only.
- Objective:: The deployed distillation runs use equal cross-entropy and KL weights, with α′=β′=0.5 and temperature τ′=2.0.Tool-routing demonstrations are mixed into the same run with the task loss only.
- Objective:: The adapted model is exported to W8A16 ONNX for GPU targets or Q4 K M GGUF for Arm CPU targets.The 3B rank-6 occupies 1.85 GB at these formats.
- Objective:: The compact model serves as the reasoning core in a task-orchestrator assistant that delegates work to external agents.The assistant follows the task-orchestrator design described in the paper.
- Objective:: Queries are routed through grammar-constrained JSON calls to machine-manual retrieval or a vision tool, after which the model synthesizes the grounded answer.Retrieval uses dense embeddings with top-4 contexts; deployment uses ONNX Runtime GenAI or llama.cpp depending on hardware.
- Objective:: The full stack runs on premises, so documents and camera images never leave the site.Development uses a single 24-GB GPU partition.
- Objective:: The selected sub-network routes each query through a grammar-constrained JSON tool call and synthesizes the grounded answer from the tool output.This is the operational flow illustrated in Fig. 2.
VI. EXPERIMENTAL SETUP
The evaluation compares compressed and adapted models using held-out manufacturing questions, multiple RAG and efficiency metrics, and deployment-relevant baselines. Judged quality is reported relative to the unpruned base under identical retrieval conditions, while separate campaigns limit absolute cross-table comparisons.
- Models and deployment: The study evaluates Llama-3.2-3B and 1B supernetworks, with platform-specific sub-networks selected for Jetson, RevPi, and UNO Q deployment.The deployed 3B systems use rank 6, while the UNO Q uses 1B rank 5 because its memory ceiling excludes 3B candidates.
- Metrics: RAG quality is the mean of faithfulness, answer relevancy, and context utilization, judged under an identical retrieval stack.Table I reports changes relative to the unpruned base model.
- Evaluation data: 633 held-out manual questions support the main quality comparison, while Table II uses a separate 291-question campaign.The two campaigns use one judge pass each, so their absolute quality values are not directly comparable.
- Main quality result: Extraction at deployed rank 6 reduces judged answer quality by 13.7% relative to the base, while retrieval-grounded distillation returns it to within 4.6%.The recovery estimate uses 633 paired questions and has a bootstrap 95% confidence interval of [−5.9%, −3.3%] of the base score.
B. Stage-2 decomposition and baselines
Stage-2 ablations show that grounded distillation and RAFT distractors improve the extracted model, while selection comparisons show that a capability-constrained blend preserves quality, capability, and throughput better than simpler rules.
- Stage-2 decomposition: 0.765 is the deployed Stage-2 recipe quality, after grounded loss, softened distillation, and RAFT distractors build cumulatively from extraction.Context-free supervised finetuning instead degrades the extracted model from 0.701 to 0.654, while grounded task loss reaches 0.727.
- Selection baselines: Every simpler selection rule forfeits at least one objective: size and speed retain only 69-74% of base capability, while quality-only ignores throughput.The constrained blend keeps capability, judged quality, and throughput together.
- Deployed quality: Rank 6 cores on Jetson and RevPi score 0.773 faithfulness, 0.845 answer relevancy, and 0.811 context utilization.The native 1B core scores 0.591/0.749/0.682 in its own judging session, so the 3B grid is used where memory permits.
- Selection evaluation: Table III compares selection strategies on the RevPi 3B grid using quality and throughput from a 256-token selection cell, with selected rank versus capability floor below.The table’s throughput values therefore differ from Table IV’s 128-token cell.
- Tool routing: Tool routing reaches 40/40 for every fine-tuned core after 37 routing demonstrations join training.This costs 1.3-2.5 points across the RAG metrics.
E. Ablations
The ablations test supernetwork training and calibrated sampling, while the deployment demo illustrates retrieval and vision tool use on the Jetson tier.
- Supernetwork training: At mid-grid deployment ranks, trained supernetworks hold approximately 0.72 quality, whereas calibrated sampling without supernetwork training collapses to 0.44-0.50.At full width, the ablation arms are indistinguishable at 0.77-0.80.
- Deployment demonstration: The Jetson demo shows the assistant answering a manual question through retrieval and a person-count query through the vision tool.Both interactions are presented through the on-device web interface.
- Sampling calibration: At deployed rank 6, calibrated sampling reaches 0.783 versus 0.738, while random sampling is better by 0.13-0.15 below rank 8.The result indicates calibrated sampling benefits larger ranks used for deployment but not the smallest tail.
F. Discussion
The discussion evaluates deployment across heterogeneous edge hardware and emphasizes that fixed retrieval, single-judge scoring, and an arbitrary capability floor constrain interpretation. Weight sharing makes per-device reselection inexpensive without retraining shared weights.
- Limitations: Retrieval recall is unmeasured, so every RAG score is conditioned on the fixed retriever’s unknown recall.A single LLM judge remains another limitation, and separately judged 3B and 1B grids make cross-grid comparisons indicative only.
- Selection: Weight sharing enables candidates to be rescored or the selection recipe to be re-instantiated at smaller scale without retraining shared weights.Each device’s core is selected by rerunning the procedure using anchor measurements from that device.
- Platforms: Three edge platforms use identical documents, prompts, routing grammar, and queries, while model scale, quantization, and runtime vary.The RevPi and Jetson use 3B rank 6; the UNO Q uses 1B rank 5 because its memory ceiling excludes 3B candidates.
- Evaluation: RAG quality is judged with interleaved, shuffled answers, while latency and throughput are measured alongside memory and energy.The evaluation attributes cross-platform RAG differences to the reasoning core because retrieval is identical everywhere.
C. Results and discussion
The deployed pipeline improves practical efficiency across three edge tiers while preserving judged retrieval-augmented quality and tool routing. Standby power varies substantially, and the selected models improve most measured deployment costs relative to unpruned references.
- Latency and energy: About 20 s on the Jetson versus one to over two minutes on CPU tiers characterizes full-query latency, with prefill dominating CPU execution.The full query includes routing, retrieval, and synthesis at production lengths.
- Latency and energy: 1.3-5.0 W standby varies by 4× across tiers, while per-inference energy remains similar at 117-170 J.At sparse duty cycles, standby therefore dominates total energy.
- Quality and deployment: All three tiers improve latency, energy, and memory relative to same-device unpruned ranks, except for an insignificant 2% memory increase on the UNO Q.Table IV reports same-session deltas against the unpruned rank-1 model on each device.
- Quality and deployment: The selected sub-networks retain ∼95% of the unpruned model’s judged RAG quality and route tools without error across the three edge tiers.The same assistant spans a 2-GB Arduino UNO Q and a Jetson Orin Nano.