Source-linked AI summary
What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
TL;DR
The impact of pruning on context-grounded smart-home tool calling remains poorly understood, despite pruning’s potential to reduce LLM deployment costs. This study evaluates pruning across architectures and task dimensions, finding that dense models degrade sharply after narrow safe regions while MoE models tolerate more pruning, and that specificity can degrade before schema-level intent.
Problem
How pruning sensitivity varies across architectures, action components, and smart-home task complexity remains unclear, although pruning can reduce LLM deployment costs.
Source: §1 · p. 1Method
The study compares pruning across Qwen architecture designs and measures accuracy by action component and task-complexity category against unpruned models.
Source: §3.1 · p. 2, §3.3 · p. 3, §3.5 · p. 4Results
Takeaways & Limitations
Pruning should be evaluated beyond overall accuracy, including architecture, workload, and failure mode, to identify regimes that reduce cost while preserving reliable smart-home tool execution.
Source: §6 · p. 7All evaluated models are from the Qwen family, and the evaluation is restricted to smart-home tool calling, with some complexity categories present in only a subset of datasets.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. We systematically study pruning-induced degradation in smart-home tool calling across four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert pruning methods. After post-pruning supervised fine-tuning (SFT), we evaluate more than 19,500 instances from three smart-home datasets. Beyond aggregate task accuracy, we characterize degradation along two dimensions: action components (i.e., operation, device, argument, and value) and task complexity. Our results show that dense models have narrow safe pruning regions followed by sharp degradation, while MoE models tolerate substantially more pruning. Pruning degrades grounded specificity before schema-level intent, and aggressive dense pruning can induce systematic over-refusal. These findings highlight the importance of evaluating pruning beyond aggregate accuracy when selecting pruned LLMs for reliable tool execution.
1 Introduction
Pruning offers a way to reduce LLM deployment costs, but its effects on context-grounded smart-home tool execution remain unclear. This study evaluates where pruning degrades performance and finds that the effects vary by architecture, action component, and request complexity.
1 Introduction
- Pruning removes structural components such as Transformer layers, channels, or experts to reduce deployment costs while preserving hardware-friendly model structures.The deployment question is not only how much an LLM can be pruned, but which capabilities are lost as capacity is removed.Source: §1 · p. 1
- Smart-home tool execution requires selecting an operation, device, argument, and value, while requests range from simple operations to contextual, partially executable, and infeasible tasks.Because these decisions and request types can be affected differently, aggregate accuracy can hide distinct pruning-induced failure modes.Source: §1 · p. 1
- The study evaluates four LLMs across three architecture families and four structured-pruning classes, with post-pruning SFT and more than 19,500 instances from three smart-home datasets.Degradation is analyzed across operation, device, argument, and value components, and five task-complexity categories.Source: §1 · pp. 1–2
- Dense models show narrow safe pruning regions before sharp degradation, while the MoE model tolerates greater expert removal; grounded specificity degrades before schema-level intent.Aggressive dense pruning can also shift models toward systematic over-refusal of executable requests.Source: §1 · p. 2
2 Related Work
Prior work studies structured pruning and LLMs for smart-home control, but these lines of research leave the reliability of pruned models for smart-home tool execution largely unexplored.
2 Related Work
- Most pruning studies evaluate aggregate accuracy, which can obscure capability-specific degradation.Pruning methods remove components at different structural levels, including width, depth, hybrid depth-and-width, and MoE levels.Source: §2 · p. 2
- Smart-home research examines natural-language interfaces, device interaction, and increasingly complex or environment-aware tasks, primarily with unpruned LLMs.Examples include HomeBench’s valid and invalid instructions and SimuHome’s temporal and environment-aware interactions.Source: §2 · p. 2
- The reliability of pruned LLMs for smart-home tool execution remains largely unexplored.This gap lies between pruning studies that mainly report aggregate accuracy and smart-home studies focused on unpruned models.Source: §2 · p. 2
3 Evaluation Framework
The evaluation framework treats smart-home tool calling as context-grounded structured prediction, then examines pruning effects across action components and task complexity. It compares architectures and compatible pruning methods using accuracy measures relative to unpruned models.
3 Evaluation Framework
- The framework evaluates pruning degradation at two complementary granularities: action components identify what parts of execution are affected, and task complexity identifies under which conditions degradation occurs.The overview spans action components and task-complexity levels.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Problem Formulation
- Each instance combines a user request, household context, and available actions; each required execution is represented by operation, target device, argument names, and values.For example, “dim the bedroom light to 50” maps to bedroom.light, set, brightness, and 50.Source: §3.1 · p. 2
3.2 Smart-Home Task-Complexity Taxonomy
- Requests are partitioned into five mutually exclusive categories: Simple, Medium, Complex, Partially Executable, and Infeasible.They distinguish single-device tasks, multiple or underspecified requests, requests requiring prior context, partially feasible actions, and requests with no feasible action.Source: §3.2 · pp. 2–3
3.3 LLM Architecture Families
- Four Qwen-family models span dense Transformer, dense hybrid, and sparse MoE designs, enabling comparisons across architecture and parameter scale.The design compares Qwen3-4B with Qwen3.5-4B at approximately fixed scale, Qwen3.5-4B with Qwen3.5-9B, and includes Qwen3.6-35B-A3B as a sparse MoE model.Source: §3.3 · p. 3
3.4 Pruning Methods
- The study uses depth, width, hybrid, and expert pruning, applying methods to compatible architectures and holding each pruned architecture fixed during shared SFT.ShortGPT and Angular provide depth pruning, FLAP width pruning, 2SSP hybrid pruning, and REAP expert pruning.Source: §3.4 · pp. 3–4
3.5 Pruning-Degradation Evaluation
- Pruning degradation is measured relative to the corresponding unpruned model using overall, action-component, and task-complexity accuracy changes.Overall task accuracy requires the complete execution; action components are scored individually, while task accuracy is reported for each complexity category.Source: §3.5 · p. 4
4 Experiments
The experiments evaluate post-pruning models on three smart-home datasets and find that degradation depends on architecture, action component, and task complexity. Dense pruning can also shift executable requests toward refusal.
4.1 Experimental Setup
- The evaluation uses training and test splits from SHTC, HomeBench, and HAR, with the same SFT mixture healing all pruned models.For proprietary SHTC data, the paper reports performance changes relative to each corresponding unpruned model.Source: §4.1 · p. 4
4.2 Results
- Dense pruning shows sharp accuracy cliffs, whereas MoE pruning retains a wider plateau; REAP removes 70% of experts with a nonsignificant −0.37% change.REAP reduces stored parameters from 35B to approximately 12.5B while preserving routing and eight active experts per token.Source: §4.2 · pp. 4–5, Figure 4 · p. 6
- At 50% dense pruning, mean losses are 24.2% vs. 21.1% for device vs. operation and 20.7% vs. 14.5% for value vs. argument.The authors attribute this specificity-first degradation to instance-specific grounding demands, compared with the repeated schema for operation and argument.Source: §4.2 · p. 5, Figure 5 · p. 6
- At 50% pruning, executable-request refusal rises from 7.7% unpruned to 56.2% with Angular and 80.8% with FLAP.Yet Angular and FLAP still refuse 86.8% and 99.9% of infeasible requests, respectively.Source: Figure 7 · p. 6, §4.2 · p. 6
- Partially Executable requests lose about 10.3% accuracy per additional 10% pruning, more than Simple (7.7%) and Medium (7.2%) requests.These requests require coordinating feasible and infeasible portions while executing and rejecting selectively.Source: §4.2 · pp. 5–6, Figure 6 · p. 6
5 Discussion and Future Work
The discussion emphasizes preserving environment-grounded specificity and evaluating pruning beyond aggregate accuracy. It recommends conservative dense pruning and future recovery methods targeted to grounding and execute-versus-reject behavior.
5 Discussion and Future Work
- Smart-home pruning must preserve contextual device and value predictions as well as general operation and argument capability.Partially Executable requests are especially sensitive, and aggressive pruning can shift models toward over-refusal.Source: §5 · p. 6
- For bounded-domain MoEs, expert pruning is a promising starting point; dense LLM pruning should be more conservative and tested across multiple ratios and criteria.Source: §5 · p. 6
- Future recovery methods should target environment grounding and calibrate execution versus rejection using feasible and infeasible requests.Suggested targets include distractor-rich device resolution, precise values, multi-device requests, and partially executable instructions.Source: §5 · p. 6
6 Conclusion
The paper evaluates pruning effects on context-grounded smart-home tool calling across architectures, pruning methods, and severities. It finds that grounded specificity degrades before schema-level intent, and that aggressive dense pruning can induce over-refusal.
6 Conclusion
- The study examines pruning-induced degradation across architectures, methods, and severities at both action-component and task-complexity levels.Its fine-grained evaluation covers context-grounded smart-home tool calling.Source: §6 · pp. 6–7
- Device and value prediction degrade before operation and argument prediction, while aggressive dense pruning can induce over-refusal.The study argues that pruning should be evaluated beyond overall accuracy, including by architecture, workload, and failure mode.Source: §6 · p. 7
- Fine-grained evaluation is important for identifying pruning regimes that reduce model cost while preserving reliable smart-home tool execution.Source: §6 · p. 7
Limitations
The study’s evidence is bounded by its model families, pruning-method coverage, and smart-home evaluation scope. The authors identify extensions to other families, domains, methods, and deployment-level efficiency measures as future work.
Limitations
- All evaluated models belong to the Qwen family, so whether the observed pruning patterns generalize to other model families remains unstudied.Source: Limitations · p. 7
- Pruning methods are not evaluated factorially because some are architecture-specific, including FLAP and 2SSP for Gated DeltaNet-based models.Source: Limitations · p. 7
- Evaluation is restricted to smart-home tool calling, and some complexity categories appear in only a subset of the three datasets.The authors identify this as limiting broader generalization across domains.Source: Limitations · p. 7
A Additional Experimental Setup
The appendix details the experimental setup and prompt, then reports action-component, complexity-level, and operation-F1 results. Across these analyses, specificity degrades before schema-level intent, while pruning sensitivity varies by task category.
A Additional Experimental Setup
- Dense pruning ratios span 10%–50%, while MoE expert pruning spans 10%–90%; SFT balances 16,667 examples each from SHTC, HomeBench, and HAR.SFT uses one epoch with 1,563 optimizer steps; inference uses greedy decoding at temperature 0, with sampling settings also evaluated.Source: A Additional Experimental Setup · p. 8
B Prompt Example
- HomeBench prompts provide current device state, capabilities, valid ranges, callable methods, and output format; unsupported devices or attributes require “error_input”.Source: B Prompt Example · p. 8, Figure 8 · p. 9
C Action-Component Accuracy
- Device and value accuracy often decline faster than operation and argument accuracy across HomeBench, HAR, and SHTC, while REAP-pruned MoE stays stable until the most aggressive expert pruning.This pattern indicates that pruning tends to damage grounded specificity before schema-level intent.Source: C Action-Component Accuracy · p. 10
D Complexity-Level Accuracy
- Under aggressive dense pruning, degradation varies by category across HomeBench, HAR, and SHTC, while REAP-pruned MoE remains stable until the highest expert-pruning ratios.Baseline task difficulty does not directly determine pruning sensitivity.Source: D Complexity-Level Accuracy · p. 10
E Operation-Level F1 Scores
- On SHTC, d loses 28.7% of F1 from M0 to the most aggressive ratio, more than p, v, or o.On HAR, d loses 17.6% points against 9.5% for p, 9.6% for v, and 7.8% for o.Source: E Operation-Level F1 Scores · p. 10
F.1 Robustness Across the Pruning Ratios
The pruning sweep can preserve aggregate accuracy while changing which requests succeed. Instance-level stability varies across datasets, request categories, and action components.
F.1 Robustness Across the Pruning Ratios
- The analysis classifies outcomes across ten checkpoints from 0% to 90% to track instance correctness throughout pruning, rather than variation across random seeds.Source: §F.1 · p. 10
- Despite a nearly flat aggregate curve, HomeBench exact-match outcomes flip for 14.1% of instances, rising to 33.7% for Partially Executable and 24.9% for Medium requests.Source: §F.1 · p. 10
- HAR has a 25.1% exact-match flipping band, concentrated in device grounding (24.4%) and Simple requests (27.4%); on SHTC, Medium flips most (29.4%) and operation selection least (11.0%).Source: §F.1 · p. 10
F.2 Robustness Across Seeds
Across-seed robustness was measured over three repeated runs; results show that task difficulty and seed instability can diverge.
F.2 Robustness Across Seeds
- Robustness used three runs with identical weights and sampling settings (temperature 0.7), classifying outcomes as always correct, always wrong, or flipping.Flipping measures cross-seed instability; always-wrong outcomes form a seed-invariant failure core.Source: §F.2 · p. 11
- On SHTC, Medium and Complex flipped more often than Simple: 12.8% and 11.6% versus 6.3%.Source: §F.2 · p. 11
- 24.1% of HomeBench Medium instances flipped across seeds, versus 7.1% of Infeasible; HAR Medium instead was 79.7% always wrong with 4.5% flipping.Figure 22 distinguishes outcomes that are always correct, flipping across seeds, or always wrong.Source: §F.2 · p. 11, Figure 22 · p. 23