Source-linked AI summary
Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task
Enrique Barba Roque, Luís Cruz, Annibale Panichella
TL;DR
Large language models impose substantial computational and energy costs, while FLOPs may not reliably capture actual energy consumption. This paper extends Morph with measured-energy surrogate objectives and applies it to classification and CodeT5+ summarization, finding lower energy and memory with modest accuracy trade-offs.
Problem
The paper asks whether FLOPs reliably proxy energy consumption for distilled LLMs and whether energy-aware distillation generalizes from classification to generation tasks.
Method
The study evaluates Morph, adds CPU and GPU energy-surrogate models based on measured consumption, and applies the extended method to CodeT5+ code summarization.
Results
FLOPs are often unreliable indicators of inference energy, while energy-surrogate optimization yields up to 90% lower energy and 86% smaller memory footprint with modest accuracy impact.
Takeaways & Limitations
Direct energy surrogates can guide distilled student models toward improved energy efficiency beyond what FLOPs-based optimization provides.
Takeaways & Limitations
The generation evaluation uses ROUGE-L, which overlooks semantic similarity and avoids human or LLM judgment because repeated evaluation would be costly.
Abstract
from arXiv · showhide
Background: Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustainability concerns and hinder their use on consumer hardware and resource-constrained platforms. A common way to report the computational cost of an LLM in the literature and industry is to use the number of Floating Point Operations (FLOPs) required to perform a pass over the network. Aims: This paper investigates the implications of energy-aware knowledge distillation for SE, aiming to improve model efficiency while maintaining performance and to determine whether FLOPs is a reliable energy-aware metric. Method: We conduct a controlled experiment using Morph, a Many-Objective Optimization-based distillation methodology, to empirically examine whether FLOPs accurately reflect energy consumption in Clone Detection and Vulnerability Prediction tasks. We extend this methodology to include energy-surrogate models that directly estimate CPU and GPU energy consumption during optimization, and we apply Morph to generative tasks using CodeT5+ for code summarization. Results: Our results show that FLOPs is not always a reliable indicator of energy consumption, and better results can be achieved by using energy-surrogate models. Distilled student models can reduce inference energy consumption by up to 90\% and memory usage by 86\%, with only modest accuracy trade-offs. Conclusions: Energy-aware knowledge distillation when guided by direct energy surrogates rather than FLOPs can improve the energy consumption, sustainability, and deployability of LLMs for SE applications, enabling efficient models on consumer hardware.
1 Introduction
This paper examines whether FLOPs reliably represent energy consumption in distilled LLMs for Software Engineering and extends Morph toward direct energy optimization and generation tasks. It evaluates energy-aware distillation across classification and code summarization, targeting smaller models with modest accuracy trade-offs.
- Motivation: LLMs support SE classification and generation tasks but require substantial computational resources and energy at scale.These demands motivate efficiency improvements for practical deployment.
- Motivation: Inference dominates lifetime energy usage for high-volume models, making compression relevant to runtime energy and local IDE deployment.Compressed models reduce per-query compute and memory footprints.
- Research gap: FLOPs may fail as an energy proxy because they omit memory access, data movement, and hardware-level execution factors.Their hardware-agnostic simplicity does not guarantee correspondence with measured consumption.
- Approach: Morph is extended to optimize measured energy directly using energy-surrogate models that estimate consumption from hyperparameter combinations.The extension replaces FLOPs-based efficiency estimation with profiler-derived energy objectives.
- Evaluation: The study applies the extended methodology to CodeT5+ code summarization in addition to Clone Detection and Vulnerability Prediction.Generation is evaluated using accuracy, model size, and energy consumption.
- Results: Up to 90% lower energy and 86% smaller memory footprint are achieved with modest accuracy trade-offs.These outcomes are reported for the extended application to CodeT5+ code summarization.
2 Background
Knowledge distillation compresses a large teacher into a smaller student, while Many-Objective optimization searches student configurations for trade-offs among competing objectives. Morph combines evolutionary search with surrogate models to explore this space efficiently.
- Knowledge distillation: Knowledge distillation trains a smaller student model to reproduce a larger teacher’s responses with fewer resources.The student learns from the teacher rather than directly from the dataset alone.
- Knowledge distillation: Distillation can reduce dataset and convergence costs, but complex generation tasks may require pretrained or copied student weights.Binary classification students may be randomly initialized, whereas generation students need more prior knowledge.
- Morph: Morph formulates student hyperparameter selection as a Many-Objective optimization problem over variables such as layers and attention heads.The search space grows rapidly with the number of configurable architectural choices.
- Morph: Morph uses AGE-MOEA to evolve configurations toward a Pareto front balancing competing objectives.Pareto-optimal solutions cannot improve one objective without worsening another.
- Morph: Surrogate models predict accuracy and prediction flips for sampled configurations, reducing the need to train and evaluate every possible student.Morph samples the configuration space before evolutionary exploration.
3 Related Work
Prior work applies multi-objective distillation mainly to binary classification and commonly treats FLOPs as an energy proxy. This paper addresses the remaining questions of FLOPs validity for LLMs and Morph’s applicability to generation.
- Multi-objective distillation: Earlier distillation methods optimized combinations of accuracy, FLOPs, model size, and later robustness using genetic algorithms.Compressor used a single objective, while Avatar introduced three objectives.
- Morph: Morph extends this line of work with robustness, surrogate models, and Many-Objective optimization but still uses FLOPs for energy-related optimization.The relationship between FLOPs and energy depends on hardware and execution optimizations.
- Scope gap: Previous approaches were evaluated on Clone Detection and Vulnerability Prediction rather than generation tasks.Binary classification permits simpler models and more aggressive compression than generation.
- Scope gap: The paper applies Morph to Code Summarization to test whether its benefits extend beyond binary classification.Generation is treated as the new target application area.
- FLOPs and energy: FLOPs-based energy validity remains unsettled because prior findings do not establish whether the relationship transfers reliably to LLMs.Many LLM studies nevertheless continue to use FLOPs to represent energy costs.
4 Methodology
The methodology measures energy for Morph’s candidate students, replaces FLOPs with energy surrogates for optimization, and adapts distillation objectives and training for CodeT5+ generation. It balances model size, energy, and task effectiveness using ROUGE-L for summarization.
- Energy-aware optimization: For RQ1, the study measures energy usage of distilled models on Morph’s Pareto front while retaining the original approach.For RQ2, energy becomes an additional optimization objective.
- Energy-aware optimization: Energy surrogates predict CPU and GPU consumption from sampled hyperparameter configurations, replacing FLOPs as the efficiency objective.The predicted CPU and GPU values are combined during optimization.
- Generation objectives: For generation, Morph optimizes summary similarity while minimizing inference energy and model size.The generation formulation removes the original robustness metric and uses task-specific effectiveness measures.
- Generation objectives: ROUGE-L is selected as the Code Summarization effectiveness metric because human or LLM judgments would make repeated model evaluation costly.ROUGE-L is an n-gram-based metric reported to align better with human judgment than several alternatives for code-related tasks.
- Generation distillation: Weight subcloning initializes generation students by copying important teacher weights ranked from neuron and attention-head behavior.The cited study reports 4x faster training for students initialized this way.
- Generation distillation: Online distillation is used because generative logits are too large to store, requiring teacher and student models to run together for each batch.This avoids logits storage but increases computational demand through simultaneous model execution.
- Generation distillation: The generation loss averages Kullback-Leibler divergence with cross-entropy to align student distributions with teacher outputs and ground truth.KLD compares student and teacher token distributions, while cross-entropy preserves ground-truth alignment.
5 Evaluation
The evaluation tests Morph’s energy-aware distillation across classification and generation tasks, asking whether FLOPs reliably represent energy and whether direct energy objectives improve optimization.
- Evaluation goals: The study evaluates FLOPs as an energy proxy and tests energy-aware Morph distillation on Clone Detection, Vulnerability Prediction, and code summarization.The evaluation includes classification tasks with GraphCodeBERT and generation with CodeT5+.
- RQ1: FLOPs and energy: Morph generates Pareto-front models with varying FLOPs, whose measured inference energy is analyzed using statistical tests and hyperparameter correlations.The Pareto front provides diverse trade-offs, while ANOVA examines relationships among energy, FLOPs, and model hyperparameters.
- RQ2: Energy optimization: The modified Morph replaces the FLOPs objective with profiled energy predictions from a surrogate model and compares both approaches across accuracy, size, robustness, FLOPs, and inference energy.Twenty models are distilled for each approach, with median and interquartile-range results compared using the same approximately 3 MB selection criterion.
- RQ3: Generation: CodeT5+ students are evaluated against the teacher using ROUGE-L, model size, inference energy, and manual inspection of generated summaries.The generation-task evaluation also considers whether outputs remain suitable under human judgment.
- Measurement setup: Inference energy measurements exclude tokenization and decoding and repeatedly run the test dataset to reduce loading and environmental variance.Measurements use EnergiBridge on an AMD CPU and NVIDIA GPU server.
6 Results
Results show that FLOPs are task-dependent rather than a reliable universal energy proxy, while direct energy optimization reduces energy substantially in some settings and supports efficient code summarization students.
- RQ1: FLOPs and energy: Clone Detection shows no clear FLOPs–energy relationship: a 0.20 GFLOPs model uses around 8,001 J, while a 2.4 GFLOPs model uses 4,423 J.The latter uses approximately 44% less energy despite having more FLOPs.
- RQ1: FLOPs and energy: For Clone Detection, FLOPs fail as an energy proxy (p = 0.17), whereas Vulnerability Prediction shows a statistical association (p = 6.48 × 10^-7).The task-dependent contrast and models with similar FLOPs but different energy indicate that FLOPs are not directly proportional to energy.
- RQ2: Energy optimization: 39% lower median inference energy is achieved for Clone Detection when surrogate energy replaces FLOPs, from 5355.83 J to 3604.50 J (p < 0.01, Â12 = 0.925).The energy-optimized models use 21% more GFLOPs, showing that lower FLOPs did not determine lower energy in this task.
- RQ2: Energy optimization: For Vulnerability Prediction, energy optimization changes median energy by +30% and GFLOPs by +36%, but differences are not statistically significant (p > 0.2), with accuracy also non-significantly different (p > 0.3).This is the task where FLOPs already correlate with energy.
- RQ3: Generation: 90% lower inference energy is obtained for the most efficient code-summarization student, while the most accurate student reduces model size by 86%.ROUGE-L decreases from 0.229 to 0.199 (−13%) for the most accurate student and to 0.194 (−15%) for the most efficient student.
7 Discussion
The discussion finds that FLOPs can diverge substantially from inference energy because they omit runtime, hardware, software, and task-specific factors. Energy-surrogate models can therefore improve optimization outcomes, while distilled models offer substantial efficiency gains with task-dependent accuracy trade-offs.
- FLOPs and energy: FLOPs show no correlation with Clone Detection energy but a positive correlation with Vulnerability Prediction energy, without establishing a direct relationship in either task.Models with similar FLOPs can consume very different amounts of energy, and vice versa.
- Why FLOPs diverge: Runtime optimizations, memory access, data movement, activation functions, padding, and hardware behavior make FLOPs an incomplete energy proxy.FLOPs count mathematical operations and may treat padded or optimized-away computations as ordinary work.
- Task effects: For Clone Detection, doubled padding creates more “empty” FLOPs, helping explain the task’s lack of correlation between FLOPs and energy.Concatenating two code blocks doubles the effective maximum sequence length used for tokenization.
- Energy-aware optimization: Energy-surrogate models produce more efficient models when FLOPs cannot reflect energy, while showing no statistically significant final-result effect where FLOPs and energy correlate.Energy measurement adds costly sampling and evaluation steps, particularly for larger students and complex tasks such as Code Summarization.
- Generation tasks: Code Summarization distillation reduces student size by up to 86% and inference energy by up to 90%, but all students show a modest ROUGE-L decrease relative to the teacher.The acceptability of this performance trade-off depends on the specific use case.
- Architecture flexibility: Many-objective distillation expands student-architecture flexibility through weight subcloning rather than restricting configurations to pretrained smaller checkpoints.The approach copies important teacher weights, enabling a wider range of hyperparameter configurations.
8 Threats to Validity
The paper identifies validity threats in its generative-task design, especially the use of a standard distillation loss and ROUGE-L as the optimization objective.
- Distillation loss: The Code Summarization experiments use standard KL-divergence and cross-entropy distillation losses, although newer losses and paradigms may improve results.The authors selected the standard loss to focus on exploring the configuration space with many-objective optimization.
- Evaluation metric: ROUGE-L measures n-gram overlap and can overlook semantically equivalent summaries expressed through different paraphrases.Semantic evaluation would require human judgment or a reliable LLM, increasing time and financial costs.
9 Conclusion
The study concludes that FLOPs are often unreliable for selecting energy-efficient distilled models, whereas energy surrogates improve optimization when FLOPs fail. CodeT5+ distillation also achieves substantial size and energy reductions with modest accuracy costs.
- FLOPs as an energy proxy: FLOPs do not consistently track student-model inference energy, and lower FLOPs does not always identify lower-energy models.Clone Detection lacks statistical correlation, while Vulnerability Prediction shows correlation without a direct relationship.
- Energy surrogates: 39% more energy-efficient Clone Detection models result from surrogate energy models estimating consumption before training.The surrogates address the gap between FLOPs and energy efficiency during optimization.
- Generative distillation: CodeT5+ students achieve up to 86% size reduction and up to 90% lower energy usage, with a modest accuracy impact.The approach was evaluated for Code Summarization as a generative SE task.
10 Data Availability
A replication package is available in Zenodo and contains the materials needed to reproduce distillation, energy measurement, and analysis workflows.
- Replication package: The Zenodo replication package includes distillation scripts, energy-measurement scripts, hardware data, and analysis scripts for plots and tables.It contains scripts for both surrogate and final models.