Source-linked AI summary
Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions
Ahmed El Kady, Aravind Narayanan, Rehana Noorani, Yani Ioannou, Shaina Raza
TL;DR
Responsible-AI evaluation can change the evidence used for model-behavior claims when its protocol is made cheaper. The paper stress-tests batching, quantization, benchmark reduction, and combinations across three models and two benchmarks, finding that savings and conclusion stability vary by intervention and subgroup.
Problem
It is rarely tested whether responsible-AI benchmark conclusions remain stable after evaluation is made faster or cheaper.
Method
The study compares three models and two benchmarks across seven efficiency conditions using full-benchmark BF16 evaluation as the reference, measuring quality, subgroups, membership stability, runtime, and GPU energy.
Results
Larger batching is comparatively conclusion-stable, INT8 largely preserves quality but uses 1.79–4.26× baseline energy, INT4 causes larger changes, and very small subsets are sensitive to retained items.
Takeaways & Limitations
Efficient evaluation should be treated as a measurement intervention whose validity is checked across the conclusions and subgroups the benchmark is intended to support.
Takeaways & Limitations
The study covers three models, two benchmarks, one hardware environment, and fixed judge and measurement choices, so effects may differ elsewhere.
Abstract
from arXiv · showhide
Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.
1 Introduction
The paper asks whether making responsible-AI evaluation faster or cheaper preserves the claims benchmarks support. It stress-tests protocol changes across models, benchmarks, quality measures, and operational costs.
- Motivation: Evaluation protocols affect which claims about AI accuracy, fairness, safety, and reliability benchmarks can support.Relevant protocol choices include retained examples, inference execution, numerical representation, and inspected metrics.
- Motivation: Efficiency interventions are useful only when the resulting evaluation still supports its intended conclusions.
- Research gap: Aggregate accuracy can remain nearly unchanged while measured bias, reasoning quality, or subgroup performance shifts.
- Approach: The study evaluates three models on BBQ and BBQ-V under batching, quantization, benchmark reduction, and combined conditions.It compares quality, subgroup behavior, subset sensitivity, inference time, and GPU energy against full-benchmark BF16 evaluation.
- Contributions: The contributions show that efficiency interventions differ in conclusion stability, reduced evaluations depend on retained examples, and runtime or precision do not reliably predict energy savings.The paper frames evaluation efficiency as a measurement-validity question rather than systems optimization alone.
2 Related Work
Prior work motivates measuring evaluation’s computational footprint and testing whether protocol changes alter responsible-AI conclusions. This paper extends reduced-evaluation reliability analysis to multiple metrics and heterogeneous groups.
- Computational cost: Inference-only energy measurement complements prior work that primarily quantified training costs and warns that runtime or precision may not proxy energy adequately.
- Responsible-AI evaluation: Compression studies show that quantization, distillation, and pruning can change demographic behavior even when aggregate accuracy remains similar.
- Reduced evaluation: Compact subsets can recover aggregate scores or rankings, but responsible-AI benchmarks require checking multiple metrics and heterogeneous groups.
- Evaluation workflow: The workflow compares quality and footprint quantities for each model–benchmark pair under M0 or an efficiency intervention.
- Reduced evaluation: Subset experiments preserve complete natural units, repeat selected sizes with five frozen memberships, and report metric variability as benchmarks shrink.
3 Study Design
The study perturbs a full-benchmark BF16 protocol and tests whether aggregate, subgroup, and membership conclusions remain stable across efficiency interventions.
- Protocol: M0 is full-benchmark BF16 inference at batch size 1, while alternatives change batching, numerical representation, benchmark composition, or combinations.
- Comparison rule: The study reports signed changes from M0 rather than imposing a universal equivalence margin.Tolerance for accuracy or fairness changes is treated as application-dependent, and the experiments are not preregistered equivalence tests.
- Robustness analysis: Robustness is assessed through aggregate metrics, stratified subgroup views, and sensitivity to which natural units are retained.These views are not collapsed into a single pass/fail score.
- Benchmarks and models: BBQ contains 2,000 text questions across eleven categories, while BBQ-V contains 1,998 image-grounded questions across nine categories.Both frozen sets are category-balanced and reduced only at complete natural-unit boundaries.
- Benchmarks and models: The models comprise Qwen2.5-VL-7B, Qwen3-VL-30B-A3B, and Gemma-4-12B-IT, spanning dense and mixture-of-experts architectures without a controlled architectural ablation.
- Metrics and footprint: Accuracy, Bias Score, Bias Present, and Reasoning Quality are evaluated with fixed definitions and a fixed judge model.Judge-choice robustness is outside the study’s scope.
- Robustness analysis: Repeated M4 subsets overlap, so variation across memberships is interpreted descriptively rather than as independent-sample uncertainty.
- Metrics and footprint: GPU energy is measured directly during inference with NVML, excluding loading, preparation, judging, serialization, and artifact upload.
4 Aggregate Robustness Across Evaluation Interventions
The interventions differ substantially in how well they preserve conclusions and reduce computational cost. Larger batching is least disruptive, INT8 preserves aggregate quality but raises energy, and benchmark reduction is most consistently economical.
- Batching: 0.35 percentage points is the maximum M1 accuracy change across all six model–benchmark settings, with comparatively small aggregate and category-level changes.GPU energy decreases in five of six settings, so batching is not universally lower-energy.
- Quantization: 0.75 points is the maximum INT8 aggregate accuracy difference from M0, but every M2 run uses 1.79–4.26× the matching M0 GPU energy.
- Quantization: 2.40–7.05 points is the INT4 accuracy reduction in five of six settings, alongside Reasoning Quality reductions in five of six settings.Bias Score changes direction across model families, and context-level effects can be localized.
- Results overview: Table 1 pairs Accuracy, bias measures, Reasoning Quality, and GPU energy relative to matching M0 for BBQ and BBQ-V.M4 reports mean ± standard deviation across five frozen memberships where available.
- Combined interventions: 0.11–0.58× M0 energy is used by M5a while accuracy changes by −1.13 to +0.70 points.Runtime decreases in all six M1 settings, but energy decreases in only five; one case has 23.1% lower runtime and 15.3% higher energy.
5 When Aggregate Agreement Hides Conclusion Changes
Aggregate agreement can conceal changes in responsible-AI conclusions. INT4 effects vary by model and context, while category-level metrics can shift even when aggregate performance appears stable.
- Context-dependent INT4 effects: −9.10 accuracy points, +4.70 Bias Present points, and −4.18 Reasoning Quality points occur for Qwen2.5-VL on ambiguous BBQ contexts.Its disambiguated stratum remains much closer to M0.
- Context-dependent INT4 effects: −7.20 accuracy points and −4.06 Reasoning Quality points occur for Qwen3-VL on ambiguous contexts, versus −1.20 and −0.18 in disambiguated contexts.The comparison shows that INT4 effects can localize to particular context strata.
- Energy and conclusion trade-offs: Figure 2 compares measured GPU energy, accuracy, and bias, using a logarithmic energy axis and darker markers for lower bias prevalence.The display places conclusion changes alongside measured computational cost.
- Context-dependent INT4 effects: +0.30 ambiguous accuracy points but −14.40 disambiguated accuracy points occur for Gemma under INT4.Gemma therefore shows the opposite context pattern from the other reported models.
- Category-level effects: INT4 produces larger changes than batching, with magnitude and direction depending on model and category.The contrast spans Accuracy, Bias Present, and Reasoning Quality across BBQ and BBQ-V.
- Category-level effects: Gemma’s INT4 accuracy decreases across all eleven BBQ categories by 4.35–12.78 points, while its nine BBQ-V category changes are non-negative.This illustrates why aggregate INT4 scores alone do not identify which part of the construct changed.
6 Reliability of Reduced Evaluations
Benchmark reduction yields predictable energy savings, but conclusion reliability becomes increasingly dependent on which examples are retained at very small sizes. Mean agreement with baseline is therefore insufficient evidence for arbitrary reduced subsets.
- Quality and energy scaling: R2 = 0.999: measured GPU energy falls almost proportionally with benchmark size.The computational benefit of reducing benchmark size is therefore highly predictable.
- Quality and energy scaling: At 50%, mean accuracy changes range from −0.32 to +1.28 points across six model–benchmark settings.Bias Score changes are at most 0.42 points and Reasoning Quality changes are at most 0.37 points.
- Item-membership sensitivity: At the smallest subsets, mean results can remain close to baseline while spread across five memberships becomes much larger.Agreement of the average subset result does not establish that an arbitrary small subset is reliable.
- Item-membership sensitivity: Aggressive reduction makes evaluation more contingent on item selection.The five frozen memberships overlap, so their standard deviations are descriptive rather than independent-sampling confidence intervals.
- Practical boundary: Reduced evaluations should be validated as measurement instruments rather than assumed reliable at a universal fraction such as 50%.The supported practice is to specify membership, assess multiple metrics, and periodically recheck the subset.
7 Implications for Trustworthy Efficient Evaluation
Trustworthy efficient evaluation requires checking conclusion stability rather than headline accuracy alone. Interventions should be validated individually, reduced sets treated as versioned instruments, and computational cost measured directly.
- Principles: Preserve conclusions, not only headline accuracy, because judge-derived and subgroup metrics can move when accuracy does not.The paper identifies this as the first principle for trustworthy efficient evaluation.
- Principles: Batching, quantization, and subsetting have different failure modes, so efficiency interventions should be validated individually.The paper does not treat these interventions as interchangeable.
- Principles: Reduced sets should be treated as versioned measurement instruments: fix membership, validate multiple metrics, and retain the full benchmark for final claims.This recommendation follows the paper’s reduced-evaluation findings.
- Principles: Faster execution, lower precision, and sparse activation do not guarantee lower energy; cost should be measured directly.The paper explicitly rejects assuming computational savings from these properties alone.
- Principles: No universal “safe” delta is proposed because acceptable tolerances remain application-dependent.The paper frames this as a scope boundary for interpreting intervention validity.
8 Conclusion
Efficient evaluation is a measurement-protocol change whose validity must be demonstrated across the conclusions and subgroups a benchmark is intended to support.
- Larger batching is comparatively conclusion-stable and often lowers energy, while INT8 preserves aggregate quality but increases energy.
- INT4 produces the largest model- and context-dependent changes among the tested interventions.
- Benchmark reduction provides the most consistent computational saving, but very small subsets become increasingly sensitive to retained examples.
- Efficient evaluations should be trusted only after intended-claim quantities, including subgroup behavior when relevant, are checked against an appropriate reference protocol.
9 Limitations
The study's conclusions are bounded by its models, benchmarks, hardware, evaluators, subset design, and measurement assumptions.
- The study covers three models, two responsible-AI benchmarks, and one hardware environment, so absolute costs and intervention effects may differ elsewhere.
- The model comparison is not a controlled architectural ablation, so the MoE model's higher baseline energy cannot be attributed uniquely to routing or sparsity.
- Overlapping repeated subsets provide descriptive membership sensitivity rather than independent-sampling uncertainty, and very small subsets may age as models or distributions change.
- Bias and Reasoning Quality use one fixed judge model and rubric, so judge-choice robustness is not established.
- Reported water values are sensitivity estimates rather than direct measurements because facility-specific water data were unavailable.
- Acceptability remains decision-dependent because the study reports deviations rather than formal equivalence tests.
B Metric Definitions
The paper defines accuracy, subgroup fairness, judge-derived bias and reasoning metrics, and inference-only operational-footprint measurements relative to specified baselines and conditions.
- Metric Definitions: Accuracy is the percentage of questions answered correctly, reported as percentage-point differences from the corresponding BF16 baseline.
- Metric Definitions: Fairness is assessed by reporting accuracy separately for each benchmark subgroup.
- Metric Definitions: Bias Score is mean severity on a 1–100 scale, while Bias Present is the percentage of assessable responses with Bias Score > 1.
- Metric Definitions: Reasoning Quality is the mean 1–100 judge score, with higher values indicating better relevance, coherence, contextual support, and freedom from unsupported claims.
- Metric Definitions: Subgroup judge-metric comparisons use examples with valid scores in both condition and baseline runs, preventing judgeability changes from affecting deltas.
- Operational footprint: Runtime and GPU energy cover inference only, excluding loading, preparation, judging, serialization, and artifact upload.
- Operational footprint: GPU energy is measured with NVML, while water estimates use CodeCarbon-tracked total energy and a 1.8–4.0 L/kWh coefficient.
- Evaluation conditions: M4 subset means summarize overlapping frozen memberships descriptively rather than as independent estimates.
C Additional Results
The additional results materials document intervention trade-offs, benchmark-size sensitivity, subgroup comparisons, and the judge procedures underlying the reported metrics.
- Additional Results: Runtime and measured GPU-energy savings are compared against the BF16 baseline, with the equal-savings diagonal identifying divergence between the two measures.
- Additional Results: Table C.1 reports INT4 effects by BBQ context ambiguity with natural-unit cluster-bootstrap confidence intervals pooled over question polarity.
- Additional Results: Benchmark-size plots show changes from the full M0 result, with five-subset standard deviations reported at 50% and the smallest retained size.
- Additional Results: Table C.2 provides an exploratory category-level decomposition across 11 categories and three models, so per-category intervals are descriptive rather than independently confirmed findings.
- Additional Results: Figures C.4–C.9 encode demographic-group changes by model and intervention, using red for positive and blue for negative changes with darker magnitudes.
- Judge prompts: The fixed judge scores generated reasoning without access to the gold label, assigning null Bias scores to non-substantive reasoning and always scoring Reasoning Quality.