PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems

Wooyoung Jung

arXiv:2610.00073v1cs.HC

TL;DR

Existing home energy management systems offer limited sustained interaction and preference elicitation. This study proposes a 23-metric, GQM-grounded framework and validates automated scoring against human coders; the framework distinguishes four configurations, with 12 of 15 LLM-scored metrics reaching ICC ≥ 0.73.

  • Problem

    Existing HEMS often treat occupants as passive recipients, offering limited sustained interaction, explicit preference elicitation, and interpretability.

    Source: §1 · p. 2

  • Method

    The study proposes a 23-metric, GQM-grounded framework across five categories and applies it to four configurations using 970 multi-turn conversations.

    Source: §1 · p. 2, §7 · p. 22

  • Results

    Across four configurations, the framework demonstrated discriminative validity; 12 of 15 LLM-scored metrics achieved ICC ≥ 0.73, with three reaching perfect agreement.

    Source: §1 · p. 2, §7 · p. 22

  • Takeaways & Limitations

    The framework provides diagnostic granularity for distinguishing failure modes such as over- versus under-personalization and schedule misalignment versus constraint violation.

    Source: §7 · p. 22

    Evaluation used simulated rather than real occupants, which may yield more cooperative and less ambiguous queries and represent an optimistic upper bound on performance.

    Source: Limitations · p. 21

Abstract

from arXiv · show

The growing complexity in home energy management (HEM) demands advanced systems that guide occupants toward informed energy decisions reflecting their background, preferences, and context. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. Whether such systems deliver effective interactions across the extended multi-turn dialogues typical of real-world use remains an open question. This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. Its reliability is validated against three trained human coders: after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC >= 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0.04-0.28). To demonstrate the framework's effectiveness, 970 dialogues -- 16 scenarios and five personas -- were generated and evaluated across four conversational HEMS configurations spanning a sophistication gradient, from a vanilla LLM with raw energy data to a multi-agent HEMS. The framework distinguished the four configurations across multiple evaluation dimensions, revealing their respective strengths and weaknesses. This study contributes to conversational HEMS by providing a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.

1 Introduction

Growing household complexity calls for HEMS that coordinate energy assets while adapting to occupants, but existing systems offer limited sustained interaction and personalization. Prior conversational HEMS evaluations largely use single-turn or single-task accuracy measures, leaving multi-turn interaction quality and other domain-specific dimensions underassessed.

1 Introduction

  • Residential energy complexity increases the need for systems that coordinate heterogeneous assets and balance household costs, comfort, demand response, and grid reliability.Homes combine distributed generation, high-power devices, and increasingly unpredictable conditions.Source: §1 · p. 1
  • Existing HEMS often rely on centralized optimization, static rules, or single-agent controls, with limited sustained interaction, preference elicitation, and decision interpretability.Occupants are typically treated as passive recipients, limiting ongoing feedback and personalization.Source: §1 · pp. 1–2
  • Conversational HEMS show promise for context-adapted decisions and more transparent interaction, but are commonly evaluated with accuracy-focused, single-turn or single-task measures.Such evaluations miss the sustained multi-turn exchanges expected across inquiries and device control.Source: §1 · p. 2
  • The study proposes a GQM-grounded framework of 23 metrics in five categories, validates LLM-as-judge scoring against human coders, and compares four HEMS configurations.The comparison is intended to establish discriminative validity across scenarios and personas.Source: §1 · p. 2

2 Related Work

  • Prior HEMS studies leave multi-turn dialogue, interaction quality, factual grounding, efficiency, and cross-domain coordination insufficiently assessed.Most use researcher-designed single-turn queries and narrow system-oriented metrics.Source: §2 · pp. 3–4

3 Evaluation Framework

The framework uses Goal-Question-Metric methodology to derive domain-specific measures of conversational HEMS performance. It combines broad evaluation categories with finer-grained metrics that diagnose response appropriateness and control behavior.

3 Evaluation Framework

  • GQM derives metrics through goals, operational questions, and quantitative measures, with goals informed by HEMS capabilities, occupant-interaction evidence, and identified evaluation gaps.Its top-down traceability links each metric to an explicit goal.Source: §3 · p. 4
  • The 23 metrics span task performance, factual accuracy, interaction quality, control capability, and system efficiency, with interaction quality divided into four subdimensions.These are defined to assess conversations and, where applicable, device-control actions.Source: §3 · p. 5, Table 2 · p. 5
  • Personalization calibration distinguishes appropriate data-backed answers from general explanations, identifying under-personalization and over-personalization as mismatches.The matrix measures whether personalization fits the question type.Source: §3 · p. 6, Table 3 · p. 7
  • GQM traceability, modular application, and diagnostic granularity support targeted assessment; control metrics apply when device control is present, while efficiency metrics apply universally.The framework separates personalization errors and decomposes action transparency into independent behaviors.Source: §3 · pp. 6–7

4 Framework Validation

The evaluation compares four HEMS configurations using simulated-user multi-turn dialogues across scenarios and personas. It first validates the LLM-as-judge pipeline against human ratings, then applies the calibrated pipeline across all configurations.

4.1 Experimental Design

  • The LLM-as-user approach supports large-scale system diagnostics, but the authors explicitly distinguish it from human user studies.The dialogue set was manually checked for flow, with degenerate conversations regenerated.Source: §4.1 · pp. 8, 10
  • Four configurations form a sophistication gradient from a vanilla LLM using raw energy data to structured reports, chain-of-thought guidance, and the multi-agent HEMA architecture.Structured-report baselines received precomputed analyses also available to HEMA for a fair comparison.Source: §4.1 · pp. 8, 10
  • The study used two weeks of data from one feature-rich Austin household and generated multi-turn dialogues with an LLM-simulated user across scenarios and personas.The study describes the single-household choice as an evaluation of system capabilities rather than household-data effects.Source: §4.1 · pp. 7–9

4.2 System Performance Evaluation

  • Eight metrics were computed from logs, while 15 judgment-based metrics were scored by an LLM pipeline whose reliability was checked with three trained human coders before cross-configuration scoring.After rubric calibration, the pipeline used structured prompts and three-run median voting for selected extracted categories across 970 conversations.Source: §4.2 · pp. 10–11

5 Results

The results validate the LLM-as-judge pipeline for most evaluated metrics and show that the four HEMS configurations differ across task performance, factual accuracy, personalization, efficiency, and device control.

5.1 Human Validation Results

  • Twelve of 15 LLM-scored metrics reached ICC(2,1) ≥0.73, while restricted-range metrics had MAEs of 0.041–0.281.Three metrics achieved perfect agreement; three-run median voting raised answered-question ratio agreement from ICC 0.593 to 0.804.Source: §5.1 · pp. 12–13

5.2 Analytical and Informative Capabilities

  • 99.6% (HEMA) versus 93.3% (Vanilla+Raw) goal achievement; only this pair differed significantly after correction.HEMA also reached 100% on utility-rate details by retrieving tiered pricing, while baseline systems ranged from 64% to 80%.Source: §5.2 · p. 13, Figure 2 · p. 14
  • Vanilla+Raw factual accuracy was 56.4%, versus at least 95.8% for structured-report or analysis-tool systems; mean errors were below 1% for the latter.HEMA produced 7.19 factual claims per conversation, 52% more than Vanilla+Energy Report, while maintaining comparable accuracy.Source: §5.2 · pp. 14–15, Table 7 · p. 14
  • HEMA had the lowest over-personalization (0.08 ± 0.29) and highest appropriate general-response count (1.69 ± 1.20), while under-personalization remained low across configurations.Vanilla+Energy Report had the highest over-personalization (0.64 ± 0.65), concentrated in conceptual scenarios.Source: §5.2 · pp. 15–16, Figure 4 · p. 16
  • Vanilla+Raw used approximately 258,000 tokens per conversation, compared with 4,500–6,700 for report baselines and approximately 37,700 for HEMA.HEMA averaged 14.10-second latency (±3.48); Vanilla+Energy Report+CoT was fastest on average, while HEMA and CoT produced more concise responses than Vanilla+Raw.Source: §5.2 · pp. 16–17, Table 9 · p. 17

5.3 Smart Device Control Analysis

  • Across seven control-focused scenarios, goal success was 97.1%; target-device accuracy was 94.2% ± 12.7%, and mode correctness and constraint compliance were 100%.Control metrics were computed only for HEMA; response latency across control scenarios averaged 12.1 seconds, with token use ranging from 17,687 ± 1,660 in demand response to 121,672 ± 102,974 in device status checks.Source: §5.3 · pp. 17–19
  • Schedule correctness was 88.0% ± 32.5% across 25 load-shifting runs, while HEMA completed the full demand-response cycle in nine of 10 runs.Pool-pump scheduling was 100% correct; EV charging was 77.8% ± 41.6%, and water-heater optimization was 88.9% ± 31.4%.Source: §5.3 · pp. 17–18, Figure 5 · p. 18
  • Action confirmation (84.3% ± 36.7%) and explanation (83.6% ± 36.8%) rates exceeded information-before-action (41.6% ± 46.6%) across control scenarios.Information-before-action was especially low for demand response (5.0% ± 15.8%) and device status checks (10.0% ± 31.6%).Source: §5.3 · p. 18, Figure 6 · p. 19

6 Discussion and Limitations

The discussion shows that single-metric evaluation can miss important differences in personalization and control, while multi-turn assessment reveals sustained performance and persona–scenario effects.

Necessity of Multi-Dimensional Evaluation of Conversational HEMS (RQ#1)

  • Vanilla+Energy Report produced 0.64 ± 0.65 over-personalized responses per conversation, versus 0.08 ± 0.29 for HEMA, a distinction accuracy-only evaluation would miss.The passage attributes this failure mode to architectural routing decisions rather than data access.Source: §6 · p. 19
  • Answered question ratio was statistically flat across configurations (71.7–75.3%, p=0.530), so it did not reveal substantial personalization-calibration differences.Surface-level question coverage remained similar despite differences detected by the broader framework.Source: §6 · pp. 19–20
  • Multi-turn evaluation exposed sustained goal achievement, per-turn productivity, and persona–scenario effects, including a challenging confused-newcomer/rate-details pairing.HEMA had higher completion rates within a comparable conversational footprint of 6.62 turns.Source: §6 · p. 20
  • Control metrics exposed residual vulnerabilities: schedule correctness was 88.0%±32.5% and target device accuracy was 94.2% ± 12.7%, despite 100% mode correctness and constraint compliance.Demand response requires a distinct criterion because off-peak placement is a failure signal in that setting.Source: §6 · p. 20

Scalability of the LLM-as-Judge Approach (RQ#2)

The LLM-as-user approach supported large-scale dialogue generation, while human validation found that judge reliability varied by metric and improved after rubric calibration.

Scalability of the LLM-as-Judge Approach (RQ#2)

  • The LLM-as-user approach enabled 970 conversations with controlled variation, a scale described as infeasible with human subjects.Different LLMs mitigated implicit alignment risk, while structured persona traits and goal-completion criteria addressed known reliability concerns [45].Source: Scalability of the LLM-as-Judge… · p. 20
  • Twelve of fifteen LLM-scored metrics achieved ICC(2,1) ≥0.73, while three restricted-range metrics were validated via low MAE.The passage does not provide the MAE values for those metrics.Source: Scalability of the LLM-as-Judge… · p. 20
  • Iterative rubric calibration resolved systematic LLM inflation in personalization metrics caused by a counting-granularity mismatch.The reported calibration protocol used independent coding, structured reconciliation, and rubric refinement.Source: Scalability of the LLM-as-Judge… · p. 20
  • The authors recommend automated scoring for most metrics, with human calibration for metrics involving subjective classification.They describe the iterative validation protocol as replicable for domain-specific conversational AI pipelines.Source: Scalability of the LLM-as-Judge… · p. 20

Distinguishing Conversational HEMS Performance (RQ#3)

HEMA was more robust across query types than non-agentic configurations, but used substantially more tokens and had higher latency; its advantages were most relevant to complex interactions.

Distinguishing Conversational HEMS Performance (RQ#3)

  • HEMA maintained 80%+ goal achievement across nearly all persona-scenario combinations, while non-agentic configurations ranged from 0% to 100%.For specific utility rate details, all three non-agentic configurations failed while HEMA achieved 100%.Source: Distinguishing Conversational HEMS… · p. 20
  • HEMA consumed approximately 37,700 tokens per conversation, 5.6 to 8.4 times more than report-based configurations, and had higher response latency than Vanilla+Energy Report+CoT.Its factual accuracy rate was 95.8%, compared with 98.4% for Vanilla+Energy Report+CoT; the passage attributes this difference to a denominator effect as HEMA generated 52% more factual claims per conversation (7.19 vs. 4.73).Source: Distinguishing Conversational HEMS… · pp. 20–21
  • Non-agentic configurations performed comparably at lower cost on well-scoped, single-domain tasks, while the agentic architecture’s value emerged for complex, multi-faceted, or knowledge-requiring interactions.HEMA's factual accuracy percentage was marginally lower, but it generated more factual claims per conversation and had a higher absolute accurate claim count.Source: Distinguishing Conversational HEMS… · pp. 20–21
  • The self-consistency classifier achieved 99.0% goal achievement, while ReAct grounded numerical claims in tool outputs and agent specialization improved personalization judgment.Independent agent operation also led to redundant tool calls in cross-domain scenarios; the authors suggest inter-agent communication as future work [16].Source: Distinguishing Conversational HEMS… · p. 21

Practical Considerations for Deployment

Deployment considerations include remote API latency and errors, as well as variation in latency, cost, and reasoning capability across LLM choices.

Practical Considerations for Deployment

Limitations

The evaluation is limited by simulated users and a single-household setting, as well as independent-agent and isolated-session designs and outcomes outside the framework’s scope.

  • LLM-simulated occupants may make more cooperative, less ambiguous queries than real users, potentially making system performance an optimistic upper bound.The five personas and 16 scenarios were informed by empirical occupant interactions; the authors call for replication with residential participants to test ranking stability.Source: Limitations · p. 21
  • The study uses one Austin household for controlled ground-truth definition, so it does not generalize performance across the broader U.S. household population.Replication across utility structures, climates, and device configurations is a priority.Source: Limitations · pp. 21–22
  • Agents operate independently, which can cause redundant computation in cross-domain scenarios but enables errors to be attributed to specific agents.Source: Limitations · p. 22
  • Isolated sessions leave sustained collaboration as preferences evolve untested, and the framework does not assess actual energy savings, long-term satisfaction, or scheduling optimality.The authors identify longitudinal evaluation and extensions to those outcomes as future directions.Source: Limitations · p. 22

7 Conclusion

The study presents a 23-metric, GQM-grounded framework and validates an LLM-as-judge pipeline for evaluating conversational HEMS. Its comparisons distinguish system configurations, with interaction-quality measures—particularly over-personalization—more discriminating than factual accuracy for agentic versus non-agentic architectures.

7 Conclusion

  • The evaluation compared four configurations using 970 multi-turn conversations across five personas and 16 scenarios; future work will test multiple households and real residential users.The scenarios and personas were generated using an LLM-simulated-user methodology.Source: §7 · p. 22
  • Interaction-quality metrics, particularly over-personalization, were more discriminating than factual accuracy for differentiating agentic from non-agentic configurations.Source: §7 · p. 22
  • The GQM-grounded framework comprises 23 metrics across task performance, factual accuracy, interaction quality, control capability, and system efficiency.Its diagnostic granularity distinguishes over- from under-personalization and schedule misalignment from constraint violation.Source: §7 · p. 22
  • Twelve of fifteen LLM-scored metrics achieved ICC ≥ 0.73 against human coders, with three reaching perfect agreement.Source: §7 · p. 22

Declaration of generative AI and AI-assisted technologies in the writing process

The author reports using AI tools to proofread sentences and taking responsibility for the publication after reviewing and editing the content.

A Home Energy Management Assistant (HEMA)

HEMA uses a multi-agent architecture with specialized Analysis, Knowledge, and Control agents coordinated by a self-consistency classifier. Agents use ReAct tool calling, with deterministic state-graph routing implemented in LangGraph.

A Home Energy Management Assistant (HEMA)

Loading 2610.00073v1…