Source-linked AI summary
Resilience Matters for Embodied Agents System: New Metrics, Systematic Evaluation, and Optimization
Yapeng Liu, Yuanzhao Zhai, Xudong Gong, Dawei Feng, Bo Ding, Lin Wang, Huaimin Wang
TL;DR
EAS evaluations often reduce execution to coarse outcomes, leaving recovery, stability, and stress adaptation insufficiently measured in dynamic physical environments. This paper introduces a resilience framework with process-level metrics and an evaluation layer, then benchmarks and optimizes EAS methods across household tasks. The results distinguish methods with similar success rates, reveal multidimensional trade-offs, and support metrics-guided resilience improvements.
Problem
Outcome-centric EAS evaluations collapse diverse execution trajectories into coarse scores and do not measure how systems recover, stabilize, and extend under stress.
Method
The paper defines Rebound, Stability, and Graceful Extensibility metrics and implements a non-intrusive evaluation layer using execution signals, stage baselines, controlled stress, and LLM judgment.
Results
Across 400 household tasks and 10 EAS methods, resilience evaluation reveals process-level disparities, distinct resilience signatures, and trade-offs that outcome metrics miss.
Takeaways & Limitations
Resilience should be treated as a multidimensional execution profile and configured according to deployment-specific requirements rather than reduced to a single ranking.
Takeaways & Limitations
Future work must validate whether simulation-derived resilience profiles transfer to physical robots and further analyze trade-offs among resilience aspects.
Abstract
from arXiv · showhide
Embodied Agents System (EAS) are increasingly deployed in open-world physical domains, where reliability directly dictates deployment quality and human-agent trust. However, existing evaluations rely on outcome-centric metrics as success rate or safety scores that collapse diverse execution trajectories into coarse scores, obscuring the dynamic processes underlying agent behavior. Therefore, they ignore a critical property of EAS -- which we define as the Resilience -- that reflects how EASs recover, stabilize, and extend under perturbations and across iterative updates. The lack of resilience is particularly critical in open-world environments due to continuous unexpected disruptions, thus directly affecting the quality of EAS deployment. To address this problem, we gain insight from the resilience-engineering concepts to EAS groundings and propose a novel resilience evaluation framework that can be flexibly applied to any EAS. Specifically, we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility across embodied tasks execution, providing a practical grounding for EAS resilience analysis. We further implement the resilience evaluation layer that transforms execution process into assessments for diagnosis and optimization. Across 400 household tasks with 10 EAS, we reveal the process-level distinction hidden by outcome metrics, including recovery cost differences among successful episodes ($ΔC_{rec}=25.2$), increased instability and task-family degradation. Metrics-guided optimizations reduce recovery cost and increase stability, graceful extensibility completion, showing the diagnostic effect of resilience evaluation. Our results reveal a trade-off among resilience characteristics, suggesting that a resilient EAS construction should be configured according to deployment-specific requirements.
1 Introduction
The paper identifies Resilience as a missing EAS property describing recovery, stabilization, and extension under stress, then proposes metrics and an evaluation layer to expose process-level differences beyond outcomes.
- Motivation: Outcome metrics can rate two EASs equally despite fundamentally different execution processes under household instructions.One trajectory handles stress through resilient mechanisms, while another reaches success through repeated backtracking, replanning, and unsafe behaviors.
- Resilience concept: Resilience describes how EASs recover, stabilize, and extend under stress.The framework grounds this property in embodied execution rather than outcome scores alone.
- Framework: The proposed metrics expose Rebound, Stability, and Graceful Extensibility throughout task execution.They include recovery cost Crec, stability sensitivity β, stress response mapping Mf(λ), and stress capacity λ∗.
- Framework: The Resilience Evaluation Layer extracts trajectory, monitor, and LLM-judge signals, aggregates them through stage baselines and controlled stress intensities, and supports diagnosis and optimization.It is designed to enable non-intrusive comparison, diagnosis, and optimization guidance.
- Empirical findings: 400 household tasks across 10 EAS methods reveal process-level disparities that outcome metrics do not capture.Physical perturbations increase recovery cost from 11.6 to 36.3, while semantic perturbations raise instability to ∆β = 0.32.
- Empirical findings: The study reports distinct resilience signatures and practical optimization effects, while treating resilience trade-offs as a design principle.The contributions include benchmarking brittle and resilient behaviors and using resilience-guided diagnosis to improve EAS resilience.
2 Related Works
Prior EAS evaluation emphasizes outcomes or perturbation safety checks, while resilience engineering motivates process-level analysis of endurance, adaptation, recovery, and graceful degradation during execution.
- Outcome-centric evaluation: Recent EAS evaluation remains largely outcome-centric, limiting distinction between robust adaptation and brittle success.Outcome measures such as success metrics compress behavior in settings with physical irreversibility, partial observability, and dynamic constraints.
- Reliability diagnostics: Perturbation-based reliability studies mainly assess constraint violations or hazardous states rather than normal process-level execution.The paper positions resilience metrics as complementary diagnostics for behavior under inner stress and perturbations.
- Resilience engineering: Resilience engineering frames resilience as the capability to endure, adapt to, and recover from disruptions.Woods decomposes it into rebound, robustness, graceful extensibility, and sustained adaptability.
- EAS grounding: The paper grounds these concepts in three EAS aspects: Rebound, Stability, and Graceful Extensibility.Rebound captures closed-loop recovery burden, Stability captures decision consistency, and Graceful Extensibility captures bounded degradation and operational margin under stress.
3 The Proposed EAS Resilience Evaluation Framework
The framework maps execution dynamics and local stage baselines into three resilience aspects—Rebound, Stability, and Graceful Extensibility—using trajectory, runtime, monitor, and judge signals. Evaluation and benchmarking reveal process-level differences, multidimensional trade-offs, and resilience profiles that outcome metrics alone do not capture.
- Framework Overview: The evaluation layer maps episode artifacts and execution dynamics into Rebound, Stability, and Graceful Extensibility assessments.Episode artifacts combine trajectories, runtime logs, monitors, and LLM-judge signals; local stage baselines provide common references for metric computation.
- 3.1 Metrics Design: Recovery Cost measures extra work needed to return to acceptable execution, decomposed into cognitive recovery, physical recovery, and state debt.The components are converted into covariance-adjusted deviations to account for scale differences and correlations within recovery channels.
- 3.1 Metrics Design: Stability β captures execution sensitivity, with higher values indicating more replanning, value fluctuation, and progress instability under similar conditions.The metric is computed over controlled perturbations and task-family episode sets.
- 3.1 Metrics Design: Graceful Extensibility models performance as stress severity increases, measuring whether degradation remains bounded and predictable rather than collapsing catastrophically.Its quantitative indicator is the largest stress level at which acceptable execution remains achievable.
- 3.2 Evaluation and Benchmarking: Resilience metrics distinguish executions with similar outcomes and expose complementary properties not reducible to success rate, safety, or completion alone.Across benchmarked methods, resilience profiles reveal different recovery, stability, and stress-response patterns.
- 3.2 Evaluation and Benchmarking: No current method dominates every resilience aspect: CoPAL favors rebound and stress capacity, InnerMono favors stability, and AgentEvolver has relatively low recovery cost but weak stress capacity.Reported values include CoPAL Crec = 11.9 and λ∗ = 0.85, InnerMono β = 0.149, and AgentEvolver Crec = 19.2 and λ∗ = 0.21.
4 EAS Optimization: Metric-Guided Resilience Diagnostics
Metric-guided resilience optimization uses resilience profiles to minimally repair EAS execution and compare original with optimized variants. Targeted improvements can improve resilience while creating trade-offs among resilience aspects.
- Optimization framework: Metric-guided optimization uses resilience metrics as diagnostic signals to minimally repair the execution loop rather than replace the original planner.Three targeted optimizations address Rebound, Stability, and Graceful Extensibility under identical evaluation conditions.
- Evaluation scope: Formal GE is reported only for GE stress-tests because it requires a stress-response curve over the λ grid.
- Trade-offs: Targeted optimization of one resilience aspect can compromise another, with Graceful Extensibility optimization associated with high recovery cost Crec.This supports treating resilience as a multidimensional structural balance.
- Rebound optimization: Feedback-guided recovery monitors world-state evidence, execution feedback, and stagnation to diagnose recoverable faults after local failures.The intervention targets high recovery cost and long recovery windows.
- Stability optimization: Consistency-state recording addresses unstable replanning that can oscillate, repeat completed subtasks, or drift from the current execution context.
- Graceful Extensibility optimization: Boundary long-tail control targets Graceful Extensibility failures in which repeated waiting or near-duplicate feedback inflates execution cost and obscures adaptive capacity.
5 Conclusion and Future Work
The paper presents a resilience evaluation framework for EASs and shows that resilience metrics expose pathologies and trade-offs beyond outcome metrics. Future work focuses on validating transfer to physical robots, analyzing trade-offs theoretically, and adapting optimization to deployment conditions.
- Conclusion: The framework introduces Rebound, Stability, and Graceful Extensibility to expose high-cost recovery and catastrophic forgetting hidden by outcome-centric metrics.
- Conclusion: Empirical analysis identifies distinct Resilience Archetypes and quantifiable trade-offs among execution, stability, and adaptability across representative baselines.
- Conclusion: The optimization framework supports root-cause diagnosis and non-degradation contracts for building resilient and trustworthy EASs.
- Future Work: Future work includes Sim2Real validation, theoretical analysis of resilience trade-offs, and deployment-aware optimization for target application conditions.