PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Anomaly Detection with Large Language Models

Amine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa Nesnas, Marco Pavone

arXiv:2305.11307v2cs.RO

TL;DR

Robots can fail when individually ordinary observations combine in semantically misleading ways, beyond what component-level checks address. This paper uses LLMs to monitor vision-based policies for such anomalies; experiments report agreement with human reasoning across autonomous-driving and manipulation settings. The approach’s prompts require manual, system-specific design and may reflect designers’ biases or oversights.

  • Problem

    System-level failures can arise from semantic context that component-level monitoring may not capture, while determining whether a scene suits a downstream task remains unresolved.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    The paper introduces an LLM monitor for vision-based policies that uses textual scene descriptions to reason about potential semantic anomalies.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    The LLM-based monitor matches human intuition in experiments on end-to-end policies and classical autonomy stacks with learned perception.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The experiments indicate potential for LLMs to identify semantic anomalies involving subtle distinctions that classical OOD methods are not equipped to handle.

    Source: §5 · p. 12

    The approach requires manual prompt engineering for each system and is subject to designers’ biases and oversights.

    Source: §5.2.2 · p. 15

Abstract

from arXiv · show

As robots acquire increasingly sophisticated skills and see increasingly complex and varied environments, the threat of an edge case or anomalous failure is ever present. For example, Tesla cars have seen interesting failure modes ranging from autopilot disengagements due to inactive traffic lights carried by trucks to phantom braking caused by images of stop signs on roadside billboards. These system-level failures are not due to failures of any individual component of the autonomy stack but rather system-level deficiencies in semantic reasoning. Such edge cases, which we call semantic anomalies, are simple for a human to disentangle yet require insightful reasoning. To this end, we study the application of large language models (LLMs), endowed with broad contextual understanding and reasoning capabilities, to recognize such edge cases and introduce a monitoring framework for semantic anomaly detection in vision-based policies. Our experiments apply this framework to a finite state machine policy for autonomous driving and a learned policy for object manipulation. These experiments demonstrate that the LLM-based monitor can effectively identify semantic anomalies in a manner that shows agreement with human reasoning. Finally, we provide an extended discussion on the strengths and weaknesses of this approach and motivate a research outlook on how we can further use foundation models for semantic anomaly detection.

1 Introduction

Rare, system-level failures can arise when individually ordinary observations combine in semantically tricky ways, which component-level monitoring may miss. The paper proposes LLM-based monitoring for vision-based policies and evaluates it in autonomous driving and object manipulation.

1 Introduction

  • Component-level OOD monitoring can miss system-level faults because it checks individual components rather than the autonomy stack’s contextual reasoning.The paper argues that the relation between objects and their surroundings can make otherwise correct detections misleading.Source: §1 · p. 2
  • Semantic anomalies are system-level OOD inputs formed by unusual combinations of individually in-distribution observations, where object context can trigger failures.Examples include inactive traffic lights carried by a truck and stop signs on roadside billboards.Source: §1 · p. 2
  • The proposed monitor uses an LLM to reason about whether observed objects could cause task-relevant confusion and policy errors in vision-based policies.The paper evaluates the framework with an autonomous-driving state-machine policy and an end-to-end learned object-manipulation policy.Source: Figure 2 · p. 2, §1 · p. 3
  • Experiments report effective anomaly recognition with characterizations that generally agree with human reasoning, followed by discussion of the approach’s strengths and failure modes.The paper uses a CARLA reconstruction of the inactive-traffic-light scenario to demonstrate contextual reasoning for semantic anomaly detection.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

The related-work section reviews anomaly detection, semantic reasoning, and robotics applications of LLMs. It contrasts component-focused OOD approaches with the paper’s use of LLMs to reason about task-relevant scene context.

2.1 Out-of-Distribution Detection

  • OOD methods detect novel or dissimilar inputs using classifiers, distribution or uncertainty estimates, reconstruction errors, and comparisons across models or sensors.Some approaches also use supervised success/failure examples or domain-specific supervisory signals.Source: §2.1 · p. 3
  • Existing OOD monitors focus on component inference errors or uncertainty, while the proposed LLM approach reasons over possible errors in interpreting a scene for a task.The paper describes its approach as zero-shot and applicable without training data or task-specific component redesign.Source: §2.1 · pp. 3–4

2.2 Large Language Models in Robotics

  • Prior robotics work integrates LLMs into subsystems such as navigation, planning, and policy synthesis; this paper instead uses an LLM to monitor an independent robot’s observations.The paper situates its work alongside zero-shot reasoning and vision-language models, noting that scene suitability for a downstream task remains unresolved.Source: §2.2 · p. 4

3 LLM-Based Semantic Anomaly Detection

The framework converts a robot’s visual observations into scene descriptions and prompts an LLM to identify factors that may cause policy errors or task confusion. Its visual-to-text stage is flexible, and its prompts use few-shot examples and chain-of-thought reasoning.

3 LLM-Based Semantic Anomaly Detection

  • At sampling times, the monitor turns visual observations into text and prompts an LLM to identify potential causes of policy error, unsafe behavior, or task confusion.Its task and system specificity depends on the prompt; the monitor is not explicitly grounded in system capabilities or internal processes.Source: §3 · pp. 4–5
  • The visual-to-text stage is approach-agnostic, and this work uses an open-vocabulary object detector to describe scene objects and their context.The authors note that conversion could range from classical image processing to vision-language models, depending on task requirements.Source: §3 · p. 5
  • Few-shot examples prime the LLM with common pitfalls, while chain-of-thought prompting adds task- and system-specific structure to its reasoning.The paper uses both prompting strategies in its framework instantiations.Source: §3 · p. 5

4 Experiments

The experiments evaluate an LLM-based semantic-anomaly monitor in autonomous driving and manipulation, finding strong agreement with expected reasoning in driving scenarios but limited versatility with erratic learned policies. In CARLA, the monitor outperforms two OOD baselines on semantic-anomaly detection, while the approaches show complementary strengths on perception errors.

4 Experiments

  • In manipulation experiments, LLM anomaly characterizations tend to align with human expectations, but erratic or inexplicable learned-policy failures limit the approach’s versatility.The manipulation setting uses an end-to-end learned visuomotor policy.Source: §4 · p. 5

4.1.1 Experimental Setup

  • The CARLA driving evaluation uses five scenario classes, including nominal interactions and anomalies involving sign imagery, a truck-carried traffic light, or strange objects.The scenarios use a finite-state-machine planner and handcrafted cases across public CARLA maps.Source: §4.1.1 · pp. 5–6

4.1.2 Anomaly Detection Methodology

  • At 2 Hz, OWL-ViT converts observations into scene descriptions that a prompt uses to elicit chain-of-thought anomaly reasoning from text-davinci-003.The method is intended to flag scene factors that could cause erroneous, unsafe, or unexpected behavior.Source: §4.1.2 · p. 6, Figure 3 · p. 7

4.1.3 Results

  • The monitor performs strongly on anomalous stop-sign and traffic-light cases and nominal stop-sign cases, but has elevated false positives on nominal traffic-light episodes.The reported results cover four scenario types and their detection rates.Source: §4.1.3 · p. 7, Table 1 · p. 8
  • CARLA visual differences from OWL-ViT’s real-world training images were associated with detector misclassifications and hallucinated objects, including false billboard predicates.The authors hypothesize that background lighting effects create the illusion of a flat background; Table 2 reports results using ground-truth scene labels.Source: §4.1.3 · pp. 7–8, Table 2 · p. 8
  • The LLM monitor significantly outperforms SCOD and Mahalanobis distance on CARLA semantic-anomaly detection, with baseline thresholds set for a 5% false positive rate.The comparison is reported in Table 3.Source: §4.1.3 · p. 8, Table 3 · p. 9
  • For perception errors, the LLM monitor is unable to detect them, whereas SCOD can; Mahalanobis shows some capacity but performs worse than SCOD.The approaches are complementary: SCOD and Mahalanobis also detect some semantic anomalies.Source: §4.1.3 · p. 9, Table 4 · p. 9

4.2 Learned Policy

In CLIPort manipulation, distractors—especially semantic ones—degraded policy performance. LLM classifications aligned with human intuition, but failure prediction was limited by unexplained policy errors.

4.2.1 Experimental Setup

  • The experiment tested a pick-and-place task across baseline, neutral-distractor, and visually similar semantic-distractor variants, using 250 simulated episodes per variant.Neutral objects were visually distinct from task objects; semantic distractors resembled the blocks or bowls.Source: §4.2.1 · p. 10

4.2.2 Anomaly Detection Methodology

  • For each distractor episode, the LLM received the task specification and a scene description, then reasoned about visual similarities that could confuse the policy.Scene descriptions used privileged simulator information because pretrained detectors were not suitable for these visuals.Source: §4.2.2 · p. 10
  • The autoencoder baseline used L2 reconstruction error on CLIPort training-task images and flagged scenes exceeding a threshold set from successful training examples.The threshold was the 95% quantile of reconstruction losses on training examples where CLIPort succeeded.Source: §4.2.2 · pp. 10–11

4.2.3 Results

  • Distractors reduced CLIPort task success and mean reward, with semantic distractors producing the largest average performance drop.Mean reward grants 0.5 for each block placed in a bowl.Source: Figure 6 · p. 11, §4.2.3 · p. 11
  • The policy often misinterpreted semantic distractors as task-relevant objects, while erratic failures also occurred across all variants and increased with neutral distractors.One example was mistaking a purple-headed spatula for a purple block.Source: §4.2.3 · p. 11
  • The LLM flagged most semantic distractors and ignored most neutral ones, resembling human classifications; ambiguous color descriptions caused disagreements.For example, it missed a light brown fedora near an orange bowl because it treated the colors as different.Source: §4.2.3 · p. 11, Table 5 · p. 12
  • The autoencoder flagged more neutral distractors than the LLM and exceeded random guessing by only a small margin on semantic distractors.Source: §4.2.3 · p. 11, Table 5 · p. 12
  • LLM anomaly detections corresponded to many policy failures with semantic distractors but also produced many false positives; neutral-only failure prediction remained difficult.The authors report that inexplicable policy failures confounded failure prediction.Source: §4.2.3 · p. 12

5 Discussion

The discussion illustrates where the monitor succeeds and errs, then considers how richer inputs, system-specific grounding, and complementary monitoring could address its limitations.

5 Discussion

  • The discussion presents LLM monitoring as promising for contextual anomaly detection, while noting that manipulation experiments expose subtle distinctions classical OOD methods are not equipped to handle.These are described as illustrative strengths and limitations, not as a comprehensive evaluation of every failure mode.Source: §5 · p. 12

5.1 Qualitative Analysis

  • In the driving examples, the monitor identifies inactive traffic lights on trucks and stop signs on billboards as anomalous, while classifying ordinary intersections as normal.The authors also report a false alert for a roadside building and a missed billboard stop sign; changing scene-element order inverted the classifications in these two cases.Source: §5.1 · pp. 12–13

5.2.1 Multimodal Context

  • Ambiguous or underspecified natural-language scene descriptions can mislead the monitor, as with “house on the sidewalk,” which it interprets as a road obstruction.The authors suggest that incorporating images alongside text could preserve context and reduce information loss from visual parsing.Source: §5.2.1 · p. 13

5.2.2 Grounding System Capabilities

  • Few-shot and chain-of-thought prompts convey system capabilities only indirectly and incompletely, and require manual, system-specific engineering that may miss updates or reflect designers’ biases.The authors suggest system-specific fine-tuning or tighter integration with the autonomy stack as alternatives for consideration.Source: §5.2.2 · pp. 14–15

5.2.3 Complementary Anomaly Detection and Mitigation

  • The authors recommend studying how LLM monitoring complements other OOD methods, since data-driven systems can fail unexpectedly or inexplicably in the CLIPort experiments.Fault mitigation and recovery are outside this work’s scope, though structured LLM queries could potentially support recovery or requests for supervisor assistance.Source: §5.2.3 · p. 15

6 Conclusion

Semantic anomalies are challenging corner cases for robotic systems. The paper uses LLM reasoning to monitor them and reports agreement with human intuition across two policy settings, while identifying directions for further development.

6 Conclusion

  • The study targets semantic anomalies, a difficult subset of corner cases that may degrade performance or pose safety hazards and require human-like insight and reasoning.The authors argue that LLMs’ extensive training data and contextual knowledge make them compelling tools for detecting these anomalies.Source: §6 · p. 15
  • The LLM-based monitor matches human intuition in experiments involving both end-to-end policies and classical autonomy stacks with learned perception.The authors also note that semantic anomalies do not always correspond to semantically explainable failures, particularly for end-to-end policies that can behave erratically.Source: §6 · p. 15
  • The authors identify tighter coupling of robot perception to prompts, grounding outputs in system capabilities, and testing complementarity with other OOD monitors as future directions.Source: §6 · p. 15

Declarations

The declarations describe LLM use, funding, consent, availability statements, and author contributions.

6.1 LLM Usage Statement

  • OpenAI’s text-davinci-003 was used for all experiments and evaluation; LLMs were not used to compose the manuscript.Source: §6.1 · p. 15

6.2 Funding

  • NASA’s University Leadership initiative and Toyota Research Institute funded the work, and Amine Elhafsi received support from a NASA NSTGRO fellowship.The article states that its opinions and conclusions are solely those of its authors.Source: §6.2 · p. 15

6.6 Consent for Publication

  • The authors endorsed the content, consented to submission, and ensured consent from the responsible authorities at their institutions or organizations.Source: §6.6 · p. 16

6.7 Availability of Data and Materials

  • Relevant documentation, data, and/or code are available upon request to verify the reported results.Source: §6.7 · p. 16

6.8 Code Availability

  • The authors state that they plan to open-source their code.Source: §6.8 · p. 16

6.9 Author Contributions

  • Amine Elhafsi led methodology, prompt tuning, implementation, and experiments, while the other authors contributed to the driving stack, baselines, analysis, advice, and manuscript writing.All authors reviewed and revised the manuscript.Source: §6.9 · p. 16

Appendix A Additional Details: Reasoning-Based Policy

The appendix gives a template for prompting an autonomous-vehicle monitor to assess observed objects and classify a scene. Its example treats a pedestrian carrying a bright green balloon as anomalous because the balloon could be mistaken for a traffic signal.

Appendix A Additional Details: Reasoning-Based Policy

  • The template places a scene description into a prompt that asks the LLM to act as a vision-based vehicle fault monitor and assess whether observations could lead to unsafe, unpredictable, or rule-violating actions.It asks the monitor to reason about each observed object and then classify the overall scene as normal or abnormal.Source: Appendix A · pp. 16–17
  • In the example, the monitor classifies a cyclist on the sidewalk and a car on the road as normal observations.It treats the pedestrian carrying a bright green balloon as anomalous because the balloon could be mistaken for a green traffic signal and prompt the vehicle to proceed when it should stop.Source: Appendix A · pp. 16–17
  • The example concludes with an overall scene classification of anomaly after reasoning that the balloon could be interpreted as a legal traffic signal.Source: Appendix A · p. 17

B.1 Prompt Template

The prompt casts the LLM as a fault monitor for a manipulator that puts colored blocks into matching bowls, asking it to flag visible objects that could be mistaken for a target. [sic]

B.1 Prompt Template

  • For each scene object, the prompt requests an appearance description and an assessment of shape and color similarity to the target block and bowl.Source: §B.1 · p. 18
  • The prompt asks whether each object could be misidentified as a block or bowl and whether any object could cause the robot to make a mistake.Source: §B.1 · p. 18
  • The authors use zero-shot prompting because few-shot examples tended to overfit and impair monitoring across diverse household distractors.They report that zero-shot prompting encouraged the LLM to use its inherent knowledge of common objects.Source: §B.1 · p. 18

B.2 Semantic and Neutral Distractors

Table B1 catalogs distractor objects and their experimental uses, including semantic distractors that also serve as neutral distractors when target colors differ.

B.2 Semantic and Neutral Distractors

  • Table B1 pairs distractor images with their experimental uses and notes that semantic distractors also serve as neutral distractors when target colors differ.Source: Table B1 · p. 19
  • The C-Clamp is a neutral distractor, while the Nintendo Cartridge is a semantic distractor for the gray block.Source: §B.2 · pp. 19–20
  • The Red Cup is a semantic distractor for the red bowl.Source: §B.2 · p. 21

Related papers

Loading 2305.11307v2…