This collaborative work between NVIDIA Research and Stanford University was awarded the Outstanding Paper Award at the 2024 Robotics: Science and Systems conference. __________________________________ Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot generalization capabilities that make them a promising technology towards detecting and mitigating out-of-distribution failure modes of robotic systems. In this work, we present a two-stage reasoning framework: 1: A fast binary anomaly classifier analyzes observations in an LLM embedding space, which may then trigger a slower fallback selection stage. 2: The fall back selection stage utilizes the reasoning capabilities of generative LLMs. We show that our fast anomaly classifier outperforms autoregressive reasoning with state-of-the-art GPT models, even when instantiated with relatively small language models. This enables our runtime monitor to improve the trustworthiness of dynami...