CHRIS-Bot: A Robot for Dialogue and Scene Understanding of Anomalous Environments in Virtual Reality

Author(s):

Stephanie M. Lukin, Jaci South, and Shawn Bowser

Abstract:

Robots can play a critical role in supporting human teammates; however, there are many challenges to ensuring effective collaboration under unknown or anomalous conditions. Natural language is a useful method for allowing humans to issue instructions at a high level. However, we further enhance the human-robot dialogue paradigm by increasing the robot’s ability to provide common ground for the conversation by performing scene understanding and reporting back on its findings. We offer the following contributions in this report: 1) a human-robot vignette centered around the train derailment in East Palestine, Ohio, USA, in February 2023 that we modeled in a simulated platform; 2) a robot implementation to autonomously navigate this 3-D space as dictated by natural language instructions using sentence embeddings and cosine similarity for the robot’s dialogue management; and 3) scene understanding using Vision-Language Models to analyze a visual snapshot of the environment and generate a textual analysis for the human teammate. We conclude with a list of planned tasks to evaluate the models at the algorithmic level as well as for their efficacy in assisting a human in information gathering and disaster relief tasks.