Learning to Understand Anomalous Scenes from Human Interactions

Author(s):

Stephanie M Lukin, Rahul Sharma, and Michael Bellissimo

Abstract:

At the US Army Combat Capabilities Development Command Army Research Laboratory, we are studying behavior, building data sets, and developing technology for anomaly classification and explanation, in which an autonomous agent generates natural language descriptions and interpretations of environments that may contain anomalous properties. This technology will support decision-making in uncertain conditions and resilient autonomous maneuvers where a Soldier and robot teammate complete exploratory navigation tasks in unknown or dangerous environments under network-constrained circumstances (e.g., search and rescue following a natural disaster). We detail our contributions in this report as follows: we designed an anomaly taxonomy drawing upon related work in visual anomaly detection; we designed two experiments taking place in virtual environments that were manipulated to exhibit anomalous properties based on the taxonomy; we collected a small corpus of human speech and human–robot dialogue for an anomaly detection and explanation task; and finally, we designed a novel annotation schema and applied it to a subset of our corpus.