Building Shared Situational Awareness across Heterogeneous Units for Improved Squad Lethality

Author(s):

Chloe Callahan-Flintoft, Joyce Tam, Osben Toulson, Richard Diego, Kevin King, Justin Brody, Joseph Conroy, Andrew Tweedell, Thomas Rohaly, Harrison Crowell, Heather Roy, and Jonathan Touryan

Abstract:

Human–machine integration (HMI) requires a shared understanding of context; that is, what are the mission objectives, and what are the environmental parameters that might affect execution of the mission? To address this need, we focused on building shared understanding of the surroundings by compiling gathered visual information during a mission-relevant scenario, hereafter referred to as situational awareness (SA). Humans build SA through a collection of cognitive mechanisms that allow them to attend or ignore information depending on the parameters of the task and the environment. Sharing this information with autonomy could enable heterogenous human–machine units to respond adaptively in dynamic real-world mission scenarios. However, to do so requires human SA to be characterized in a machine-readable format. Toward that aim, in the current work, two Soldiers equipped with HoloLens headsets and sensor-augmented rifles searched a mock urban environment for targets. A theory-grounded, neural model of human visual attention provided a quantitative characterization of the information in each Soldier’s visual field most likely to be actively perceived and remembered by the Soldier. To amass this information across multiple human members of the squad as well as other robotic systems, we built the networking capabilities necessary to record synchronized multimodal data streams and identified challenges in localizing these data streams into a shared coordinate system. Finally, we discuss lessons learned, remaining gaps, and suggestions on how future work could close those gaps to achieve bidirectional, adaptive HMI.