Final Report on Multimodal Inference of Human State: Tracking Cognition in a Risky Environment

Author(s):

Jason S. Metcalfe, W. David Hairston, Namazbai Ishmakhametov, Mohammad Y. M. Naser, and Sylvia Bhattacharya

Abstract:

This report details the execution and outcomes of a 3-year collaboration between the U.S. Army Combat Capabilities Development Command Army Research Laboratory and Kennesaw State University that focused multimodal data integration to improve human state inference in complex, high-risk environments. The project advanced signal processing techniques and machine learning algorithms for classifying human states related to ongoing, task-related events. Accompanied by seven publications, data, and a code repository, the primary contribution from this program was the Sequential, Event-Based Multimodal Fusion framework, which enhances the integration of diverse physiological, behavioral, and environmental data streams. This framework improved accuracy of classifying events from DEVCOM Army Research Laboratory within real-world driving experiments. These advancements support ARL’s mission toward human–machine integration for future operational formations. The project also contributed to talent development by involving 28 students across various disciplines, providing a cross-functional team of students with hands-on experience in multimodal data fusion and Army-focused research. This engagement supported career transitions into industry and academia, strengthening the national workforce that will manifest future Army technology-based advantages. This project further established an interdisciplinary research ecosystem, and the outcomes advance ARL’s objectives in the broader scientific space of multimodal machine learning and classification.