Long-Term Analysis of a Human-AI Collaboration Study Using a Mobile Game

Author(s):

Torin Adamson, Arvind Pillai, Andrew Campbell, Lydia Tapia, Lidia S Obregon, and Evan C Carter

Abstract:

Human-AI interaction is typically studied in laboratory settings where participants spend minutes to hours per session. The benefits of this approach are well-known: excellent experimental control and sophisticated sensing of behavioral and nonbehavioral data (e.g., psychophysiology) to name a few. The downsides are discussed less frequently. The burden on experimenters and participants from sessions, difficulties in collecting massive data sets, and the inability to analyze longtime- scale processes limit the produced knowledge. Our experimental setup combines mobile devices and wearable sensors as a means of addressing these limitations. Via a mobile game application called “Busy Beeway,” for studying human-AI collaboration, we require participants to balance their strengths and weaknesses with autonomous partners in obstacle avoidance tasks. Participants provide game data daily over several months while their “context”— heart-rate, activity, sleep, environment—is measured continuously by StudentLife. In this technical report, we present an initial survey of interaction styles that participants developed over time with different autonomous capabilities and across changing context. Additionally, we present how this data can be used to predict human-AI collaborative performance, thus providing a way to suggest human-AI pairings.