STAIT-Replay: Multimodal 3D Spatiotemporal Stream Replay for Enabling Common Grounding in Human–Agent Teams

Author(s):

Russell Cohen Hoffing

Abstract:

Researchers and analysts attempting to understand real-world, naturalistic 3D spatiotemporal datasets derived from humans, autonomous agents, and mixed human–agent teams contend with cognitive and technical bottlenecks due to massive spatially and temporally heterogeneous sensor streams. These datasets comprise high-frequency poses, LiDAR point clouds, 2D camera feeds, conversational dialogue, eye gaze vectors, and discrete interaction events. For instance, an analyst attempting to evaluate what an autonomous ground robot “sees” via LiDAR scans must inspect 3D point clouds in isolation—a process that frequently leaves object identities ambiguous. Similarly, analyzing dialogue transcripts without spatial context obscures the situational awareness motivating communication. Synchronously pairing 3D spatiotemporal data, 2D video feeds, pose, trajectories, and occupancy grids provide the critical visual and spatial grounding necessary to understand this otherwise obtuse data. We present STAIT-Replay, an open-source, interactive 3D spatiotemporal visualization and auditing tool developed as part of the broader STAIT (SpatioTemporal Analysis and Inference Toolkit) suite and built on top of the Rerun visualization framework. STAIT-Replay enables unified visualization and interaction with heterogeneous 3D spatial data, 2D visual feeds, dialogue, eye gaze rays, and discrete event streams on a unified timeline for interactive playback.