A Study of the Time Delay between Head Gestures and Pitch Patterns

We investigated the pragmatic effects of gesture-speech lag by asking participants to narrate four different stories in both English and in their mother tongue, in four conditions sync, video or audio lag (15 ms), audio only conditions sync, video or audio lag (15 ms), audio only. All three video groups rated the task as less difficult compared to the audio only group and performed better. The scores were slightly lower when sound preceded gestures (video lag), but not when gestures preceded sound (audio lag). Participants thus compensated for delays of 15 milli seconds in either direction, apparently without making a conscious effort. This greatly exceeds the previously reported time window for automatic multimodal integration.