A Multimodal Understanding of the Eye-Mind Link

The eye-mind link postulates a relationship between overt attention and covert cognitive processes. Eye movements are the key measure used to reflect attention allocation, and have long been used to develop explanatory and predictive models of mental processing. However, there is very little research investigating the direct relationship between eye movements and the processing occurring in the brain. The proposed research aims to leverage a multimodal dataset where both eye tracking and the cortical hemodynamic responses in the brain were recorded in parallel. The goal is to develop a flexible methodology that can relate eye movements to the brain activations occurring in parallel, including state-of-the-art transformer models to extract meaningful information from the two very different time series. The goal is to support the understanding of the eye-mind link through a focus on the "mind" while also working towards multimodal modeling of cognitive processes. © 2024 Copyright held by the owner/author(s).