Related publications, videos and repositories:
Demo: Catch My Eye: Gaze-Based Activity Recognition in an Augmented Reality Art Gallery [Demo abstract PDF] [Demo Video]
T. Scargill, G. Lan, M. Gorlatova
In Proc. IEEE IPSN’22, May 2022
EyeSyn: Psychology-inspired Eye Movement Synthesis for Gaze-based Activity Recognition [Paper PDF]
G. Lan, T. Scargill, M. Gorlatova
In Proc. IEEE/ACM IPSN’22, May 2022
Selected media coverage: vice.com, hackster.io. Highlighted in the university-wide Duke Daily and in the NSF-wide Research News newsletters.
GazeGraph: Graph-based Few-Shot Cognitive Context Sensing from Human Visual Behavior [Paper PDF]
G. Lan, B. Heit, T. Scargill, M. Gorlatova
In Proc. ACM SenSys 2020, Nov. 2020.
Eye tracking capabilities have been added to state-of-the-art AR headsets such as the Magic Leap One and Microsoft HoloLens 2, with a view to supporting gaze-based interactions within applications. Recent studies have examined how gaze point may be used to improve both user experiences and system efficiency, by adjusting the rendering quality of virtual content according to its position in the field of view.
However, a survey of the eye tracking literature shows us that there exists the potential for much more sophisticated applications of eye tracking in AR. Beyond simply using the current gaze point, we could aggregate those gaze points to generate heatmaps, combine sequences of gaze points to create scanpaths, and use other data such as blink detection in novel ways. In turn, these inputs could facilitate the extraction of a user’s cognitive state, intent and interests, and help us tailor virtual content accordingly.
One example of this is in the provision of contextual and personalized content. A recently published paper produced by the lab (PDF here) explored how different virtual content may be rendered, depending on both environmental conditions and user attributes/preferences. Eye tracking is unique in that it can provide otherwise unknowable cognitive attributes, allowing us to anticipate a user’s needs and act accordingly. Secondly, we propose that gaze data may be used to maximize system efficiency; if an object or region in the scene receives little attention, we do not need a high-fidelity representation of it, and storage/computation savings can be made.
A use case that captures these possibilities particularly well is the concept of an augmented reality art gallery. In a recent project we developed a testbed to simulate this scenario, along with a machine learning algorithm that analyzed a user’s level of engagement through their eye movement data (see video below). We anticipate this data could be used both to provide in-app content suggestions to users, and for application owner/developer analytics. We plan to extend this work by examining the creation/analysis of heatmaps and scanpaths, and including 3D virtual content such as sculptures.
