All eyes on you – the wonderous world of Visual World Paradigm
Muzna Shehzad
"The eyes, chico, they never lie" – Al Pacino might have been onto something when he said this iconic line back in 19831. Fast forward a few years and we had a research paradigm that used eye movement data, gaining popularity in the 1990s2. I would like to take a moment to clarify that though Al Pacino emphasized the importance of considering what the eyes have to say, he had no role to play in this paradigm’s rise to fame. This paradigm, now known as the Visual World Paradigm, was pioneered by Cooper and by Tanenhaus and colleagues3, and the use of this paradigm revolutionized the field of psycholinguistics2.
Visual World Paradigm
The Visual World Paradigm (VWP), as the name may give away, is an eye-tracking paradigm. It consists of a visual display of pictures of objects that a participant views while listening to an utterance. The utterance can range from a story to only a single word4. How the visual display and the utterance are set up will depend on the research aims. In a VWP experiment, the visual display mainly constitutes a target, competitor, and distractor objects. During the VWP experiments, the participant’s eye movements are tracked and recorded using an eye-tracking system that facilitates in collecting high quality eye movement data.
To illustrate, let’s take an example from one of the most widely known research papers on VWP that study sentence processing in light of verb interpretation5:
Figure 1. VWP example
In this example (figure 1), the participant hears either the sentence ‘The boy will move the cake’ or ‘The boy will eat the cake’. In the visual array, the boy is the subject, the cake is the target object, and the other three objects are the distractors. Participants tend to look at the only edible object when hearing the verb ‘eat’ relative to when they hear the verb ‘move’ – showing how verb was used to predict what will be said5. As you may have noticed, all objects are moveable, and thus the verb information does not restrict the interpretation of the object.
The Visual World Paradigm has been used for various research questions spanning from sentence and phonetic processing, situation-specific interpretation, bilingual word recognition, to analysing the impact of varying contexts. This paradigm’s vast and versatile use has proven to be useful to explore the underlying cognitive processes2. One important cognitive process that the VWP is uniquely suited to investigate is prediction. Researchers can use the visual world paradigm to investigate the ‘time course of prediction’ as it allows the experimenters to observe and record anticipatory eye movements. Briefly, anticipatory eye movements can be understood as the participant looking to a target object/picture before the target word is spoken6.
Eye-tracking
Eye-tracking is a technique that uses pupil centre corneal reflection (PCCR) to track and record eye behaviour in real time across time and tasks7. In the PCCR technique, reflections in the pupil and cornea, caused by using infrared light, are detected and tracked8. The recorded eye behaviour can include pupil dilation (pupillometry) and various types of eye movements such as fixations (when the eye is relatively still and taking in information) and saccades (eye movements between fixations)9. It can also be used to measure when and for how long the participant fixated on the Area of Interest (AOI) which can be defined before or after the data collection stage10.
One useful aspect of eye-tracking is its integration with other technologies, such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), broadening its application. Eye-tracking is a useful technique that can be used for various research and clinical purposes across many different disciplines7. With careful planning in designing an eye-tracking study, it can be a powerful tool to obtain high quality data.
Figure 2. Experimenter view
HS-PrediCt’s Visual World Paradigm studies
Taking the assumption that the eyes don’t lie to our hearts, we decided to use the Visual World Paradigm in several of our studies. As some of you may already be aware, the HS-PrediCt research team investigates the mechanism of prediction during interaction, specifically in people with hearing impairment.
Figure 3. Eyelink 1000 Plus by SR Research13
Keeping this goal in mind, we used the Eyelink 1000 Plus by
SR Research. SR Research has been developing eye-tracking technology for the last 25 years and their eye-trackers are used widely by the scientific community
11. We opted for the Eyelink 1000 Plus due to its ability to gather high quality data and the flexibility it provides, making it an ideal eye-tracker to use in various experimental setups. As illustrated in figure 4, we opted for the desktop mount with a chin rest setup for this experiment.
Hearing impairment has been found to adversely impacts some cognitive processes during listening, such as sentence processing and word recognition that can be key elements to prediction12. Our study compares the aspects of prediction between hearing impaired (HI) older adults and normal hearing (NH) older adult participants. We aim to investigate whether one group is less likely to predict, overly rely on prediction and/or takes longer to predict.
Figure 4. Experiment setup
We are excited to share that intriguing results are emerging from this study. Stay tuned for further updates!
References
1. Scarface. Directed by: Brian De Palma. USA: Universal Pictures; 1983.
2. Huettig F, Rommers J, Meyer AS. Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychol (Amst). 2011 Jun 1;137(2):151–71.
3. Allopenna PD, Magnuson JS, Tanenhaus MK. Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models. J Mem Lang. 1998 May 1;38(4):419–39.
4. McDonald M, Assef BL, Li SWY, Zamuner T. The visual world paradigm for indexing phonological and semantic lexical co-activation in children: A methodological overview [Internet]. PsyArXiv; 2023 [cited 2023 Oct 9]. Available from: https://psyarxiv.com/c3jsf/
5. Altmann GTM, Kamide Y. Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition. 1999 Dec 17;73(3):247–64.
6. Amos RM, Seeber KG, Pickering MJ. Prediction during simultaneous interpreting: Evidence from the visual-world paradigm. Cognition. 2022 Mar 1;220:104987.
7. Carter BT, Luke SG. Best practices in eye tracking research. Int J Psychophysiol. 2020 Sep 1;155:49–62.
8. Farnsworth B. What is Eye Tracking and How Does it Work? - iMotions [Internet]. 2019 [cited 2023 Oct 27]. Available from: https://imotions.com/blog/learning/best-practice/eye-tracking-work/
9. Purves D, Augustine GJ, Fitzpatrick D, Katz LC, LaMantia AS, McNamara JO, et al. Types of Eye Movements and Their Functions. In: Neuroscience 2nd edition [Internet]. Sinauer Associates; 2001 [cited 2023 Oct 10]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK10991/
10. Cullipher S, Hansen SJR, VandenPlas JR. Eye Tracking as a Research Tool: An Introduction. In: Eye Tracking for the Chemistry Education Researcher [Internet]. American Chemical Society; 2018 [cited 2023 Oct 14]. p. 1–9. (ACS Symposium Series; vol. 1292). Available from: https://doi.org/10.1021/bk-2018-1292.ch001
11. Fast, Accurate, Reliable Eye Tracking [Internet]. [cited 2023 Oct 14]. About. Available from: https://www.sr-research.com/about/
12. Wendt D, Kollmeier B, Brand T. How Hearing Impairment Affects Sentence Comprehension: Using Eye Fixations to Investigate the Duration of Speech Processing. Trends Hear. 2015 Apr 22;19:2331216515584149.
13. Fast, Accurate, Reliable Eye Tracking [Internet]. [cited Oct 23]. EyeLink 1000 Plus. Available from: https://www.sr-research.com/eyelink-1000-plus/