The secret life of subtitles: Investigating multimodal cognitive processing and psycholinguistic effects in English L1 and L2 viewers

When:17 Nov 2015, 3pm - 4:30pm
Venue:Morven Brown 209, UNSW (Map Ref: C20)
Who:Dr. Stephen Doherty, UNSW Australia
Dr Stephen Doherty


In a time when our consumption of multimedia products continues to increase, the usage of text-based stimuli within media and across its diverse platforms also continues to develop. Subtitles, in particular, are becoming more and more popular due to an interaction between the factors of globalisation, language diversity, economic viability, and technological feasibility. While research into subtitling has paved the way for developments in accessibility, audio-visual translation, and intercultural communication, a scarcity of empirical research into the cognitive processing of subtitles has led to a gap between academic research and wider application. This paper reports on two current large-scale experimental studies that are investigating the cognitive processing and psycholinguistic effects of English same-language subtitles in viewers with English as their first (L1) and second language (L2).

Firstly, we explore the effect that the modification of linguistic features of subtitles has on English L1 and L2 viewers (n = 81). Focusing here on evidential adverbs, we confirm the susceptibility of same-language subtitles to be manipulated covertly, and demonstrate the significant but distinctive effects of this manipulation on both language groups in terms of comprehension, interpretation, and cognitive load. We also show how divided attention is a significant moderator of this, and argue for its inclusion as part of more detailed participant profiling in future studies.

Secondly, we investigate the impact of same-language subtitles on psychological immersion as a function of the viewer’s language. English L1 and L2 viewers (n = 173) from three universities were assigned randomly to two experimental conditions in which they watched a popular American TV drama either with, or without, English same-language subtitles. Taking participants’ language profile and viewing habits into account as covariates, our results show that subtitles do not significantly reduce immersion for the L1 viewers, and actually benefit the L2 viewers in terms of scaffolding immersion by means of increased viewer interaction with the textual rendition of the narrative.

Finally, we reflect on the development of our methodology, and, using data from these studies, we explore further applications of this approach in second language acquisition and cognitive science.

About Stephen Doherty

Dr Stephen Doherty is a lecturer and researcher in the School of Humanities and Languages at UNSW, where he teaches courses on media translation, translation technologies, and translation theory. Using eye-tracking, psychometrics, and EEG methodologies, his research focuses on the cognitive aspects of translation and includes translation process studies, psycholinguistics, subtitling, and translation technologies.

View Past Events