Model of Narrative Nowness for Neurocinematic Experiments

Authors Janne Kauttonen, Mauri Kaipainen, Pia Tikka

Thumbnail PDF


  • Filesize: 0.58 MB
  • 11 pages

Document Identifiers

Author Details

Janne Kauttonen
Mauri Kaipainen
Pia Tikka

Cite AsGet BibTex

Janne Kauttonen, Mauri Kaipainen, and Pia Tikka. Model of Narrative Nowness for Neurocinematic Experiments. In 2014 Workshop on Computational Models of Narrative. Open Access Series in Informatics (OASIcs), Volume 41, pp. 77-87, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2014)


Cognitive neurosciences have made significant progress in learning about brain activity in situated cognition, thanks to adopting stimuli that simulate immersion in naturalistic conditions instead of isolated artificial stimuli. In particular, the use of films in neuroscientific experiments, a paradigm often referred to as "neurocinematics", has contributed to this success. The use of cinematic stimuli, however, has also revealed a fundamental shortcoming of neuroimaging studies: The lack of conceptual and methodological means to handle the viewers' experience of narrative events in their temporally extended contexts in the scale of full cinematic narrative, not to mention life itself. In order to give a conceptual structure to the issue of temporal contexts, we depart from the "neurophenomenological" approach to time consciousness by neurobiologist Francisco Varela, which in turn builds on Husserl's phenomenology of time. More specifically, we will discuss the experience of narrative tension, determined by backward-looking conceptualizing retention, and forward-looking anticipatory protention. Further, this conceptual structure is built into a preliminary mathematical model, simulating the dynamics of decaying and refreshing memory traces that aggregates a "retentive" perspective for each moment of nowness, which in turn may trigger anticipations for coming events, in terms of Varela and Husserl, protentions. The present tentative mathematical model is constructed using simple placeholder functions, with the intention that they would eventually be replaced by models based on empirical observations on the psychological capabilities that support narrative sensemaking. The final goal is a model that successfully simulates the way how the memory system maintains narrative tension beyond the transient nowness window, and thereby allows mappings to observed brain activity with a rich temporal system of narrative contexts.
  • computational
  • neurocinematics
  • narrative
  • retention
  • protention


  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    PDF Downloads


  1. Erik M. Altmann and Wayne D. Gray. Forgetting to remember: the functional relationship of decay and interference. Psychological science, 13(1):27-33, 2002. Google Scholar
  2. Andreas Bartels and Semir Zeki. The chronoarchitecture of the human brain—natural viewing conditions reveal a time-based anatomy of the brain. NeuroImage, 22(1):419-433, 2004. Google Scholar
  3. Andreas Bartels, Semir Zeki, and Nikos K. Logothetis. Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. Cereb Cortex, 18(3):705-717, 2008. Google Scholar
  4. Alexander C. Berg, Tamara R. Berg, Daume Daume III, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi. Understanding and predicting importance in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI), page 3562–3569, 2012. Google Scholar
  5. Gordon D. A. Brown and Stephan Lewandowsky. Forgetting in memory models: Arguments against trace decay and consolidation failure. In Sergio Della Sala, editor, Forgetting, pages 49-75. Psychology Press, Brighton, 2010. Google Scholar
  6. Eva Cadez. Integration of memory, perception and attention in episode processing. Journal of integrative neuroscience, 13(1):143-170, 2014. Google Scholar
  7. Eva Cadez and Evan Heit. Forgetting curves emerge from dynamics of integrated memory. In Proceedings of the 33th Annual Conference of the Cognitive Science Society, page 1673–1678, 2011. Google Scholar
  8. Antonio R. Damasio. The somatic marker hypothesis and the possible functions of the prefrontal cortex. Philosophical Transactions of the Royal Society B: Biological Sciences, 351:1413-1420, 1996. Google Scholar
  9. Stefan Frank, Mathieu Koppen, Leo Noordman, and Wietske Vonk. World knowledge in computational models of discourse comprehension. Discourse Processes, 45(6):429-463, 2008. Google Scholar
  10. Stefan L. Frank and Gabriella Vigliocco. Sentence comprehension as mental simulation: An information-theoretic perspective. Information, 2(4):672-696, 2011. Google Scholar
  11. Uri Hasson, Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin, and David J. Heeger. Neurocinematics: The neuroscience of film. Projections, 2(1):1-26, 2008. Google Scholar
  12. Uri Hasson, Yuval Nir, Ifat Levy, Galit Fuhrmann, and Rafael Malach. Intersubject synchronization of cortical activity during natural vision. Science, 303(5664):1634-1640, 2004. Google Scholar
  13. Uri Hasson, Eunice Yang, Ignacio Vallines, David J. Heeger, and Nava Rubin. Hierarchy of temporal receptive windows in human cortex. The Journal of Neuroscience, 28(10):2539-2550, 2008. Google Scholar
  14. Edmund Husserl. The Phenomenology of Internal Time Consciousness. Indiana University Press, Bloomington, 1964. Husserl’s writings in German language Zur Phänomenologie des Inneren Zeitbewusstseins originate from 1893- 1917. Google Scholar
  15. Iiro P. Jääskeläinen, Jyrki Ahveninen, Mark L. Andermann, John W. Belliveau, Tuukka Raij, and Mikko Sams. Short-term plasticity as a neural mechanism supporting memory and attentional functions. Brain Res., 1422:66-81, 2011. Google Scholar
  16. Iiro P. Jääskeläinen, Katri Koskentalo, Marja H. Balk, Taina Autti, Jaakko Kauramäki, Cajus Pomren, and Mikko Sams. Inter-subject synchronization of prefrontal cortex hemodynamic activity during natural viewing. Open Neuroimaging Journal, 2:14-19, 2008. Google Scholar
  17. T. Florian Jaeger and Harry Tily. On language 'utility': processing complexity and communicative efficiency. Wiley Interdisciplinary Reviews: Cognitive Science, 2(3):323-335, 2011. Google Scholar
  18. William James. The principles of psychology. New York: Holt, 1890. Google Scholar
  19. John Jonides, Richard L. Lewis, Derek Evan Nee, Cindy A. Lustig, Marc G. Berman, and Katherine Sledge Moore. The mind and brain of short-term memory. Annual review of psychology, 59:193-224, 2008. Google Scholar
  20. Michael J. Kahana and Mark Adler. Note on the power law of forgetting. Available at, 2002.
  21. Mauri Kaipainen and Antti Hautamäki. Epistemic pluralism and multi-perspective knowledge organization: Explorative conceptualization of topical content domains. Knowledge Organization, 38(6):503-514, 2011. Google Scholar
  22. Jukka-Pekka Kauppi, Iiro P. Jääskeläinen, Mikko Sams, and Jussi Tohka. Inter-subject correlation of brain hemodynamic responses during watching a movie: Localization in space and frequency. Front. Neuroinformatics, 4:5, 2010. Google Scholar
  23. Juha M. Lahnakoski, Juha Salmi, Iiro P. Jääskeläinen, Jouko Lampinen, Enrico Glerean, Pia Tikka, and Mikko Sams. Stimulus-related independent component and voxel-wise analysis of human brain activity during free viewing of a feature film. PloS ONE, 7:e35215, 2012. Google Scholar
  24. Benoît Lemaire, Guy Denhière, Cédrick Bellissens, and Sandra Jhean-Larose. A computational model for simulating text comprehension. Behavior research methods, 38(4):628-637, 2006. Google Scholar
  25. Yulia Lerner, Christopher J. Honey, Lauren J. Silbert, and Uri Hasson. Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. Journal of Neuroscience, 31:2906-2915, 2011. Google Scholar
  26. Richard L. Lewis, Shravan Vasishth, and Julie A. Van Dyke. Computational principles of working memory in sentence comprehension. Trends in cognitive sciences, 10(10):447-454, 2006. Google Scholar
  27. Sanna Malinen, Yevhen Hlushchuk, and Riitta Hari. Towards natural stimulation in fmri: issues of data analysis. Neuroimage, 35:131–139, 2007. Google Scholar
  28. Lauri Nummenmaa, Enrico Glerean, Mikko Viinikainen, Iiro P. Jääskeläinen, Riitta Hari, and Mikko Sams. Emotions promote social interaction by synchronizing brain activity across individuals. PNAS, 109:9599-9604, 2012. Google Scholar
  29. Robert Pugliese, Pia Tikka, and Mauri Kaipainen. Navigating story ontospace: Perspective-relative drive and combinatory montage of cinematic content. In Raivo Kelomees and Chris Hales, editors, Expanding Practices in Audiovisual Narrative. Cambridge Scholars Publishing, 2014. Google Scholar
  30. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image annotations using amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (Los Angeles, CA), page 139–147, 2010. Google Scholar
  31. Jelena Rosic and Pia Tikka. Annotation of film content for a neurocinematic analysis: Implications for embodied approaches to filmmaking. Presentation at Society for Cognitive Studies of the Moving Image 2013 (Berlin), 2013. Google Scholar
  32. Jesse Q. Sargent, Jeffrey M. Zacks, David Z. Hambrick, Rose T. Zacks, Christopher A. Kurby, Heather R. Bailey, Michelle L. Eisenberg, and Taylor M. Beck. Event segmentation ability uniquely predicts event memory. Cognition, 129:241-255, 2013. Google Scholar
  33. Sverker Sikström. Forgetting curves: implications for connectionist models. Cognitive Psychology, 45(1):95-152, 2002. Google Scholar
  34. Julie A. Van Dyke and Clinton L. Johns. Memory interference as a determinant of language comprehension. Language and linguistics compass, 6(4):193-211, 2012. Google Scholar
  35. Nico G. Van Kampen. Stochastic processes in physics and chemistry. North-Holland Publishing, New York, 1982. Google Scholar
  36. Francisco Varela. Neurophenomenology. a methodological remedy for the hard problem. Journal of Consciousness Studies, 3(4):330-349, 1996. Google Scholar
  37. Franscisco J. Varela. The specious present: A neurophenomenology of time consciousness. In Jean Petitot, Franscisco J. Varela, Barnard Pacoud, and Jean-Michel Roy, editors, Naturalizing Phenomenology: Issues in Contemporary Phenomenology and Cognitive Science, pages 266-329. Stanford University Press, 1999. Google Scholar
  38. Wayne A. Wickelgren. Time, interference, and rate of presentation in short-term recognition memory for items. Journal of Mathematical Psychology, 235:219–235, 1970. Google Scholar
  39. Wayne A. Wickelgren. Single-trace fragility theory of memory dynamics. Memory and cognition, 2(4):775-780, 1974. Google Scholar
  40. Peter Wittenburg, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes. Elan: a professional framework for multimodality research. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 1556-1559, 2006. Available at URL:
  41. Kiwon Yun, Yifan Peng, Dimitris Samaras, Gregory J. Zelinsky, and Tamara L. Berg. Exploring the role of gaze behavior and object detection in scene understanding. Front. Psychol., 4:917, 2013. Google Scholar
  42. Jeffrey M. Zacks, Nicole K. Speer, Khena M. Swallow, Todd S. Braver, and Jeremy R. Reynolds. Event perception: A mind/brain perspective. Psychol. Bull., 133(2):273-293, 2007. Google Scholar
  43. Jeffrey M. Zacks, Nicole K. Speer, Khena M. Swallow, and Corey J. Maley. The brain’s cutting-room floor: segmentation of narrative cinema. Front. Hum. Neurosci., 4:1-15, 2010. Google Scholar
Questions / Remarks / Feedback

Feedback for Dagstuhl Publishing

Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail