The ILCB Docs & Postdocs Group

Research Project : Large-scale exploratory modeling of early language acquisition: integrating prosodic and phonetic learning

Short Summary:

One of the remaining challenges in developmental research is to link learning mechanisms to the observable learning outcomes they predict in ecological environment at developmental timescales. We propose to leverage the recent breakthroughs in computational modeling to systematically investigate two fundamental milestones in early language acquisition: the discovery of the rhythm and the discovery of the sound system of the native language. We will simulate learning on naturalistic speech in amounts relevant to developmental timescales in order to delineate critical computational principles for language learning, and we will evaluate the models’ predictions using empirical behavioral and neurophysiological data. In a first work package, we propose, for the first time, to model pre-natal rhythm perception and prosodic learning, in order to directly investigate the impact of fetal exposure on the acquisition of prosody. In a second work package, we will conduct an exploratory modeling study of early phonetic learning. We will contrast a variety of representation learning algorithms that parallel some hypothesized phonetic learning mechanisms to identify which parameters are critical to capture well-characterized effects in the developmental literature. In the last work package, we will build an integrated prosodic and phonetic learning algorithm in order to investigate how prosodic processing impacts phonetic learning, and to test whether it explains infant behavioral and neurophysiological data better than state-of-the-art speech recognition models. Overall, this project will foster two-way interactions between the developmental science and computational modeling communities, both informing cognitive theories of language acquisition in a principled way and promoting the development of infant-inspired learning algorithms.

Research Project : Statistical modelling of vocal productions of human babies during their first year of life

Short Summary: This thesis project is part of the "Babyvoc" project, which aims to study the developmental trajectory of vocal productions in human babies during their first year of life. From sound recordings of human babies made once a month for three days at home between 0 and 12 months, we currently have a database comprising 16 babies and more than 1,000 hours of recordings. Thanks to an automatic processing procedure based on a convolutional neural network (Bonafos, Pudlo, Freyermuth, Legou, Fagot, Tronçon & Rey, 2023), we are able to extract the relevant segments corresponding to babies' vocal productions, i.e., currently 200 hours. The aim of this project is to draw on the latest developments in Object Oriented Data Analysis (OODA, e.g., Marron & Dryden, 2021) and Topological Data Analysis (TDA, e.g., Chazal & Michel, 2021; Wasserman, 2018) to characterize the space of vocal productions for each individual during the first 12 months of life. This project could thus enable us to gain a better understanding of language development in its early phases, and also to develop a diagnostic tool for detecting atypical developmental trajectories.

Research Project : A physiological take on Turing test: using fMRI to assess the social competence of an autonomous conversational robot

Short Summary: The present preoject starts from the recent debates surrounding the social competence of artificial agents, particularly conversational agents based on large language models (LLMs) like chatGPT. It draws inspiration from Alan Turing question, "Can Machines think?" to explore the concept of the Total Turing Test. The current implementations of the Turing Test suffer from a major limitation: challenges such as the Loebner Prize ask participants to recognize whether the conversational chatbot is a human or a machine. By priming the possible artificial nature of the interacting agent, such challenges induce a specific "testing" stance from the human participants. We propose three solutions from social cognitive neuroscience though a multidisciplinary approach that aims to investigate human-robot conversational interactions using synchronized behavioural and neurophysiological data. Firstly, we will concentrate on the role of phonological information in voice processing, investigating the influence of sensory cues on participants' engagement with artificial agents using an existing corpus. by analysing this data, we aim to unravel the significance of sensory inputs in shaping the dynamics of conversation. Then, w'll investigate the adoption of an intentional stance during interactions with unfamiliar agents, hypothesizing it is influenced by a dynamic interplay between sensory information and contextual factors. By employing fMRI technology, the project aims to objectively and non-invasively measure the adoption of the intentional stance, avoiding the need for subjective reports. Importantly, this approach avoids priming participants about the potential artificial nature of the interacting agent, which distinguishes it from traditional implementations of the Turing Test. To facilitate data collection, we will develop a robust methodology encompassing the utilization of a cover story and the design of a comprehensive Total Turing test corpus. The objectives of the project include analysing brain responses involved in sensorimotor and contextual information during a behavioural protocol testing social conversation, identifying brain areas involved in vocal processing, and assessing a multimodal approcach encompassing the analysis of behavioral and physiological data, our project aims to contribute to the broader understanding of the intentional stance and the social competence of artificial agents. We strive to shed light on the intricate machanisms underlying human-robot conversational interactions, paving the way for future advancements in this burgeoning field.

Jacques Pesnot Lerousseau

Research Project : In-Context Rule Generalisation in Transformers and Humans

Project Summary : Recent artificial neural networks like transformers have recently demonstrated an ability to use the context to extract a rule, without needing an actual modification of their weights. Can this “in-context learning” phenomenon account for human behaviour too? To address this issue, we will investigate the same abstract sequence learning and contextual information processing in both humans and transformers by combining modelling, behaviour and brain imaging experiments. For this, we will create a transitive inference learning paradigm where agents have to learn an abstract representation during training to do few shot inferences in a subsequent evaluation phase. We will compare these in-context learners to human brain responses to those abstract sequences. Overall, this project lays the foundations to uncover in-context rule generalisation in brains and algorithms.

Research Project : The Ontogenetic and Phylogenetic Foundations of the Linguistic Brain

Project Summary : Determining the structural changes in human brain evolution that enable our capacity for language in an ongoing active area of research. This quest to understand human language processing involves research into a wide variety of processes that support language, including vocal perception and production; semantic comprehension, sequencing, articulation, and gestural communication. The substrates for these abilities recruit a distributed network of cortical areas connected by a number of large white matter tracts, some of which are more well-characterized than others thanks to the recent rise of diffusion imaging. In order to determine which components are important for human language, we need to identify how these networks are organized in related species. Many species of primate are being used and/or developed as animal models to interrogate brain structure and function with the goal of being able to generalize these findings to humans. However, humans have both a unique evolutionary history and a protracted developmental path which both feature in our capability for complex language. Within a comparative framework with human language network strutures, the present project will characterize the organization of language-homolog brain areas and its structural connectivity in other primate species. This approach will use recent state-of-the-art analyses of diffusion-weighted images and critical white matter tracts of interest not only across different primate taxa but also across different age classes by including longitudinal data base of cerebral images in baboons from birth to adulthood This projet will thus provide insight into the a) structural and    b) developmental modifications that permit language of the substrate of human language in the primate lineage, and c) create valuable comparative resources for researchers investigating cognitive processes in model species.

Research Project : Investigating the coding of voice identity in voice-selective areas of macaques

Project Summary : This project aims to investigate how neurons in Temporal Voice Areas (TVA) encode voice identity. In ILCB’s BaNCo team there is world-unique ability to record single neuron spiking activity in fMRI-localized voice-selective areas in the macaque brain. Thanks to these recordings, I recently confirmed the existence of ‘voice cells’ in the anterior TVA (aTVA) of macaques. However, I also discovered an unsuspected population of human-selective cells that appear to encode information in human vocalizations (Giamundo et al., 2023, in prep.). The present project aims to test further these two subpopulations, focusing on one of the main types of voice information: identity.
Research in neuronal processing of facial identity has demonstrated that neurons in anterior face patches show viewpoint invariance (Freiwald & Tsao, 2010), and that individual neurons encode specific directions in a multidimensional face space (Chang & Tsao, 2017). My main hypothesis is that neurons in the aTVA follow similar principles of voice identity coding to those of face patch neurons; specifically, that they will: (i) exhibit caller identity invariance; (ii) encode specific dimensions of the voice identity multidimensional space. A secondary hypothesis is that encoding of identity information will differ in human vs. macaque-selective neurons.
Results will improve our understanding of neural mechanisms involved in voice processing, and will contribute to advance our comprehension of the evolution of communication systems in primates.

Research Project : Vers une approche comparée de la complexité des signaux de communication : apports croisés de l’éthologie et de la linguistique

Short Summary: De plus en plus de chercheu.e.s en linguistique, en psychologie comparée et en éthologie estiment qu’une approche comparative des systèmes de communication chez les primates humains et non-humains centrée sur la syntaxe et sur le canal vocal/auditif est trop restrictive pour comprendre au mieux la complexité des systèmes de communication et les origines évolutives du langage humain. Ce projet a pour but d’étudier la complexité de la communication multimodale lors d’interactions sociales spontanées chez les babouins de Guinée en mêlant une approche quantitative empruntée à l’éthologie (analyse de séquences comportementales, clustering, mesure de la complexité structurelle et contextuelle sur la base de l’entropie de Shannon) et une méthode qualitative empruntée à la linguistique interactionnelle (analyse de paires adjacentes, turn-taking, réparation de l’interaction, etc.). Les babouins sont un modèle de choix pour l’étude de l’histoire évolutive du langage humain puisqu’ils possèdent une socialité complexe et un habitat naturel ouvert, proches de ceux rencontrés dans l’histoire des sociétés humaines. Nous souhaitons développer des outils quantitatifs permettant de comparer la complexité de systèmes de communication de diverses espèces de primates qui ne soient pas centrés sur des caractéristiques définitoires du langage humain telles que la syntaxe. Nous souhaitons également transposer à l’étude de la communication des primates non-humains les méthodes et outils d’analyse de la linguistique interactionnelle et ainsi, identifier plus précisément ce qui, dans la complexité de l’interaction, constitue un socle commun aux primates humains et non-humains.

Short Summary: The goal of this project is to study the neurobiology of syntax, the brain’s ability to structure linguistic elements and create meaning, focusing on two understudied topics: (1) Spatiotemporal dynamics, namely ‘how do the neural sources of basic combinatorial syntax manifest over time?’; (2) Language modality, namely ‘do the perception and production of combinatorial syntax rely on the same neural machinery?’. Addressing these two questions is important because the quite disperse neurobiological models of syntax make different predictions with regard to the degree of spatiotemporal and language modality integration. Hence, investigation these two dimensions will provide insights to constrain brain language models from a novel perspective. More concretely, in this project we will focus on the two core brain areas routinely associated with syntactic processing, the left posterior temporal lobe (LPTL) and the left inferior frontal gyrus (LIFG), and test (1) whether they form a parallel integrated network or rather serve distinct functions over time; and (2) whether their dynamics are identical or not when perceiving vs producing syntax. To test these predictions, we adapt Pylkkänen and colleagues’ minimalistic paradigm to syntax and contrast the final word of simple three-word noun phrases (NPs) that differ in their syntactic tree and computation (e.g., a left-branching Adjective-NP ‘joli buisson fleuri’ vs. a right-branching Adverb-NP ‘buisson joliment fleuri’). This paradigm will be used in three Work Packages (WPs): In WP1 we will test the spatiotemporal dynamics (high-density EEG) in perception (exp1) and production (exp2) separately. In WP2 we will test (MEG) perception and production within the same brain (exp3). And finally, in WP3 we will test the two interlocutors simultaneously (EEG hyper-scanning) while interacting with each other using the minimal NPs (exp4). In sum, this project will assess the degree of spatiotemporal integration of the syntactic parser across the language modalities, allowing to constrain neurobiological models of syntax in function of linguistic behavior and social interaction.

Short Summary: Neural investigations of reading have mainly focused on describing an average generic left-lateralized reading network, with two  dissociated routes for mapping orthography to phonology, and orthography to lexical forms and meaning, in line with behavioral and computational dual route models of reading. To date, variability in the neural substrates of reading has been associated with some features of the writing system, and with readers’ experience and abilities. For instance, reading through grapheme-to-phoneme mapping, i.e. at a fine grain of orthographic processing, results in consistently more left-lateralized activations in regions of the ventral occipito-temporal cortex (vOT) associated with orthographic processing, compared to mapping whole words (or logograms) to phonology. It remains unclear, however, how to reconcile these various neural findings, implicating different neural features, across ages, abilities, languages, and contexts. Here, we propose a unifying view of the reading network, according to which phonological decoding and orthographic grain size are general organizing principles of the reading brain network across multiple scales (cross-linguistic, inter-individual and intra-individual), and across ages. We predict that increased reliance on fine- grained orthographic processing and on phonological decoding is associated with three neural markers: stronger left-lateralization of vOT reading activations, stronger medial than lateral activations in the left vOT (around the putative grapheme area), and increased reliance on phonological regions of the dorsal reading route. To support to this view, we propose to analyze how these three neural markers are associated with phonology and grain size across ages and scales (across languages, across individuals and across stimuli and contexts) in two datasets: an existing longitudinal and multilingual child dataset (SF immersion dataset), and an adult multimodal dataset to be acquired at the ILCB (ILCB adult dataset). This new multimodal dataset will include 1) behavioral measures of phonological demands and reading grain-size, 2) functional MRI investigations of underlying neural correlates (the three markers) and 3) tests of the causal role of the right-vOT in coarse-grained orthographic processing using TMS. With this project, we hope to advance the understanding of the neural underpinnings of reading, by establishing the role of grain size and reliance on phonology as general organizing principles of the reading brain.

Project Summary : Empathy, the capacity to share emotional experience of others is fundamental to prosocial behaviours such as cooperation and communication in humans. Investigations into the development of empathy in preverbal infants have solely focussed on their responses to other's facial and vocal markers of emotions. These studies show that infants discriminate emotions, possibly based on their external expressions, but remain inconclusive about infants' comprehension of the meaning of emotions and the underlying mental states. In addition, there is little to no work on whether and how preverbal infants may communicate their empathic motivation as a response to other's emotions. In this project, using behavioural and neurophysiological methods, I propose to investigate gestures in preverbal infants as a way to fill  the existing gaps in our knowledge on early developmental milestones of empathy. Gestures are nonverbal communicative signals produced intentionally to convey a conceptual meaning or achieve a social goal. Thus, because they are inherently a medium of communication of concepts, gestures provide an avenue to study at which age infants understand the conceptual meanings of emotions. Since infants are capable of intentional communicating through gestures from 8 months of age, they may begin empathic responding to other’s emotional states by producing gestures even before speech. The present project proposes thus to be the first systematic investigation of comprehension and production of gestures during emotion-eliciting events. The results generated will contribute to our understanding of the early cognitive development of empathy; additionally, identifying gesture markers of empathy could be applied into further investigations of empathy development in nonhuman primates, using a comparative framework.

Project Summary : Dysgraphia, that is characterized by an alteration of handwriting, is often found in developmental coordination disorder (DCD) and/or developmental dyslexia (DD) (Afonso et al., 2019; Biotteau et al., 2019). Interestingly, difficulties to perceive and produce rhythm have also been reported in the latter two disorders (Lense et al., 2021) and could contribute to dysgraphia. However, since graphomotor alteration may be different in DCD and DD (Afonso et al., 2019; Jover & Huau, 2021), the relationship between dysgraphia and rhythm could depend on the associated disorder. This investigation has important implications on both diagnosis and remediation perspectives to help children with dysgraphia. Indeed, a growing number of studies have shown that rhythmic interventions could help patients suffering of motor disorders (e.g. Dalla Bella et al., 2017) and language disorders (e.g. Thomson et al., 2013). Clearly, this project aims to study how rhythm is impacted in children with dysgraphia, in specifying their co-morbidity, and then to evaluate the effects of rhythmic background listening on handwriting in these same children. First, rhythm impairments in children with dysgraphia will be characterized using behavioral and EEG measures of rhythm perception and production. It is expected that children with dysgraphia will perform worse than children without dysgraphia on these rhythm tasks. EEG measures would permit to specify the impacted processes, based on measuring beat-related steady-state evoked potentials (SS-EPs, Nozaradan et al., 2015) in an attempt to disentangle neural sensory and motor contributions to behavioral beat synchronization in children (e.g., Colling et al., 2017). We expect the children to present distinct profiles depending on the presence of DCD and DD, singular or associated. Besides, we expect the handwriting difficulties to coincide with the rhythm ones. The second aim of the project is to investigate the potential benefits of a rhythmic background on handwriting performance in children with dysgraphia. To achieve this second objective, a first study will be conducted in typically developing children (TD) to investigate how rhythmic background presented at different tempi influences handwriting speed and fluency. Then, the benefits of a rhythmic background on handwriting fluency will be formally tested in children with and without dysgraphia. To this aim, children will realize pre-writing and word writing tasks while listening to a rhythmic background with different tempi (matching with the individual handwriting tempo or with an increased/decreased tempo) and in a non-rhythmic background condition. We hypothesize that writing frequency and fluency will improve in the rhythmic background condition. In addition, differential benefits are expected depending on the motor and/or reading disorder associate to dysgraphia..

Research Project : Separation and characterization of region-specific activities in the auditory cortices during speech perception.

Short Summary:

The objective of this project is to characterize theta-gamma interactions in speech processing thanks to intracerebral EEG (iEEG) and advanced signal processing. The methodology will be based on blind source separation (BSS) techniques applied to iEEG, disentangling brain activity from spatially close regions that are mixed at the sensor level.

In the human auditory cortex, theta (4-8 Hz) and low-gamma (25-45 Hz) oscillation have been suggested to parse syllabic and phonemic information, respectively. This process would be asymmetric, with predominant theta activity in the right hemisphere, and left hemisphere preferentially oscillating in the gamma range. How this functional lateralization is implemented, and the putative role of the different oscillations, is widely debated. Intracerebral EEG is a method of choice for answering these questions, as it is recording directly from brain structures. Still, each iEEG sensor records a combination of the activity of close and distant structures whose electric fields spread through the brain to the sensor. Therefore, it is arduous, if not impossible, to isolate the activity of only one structure from raw electrophysiological recordings.

This project is divided in two main stages. First, we will apply BSS on auditory evoked potentials of right and left auditory cortices, looking to separate the electrophysiological activities stemming from primary, secondary and associative cortices, and analyzing their intrinsic spectrotemporal dynamics and inter-hemispheric differences. Second, we will focus on the analysis of theta and gamma interactions in the left auditory cortex while natural speech was perceived. Thanks to BSS on iEEG, we expect to be able to segregate region-specific oscillations, and hypothesize that theta responses to the stimuli are asymmetric and that multiple theta-gamma interactions coexist between structures.

In summary, this project will use a unique combination of intracerebral data recorded during speech perception and a new signal analysis method that will be able to address long lasting questions on frequency interactions. It is thus expected to improve our understanding of the interactions between subregions of the auditory cortex, their lateralized interactions, with an impact on the fine mapping of language regions. Importantly, it will give insights on the interrelation between region-specific oscillations during speech perception. We will also provide to the ILCB the methods and software to disentangle activities from different substructures of the auditory cortex with iEEG.

Research Project: Functional role of oscillatory dynamics in motor cortex during speech perception

Project Summary :
While our knowledge on the brain structures underpinning speech perception has greatly advanced in the last decades, the neurophysiological mechanisms that can explain how humans process speech are still largely unknown.
In particular, influential theories about speech perception do not agree on the role of the motor system (Skipper et al., 2017): Dual-stream theories suggest that the motor system is not crucial (Hickok & Poeppel, 2007; Hickok, 2014), whereas opposing theories ascribe a fundamental role to the motor system in speech perception (Barnaud et al., 2018; Pulvermüller & Fadiga, 2010). A fruitful approach to understand the neural mechanisms underlying speech perception is to investigate cortical oscillations. Cortical oscillations refer to synchronized rhythmic brain activity, which is hypothesized to be important for structuring, binding and consolidating complex information in the cerebral cortex (e.g. Buzsáki & Draguhn, 2004). Given the intriguing possibility that cortical oscillations may offer a link between brain and behavior, in particular for higher-order cognitive processes such as language perception (e.g. Buzsáki, 2010), the current project sets out to investigate how brain oscillations in the motor cortex impact speech comprehension.
The objectives and hypotheses of this project are guided by current observations and proposals regarding the nature of cortical oscillations for (1) the extraction of speech sounds and (2) the extraction of meaning when perceiving language. With regard to speech sound processing (1), it has been suggested that cortical oscillations “provide the [temporal] infrastructure to parse and decode connected speech” (Giraud & Poeppel, 2012). At the level of the auditory cortex, low-frequency neural oscillations entrain to the (quasi-)rhythmic structure of the speech signal and causally contribute to speech comprehension (Peelle, 2018; Riecke et al., 2018; Zoefel et al., 2018). Moreover, neural entrainment to speech is also observed in regions beyond the auditory cortex, and in particular in the motor cortex, at the phrasal (0.6-1.3 Hz), lexical (1.3-3 Hz), and syllabic rates (3.5-4.5 Hz) (e.g. Keitel et al., 2018; Assaneo & Poeppel, 2018). One hypothesis is that such entrainment in these frequencies reflects temporal prediction derived from the temporal regularities presented in speech (e.g. Morillon & Baillet, 2017). In a similar vein, cortical oscillations related to the lexical meaning of spoken words (2) have been observed over auditory and motor cortices in the high-frequency range (beta and gamma; e.g. Pulvermüller et al., 1996; Canolty et al., 2007).
These large-scale synchronizations between fronto-central and superior temporal brain regions are hypothesized to reflect the binding of sensorimotor experiences into lexical categories (e.g. Strijkers, 2016; Garagnani et al., 2017). However, at present the functional role (if any) of such motor oscillatory activity in speech perception remains debated. In the current project, we set out to investigate the exact nature of oscillatory dynamics in the motor cortex for key components of speech perception (i.e. sound- and meaning-extraction) with two complementary studies
that each containing a behavioral and neurophysiological (magnetoencephalography; MEG) part. The behavioral experiments will assess whether activation of the motor cortex improves speech perception (and if so, under which conditions), and the MEG experiments will assess whether these potential behavioral improvements are indeed driven by frequency-specific cortical oscillations and enhanced functional coupling between motor and auditory cortical regions. In this manner, the results of this project may provide valuable insights for the theoretical development of sensorimotor integration during language processing and even highlight that specific oscillatory patterns (different frequency ranges) drive different processes involved in the perception of speech.

Research Project: The neural correlates of embodied L2 learning

Project Summary :
Within the framework of embodied semantics, the overlap between sensorimotor and language processes could have important implications for second language (L2) acquisition. Neurolinguistic studies have established motor-language interactions and behavioral studies have shown that coupling word encoding with corresponding, relevant gestures enhances language learning. The present study will explore these aspects of language processing and acquisition, using virtual reality (VR), to investigate how performing naturalistic actions during the learning of L2 action verbs may enhance mapping between word form and meaning. We will examine whether “embodied learning”, or learning that occurs using specific physical movements that are coherent with the meaning of new words, creates linguistic representations that produce greater motor resonance (activity in the motor cortex), due possibly to stronger and more specific motor traces, compared to a control condition. In addition, we will investigate whether embodied learning leads to better retention; indeed the implication of both linguistic and motor areas should lead to a more complete semantic representation and increased learning. Training will take place over two days using auditory verbs and a VR oculus headset and a hand controller. Native French speakers will learn the same set of L2 verbs in Serbian, in either the “embodied learning” or the control condition. The “embodied learning” group will learn the action verbs using different specific movements to manipulate objects for each verb; the control group will simply point to the virtual objects. Both pre and post training, time-frequency analysis will be carried out on the EEG data to measure mu and beta suppression, associated with motor activation, while participants passively listen to the new verbs. EEG and behavioral accuracy will also be used to assess learning in an audio-visual match-mismatch task. We expect that the “embodied learning” group will show greater motor activation during verb processing post-training, and that this will correlate with improved learning as indexed by a greater N400 effect and improved behavioral results compared to the control condition. If so, this would suggest that embodied learning adds a motor trace to lexical items, which would support theories of embodied semantics.

Research Project :

Development of Children's Communicative

Project Summary :
Research Lab: CoCoDev
The study of how the ability for coordinated communication emerges in development is both an exciting scientic frontier | at the heart of debates about the uniqueness of human cognition (Tomasello, 2014) | as well as an important applied issue for AI (Antle, 2013).
Early signs of coordination (e.g., through gaze and smile) can be found in preverbal infants (Yale, 2003), but the ability to engage in coordinated verbal communication (Clark, H. & Brennan, 1991) takes years to mature.
Learning such coordination, especially with the caregivers, is crucial for the child's healthy cognitive development (Ho, 2006; Gelman, 2009).
Very few studies examined the nature of children's communicative coordination and its development in the natural environment (that is, outside controlled laboratory studies).
Further, existing naturalistic studies (e.g., Clark, E. 2015), though insightful, have been based on anecdotal observations, leading to rather qualitative conclusions.
Thus, previous work did not provide any theoretical model that could explain, quantitatively, the naturally occurring data, let alone provide a basis for theory-informed applications. This project will contribute to ll this gap.
We will combine AI tools from NLP and Computer Vision to study the multimodal dynamics of children's communicative coordination with caregivers, laying the foundation for a data-driven model that would 1) provide us with a scientic understanding of the natural phenomena and 2) guide us through the design of child-computer interaction systems that can be used to test and evaluate the model.

References
Antle (2013). Research opportunities: Embodied child{computer interaction. International Journal of Child-Computer Interaction.
Clark, E. (2015) Common ground. The Handbook of Language Emergence.
Clark, H. & Brennan (1991). Grounding in communication. Perspectives on socially shared cognition.
Gelman (2009). Learning from others: Children's construction of concepts. Annual review of psychology.
Ho (2006). How social contexts support and shape language development. Developmental Review.
Tomasello (2014). A natural history of human thinking. Cambridge, MA: Harvard University Press.
Yale, Messinger, Cobo-Lewis, & Delgado (2003). The temporal coordination of early infant communication. Developmental Psychology.

E- mail : mitja.nikolaus@univ-amu.fr
Web link :
Curriculum Vitae : CV-MitjaNikolaus

Research Project: Bridging communication in behavioural and neural dynamics

Project Summary :
The aim of this project is to bridge interpersonal verbal coordination and neural dynamics. In practice, we will collect neurophysiological data on individuals (mostly patients with intracranial recordings) performing different interactive language tasks. We will use natural language processing methods to estimate objective features of verbal coordination on speech/language signals. Then we will use machine learning and information theory driven approaches to bridge the dynamics of the coordinative verbal behavior to spatio-temporal neural dynamics.
More precisely, we plan to use several tasks that have been proven to be efficient in the study of verbal interactions. Some tasks are rather constrained and controlled (allowing to manipulate the coordinative dynamics) while others assess conversation in more natural conditions. Speech recordings allow quantifying coordination at different linguistic levels in a time resolved manner. These metrics can then be used to interpret changes in neural dynamics as a function of verbal coordination. We plan to use different approaches, a machine learning approach (decoding the speech signal of the speaker based on the neural signal of the listener) as well as information-theoretic approach (to model to what extent the relation between neural signals and upcoming speech is influenced by the current level of coordination estimated by convergence, for instance).
Overall, this project will allow gathering a better understanding of the link between behavioural coordinative dynamics and neural dynamics. For instance, compared to simple coordinative dynamics, more difficult coordinative behaviour will probably require a change in the ratio between top-down and bottom-up connections between frontal regions and temporal regions in specific frequency bands (increase of top-down beta and decrease of bottom-up gamma).
The strength of this project is to merge sophisticated coordination designs, advanced analysis of verbal coordination dynamics and front edge neuroscience tools with unique neural data in humans.

Research Project: Ouvrir une fenêtre sur l'esprit des lecteurs : Détermination par TMS et EEG du réseau cortical impliqué dans le comportement oculomoteur de lecture

Project Summary :

Les mouvements oculaires pendant la lecture ont été étudiés depuis plus d’un siècle, révélant un comportement très stéréotypé, en dépit même d’une importante variabilité de l’amplitude des saccades et des positions des fixations sur les lignes de texte. La majorité des modèles proposés pour rendre compte de ce comportement repose sur un guidage cognitif du regard, et suppose donc un contrôle essentiellement descendant. Ces modèles descendants sont néanmoins contredits par le fait rapporté récemment qu’un modèle analphabète de programmation des saccades dans le colliculus supérieur, une structure sous-corticale multi-intégrative, prédise assez fidèlement le comportement oculomoteur des lecteurs simplement à partir de traitements visuels précoces effectués dès la rétine (contraste de luminance). Ce résultat suggère au contraire un rôle secondaire du néocortex dans le contrôle oculomoteur pendant la lecture.
La thèse envisagée aura pour but d’une part de caractériser le réseau cortical impliqué dans le contrôle oculomoteur pendant la lecture, et d’autre part de déterminer la dynamique temporelle d’activation de ces différentes aires corticales. Ces recherches reposeront d’abord sur l’utilisation de la stimulation magnétique transcrânienne (TMS), permettant d’inactiver transitoirement une aire corticale donnée chez des participants sains, conjointement à l’enregistrement des mouvements oculaires pendant une tâche de lecture de phrases. L’effet de l’inactivation d’une aire corticale donnée sur les comportements oculomoteurs classiquement observés renseignerait donc de son implication dans la lecture. Dans un second temps, les études TMS seront complétées par une approche basée sur des enregistrements électroencéphalographiques (EEG).

English:

Eye movements during reading have been studied for more than a century, revealing a very stereotyped behaviour, despite a significant variability in the amplitude of saccades and the positions of fixations on the lines of text. Most of the models proposed to account for this behaviour are based on a cognitive guidance of the gaze, and therefore presuppose an essentially top-down control. These top-down models are nevertheless contradicted by the recently reported fact that an illiterate model of saccade programming in the superior colliculus, a multi-integrative subcortical structure, fairly accurately predicts the oculomotor behaviour of readers simply from early visual processing (luminance contrast). This result suggests on the contrary a secondary role of the neocortex in oculomotor control during reading.
The thesis aims on the one hand to characterize the cortical network involved in oculomotor control during reading, and on the other hand to determine the temporal dynamics of activation of these different cortical areas. This research is primarily based on the use of transcranial magnetic stimulation (TMS), which temporarily inactivates a given cortical area in healthy participants, in conjunction with the recording of eye movements during a sentence-reading task. The effect of the inactivation of a given cortical area on the oculomotor behaviours classically observed would therefore indicate its involvement in reading. In a second step, TMS studies will be complemented by an approach based on electroencephalographic (EEG) recordings.

Research Project: Growing and learning with laughter

Project Summary :
Laughter, emerging around 3 months of age, is one of the earliest means that an infant has to convey meaning, practising turn-taking, attention sharing, directing other’s attention and contribute to interaction at the same level of an adult. Through development its use becomes more and more sophisticated both from a semantic and pragmatic perspective, being closely entangled with language production to convey meaning multimodally. Laughter both when occurring in relation to humour or not, can give us important insights into the child’s cognitive, linguistic and pragmatic development on different levels of observation. Nevertheless, there is a dearth of research on the development of laughter, especially in interaction.
The goal of my current project is to deepen our characterization of laughter development longitudinally in itself and in relation to speech, language, humour and pragmatic abilities.
I will conduct longitudinal cross-linguistic corpus studies investigating laughter behaviour in child-caregiver interactions. I will compare the development of laughter behaviour in children with atypical language or pragmatic development with that of typically developing children during the first years of life. The main aim is to test the hypothesis that laughter can be an early bio-marker concerning language and pragmatic development.
In addition to corpus studies, I will run experiments that will contribute to deepening our understanding of laughter perception and interpretation in typical development and in children within the autistic spectrum, as well as shedding light on the role of laughter in pragmatic reasoning and non-literal meaning processing, with a special interest for irony.

Research Project :

Wavelet-based muldimensional characterizaon of brain networks in language tasks

Project Summary :

Research Project :

Multimodal study of functional organization of the Visual Word Form Area and its communication with the spoken language system

Project Summary :

At ILCB, the goal of my research is to investigate 1) the fine-scale spatial organization of functionally segregated neuronal populations within the visual word form area (VWFA) and 2) the activation time course of the VWFA in response to speech as well as the temporal dynamics of the communication between this area and the spoken language network. Broadly, I’m interested in the functional integration and segregation of the language system.

Research Project :

INFORMATION DYNAMIC METRICS TRACK THE EMERGENCE OF COGNITIVE INFORMATION
PROCESSING FROM NEURAL CIRCUIT DYNAMICS

Project Summary :

Cognitive function arises from the coordinated activity of neural populations distributed over largescale brain networks. However, it is challenging to understand how specific aspects of neural dynamics translate into operations of information processing, and, ultimately, cognitive functions. To address this question, we combine novel approaches from information theory with computational simulations of canonical neural circuits, emulating well-defined cognitive functions. Specifically, we simulate circuits composed of one or multiple brain areas, each modeled as a 1D ring network of simple rate units. Despite its simplicity, such model can give rise to rich neuronal dynamics [1]. These models can be used to reproduce functions such bottom-up transfer of stimuli, working memory and even top-down attentional modulation [2].
We then apply recent tools from the Information Dynamics framework to simulated data. Information Dynamics is a novel theoretical approach that formalize the decomposition of generic information processing into “primitive” operations of active storage, transfer and modification of information [3]. In particular, we analyze simulated recordings from our models, quantifying how its nonlinear dynamics implement specific mix of these different primitive processing operations, varying depending on the emulated cognitive function. For instance, we show that the neuronal subsets maintaining sensory representations in working memory (via reverberant self-sustained activity) can be revealed by high values of the active Information Storage metric. Or, the integration of top-down signals (mediated by nonlinear interactions between active sub-populations) is detected by increased values of information modification.
Our models thus highlight transparently the capacity of information dynamics metrics to characterize which network units participate to cognition-related information processing, and how they do it. Such capability can be exploited for the analysis of actual human MEG datasets.
References
1. Roxin, A., Brunel, N., Hansel, D. (2005). Physical Review Letters 94(23), 238103
2. Ardid, S., Wang, X., Compte, A. (2007). J Neurosci 27(32), 222
3. Wibral, M., Priesemann, V., Kay, J., Lizier, J., Phillips, W. (2017). Brain and cognition 112, 25

Research Project :

Breaking the acoustical code of brain by interpreting machine hearing

Project Summary :

I'm a sound and hearing researcher interested to decypher the neurocomputational bases of audition. My researches combine advanced mathematical modeling of sound signals with statistical learning techniques, behavioral testing and neuroinspired techniques in order understand how these processes guide human communication and behaviour.

I'm currently post-doc between the Perception, Representation, Image, Sound, Music lab (PRISM) and the Laboratoire d'Informatique & Systèmes lab (LIS) in Marseille through the Institute of Language Communication & the Brain (ILCB) of Aix-Marseille University. I'm advised by Richard Kronland-Martinet (PRISM) and Valentin Emiya & Stéphane Ayache (LIS).

I'm grateful having been advised by Daniel Pressnitzer & Christian Lorenzi at the Ecole Normale Supérieure de Paris, Stephen McAdams & Philippe Depalle at McGill University in Montreal, and by Sølvi Ystad & Mistuko Aramaki at the CNRS Mechanics and Acoustic Lab in Marseille.

Selected publications :

• Thoret, E., Andrillon, T., Leger, D., Pressnitzer, D. (2020) Probing machine-learning classifiers using noise, bubbles, and reverse correlation, bioRxiv, bioRxiv 2020.06.22.165688, 10.1101/2020.06.22.165688
• Thoret, E., Caramiaux, B., Depalle, P., McAdams, S. (In press) Learning metrics on spectrotemporal modulations reveals the perception of musical instrument timbre, Nature Human Behaviour. 10.1038/s41562-020-00987-5
• Thoret, E., Depalle, P., McAdams, S. (2016) Perceptually salient spectro-temporal modulations for recognition of sustained musical instruments. The Journal of the Acoustical Society of America, 140(6), EL478-EL483. 10.1121/1.4971204
• Thoret, E., Aramaki, M., Kronland-Martinet, R., Velay, J. L., Ystad, S. (2014) From Sound to Shape: Auditory Perception of Drawing Movements, Journal of Experimental Psychology: Human Perception and Performance, 40(3), 983-994.
• Thoret, E., Aramaki, M., Bringoux L., Ystad S., Kronland-Martinet R. (2016) Seeing circles and drawing ellipses: when sound biases reproduction of visual motion. PLoS one, 11(4):e0154475. 10.1371/journal.pone.0154475

Research Project :

Nested cortical sulci organisation models for human and non-human primate inter-species comparisons

Project Summary :

Inter-species comparisons of brain organization between human and non-human primates can provide insights into how uniquely human abilities, such as speech and language, emerged through primate brain evolution. While brain organization can be described in many ways, we focus primarily on cortical folding patterns, or sulci, which are critical landmarks that are strongly tied to the functional and histological features of the brain.

The first goal of my project is to construct the first cortical sulci models that describe the organisation of brain folding patterns (sulci) in four primate species: macaques, baboons, chimpanzees and humans. On the basis of common/homologous sulci, these models allow the registration of individual brains both, within the same species, and across species, for brain comparisons. The second goal of my project is then to apply these models to study how the primate vocal control brain network has changed across the four primate species to understand how speech and language areas emerge in the human brain.