Salammbo is based on previous work by the PIs & members that investigated speech motor control, adaptation and learning in relation with breathing and limb movements, with contributions to the following topics:

  1. Limb motor control, speech and learning
  2. The interplay between speech production and motion
  3. The interplay between respiration and speech production
  4. The interplay between respiration, motion and speech production

Limb motor control, speech and learning

We previously addressed the link between limb motor control, speech and learning at different levels. In a conceptual review on speech and limb motor control (Grimme, Fuchs, Perrier & Schöner, 2011) we discussed differences and commonalities between limb and speech motor control. This work shows that the two systems share task specificity of the motor goals and the sequential order of targets (in object oriented limb motion). Inter-gestural coordination is a major topic in speech production, but less so in limb control, which may at least partially be influenced by the peculiarities (soft tissue dynamics) of the speech production apparatus and its high speed. In this respect, the two systems differ largely. That may have consequences on planning ahead and coordination of subsequent targets between the two systems.

Rochet-Capellan & Ostry (2011) and Rochet-Capellan et al. (2012) addressed how far speech motor learning is related to training experience and provide evidence for specificity in speech motor learning, as previously observed for limbs. We also used transfer of sensory-motor learning in speech as a window to the nature of the units of speech production (Caudrelier, Schwartz, Perrier, Gerber & Rochet-Capellan, 2018).

Our work on transfer of speech motor adaptation used formants perturbations. As displayed on this figure, when F1 is decreased and F2 increased in speaker's auditory feedback ([e] toward [I]), the speaker changes F1 and F2 in the opposite direction.

In a recent study, we observed that making gestures during anatomy learning improves vocabulary recall, as compared with merely seeing gestures, especially when participants were tested 2-3 days after the learning session (Cherdieu, Palombi, Gerbier, Troccaz & Rochet-Capellan).

Illustration of Cherdieu & al. (2017) methods & results: making gestures help long term retention of a forearm anatomy course as compared to seeing gestures

In a collaborative project with researchers in human-computer interaction, we found adaptation of finger tracking to visual feedback delay on a tactile screen over several days of learning (Cattan, Rochet-Capellan, Perrier, & Bérard, 2017; Cattan, Perrier, Bérard, Gerber & Rochet-Capellan, to appear).

In January 2017, the two PIs organized a winter school on “Speech production and perception: Learning and memory” with Joanne Cleland (University of Strathclyde) and a co-edited book on the topic will be published soon.

The interplay between speech production and motion

Members of the project also worked on mutual adaptation between speech and manual gestures.

Rochet-Capellan, Laboissière, Galván & Schwartz (2008) investigated the coordination between jaw and finger pointing motions in native speakers of Brazilian Portuguese. Bi-directional adaptations were found between the finger and the jaw that aim at making the part of speech that shows (in this case, lexical stress) closer to the part of the gesture that shows (in this case, the apex of the pointing gesture).

Example of setup using Optotrak cameras to track finger and jaw motion in a pointing and naming task in Rochet-Capellan & al. (2008) or Roustan & Dohen. Plot on the right illustrate the finger and jaw trajectory in a pointing alone, naming alone vs. pointing and naming task.

In line with this work, B. Roustan (PhD supervised by M. Dohen) investigated the coordination between manual gestures (pointing, beat, button press) and contrastive focus in native speakers of French. Results show that the adaptation of speech-hand coordination is driven both by motor coupling constraints (grouping according to motor effort requirements) and communicative aims (grouping according to communicative function) (Roustan, 2012 – PhD Thesis; Roustan & Dohen, 2010a, 2010b).

S. Fuchs, K. Stoltmann and U. Reichel (Stoltmann & Fuchs 2017a, 2017b, Fuchs & Reichel, 2016) used a game that children often play (counting out rhymes, like "Eeny, meeny, miny, moe" in English) to study the relation between index finger pointing gestures and number of syllables in the motion capture lab. German and Polish-speaking participants were recorded under different time constraints. A reorganization between speech and arm gestures were expected, because pointing gestures are slower and may not adapt to the fast speech rate. However, within a speaker coordination was kept stable while the exact ratio of number of syllables to pointing gesture differed among speakers.

Set up used in Stoltmann & Fuchs (2017a, 2017b), Fuchs & Reichel (2016) in studying the relationships between the number of syllables and the pointing gestures.

J.-M. Colletta and collaborators investigated spontaneous gestures in narrative tasks in typically developing children and adult speakers of different languages (Colletta et al., 2015; Colletta, Pellenq, & Guidetti, 2010). Similarities in cross-cultural developmental trends were observed: age has a strong effect on language complexity and discourse structure but also on manual gestures that co-develop to more complex narratives.

Gesture can help children with Down syndrome (ComEns project) or other speech issues to learn and produce words, see Dohen (to appear)

Using similar methods, A. Rochet-Capellan and M. Dohen investigated how gestures can help speech production in children with language specific impairment (GesLI project, Dohen et al., 2016) and Down syndrome (ComEns project).

The interplay between respiration and speech production

The PIs of Salammbo have been working on breathing and speech production since 2011, specifically with respect to speech production and planning in read and spontaneous speech (Fuchs, S., Petrone, C., Krivokapić, J., & Hoole, 2013; Rochet-Capellan & Fuchs, 2013) and to turn-taking coordination in dialogue (Rochet-Capellan & Fuchs, 2014).

Example of study investigating the link between breathing and turn-taking in face to face interaction. Both the structure of the breath group and breathing cycle parameters, as well as between-speakers' breathing coordination were analyzed (see Rochet-Capellan & Fuchs, 2014).

Among other results, we found that breathing is a quite flexible and adaptive rhythm tightly coupled to speech production. No overall coordination was observed in breathing cycles among the interlocutors in dialogue, but systematic coordinative patterns were observed in turn taking and turn holding.

In recent projects we additionally addressed whether the respiratory kinematics during exhalation are the driving forces for a common intonation feature: f0 declination. For interpausal units we did not find evidence of a correlation between the slope of f0 declination and the slope of respiratory volume changes (Fuchs, Petrone, Rochet-Capellan, Reichel, & Koenig, 2015). For words under focus, we found a strong relation between respiration and speech intensity, but again no clear correlation with f0 (Petrone, Fuchs & Koenig 2017).

Example of study investigating the link between exhalation profiles and F0 declination (Fuchs, Petrone, Rochet-Capellan, Reichel, & Koenig, 2015).

Preliminary work on the interaction between limb motion, respiration and speech

We recorded acoustical signals, breathing kinematics and limb movements of participants while speaking only, biking only, or speaking and biking at the same time with various biking efforts. When speaking and biking, subjects were not able to ventilate as fast as when they biked alone and not as slowly as when they only spoke.

Motion and breathing tracking setup in preliminary work and first results about the adaptation of speech breathing to physical effort.

These first results are interpreted with respect to a shared resource (respiration) and conflicting activities (speech versus aerobic motion) acting on this shared resource. When the physical task was more demanding (Be and SBe), breathing frequency increased in comparison to the less demanding condition (B, SB). Speakers consistently compensated for the shorter breathing cycles with an increase in speech rate. They also elevated loudness and f0 (Fuchs, Reichel & Rochet-Capellan, 2015).

Work done with this first corpus helped us to solve a number of recording and analysis issues. Additionally, we ran a pilot study with a mini-bike that will allow us to compare arm and leg movements. This work provides a basis for Salammbo.

Cited references

  • Cattan, E., Rochet-Capellan, A., Perrier, P., & Bérard, F. (2017, May). Does Practice Make Perfect?. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 5619-5629). ACM.
  • Cattan, E., Perrier, P., & Bérard, F., Gerber, S. & Rochet-Capellan, A., (2018). Adaptation to visual feedback delays on touchscreens with hand vision, Experimental Brain Research, 1-11.
  • Caudrelier, T., Schwartz, J. L., Perrier, P., Gerber, S., & Rochet-Capellan, A.(2018). Transfer of Learning: What Does It Tell Us About Speech Production Units?. Journal of Speech, Language, and Hearing Research, 1-13.
  • Cherdieu, M., Palombi, O., Gerber, S., Troccaz, J., & Rochet-Capellan, A. (2017). Make Gestures to Learn: Reproducing Gestures Improves the Learning of Anatomical Knowledge More than Just Seeing Gestures. Frontiers in psychology, 8, 1689.
  • Colletta, J. M., Guidetti, M., Capirci, O., Cristilli, C., Demir, O. E., Kunene-Nicolas, R. N., & Levine, S. (2015). Effects of age and language on co-speech gesture production: an investigation of French, American, and Italian children's narratives. Journal of child language, 42(01), 122-145.
  • Colletta, J. M., Pellenq, C., & Guidetti, M. (2010). Age-related changes in co-speech gesture and narrative: Evidence from French children and adults. Speech Communication, 52(6), 565-576.
  • Fuchs, S., Petrone, C., Krivokapić, J., & Hoole, P. (2013). Acoustic and respiratory evidence for utterance planning in German. Journal of Phonetics, 41(1), 29-47.
  • Fuchs, S., Petrone, C., Rochet-Capellan, A., Reichel, U. D., & Koenig, L. L. (2015). Assessing respiratory contributions to f0 declination in German across varying speech tasks and respiratory demands. Journal of Phonetics, 52, 35-45.
  • Fuchs, S., Reichel, U. D., & Rochet-Capellan, A. (2015). Changes in speech and breathing rate while speaking and biking. In Proceedings of the ICPhS, Glasgow.
  • Fuchs, S. & Reichel, U.D. (2016). On the relationship between pointing gestures and speech production in German counting out rhymes: Evidence from motion capture data and speech acoustics. In Christoph Draxler & Felicitas Kleber (eds.), Proceedings of P&P 12, 1-4. München: LMU .
  • Grimme, B., Fuchs, S., Perrier, P., & Schöner, G. (2011). Limb versus speech motor control: A conceptual review. Motor control, 15(1), 5-33.
  • Petrone, C., Fuchs, S. & Koenig, L.L. (2017). . Relations among subglottal pressure, breathing, and acoustic parameters of sentence level prominence in German. The Journal of the Acoustical Society of America, 141, 1715–1725.
  • Rochet-Capellan, A., Bailly, G., & Fuchs, S. (2014). Is breathing sensitive to the communication partner? In Proceedings of the 7th Speech Prosody Conference (pp. 613-618).
  • Rochet-Capellan, A., & Fuchs, S. (2014). Take a breath and take the turn: how breathing meets turns in spontaneous dialogue. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 369(1658), 20130399.
  • Rochet-Capellan, A., & Fuchs, S. (2013). The interplay of linguistic structure and breathing in German spontaneous speech. In Proceedings of the 14th Interspeech, p. 1228ff.
  • Rochet-Capellan, A., Laboissière, R., Galván, A., & Schwartz, J. L. (2008). The speech focus position effect on jaw–finger coordination in a pointing task. Journal of Speech, Language, and Hearing Research, 51(6), 1507-1521.
  • Rochet-Capellan, A., & Ostry, D. J. (2011). Simultaneous acquisition of multiple auditory–motor transformations in speech. The Journal of Neuroscience, 31(7), 2657-2662.
  • Rochet-Capellan, A., Richer, L., & Ostry, D. J. (2012). Nonhomogeneous transfer reveals specificity in speech motor learning. Journal of neurophysiology, 107(6), 1711-1717.
  • Roustan, B. (2012). Étude de la coordination gestes manuels/parole dans le cadre de la désignation (Doctoral dissertation, Grenoble).
  • Roustan, B., & Dohen, M. (2010b). Gesture and Speech Coordination: The Influence of the Relationship Between Manual Gesture and Speech. In Interspeech 2010.
  • Roustan, B., & Dohen, M. (2010a). Co-production of contrastive prosodic focus and manual gestures: temporal coordination and effects on the acoustic and articulatory correlates of focus. In Speech Prosody 2010
  • Stoltmann, K. & Fuchs, S. (2017a). Syllable-pointing gesture coordination in Polish counting out rhymes: The effect of speech rate. In Silvia Bonacchi & Maciej Karpiński (eds.), Journal of Multimodal Communication Studies 4 (1-2): Special issue: Gesture and Speech in Interaction, 63-68.
  • Stoltmann, K. & Fuchs, S. (2017b). The influence of handedness and pointing direction on deictic gestures and speech interaction: Evidence from motion capture data on Polish counting-out rhymes. In Ouni, Slim, Chris Davis, Alexandra Jesse & Jonas Beskow (eds.), Proceedings of the 14th International Conference on Audio-Visual Speech Processing (AVSP2017). S1 – Gaze and Handedness, D1.S1.4 (5 Seiten). Stockholm: KTH.