Knowledge Base

How Does It Work?

Unlike video, where each frame can be analyzed independently, audio-based emotion recognition requires a time window to extract patterns. A single moment of sound lacks information to determine emotional state, as vocal features such as pitch, loudness, and speech rate can only be measured over time.

To account for this, each video frame is analyzed alongside the preceding second of audio, meaning that voice analysis begins only after enough audio has been processed to detect a signal.

Detecting Voice Activity

Voice analysis in FaceReader employs Voice Activity Detection (VAD) based on a Gaussian Mixture Model (GMM) to filter out audio that clearly falls outside the typical range of human speech. Although it does not explicitly classify segments as speech or non-speech, this approach helps reduce the impact of background noise and other irrelevant sounds.

While GMM-based VAD is generally reliable under typical conditions, it is not immune to misclassifications, particularly in environments with overlapping speech, high levels of ambient noise, or non-speech sounds within the speech frequency range. When a valid voice signal is detected, the following vocal measures will be calculated and reported by FaceReader.

Emotions

Emotions are estimated based on a deep learning model that analyzes a broad set of vocal and audio features. These features are extracted from diverse datasets covering multiple languages and include both natural and acted speech to reflect a wide range of expressive variations. The model accounts for differences in gender, ethnicity, and speaking style, aiming to provide a robust estimation of the expressed emotional state.

The model focuses on prosodic features such as pitch, loudness, speech rate, and intonation, rather than the semantic content of speech. This approach is less dependent on the spoken language, as it emphasizes how something is said rather than the specific words used. However, because the model was trained primarily on English-language data, its performance is highest for English speakers.

Valence

Valence represents the emotional positivity or negativity of speech and is derived from speech emotion recognition.

• Positive emotions, such as happiness, contribute to a higher valence score.
• Negative emotions, including anger and sadness, result in a lower valence score.

Valence is normalized to a scale from –1 to 1, where –1 indicates strong negative affect, 0 is neutral, and 1 reflects strong positive affect.

Arousal

Arousal is estimated using both loudness and speech rate, which are also included as individual vocal measures in FaceReader. Higher loudness and faster speech are typically associated with excitement or stress, while lower loudness and slower speech indicate calmness or disengagement.

Arousal is normalized to a 0 to 1 scale, with 0.5 being average arousal. When interpreted alongside valence, arousal helps differentiate between emotions that share a similar emotional direction but vary in intensity. For example, sadness (low arousal) versus anger (high arousal), both of which have negative valence.

Loudness

Loudness is a custom volume-independent measure of sound intensity. Variations in microphone sensitivity and speaker positioning can significantly affect the absolute volume of a recording.

To address this, the loudness measure is computed using a continuously updated, normalized audio signal. For each frame, the system analyzes the past second of the audio signal. Within this one-second window, the audio is normalized to a maximum absolute amplitude of 1.0.

This ensures that the measurement reflects the speaker's vocal dynamics — how energetically a person is speaking within their own dynamic range, rather than the recording conditions.

Speech Rate

The speech rate measure estimates how quickly a person is speaking. The algorithm detects peaks in the audio signal, typically corresponding to syllables, by identifying bursts of vocal energy.

These peaks are counted over the past one-second window of audio, using an upper bound of 160 syllables per minute, which is suitable for capturing various emotional states. The count is normalized to a value between 0 and 1. A higher value indicates faster speech, while a lower value reflects slower or more deliberate speaking.

Calibration

Voice data is not automatically calibrated during acquisition. However, in the Project Analysis window, you can apply baseline correction to your voice data. This works by normalizing each voice feature based on its average across the selected samples (e.g., within a subject, session, or stimulus condition, depending on the analysis context).

The result is a mean-centered dataset where inter-subject or inter-session comparisons are more meaningful. This approach is particularly useful in group analyses where absolute voice feature values are less informative than relative changes or patterns.

Source: EthoVision XT 18 - THC - Trial and Hardware Control, Noldus Information Technology