Research on Extracting Facial Image for Bimodal Emotion Recognition Based on Speech Signal and Facial Expression

On bimodal emotion recognition based on speech signal and facial expression, a method is proposed to extract the corresponding image farme combining speech signal frame on the timeline. The method overcomes the problem which the key frame of static facial expressional image can't be pinpointed as the recognized object on video. It is also use to combine nonlinear features of speech signal to conduct the study of emotional recognition. Firstly, linear and nonlinear features of the speech signal are extracted as the speech emotion feature. After pre-processing speech signal frames and adds window according to the overall frames of video, the mean of the absolute value of all framing signal sampling points' amplitude is computed, and ten images,which correspond to ten largest absolute values of speech signal, can be selected. And then, ten images' feature vectors are extracted and their mean is computed as last feature vector of facial expression of video. Finally, the support vector machine classifier is used to recognize emotion through feature-level fusion.Experiments show that the extracted facial features can better reflect the emotional state, and the recognition effect is achieved better though fusion of speech signal features.