Facial Expression Classification using Visual Cues and Language

Abstract:

In this work, we attempt to tackle the problem of correlating language with facial expressions to learn in an unsupervised manner, the adjectives used to describe emotions. The problem is divided into two parts - i) a supervised method for classifying facial emotions using visual cues and ii) an unsupervised algorithm to extract keywords out of commentary on videos depicting facial expressions. We use a method based on Gabor filters and Support Vector Machines to classify emotions into 7 categories - Anger, Surprise, Sadness, Happiness, Disgust, Fear and Neutral. We explore various dimensionality reduction and feature selection methods like PCA and AdaBoost. The Extended Cohn-Kanade database is used for testing the algorithms. We achieve an accuracy of 94.72% for a 7-way forced choice SVM classifier after feature selection using Adaboost which is a significant improvement over many previous successful approach based on PCA and LDA and Local Binary Patterns. In the next step, we obtain commentary on 40 videos depicting 4 emotions - Anger, Sadness, Happiness and Surprise and cluster keywords obtained using a maximum co-occurrence method to discover descriptors for these emotions.

Figure 1: Gabor filters applied to an image in the CK+ dataset


Figure 2: Results from our algorithm


Figure 3: Accuracies compared to previous methods for facial expression recognition

---

---