Disenchantment with Emotion Recognition Technologies: A Comparison Between Humans Observers and Automatic Classifiers

ABSTRACT: In the wake of rapid advances in automatic affect analysis, commercial automatic classifiers for facial affect recognition have attracted considerable attention in recent years. While several options now exist to analyse dynamic video data, less is known about the relative performance of these classifiers, in particular when facial expressions are spontaneous rather than posed.

In the study “A performance comparison of eight commercially available automatic classifiers for facial affect recognition” published in Plos One (Dupré et al., 2020), we tested eight commercially available automatic classifiers either stand alone software, SDK or API for emotion recognition. In addition to comparing the recognition performance between classifiers, their results are also compared with the recognition of human observers used as ground truth. A total of 937 videos were sampled from two large databases that conveyed the basic six emotions (happiness, sadness, anger, fear, surprise, and disgust) either in posed (BU-4DFE) or spontaneous (UT-Dallas) form.

Results revealed a recognition advantage for human observers over automatic classification. Among the eight classifiers, there was considerable variance in recognition accuracy ranging from 48% to 62%. Subsequent analyses per type of expression revealed that performance by the two best performing classifiers approximated those of human observers, suggesting high agreement for posed expressions. However, classification accuracy was consistently lower (although above chance level) for spontaneous affective behaviour.

The findings indicate potential shortcomings of existing out-of-the-box classifiers for measuring emotions, and highlight the need for more spontaneous facial databases that can act as a benchmark in the training and testing of automatic emotion recognition systems. Issues regarding the poor generalization capacity of automatic classifiers have recently led to a call for new regulations in the use of affective computing technologies, especially when applied to organizational and decision-making processes.