A Comprehensive Investigation on Emotional Detection in Deep Learning

Authors

  • Anand M  Research Scholar, Department of Computing Technologies, SRM Institute of Science & Technology, Kattankulathur, Tamil Nadu, India
  • Dr. S. Babu  Associate Professor, Department of Computing Technologies, SRM Institute of Science & Technology, Kattankulathur, Tamil Nadu, India

DOI:

https://doi.org//10.32628/CSEIT228111

Keywords:

Deep Learning, Emotional, Facial Expression

Abstract

Emotion recognition is a substantial problem in the field of Big Data. In a wide range of applications, reliable categorization, analysis, and interpretation of emotional content is greatly desired. In this paper, we look at how Deep Learning models perform on an emotion perception test. Facial Expression Recognition (FER) plays an important role in machine learning tasks. Deep Learning models do well in FER tasks, but they lack explanation for their conclusions. Based on the notion that facial expression is a mixture of facial muscle movements, we discover a link between Facial Action Coding Units (AUs) and Emotion label in the CK+ Dataset. In this study, we offer a model that uses AUs to explain the classification outcomes of a Convolutional Neural Network (CNN) model. The CNN model is trained using the CK+ Dataset and identifies emotions using extracted characteristics. The CNN model's retrieved features and emotion classes are used by the explanation model to classify multiple AUs. Explanation model creates AUs quite effectively with only characteristics and emotion classes acquired from the CNN model, according to our trial. Experimental research was constructed, and several deep learning approaches were tested using publically available datasets. The findings are highly intriguing, highlighting the benefits of each strategy and training style studied.

References

  1. HAOPENG WANG, DIANA P. TOBÓN V, M. SHAMIM HOSSAIN, (Senior Member, IEEE), AND ABDULMOTALEB EL SADDIK “Deep Learning (DL)-Enabled System for Emotional Big Data” VOLUME 9, 2021.
  2. C.-N. Anagnostopoulos, T. Iliou, et I. Giannoukos, « Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011 », Artif. Intell. Rev., vol. 43, no 2, p. 155‑177, févr. 2015, doi: 10.1007/s10462-012-9368-5.
  3. L. Shu et al., « A Review of Emotion Recognition Using Physiological Signals », Sensors, vol. 18, no 7, p. 2074, juill. 2018, doi: 10.3390/s18072074.
  4. C. Marechal et al., « Survey on AI-Based Multimodal Methods for Emotion Detection », in High-Performance Modelling and Simulation for Big Data Applications: Selected Results of the COST Action IC1406 cHiPSet, J. Kołodziej et H. González-Vélez, Éd. Cham: Springer International Publishing, 2019, p. 307‑324.
  5. M. H. Alkawaz, D. Mohamad, A. H. Basori, et T. Saba, « Blend Shape Interpolation and FACS for Realistic Avatar », 3D Res., vol. 6, no 1, p. 6, janv. 2015, doi: 10.1007/s13319-015-0038-7.
  6. P. V. Rouast, M. Adam, et R. Chiong, « Deep Learning for Human Affect Recognition: Insights and New Developments », IEEE Trans. Affect. Comput., p. 1‑1, 2018, doi: 10.1109/TAFFC.2018.2890471.
  7. C. Shan, S. Gong, et P. W. McOwan, « Facial expression recognition based on Local Binary Patterns: A comprehensive study », Image Vis. Comput., vol. 27, no 6, p. 803‑816, mai 2009, doi: 10.1016/j.imavis.2008.08.005.
  8. T. Jabid, M. H. Kabir, et O. Chae, « Robust Facial Expression Recognition Based on Local Directional Pattern », ETRI J., vol. 32, no 5, p. 784‑794, 2010, doi: 10.4218/etrij.10.1510.0132.
  9. S. Zhang, L. Li, et Z. Zhao, « Facial expression recognition based on Gabor wavelets and sparse representation », in 2012 IEEE 11th International Conference on Signal Processing, oct. 2012, vol. 2, p. 816‑819, doi: 10.1109/ICoSP.2012.6491706.
  10. R. Gross, I. Matthews, J. Cohn, T. Kanade, et S. Baker, « Multi-PIE », Proc. Int. Conf. Autom. Face Gesture Recognit. Int. Conf. Autom. Face Gesture Recognit., vol. 28, no 5, p. 807‑813, mai 2010, doi: 10.1016/j.imavis.2009.08.002.
  11. M. Pantic, M. Valstar, R. Rademaker, et L. Maat, « Web-based database for facial expression analysis », in 2005 IEEE International Conference on Multimedia and Expo, juill. 2005, p. 5 pp.-, doi: 10.1109/ICME.2005.1521424.
  12. M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, et K. Scherer, « The first facial expression recognition and analysis challenge », in Face and Gesture 2011, mars 2011, p. 921‑926, doi: 10.1109/FG.2011.5771374.
  13. A. Dhall, R. Goecke, S. Lucey, et T. Gedeon, « Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark », in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), nov. 2011, p. 2106‑2112, doi: 10.1109/ICCVW.2011.6130508.
  14. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, et I. Matthews, « The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression », in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, juin 2010, p. 94‑101, doi: 10.1109/CVPRW.2010.5543262.
  15. I. J. Goodfellow et al., « Challenges in Representation Learning: A Report on Three Machine Learning Contests », in Neural Information Processing, Berlin, Heidelberg, 2013, p. 117‑124, doi: 10.1007/978-3-642 -42051-1_16.
  16. M. Lyons, M. Kamachi, et J. Gyoba, « The Japanese Female Facial Expression (JAFFE) Database ». Zenodo, avr. 14, 1998, doi: 10.5281/zenodo.3451524.
  17. Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, et M. J. Rosato, « A 3D facial expression database for facial behavior research », in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), avr. 2006, p. 211‑216, doi: 10.1109/FGR.2006.6.
  18. W.-J. Yan et al., « CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation », PLoS ONE, vol. 9, no 1, janv. 2014, doi: 10.1371/journal.pone.0086041.
  19. G. Zhao, X. Huang, M. Taini, S. Z. Li, et M. Pietikäinen, « Facial expression recognition from near-infrared videos », Image Vis. Comput., vol. 29, no 9, p. 607‑619, août 2011, doi: 10.1016/j.imavis.2011.07.002.
  20. A. Mollahosseini, B. Hasani, et M. H. Mahoor, « AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild », IEEE Trans. Affect. Comput., vol. 10, no 1, p. 18‑31, janv. 2019, doi: 10.1109/TAFFC.2017.2740923.
  21. S. Li, W. Deng, et J. Du, « Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild », 2017, p. 2852‑2861.
  22. O. Langner, R. Dotsch, G. Bijlstra, D. H. J. Wigboldus, S. T. Hawk, et A. van Knippenberg, « Presentation and validation of the Radboud Faces Database », Cogn. Emot., vol. 24, no 8, p. 1377‑1388, déc. 2010, doi: 10.1080/02699930903485076.
  23. A. Mollahosseini, D. Chan, et M. H. Mahoor, « Going deeper in facial expression recognition using deep neural networks », in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), mars 2016, p. 1‑10, doi: 10.1109/WACV.2016.7477450.
  24. A. T. Lopes, E. de Aguiar, A. F. De Souza, et T. Oliveira-Santos, « Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order », Pattern Recognit., vol. 61, p. 610‑628, janv. 2017, doi: 10.1016/j.patcog.2016.07.026.
  25. M. Mohammadpour, H. Khaliliardali, S. M. R. Hashemi, et M. M. AlyanNezhadi, « Facial emotion recognition using deep convolutional networks », in 2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI), déc. 2017, p. 0017‑0021, doi: 10.1109/KBEI.2017.8324974.
  26. J. Cai, O. Chang, X. Tang, C. Xue, et C. Wei, « Facial Expression Recognition Method Based on Sparse Batch Normalization CNN », in 2018 37th Chinese Control Conference (CCC), juill. 2018, p. 9608‑9613, doi: 10.23919/ChiCC.2018.8483567.
  27. Y. Li, J. Zeng, S. Shan, et X. Chen, « Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism », IEEE Trans. Image Process., vol. 28, no 5, p. 2439‑2450, mai 2019, doi: 10.1109/TIP.2018.2886767.
  28. G. Yolcu et al., « Facial expression recognition for monitoring neurological disorders based on convolutional neural network », Multimed. Tools Appl., vol. 78, no 22, p. 31581‑31603, nov. 2019, doi: 10.1007/s11042-019-07959-6.
  29. A. Agrawal et N. Mittal, « Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy », Vis. Comput., janv. 2019, doi: 10.1007/s00371-019-01630-9.
  30. D. K. Jain, P. Shamsolmoali, et P. Sehdev, « Extended deep neural network for facial emotion recognition », Pattern Recognit. Lett., vol. 120, p. 69‑74, avr. 2019, doi: 10.1016/j.patrec.2019.01.008.
  31. D. H. Kim, W. J. Baddar, J. Jang, et Y. M. Ro, « Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition », IEEE Trans. Affect. Comput., vol. 10, no 2, p. 223‑236, avr. 2019, doi: 10.1109/TAFFC.2017.2695999.
  32. Z. Yu, G. Liu, Q. Liu, et J. Deng, « Spatio-temporal convolutional features with nested LSTM for facial expression recognition », Neurocomputing, vol. 317, p. 50‑57, nov. 2018, doi: 10.1016/j.neucom.2018.07.028.
  33. D. Liang, H. Liang, Z. Yu, et Y. Zhang, « Deep convolutional BiLSTM fusion network for facial expression recognition », Vis. Comput., vol. 36, no 3, p. 499‑508, mars 2020, doi: 10.1007/s00371-019-01636-3.
  34. S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, et J. F. Cohn, « DISFA: A Spontaneous Facial Action Intensity Database », IEEE Trans. Affect. Comput., vol. 4, no 2, p. 151‑160, avr. 2013, doi: 10.1109/T-AFFC.2013.4.
  35. M. F. Valstar et al., « FERA 2015 - second Facial Expression Recognition and Analysis challenge », in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), mai 2015, vol. 06, p. 1‑8, doi: 10.1109/FG.2015.7284874.
  36. M. Pantic et L. J. M. Rothkrantz, « Toward an affect-sensitive multimodal human-computer interaction », Proc. IEEE, vol. 91, no 9, p. 1370‑1390, sept. 2003, doi: 10.1109/JPROC.2003.817122.
  37. S. Zhang, S. Zhang, T. Huang, et W. Gao, « Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition », in Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, New York, NY, USA, 2016, p. 281–284, doi: 10 .1145/2911996.2912051.
  38. F. Ringeval et al., « Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data », Pattern Recognit. Lett., vol. 66, p. 22‑30, nov. 2015, doi: 10.1016/j.patrec.2014.11.007.
  39. S. Srivastava, P. Gupta and P. Kumar, "Emotion Recognition Based Emoji Retrieval Using Deep Learning," 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), 2021, pp. 1182-1186, doi: 10.1109/ICOEI51242.2021.9452832.
  40. R. R. Subramanian, C. S. Niharika, D. U. Rani, P. Pavani and K. P. L. Syamala, "Design and Evaluation of a Deep Learning Algorithm for Emotion Recognition," 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), 2021, pp. 984-988, doi: 10.1109/ICICCS51141.2021.9432336.
  41. A. Poulose, C. S. Reddy, J. H. Kim and D. S. Han, "Foreground Extraction Based Facial Emotion Recognition Using Deep Learning Xception Model," 2021 Twelfth International Conference on Ubiquitous and Future Networks (ICUFN), 2021, pp. 356-360, doi: 10.1109/ICUFN49451.2021.9528706.
  42. H. Lee and K. Hong, "A study on emotion recognition method and its application using face image," 2017 International Conference on Information and Communication Technology Convergence (ICTC), 2017, pp. 370-372, doi: 10.1109/ICTC.2017.8191005.
  43. N. Naik and M. A. Mehta, "Hand-over-Face Gesture based Facial Emotion Recognition using Deep Learning," 2018 International Conference on Circuits and Systems in Digital Enterprise Technology (ICCSDET), 2018, pp. 1-7, doi: 10.1109/ICCSDET.2018.8821186.
  44. T. L. Praveena and N. V. M. Lakshmi, "Multi Label Classification for Emotion Analysis of Autism Spectrum Disorder Children using Deep Neural Networks," 2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA), 2021, pp. 1018-1022, doi: 10.1109/ICIRCA51532.2021.9545073.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2022-02-28

Issue

Volume 8, Issue 1, January-February-2022

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
Anand M, Dr. S. Babu, " A Comprehensive Investigation on Emotional Detection in Deep Learning, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 8, Issue 1, pp.115-122, January-February-2022. Available at doi : https://doi.org/10.32628/CSEIT228111