Multiple Instances Based Emotion Detection Using Discriminant Feature Tracking

Authors

  • Abhishek Kilak  Malaviya National Institute of Technology, Jaipur, Rajasthan, India
  • Namita Mittal  Malaviya National Institute of Technology, Jaipur, Rajasthan, India

Keywords:

Emotion detection, facial expression, emotion recognition, facial images, Indian faces.

Abstract

Automatic recognition of emotions from the facial expressions continues to be an important aspect in the field of evolution of new age computing systems. Emotion detection capable computer systems is an ongoing research field that has a numerous applications ranging from recommender systems, human-computer interaction, robotics and affective systems. The various features that increase the complexity of emotion recognition systems include ethnicity, gender, pose, occlusion, beard, moustache etc. The type of database used for learning by systems is of crucial importance. Many databases exist for this purpose but none of them is for posed Indian faces. In this paper,we bridge this gap by providing Bharat Database which contains facial images of Indian people. The database has posed expressions of 102 participants and has 896 images. The participants were asked to pose for different emotions by showing them images eliciting those emotions as well as with the help of expert artists. The annotation was done using polling by a panel of three experts.Experiments were conducted on the database using different algorithms and results are presented for reference. This database will further help the community involved in developing of algorithms for emotion recognition. In this paper we propose two emotion detection approaches, the first one is based on Compact Local Binary Pattern and is used for construction of hybrid features which increases emotion detection accuracy. Second approach is Enhanced Feature Extraction using multiple Patches face on images from indigenously developed Bharat Database of Indian Faces, Japanese Female Facial Expression Database and Karolinska Directed Emotional Faces. The results show its applicability for construction of emotion detection systems.

References

  1. Ekman, Paul. "An argument for basic emotions." Cognition & emotion 6, no. 3-4 (1992): 169-200.
  2. Lucey, Patrick, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression." In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, pp. 94-101. IEEE, 2010.
  3. Lyons, Michael J., Shigeru Akamatsu, Miyuki Kamachi, Jiro Gyoba, and Julien Budynek. "The Japanese female facial expression (JAFFE) database." In Proceedings of third international conference on automatic face and gesture recognition, pp. 14-16. 1998.
  4. Yin, Lijun, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J. Rosato. "A 3D facial expression database for facial behavior research." In Au- tomatic face and gesture recognition, 2006. FGR 2006. 7th international conference on, pp. 211-216. IEEE, 2006.
  5. Valstar, Michel, and Maja Pantic. "Induced disgust, happiness and surprise: an addition to the mmi facial expression database." In Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, p. 65. 2010.
  6. Sneddon, Ian, Margaret McRorie, Gary McKeown, and Jennifer Hanratty. "The belfast induced natural emotion database." IEEE Transactions on Affective Computing 3, no. 1 (2012): 32-41.
  7. Happy, S. L., Priyadarshi Patnaik, Aurobinda Routray, and Rajlakshmi Guha. "The Indian Spontaneous Expression Database for Emotion Recog- nition." IEEE Transactions on Affective Computing 8, no. 1 (2017): 131-142.
  8. Mavadati, S. Mohammad, Mohammad H. Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F. Cohn. "Disfa: A spontaneous facial action intensity database." IEEE Transactions on Affective Computing 4, no. 2 (2013): 151-160.
  9. Viola, Paul, and Michael Jones. "Rapid object detection using a boosted cascade of simple features." In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, pp. I-I. IEEE, 2001.
  10. Dalal, Navneet, and Bill Triggs. "Histograms of oriented gradients for human detection." In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, pp. 886-893. IEEE, 2005.
  11. Jolliffe, Ian. Principal component analysis.John Wiley & Sons, Ltd, 2002. 12Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24, no. 7 (2002): 971-987.
  12. Zhao, Guoying, and Matti Pietikainen. "Dynamic texture recognition using local binary patterns with an application to facial expressions." IEEE transactions on pattern analysis and machine intelligence 29, no. 6 (2007): 915-928.
  13. Rudovic, Ognjen, Maja Pantic, and Ioannis Patras. "Coupled Gaussian processes for pose-invariant facial expression recognition." IEEE trans- actions on pattern analysis and machine intelligence 35, no. 6 (2013): 1357-1369.
  14. Tong, Yan, Jixu Chen, and Qiang Ji. "A unified probabilistic framework for spontaneous facial action modeling and understanding." IEEE trans- actions on pattern analysis and machine intelligence 32, no. 2 (2010): 258-273.
  15. Ojansivu, Ville, and Janne HeikkilAd'. "Blur insensitive texture classi- fication using local phase quantization." In International conference on image and signal processing, pp. 236-243. Springer Berlin Heidelberg, 2008.
  16. Dalal, Navneet, and Bill Triggs. "Histograms of oriented gradients for human detection." In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, pp. 886-893. IEEE, 2005.
  17. Huang, Di, Caifeng Shan, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. "Local binary patterns and its application to facial image analysis: a survey." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41, no. 6 (2011): 765-781.
  18. Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24, no. 7 (2002): 971-987.
  19. Valstar, Michel F., Bihan Jiang, Marc Mehu, Maja Pantic, and Klaus Scherer. "The first facial expression recognition and analysis challenge." In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, pp. 921-926. IEEE, 2011.
  20. Jiang, Bihan, Michel F. Valstar, and Maja Pantic. "Action unit detection using sparse appearance descriptors in space-time video volumes." In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, pp. 314-321. IEEE, 2011.
  21. Ahonen, Timo, Abdenour Hadid, and Matti Pietikainen. "Face descrip- tion with local binary patterns: Application to face recognition." IEEE transactions on pattern analysis and machine intelligence 28, no. 12 (2006): 2037-2041.
  22. Zhao, Guoying, and Matti Pietikainen. "Dynamic texture recognition using local binary patterns with an application to facial expressions." IEEE transactions on pattern analysis and machine intelligence 29, no. 6 (2007): 915-928.
  23. Taheri, Sima, Qiang Qiu, and Rama Chellappa. "Structure-preserving sparse decomposition for facial expression analysis." IEEE Transactions on Image Processing 23, no. 8 (2014): 3590-3603.
  24. Shan, Caifeng, Shaogang Gong, and Peter W. McOwan. "Facial expres-sion recognition based on local binary patterns: A comprehensive study." Image and Vision Computing 27, no. 6 (2009): 803-816.
  25. Lv, Hai-Rong, Zhong-Lin Lin, Wen-Jun Yin, and Jin Dong. "Emotion recognition based on pressure sensor keyboards." In Multimedia and Expo, 2008 IEEE International Conference on, pp. 1089-1092. IEEE, 2008.
  26. Soleymani, Mohammad, Sadjad Asghari-Esfeden, Yun Fu, and Maja Pantic. "Analysis of EEG signals and facial expressions for continuous emotion detection." IEEE Transactions on Affective Computing 7, no. 1 (2016): 17-28.
  27. Matlovic, Tomas, Peter Gaspar, Robert Moro, Jakub Simko, and Maria Bielikova. "Emotions detection using facial expressions recognition and EEG." In Semantic and Social Media Adaptation and Personalization (SMAP), 2016 11th International Workshop on, pp. 18-23. IEEE, 2016.
  28. Kim, Kyung Hwan, Seok Won Bang, and Sang Ryong Kim. "Emotion recognition system using short-term monitoring of physiological signals." Medical and biological engineering and computing 42, no. 3 (2004): 419- 427.
  29. Boiten, Frans A., Nico H. Frijda, and Cornelis JE Wientjes. "Emotions and respiratory patterns: review and critical analysis." International Journal of Psychophysiology 17, no. 2 (1994): 103-128.
  30. Glowinski, Donald, Antonio Camurri, Gualtiero Volpe, Nele Dael, and Klaus Scherer. "Technique for automatic emotion recognition by body ges- ture analysis." In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW'08. IEEE Computer Society Conference on, pp. 1-6. IEEE, 2008.
  31. Castellano, Ginevra, Santiago D. Villalba, and Antonio Camurri. "Recog- nising human emotions from body movement and gesture dynamics." In International Conference on Affective Computing and Intelligent Interac- tion, pp. 71-82. Springer, Berlin, Heidelberg, 2007.
  32. Tian, Yingli, Takeo Kanade, and Jeffrey F. Cohn. "Facial expression recognition." In Handbook of face recognition, pp. 487-519. Springer London, 2011.
  33. Bowyer, Kevin W., Kyong Chang, and Patrick Flynn. "A survey of ap- proaches and challenges in 3D and multi-modal 3D+ 2D face recognition." Computer vision and image understanding 101, no. 1 (2006): 1-15.
  34. Jeni, LAa˛szlAs¸ A., Jeffrey F. Cohn, and Takeo Kanade. "Dense 3D face alignment from 2D video for real-time use." Image and Vision Computing 58 (2017): 13-24.
  35. Kamarol, Siti Khairuni Amalina, Mohamed Hisham Jaward, Heikki KAd'lviAd'inen, Jussi Parkkinen, and Rajendran Parthiban. "Joint facial expression recognition and intensity estimation based on weighted votes of image sequences." Pattern Recognition Letters92 (2017): 25-32.
  36. Valstar, Michel F., Enrique SAa˛nchez-Lozano, Jeffrey F. Cohn, Laszlo A. Jeni, Jeffrey M. Girard, Zheng Zhang, Lijun Yin, and Maja Pantic. " FERA 2017-Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge”. arXiv preprint arXiv:1702.0417 (2017).
  37. Battocchi, Alberto, Fabio Pianesi, and Dina Goren-Bar. "A first eval- uation study of a database of kinetic facial expressions (dafex)." In Proceedings of the 7th international conference on Multimodal interfaces, pp. 214-221. ACM, 2005.
  38. Baron-Cohen, Simon. "The Eyes as Window to the Mind." The American journal of psychiatry 174, no. 1 (2017): 1-2.
  39. Shen, Xiaolu, Xuetao Feng, Jungbae Kim, Hui Zhang, Youngkyoo Hwang, and Ji-yeun Kim. "An extensible framework for facial motion tracking." In Consumer Electronics (ICCE), 2013 IEEE International Conference on, pp. 288-291. IEEE, 2013.
  40. Viola, Paul, and Michael Jones. "Rapid object detection using a boosted cascade of simple features." In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, pp. I-I. IEEE, 2001.
  41. Bashyal, Shishir, and Ganesh K. Venayagamoorthy. "Recognition of facial expressions using Gabor wavelets and learning vector quantization." Engineering Applications of Artificial Intelligence 21, no. 7 (2008): 1056- 1064.
  42. Chen, Ying, Chunjian Hua, and Ruilin Bai. "Sequentially adaptive active appearance model with regression-based online reference appearance template." Journal of Visual Communication and Image Representation 35 (2016): 198-208.
  43. Wang, Jun, Lijun Yin, Xiaozhou Wei, and Yi Sun. "3D facial expression recognition based on primitive surface feature distribution." In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2, pp. 1399-1406. IEEE, 2006.
  44. Lorincz, Andras, Laszlo Jeni, Zoltan Szabo, Jeffrey Cohn, and Takeo Kanade. "Emotional expression classification using time-series kernels." In Proceedings of the IEEE Conference on computer vision and pattern recognition workshops, pp. 889-895. 2013.
  45. Sariyanidi, Evangelos, Hatice Gunes, and Andrea Cavallaro. "Automatic analysis of facial affect: A survey of registration, representation, and recognition." IEEE transactions on pattern analysis and machine intelli- gence 37, no. 6 (2015): 1113-1133.
  46. Liu, Jingjing, Bo Liu, Shaoting Zhang, Fei Yang, Peng Yang, Dimitris N. Metaxas, and Carol Neidle. "Non-manual grammatical marker recognition based on multi-scale, spatio-temporal analysis of head pose and facial expressions." Image and Vision Computing 32, no. 10 (2014): 671-681.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2017-09-30

Issue

Volume 2, Issue 7, September-2017

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
Abhishek Kilak, Namita Mittal, " Multiple Instances Based Emotion Detection Using Discriminant Feature Tracking, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 2, Issue 7, pp.232-250, September-2017.