Facial Expression Detection and Recognition through VIOLA-JONES Algorithm and HCNN using LSTM Method

Authors

  • Dinesh Kumar P  Ph.D. Research Scholar, PG and Research Department of Computer Science, Dr. N.G.P. Arts and Science College, Coimbatore, Tamilnadu, India
  • Dr. B. Rosiline Jeetha  Professor and Head, PG and Research Department of Computer Science, Dr. N.G.P. Arts and Science College, Coimbatore, Tamilnadu, India

DOI:

https://doi.org//10.32628/CSEIT2173143

Keywords:

Face geometry, feature extraction, LBP, LSTM, SIFT, HCNN, f-measure.

Abstract

Facial expression, as one of the most significant means for human beings to show their emotions and intensions in the process of communication, plays a significant role in human interfaces. In recent years, facial expression recognition has been under especially intensive investigation, due conceivably to its vital applications in various fields including virtual reality, intelligent tutoring system, health-care and data driven animation. The main target of facial expression recognition is to identify the human emotional state (e.g., anger, contempt, disgust, fear, happiness, sadness, and surprise ) based on the given facial images. This paper deals with the Facial expression detection and recognition through Viola-jones algorithm and HCNN using LSTM method. It improves the hypothesis execution enough and meanwhile inconceivably reduces the computational costs. In feature matching, the author proposes Hybrid Scale-Invariant Feature Transform (SIFT) with double δ-LBP (Dδ-LBP) and it utilizes the fixed facial landmark localization approach and SIFT’s orientation assignment, to obtain the features that are illumination and pose independent. For face detection, basically we utilize the face detection Viola-Jones algorithm and it recognizes the occluded face and it helps to perform the feature selection through the whale optimization algorithm, once after compression and further, it minimizes the feature vector given into the Hybrid Convolutional Neural Network (HCNN) and Long Short-Term Memory (LSTM) model for identifying the facial expression in efficient manner.The experimental result confirms that the HCNN-LSTM Model beats traditional deep-learning and machine-learning techniques with respect to precision, recall, f-measure, and accuracy using CK+ database. Proposes Hybrid Scale-Invariant Feature Transform (SIFT) with double δ-LBP (Dδ-LBP) and it utilizes the fixed facial landmark localization approach and SIFT’s orientation assignment, to obtain the features that are illumination and pose independent. And HCNN and LSTM model for identifying the facial expression.

References

  1. Pantic, Maja, and Ioannis Patras. "Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 36.2 (2006): 433-449. Available from: DOI: 10.1109/TSMCB.2005.859075.
  2. Ma, Liying, and KhashayarKhorasani. "Facial expression recognition using constructive feedforward neural networks." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 34.3 (2004): 1588-1595. Available from: DOI: 10.1109/TSMCB.2004.825930
  3. Liu, Wei-feng, and Yan-jiang Wang. "Expression feature extraction based on difference of local binary pattern histogram sequences." 2008 9th International Conference on Signal Processing. IEEE, 2008.pp (2082 – 2084). Available from: DOI: 10.1109/ICOSP.2008.4697555.
  4. Zhao, Shuwen, et al. "Feature Selection Mechanism in CNNs for Facial Expression Recognition." BMVC. 2018.pp(35-41). Available from: DOI: 10.1109/EUVIP.2010.5699141.
  5. Jia, Haipeng, et al. "Accelerating viola-jones face detection algorithm on gpus." 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems. IEEE, 2012.Available from:DOI: 10.1109/FCCM.2010.12.
  6. Farfade, Sachin Sudhakar, Mohammad J. Saberian, and Li-Jia Li. "Multi-view face detection using deep convolutional neural networks." Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 2015.pp.643–650. Available from:https://doi.org/10.1145/2671188.2749408.
  7. El-Bakry, H. M., & Zhao, Q. (2004). Fast object/face detection using neural networks and fast Fourier transform. International Journal of Signal Processing, 1(3), 182-187. Available from:https://www.researchgate.net/publication/37432768.
  8. Satiyan, M., and R. Nagarajan. "Recognition of facial expression using Haar-like feature extraction method." 2010 International Conference on Intelligent and Advanced Systems. IEEE, 2010. (pp. 1-4) . Available from:DOI: 10.1109/ICIAS.2010.5716228
  9. Kirana, KartikaCandra, Slamet Wibawanto, and Heru Wahyu Herwanto. "Facial Emotion Recognition Based on Viola-Jones Algorithm in the Learning Environment." 2018 International Seminar on Application for Technology of Information and Communication. IEEE, 2018. (pp. 406-410). Available from: DOI: 10.1109/ISEMANTIC.2018.8549735.
  10. Du, Cheng-Jin, and Da-Wen Sun. "Recent developments in the applications of image processing techniques for food quality evaluation." Trends in food science & technology 15.5 (2004): 230-249.Available from: https://doi.org/10.1016/j.tifs.2003.10.006.
  11. Mian, Ajmal, Mohammed Bennamoun, and Robyn Owens. "An efficient multimodal 2D-3D hybrid approach to automatic face recognition." IEEE transactions on pattern analysis and machine intelligence 29.11 (2007): 1927-1943. Available from: DOI:10.1109/TPAMI.2007.1105.
  12. Zheng, Yuefeng, et al. "A Novel Hybrid Algorithm for Feature Selection Based on Whale Optimization Algorithm." IEEE Access 7 (2018): 14908-14923. Available from: DOI: 10.1109/ACCESS.2018.2879848.
  13. Sharawi, Marwa, Hossam M. Zawbaa, and EidEmary. "Feature selection approach based on whale optimization algorithm." 2017 Ninth International Conference on Advanced Computational Intelligence (ICACI). IEEE, 2017. (pp. 163-168). Available from: DOI: 10.1109/ICACI.2017.7974502.
  14. Ekweariri, Augustine Nnamdi, and Kamil Yurtkan. "Facial expression recognition using enhanced local binary patterns." 2017 9th International Conference on Computational Intelligence and Communication Networks (CICN). IEEE, 2017. (pp. 43-47). Available from: DOI: 10.1109/CICN.2017.8319353.
  15. Akbulut, Yaman, et al. "Deep learning based face liveness detection in videos." 2017 international artificial intelligence and data processing symposium (IDAP). IEEE, 2017. (pp. 1-4). Available from: DOI: 10.1109/IDAP.2017.8090202.
  16. Sunaryo, Musthofa, and MochammadHariadi. "Preprocessing on Digital Image using Histogram Equalization: An Experiment Study on MRI Brain Image." International Journal of Computer Science and Information Technologies 7.4 (2016): 1723-1727. Available from: DOI: 10.1109/IDAP.2017.8090202.
  17. Ren, Jianfeng, Nasser Kehtarnavaz, and Leonardo Estevez. "Real-time optimization of Viola-Jones face detection for mobile platforms." 2008 IEEE Dallas Circuits and Systems Workshop: System-on-Chip-Design, Applications, Integration, and Software. IEEE, 2008. (pp. 1-4). Available from: DOI: 10.1109/DCAS.2008.4695921.
  18. Purandare, V., &Talele, K. T. (2014, April). Efficient heterogeneous face recognition using scale invariant feature transform. In 2014 International Conference on Circuits, Systems, Communication and Information Technology Applications (CSCITA) (pp. 305-310). IEEE. Available from: DOI: 10.1109/CSCITA.2014.6839277.
  19. Shen, Fang, Jing Liu, and Peng Wu. "Double δ-LBP: A Novel Feature Extraction Method for Facial Expression Recognition." Chinese Conference on Image and Graphics Technologies. Springer, Singapore, 2018.pp 370-379.Available from:DOI: 10.1109/CSCITA.2014.6839277.
  20. Sharawi, Marwa, Hossam M. Zawbaa, and EidEmary. "Feature selection approach based on whale optimization algorithm." 2017 Ninth International Conference on Advanced Computational Intelligence (ICACI). IEEE, 2017. Available from:DOI: 10.1109/ICACI.2017.7974502.
  21. Nwosu, Lucy, et al. "Deep convolutional neural network for facial expression recognition using facial parts." 2017 IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech). IEEE, 2017. Available from: DOI 10.1109/DASC-PICom-DataCom-CyberSciTec.2017.213.
  22. Gao, Tingwei, Yueting Chai, and Yi Liu. "Applying long short term memory neural networks for predicting stock closing price." 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS). IEEE, 2017. (pp. 575-578). Available from: DOI: 10.1109/ICSESS.2017.8342981.
  23. Canedo, D. and Neves, A.J., 2019. Facial Expression Recognition Using Computer Vision: A Systematic Review. Applied Sciences, 9(21), p.4678. Available from: DOI: 10.3390/app9214678;
  24. Wati, V., Kusrini, K. and Al Fatta, H., 2019, July. Real Time Face Expression Classification Using Convolutional Neural Network Algorithm. In 2019 International Conference on Information and Communications Technology (ICOIACT) (pp. 497-501). IEEE. Available from: doi: 10.1109/ICOIACT46704.2019.8938521.
  25. Mahmood, A., Hussain, S., Iqbal, K. and Elkilani, W.S., 2019. Recognition of facial expressions under varying conditions using dual-feature fusion. Mathematical Problems in Engineering, 2019. Available from: doi:10.1155/2019/9185481.
  26. Siddiqui, M. F., Siddique, W. A., Ahmedh, M., &Jumani, A. K. (2020). Face Detection and Recognition System for Enhancing Security Measures Using Artificial Intelligence System. INDIAN JOURNAL OF SCIENCE AND TECHNOLOGY, 13(09), 1057-1064.Available from: DOI: 10.17485/ijst/2020/v013i09/149298.
  27. Lucey, P., Cohn, J.F., Kanade, T., et al.: ‘The extended Cohn–Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression’. IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, 2010, 36, (1), pp. 94–101. Available from: DOI: 10.1109/CVPRW.2010.5543262.
  28. Siddiqi, Muhammad Hameed, et al. "Real Time Human Facial Expression Recognition System using Smartphone." INTERNATIONAL JOURNAL OF COMPUTER SCIENCE AND NETWORK SECURITY 17.10 (2017): 223-230.Available from: DOI: 10.1109/CVPRW.2014.25.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2021-06-30

Issue

Volume 7, Issue 3, May-June-2021

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
Dinesh Kumar P, Dr. B. Rosiline Jeetha, " Facial Expression Detection and Recognition through VIOLA-JONES Algorithm and HCNN using LSTM Method, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 7, Issue 3, pp.463-480, May-June-2021. Available at doi : https://doi.org/10.32628/CSEIT2173143