A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition
DOI:
https://doi.org/10.32628/CSEIT251116171Keywords:
Face Expressions, Face Emotion Recognition, Deep Learning, VGG-19, ResNet-50, Inception-V3, MobileNetAbstract
Deep learning has substantially enhanced facial emotion recognition, an essential element of hu-man–computer interaction. This study evaluates the performance of multiple architectures, including a custom CNN, VGG-16, ResNet-50, and a hybrid CNN-LSTM framework, across FER2013 and CK+ datasets. Preprocessing steps involved grayscale conversion, image resizing, and pixel normaliza-tion. Experimental results show that ResNet-50 achieved the highest accuracy on FER2013 (76.85%), while the hybrid CNN-LSTM model attained superior performance on CK+ (92.30%). Per-formance metrics such as precision, recall, and F1-score were used for evaluation. Findings high-light the trade-off between computational efficiency and recognition accuracy, offering insights for developing robust, real-time emotion recognition systems.
📊 Article Downloads
References
P. Naga, S. Das Marri, and R. Borreo, “Facial emotion recognition methods, datasets and technologies: A literature survey,” Mater Today Proc, vol. 80, pp. 2824–2828, Jan. 2023, doi: 10.1016/j.matpr.2021.07.046. DOI: https://doi.org/10.1016/j.matpr.2021.07.046
M. K. Chowdary, T. N. Nguyen, and D. J. Hemanth, “Deep learning-based facial emotion recognition for human–computer interaction applications,” Neural Comput Appl, vol. 35, no. 32, pp. 23311–23328, Nov. 2023, doi: 10.1007/s00521-021-06012-8. DOI: https://doi.org/10.1007/s00521-021-06012-8
T. T. Nguyen et al., “Deep Learning for Deepfakes Creation and Detection: A Survey,” Sep. 2019, doi: 10.1016/j.cviu.2022.103525.
A. Saravanan, G. Perichetla, and Dr. K. S. Gayathri, “Facial Emotion Recognition using Convolutional Neural Networks,” Oct. 2019, [Online]. Available: http://arxiv.org/abs/1910.05602
W. Dias et al., “Cross-dataset emotion recognition from facial expressions through convolutional neural networks A R T I C L E I N F O,” J Vis Commun Image Represent, 2023, doi: 10.5281/zenodo.4696032.
F. Hernández-Luquin and H. J. Escalante, “Multi-branch deep radial basis function networks for facial emo-tion recognition,” Neural Comput Appl, vol. 35, no. 25, pp. 18131–18145, Sep. 2023, doi: 10.1007/s00521-021-06420-w. DOI: https://doi.org/10.1007/s00521-021-06420-w
N. Siddiqui, R. Dave, T. Bauer, T. Reither, D. Black, and M. Hanson, “A Robust Framework for Deep Learn-ing Approaches to Facial Emotion Recognition and Evaluation,” IEEE.
I. Dominguez-Catena, D. Paternain, and M. Galar, “Assessing Demographic Bias Transfer from Dataset to Model: A Case Study in Facial Expression Recognition,” May 2022, [Online]. Available: http://arxiv.org/abs/2205.10049
T. Kumar Arora et al., “Optimal Facial Feature Based Emotional Recognition Using Deep Learning Algo-rithm,” Comput Intell Neurosci, vol. 2022, 2022, doi: 10.1155/2022/8379202. DOI: https://doi.org/10.1155/2022/8379202
R. Daněček and M. Black, “EMOCA: Emotion Driven Monocular Face Capture and Animation.” [Online]. Available: https://emoca.is.tue.mpg.de.
R. Appasaheb Borgalli and S. Surve, “Deep Learning Framework for Facial Emotion Recognition using CNN Architectures,” in Proceedings of the International Conference on Electronics and Renewable Systems, ICEARS 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 1777–1784. doi: 10.1109/ICEARS53579.2022.9751735. DOI: https://doi.org/10.1109/ICEARS53579.2022.9751735
A. Malik, M. Kuribayashi, S. M. Abdullahi, and A. N. Khan, “DeepFake Detection for Human Face Images and Videos: A Survey,” IEEE Access, vol. 10. Institute of Electrical and Electronics Engineers Inc., pp. 18757–18775, 2022. doi: 10.1109/ACCESS.2022.3151186. DOI: https://doi.org/10.1109/ACCESS.2022.3151186
Y. S. Lee and W. H. Park, “Diagnosis of Depressive Disorder Model on Facial Expression Based on Fast R-CNN,” Diagnostics, vol. 12, no. 2, Feb. 2022, doi: 10.3390/diagnostics12020317. DOI: https://doi.org/10.3390/diagnostics12020317
T. Dar, A. Javed, S. Bourouis, H. S. Hussein, and H. Alshazly, “Efficient-SwishNet Based System for Facial Emotion Recognition,” IEEE Access, vol. 10, pp. 71311–71328, 2022, doi: 10.1109/ACCESS.2022.3188730. DOI: https://doi.org/10.1109/ACCESS.2022.3188730
K. , R. Y. , & M. R. (2022). C. facial emotional expression recognition using cnn deep features. E. L. 30(4), 1402-1416. Slimani, “Compound facial emotional expression recognition using cnn deep features,” 2022.
A. Pise, H. Vadapalli, and I. Sanders, “Facial emotion recognition using temporal relational network: an application to E-learning,” Multimed Tools Appl, vol. 81, no. 19, pp. 26633–26653, Aug. 2022, doi: 10.1007/s11042-020-10133-y. DOI: https://doi.org/10.1007/s11042-020-10133-y
D. Mamieva, A. B. Abdusalomov, M. Mukhiddinov, and T. K. Whangbo, “Improved Face Detection Method via Learning Small Faces on Hard Images Based on a Deep Learning Approach,” Sensors, vol. 23, no. 1, Jan. 2023, doi: 10.3390/s23010502. DOI: https://doi.org/10.3390/s23010502
A. R. Khan, “Facial Emotion Recognition Using Conventional Machine Learning and Deep Learning Methods: Current Achievements, Analysis and Remaining Challenges,” Information (Switzerland), vol. 13, no. 6. MDPI, Jun. 01, 2022. doi: 10.3390/info13060268. DOI: https://doi.org/10.3390/info13060268
S. Wang, J. Qu, Y. Zhang, and Y. Zhang, “Multimodal Emotion Recognition From EEG Signals and Facial Expressions,” IEEE Access, vol. 11, pp. 33061–33068, 2023, doi: 10.1109/ACCESS.2023.3263670. DOI: https://doi.org/10.1109/ACCESS.2023.3263670
S. Dwijayanti, M. Iqbal, and B. Y. Suprapto, “Real-time Implementation of Face Recognition and Emotion Recognition in a Humanoid Robot Using a Convolutional Neural Network,” IEEE Access, 2022, doi: 10.1109/ACCESS.2022.3200762. DOI: https://doi.org/10.1109/ACCESS.2022.3200762
S. Gupta, P. Kumar, and R. K. Tekchandani, “Facial emotion recognition based real-time learner engage-ment detection system in online learning context using deep learning models,” Multimed Tools Appl, vol. 82, no. 8, pp. 11365–11394, Mar. 2023, doi: 10.1007/s11042-022-13558-9. DOI: https://doi.org/10.1007/s11042-022-13558-9
N. Rawal and R. M. Stock-Homburg, “Facial Emotion Expressions in Human–Robot Interaction: A Survey,” Int J Soc Robot, vol. 14, no. 7, pp. 1583–1604, Sep. 2022, doi: 10.1007/s12369-022-00867-0. DOI: https://doi.org/10.1007/s12369-022-00867-0
H. K. Wong and A. J. Estudillo, “Face masks affect emotion categorisation, age estimation, recognition, and gender classification from faces,” Cogn Res Princ Implic, vol. 7, no. 1, Dec. 2022, doi: 10.1186/s41235-022-00438-x. DOI: https://doi.org/10.1186/s41235-022-00438-x
M. F. Alsharekh, “Facial Emotion Recognition in Verbal Communication Based on Deep Learning,” Sensors, vol. 22, no. 16, Aug. 2022, doi: 10.3390/s22166105. DOI: https://doi.org/10.3390/s22166105
A. Farkhod, A. B. Abdusalomov, M. Mukhiddinov, and Y. I. Cho, “Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces,” Sensors, vol. 22, no. 22, Nov. 2022, doi: 10.3390/s22228704. DOI: https://doi.org/10.3390/s22228704
Y. Ding, N. Robinson, S. Zhang, Q. Zeng, and C. Guan, “TSception: Capturing Temporal Dynamics and Spa-tial Asymmetry From EEG for Emotion Recognition,” IEEE Trans Affect Comput, vol. 14, no. 3, pp. 2238–2250, Jul. 2023, doi: 10.1109/TAFFC.2022.3169001. DOI: https://doi.org/10.1109/TAFFC.2022.3169001
M. Mukhiddinov, O. Djuraev, F. Akhmedov, A. Mukhamadiyev, and J. Cho, “Masked Face Emotion Recogni-tion Based on Facial Landmarks and Deep Learning Approaches for Visually Impaired People,” Sensors, vol. 23, no. 3, Feb. 2023, doi: 10.3390/s23031080. DOI: https://doi.org/10.3390/s23031080
Y. Cai, X. Li, and J. Li, “Emotion Recognition Using Different Sensors, Emotion Models, Methods and Da-tasets: A Comprehensive Review,” Sensors, vol. 23, no. 5. MDPI, Mar. 01, 2023. doi: 10.3390/s23052455. DOI: https://doi.org/10.3390/s23052455
S. Talala, S. Shvimmer, R. Simhon, M. Gilead, and Y. Yitzhaky, “Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning,” Sensors, vol. 24, no. 8, Apr. 2024, doi: 10.3390/s24082620. DOI: https://doi.org/10.3390/s24082620
R. Pereira et al., “Systematic Review of Emotion Detection with Computer Vision and Deep Learning,” Sen-sors, vol. 24, no. 11, p. 3484, May 2024, doi: 10.3390/s24113484. DOI: https://doi.org/10.3390/s24113484
Techno-Societal 2018. Springer International Publishing, 2020. doi: 10.1007/978-3-030-16848-3.
C. V. R. Reddy, U. S. Reddy, and K. V. K. Kishore, “Facial emotion recognition using NLPCA and SVM,” Traitement du Signal, vol. 36, no. 1, pp. 13–22, Feb. 2019, doi: 10.18280/ts.360102. DOI: https://doi.org/10.18280/ts.360102
M. Sajjad, M. Nasir, F. U. M. Ullah, K. Muhammad, A. K. Sangaiah, and S. W. Baik, “Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services,” Inf Sci (N Y), vol. 479, pp. 416–431, Apr. 2019, doi: 10.1016/j.ins.2018.07.027. DOI: https://doi.org/10.1016/j.ins.2018.07.027
M. Nazir, Z. Jan, and M. Sajjad, “Facial expression recognition using weber discrete wavelet transform,” Journal of Intelligent and Fuzzy Systems, vol. 33, no. 1, pp. 479–489, 2017, doi: 10.3233/JIFS-161787. DOI: https://doi.org/10.3233/JIFS-161787
N. Zeng, H. Zhang, B. Song, W. Liu, Y. Li, and A. M. Dobaie, “Facial expression recognition via learning deep sparse autoencoders,” Neurocomputing, vol. 273, pp. 643–649, Jan. 2018, doi: 10.1016/j.neucom.2017.08.043. DOI: https://doi.org/10.1016/j.neucom.2017.08.043
M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino, and J. Torresen, “A facial expression recog-nition system using robust face features from depth videos and deep learning,” Computers and Electrical Engineering, vol. 63, pp. 114–125, Oct. 2017, doi: 10.1016/j.compeleceng.2017.04.019. DOI: https://doi.org/10.1016/j.compeleceng.2017.04.019
S. A. Al-agha, H. H. Saleh, and R. F. Ghani, “Geometric-based Feature Extraction and Classification for Emotion Expressions of 3D Video Film,” Journal of Advances in Information Technology, pp. 74–79, 2017, doi: 10.12720/jait.8.2.74-79. DOI: https://doi.org/10.12720/jait.8.2.74-79
D. Ghimire, J. Lee, Z. N. Li, and S. Jeong, “Recognition of facial expressions based on salient geometric features and support vector machines,” Multimed Tools Appl, vol. 76, no. 6, pp. 7921–7946, Mar. 2017, doi: 10.1007/s11042-016-3428-9. DOI: https://doi.org/10.1007/s11042-016-3428-9
J. Wang and H. Yang, “Face detection based on template matching and 2DPCA algorithm,” in Proceedings - 1st International Congress on Image and Signal Processing, CISP 2008, 2008, pp. 575–579. doi: 10.1109/CISP.2008.270. DOI: https://doi.org/10.1109/CISP.2008.270
P. ping Wu, H. Liu, X. wu Zhang, and Y. Gao, “Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics,” Frontiers of Information Technology and Elec-tronic Engineering, vol. 18, no. 7, pp. 955–967, Jul. 2017, doi: 10.1631/FITEE.1600041. DOI: https://doi.org/10.1631/FITEE.1600041
IEEE Computer Society and Institute of Electrical and Electronics Engineers, 12th IEEE International Confer-ence and Workshops on Automatic Face and Gesture Recognition : FG 2017 : proceedings : 30 May - 3 June 2017, Washington, D.C.
D. J. Kim, “Facial expression recognition using ASM-based post-processing technique,” Pattern Recognition and Image Analysis, vol. 26, no. 3, pp. 576–581, Jul. 2016, doi: 10.1134/S105466181603010X. DOI: https://doi.org/10.1134/S105466181603010X
J. Y. R. Cornejo, H. Pedrini, and F. Flórez-Revuelta, “Facial Expression Recognition with Occlusions based on Geometric Representation.”
M. H. Siddiqi, R. Ali, A. M. Khan, E. S. Kim, G. J. Kim, and S. Lee, “Facial expression recognition using ac-tive contour-based face detection, facial movement-based feature extraction, and non-linear feature selec-tion,” Multimed Syst, vol. 21, no. 6, pp. 541–555, Nov. 2015, doi: 10.1007/s00530-014-0400-2. DOI: https://doi.org/10.1007/s00530-014-0400-2
K. Y. Chang, C. S. Chen, and Y. P. Hung, “Intensity rank estimation of facial expressions based on a single image,” in Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013, 2013, pp. 3157–3162. doi: 10.1109/SMC.2013.538. DOI: https://doi.org/10.1109/SMC.2013.538
S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” Proc Natl Acad Sci U S A, vol. 111, no. 15, Apr. 2014, doi: 10.1073/pnas.1322355111. DOI: https://doi.org/10.1073/pnas.1322355111
“Expressions of Emotion Dataset [5]. We trained the network on CFEE dataset and tested the trained model on RaFD. 2. Related Work.”
K. Slimani, K. Lekdioui, R. Messoussi, and R. Touahni, “Compound facial expression recognition based on highway CNN,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Mar. 2019. doi: 10.1145/3314074.3314075. DOI: https://doi.org/10.1145/3314074.3314075
S. Li, W. Deng, and J. Du, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild.” [Online]. Available: http://whdeng.cn/RAF/model1.html
Institute of Electrical and Electronics Engineers and IEEE Signal Processing Society, 2018 IEEE International Conference on Image Processing : proceedings : October 7-10, 2018, Megaron Athens International Con-ference Centre, Athens, Greece.
Y. Fan, J. C. K. Lam, and V. O. K. Li, “Multi-Region Ensemble Convolutional Neural Network for Facial Ex-pression Recognition.”
S. Ghosh, A. Dhall, and N. Sebe, “Automatic Group Affect Analysis in Images via Visual Attribute and Fea-ture Networks,” in Proceedings - International Conference on Image Processing, ICIP, IEEE Computer Soci-ety, Aug. 2018, pp. 1967–1971. doi: 10.1109/ICIP.2018.8451242. DOI: https://doi.org/10.1109/ICIP.2018.8451242
Institute of Electrical and Electronics Engineers and IEEE Signal Processing Society, 2018 IEEE International Conference on Image Processing : proceedings : October 7-10, 2018, Megaron Athens International Con-ference Centre, Athens, Greece.
V. Vielzeuf, C. Kervadec, S. Pateux, A. Lechervy, and F. Jurie, “An Occam’s Razor View on Learning Audio-visual Emotion Recognition with Small Training Sets,” Aug. 2018, [Online]. Available: http://arxiv.org/abs/1808.02668 DOI: https://doi.org/10.1145/3242969.3264980
D. Acharya, Z. Huang, D. P. Paudel, and V. Gool, “Covariance Pooling for Facial Expression Recognition.”
Y. Li, J. Zeng, S. Shan, and X. Chen, “Patch-Gated CNN for Occlusion-aware Facial Expression Recognition.”
J. Deng, G. Pang, Z. Zhang, Z. Pang, H. Yang, and G. Yang, “CGAN Based Facial Expression Recognition for Human-Robot Interaction,” IEEE Access, vol. 7, pp. 9848–9859, 2019, doi: 10.1109/ACCESS.2019.2891668. DOI: https://doi.org/10.1109/ACCESS.2019.2891668
K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao, “Region Attention Networks for Pose and Occlusion Ro-bust Facial Expression Recognition,” May 2019, [Online]. Available: http://arxiv.org/abs/1905.04075
T. H. Vo, G. S. Lee, H. J. Yang, and S. H. Kim, “Pyramid with Super Resolution for In-the-Wild Facial Ex-pression Recognition,” IEEE Access, vol. 8, pp. 131988–132001, 2020, doi: 10.1109/ACCESS.2020.3010018. DOI: https://doi.org/10.1109/ACCESS.2020.3010018
S. Li, W. Deng, and J. Du, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild.” [Online]. Available: http://whdeng.cn/RAF/model1.html
Z. You et al., Eds., Biometric Recognition, vol. 9967. in Lecture Notes in Computer Science, vol. 9967. Cham: Springer International Publishing, 2016. doi: 10.1007/978-3-319-46654-5.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 International Journal of Scientific Research in Computer Science, Engineering and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
