Comparison and Implementation of CNN Facial Emotion Recognition Model with Hyperparameter Analysis on Multiple Datasets

Xaviera Valentina Tandianto; Dwi Hosanna Bangkalang; Nina Setiyawati

doi:10.34148/teknika.v15i1.1445

Authors

Xaviera Valentina Tandianto Department of Information System, Faculty of Information Technology, Satya Wacana Christian University, Salatiga, Central Java, Indonesia
Dwi Hosanna Bangkalang Department of Information System, Faculty of Information Technology, Satya Wacana Christian University, Salatiga, Central Java, Indonesia
Nina Setiyawati Department of Informatics Engineering, Faculty of Information Technology, Satya Wacana Christian University, Salatiga, Central Java, Indonesia

DOI:

https://doi.org/10.34148/teknika.v15i1.1445

Keywords:

Convolutional Neural Network, Facial Emotion Recognition, Hyperparameter Analysis, Webcam Implementation

Abstract

This study presents a systematic comparison and implementation of a Convolutional Neural Network (CNN) for Facial Emotion Recognition (FER) across multiple public datasets, namely FER-2013, FER+, RAF-DB, and AffectNet. Unlike previous studies that focused on a single dataset or different model architectures, The main contributions of this research consist of three aspects. First, a five-layer integrated CNN architecture is used to enable fair cross-dataset evaluation within a consistent training and testing framework. Second, structured hyperparameter tuning is performed, including variations in learning rate, batch size, filter configuration, and dropout rate, resulting in a stable and reproducible model configuration. Third, an in-depth analysis was conducted to explore the impact of annotation quality and dataset complexity on model performance. The experimental results show that FER+ achieved the highest accuracy and weighted F1 score thanks to better label consistency, followed by RAF-DB, while FER-2013 and AffectNet experienced a decline in performance due to label noise and higher pose and lighting variations. Further confusion matrix analysis shows that happy and neutral expressions are classified more reliably, while negative emotions such as anger, fear, and disgust remain challenging. To validate practical application, the best-performing model was implemented in a webcam-based facial expression recognition prototype using Python and OpenCV, demonstrating reliable frame-level emotion inference under controlled real-time conditions.

Downloads

Download data is not yet available.

References

[1] J. Shen, J. Yu, H. Zhang, M. A. Lindsey, and R. An, “Artificial intelligence-powered social robots for promoting physical activity in older adults: A systematic review,” Dec. 01, 2025, Elsevier B.V. doi: 10.1016/j.jshs.2025.101045.

[2] T. Kujani and V. D. Kumar, “Emotion Understanding from Facial Expressions using Stacked Generative Adversarial Network (GAN) and Deep Convolution Neural Network (DCNN),” International Journal of Engineering Trends and Technology, vol. 70, no. 10, pp. 98–110, 2022, doi: 10.14445/22315381/IJETT-V70I10P212.

[3] D. de M. Fabrício, B. L. C. Ferreira, M. A. Maximiano-Barreto, M. Muniz, and M. H. N. Chagas, “Construction of face databases for tasks to recognize facial expressions of basic emotions: a systematic review,” Dementia e Neuropsychologia, vol. 16, no. 4, pp. 388–410, 2022, doi: 10.1590/1980-5764-DN-2022-0039.

[4] N. F. Alkhaleeli, “Facial Expression Recognition and Emotion Detection with CNN methods And SVM Classifiers,” Journal of Millimeterwave Communication, Optimization and Modelling, vol. 4, no. January 2023, 2024.

[5] E. Mansouri-Benssassi and J. Ye, “Generalisation and robustness investigation for facial and speech emotion recognition using bio-inspired spiking neural networks,” Soft comput., vol. 25, no. 3, pp. 1717–1730, 2021, doi: 10.1007/s00500-020-05501-7.

[6] C. Duan, “A survey of facial expression recognition in the wild,” Applied and Computational Engineering, vol. 6, no. 1, pp. 98–106, 2023, doi: 10.54254/2755-2721/6/20230760.

[7] Y. Fan et al., “FER-PCVT: Facial Expression Recognition with Patch-Convolutional Vision Transformer for Stroke Patients,” Brain Sci., vol. 12, no. 12, 2022, doi: 10.3390/brainsci12121626.

[8] Z. Y. Huang et al., “A study on computer vision for facial emotion recognition,” Sci. Rep., vol. 13, no. 1, pp. 1–13, 2023, doi: 10.1038/s41598-023-35446-4.

[9] J. Wang, “Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing,” Front. Neurosci., vol. 18, 2024, doi: 10.3389/fnins.2024.1449527.

[10] M. I. Abdi and B. Ergen, “Facial Expression Based Emotion Recognition,” Journal homepage: Journal of Electrical Engineering and Computer (JEECOM), vol. 7, no. 1, 2025, doi: 10.33650/jeecom.v4i2.

[11] D. Pruthviraja, U. M. Kumar, S. Parameswaran, V. G. Chowdary, and V. Bharadwaj, “Deep convolutional neural network architecture for facial emotion recognition,” PeerJ Comput. Sci., vol. 10, pp. 1–20, 2024, doi: 10.7717/peerj-cs.2339.

[12] B. H. Pansambal and A. B. Nandgaokar, “Integrating Dropout Regularization Technique at Different Layers to Improve the Performance of Neural Networks,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 4, pp. 716–722, 2023, doi: 10.14569/IJACSA.2023.0140478.

[13] M. Ramdhany Edy, “Deteksi Emosi dari Ekspresi Wajah dengan Deep Learning,” INTEC Journal: Information Technology Education Journal, vol. 3, no. 2, 2024.

[14] M. Smaida, S. Yaroshchak, and A. Y. Ben Sasi, “Learning Rate Optimization in CNN for Accurate Ophthalmic Classification,” International Journal of Innovative Technology and Exploring Engineering (IJITEE), vol. 3075, no. 4, pp. 211–216, 2021, doi: 10.35940/ijitee.B8259.0210421.

[15] A. R. Fadillah, C. Sri, and K. Aditya, “Gfpgan Upscaling for Human Facial Expression Classification Using Vgg19 Architecture,” vol. 11, no. 1, pp. 18–26, 2025, doi: 10.33480/jitk.v11i1.6588.INTRODUCTION.

[16] and Y. Q. Kai Wang , Xiaojiang Peng , Jianfei Yang , Debin Meng, “Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition,” IEEE Transactions on Image Processing, vol. 29, pp. 4057–4069, 2020.

[17] Y. K. Bhatti, A. Jamil, N. Nida, M. H. Yousaf, S. Viriri, and S. A. Velastin, “Facial Expression Recognition of Instructor Using Deep Features and Extreme Learning Machine,” Comput. Intell. Neurosci., vol. 2021, 2021, doi: 10.1155/2021/5570870.

[18] F. Y. Rahadika, N. Yudistira, and Y. A. Sari, “Facial Expression Recognition using Residual Convnet with Image Augmentations,” Jurnal Ilmu Komputer dan Informasi, vol. 14, no. 2, pp. 127–135, Jul. 2021, doi: 10.21609/jiki.v14i2.968.

[19] W. Yu and H. Xu, “Co-attentive multi-task convolutional neural network for facial expression recognition,” Pattern Recognit., vol. 123, p. 108401, 2022, doi: 10.1016/j.patcog.2021.108401.

[20] H. B. Khoirullah, N. Yudistira, and F. Abdurrachman Bachtiar, “Facial Expression Recognition Using Convolutional Neural Network with Attention Module,” INTERNATIONAL JOURNAL ON INFORMATICS VISUALIZATION, Dec. 2022, doi: 10.30630/joiv.6.4.963.

[21] A. Y. Nawaf and W. M. Jasim, “A pre-trained model vs dedicated convolution neural networks for emotion recognition,” International Journal of Electrical and Computer Engineering, vol. 13, no. 1, pp. 1123–1133, Feb. 2023, doi: 10.11591/ijece.v13i1.pp1123-1133.

[22] M. C. Gursesli, S. Lombardi, M. Duradoni, L. Bocchi, A. Guazzini, and A. Lanata, “Facial Emotion Recognition (FER) Through Custom Lightweight CNN Model: Performance Evaluation in Public Datasets,” IEEE Access, vol. 12, pp. 45543–45559, 2024, doi: 10.1109/ACCESS.2024.3380847.

[23] L. Duongthang, “Driver facial emotion tracking using an enhanced residual network with weighted fusion of channel and spatial attention,” Sci. Rep., vol. 15, no. 1, pp. 1–10, 2025, doi: 10.1038/s41598-025-97451-z.

[24] G. J. P. Hakim et al., “Real-Time Facial Emotion Detection Application with Image Processing Based on Convolutional Neural Network (CNN),” International Journal of Electrical Engineering, Mathematics and Computer Science, vol. 1, no. 4, pp. 27–36, Nov. 2024, doi: 10.62951/ijeemcs.v1i4.123.

[25] O. Fassya Enbiva and R. Toyib, “Penerapan Metode Haar Cascade Untuk Deteksi Pola Wajah Berdasarkan Perkembangan Usia,” Jurnal Informatika Teknologi dan Sains, Aug. 2025.