Facial Expression Recognition inCovid-19 Pandemic EraUsing 2 Stage Convolutional Neural Network

Authors

  • Kevin Kurniawan Bina Nusantara University, Jakarta
  • Lili Ayu Wulandhari Bina Nusantara University, Jakarta
  • Antonius Rildo Pramudya Gondosiswojo Bina Nusantara University, Jakarta

DOI:

https://doi.org/10.30865/json.v5i1.6768

Keywords:

Pandemic Era, Covid-19, Facial Expression Recognition, Face Mask, Occlusion, Convolutional Neural Network

Abstract

In the Covid-19 pandemic era, the use of face mask has become mandatory for all citizens to prevent the spread of the virus. This regulation has becomes a big problem for the Facial Expression Recognitions (FER) applications because face mask cover more than halfof human faces. Starting from this problem, this experiment attemps to produce a robust network which can perform well in both conditions: recognizing expressions with and without a face mask. Dataset used in this experiment is FER2013, with a preprocessing step to produce a FER2013 masked. This research uses 2-stage network, where the first network is used to recognize whether the subject is wearing a mask or not, and then the second network is used to recognize the expression based on the result from the first stage. The network in this experiment is based on Convolutional Neural Network (CNN), with Imagenet as our pretrainedmodel and EfficientNet as our architecture model.Ourproposed model has shown quite good performance in recognizing expressions, even when the data consists of subjects who use and do not use masks with an accuracy of 57.55%.

References

P. Saxena, “Real-time emotion recognition: Potential use cases and challenges,†indiaai.gov.in, 2021. .

Tinaliah and T. Elizabeth, “Penerapan Convolutional Neural Network Untuk Klasifikasi Citra Ekspresi Wajah Manusia Pada MMA Facial Expression Dataset,†J. Tek. Inform. dan Sist. Inf., vol. 8, no. 4, pp. 2051–2059, 2021, doi: 10.35957/jatisi.v8i4.1437.

S. Minaee, M. Minaei, and A. Abdolrashidi, “Deep-emotion: Facial expression recognition using attentional convolutional network,†Sensors, vol. 21, no. 9, pp. 1–16, 2021, doi: 10.3390/s21093046.

S. T. Zulkarnain and N. Suciati, “Modified Local Ternary Pattern With Convolutional Neural Network for Face Expression Recognition,†JUTI J. Ilm. Teknol. Inf., vol. 19, no. 1, p. 10, 2021, doi: 10.12962/j24068535.v19i1.a1031.

R. Septian, D. I. Saputra, and S. Sambasri, “Klasifikasi Emosi Menggunakan Convolutional Neural Networks,†2019, doi: 10.13733/j.jcam.issn.2095-5553.2022.11.026.

L. Pham, T. H. Vu, and T. A. Tran, “Facial expression recognition using residual masking network,†Proc. - Int. Conf. Pattern Recognit., pp. 4513–4519, 2020, doi: 10.1109/ICPR48806.2021.9411919.

Syahromi, “Facial Expression Recognition Menggunakan Algoritma Mtcnn (Multi-Task Cascaded Convolutional Neural Network) Dan Vgg16,†Apl. dan Anal. Lit. Fasilkom UI, pp. 4–25, 2022.

Ministry of Health, “Peta Sebaran,†Https://Covid19.Go.Id/Peta-Sebaran, p. https://covid19.go.id/peta-sebaran, 2022.

Haeruddin, Herman, and Patrick Pratama Hendri, “PENGEMBANGAN APLIKASI EMOTION RECOGNITION DAN FACIAL RECOGNITION MENGGUNAKAN ALGORITMA LOCAL BINARY PATTERN HISTOGRAM (LBPH) DAN CONVOLUTIONAL NEURAL NETWORK (CNN),†J. Teknol. Terpadu, vol. 9, no. 1, pp. 49–55, 2023.

B. Yang, J. Wu, and G. Hattori, “Facial Expression Recognition with the advent of face masks,†ACM Int. Conf. Proceeding Ser., pp. 335–337, 2020, doi: 10.1145/3428361.3432075.

G. Castellano, B. De Carolis, and N. Macchiarulo, “Automatic emotion recognition from facial expressions when wearing a mask,†ACM Int. Conf. Proceeding Ser., 2021, doi: 10.1145/3464385.3464730.

A. Anwar and A. Raychowdhury, “Masked Face Recognition for Secure Authentication,†pp. 1–8, 2020.

A. Lioga Seandrio, A. Hendrianto Pratomo, and M. Y. Florestiyanto, “Implementation of Convolutional Neural Network (CNN) in Facial Expression Recognition,†Telemat. J. Inform. dan Teknol. Inf., vol. 18, no. 2, pp. 211–221, 2021, doi: 10.31515/telematika.v18i2.4823.

M. Fakhrudin and A. Fatoni, “Pengaruh Risiko Spesifik Bank dan Good Corporate Governance Terhadap Stabilitas Bank Pembiayaan Rakyat Syariah,†vol. 4, no. 1, pp. 116–127, 2023, doi: 10.47065/jtear.v4i1.734.

D. Alamsyah and D. Pratama, “Implementasi Convolutional Neural Networks (CNN) untuk Klasifikasi Ekspresi Citra Wajah pada FER-2013 Dataset,†J. Teknol. Inf., vol. 4, no. 2, pp. 350–355, 2020, doi: 10.36294/jurti.v4i2.1714.

Lia Farokhah, “Implementasi Convolutional Neural Network untuk Klasifikasi Variasi Intensitas Emosi pada Dynamic Image Sequence,†J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 4, no. 6, pp. 1070–1076, 2020, doi: 10.29207/resti.v4i6.2644.

S. Sriyati, A. Setyanto, and E. E. Luthfi, “Literature Review: Pengenalan Wajah Menggunakan Algoritma Convolutional Neural Network,†J. Teknol. Inf. dan Komun., vol. 8, no. 2, 2020, doi: 10.30646/tikomsin.v8i2.463.

P. Musa, W. K. Anam, S. B. Musa, W. Aryunani, R. Senjaya, and P. Sularsih, “Pembelajaran Mendalam Pengklasifikasi Ekspresi Wajah Manusia dengan Model Arsitektur Xception pada Metode Convolutional Neural Network,†Rekayasa, vol. 16, no. 1, pp. 65–73, 2023, doi: 10.21107/rekayasa.v16i1.16974.

Y. Fajriyanti, A. H. Rahmah, S. Ulfa, and E. Hadiyanti, “Analisis Motivasi Kerja Generasi Z Yang Dipengaruhi Oleh Lingkungan Kerja dan Komitmen Kerja,†vol. 4, no. 1, pp. 107–115, 2023, doi: 10.47065/jtear.v4i1.808.

S. P. Ristiawanto, B. Irawan, and C. Setianingsih, “PENGENALAN EKSPRESI WAJAH BERBASIS CONVOLUTIONAL NEURAL NETWORK MENGGUNAKAN ARSITEKTUR RESIDUAL NETWORK-50,†e-Proceeding Eng., vol. 8, no. 5, pp. 6442–6454, 2021.

P. P. Kusdiananggalih, E. Rachmawat, and Risnandar, “Pengenalan Ekspresi Wajah Menggunakan Deep Convolutional Neural Network,†J. Tugas Akhir Fak. Inform. e-Proceeding, vol. 8, no. 2, p. 75, 2021, doi: 10.35200/explore.v11i2.478.

J. Omar, N. H. Shabrina, A. N. Bhakti, and A. Patria, “Emotion Recognition Using Convolutional Neural Network (CNN),†Ultim. Comput. J. Sist. Komput., vol. 13, no. 1, 2021, doi: 10.1088/1742-6596/1962/1/012040.

Downloads

Published

2023-09-29

How to Cite

Kurniawan, K., Wulandhari, L. A., & Gondosiswojo, A. R. P. (2023). Facial Expression Recognition inCovid-19 Pandemic EraUsing 2 Stage Convolutional Neural Network. Jurnal Sistem Komputer Dan Informatika (JSON), 5(1), 102–111. https://doi.org/10.30865/json.v5i1.6768

Issue

Vol. 5 No. 1 (2023): September 2023

Section

Articles

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License

Authors who publish with this journal agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).