Penerapan Model Hybrid Convolutional Neural Network dan Long Short-Term Memory untuk Pengenalan Real-Time Sistem Isyarat Bahasa Indonesia (SIBI)

Authors

  • Syahrul Hidayat Universitas Pembangunan Nasional “Veteran” Jawa Timur, Surabaya
  • Yisti Vita Via Universitas Pembangunan Nasional “Veteran” Jawa Timur, Surabaya
  • Eka Prakarsa Mandyartha Universitas Pembangunan Nasional “Veteran” Jawa Timur, Surabaya

DOI:

https://doi.org/10.30865/mib.v8i3.7837

Keywords:

The Indonesian Sign Language System (SIBI), Deaf, Sign Recognition, CNN-LSTM, Real-Time

Abstract

The Indonesian Sign Language System (SIBI) is an essential means of communication for the deaf and speech-impaired community in Indonesia. However, the limited public understanding of SIBI often hinders effective communication. This study develops a real-time SIBI sign recognition model to facilitate effective communication for the deaf and speech-impaired in Indonesia. The proposed method integrates a hybrid CNN-LSTM model to process the spatial and temporal information from the data. The study evaluates the model's performance on 25 types of SIBI signs. The dataset used consists of image sequences captured in real-time. Training is conducted with various parameters, including batch size, learning rate, and epochs. Model evaluation is carried out using accuracy, precision, recall, and f1-score metrics. The training and validation results show an increase in accuracy with the number of epochs: 87% at 10 epochs, 93% at 25 epochs, and 100% at 50 epochs. In real-time detection tests, the model with the image sequence dataset accurately detected SIBI signs in environments and with objects consistent with the dataset. The real-time detection program generates SIBI sign predictions in text form and sentences. The output of this research is efficient and accurate SIBI sign recognition technology. This research is expected to facilitate more effective communication for the deaf and speech-impaired community in Indonesia.

References

F. Iffah and Y. Fitri Yasni, “Manusia Sebagai Makhluk Sosial,†2022.

U. Mahadi, “Komunikasi yang Baik,†J. Public Policy Adm. Silampari, vol. 2, no. 2, pp. 80–90, 2021.

A. S. Nugraheni, A. P. Husain, and H. Unayah, “Optimalisasi Penggunaan Bahasa Isyarat Dengan Sibi Dan Bisindo Pada Mahasiswa Difabel Tunarungu Di Prodi Pgmi Uin Sunan Kalijaga,†J. Holistika, vol. 5, no. 1, p. 28, 2023, doi: 10.24853/holistika.5.1.28-33.

A. Pratiwi and A. Amri, “Penggunaan Sistem Isyarat Bahasa Indonesia (SIBI) sebagai media komunikasi (studi pada siswa tunarungu di SLB),†J. Ilm. Mhs. FISIP Unsyiah, vol. 4, no. 3, pp. 1–12, 2019, [Online]. Available: www.jim.unsyiah.ac.id/FISIP

Z. Zhang, P. Cui, and W. Zhu, “Deep Learning on Graphs: A Survey,†IEEE Trans. Knowl. Data Eng., vol. 34, no. 1, pp. 249–270, 2022, doi: 10.1109/TKDE.2020.2981333.

G. Gumelar, H. Hafiar, and P. Subekti, “Bahasa Isyarat Indonesia Sebagai Budaya Tuli Melalui Pemaknaan Anggota Gerakan Untuk Kesejahteraan Tuna Rungu,†INFORMASI, vol. 48, no. 1, p. 65, Jul. 2018, doi: 10.21831/informasi.v48i1.17727.

K. Xia, J. Huang, and H. Wang, “LSTM-CNN Architecture for Human Activity Recognition,†IEEE Access, vol. 8, pp. 56855–56866, 2020, doi: 10.1109/ACCESS.2020.2982225.

H. M. Putri, F. Fadlisyah, and W. Fuadi, “Pendeteksian Bahasa Isyarat Indonesia Secara Real-Time Menggunakan Long Short-Term Memory (Lstm),†J. Teknol. Terap. Sains 4.0, vol. 3, no. 1, p. 663, 2022, doi: 10.29103/tts.v3i1.6853.

G. Khartheesvar, M. Kumar, A. K. Yadav, and D. Yadav, “Automatic Indian sign language recognition using MediaPipe holistic and LSTM network,†Multimed. Tools Appl., 2023, doi: 10.1007/s11042-023-17361-y.

P. Sinha, D. Kumar, and A. Prakash, “Real Time Sign Language Prediction Using Cnn and Lstm,†no. 04, pp. 7304–7316, 2023.

F. Marpaung, F. Aulia, and R. C. Nabila, Computer Vision Dan Pengolahan Citra Digital. 2022.

Y. Vita Via, W. S.J. Saputra, M. Idham Fachrurrozi, E. Yulia Puspaningrum, F. Tri Anggraeny, and S. Rohman Nudin, “Object Localization and Detecting Alphabet in Sign Language BISINDO Using Convolution Neural Network,†Rom. J. Ofapplied Sci. Technol., vol. XVI, no. 1, pp. 143–149, 2023.

K. R. Nur Manab, E. P. Mandyartha, and A. M. Rizki, “Rancang Bangun Sistem Deteksi Huruf Rusia Berbasis Web Flask,†Pros. Semin. Nas. Inform. Bela Negara, vol. 2, pp. 156–160, 2021, doi: 10.33005/santika.v2i0.108.

L. Alzubaidi et al., Review of deep learning: concepts, CNN architectures, challenges, applications, future directions, vol. 8, no. 1. Springer International Publishing, 2021. doi: 10.1186/s40537-021-00444-8.

M. I. Choldun R and K. Surendro, “Klasifikasi Penelitian Dalam Deep Learning,†Improv. J. Ilm. Manaj. Inform., vol. 10, no. 1, pp. 25–33, 2018.

E. Rasywir, R. Sinaga, and Y. Pratama, “Evaluasi Pembangunan Sistem Pakar Penyakit Tanaman Sawit dengan Metode Deep Neural Network (DNN),†J. Media …, vol. 4, no. 5, pp. 1206–1215, 2020, doi: 10.30865/mib.v4i4.2518.

A. Roihan, P. A. Sunarya, and A. S. Rafika, “Pemanfaatan Machine Learning dalam Berbagai Bidang: Review paper,†IJCIT (Indonesian J. Comput. Inf. Technol., vol. 5, no. 1, pp. 75–82, 2020, doi: 10.31294/ijcit.v5i1.7951.

R. R. Pratama, “Analisis Model Machine Learning Terhadap Pengenalan Aktifitas Manusia,†MATRIK J. Manajemen, Tek. Inform. dan Rekayasa Komput., vol. 19, no. 2, pp. 302–311, 2020, doi: 10.30812/matrik.v19i2.688.

I. W. Widya, I. Gede, and A. Wibawa, “Klasifikasi Bentuk Wajah Manusia Menggunakan Metode Convolutional Neural Network (CNN),†vol. 1, no. November, pp. 373–378, 2022.

M. R. Alwanda, R. P. K. Ramadhan, and D. Alamsyah, “Implementasi Metode Convolutional Neural Network Menggunakan Arsitektur LeNet-5 untuk Pengenalan Doodle,†J. Algoritm., vol. 1, no. 1, pp. 45–56, 2020, doi: 10.35957/algoritme.v1i1.434.

A. Sherstinsky, “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network,†Phys. D Nonlinear Phenom., vol. 404, p. 132306, 2020, doi: https://doi.org/10.1016/j.physd.2019.132306.

M. W. P. Aldi, Jondri, and A. Aditsania, “Analisis dan Implementasi Long Short Term Memory Neural Network untuk Prediksi Harga Bitcoin,†e-Proceeding Eng. Vol.5 No.2, vol. 5, no. 2, pp. 3548–3555, 2018.

Downloads

Published

2024-07-27

Issue

Vol. 8 No. 3 (2024): Juli 2024

Section

Articles

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License

Authors who publish with this journal agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).