Perbandingan Kinerja dan Efisiensi Model NLP pada Analisis Sentimen Ulasan Aplikasi Layanan Publik Digital

Irfan Setiawan; Widyastuti Andriyani

doi:10.30865/json.v7i4.9774

Authors

Irfan Setiawan Universitas Teknologi Digital Indonesia
Widyastuti Andriyani Universitas Teknologi Digital Indonesia

DOI:

https://doi.org/10.30865/json.v7i4.9774

Keywords:

Analisis Sentimen, Aplikasi Layanan Publik Digital, IndoBERT, IndoBERTweet, Machine Learning, Natural Language Processing, Studi Komparatif

Abstract

Transformasi digital layanan publik di Indonesia menghasilkan volume ulasan pengguna yang besar pada Google Play Store, khususnya untuk aplikasi layanan publik digital seperti Identitas Kependudukan Digital (IKD), BPJS Kesehatan Mobile, dan MyPertamina. Penelitian ini bertujuan untuk membandingkan kinerja dan efisiensi komputasi lima model dari tiga generasi pendekatan Natural Language Processing (NLP), yaitu Naive Bayes, Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), IndoBERT, dan IndoBERTweet dalam tugas analisis sentimen ulasan berbahasa Indonesia. Evaluasi dilakukan pada performa klasifikasi menggunakan accuracy, macro precision, macro recall, dan macro F1-Score serta efisiensi komputasi melalui waktu pelatihan, waktu inferensi, dan penggunaan memori. Dataset dikumpulkan melalui scraping ulasan Google Play Store dengan strategi pelabelan otomatis berbasis rating bintang (weak labels), yang keandalannya divalidasi dengan subset sampel menggunakan Cohen’s Kappa. Penggunaan label lemah ini merupakan keterbatasan yang perlu dipertimbangkan dalam interpretasi hasil, mengingat hanya sebagian kecil data yang divalidasi secara manual. Penelitian ini mengisi gap literatur pada domain aplikasi layanan publik digital Indonesia yang masih kurang dieksplorasi dalam konteks komparasi model NLP lintas generasi, sekaligus menghasilkan implikasi praktis pemilihan model berdasarkan kondisi infrastruktur komputasi yang tersedia, dengan mempertimbangkan keterbatasan generalisasi hasil pada domain dan skala dataset yang berbeda. Hasil eksperimen menunjukkan IndoBERTweet mencapai performa tertinggi dengan macro F1-Score 0,8957, sementara Naive Bayes dan SVM dapat dijalankan dalam waktu di bawah 0,15 detik tanpa GPU dengan macro F1-Score masing-masing 0,8345 dan 0,8402.

References

[1] Republik Indonesia, Peraturan Presiden Nomor 95 Tahun 2018 tentang Sistem Pemerintahan Berbasis Elektronik. Jakarta, Indonesia, 2018.

[2] Kementerian Energi dan Sumber Daya Mineral, Keputusan Menteri ESDM Nomor 37.K/HK.02/MEM.M/2022 tentang Jenis Bahan Bakar Minyak Khusus Penugasan (JBKP). Jakarta, Indonesia, 2022.

[3] Republik Indonesia, Undang-Undang Nomor 25 Tahun 2009 tentang Pelayanan Publik. Jakarta, Indonesia, 2009.

[4] L. Ashbaugh and Y. Zhang, “A Comparative Study of Sentiment Analysis on Customer Reviews Using Machine Learning and Deep Learning,” Computers, vol. 13, no. 12, pp. 1–16, Dec. 2024, doi: 10.3390/computers13120340.

[5] A. Patel, P. Oza, and S. Agrawal, “Sentiment Analysis of Customer Feedback and Reviews for Airline Services using Language Representation Model,” Procedia Comput. Sci., vol. 218, pp. 2459–2467, 2023, doi: 10.1016/j.procs.2023.01.221.

[6] H. C. Husada and A. S. Paramita, “Analisis Sentimen Pada Maskapai Penerbangan di Platform Twitter Menggunakan Algoritma Support Vector Machine (SVM),” Teknika, vol. 10, no. 1, pp. 18–26, Feb. 2021, doi: 10.34148/teknika.v10i1.311.

[7] W. Andriyani, Y. Astuti, B. A. Wisesa, and D. Hengki, “Analisis Sentimen pada Ulasan Produk dengan SVM dan Word2Vec,” JIKO (Jurnal Informatika dan Komputer), vol. 8, no. 1, pp. 173–185, Feb. 2024, doi: 10.26798/jiko.v8i1.1498.

[8] L. Xiaoyan, R. C. Raga, and S. Xuemei, “GloVe-CNN-BiLSTM Model for Sentiment Analysis on Text Reviews,” J. Sens., vol. 2022, 2022, doi: 10.1155/2022/7212366.

[9] S. Mutmainah, D. H. Fudholi, and S. Hidayat, “Analisis Sentimen dan Pemodelan Topik Aplikasi Telemedicine Pada Google Play Menggunakan BiLSTM dan LDA,” Jurnal Media Informatika Budidarma, vol. 7, no. 1, p. 312, Jan. 2023, doi: 10.30865/mib.v7i1.5486.

[10] J. M. Halim, P. N. Hoshe, S. Soenarto, and R. Sutoyo, “Comparative Analysis of Machine Learning, Deep Learning, and IndoBERT Models Using SPAMID-PAIR Dataset,” in Proceedings - 2024 International of Seminar on Application for Technology of Information and Communication: Smart And Emerging Technology for a Better Life, iSemantic 2024, Institute of Electrical and Electronics Engineers Inc., 2024, pp. 253–259. doi: 10.1109/iSemantic63362.2024.10762382.

[11] K. S. Nugroho, A. Y. Sukmadewa, H. Wuswilahaken DW, F. A. Bachtiar, and N. Yudistira, “BERT Fine-Tuning for Sentiment Analysis on Indonesian Mobile Apps Reviews,” in Proceedings of the 6th International Conference on Sustainable Information Engineering and Technology, in SIET ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 258–264. doi: 10.1145/3479645.3479679.

[12] F. Koto, J. H. Lau, and T. Baldwin, “INDOBERTWEET: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Nov. 2021, pp. 10660–10668. doi: 10.18653/v1/2021.emnlp-main.833.

[13] I. N. Kusuma et al., “Analisis Sentimen Pada Pengguna Aplikasi Dana Menggunakan Algoritma Naive Bayes,” Jurnal Mahasiswa Teknik Informatika, vol. 8, no. 2, p. 1470, Apr. 2024, doi: 10.36040/jati.v8i2.9041.

[14] N. Aulia, S. N. Sari, and N. Wakhidah, “Analisis Sentimen Aplikasi Get Contact di APP Store Menggunakan Metode SVM (Support Vector Machine),” Jurnal Informatika: Jurnal pengembangan IT, vol. 10, no. 1, pp. 139–148, 2025, doi: 10.30591/jpit.v10i1.8057.

[15] A. Rahman Isnain, H. Sulistiani, B. Miftaq Hurohman, A. Nurkholis, and Styawati, “Analisis Perbandingan Algoritma LSTM dan Naive Bayes untuk Analisis Sentimen,” Jurnal Edukasi dan Penelitian Informatika, vol. 8, no. 2, pp. 299–303, Aug. 2022, doi: 10.26418/jp.v8i2.54704.

[16] H. Jayadianti, W. Kaswidjanti, A. Utomo, S. Saifullah, F. A. Dwiyanto, and R. Drezewski, “Sentiment analysis of Indonesian reviews using fine-tuning IndoBERT and R-CNN,” ILKOM Jurnal Ilmiah, vol. 14, no. 3, pp. 348–354, 2022, doi: 10.33096/ilkom.v14i3.1505.348-354.

[17] R. Maulana, A. Voutama, and T. Ridwan, “Analisis Sentimen Ulasan Aplikasi MyPertamina pada Google Play Store menggunakan Algoritma NBC,” Jurnal Teknologi Terpadu, vol. 9, no. 1, pp. 42–48, Jul. 2023, doi: 10.54914/jtt.v9i1.609.

[18] A. I. Tanggraeni and M. N. N. Sitokdana, “Analisis Sentimen Aplikasi E-Government Pada Google Play Menggunakan Algoritma Naïve Bayes,” JATISI, vol. 9, no. 2, pp. 785–795, Jun. 2022, doi: 10.35957/jatisi.v9i2.1835.

[19] Tarwoto, R. Nugroho, N. Azka, and W. S. R. Graha, “Analisis Sentimen Ulasan Aplikasi Mobile JKN di Google PlayStore Menggunakan IndoBERT,” Jurnal JTIK (Jurnal Teknologi Informasi dan Komunikasi), vol. 9, no. 2, pp. 495–505, Apr. 2025, doi: 10.35870/jtik.v9i2.3340.

[20] Dhendra and V. G. Utomo, “Benchmarking IndoBERT and Transformer Models for Sentiment Classification on Indonesian E-Government Service Reviews,” Jurnal Transformatika, vol. 23, no. 1, pp. 86–95, Jun. 2025, doi: 10.26623/transformatika.v23i1.12095.

[21] A. A. Chamid, Widowati, and R. Kusumaningrum, “Labeling Consistency Test of Multi-Label Data for Aspect and Sentiment Classification Using the Cohen Kappa Method,” Ingenierie des Systemes d’Information, vol. 29, no. 1, pp. 161–167, Feb. 2024, doi: 10.18280/isi.290118.

[22] N. Aliyah Salsabila, Y. Ardhito Winatmoko, A. Akbar Septiandri, and A. Jamal, “Colloquial Indonesian Lexicon,” in 2018 International Conference on Asian Language Processing (IALP), Nov. 2018, pp. 226–229. doi: 10.1109/IALP.2018.8629151.

[23] D. Jurafsky and J. H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models, 3rd ed. 2026. Accessed: Apr. 16, 2026. [Online]. Available: https://web.stanford.edu/~jurafsky/slp3/

[24] M. R. Haziq, Y. Sibaroni, and S. S. Prasetyowati, “Word Embedding Optimization In Sentiment Analysis Of Reviews On Mytelkomsel App Using Long Short-Term Memory And Synthetic Minority Over-Sampling Technique,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 6, pp. 1581–1589, Dec. 2024, doi: 10.52436/1.jutif.2024.5.6.2498.

[25] J. Eisenstein, Introduction to Natural Language Processing. Cambridge, MA, USA: The MIT Press, 2019.

[26] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947, doi: 10.1007/BF02295996.

[27] O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Sci. Rep., vol. 14, no. 1, p. 6086, 2024, doi: 10.1038/s41598-024-56706-x.

[28] F. Wang, “Comparative Evaluation of Sentiment Analysis Methods: From Traditional Techniques to Advanced Deep Learning Models,” Applied and Computational Engineering, vol. 105, pp. 23–29, Apr. 2024, doi: 10.54254/2755-2721/105/2024TJ0056.

[29] Y. Jiang, P. C.-I. Pang, D. Wong, and H. Y. Kan, “Natural Language Processing Adoption in Governments and Future Research Directions: A Systematic Review,” Applied Sciences, vol. 13, no. 22, 2023, doi: 10.3390/app132212346.

[30] P. S. Ghatora, S. E. Hosseini, S. Pervez, M. J. Iqbal, and N. Shaukat, “Sentiment Analysis of Product Reviews Using Machine Learning and Pre-Trained LLM,” Big Data and Cognitive Computing, vol. 8, no. 12, pp. 1–18, Dec. 2024, doi: 10.3390/bdcc8120199.

[31] J. R. Jim, M. A. R. Talukder, P. Malakar, M. M. Kabir, K. Nur, and M. F. Mridha, “Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review,” Natural Language Processing Journal, vol. 6, p. 100059, 2024, doi: 10.1016/j.nlp.2024.100059.