Implementasi Model Sentence-BERT untuk Peningkatan Semantic Retrieval pada Sistem Question Answering Layanan Helpdesk Ticketing
DOI:
https://doi.org/10.30865/jurikom.v15i5.10021Keywords:
Question Answering, Closed-Domain QA, Semantic Retrieval, SBERT, Helpdesk SystemsAbstract
Sistem Question Answering (QA) terutama closed-domain QA memberikan efisiensi dan konsistensi pada layanan helpdesk untuk menyajikan jawaban yang relevan dan akurat. Secara tradisional sistem QA sangat bergantung pada kesamaan kata antara query dan dokumen, sehingga rentan terhadap permasalahan lexical gap. Untuk mengatasi keterbatasan tersebut, pendekatan semantik dikembangkan guna menangkap kesamaan makna secara kontekstual. Penelitian ini bertujuan untuk mengevaluasi dan membandingkan kinerja metode retrieval berbasis leksikal, yaitu Term Frequency-Inverse Document Frequency (TF-IDF) dan Best Matching 25 (BM25), dengan pendekatan semantik menggunakan Sentence Bidirectional Encoder Representations from Transformers (SBERT) pada sistem closed-domain QA. Evaluasi dilakukan menggunakan dataset helpdesk yang dikumpulkan secara mandiri, serta divalidasi menggunakan dataset benchmark, seperti SQuAD v1.1, IDK-MRC, TechQA, dan COVID-QA. Kinerja sistem diukur menggunakan metrik information retrieval standar, meliputi Recall@k, Mean Reciprocal Rank (MRR), dan Normalized Discounted Cumulative Gain (NDCG). Hasil eksperimen menunjukkan bahwa SBERT memberikan kinerja yang sangat kompetitif dan secara umum mencapai hasil terbaik pada dataset yang dievaluasi. Pada dataset Helpdesk, SBERT mencapai Recall@1 99,814%, MRR 99,902%, dan NDCG@3 99,922%. Namun, pada dataset TechQA, TF-IDF dan BM25 menghasilkan Recall@1 yang lebih tinggi dibandingkan SBERT. Temuan ini menunjukkan keunggulan pendekatan semantik tidak selalu berlaku pada setiap dataset, terutama ketika pertanyaan dan jawaban memiliki kemiripan leksikal yang kuat. Secara keseluruhan, hasil penelitian menunjukkan SBERT merupakan pendekatan yang efektif dan kompetitif untuk sistem closed-domain QA.
References
[1] Republik Indonesia, Undang-undang (UU) Nomor 24 Tahun 2011 tentang Badan Penyelenggara Jaminan Sosial. 2011. [Online]. Available: https://peraturan.bpk.go.id/Details/39268
[2] D. Wang, D. Zhong, and L. Li, “A comprehensive study of the role of cloud computing on the information technology infrastructure library (ITIL) processes,” Libr. Hi Tech, vol. 40, no. 6, pp. 1954–1975, Dec. 2022, doi: 10.1108/LHT-01-2021-0031.
[3] A. K. Swain and V. R. Garza, “Key Factors in Achieving Service Level Agreements (SLA) for Information Technology (IT) Incident Resolution,” Inf. Syst. Front., vol. 25, no. 2, pp. 819–834, Apr. 2023, doi: 10.1007/s10796-022-10266-5.
[4] D. MacLean and R. Titah, “Implementation and impacts of IT Service Management in the IT function,” Int. J. Inf. Manage., vol. 70, p. 102628, Jun. 2023, doi: 10.1016/j.ijinfomgt.2023.102628.
[5] A. Zangari, M. Marcuzzo, M. Schiavinato, A. Gasparetto, and A. Albarelli, “Ticket automation: An insight into current research with applications to multi-level classification scenarios,” Expert Syst. Appl., vol. 225, p. 119984, Sep. 2023, doi: 10.1016/j.eswa.2023.119984.
[6] F. Al-Hawari and H. Barham, “A machine learning based help desk system for IT service management,” J. King Saud Univ. - Comput. Inf. Sci., vol. 33, no. 6, pp. 702–718, Jul. 2021, doi: 10.1016/j.jksuci.2019.04.001.
[7] B. M. Osman and A. M. S. Muse, “Enhancing Helpdesk Efficiency through Management Information System – A Resource Allocation Study in Technology Firms,” Qual. Innov. Prosper., vol. 28, no. 3, pp. 63–81, Nov. 2024, doi: 10.12776/qip.v28i3.2034.
[8] B. Bernárdez, J. I. Panach, J. A. Parejo, A. Durán, N. Juristo, and A. Ruiz-Cortés, “An empirical study to evaluate the impact of mindfulness on helpdesk employees,” Sci. Comput. Program., vol. 230, p. 102977, Aug. 2023, doi: 10.1016/j.scico.2023.102977.
[9] A. Revina, K. Buza, and V. G. Meister, “IT Ticket Classification: The Simpler, the Better,” IEEE Access, vol. 8, pp. 193380–193395, 2020, doi: 10.1109/ACCESS.2020.3032840.
[10] S. S. Ali Zaidi, M. M. Fraz, M. Shahzad, and S. Khan, “A multiapproach generalized framework for automated solution suggestion of support tickets,” Int. J. Intell. Syst., vol. 37, no. 6, pp. 3654–3681, Jun. 2022, doi: 10.1002/int.22701.
[11] S. Ahmed et al., “An Empirical Analysis of State-of-Art Classification Models in an IT Incident Severity Prediction Framework,” Appl. Sci., vol. 13, no. 6, p. 3843, Mar. 2023, doi: 10.3390/app13063843.
[12] M. A. Calijorne Soares and F. S. Parreiras, “A literature review on question answering techniques, paradigms and systems,” J. King Saud Univ. - Comput. Inf. Sci., vol. 32, no. 6, pp. 635–646, Jul. 2020, doi: 10.1016/j.jksuci.2018.08.005.
[13] A. Farea and F. Emmert-Streib, “Understanding question-answering systems: Evolution, applications, trends, and challenges,” Eng. Appl. Artif. Intell., vol. 156, p. 110997, Sep. 2025, doi: 10.1016/j.engappai.2025.110997.
[14] G. M. Biancofiore, Y. Deldjoo, T. Di Noia, E. Di Sciascio, and F. Narducci, “Interactive Question Answering Systems: Literature Review,” ACM Comput. Surv., vol. 56, no. 9, pp. 1–38, Oct. 2024, doi: 10.1145/3657631.
[15] M. A. Kia, A. Garifullina, M. Kern, J. Chamberlain, and S. Jameel, “Adaptable Closed-Domain Question Answering Using Contextualized CNN-Attention Models and Question Expansion,” IEEE Access, vol. 10, pp. 45080–45092, 2022, doi: 10.1109/ACCESS.2022.3170466.
[16] A. Formica, I. Mele, and F. Taglino, “A template-based approach for question answering over knowledge bases,” Knowl. Inf. Syst., vol. 66, no. 1, pp. 453–479, Jan. 2024, doi: 10.1007/s10115-023-01966-8.
[17] K. Nassiri and M. Akhloufi, “Transformer models used for text-based question answering systems,” Appl. Intell., vol. 53, no. 9, pp. 10602–10635, May 2023, doi: 10.1007/s10489-022-04052-8.
[18] Q. Guo, S. Cao, and Z. Yi, “A medical question answering system using large language models and knowledge graphs,” Int. J. Intell. Syst., vol. 37, no. 11, pp. 8548–8564, Nov. 2022, doi: 10.1002/int.22955.
[19] M. Laugiwa and E. Yulianti, “Question Answering through Transfer Learning on Closed-Domain Educational Websites,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 9, no. 1, pp. 104–110, Feb. 2025, doi: 10.29207/resti.v9i1.6163.
[20] N. Fujishiro, Y. Otaki, and S. Kawachi, “Accuracy of the Sentence-BERT Semantic Search System for a Japanese Database of Closed Medical Malpractice Claims,” Appl. Sci., vol. 13, no. 6, p. 4051, Mar. 2023, doi: 10.3390/app13064051.
[21] M. D. S. Antariksa, A. Sugiharto, and B. Surarso, “BERT Model Fine-tuned for Scientific Document Classification and Recommendation,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 9, no. 4, pp. 754–764, Aug. 2025, doi: 10.29207/resti.v9i4.6789.
[22] K. A. Hambarde and H. Proença, “Information Retrieval: Recent Advances and Beyond,” IEEE Access, vol. 11, pp. 76581–76604, 2023, doi: 10.1109/ACCESS.2023.3295776.
[23] N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, and B. Alshemaimri, “BERT applications in natural language processing: a review,” Artif. Intell. Rev., vol. 58, no. 6, p. 166, Mar. 2025, doi: 10.1007/s10462-025-11162-5.
[24] L. R. Aini and Evi Yulianti, “Expertise Retrieval Using Adjusted TF-IDF and Keyword Mapping to ACM Classification Terms,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 9, no. 3, pp. 497–505, May 2025, doi: 10.29207/resti.v9i3.6397.
[25] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 3980–3990. doi: 10.18653/v1/D19-1410.
[26] K. Peyton and S. Unnikrishnan, “A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs,” Results Eng., vol. 17, p. 100856, Mar. 2023, doi: 10.1016/j.rineng.2022.100856.
[27] J. Seo, S. Lee, L. Liu, and W. Choi, “TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation,” IEEE Access, vol. 10, pp. 39119–39128, 2022, doi: 10.1109/ACCESS.2022.3164769.
[28] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2016, pp. 2383–2392. doi: 10.18653/v1/D16-1264.
[29] R. A. Putri and A. Oh, “IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2022, pp. 6918–6933. doi: 10.18653/v1/2022.emnlp-main.465.
[30] V. Castelli et al., “The TechQA Dataset,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 1269–1278. doi: 10.18653/v1/2020.acl-main.117.
[31] T. Möller, A. Reina, R. Jayakumar, and M. Pietsch, “COVID-QA: A question answering dataset for COVID-19,” in Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, 2020. [Online]. Available: https://aclanthology.org/2020.nlpcovid19-acl.18/
