Transformer-based complaint classification: evidence from 951 ivr line customer contact center of the commercial bank of ethiopia
No Thumbnail Available
Date
2026-11-02
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
. Addis ababa university
Abstract
Effective customer complaint handling is essential for delivering quality services in banks. CBE records thousands of complaints on a daily basis through its toll-free number 951 using the Interactive Voice Response (IVR). It has become clear from the current manual complaint classification strategy that there are significant problems within this system. Customer service agents were found to be classifying customer complaints randomly without paying attention to their contents, resulting in a classification accuracy of 7.0% and a macro F1 score of 6.0%. The DSR method has been used to create a Natural Language Processing (NLP) tool for classifying CBE 951 IVR customer complaints into one of the six categories: Operational Related, Behavioural Related, Fraud Related, General, Suggestion/Comment, and Product/Procedure Related. A total of 9,762 customer complaint cases were used to train and validate four transformer models such as BERT-base, finbert, afriberta, and XLM-R. The Triple-Stream Ensemble which is a combination of three transformers including finbert, afriberta, and XLM-R recorded a high validation accuracy of 88.2% and macro-F1-score of 87.5%. On the other hand, TF-IDF + SVM classical baseline experimented on the same dataset gained weighted F1 = 78.19%. The most performing transformer in this study was the finbert batch size 32 that reported a weighted F1-Score of 0.9025 and accuracy of 90.29%. In this case, the proposed solutions have resulted in a huge increase of 81.2% improvement over the baseline model. This shows the usefulness of using transformers designed specifically for certain languages/ domains.
Description
Keywords
Customer Complaint Classification, NLP, Transformers, FinBERT, AfriBERTa, XLM-R, CBE, Design Science Research, 951 IVR.