Evaluating FAQ Based Generative AI Chatbot Performance in Islamic Banking Customer Service

Nurul Izzah Ahlam, Mohammad Givi Efgivia

This study designs and evaluates a web-based customer service chatbot for Islamic banking services. The chatbot integrates the Google Gemini API with a validated Frequently Asked Questions knowledge base to generate responses based on institutional service information. The system was developed using the Flask framework and evaluated through Black Box Testing, FAQ-based response evaluation, and chat log analytics. The evaluation used 310 validated FAQ records and 1,000 anonymized raw chat log records, which were filtered into 360 valid categorized records for interaction-pattern analysis. Based on 310 FAQ-based test questions, the chatbot produced 287 correct responses and 23 incorrect responses. The results show an accuracy of 92.58%, precision of 95.67%, recall of 96.63%, F1-score of 96.14%, success rate of 92.58%, and an average response time of 1.84 seconds. Chat log analysis further indicates that Mobile Banking was the most frequent service category requested by users. These findings suggest that combining Generative AI with a structured FAQ knowledge base can support fast and relevant customer service responses in Islamic banking.