AI Improves Customer Query Resolution in Banking

By Milan Rojan

A peer-reviewed study has found that banking-focused AI models trained to recognise uncertainty can resolve more customer queries while operating at significantly lower cost than larger frontier models.

The research examined the performance of a production-grade, 12-billion-parameter language model deployed in banking environments and evaluated its ability to provide accurate, evidence-based responses while avoiding unsupported answers.

The findings addressed a longstanding challenge in customer-facing banking AI: balancing helpfulness with accuracy. Rather than encouraging the model to answer every query, the researchers trained it to recognise when available information was insufficient and respond with an explicit “I don’t know”. According to the study, this approach reduced unsupported responses while improving the overall resolution of customer enquiries.

Over a seven-month deployment at a large US financial institution, the model increased customer query resolution by 7.1 percentage points across a sample of 3,297 queries. It recorded a 12% refusal rate, compared with 4.3% for an untuned base model and 20.2% for GPT-4.1, indicating a balance between declining uncertain requests and providing supported responses.

The study also found that the banking-specific model achieved a higher independent evaluation score than GPT-4.1, recording 6.21 compared with 5.72 on a 10-point scale. Researchers also reported stronger citation grounding, improved performance on FinanceBench, and operating costs of around $0.001 per query on a single GPU, making it approximately 20 to 50 times less expensive and three to five times faster than GPT-4.1. Training the model reportedly cost around $1,800.

In addition, the researchers found that the sequence of training data had a significant impact on model performance. Teaching general financial knowledge before introducing calibrated refusal behaviour produced better results than combining all training data simultaneously, which reduced answer quality and increased refusal rates.

The findings have contributed to ongoing discussions around trustworthy AI deployment in financial services, where explainability, regulatory compliance and operational efficiency remain key priorities. The research has suggested that models trained to acknowledge uncertainty, rather than generate confident responses without sufficient evidence, could help financial institutions improve customer service while supporting more reliable AI-driven decision-making in regulated environments.