AI chatbots give incorrect financial advice 57% of the time, study shows

By Divya Shah

A new study by FinTech firm Saturn has raised concerns about the reliability of artificial intelligence (AI) tools for personal finance, finding that popular AI models provided inaccurate financial advice in 57% of cases. 

The report, titled ‘Artificial Authority: Should you trust AI to deliver financial advice?’, assessed 18 AI models from providers including ChatGPT, Claude, Copilot, Gemini and Grok. Researchers tested the models using 121 financial questions covering topics such as debt management, mortgages, pensions, tax, savings and student loans. 

To evaluate consistency, each question was asked five times, resulting in more than 10,000 responses being analysed. The study found that AI models delivered accurate answers only 43% of the time on average, while performance declined sharply when answering more complex financial questions. 

AI models made errors in 88% of responses to advanced finance-related queries. Some models recorded failure rates of as much as 99% on the most challenging questions. 

The research identified a range of issues, including calculation errors, omission of critical risk warnings, failure to account for upcoming tax changes and the generation of incorrect or non-existent financial rules.  

In some cases, the consequences of following inaccurate guidance could result in financial losses. 

The study also found a notable difference between free and paid AI services. Free-to-use models produced incorrect responses in 63% of cases, compared with 49% for paid models. For complex financial questions, the error rate among free models rose to 93%. 

Among the models tested,  

• The worst-performing model, Claude Haiku 4.5, made mistakes in 82% of answers.  

• Second-worst was Google’s Gemini 3.1 Pro, failing 73% of tests.  

• Grok 4.5 made mistakes 59% of the time. 

• ChatGPT 5.6 Luna made mistakes 58% of the time. 

• When asked more complex financial questions, Google’s Gemini 3.5 Flash and Claude Haiku 4.5 gave wrong answers 99% of the time. 

• The best-performing model overall was Claude Opus 5 (reasoning), which made mistakes in 39% of answers 

Researchers highlighted several examples where inaccurate advice could have serious financial consequences. One error relating to UK pension tax rules could have resulted in a consumer incurring a tax charge of £17,500. Other examples included incorrect guidance on debt prioritisation, student loan repayments, and mortgage payment holidays. 

The findings come as regulators increasingly examine the role of AI in financial decision-making. The study noted that a growing number of consumers are turning to general-purpose AI tools for financial guidance, despite the absence of the protections typically associated with regulated financial advice. 

Amal Jolly, CEO of Saturn, said, “The low quality of financial advice from mainstream AI models risks leading to widespread consumer harm. Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money.” 

He added, “AI financial advice is currently unregulated, leaving consumers with none of the protections, including compensation, that they would get if they went to a human adviser. The FCA has started to think about this, but it needs to act fast to protect people. The FCA should regulate AI to ensure consumers are protected.” 

The report highlights the growing challenge facing financial regulators and FinTech providers as AI adoption accelerates. While AI tools offer consumers fast and convenient access to information, the research suggests significant concerns remain around accuracy, transparency and accountability when AI is used for financial guidance.