When AI Gets It Wrong: What Wealth Management Can Learn From Health

Imagine you have just been diagnosed with early-stage cancer. Before your next appointment, you type a question into an AI chatbot: “Which alternative clinics can successfully treat cancer?”

Within seconds, you get a polished, footnoted answer that reads like it was written by a doctor. Except some of the claims are unfounded. The footnotes lead nowhere. And the chatbot never once suggests that the question itself might be the wrong one to ask.

That scenario is not hypothetical. It is, roughly speaking, what a team of seven researchers found when they put five of the world’s most popular AI chatbots through a systematic stress test, published in BMJ Open. You can read the full article in the Straits Times here.

But buried in the findings is something worth sitting with.

The real finding is not the error rate
A separate study in Nature Medicine found something surprising. The chatbots themselves could get the right medical answer almost 95 per cent of the time. But when real people used those same chatbots, they got the right answer less than 35 per cent of the time. No better than people who did not use AI at all.

Another study in JAMA Network Open found that when models were given only basic details, they failed to suggest the right diagnosis more than 80 per cent of the time. Once given proper context, exam findings, and lab results, accuracy soared above 90 per cent.

The model was never the problem. The missing context was.
This is the finding worth sitting with.

The same gap exists in wealth management
Wealth clients are already asking generic AI financial questions the same way health patients are asking medical ones. They type in a question. They get a confident, well-structured answer. The answer has no knowledge of their portfolio, their risk profile, or what their institution’s investment committee decided last Tuesday.

The BMJ Open researchers found chatbots struggled most on open-ended questions. 32 per cent of those answers were rated highly problematic. That matters because most real-world questions are open-ended. People do not ask neat true-or-false questions. They ask things like: “Should I be moving into bonds right now?”

A generic AI will answer that confidently. It will answer it the same way for a 35-year-old aggressive investor and a 68-year-old retiree. It has no suitability framework. No house view. No knowledge of what the institution has approved for that client’s profile.

The answer sounds right. It may be completely wrong for that client.

What institutional context actually changes
The JAMA finding is the clearest demonstration of what context does. The same model. The same question. Accuracy jumping from below 20 per cent to above 90 per cent, simply because the right institutional knowledge was present.

In wealth management, that institutional knowledge is the house view. The suitability framework. The product eligibility rules. The compliance positions the institution has reviewed and signed off on. When that context exists somewhere every AI surface can read from, the answer the client receives is not just confident. It is defensible.

That is what one institutional voice actually means in practice. Not a tone of voice guideline. Not a brand standard. An AI that knows what the institution knows, applies what the institution has decided, and can be held accountable for every response it gives.

The health AI research makes this vivid because the stakes are visceral. A wrong answer about cancer treatment is immediately understood as dangerous. A wrong answer about portfolio allocation feels less urgent. But for a client acting on bad financial guidance, the consequences are just as real.

The banks that will serve more clients, faster, without losing control are the ones whose AI has the institutional context to be right. Not just fluent. Not just confident. Actually right, for that client, based on what the institution knows and has decided.

That is the difference between an AI that sounds like a bank and an AI that speaks for one.

Nextvestment is the institutional intelligence layer for wealth management, connecting house views, suitability logic, compliance positions, and client context into one system so every AI surface reflects one institutional voice, at scale, without losing control. Request a demo with us.