Safeguarding against AI’s existential risks and small mistakes

AI’s risks and potential failings are often framed as both existential and low-level. On the existential side we have AI CEOs and former employees at major AI labs talking about the per centage risk that artificial intelligence ends human life on this planet. On the low-level side we have the experience of daily users, stories about frustrating AI ‘hallucinations,’ including a ChatGPT-planned holiday to Montreal with hotel rooms booked in Quebec City. For Conquest Planning, both ends of the AI risk spectrum have to be considered.

Brad Joudrie, CEO of Conquest Planning, says that the financial planning software firm’s integrated AI agent SAM (short for Strategic Advice Manager) has been built and maintained with those areas of risk in mind. Joudrie acknowledges that AI integrations in software come with certain elements of risk, including hallucination, agentic misalignment, and data sovereignty. He outlined what his firm does to control for those risks while seeking to give advisors access to the efficiencies that AI can provide.

“Think our industry and industries outside of it have already acknowledged that there is ultimately a threat depending on how pervasive these tools and technology are. One of the ways that we as an industry can protect ourselves is by understanding and doing the right amount of diligence,” Joudrie says. “In the work we’re doing that’s connected to agents, one of the things that those agents can stand on is that the auditability and the traceability that exists in the calculations that we’re building.”

Controlling for hallucination and error in AI

Joudrie emphasizes auditable and traceable calculations in SAM because that allows users to more clearly double-check the work done by the AI agent. AI agents have become a popular means of expediting workflows and introducing new angles of discussion between advisors and clients. While Joudrie sees why that popularity is growing, he stresses the need for the underlying data that the agents use to be verifiable by the advisor. Without that clarity the ‘small’ error of a hallucination can become a core fault in the plan and pose real long-term risks to the client.

Financial plans built in their software platform need to be living entities, pieces that can be revised and reworked based on clients’ changing circumstances. By ensuring that the work SAM does in building that plan can be traced back to initial inputs and run through the correct calculations, the plans can be revised without the risk of a core error. Joudrie insists that because of the emphasis on diligence and traceability, Conquest’s engine “doesn’t hallucinate.”

One of the sources of hallucination in other AI large language models is the sheer volume of information they pull from across the whole internet to answer a question. One of the tools Conquest has elected to use to avoid that is what they call a “small language model” which only pulls from conquest’s native environment. That more limited scope ensures that the tool is tied only to what has already been put into the system.

Managing existential risks in AI

Joudrie acknowledges that even for the kind of limited-scope and disciplined agents that his firm uses, there is a risk that AI agents can act in their own best interests rather than the interests of their users. This is one of the core risks being highlighted by AI CEOs like Anthropic’s Dario Ammodei in their statements about the risk of an AI-driven apocalypse. Some of the solution to this risk, especially in financial services, will come from regulators and dealers as they develop more complete forms of AI policy. However, Joudrie says that his team has factored this risk into their emphasis on traceability. If it appears that the AI agent is making suggestions or plans that serve its interests over the clients, the roots of those suggestions can be identified and corrected through those more traceable pathways.

Core cybersecurity principles also play a role in Conquest’s approach. They recently completed a new AI designation called ISO 42001, the world’s first international standard for an AI management system. They have also maintained a data sovereignty policy from their outset. All of Conquest’s data is stored in the country that their customers are based in. Canadian data is domiciled in Canada, UK in the UK, US in the US, all through Amazon Web Services in each country.

The other real risk to clients and advisors that Conquest stays cognizant of is the ease of using a less secure open tool. Joudrie says that many individual investors are turning to public AI tools like ChatGPT and Claude for financial advice, putting their personal information into platforms that have none of the security protocols that a tool like Conquest has. They do that, in part, because those tools are easy to use. As much as Conquest is built to be compliant and secure, Joudrie says the platform has to be user-friendly so that the innate human preference for the easy path doesn’t lead people into danger.

“People are going to at times gravitate to easy, and easy might be plugging some data into a service like Claude or OpenAI to get a financial plan, but that’s really not the best way to approach it,” Joudrie says. “So as a financial professional, you’re not sacrificing any of the speed that these engines can solve for. With Conquest, I would encourage those financial advisors that are leveraging our service to get your customer or client to stop [using public LLMs for advice]. But number two, you know, you can take that data that was produced out of those engines, load it up directly through our LLM upload, and actually construct and automatically build a financial plan in Conquest, which is going to identify those gaps or errors. So we’ve essentially looked to solve for this challenge as well.”