The bank quant who wants to stop genAI hallucinating

Ever since generative AI reached mainstream consciousness, the idea of bot hallucinations has sparked a mix of concern and amusement. As long as the practical uses for generative AI were limited, the concern was mostly theoretical. However, fears are mounting as robo-generated falsehoods have led to a slew of lawsuits for offenses ranging from defamation to negligence. 

Interpretability for complex LLMs can be accomplished by understanding how the corpus of queries and answers are embedded

Agus Sudjianto, Wells Fargo

In one instance, Air Canada was ordered last month to compensate a customer after its AI chatbot offered incorrect information on refunds.

The legal threats are focusing attention on the validation and monitoring of AI chatbots, and generative AI tools more broadly. Behind the scenes, Agus Sudjianto, the recently retired head of model risk at Wells Fargo, has been working on a tool that could help the financial industry do just that. 

“Proper model validation is a very critical piece before models are deployed to make sure that risks are known and mitigated. However, in the case of genAI, I have seen that people deployed models without validation,” Sudjianto says.

This year, Sudjianto plans to release and patent the tool, which will help to validate and monitor genAI models, first rolling it out to the banking sector he’s most familiar with, and then extending it to other industries. It is not Sudjianto’s first time entering this arena. In 2022, he released, along with others, the Python Interpretable Machine Learning (PiML) toolbox on GitHub. The tool, downloaded more than 200,000 times, offers a way to test and validate machine learning models. These are notoriously harder to test and validate than statistical models, and genAI models are a notch above.

Some suggest that full explainability of outputs—a key concept in the AI model world—will be difficult to reach in the case of genAI models. Sudjianto aims to prove them wrong, and to show why these criteria must also be applied to large language models (LLMs)—the type of model that underpins genAI tools such as ChatGPT. More than that, he doesn’t think it is viable to adopt genAI without having an established validation process in place first.

“If people rushed the deployment, we should not be surprised that they encountered problems down the road—that’s the concern I have,” says Sudjianto.

Box clever

In January, Sudjianto left Wells Fargo, where he had been the head of corporate model risk for 10 years. It was the latest stage in a career spent looking under the bonnet of increasingly complex algorithms. After finishing his doctorate in neural networks in the 1990s, Sudjianto worked for Ford Motor Company as a product development manager, Bank of America as head of quantitative risk, and Lloyds as director of analytics and modeling. 

One thing linked all these positions: Sudjianto’s interest in the development of machine learning tools. 

When Sudjianto started at Wells Fargo, banks were under pressure to upgrade their model risk management framework to cope with the qualitative element of the US Federal Reserve’s stress-testing process. As part of this work, Sudjianto and his team developed an explainability technique that was subsequently adopted by other financial firms. 

Understanding the inner workings of AI models—or peering into the so-called black box—is a long-standing concern of regulators. These worries are just as salient with the new breed of generative AI chat engines, which can be difficult to control and have unpredictable outputs. Sudjianto believes that putting clear boundaries and constraints around genAI models will be the way to use them safely. 

He thinks effective testing and validation of genAI models starts at the development stage, including setting boundaries on the input and/or output of LLMs. In a general model, inputs and outputs are constrained, whereas genAI models can in theory draw on any available information. 

GenAI models need to be bounded by purpose, Sudjianto says. For example, if models are being used for credit products, they should be limited to retrieving credit-related information. By limiting the model in this way, the output becomes more manageable to test and validate.

“[It’s] dealing with a slice instead of dealing with the entire universe,” says Sudjianto.

Typically, this bounding can be done by using retrieval augmented generation (RAG), rather than free-flowing mode. The RAG technique gives an LLM a specific context in the prompt to answer specific questions with answers that are tied to facts retrieved from a database. This doesn’t eliminate the risk of hallucination, but it narrows down the potential causes, and exercises more control over outputs than just fine-tuning the model. 

In the case of genAI, I have seen that people deployed models without validation

Agus Sudjianto, Wells Fargo

Once bounded, these models can be tested more thoroughly. Model weakness can be identified using various evaluation metrics to check the outputs are robust and relevant to changes in context or input data. This is also the point at which validators must check there are no signs of hallucination or toxic chatbot outputs that might drive away end-users. 

Sudjianto says testing and validation within his tool will be an automatic process. The tool will deal with the issue of scale by harnessing genAI to test genAI. Using AI in model testing and validation has been suggested by model risk experts before. Sudjianto says this is exactly how the tool will manage to test a wide combination of questions and prompts that come from LLMs. 

The use of genAI to test itself might ring alarm bells for some, especially given worries around bias and hallucinations. The human will not be completely taken out of the loop with this tool, says Sudjianto. At the last stage, humans should complete a final verification of a sample of outputs, guided by statistical techniques. 

All of these steps should help the model owner gain a better understanding of the relationship between the inputs and outputs of genAI models—helping to lift the lid of the black box.

No compromise

Testing the mechanics of the model is one thing. Interpreting what the model tells you is another. Sudjianto says quants have previously approached the interpretability of model outputs by referencing individual parameters. While this can be done in traditional statistical models with a few parameters, the approach will not work for machine learning models with far higher numbers of inputs. Sudjianto says a new paradigm is needed.

“For foundational models in genAI, we are dealing with a billion or trillion parameters,” he says. “Instead of looking at individual parameters, one should look at the relationships between inputs and outputs which are the results of a collection of parameters.”

Sudjianto draws a comparison with particle physics, where statistical mechanics explains phenomena using ensemble behavior, instead of trying to track individual particles. “Interpretability for complex LLMs can be accomplished by understanding how the corpus of queries and answers are embedded,” he says. 

There are still those who doubt genAI models can be validated in any conventional sense. Rather, some have suggested that observability might be a better aim than explainability. Observability means inferring the condition of the model from the quality of its outputs. 

However, Sudjianto says these sorts of compromises are not needed. He thinks observability is important for monitoring genAI, but it is essentially a reactive form of risk management.

“Instead, just like other types of models, one should thoroughly test and validate the models before deployment so that model risk management can be done proactively,” says Sudjianto. “We need to have an assurance before model deployment that it will perform well, understand the risks and know how to prevent or to manage them.”

In fact, he suggests the financial world is in a prime position to offer these validation techniques. The leading global banks are heavily regulated, which has driven substantial investment into model validation. Each of them, including Wells Fargo, employs hundreds of specialists with advanced postgraduate qualifications to carry out rigorous processes. This has also permitted model risk teams to expand into research on neural networks and LLMs.

“Other, less regulated industries may not be able to afford it,” says Sudjianto. “That’s where I want to end my career: to make model risk management—particularly model validation for complex AI models—accessible to broader communities, democratizing model risk management and model validation.”