US banks seek to open vendors’ black box on green data

The US Federal Reserve and the country’s largest banks have had their differences, especially in recent months. But the regulator’s first ever climate scenario analysis has revealed a common cause: frustration over the climate risk models provided by third party vendors.

The analysis was launched in January 2023, with the findings published in May this year. Among its conclusions from the exercise, the Fed observed: “Participants noted that the lack of historical data and the proprietary nature of vendor models inhibited their ability to independently assess model performance.” According to seven sources spoken to for this article, this sentiment is widespread across the banking sector, including the six global systemically important banks (G-Sibs) that participated in the scenario analysis.

“If we want informed users and a healthy discussion about these models, then we need better disclosure about the inputs and assumptions and more systematized disclosure,” says Ilmi Granoff, senior fellow at the Sabin Center for climate change law and a member of the US Treasury’s climate-related financial risk advisory committee.

In fact, experts question the value of risk assessments if the inputs and models cannot be explained. Ramalingam Saravanan, head of the department of atmospheric sciences at Texas A&M University, does not think the process banks and their vendors used for the scenario analysis was “open and transparent”.

Participants noted that the lack of historical data and the proprietary nature of vendor models inhibited their ability to independently assess model performance

Federal Reserve

“The danger is that you get a false position if you have numbers that you think are more precise than they actually are,” says Saravanan.

The Fed’s climate analysis was a learning exercise, and participating banks did not face supervisory or capital penalties. Its objective was to help supervisors better understand banks’ climate risk-management practices and challenges, and for both banks and supervisors to identify, estimate, monitor and manage climate-related financial risks. Alexander Pui, senior vice-president of climate and sustainability consulting at risk consultancy Marsh McLennan, thinks the analysis was worthwhile precisely because it flagged up the shortcomings in existing climate risk measurement.

“It is encouraging to see participating banks in the Fed Reserve pilot push for greater transparency to enable robust model validation similar in standard to credit risk, for example,” says Pui. “There seems to be an acknowledgment that stress test results cannot directly be applied to credit decisioning, as they are not fit for purpose.”

Developing climate models that can integrate into everyday risk management is important, because the Financial Stability Board ultimately wants both banks and their supervisors to move in this direction. To get to the desired destination, says Pui, third party vendors will need to be held to the same transparency standards for climate models as for more established models in areas such as market risk or credit risk.

“There should be greater scrutiny and transparency for the industry to move forward, because it serves no-one’s interests for these big players to fail,” says Pui. “I would like to see the whole industry move to a more scientific base—methodologies and rating metrics need to be reviewed with more rigour, because the risks from maladaptation or misinterpreting these numbers are significant.”

As a consequence of the test, the Fed indicated all six banks plan to invest in data, models and expertise, such as more granular climate and exposure data, enhanced modeling capabilities and better customized scenarios suited to their business models and vulnerabilities. Some are considering shifting from vendor models to in-house solutions. However, more pessimistic sources suggest regulators will need to lead by example and enhance their own understanding of climate risk, in order to shift the financial sector in the desired direction.

Roughly right, precisely wrong

The majority of sources spoken to are concerned with the unproven scientific analysis underpinning third party climate risk models that were used by banks in the physical component of the Fed’s exercise, and the risk that their black box nature may be obfuscating the uncertainty in their projections.

A major problem is the misleading “down-scaling” of high-credibility but larger resolution general circulation models (GCMs) that simulate the physics of the climate. GCMs, such as those developed by vast networks of scientists within the UN’s Intergovernmental Panel on Climate Change (IPCC), represent physical processes in the atmosphere, ocean, cryosphere and land surface. Such models are so complex they can only be developed and run on supercomputers.

Whereas GCMs represent a top-down approach to modeling impacts, meaning the physical outputs are linked to projected changes in the whole global environment and economy, bottom-up approaches such as catastrophe models were developed to assess acute risk in certain locations at a granular scale. Bottom-up approaches now also include a new universe of providers of climate data and modeling services, which translate GCM outputs into financial risk terms, such as an expected loss.

We see certain companies cropping up who are making these outlandish claims about how accurately they can predict the future

Jay Guin, Verisk

Marsh McLennan’s Pui believes the vendor market has opted for increased granularity because “it is an easier sell” to banks that need to estimate the losses in their specific portfolios. But he worries “the market can often confuse precision for accuracy”.

According to Boston University law professor Madison Condon, climate model forecast reliability depends on the information being sought, such as how specific you are trying to be on the type of event, the timing and time horizon of the risk manifesting, and the location and precision of that location. Uncertainty typically goes up as these lenses narrow, but many new climate risk modelers have staked a commercial niche by downscaling GCMs to arrive at very granular information, for example based on individual postal codes.

“We see certain companies cropping up who are making these outlandish claims about how accurately they can predict the future,” says Jay Guin, chief research officer of extreme event solutions at catastrophe modeler Verisk, who distinguishes between the decades-old business of catastrophe modeling and the much more recent business of climate risk modeling. 

These companies’ methodologies and therefore their range of predictions vary significantly, Guin adds: “This is where I have seen expressions of concern from academia, because they are the ones who are building these models and have a very deep understanding of what the models are capable of doing and what they are not capable of doing.”

Incoherent outcomes

The consequences are borne out in the Fed’s scenario analysis results. Unlike other jurisdictions, the regulator opted to split out physical and transitions risks in its climate scenario analysis in a bid to focus on banks’ abilities to model these components separately. The findings revealed wide differences in the six G-Sibs’ approaches to these two components, driven by banks’ business models, views on risk, access to data, and participation in other jurisdictions’ similar exercises.

“Some” of the six G-Sibs were undertaking climate scenario analysis for the first time, and “many” relied on external vendors to fill modeling and data gaps. According to the Fed, “significant” challenges included lack of data on building characteristics, insurance coverage, and counterparties’ plans to manage climate risks.

Most but not all banks relied on existing credit risk models to estimate portfolio impact and assumed historical relationships between model inputs and outputs would hold. However, some indicated to the Fed that models could be “enhanced” to better capture climate transmission channels and their impacts.

Those banks that reviewed their modeling frameworks in light of the Fed’s exercise were hampered by limited data, the lack of back-testing capabilities for a risk that has not yet fully materialized, the non-linear nature of climate risks and a 10-year scenario horizon, according to the regulator. The Fed’s report also noted “heavy reliance on judgment, limited reliability of model outputs, and time constraints”.

These shortcomings manifested themselves in probability of default and loss-given-default estimates for credit exposures, which showed “significant heterogeneity in impact across sectors, regions, and counterparties” between different banks. Even for common obligors, estimates of transition risk impacts diverged by 100 basis points for 20% of loans and by 500bp for 6% of loans, according to the Fed.

One of the most concerning findings in the common shock physical risk module, according to three sources, was that projected commercial real estate losses modeled for a one in 200-year hurricane versus a one in 100-year hurricane decreased slightly. This is “physically not sensible”, says Scott St George, head of weather and climate research at insurance broker Willis Tower Watson. The rarer event would by definition be larger and more severe, so finding it causes lower losses is statistically incoherent and counterintuitive.

“It shouldn’t go down, it should go up, and the fact that it goes down indicates the tail risk behavior of extreme storms may not be represented correctly,” says St George. “It makes me a little bit concerned that the banks are struggling to represent the impact of very rare climate related perils… and reminds me that this is a practice that should not be taken literally as an indication of exposure in banks.”

One climate modeling source is also concerned that in the idiosyncratic shock part of the physical risk module banks, reported increased losses by almost 450 basis points for commercial real estate loans versus increased losses by approximately 150 basis points for residential mortgages. “That’s insane—that’s just not even believable,” says the source, pointing out that global insurance loss rates for commercial real estate are lower than for residential property.

Yanjun Liao, an economist at the non-profit research institution Resources for the Future, believes this could be accounted for by the design limitations of the exercise, which did not require the modeling of indirect impacts or future changes to insurance rates or insurance availability. That could mean losses are underestimated if some properties in banks’ portfolios become uninsurable.

“I think that is probably truer for the residential than the commercial sector, so this could represent a lower bound of the potential impact,” says Liao.

Show us your workings

The problems with physical risk models are not limited to their use in banking. Catastrophe models, are a mainstay of insurance sector risk management. Yet many insurers don’t have insight into their workings, says Liao.

In the academic and public sector climate modeling community, the source code for running models is available and results are published in peer-reviewed literature, allowing for the analysis of their strengths and weaknesses.

“Once you get to catastrophe models and these transition models, they are proprietary,” says Saravanan. “I can’t assess them, and I’m not sure people in the field have the information to decide if this model is good or bad.”

The broader issue has already come to the attention of the President’s Council of Advisers on Science and Technology. In April 2023, it recommended that federal government should challenge the private sector to improve its tools for predicting the severity and frequency of weather hazards and the human and economic losses they will cause.

Saravanan and St George took part in a later industry roundtable arranged by the White House council of economic advisers in September 2023 that also grappled with the issue of scientific quality and reliability of climate risk data in the private sector. Participants discussed how to improve scientific focus, transparency and peer-reviewed research in climate risk modeling, how to strengthen the role of Federal climate science institutions, and how to improve public-private cooperation.

Vendors respond

Risk.net, a sibling publication of WatersTechnology, contacted a number of climate model vendors for comment. Guin at Verisk says they are seeking to make their models “feel” less opaque, while protecting their proprietary information from competitors. As well as peer-reviewing new models and updates internally, catastrophe modeler Verisk selects external academics to peer review “on a need basis… certain components”, says Guin.

Two years ago, Verisk assembled a climate advisory council of academics in response to stakeholders’ concerns about whether the company was “really representing the right view of risk given that the climate is changing”, says Guin. Verisk’s core customers are insurance firms that are highly sophisticated and constantly challenge Verisk model assumptions, he says. Typically, insurers will benchmark views of risk for a particular hazard across several model vendors, to act as a “check and balance”, Guin explains.

Climate models offered by Moody’s were used by some of the six US banks for the Fed scenario analysis, Risk.net understands, as well as in other regulatory exercises. Moody’s, like Verisk, says it spends significant time supporting clients to understand the models, including how they work and their assumptions, so that firms may reach the right decisions for their view of risk.

“These clients went through extensive internal validation exercises with their model risk management teams prior to approving our models for use in these exercises—all of which clients were able to successfully complete with the outputs, documentation and guidance available from Moody’s,” says Theresa Lederer, associate director of risk management at Moody’s.

A spokesperson for MSCI, another major climate risk model vendor, says: “MSCI climate solutions follow rules-based, transparent methodologies that are publicly available on our website.”

Banks’ knowledge gap

Three sources who do not work for vendors say at least some of the responsibility for improvement has to lie with the banks themselves. Even if there were greater vendor model transparency, the average bank risk department lacks the scientific expertise to effectively vet models. Catastrophe models, for instance, are designed by teams of hundreds of doctorate-level engineers and scientists from highly specialized sub-disciplines.

“Even if you have a doctorate in climate science, most of the time it’s hard to understand what they are doing, what goes in, what comes out, and so I imagine if you don’t have a background in climate science or you don’t have the time to get informed about it, it can be very difficult to pick one provider over another or to really understand what might be reasonable and what might be a very risky tool,” says St George.

Sources believe banks ought to employ and train scientists in their risk function. Some G-Sibs have begun to do so—JP Morgan employs oceanography and atmospheric physics experts in its London office. According to the Fed, several participants in the scenario analysis wanted to further develop in-house modelling capabilities “in order to reduce reliance on third-party vendors”. However, one source is pessimistic about whether climate modelers who lack a traditional financial risk background would be awarded sufficient seniority to be taken seriously.

“He or she will be put in a position where they won’t have enough influence,” says the source. “They won’t be a head of department and they will always be deferring to someone with a finance background, who is suddenly miraculously able to learn all the nuances and all the complexities of climate science rather than the other way around.”

Over to the regulators?

Some in the sector are calling for greater oversight and regulation of climate risk models to prevent the mis-selling of climate services based on unscientific assumptions. They are urging clearer guidance on the use, application and limitations of these tools. However, several sources also argue that the problem with the climate modeling skills gap extends from regulated entities to financial regulators themselves.

“You need to have climate scientists in the room within every stakeholder to be able to at least have a more nuanced conversation,” says Pui, who between 2022 and 2023 developed climate scenario analytics for Commonwealth Bank.

Many, if not most regulators still rely on external consultancies for advice designing climate scenario analysis exercises, and on third party vendors for the models and data underpinning these exercises. Half of the data contained in the directory compiled by the global regulatory Network for Greening the Financial System is proprietary or requires a subscription.

You need to have climate scientists in the room within every stakeholder to be able to at least have a more nuanced conversation

Alexander Pui, Marsh McLennan

The European Central Bank used Moody’s flood risk and asset location data for its 2021 supervisory climate stress test, and Canadian supervisors the Office of the Superintendent of Financial Institutions Canada will use RiskThinking.ai as the sole provider of data for its 2024 standardized climate scenario analysis. Ron Dembo, founder and chief executive of RiskThinking.ai, says he hopes the exercise will lead to Canadian banks hiring the firm’s services, or will at least boost awareness of the company’s forward-looking data and stochastic models.

Condon has argued that US financial regulators must develop long-lasting collaborative relationships with non-financial US agencies that have environmental expertise.

In 2022, the US Treasury proposed creating a climate data and analytics hub housed at the financial regulators—the Office of Financial Research would be an obvious home. Condon has said agencies with climate expertise, such as the National Oceanographic and Atmospheric Administration, need to be involved in this hub. These bodies have “a wealth of climate data and scientific knowledge, yet collaboration between these agencies and the financial regulators is limited”, according to Condon’s paper.

Such a central hub might develop open source catastrophe models whose performance could be compared, and regulators could then use these models as the basis for their climate stress testing. The models would be available to the private sector to develop their own tools. However, Saravanan notes that regulators would still face the problem that most damage loss claims data is proprietary.

“If a hurricane hits a particular part of the US, what are the damages? If you know for several hurricanes we can calibrate the model, but without that data we can’t,” says Saravanan. “Insurance companies don’t want to hand the data out to their competitors.”

The US could take steps to obtain such data from insurance firms in an aggregated format, says Saravanan, while preserving the anonymity of individual policyholders and respecting proprietary information.