Jikin Shah is something of a human Swiss Army knife. He studied instrumentation and control engineering at Gujarat University, about an hour’s flight north of Mumbai. After gaining some work experience, he moved to the US and earned a master’s degree at Marist College in New York.

In 2000, he landed at MetLife and while working at the insurance giant, decided to pursue an MBA in finance. By 2008, he had earned the MBA and worked in five different roles at MetLife, ranging from building out its middleware practice to managing its three digital platforms for institutional, individual, and broker clients.

Shah decided a new challenge was in order and moved to Atlanta to work at regional bank SunTrust. He spent 11 years there in seven different roles, including head of consumer channels and head of the bank’s business accelerator unit. Then, he served as head of enterprise transactions operations technology. Finally, he was head of financial crime and compliance tech.

These roles gave Shah an impressive mix of technological and business expertise, but his resume lacked experience in technology infrastructure at a time when financial services firms were increasingly moving to the cloud.

So in 2019 he joined the Royal Bank of Canada (RBC) as vice president of architecture, innovation and cloud, and today serves as senior vice president of technology infrastructure.

Changing forecasts

While there is a lot of talk about public cloud adoption, most big banks’ critical workloads are still run on-premises.

RBC has taken a cautious approach to the public cloud. First, the bank focused on building out its private cloud capabilities before moving to a hybrid model. Shah says that some 600 applications run on this private–public hybrid model, with 80% of them using a private architecture. But, he says, that number will shift.

“In the next three to four years, we will see roughly an equal balance between on-premises and multi-cloud and maybe more at 40:60 in four or five years,” he says. “Early in our strategy, we anticipated that shifting operational and security functions toward the developer community would have a learning curve that would hinder our acceleration to cloud. As such, we built an internal platform where we’ve automated many of these functions for our developers. It was primarily built internally with the support of external–local, as well as vendor resources to complement our internal team.” The strategy, design, and the engineering IP belong to RBC.

RBC isn’t rushing to the public cloud, however, because it has taken what Shah calls a “crawl, walk, run” approach when it comes to cloud deployment. Specific to tech infrastructure, Shah wants RBC to be software-defined—an organization where infrastructure is powered by software as much as possible—and where manual processes are nixed from the equation.

“How do you untangle that from that vision perspective, and simplify it? So running it as a business, it starts from software-defined because I want to identify every opportunity to eliminate the human touch and have that as a straight-through processing [tool], as a request, as an API, as a micro-service,” he says.

Vinod Jain, senior analyst at consultancy Aite-Novarica Group, says it’s difficult to move bread-and-butter, in-house built applications to the cloud because there tends to be a lack of documentation around how the platform was built. On the other hand, transparency is a competitive advantage for vendors vying for business.

“The homegrown application is a challenge because of lack of documentation—nobody knows exactly what’s hidden underneath that,” Jain says. “So it’s more difficult to move or migrate the homegrown application to a cloud platform. And when we want to build the services around it, it’s a bigger investment. It can be done, but it’s just additional work that needs to be done.”

For Shah, the next step after being software-defined is for the architecture to be automation-enabled, which entails listing all the functions in the bank that are still not automated to see where a machine could do the job better and more efficiently. The third step is for the bank to have a tech-delivery model that is AI for IT operations (AIOps)-influenced.

RBC was founded in 1864 and in the ensuing 158 years has undergone mergers and acquisitions and the installation of new third-party systems in addition to internally built tools. Large longstanding banks must manage technical debt—there’s a reason why a cottage industry of Cobol developers still exists. It can be a nightmare to replace critical legacy systems, which is why they persist.

To get RBC’s infrastructure development to be software-defined, automation-enabled, and AIOps-influenced, Shah’s team must identify processes that have not yet been automated, and either shift them to the bank’s hybrid-cloud model, or make them have “cloud characteristics,” with the end goal of “RBC growing to become a truly end-to-end digital enterprise.”

Shifting winds

Over the last decade, financial services firms have become more comfortable with the security and requirements of cloud infrastructures (though less comfortable than other less-regulated industries). At the same time, they have legacy systems that are accruing technical debt and, thus, hindering investment in innovation.

In the same vein, cloud has helped facilitate the rise of software-as-a-service (SaaS) and managed services. Additionally, vast quantities of data can be stored, processed, and analyzed far more quickly and efficiently, and delivery mechanisms, such as the use of APIs, have improved. And the cherry on top has been the democratization of AI tools, specifically around machine learning and natural language processing.

Where banks tend to struggle is that middle layer between new (or evolved) technologies and legacy platforms that are upwards of 30 to 40 years old. That’s what Shah and his team are currently looking to address.

“When I took this role 1.5 years ago, I truly made it the priority, where multi-cloud means everything we do in tech infrastructure needs to be cloud characteristics,” he says. “So if it is a public cloud, you know that. Private cloud, you know that. But whatever is left, I am challenging my team and myself and my partners to identify opportunities to insert those automation and AIOps and drive cloud-type characteristics.”

This is also where Kubernetes containers for software development come into play.

“We are Kubernetes right now in private cloud and public cloud,” Shah says. “What we are seeing now as a next thing is  how we can enable the business to architect an application where it is designed to run primarily on private cloud, but for a surge or spike-type scenario, it goes and bursts into public cloud. We don’t have millions of dollars of investment to increase our private cloud offering just for that one single month scenario.”

So, for example, in late January 2021, trading volumes by share count exceeded peaks originally set during the 2008 financial crisis. Followers of the Reddit board r/WallStreetBets were short-squeezing stocks of video game retailer GameStop and movie theater chain AMC in a move against hedge funds that had been short selling the stock. The high trading volume triggered service disruptions across the industry.

At the time, Shah says the bank saw a 3–5x increase in trading volume as a result of the GameStop/AMC activity. RBC’s private clouds are architected in a way that when they achieve about 90% capacity, they “burst” into the public cloud (the bank uses Amazon Web Services and Microsoft Azure) to handle the excess demand. Once traffic levels off, that flow is redirected back to the on-premises instances. Kubernetes serves as the layer that assists in that traffic flow.

Shah wants to make the process more robust so that it can work not just in cases of volume spikes, but also if there’s an outage at AWS or Azure, or a major security concern. Seamlessly failing over from, for example, an AWS environment to an Azure environment (or Google Cloud or IBM Cloud) isn’t currently possible for the most part.

“We feel that there will be a time where we will be able to move the workload from one place to another for this type of spike scenario. But also, if we find ourselves in a situation where there’s a significant security issue with one provider, we don’t want to put our entire bank into them,” he says.

“I’m not rooting for a doomsday scenario for any of the cloud providers, but as a thought leader, we are challenging ourselves to better position ourselves so if that time comes, we should be able to take our workload and switch it to another,” he says. “In order to do that, a lot of planning is required. Just having a Kubernetes orchestration layer across private–public cloud is a starting point, but you have to go all the way up to the design of the application to make sure that it is designed, architected, and tested properly to support that type of movement.”

He adds that while Kubernetes gives users those technical standards in terms of interoperability between cloud layers, in order to get that value, you need to lay the foundation around proper design, architecting, and testing.

“The fundamental thing is, how do you stop developers from using a unique feature in AWS that is not available in Azure? So you have Kubernetes, it is portable. But if you use cloud-native functionality, you can’t do it,” Shah says. “So when we say multi-cloud, what it means is we are trying to provide faster training wheels to our developer community to leverage Kubernetes and use cloud—private or public—but we are restricting them from a unique cloud-native functionality, unless they come and explain a genuine business case, and when they do that, we educate them around the price it comes with.”

Lake effect

As the tech infrastructure unit looks to codify these practices, a key next step is that third tier of AIOps.

To look at this in motion, Shah points to an operational data lake the bank has built, which brings in data from systems, applications, and networks from around the organization. Currently, the lake employs about nine AI models that sift through that information. When a change request is submitted, the system will use AI to predict if there’s a higher probability of something going wrong with that change, as well as a confidence score.

For example, an employee might submit a change request for a payment application for March 31, 2022. The person filing the request believes it to be low risk, but the AI tags it as a moderate risk, which alerts the user to go back and check to make sure the request won’t cause a disruption. For future tech rollouts, AI capabilities will need to feature prominently in the decision-making process, Shah says.

“The latest [IBM] mainframe that we will roll out in the next two years or so will feature AI out of the box that helps us understand how we can optimize the Mips (millions of instructions per second) usage for the mainframe. It will have out-of-the-box features and functionality for us to understand what might not be operating at the optimum level,” Shah says.

For Shah, it all comes down to a change in thinking about tech development, a belief that has been formed over almost three decades in a range of tech and business roles.

“What I saw typically in large organizations is, here’s what I want, here’s what I need, so that I can do this,” he says. “I operate in the reverse. Here’s what I’m going to make given what I have and showcase what good looks like, and I always see the money and resources following that.”