Banks Grapple with VPN Capacity Amid Covid-Induced Network Strain

Financial services companies are under increasing pressure to avoid overloading virtual private networks (VPNs), as the coronavirus pandemic has forced the bulk of their employees to work from home.

A firm’s VPN is typically built to support 20% to 30% of staff working remotely at any given time, but under current circumstances that has increased to 80% to 100%, says Veronica Bocarova, a principal analyst at Cullen International, an independent provider of regulatory intelligence in the communications sector. In some cases, firms are having to allow access to the VPN on a rotational basis to prevent overwhelming the system.

“You realize that you cannot have everybody working at the same time, so you have to schedule sessions, or provide VPN-sharing across employees, like tunnel sharing. So, we have lots of issues that suddenly chief information officers and IT bosses must work out,” Bocarova says.

Video conferencing, streaming videos, and hosting webinars consume the most network bandwidth, a problem that is compounded by these types of activities surging over the past few weeks.

Getting the right equipment to the right people during a global pandemic, when travel is drastically limited, is not a simple undertaking. In India, HSBC had to use a cargo plane to transport 600 laptops more than 500 miles from Pune to Bangalore. And the work didn’t stop there.

High-speed connectivity is crucial for some traders, effectively eliminating the option to work from home. James Kemp, managing director at the Association for Financial Markets in Europe, says that 5% to 10% of sell-side traders are working in their main offices or at business continuity sites.

Traders working in fast-moving markets like foreign exchange require near-real-time access to data. In this case, plugging into the main system remotely can be problematic as a firm’s global staff will be simultaneously sending data across the network, causing traffic bottlenecks and latency issues.

“You’ve currently got that split going on: if you’ve got a less-active marketplace with slower pricing, you can probably trade that more easily from your remote home location. If you have a fast-moving, latency-dependent, electronic ‘flow’ market with huge numbers of updates—and I am talking possibly millions, if not billions, a day—you need to be at the end of a very thick pipe,” Kemp says.

Some staff have also been allocated to work at their employer’s main sites on a rotational basis to alleviate the pressure on the private network. In the first few weeks of March, Gareth Coltman, global head of trading at MarketAxess, recalls speaking to a bank lead, who said the bank was experiencing issues connecting with clients over its VPN.

“They didn’t have enough servers to support that kind of connectivity,” he says.

And in that case, the bank had some of its traders take turns working from the office to offset some of the limitations in its IT stack.

Adjusting Tech Stacks

In a normal scenario, firms can upgrade their VPN capacity to make room for an expected increase in traffic and usage, but in the current crisis, it’s not as easy as that, says Bocarova. Firms had little to no time to prepare as lockdown measures were swiftly implemented across the globe, first affecting the Asia bloc, then Europe and the US.

Scaling VPN capacity to support entire workforces is a major undertaking that can take weeks to implement, requiring IT teams to order technical equipment and install VPN access on local devices. Some of this work means that firms must have IT personnel physically present on site to tune networks, manage traffic, and upgrade servers.

For HSBC, the task involved extending its VPN capacity to support its 235,000-plus employees and client interactions on its network globally. The bank has had to more than double its VPN capability since mid-March across offices in the UK, US, India, China, Hong Kong, Dubai, the Middle East, and Australia.  

Prior to the coronavirus outbreak, a typical day would only see 10,000 to 20,000 employees logging into the VPN at any one time, says John Hinshaw, group chief operating officer at HSBC, who joined the bank in December 2019. 

Adding to the scope of the challenge, staff accessing the network also needed the right computers and connectivity equipment.

Many HSBC personnel already had company-supplied laptops and the necessary setup. Those who didn’t were placed into one of three groups for the purposes of the deployment, which took place over the course of two weeks, prior to which HSBC staff in China were already working from home. The first group of 28,000 employees were sent new pre-configured laptops. The second group, comprising around 20,000 staff, primarily in Asia, were authorized to take their desktop computers home with them. The third group of around 13,000 people was set up with access to the VPN via HSBC’s Virtual Connect product from their own devices.

Getting the right equipment to the right people during a global pandemic, when travel is drastically limited, is not a simple undertaking. In India, for example, HSBC had to use a cargo plane to transport 600 laptops more than 500 miles from Pune to Bangalore.

And the work didn’t stop there.

“Then we had to tune the network, which was designed for traffic to flow predominantly from building to building, not from home to home, through all the VPN connections. So we had to really look at the network traffic and understand how it needed to be adjusted,” Hinshaw says.

The bank increased the capacity in parts of its datacenters and Cisco switch capabilities. It adjusted the way in which certain software programs operated and computer logins were entered, and changed other system functions to control the amount of bandwidth they consumed. HSBC also accelerated its implementation of Zoom video conferencing across the company. As part of that rollout, Zoom was also configured to operate on the bank’s corporate network, and servers are monitored in each country to ensure they can cope with the traffic. HSBC had to provide a list of essential employees to governments, which included employees working in datacenters and those managing the network. 

“Employees would go into the datacenters, upgrade the switches, the servers, or the network capability, and that would be either our direct employees or we would partner with technology providers,” Hinshaw says.

One of the final steps involved instructing staff members to use their home Wi-Fi to connect to the VPN client portal, which they access using an RSA authentication app on their mobile devices. Users enter a password into the app and it generates an eight-digit number that allows them to access the VPN on a work device.

During that process, the bank also put out a request for employees to come forward to support the IT desk in helping other, less tech-savvy employees, get set up. In response, 500 volunteers across the company stepped forward.

“If you’d asked me a couple of months ago, ‘Could you have the whole bank working from home and still be providing all services to customers in a seamless manner?’, I would have been surprised,” Hinshaw says.

Public Networks

Beyond VPNs, public networks, as well as telecommunication and internet service providers (ISPs) are also having to cope with a huge increase in usage. In a joint statement with the European Commission on March 19, the Body of European Regulators for Electronic Communications (Berec) announced that it would set up a system to monitor the internet traffic situation through the national regulatory authorities (NRAs) in each member state to enable a response to capacity issues.

Berec’s fortnightly summary reports showed a huge increase in fixed and mobile network usage during the Covid-19 crisis in March. Its latest report presents a more mixed picture, showing that traffic is increasing, stabilizing, or decreasing to different degrees, according to the local NRAs.

For the most part, the major network vendors have the ability to scale their server and network capacity, but whether that is sufficient—and sustainable for an extended period of time—is another question.

“I think we have discovered, even from a personal perspective, the challenges of bandwidth for remote working,” Kemp says. “The internet is not without problems. Even you and I speaking here now, it’s pixelating, and we are finding that the ISPs are struggling, and the telecommunication providers are struggling, with the amount of data going over them.”

Looking ahead, Kemp says that in the aftermath of the coronavirus, there will be new questions about the definition of operational resiliency from a regulatory standpoint. Factors such as internet access, connectivity, VPN capacity, server bandwidth, remote workplace setups, and more, will all be scrutinized.

“It will challenge us in our definition of what we mean by operational resilience,” he says. “Traditionally, we would have thought it means, for example, we get a cyber attack and we go down for two hours and plan for how quickly we get the system back up. I don’t think anybody ever envisaged that operational resilience scenarios might mean 80% to 100% of our staff working, trading or running operations over the internet from home, and using laptops. What does that mean for cyber attacks and security? Whoever thought that you would need to challenge the bandwidth of the internet service providers and telecommunications vendors?” he says.

After the crisis ebbs and work practices start to normalize, the pandemic will almost certainly shape future regulation, such as the proposed rules on operational resilience in the UK, and the role of public networks as third parties to systemically important financial institutions.

Security Hygiene

A VPN enables employees to work and send data across a private network securely by routing their device’s public internet connection through the corporate server.

But the security of the technology only goes so far, says Adrian Scrase, chief technology officer at the European Telecommunications Standards Institute (ETSI). How the network is deployed, and how an individual uses it, plays a major part in the overall security of the network.

“Once you have products that are built and are inherently secure, you then have to deploy them in a secure way, and the end-user has to operate them in a secure way. So, while we provide sort of the first level of protection, up to the provisioning of the equipment, there are many other actors that also need to behave in a secure way, including the end-user, if you’re going to have end-to-end security,” Scrase says.

The dramatic uptick in the use of video conferencing services was closely followed by a backlash over the security of some of these tools, such as Zoom. Other concerns have emerged as a result of how quickly some of the platforms were released to the market to meet the needs of financial services firms. Collaboration platform vendor Symphony’s Meetings tool was launched in mid-April, and chief executive David Gurle recently told WatersTechnology that the provider plans to release a fully compliant, end-to-end encrypted, and cloud-enabled second iteration of the platform by June 30 of this year.

Scrase similarly highlights the need to employ these types of platform in a secure manner, such as by using authentication methods and verifying router security.

“The product itself might not be inherently unsecure, but it may enable you to operate in a non-secure way. So, unless password protection is the default setting, you could enable people to have communication paths that are not password protected, and that’s just bad behavior by the end-user,” he says.

Security practices might seem obvious, but the worry is that employees could become complacent when working from home away from the usual safeguards. Increased remote working raises new security considerations around authentication, secure networks, vulnerability to cyber attacks, and more.

“Inside these firms, IT security risk and compliance are key parts of that workflow, and internal policies are set up to take advantage of the fact that you’ve got things like building security, network security, desktop security—those things are all there. And suddenly, if you put people [in remote working environments], you need to start thinking about all those problems and how you’re going to solve them,” MarketAxess’ Coltman says.

5G Adoption

The coronavirus pandemic has highlighted the importance of staying connected, both from a personal and a professional standpoint, relegating activities previously conducted largely in person—everything from work, school and meetings, to connecting with family and friends—to cyberspace. This sudden and monumental shift points to an increased appetite for higher-speed and higher-capacity 5G networks.

Current home Wi-Fi networks are subject to bandwidth-sharing with other household members, and internet connectivity and stability issues.

The hope is that 5G could alleviate some of those pain points in the near future, promising the delivery of high-performance and low-latency internet connectivity for both retail consumer and industrial use.  

From a business continuity perspective, says Hinshaw, 5G could prove invaluable in ensuring workers can remain online and connected, as it purports to be more resilient than current 4G LTE networks.

As supply chains have halted, there is an expectation that the technology’s broader rollout plans will be delayed. However, ETSI’s Scrase says network operators are still on course to deploy widespread 5G coverage as soon as the third or fourth quarter of this year. This could be a crucial time as nations’ efforts to re-open their economies run the risk of sparking a second wave of infections.

“If we should, regrettably, end up with a second peak in a few months’ time, it’s quite likely that we will have 5G deployed by then, and we’ll be in an even better situation to cope with increased traffic,” he says.