As augmented reality and virtual reality devices become more accessible, financial institutions are seeing the technology’s potential to overhaul how their staff communicate and bring data to life in ways that simply weren’t possible before. But projects may still have to wait years before they make their way to the trading floor. By Emilia David.

One day, in the future, a trader will pick up a shiny pair of glasses, put it over their eyes and instantly bring up the image of a counterparty that, despite being thousands of miles away, looks like they’re sitting right in front of them.

That is one of the scenarios financial institutions envisage as they experiment with virtual reality (VR) and augmented reality (AR)—especially now that commercial devices can now be easily bought from Best Buy for a few hundred dollars. Once clunky headsets that rendered cartoonish surroundings, modern technology has advanced to the point where, ergonomically and visually, the everyday use of VR has become a possibility.

VR, AR, and mixed reality (MR) have been generating interest since the first ranges of modern devices were released to consumers a few years ago, and interest reached a fever pitch when Facebook bought Oculus VR—maker of VR headsets—in 2014. Data from the VR/AR Association, an industry lobby group, estimates total market size for the technology to reach $108 billion to $215 billion by 2021. Investments into the technology also reached $3.5 billion in 2016, based on figures from a report released by Goldman Sachs in 2016, Virtual and Augmented Reality: Understanding the Race for the Next Computing Platform.

Much of the consumer technology in VR and AR—especially VR—is geared toward the entertainment sector, either for movies or deeply immersive gaming experiences. The Goldman report, however, sees enterprise VR becoming a $180 billion market by 2025.

And it’s these potential applications that are enticing some banks and financial institutions to experiment with the technology.

Fidelity Labs’ head of emerging technologies, Adam Schouela, says AR and VR offer a fresh approach for new ways to understand data, build better services and even free up cognitive space for research. 

“We want to be ready and develop applications for when these form factors take off,” says Schouela. “VR has been a preferred training tool for maybe 40 years, but the technology has become more accessible so we can better leverage its blank canvas.” 

He adds Fidelity Labs, and many others in the financial industry, see AR and VR as a more visceral way of presenting data so the information has much more of an impact. 

Eyes Front

Though VR, AR and MR all depend on a device to plunge users into essentially different worlds, they are not all the same. VR devices such as Oculus Rift and the HTC Vive platform totally immerse a person in a virtual world by taking much of the actual world out. AR devices, such as Microsoft HoloLens or Glass, augment the real world with virtual information. That can take the form of overlaying an object from a device onto the street or wall, seeming as if it belongs there, to heads-up displays commonly seen in military applications. It is less immersive and allows a person to retain awareness of their surroundings. MR, sometimes referred to as extended reality, functions like AR but with the added flourish of a physical reaction occurring in the world when an action is performed.

“We’re excited about the possibilities enabled by Microsoft HoloLens and see a wide array of use-cases for all,” a Microsoft spokesperson tells Waters. “We’ve been on a decades-long journey to make computing more personal, and this is a logical extension of that path. Bringing computing into the three-dimensional world that humans have always existed in is the next step in making computing truly more personal.”

Many large financial institutions have been experimenting with AR and VR, mostly in proofs of concepts in the past few years. So far, experimental projects have ranged from mapping out data in three dimensions to letting customers visit a virtual bank, running the gamut from services for institutional investors to simply making the banking experience easier for the retail customer.

Citi and its development partner, 8ninths, launched a virtual trading desk using Microsoft’s HoloLens in 2016. BNP Paribas also began several experiments with the technology including a program offering a virtual walkthrough of a property being sold, chat with a bank representative, and one teaching business continuity procedures. Wells Fargo hopes to put out a product where customers can converse with a bank teller through VR. Citi did not respond to requests for comment on the current state of that project, however, and BNP Paribas declined to comment.

Fidelity, through its Fidelity Center for Applied Technology, launched several projects including Cora, a proof of concept it worked on with Amazon. The idea behind Cora, says Schouela, was to see how to marry voice technology with the 360-degree blank canvas of VR. Cora is a virtual “host” that can answer questions prompted by voice commands and find information quickly. The company also currently uses VR technology to train its customer service agents.

“Within VR we have four main use-cases that we spin out—data visualization, that’s the vein Cora is in; customer education to help them understand more complex financial concepts; employee training, like our empathy training module for our contact center associates; and collaboration to see how might this experience be better than a voice or video call,” he says. “We have real experiments to try out to see where this technology is better than the ones we’re using today.”

Schouela says Fidelity has seen good results from the customer service training proof of concept: Those who trained using VR—which lets users run through customer service scenarios and virtually visit a client—showed better customer satisfaction ratings overall. 

It isn’t just banks playing around with AR and VR, though. Exchanges are also looking at how the technology can work for them. 

Nasdaq’s chief information officer and chief technology officer Brad Peterson says the exchange is experimenting with AR in its surveillance function. Using HoloLens, the group has trialed the use of AR in the NOC, its market control center in midtown Manhattan, which oversees the operation of the second-largest exchange in the world.

“We’re starting to see, in things like surveillance and workgroups, pretty exciting interfaces that combine voice and gesture navigation,” Peterson says. “Let’s say we’re at our surveillance room. In a traditional one, we all have our own screen. If we’re working together with AR, all of a sudden we can be working on all the markets and maybe we can be working on a factor of magnitude more efficiently.”

He adds Nasdaq wants groups within the exchange, say those in operations and surveillance, to be able to collaborate using this technology. Nasdaq has worked with students from the Massachusetts Institute of Technology to integrate voice commands with an AR workflow to pull up data during a meeting and enhance working groups. AR, in particular, works well for this, Peterson says—rather than headsets that obscure and isolate individuals, AR instead acts as a force multiplier and encourages collaboration.

And yet, the financial services industry still has a lot of catching up to do. 

Other industries have managed to use AR and VR and integrate them into their workflows. A spokesperson for Microsoft tells Waters many of projects developed for the HoloLens—which also include many MR programs—“demonstrate how HoloLens is already transforming businesses including Ford, thyssenkrupp, and Stryker, among others.” 

Glass—the product formerly known as Google Glass—has transformed itself as a tool for manufacturers to quickly double check schematics or data and for healthcare professionals to chart medical information. Glass is no longer available for retail customers but is offered exclusively to enterprise users. 

Extending further into healthcare, some are using commercial hardware and custom-built environments for the diagnosis and potential treatment of mental health patients.

Despite success in other industries, however, there are some considerations the financial industry has to deal with if it wants widespread use of AR and VR technology, particularly if the use case a bank has in mind is more client-facing than within the organization. 

Blind Spots

Mihir Shah, a senior director at consultancy Synechron, points out there are issues to contend with before VR and AR are ready for prime time, particularly in terms of cost, distribution and security around the use of the technology. That’s not to mention the specialized skillset required in computer graphics and special effects software packages, talent that’s more likely to be found in the Presidio rather than Park Avenue.

“The cost of it really makes it something that only big firms can do, since it takes millions just to develop a base application with strong user experience, plus the high processing software needed to build the environment,” Shah says. “But other than that you have think about how you’re going to distribute the device to users and since some applications depend on the internet, you also need to determine how to secure that connection.”

Adoption is also dependent on how quickly technology companies can refine and reduce prices on their devices, according to Dimitry Parilov, managing director for data solutions and products at vendor dxFeed.

“Only the development of the hardware really holds adoption back because companies are pilot testing projects and they want these to be device agnostic. These are still mostly in development so we’re waiting for them to really get ahead,” says Parilov. 

DxFeed is in the process of building out an AR application that allows clients like banks and proprietary trading shops to build out a simulated three-dimensional model of a graph. Parilov notes that with technology like AR, traders essentially can have as many screens as they want to and even pull out certain data points. While people are definitely interested, he says, firms are cautious and are taking their time deciding if they will invest in the technology.

Fidelity’s Schouela, Synechron’s Shah and dxFeed’s Parilov all say that full commercial production is still a ways off. Shah even predicts commercial production of financial services applications could take four to five years, assuming firms have figured out the right use-case, data security and hardware distribution for users. 

And it isn’t just the cost of the devices: While AR and VR developed at the same time, the hardware did not. AR devices, visors like HoloLens and Glass, must be able to track physical surroundings at all times, and as such have to be mobile in nature while incorporating enough processing power and optical hardware to perform at a high level, consistently. Meanwhile, VR headsets can be connected to a cable and while bulk may be an issue, it doesn’t have to be a completely lightweight visor. 

But the biggest drawback of all is one that befalls all new innovations—the tendency to use the technology like a hammer, says Schouela, to solve every single problem. 

Trick of the Eye

Despite how far away the commercial production of a lot of the prototype AR and VR applications are, Shah notes the financial services industry—especially firms with larger research and development budgets—absolutely can play around with the possibilities it offers. 

“The financial markets are going to follow these investments into AR and VR because they don’t want to be left behind,” he says. “Banks are definitely investing in applications but none of those proofs-of-concepts are ready to go into production and the reason for that is the reach of the technology. The technology can only be successful if there’s good content; some applications I’ve seen are so basic, like an ATM-type application, so if there is better content it will really grow.” 

Firms are not deterred by the drawbacks, however, and as demonstrated by Nasdaq are actively building out technologies to meet specific needs. For Fidelity’s Schouela, it’s all about anticipating trends and platforms that can communicate information better. 

“Years ago we built on the Pebble watch, just in case that took off. That’s why we do these things. We really want to understand the technology better and we want to see where it is different from the old technology that we have,” he says. “The technology is changing rapidly and it will continue to evolve, so now we’re at a time of change, definitely in an experimental phase.”