Multicast in the cloud—no longer a pipe dream

In October 2022, network provider and market data carrier Colt Technology Services announced that, together with Amazon Web Services, it had completed a cloud co-location proof of concept that demonstrated the viability of hosting and distributing multicast data in the cloud for capital markets customers. 

Wesley Elder, capital market product director at Colt, tells WatersTechnology that the thinking behind this PoC was to establish if multicast data in the cloud was even a possibility.  “In order to deliver any kind of solution, we first had to understand what the cloud is capable of. Is this just a pipe dream? Is this something that will ever be a reality?” 

But AWS executives were confident that once Colt could get the data in, it would be able to support the bandwidth, traffic, and everything else necessary.  

As part of the initial project, Colt and AWS built virtual distribution points of presence (PoPs) in the AWS Cloud, essentially allowing customers to lift and shift applications into the cloud environment without needing physical infrastructure. 

This resulted in Colt bringing its multicast data service for capital markets to life with AWS in June 2023. But between the PoC and the service going live, there was a lot of work involved. 

Part of the challenge of distributing multicast data in the cloud is that cloud service providers don’t accept multicast data in their cloud environments to prevent what is called a ‘multicast flood’ or ‘multicast storm’.

A multicast flood or storm happens when there is excessive traffic, which leads to an over-utilization or degradation of network performance. 

Russell Toop, Colt’s team lead for capital markets in Asia, says that while unicast—a single point-to-point data transfer—is still used today, multicast—a one-to-many communication model and the model that most exchanges use to distribute market data—is the model that Colt is focused on. 

“If you think of Asia, you’ve got Japan, Singapore, Hong Kong exchange market data. We could have a client in the US who wants to receive that data, run their algorithms, their AI tools, etcetera, on the data. We can send it using multicast from one point in Asia, where we receive it, to one point in the US, so that would leverage just one size of traffic. Once it gets to the US, we can send it to five clients from that point, and that’s really the benefit of multicast over unicast,” Toop says.  

Using unicast would mean that Colt would have to send the data once to each single client, which would take up more bandwidth utilization between locations. Traditionally, unicast is considered more reliable as the technology has retransmission, bandwidth controls, and various other capabilities that make it better for distributing between long distances. 

But, as Elder explains, Colt has designed its network to enable it to distribute multicast in the same way. “We have over provisioned bandwidth, over provisioned our capability, and we can essentially say our multicast delivery is as reliable as unicast delivery. And that’s what we’ve been doing on-premise and distributed between all our PoPs already today. This is just about extending that into the cloud,” he says. 

Flooding the network

Elder says the biggest challenge was getting the data into AWS. “If you tried to send multicast data to AWS, it’s blocked. They simply won’t accept it; it’s completely banned,” he says. 

He says multicast data in the cloud is like pouring water into the network—it goes to every endpoint in the network, whether that’s the intention or not. Particularly in a shared environment, if everyone starts “pouring water” into a network, that network will be flooded—quite literally flooded with data. 

“Most infrastructure that’s shared … actively ban it because they don’t want it flooding the network,” Elder says. “You get into something that’s referred to as a ‘multicast storm’, which sounds a little bit crazy, but that’s where everyone’s pushing a lot of data and all that data is going to everyone, and everyone’s duplicating it and pushing it back to each other. So, unless you control every single piece of the hardware, the firm that’s providing this generally blocks it.” 

For this service, Colt converts the data into a unicast-type packet, allowing the data to get through AWS’s infrastructure. Then, Colt converts it back to multicast to distribute—a process that takes “a few microseconds”. 

You get into something that’s referred to as a ‘multicast storm’, which sounds a little bit crazy.

Wesley Elder, Colt Technology Services

We’re not changing the format, we’re not changing the original payload. We’re finding a way to get the data through the firewalls or the barriers that exist. Then, once we get it in there, we’ve created technology to then convert it back to multicast to push it to clients, and it looks exactly the same as it did when it came from an exchange,” he says. 

For latency-sensitive trading desks, those few microseconds may be too much to serve as their primary datafeed. However, the vendor believes its cloud offering will still appeal to its customer base among firms in the high-frequency trading realm.

For example, they could use this multicast service for disaster recovery, Elder says, or they might move some of their infrastructure or applications that aren’t latency-sensitive from co-location facilities to the cloud. Those firms could also use it as a way to test entry into a new market instead of building out co-lo, as getting racks and hardware for connectivity can be challenging and costly. While HFT firms tend to be extremely latency sensitive, certain workloads may be less latency sensitive. 

“Rather than them going to, say, Thailand, and building a whole thing, we could get that data to them, they can do some testing, and then if it makes sense, then they go and build colo. So it’s more as a test bed or a sandbox for them to test and then think about moving into new markets,” Elder says.

Broader interest

For now, one user of Colt’s multicast market data in the cloud service is AsiaNext, an institutional-grade digital exchange that is a joint venture between SBI Digital Asset Holdings and SIX. AsiaNext declined to comment for this story. However, accessing real-time data in the cloud is a problem affecting a broad swathe of larger firms, and is one of the problems that members of a working group led by Goldman Sachs and JP Morgan are trying to solve. 

The argument is that real-time data isn’t a naturally good fit for cloud as firms that have real-time ticker plants will likely have exotic on-premise setups that use specific hardware technology and dedicated networks and multicast. “There is a lot more complexity and a lot more concern around deterministic latency, where you don’t have these big outliers and jitter,” a senior executive who attended a closed-door engineering session at the World Financial Information Conference in October 2022 told WatersTechnology.  

But those consuming a normalized datafeed from any of the big market data vendors would be classified as medium latency sensitive.

“Those medium-latency sensitive people don’t need to be in co-lo; they may be in proximity, or they may just be in that city trading from their office. For those types of firms where a few milliseconds clearly do not matter, they’re perfect for this use case,” the executive said.  

Again, the focus was for Colt to get data into the AWS cloud in the multicast format so that clients can “lift and shift” applications easily. During the process, Colt identified a few limitations that, once solved, would make the service more performant. 

In the cloud, [we copy] a message to multiple participants simultaneously [and that] happens in microseconds rather than nanoseconds. But functionally, it’s still multicast.

John Kain, AWS

“Some of the multicast streams AWS has seen for other industries like media, where it’s video, and when it’s not as latency-sensitive, has been fine,” says Elder. 

But when it comes to financial market data, the order in which data packets arrive is extremely important. Elder explains that if data arrives out of order, then that would mean firms consuming that market data would need to painstakingly reassemble the order book for that exchange in the correct order. 

“So AWS did some tweaking to their infrastructure to make sure that any time data was being sent through their Transit Gateway, it always took a single path,” he says. “Because any time it diverged or took multiple paths, then packets could arrive in essentially a different order.” 

While this issue didn’t come up during the testing phase, Colt has monitoring software—which it also runs within stock exchanges—in place in the cloud that it can use to compare the exact packet ordering that it received in the co-location facility versus what it received in the cloud.

AWS had worked to emulate multicast in the cloud within its Transit Gateways—hubs that connect AWS clouds to companies’ on-premises networks. The only difference, John Kain, head of worldwide financial services market development at AWS, told WatersTechnology, is how the multicast occurs and the performance. 

Kain, who spoke with WatersTechnology for a separate story, said: “If multicast is done on a network switch in a datacenter, it’s running on hardware. In the cloud, we use a software process—essentially an intelligent network—and the process of copying a message to multiple participants simultaneously happens in microseconds rather than nanoseconds. But functionally, it’s still multicast.”