Starting two years ago, Grasshopper, the Singapore-based proprietary-trading firm, has been using the Google Cloud Platform (GCP) to build out its quantitative research and data-processing platforms to improve its trading capabilities.

By leveraging GCP tools such as TensorFlow, an open-source machine learning library, BigQuery, the same data warehouse on which everyday Internet users search Google, and Cloud DataFlow, which is used for processing both batch and real-time data streaming, Grasshopper and the cloud giant have worked closely on several internal projects.

One of those projects is the firm’s in-house Java application, Ahab, which was launched a year ago. Ahab, which is named for Herman Melville’s fictional whaling captain, allows traders to “listen” to live market data and make better and faster trading decisions. It is built on Apache Beam, the open-source programming model.

Grasshopper is now able to query 100 billion rows of market data in less than 30 seconds. Before, when the firm would go to process that data, or look to do large studies on how to look at the market, it took about a week for the firm do the necessary calculations with the compute power it had.

Tan T-Kiang, who holds the dual-title of chief investment officer and chief technology officer for Grasshopper, says the data that flows through the application can be thought of as fish, gobbled up by a bottomless whale. Moby-Dick metaphor aside, Ahab sources its data directly from exchanges, and calculates, in real-time, the value of order books, or the lists of buy and sell orders at any trading venue, and how that can impact a stock’s price. The data is tied into GCP’s Solace PubSub+ tool, which handles and sorts information from multiple sources, thus eliminating the need for Grasshopper’s engineers to deal with basic network connectivity. The resulting data log then gets stored inside BigQuery.

“One of the biggest things we’ve been trying to solve is that when you start off, let’s say, 10 years ago, you built a database, and the market was probably 10 times smaller in terms of data. That system was fine, and then a few years later or even a year later, you have to re-tool because it’s not good enough anymore,” Tan says. “Or you, as a hedge fund or an HFT firm, decided to add five more markets to what you’re covering. And what you’re storing now is maybe 50 times or 100 times more data.”

The scalable, adaptable environment GCP offers has allowed the firm to optimize its “cycle of innovation,” and focus on the things it should: numbers, data science, and engineering. Before partnering with Google, Tan says maintaining infrastructure and managing anomalies, such as market spikes outside of regular market hours, monopolized too much time and effort from Grasshopper’s developers.

“That’s an issue that a firm like mine shouldn’t be focused on,” says Tan. “[Those things] shouldn’t be our core competency. So we went out there to look for someone who could help us.”

• READ MORE: Scotiabank has significantly improved the time it takes to run XVA calculations, and has big plans to further expand its XVA cloud GPU program. Click here to read more.

When conducting research now, the firm is able to query 100 billion rows of market data in less than 30 seconds, Tan adds. Before, when the firm would go to process that data, or look to do large studies on how to look at the market, it took about a week for the firm do the necessary calculations with the compute power it had. Using the cloud’s parallel computing capabilities, the time to run the same calculations has shrunk to “maybe 10 minutes,” he says.

The firm used to use various other open-source databases, which could be successful in the short-term, but they were ultimately faultier, or often funding dried up. And Grasshopper’s teams spent a lot of time considering how many terabytes of storage to buy in a way that would both serve their present needs and future-proof themselves. 

“[Now] that question just sort of goes away,” Tan says. “We just don’t talk about it anymore. And by not talking about that, you actually help your rate of innovation.”

Big, and Getting Bigger

Ulku Rowe, technical director of financial services at Google Cloud, says financial services is already a data-heavy business—and that weight isn’t going to get any lighter. On top of normal market data, there’s constant news, global events, regulatory shifts and changing market conditions.

“It’s a tough problem, but it’s also a massive business opportunity for those that can create the best trading strategies, the best risk management models, and those that can actually do it fairly quickly,” Rowe says. “And nowhere is this need as acute as high-frequency trading.”

Looking at the big picture, Tan says what they’re trying to do is predict prices as accurately and consistently as possible. They start with big data, study those massive datasets on the cloud, and then produce summary tables of possible events.

“If this type of market condition happens that way, then there is a high likelihood that the market will go up in the next one second,” Tan says. “And if this other situation happens, there’s a high likelihood that the market will go down in the next one second. All we’re doing is understanding what the probabilities are.”