Chicago-based data and investment research provider Morningstar has revamped its tick data delivery architecture into a cloud-based solution. For the project, it is partnering with Australian big data technology platform vendor RoZetta Technology.

The new version of Morningstar’s Tick Data Solution will go live at the end of July, using a platform built by RoZetta, running on Amazon’s cloud environment. Officials say the revamp will make it quicker and easier for clients to access tick data from the vendor.

Our challenge was to get [15 years of tick data] into the hands of clients—and the way that the data was traditionally stored made that challenging.

Matt Spedden

“Over the past couple of years, tick data has risen on firms’ priority lists,” says Matt Spedden, global head of equities and market data solutions at Morningstar. He adds that the service should appeal to financial professionals performing best execution, transaction cost analysis, and regulatory compliance: “Tick data is one of the ingredients that customers need in order to fulfill those tasks.”

The data can also be used for market surveillance and monitoring by firms’ compliance departments and regulators, in the front office for back-testing trading strategies, for market and liquidity analysis, and by software providers to populate analytics and research applications, and to back-fill other datasets.

Morningstar’s existing Tick Data service stems from its acquisition of London-based ticker plant and datafeeds vendor Tenfore Systems in 2003. Since then, Morningstar has collected and stored—and in most cases, commercialized—around 2.5 petabytes of tick-level market data, which is growing exponentially year-on-year, covering 200 stock trading venues, or roughly 99% of global equities coverage.

The dataset includes historical tick data back to 2003, with 10 years of US composite data, exchange messages and outage information, and the ability to filter by symbol or exchange, and to view market-by-order or market-by-price. Data points include trade date and time, exchange time, volume, trade price, last bid and last offer.

“Our challenge was to get that into the hands of clients—and the way that the data was traditionally stored made that challenging,” Spedden says, citing the vendor’s legacy storage and extraction environment, which used older technologies and typically delivered data via flat file or physical media, such as hard drives, for large volumes of data.

Specifically, the legacy technologies extracted data from an on-site storage area network using a single-threaded process. Then, the data could be copied to hard drives and shipped via courier service.

“This single-threaded extraction meant data could only be extracted chronologically, and while we could create multiple single threads, this was a manual process, adding time and complexity to the extraction process,” Spedden says. “The data and new technology is all cloud-hosted, which means we can automatically scale the number of server instances—and therefore the number of threads—based on utilization. This means we can dramatically increase the speed of extracting the data, putting the data into the hands of our clients quicker.”

Buy vs. Build

Morningstar as a whole is moving to more cloud-based delivery mechanisms. In this case, the market data group decided to engage a third party with proven expertise in this space, rather than to build the cloud platform from scratch in-house, and turned to Sydney-based RoZetta, which had supplied technology to Thomson Reuters’ (now Refinitiv) tick data platform for more than a decade—starting when RoZetta was known as SIRCA, an Australian non-profit technology research and innovation entity—before parting ways so the vendor could take the function in-house around two years ago.

Historically, it would have taken eight weeks to do that sort of extraction in the old environment. In the new environment, that’s reduced to a couple of hours.

Matt Spedden

“We made the decision to use a third party because of the experience they had [of working with Amazon] … and because they have a deep understanding of exchange data globally,” Spedden says. “And we had a longstanding relationship with RoZetta in Sydney, so expanding that was a natural way for this to evolve.”

Each company brought its specific skills to the table: Morningstar brought its expertise in market data and large datasets, while RoZetta brought its experience of working closely with Amazon as an AWS gold partner with more than 15 years’ experience building tick data platforms. The result is a customized tick data platform housed on RoZetta’s cloud-based technology stack.

“If a customer came to us and wanted all the US composite data for days, or even a year, historically it would have taken eight weeks to do that sort of extraction in the old environment. In the new environment, that’s reduced to a couple of hours,” he says. “We believe that turning around the data so quickly is a bit of an advantage.”

Clients will be able to access the data by visiting a password-protected Morningstar website that provides access to a variety of mechanisms by which they can obtain the data, including a front-end and an API—though clients can also still request hard drives if they want.