Esma to ink deal for big data capabilities

The European Securities and Markets Authority (Esma) is in the final stages of formalizing a deal with a managed service provider that will give the regulator the big data capabilities to operate on the Microsoft Azure cloud platform. The European Commission will officially sign the new contract at the end of June, Paul Hussein, team lead of supervision and analysis systems at Esma, tells WatersTechnology.

The regulator’s legacy systems have started to crack with the strain of ingesting a slew of data reported to it by financial firms under a range of far-reaching regulatory regimes, such as the Markets in Financial Instruments Regulation (Mifir). Hussein says Esma is shelving its Oracle relational database, which is unable to meet its urgent need for scalable computing power.

“We’re pushing the limits of what we can do with a relatively simple Oracle,” he says. “The data that is sitting within the Oracle databases—we have to query it, we have to analyze it, and we’re finding that querying the data sitting in this old tech is taking too long.”

Esma is unable to disclose the name of the managed service provider it is partnering with until the deal is signed, Hussein says.

Updating the regulator’s capabilities in this regard has become more urgent as it prepares to take on more responsibilities. Following a review in 2019, the European Supervisory Authorities (ESAs), of which Esma is one, decided to grant it more powers. Starting next year, the regulator will authorize and supervise the Data Reporting Service Providers (DRSPs), which include Authorized Reporting Mechanisms (Arms) and Approved Publication Arrangements (APAs).

Once the new third-party system is fully implemented, Esma will be able to leverage its big data components, such as self-service analytics, search functions, data integration services, and a data dictionary. Additionally, its data scientists will be able to code new functions on top of the self-service analytics.

Rather than using vendor Talend’s extract, transform, and load capabilities to push reporting data into Oracle, as is currently the case, Esma’s data science team will transfer that data to its Microsoft Azure cloud environment, with the aim of using the new third-party capabilities to perform more intense calculations in Python and apply tools such as Apache Hive.

“It means that when we come to manipulate the data, we’re not held back by the capabilities of the platform in terms of computing, and we can easily distribute the analysis,” Hussein says.

In June last year, the regulator completed its data and operations migration to Microsoft’s cloud, a move that will enable the big data project. The project consists of two parts: integrating the new vendor capabilities on Azure, then applying them to a real use case.

IT advisory firm Gartner helped Esma write the technical specifications for the procurement process. Hussein says that rather than asking for a specific out-of-the-box solution, Esma wanted individual third-party tools that could be spun up on the Azure cloud on demand.

This pay-as-you-go model is necessary for an organization with a limited tech budget, he says. “There are capabilities that you can switch on and pay per month, and some are per user, some per gigabyte or terabyte. So, there are different cost models for different functions, or what Gartner calls [control] ‘towers.’” 

Prior to embarking on the project, Esma studied implementations carried out by other financial authorities, including the European Central Bank (ECB) and the national competent authorities (NCAs), which have delegated regulatory power in each EU country. Hussein realized that not only was it enormously challenging to run big data projects on-premise, it was also quite expensive.

“That is what we saw as the downside of how other regulatory bodies are running big data platforms,” he says. “They manage it in-house, and they run it on their own servers. But if you pay for 20 servers, they are sitting there most of the day doing nothing. Being small, we want to optimize our costs.”

The second leg of the project will involve Esma applying its new capabilities to a real use case: manipulating reported data collected from the DRSPs. The regulator now has to collect and process this data as part of the responsibilities assigned to it after the ESA review. For instance, it will have to compare trade data published by the APAs with transaction data reported to the NCAs by Arms and trading venues.  

Dealing with new data flows from multiple institutions across the EU will require Esma to develop a workflow for managing data quality and inconsistencies. Kathryn Rogers, a partner at UK-based law firm Cripps Pemberton Greenish, says this will be one of the biggest impediments for such a project.

“The data will be stored in a different way in financial institutions’ systems and will be reported in a different way to the regulator, so trying to compare data from across different institutions is also going to be difficult,” she says. “You first need to get all the data set into a cohesive style before you can even start comparing.”

Calling on the big guns

Esma is relatively small in comparison with other financial authorities. It has roughly 418 employees, compared with 3,500 at the ECB and 4,200 at the UK’s Financial Conduct Authority. Hussein says the regulator had to leverage the power of the EC to access large vendor contracts.

“The commission is one of the biggest—if not the biggest—IT companies in Belgium,” he says. One benefit of this that the EC can create contracts that Esma would unable to achieve on its own. “Some of the contracts in the commission are eye-wateringly big, and it attracts good vendors and good prices,” Hussein says.

In a typical tendering process, a company publishes a specific technical requirement, and vendors respond with an offer on their services. With the help of the EC, however, Esma was able to use an electronic public procurement tool called the Dynamic Purchasing System, in which vendors must qualify to join a marketplace of suppliers from which authorities can choose. 

“It’s like the reopening of contractual competitions, whereby you to have to qualify to enter,” Hussein says. “There is a qualifying criterion—for example, you’ve done this stuff before, you’re a big or reasonable company, you’ve done managed services, you’ve done public cloud—then you get on a list, and only that list receives the publication of these mini-tenders.”

‘Risk-averse manner’

As Esma is publicly funded, it has limited means to invest in new technologies. Even when it obtains approval for a project, there is no room for error, Hussein says. 

“As we only have enough money to keep ticking over with the [responsibilities we have], and the systems we have, and then just enough money to build a new system, it’s very risky to look at new types of technologies. So, we must have a risk-averse manner, by going slow and steady rather than fast,” he says.

Rather than having a budget dedicated to exploring new tech, Esma must prototype solutions in the context of an approved project. The contract with the new third party, for example, is funded as part of delivering the DRSP project to help the regulator meet its new supervisory responsibilities.

Cripps Pemberton Greenish’s Rogers says public bodies struggle to secure funding for tech projects because they are not driven by commercial incentives, and they have shallower pockets than banks or asset managers.

“Financial institutions will be doing these things because it helps them improve a product or comply with their regulatory obligations in terms of assessing future risks,” she says. “The regulator’s reason to do it is so that they are able to better monitor what institutions are doing. But, obviously, they don’t get the financial benefits of it in the same way.”

To fund some of Esma’s added powers resulting from changes to Mifir, the regulator has proposed that DRSPs pay fees. The public consultation for the proposals ended on January 4, and Esma published a final report and submission of the advice to the EC on March 26. 

Steep learning curve

Rogers says that getting the tech right for a big data project is only half the battle. The data analyst tasked with implementing the project will have to be sufficiently skilled at interrogating and interpreting the reported data. One of the most challenging aspects is ensuring that causations can be accurately drawn from correlations in the data.

“The hardest part is how you actually interpret that data. Large amounts of data can increase the risk of incorrectly drawing assumptions from correlation. Just because you have a correlation in the data doesn’t mean that you know the causation,” she says.

Attracting the right talent can also be problematic for a regulator. Public authorities are competing in a job market where data scientists are already in high demand, and they can’t offer the private sector salaries and experience with bleeding-edge tech that big tech firms and financial institutions can. Rogers says this is one reason regulators tend to be behind the curve when it comes to technology and innovation. 

“The speed of change in a regulator is slower, the salaries are likely to be lower, the potential for job progression is slower, and your ability to go back into working for a financial institution is likely to be impacted because you haven’t been exposed to the same [technology] environment,” she says.

Esma employs 50 data scientists and analysts across its various departments. Alexandru Dincov heads the information and communications technology (ICT) unit, which is made up of 38 IT and data specialists. The unit is split into three teams, one of which is the supervision and data analysis systems (SAS) group, led by Hussein. The SAS team provides support for the ICT systems, along with data engineering and data science work. Hussein recently finished building out his SAS team of nine people, including a project manager, service manager, architect, support engineer, business analyst, data engineer, data scientist, and team leader.

In the coming months, following the official signing of the managed service provider contract, Hussein’s team will be tasked with learning and testing the regulator’s new big data capabilities. Upon completion of their training, they will then apply the tools to the DRSP datasets. If the project proves successful, the new solution could be used for other regulatory reporting data.

But there is no rush to do this right now, Hussein says.

“The first thing everybody asks is, ‘Are you going to run the existing 20 other systems on Azure [and apply the new vendor capabilities]?’ And the answer is no, not in the beginning,” he says. “It’s a complex technology and there’s a lot of learning to do, so we’ll run this first system, make sure that it is successful, and then we’ll see.”