Update: Bloomberg Cites Hardware, Software Failures for Terminal Network Outage

According to a Bloomberg spokesperson, software and hardware failures precipitated excessive volumes of network traffic, overwhelming terminals and forcing them to disconnect from its network. Sources say the outage appears to have been global and affected many─if not all─clients, though Bloomberg's other data products, such as its B-Pipe datafeed and back-office data services, were not affected by the outage.

The issue began just before UK market open, leaving many screens dark for the UK trading day. At 2:57am ET, the vendor tweeted "We are currently restoring service to those customers who were affected by today's network issue, and are investigating the cause." In response, some users tweeted back that their service was already back up and running.

At 4:25am ET, Bloomberg tweeted that it had restored service to "most customers following disruption to parts of our network," adding that the vendor was "making progress bringing the full network back online." Then at 5:13am, the vendor tweeted that "there is no indication at this point that this is anything other than an internal network issue."

According to a source at a US asset manager, the process of restoring service was a gradual one, and that the firm experienced "sporadic problems with individuals reconnecting" for several hours after the UK market open.

Sources say Bloomberg did not immediately communicate the cause of the outage, with some saying this is not unusual practice for the vendor.

"We'll probably never know the true underlying reason behind the ‘outage' today. Even if some piece of hardware failed somewhere, it still leaves open the question of whether there was a backup system in place, and if there was, why it didn't work? And if there was and it didn't work, why it took two hours to fix things?" says a source at a data management provider.

An executive at UK data user group Ipug says that while outages in general are not uncommon, it is rare for Bloomberg service to be interrupted for such a long period. "This is the first time we've seen such a long outage compared to past outages, which have to do with quality of data rather than timeliness. Bloomberg is usually a stable environment," the Ipug executive says, adding that Bloomberg rarely provides detailed explanations of the cause of any problems.

However, on this occasion, Bloomberg apologized to customers for the outage, and provided the following explanation once service had been fully restored: "We experienced a combination of hardware and software failures in the network, which caused an excessive volume of network traffic. This led to customer disconnections as a result of the machines being overwhelmed. We discovered the root cause quickly, isolated the faulty hardware, and restarted the software. We are reviewing our multiple redundant systems, which failed to prevent this disruption," the Bloomberg spokesperson says.