Goldman Sachs Revamps Virtualization Infrastructure

Goldman Sachs is undergoing a multi-year pilot program to modernize its highly customized virtualization environment and leverage the use of container-native virtualization (CNV).

Toward the end of 2018, the investment bank partnered with IBM subsidiary Red Hat, a provider of open-source technologies. The partnership will develop Goldman’s virtualization environment and allow it to effectively manage its virtual machines (VMs) with the implementation of Red Hat’s OpenShift CNV platform. The technology is built on Kubernetes, an open-source system for deploying, scaling, and managing containerized applications.

OpenShift can also operate as a unified development platform where developers can build, modify, and deploy applications in containers and VMs within a shared system.

VMs are software computers that have the same functionality as their physical counterparts. They run applications and operating systems separate from the rest of the system within a host computer. VMs are heavyweight technologies, so running multiple instances on a single physical machine can impact performance. They are also limited in their ability to port applications from on-premise to the cloud. Containers, on the other hand, can run multiple applications on a single operating system.

Over the past decade, Goldman faced significant challenges when it came to scaling its virtual environment and VM management applications, also known as hypervisors, such as VMware’s ESXi. The required architecture would need to support not only hundreds of thousands of VMs, but also tens of thousands of hypervisors, said Michael Hanulec, senior engineer at Goldman Sachs. Hanulec was speaking during a presentation on modernizing virtualization environments at Red Hat’s Virtual Summit on April 28.

In 2010, the bank stopped using VMware’s vSphere and vCenter Server when managing its hypervisors and VMs. Instead, the development team built an in-house control center that leveraged APIs. This acted as the foundation for unraveling virtualization stacks over subsequent years, and put the bank in a better position to scrutinize vendor contracts.

Today, Goldman Sachs runs more than 37,000 hypervisors, 99% of them still on ESXi. The rest run on Microsoft’s Hyper-V, on which the bank has over 225,000 virtual machines. Around 80,000 of those are virtual desktop infrastructures, which operate from datacenters. The remaining VMs comprise Goldman Sachs’ stateless computing environments, which are run on Linux or Windows and are stored on the bank’s private cloud.

In terms of hardware, Goldman uses both Hewlett-Packard and Dell, and primarily Intel’s Haswell central processing units. For a short spell, the bank also outsourced to tier-two tech vendors in the computer hardware space.

“We’ve since stopped ordering from those vendors. Number one, because the pricing was catching up, and number two, getting the orders fulfilled with [computer] parts and all of that was becoming an issue. And [tech] support,” Tom O’Connell, vice president and senior engineer at Goldman Sachs, said during the virtualization presentation.

In the first step of the virtualization project with Red Hat, the bank is replacing its existing ESXi hypervisors with OpenShift’s CNV worker nodes. This will mean that instances and virtual machines can be spun up to the container platform, rather than a hypervisor. O’Connell said the first phase will begin by replacing the platform underneath the workflow, but users will notice no difference in the front end.

Stepping Stones

Goldman Sachs and Red Hat first started talks about the use of OpenShift following the vendor’s annual summit in May 2018.

A few months later, Red Hat was given the task of matching the performance of a hypervisor against its CNV, which proved successful. At the time, the vendor also coded supplementary Kubernetes feature sets into the CNV that are available in KubeVirt, an open-source project that helps with the management of VMs.

Further along the development and testing pipeline, Red Hat helped Goldman improve and support its layer-two networking, or ethernet. This was particularly important as the bank uses Pre-Boot Execution (PiXE boot) provisioning, which allows workstations to boot from a server on a network prior to booting the operating system or local hard drive.

Some of the other enhancements and bug fixes that were put in place throughout 2019 include nested virtualization capabilities, to allow a firm to run one hypervisor inside another, and enhancing Windows hotspot (hot-plug in or removal) support for using Kubernetes.

Red Hat is currently working on a range of features, such as auto-pinning a VM to a host or multiple hosts so that it stays in place, as well as custom partitioning to control workflows going to local disks.

Goldman’s pilot environments are up and running on OpenShift 3.11. The next major step will involve adapting all of the work into OpenShift 4.0 and testing its capabilities at scale.

“A huge amount of the work has been ongoing in the project to make the OpenShift 4.0 environment for an investment bank high security and heavily proxied,” said Mike Pagan, senior solution architect at Red Hat.