PitchBook moves to a microservices infrastructure – scaling the business through scalable tech

PitchBook is a data company. Its reason for being is to provide a platform that tracks a plethora of different aspects of both private and public markets. Want to know about what’s happening in venture capital, private equity or M&A? Chances are PitchBook can give you the answer. The company is a subsidiary of Morningstar and has offices in Seattle, New York, and London.

But here’s the thing, though. PitchBook was founded in 2007 when cloud computing was pretty much just beginning and there was no real awareness of what it meant. In those days, enterprise IT agility meant leveraging virtualization to gain efficiencies. Now don’t get me wrong, moving from a paradigm of racking and stacking physical servers to being able to spin up virtual servers at will is a big deal, it’s just that since 2007, there has been massive further innovation in the infrastructure space.

So if you’re PitchBook, built in the early days of the cloud in a monolithic way, and you want to scale to your stated business ambition of hosting data about 10 million companies, what do you do? Well, one thing you can do is to rethink your entire infrastructure footprint to take advantage of modern approaches. And this is what PitchBook has done, moving from a monolithic infrastructure to microservices, which should enable PitchBook developers to easily scale the platform.

“Breaking from a monolithic environment will allow us to easily make changes under the hood of different modules without affecting any of the other services tied to it. This ultimately is pushing the PitchBook Platform into a new era, defined by greater scale and usability,” said Alex Legault, lead product manager at PitchBook. “With an aggressive product roadmap that involves loading massive datasets, leveraging modern cloud techniques and enabling more machine learning, a microservices infrastructure will provide the right framework to execute on our plans, quickly and efficiently.”

The PitchBook journey piqued my interest and so I sat down (in the modern sense of the word where “sit down” means “get email answers to questions”) with Legault to learn more about this journey. Without further ado, here’s the PitchBook story.

What tech are you using? K8S? Docker? Mesos? Serverless?

We made a lot of moves to new tech with our front end in this release: React, ES2016 (EcmaScript 2016 – version of Javascript). Spring too. We’re currently evaluating Docker and K8S.

Why did you make the decision to migrate to microservices?

Our clients need to move fast and require timely access to data and new datasets. To meet these needs, we require an architecture that will allow our product team to run fast and scale. Microservices provides this. At PitchBook, we’re at a critical inflection point where we’re growing at a rapid pace, and the platform needs to keep up, both from a data perspective as well as from a feature set and scalability standpoint. While a monolithic infrastructure could have met our needs, as our platform gets bigger and more complex, it would get increasingly challenging to make changes or updates. With microservices, each service becomes its own module allowing our developers to easily make changes without impacting other services.

+ MORE ON NETWORK WORLD What you need to know about microservices +

In some instances, microservices can lend itself to an explosion of modules/services that need to be managed within an enterprise. Did you think about that going into this migration and what sorts of management technology have you implemented to avoid the chaos that some companies are facing?

Moving to microservices naturally creates the problem of module explosion. There are few recipes to avoid or minimize this:

1) Mini-services versus nano-services approach. We tried not to be too idealistic and not design microservices as “nano”-services. Getting too small and too specific with the services can quickly introduce a headache. For us, it made sense to start with bigger modules, which we call “mini”-services first, then adapt and split down further when necessary. Each team can control this process and split things only when it serves a real purpose or advantage to do so.

2) Unify the service interface and infrastructure, use containerization and orchestration. Our ideal end state is a fully programmable and automated infrastructure (IAC), which requires a formalized DevOps function. Can’t state enough how important having good DevOps folks is in making this transition successful.

What will this switch allow you to do? What’s next in the road map where microservices will play a huge role?

There are several benefits microservices provides us, including:

· It will allow us to speed up delivery of new features, innovations and data sets. Our goal is to eventually host 10 million private and public companies within the platform and microservices will help us get there faster and with scale.

· We can also more easily adopt different technology where needed and aren’t bound to the same databases or languages in any part of the application.

· Redeployment will become easier. While the system is more fragmented, it’s less fragile so when individual services are down, it doesn’t bring down the entire system.

· Allows us to scale individual services that are the bottleneck, it’s not just one big instance anymore. This helps us with scaling as our datasets grow.

On the horizon, we have several initiatives related to high-speed data visualization and analysis. We have such great datasets, so how can we generate and surface more insights to customers. Microservices will play a huge role in enabling this.

How will your customers benefit from the switch?

We’re all about serving our customers which is why we made this move. Institutional investors are under more pressure than ever before to make intelligent investment decisions and generate higher returns, making access to quality data absolutely essential. New technology that can help us recommend, analyze and surface personalized insights to customers is hitting the jackpot –  we‘re confident microservices can unshackle us so we can go after these initiatives. Customers can expect to start seeing more releases, more innovation and a platform that can handle much larger scale while staying fast.

MyPOV

Technology is a progression – mainframes to physical x86 to virtualization. Microservices is but the latest move in this process and we can already see things on the horizon (event-driven infrastructure, for example) that will take organizations like PitchBook to the next level. It is interesting to have a glimpse inside and explore the thinking that goes into a significant platform shift.