Colocation vs. cloud: SEO firm finds cloud to be cost prohibitive for its high density computing

A software firm in Singapore claims it would cost more than $400 million over three years if it were to migrate from its existing colocation setup and move its workloads to the Amazon Web Services (AWS) cloud. Notably, the firm runs a very compute-intensive environment, and high density computing can be very expensive to duplicate in cloud environments.

Ahrefs, which develops search engine optimization tools, made the $400 million claim in a March 9 blog post by one of the company’s data center operations executives, Efim Mirochnik. Mirochnik compared the cost of acquiring and running its 850 Dell servers in a colocation provider’s data center with the cost of running a similar configuration in AWS.

With colocation, a company buys its own IT equipment, but instead of building and maintaining a data center for it, the equipment is hosted by a colocation provider. This way the customer can manage its own IT systems and leave things like power management and cooling to the colocation provider.

There’s a wide swath of colocation providers, ranging from firms with just a few locations to global giants like Equinix. According to Allied Market Research, the colocation market was worth $46 billion in 2020 and is projected to top $202 billion by 2030, growing at a CAGR of nearly 16% from 2021 to 2030.

Mirochnik calculated Ahrefs’ cost per server per month at $1,550, including the cost of acquisition, for the firm’s colocation setup. That estimate factors in the cost of renting space at the colocation facility as well as electricity use, the cost of IP transit and dark fiber between the data center and point of presence, and network hardware.

To get the same hardware through AWS, he estimated the company would pay $17,557 per month for an equivalent server, including storage and data-transfer costs.

However, Mirochnik explained the comparison isn’t quite even. Ahrefs has relatively new hardware with high core-count CPUs, 2TB RAM, 2x 100Gbps per server, and, on average, 16 15TB drives. You won’t find that kind of configuration on any cloud service provider, even AWS. So Mirochnik compared one Ahrefs server to two Amazon Elastic Compute Cloud (EC2) instances:

“The cost structure for AWS is different from the colocation,” Mirochnik explained. “Unfortunately, AWS doesn’t provide an EC2 instance with the number of cores we have. So we found an EC2 configuration with half the cores and 1TB RAM. We then compared one Ahrefs server cost to the cost of two such EC2 instances.”

Storage equivalency also required some modifications. For the cost comparison, Mirochnik priced Amazon block storage, which isn’t as fast as Ahrefs’ NVMe drives.

“Apart from EC2 instances, we added Elastic Block Storage (EBS). It is not an accurate replacement for directly attached storage, as we use big and fast NVMe drives in the servers. To make things simple, we chose cheaper gp3 EBS (much slower than ours, though). Its cost consists of two parts: storage size and charges for IOPS.”

The Amazon storage may have been slower than Ahrefs’ NVMe drives, but it sure was pricey. Ahrefs estimated the monthly cost of EBS at $11,486 – which accounts for 65% of the $17,557 total monthly cost. The EC2 instances came in at $5,606, and data transfer costs were estimated at $464.

Given the enormous cost and lesser storage capability, AWS was no replacement for an on-premises configuration for Ahrefs.

“A replacement EC2 instance with similar usable SSD space in AWS would cost us roughly the same amount of money for 11.3 servers in a colocation data center. Accordingly, that means our 20-server rack would transform into just ~2 servers,” Mirochnik wrote.

At $1,550 per server, Mirochnik calculated the cost of running its own 850 servers in a colocation data center at $1.3 million per month. Over a 30-month period, it would cost $39.5 million.

By comparison, Mirochnik estimated the monthly cost for AWS EC2 instances with an equivalent amount of computing as the on-prem setup at $14.9 million, which translates to $447.7 million over 30 months.

The $447.7 million price for AWS would have greatly exceeded the $257 million in revenue Ahrens reported during the same 30-month period. It doesn’t take a CFO to figure out that the company would be running at a massive loss in this scenario.

“Ahrefs would hardly survive if a cloud was our primary infrastructure destination,” Mirochnik concluded.

HPC not ideal for cloud environments

Ashish Nadkarni, group vice president and general manager of IDC’s worldwide infrastructure research organization, notes that Ahrefs operates a high-performance computing environment with the gear it uses, and the cloud is not built for sustained high-performance computing in general.

“The economies of scale start to become less compelling as you start using more premium products or premium services from these corporate players,” Nadkarni said.

If you implement an HPC environment in the cloud without doing your homework first, you could end up spending two to three times more over a period of three to four years just in operational costs alone, Nadkarni warned.

“Nobody’s saying get rid of public cloud services. Absolutely not. What we do say is you want to place the workload at a location in an operating environment that is doing justice to the workload itself,” he said.

If you have an HPC environment that only needs to run two hours a day, then you’re probably not serving yourself well by going with an on-prem solution. But if you have an HPC environment that runs 24 hours a day, 7 days a week, 365 days a year, then you should think about whether it is best to invest in your own on-premises or colocation infrastructure, he added.