Supply-chain constraints spike data-center outages

Shortages due to supply-chain problems has faded from the headlines, but they persist and are actually getting worse in some sectors.

A report from AFCOM, an organization of data-center professionals, found that 44% of data-center operators surveyed had suffered downtime or an outage because they couldn’t get needed parts.

In the report “State of the Data Center 2023,” 94% of respondents said that they have had supply-chain issues of one kind or another. The most commonly cited items in short supply (59%) are basic IT equipment such as servers and switches, followed by power systems (51%) including generators and UPSes.

But even acquiring security systems (34%) such as secure doors and cameras and building materials like concrete (35%) were an issue for many respondents.

Bill Kleyman, program chair for AFCOM and a data-center consultant, said what really jumped out at him was the news that last year 44% of data-center operators experienced some type of outage due to parts being unavailable. That number the year before was 25%.

Continuing issues with the supply chain itself–such as Covid-related shutdowns–are compounded by operators seeking to grow data centers faster.

“A lot of private equity firms have bought quite a few data centers, including my former company [Switch], and they are being tasked with bringing up as much capacity as they can as quickly as they can to support a very hungry market,” Kleyman said.

So you have an industry being pressured to grow and expand as quickly as possible trying to get parts in short supply.

Kleyman said he knew of one data center that was all but complete and needed just one specific part that was manufactured in Italy that it couldn’t get. The owner had to wait an three months to get the piece to bring the facility online.

He cited other specifics, like a UPS container that took 50 weeks to be delivered when it used to be delivered in eight and a generator enclosure that took more than 50 weeks when it used to be delivered in 10. Chillers for cooling can take up to 70 weeks when it used to be 16, he said.

“I think we are realizing just how fragile the supply chain really is. The demand is just huge. So a lot of these manufacturers simply trying to catch up to what the requests are. The pandemic really was an accelerator. And now we’re trying to sort of dig out from the supply chain issues,” Kleuman said.

Buyers aren’t exactly helping. Kleyman said customers are binge buying pieces of equipment that they might need, causing a lot of data centers to stock up on gear where they used to buy just what they need and nothing more.

The problem is more common among hyper scale data centers than enterprise data centers, which don’t turn over equipment is rapidly. The shortage affects on-prem data centers less than colocation facilities because they are more willing to extend the life of their gear. “If they can’t get something, they’re like, ‘All right, that’s cool. We’ll wait for it,’” said Kleyman.

Recommendations

His advice is for data centers to stockpile critical parts when they can, “Because yes, you do have to house more inventory in your facilities, but it does buffer challenges to getting that part.”

Kleyman recommends leaders look to partners that can detail how they will deal with specific disruptions such as air-freight or rail problems.

He said operators should consider the used/recycled market for IT equipment, such as ITRenew. If that means getting GPUs a few generations old, so be it. They are stil powerful systems that may take longer to run certain workloads than the most advanced technology, but at least the tasks get done.

“It doesn’t work for everybody, but I really feel that the circular economy where you can get this stuff is massively important, because hyperscalers retire their infrastructure pretty regularly,” he said.

Finally, do what 29% of respondents in the survey did and start sourcing from multiple vendors. IT shops generally prefer to buy from one vendor that they know and work well with. In a shortage, that may not be a viable option anymore.