Diversity of AI workloads means there’s a market for many types of chips.
AI is a multifaceted discipline, and there’s room for many different chips beyond Nvidia GPUs, although consolidation is inevitable. That’s the conclusion of a research report from the consultancy J.Gold Associates.
The processor market for AI-enabled systems is vast and highly varied given the many uses and work conditions for AI, including the data center, cloud, and edge. No current vendor will be able to capture it all, and specialized vendors will be coming online in the next one or two years to further diversify the available solutions, concludes Jack Gold, president of the consultancy.
The vendor landscape will expand beyond the few now in a leadership role, Gold expects. Shifting needs and high growth for AI-enabled systems will offer major opportunities to a variety of vendors that will concentrate on specific AI system types and processing areas, he predicts.
“There is enough width and breadth in the marketplace to float all boats, so to speak,” Gold said. “If we look out two or three years, the majority of AI workflows are not going to be on [Nvidia] H100s – you know, high-end machine learning stuff. It’s going to be inference loads. It’s going to be in the edge primarily, but it’s also going to be on PCs and mobile devices. IoT will have some. So, there’s going to be a wide array of workloads in AI that are not only deployed on high-end Nvidia chips.”
Breaking down the different markets, and starting with cloud and hyperscalers, Gold sees AWS and company offering near-Nvidia levels of performance to their customers with their own custom chip technologies but at a lower price than the expensive Nvidia chips.
“We expect the hyperscaler market to offer a wide array of processors to meet the growing diversity of AI training needs, as well as some higher end inference based workloads. While in the short term, Nvidia’s dominance in this segment is likely protected, longer term (2+ years) we expect a significant dilution of its market share,” he wrote.
In the data center, he expects to see more traditional data center servers running AI workloads as they move toward inference-based workloads, as well as fine tuning and RAG optimizing of existing models. Inferencing is much less process-intensive than training and can be done on traditional CPUs instead of more expensive GPUs.
This is opening up an opportunity for AI as a service, provided by major cloud service providers, where a company can have the AI training done on the expensive hardware without having to make a major capital investment in hardware they only need once and then do the updates or inferencing with their own gear.
“It’s also likely that as newer, more efficient modeling methods are developed, they will increasingly be run on traditional servers, both from a cost/performance advantage perspective as well as for greater compute availability. This will benefit the traditional players who have a well-established data center business,” Gold wrote.
On the edge, Gold expects the vast majority of AI workloads to migrate to edge-based systems over the next two or three years. What qualifies as the edge is a wide range of systems and processing capabilities – from small internal processing in sensor arrays to heavy machinery, autonomous vehicles and medical diagnostics, just to name a few.
Gold predicts that open-source platforms and development environments will play a key role in this space as opposed to proprietary solutions like Nvidia’s CUDA. “Open and compatible ecosystems like Arm and x86 from will have significant advantages as they create compatibility from small to large computing needs. They allow up scaling or down scaling as the processing requires as well as ease of porting solutions and reuse,” he wrote.
The IoT space has a lot of overlap with edge computing, and therefore there is a need for an open ecosystem to provide scalable solutions, much like the edge. It’s just that with IOT, the devices tend to be smaller and lower power, but there are plenty of players in that field.
With so much hype surrounding AI, there has been a significant number of startups in the AI processor market in the last few years, with more to come in the next few years. But since they are relatively new players with a lack of established market presence and proven capability, it’s difficult to position them effectively as they stake out their particular niches, he said.
“We do expect a few of the new entrants to ultimately be successful, while many of the others will either fade away or be acquired in the next two to three years,” he wrote. He cited Cerebras as a notable startup, with its wafer scale technology that is positioned at the high end of the market and challenging the dominance of Nvidia.
