Enterprises are investing heavily in AI infrastructure, but they are struggling to keep track of the costs and economics behind their spending, according to new research from VentureBeat Pulse. Across 107 surveyed organizations, a significant compute gap has emerged — a disconnect between the rapid pace of infrastructure investment and the limited visibility into actual compute costs and utilization.
The data reveals that while companies are planning to shift workloads to specialized AI clouds and alternative accelerators, they are still largely reliant on general-purpose cloud providers like Google Cloud and Microsoft Azure. Despite this, a clear majority — 64% — intend to switch or add providers within a year, with 38% planning to act within the next quarter.
Interestingly, price is not the primary driver in these decisions. Instead, enterprises are prioritizing integration with existing systems and total cost of ownership (TCO), with cost per million tokens ranking last in importance. However, the research shows that only 44% of respondents rigorously track their compute costs, leaving many buyers operating without a clear understanding of their economic investment.
Another concerning trend is the underutilization of existing hardware. A staggering 83% of enterprises reported GPU utilization at 50% or less, with nearly half running at 25% or below. This inefficiency is compounded by a lack of measurement tools, with many companies unable to quantify the return on their AI infrastructure investments.
The report also highlights a growing concern about the next bottleneck in large-scale inference: the shift from compute to memory bandwidth. While this constraint is expected to reshape inference architecture and cost, about 18% of surveyed enterprises either do not recognize it or have not begun to address it.
Overall, the findings suggest that while enterprises are eager to invest in AI infrastructure, they are doing so without the proper tools or visibility to ensure they are spending wisely. The compute gap, therefore, is not just a matter of capacity but of understanding what they are spending on. As companies prepare for a wave of re-platforming, the question remains: will they build the instrumentation to manage that spending effectively, or will they continue to invest blindly?
