The AI Compute Gap: Enterprises Struggle to Measure Infrastructure Costs

A recent survey of 107 UK enterprises reveals a significant gap between the rapid investment in AI infrastructure and the ability to effectively measure its costs. While many organizations are keen to enhance their AI capabilities, only 21% currently run AI in production at scale. Despite this, nearly half (45%) plan to evaluate AI-specialized cloud services within the next year, indicating a shift towards more advanced infrastructure options.

The survey highlights that a staggering 83% of enterprises report GPU utilization at 50% or less, suggesting inefficiencies in their current setups. Furthermore, less than half (44%) can rigorously track the costs associated with their AI compute resources, leading to concerns about the sustainability of their investments. The majority of enterprises are prioritizing integration and total cost of ownership over headline prices when selecting infrastructure providers, with 64% indicating plans to switch or add providers within the next year.

As organizations grapple with these challenges, the focus is shifting from traditional compute resources to memory bandwidth as the next critical constraint in large-scale AI inference. However, many enterprises remain unaware of this emerging issue, complicating their strategic planning.

The findings underscore a pressing need for better visibility into AI infrastructure costs as enterprises continue to ramp up spending without a clear understanding of their economic implications.

Source: venturebeat.com – https://venturebeat.com/ai/the-ai-compute-gap-enterprises-are-buying-infrastructure-faster-than-they-can-measure-what-it-costs