AI is evolving at an unprecedented pace, driving an urgent need for more powerful and efficient data centers. In response, nations and companies are ramping up investments into AI infrastructure. According to Forbes, AI spending from the Big Tech sector will exceed $250 billion in 2025, with the bulk going towards infrastructure. By 2029, global investments in AI infrastructure, including data centers, networks, and hardware, will reach $423 billion.
However, the rapid AI innovations also put unprecedented strain on data center networks. For instance, Meta’s recent paper on the Llama 3 405B model training cluster shows it requires over 700 TB of memory and 16,000 NVIDIA H100 graphical processing units (GPUs) during the pre-training phase. Epoch AI estimates that AI models will need 10,000 times more computational power by 2030 than today’s leading models.
If your organization has a data center, it’s only a matter of time before AI is deployed there. In this blog, we’ll explore the key challenges of AI cluster scaling and reveal why “the network is the new bottleneck.”
The Rise of AI Clusters
An AI cluster is a large, highly interconnected network of computing resources that handles AI workloads.
Unlike traditional computing clusters, AI clusters are optimized for tasks such as AI model training, inference, and real-time analytics. They rely on thousands of GPUs, high-speed interconnects, and low-latency networks to meet the intensive computational and data throughput requirements of AI.
Building AI clusters
An AI cluster is, at its core, functions like a mini network. Building an AI cluster involves connecting the GPUs to form a high-performance computing network where data can flow seamlessly between GPUs. Robust network connectivity is essential, as distributed training relies on the coordination of thousands of GPUs over extended periods.
Key components of AI clusters
AI clusters consist of multiple essential components, as shown in Figure 1.
- Compute nodes behave like the brain of the AI clusters, with thousands of GPUs connecting to the top-of-rack switches. As problem complexity increases, so does the need for GPUs.
- High-speed interconnects such as Ethernet enable rapid data transfer between compute nodes.
- Networking infrastructure includes network hardware and protocols supporting data communications between multiple thousands of GPUs over extended periods.
Scaling AI Clusters
AI clusters scale to meet the growing AI workloads and complexities. Until recently, network bandwidth, latency, and other factors had limited AI clusters to around 30,000 GPUs. However, xAI’s Colossus supercomputer project shattered this barrier by scaling to over 100,000 NVIDIA H100 GPUs — a breakthrough made possible by advancements in networking and memory technologies.
Key Scaling Challenges
As AI models grow to trillions of parameters, scaling AI clusters involves myriad technical and financial hurdles.
Network challenges
GPUs are effective at performing math calculations in parallel. However, when thousands — or even hundreds of thousands — of GPUs work together on the same task in an AI cluster, if even one GPU lacks the data it needs or encounters delays, every other GPU stalls. Such prolonged packet latency or packet loss contributed by a congested network can cause packet retransmissions, significantly increasing job completion time (JCT) and leaving millions of dollars’ worth of GPUs sitting idle.
Additionally, AI workloads generate a dramatic rise in east-west traffic (data moving between nodes within the data center), potentially leading to network congestion and latency issues if the traditional network infrastructure isn’t optimized for these loads.
Interconnect challenges
As AI clusters expand, traditional interconnects may struggle to provide the necessary throughput. To avoid bottlenecks, organizations must upgrade to higher-speed interconnects, such as 800G or even 1.6T solutions. However, deploying and validating such high-speed links is no easy feat to meet the rigorous requirements of AI workloads. The high-speed serial paths must be carefully tuned and tested for the best signal integrity, lower bit error rates, and reliable long forward error correction (FEC) performance. Any instability in high-speed serial paths can degrade reliability and slow down AI training. Companies need highly accurate and efficient test systems to validate them before deployment.
Financial challenges
The total cost of scaling AI clusters goes well beyond the expense of GPUs. Organizations must factor in power, cooling, networking equipment, and broader data center infrastructure. However, accelerating AI workloads through better interconnects and optimized network performance can shorten training cycles and free up resources for additional tasks. Each day saved on training can translate into significant cost reductions, making the financial stakes as high as the technical ones.
Validation Challenges
Optimizing an AI cluster’s network performance requires testing and benchmarking the performance of both the network fabric and the interconnects between GPUs. However, validating these components and systems is challenging because of the intricate relationships among hardware, architectural design, and dynamic workload characteristics. There are three common validation issues.
No 1. Lab deployment constraints
The high cost of AI hardware, limited equipment availability, and the need for specialized network engineers make full-scale replication impractical. Additionally, lab environments often have space, power, and thermal constraints that differ from real-world data center conditions.
No 2. Impact on production system
Testing on a production system reduces can be disruptive, potentially affecting critical AI operations.
No 3. Complex AI workloads
The diverse nature of AI workloads and data sets — varying significantly in size and communication patterns — makes it difficult to reproduce issues and benchmark consistently.
As AI reshapes the data center landscape, future-proofing network infrastructure is crucial to staying ahead of rapidly evolving technologies and standards. Keysight’s advanced emulation solutions provide a critical advantage by enabling comprehensive validation of network protocols and operational scenarios before deployment. Explore how we empower network engineers to navigate AI workload complexities and optimize network performance — ensuring scalability, efficiency, and readiness for AI demands.