AI Compute & Infrastructure

AI runs on compute. Every chatbot response, every generated image, every model training run requires massive computational resources. Understanding AI infrastructure helps you understand why AI costs what it does, why NVIDIA is worth trillions, and what the bottlenecks are.

The GPU Revolution

GPUs (Graphics Processing Units) weren't built for AI. They were built for rendering video game graphics. But it turns out that the same parallel processing that makes GPUs great at rendering millions of pixels is perfect for the matrix multiplications that neural networks need.

NVIDIA dominates this market. Their A100 and H100 GPUs are the workhorses of AI training. A single H100 costs around $30,000 and can perform 1,979 teraflops of FP8 computation. Training a frontier model like GPT-4 reportedly required 25,000+ A100 GPUs running for months.

The latest generation — the B200 and GB200 — push performance even further. NVIDIA's CUDA software ecosystem is as important as their hardware. Most AI frameworks (PyTorch, TensorFlow, JAX) are optimized for CUDA, creating a moat that competitors struggle to cross.

Training vs. Inference

There are two fundamentally different compute workloads in AI:

Training is where you build the model. It requires massive clusters of GPUs running in parallel for weeks or months, processing trillions of tokens. Training GPT-4 reportedly cost over $100 million in compute alone. This is a one-time (or few-time) cost for each model version.

Inference is where you use the model. Every time ChatGPT generates a response, that's inference. Inference is cheaper per request but scales with usage. As AI adoption grows, inference compute needs are exploding — some estimates suggest inference will account for 90%+ of total AI compute spending by 2027.

The hardware needs are different too. Training prioritizes raw throughput (process as much data as fast as possible). Inference prioritizes latency (respond to users quickly) and efficiency (minimize cost per request). This is why specialized inference chips and optimization techniques like quantization matter so much.

The Cloud Providers

Most AI compute runs in the cloud. The big three are:

AWS: The largest cloud provider. Offers NVIDIA GPUs, their own Trainium/Inferentia chips, and SageMaker for managed ML. Amazon is investing heavily in custom silicon to reduce dependence on NVIDIA.

Google Cloud (GCP): Home to TPUs (Tensor Processing Units), Google's custom AI chips. TPUs offer competitive performance for transformer training and are what Google uses to train Gemini. Also offers NVIDIA GPUs.

Microsoft Azure: OpenAI's cloud partner. Massive GPU clusters for OpenAI's training runs. Azure's AI infrastructure is a key competitive advantage.

Specialized providers like CoreWeave, Lambda Labs, and Together AI focus specifically on GPU compute for AI, often offering better pricing and availability for GPU-specific workloads.

The Chip Competition

NVIDIA isn't the only game in town, though they're the dominant one:

AMD: Their MI300X GPU is a serious competitor, especially for inference. Meta and Microsoft have made large MI300X purchases. AMD's ROCm software stack is improving but still lags CUDA.

Google TPUs: Custom chips optimized for transformer workloads. Not available for purchase, only through Google Cloud. Very competitive for training large models.

Intel: Gaudi accelerators target training workloads. Competitive on price/performance for specific model types.

Custom silicon: Amazon (Trainium), Microsoft (Maia), and Meta are all building their own AI chips. The goal: reduce costs and NVIDIA dependence. Apple's M-series chips have also become popular for running smaller open-source models locally.

Making AI Faster and Cheaper

Hardware is only part of the story. Software optimizations are just as important:

• Quantization: Reducing the precision of model weights (from 16-bit to 8-bit or 4-bit). Dramatically reduces memory usage and speeds up inference with minimal quality loss. This is how Llama 70B runs on consumer GPUs.
• Distillation: Training a smaller model to mimic a larger one. The small model captures most of the big model's capability at a fraction of the cost.
• Flash Attention: An algorithm that makes the attention mechanism in transformers much more memory-efficient. Now standard in most model implementations.
• Speculative decoding: Using a small, fast model to draft tokens and a large model to verify them. Can speed up inference 2-3x.
• Model parallelism: Splitting large models across multiple GPUs. Tensor parallelism, pipeline parallelism, and expert parallelism each handle different aspects of distribution.

The Energy Question

AI's energy consumption is a real concern. A single ChatGPT query uses roughly 10x the energy of a Google search. Training frontier models consumes as much electricity as a small city for months. Data centers housing AI hardware are driving significant new demand for electricity.

The industry is responding with more efficient chips, better cooling systems, and renewable energy commitments. But as AI adoption scales, energy consumption will only grow. This is a genuine infrastructure challenge, not just a PR problem.