Grok 4 Colossus: The Memphis Substation Did More Work Than the Math Units

Everyone is going to screenshot the Grok 4 benchmark slides to death this week, so let me start somewhere less crowded. One hundred thousand H100S are sitting in a converted appliance factory in Memphis, pulling somewhere between 150 and 200 megawatts when the cluster is hammering on a training run. The Grok 4 Colossus cluster is the part of this launch that took real engineering. The numbers everyone is about to argue about fall out of it.

xAI shipped Grok 4 on July 9, five months after Grok 3 landed in February, and the pitch is the one everyone in the frontier club is making now: a reasoning model that thinks before it answers, tuned hard for math, science, and code. Access is the usual menu. Grok.com is behind an X Premium+ paywall, or the API if you’re building on it. It’s also just shoved into X whether you asked for it or not. Personally, the interesting question was never where you type the prompt. It is whether the architecture and the iron underneath make the noise.

On architecture, xAI has stayed quiet. No full technical report at launch, so most of what I can say with confidence comes from inference and from what they have admitted before. Grok 1 was an open-weights release back in March 2024, and that one we know cold: 314B total parameters, Mixture-of-Experts, eight experts with two firing per token, so roughly 86B parameters actually active on any given forward pass. xAI never confirmed that Grok 3 or Grok 4 kept the MoE structure, but sparse activation is how everyone scales now, because it lets you pile on total parameter count for knowledge capacity without paying for all of it on every token. Betting against MoE here would be strange.

What actually changed between Grok generations is the reasoning phase. Grok 4 runs an extended thinking pass before it generates a visible response. Internally, it produces a scratchpad of reasoning tokens, runs self-verification over them, burns a compute budget that scales with the problem’s difficulty, and then conditions the visible answer on that hidden trace. Those thinking tokens are generated autoregressively, just like everything else; they are just masked out of what you see. It’s the trick OpenAI’s o1 and o3 pull and what Claude does in its extended thinking mode. Nobody in this tier is doing anything architecturally novel anymore. The differentiation lies in how well you trained the model and how much compute you are willing to set on fire per query, which is exactly why Memphis is the only part of this launch that is genuinely hard to copy.

Which brings me back to the cluster. Colossus came online in 2024 and trained Grok 3 first; xAI stood up the initial phase, 100,000 HH100Sin SXM5 form factor, inside a gutted Electrolux factory in something like 122 days. Under standard DGX H100 nodes, that is about 12,500 boxes, each with eight GPUs. Inside a node, they are stitched together with NVLink at 900 GB/s per GPU, and between nodes, it rides on NVIDIA’s Spectrum-X Ethernet fabric. Do the FP8 math, and you land around 396 exaFLOPS of sparse peak, or roughly 198 exaFLOPS at BF16 if you want a number that resembles reality during a training run. The figure that actually keeps me up at night is the aggregate memory: something on the order of 8 petabytes of HBM across the cluster, with hundreds of petabytes per second of bandwidth when all 100,000 chips read at once.

For Grok 4, xAI talked about pushing toward 200,000 H100-equivalent GPUs and folding in H200S, with GB200 racks on the roadmap, in a market where just laying hands on H100S at this volume was the hard part for everyone else. Treat the exact second-phase numbers as unconfirmed until they show receipts; the direction is obvious, and the reason is the H200’s memory. Compute stays put: the H200 does the same 3,958 TFLOPS at FP8 and 1,979 at BF16 as the H100, and it draws the same 700 watts doing it. What moves is the memory, HBM3e at 141GB instead of 80GB, and 4.8 TB/s of bandwidth, where the H100 gave you 3.35. That is 76% more capacity and 43% more bandwidth on identical math units. For training a giant sparse model, the extra HBM means you fit bigger shards per GPU and shove less data across the slow inter-node links. Memory, not raw FLOPS, is the constraint that bites at this scale, the same memory-first bet AMD has been pushing while everyone stared at FLOPS charts.

To actually use 100,000 GPUs on a single model, you have no choice but to split the work across every axis at once. Tensor parallelism slices individual weight matrices, usually into eight ways to stay inside one NVLink domain, with an all-reduce inside the node where the 900 GB/s link can absorb it. Pipeline parallelism chops the model’s layers into stages spread across nodes, with point-to-point traffic riding the Ethernet fabric. Whatever GPUs are left over go to data parallelism, replicating the whole tensor-by-pipeline unit and synchronizing gradients across the copies. Run the arithmetic on a 100K cluster with tensor parallelism 8 and pipeline parallelism 16, and you are left with roughly 781 data-parallel workers. None of that fits in memory naively, so you also need ZeRO-style sharding of optimizer states, gradients, and parameters across the data-parallel dimension. That is not optional at this scale. It is the only reason the run starts at all.

The full communication-cost accounting is a rabbit hole, so skip it. The short version: at 100K GPUs, the Ethernet topology and how you schedule all-reduces across pipeline stages matter more than the GPU spec sheet. A poorly tuned collective communication library will idle half your cluster while waiting for gradient synchronization, and that is a software problem, not a hardware one. Franchement, that is where the real operational expertise lives. The chips were never really the scarce thing. Powering a hundred thousand of them and keeping them all busy without the run collapsing, that was the hard part, and it’s the part nobody puts on a slide.

xAI has been making claims faster than it has been publishing verifiable scores, and that irritates me. The headline pitch is top-percentile math. Take AIME 2025, the 15-question competition exam: GPT-4o lands around 4-6 out of 15, while o3 sits at 13-14. FrontierMath is the deliberately unpublished problem set built to defeat memorization, where GPT-4-class models scored under 2% at launch, and o3 jumped well past that. On GPQA Diamond for graduate-level science, human experts hit roughly 69%, GPT-4o managed about 53%, Claude 3.5 Sonnet around 65%, and o3 climbed to 87.7%. SWE-bench Verified, the test for resolving real GitHub issues, put Claude 3.5 Sonnet at around 49% and o3 at around 71.7% on the early-2025 board. Those are the yardsticks Grok 4 has to meet to back up the word “frontier”. I’ve seen plenty of marketing positioning. What I haven’t seen is a single independently reproduced number I’d stake my name on, so until a neutral third party runs these, every claimed score goes in the trust-but-verify pile.

Serving a model this size is its own bill. A 256K-token context window builds a monster KV cache, tens of gigabytes per single request,t once you account for two cache tensors per token per layer across every attention head. There are ways to claw it back, none of them free. The standard move is vLLM-style PagedAttention, smarter paging for the cache,e so you stop wasting memory on fragmentation. After that, you’re into compromises: quantizing the cache to INT8 or FP8, or sliding-window attention on the layers that won’t miss the long tail. For a big, sparse model, you also lean on speculative decoding, where a small, fast draft model proposes a handful of tokens and Grok 4 verifies them in parallel, accepting the run until the first mismatch and resampling from there. Done right,ght that is a 2x to 4x throughput win on normal text, and it is the difference between an inference bill that is merely brutal and one that is catastrophic.

Putting a real number on the cost is a fool’s errand right now. xAI hasn’t published utilization figures, and the API pricing will move the moment they feel competitive heat from Anthropic or Google. Raw token throughput at scale is a solved problem if you’re willing to build custom silicon, and xAI isn’t doing that. They are buying NVIDIA at volume and praying the margin math works out. The HBM supply chain that makes all of this possible is its own geopolitical story, and not one that resolves cleanly in xAI’s favor if export controls tighten further.

On paper, Grok 4 is the same animal as everything else: a sparse reasoning transformer with a hidden thinking phase and a 256K window. Gemini 2.5 Pro out-reaches it on context with a full million tokens, and for actual openness, you’re looking at Meta’s open-weights line, most recently Llama 4 Maverick, a roughly 400B MoE that will show you its weights. So the model card is unremarkable. The wild part is that xAI built the biggest single-site GPU cluster on Earth in about a year to feed it—capital and logistics, wearing an AI-research costume.

Anyone with a thinking-mode transformer and a few exaFLOPS to burn can chase O3’s numbers. What nobody else pulled off was raising 100,000 HH100Sin in a dead appliance factory faster than anyone thought possible, then keeping the power flowing into it without the whole run falling over. That last part is the actual moat. The benchmarks are just what gets the screenshots.