Apple’s M4 chip hit 38 TOPS on the Neural Engine, and for once, I am not rolling my eyes at an Apple performance claim. The number is real. The architecture shift is real. And the competitive pressure it puts on Qualcomm and MediaTek in the on-device AI race is very real.
But here is what matters more than the 38 TOPS figure: Apple is finally treating on-device AI inference as a product pillar, not a spec-sheet footnote. The M4, unveiled on May 7, 2024, features a 16-core Neural Engine built on second-generation 3-nanometer technology, paired with a new 10-core CPU and 10-core GPU. It supports real-time image processing and on-device machine learning inference with lower latency, and it signals that Apple is done playing catch-up in the mobile AI space.
Meanwhile, Google DeepMind continues to refine its cloud-first Gemini family of multimodal large language models. Gemini 1.5 Pro just hit general availability with a 1-million-token context window and multimodal reasoning improvements across text, images, and code. Google also introduced “TurboQuant,” a compression method requiring up to six times less working memory while maintaining performance, and is experimenting with “Gemini Diffusion,” a text diffusion model for faster, more coherent text generation. Every one of these advancements is optimized for centralized, cloud-based inference.
And then there is Intel, stuck in the middle, trying to serve both markets with its Xeon 6 CPUs. The Xeon 6 family includes Sierra Forest (E-cores, up to 144 cores, launched early June 2024) and Granite Rapids (P-cores, up to 128 cores, expected Q3 2024). These processors boast integrated AI acceleration and support for TensorFlow and PyTorch, and they are designed to handle data center workloads. But the strategic positioning is muddled. Intel is hedging its bets, trying to be the infrastructure spine for both on-device and cloud AI, and that indecision will cost them.
The AI stack is splitting into two incompatible architectures, and the companies that try to straddle both will lose to the specialists.
The On-Device Thesis: Why 38 TOPS Matters
Let me be clear about what the M4’s 38 TOPS actually represents. For context, the Snapdragon X Elite’s Hexagon NPU hits 45 TOPS, and the Snapdragon 8 Gen 3 hits 73 TOPS in burst mode. Apple’s 38 TOPS is competitive, not class-leading. But Apple’s advantage is the software stack. The Neural Engine is integrated into Core ML and the OS-level inference pipeline in a way that Snapdragon’s Hexagon is not yet matched by Android OEMs.
On-device AI inference offers three critical advantages: latency, privacy, and sovereignty. When your AI model runs locally, you cut out the round-trip time to a remote server. You keep your data on your device, not in someone else’s cloud. And you control the update cycle and the weights.
Apple understands this. Qualcomm understands this. MediaTek is starting to understand this. The entire mobile AI competitive landscape is shifting toward NPUs and Neural Engines that can handle real-world inference tasks without phoning home to a data center.
This is not just a performance race. It is an architecture decision with profound implications for app developers and enterprise buyers. If your application depends on low-latency AI (augmented reality, real-time translation, voice processing), on-device inference is the only viable path. If your application handles sensitive data (medical records, financial transactions, personal communications), on-device inference is the only compliant path.
Apple’s M4 signals that the company is done treating AI as a cloud service problem. They are building the inference stack into the device itself, and that changes everything.
The Cloud-First Thesis: Why Centralized Inference Still Wins for Training
On the other side of the split, Google, Anthropic, and OpenAI are doubling down on cloud-first AI. And for good reason. Training large language models requires infrastructure that no device can match. Google’s TurboQuant compression method is not designed to make on-device inference feasible. It is designed to make cloud inference more efficient, reducing the memory footprint of models running on TPUs in data centers.
Gemini Diffusion, Google’s experimental text diffusion model, is another cloud-first optimization. It is faster and more coherent than previous approaches, but it still assumes centralized compute. The entire Gemini family is architected around the assumption that the heavy lifting happens in Google’s data centers, not on your phone or laptop.
This is not a weakness. It is a strategic choice. Cloud-first AI offers scale, flexibility, and rapid iteration. When OpenAI ships a new GPT model, every user gets the update instantly. When Apple ships a new Neural Engine, you need to buy new hardware.
The cloud-first thesis is that centralized inference will always outperform distributed inference for complex, evolving models. And for many use cases (content generation, deep research, multi-modal synthesis), that thesis holds.
But here is the rub: cloud-first AI cedes control. If your AI runs on Google’s TPUs, Google controls both the infrastructure and the weights. If your AI runs on Anthropic’s Claude, Anthropic controls the update cycle and the policy guardrails. That is fine for consumer applications. It is a dealbreaker for regulated industries and sovereign governments.
The Infrastructure Spine: Intel Xeon 6 and the Middle Ground Problem
Intel’s Xeon 6 launch is not a CPU update. It is a defensive repositioning against AMD’s EPYC dominance in the data center and Ampere’s energy efficiency play. Sierra Forest’s 144 E-cores and Granite Rapids’s 128 P-cores are impressive on paper, and the integrated AI accelerators are real (supporting TensorFlow, PyTorch, and other ML frameworks).
But Intel is trying to be the infrastructure spine for both on-device and cloud AI, and that strategy is incoherent. On-device AI workloads do not run on Xeon CPUs. They run on NPUs, Neural Engines, and Hexagon processors. Cloud AI workloads increasingly run on specialized accelerators (Nvidia H100s, Google TPUs, AMD Instinct MI300), not general-purpose CPUs.
Xeon 6 is positioned as a hybrid solution, capable of handling traditional data center workloads (databases, virtualization, networking) while also accelerating AI inference. And for enterprises that need both capabilities in a single platform, Xeon 6 makes sense. But the market is moving toward specialization, not generalization.
AMD’s EPYC Genoa and Bergamo lines are eating Intel’s lunch in raw compute density and energy efficiency. Ampere’s Altra Max is winning the cloud-native, ARM-based server market. Nvidia’s Grace CPU is designed explicitly to pair with H100 GPUs for AI training. Intel’s competitive advantage (x86 compatibility, broad software support, established supply chains) is eroding in a world where workloads are increasingly containerized, cloud-native, and accelerator-dependent.
The Xeon 6 AI accelerator integration is a band-aid on a strategic wound. Intel is fighting a credibility war, not a technology war. Xeon 6 is good enough to keep them in the game. Whether it is good enough to win back market share depends on execution at scale, and Intel’s track record there is mixed.
The Sovereignty Question: Who Controls the Weights, Who Controls the Compute
I keep coming back to the same question: who controls the inference?
If your AI runs on Apple’s Neural Engine, Apple controls the update cycle but you control the weights. If your AI runs on Qualcomm’s Hexagon NPU, Qualcomm controls the hardware roadmap but you control the model. If your AI runs on Google’s TPUs, Google controls both the infrastructure and the policy layer.
This is not an abstract question. It is the defining architectural decision for enterprises, governments, and regulated industries deploying AI at scale. The European Union’s AI Act, China’s export controls on AI chips, and the U.S. restrictions on Nvidia H100 exports to certain countries all hinge on this sovereignty question.
On-device inference offers sovereignty. You own the hardware. You own the weights. You control the update cycle. The tradeoff is performance and scalability. Cloud-first inference offers scale and flexibility. The tradeoff is control.
The companies that picked a side early (Apple and Qualcomm for on-device, Anthropic and OpenAI for cloud) have already won their lanes. The ones still hedging (Google, Intel) are burning capital to stay in both races and winning neither.
Google’s Gemini family is cloud-first, but Google also ships TPUs for on-premise deployment and integrates AI accelerators into Pixel phones. Intel’s Xeon 6 is a data center chip, but Intel also acquired Habana Labs to build AI accelerators and invested in Movidius for edge AI. These are not coherent strategies. They are hedges.
And hedges do not win markets. Specialists do.
The future is not hybrid. The future is specialists who own their stack, end to end. Apple owns the device (M4 Neural Engine, Core ML, tight OS integration). Anthropic owns the cloud (Claude models, Constitutional AI, API-first deployment). The companies stuck in the middle will discover, as Intel is discovering with Xeon 6, that serving two masters means winning neither market.
For more on the AI sovereignty question as it has played out since this analysis, see what happened when the US government pulled Anthropic’s access for foreign nationals. For a broader look at Qualcomm’s competitive positioning in the on-device AI race, see the Snapdragon X Series deep dive from CES 2026.
The split is permanent. The question is not whether you will pick a side. The question is whether you will pick early enough to matter.