Inference Runs in Two Places That Barely Talk to Each Other

The Snapdragon X Elite NPU showed up at my local Microsoft Store the same week the US government quietly walked back its restriction on NVIDIA’s H20, the data center chip it had blocked three months earlier in April, a part NVIDIA had built to be compliant in the first place. One is a laptop you grab off a shelf. The other is a 500W accelerator caught inside a geopolitical chess move. They read like separate conversations. They are the same story told from opposite ends of the compute stack, and the thing tying them together is the dullest spec on either datasheet: memory bandwidth.

Both are about where inference actually runs. Right now it runs in two places that are barely on speaking terms.

What the Snapdragon X Elite is

The Snapdragon X Elite in the top Surface SKU is the X1E-84-100, a 4nm TSMC part with twelve Oryon cores, each identical. No efficiency cluster, no big.LITTLE split. Qualcomm built a homogeneous Oryon arrangement running up to 3.8 GHz boost and holding roughly 3.4 GHz all-core under thermal load, and that single decision tells you what the chip is for. Mobile silicon gets tuned for burst-and-idle, where you sprint for a notification and then sleep. A flat twelve-core layout is tuned for the opposite, a workload that just keeps pushing. That is a PC philosophy wearing an ARM badge.

The number everyone quotes is 45 TOPS, and it belongs to the Hexagon NPU, not the CPU or the Adreno 830 GPU. The Adreno does 4.6 TFLOPS FP32, speaks Vulkan 1.3 and DX12 Ultimate, and can grind INT8 matrix ops if you ask, but it is not the inference path. The Hexagon Tensor Processor is. It is a systolic array that performs the matrix-matrix multiply and sits underneath every transformer, natively supporting FP16, INT8, and INT4 in hardware rather than emulating low-precision operations in software.

Coverage mangles that 45 TOPS constantly. TOPS is an INT8 multiply-accumulate count. Drop to FP16 and your effective throughput is roughly half, call it 22 TFLOPS-equivalent, which Qualcomm does not bother publishing because it is a worse-looking number. What actually matters is what the thing does. A 7B model like Llama 3 8B at INT4 runs around 20 to 30 tokens per second on the NPU path depending on context length. SDXL at 512×512 lands 8 to 12 iterations per second, so a 20-step generation finishes in under three seconds. Whisper large-v3 transcribes at about 0.3x realtime, ten minutes of audio in three. None of those are datasheet maximums dreamed up in a lab. They are what you get through Windows ML and the ONNX Runtime QNN execution provider, which is the road basically every Copilot+ feature drives down.

The bandwidth wall

The microarchitecture underneath is a 2D systolic array and Qualcomm will not say how big. Working backward from 45 TOPS and the 4nm node, the estimates land somewhere around 256×256 INT8 MAC units at roughly 1.7 GHz effective. Around the tensor unit sits a scalar unit for the activation functions and softmax, a 1024-bit SIMD vector unit, and the HMX instruction set that exposes the whole thing to the compiler. HMX is the part developers actually touch, through Qualcomm’s AI Engine Direct SDK or, more sensibly, through the QNN execution provider that translates your ONNX graph to HMX automatically so you never write a line of it by hand.

At 135 GB/s, the LPDDR5x bandwidth is the real ceiling, not the compute array. A 7B model at INT4 is about 3.5 GB of weights, and dragging those weights from DRAM into the NPU’s on-chip SRAM on every forward pass is what costs you. There is an SRAM cache on die, undisclosed, probably 4 to 8 MB judging by area. Fit your model or your layer inside that SRAM and you flirt with the 45 TOPS ceiling. Miss it and you are hammering DRAM, bandwidth-bound at 135 GB/s, and the compute number is a fantasy. Hold onto that bandwidth wall, because the H20 is the exact same problem turned inside out.

Why shipping the Surface is a market story

Microsoft’s Copilot+ certification demands a minimum of 40 TOPS or you do not get the feature set. The X Elite clears it at 45. Intel’s Lunar Lake NPU does 48. AMD’s Strix Point does 50. Three silicon vendors converging on 40-plus TOPS inside the same twelve months is not the market discovering a natural equilibrium. It is Microsoft using a certification badge as a cattle prod, and it worked.

The 40 TOPS number itself is not arbitrary. It fell out of the requirement that Recall’s background indexing model, a 1 to 2B parameter vision encoder running at INT8, eats no more than about 15% of NPU capacity so the foreground tasks have room. Recall does semantic search over a rolling screenshot history. Cocreator in Paint runs a distilled diffusion model around 1B parameters for live image generation, Live Captions does Whisper plus translation entirely on-device, and Auto SR upscales game frames the way DLSS does except on the NPU instead of dedicated tensor cores. Every one of those carries a memory footprint and a throughput budget, and the floor was reverse-engineered so they do not trample each other.

The stack from feature to silicon is short, and its fragmentation is the whole developer problem:

Application Layer
    ↓
Windows Copilot Runtime (WCR)
    ↓
Windows ML API (WinML 2.0)
    ↓
ONNX Runtime (ORT), version 1.18+
    ↓
QNN Execution Provider (for Qualcomm)
DirectML Execution Provider (for Intel/AMD)
    ↓
Hardware Abstraction Layer
    ↓
Hexagon NPU (Qualcomm) / NPU tile (Intel) / XDNA2 (AMD)

ONNX Runtime is the universal joint. Microsoft bet the whole Copilot+ ecosystem on ONNX as the interchange format, so anything you can export from PyTorch, TensorFlow, or JAX can in principle ride the NPU through this path. In principle. The QNN provider for Qualcomm is mature and well-tuned. The DirectML provider for AMD is younger and has real performance holes on certain operator types, the kind of detail that never makes it into a keynote.

Qualcomm runs on a September clock. The 8 Gen 2 landed at Snapdragon Summit 2022, the 8 Gen 3 at 2023, the X Elite and the 8 Elite both at Summit 2024, and this year’s Summit is set for late September in Maui. Next-gen PC silicon arrives on that same cadence, and the analyst estimates I trust put the NPU target somewhere in the 60 to 80 TOPS range. The plausible mechanism is the node shrink, 4nm N4P moving to TSMC 3nm, either N3E or N3P, which buys roughly 15 to 20% perf-per-watt at equal area. Stack architectural HMX improvements on top and 60 to 80 TOPS is reachable without blowing out the thermal envelope. If you write NPU software, the thing to watch is the QNN SDK refresh that always ships alongside new silicon, the updated ONNX Runtime QNN builds, and any changes to AIMET, Qualcomm’s quantization toolkit, since that is where the practical breakage lives.

The H20: a chip cut down to fit the rules

The H20 is a Hopper part, the same GH100 die as the H100, deliberately cut down to slip under the export thresholds in force when NVIDIA designed it. They gutted compute by about 85% versus the H100 and then went the other way on memory, fitting 96 GB of HBM3 at 4.0 TB/s against the H100’s 80 GB at 3.35 TB/s. Read that and the design intent is naked. This chip is built for memory-bandwidth-bound inference, the regime where your bottleneck is shoveling 70B-parameter weights out of HBM and not the FLOPS doing the math. Same wall as the Snapdragon X Elite NPU, just with HBM3 at 4.0 TB/s instead of LPDDR5x at 135 GB/s. NVLink bandwidth stayed at 900 GB/s, identical to the H100. FP16 TFLOPS dropped from 989 to 148. INT8 TOPS dropped from 3,958 to 296. BIS left the memory subsystem completely untouched, which is exactly what made the April restriction feel like a surprise to everyone who had read the original spec sheet.

The restriction timeline

BIS restricts the H100 and A100 in October 2022. NVIDIA builds the compliant H20, and from late 2023 through March 2025 it ships to Alibaba, Tencent, ByteDance, and Baidu to the tune of an estimated $15 to $20 billion. Then on April 9, 2025, BIS turns around and restricts the H20 too, arguing the memory subsystem alone gives you enough capability once you network thousands of them. NVIDIA eats a $5.5 billion inventory charge on chips it can suddenly no longer sell. And in July 2025 the government reverses course and clears the H20 to go back on sale into China.

The technical argument for the April restriction was not dumb. It was a systems-level case, not a chip-level one. Wire enough H20s together over 900 GB/s NVLink and the aggregate memory bandwidth compensates for the compute deficit in bandwidth-bound training. Eight H20s give you 1,184 TFLOPS FP16, 768 GB of HBM, and 32 TB/s of within-node bandwidth. Eight H100s give you 7,912 TFLOPS, 640 GB, and 26.8 TB/s. The H20 node wins on memory capacity and bandwidth, then loses on compute by a factor of nearly seven. For training a big transformer, where the forward and backward passes are compute-bound, that cluster is a dog. The restriction only works as an argument if you accept that memory-bandwidth-bound training is the threat model that matters, and that is a contestable position dressed up as a settled one.

The reversal cannot undo the momentum it handed to Huawei. The Ascend 910B and 910C picked up real deployment share while the H20 was locked out, and a Chinese cloud provider that already rebuilt its infrastructure around Ascend is not ripping it out the week H20 procurement reopens. The whiplash is worse for forecasting than for any single quarter, because the whole restrict-reverse-restrict pattern teaches every Chinese buyer to over-procure the instant the door cracks open, which manufactures demand spikes nobody can model. NVIDIA recognizes the written-down inventory, the Q3 guidance bumps, and underneath all of it the market structure has already drifted somewhere policy cannot pull it back.

What the H20 actually does well

For anyone building on it, the H20’s pitch is inference on large models. The 96 GB of HBM3 holds more than the H100’s 80 GB, so a 70B model at FP16, around 140 GB, needs two H20S instead of two H100S, and the pair hands you better aggregate bandwidth for the serving workload. Two H20S do about 800 tokens per second at roughly 40% of the cost-per-token versus an H100 pair doing 1,200. Two A100 80GB cards land around 600 tokens per second at about 60% of the H100 cost. That spread is the entire reason Chinese clouds built inference fleets on a chip with a fraction of the compute. For serving, they were right.

The vLLM configuration that actually extracts this:

from vllm import LLM, SamplingParams

# H20-optimized configuration for 70B model serving
llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    tensor_parallel_size=2,          # 2x H20 for 70B model
    gpu_memory_utilization=0.92,     # H20 has 96GB HBM3; use 88GB
    max_model_len=8192,
    quantization="fp8",              # FP8 quantization for H20
    enforce_eager=False,             # Enable CUDA graph capture
    max_num_batched_tokens=32768,    # Large batch for throughput
    max_num_seqs=256,                # Max concurrent sequences
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

# The H20's 4.0 TB/s memory bandwidth is the key advantage here
# Weight loading per forward pass: ~140GB / 4.0 TB/s = ~35ms
# Compare to H100: ~140GB / 3.35 TB/s = ~42ms
# H20 is ~17% faster for memory-bandwidth-bound 70B inference
outputs = llm.generate(prompts, sampling_params)

That comment block is the whole H20 thesis in three lines of arithmetic. 140 GB of weights out of 4.0 TB/s is 35ms per forward pass against the H100’s 42ms, so on bandwidth-bound 70B serving the cut-down chip runs about 17% faster than the chip it was cut from. Which is exactly why BIS got nervous, and exactly why the nervousness was aimed at the wrong workload.

The two-backend tax

A real AI product in 2025 ships against two backends, not one, and maintaining both is the cost nobody puts in the pitch deck. The on-device path is ONNX, INT4 or INT8, through QNN or DirectML or CoreML, for the privacy-sensitive and latency-critical stuff under 7B. The cloud path is FP16 or FP8 through vLLM or TGI for anything bigger or any workload with real concurrency. Targeting the Hexagon NPU, the workflow that actually works looks like this:

# Step 1: Export PyTorch model to ONNX
import torch
import torch.onnx

model = load_your_model()  # e.g., Llama 3 8B
dummy_input = torch.randint(0, 32000, (1, 512))  # batch=1, seq_len=512

torch.onnx.export(
    model,
    dummy_input,
    "model_fp32.onnx",
    opset_version=17,
    input_names=["input_ids"],
    output_names=["logits"],
    dynamic_axes={"input_ids": {1: "sequence_length"}}
)

# Step 2: Quantize to INT8 using ONNX Runtime quantization tools
from onnxruntime.quantization import quantize_static, QuantType, QuantFormat
from onnxruntime.quantization.calibrate import CalibrationDataReader

class ModelCalibrationReader(CalibrationDataReader):
    def __init__(self, calibration_dataset):
        self.dataset = iter(calibration_dataset)

    def get_next(self):
        try:
            batch = next(self.dataset)
            return {"input_ids": batch["input_ids"].numpy()}
        except StopIteration:
            return None

calibration_reader = ModelCalibrationReader(calibration_data)

quantize_static(
    model_input="model_fp32.onnx",
    model_output="model_int8.onnx",
    calibration_data_reader=calibration_reader,
    quant_format=QuantFormat.QDQ,  # QDQ format required for QNN EP
    per_channel=True,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
)

# Step 3: Run inference with QNN Execution Provider
import onnxruntime as ort

session_options = ort.SessionOptions()
session_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

# QNN EP configuration for Snapdragon X Elite
qnn_options = {
    "backend_path": "QnnHtp.dll",  # HTP = Hexagon Tensor Processor
    "htp_performance_mode": "burst",  # Options: burst, balanced, low_power
    "htp_graph_finalization_optimization_mode": "3",  # Highest optimization
    "enable_htp_fp16_precision": "0",  # Use INT8, not FP16
}

session = ort.InferenceSession(
    "model_int8.onnx",
    sess_options=session_options,
    providers=[("QNNExecutionProvider", qnn_options), "CPUExecutionProvider"]
)

# Run inference
input_ids = np.array([[1, 2, 3, 4, 5]], dtype=np.int64)
outputs = session.run(None, {"input_ids": input_ids})

That QDQ format note matters more than it looks. Get the quant format wrong and the QNN provider silently falls back to CPU, and you spend an afternoon wondering why your NPU is doing nothing.

For LLMs specifically, llama.cpp has eaten the edge. As of July 2025 it carries a Qualcomm QNN backend, and the --n-gpu-layers flag you know from CUDA gets repurposed to offload transformer layers onto the Hexagon NPU:

# Build llama.cpp with QNN support
cmake -B build -DGGML_QNN=ON -DQNN_SDK_PATH=/path/to/qnn-sdk
cmake --build build --config Release

# Run inference with NPU offloading
./build/bin/llama-cli --model ./models/llama-3-8b-instruct-q4_k_m.gguf --n-gpu-layers 32 --ctx-size 4096 --threads 4 --prompt "Explain the Snapdragon X Elite NPU architecture"

For a 7B with 32 layers you offload all 32 and walk away. For a 13B with 40 layers you offload 20 to 30 and leave the rest on CPU, and the exact split is governed by that same SRAM-versus-bandwidth tension, not by some clean rule. The optimal offload number is something you find by trying it. The marketing slide skips that part.

Where the model-size line sits

Up to 7B at INT4, about 3.5 GB, the NPU is viable and interactive at 20 to 30 tokens per second. Push to 13B, and you are still on the NPU, but the bandwidth wall starts biting: 10 to 20 tokens per second, and it feels like wading through mud. Past 13B the current-gen NPU just is not the tool, and at 70B and up you are on a GPU, plural, with no argument left to have.

That line is not fixed. Every NPU generation drags it upward, and if the next Snapdragon really delivers 60 to 80 TOPS with a fatter on-chip SRAM, 13B at interactive speed becomes plausible and the set of features you can run with no cloud at all gets meaningfully bigger. I am not going to get into the question of whether on-device models ever close the quality gap with frontier cloud models at the same parameter count, because that is a separate fight and the answer changes every six months anyway.

The split between edge and data center is structural, not a phase we are passing through on the way to some grand convergence. The Hexagon NPU is fixed-function tensor acceleration, INT4 and INT8 as first-class citizens, sipping 5 to 8W during active inference, built to serve exactly one user at a time as fast as possible. The H20 is a programmable shader array with Tensor Cores bolted on, 500W, discrete HBM3, built to batch hundreds of concurrent requests at once. One optimizes for latency, the other for throughput. Type into a chat box and latency is the only number you feel. Pay an API bill and throughput is the only number that matters.

If you are building on H20 capacity, model the next restriction cycle into your planning today and keep real fallback room on H100 or domestic silicon, because rented capacity comes with a kill switch and the April-to-July whiplash proved who holds it. I would rather bet on the chip than on the room. The chip at least has to obey physics.