For years, on-device AI was a bullet point on a phone spec sheet, the NPU nobody really used. Personally, I think that era just ended. Somewhere between the laptops, the robots, and the phones launched this year, the question quietly flipped. It is no longer about which AI model you use. It is where that model actually runs. And that turns out to be a business decision, not a feature.
What actually changed
The NPU stopped being a marketing number. Look at the hardware that shipped or got announced in the last few months. Laptops with neural engines pushing 80 TOPS. Robot brains pushing 700. Even $300 Windows laptops now ship with real on-device inference.
Meanwhile, the frontier-model race has hit the point where most people cannot tell you what the latest half-version-bump actually does for them. The models are good enough. The interesting fight moved underneath them.
So the new question, the one I keep seeing show up in serious strategy conversations, is brutally simple: where does the inference physically happen? On the chip in your hand, or on a server you rent by the token?
Why that is a business decision, not a tech one
If your product sends every single action to the cloud, you are signing up for four things at once. You pay per call, forever. You add latency on every request. You hand sensitive data to someone else’s servers, which your compliance people may hate. And you make your whole product dependent on a vendor’s pricing and uptime. Run that same work locally and all four of those flip in your favor.
The honest answer for most teams is not all-or-nothing, it is hybrid. Keep the fast, sensitive, high-frequency stuff on the device. Send only the heavy reasoning out to a big cloud model when you need it. That is the architecture, and notice that it is an architecture choice, made by someone thinking about cost and risk, not a checkbox a product manager ticks.
Where on-device AI is already the right call
This is not theoretical. The use cases that already make sense locally are the narrow, repeatable ones: transcription, OCR and document scanning, biometric checks, recommendations, and assistance inside vehicles and industrial machines. Notice the pattern. High frequency, privacy-sensitive, latency-allergic. Exactly the workloads you do not want to round-trip to a server. In Europe especially, where privacy expectations shape what you can even sell, keeping data on the device is becoming a feature you lead with, not a constraint you apologize for.
The strategic shift nobody is pricing in yet
Here is the part I find most interesting. For a couple of years, buying AI meant choosing a model. In 2026 it increasingly means choosing a platform: a chip, a runtime, a control plane, and a security posture. Those are stickier, multi-year decisions, and they are a lot harder to walk back than swapping an API key.
The unlock is the workloads that compliance teams have flat-out refused to put in the cloud. Run an agent on-device and suddenly that conversation changes, because the data never leaves. I think where the inference happens becomes a board-level question within a year, the same way where does our data live did a decade ago. Even Nvidia gets it from the other side, busy shrinking the data center into a desktop.
The part the hype skips
Let me not oversell it, because I have been guilty of that before. On-device AI is a tradeoff, not a miracle. Local models drain battery and throw off heat. Device fragmentation is a nightmare, because the experience on a flagship is not the experience on a three-year-old budget phone. And a lot of installed hardware simply is not strong enough to run anything useful yet. Anyone promising you fully local everything by next quarter is selling something.
I have been pushing the AIoT and on-device argument for years, so watching it become the main conversation instead of the footnote is satisfying. But the lesson is not that on-device wins. The lesson is that the winners will not be whoever has the slightly better model. They will be whoever makes the smartest call about where the work runs, and builds the hybrid plumbing to back it up. That is less sexy than a benchmark chart. It is also where the actual money and the actual risk now live.