Forty tokens a second, fully offline, airplane mode on, not a single packet leaving the phone. That’s Qwen3 1.7B running on a Galaxy S25 Ultra through MLC Chat, and it’s the closest thing to pocket magic I’ve played with this year. It’s also, I’m sorry to report, the exception that exposes the rule. Because the moment you step off that one specific path, the fast silicon Qualcomm keeps bragging about quietly checks out, and your shiny on-device model starts grinding away on the CPU like it’s 2019.
That gap is the whole story of running a local LLM on Android in 2026, and almost nobody tells you about it before you’ve already sideloaded three apps and watched your battery drop 50% in ninety minutes. So let me lay out what actually works, what’s quietly faking it, which models are worth your storage, and the de-Googled angle that makes the privacy real, with the one catch that comes attached.
The apps you’d actually install
The field has settled into a handful of real options, and they split cleanly by what you care about. If you want the absolute fastest inference and you’re on a Snapdragon 8 Elite phone, it’s MLC Chat, full stop. It’s the one app I can confirm actually lights up the Hexagon NPU, and it gets there by compiling each model specifically for your chip rather than handing a generic file to the processor. That’s why it hits roughly 22 tokens a second on Phi-4 Mini where the CPU apps sit around 16, and why it stretches past 40 on a small model like Qwen3 1.7B. The price you pay is steep in its own way: you’re stuck with a curated, pre-compiled model list, you can’t drop in an arbitrary file from Hugging Face, and you have to sideload the APK because it isn’t on the Play Store.
PocketPal AI is the one I’d hand to most people. It’s open source, built on llama.cpp under a React Native skin, and it lets you search the entire Hugging Face GGUF catalog from inside the app, pick your quantization, and download without ever leaving it. It sends nothing home by default, which matters more than it sounds. The tradeoff is that it runs on CPU or the GPU via Vulkan, never the NPU, so it’s slower than MLC on the same phone and the interface can stutter on cheaper hardware.
Past those two it’s about temperament. Ollama through Termux is the power-user route: the full Ollama ecosystem, an OpenAI-compatible API, tool use, anything in the library your RAM can hold, all behind a terminal you have to set up yourself. Maid is the F-Droid purist’s pick, fully offline with direct GGUF import and no Play Store anywhere near it. Layla smooths the on-ramp for beginners with a curated model list and proper share-sheet support. And Google’s own AI Edge Gallery is worth a look as the official demo of what mobile-tuned silicon can do, except it only runs Gemma, so the second you want Llama or Qwen or a fine-tune you’ve outgrown it.
| App | Engine | NPU | Model freedom | Phi-4 Mini, S25 Ultra |
|---|---|---|---|---|
| MLC Chat | TVM compile | Yes (Hexagon) | Pre-compiled list only | ~22 tok/s |
| PocketPal AI | llama.cpp | No (CPU/Vulkan) | Any GGUF | ~16 tok/s |
| Maid | llama.cpp | No (Vulkan) | Any GGUF | ~18 tok/s |
| Layla | llama.cpp | No (CPU) | Curated | ~14 tok/s |
| Ollama (Termux) | llama.cpp | No (CPU) | Full Ollama library | ~10 tok/s |
The NPU wall
Look at that table again and the uncomfortable truth jumps out. Every app on it except MLC runs your model on the CPU or the GPU, and the one component in your phone purpose-built for exactly this work, the NPU, sits idle. This is the thing that took me a while to accept: in 2026 “on-device” almost never means “on the neural engine.”
The reason is structural. The open model ecosystem speaks GGUF, the format llama.cpp popularized, and GGUF was designed to run anywhere, which in practice means the CPU with maybe a GPU assist. NPUs don’t work like that. The Hexagon block in a Snapdragon, the TPU in a Tensor, the APU in a MediaTek chip, each wants a model compiled and quantized for its own instruction set, fed through a vendor toolchain. MLC pulls it off by doing that compile step ahead of time for a fixed set of models, which is exactly why its library is locked. Google’s official answer, LiteRT-LM, can absolutely target the Qualcomm, Tensor, and MediaTek NPUs through vendor dispatch libraries, but doing it means building binaries with the Android NDK, pushing a .litertlm file and the QAIRT runtime to /data/local/tmp/ over adb, and checking whether your exact SoC is even on the support list yet. That’s a developer workflow, not a Tuesday-night app install.
And the one piece of on-device AI that does ride the NPU cleanly, Gemini Nano, isn’t your model at all. It’s Google’s, roughly a gigabyte, delivered and managed through AICore, doing the smart-reply and summarize work behind the system. You don’t load it, you don’t swap it, and as we’ll get to in a second, you can lose it entirely. So the state of play is blunt: open models you control run on the slow path, and the fast path runs closed models you don’t. I keep coming back to where Qualcomm’s AI actually lives on its silicon, and the answer on a phone is that the good part is mostly fenced off from you.
What to actually run
Model choice is where you claw performance back, because the right small model on the CPU beats the wrong big one on any silicon. The mobile-first standouts right now are Google’s Gemma family, with Gemma 4 and the distilled, multimodal Gemma 3n built specifically to fit phone RAM, plus Qwen3 in its 1.7B trim, Llama 3.2 at 1B or 3B, and Phi-4 Mini, which punches above its size on reasoning. Quantization is the lever that matters most: Q4_K_M is the sweet spot almost everyone lands on, small enough to load and good enough to trust, and I’d only step up to a heavier quant when I needed a model to be careful, like anything medical or legal where a 4-bit version starts hallucinating specifics.
The hardware floor is real. You want a modern 64-bit ARM chip and at least 6 to 8 GB of RAM to run anything comfortably, and 12 GB if you want a 3B-class model to breathe. Expect roughly 30 to 40 tokens a second on a sub-1B model, dropping toward 10 to 20 on a 3B, and remember those numbers are best-case before the chip heats up. Sustained inference is one of the heaviest things you can ask a phone to do, the back gets warm, and the moment thermal throttling kicks in your tokens-per-second falls off a cliff.
The de-Googled catch
Here’s where the privacy story gets real and complicated at once. If your goal is total privacy, the logical move is a de-Googled OS, and in 2026 that means GrapheneOS as the hardened gold standard on Pixel hardware, CalyxOS as the friendlier middle path with microG and broader device support, LineageOS for sheer device coverage at the cost of weaker hardening, or /e/OS on a Fairphone if repairability is your thing. GrapheneOS even ported to Android 17 the same day Google shipped it this June, and it’s branching past Pixels through a Motorola partnership aimed at 2027. Owning your stack at the OS level is the same instinct as running your own DNS, just pushed all the way down to the phone.
The catch is the one nobody flags up front: de-Googling kills Gemini Nano. AICore is a Google Play Services component, so a phone without Google services has no Nano, no system-level on-device AI, none of the smart-reply magic baked into stock. That sounds like a loss until you realize it’s actually the cleanest argument for the independent apps. MLC, PocketPal, Ollama, none of them need a shred of Google code to function, so they run perfectly on GrapheneOS or CalyxOS. De-Googling doesn’t take local AI away from you, it just takes away Google’s version and forces you onto the one you actually control. The genuine cost lives elsewhere: a de-Googled phone fails Play Integrity, which is what trips up banking apps and the age-verification systems that have gotten aggressive this year, and the sideloading you need for an app like MLC keeps bumping into Google’s tightening developer-verification regime. F-Droid and the Aurora Store are the usual escape hatches there.
When it breaks, and it will
The failures are predictable, so here’s the short triage. If a model won’t load or the app dies on launch, you’re out of RAM, drop to a smaller model or a lighter quant before anything else. If generation is painfully slow, you’ve got two moves: shrink the model, or get onto MLC so the NPU does the work instead of the CPU. If the phone turns into a hand-warmer and output crawls mid-answer, that’s thermal throttling, so flip on the app’s power-saving mode, kill background apps to free memory, and keep sessions short rather than running one marathon generation. If you specifically want the NPU and your app isn’t using it, accept that almost none of them will and switch to MLC, because wishing won’t change the architecture. And if an APK won’t install, that’s the sideloading clampdown, so pull it from F-Droid or Aurora instead of chasing a random mirror.
I’m not getting into on-device image generation here, the Stable-Diffusion-on-Snapdragon side that an app like Off Grid pulls off, because that’s a different beast with its own thermal nightmares and deserves its own post.
So is it worth it? For privacy, unequivocally yes. Nothing matches an LLM that runs in airplane mode with zero data leakage, no account, no subscription, no rate limit, and on an open-source app you can actually audit. That’s the whole open-weights argument made physical, sitting in your hand instead of someone’s datacenter. What you give up is real and worth saying plainly: a 1B-to-7B model is not going to reason like a frontier model, the battery cost is brutal, and the NPU that should make all of this effortless is mostly locked behind vendor toolchains and closed models. I run this stuff anyway, because the privacy is the point and the speed is good enough for the quick, sensitive, offline things I actually want it for. But anyone selling you on-device AI as the effortless future is skipping the part where the best silicon in your phone is still, in mid-2026, mostly off-limits to the models you’d choose yourself.