A 35B AI Model on a Phone, With an Asterisk

Edge0 released a demo this week showing a 35-billion-parameter model running on an iPhone with a reported peak active-memory footprint of 2.9 GiB. That headline is accurate and incomplete. The full checkpoint still occupies about 23 GB of local storage, the memory figure holds at short context lengths, and it decodes at 15 tokens per second. Those three numbers change what the result means for a business.

What Edge0 published

The release is an open-source streaming inference framework under Apache 2.0 on GitHub, with two checkpoints on Hugging Face: a 35B model derived from Qwen3.5-MoE and an 8B based on Ling 3.0. Edge0 reports peak active memory of about 2.9 GiB for the 35B and about 1.0 GiB for the 8B, both at short contexts, with decode running at 15 tokens per second and benchmark scores averaging 3.9 points below the fp16 base.

These are the project's own reported figures from its release rather than independently reproduced benchmarks, which is worth holding in mind before quoting them to anyone.

How it works, in one paragraph

A mixture-of-experts model activates only a fraction of its parameters for any given token. Edge0 takes advantage of that by memory-mapping the expert weights on local storage and fetching them on demand, so the working set tracks the experts actually selected rather than the model's total size. Frequently used experts stay cached in memory, the rest sit on disk until requested. A small trained prerouter head predicts which experts the next token will need, letting the runtime start reading those weights while it finishes the current one, which Edge0 reports as up to 59% higher decode throughput.

That is a genuinely clever piece of engineering. It is also why the 23 GB figure is not a footnote: the weights have to be somewhere, and the technique moves them from RAM to storage rather than making them disappear.

Why phone-class matters differently from server-class

Running AI on your own hardware has been viable for a while, and we have covered it several times: local LLMs for Canadian business, Ollama going mainstream, and offline AI. All of those assume a machine: a workstation, a server in a cupboard, something with a power cable.

A phone changes the question from whether you can host a model to where the model can be. Basements with no signal. Rural sites. A van between jobs. An inspection in a building with metre-thick walls. Hardware your staff already carry and already charge.

For businesses whose work happens in those places, the comparison is not this model against a cloud API. It is this model against having nothing, which is a very different calculation and makes 15 tokens per second look considerably better.

The privacy angle is the underrated one

A model running entirely on a device sends nothing anywhere. No vendor terms about training on your inputs, no retention question, no data processing agreement, no region to specify. For work covered by a client confidentiality clause or handled under a strict residency requirement, that removes an entire category of problem rather than mitigating it.

It creates a different one, which is that a device holding a model and its cached context is a device that can be lost. Device management, encryption and remote wipe become the controls that matter, and the assessment questions shift accordingly. Our note on privacy impact assessments for AI covers the shape of that, and AI data residency in Canada covers what on-device does to the residency question.

What not to conclude

This is a research release, not a product. Nobody supports it, nobody is liable for it, and it will not appear in your mobile device management console. The figures come from the people who built it. Battery drain and thermal behaviour under sustained inference are real questions that a demo does not answer, and 23 GB per device is a procurement conversation on a fleet of phones.

The direction is clear enough to plan around even if the release is not. Capability that needed a data centre two years ago and a workstation last year now fits, awkwardly, in a pocket. The useful move this quarter is not a deployment. It is writing down which of your tasks genuinely need to work without a connection, so that when a supported product appears you can evaluate it in a week rather than starting the analysis then. That list is also worth having for its own sake, because most businesses discover it is shorter than they assumed. The related decision about hosting on your own hardware is in ChatGPT versus local models.

Frequently Asked Questions