On July 7, 2026, I stood at the front of the My Porsche App Tech Day in Ludwigsburg with a microphone in one hand and a fairly opinionated claim on every screen behind me: not every on-device AI is Apple Intelligence. Jonathan, a friend, had helped get me into the room. Once I was there, the ninety minutes were mine.
I was nervous in a very specific way. I was a good bit younger than most of the people looking back at me, and many of them had been shipping software longer than I had been studying it. I was not there to give a harmless overview, either. I wanted to argue that Apple's Foundation Models are only one option on Apple platforms, and that open-weight models running through MLX can be the more practical foundation for a real product.
That changes what preparation means. A polished slide is not enough when the room can interrogate the assumption underneath it. Every number needed a measurement behind it, every comparison needed a limitation, and every confident sentence needed to survive the next question. The experience taught me that being the least experienced person in a room is manageable. Being careless with the evidence is not.
A Claim With a Target on It
On Apple platforms, the phrase on-device AI has started to collapse into Apple Intelligence. That is understandable. Apple owns the operating system, ships a system model, and gives developers a native Foundation Models framework with guided generation and tool calling. It is the obvious option to evaluate first.
It is not the option I built NeatPass on. NeatPass turns a screenshot, PDF, or photo of a ticket into an Apple Wallet pass without uploading the source file. A small Qwen model ships inside the app and runs through mlx-swift on the phone. The model handles the fuzzy reading; deterministic code handles the barcode, validation, signing, and pass assembly. That architecture is already used by more than 37,500 people and has created more than 400,000 passes.
So the talk was not a speculative pitch for someday. I had a product in my pocket whose on-device model works on iPhones going back to the iPhone 12, including devices that cannot run Apple Intelligence. The download is larger because the weights ship with the app, but the model is versioned with the product, works in every supported market, and cannot be replaced by an operating-system update overnight.
The strongest version of that argument still had to preserve Apple's advantages. The system model adds zero bytes to an app bundle. Its structured-output API is excellent. Its tool protocol is far less plumbing than building the same loop yourself. If I presented the comparison as Apple losing six rounds, someone in the room would rightly dismantle it. The real question was which constraints mattered for a specific product, not which logo won.
Six Rounds, One Machine
I put Apple's Foundation Model head to head with Qwen and SmolLM models through MLX across six rounds: latency and throughput, structured output, tool calling, context, footprint, and failure modes. The measurements came from one Apple M2 Pro. For the latency run I used the same long-form prompt, five runs per model, and the median. Apple's framework does not expose token counts, so I labelled its throughput as an estimate from characters rather than pretending the APIs offered identical data.
Three conclusions carried most of the argument:
- The start and the stream. On my M2 Pro, SmolLM2 1.7B produced its first token after 218 ms; Apple's model took 447 ms. The open models then streamed at 106 to 110 tokens per second, while Apple's throughput was an estimate of roughly 65. I put the machine, prompt, run count, and estimation method on the slide so the numbers could be challenged as the measurements they were.
- Reach and control. A one-to-two-billion-parameter model can run on far more of the phones people already carry. Shipping the weights also gives the app a model version it can test and keep stable.
- Shape is not truth. Apple can guarantee a generated Swift structure, and open models can grammar-lock JSON. Neither guarantee says that the venue, seat, or tool argument is factually correct. Both still need validation and guardrails.
iOS 27 handed me a better version of my own argument before I reached Porsche. Apple had announced AFM 3, an 8,192-token context window, and a sparse 20B model for its newest hardware. At the time of the talk, those models could not be independently benchmarked. AFM 3 was not shipping on iPhone or iPad in the EU, although the Mac was unaffected, and the 20B model required an M3-class chip with 12 GB of memory or an iPhone 17 Pro. Apple also opened the Foundation Models framework through a LanguageModel protocol, including an open-source adapter for MLX models. Apple itself was making room for on-device AI that was not Apple Intelligence. The other product constraints remained: Apple added no bundle size, while open models offered 32,000 to 262,000 tokens of context, exposed every tool hop, and could be pinned across older devices and markets.
Most importantly, the benchmark did not stop at the slide. I built a public SwiftUI lab that ran the same jobs both ways, then measured them live in the room. People could watch the first-token clock, inspect the structured output, and run the models on their own Macs. The full methodology and all six rounds are in the deck. The article-sized version is simpler: if a claim can be reproduced in front of the audience, it becomes a much better starting point for disagreement.
The Part You Cannot Rehearse
I had explicitly told people to interrupt. Questions in the middle beat questions politely stored until the final slide, especially in a ninety-minute technical session. That invitation also removed my safest option. I could not rush through a fragile point and hope nobody returned to it.
The questions focused on tolerance inside the product. How many inaccurate answers would users accept before they stopped trusting the feature? How should a response be validated? Where could deterministic heuristics remove work from the model altogether? We talked about grounding fields in source data, rejecting unsupported values, retrying with the validation error, and reserving the model for the genuinely fuzzy parts. Once an output can trigger a vehicle action, those checks stop being implementation polish.
After that, I drew a much tighter boundary around the model. In NeatPass it is one fallible component: deterministic code decodes the barcode, heuristics handle labelled fields, the model reads the ambiguous fields, and another deterministic layer validates the result before building the pass. The pipeline assumes the model will sometimes be wrong.
The benchmark measured speed. The product discussion defined the conditions under which an output was worth accepting: source-grounded fields, deterministic checks, controlled retries, and no action on unsupported data. Those conditions matter more than squeezing another few tokens per second out of the model.
The Risk Missing From My Slides
One of my core arguments for open weights was ownership of the failure mode. A small model will invent things. NeatPass has seen a North Macedonia concert become Coldplay in London and an Olympiahalle ticket become the Kleine Olympiahalle, even though that extra word never appeared in the source. Those errors are bad, but they are visible to my code. I can check whether a generated value appears in the ticket, reject it, retry it, and pin the model while I improve the guardrail.
The system model introduces failures an app controls less directly: a guardrail can refuse a harmless prompt, availability can depend on hardware or region, and the model can change with the OS. My line in the talk was to choose the failure your UX can actually catch. It is still a useful line, but the discussion exposed a cost I had not put beside it.
If the weights ship in the app bundle, the weights ship to the user. They can be extracted and copied. Version ownership is also distribution. For a public base model such as Qwen, that is expected. For a proprietary or heavily fine-tuned model whose weights are part of the company's defensible IP, it can change the architecture completely.
This is not an abstract licensing footnote. A team can spend substantial time and data making a small model unusually good at one task, then package the result into the easiest possible delivery mechanism for someone who wants to inspect it. Encryption can raise the effort, but the app eventually needs the usable weights on hardware the user controls. A secret that must be decrypted on somebody else's device is a difficult secret to keep.
That does not overturn the NeatPass architecture. Its base weights are already open, and most of the product-specific value sits in the deterministic pipeline, prompts, validation, retries, and Wallet integration around them. It does qualify the broader argument. Carrying your own model gives you control over reach, versions, and guardrails, but it also means carrying the model's redistribution and IP risk. The room made the thesis better by finding the invoice attached to the benefit.
Being the Least Experienced Person in the Room
Before the talk, I treated the experience gap mostly as a reason to be nervous. That was not entirely irrational. Experienced engineers can spot when a benchmark hides an unfair setup, when a neat architecture ignores deployment, or when a confident speaker is borrowing certainty from a graph. The solution was not to imitate their experience. I could not manufacture years I had not worked.
What I could do was make my narrower experience unusually inspectable. I had shipped this architecture. I had the harness. I knew which number was exact, which was estimated, which result applied only to my M2 Pro, and where Apple's option was genuinely better. That did not make me the authority on every vehicle or AI system in the room. It made me responsible for the part I had chosen to claim.
Once the questions started, the age difference became less important than I had expected. The conversation worked because the room did not need a performance of seniority from me. It needed a concrete system, measurements it could challenge, and honest boundaries around what those measurements meant. In return, I got perspectives that no amount of rehearsing alone would have produced.
That is probably the most useful speaking lesson I took home. Confidence does not have to mean sounding certain about everything. In a technical room, it can mean being precise about the thing you know, calm about the thing you do not, and genuinely interested when someone finds a risk your preparation missed.
What Survived the Room
The main conclusion survived, with sharper edges. Apple's Foundation Models are a strong option when zero bundle footprint, native APIs, and system integration dominate the decision. Open weights through MLX are a strong foundation when device reach, long context, version stability, auditability, or task-specific model choice matter more. Often the useful architecture is not ideological at all: own weights as the reliable floor, the system model as a capable tier where it is available.
The new part is the question I would now add before recommending that floor: what happens when somebody extracts it? If the answer threatens the business, the model cannot be treated like an ordinary bundled dependency. That constraint belongs beside latency, context, footprint, and hardware reach, not in a security review after the architecture is already fixed.
I walked into Porsche worried that a room with more experience would expose the limits of my argument. It did, just not in the way I feared. The measurements held up. The discussion found what the measurements did not cover. That is a much better outcome than giving a flawless talk that leaves the thesis exactly as it entered.
The complete benchmark, sources, and six-round comparison are in my slides. The runnable On-Device Lab contains the SwiftUI demos and the headless harness, so you can challenge the numbers on your own Apple Silicon instead of taking mine on faith. That is the standard the room pushed me toward, and the one worth keeping.
