LW - AiPhone by Zvi

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AiPhone, published by Zvi on June 13, 2024 on LessWrong.
Apple was for a while rumored to be planning launch for iPhone of AI assisted emails, texts, summaries and so on including via Siri, to be announced at WWDC 24.
It's happening. Apple's keynote announced the anticipated partnership with OpenAI.
The bottom line is that this is Siri as the AI assistant with full access to everything on your phone, with relatively strong privacy protections. Mostly it is done on device, the rest via 'private cloud compute.' The catch is that when they need the best they call out for OpenAI, but they do check with you first explicitly each time, OpenAI promises not to retain data and they hide your queries, unless you choose to link up your account.
If the new AI is good enough and safe enough then this is pretty great. If Google doesn't get its act together reasonably soon to deliver on its I/O day promises, and Apple does deliver, this will become a major differentiator.
AiPhone
They call it Apple Intelligence, after first calling it Personal Intelligence.
The pitch: Powerful, intuitive, integrated, personal, private, for iPhone, iPad and Mac.
The closing pitch: AI for the rest of us.
It will get data and act across apps. It will understand personal context. It is fully multimodal. The focus is making everything seamless, simple, easy.
They give you examples:
1. Your iPhone can prioritize your notifications to prevent distractions, so you don't miss something important. Does that mean you will be able to teach it what counts as important? How will you do that and how reliable will that be? Or will you be asked to trust the AI? The good version here seems great, the bad version would only create paranoia of missing out.
2. Their second example is a writing aid, for summaries or reviews or to help you write. Pretty standard. Question is how much it will benefit from context and how good it is. I essentially never use AI writing tools aside from the short reply generators, because it is faster for me to write than to figure out how to get the AI to write. But even for me, if the interface is talk to your phone to have it properly format and compose an email, the quality bar goes way down.
3. Images to make interactions more fun. Create images of your contacts, the AI will know what they look like. Wait, what? The examples have to be sketches, animations or cartoons, so presumably they think they are safe from true deepfakes unless someone uses an outside app. Those styles might be all you get? The process does seem quick and easy to generate images in general and adjust to get it to do what you want, which is nice.
Resolution and quality seems fine for texting, might be pretty lousy if you want better. Image wand, which can work off an existing image, might be more promising, but resolution still seems low.
4. The big game. Take actions across apps. Can access your photos, your emails, your podcasts, presumably your everything. Analyze the data across all your apps. Their example is using maps plus information from multiple places to see if you can make it from one thing to the next in time.
Privacy
Then at 1:11:40 they ask the big question. What about privacy? They say this all has 'powerful privacy.' The core idea is on-device processing. They claim this is 'only possible due to years of planning and investing in advanced silicon for on device intelligence.' The A17 and M1-4 can provide the compute for the language and diffusion models, which they specialized for this. An on-device semantic index assists with this.
What about when you need more compute than that? Servers can misuse your data, they warn, and you wouldn't know. So they propose Private Cloud Compute. It runs on servers using Apple Silicon, use Swift for security (ha!) and are secure. If necessary, only the necessary d...