We’re a little over a week away from the WWDC keynote, in which Apple is expected to announce a slew of new AI features as part of iOS 18.
This is expected to be the point at which Siri gets generative AI smarts, thanks to a deal with OpenAI to incorporate ChatGPT tech …
2024: The year of generative AI features
We’ve noted before that it’s not fair to act like Apple is an AI newbie. The company has made extensive use of machine learning in its Camera app in particular, with computational photography features used for everything from exposure bracketing to artificial shallow depth of field effects.
We’ve also observed that while Siri may have been left behind by other intelligent assistants, that’s partly an unfair comparison, and partly because Apple has given greater priority to protecting the privacy of user data (albeit with some fails).
But at the same time, there’s no doubt that generative AI – flawed as it may be – has completely changed expectations around what an intelligent assistant ought to be able to do. In particular, we now expect way more conversational interactions, and a far greater understanding of context and intent.
That’s why Apple has decided to make AI features a core focus in this year’s software updates – but what exactly are we expecting?
A way smarter Siri
Apple’s top AI priority has to be making Siri much smarter than its often stilted replies, failure to understand what we want it to do, and the dreaded “I found this on the web” response.
The company is believed to be essentially rebuilding Siri to be able to be powered by large language models (LLM). It’s likely that this will be some combination of Apple’s own LLM, and OpenAI’s ChatGPT tech.
However, the enormous size of this task means that the full rollout isn’t expected until sometime next year. It’s likely that Apple will release an interim update in the initial release of iOS 18.
Specifics we’re expecting to see include:
- Smarter understanding of what you want Siri to do
- Spoken answers to questions instead of showing web links (with a ChatGPT disclaimer)
- Awareness of your contacts, appointments, locations, and so on, for smarter responses
- Ability to summarise everything from notifications and messages to lengthy documents
- Ability to interact with your apps – eg. “Move this note to my New Home folder”
- String multiple commands together – eg. “Crop this photo to a square and send it to Sam”
Stringing commands together, which is expected to arrive in a later update, would address one of the biggest frustrations with Siri’s control of HomeKit devices. You cannot currently say something like “Hey Siri, switch off the living room lights and run the Garage Work scene.”
- iOS 18: Siri to gain extensive knowledge to trigger individual app functions, powered by AI
- iOS 18 Siri will better understand people, events, locations – report
- iOS 18 rumored to include Humane Ai Pin-style iMessage summary feature using AI
- iOS 18 Siri: Three clues about what it may be able to do
AI photo editing capabilities
Reports on this have been fairly general, but there have been some specific pointers – like AI-powered object removal, to get rid of unwanted people and objects in photos.
A new Clean Up feature, which will allow users to remove objects from images through the use of generative AI software.
We also know that the company has been working on text-based photo editing, where you simply tell the device what it is you want to achieve.
For example, a user can simply say “make the sky more blue” and MGIE will interpret that as “identify and select the sky, then increase the saturation in this area by 20%.”
We don’t know how many of these capabilities will be incorporated into iOS 18, but the model outlined by Apple is capable of:
- Cropping, resizing, rotating, flipping
- Adding filters
- Identifying and editing specific photo elements, like eyes, hair, and clothes
While the paper (and demo) use typed text, the eventual implementation would seem more likely to be a Siri command; “Hey Siri, remove the post behind the woman on the right.”
Again, full rollout of these capabilities is likely to take time.
- Apple AI work continues: Editing photos using text commands
- Apple Keyframer generates AI animation from a still image and text prompt
- Apple researchers reveal new AI breakthrough for training LLMs on images and text
Custom emoji
Stories on new emoji coming to iOS are always popular here, suggesting that many iPhone owners want to be able to use the graphic elements to express a wider range of emotions in messages.
A Bloomberg report says that Apple will use generative AI to allow users to create custom emoji.
One standout feature will bring generative AI to emojis. The company is developing software that can create custom emojis on the fly, based on what users are texting.
This may be something you ask for, or it could be that iOS 18 will automatically suggest a custom emoji when you hit the button to bring up the emoji menu.
Other possibilities
There have been sufficient Siri reports to make a major revamp here a sure bet; the photo editing capabilities seem extremely likely; and Gurman is confident about custom emoji. But there have also been additional reports of things Apple is working on, and which we might see introduced in iOS 18 and macOS 15.
Smarter Spotlight searches
One report says that Apple is “considering” using AI to enable smarter Spotlight searches. In particular, it says that it would be able to dig deep into apps. This might, for example, enable searches like these:
“Show me all the notes and drawings I’ve made about home improvements”
“Show me all my cheese recipes”
AI-assisted coding in Xcode
ChatGPT is widely used to generate code, with Microsoft’s GitHub Copilot also helping developers complete their code. One report says Apple is testing a similar Xcode capability in-house, with a view to launching it publicly once it is working well.
AI-powered health & fitness coaching
A somewhat vaguer report suggests that Apple is working on an AI-powered health & fitness coaching feature.
The goal is to “keep users motivated to exercise, improve eating habits and help them sleep better.” The platform will use a combination of artificial intelligence and Apple Watch data to give users suggestions on lifestyle changes that will help improve their overall health and wellness.
AI-enhanced journaling
Ahead of the launch of the Apple’s Journal app, Bloomberg also suggested that AI features might be used to analyse the wording of journal entries to detect the user’s mood. We discussed yesterday how the company could use AI features in a similar way to the Autobiographer app to super-charge Journal.
What are you most hoping for?
What AI-powered features are you most hoping to see in iOS 18 and macOS 15?
Please share your thoughts in the comments.
Photo by randa marzouk on Unsplash
FTC: We use income earning auto affiliate links. More.
Comments