Concept imagines how Siri could use AI to read and interact with what’s on the screen

Apple is rumored to announce many new AI features for its devices next week at WWDC 2024, and some of the rumors say that the company will use generative AI to make Siri better. With this in mind, designer João Bortolotti created a concept imagining what it would be like if Siri could read and interact with what’s on the screen.

Concept combines Siri with the power of AI

The Siri AI concept created by Bortolotti is quite simple, but it shows us the endless possibilities that users would have if Siri gets updated with such capabilities.

“I believe that Apple can use the power of LLMs and Shortcuts Actions to create live automations and give complex responses in a transparent way, protecting itself from the hallucinations of models,” the designer said.

In one of the examples he showed, the user could ask Siri to “sum up the prices” shown in an app or website, and then Siri would read the content to identify the prices and show the result. Another scenario imagined by the concept is asking Siri AI to summarize an open document.

But can such features become real? In fact, yes. Earlier this year, Apple published a paper on a new language model called ReALM (Reference Resolution As Language Modeling) that can do things like read what’s shown on the user’s screen, with everything happening locally on the device. It’s worth noting that OpenAI recently demoed a similar feature in the ChatGPT app.

What to expect for Siri with iOS 18

When it comes to real rumors, Bloomberg recently reported that Siri will use AI to do things like open documents, move notes to another folder, delete emails, or summarize an article. “For example, they could ask Siri to summarize a recorded meeting and then text it to a colleague in one request,” said the report.

However, it seems that these advanced AI features for Siri won’t be available to users until next year with a future iOS 18 update.

Earlier today, it was revealed that Apple plans to call these features “Apple Intelligence,” which is a clever way of taking advantage of the “AI” acronym. In addition to Siri enhancements, iOS 18 will have AI features to summarize notifications and messages, create auto-replies for conversations, generate new emoji, transcribe voice recordings, and edit photos.

Apple Intelligence will be powered by a combination of on-device and cloud server capabilities, depending on the task at hand.

Are you excited to see Apple’s AI features in iOS 18? Let us know in the comments below.