Skip to main content

ChatGPT will soon be able to see everything happening on your screen

Yesterday in its spring update, OpenAI launched a fully-native Mac app for ChatGPT and also debuted its new flagship model, GPT-4o. There was no shortage of remarkable demos and excitement, especially for Apple users mulling the rumored Apple-OpenAI partnership for iOS 18.

One key thing that caught my interest in the full event video was a forthcoming ChatGPT app feature that was demoed but without any real details shared. What feature? The ability for ChatGPT to see everything happening on your device’s screen.

Ready to trust ChatGPT with your on-screen content?

At launch, the ChatGPT Mac app supports manually capturing a screenshot, or uploading a file, to then query ChatGPT about it. But in the future, as seen in the video below, you’ll be able to click a button that gives the app access to anything and everything on your display.

This feature will apparently not be limited to the Mac, as the iOS & iPadOS app demonstrated the same technology in the tutoring example below.

Top comment by Bobby

Liked by 8 people

I'm assuming we can opt out of this. I don't trust Sam Altman. I won't be upgrading to ios18 unless the privacy controls are spot on.

View all comments

It’s unknown when these screen viewing features will be available in ChatGPT. They do, however, feel like no-brainer additions for a more seamless and interactive experience. Apple even provides APIs for developers to tap into a device’s screen contents—with user consent, of course. So OpenAI isn’t doing anything nefarious by adding these features.

9to5Mac’s Take

ChatGPT having access to the full contents of your screen could certainly prove a bit creepy, but it also will undoubtedly be incredibly useful.

Most likely there will be time limitations on the app’s access to your display, requiring you to hit a button every time you want to grant access. But in theory, some day it makes sense that AI tools will have perpetual access to everything you can see, enabling more seamless help with minimal friction.

An AI assistant that can see everything you see across your devices seems like something Apple could enable at a system level across its platforms. If Siri were supercharged with new AI intelligence, then with its deep system integration it could prove a true asset for users in iOS 18 and beyond.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

Author

Avatar for Ryan Christoffel Ryan Christoffel

Ryan got his start in journalism as an Editor at MacStories, where he worked for four years covering Apple news, writing app reviews, and more. For two years he co-hosted the Adapt podcast on Relay FM, which focused entirely on the iPad. As a result, it should come as no surprise that his favorite Apple device is the iPad Pro.

Manage push notifications

notification icon
We would like to show you notifications for the latest news and updates.
notification icon
You are subscribed to notifications
notification icon
We would like to show you notifications for the latest news and updates.
notification icon
You are subscribed to notifications