ChatGPT will soon be able to see everything happening on your screen

Yesterday in its spring update, OpenAI launched a fully-native Mac app for ChatGPT and also debuted its new flagship model, GPT-4o. There was no shortage of remarkable demos and excitement, especially for Apple users mulling the rumored Apple-OpenAI partnership for iOS 18.

One key thing that caught my interest in the full event video was a forthcoming ChatGPT app feature that was demoed but without any real details shared. What feature? The ability for ChatGPT to see everything happening on your device’s screen.

Ready to trust ChatGPT with your on-screen content?

At launch, the ChatGPT Mac app supports manually capturing a screenshot, or uploading a file, to then query ChatGPT about it. But in the future, as seen in the video below, you’ll be able to click a button that gives the app access to anything and everything on your display.

This feature will apparently not be limited to the Mac, as the iOS & iPadOS app demonstrated the same technology in the tutoring example below.

It’s unknown when these screen viewing features will be available in ChatGPT. They do, however, feel like no-brainer additions for a more seamless and interactive experience. Apple even provides APIs for developers to tap into a device’s screen contents—with user consent, of course. So OpenAI isn’t doing anything nefarious by adding these features.

9to5Mac’s Take

ChatGPT having access to the full contents of your screen could certainly prove a bit creepy, but it also will undoubtedly be incredibly useful.

Most likely there will be time limitations on the app’s access to your display, requiring you to hit a button every time you want to grant access. But in theory, some day it makes sense that AI tools will have perpetual access to everything you can see, enabling more seamless help with minimal friction.

An AI assistant that can see everything you see across your devices seems like something Apple could enable at a system level across its platforms. If Siri were supercharged with new AI intelligence, then with its deep system integration it could prove a true asset for users in iOS 18 and beyond.