Skip to main content

Facebook’s iOS app uses AI and VoiceOver to describe the contents of photos to blind users

facebook-voiceover

At a time when so many Facebook posts comprise a photo and a brief comment, there’s one group of people who get rather left out of the picture: those who are blind and partially sighted. That’s a problem Facebook is fixing, starting from today. The iOS app now uses artificial intelligence to figure out the content of photos, and Apple’s VoiceOver feature to read aloud a description of them.

The Verge got a demo of the feature, which Facebook calls ‘automatic alt text.’

Automatic alt text, which is coming to iOS today and later to Android and the web, recognizes objects in photos using machine learning […] While still in its early stages, the technology can reliably identify concepts in categories including transportation (“car,” “boat,” “airplane”), nature (“snow,” “ocean,” “sunset”), sports (“basketball court”), and food (“sushi”). The technology can also describe people (“baby,” “smiling,” beard”), and identify a selfie.

Facebook said that there were two approaches it could have taken to the problem, and it chose the one it believed would be most successful …

Matt King, a Facebook engineer who is blind, […] said “We could probably require people when they upload a photo: ‘please describe this for blind people.’ It would drive people nuts — that would never work at scale.” (This is the actual approach Twitter is taking to the problem, though adding descriptions is optional.)

Facebook says that it was conscious of the risks of misidentification, so the algorithm only tags photos with keywords if it is 80% certain – but applies a higher confidence requirement to sensitive topics, such as someone’s race.

King says the company is aware that this is just a start, and that there are still ‘millions’ of things the AI can’t identify, but that it will improve over time and make a huge difference to blind users.

The impact of doing something like this is really telling people who are blind, your ability to participate in the social conversation that’s going on around the world is really important to us. It’s saying as a person, you matter, and we care about you. We want to include everybody — and we’ll do what it takes to include everybody.

VoiceOver is one of a range of accessibility features built into iOS. Apple last year created a catalog of apps that take advantage of the feature, the same year that the American Foundation for the Blind honored Apple for the technology.

AD_Clear_Pecan_728x90

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

  1. Jake Becker - 8 years ago

    “She’s beautiful, she’s rich, she’s got huge…tracts of land..”

  2. PhilBoogie - 8 years ago

    That’s fantastic news! Most of my clients are blind, pity they don’t use Facebook.

  3. Doug Aalseth - 8 years ago

    Can’t wait for the inevitable mis tagged pictures to get posted.
    Shot of a bikini clad woman laying in the sun “Sun, Beach, Mountains”

  4. Robert Wilson - 8 years ago

    It is amazing how far tech has come with helping people with disabilities.

    Being a visually impaired person as well as having a totally blind wife and many friends who are also blind or visually impaired I see first hand the advances devices like the iPhone and others have made.

    One of the biggest thing about the iPhone is it has replaced several devices needed by blind people down to a single device. Let’s see I use an app the turns my iPhone into a digital magnifier to read menus and what not. The standalone device for that is like $500 or more. I also use an app to let us use our iPhone to play the talking books from the national Library service that you used to have a very bulky player owned by them or plunk out $300 or more for a standalone unit that you then had to get unlocked by them to access the books.

    Other stuff the iPhone can do for blind people that I know about from my friend. There is an app that will tell them the denomination of paper money they pass by the camera. That replaces a device costing around $200. An app that can tell the color of something in front of the camera. Not sure what a device for that cost. There an app that uses the camera and will use difrent sound clicks and pitch to let you how light or dark a room is. Great for totally blind people that have sighted guest come by and have the lights on and after they leave they can’t tell if everything is off that app dose the trick. There even a couple of apps that you can point at stuff and they tell you what it is. One uses the barcode on the item and looks it up. The other uses OCR and will read the text on like a sign.

    That is just the stuff in currently aware of. I wonder what the future has instore.

    Now on a sadder note tech is also restricting some folks with vision problems. I’ll just use this one example being talked about on my school alumni list since the school is just blind people. Vending machine are starting go touch screen and no accessibility built in. So if your blind you can’t get a drink from it without help. At least with the old ones as long as you know from the past if it was a coke or pepsie machine you always knew the first selection was the main drink of that company but but with the full touch screen there no way to do that since there no button for each drink. Now for an easy fix. Since more and more vending machines are tied to a network since they can take credit cards. They should come up with a way it and say your iPhone can talk to each other. Then you can use voiceover on the phone macke selection one phone pay with phone and the machine spits out your item.

    • Ben Lovejoy - 8 years ago

      Really interesting stuff, Robert. Could you let me know the names of the apps?

      • Robert Wilson - 8 years ago

        The Magnifier I use to read menus and other print I can’t read with my bad eyes. There is a free version but I got to paid version to support the developer.
        Magnifying Glass With Light Pro – Restaurant Menu Reader by Falcon In Motion LLC
        https://appsto.re/us/-FtFC.i

        The money identification app that several blind friends use.
        LookTel Money Reader by IPPLEX
        https://appsto.re/us/o_I4y.i

        This is the one for the audio books but you have to be a registered user of the service and to be eligible you either need to be blind visually impaired or some other disability disorder that prevents you from reading regular books. Once a user has been added to the system all book are free.
        Just a tidbit the books are not like your audiobook from audible or others they are not done by professionals. They are all regular print books read by volunteers.

        BARD Mobile by Library of Congress
        https://appsto.re/us/smpcQ.i

        As for the other apps I would have to get together with my totally blind friends to get the names of other apps. I’m sure if you google iOS apps for blind users or something similar to that you will find more info faster than I can get back to you.

      • Ben Lovejoy - 8 years ago

        Many thanks, Robert. It’s interesting stuff, so I’m thinking about investigating what’s out there and writing a piece about it.

      • Robert Wilson - 8 years ago

        No problem glad I could help. I’ll message one of my former classmates who works in the accessibility industry testing stuff and giving demos. Maybe I can get some more links to stuff from him if he gets time. Might even be able to put him in contact with you but not sure on that.

        Is there a way he or I could send you more info? If you don’t want to put it on here or to be sure it reaches me I’m logged in from Facebook you can send me a private message threw there.

      • Ben Lovejoy - 8 years ago

        Thanks, Robert – I sent you a PM on Facebook (which will be in your Other tab)

  5. moo083 - 8 years ago

    I will say though, as interesting as this is, it is not without issues. These AIs have been known to be accidentally racist, describing black people as gorillas and such. So don’t be surprised if there is a story about this some time from now describing a similar situation.

  6. Megan Fuentes - 8 years ago

    The voice over on Facebook has been great for my husband who is blind. But, one complaint I have is that the app doesn’t read all photos only certain ones. We have been married for six years and he hasn’t “seen” me since I was 19 and I’m 30 now. Is there a way the photos can be described even more? Because I would love for him to be able to “see” our wedding pictures

Author

Avatar for Ben Lovejoy Ben Lovejoy

Ben Lovejoy is a British technology writer and EU Editor for 9to5Mac. He’s known for his op-eds and diary pieces, exploring his experience of Apple products over time, for a more rounded review. He also writes fiction, with two technothriller novels, a couple of SF shorts and a rom-com!


Ben Lovejoy's favorite gear