We scroll through Facebook every day, looking at pictures and videos, commenting and liking. Now, imagine that experience without actually being able to see all those photos and videos. How would you know what people were talking about? How would you know what to say to join in on the conversation?
"It was kind of a letdown," Facebook's accessibility specialist Matt King, who is blind, tells Refinery29 of what it was like using the social network without sight. "Even though people didn't mean to, they were really excluding me from the conversation. It was an inherently visual conversation."
King lost most of his vision while he was in college. He has since spent his career — 25 years at IBM and close to a year at Facebook — dedicated to making technology available to everyone. After joining Facebook in its early years and struggling to navigate its interface, now he, and the rest of the Facebook accessibility team, are really opening up the social networking conversation with a new product, Automatic Alternative Text.
Normally, a visually impaired person uses some form of screen reader to audibly read out text that's on screen (on an iPhone, you can do this by switching on VoiceOver in its settings). However, in today's world, much of what's happening online is in pictures and videos. Automatic Alt Text comes in to translate those images into words. You can see it in action in the video below.
I'm inspired by this video about our artificial intelligence research at Facebook.Our AI can now look at a photo, figure out what's in it and help explain it to you. This is especially helpful if you're blind or can't see the photo. We see AI as helping computers better understand the world -- so they can be more helpful to people.We're still early with this technology, and you can already start to imagine how helpful it will be in the future.
Posted by Mark Zuckerberg on Tuesday, November 3, 2015
"That whole saying of, 'A picture is worth a thousand words,' I think it's true, but unless you have somebody to describe it to you, even having three words helps flesh out all the details that I can't see. That makes me feel included," a woman explains in the video.
In the past, she'd scroll through her feed and have to use the comments underneath to demystify what the image was of. With Automatic Alt Text, she can get some semblance of what a friend may be "So happy!" about without resorting to detective work. (A "So happy!" picture of a steamy slice of pizza doesn't necessitate quite the same response as that of a smiling, newly engaged couple, for instance.)
"It’s hard, if you’re sighted, to put yourself in someone else’s shoes," Jeff Wieland, who heads up Facebook's accessibility team, says. "If you don’t know someone who’s blind, people say, 'Well do blind people want to know what’s in a photo? If they’ve never seen a beach?' The emotion of hearing someone is smiling, that warms your heart whether you can see or not."
Automatic Alt Text works using image recognition to identify objects in photos. It generates, and reads out, a description of a photo, listing out items that may be in the picture. It can identify modes of transportation (things such as a car, bus, bicycle, or boat), nature (words such as outdoor, mountain, tree, and snow), words relating to someone's appearance (glasses, baby, smiling), sports, foods, and selfies.
While Automatic Alt Text isn't incredibly robust at this point, it is accurate, and it's exceptionally more helpful than not having any photo description at all.
And Facebook isn't the only company working on this kind of technology. At its developer conference last week, Microsoft also showed a demo of how its artificial intelligence systems help one of its blind engineers identify what's happening in front of him in the real world. Just when you thought AI was doomed, or doomed to remain incredibly creepy, here it is finally being used for the betterment of mankind.
As for Facebook's new feature, you can try the experience yourself. On your phone, go to Settings, General, Accessibility, and then turn on VoiceOver. Then, go to the Facebook app. When you swipe over a photo, you'll hear the app read out some of the different items the image may include.
"It’s not just about making [Facebook] possible to use," King says, "but making it possible for someone with a disability to get as much enjoyment out of the product as anyone else — to be equally engaged and connected, and feel equal in society, [like], 'Hey, we matter, we’re part of it, too.'"
Like what you see? How about some more R29 goodness, right here?
Why We Celebrate Pi Day On 14th March
The Creator Of The Cronut Is Behind Today's Pi Day Google Doodle