{"id":62,"date":"2021-12-03T20:59:31","date_gmt":"2021-12-04T01:59:31","guid":{"rendered":"https:\/\/unsung.davidpogue.com\/?p=62"},"modified":"2023-10-27T21:00:33","modified_gmt":"2023-10-28T01:00:33","slug":"how-apple-and-microsoft-built-the-seeing-eye-phone","status":"publish","type":"post","link":"https:\/\/www.unsungscience.com\/index.php\/2021\/12\/03\/how-apple-and-microsoft-built-the-seeing-eye-phone\/","title":{"rendered":"How Apple and Microsoft Built the Seeing-Eye Phone"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Season 1 \u2022 Episode 7<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Your smartphone can see, hear, and speak\u2014even if you can\u2019t. So it occurred to the engineers at Apple and Microsoft: Can the phone be a talking companion for anyone with low vision, describing what it\u2019s seeing in the world around you?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today, it can. Thanks to some heavy doses of machine learning and augmented reality, these companies\u2019 apps can identify things, scenes, money, colors, text, and even people (\u201c30-year-old man with brown hair, smiling, holding a laptop\u2014probably Stuart\u201d)\u2014and then speak, in words, what\u2019s in front of you, in a photo or in the real world. In this episode, the creators of these astonishing features reveal how they turned the smartphone into a professional personal describer\u2014and why they care so deeply about making it all work.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Guests: Satya Nadella, Microsoft CEO. Saqib Shaikh, project lead for Microsoft\u2019s Seeing AI app. Jenny Lay-Flurrie, Chief Accessibility Officer, Microsoft. Ryan Dour, accessibility engineer, Apple. Chris Fleizach, Mobile Accessibility Engineering Lead, Apple. Sarah Herrlinger, Senior Director of Global Accessibility, Apple.<\/p>\n\n\n\n<figure class=\"wp-block-audio\"><audio controls src=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3\"><\/audio><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Episode transcript<\/h2>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Intro<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Your smartphone can see, hear, and speak\u2014even if you can\u2019t. So the accessibility engineers at Apple and Microsoft wondered: Could the smartphone ever be smart enough to serve as a talking camera for people who are blind or have low vision? Could it describe what it\u2019s seeing in the world around you, or photos in front of you?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>App:<\/em><em>&nbsp;<\/em><em>A group of people sitting around a table playing a board game.&nbsp;<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny:&nbsp; Seeing AI is one of the most incredible revolutionary products I think we\u2019ve ever put out there. \/I get emotional when I think about what employees have created.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today, the origin stories of two amazing accessibility features from Microsoft and Apple. I\u2019m David Pogue, and this is \u201cUnsung Science.\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Season 1, Episode 7: How Apple and Microsoft Built the Seeing-Eye Phone. We\u2019re releasing this episode on December 3, the International Day of Persons with Disabilities, which the United Nations created in 1992. And the reason that\u2019s so appropriate will become obvious within the next 90 seconds.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">About eight years ago, I was hired to host a panel at a corporate event. And backstage, I spotted one of the other panelists waiting. She was using her iPhone in a way I\u2019d never seen before.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>App:&nbsp;<\/em><em>Voiceover fast<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Her screen was off\u2014it was just black\u2014and she was sliding her finger around. She was using VoiceOver, Apple\u2019s screen-reading feature for blind people like her. It speaks the name of every icon, button, list item, and text bubble beneath your finger. Over time, she\u2019d gotten so good at it that she\u2019d cranked the speaking rate up. It was so fast that I couldn\u2019t even understand it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>App:&nbsp;<\/em><em>Voiceover fast<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">She gave me a little demo\u2014and it occurred to me that her phone\u2019s battery lasts twice as long as mine, because her screen never lights up\u2014and of course her privacy is complete, since nobody can&nbsp;<em>see&nbsp;<\/em>anything on her screen. She usually uses earbuds.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But VoiceOver was just the iceberg tip. Turns out Apple has essentially written an entire shadow operating system for iPhones, designed for people with differences in hearing, seeing, muscle control, and so on. The broad name for these features is accessibility.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, in popular opinion these days, the big tech companies are usually cast as the bad guys. But Apple and Microsoft have entire design and engineering departments that exist&nbsp;<em>solely&nbsp;<\/em>to make computers and phones usable by people with disabilities. And they\u2019re totally unsung.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah&nbsp;<strong><\/strong>Our first Office of Disability was actually started in 1985, which was five years before the Americans with Disabilities Act came to pass. \/<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah Herrlinger is Apple\u2019s Senior Director of Global Accessibility. She describes her job like this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah My job is to look at accessibility at the 30,000-foot level, and make sure that any way that Apple presents itself to the world, that we\u2019re treating people with disabilities with dignity and respect.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David What is the business case to be made for designing features that are, at their, at their core, intended for a subset of your audience?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah You can look at public statistics that tell us that 15 percent of the world\u2019s population has some type of disability.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2014 that number grows exponentially over the age of 65.&nbsp; Whether you think so or not, you\u2019re probably going to be turning on some of these features.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But for us, we don\u2019t look at this as a traditional business case, and don\u2019t focus on ROI around it. You know, we believe it\u2019s just good business to treat all of our customers with dignity and respect, and that includes building technology that just works no matter what your needs might be.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I think you might be surprised how far these companies go. Like on the Mac: they\u2019ve added this entire spoken interface, so that you can move the mouse, click, drag, double-click, Shift-click, open menus, type, edit text\u2014all with your voice. Same thing on the iPhone and iPad. Very handy if you can\u2019t use your hands\u2014whether they don\u2019t work, or just because they\u2019re full of groceries. Or greasy.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s the Apple ad that introduced this Voice Control feature. It\u2019s basically a demo by outdoor enthusiast Ian Mackay, who\u2019s paralyzed from the neck down. Here\u2019s him, opening the Photos app and choosing a photo to send as a text message to his buddy:&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>APP: Voice control audio<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">IAN: Wake up. Open Photos. Scroll up. Show numbers. 13. Click Share. Tim. Next field. Let\u2019s ride this one today. Thumbs-up emoji. Click Send.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Did you hear that \u201cShow numbers\u201d business? That\u2019s how you tell the machine to click something on the screen that\u2019s&nbsp;<em>unlabeled<\/em>\u2014like the thumbnails of your photos. When you say \u201cshow numbers,\u201d little blue number tags appear on every single clickable thing on the screen, and you just say the one you want to click.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">IAN: Show numbers. 13.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But I mean, there are also features for people who have trouble hearing, like your phone\u2019s LED flash can blink when you have a notification. There\u2019s another feature that lets you use your iPhone as a remote microphone\u2014you set it on the table in front of whoever\u2019s talking, and it transmits their voice directly into your AirPods or hearing aids.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And there\u2019s a feature that pops up a notification whenever the phone hears a sound in the background that should probably get your attention, like the doorbell, a baby crying, a dog or a cat, a siren, or water running somewhere. Here\u2019s my cat Wilbur and me testing out this Sound Detection feature:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>SOUND: Wilbur<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And sure enough! My phone dings and says, \u201cA sound has been recognized that may be a cat.\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you\u2019re in a wheelchair, the Apple Watch can track your workouts. If you\u2019re color-blind, like me, there\u2019s a mode that adjusts colors so you can distinguish them. If you\u2019re paralyzed, they\u2019ve got features to let you operate the Mac with a head switch, blink switch, joystick, or straw puffer. They\u2019re even trying to make the Apple Watch useful if you can\u2019t tap it. Here\u2019s Sarah Herrlinger:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah We brought Assistive Touch to Apple Watch as a way for individuals who have upper body limb differences. So someone who might be an amputee, or have another type of limb difference, to navigate and to use the device \u2014 without ever having to touch the screen itself.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I tried it out. Each time you tap two of your fingers together, on the arm that\u2019s wearing the watch, the next element on the watch\u2019s screen lights up. Tap, tap, tap\u2014and when you get to what you want to open or click, you make a quick fist to click it.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And it\u2019s not just Apple. A few years ago, for a story on \u201cCBS Sunday Morning,\u201d I interviewed Microsoft CEO Satya Nadella. And at one point, he described how his life changed the day his son Zane was born\u2014with quadriplegia.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">NADELLA: 12:46:49:15 Even a few hours before Zain was born, if somebody had asked me\u2013 \u201cWhat are the things that you are thinking about?\u201d&nbsp; I would have been mostly thinking about\u2013 \u201cHow will our weekends change?\u201d And about childcare and what have you.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And so obviously after he was born, our life drastically changed. To be able to see the world through his eyes and then recognize my responsibility towards him, that I think has shaped a lot of who I am today.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">POGUE: But how does something as emotional\/ as that empathy that\u2013 that you\u2019ve developed translated into something as nuts-and-bolts-y and number-crunch-y as running a huge corporation?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">NADELLA: There\u2019s no way you can motivate anyone if you can\u2019t see\u2013 the world through their eyes. There\u2019s no way\u2013 you can get people\u2013 to bring their&nbsp; A game&nbsp; if you can\u2019t create an environment in which they can contribute. But the creation of that environment requires you to be in touch with what are they seeking? What motivates them? What drives them? So as a leader or as a product creator, can draw a lot from I would say this sense of empathy.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So it\u2019s no coincidence that later in our shoot, the one new Microsoft software product the CEO was most eager to show our cameras was something called Seeing AI. This was, by the way, a Microsoft app that runs only on the&nbsp;<em>iPhone<\/em>. From&nbsp;<em>Apple<\/em>.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s a clip from that \u201cSunday Morning\u201d story. Nadella and I are in a company snack bar with Microsoft engineer Angela Mills, who\u2019s legally blind. She showed me how Seeing AI works.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>SOUND: Angela demo<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">MILLS: &nbsp; So if I now hold it up\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DP (VO): It helps her read text\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">PHONE: \u201cCarob Malted Milk Balls.\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DP (VO): \u2026recognize objects\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">PHONE: Banana. Orange.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DP: That is oranges!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">MILLS: Yup! And then\u2026Take picture.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DP (VO): \u2026and even identify faces.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">PHONE: 49 year old man with brown hair looking happy.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">POGUE: Wow, it left out tall and handsome. But that\u2019s pretty close! (laughter)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So today, you\u2019re going to hear two stories. Backstories of how two very cool accessibility features came to be. By what\u2019s probably no coincidence at all, both features were invented&nbsp;<em>by&nbsp;<\/em>disabled employees at these corresponding companies\u2014Apple and Microsoft.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Incidentally, this is the first episode of \u201cUnsung Science\u201d that involves commercial consumer corporations. There are a couple of others later this season. I just want to make clear that these companies did not, do not, and can not pay to be featured as a topic on this show; it wasn\u2019t even their idea. In fact, I hounded&nbsp;<em>them&nbsp;<\/em>for interviews, they didn\u2019t hound me. Sometimes, for-profit corporations make cool scientific or technical breakthroughs, too.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[pause\/music]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OK. So our first story is that Seeing AI app from Microsoft. What a great name, too, right? Like seeing eye dog, but seeing AI, for artificial intelligence?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Anyway, the app has ten icons across the bottom; you tap one to tell it what kind of thing you want it to recognize. It can be text\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">APP: Caution: Non-Potable Water.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Barcodes \u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">APP: Ronzoni Gluten-Free Spaghetti.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">People\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">APP: 30-year-old woman with red hair looking angry.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Currency\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">APP: 10 U.S. dollars. One U.S. dollar.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Scenes\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">APP: A bus that is parked on the side of the road.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Colors\u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Gray and white. Red.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Or handwriting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Sorry Dad\u2014I ate the last of your birthday cake.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There\u2019s even a mode that uses low or high-pitched notes to tell you how dark or bright it is where you are, or as you move from room to room.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: (Warbles low and high)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The app can also tell you what\u2019s in a photo.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: &nbsp; One face. A woman kissing a llama.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The subject of that picture isn\u2019t just any woman kissing a llama\u2014it\u2019s my wife, Nicki. She\u2019s always had a thing for llamas.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Anyway. The man behind the app is Saqib Shaikh (Sokkibb Shake), who Microsoft introduced in a YouTube video like this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">SAQIB: I\u2019m Saqib Shaikh. I lost my sight when I was seven. I joined Microsoft ten years ago as an engineer. And one of the things I\u2019ve always dreamt of since I was at university was this idea of something that could tell you at any moment what\u2019s going on around you.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In 2014, he got his shot. Microsoft held its first-ever, company-wide hackathon\u2014a programming contest to see which engineering team could come up with the coolest new app in one mostly sleepless week. And Saqib\u2019s app won.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib: I gotta say, it was very basic, very early.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It could read text, and do a bit of face recognition to help you identify your friends while you\u2019re walking past, and a few other things like recognizing colors and so forth. But&nbsp; describing images\u2014 that didn\u2019t come to a year or more later.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And, you know, it got some attention, but it wasn\u2019t really till the next year we thought, \u201clet\u2019s do it again.\u201d And over time, more and more people got involved.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In 2015, a more mature version of Seeing AI won Microsoft\u2019s hackathon&nbsp;<em>again.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib<strong>&nbsp;<\/strong><strong><\/strong>So when we won the second hackathon, the company wide, I told my manager, look, we have this opportunity. And he was just like, \u201cOK, I am just going to give you two months to see what you could do full time.\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But then it never stopped being my project. And before we knew it, we were on stage with the CEO at the Build conference in 2016, which was a pivotal moment and just so incredible.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That CEO was, of course, Satya Nadella, who brought Saqib onto the stage.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">NADELLA: It\u2019s such a privilege to share the stage with Saqib. You know, Saqib took his passion, his empathy, and he\u2019s gonna change the world.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[pause\/music]&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Seeing AI relies on a form of artificial intelligence called machine learning.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib<strong>&nbsp;<\/strong><strong><\/strong>So the way machine learning works is, you have these algorithms called neural networks, which are\u2014you\u2019re not giving them steps like \u201crecognize this, do this, look for this type of color or this line.\u201d Instead, you\u2019re taking many, many examples, hundreds of thousands of examples, of different photos. And then someone is teaching the computer by describing it. And that could be writing a sentence about it, such as \u201cthis is a living room with such and such.\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You\u2019re teaching the system by giving it, \u201cThis is the real answer.\u201d And then this so-called neural network will learn over many, many iterations that this is the concept that makes this thing over here a couch. And this is the thing that makes this other thing over here a car.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David I see. So&nbsp; does the machine learning write code, or is it just a black box and you\u2019ll never really know why it thinks this banana is a banana?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib In many ways, it is a black box where the system has learned the association between this banana and the word banana.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib even mentioned a feature I\u2019d missed completely:&nbsp;<em>feeling&nbsp;<\/em>your photos.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib You can kind of feel the little clicks around the edges of objects and you hear that, \u201cwow, there\u2019s a car in the bottom left and there\u2019s a house over on the right\u201d and \u201coh, there\u2019s my friend over there,\u201d and you\u2019re kind of tracing a photo with your finger on a flat piece of glass.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And sure enough: In the Scene mode, if you tap the Explore button, you can run your finger over the photo on the screen and feel little vibrations\u2014with accompanying sounds\u2014as your fingertip bumps into objects in the picture.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Move your finger over the screen to explore. Couch. Table. Chair 1. Chair 2. Monitor. Desk. Keyboard instrument.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sometimes, Seeing AI is incredibly, freakishly specific in its descriptions. I tried to baffle it with a photo of me standing in front of this giant wall of stacked shipping containers at a seaport. I figured it\u2019d have no chance of figuring out what they were. But here\u2019s what it said:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: A man standing in front of a row of shipping containers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Wow.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OK, something harder then. My daughter Tia has this hobby, where she carves incredibly lifelike images into pumpkins. Like, four-level grayscale photographic-looking carvings. One year, she carved actress Shailene Woodley\u2019s face into a pumpkin. And in my phone\u2019s photo roll, there\u2019s a shot of her pumpkin&nbsp;<em>next&nbsp;<\/em>to the Shailene Woodley&nbsp;<em>picture&nbsp;<\/em>that Tia used as a model. And here\u2019s what Seeing AI said:&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Two faces\u2014probably a collage of Shailene Woodley.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I mean, what the ever-loving\u2014how did it know&nbsp;<em>that?&nbsp;<\/em>It knew that the woman on the pumpkin was Shailene Woodley? And that there were two Shailene Woodleys\u2014one in a photo, and one on a squash<em>?&nbsp;<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On the other hand, I should do some expectation setting here: The app is not flawless, and it\u2019s not even complete\u2014several of the recognition features are still labeled Preview. It sometimes gets easy ones wrong:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Probably a white horse with a long mane.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Uh, nope\u2014it\u2019s very clearly a llama. And especially in barcode mode, the app often just gives you a shrug:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Processing. Not recognized.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, it\u2019s worth mentioning that the most impressive descriptions arrive only if you let the app send a photo to Microsoft\u2019s servers for analysis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Processing.&nbsp;<em>(musical notes)&nbsp;<\/em>Probably a cat standing on a table.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Get down from there, Wilbur!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Still, Seeing AI has gotten better and better since Microsoft released it in 2017. Saqib says he uses it almost every day.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib It can range from just reading which envelope is for me in the mail. Versus for my wife. In a hotel room to see, OK, all these little toiletry bottles, which one\u2019s going to be the shower gel and the shampoo. You don\u2019t want to get that the wrong way round!&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A lot of these things then puts me in the driving seat instead of having to ask someone for help.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Is there anything technologically that\u2019s holding you back from making the app the dream app?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib&nbsp;<strong><\/strong>I would love to be able to have AI that understands time, not only that this is a photo, but what\u2019s going on over time, like, \u201cthe man\u2019s walking down the corridor and just picked this up.\u201d&nbsp; And there are scientists around the world solving each one of these small problems.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[Pause]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny:&nbsp; Seeing AI\u2026Seeing AI is one of the most incredible revolutionary products I think we\u2019ve ever put out there. I get emotional when I think about what employees have created.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny Lay-Flurrie is Microsoft\u2019s chief accessibility officer. She\u2019s deaf, so we chatted over Zoom with the help of her sign-language interpreter Belinda, which I thought was very cool. I started by asking her the same devil\u2019s-advocate question I\u2019d asked Apple\u2019s accessibility head.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Microsoft is&nbsp; very proud of its work in accessibility.&nbsp; From a business standpoint:&nbsp; you\u2019re doing a lot of expense and effort for a subset of potential customers.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny:&nbsp; Well, I disagree, of course.&nbsp; A billion people is not a subset.&nbsp; Disability is an enormous part of \u2014of community. It\u2019s part of being human.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David I mean, you say there\u2019s a billion people who are helped by these technologies. Do you think the people who could use these features and these apps \u2026<em>know<\/em>&nbsp;about these features and these apps?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny No, I don\u2019t think that enough people know about what is available today with modern accessibility digital tools. So that comes back to, how do we educate and get it into people\u2019s hands?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David&nbsp;<strong><\/strong>Well, you could go on big podcasts and talk about it, that would help.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny Game on! Let\u2019s do that!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After the ad break, I\u2019ll tell you the story of a cool Apple accessibility feature I\u2019ll bet you didn\u2019t know existed. Meanwhile, let\u2019s take a moment to acknowledge: Man, Microsoft and Apple both allowing their chief accessibility people to join in on the same podcast? I thought these companies are, like, arch-rivals?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But Jenny Lay-Flurrie surprised me. In accessibility, they have a kind of truce.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny I would say that actually accessibility is one part of the tech industry where we don\u2019t compete.&nbsp; We collaborate.&nbsp; On any given day, I\u2019m chatting with my peers in all of the companies.&nbsp; This is bigger than us. This isn\u2019t about one company versus another.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Inclusion is not where it needs to be, and technology is one powerful means to help address that. So, yeah, I would say this is an industry wide, and the maturity over the last five years has been incredible. Incredible! And it makes me bluntly, just stupidly excited about where we\u2019ll be five years from now.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I\u2019ll be back after the ads.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\"><strong>Break \u2013&nbsp;<\/strong><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Before the break, you heard about Seeing AI, a Microsoft app for the iPhone for blind people that\u2019s designed to describe the world around you. Apple has something like that, too. It\u2019s an option that\u2019s available when you turn on the VoiceOver screen reader.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once again, you could quibble with some of the descriptions, like in this photo of me in front of a fireplace:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Voice:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A photo containing an adult in clothing.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Well,&nbsp;<em>that&nbsp;<\/em>narrows it down! Or how about this one:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Voice:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A person wearing a red dress and posing for a photo on a wooden bridge.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Well, that\u2019s all true\u2014and the red dress part is really impressive. But I\u2019d say the&nbsp;<em>main&nbsp;<\/em>thing in this photo is that she\u2019s in a cave, surrounded by stunning white stalactites. And I know VoiceOver knows what those are, because in the very next picture, it says:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Voice:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A group of stalactites hanging from the ceiling of a cave.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But most of the time, VoiceOver scores with just unbelievable precision and descriptiveness. Listen to these:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Voice:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A person wearing sunglasses and sitting at a table in front of a Christmas tree. Two adults and a child posing for a photo in a wagon with pumpkins. A group of people standing near llamas in a fenced area. Maybe Nicki.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OK, what!? It not only got the llamas, but it identified my wife!? Well, I guess I know how it did that. The Photos app learns who\u2019s in your photos, if you tell it\u2014so this feature is obviously tapping into that feature. But it really got me there.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Oh\u2014and VoiceOver Recognition even describes what\u2019s in&nbsp;<em>video&nbsp;<\/em>clips.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Voice: Video: A person holding a guitar and sitting on a couch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris<strong>&nbsp;<\/strong><strong><\/strong>[00:04:36] \/ I totally agree that these machines are now doing things that are ridiculous.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris Fleizach is in charge of the team at Apple that makes all the accessibility features for iPhone, Apple TV, Apple Watch, and so on. Including that photo-describing business, whose official name is VoiceOver Recognition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris&nbsp;<em>(continuing):&nbsp;<\/em>&nbsp;This is a combination of machine learning, the vision processing, and to string together a full sentence description.&nbsp; And so we can grab this screenshot, feed it through this machine learning, vision-based algorithm, pop out a full sentence in under a quarter of a second\u2014and do it all on device.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By \u201cOn device,\u201d he means that all of this happens right on the phone in your hand. No internet needed. No sending images off to some computer in the cloud for processing, which is good for both privacy and speed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris: Before, you could have done that, but you\u2019d have to send it off to a server to be processed in some data farm and it would take 10 seconds to come back.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Getting to this point didn\u2019t happen overnight, of course.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris: the image descriptions, that\u2019s something that we started working on years and years ago when I essentially saw a prototype that someone else was working on at Apple. And they said, \u201cWell, look at this cool thing. I can take this photo and turn it into a sentence, and it\u2019s sort of OK.\u201d And I said, \u201cwe need that. We need that now. How do we make that happen?\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And, you know, four years later, with the involvement of 25 different people across Apple, we finally have this on-device image description capability.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[pause]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But we are gathered here today to hear the origin story of a different accessibility feature. Not object recognition, but people recognition. Actually, not even that. People&nbsp;<em>distance&nbsp;<\/em>recognition.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David How do you pronounce your name?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan Dour.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Okay. Like the word.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan Yes. Well, I frequently hear people say \u201cdower,\u201d but I\u2019m definitely not a dour person. I\u2019m a doer.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Oh, that\u2019s very good.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So this is Ryan Dour, D-O-U-R.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David So what is your actual job at Apple? I mean, you\u2019re not \u2014 you\u2019re not Idea Man.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan Well, no.&nbsp; I work on the software engineering Accessibility Quality Assurance team. So my job at Apple is to test and qualify all of our accessibility features, and then to make sure that those features work with many of our products.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan says that, conveniently enough, Apple\u2019s various accessibility features for low vision have evolved at just about the right speed to keep up with his own deteriorating vision.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan When I was a kid,&nbsp; I had some vision, and over my lifetime, as my vision went from low vision to, well, quite frankly, no vision, there\u2019s \u2014 Apple\u2019s always been sort of at that forefront of technology, such that it actually followed my progress. You know, I went from actually using, you know, Close View and Zoom to, \u201cOh, okay, my vision\u2019s getting to the point where I can\u2019t really use Zoom effectively anymore. Oh, but\u2014but here comes Spoken User Interface Public Preview right on time.\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, when you can\u2019t see, it takes some ingenuity to navigate the world\u2014and if you haven\u2019t been there, some of the stickiest situations may not occur to you. And for Ryan, one of the most awkward social moments is standing in lines. How do you know when it\u2019s time to shuffle forward, even if you\u2019ve got your cane?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[Ambi]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan So imagine at a theme park, we\u2019re in line at a theme park. The person in front of me is moved up, but I don\u2019t want to constantly be tapping them, and there\u2019s tons of voices and lots of chatter going around, so I don\u2019t necessarily hear that they have specifically moved up. The person behind me now is waiting. And they\u2019re not noticing. When they finally do notice, they\u2019re getting annoyed. I feel like a rubber band. I\u2019m bouncing between the person in front of me, tapping them, stopping, and then the person behind me is \u2014&nbsp;if, if I don\u2019t move up quickly enough, saying, you know, \u201cgo ahead, move up.\u201d And imagine doing that for an hour while you\u2019re waiting for a rollercoaster.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With a dog, by the way,&nbsp; you\u2019re relying on your dog to wait in line. But their goal, as trained, is to actually get you&nbsp;<em>around<\/em>&nbsp;objects. So very frequently, if there\u2019s a space, the dog will say, \u201cOK,&nbsp; it\u2019s time to move up and around the people in front of you.\u201d And that, that can become an issue as well.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OK. With that background, you\u2019ll now be able to understand the significance of the meeting Ryan attended at Apple one day in the summer of 2019.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That would be 2019, PC\u2014pre-Covid.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan We had this meeting with our video team and they were introducing us to a new technology. We all know it now as Lidar.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Lidar stands for&nbsp;\u201dlight detection and ranging.\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[music]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A lidar lens shoots out very weak laser beams\u2014you can\u2019t see \u2018em, and they can\u2019t hurt your eyes\u2014and measures the infinitesimal time it takes for them to bounce off of nearby objects and return to the lens, like a bat using echolocation. That way, lidar can build a 3D understanding of the shapes and distances of things around it. It\u2019s the same idea as radar, except using light to measure distance instead of radio waves.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They put lidar on planes, aimed at the ground, to measure crop yields, plant species, or archaeological features. They\u2019ve used it to map the surface of the moon and the sea floor. A lot of self-driving car features rely on lidar\u2014and so do speed guns that the police use to give you speeding tickets.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But in 2020, Apple built lidar into the back of its iPhone 12 Pro. The iPhone 13 Pro has it, too. The big idea was that this lidar would permit software companies to create really cool augmented-reality apps\u2014you know, where the screen shows realistic 3-D objects in the room with you that aren\u2019t really there, like an elephant or a sports car, even as you move around or change angles.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OK, back to Ryan Dour\u2019s meeting at Apple. You\u2019ll hear him refer to haptics\u2014that\u2019s the little clicky vibrations that smartphones make.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan: They had this demo app that would provide haptics based on how far away the person was. Or other things\u2014like, it wasn\u2019t just people at the time, it was just simply the output of the Lidar sensor itself, provided in a haptic way.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And we all sat around, a bunch of us from the accessibility team and video team. And we thought about \u201cwhat are some really great things we could do with this?\u201d And a lot of ideas were sort of thrown around in the\u2014 in the meeting.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp; But towards the very end of it, I remember we had this lightbulb moment where I said, \u201cHey, wait a minute. Let\u2019s go out into the hall.\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And so we went out into the hall at Apple Park, and I said, \u201cOK\u201d\u2013I had an engineer stand in front of me and I said, \u201cLet\u2019s pretend we\u2019re at a coffee shop and you\u2019re in line in front of me.&nbsp; Whenever you want to, at random, I want you to just go ahead and walk forward.\u201d And I held out this\u2014this app with this haptic prototype, and I could feel like bup bup bup bup bup boop boop\u2026 and it\u2019s like, \u201cOh, OK, that person moved up.\u201d&nbsp; I said, \u201cYou know what? This has some serious practical uses. I think that this is going to be something we should really consider in the future.\u201d And that was sort of the end of that meeting.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It sounded like a cool enough idea\u2014just not cool enough to act on immediately. The idea was filed away.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But then, in March 2020, Ryan\u2019s idea got a big, hard push from a source that nobody saw coming: COVID-19. I think we can admit that for a lot of people, the pandemic wasn\u2019t a great time. But it was even worse for people without sight.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan: I was feeling a lot of apprehension in places that I\u2019d never felt apprehension before! Like a grocery store or, you know, my local coffee shop. Just wondering, \u201cwhere can I stand in the room where I\u2019m not in somebody else\u2019s bubble and nobody\u2019s going to be in my bubble?\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I was definitely not looking forward to potentially catching this, mostly because I thought, I don\u2019t want to lose my taste. I already can\u2019t see; I\u2014I don\u2019t want to lose my taste and smell.&nbsp; And so, I was incredibly careful. I would say, maybe more cautious than others about trying to keep my distance, and, and also really being concerned that I didn\u2019t want to be that vector that brings that to somebody else, either.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And it was like, \u201cOkay, you know what? That, that, that feature that we\u2019d been talking about doing \u2014&nbsp;why don\u2019t we do it now? Why don\u2019t we do it, why don\u2019t we do it&nbsp;<em>right<\/em>&nbsp;now?\u201d&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In other words, this idea of using Lidar to detect how far away people were now had&nbsp;<em>two&nbsp;<\/em>purposes. It could help blind people know when it\u2019s time to move ahead in a line\u2014and it could help&nbsp;<em>anybody&nbsp;<\/em>know if they\u2019re observing six feet of social distancing.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But Ryan\u2019s first experiment with a prototype of the app, back in 2020, wasn\u2019t exactly a triumph.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan So we built this prototype and I took it out onto the streets of San Francisco. And the first thing that we encountered was, \u201cOh my gosh, this is a cacophony of feedback!\u201d It was detecting poles. It was detecting garbage cans. It was detecting dogs and people in cars and all sorts of things.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I\u2019m hearing, you know, \u201ceight three five eight nine six three five.\u201d I\u2019m like, Whoa, okay, what\u2019s going on here? And then with the sounds feedback, it was just all over the place. You know, the beeps were close, the beeps or further apart, and it was just \u2014 I didn\u2019t even know what object was being detected at that point.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And at that point, it wasn\u2019t actually a useful tool.&nbsp; Everything was kind of setting me off in this really weird walking pattern of \u2014 no different than if my cane had been hitting a bunch of objects that make me stop and think for a moment.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Right.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan<strong>&nbsp;<\/strong>And so, we started to consider, what are the other technologies we can use here?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The answer was to rope in the Apple engineers who worked on augmented reality.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan: ARKit, which is our augmented reality software development framework for, for developers&nbsp; has a feature, People Occlusion. So you may have used an augmented reality app where a body part like an arm or even another person gets in the way of the view, right? That the objects that you\u2019re virtually viewing in your augmented reality game or, you know, whatever the environment is \u2014 when people get into view, they, they block it off. That\u2019s actually incredibly useful. It\u2019s amazingly powerful, and that is part of the machine learning process.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s an example of People Occlusion. There\u2019s a super cool, free augmented-reality app called JigSpace, that lets you place your choice of all kinds of 3-D objects right in front of you, as viewed on your phone\u2019s screen: a life-sized printing press, or lunar lander, or combine harvester, or whatever. And right there in your living room, you can walk around this thing, come up close to it, and so on.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Well, suppose I\u2019ve got this app, and now there\u2019s a huge coral reef in my living room. Hey\u2014it could happen.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Sound: Coral<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, if somebody walks in front of that reef, close to your phone, their body looks like it\u2019s passing in front of the coral wall. But if they walk far enough away from you that they should be behind the reef, the reef blocks them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Turns out that\u2019s just the sort of intelligence Ryan\u2019s app needed to distinguish people in the environment from random clutter\u2014and to ignore his&nbsp;<em>own&nbsp;<\/em>body parts in the scene.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan For example, if I\u2019m using my cane and my hand on my cane comes into view from the bottom, OK, we can ignore that. \/ I don\u2019t need to be detecting my feet or my shirt or a finger; I need to be detecting the person in front of me. And so we\u2019ve been able to really fine tune this so that we can pick up just the people and not the dogs and the trashcans and the poles. And this results in a fantastic tool.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By the way\u2014there\u2019s a good reason that no other phones have a feature like People Detection: It really needs the Lidar. Chris Fleizach\u2019s team briefly explored reproducing the feature using only regular iPhone cameras to see the world. But regular cameras don\u2019t see the world in 3-D, the way Lidar does.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Chris And so, yeah, hard decisions were made. We would have loved to have brought this to more devices, but it wouldn\u2019t have been good enough.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So\u2014in the beginning, the phone thought that everything in the environment was a person. But in the end, the solution was combining the depth information from the Lidar with the screening-out abilities of the augmented-reality software kit.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you have an iPhone 12 or 13 Pro, you can try the People Detection feature yourself. It\u2019s hiding in the Magnifier app. You open Magnifier, and tap the Settings sprocket. Tap People Detection. You can specify your social-distance threshold, like 6 feet, and also how you want the app to tell you when it detects people nearby.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You have three options: Sounds, which plays faster boops as someone gets closer to you and then switches to a higher note when they\u2019re within six feet;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App:&nbsp;<em>Boops<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2026Speech, which speaks how far away the nearest person is, in feet or meters;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App:&nbsp;<em>Speech counter<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2026or Haptics, which uses little vibrational clicks that play in sync with the boops. You can turn \u2018em on in any combination. Here\u2019s what it sounds like in your earbuds if you have all three turned on\u2014in this case, as I walked past somebody in the drugstore.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">App: Seven. Six. Five. Six. Seven. Eight.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s Ryan again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan So, so one subtle thing you might notice is that if you\u2019re not using headphones, instead of hearing the \u2014the feedback left to right across your, your soundstage, it will also change in volume to indicate, you know, how centered the person is.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>APP: Stereo<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Oh, that\u2019s cool! So yeah. If you\u2019ve got both earbuds in, you can hear where the person is, left to right, as they cross your path.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>APP: Stereo<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you\u2019re not wearing earbuds, the volume indicates if the person is centered in front of you.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>APP: Boops<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan has it set up so that two quick taps on the back of the phone turn the detection on, so he can whip the phone out unobtrusively and, thanks to those haptic clicks, start sensing where people are around him.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s how Ryan heard the world as he approached a food truck to pick up his order. You can also hear him talking to his guide dog.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Sound: Foodtruck<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan It took away a lot of the apprehension, especially in places like waiting in line at the grocery store to check out. And this was especially true before we had the vaccines.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Oh, yeah.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Like everybody else, Apple\u2019s accessibility team is starting to dream of a time when this pandemic is over. What will happen to social distancing then?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan: Right now, we may be using it for keeping track of where, you know, where people are, what\u2019s my bubble? But in the future, you may want to have a different threshold for how far away are people.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp; So, for example, our sound feedback provides tones that indicate when they\u2019re, when they\u2019re playing very fast, the person\u2019s very close.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;When they\u2019re playing slower, the person\u2019s further away. But as you cross over the threshold that you\u2019ve set, which by default is six feet, it drops in pitch<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>APP: Fast-slow boops<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan: \u2026 and then the distance between the tones also increases until you really kind of get out of range \u2014&nbsp;until that person is around 20 feet away and then they\u2019re not detected.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David Right.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ryan In the future, this is going to be useful for walking into that crowded coffee shop and looking for that quiet corner, or&nbsp; getting on to a subway car and figuring out like, where\u2019s the empty space I can go sit down?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[pause]&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As you\u2019ve probably figured out, I&nbsp;<em>love&nbsp;<\/em>this stuff. Some of the cleverest, most magical work going on in Silicon Valley today is in these accessibility features\u2014and so few people even know they exist!&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And you may be thinking, \u201cWell, it doesn\u2019t matter if I know about these features. I\u2019m not disabled.\u201d But the accessibility engineers I interviewed made one point over and over again: Almost every time they dream up a feature for the disabled community, it turns out to be useful for the wider public.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s Apple\u2019s Sarah Herrlinger:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sarah There are many people for whom they don\u2019t self-identify as having a disability. And yet accessibility can be more about productivity for those individuals. If you go under the accessibility panel on any of our devices, take a little bit of time to investigate what\u2019s there, I would say you\u2019re probably going to find something that will make your life easier.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Microsoft\u2019s Saqib Shake:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib This is a whole history of disability as a driver of innovation. And I could tell you a dozen stories.&nbsp; The fact that our phones can talk to us, this was part of the invention of the first reading machine.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Same with voice recognition, was for people with physical impairments. Even the touch screen, to some extent was invented by someone who had difficulty typing and wanted a lower impact way to type text messages for people who were deaf.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Microsoft accessibility head Jenny Lay-Flurrie:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny:&nbsp; Captioning! Captioning came out of creating technology for the deaf. We all use captioning in different ways. We can be sitting on a train, or trying to sneak that video without the person next to you watching.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Audiobooks is the same. Those were created as talking books for the blind. And now look at what\u2019s happening with audiobooks.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Saqib: [00:20:34] People kind of forget where the origin came from. And that\u2019s a good thing because it just blends into the fabric of life.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny: By making something accessible, you don\u2019t just make it inclusive to the cool people with disabilities. You actually give core capability that everyone can benefit from.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I told Jenny the story of that conference eight years ago, where I saw someone using VoiceOver on the iPhone for the first time. And her response is epic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">David It She was using voiceover, to operate her iPhone with the screen off, I mean, completely off.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Jenny Absolutely. The way that that individual who\u2019s blind is using her phone actually is a much more efficient use of the phone than someone who\u2019s sighted; the screen\u2019s off, the battery power lasts longer. And I\u2019ll tell you,if they\u2019re using a screen reader, they\u2019re listening to that sound at way above your normal speed. They can get through an audio book in half the time. I can sit in a room and understand what\u2019s being said, not by hearing any audio, but I can watch what\u2019s happening. I can understand to put the pieces together of what people are saying, and I don\u2019t have to be within two feet of someone. I\u2019m great at a party for that perspective.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">[music]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Disability\u2019s a strength.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We have strengths and we have expertise.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The one thing that we need to stop doing as a society is seeing disability with a sympathy lens.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So when you see a person with a disability, forget the \u201cdiverse abilities,\u201d&nbsp; \u201csuper ability,\u201d \u201cspecial ability.\u201d No, say the word \u201cdisability.\u201d We\u2019re proud of our identities. We\u2019re proud of who we are, and we\u2019re experts in who we are. And just not with sympathy, but with empathy.&nbsp;<\/p>\n<div class=\"powerpress_player\" id=\"powerpress_player_3937\"><audio class=\"wp-audio-shortcode\" id=\"audio-62-1\" preload=\"none\" style=\"width: 100%;\" controls=\"controls\"><source type=\"audio\/mpeg\" src=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3?_=1\" \/><a href=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3\">https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3<\/a><\/audio><\/div><p class=\"powerpress_links powerpress_links_mp3\" style=\"margin-bottom: 1px !important;\">Podcast: <a href=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3\" class=\"powerpress_link_pinw\" target=\"_blank\" title=\"Play in new window\" onclick=\"return powerpress_pinw('https:\/\/www.unsungscience.com\/?powerpress_pinw=62-podcast');\" rel=\"nofollow\">Play in new window<\/a> | <a href=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3\" class=\"powerpress_link_d\" title=\"Download\" rel=\"nofollow\" download=\"unsungscience-20211203.mp3\">Download<\/a><\/p>","protected":false},"excerpt":{"rendered":"<p>Your smartphone can see, hear, and speak\u2014even if you can\u2019t. So it occurred to the engineers at Apple and Microsoft: Can the phone be a talking companion for anyone with low vision, describing what it\u2019s seeing in the world around you?<span class=\"excerpt-more-link\"><a class=\"more-link\" href=\"https:\/\/www.unsungscience.com\/index.php\/2021\/12\/03\/how-apple-and-microsoft-built-the-seeing-eye-phone\/\">More <svg class=\"svg-icon\" width=\"24\" height=\"24\" aria-hidden=\"true\" role=\"img\" focusable=\"false\" viewBox=\"0 0 24 24\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M6.96954 10.2804L11.9999 15.3107L17.0302 10.2804L15.9695 9.21973L11.9999 13.1894L8.0302 9.21973L6.96954 10.2804Z\" fill=\"currentColor\"\/><\/svg><\/a><\/span><\/p>\n<div class=\"excerpt-audio-block\">\n<figure class=\"wp-block-audio\"><audio controls src=\"https:\/\/unsung.davidpogue.com\/wp-content\/uploads\/2023\/10\/unsungscience-20211203.mp3\"><\/audio><\/figure>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-62","post","type-post","status-publish","format-standard","hentry","category-uncategorized","entry"],"_links":{"self":[{"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/posts\/62","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/comments?post=62"}],"version-history":[{"count":2,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/posts\/62\/revisions"}],"predecessor-version":[{"id":65,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/posts\/62\/revisions\/65"}],"wp:attachment":[{"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/media?parent=62"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/categories?post=62"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.unsungscience.com\/index.php\/wp-json\/wp\/v2\/tags?post=62"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}