photo of two users' hands interacting with a touch-sensitive table using the Access Overlays tool to gain understanding of a large map accessible without sight

project areas

touchscreen accessibility

friendsourcing Q&A for the blind

image captioning

social tech for parents of children with special needs

social tech for autism

health info seeking

crowdworkers with disabilities

improving communication via AAC

accessible AR and VR

accessible photography

dyslexia and web search

screen reader experiences

sign language technologies

AI fairness and accessibility

accessibility & social technologies

overview

My research explores the relationship between accessibility and social technologies. In particular, I am interested in how interaction design can have a positive impact on people with disabilities by making it easier for them to connect with others in social and professional contexts. I am also interested in how interaction traces from social technologies (e.g., "big data") can provide insight into the activities, challenges, and successes surrounding technology use by people with disabilities, and/or provide early warning signs of health concerns. I manage the MSR Ability team at Microsoft Research, and am an affiliate member of MSR NExT's Enable group.

teaching & service

Together with Richard Ladner, I co-organized a joint University of Washington + Microsoft Research seminar series on the topic of "Computers & Disabilities" (CS590W) during the Fall 2010 academic quarter, and we also co-taught the same course at UW in Winter 2017. I served as a faculty panelist for the 2012 and 2017 ACM ASSETS doctoral symposium, and I was the local arrangements chair for the 2013 ASSETS conference. Richard Ladner and I co-organized the 2016 UW-MSR Summer Institute on the topic of Expanding Accessibility Research. I was the papers co-chair for the 2017 ASSETS conference, and the papers co-chair for the "Health, Accessibility, and Aging" subcommittee for CHI 2018. I was a co-organizer of ISMAR 2019 Mixed Reality & Accessibility Workshop, the CSCW 2019 Workshop on Addressing the Accessibliity of Social Media, and of the ASSETS 2019 Workshop on AI Fairness for People with Disabilities

research projects

Touchscreen Accessibility for the blind

Friendsourcing Q&A for the blind

  • survey of blind people's use of social networking sites, and experiment comparing their use of friendsourcing vs. crowdsourcing for Q&A [ CSCW 2013 paper ]
  • analysis of visual challenges facing blind people based on questions posted to a friend/crowdsourcing service [ CHI 2013 paper ]
  • proposing and studying the concept of social microvolunteering as a mechanism for friendsourcing on behalf of altruistic causes [ CHI 2015 paper ]

Image Captioning

  • study of blind Twitter users' needs regarding image captioning, and recommendations on how to make visual media in social media more accessible to blind users [ CHI 2016 paper ]
  • A study of how automated image captions in social media are interpreted by blind users and how to better support appropriate user skepticism of incorrect captions. [ CHI 2017 paper ]
  • A study of how human-in-the-loop AI can improve the utility of automatically-generated image captions for users with visual impairment. [ HCOMP 2017 paper and appendix ]
  • introducing proxemic audio interfaces as an interaction technique for immersive and varied rich description of artwork for people with visual impairment. [ IMWUT 2017 paper | Video ]
  • Caption Crawler: a system for automatically providing alt text for online images using reverse image search [ CHI 2018 paper | Video | Browser Extension ]
  • Rich Representations of Visual Content for Screen Reader Users: rich, interactive updates to traditional "alt text" representations [ CHI 2018 paper ]
  • An exploration of sonification techniques to non-visually represent imagery from social media [ Ergonomics in Design, 2019 ]
  • A study of the (under)utilization of alt text on Twitter [ WWW 2019 paper ]
  • Studies of how people who are blind's image description needs are impacted by the context in which the image is encountered [ CHI 2020 paper | ASSETS 2021 paper ]
  • Comparisons of the efficacy of new interfaces for alt text creation and editing in PowerPoint [ ASSETS 2021 paper ]

Social technologies for parents of children with special needs

  • study of Facebook and Twitter use by mothers of children aged 3 and under, including those with special needs/developmental delays [ CSCW 2014 paper ]
  • study of ways in which parents of children with special needs use social media, including suggestions for changes to social platforms to better support this demographic [ ICWSM 2014 paper ]

Social technologies for people on the autism spectrum

  • SIDES: a tabletop game for social skills development aimed at youth with Asperger's syndrome [ CSCW 2006 paper ]
  • SWARM: a wearable device to facilitate awareness of one's own or others' emotional states [ TEI 2015 paper | Video ]
  • Supporting Neurodiverse Software Engineers in Technical Workplace Settings [ ASSETS 2015 paper ]
  • Understanding the video calling technology needs of adults with autism [ CSCW 2019 paper ]
  • A study of how search engines and Twitter are used to seek and share health information, including disability issues [ CHI 2014 paper ]

Crowdsourcing employment and disability

  • A study indicating that people with disabilties view online crowdsourcing work as a potentially important source of employment, and identifying the technical obstacles to full participation in crowdwork by people with disabilities. [ CSCW 2015 paper ]
  • A study exploring senior citizens' understanding of and interest in crowd work, and reflecting on how crowd work systems might evolve to engage this audience. [ CHI 2016 paper ] [ HCOMP 2015 poster abstract ]
  • A framework for subcontracting microtasks that may benefit workers with disabilities. [ CHI 2017 paper ]

Improving communication via AAC devices

  • Reframing AAC as groupware and supplementing AAC with a mobile phone app (AACrobat) to facilitate communication partner awareness and engagement [ CSCW 2017 paper | Video ]
  • Interview study identifying limitations of current AAC devices for personality expression and presenting design guidelines and evaluation metrics for more expressive AAC [ CSCW 2017 paper ]
  • Expanding the expressiveness of AAC via supplementary "awareness displays" for conveying emotion and/or communication logistics. [ CHI 2017 paper | Video ]
  • Automating e-Tran board style input via a smartphone app for interpreting simple gaze gestures and translating them into text. [ CHI 2017 paper | Video | iOS App ]
  • Analyzing the limitations of eye gaze sensing for interactive applications and considering how applications can be re-designed to mitigate these challenges. [ CHI 2017 paper ]
  • Introducing the dynamic, cascading dwell technique for error-reduction and improved throughput in eye typing. [ CHI 2017 paper ]
  • Using computer vision to provide word and phrase suggestions to AAC systems that faciltiate discussing co-located objects. [ DIS 2017 paper ]
  • Voicesetting interfaces for improved expressivity in AAC via non-speech audio backchannel reactions, the use of emoji to trigger automated SSML markup for common emotional intonations, and a detailed eye-gaze operated prosody editor. [ CHI 2018 paper | Video ]
  • Dueto: an musical instrument for the eyes [ ASSETS 2019 Poster | Video ]

Accessible Augmented Reality and Virtual Reality

Accessible Photography

  • Understanding the accessibility needs of users with motor impairments. [ CHI 2018 paper ]
  • An investigation of the use of photo-capture and photo-sharing apps by low-vision teens, including Instagram and Snapchat. [ CHI 2018 paper ]
  • Understanding how people use web search as a linguistic support tool [ WWW 2017 paper ]
  • Interview, survey, and online study demonstrating the differential needs of dyslexic and non-dyslexic web searchers. [ CHI 2018 paper ]
  • Analysis of how web page features impact readability for users with dyslexia. [ SIGIR 2018 paper ]
  • Analysis of how web browser Reader View layouts impact readibility for users with and without dyslexia. [ CHI 2019 paper ]
  • An eye tracking study of web search by people with and without dyslexia [ SIGIR 2020 paper ]

Novel Screen Reader Interactions

    Bridging Screen Readers and Voice Assistants for Eyes-Free Web Search [ ASSETS 2019 paper | Video ]
  • Rich Representations of Visual Content for Screen Reader Users: rich, interactive updates to traditional "alt text" representations [ CHI 2018 paper | Video ]

Sign Language Technologies

  • Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective [ ASSETS 2019 paper ]

AI Fairness and Accessibility

  • Understanding the Representation and Representativeness of Age in AI Data Sets [ (arXiv pre-print coming soon) ]
  • Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs [ arXiv pre-print ]
  • Designing an Online Infrastructure for Collecting AI Data from People with Disabilities. [ FAccT 2021 paper ]
  • Sense and Accessibility: Understanding People with Physical Disabilities' Experiences with Sensing Systems. [ ASSETS 2020 paper ]
  • Toward Fairness in AI for People with Disabilities: A Research Roadmap [ ASSETS 2019 Workshop Paper ]
  • AI and Accessibility: A Discussion of Ethical Considerations [ CACM Viewpoint, June 2020 ]
  • Disability, Bias, and AI [ AI Now whitepaper ]
  • Radical AI Podcast - Ability and Accessibility in AI [ Podcast recording ]